UCC Wheel Meeting on 18th April 2020

Wheel Meeting Minutes - Saturday 2020-04-18 14:00

VENUE: https://meetings.ucc.asn.au/

Meeting opened 14:06

Attendance

• Present
• [TPG]
• [BRD]
• [BOB]
• [CFE]
• [333]
• [MPT]
• [MTL]
• [NTU]
• [SJH]
• [LE@]

Apologies

• None

Absent

• None

Schedule next meeting

• Schedule/delegate reminders of next meeting
• [TPG]: Monthly is working well
• [BOB]: Saturday May 23rd suggestion for next meeting
• Scheduled for 2020-05-24 14:00

Standing items (brief)

SWS

• Done

Status Check: Regular Updates

• eg. Debian Oldstable 9 "Stretch" --> Debian Stable 10 "Buster"
• Discord-irc.ucc.asn.au could use a rebuild, it's Jessie
• OcsInventory, uccmonitor (see https://wiki.ucc.asn.au/MissionControl) for an overview
• Out-of-date servers:
  • meersau (stretch)
  • discord-irc (jessie)
  • gitlab ()
  • unisfa-koha (stretch)
  • Contact Blair, Chase, or Felix from UniSFA to arrange downtime
  • salmon (? container, hard to upgrade)
  • mooneye (working on it)
  • Dead disk, needs replacing or new machine STAT
  • mussel (stretch)
  • murasoi (stretch)
  • motsugo (stretch)
• Tuesday 21 Apr upgrade event - "Let's Break UCC"

Status Check: Password/Key Rotations

• DEFER
• https://en.wikipedia.org/wiki/Pro_re_nata

Status Check: Backups

• [NTU]: We should have done that offsite file-restore demo
• Communal decision to remove off mollitz webcam-archives
• Full disks
  • [TEC]: If we can clean the old warnings, include webcamrestore, we can get timely new uccmonitor alerts
• Dead disks
• Mollitz (Dell 2950) can't have disks larger than 2TiB, and is old, so we should look at replacing
• Other options:
  • Motsugo could be replaced
  • Old Motsugo coule become the new backup server
  • Worst case: could deploy the Cisco UCS220 or Mudkip
  • Mudkip has plenty of space drive caddies
  • Cisco for user server?
• ACTION: Get Cisco server to a workable state

New Matters

• [BRD] allocating more memory to minecraft2019
• [RME] & [333]: Migrate to magikarp?
  • [NTU]: Save to molmol NFS or Ceph? (fast to migrate); or
  • Save to magikarp local Optane dm-cache? (fast to access)
• [333]: minecraft2019:/, currently hosted on loveday, is 128GB? Could be shrunk? Will take a while to migrate
  • Run MineOS and a bunch of minecraft servers: maybe 128GB is right-sized?
  • Check with [BRD] & [MDD] about the rootfs shrinking?
• ACTION: [333] to migrate and look at shrinking system disk if necessary
• [333] Discuss current security of remote management, and whether we need to lock down the 192.168.2.0 subnet further (ie. no access from motsugo)
  • eg. some management controllers are positively ancient and are unpatched
• See this article for an example of a nasty bug (hint: we have 3 HPs):
  • https://it.slashdot.org/story/19/07/08/007202/critical-bug-last-year-allowed-bypassing-authentication-on-hpe-ilo4-servers-with-29-a-characters
• Wheel-only jumpbox or VPN? Log traffic to/from that subnet?
• ipmiview tool is handy? (By SuperMicro)
• ssh in to some of them - which ones to be decided. Probably runs dropbear
• ACTION: [MTL] will spin up a jumpbnox VM for testing
  • ...but a physical host is better than a V<
• [MTL] mailauesi - imaps, pop3s, submission: currently wheel-only logins
• Live testing: imap works, smtp doesn't?
  • Submission replaces smtp
• Deprecate secure.ucc.asn.au for people's mail clients
• https:gitlab.ucc.asn.au/ucc-systems-ucc-ansible-soe
• [MTL] maculatus - new home VM for flame
• libc6:i386 installed, old flame MudOS driver runs
• listenes via firewall rules to telnet port 4242, and many other ports (like ssh.ucc.asn.au)
• possibly no one uses most of those anymore?
• webtty
• [MTL] mailfish - new home for mail services from mooneye and mailman
• postfix off switch, similar to postgrey, stop accepting deliveries if AD or NFS is down
• Remaining stuff on mooneye - DNS (rearchitect? [MTL], see below)
• [MTL] Folding@Home
  • Rolled out on Clubroom Desktops
  • https://stats.foldingathome.org/team/261452
  • Ranked 11,724 of 251,202
• Windows Clubroom machines?
• Try desktop: Christmas
• WSL+OpenSSHd -> running chocolately
• can rdesktop in via guacamole/maaxen?
• ACTION: [MTL] to do above.
• [TEC]: GPU Server
  • Bob raised on Facebook group: https://www.facebook.com/universitycomputerclub/permalink/3767237643318123
  • [DBA] David Adams, Cursor XYZ
• [NTU] Proxmox Cluster Upgrades and Updates
• [CFE] We have a virtual cluster we can test upgrading Proxmox PVE v5 -> v6

root@magikarp:~# pveversion
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-24-pve)

• [MPT] 4G backup uplink

• Set up: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2020-April/005323.html

• Can be used for outgoing with source-based policy routing

• Has CGNAT IPv4 /64 address?

• Probably don't want to fail2ban on that if it all comes from the one IP address

• Needs a VPN?

• Wireguard?

• ACTION: [MSH] proof-of-concept on murasoi?

• TINC?

• Cloudflare! All! The! Things! ?!?!?!?

• Still Priority #1

• [MTL] UCC webservers, etc (was: mussel)_

• More SSL expiries coming in June

• If only we can make a CNAME change to ucc.gu.uwa.edu.au?

• letsencrypt might see our secondaries?

• ACTION: [NTU][MTL][333] try it out and extend our expiries

• Set up offsite ucc.asn.au nameserver

• We've been using and developing iodine

• [MPT] has moved his domains to Cloudflare Free tier

  • Using dynamic DNS, via Cloudflare API: token based, not ddclient?

Matters Arising Previously

ACTION: Annual Account Locking - if it has not happened by now, it's overdue

• Are the rejoining member/password reset/new member account procedures going OK?
  • Committee: Make it an event!
  • Account Locking Stream in Charity UnVigil 2020-05-13 through 2020-05-16

ACTION: [333] to document remote management options for our critical servers

• Initial document in place, further testing needs to be done
• More details to be added
• See /home/wheel/docs/RemoteManagement.org

ACTION: [333] to sort out iDRAC for Mooneye as a priority

• Live demo for mudkip! HP iLO
  • ssh
  • vsp: Virtual serial port
  • systemctl enable getty@ttyS1
  • power reset
• Mail server from scratch on it, and a point a ucc subdomain at it, by the end of this weekend (2020-03-22)
• [333]: Played with EC2 instance, set up a DNS A record for it, and ran push.sh
• Backed up mooneye:/etc/bind/domains to https://gitlab.ucc.asn.au/ucc-systems/ucc-domains.git, cloned it onto cloud-mooneye
• Copied the named.config.local across
• Commented out mooneye-specific parts (like LetsEncrypt stuff and other referenced secrets only on mooneye)
• Tried running zonemake.py, only to find that it needed a package installed
• Once package was installed, it still wouldn't work due to symlinks to more stuff on mooneye (like /home/other/www/members.conf)
• [333] work more independently gitlab vs zonemake.py vs AD getent
• [MTL] [NTU] separate authoritative and recursive - DJB was right!
• [NTU] has been testing knot-DNS

ACTION: [333][THA][TEC] to buy 1TB SSDs for magikarp + mudkip

• Passed by committee on 2020-10-04.txt
• Austin Computers: ~$500 Each? MLC, not SLC/QLC https://.www.austin.net.au
• Samsung-mz-76q1t0-2-5-1000-gb-serial-ata-iii-v-nand-mlc.html
• But the MLC is a lie!

Meeting closed 17:04