UCC Wheel Meeting on 18th April 2020
Wheel Meeting Minutes - Saturday 2020-04-18 14:00
Meeting opened 14:06
- [[email protected]]
Schedule next meeting
- Schedule/delegate reminders of next meeting
- [TPG]: Monthly is working well
- [BOB]: Saturday May 23rd suggestion for next meeting
- Scheduled for 2020-05-24 14:00
Standing items (brief)
Status Check: Regular Updates
- eg. Debian Oldstable 9 "Stretch" --> Debian Stable 10 "Buster"
- Discord-irc.ucc.asn.au could use a rebuild, it's Jessie
- OcsInventory, uccmonitor (see https://wiki.ucc.asn.au/MissionControl) for an overview
- Out-of-date servers:
- meersau (stretch)
- discord-irc (jessie)
- gitlab ()
- unisfa-koha (stretch)
- Contact Blair, Chase, or Felix from UniSFA to arrange downtime
- salmon (? container, hard to upgrade)
- mooneye (working on it)
- Dead disk, needs replacing or new machine STAT
- mussel (stretch)
- murasoi (stretch)
- motsugo (stretch)
- Tuesday 21 Apr upgrade event - "Let's Break UCC"
Status Check: Password/Key Rotations
Status Check: Backups
- [NTU]: We should have done that offsite file-restore demo
- Communal decision to remove off mollitz webcam-archives
- Full disks
- [TEC]: If we can clean the old warnings, include webcamrestore, we can get timely new uccmonitor alerts
- Dead disks
- Mollitz (Dell 2950) can't have disks larger than 2TiB, and is old, so we should look at replacing
- Other options:
- Motsugo could be replaced
- Old Motsugo coule become the new backup server
- Worst case: could deploy the Cisco UCS220 or Mudkip
- Mudkip has plenty of space drive caddies
- Cisco for user server?
- ACTION: Get Cisco server to a workable state
- [BRD] allocating more memory to minecraft2019
- [RME] & : Migrate to magikarp?
- [NTU]: Save to molmol NFS or Ceph? (fast to migrate); or
- Save to magikarp local Optane dm-cache? (fast to access)
- : minecraft2019:/, currently hosted on loveday, is 128GB? Could be shrunk? Will take a while to migrate
- Run MineOS and a bunch of minecraft servers: maybe 128GB is right-sized?
- Check with [BRD] & [MDD] about the rootfs shrinking?
- ACTION:  to migrate and look at shrinking system disk if necessary
-  Discuss current security of remote management, and whether we need to lock down the 192.168.2.0 subnet further (ie. no access from motsugo)
- eg. some management controllers are positively ancient and are unpatched
- See this article for an example of a nasty bug (hint: we have 3 HPs):
- Wheel-only jumpbox or VPN? Log traffic to/from that subnet?
- ipmiview tool is handy? (By SuperMicro)
- ssh in to some of them - which ones to be decided. Probably runs dropbear
- ACTION: [MTL] will spin up a jumpbnox VM for testing
- ...but a physical host is better than a V<
- [MTL] mailauesi - imaps, pop3s, submission: currently wheel-only logins
- Live testing: imap works, smtp doesn't?
- Submission replaces smtp
- Deprecate secure.ucc.asn.au for people's mail clients
- [MTL] maculatus - new home VM for flame
- libc6:i386 installed, old flame MudOS driver runs
- listenes via firewall rules to telnet port 4242, and many other ports (like ssh.ucc.asn.au)
- possibly no one uses most of those anymore?
- [MTL] mailfish - new home for mail services from mooneye and mailman
- postfix off switch, similar to postgrey, stop accepting deliveries if AD or NFS is down
- Remaining stuff on mooneye - DNS (rearchitect? [MTL], see below)
- [MTL] [email protected]
- Rolled out on Clubroom Desktops
- Ranked 11,724 of 251,202
- Windows Clubroom machines?
- Try desktop: Christmas
- WSL+OpenSSHd -> running chocolately
- can rdesktop in via guacamole/maaxen?
- ACTION: [MTL] to do above.
- [TEC]: GPU Server
- Bob raised on Facebook group: https://www.facebook.com/universitycomputerclub/permalink/3767237643318123
- [DBA] David Adams, Cursor XYZ
- [NTU] Proxmox Cluster Upgrades and Updates
- [CFE] We have a virtual cluster we can test upgrading Proxmox PVE v5 -> v6
[email protected]:~# pveversion pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-24-pve)
- [MPT] 4G backup uplink
- Set up: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2020-April/005323.html
- Can be used for outgoing with source-based policy routing
- Has CGNAT IPv4 /64 address?
- Probably don't want to fail2ban on that if it all comes from the one IP address
- Needs a VPN?
- ACTION: [MSH] proof-of-concept on murasoi?
Cloudflare! All! The! Things! ?!?!?!?
- Still Priority #1
- [MTL] UCC webservers, etc (was: mussel)_
- More SSL expiries coming in June
- If only we can make a CNAME change to ucc.gu.uwa.edu.au?
- letsencrypt might see our secondaries?
- ACTION: [NTU][MTL] try it out and extend our expiries
- Set up offsite ucc.asn.au nameserver
- We've been using and developing iodine
- [MPT] has moved his domains to Cloudflare Free tier
- Using dynamic DNS, via Cloudflare API: token based, not ddclient?
Matters Arising Previously
ACTION: Annual Account Locking - if it has not happened by now, it's overdue
- Are the rejoining member/password reset/new member account procedures going OK?
- Committee: Make it an event!
- Account Locking Stream in Charity UnVigil 2020-05-13 through 2020-05-16
ACTION:  to document remote management options for our critical servers
- Initial document in place, further testing needs to be done
- More details to be added
- See /home/wheel/docs/RemoteManagement.org
ACTION:  to sort out iDRAC for Mooneye as a priority
- Live demo for mudkip! HP iLO
- vsp: Virtual serial port
- systemctl enable [email protected]
- power reset
- Mail server from scratch on it, and a point a ucc subdomain at it, by the end of this weekend (2020-03-22)
- : Played with EC2 instance, set up a DNS A record for it, and ran push.sh
- Backed up mooneye:/etc/bind/domains to https://gitlab.ucc.asn.au/ucc-systems/ucc-domains.git, cloned it onto cloud-mooneye
- Copied the named.config.local across
- Commented out mooneye-specific parts (like LetsEncrypt stuff and other referenced secrets only on mooneye)
- Tried running zonemake.py, only to find that it needed a package installed
- Once package was installed, it still wouldn't work due to symlinks to more stuff on mooneye (like /home/other/www/members.conf)
-  work more independently gitlab vs zonemake.py vs AD getent
- [MTL] [NTU] separate authoritative and recursive - DJB was right!
- [NTU] has been testing knot-DNS
ACTION: [THA][TEC] to buy 1TB SSDs for magikarp + mudkip
- Passed by committee on 2020-10-04.txt
- Austin Computers: ~$500 Each? MLC, not SLC/QLC https://.www.austin.net.au
- But the MLC is a lie!
Meeting closed 17:04