Debian 10 (aka Buster) doing its thing

July 26th is Systems Administrator Appreciation Day! A day where office workers everywhere should be bringing in delicious treats for the IT department to ensure goodwill between techie and Luddite remains in full force.

I’m not quite sure why I wanted to be a systems administrator some 22-odd years ago, but here I am – still a systems administrator. I suppose it all started when I started visiting my dad at work and being fascinated by the telex machines and computer systems, including the multi-user DEC systems they had.

SysAdmin Man Begins – taken from my very first C.V.

After leaving university I started building PCs for a local company in Norwich, then set-up and managed Linux and Windows servers for the same company when they became an ISP. That job was a jack of all trades and also included writing software to configure the TCP/IP stack for dial-up for Windows machines, web design, and technical support.

After a few more years in the ISP industry, I went to work for The Moving Picture Company (MPC) in the film and television industry, sysadminning the infrastructure for high-end visual effects for major movies and TV shows. After that, my first taste of systems engineering in a software development firm that specialised in VFX software, before moving on back to the ISP/web hosting industry for 9 years.

Now I work in e-commerce and handle corporate infrastructure as well as that of client websites. All the years of experience from the above come into play. It’s been an interesting journey so far. Not sure what else fate has in store for me, but I’m sure I’ll be a sysadmin until the day I die.

Fellow sysadmins, I salute you.

You’ll have seen the adverts on TV. Well, I did too. And I thought – have they possibly changed in the few decades I’ve known them? They’ve always been in the back of mind – but not in a good way (especially when it came to domains). Has the rebranding done any good?

Bargain Hunt

I like a bargain as much as anybody else does, and although I’ve been very happy with DigitalOcean, 1&1 IONOS’ VPS service for £1.20/month for 6 months before another 6 months of £24/month seemed quite reasonable for the specifications on offer (4 vCPU, 8Gb RAM and 160Gb SSD).

I know my own address, thanks..

So I signed up early last week. The first thing that drove me insane was their postcode/address lookup function when entering your address as a new customer. I have constant problems with postcode databases not getting my address properly and 1&1 are no different. After entering my postcode, the system told me my address was wrong and I couldn’t move forward with completing the registration form unless I accepted their version of my address (which is wrong). So I just accepted it. When it came to payment, a similar problem, but the system seemed to accept it and was charged £1.20 just fine.

It wasn’t until later the following day I received the account set-up confirmation email and I proceeded to log in and start getting things set-up. The very first to do was to lock down the server so that only I could connect to it from my home and work IP addresses for the purposes of SSH access (command line access). 1&1 IONOS comes with a firewall, so I started to configure it. As I also use CloudFlare for caching, WAF and firewall, I started to configure the IONOS firewall for that – though I note that the documentation for the firewall doesn’t mention you can use CIDR notation for the allowed IPs. The web form will accept them though! According to the official firewall docs, you can specify a range of IPs with a dash, but since CIDR is a perfectly normal and standard notation for IP ranges, I’d try that (it saves typing). After a while (as CloudFlare has a fair number of IP ranges), everything looked set to go. CloudFlare’s servers were the only ones that could connect to TCP port 443.

Let me explain how CloudFlare works, as you’ll find that neither 1&1 IONOS engineers or my “personal consultant” understand how systems like CloudFlare or Akamai work (I’ve been using CloudFlare for at least 7 or 8 years, and Akamai for 2).

How does CloudFlare work?

When you request a page from my blog, the request goes to CloudFlare. CloudFlare does a few security checks first of all, then, if you’re not a naughty bot or person, it checks to see if the page already exists in its cache. If not, CloudFlare – and ONLY CloudFlare – will connect to my VPS securely to retrieve the page and serve it to you. You, as a requester cannot bypass CloudFlare to get to my VPS directly unless I specifically disable proxying within CloudFlare (my DNS is hosted with CloudFlare so any changes I make should be almost immediate).

Too hot to trot?

When I set-up the 1&1 IONOS VPS server, it took me about 30 minutes to get everything running including moving everything off DigitalOcean and installing MySQL, PHP and nginx. I’ve written scripts which perform much of the set-up for me – and everything is checked into BitBucket so that I can retrieve those scripts at any time from anywhere. I also have many backups at Backblaze B2, courtesy of rclone (written and maintained by my former boss at Memset Hosting Ltd.)

ALAS!

CloudFlare could not talk to the 1&1 IONOS VPS. Connection timed out every time. I set-up a firewall rule to allow myself direct access to the VPS via port 443) to test that the LNMP stack was working correctly. It was. Output from netstat showed everything was fine. No local firewall was running, and iptables rules were clear and set to accept. And yes, I had changed the IP addresses in CloudFlare’ DNS to the new shiny VPS.

How about you try turning it on and off again?

So I utilised 1&1 IONOS’ live chat system for technical support. They’re fast, but they wanted to know why I was locking off port 443 to specific IPs. I explained I was using CloudFlare. I checked with them if the syntax of the firewall rules were correct. Apparently, they were. Their advice? Open port 443 to the world. I asked them if they had any experience with CloudFlare or Akamai or any other similar service. The whole point with these systems is that it acts as a barrier between the internet at large and your origin servers. The origins which host your application should never be exposed externally but only through CloudFlare, Akamai or whoever.

So I called my “personal consultant” for help by submitting a request for a callback. Within a minute or two I was connected. I explained the problem to him and he went away and spoke to the technical people. Their explanation was how CloudFlare was returning client IPs. Which is absolute bull. See my explanation further above. The connecting IPs are the ones that I defined in the firewall. The same IPs I had been using at DigitalOcean with their firewall. Client IPs that come in are passed to the original server in the form of a header (and my nginx configuration looks at that header and parses the real IP which is then available in the server logs) – but that’s got nothing whatsoever to do with CloudFlare’s servers connecting to my VPS.

No experienced sysadmin should touch 1&1 with a bargepole..

While I was still talking to the chap on the phone, the blog suddenly spluttered into life. But it is not obvious why. The firewall rules didn’t look to have changed. But still, I didn’t like the explanation whatsoever from their technical department as to how CloudFlare operates and the encouragement of opening TCP port 443 to the world. I had to explain that I’m a systems administrator of some 22 years, having worked for two Academy Award-winning VFX companies, and now help manage multi-million-pound websites for some very high profile clients and have extensive experience with CloudFlare. So I cancelled the account there and then.

ALAS!

I was transferred to the US division of 1&1 for cancellation. But after 10 minutes or so, I was put through to the right person who cancelled the account for me. And I received this email:

Every. Single. Image. Broken.

SIGH.

I’ve gone back to DigitalOcean again (~£10/month for third of the resources). But in order to test my DR (disaster recovery) plan, wiped the old server, set-up a new one, and restored everything from my Backblaze B2 backups. It all works perfectly.

CloudFlare had no problem connecting to my new VPS at DigitalOcean. New IP and everything. That’s how 1&1 IONOS should have worked out of the box. I blame their firewall and their documentation. And possibly lack of experience of IDS/WAF/CDN systems such as CloudFlare.

I gave up on WordPress.com because I felt I wasn’t getting value for money. Unless I forked out more money than I’m paying now – and annually upfront no less – there was no Google Analytics access and I disliked having to give up the ‘www’ subdomain. Then there are other technical matters which just couldn’t cut the mustard. So I’ve gone back to using CloudFlare, a CDN (content delivery network) and WAF (web application firewall), which sits in front of my VPS (virtual private server) to protect the server and WordPress application. As an added bonus, I was able to enable DNSSEC too.

However, one of the problems I have had with CloudFlare in the past is making it play nicely with WordPress.com’s Jetpack plugin. This provides additional features which are nice to have, but more importantly, allows me to use the WordPress iOS app to create and edit posts on the fly. Very handy if I have my iPad Pro with me and have the urge to write a blog post.

One of my favourite (relatively) new features of CloudFlare is the Firewall. This allows anybody to create a series of rules which grants or denies access to the underlying application. This is a big step up from the simple whitelisting/blacklisting feature which was very limiting and as a simple $20/month Pro subscriber didn’t allow me to block entire countries (a few of which are almost always entirely responsible for attacks and dodgy bots).

To get the Jetpack plugin to work properly, I had to create a brand new rule to allow a series of IPs from Automattic (who make WordPress) to access the blog.

CloudFlare’s new firewall editor is a big step up from simple whitelisting/blacklisting

The rules page is very simple:

We allow access only from WordPress.com IPs & two URLs

The /?rest_route= URI was a result of examining the output of the firewall logs. I’ve not seen any other calls from WordPress.com using that URI as yet (but then again, I haven’t used it in anger fully as yet), so it might not be necessary. But certainly, WordPress.com will use xmlrpc.php.

It works!

Prior to this, whenever I tried to associate Jetpack with WordPress.com, it would fail authentication, refresh the page, seemingly authenticate and that would be it. Nothing else would work. By applying the above firewall rule has made everything work as it should.