And ditch insecure and weak TLS ciphers or risk attack

SSL, or TLS as it should be called these days, is THE de rigueur of modern web site hosting. Well, not so much de rigueur, but more of a necessity. It’s not just about security (encryption between your web browser and the webserver), but SEO (search engine optimisation) requires an SSL/TLS certificate as search engines such as Google are prioritising SSL/TLS protected sites above non-secure sites (see http://www.bafta.org, an example of a web site which could – and indeed should – be using an encryption connection throughout, but doesn’t).

And it’s not just all about encryption. With the HTTP/2 protocol – assuming your web server supports it – can provide a number of improvements that can significantly boost the performance of your site as well.

SSL/TLS certificates used to cost a fortune and were difficult to manage. Every year or so, you’d have to create a new certificate signing request (and private key, if necessary) and then submit the CSR to an SSL vendor. You’d then have to verify you own the domain either by placing a text file on your website, or an entry in DNS. And you’d be paying a pretty penny in the process. And that’s just to protect one URL (or, in the case of most SSL vendors – actually two – one for a subdomain (such as ‘www’), and the other for the bare domain (such as ‘drake.org.uk’). If you wanted to protect a whole bunch of subdomains, you could buy a wildcard SSL certificate. These are even more expensive (though the cheapest I found was $45 per year), but can be deployed across multiple servers and hostnames under the same domain.

Then came along Let’s Encrypt. It’s a free certificate authority that issues free single hostname and wildcard SSL certificates. It’s easily automated and requires very little effort. Wildcard SSL certificates are relatively new – and most people end up issuing single domain certificates through the “certbot” utility.

But it’s just as easy to get a wildcard cert which can be renewed automatically. Usually, like me, you’d run certbot with the –nginx command which sorts out your nginx configuration for you. But if you wanted a wildcard certificate instead, it requires a bit extra work:

certbot-auto certonly --manual --preferred-challenges=dns \
--email [email protected] \
--server https://acme-v02.api.letsencrypt.org/directory \
--agree-tos -d *.wombats-are-cool.com

You’ll then be prompted by certbot to add a DNS entry to your domain (in this example, wombats-are-cool.com) and then it’ll go off and verify it exists and issue the certificate. Keep your DNS TTL record for a quick resolution.

Once issued, you’d just alter your nginx server block with:

ssl_certificate /etc/letsencrypt/live/wombats-are-cool.com/fullchain.pem; # managed by Certbot

ssl_certificate_key /etc/letsencrypt/live/wombats-are-cool.com/privkey.pem; # managed by Certbot

Then shove the following in /etc/crontab:

0 0,12 * * * root python -c 'import random; import time; time.sleep(random.random() * 3600)' && /usr/local/bin/certbot-auto renew

(add > /dev/null 2>&1 to taste)

A free wildcard SSL certificate which will automatically renew itself. An alternative to Let’s Encrypt is to use a WAF or CDN such as Cloudflare or Sucuri – both will offer to install a certificate at the edge (e.g. their servers – all traffic will go through their datacentres before being passed to your origin server). This requires a bit more set-up, especially if you want to the WAF/CDN to connect over HTTPS to the origin server. There are a number of approaches to this – including, ironically, using Let’s Encrypt.

Now, don’t forget to disable SSLv3, TLS v1.0 and v1.1 and use strong ciphers. Don’t do what many web site owners do, and accept any old nonsense.

In the following example (from a well known UK multi-media facility), the highlighted protocols are terribly insecure and will fail you in any vulnerability scan, and a temptation for intruders and automated bots. TLS v1.1 isn’t worth keeping around – I’ve been looking at the stats of a very high volume e-commerce web site shows that barely anybody uses it. I’ve configured many web sites to use TLS v1.2 at a minimum and it has had absolutely no impact on browser compatibility.

PORT STATE SERVICE
443/tcp open https
| ssl-enum-ciphers:
|
SSLv3:
| ciphers:
| TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (dh 2048) - C
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 2048) - A
| TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (secp256r1) - C
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_RC4_128_SHA (secp256r1) - C
| TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
| compressors:
| NULL
| cipher preference: client
| warnings:
| 64-bit block cipher 3DES vulnerable to SWEET32 attack
| Broken cipher RC4 is deprecated by RFC 7465
| CBC-mode cipher in SSLv3 (CVE-2014-3566)
|
TLSv1.0:
| ciphers:
| TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (dh 2048) - C
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 2048) - A
| TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (secp256r1) - C
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_RC4_128_SHA (secp256r1) - C
| TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
| compressors:
| NULL
| cipher preference: client
| warnings:
| 64-bit block cipher 3DES vulnerable to SWEET32 attack
| Broken cipher RC4 is deprecated by RFC 7465
|
TLSv1.1:
| ciphers:
| TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (dh 2048) - C
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 2048) - A
| TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (secp256r1) - C
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_RC4_128_SHA (secp256r1) - C
| TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
| compressors:
| NULL
| cipher preference: client
| warnings:
| 64-bit block cipher 3DES vulnerable to SWEET32 attack
| Broken cipher RC4 is deprecated by RFC 7465
| TLSv1.2:
| ciphers:
| TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (dh 2048) - C
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 2048) - A
| TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 2048) - A
| TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 2048) - A
| TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 2048) - A
| TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 2048) - A
| TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (secp256r1) - C
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_RC4_128_SHA (secp256r1) - C
| TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
| TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
| TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
| TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
| compressors:
| NULL
| cipher preference: client
| warnings:
| 64-bit block cipher 3DES vulnerable to SWEET32 attack
| Broken cipher RC4 is deprecated by RFC 7465
|_ least strength: C

Or a more visual representation of the above:

Exposing the versions of your server’s web server, OpenSSL and PHP is also a Bad Thing(tm). Which of course, our poor saps absolutely do:

Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.6.30

Don’t do what these people do. Pay attention to your SSL/TLS settings as well as your certificate.

Meanwhile, I’m happy with this:

You’ll have seen the adverts on TV. Well, I did too. And I thought – have they possibly changed in the few decades I’ve known them? They’ve always been in the back of mind – but not in a good way (especially when it came to domains). Has the rebranding done any good?

Bargain Hunt

I like a bargain as much as anybody else does, and although I’ve been very happy with DigitalOcean, 1&1 IONOS’ VPS service for £1.20/month for 6 months before another 6 months of £24/month seemed quite reasonable for the specifications on offer (4 vCPU, 8Gb RAM and 160Gb SSD).

I know my own address, thanks..

So I signed up early last week. The first thing that drove me insane was their postcode/address lookup function when entering your address as a new customer. I have constant problems with postcode databases not getting my address properly and 1&1 are no different. After entering my postcode, the system told me my address was wrong and I couldn’t move forward with completing the registration form unless I accepted their version of my address (which is wrong). So I just accepted it. When it came to payment, a similar problem, but the system seemed to accept it and was charged £1.20 just fine.

It wasn’t until later the following day I received the account set-up confirmation email and I proceeded to log in and start getting things set-up. The very first to do was to lock down the server so that only I could connect to it from my home and work IP addresses for the purposes of SSH access (command line access). 1&1 IONOS comes with a firewall, so I started to configure it. As I also use CloudFlare for caching, WAF and firewall, I started to configure the IONOS firewall for that – though I note that the documentation for the firewall doesn’t mention you can use CIDR notation for the allowed IPs. The web form will accept them though! According to the official firewall docs, you can specify a range of IPs with a dash, but since CIDR is a perfectly normal and standard notation for IP ranges, I’d try that (it saves typing). After a while (as CloudFlare has a fair number of IP ranges), everything looked set to go. CloudFlare’s servers were the only ones that could connect to TCP port 443.

Let me explain how CloudFlare works, as you’ll find that neither 1&1 IONOS engineers or my “personal consultant” understand how systems like CloudFlare or Akamai work (I’ve been using CloudFlare for at least 7 or 8 years, and Akamai for 2).

How does CloudFlare work?

When you request a page from my blog, the request goes to CloudFlare. CloudFlare does a few security checks first of all, then, if you’re not a naughty bot or person, it checks to see if the page already exists in its cache. If not, CloudFlare – and ONLY CloudFlare – will connect to my VPS securely to retrieve the page and serve it to you. You, as a requester cannot bypass CloudFlare to get to my VPS directly unless I specifically disable proxying within CloudFlare (my DNS is hosted with CloudFlare so any changes I make should be almost immediate).

Too hot to trot?

When I set-up the 1&1 IONOS VPS server, it took me about 30 minutes to get everything running including moving everything off DigitalOcean and installing MySQL, PHP and nginx. I’ve written scripts which perform much of the set-up for me – and everything is checked into BitBucket so that I can retrieve those scripts at any time from anywhere. I also have many backups at Backblaze B2, courtesy of rclone (written and maintained by my former boss at Memset Hosting Ltd.)

ALAS!

CloudFlare could not talk to the 1&1 IONOS VPS. Connection timed out every time. I set-up a firewall rule to allow myself direct access to the VPS via port 443) to test that the LNMP stack was working correctly. It was. Output from netstat showed everything was fine. No local firewall was running, and iptables rules were clear and set to accept. And yes, I had changed the IP addresses in CloudFlare’ DNS to the new shiny VPS.

How about you try turning it on and off again?

So I utilised 1&1 IONOS’ live chat system for technical support. They’re fast, but they wanted to know why I was locking off port 443 to specific IPs. I explained I was using CloudFlare. I checked with them if the syntax of the firewall rules were correct. Apparently, they were. Their advice? Open port 443 to the world. I asked them if they had any experience with CloudFlare or Akamai or any other similar service. The whole point with these systems is that it acts as a barrier between the internet at large and your origin servers. The origins which host your application should never be exposed externally but only through CloudFlare, Akamai or whoever.

So I called my “personal consultant” for help by submitting a request for a callback. Within a minute or two I was connected. I explained the problem to him and he went away and spoke to the technical people. Their explanation was how CloudFlare was returning client IPs. Which is absolute bull. See my explanation further above. The connecting IPs are the ones that I defined in the firewall. The same IPs I had been using at DigitalOcean with their firewall. Client IPs that come in are passed to the original server in the form of a header (and my nginx configuration looks at that header and parses the real IP which is then available in the server logs) – but that’s got nothing whatsoever to do with CloudFlare’s servers connecting to my VPS.

No experienced sysadmin should touch 1&1 with a bargepole..

While I was still talking to the chap on the phone, the blog suddenly spluttered into life. But it is not obvious why. The firewall rules didn’t look to have changed. But still, I didn’t like the explanation whatsoever from their technical department as to how CloudFlare operates and the encouragement of opening TCP port 443 to the world. I had to explain that I’m a systems administrator of some 22 years, having worked for two Academy Award-winning VFX companies, and now help manage multi-million-pound websites for some very high profile clients and have extensive experience with CloudFlare. So I cancelled the account there and then.

ALAS!

I was transferred to the US division of 1&1 for cancellation. But after 10 minutes or so, I was put through to the right person who cancelled the account for me. And I received this email:

Every. Single. Image. Broken.

SIGH.

I’ve gone back to DigitalOcean again (~£10/month for third of the resources). But in order to test my DR (disaster recovery) plan, wiped the old server, set-up a new one, and restored everything from my Backblaze B2 backups. It all works perfectly.

CloudFlare had no problem connecting to my new VPS at DigitalOcean. New IP and everything. That’s how 1&1 IONOS should have worked out of the box. I blame their firewall and their documentation. And possibly lack of experience of IDS/WAF/CDN systems such as CloudFlare.