A Man-in-the-Middle Attack

I host my own instance of miniflux, an RSS reader. I do it as a hobby, I enjoy the learning opportunities that come along the way.

One such opportunity presented itself in November 2023.

Back then, a Man-in-the-Middle attack was reported against jabber.ru. You can go read the full details on that blog post, but let’s go over its main aspects. Without the attacker, a client connects directly to the jabber.ru server over TLS:

XMPPclientconTnLeSctionjabber.ruserver

TLS encrypts the connection. A certificate is also presented when the connection is established, to ensure authenticity and integrity. For this to work, the Certificate Authority (CA), a trusted third party, signs the certificate. In doing so, it affirms that it saw a given certificate associated with a particular domain1. This protects against some Man-in-the-Middle attacks, where an attacker inserts itself between a client (say on a public wifi network) and the website, as in the next diagram. If that attacker does not manage to similarly be between the CA and the website, then it will be unable to decrypt or alter the exchange without being detected, because it can’t present a certificate signed by the CA.

Nonetheless in November 2023, a Man-in-the-Middle attack against jabber.ru succeeded. In that instance, the attacker leveraged a privileged position on the network. They intercepted all TLS traffic to the victim’s server, then tricked2 the Let’s Encrypt CA into issuing a certificate for the attacker server. Then, they could decrypt the traffic from the client, re-encrypt it, potentially alter it and send it to the server.

XMPPclientconTnLeSctionattackerservercnoenwneTcLtSionjabber.ruserver

There was no need to compromise the server for this attack to work. It simply leveraged a privileged network position.

In a reply on his blog, ACME developer Hugo Landau points out the following mitigation:

Mitigation. The second area of consideration is mitigation, in which the unauthorized issuance of TLS certificates is prevented from happening in the first place. The entire point of a TLS certificate is, of course, to prevent a man-in-the-middle attack. The fundamental problem here is that the “Domain Validation” model by which CAs validate control of a domain name is ironically itself vulnerable to man-in-the-middle attacks, especially if an attacker can intercept not just some but all traffic to a victim site (as happened in this case).

Some years ago I authored ACME-CAA (RFC 8657), now implemented by Let’s Encrypt, which can mitigate this in some circumstances. The basic idea is that you can configure a DNS record which specifies that only a specific account of a specific CA is authorised to issue certificates for a domain. Thus simply using the same CA isn’t enough; you must gain access to the same account at the CA. With Let’s Encrypt, this means gaining access to the ACME private key used to request a certificate. Based on what we know about the attack, it would have been prevented by deploying this extension.

https://www.devever.net/~hl/xmpp-incident#narrow

This is just one mitigation strategy and it has caveats. I encourage you to go read the rest of the post, where Hugo also explain how to detect the attack.

This got me thinking: how would I implement this mitigation for my self-hosted services? In this post, we will use the CAA DNS record to limit issuance to one particular account registered at one particular CA, as suggested above.

The Setup

TLS Proxy

My setup looks like this:

WebbrowsercoTnLnSectioncaddryemsoetrevesrervers(uonnciikcxektnaomtehmeOirlniisvfeilrnuvexi)ces...

Here caddy acts as a TLS proxy. It manages the TLS certificate, requesting a new one from Let’s Encrypt as needed. It also terminates the TLS connection from the browser and then connects to the underlying service, like miniflux. This example could of course be generalized to any service proxied by caddy, in particular services handling more personal data, like a self-hosted webmail or contact server.

ℹ️ Note

Here, we do not have a CDN in front of our site.
The whole point is to restrict the certificates issued to a particular Let’s Encrypt account that we control. A CDN terminates the TLS connection (usually), so it needs a TLS certificates for the domain.

In this context, the server is presumably in one’s house or in a datacenter nearby. There is thus less of a need for an intermediary cache that would be closer to the user.

DNS

The domain hosting the RSS reader, r.cj.rs, is in fact a CNAME record pointing to the underlying machine hosting the various services, Olivine. It looks like this:

dog r.cj.rs

CNAME r.cj.rs.             5m00s   "olivine.joly.eu.org."
    A olivine.joly.eu.org. 5m00s   132.145.68.59

CAA Records

Going back to the mitigation we want to implement:

The basic idea is that you can configure a DNS record which specifies that only a specific account of a specific CA is authorised to issue certificates for a domain.

https://www.devever.net/~hl/xmpp-incident#narrow

Let’s Encrypt provides some documentation for the CAA record (which stands for Certification Authority Authorization). The key points are:

  • CAA records closest to the domain take priority: a CAA on r.cj.rs takes precedence over one on cj.rs.
  • CAA follows CNAME redirects.
  • The issue and issuewild properties control the certificate authority allowed. If only issue is present, then the same constraints apply to both normal and wildcard certificates.
  • The accounturi parameter can restrict issuance to a particular account, like https://acme-v02.api.letsencrypt.org/acme/acct/1234567890

So in our case, we can simply add an issue CAA record with the accounturi parameter on the DNS record directly pointing to the machine (olivine.joly.eu.org). Then other domains can alias with CNAME to that DNS record and will inherit the CAA constraints. And if a service moves to a different server, the CAA constraint will be that of the new server, if any.

Finding the accounturi Caddy Uses

Caddy automatically creates a Let’s Encrypt account if none is configured. It then issues certificates against that account. But where is the account number?

By default on Ubuntu, the parameters for that account are stored in /var/lib/caddy/.local/share/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json3. There, the location key contains the accounturi:

{
  // ...
  "location": "https://acme-v02.api.letsencrypt.org/acme/acct/1968958296"
}

Putting It All Together

Summing up the above, we know that we need a CAA record with the issue property and the same accounturi as caddy. It looks like this in the Bind format:

IN CAA   0 issue "letsencrypt.org;accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1968958296"

This record is placed on olivine.joly.eu.org, where a A (or AAAA) record contains the IP address of the Olivine server. In turn, the RSS reader host is a CNAME pointing to that A record.

Testing

To test that the new setup works, we can try to issue a certificate for a new domain. Just add it to the Caddy configuration as usual, for instance like this:

test-caa.cj.rs {
  respond "Hello, world!"
}

Caddy logs should show that the certificate was successfully issued:

{
  "level":"info",
  // ...
  "msg":"certificate obtained successfully",
  "identifier":"test-caa.cj.rs"
}

You can similarly test that changing the accounturi in the CAA. After the various caches expire, attempts to issue a certificate fail:

{
  "level":"error",
  // ...
  "msg":"could not get certificate from issuer",
  "error":"HTTP 0 urn:ietf:params:acme:error:caa - During secondary validation: While processing CAA for test-caa.cj.rs: CAA record for olivine.joly.eu.org prevents issuance"
}

Just a Mitigation in the End

We have limited the certificates that can be issued to a particular CA and to a particular account. In turn, that account is tied to a private key stored on the server.

However, as Hugo Landau explains, this is only a mitigation. A well resourced attacker could still succeed. For instance, they could alter the CAA record read by the CA.

Thus, complementary defenses include monitoring certificate transparency to spot suspicious issuance. Cloudflare offers such a service. So does crt.sh. While CAs sometimes emit certificates without logging them to Certificate Transparency4, those certificates should be rejected by most browsers. In our case, with web services, that would be enough. Other clients communicating over TLS might not reject them though, like XMPP clients of the jabber.ru example.

Despite the limitations of this CAA-based approach, it was interesting to learn more about it and see how the various pieces fit together. I hope you enjoyed reading!

  1. At least that’s the basic domain validation. Other types exist, but they are not necessarily that much more reliable in practice. ↩︎

  2. There is little the certificate authority can do, it’s not really their fault if someone manipulates traffic between them and the server to verify. They do use multi-perspective validation as mitigation measure though. ↩︎

  3. That same folder also holds the private key used to sign in with the account. ↩︎

  4. A customer can request that, if they want to avoid publishing a domain name in a public log. ↩︎