Mutual TLS in Elixir Part 1: HTTPoison

Using HTTPoison to include a client certificate is mostly straightforward but there a few caveats.

Mutual TLS (mTLS) builds upon TLS by adding client authentication. A client can include a certificate to identify itself and the server can verify this certificate. This blog post will focus on the client perspective — providing a client certificate and verifying the server certificate.

Although performing Mutual TLS at the application layer is becoming less common with the rise of TLS offloading and sidecar proxies like istio, doing mTLS straight from Elixir still has some benefits — it’s simpler and requires less additional infrastructure.

I was recently setting up Mutual TLS in Elixir and I ran into a few challenges. Part 1 of this blog post will go over how I overcame those challenges to arrive at some simple code that performs Mutual TLS securely. Part 2 will go over more advanced concepts like testing and intermediate CAs.

Challenges

The HTTPoison readme gives a simple example of using a client certificate:

And the example below shows the use of the :ssl options for a post request to an api that requires a client certification.

That code works but:

• all the default ssl options get overridden
• it uses an outdated CA bundle for server authentication

This is dangerous because that code isn’t verifying server certificates. Also the outdated CA bundle could lead to us trusting server certificates issued by a CA that recently been compromised.

This post will go over solutions to those problems. We will use a dummy client certificate I generated. You can clone https://github.com/MichaelViveros/blog_mtls_elixir to get a template project with that certificate and the necessary dependencies.

Also feel free to use your own client certificate if you have one.

Default SSL Options

Let’s define a module with one function to make a request with our client certificate.

We can make a request to a test site like server.cryptomix.com which requests a client certificate and returns information about the client certificate. And I swear, this site is legit and not an elaborate ploy for me to steal all your bitcoin 😛 (corresponding ssl labs report).

Here’s the code to send a request to server.cryptomix.com :

It prints:

We can see that the server successfully received our certificate and returned some information about it like the serial number, expiry, subject domain name (SSL_CLIENT_S_DN) and issuer domain name (SSL_CLIENT_I_DN).

Note that server.cryptomix.com doesn’t actually perform client authentication by verifying the client certificate. Part 2 of this blog post will go over how to properly test client authentication.

Now that we’ve confirmed we can properly send our client certificate, let’s see how the code handles server certificates.

badssl.com is a site where you can test how your code handles “bad” servers, Ex. a server with an expired certificate. Our code should return an error if the server is “bad”.

Running that code actually doesn’t return an error 😧. The request was made successfully even though the server certificate is expired.

Oddly enough, making a request without a client certificate will correctly return a :certificate_expired error.

Returns:

The request without the client certificate sets some default ssl options for server authentication. The request with the client certificate loses all those default options since HTTPoison (or more specifically, hackney) causes them to get overridden. Yikes!

HTTPoison is built on top of the Erlang library hackney. Hackney uses the Erlang ssl library to do TLS/SSL. By default, hackney will set Erlang ssl options like:

• verify: verify_peer - verify the server certificate isn't expired, is signed by a trusted CA, ...
• cacerts: [...] - trusted CAs
• and more … see hackney’s connection and ssl modules for more info

When we pass in our own SSL options for the client certificate, all the default options get overridden (hackney source code). Since verify: verify_peer isn't being set anymore, the expired server certificate isn't verified and no error is returned.

The hackney readme mentions this behaviour but the HTTPoison readme doesn’t say anything about it.

To fix this, we can use hackney’s hackney_connection.ssl_opts/2. Passing in [] as the second argument will cause hackney to return its default ssl options. The first argument is the hostname of the server we're connecting to which is used during hostname verification.

Returns:

We can update our code to extract the hostname from the url and then call ssl_opts() to get hackney's default ssl options. Note that our code will have to convert the hostname from a string to a charlist since hackney is an older Erlang library that doesn't support Elixir strings.

Here’s the updated code:

Now making a request to the server with the expired certificate will return an error:

Returns:

Yay! We were able to use HTTPoison to send our client certificate and verify the server certificate.

CA Bundle

One of the default ssl options mentioned above is cacerts which corresponds to the list of trusted CAs used during server authentication. Hackney uses the certifi library which contains Mozilla's canonical list of trusted CAs. This list is widely used across the industry. But there's one problem:

The latest commit was over 6 months ago … Mozilla’s CA bundle has been updated numerous times since then. An outdated CA bundle is dangerous:

• it could include a CA that has been compromised
• it could exclude a new CA that has been added

For example, a new CA called Trustwave was recently added to Mozilla’s list. Our code will incorrectly return an :unknown_ca error when connecting to a server with a Trustwave certificate.

Returns:

Fortunately, the CAStore library can save us! It has a much more updated CA bundle.

CAStore will release a new version whenever Mozilla’s CA bundle changes. The latest version includes new CAs like Trustwave (see this commit). CAStore also provides a mix task to automatically update your app’s CA bundle which can be hooked up with CI tools like CircleCI.

Note that CAStore was developed as part of mint, the new Elixir http client, but you can still use CAStore with other http clients like HTTPoison.

We can add CAStore to our mix.exs:

CAStore contains a file with its CA bundle which it exposes via:

Returns:

Instead of using Erlang ssl’s cacerts option to pass in this file, we can use the equivalent cacertfile option. But beware, there is another dangerous override lurking beneath our code! By default, cacerts will override cacertfile so we'll have to explicitly unset cacerts.

Now requests to a server with a Trustwave certificate will work.

Returns:

Conclusion

Although HTTPoison provides a simple way to include a client certificate, there are some extra steps involved to ensure you are performing Mutual TLS securely.

We had to add back the default ssl options that were overridden to ensure we were correctly verifying the server certificate. Then we had to use an updated CA bundle to ensure our list of trusted CAs is up-to-date. An updated CA bundle is important so that we exclude any CAs that have been compromised and include any new CAs that have been added.

Here’s the final code:

Now we can perform Mutual TLS in Elixir simply and securely! 🔒

Check out Part 2 of this blog which will go over more advanced concepts like properly testing client authentication and setting up intermediate certificates.