When working with ASP.Net Core in Docker containers, it can be cumbersome to deal with certificates. While there is a documentation about setting certificate for dev environment, there’s no real guidance on how to make it work when deploying containers in a Swarm for example.
In this article we are going to see how to take advantage of Docker secrets to store ASP.Net Core Kestrel certificates in the context of Docker Swarm.
Hosting the service
First of all, we are going to create à Swarm service on our machine that use the sample Asp.Net Core app. The purpose of this article is to make SSL work in the container withoutchanging anything to an existing image.
docker service create --name mywebsite --publish published=8080,target=80,mode=host microsoft/dotnet-samples:aspnetapp
We are creating service mywebsite, publishing only one port 8080 bound to the port 80 in the container using the host mode and using the image microsoft/dotnet-samples:aspnetapp. Please note that you can use others configuration (for example expose port in routing mesh mode).
Preparing the certificate
We need a certificate. It can be created via an external certificate authority but here for the sake of the article, we are going to create a self signed certificate (of course, don’t use this in production). We are using Powershell for this task (you can skip this if you already have a pfx certificate signed by a real CA).
$cert = New-SelfSignedCertificate -DnsName "mywebsite" -CertStoreLocation "cert:\LocalMachine\My"
$password = ConvertTo-SecureString -String "mylittlesecret" -Force -AsPlainText
$cert | Export-PfxCertificate -FilePath c:\temp\mywebsite.pfx -Password $password
Once you have your pfx, we are goind to unprotect it from the password. It might be seem unsecure but when it will be added to the Docker secret store, it will be stored securedly. For this task I will use OpenSSL (not possible with Powerhsell as far as I know). OpenSSL is provided with Git for example.
& 'C:\Program Files\Git\mingw64\bin\openssl.exe' pkcs12 -in c:\temp\mywebsite.pfx -nodes -out c:\temp\mywebsite.pem -passin pass:mylittlesecret
& 'C:\Program Files\Git\mingw64\bin\openssl.exe' pkcs12 -export -in c:\temp\mywebsite.pem -out c:\temp\mywebsite.unprotected.pfx -passout pass:
Now that the pfx is un protected, we can add it to the docker store certificate and display it.
docker secret create kestrelcertificate c:\temp\mywebsite.unprotected.pfx
docker secret ls
ID NAME DRIVER CREATED UPDATED
iapy6rolt7po1mwm9aw6z0qc5 kestrelcertificate 13 minutes ago 13 minutes ago
Our secret being in the store, you can delete (or store securely somewhere else your pfx).
Making it work
We can now update our service to take in account this secret. When adding a secret to a service, Docker will create a file in a specific directory containing the value of the secret. On Windows it’s c:\programdata\docker\secrets.
Let’s update our service and see what happened inside the container.
docker service update --secret-add kestrelcertificate mywebsite
docker exec 4b51e736ce65 cmd.exe /c dir c:\programdata\docker\secrets
Volume in drive C has no label.
Volume Serial Number is 3CBB-E577
Directory of c:\programdata\docker\secrets
11/15/2018 10:38 PM <DIR> .
11/15/2018 10:38 PM <DIR> ..
11/15/2018 10:38 PM <SYMLINK> kestrelcertificate [C:\ProgramData\Docker\internal\secrets\iapy6rolt7po1mwm9aw6z0qc5]
1 File(s) 0 bytes
2 Dir(s) 21,245,009,920 bytes free
We can see that our secret exists and is named kestrelcertificate, as we named it in the command line.
We can therefore update our service to remove the old binding on port 80, replace it with a binding on port 443, tell Kestrel to use this port and finally give Kestrel the path of our secret.
This can be done with only one command:
docker service update --publish-rm published=8080,target=80,mode=host --publish-add published=8080,target=443,mode=host --env-add ASPNETCORE_URLS=https://+:443 --env-add Kestrel__Certificates__Default__Path=c:\programdata\docker\secrets\kestrelcertificate mywebsite
Wait a while that your service update, try to browse and it should work ! Well, actually it should only works on Linux.
Making it work on Windows
If you try to have a look a the logs generated by your service, you should end with something like this.
docker service logs mywebsite
mywebsite.1.uy3vm8txwxec@nmarchand-lt | crit: Microsoft.AspNetCore.Server.Kestrel[0]
mywebsite.1.uy3vm8txwxec@nmarchand-lt | Unable to start Kestrel.
mywebsite.1.uy3vm8txwxec@nmarchand-lt | Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Unspecified error
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(Byte[] rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.LoadCertificate(CertificateConfig certInfo, String endpointName)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.LoadDefaultCert(ConfigurationReader configReader)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.Load()
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.ValidateOptions()
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
mywebsite.1.uy3vm8txwxec@nmarchand-lt |
mywebsite.1.uy3vm8txwxec@nmarchand-lt | Unhandled Exception: Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Unspecified error
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(Byte[] rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.LoadCertificate(CertificateConfig certInfo, String endpointName)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.LoadDefaultCert(ConfigurationReader configReader)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.Load()
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.ValidateOptions()
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Hosting.Internal.WebHost.StartAsync(CancellationToken cancellationToken)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token, String shutdownMessage)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host)
mywebsite.1.uy3vm8txwxec@nmarchand-lt | at aspnetapp.Program.Main(String[] args) in C:\app\aspnetapp\Program.cs:line 18
We can see a nasty bug of Windows here (Github issue).
What did happen ? If you look closely at the dir command we made in the container, you’ll see that the secret is not really a file but instead a symbolic link to an other file. Unfortunately, Windows is unable to use a certificate that is a symlink. One solution could be to manually read the certificate with File.ReadAllBytes() and pass it to the constructor of X509Certificate. However, it would be against the purpose of this article which is to not modify the Docker image.
We can find a workaround by browsing the Docker documentation which states that the real file containing the secret (which in fact is the target of the symlink) can be found in the path c:\programdata\docker\internal\secrets\<secretid> where secretid is the id of the secret (as shown by docker secret ls).
We can update our service to change the path by updating the environment variable. It now works also on windows!
docker service update --env-rm
Kestrel__Certificates__Default__Path=c:\programdata\docker\secrets\kestrelcertificate --env-add Kestrel__Certificates__Default__Path=c:\programdata\docker\internal\secrets\iapy6rolt7po1mwm9aw6z0qc5 mywebsite
docker logs 7b54cdc42a86
Hosting environment: Production
Content root path: C:\app
Now listening on: https://[::]:443
Application started. Press Ctrl+C to shut down.
Final word
We have seen in this article how to use Docker secrets to store ASP.Net Core Kestrel certificates in our Docker Swarm. However, please keep in mind that the Windows workaround should be used with care as written in the Docker documentation.
Another word also about SSL Offloading : I know that usually the reverse proxy (Nginx, Traefik, etc.) is used to be the SSL termination but sometimes you still want SSL end to end.
