There is very little to no good documentation from Cisco, Microsoft, or Apple when it comes to authenticating a VPN tunnel using SSL certificates. I ran across a customer issue last week where, for compliance reasons, a customer needed to authenticate their VPN tunnel using an SSL certificate, rather than a pre-shared key. No problem, I have set this up several times in the past. The curve ball here was that the end users were using 64 Bit Mac OS X Lion.
After struggling with this for a bit and doing a good deal of research, this seems to be the “great unsolved problem” of Mac OS X. I did get a clue as to what the issue was by finding a patched version of racoon, the Mac OS X vpn daemon here https://github.com/p120ph37/darwin-racoon-cisco-cert-fix. By reading this smart gentlemen’s write up, I learned what the issue actually was.
As you probably know, the reason we use SSL certificates to identify VPN tunnels is to guarantee the identity of the VPN client and the VPN server. This is done by exchanging SSL certificates issued by a common certificate authority. In general, this works the same way from both from the VPN client side and the VPN server side. The VPN client gets the VPN server’s SSL certificate during the Phase 1 negotiation and vice versa. The VPN client then uses a stored copy of the CA certificate that issued the VPN server’s certificate to verify the authenticity of the certificate. That is, the client essentially verifies “am I really connecting to whom I think I am connecting”?
Here is where the Mac OC X rub comes in. The Cisco VPN client retrieves the SSL certificate and checks the subject name of the certificate. That subject name will usual be the FQDN that the VPN client used to connect to the VPN server or the IP address if a DNS name was not used. In my experience, the Cisco VPN client really does not even care about the subject name, but it seems to blindly accept any SSL certificate. As long as the certificate “checks out” when validated by the CA certificate, the Cisco VPN client says “Good enough” and accepts the identity of the VPN server.
Mac OS X’s built in VPN client, raccoon, is a lot stricter when it comes to SSL identity checks. And, it has a VERY odd behavior, At least it seems odd to me, as I have not read the RFCs for this kind of thing. Rather than look at the subject name, the Mac OSX built-in VPN client looks at the SUBJECT ALTERNATIVE NAME (SAN). If the SAN is missing or does not match the FQDN or IP address used to connect to the VPN server, raccoon will assume this is a forged SSL certificate and fail Phase 1 vpn negotiation. The /var/log/system.log will report a failure on the SSL message 6 of the main-mode Phase 1 negotiation like so:
May 25 11:11:26 Aarons-MacBook-Pro-2 racoon: IKE Packet: transmit success. (Initiator, Main-Mode message 1).
May 25 11:11:26 Aarons-MacBook-Pro-2 racoon: IKE Packet: receive success. (Initiator, Main-Mode message 2).
May 25 11:11:26 Aarons-MacBook-Pro-2 racoon: IKE Packet: transmit success. (Initiator, Main-Mode message 3).
May 25 11:11:26 Aarons-MacBook-Pro-2 racoon: IKE Packet: receive success. (Initiator, Main-Mode message 4).
May 25 11:11:26 Aarons-MacBook-Pro-2 racoon: IKE Packet: transmit success. (Initiator, Main-Mode message 5).
May 25 11:11:26 Aarons-MacBook-Pro-2 racoon: IKEv1 Phase1 AUTH: failed. (Initiator, Main-Mode Message 6).
So the key to this whole issue revolves around the SSL certificate that is on the firewall; the firewall’s identity certificate. If you follow most guides for SSL VPN authentication, they will have you generate the SSL certificate signing request (CSR) on the firewall device itself, which normally makes a lot of sense. However, since the Mac OS X native client REQUIRES a SAN certificate, and the firewall does not give you this option when creating a CSR, you will need to create the CSR off of the ASA with the SAN fields properly identified. You can then import this certificate and the private key into the ASA with a few OpenSSL gyrations.
I am going to go over how to set this up using a Microsoft standalone, non-AD integrated certificate authority (CA) and then demonstrate how to configure the Mac OS X VPN client to work properly with the ASA for VPN certificate authentication. This can also be done using free tools in Linux, however, I will be using the Microsoft Server for the demonstration.
Part 1: Install the PKI and tools and generate the needed keys and certificates:
1. Install the Certificate Authority role on Server 2008 (Standard edition or better) by opening Server Manager and clicking ADD ROLES. Select the checkbox “Active Directory Certificate Services”. Note that you do NOT have to have an AD infrastructure setup. This can be used as a stand-alone server, even though the title implies it is AD integrated only. When asked for the subject name of the CA, enter something that makes sense for your organization. In my case, I have used the subject name and common name (CN): ca.networksa.org.
2. Now let’s prepare an MMC to get access to all of the tools that we need on Windows 2K8 to generate a CSR, sign the CSR, and finally export the signed certificate. Click Start –> Run and type MMC to get a MMC framework. Click File–> Add/Remove Snap-in and select the Certificate Authority Server snap in. Then, also select the Certificates Snap-in. When prompted for the types of certificates that you would like to manage, select Computer Account on the local computer.You now have a rudimentary Public Key Infrastructure (PKI) set up with the needed tools to manage the PKI for our purposes.
3. Now we are going to generate a CSR for the ASA firewall. In your MMC, expand the Certificates Tool.
4. Now that we have the CSR generated with all of the critical fields, we are ready to sign the request with our CA.
5. We are now ready to export the certificate and add it to our Personal Certificate Store to match the certificate with the Private Key. .
6. We are almost there. Now, we need to export the certificate and the private key as a PFX file.
7. The last thing we will need from the Windows server at this time is the CA’s public certificate.
We now have everything we need from the CA to begin configuring our ASA. In part 2. we will cover the ASA configuration using all of the pieces that we have created here in part 1!