Connecting to a VPN server via L2TP/IPsec, behind layers of NAT

Morgan, an Apple Inc. consultant and overall kick ass friend of mine, and I were scratching our heads a few weeks ago over an issue in which her Windows XP and Windows Vista machines could not connect to an Xserve running Mac OS X Leopard Server. Naturally, her Mac OS X clients could connect without problem, threatening my manhood...

After some hardcore research, we stumbled, literally, across a few Microsoft support articles all explaining how to allow connections to devices behind NAT.

By default, Windows Vista and the Windows Server 2008 operating system do not support Internet Protocol security (IPsec) network address translation (NAT) Traversal (NAT-T) security associations to servers that are located behind a NAT device. Therefore, if the virtual private network (VPN) server is behind a NAT device, a Windows Vista-based VPN client computer or a Windows Server 2008-based VPN client computer cannot make a Layer Two Tunneling Protocol (L2TP)/IPsec connection to the VPN server. This scenario includes VPN servers that are running Windows Server 2008 and Microsoft Windows Server 2003.

For those curious about the why, it boils down to the fact that Microsoft has uncovered a rare issue with NAT devices (not Windows) that “may cause security association negotiation traffic to be misrouted”. To be safe, this sort of communication is disabled out of the box.

To re-enable the old behavior, accepting the risk, we navigated to HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesIPsec and created a AssumeUDPEncapsulationContextOnSendRule DWORD value, setting its value to 0, 1, or 2, depending on the configuration of each client.

• 0 (default)
A value of 0 (zero) configures Windows XP SP2 so that it cannot initiate IPsec-secured communications with responders that are located behind network address translators.

• 1
A value of 1 configures Windows XP SP2 so that it can initiate IPsec-secured communications with responders that are located behind network address translators.

• 2
A value of 2 configures Windows XP SP2 so that it can initiate IPsec-secured communications when both the initiators and the responders are behind network address translators.

I made it out alive with only minor cuts and scrapes.