Enabling IPSec VPN connections to ISA 2006

by Ed Fisher on 2007-08-09

in Security

One part of my current job is to manage VPN access to the corporate intranet. This fortunately does NOT involve user access currently, as that is outsourced, but it does involve third party access, and since our outsource solution doesn’t support user password changes, I am in the process of evaluating alternatives. One of the things you can use MS ISA 2006 for is client VPN connectivity. I see several advantages to this, and am evaluating ISA for VPN as a result.

  1. Client software is included in all current Windows operating sytems
  2. Configuration can be managed through Group Policy, and other methods, that do not require the installation of a fat client
  3. Automatically integrates with Active Directory, unless you want otherwise
  4. Offers granular access control, and network quarantine
  5. Definitely can prompt a user to change their password at expiration

Setting up ISA 2006 to accept VPN connections is pretty straight forward, as long as you are willing to use PPTP. While I do feel that PPTP is secure, IPSec is better, and is supposed to work, so why not use it? Well, in short, while the ISA server will support it, and accept IPSec connections without any additional configuration, the clients will not be able to use it if there is NAT somewhere in the data flow, without some extra work.
\
we interrupt this post for the following…
Remember, NAT is evil. Say it. SAY IT BIATCH! NAT IS EVIL!!! we now return you to our original posting, already in progress…
\
Sorry about that. Anyway, Windows clients will attempt to make a IPSec connection, but, upon detecting that NAT is in place, will fail to complete the ISAKMP negotiation. No entry will appear in the system log, and a network trace will just show the client continuing to try ISAKMP until timeout. Since these exchanges are already encrypted, a trace won’t tell you much.

The problem is that once NAT is detected, the client will not use IPSec unless it is encapsultated within UDP. This is turned off by default, but it will continue to try. Genius. What you need to do is enable UDP encapsulation using a registry hack, which is different in Vista from XP, but ultimately, we want a key called AssumeUDPEncapsulationContextOnSendRule .

For Vista http://support.microsoft.com/kb/926179

  1. Log on to Vista as an administrator.
  2. Launch regedit.
  3. Browse to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent
  4. Right-click, New, DWORD (32-bit) Value.
  5. Type AssumeUDPEncapsulationContextOnSendRule, and then press ENTER.
  6. Double-click AssumeUDPEncapsulationContextOnSendRule
  7. In the Value Data box, type one of the following values, depending on your situation:
    • 0 (default) configures Windows so that it cannot establish security associations with servers that are located behind network address translators.
    • 1 this configures Windows so that it can establish security associations with servers that are located behind network address translators.
    • 2 This configures Windows so that it can establish security associations when both the server and the Windows Vista client computer are behind network address translators.
  8. Click OK, and then exit Registry Editor.
  9. Restart the computer.


For XP http://support.microsoft.com/?id=818043

  1. Log on to XP as an administrator
  2. Click Start, click Run, type regedit, and then click OK.
  3. Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPsec
  4. On the Edit menu, point to New, and then click DWORD Value.
  5. In the New Value #1 box, type AssumeUDPEncapsulationContextOnSendRule, and then press ENTER.
  6. Right-click AssumeUDPEncapsulationContextOnSendRule, and then click Modify.
  7. In the Value Data box, type one of the following values, depending on your situation:
    • 0 (default) configures Windows so that it cannot establish security associations with servers that are located behind network address translators.
    • 1 this configures Windows so that it can establish security associations with servers that are located behind network address translators.
    • 2 This configures Windows so that it can establish security associations when both the server and the Windows XP SP2-based client computer are behind network address translators.
  8. Click OK, and then quit Registry Editor.
  9. Restart the computer.

Reboots are required because a system driver is responsible for this and only reads the settings at start up. If your ISA server is directly connected to the Internet, with a live ip.addr, you will use a value of 1. If your ISA server is itself behind another firewall that is NATing the public address, use 2.

If your ISA Server is behind another firewall, you will need to permit udp 500 and 4500 in from the Internet. In PIXish, that is

access-list from_internet permit udp any host w.x.y.z eq 500
access-list from_internet permit udp any host w.x.y.z eq 4500

Then, all you need to do is configure your client to make the IPSec connection, and establish the preshared key if applicable.

here endeth the lesson

You might also enjoy:

  1. enabling NLB on ISA 2006
  2. Enabling hyperlinks for remote connections in Windows
  3. howto://troubleshoot microsoft vpn connections part three-tales from the trenches
  4. howto://troubleshoot microsoft vpn connections part two-client side issues

Leave a Comment

CommentLuv Enabled

Previous post:

Next post: