[Swan] Site-to-site with public member addresses, routing trouble

[Jesse Butcher]

Hello,

This is my first post to this list so please let me know if I've broken
any protocols or best practices.

* Barf dump here: https://s3-us-west-2.amazonaws.com/signet-dist/ipsec.barf.tgz
* ASCII art attached (it's a bit wide).

I'm trying to configure an IPsec tunnel with a client.  The client has already reserved all RFC1918 address space so we're required to configure the members with public IP's.  We're only trying to grant traffic between one remote host and two local so all members are defined as /32 subnets.  FWIW, they are using a Cisco ASA.  Both sides are in NAT'd LAN's.

We have successfully established SA's with no errors but I am having trouble configuring the routing on our side.  

Here's a snippet of the left and right sections in the config:
|    # left == HCL Network (local)
|    # ---------------------------
|    leftid=66.11.27.153
|    left=10.250.248.14
|    leftsubnets={66.11.24.25/32 66.11.24.26/32}
|
|
|    # right == URMC (remote)
|    # ----------------------
|    rightid=128.151.71.71
|    right=128.151.71.71
|    rightsubnet=128.151.70.41/32

And a snippet of `ip xfrm policy`:
|    src 66.11.24.26/32 dst 128.151.70.41/32
|            dir out priority 2080 ptype main
|            tmpl src 10.250.248.14 dst 128.151.71.71
|                    proto esp reqid 16393 mode tunnel
|    src 66.11.24.25/32 dst 128.151.70.41/32
|            dir out priority 2080 ptype main
|            tmpl src 10.250.248.14 dst 128.151.71.71
|                    proto esp reqid 16389 mode tunnel

My thinking was since the tunnels are (66.11.24.25<==>128.151.70.41) and (66.11.24.26<==>128.151.70.41) and routing wasn't working out-of-the-box that perhaps I needed the source of the packets to match their public IP's.  So I added their respective pub IP's and added routes with specified "src".

|    # Add pub IP to NIC
|    [Member A]# ip address add 66.11.24.25/32 dev eth1
|
|    # Send traffic destined for client host through our endpoint and set the source as pub IP
|    [Member A]# ip route add 128.151.70.41 via 10.250.248.14 dev eth1  src 66.11.24.25
|
|
|    [Member A]# ip route
|    128.151.70.41 via 10.250.248.14 dev eth1
|    10.250.248.0/24 dev eth1  proto kernel  scope link  src 10.250.248.11
|    169.254.0.0/16 dev eth1  scope link  metric 1002
|    default via 10.250.248.5 dev eth1
|
|    <repeat for Member B>

Still not working.  I can see with tcpdump that the ping packet goes from Member A to our local endpoint but the endpoint uses the internet to forward the packet instead of the tunnel.  It presents as "Time to live exceeded", replied by their internet gateway (128.151.71.7)

I'm relatively new to IPsec and I've successfully set up a few different tunnels so far but I haven't run into this.

What am I doing wrong?  Thanks in advance to anyone willing to take a look.

--
Jesse Butcher

-------------- next part --------------
  LOCAL                                                               REMOTE
+---------------------------------------------------+              +-------------------------------------------------------+
|                                                   |              |                                                       |
| +-------------------+                             |              |                                                       |
| |66.11.24.25        |                             |              |                                                       |
| |      10.250.248.11+---+                         |              |                                                       |
| +-------------------+   |                         |              |                                                       |
|       MEMBER A          |      LOCAL ENDPOINT     |              |  REMOTE ENDPOINT                                      |
|                         |  +--------------------+ |              | +---------------+                   MEMBER C          |
|                         |  |        66.11.27.153+-----INTERNET-----+128.151.71.71  |            +----------------------+ |
|                         +--+10.250.248.14       | |              | |        ?.?.?.?+------------+172.16.157.127        | |
|                         |  +--------------------+ |              | +---------------+            |         128.151.70.41| |
|       MEMBER B          |                         |              |                              +----------------------+ |
| +-------------------+   |                         |              |                                                       |
| |66.11.24.26        |   |                         |              |                                                       |
| |      10.250.248.12+---+                         |              |                                                       |
| +-------------------+                             |              |                                                       |
|                                                   |              |                                                       |
+---------------------------------------------------+              +-------------------------------------------------------+