[Swan] libreswan subnet to subnet (on top of Openstack Neutron)

[Charlie Brown (charlibr)]

* Libreswan 3.15 ipsec (libreswan-3.15-5.el7_1.x86_64)
* Openstack Icehouse
* RHEL 7.1 boxes used as routers

Followed this document (with modifications) https://libreswan.org/wiki/Subnet_to_subnet_VPN_with_PSK

I am attempting to create libreswan ipsec tunnels between tenants in an Openstack Icehouse cloud.  I have created a shared subnet (10.255.255.0/24) that the RHEL boxes attach to in a multi-homed fashion.  Each box has two interfaces. net.ipv4.ip_forward is enabled (1) on each box.

    Router in tenant A                           Router in tenant B
    ------------------------                     ------------------------
    eth0      eth1           <=================> eth1           eth0
    10.0.1.1  10.255.255.10                      10.255.255.20  10.0.2.1
    ------------------------                     ------------------------


Routing

    Tenant A: 10.0.1.0/24
       * Subnet hosts use gateway: 10.0.1.1
    Tenant B: 10.0.2.0/24
       * Subnet hosts use gateway: 10.0.2.1
    Shared segment: 10.255.255.0/24

    10.0.2.0/24 via 10.255.255.20
    10.0.1.0/24 via 10.255.255.10


I've setup the VM ports to allow tenant A's network to be announced on Tenant B's eth0 interface, and vice-versa, by setting the port's allowed_address_pairs to the opposite network's range (this is an Openstack-ism).  I can connect (ping,ssh) across the shared network to arbitrary hosts on the other end.  This all works without libreswan in the middle.

I add libreswan, and things break.

Libreswan config router tenant A
--------------------------------
    config setup
        protostack=netkey
        plutodebug=all
        plutorestartoncrash=yes
        dumpdir=/var/run/pluto

    conn base
        authby=rsasig
        left=10.255.255.10
        leftsourceip=10.0.1.1
        leftsubnet=10.0.1.0/24
        leftrsasigkey=<redacted>

    tenant_a
        also=base
        right=10.255.255.10
        rightsubnet=10.0.2.0/24
        rightrsasigkey=<redacted>
        auto=start

Libreswan config router tenant B
--------------------------------
    config setup
        protostack=netkey
        plutodebug=all
        plutorestartoncrash=yes
        dumpdir=/var/run/pluto

    conn base
        authby=rsasig
        left=10.255.255.20
        leftsourceip=10.0.2.1
        leftsubnet=10.0.2.0/24
       leftrsasigkey=<redacted>

    tenant_b
        also=base
        right=10.255.255.20
        rightsubnet=10.0.1.0/24
        rightrsasigkey=<redacted>
        auto=start

Running this config on each end has the tunnel coming up without any errors.  When the tunnel comes up, libreswan adds a new route with a higher priority than the initial static route:

     10.0.2.0/24 dev eth1  scope link  src 10.0.1.1
     10.0.2.0/24 via 10.255.255.20 dev eth1  proto static  metric 100

and the same on the remote end:

    10.0.1.0/24 dev eth1  scope link  src 10.0.2.1
    10.0.1.0/24 via 10.255.255.10 dev eth1  proto static  metric 100

I can ping the remote router's eth0 interface IP address but not any hosts in the remote subnet.  This is true from either direction.

With this config, i can reach the remote end:

    traceroute to 10.0.2.1 (10.0.2.1), 64 hops max, 52 byte packets
     1  172.31.228.1 (172.31.228.1)  29.907 ms  29.742 ms  29.888 ms
     2  10.0.1.1 (10.0.1.1)  30.338 ms  29.795 ms  29.985 ms
     3  10.0.2.1 (10.0.2.1)  29.645 ms  30.893 ms  30.823 ms

But cannot route past 10.0.2.1.  I get a 'Network unreachable' error:

     1  172.31.228.1 (172.31.228.1)  29.907 ms  29.742 ms  29.888 ms
     2  10.0.1.1 (10.0.1.1)  30.338 ms  29.795 ms  29.985 ms
     3  10.0.1.1 (10.0.1.1)  3035.382 ms  !H 30.893 ms  30.823 ms

Any idea what is going on here?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libreswan.org/pipermail/swan/attachments/20160921/49d04141/attachment-0001.html>