[Swan] Forwarding between multiple LibreSwan/IPSEC tunnels

[Paul Wouters]

On Wed, 23 Nov 2016, Jason Martin wrote:

> Subject: [Swan] Forwarding between multiple LibreSwan/IPSEC tunnels

> In this scenario there are two VPCs being connected, and a
> instance that happens to be in a 3rd VPC is performing the
> routing and acting as a hub.

> My issue is that the hub is able to reach both East and West,
> but packets from either end arrive on hub but reach no further.

If you configured tunnels properly (from east to west terminating
at the middle box) then my guess is firewalling, NATing, forwarding
or rp_filter to be the culprit.

> I notice that counters 'XfrmInNoPols' increments  when I attempt to ping.
> I suspect the issue is somewhere in the XFRM config, but its not
> clear to me how to resolve the issue (if that is even the
> issue).
>
> Topology:
> West (172.19.0.0/16) - (hub) - East (172.18.0.0/16). Hub is
> connecting to both ends via VGW's, so cleartext packets for
> east/west never leave Hub. As per normal VGW behavior, two
> tunnels exist between each end and HUB.

west should have a conn with hub:

conn tunnel
 	left=westip
 	right=hubip
 	leftsubnet=172.19.0.0/16
 	rightsubnet=172.18.0.0/16

east should have a similar conn with hub. So east and west
have 1 tunnel, and hub has two tunnels. Is that what you have?

> The basis for this configuration is
> https://github.com/patrickbcullen/Openswan-VPC, modified to
> support a 2nd set of tunnels. One oddity about this script is it
> set ups a 'network namespace'
> (http://man7.org/linux/man-pages/man8/ip-netns.8.html) to handle
> all the ipsec and routing.

Seems a bit overengineered :)
If using namespaces, you might have to ensure that the forwarding and
rp_filter and NAT iptables rules are properly setup for that namespace
too. I'm a little confused how you would have the hubip in the
configuration within the special namespace if it is also the main IP
for the host itself?

> The hub can ping nodes in east and west via the IPSEC tunnels.
> The VGW's agree that ipsec and BGP is up, the the East/West
> subnets see the propagated routes. The hub has routes to both
> East and West. Iptables is fully open. rp_filter is set to 0 and
> forwarding / ip_forward is set to 1 in sysctl.

Run "ipsec verify" to confirm it all is set as expected?

> I set up a ping generator in West that is attempting to ping
> East. The packets reach the openswan network namespace in hub:
>
> 16:38:49.311665 IP 35.163.220.45 > 169.254.255.3:
> ESP(spi=0x0a790d98,seq=0x4f5), length 132
> 16:38:49.311665 IP 172.19.58.64 > 172.18.57.207: ICMP echo
> request, id 411, seq 1113, length 64
>
> I have NFLOG / ulogd2 setup in iptables. It shows:
>
> RAW-PREROUTING IN=eth0 OUT= MAC=d6:fd:61:4b:73:42:6a:3a:bb:e2:33:75:08:00 SRC=172.19.58.64 DST=172.18.57.207 LEN=84 TOS=00 PREC=0x00 TTL=254 ID=49803 DF PROTO=ICMP TYPE=8 CODE=0 ID=411 SEQ=1155 MARK=0
> NAT-PREROUTING IN=eth0 OUT= MAC=d6:fd:61:4b:73:42:6a:3a:bb:e2:33:75:08:00 SRC=172.19.58.64 DST=172.18.57.207 LEN=84 TOS=00 PREC=0x00 TTL=254 ID=49803 DF PROTO=ICMP TYPE=8 CODE=0 ID=411 SEQ=1155 MARK=0
>
> However the packet never reaches the FORWARD iptables chain:
>
> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>
> Pinging from East to West fails similarly.
>
> The hub can ping both the source and destination:
>
> # ping -c 1 172.18.57.207
> 64 bytes from 172.18.57.207: icmp_seq=1 ttl=254 time=1.74 ms
> # ping -c 1 172.19.58.64
> 64 bytes from 172.19.58.64: icmp_seq=1 ttl=254 time=94.3 ms
>
> Any suggestions on what might be blocking packets from
> transiting hub? I've seen hints that 'eroute' might be filtering
> the packets, but when I try to look at the eroute table with
> 'ipsec eroute' I get an error that NETKEY and eroute aren't
> compatibile.

You can check using "ip xfrm pol" and "ip xfrm state" what the
kernel state is. You'd have to run this within the namespace.

Oh, you included the barf which has the xfrm settings

> + ip xfrm policy
> src 169.254.12.220/30 dst 0.0.0.0/0

Not sure why you have 169.254 ipsec policies? Either you lied
and rewrote the above IP addresses from the real ones used,
or you are surely missing IPsec policies in this namespace?
Or perhaps barf was run in the wrong namespace?

> XfrmInTmplMismatch      	8649

Well, 169.254 does not match 172.16 but how could you get
the wrong policies? Possibly namespace confusion?

> # begin conn awstunnel1
> conn awstunnel1
> 	left=169.254.255.2
> 	leftid="169.254.255.2"
> 	leftsubnet=169.254.12.52/30
> 	left=169.254.255.2
> 	right=35.163.197.247
> 	rightid="35.163.197.247"
> 	rightsubnet=0.0.0.0/0
> 	right=35.163.197.247

So my guess is you rewrote some IP addresses in your email?

I guess I was expecting you wanted to connect two subnets to
each other but you are using 0.0.0.0/0 so that means you want
to extrude a subnet ? That is the subnets are given connectivity
to the world, and not just to each other? That does not match
what you were trying to do as written in your email?

Paul