[Swan] Forwarding between multiple LibreSwan/IPSEC tunnels
Jason Martin
jhmartin at toger.us
Sun Nov 27 22:06:06 UTC 2016
Thank you for looking at this. None of the addresses in the barf
output bave been modified. I should add that there nat rules
outside of the namepace that govern namespace traffic:
# Generated by iptables-save v1.4.18 on Sun Nov 27 21:50:44 2016
*nat
:PREROUTING ACCEPT [262:51071]
:INPUT ACCEPT [97:5112]
:OUTPUT ACCEPT [4310:441721]
:POSTROUTING ACCEPT [4318:442304]
-A PREROUTING -s 35.163.197.247/32 -i eth0 -j DNAT
--to-destination 169.254.255.2
-A PREROUTING -s 35.163.220.45/32 -i eth0 -j DNAT
--to-destination 169.254.255.3
-A PREROUTING -s 52.45.134.147/32 -i eth0 -j DNAT
--to-destination 169.254.255.4
-A PREROUTING -s 52.45.232.151/32 -i eth0 -j DNAT
--to-destination 169.254.255.5
-A POSTROUTING -d 35.163.197.247/32 -j SNAT --to-source
172.28.10.214
-A POSTROUTING -d 35.163.220.45/32 -j SNAT --to-source
172.28.10.214
-A POSTROUTING -d 52.45.134.147/32 -j SNAT --to-source
172.28.10.214
-A POSTROUTING -d 52.45.232.151/32 -j SNAT --to-source
172.28.10.214
COMMIT
Is adding the additional connections the same as changing the
existing configs to left/right subnets? When I try for example
merely adding the external subnets (leaving 0/0 in place):
aws1.conf: leftsubnets={169.254.12.53/30 172.18.0.0/16}
aws1.conf: rightsubnets={0.0.0.0/0 172.19.0.0/16}
aws2.conf: leftsubnets={169.254.12.221/30 172.18.0.0/16}
aws2.conf: rightsubnets={0.0.0.0/0 172.19.0.0/16}
aws3.conf: leftsubnets={169.254.47.13/30 172.19.0.0/16}
aws3.conf: rightsubnets={169.254.47.13/30 172.18.0.0/16}
aws4.conf: leftsubnets={169.254.47.1/30 172.19.0.0/16}
aws4.conf: rightsubnets={0.0.0.0/0 172.18.0.0/16}
, I am no longer able to connect to the remote BGP daemons.
Since nothing has been removed from the config, the cause for
that is also not obvious to me. The tunnels appear to be up
(using awstunnel3 as an example):
000 #5: "awstunnel3/1x1":4500 STATE_QUICK_I2 (sent QI2, IPsec SA established); EVENT_SA_REPLACE in 2402s; newest IPSEC; eroute owner; isakmp#3; idle; import:admin initiate
000 #5: "awstunnel3/1x1" esp.de2eb6e6 at 52.45.134.147 esp.f3659801 at 169.254.255.4 tun.0 at 52.45.134.147 tun.0 at 169.254.255.4 ref=0 refhim=0 Traffic: ESPin=0B ESPout=1KB! ESPmax=4194303B
000 #6: "awstunnel3/1x2":4500 STATE_QUICK_I2 (sent QI2, IPsec SA established); EVENT_SA_REPLACE in 2332s; newest IPSEC; eroute owner; isakmp#3; idle; import:admin initiate
000 #6: "awstunnel3/1x2" esp.ea58a6f4 at 52.45.134.147 esp.87d9c538 at 169.254.255.4 tun.0 at 52.45.134.147 tun.0 at 169.254.255.4 ref=0 refhim=0 Traffic: ESPin=0B ESPout=0B! ESPmax=4194303B
000 #7: "awstunnel3/2x1":4500 STATE_QUICK_I2 (sent QI2, IPsec SA established); EVENT_SA_REPLACE in 2628s; newest IPSEC; eroute owner; isakmp#3; idle; import:admin initiate
000 #7: "awstunnel3/2x1" esp.d52b1a86 at 52.45.134.147 esp.d44d8ee3 at 169.254.255.4 tun.0 at 52.45.134.147 tun.0 at 169.254.255.4 ref=0 refhim=0 Traffic: ESPin=71B ESPout=0B! ESPmax=4194303B
000 #8: "awstunnel3/2x2":4500 STATE_QUICK_I2 (sent QI2, IPsec SA established); EVENT_SA_REPLACE in 2310s; newest IPSEC; eroute owner; isakmp#3; idle; import:admin initiate
000 #8: "awstunnel3/2x2" esp.c8f6616a at 52.45.134.147 esp.c4b87ed at 169.254.255.4 tun.0 at 52.45.134.147 tun.0 at 169.254.255.4 ref=0 refhim=0 Traffic: ESPin=4KB ESPout=0B! ESPmax=4194303B
000 #3: "awstunnel3/2x2":4500 STATE_MAIN_I4 (ISAKMP SA established); EVENT_SA_REPLACE in 27571s; newest ISAKMP; lastdpd=8s(seq in:5568 out:0); idle; import:admin initiate
I think the reason the script author went with network
namespaces was to ease configuration that allows BGP to pair
with each end.
I thought 0.0.0.0/0 meant accept whatever parameters were
provided by the remote end; the subnets already have direct
access to the world through their amazon IGW. The goal is to
allow this hub instance to connect multiple amazon VPCs
together, using BGP so that the tunnels auto-failover their
routing. The remote networks are /16's in the 172.x.x.x space,
though the gateways I am connecting to are not part of the 172
space -- some thing happens inside AWS to make that part work.
That complicates the configuration though as the remote subnet
is not 'only' 172.x.x.x/16, since I need to speak with the
remote BGP neighbor over ipsec as well to communicate the
routes, and they are at the 169 addresses.
VGW=Amazon infrastructure config
CGW=router config.
outside_ip=public ips
inside_ips='fake' ips used to connect bgp inside the tunnel and as next-hop.
These values are all supplied by aws, verbatim below except the secret.
# West
TUNNEL1_SECRET=x
VGW_TUNNEL1_OUTSIDE_IP=35.163.197.247
CGW_TUNNEL1_INSIDE_IP=169.254.12.54
VGW_TUNNEL1_INSIDE_IP=169.254.12.53
TUNNEL2_SECRET=x
VGW_TUNNEL2_OUTSIDE_IP=35.163.220.45
CGW_TUNNEL2_INSIDE_IP=169.254.12.222
VGW_TUNNEL2_INSIDE_IP=169.254.12.221
# East
TUNNEL3_SECRET=x
VGW_TUNNEL3_OUTSIDE_IP=52.45.134.147
CGW_TUNNEL3_INSIDE_IP=169.254.47.14
VGW_TUNNEL3_INSIDE_IP=169.254.47.13
TUNNEL4_SECRET=x
VGW_TUNNEL4_OUTSIDE_IP=52.45.232.151
CGW_TUNNEL4_INSIDE_IP=169.254.47.2
VGW_TUNNEL4_INSIDE_IP=169.254.47.1
Adding the remote subnets to the left/right pair makes logical
sense, but for some reason doing so causes BGP connectivity to
fail.
Thank you,
-Jason Martin
On Sat, Nov 26, 2016 at 08:16:25AM -0500, Paul Wouters wrote:
> On Wed, 23 Nov 2016, Jason Martin wrote:
>
> >Subject: [Swan] Forwarding between multiple LibreSwan/IPSEC tunnels
>
> >In this scenario there are two VPCs being connected, and a
> >instance that happens to be in a 3rd VPC is performing the
> >routing and acting as a hub.
>
> >My issue is that the hub is able to reach both East and West,
> >but packets from either end arrive on hub but reach no further.
>
> If you configured tunnels properly (from east to west terminating
> at the middle box) then my guess is firewalling, NATing, forwarding
> or rp_filter to be the culprit.
>
> >I notice that counters 'XfrmInNoPols' increments when I attempt to ping.
> >I suspect the issue is somewhere in the XFRM config, but its not
> >clear to me how to resolve the issue (if that is even the
> >issue).
> >
> >Topology:
> >West (172.19.0.0/16) - (hub) - East (172.18.0.0/16). Hub is
> >connecting to both ends via VGW's, so cleartext packets for
> >east/west never leave Hub. As per normal VGW behavior, two
> >tunnels exist between each end and HUB.
>
> west should have a conn with hub:
>
> conn tunnel
> left=westip
> right=hubip
> leftsubnet=172.19.0.0/16
> rightsubnet=172.18.0.0/16
>
> east should have a similar conn with hub. So east and west
> have 1 tunnel, and hub has two tunnels. Is that what you have?
>
> >The basis for this configuration is
> >https://github.com/patrickbcullen/Openswan-VPC, modified to
> >support a 2nd set of tunnels. One oddity about this script is it
> >set ups a 'network namespace'
> >(http://man7.org/linux/man-pages/man8/ip-netns.8.html) to handle
> >all the ipsec and routing.
>
> Seems a bit overengineered :)
> If using namespaces, you might have to ensure that the forwarding and
> rp_filter and NAT iptables rules are properly setup for that namespace
> too. I'm a little confused how you would have the hubip in the
> configuration within the special namespace if it is also the main IP
> for the host itself?
>
> >The hub can ping nodes in east and west via the IPSEC tunnels.
> >The VGW's agree that ipsec and BGP is up, the the East/West
> >subnets see the propagated routes. The hub has routes to both
> >East and West. Iptables is fully open. rp_filter is set to 0 and
> >forwarding / ip_forward is set to 1 in sysctl.
>
> Run "ipsec verify" to confirm it all is set as expected?
>
> >I set up a ping generator in West that is attempting to ping
> >East. The packets reach the openswan network namespace in hub:
> >
> >16:38:49.311665 IP 35.163.220.45 > 169.254.255.3:
> >ESP(spi=0x0a790d98,seq=0x4f5), length 132
> >16:38:49.311665 IP 172.19.58.64 > 172.18.57.207: ICMP echo
> >request, id 411, seq 1113, length 64
> >
> >I have NFLOG / ulogd2 setup in iptables. It shows:
> >
> >RAW-PREROUTING IN=eth0 OUT= MAC=d6:fd:61:4b:73:42:6a:3a:bb:e2:33:75:08:00 SRC=172.19.58.64 DST=172.18.57.207 LEN=84 TOS=00 PREC=0x00 TTL=254 ID=49803 DF PROTO=ICMP TYPE=8 CODE=0 ID=411 SEQ=1155 MARK=0
> >NAT-PREROUTING IN=eth0 OUT= MAC=d6:fd:61:4b:73:42:6a:3a:bb:e2:33:75:08:00 SRC=172.19.58.64 DST=172.18.57.207 LEN=84 TOS=00 PREC=0x00 TTL=254 ID=49803 DF PROTO=ICMP TYPE=8 CODE=0 ID=411 SEQ=1155 MARK=0
> >
> >However the packet never reaches the FORWARD iptables chain:
> >
> >Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> >
> >Pinging from East to West fails similarly.
> >
> >The hub can ping both the source and destination:
> >
> ># ping -c 1 172.18.57.207
> >64 bytes from 172.18.57.207: icmp_seq=1 ttl=254 time=1.74 ms
> ># ping -c 1 172.19.58.64
> >64 bytes from 172.19.58.64: icmp_seq=1 ttl=254 time=94.3 ms
> >
> >Any suggestions on what might be blocking packets from
> >transiting hub? I've seen hints that 'eroute' might be filtering
> >the packets, but when I try to look at the eroute table with
> >'ipsec eroute' I get an error that NETKEY and eroute aren't
> >compatibile.
>
> You can check using "ip xfrm pol" and "ip xfrm state" what the
> kernel state is. You'd have to run this within the namespace.
>
> Oh, you included the barf which has the xfrm settings
>
> >+ ip xfrm policy
> >src 169.254.12.220/30 dst 0.0.0.0/0
>
> Not sure why you have 169.254 ipsec policies? Either you lied
> and rewrote the above IP addresses from the real ones used,
> or you are surely missing IPsec policies in this namespace?
> Or perhaps barf was run in the wrong namespace?
>
> >XfrmInTmplMismatch 8649
>
> Well, 169.254 does not match 172.16 but how could you get
> the wrong policies? Possibly namespace confusion?
>
> ># begin conn awstunnel1
> >conn awstunnel1
> > left=169.254.255.2
> > leftid="169.254.255.2"
> > leftsubnet=169.254.12.52/30
> > left=169.254.255.2
> > right=35.163.197.247
> > rightid="35.163.197.247"
> > rightsubnet=0.0.0.0/0
> > right=35.163.197.247
>
> So my guess is you rewrote some IP addresses in your email?
>
> I guess I was expecting you wanted to connect two subnets to
> each other but you are using 0.0.0.0/0 so that means you want
> to extrude a subnet ? That is the subnets are given connectivity
> to the world, and not just to each other? That does not match
> what you were trying to do as written in your email?
>
> Paul
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL: <https://lists.libreswan.org/pipermail/swan/attachments/20161127/f83f5e77/attachment.sig>
More information about the Swan
mailing list