[Swan] Advice on troubleshooting AWS VPC site-to-site VPN connection.

Mon Oct 18 19:07:48 UTC 2021

Just to summarize my results so far:

left(CentOS8-libreswan)<---VPN--->right(AWS-virtual-private-gateway)---->AWS-EC2(10.0.2.252 & 54.151.19.230)

ping from left to EC2(10.0.2.252)

tcpdump shows that a ping packet leaves left as UDP-encapsulated ESP packet:

11:03:17.230139 IP xxx.xxx.xxx.19.ipsec-nat-t > 52.9.186.125.ipsec-nat-t: UDP-encap: ESP(spi=0xc17c2033,seq=0x15), length 132

Then arrives and leaves the EC2 instance as regular ICMP packets

18:03:17.233191 IP xxx.xxx.xxx.19 > 10.0.2.252: ICMP echo request, id 5630, seq 1, length 64
18:03:17.233222 IP 10.0.2.252 > xxx.xxx.xxx.19: ICMP echo reply, id 5630, seq 1, length 64

and arrives back at left as a regular icmp packet.

11:03:17.234656 IP 54.151.19.230 > xxx.xxx.xxx.19: ICMP echo reply, id 5630, seq 1, length 64

and ipsec trafficstatus shows only outgoing ESP traffic. There are no inBytes. This is what is concerning me the most.

# ipsec trafficstatus
006 #19: "conn-to-aws-1/1x1", type=ESP, add_time=1634578592, inBytes=0, outBytes=0, id='52.9.186.125'
006 #21: "conn-to-aws-1/1x3", type=ESP, add_time=1634578675, inBytes=0, outBytes=0, id='52.9.186.125'
006 #20: "conn-to-aws-1/2x1", type=ESP, add_time=1634578643, inBytes=0, outBytes=0, id='52.9.186.125'
006 #15: "conn-to-aws-1/2x3", type=ESP, add_time=1634578312, inBytes=0, outBytes=1344, id='52.9.186.125'
006 #16: "conn-to-aws-2/1x1", type=ESP, add_time=1634578373, inBytes=0, outBytes=0, id='54.151.120.103'
006 #22: "conn-to-aws-2/1x2", type=ESP, add_time=1634578698, inBytes=0, outBytes=0, id='54.151.120.103'
006 #17: "conn-to-aws-2/2x1", type=ESP, add_time=1634578410, inBytes=0, outBytes=0, id='54.151.120.103'
006 #18: "conn-to-aws-2/2x2", type=ESP, add_time=1634578525, inBytes=0, outBytes=0, id=’5hen a 4.151.120.103’

Then pinging the public IP of the EC2 instance (54.151.19.230) behaves as expected with icmp packets leaving left:

11:05:19.450900 IP xxx.xxx.xxx.19 > 54.151.19.230: ICMP echo request, id 5648, seq 1, length 64

arriving and leaving the EC2 instance:

18:05:19.452647 IP xxx.xxx.xxx.19 > 10.0.2.252: ICMP echo request, id 5648, seq 1, length 64
18:05:19.452678 IP 10.0.2.252 > xxx.xxx.xxx.19: ICMP echo reply, id 5648, seq 1, length 64

arriving back at left:

11:05:19.454082 IP 54.151.19.230 > xxx.xxx.xxx.19: ICMP echo reply, id 5648, seq 1, length 64

It seems that maybe the AWS virtual private gateway is not routing properly?

Any ideas would be greatly appreciated.

and no XFRM errors in the kernel tables:

cat /proc/net/xfrm_stat
XfrmInError             	0
XfrmInBufferError       	0
XfrmInHdrError          	0
XfrmInNoStates          	0
XfrmInStateProtoError   	0
XfrmInStateModeError    	0
XfrmInStateSeqError     	0
XfrmInStateExpired      	0
XfrmInStateMismatch     	0
XfrmInStateInvalid      	0
XfrmInTmplMismatch      	0
XfrmInNoPols            	0
XfrmInPolBlock          	0
XfrmInPolError          	0
XfrmOutError            	0
XfrmOutBundleGenError   	0
XfrmOutBundleCheckError 	0
XfrmOutNoStates         	0
XfrmOutStateProtoError  	0
XfrmOutStateModeError   	0
XfrmOutStateSeqError    	0
XfrmOutStateExpired     	0
XfrmOutPolBlock         	0
XfrmOutPolDead          	0
XfrmOutPolError         	0
XfrmFwdHdrError         	0
XfrmOutStateInvalid     	0
XfrmAcquireError        	0

Thanks again.

Scott

> On Oct 14, 2021, at 7:00 PM, Paul Wouters <paul at nohats.ca> wrote:
> 
> On Thu, 14 Oct 2021, Scott Classen wrote:
> 
>> Now I have an EC2 instance that is associated with my AWS VPC with a public and private IP address
>> 
>> Private: 10.0.1.252
>> Public: xxx.xxx.79.208
>> 
>> I can ping the public address no problem, but when I ping the private address the pings appear to go out as UDP-encap: ESP packets (I think?) but they return from the public address! See this tcpdump output:
>> 
>> # ping -c3 10.0.1.252
>> PING 10.0.1.252 (10.0.1.252) 56(84) bytes of data.
>> 64 bytes from xxx.xxx.79.208: icmp_seq=1 ttl=240 time=3.44 ms
>> 64 bytes from xxx.xxx.79.208: icmp_seq=2 ttl=240 time=5.13 ms
>> 64 bytes from xxx.xxx.79.208: icmp_seq=3 ttl=240 time=3.46 ms
> 
> 
> So this looks good. The ping is using the internal IP and gets
> a response on the internal IP
> 
>> # tcpdump -ni enp2s0f0 udp port 500 or udp port 4500 or icmp
>> 13:30:54.113295 IP xxx.xxx.85.19.ipsec-nat-t > 52.9.186.125.ipsec-nat-t: UDP-encap: ESP(spi=0xc9530708,seq=0x20), length 132
>> 13:30:54.116788 IP xxx.xxx.79.208 > xxx.xxx.85.19: ICMP echo reply, id 56943, seq 1, length 64
>> 13:30:55.114989 IP xxx.xxx.85.19.ipsec-nat-t > 52.9.186.125.ipsec-nat-t: UDP-encap: ESP(spi=0xc9530708,seq=0x21), length 132
>> 13:30:55.118301 IP xxx.xxx.79.208 > xxx.xxx.85.19: ICMP echo reply, id 56943, seq 2, length 64
>> 13:30:56.116497 IP xxx.xxx.85.19.ipsec-nat-t > 52.9.186.125.ipsec-nat-t: UDP-encap: ESP(spi=0xc9530708,seq=0x22), length 132
>> 13:30:56.119892 IP xxx.xxx.79.208 > xxx.xxx.85.19: ICMP echo reply, id 56943, seq 3, length 64
> 
> This looks good a little odd. The fact you see decrypted traffic is not
> wrong. The reason you see this is an artifact of how the linux kernel
> implements the hooks which tcpdump uses to see traffic.  Basically,
> it sees outgoing traffic after encrypt, and incoming traffic after
> decrypt. If you tcpdumped on the machine in front of your linux machines,
> you would see traffic as UDP-encap: ESP both ways.
> 
> You can run "ipsec trafficstatus" and look at the byte counters to
> confirm traffic is properly encrypted in both directions.
> 
> You can also check 'cat /proc/net/xfrm_stat' to see any problems. All
> the counters in this file should be 0.
> 
> Paul