[Swan] getting traffic in 1 direction only

[Marc]

If I do a ping on the ipsec client, I can see it arrive on the destination host with a tcpdump 

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:51:25.694274 IP 192.168.x.3 > 192.168.x.15: ICMP echo request, id 34, seq 278, length 64
18:51:26.677873 IP 192.168.x.3 > 192.168.x.15: ICMP echo request, id 34, seq 279, length 64
18:51:27.716212 IP 192.168.x.3 > 192.168.x.15: ICMP echo request, id 34, seq 280, length 64
18:51:28.722770 IP 192.168.x.3 > 192.168.x.15: ICMP echo request, id 34, seq 281, length 64

However when I do ping on the destination host to the ipset client I do not even see this ping enter the interface of the host where libreswan is running.

How should I resolve this? I have tried a bit with arping and enabling/disabling proxy arp on the host interface, but nothing seems to work
(iptables is off and forwarding is on)