[Swan] XFRM pCPU Load distribution in KVM Muti-queue virtio-net

[Rav Ya]

Hi Antony,

Thank you for your time.

I have been referring to this page (https://libreswan.org/wiki/XFRM_pCPU)
and it doesn't say that XFRM is only supported for ikev2. I am setting up a
shared VTI for 500 Remote Clients IPSec (xAUTH using PAM, IKEv1) tunnels. I
have attached my ipsec.conf at the bottom of this email.

*What I understand from your response: Please correct me*
1. Lbreswan experimental versions only support pCPU with IKEv2. (Lod
balancing one big IPSec flow over multiple vCPUs.)

*Question: *For my use case (500 Clients, xAUTH using PAM, IKEv1 ) the SAs
per client will be created per vCPU.

   - The vCPU will be picked randomly (How will the 500 SAs be
   distributed?) 500/6 = 82 SAs per CPU.
   - There shall be no duplicate SAs for a single connection over multiple
   vCPU because there is no pCPU XFRM. Correct?
   - Is there a way fro me to check how any SAs got allocated to a vCPU on
   my system?

*My Observation: *When I start pushing traffic across all the 500 SAs

   - Some times the load isn't distributed evenly and I see some vCPUs
   geting overutilized and start slowing down the Libreswan packet processing
   rate.
   - The Libreswan server itn't able to process packets fast enough and the
   TAP interface (tx queue) on the KVM virtulization host starts dropping
   packets.

Currently, my ipsec clients are using: ( Any advice?) vCPU is Intel(R)
Xeon(R) Gold 6126 CPU @ 2.60GHz passthrough Host VM
ike=3des-sha1-modp1024
esp=aes256-md5-modp1024

###########################  ipsec.conf  ###########################
config setup
        uniqueids=no

conn %default
        dpdaction=clear
        dpddelay=30s
        dpdtimeout=90s
        ikev2=no
        rekey=no
        ikelifetime=24h
        lifetime=24h
        authby=secret
        leftxauthserver=yes
        rightxauthclient=yes
        xauthby=pam
        left=10.11.251.251
        leftsubnet=0.0.0.0/0
        leftid=@libreswan
        right=%any
        vti-interface=vti01
        vti-routing=yes
        vti-shared=yes
        mark=5/0xffffffff
        replay-window=0
        nic-offload=auto
        type=tunnel
        auto=add

conn strswan1
        rightid=STRSWANAT1
        rightsubnet=10.15.0.0/30

...... 1 through N

  conn strswan500
        rightid=STRSWANAT500
        rightsubnet=10.16.0.0/30


###########################  ipsec.conf  ###########################

On Thu, Sep 17, 2020 at 12:40 PM Antony Antony <antony at phenome.org> wrote:

> On Tue, Sep 15, 2020 at 11:11:30AM -0400, Rav Ya wrote:
> > Hello Everyone,
> >
> > Please advice. Any help will be highly appreciated. Thank you in advance.
> >
> > *Test Setup: *Libreswan Server (Virtual Machine: KVM)
> > 500 IPSec Clients (xAuth using PAM-Auth)
>
> can you share your libreswan config? Where did you get libreswan with
> xauth
> and pCPU support?
>
> The libreswan expirimental versions only support pCPU with IKEv2, without
> CP(or xauth) payload, INTERNAL_IP options. It is meant for data center
> like
> envirment without NAT and without xauth. Just one fat IPsec flow, using
> AES
> GCM, with multiple CPU cores (not hyper threads).
>
> > I am running a Libreswan server in a virtual environment (VM hosted on
> > KVM/oVIRT). The logical network i.e. virtio-net virtual NIC drivers
> > supports Multiqueue. I have 6 vCPUs configured with 6 RX/TX Queue (1
> queue
> > per vCPU).
>
> vCPU and XFRM bottlenecks are hard to debug. What is the host CPU? look at
> cache miss(using kernel perf) and IRQ distributions using mpstat.
>
> > The traffic load balancing over XFRM pCPU is flaky. Initially, the load
> > gets evenly distributed and after a while, only 1 (at most 2) vCPUs get
> > utilized with soft IRQs and the rest go underutilized.
>
> > I read an article that said XFRM pCPU only supports RSS NIC and
> > recently support for vmxnet3 (VMWare) got added. The KVM and virtio
> > Multiqueue was listed under future ideas and worklist.
>
> you are likely mixing up too many things. xauth and RSS can work on its
> own.
> As far as I see you don"t need pCPU for with 500 clients. The idea behind
> pCPU work is traffic for one SA, or a few SAs, split across multiple
> CPUs.
> and pCPU only work upto number CPUs. We were focused on on Intel CPUs
> only,
> with AESNI acceleration. As I see it, it will not work very well vCPU 6
> CPUs
> and 500 clients use case.
>
> > Is there a way to work around this limitation? Is this support available
> > on  the Latest version of Kernel and Libreswan? please advise.
>
> Which crypto cypher is used?  One tip, if it is AES GCM with AESNI
> accelartion aovid hyper threading.
> CPU threads may share AESNI engine, hence lowering performance on vCPU.
> Check your specific CPU model.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libreswan.org/pipermail/swan/attachments/20200921/998fe17d/attachment.html>