[Swan] XFRM pCPU Load distribution in KVM Muti-queue virtio-net

[Antony Antony]

On Tue, Sep 15, 2020 at 11:11:30AM -0400, Rav Ya wrote:
> Hello Everyone,
> 
> Please advice. Any help will be highly appreciated. Thank you in advance.
> 
> *Test Setup: *Libreswan Server (Virtual Machine: KVM)
> 500 IPSec Clients (xAuth using PAM-Auth)

can you share your libreswan config? Where did you get libreswan with xauth 
and pCPU support? 

The libreswan expirimental versions only support pCPU with IKEv2, without 
CP(or xauth) payload, INTERNAL_IP options. It is meant for data center like 
envirment without NAT and without xauth. Just one fat IPsec flow, using AES 
GCM, with multiple CPU cores (not hyper threads).

> I am running a Libreswan server in a virtual environment (VM hosted on
> KVM/oVIRT). The logical network i.e. virtio-net virtual NIC drivers
> supports Multiqueue. I have 6 vCPUs configured with 6 RX/TX Queue (1 queue
> per vCPU).

vCPU and XFRM bottlenecks are hard to debug. What is the host CPU? look at 
cache miss(using kernel perf) and IRQ distributions using mpstat.

> The traffic load balancing over XFRM pCPU is flaky. Initially, the load
> gets evenly distributed and after a while, only 1 (at most 2) vCPUs get
> utilized with soft IRQs and the rest go underutilized. 

> I read an article that said XFRM pCPU only supports RSS NIC and
> recently support for vmxnet3 (VMWare) got added. The KVM and virtio
> Multiqueue was listed under future ideas and worklist.

you are likely mixing up too many things. xauth and RSS can work on its own.  
As far as I see you don"t need pCPU for with 500 clients. The idea behind 
pCPU work is traffic for one SA, or a few SAs, split across multiple CPUs.  
and pCPU only work upto number CPUs. We were focused on on Intel CPUs only, 
with AESNI acceleration. As I see it, it will not work very well vCPU 6 CPUs 
and 500 clients use case.

> Is there a way to work around this limitation? Is this support available 
> on  the Latest version of Kernel and Libreswan? please advise.

Which crypto cypher is used?  One tip, if it is AES GCM with AESNI 
accelartion aovid hyper threading.
CPU threads may share AESNI engine, hence lowering performance on vCPU.  
Check your specific CPU model.