[Swan-dev] IPSec offload API

[Paul Wouters]

On Mon, 28 Nov 2016, Ilan Tayari wrote:

> I would like to introduce the new kernel API for IPSec crypto offload, and open discussion regarding its support in swan daemons.
>
> This is currently just an RFC in Linux net-next, but it is stabilizing and will be submitted soon enough.
> Steffen Klassert has been working on this for a while now, and I have helped with testing and the driver for Mellanox hardware.

Thanks for reaching out to the libreswan developers!

> For anyone who is interested, here is the kernel code:
> https://git.kernel.org/cgit/linux/kernel/git/klassert/linux-stk.git/log/?h=net-next-ipsec-offload-api5
> And a version of our driver:
> https://github.com/Mellanox/ipsec-offload/tree/net-next-ipsec-offload-api3-squash
> Branches get rebased and updated every once in a while.
>
> To sum up, a new XFRM state attribute is introduced: XFRMA_OFFLOAD_DEV
> Along with the following struct:
>
> struct xfrm_user_offload {
>       int                             ifindex;
>       __u8                            flags;
> };
> #define XFRM_OFFLOAD_IPV6      1
> #define XFRM_OFFLOAD_INBOUND   2
>
> Regarding the swan daemon, we expect the user to configure HW offload explicitly (maybe per-SA, or maybe globally)
> Then the daemon will apply this attribute to the XFRM states that it wishes to offload.

Can I ask the use case for this? Why wouldn't hardware offload always be
used by the kernel when the kernel detects there is such hardware
support available? I could understand a global proc/sys value for
enabling or disabling it, but I have never seen a use case where
someone wants only some of their IPsec SA's offloaded, but not
others.

> Note that the offloaded XFRM state needs the daemon to explicitly specify the network interface ifindex, the SA direction
> (inbound or outbound), and the used L3 protocol (IPv4 or IPv6).
> A network interface needs to specify the capability flag NETIF_F_HW_ESP in order to indicate support of ESP crypto offload.
> All of this may require some extra logic from the daemon.

So is this a new layer of offloading that works separately from any
existing hardware offload? Does it only apply to nic cards? What if
someone has some dedicated non-NIC PCI offloading card like the
Intel QuickAssist?
https://www-ssl.intel.com/content/www/us/en/ethernet-products/gigabit-server-adapters/quickassist-adapter-for-servers.html

> Lastly, the offloaded XFRM state may be rejected by the driver or kernel, by returning an error from the NEWSA message.
> In such a case, the daemon may wish to fall back to non-offloaded XFRM state, or do something else (configurable?)

So we install the SA, but it fails and we have to modify and re-install
the SA? I don't see the userland ever wanting to say "if no
offload, we do not want to install this SA at all", so I don't see
the value of throwing an error only to get installed anyway without
offload. From a userland point of view it is really "install
this SA and use offload if available".

> Even if the offloaded XFRM state is accepted, the kernel might have not used the offloaded attribute.
> Therefore the daemon is expected to read back the SA, if it wishes to report to the user that it is indeed offloading.

This feels like some bad bandaid to me. So now we have to do like 3 or 4
things to deal with various possible failure modes? Why do failures
happen anyway? And again, what possible choice does userland have
other than to reluctantly tell the kernel "fine, do it without
offload" ?

> Any comments regarding this are welcome.

How well does this offloading deal with out-of-order packets and PFS?

Is there a performance difference/limitation when using one IPsec SA
versus having many IPsec SA's (current CPU/AESNI acceleration has
limits of putting one SA on one CPU only, and if you tweak it to
use multiple CPU's the out-of-order problem causes so many problems
the SA ends up getting less throughput)

How can we detect at runtime that the kernel supports this new XFRM API,
without a try+fail cycle?

> I would love to see support for all of this in libreswan as well.

We'd be happy to support it, although the failure-then-retry API is
something I'm very reluctant about. I'd rather send a flag to the
kernel to tell it what to do on offload failure, so we keep our
current API of sending 1 XFRM message and getting 1 XFRM answer.

Paul