[Swan-dev] IPSec offload API

Paul Wouters paul at nohats.ca
Wed Nov 30 13:41:41 UTC 2016


On Wed, 30 Nov 2016, Steffen Klassert wrote:

>> Can I ask the use case for this? Why wouldn't hardware offload always be
>> used by the kernel when the kernel detects there is such hardware
>> support available?
>
> There are several reasons why we want to have it configurable
> on a SA basis. First of all, the SADB on the NIC is limited.
> So you don't want to offload the first N negotiated SAs, but
> those that are expected to have high traffic rates.

You say these drivers have a limit to the number of concurrent SA's
the can support - as opposed to a limit to the number of packets
they can process? Where does that limit come from? Does the private
IPsec SA key get pushed into the card? If so, then we also have FIPS
issues to deal with - that is either those drivers need to come
inside the crypto boundary or we have to disable offload in FIPS mode.

> Second, the user should be aware what happens. The systems
> behaviour changes, a user won't see IPsec packets in the stack
> anymore.

The behaviour I was expecting was IPsec processing always works. But
it either gets offloaded or not.

> Third, you might not trust all NICs that can do ESP offload.

It seems like an unlikely corner case that you would trust some
NICs in your machine but not some others. If there is a bug in
one of them, then I'd hope the nic driver has a way of just
disabling the offload feature. I'd rather not see that being
handled in the XFRM API.

Adding tweaks and switches is easy. We already have way too many
of them. I'd really recommend keeping things as simple as possible.

>>> Note that the offloaded XFRM state needs the daemon to explicitly specify the network interface ifindex, the SA direction
>>> (inbound or outbound), and the used L3 protocol (IPv4 or IPv6).
>>> A network interface needs to specify the capability flag NETIF_F_HW_ESP in order to indicate support of ESP crypto offload.
>>> All of this may require some extra logic from the daemon.
>>
>> So is this a new layer of offloading that works separately from any
>> existing hardware offload?
>
> No, it's not really new. It works similar to other network protocol
> offloads.

I meant as compared to existing crypto offload support.

>> Does it only apply to nic cards? What if
>> someone has some dedicated non-NIC PCI offloading card like the
>> Intel QuickAssist?
>
> This is something completely different. Intel QuickAssist does pure
> crypto offload. It is (as far as I know) already supported in the
> crypto layer.

It is. I am just wondering how these different methods will interact,
or at least not interfere with each other.

>>> Lastly, the offloaded XFRM state may be rejected by the driver or kernel, by returning an error from the NEWSA message.
>>> In such a case, the daemon may wish to fall back to non-offloaded XFRM state, or do something else (configurable?)
>>
>> So we install the SA, but it fails and we have to modify and re-install
>> the SA?
>
> No, we don't. The current implementation installs the SA to software
> if the requested hardware offload fails without any user interaction.

Okay, good.

> We probably make this behaviour configurable with come flag in a
> second step, because some users might not want a software fallback
> if the hardware offload fails.

I think that is highly unlikely. I do agree it would be nice if the XFRM
response could tell us if things are working or failing, but I'm not
sure if you can tell that before a single packet encrypt/decrypt has
been attempted. I would be reluctant to add this knob, as I don't really
see the use case of "I'd rather have a broken tunnel than a slow
tunnel".

>> How well does this offloading deal with out-of-order packets and PFS?
>
> We don't get packet reordering with this, it works like it does
> without offloading. Do you refer to Perfect Forward Secrecy with PFS?
> If so, the NIC just handles the ESP packet path. It is not involved
> in the key negotiation. It takes the key from the key deamon, uses
> it and deletes it on request. So I guess there is no impact on this.

Sorry, I did not mean PFS. I meant with respect to the kernel replay
protection. If the kernel can give the nic two packets, and the last
packet for some reason encrypts faster and goes out first, the other
end receives the packets in reverse order. Especially on high speed
links this can severely degrade performance due to the maximum depth
of this out-of-order packets kept for processing. A common issue is
a packet that due to the smaller MTU inside the tunnel gets fragmented
into two packets where the second packet is much smaller than the first,
and so encrypting it takes less time. I guess this depends on the NIC
capabilities and how the kernel drives the nic.

>> Is there a performance difference/limitation when using one IPsec SA
>> versus having many IPsec SA's
>
> Most cpu intensive operations are done by the NIC. So it is not
> a big problem anymore that we have to keep a one SA on one cpu.

On a multi core machine, using multiple CPU's could yield a better
performance than offloading it to a single nic using a single CPU.

> Btw. if you want to spread the crypto operations for one SA over
> multiple cpus, you can use the pcrypt template. It sends the crypto
> request to a configurable set of cpus and guarantees to return
> the crypto requests in the same order they entered. So you can
> have crypto with multiple cpus without packet reordering.

In almost all cases I've seen, pcrypt makes things worse unless you
completely disable replay protection (using replay-window=0)

Paul


More information about the Swan-dev mailing list