[Swan-dev] IPSec offload API

[Paul Wouters]

On Thu, 1 Dec 2016, Steffen Klassert wrote:

>> You say these drivers have a limit to the number of concurrent SA's
>> the can support - as opposed to a limit to the number of packets
>> they can process? Where does that limit come from? Does the private
>> IPsec SA key get pushed into the card?
>
> Yes.
>
>> If so, then we also have FIPS
>> issues to deal with - that is either those drivers need to come
>> inside the crypto boundary or we have to disable offload in FIPS mode.
>
> I'm not so familiar when FIPS mode allows, but
> be probably have to disable offloading in FIPS mode.

Interesting. So that does mean we have to ensure we are not running in
FIPS mode before we send the XFRM offload commands.

>>> Second, the user should be aware what happens. The systems
>>> behaviour changes, a user won't see IPsec packets in the stack
>>> anymore.
>>
>> The behaviour I was expecting was IPsec processing always works. But
>> it either gets offloaded or not.
>
> IPsec processing works always, but you don't have encrypted IPsec
> packets in the stack anymore because the NIC does this, before
> the packet enters the stack on receive and after the packets leaves
> the stack on transmit.

Are you saying that the user won't see ESP packets when using tcpdump?
Even when using VTI? That would be confusing and would have to be
documented in the man page for such an option.

>>> Third, you might not trust all NICs that can do ESP offload.
>>
>> It seems like an unlikely corner case that you would trust some
>> NICs in your machine but not some others.
>
> This is a valid case in high security environments. The NIC
> has the key and can do whatever it want with it. So somebody
> might just trust the NIC if he knows the crypto code that is
> running on the NIC.

Okay.

>>
>>>> Does it only apply to nic cards? What if
>>>> someone has some dedicated non-NIC PCI offloading card like the
>>>> Intel QuickAssist?
>>>
>>> This is something completely different. Intel QuickAssist does pure
>>> crypto offload. It is (as far as I know) already supported in the
>>> crypto layer.
>>
>> It is. I am just wondering how these different methods will interact,
>> or at least not interfere with each other.
>
> If we do crypto offload to the NIC, we don't use the crypto layer.
> We do the ESP encapsulation in the stack and defer the crypto
> operations to the NIC by just not calling the crypto layer.

Does this mean any related options have no more effect either? Such as
replay-window size and replay protection and UDP encapsulation? Does
the nic support all of those? Do we need to separately configure those
or are they going to be part of the XFRM command set? I guess what you
are saying is that you are completely bypassing esp4.c/esp6.c from the
kernel? Would we still get ACQUIRE's when max packet count or max life
is reached?

>>> We probably make this behaviour configurable with come flag in a
>>> second step, because some users might not want a software fallback
>>> if the hardware offload fails.
>>
>> I think that is highly unlikely.
>
> This is also a valid case in high security environments. In
> some cases the crypto operation _must_ be processed by a
> special NIC, software crypto or in particular AESNI is not
> allowed.

Okay, understood.

>>>
>>> Most cpu intensive operations are done by the NIC. So it is not
>>> a big problem anymore that we have to keep a one SA on one cpu.
>>
>> On a multi core machine, using multiple CPU's could yield a better
>> performance than offloading it to a single nic using a single CPU.
>
> As Ilan mentioned, this NIC offloading can be magnitudes faster
> than everything you can get with spreading traffic of one SA over
> multiple cores.

Excellent :)

>>> Btw. if you want to spread the crypto operations for one SA over
>>> multiple cpus, you can use the pcrypt template. It sends the crypto
>>> request to a configurable set of cpus and guarantees to return
>>> the crypto requests in the same order they entered. So you can
>>> have crypto with multiple cpus without packet reordering.
>>
>> In almost all cases I've seen, pcrypt makes things worse unless you
>> completely disable replay protection (using replay-window=0)
>
> I have the opposite experience here. We Don't get any packet reorder
> with that.

Interesting. I guess these cards are not cheap of-the-shelves cards that
I can get to play with? :)

Paul