<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
I am running some iperf3 bandwidth tests and noticing poor performance on Libreswan v3.19. When I ran similar tests on v3.15, I didn’t run into these performance issues. On v3.15,
<div class="">I would see bandwidth around 700Mbps, where now the max I’ve seen in 300Mbps. It seems like the process ‘ksoftirqd’ is hogging the CPU. I’ve tried playing around with linux irq</div>
<div class="">settings, as well as tx and rx queue length and CPU affinity. No luck. I’m wondering if anyone has experienced this and, if so, how they have handled it.</div>
<div class="">
<div class=""><br class="">
</div>
<div class="">The iperf3 client is on one end of the IPsec tunnel’s local network. The iperf3 server is on the other end’s local network. This is a representation of my setup:</div>
<div class=""><br class="">
</div>
<div class="">iperf3 client <span class="Apple-tab-span" style="white-space:pre">
</span>—> <span class="Apple-tab-span" style="white-space:pre"></span>tunnel endpoint A
<span class="Apple-tab-span" style="white-space:pre"></span> —> <span class="Apple-tab-span" style="white-space:pre">
</span>tunnel endpoint B <span class="Apple-tab-span" style="white-space:pre"></span> —>
<span class="Apple-tab-span" style="white-space:pre"></span>iperf3 server</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span> 10.200.1.1 10.200.1.210 <span class="Apple-tab-span" style="white-space:pre">
</span> 10.200.0.210 10.200.0.92 <span class="Apple-tab-span" style="white-space:pre">
</span> 10.200.2.2 10.200.2.11</div>
<div class=""><br class="">
</div>
<div class="">The iperf3 client command I’m running is iperf3 -c 10.200.2.11 -p 54321 -t 120 -R and the iperf3 server command I’m running is iperf3 -sp 54321. I’ve also tried running</div>
<div class="">with multiple iperf3 client/servers, and the performance results are about the same.</div>
<div class=""><br class="">
</div>
<div class="">Here’s the configuration for tunnel endpoint B:</div>
<div class=""><br class="">
</div>
<div class="">config setup</div>
<div class="">
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>dumpdir=/var/run/pluto/</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>virtual-private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12,%v4:25.0.0.0/8,%v4:100.64.0.0/10,%v6:fd00::/8,%v6:fe80::/10</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>protostack=netkey</div>
<div class=""><br class="">
</div>
<div class=""># begin conn local</div>
<div class="">conn local</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>left=10.200.0.210</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>leftid="@client"</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>leftsubnet=10.200.1.0/24</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>rightid="@is1"</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>rightsubnet=10.200.2.0/24</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>rightcert=server</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>right=10.200.0.92</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>authby=rsasig</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>auto=ignore</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>type=tunnel</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>compress=no</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>pfs=yes</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>ikepad=yes</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>authby=rsasig</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>phase2=esp</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>ikev2=permit</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>esn=no</div>
</div>
<div class=""><br class="">
</div>
<div class="">Here’s the configuration for tunnel endpoint A:</div>
<div class="">
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="">
<div class="">config setup</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>dumpdir=/var/run/pluto/</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>virtual-private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12,%v4:25.0.0.0/8,%v4:100.64.0.0/10,%v6:fd00::/8,%v6:fe80::/10</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>protostack=netkey</div>
<div class=""><br class="">
</div>
<div class=""># begin conn local</div>
<div class="">conn local</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>left=10.200.0.210</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>leftid="@client"</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>leftsubnet=10.200.1.0/24</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>leftcert=client</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>rightid="@is1"</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>rightsubnet=10.200.2.0/24</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>right=10.200.0.92</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>authby=rsasig</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>auto=ignore</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>type=tunnel</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>compress=no</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>pfs=yes</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>ikepad=yes</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>authby=rsasig</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>phase2=esp</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>ikev2=permit</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>esn=no</div>
<div class=""># end conn local</div>
</div>
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="">This is what I’m seeing when I run top on tunnel endpoint A:</div>
<div class=""> PID USER PR NI VIRT RES SHR S <span class="Apple-tab-span" style="white-space:pre">
</span>%CPU <span class="Apple-tab-span" style="white-space:pre"></span>%MEM TIME+
<span class="Apple-tab-span" style="white-space:pre"></span>COMMAND</div>
<div class="">
<div class=""> 13 root 20 0 0 <span class="Apple-tab-span" style="white-space:pre">
</span>0 0 R <span class="Apple-tab-span" style="white-space:pre"> </span>99.7 <span class="Apple-tab-span" style="white-space:pre">
</span>0.0 <span class="Apple-tab-span" style="white-space:pre"> </span>24:32.94
<span class="Apple-tab-span" style="white-space:pre"></span>ksoftirqd/1</div>
<div class=""> 434 root 20 0 0 <span class="Apple-tab-span" style="white-space:pre">
</span>0 0 S <span class="Apple-tab-span" style="white-space:pre"></span>0.3 <span class="Apple-tab-span" style="white-space:pre">
</span>0.0 <span class="Apple-tab-span" style="white-space:pre"></span>0:08.04 <span class="Apple-tab-span" style="white-space:pre">
</span>xfsaild/dm-1</div>
<div class=""> 913 root 0 -20 0 <span class="Apple-tab-span" style="white-space:pre">
</span>0 0 S <span class="Apple-tab-span" style="white-space:pre"></span>0.3 <span class="Apple-tab-span" style="white-space:pre">
</span>0.0 <span class="Apple-tab-span" style="white-space:pre"> </span>0:00.39
<span class="Apple-tab-span" style="white-space:pre"></span>kworker/1:1H</div>
<div class=""> 9683 root 20 0 0 <span class="Apple-tab-span" style="white-space:pre">
</span>0 0 S <span class="Apple-tab-span" style="white-space:pre"></span>0.3 <span class="Apple-tab-span" style="white-space:pre">
</span>0.0 <span class="Apple-tab-span" style="white-space:pre"></span>0:03.94 <span class="Apple-tab-span" style="white-space:pre">
</span>kworker/0:3</div>
<div class="">21067 root 20 0 0 <span class="Apple-tab-span" style="white-space:pre">
</span>0 0 S <span class="Apple-tab-span" style="white-space:pre"></span>0.3 <span class="Apple-tab-span" style="white-space:pre">
</span>0.0 <span class="Apple-tab-span" style="white-space:pre"></span>0:01.26 <span class="Apple-tab-span" style="white-space:pre">
</span>kworker/1:0</div>
<div class="">25264 cmarker 20 0 157696 2216 1556 R <span class="Apple-tab-span" style="white-space:pre">
</span>0.3 <span class="Apple-tab-span" style="white-space:pre"> </span>0.0 <span class="Apple-tab-span" style="white-space:pre">
</span>0:00.76 <span class="Apple-tab-span" style="white-space:pre"></span>top</div>
<div class=""> 1 root 20 0 46168 6684 3956 S <span class="Apple-tab-span" style="white-space:pre">
</span>0.0 <span class="Apple-tab-span" style="white-space:pre"> </span>0.1 <span class="Apple-tab-span" style="white-space:pre">
</span>0:10.36 <span class="Apple-tab-span" style="white-space:pre"></span>systemd</div>
<div class=""> 2 root 20 0 0 <span class="Apple-tab-span" style="white-space:pre">
</span>0 0 S <span class="Apple-tab-span" style="white-space:pre"></span>0.0 <span class="Apple-tab-span" style="white-space:pre">
</span>0.0 <span class="Apple-tab-span" style="white-space:pre"></span>0:00.02 <span class="Apple-tab-span" style="white-space:pre">
</span>kthreadd</div>
</div>
<div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<br class="">
</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
Let me know if any other information would be useful, and I’ll do my best to provide it!</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<br class="">
</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
Thanks!</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
--</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
cm</div>
</div>
<br class="">
</div>
</div>
</body>
</html>