I have been shifting my residence storage over to a UniFi UNAS Professional 8 as half of a bigger homelab moderization. My essential IRONHEART (the last word PC from a couple of years again) desktop now has an Intel E610-XT2 10GbE card, the NAS is on 10GbE, and there is a Minisforum MS-01 miniPC on the identical community with a 10GbE SFP+ connection working Immich and Portainer and some different issues.
The whole lot says 10 gigabit. Home windows says 10 gigabit. UniFi says 10 gigabit. SMB copies are utilizing the right NIC however my file copies are working at round 100 to 200 megabits per second which is unhappy making.
Naturally, I blamed the NAS, and the spinning rust inside. The UNAS has six 16 TB spinning disks in RAID 6 and a pair of NVMe SSDs getting used as cache. I am additionally working Immich on the MS-01, with its photograph library residing on the UNAS, so there are many thumbnails, metadata reads, and little background writes occurring. All appear to be cheap suspects.
I switched the UNAS SSD cache from read-write to read-only. No significant distinction. I finished Immich utterly. No distinction. I checked out iostat; the disks weren’t saturated. We checked out SMB signing and Home windows Defender community scanning. Nonetheless gradual.
Then I finished testing the NAS and ran iperf3 instantly between the Home windows desktop and the MS-01:
iperf3 -c 192.168.1.222 -P 4133 Mbit/sec
Oops. The reverse take a look at was higher, however nonetheless improper:
iperf3 -c 192.168.1.222 -P 4 -R1.33 Gbit/sec
That is bizarre. Now the disks, SMB, Immich, RAID, and the NAS itself had been utterly out of the equation. This was a Home windows/NIC drawback and it is weirdly asymmetrical.
Trying on the Intel adapter statistics reveals me…
Get-NetAdapterStatistics -Identify "Ethernet - 10 Gig Intel"There have been almost one million ReceivedDiscardedPackets. Throughout one ten-second iperf3 take a look at, the counter elevated by one other 268. Why?
The E610 driver had its obtain buffers on the default 512, though it supported as much as 4096. I elevated them. I like an elevated buffer.
Set-NetAdapterAdvancedProperty `
-Identify "Ethernet - 10 Gig Intel" `
-DisplayName "Obtain Buffers" `
-DisplayValue "4096"The discarded-packet rely in the course of the subsequent take a look at went from 268 to zero, and obtain throughput jumped from 1.33 Gbit/sec to five.15 Gbit/sec. The transmit route was nonetheless horrible, principally 313 Mbit/sec. The subsequent experiment was disabling Giant Ship Offload (LSO) V2 for IPv4:
Set-NetAdapterAdvancedProperty `
-Identify "Ethernet - 10 Gig Intel" `
-DisplayName "Giant Ship Offload V2 (IPv4)" `
-DisplayValue "Disabled"Then I ran the identical iperf3 take a look at once more.
7.03 Gbit/sec
That is not a typo. 313 Mbit/sec to 7.03 Gbit/sec by altering one NIC setting. Candy sassy molassey.
LSO exists for a superb purpose: Home windows can hand giant TCP buffers to the NIC and let the adapter/driver section them into network-sized packets, decreasing CPU work. Microsoft does, nevertheless, explicitly level out that segmentation offload can cut back most sustainable throughput with some community adapters/configurations. LSO is normally helpful, however not on this case.
In my specific mixture of Home windows and the Intel E610-XT2, one thing within the IPv4 LSO path was very, very unhappy. I do not but know whether or not that is an Intel driver bug, firmware challenge, Home windows interplay, or one thing specific to this machine, so I would not flip this into random tech blogger recommendation that everybody ought to disable LSO. Measure first, minimize as soon as. Er, twice. Simply keep woke.
Lastly I went again to the take a look at that began all this and copied the identical giant file to the UNAS and Robocopy reported:
Pace : 350,201,354 Bytes/sec.
Pace : 20,038.682 MegaBytes/min.About 350 MB/sec, or 2.8 Gbit/sec of sustained real-world SMB writes to a six-disk RAID 6 NAS.
That is way more prefer it. The helpful lesson is not “disable LSO.” It was that when storage is mysteriously gradual, finally you need to cease testing storage. iperf3 eliminated the NAS, filesystem, RAID, cache, SMB and disks from the experiment in a single transfer. As soon as the uncooked community was additionally gradual, the issue turned dramatically smaller. And, typically the little checkbox labeled Giant Ship Offload is able to making your 10 gigabit card run prefer it’s 2004.
TL;DR – with LSO V2 for IPv4 enabled, Home windows-to-Linux iperf3 managed about 313 Mbit/sec. Turning off that single offload took the very same take a look at to 7.03 Gbit/sec. I’m intentionally saying on this machine as a result of LSO is often helpful and this isn’t blanket recommendation to disable it in all places.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, marketing consultant, father, diabetic, and Microsoft worker. He’s a failed stand-up comedian, a cornrower, and a e-book creator.
