I’ve had a Synology NAS for a very very long time, and lately I began shifting its contents to a brand new Ubiquiti UniFi UNAS Professional 8. This appeared prefer it must be a reasonably boring operation. Each units converse SMB, I’ve a quick community (lately upgraded to 10 gigabit internally), and Home windows has had instruments for copying recordsdata reliably between machines for many years. Naturally, it changed into an entire night of studying issues I assumed I already knew, which is why I began a weblog lol.
There was a pleasant little bit of historical past right here for me as a result of again in 2007 (good lord!) I wrote a submit known as “XCopy thought of dangerous – Robocopy or XXCopy or SyncBack.” My argument on the time was mainly that when you might be shifting sufficient recordsdata, Explorer stops being the transfer and Robocopy begins wanting fairly good. I even used /Z, Robocopy’s restartable mode, as a result of having the ability to resume {a partially} transferred file was helpful on unreliable connections.
Virtually twenty years later, /Z turned out to be one of the crucial vital issues I wanted to take away as a result of it made all the pieces hella sluggish.
The migration
The fundamental job was easy. I had shares on the Synology corresponding to:
servermusic
and matching shares on the UNAS:
UNAS-Professional-8music
I initially used Explorer, largely as a result of it was there and since generally the simple factor actually is the simple factor. That lasted till Explorer began producing errors on particular person recordsdata:
The requested operation couldn't be accomplished as a consequence of a file system limitation
My first thought was filenames. NAS migrations are stuffed with alternatives to find that one filesystem is extra permissive than one other, and there have been filenames with parentheses and different punctuation in them.
Then this failed:
servermusicAthleteTourist 5 Wires.m4p
There’s nothing particularly unique about 05 Wires.m4p, so I moved over to Robocopy to get a bit of extra data. It persistently received to 92% and returned Home windows error 665:
92% New File 4.3 m 05 Wires.m4p
ERROR 665 (0x00000299) Copying File
The requested operation couldn't be accomplished as a consequence of a file system limitation
At this level the helpful query was not “what’s flawed with that filename?” however “which a part of the trail is refusing this file?”
I copied the file from the Synology to my native Home windows desktop. That labored. I then copied the native file from Home windows to the UNAS, and that failed with the identical filesystem limitation.
That remoted the issue so the Synology may learn the file, Home windows may retailer it, and one thing about scripting this specific file to the UNAS was inflicting bother.
Alternate Knowledge Streams, once more
NTFS recordsdata can comprise named Alternate Knowledge Streams along with the extraordinary unnamed stream that we often consider because the contents of a file. That is an outdated Home windows filesystem function, and it occurs to be one I wrote about in 2007 when discussing Zone.Identifier, which Home windows can use to document the place a downloaded file got here from. I even blogged about Alternate Knowledge Streams in 2003!!! Home windows can expose these streams with DIR /R.
So I ran:
dir /r "%USERPROFILEpercentDesktop 5 Wires.m4p"
and received:
11/30/2011 02:17 PM 4,576,368 05 Wires.m4p
360,456 05 Wires.m4p:01APIC_03.jpg:$DATA
There it’s. Alongside the traditional 4.5 MB music file was a roughly 360 KB named knowledge stream known as 01APIC_03.jpg.
That additionally defined the unusual 92% failure. Robocopy was efficiently getting by means of the principle contents of the file after which encountering the extra stream. What had seemed like a failure someplace in the course of an extraordinary .m4p file was really occurring when Home windows tried to take care of the extra filesystem knowledge.
Robocopy has help for precisely this example. Microsoft paperwork X as one of many /COPY flags, that means “skip alternate knowledge streams.” So:
/COPY:DATX
means copy the file’s knowledge, attributes, and timestamps, however don’t copy the alternate streams. /DCOPY:DATX applies the corresponding conduct to directories. I retried the identical file:
robocopy "servermusicAthleteTourist" "UNAS-Professional-8musicAthleteTourist" "05 Wires.m4p" /R:0 /W:0 /COPY:DATX /DCOPY:DATX /V
and it accomplished efficiently. The vital distinction right here is that DATX doesn’t take away metadata saved inside an MP3, M4A, M4P, JPEG, or different file format. It tells Robocopy to not reproduce separate filesystem streams related to the file. In my case these further streams weren’t one thing I wanted to protect on the brand new NAS.
The copy labored, nevertheless it was sluggish
As soon as the ADS situation was understood, I began the bigger migration with a reasonably conventional-looking Robocopy command:
robocopy "servermusic" "UNAS-Professional-8music" /E /Z /MT:16 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /TEE /LOG:"%USERPROFILEpercentDesktopsynology-to-unas.log"
It ran, however efficiency was in every single place. Generally I might see a couple of hundred megabits per second, then it could drop dramatically. A small file may seem to take a seat there for a very long time. I began questioning whether or not I used to be taking a look at buffering, sluggish disks, parity calculations, SMB conduct on the UNAS, or possibly my Synology had lastly reached its limits.
So now it is “simply attempt random stuff (bisect)” time. I lowered the variety of threads. I attempted single threaded. None of that helped. Then I eliminated /Z.
Microsoft’s Robocopy documentation describes /Z as restartable mode, which lets an interrupted file resume relatively than beginning once more from byte zero. What I had forgotten is that Microsoft’s present migration steerage particularly warns that /Z ought to be used cautiously as a result of the additional logging required for restartability can considerably scale back copy efficiency.
My profitable music run ended up utilizing:
robocopy "servermusic" "UNAS-Professional-8music" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /TEE /LOG:"%USERPROFILEpercentDesktopsynology-to-unas-DATX.log"
The abstract from that run was:
Complete Copied Skipped Mismatch FAILED
Information : 15940 6991 8949 0 0
Bytes : 70.907 g 47.917 g 22.989 g 0 0
Pace : 187,185,171 Bytes/sec.
So the run that copied nearly 48 GB of remaining knowledge averaged about 187 MB/sec, with no failed recordsdata.
This was not a managed benchmark the place I modified precisely one variable whereas all the pieces else remained equivalent, so I’m not going to fake the quantity proves that /Z accounted for each little bit of the sooner slowdown. The sensible distinction was massive sufficient, nevertheless, that /Z is not one thing I’ll robotically put in a LAN migration command simply because restartability sounds fascinating. On a steady native community I might begin with out it and add it solely after I really want its semantics.
/MT is beneficial, nevertheless it helps a selected type of drawback
Robocopy’s /MT:n choice runs copies utilizing a number of threads. It helps values from 1 by means of 128, with eight threads because the default if /MT is equipped with out a quantity. Microsoft’s personal migration steerage additionally factors out that extra threads don’t robotically translate right into a sooner migration and recommends measuring thread counts towards the precise workload.
This made extra sense as soon as I finished pondering of /MT:4 as “make one file 4 instances sooner.”
Think about a music assortment with 1000’s of recordsdata of questionable provenance (I ripped them, simply kidding). There’s work related to opening a file, creating the vacation spot file, studying and writing its contents, coping with metadata, and shutting it once more. A single-threaded copy has intervals the place the community or storage might be ready whereas a type of operations completes. Having a number of recordsdata in progress directly offers Robocopy alternatives to overlap that work.
For this specific assortment, 4 threads turned out to be a great match. Sixteen wasn’t clearly serving to extra, and one thread wasn’t an enchancment. I might resist turning /MT right into a magic worth that belongs in each command line, as a result of a listing containing 50,000 pictures presents a special workload from 4 900 GB disk photos.
There’s additionally a logging price price remembering. Microsoft recommends redirecting Robocopy output to a log when utilizing multithreaded copies, and its migration steerage makes use of switches corresponding to /NP, /NFL, and /NDL when the target is throughput relatively than watching each filename scroll by.
For a migration I’m not actively watching, I might in all probability use one thing like:
robocopy "servershare" "UNAS-Professional-8share" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /NP /NFL /NDL /LOG:"%USERPROFILEpercentDesktopnas-migration.log"
Then got here the big recordsdata
Later I began copying some recordsdata that had been tons of of gigabytes every. Microsoft describes /J as unbuffered I/O and recommends it for giant recordsdata, so it appeared like the plain choice to attempt.
With /J enabled, nevertheless, NAS-to-NAS switch slowed dramatically. The helpful factor about having a Home windows machine within the center is that I may check every half of the journey individually. I took one of many very same massive recordsdata and copied it straight from the Synology to my native machine. That ran at roughly 250 MB/sec, so the Synology was completely able to studying the file at excessive velocity.
I then eliminated /J from the direct Synology-to-UNAS Robocopy command:
robocopy "servershare" "UNAS-Professional-8share" "huge-file.ext" /R:0 /W:0 /COPY:DATX /NP
and the velocity got here again.
I don’t suppose the helpful conclusion is that /J is dangerous. Microsoft recommends it for large-file copies for a cause, and it’s solely doable that it’s precisely what you need when copying from native disk to native disk or in one other community configuration. What mattered right here was that Home windows was concurrently studying from one SMB server and writing to a different SMB server, and on this specific path buffered I/O carried out significantly better.
That may be a good reminder that command-line switches describe conduct, not assured efficiency enhancements. /J modifications the I/O mannequin. /MT modifications concurrency. /Z provides restartability. Whether or not these modifications enhance a migration is determined by the remainder of the system.
The Synology was sooner than I gave it credit score for
At a number of factors I blamed the ageing Synology. It’s an outdated machine with spinning disks, so it was simple to imagine that a couple of hundred megabits per second was merely all it had left. Then I remembered that the Synology has 4 1 GbE interfaces and that SMB 3 helps Multichannel. I’m nonetheless shocked this labored so nicely.
SMB Multichannel permits an SMB session to make use of a number of community paths concurrently. Microsoft paperwork this particularly as a option to combination accessible community bandwidth, and Synology helps SMB3 Multichannel for a similar cause.
Home windows makes the lively channels simple to examine:
Get-SmbMultichannelConnection -ServerName server |
Format-Desk ServerName,Chosen,ClientIpAddress,ServerIpAddress,ClientLinkSpeed,ServerLinkSpeed,CurrentChannels
My machine reported:
ServerName Chosen ClientIpAddress ServerIpAddress ClientLinkSpeed ServerLinkSpeed
---------- -------- --------------- --------------- --------------- ---------------
server True 192.168.1.45 192.168.1.210 1000000000 1000000000
server True 192.168.1.45 192.168.1.198 1000000000 1000000000
server True 192.168.1.45 192.168.1.197 1000000000 1000000000
server True 192.168.1.45 192.168.1.26 1000000000 1000000000
All 4 1 GbE interfaces on the Synology had been taking part within the SMB connection.
My Home windows machine presently has a 2.5 GbE adapter (10 gig coming quickly), and throughout the quick copy I used to be seeing roughly 250 MB/sec arriving from the Synology. That all of the sudden made the conduct of the system a lot much less mysterious. The Synology was not restricted to the throughput of 1 gigabit Ethernet connection as a result of SMB Multichannel was permitting Home windows to make use of the 4 accessible server-side paths, whereas the two.5 GbE hyperlink on the PC was changing into the smaller community pipe.
Synology’s documentation makes an vital distinction right here between SMB Multichannel and extraordinary hyperlink aggregation. Multichannel can improve SMB efficiency for one shopper by utilizing a number of community connections, whereas standard hyperlink aggregation is mostly about combination throughput throughout a number of purchasers and companies.
Like I stated, I’ve a ten GbE adapter on the way in which for the Home windows machine, so there’s one other experiment accessible after the migration. The Synology nonetheless solely has 4 1 GbE interfaces, which provides it 4 Gb/sec of community hyperlinks in combination, however eradicating the present 2.5 GbE shopper bottleneck ought to present how a lot farther the disks and the Synology itself can go. For an older NAS that I had already mentally demoted to “the sluggish backup NAS,” it carried out surprisingly nicely.
I additionally tried rsync
Sure, I do know, what about rsync? You’re saying Home windows as a intermediary is pointless. The Synology can present rsync, and UniFi Drive can pull from an rsync server utilizing daemon mode. Ubiquiti paperwork the rsync path beneath Drive’s Backup Duties and requires daemon mode for the sort of supply.
I attempted a separate film share with rsync whereas the opposite experiments had been happening. It labored, and I noticed about 67 MB/sec. It additionally gave me a vacation spot format with some extra listing nesting that I would want to wash up afterward.
That’s not an argument that rsync is sluggish normally, nor that its listing conduct can’t be configured appropriately. It’s simply what occurred on this specific Synology-to-UNAS check. As soon as the Robocopy path was reaching roughly 187 MB/sec and preserving precisely the UNC share format I wished, there wasn’t a lot incentive for me to make rsync the first migration mechanism.
The marginally amusing consequence was that the apparently oblique path:
Synology -> SMB -> Home windows -> SMB -> UNAS
was significantly sooner in my atmosphere than asking the 2 NAS units to switch the check share straight with the rsync implementation uncovered by UniFi Drive. My guess is as a result of rsync wasn’t utilizing the 4 1gig connections linked. Let me know what you suppose within the feedback.
The Robocopy command I ended up with
For the traditional shares containing a lot of recordsdata, that is the model I might begin with now:
robocopy "servershare" "UNAS-Professional-8share" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /NP /NFL /NDL /LOG:"%USERPROFILEpercentDesktopnas-migration.log"
The switches have pretty particular jobs:
/E Copy subdirectories, together with empty ones
/MT:4 Enable 4 file-copy threads
/R:2 Retry a failed copy twice
/W:2 Wait two seconds between retries
/COPY:DATX Copy knowledge, attributes and timestamps, however skip ADS
/DCOPY:DATX Apply the corresponding listing copy flags
/XJ Exclude junction factors
/NP Do not print proportion progress
/NFL Do not log each filename
/NDL Do not log each listing
/LOG Write the helpful output to a file
I intentionally do not need /Z in there. I additionally wouldn’t robotically add /J; for a workload dominated by very massive recordsdata I might check the identical switch each with and with out it earlier than committing to a multi-terabyte run.
The /MT worth is equally empirical. 4 labored extraordinarily nicely for this Synology and this mixture of recordsdata, however I might attempt 1, 4, 8, or one other smart quantity towards a consultant slice of the actual knowledge relatively than assuming that the biggest accessible thread depend is finest.
Checking the consequence
For the recordsdata the place I care sufficient to show that the vacation spot accommodates precisely the identical bytes because the supply, PowerShell’s Get-FileHash is a handy last examine:
Get-FileHash "serversharehuge-file.ext" -Algorithm SHA256
Get-FileHash "UNAS-Professional-8sharehuge-file.ext" -Algorithm SHA256
Get-FileHash makes use of SHA-256 by default, and if the SHA-256 values match then the 2 inputs produced the identical digest. For big recordsdata this requires studying your entire file once more at each ends, so I’m unlikely to hash each music in a music assortment, however it’s a simple option to confirm the notably beneficial multi-hundred-gigabyte recordsdata after a migration.
A number of issues I’d examine earlier than blaming the NAS
What made this migration attention-grabbing was that the signs may have supported a number of believable explanations. The UNAS has a brand new RAID array, the Synology is outdated, Home windows is appearing as an SMB shopper in each instructions, the recordsdata got here from years of various functions, and Robocopy has sufficient switches to make nearly any command line look authoritative.
When a file failed, I copied it Synology-to-local after which local-to-UNAS, which uncovered the destination-side ADS drawback. When massive recordsdata had been sluggish, I copied the identical file Synology-to-local, which proved the outdated NAS may nonetheless ship about 250 MB/sec. When the community all of the sudden grew to become a lot sooner, Get-SmbMultichannelConnection confirmed that each one 4 Synology Ethernet interfaces had been taking part. When /J seemed sluggish, eradicating simply that conduct restored the throughput I used to be anticipating.
The ultimate efficiency was not the results of discovering a secret “quick Robocopy” command from a discussion board. It got here from fascinated about which options I really wished for this migration and eradicating a couple of that had been helpful in different circumstances however costly in mine.
That’s in all probability the half I’ll wish to keep in mind the following time I do that. /Z, /J, and /MT should not ranges on a efficiency slider. They modify restartability, buffering, and concurrency. Alternate Knowledge Streams are actual knowledge even when Explorer usually hides them. SMB Multichannel could make an outdated NAS with a number of gigabit interfaces far more succesful than one may assume from taking a look at any single Ethernet port.
Robocopy ended up being the quickest factor I attempted. As with all recommendation, this labored for me. Ideally you may gind extra worth within the feedback as Hacker Information of us and Home windows consultants will drop in with higher instruments and methods. Simply keep in mind, there’s multiple option to saturate a community and I fully saturated this one, so I am fairly proud of the results of my migration.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, guide, father, diabetic, and Microsoft worker. He’s a failed stand-up comedian, a cornrower, and a e-book creator.
