I’ve had a Synology NAS for a very very long time, and just lately I began transferring its contents to a brand new Ubiquiti UniFi UNAS Professional 8. This appeared prefer it must be a reasonably boring operation. Each units converse SMB, I’ve a quick community (just lately upgraded to 10 gigabit internally), and Home windows has had instruments for copying information reliably between machines for many years. Naturally, it changed into an entire night of studying issues I assumed I already knew, which is why I began a weblog lol.
There was a pleasant little bit of historical past right here for me as a result of again in 2007 (good lord!) I wrote a put up known as “XCopy thought of dangerous – Robocopy or XXCopy or SyncBack.” My argument on the time was mainly that after you’re transferring sufficient information, Explorer stops being the transfer and Robocopy begins wanting fairly good. I even used /Z, Robocopy’s restartable mode, as a result of having the ability to resume {a partially} transferred file was helpful on unreliable connections.
Virtually twenty years later, /Z turned out to be probably the most vital issues I wanted to take away as a result of it made all the things hella sluggish.
The migration
The fundamental job was simple. I had shares on the Synology resembling:
servermusic
and matching shares on the UNAS:
UNAS-Professional-8music
I initially used Explorer, principally as a result of it was there and since typically the straightforward factor actually is the straightforward factor. That lasted till Explorer began producing errors on particular person information:
The requested operation couldn't be accomplished as a consequence of a file system limitation
My first thought was filenames. NAS migrations are filled with alternatives to find that one filesystem is extra permissive than one other, and there have been filenames with parentheses and different punctuation in them.
Then this failed:
servermusicAthleteTourist 5 Wires.m4p
There’s nothing particularly unique about 05 Wires.m4p, so I moved over to Robocopy to get just a little extra data. It constantly acquired to 92% and returned Home windows error 665:
92% New File 4.3 m 05 Wires.m4p
ERROR 665 (0x00000299) Copying File
The requested operation couldn't be accomplished as a consequence of a file system limitation
At this level the helpful query was now not “what’s mistaken with that filename?” however “which a part of the trail is refusing this file?”
I copied the file from the Synology to my native Home windows desktop. That labored. I then copied the native file from Home windows to the UNAS, and that failed with the identical filesystem limitation.
That remoted the issue so the Synology might learn the file, Home windows might retailer it, and one thing about scripting this specific file to the UNAS was inflicting hassle.
Alternate Knowledge Streams, once more
NTFS information can include named Alternate Knowledge Streams along with the bizarre unnamed stream that we normally consider because the contents of a file. That is an previous Home windows filesystem function, and it occurs to be one I wrote about in 2007 when discussing Zone.Identifier, which Home windows can use to report the place a downloaded file got here from. I even blogged about Alternate Knowledge Streams in 2003!!! Home windows can expose these streams with DIR /R.
So I ran:
dir /r "%USERPROFILEpercentDesktop 5 Wires.m4p"
and acquired:
11/30/2011 02:17 PM 4,576,368 05 Wires.m4p
360,456 05 Wires.m4p:01APIC_03.jpg:$DATA
There it’s. Alongside the traditional 4.5 MB music file was a roughly 360 KB named knowledge stream known as 01APIC_03.jpg.
That additionally defined the unusual 92% failure. Robocopy was efficiently getting by the primary contents of the file after which encountering the extra stream. What had seemed like a failure someplace in the course of an bizarre .m4p file was truly occurring when Home windows tried to cope with the extra filesystem knowledge.
Robocopy has assist for precisely this example. Microsoft paperwork X as one of many /COPY flags, that means “skip alternate knowledge streams.” So:
/COPY:DATX
means copy the file’s knowledge, attributes, and timestamps, however don’t copy the alternate streams. /DCOPY:DATX applies the corresponding habits to directories. I retried the identical file:
robocopy "servermusicAthleteTourist" "UNAS-Professional-8musicAthleteTourist" "05 Wires.m4p" /R:0 /W:0 /COPY:DATX /DCOPY:DATX /V
and it accomplished efficiently. The vital distinction right here is that DATX doesn’t take away metadata saved inside an MP3, M4A, M4P, JPEG, or different file format. It tells Robocopy to not reproduce separate filesystem streams related to the file. In my case these further streams weren’t one thing I wanted to protect on the brand new NAS.
The copy labored, but it surely was sluggish
As soon as the ADS situation was understood, I began the bigger migration with a reasonably conventional-looking Robocopy command:
robocopy "servermusic" "UNAS-Professional-8music" /E /Z /MT:16 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /TEE /LOG:"%USERPROFILEpercentDesktopsynology-to-unas.log"
It ran, however efficiency was far and wide. Generally I might see a number of hundred megabits per second, then it will drop dramatically. A small file might seem to sit down there for a very long time. I began questioning whether or not I used to be taking a look at buffering, sluggish disks, parity calculations, SMB habits on the UNAS, or possibly my Synology had lastly reached its limits.
So now it is “simply attempt random stuff (bisect)” time. I diminished the variety of threads. I attempted single threaded. None of that helped. Then I eliminated /Z.
Microsoft’s Robocopy documentation describes /Z as restartable mode, which lets an interrupted file resume somewhat than beginning once more from byte zero. What I had forgotten is that Microsoft’s present migration steering particularly warns that /Z needs to be used cautiously as a result of the additional logging required for restartability can considerably scale back copy efficiency.
My profitable music run ended up utilizing:
robocopy "servermusic" "UNAS-Professional-8music" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /TEE /LOG:"%USERPROFILEpercentDesktopsynology-to-unas-DATX.log"
The abstract from that run was:
Whole Copied Skipped Mismatch FAILED
Information : 15940 6991 8949 0 0
Bytes : 70.907 g 47.917 g 22.989 g 0 0
Velocity : 187,185,171 Bytes/sec.
So the run that copied virtually 48 GB of remaining knowledge averaged about 187 MB/sec, with no failed information.
This was not a managed benchmark the place I modified precisely one variable whereas all the things else remained similar, so I’m not going to faux the quantity proves that /Z accounted for each little bit of the sooner slowdown. The sensible distinction was giant sufficient, nevertheless, that /Z is now not one thing I’ll routinely put in a LAN migration command simply because restartability sounds fascinating. On a secure native community I might begin with out it and add it solely after I really need its semantics.
/MT is helpful, but it surely helps a specific type of drawback
Robocopy’s /MT:n choice runs copies utilizing a number of threads. It helps values from 1 by 128, with eight threads because the default if /MT is equipped with no quantity. Microsoft’s personal migration steering additionally factors out that extra threads don’t routinely translate right into a quicker migration and recommends measuring thread counts towards the precise workload.
This made extra sense as soon as I finished pondering of /MT:4 as “make one file 4 occasions quicker.”
Think about a music assortment with 1000’s of information of questionable provenance (I ripped them, simply kidding). There’s work related to opening a file, creating the vacation spot file, studying and writing its contents, coping with metadata, and shutting it once more. A single-threaded copy has durations the place the community or storage might be ready whereas a type of operations completes. Having a number of information in progress without delay provides Robocopy alternatives to overlap that work.
For this specific assortment, 4 threads turned out to be match. Sixteen wasn’t clearly serving to extra, and one thread wasn’t an enchancment. I might resist turning /MT right into a magic worth that belongs in each command line, as a result of a listing containing 50,000 images presents a unique workload from 4 900 GB disk photos.
There’s additionally a logging price value remembering. Microsoft recommends redirecting Robocopy output to a log when utilizing multithreaded copies, and its migration steering makes use of switches resembling /NP, /NFL, and /NDL when the target is throughput somewhat than watching each filename scroll by.
For a migration I’m not actively watching, I might most likely use one thing like:
robocopy "servershare" "UNAS-Professional-8share" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /NP /NFL /NDL /LOG:"%USERPROFILEpercentDesktopnas-migration.log"
Then got here the big information
Later I began copying some information that had been a whole lot of gigabytes every. Microsoft describes /J as unbuffered I/O and recommends it for giant information, so it appeared like the apparent choice to attempt.
With /J enabled, nevertheless, NAS-to-NAS switch slowed dramatically. The helpful factor about having a Home windows machine within the center is that I might check every half of the journey individually. I took one of many very same giant information and copied it instantly from the Synology to my native machine. That ran at roughly 250 MB/sec, so the Synology was completely able to studying the file at excessive pace.
I then eliminated /J from the direct Synology-to-UNAS Robocopy command:
robocopy "servershare" "UNAS-Professional-8share" "huge-file.ext" /R:0 /W:0 /COPY:DATX /NP
and the pace got here again.
I don’t assume the helpful conclusion is that /J is dangerous. Microsoft recommends it for large-file copies for a purpose, and it’s solely potential that it’s precisely what you need when copying from native disk to native disk or in one other community configuration. What mattered right here was that Home windows was concurrently studying from one SMB server and writing to a different SMB server, and on this specific path buffered I/O carried out a lot better.
That may be a good reminder that command-line switches describe habits, not assured efficiency enhancements. /J modifications the I/O mannequin. /MT modifications concurrency. /Z provides restartability. Whether or not these modifications enhance a migration is dependent upon the remainder of the system.
The Synology was quicker than I gave it credit score for
At a number of factors I blamed the ageing Synology. It’s an previous machine with spinning disks, so it was straightforward to imagine that a number of hundred megabits per second was merely all it had left. Then I remembered that the Synology has 4 1 GbE interfaces and that SMB 3 helps Multichannel. I’m nonetheless shocked this labored so properly.
SMB Multichannel permits an SMB session to make use of a number of community paths concurrently. Microsoft paperwork this particularly as a strategy to combination out there community bandwidth, and Synology helps SMB3 Multichannel for a similar purpose.
Home windows makes the energetic channels straightforward to examine:
Get-SmbMultichannelConnection -ServerName server |
Format-Desk ServerName,Chosen,ClientIpAddress,ServerIpAddress,ClientLinkSpeed,ServerLinkSpeed,CurrentChannels
My machine reported:
ServerName Chosen ClientIpAddress ServerIpAddress ClientLinkSpeed ServerLinkSpeed
---------- -------- --------------- --------------- --------------- ---------------
server True 192.168.1.45 192.168.1.210 1000000000 1000000000
server True 192.168.1.45 192.168.1.198 1000000000 1000000000
server True 192.168.1.45 192.168.1.197 1000000000 1000000000
server True 192.168.1.45 192.168.1.26 1000000000 1000000000
All 4 1 GbE interfaces on the Synology had been taking part within the SMB connection.
My Home windows machine at present has a 2.5 GbE adapter (10 gig coming quickly), and in the course of the quick copy I used to be seeing roughly 250 MB/sec arriving from the Synology. That out of the blue made the habits of the system a lot much less mysterious. The Synology was not restricted to the throughput of 1 gigabit Ethernet connection as a result of SMB Multichannel was permitting Home windows to make use of the 4 out there server-side paths, whereas the two.5 GbE hyperlink on the PC was changing into the smaller community pipe.
Synology’s documentation makes an vital distinction right here between SMB Multichannel and bizarre hyperlink aggregation. Multichannel can enhance SMB efficiency for one shopper by utilizing a number of community connections, whereas standard hyperlink aggregation is usually about combination throughput throughout a number of shoppers and providers.
Like I mentioned, I’ve a ten GbE adapter on the best way for the Home windows machine, so there may be one other experiment out there after the migration. The Synology nonetheless solely has 4 1 GbE interfaces, which supplies it 4 Gb/sec of community hyperlinks in combination, however eradicating the present 2.5 GbE shopper bottleneck ought to present how a lot farther the disks and the Synology itself can go. For an older NAS that I had already mentally demoted to “the sluggish backup NAS,” it carried out surprisingly properly.
I additionally tried rsync
Sure, I do know, what about rsync? You’re saying Home windows as a intermediary is pointless. The Synology can present rsync, and UniFi Drive can pull from an rsync server utilizing daemon mode. Ubiquiti paperwork the rsync path beneath Drive’s Backup Duties and requires daemon mode for this kind of supply.
I attempted a separate film share with rsync whereas the opposite experiments had been happening. It labored, and I noticed about 67 MB/sec. It additionally gave me a vacation spot format with some extra listing nesting that I would wish to wash up afterward.
That isn’t an argument that rsync is sluggish generally, nor that its listing habits can’t be configured appropriately. It’s simply what occurred on this specific Synology-to-UNAS check. As soon as the Robocopy path was reaching roughly 187 MB/sec and preserving precisely the UNC share format I wished, there wasn’t a lot incentive for me to make rsync the first migration mechanism.
The marginally amusing consequence was that the apparently oblique path:
Synology -> SMB -> Home windows -> SMB -> UNAS
was significantly quicker in my surroundings than asking the 2 NAS units to switch the check share instantly with the rsync implementation uncovered by UniFi Drive. My guess is as a result of rsync wasn’t utilizing the 4 1gig connections linked. Let me know what you assume within the feedback.
The Robocopy command I ended up with
For the traditional shares containing numerous information, that is the model I might begin with now:
robocopy "servershare" "UNAS-Professional-8share" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /NP /NFL /NDL /LOG:"%USERPROFILEpercentDesktopnas-migration.log"
The switches have pretty particular jobs:
/E Copy subdirectories, together with empty ones
/MT:4 Enable 4 file-copy threads
/R:2 Retry a failed copy twice
/W:2 Wait two seconds between retries
/COPY:DATX Copy knowledge, attributes and timestamps, however skip ADS
/DCOPY:DATX Apply the corresponding listing copy flags
/XJ Exclude junction factors
/NP Do not print proportion progress
/NFL Do not log each filename
/NDL Do not log each listing
/LOG Write the helpful output to a file
I intentionally do not need /Z in there. I additionally wouldn’t routinely add /J; for a workload dominated by very giant information I might check the identical switch each with and with out it earlier than committing to a multi-terabyte run.
The /MT worth is equally empirical. 4 labored extraordinarily properly for this Synology and this mixture of information, however I might attempt 1, 4, 8, or one other smart quantity towards a consultant slice of the true knowledge somewhat than assuming that the most important out there thread rely is finest.
Checking the consequence
For the information the place I care sufficient to show that the vacation spot comprises precisely the identical bytes because the supply, PowerShell’s Get-FileHash is a handy closing examine:
Get-FileHash "serversharehuge-file.ext" -Algorithm SHA256
Get-FileHash "UNAS-Professional-8sharehuge-file.ext" -Algorithm SHA256
Get-FileHash makes use of SHA-256 by default, and if the SHA-256 values match then the 2 inputs produced the identical digest. For big information this requires studying the whole file once more at each ends, so I’m unlikely to hash each tune in a music assortment, however it’s a straightforward strategy to confirm the notably worthwhile multi-hundred-gigabyte information after a migration.
Just a few issues I’d examine earlier than blaming the NAS
What made this migration attention-grabbing was that the signs might have supported a number of believable explanations. The UNAS has a brand new RAID array, the Synology is previous, Home windows is appearing as an SMB shopper in each instructions, the information got here from years of various functions, and Robocopy has sufficient switches to make virtually any command line look authoritative.
When a file failed, I copied it Synology-to-local after which local-to-UNAS, which uncovered the destination-side ADS drawback. When giant information had been sluggish, I copied the identical file Synology-to-local, which proved the previous NAS might nonetheless ship about 250 MB/sec. When the community out of the blue grew to become a lot quicker, Get-SmbMultichannelConnection confirmed that each one 4 Synology Ethernet interfaces had been taking part. When /J seemed sluggish, eradicating simply that habits restored the throughput I used to be anticipating.
The ultimate efficiency was not the results of discovering a secret “quick Robocopy” command from a discussion board. It got here from fascinated about which options I truly wished for this migration and eradicating a number of that had been helpful in different circumstances however costly in mine.
That’s most likely the half I’ll need to keep in mind the following time I do that. /Z, /J, and /MT are usually not ranges on a efficiency slider. They modify restartability, buffering, and concurrency. Alternate Knowledge Streams are actual knowledge even when Explorer usually hides them. SMB Multichannel could make an previous NAS with a number of gigabit interfaces rather more succesful than one would possibly assume from taking a look at any single Ethernet port.
Robocopy ended up being the quickest factor I attempted. As with all recommendation, this labored for me. Ideally you may gind extra worth within the feedback as Hacker Information of us and Home windows consultants will drop in with higher instruments and techniques. Simply keep in mind, there’s multiple strategy to saturate a community and I fully saturated this one, so I am fairly proud of the results of my migration.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, marketing consultant, father, diabetic, and Microsoft worker. He’s a failed stand-up comedian, a cornrower, and a e-book creator.
