Re: [Hampshire] Replication speed - rsync v robocopy

Top Page

Reply to this message
Author: Pierre Cazenave
Date:  
To: Hampshire LUG Discussion List
Subject: Re: [Hampshire] Replication speed - rsync v robocopy
> So for a file system that is compatible both with Linux and Windows,
> I use FAT32 - and if I had to create a partition on a dual-boot
> Linux/Windows box for file storage, then I would also make that FAT32
> as well so that it can be mounted easily in Linux.


An alternative to FAT32 for good cross-platform compatibilty is UDF.
Unless I'm mistaken (which I may well be), Windows XP and more modern
OS's (Linux, Mac OS, Windows Vista and 7 [0]) can read UDF filesystems
(commonly used on DVDs). There is no technical reason you cannot format
a hard disk with UDF, and I have successfully done so. The notable
advantage of using UDF is that is doesn't have the limitations FAT32
does (and it's not patent encumbered either, I think). It also supports
permissions properly and there's no low filesize limit like that in FAT32.

The principal disadvantage of UDF is that you cannot have a UDF parition
with other filesystems on the same disk and expect Windows to recognise
them (Linux and Mac have no such problems); you must devote your entire
device to UDF (i.e. you're formatting /dev/sdb as opposed to /dev/sdb1
or /dev/sdb2).

I've found that formatting my USB sticks with UDF means I can transfer
large files (greater than 2GB) across the three main operating systems
(Windows, Linux and Mac) without installing extra software on any of the
machines.

Pierre

[0]
https://secure.wikimedia.org/wikipedia/en/wiki/Universal_Disk_Format#Table_of_operating_systems

>
> 2. FAT32, by default, has problems with file permissions and can
> never handle a file bigger than 2GB in size. For the second issue,
> the only way around that is to split the file into chunks less than
> 2GB in size, For the file permissions issue, I use rsync parameters
> as follows:
>
> rsync -rltDv --delete --modify-window=1 /linux-source-dir/
> /windows-fat32-dest-dir/
>
> The above stops any rsync errors due to file permissions.
>
> Incidentally, here is how I mount the FAT32 disk onto Linux - note
> that I mount that as my normal user (uid=1000) and group (gid=1000),
> and mounting it in /media puts it into Gnome's automounting control
> and puts an icon on the desktop automatically:
>
> /dev/wd         /media/wd         auto
> users,noatime,noauto,shortname=mixed,uid=1000,gid=1000 0 0

>
> 3. I format my USB drives as FAT32 in Windows, but Microsoft put a
> stupid and meaningless limit on FAT32 partitions which restricts them
> to around 150GB or so (don't quote me on that but it's around that
> level). FAT32 actually supports 4TB (or something similar) for
> partition sizes, the solution therefore is to use a tool called
> "fat32format" and just format the drive at the Windows command line,
> instead of using the Disk Manager.
>
> You can get fat32format free here:
>
> http://www.ridgecrop.demon.co.uk/index.htm?fat32format.htm
>
> 4. I know NTFS is a better filesystem and has journalling built in
> whereas FAT32 doesn't, but in my case I'm backing up to USB hard
> disks that are rarely plugged into a PC or media player to be used
> directly. But it's also worse mentioning that just about every
> camera, phone, media player, etc. out there supports FAT32 but
> probably won't support NTFS. So, personally, I keep NTFS partitions
> exclusively for Windows OS partitions but nothing else.
>
> 5. A few years ago I did play about with DeltaCopy, an rsync GUI
> program for Windows that is built on Cygwin. It did work very well
> and didn't seem to be particularly slow so you may want to take a
> look at that:
>
> http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
>
> It is free, by the way.
>
> ---
>
> I hope that helps and at least gives you some options. I'm certainly
> interested in hearing from other members here how they do this stuff
> because there doesn't seem to be one single solution that suits every
> possible circumstance - but the above is the way I do it and it works
> well for me.
>
> Regards
>
> Peter
>
>
>
>
>
> On 7 August 2011 12:41, Rob Malpass<linux@???>
> wrote:
>
>> Hi all****
>>
>> ** **
>>
>> A bit of expertise on rsync (and seemingly its windows equivalent
>> robocopy) please. I'm trying to build a mirror of this huge store
>> of files I have - currently ~750Gb so any differential copying
>> method will save a huge amount of time as each file is around
>> 10Gb.****
>>
>> ** **
>>
>> After (shamefully admits years of being too frightened by the
>> syntax!) I've just found out that****
>>
>> ** **
>>
>> rsync -avz source dest****
>>
>> ** **
>>
>> is all I need. Therein lies a bit of my concern - I have two
>> nagging problems:****
>>
>> ** **
>>
>> 1) rsync is much faster than robocopy. Also does anyone know of a
>> good GUI for rsync? With around 100Gb to replicate each time I do
>> it - I could use a progress indicator (beyond using the --progress
>> switch).****
>>
>> 2) The files are being created on an XP machine and the filesystem
>> on source and dest is ntfs. As such I'm getting files (when I see
>> them in Nautilus etc) as ****
>>
>> ** **
>>
>> "An /Example/ Of/ Something.mpg"****
>>
>> ** **
>>
>> If rsync is copying these, do I have anything to worry about when I
>> put the dest back into a windows machine?****
>>
>> ** **
>>
>> Cheers****
>>
>> Rob****
>>
>> -- Please post to: Hampshire@??? Web Interface:
>> https://mailman.lug.org.uk/mailman/listinfo/hampshire LUG URL:
>> http://www.hantslug.org.uk
>> --------------------------------------------------------------
>>
>
>
>
> -- Please post to: Hampshire@??? Web Interface:
> https://mailman.lug.org.uk/mailman/listinfo/hampshire LUG URL:
> http://www.hantslug.org.uk
> --------------------------------------------------------------