Re: [Hampshire] Replication speed - rsync v robocopy

Top Page

Reply to this message
Author: RobinT Catling
Date:  
To: Hampshire LUG Discussion List
Subject: Re: [Hampshire] Replication speed - rsync v robocopy
Before I character-assassinate UDF, anyone had cross-platform compatibility
issues with it lately?

I used to use UDF heavily years ago as it is a good utility file-system - on
paper. However, trying to read-write UDF across Windows (both XP, Vista) and
just about any version of Linux began to fall apart, I suspect because MS
deviated in it's interpretation of the UDF spec.

Mounting a UDF formatted disk on one platform and getting no FAT after a
write on the 'other' platform can be quite a distressing experience.

I haven't tried it under Windows 7.

Rgds
RC

Robin Catling
Full Circle Podcast


On 8 August 2011 10:46, Pierre Cazenave <pwcazenave@???> wrote:

> So for a file system that is compatible both with Linux and Windows,
>> I use FAT32 - and if I had to create a partition on a dual-boot
>> Linux/Windows box for file storage, then I would also make that FAT32
>> as well so that it can be mounted easily in Linux.
>>
>
> An alternative to FAT32 for good cross-platform compatibilty is UDF. Unless
> I'm mistaken (which I may well be), Windows XP and more modern OS's (Linux,
> Mac OS, Windows Vista and 7 [0]) can read UDF filesystems (commonly used on
> DVDs). There is no technical reason you cannot format a hard disk with UDF,
> and I have successfully done so. The notable advantage of using UDF is that
> is doesn't have the limitations FAT32 does (and it's not patent encumbered
> either, I think). It also supports permissions properly and there's no low
> filesize limit like that in FAT32.
>
> The principal disadvantage of UDF is that you cannot have a UDF parition
> with other filesystems on the same disk and expect Windows to recognise them
> (Linux and Mac have no such problems); you must devote your entire device to
> UDF (i.e. you're formatting /dev/sdb as opposed to /dev/sdb1 or /dev/sdb2).
>
> I've found that formatting my USB sticks with UDF means I can transfer
> large files (greater than 2GB) across the three main operating systems
> (Windows, Linux and Mac) without installing extra software on any of the
> machines.
>
> Pierre
>
> [0] https://secure.wikimedia.org/**wikipedia/en/wiki/Universal_**
> Disk_Format#Table_of_**operating_systems<https://secure.wikimedia.org/wikipedia/en/wiki/Universal_Disk_Format#Table_of_operating_systems>
>
>
>> 2. FAT32, by default, has problems with file permissions and can
>> never handle a file bigger than 2GB in size. For the second issue,
>> the only way around that is to split the file into chunks less than
>> 2GB in size, For the file permissions issue, I use rsync parameters
>> as follows:
>>
>> rsync -rltDv --delete --modify-window=1 /linux-source-dir/
>> /windows-fat32-dest-dir/
>>
>> The above stops any rsync errors due to file permissions.
>>
>> Incidentally, here is how I mount the FAT32 disk onto Linux - note
>> that I mount that as my normal user (uid=1000) and group (gid=1000),
>> and mounting it in /media puts it into Gnome's automounting control
>> and puts an icon on the desktop automatically:
>>
>> /dev/wd         /media/wd         auto
>> users,noatime,noauto,**shortname=mixed,uid=1000,gid=**1000 0 0

>>
>> 3. I format my USB drives as FAT32 in Windows, but Microsoft put a
>> stupid and meaningless limit on FAT32 partitions which restricts them
>> to around 150GB or so (don't quote me on that but it's around that
>> level). FAT32 actually supports 4TB (or something similar) for
>> partition sizes, the solution therefore is to use a tool called
>> "fat32format" and just format the drive at the Windows command line,
>> instead of using the Disk Manager.
>>
>> You can get fat32format free here:
>>
>> http://www.ridgecrop.demon.co.**uk/index.htm?fat32format.htm<http://www.ridgecrop.demon.co.uk/index.htm?fat32format.htm>
>>
>> 4. I know NTFS is a better filesystem and has journalling built in
>> whereas FAT32 doesn't, but in my case I'm backing up to USB hard
>> disks that are rarely plugged into a PC or media player to be used
>> directly. But it's also worse mentioning that just about every
>> camera, phone, media player, etc. out there supports FAT32 but
>> probably won't support NTFS. So, personally, I keep NTFS partitions
>> exclusively for Windows OS partitions but nothing else.
>>
>> 5. A few years ago I did play about with DeltaCopy, an rsync GUI
>> program for Windows that is built on Cygwin. It did work very well
>> and didn't seem to be particularly slow so you may want to take a
>> look at that:
>>
>> http://www.aboutmyip.com/**AboutMyXApp/DeltaCopy.jsp<http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp>
>>
>> It is free, by the way.
>>
>> ---
>>
>> I hope that helps and at least gives you some options. I'm certainly
>> interested in hearing from other members here how they do this stuff
>> because there doesn't seem to be one single solution that suits every
>> possible circumstance - but the above is the way I do it and it works
>> well for me.
>>
>> Regards
>>
>> Peter
>>
>>
>>
>>
>>
>> On 7 August 2011 12:41, Rob Malpass<linux@???.**co.uk<linux@???>
>> >
>> wrote:
>>
>> Hi all****
>>>
>>> ** **
>>>
>>> A bit of expertise on rsync (and seemingly its windows equivalent
>>> robocopy) please. I'm trying to build a mirror of this huge store
>>> of files I have - currently ~750Gb so any differential copying
>>> method will save a huge amount of time as each file is around
>>> 10Gb.****
>>>
>>> ** **
>>>
>>> After (shamefully admits years of being too frightened by the
>>> syntax!) I've just found out that****
>>>
>>> ** **
>>>
>>> rsync -avz source dest****
>>>
>>> ** **
>>>
>>> is all I need. Therein lies a bit of my concern - I have two
>>> nagging problems:****
>>>
>>> ** **
>>>
>>> 1) rsync is much faster than robocopy. Also does anyone know of a
>>> good GUI for rsync? With around 100Gb to replicate each time I do
>>> it - I could use a progress indicator (beyond using the --progress
>>> switch).****
>>>
>>> 2) The files are being created on an XP machine and the filesystem
>>> on source and dest is ntfs. As such I'm getting files (when I see
>>> them in Nautilus etc) as ****
>>>
>>> ** **
>>>
>>> "An /Example/ Of/ Something.mpg"****
>>>
>>> ** **
>>>
>>> If rsync is copying these, do I have anything to worry about when I
>>> put the dest back into a windows machine?****
>>>
>>> ** **
>>>
>>> Cheers****
>>>
>>> Rob****
>>>
>>> -- Please post to: Hampshire@??? Web Interface:
>>> https://mailman.lug.org.uk/**mailman/listinfo/hampshire<https://mailman.lug.org.uk/mailman/listinfo/hampshire>LUG URL:
>>> http://www.hantslug.org.uk
>>> ------------------------------**------------------------------**--
>>>
>>>
>>
>>
>> -- Please post to: Hampshire@??? Web Interface:
>>
>> https://mailman.lug.org.uk/**mailman/listinfo/hampshire<https://mailman.lug.org.uk/mailman/listinfo/hampshire>LUG URL:
>> http://www.hantslug.org.uk
>> ------------------------------**------------------------------**--
>>
>
> --
> Please post to: Hampshire@???
> Web Interface: https://mailman.lug.org.uk/**mailman/listinfo/hampshire<https://mailman.lug.org.uk/mailman/listinfo/hampshire>
> LUG URL: http://www.hantslug.org.uk
> ------------------------------**------------------------------**--
>




--
Rgds
RC

Robin Catling
Full Circle Podcast