I could do with some help getting my bulk file copy to work, please.
Scenario
I have tens of GB of photos that I would like to upload to my servage account overnight when my ADSL bandwidth is free. The photos get added to on a regular basis and I would like any additions to be copied up automatically overnight. It will take many nights to complete the initial upload so the process has to cope with being stopped and restarted. If I am ever stupid enough to delete any files or folders locally I want to ensure that the delete is not propagated to the server.
Attempted Solution
Use curlftpfs to mount my servage space as a local directory and then use rsync to copy the files over.
curlftpfs
*******:$$$$$$$@ftp.servage.net /mnt/ftp
rsync -a /root/pictures /mnt/ftp
Problem
This worked once for a short while for a few hundred MB but now when ever I try to run it I it fails. Everything is OK while it is check existing directories and it starts uploading the first photo and then fails. I think this is at the point that it is finishing the first photo.
There an rsync patch on samba.org for using fsync() to close the file, I think.
fsync() patch
http://samba.org/ftp/unpacked/rsync/patches/fsync.diffuse
It would probably be quicker to write my own script using just ftp, but I wanted to use the available tools.
Does anyone have any views on whether if i compile this in it might help or any other hints?
I know "compile and see", I just thought I'd ask before going to the bother. Is this even a sensible strategy architecturally?
*********************************************************
output from rsync
*********************************************************
sh-3.00# rsync -h
rsync version 2.6.4 protocol version 29
sh-3.00# rsync -vai --bwlimit=50 --stats /root/pictures /mnt/ftp
building file list ... done
.d..t.... pictures/
.d..t.... pictures/digital camera/
.d..t.... pictures/digital camera/01/
>f+++++++ pictures/digital camera/01/IMGP0013.JPG
>f+++++++ pictures/digital camera/01/IMGP0014.JPG
rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown" [sender]: Broken pipe (32)
rsync: close failed on "/mnt/ftp/pictures/digital camera/01/.IMGP0013.JPG.0lwcXm": Operation not permitted (1)
rsync error: error in file IO (code 11) at receiver.c(642)
rsync: connection unexpectedly closed (538 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(420)
rsync: connection unexpectedly closed (70 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(420)
****************************************************
output from curlftpfs running at the same time, started first!
****************************************************
sh-3.00# curlftpfs -V
curlftpfs 0.9 libcurl/7.15.4 fuse/2.5
sh-3.00# curlftpfs -d
*******:$$$$$$$@ftp.servage.net /mnt/ftp
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.7
INIT: 7.5
unique: 1, error: 0 (Success), outsize: 40
unique: 2, opcode: STATFS (17), nodeid: 0, insize: 40
...
FLUSH[134774616]
unique: 537, error: -1 (Operation not permitted), outsize: 16
unique: 538, opcode: RELEASE (18), nodeid: 5, insize: 56
RELEASE[134774616] flags: 0x8002
unique: 538, error: 0 (Success), outsize: 16
unique: 539, opcode: LOOKUP (1), nodeid: 1, insize: 49
LOOKUP /pictures
NODEID: 2
unique: 539, error: 0 (Success), outsize: 136
unique: 540, opcode: LOOKUP (1), nodeid: 2, insize: 55
LOOKUP /pictures/digital camera
NODEID: 3
unique: 540, error: 0 (Success), outsize: 136
unique: 541, opcode: LOOKUP (1), nodeid: 3, insize: 43
LOOKUP /pictures/digital camera/01
NODEID: 4
unique: 541, error: 0 (Success), outsize: 136
unique: 542, opcode: LOOKUP (1), nodeid: 4, insize: 61
LOOKUP /pictures/digital camera/01/.IMGP0013.JPG.0lwcXm
NODEID: 5
unique: 542, error: 0 (Success), outsize: 136
unique: 543, opcode: UNLINK (10), nodeid: 4, insize: 61
UNLINK /pictures/digital camera/01/.IMGP0013.JPG.0lwcXm
unique: 543, error: 0 (Success), outsize: 16
unique: 544, opcode: FORGET (2), nodeid: 5, insize: 48
FORGET 5/2
delete: 5
unique: 545, opcode: STATFS (17), nodeid: 0, insize: 40
unique: 545, error: 0 (Success), outsize: 96
unique: 546, opcode: STATFS (17), nodeid: 0, insize: 40
unique: 546, error: 0 (Success), outsize: 96
unique: 547, opcode: STATFS (17), nodeid: 0, insize: 40
unique: 547, error: 0 (Success), outsize: 96
TIA
_________________
Will
%20to%20close%20the%20file,%20I%20think.%0A>%20fsync()%20patch%0A>%20http://samba.org/ftp/unpacked/rsync/patches/fsync.diffuse%0A>%20%0A>%20It%20would%20probably%20be%20quicker%20to%20write%20my%20own%20script%20using%20just%20ftp,%20but%20I%20wanted%20to%20use%20the%20available%20tools.%0A>%20Does%20anyone%20have%20any%20views%20on%20whether%20if%20i%20compile%20this%20in%20it%20might%20help%20or%20any%20other%20hints?%0A>%20I%20know%20%22compile%20and%20see%22,%20I%20just%20thought%20I'd%20ask%20before%20going%20to%20the%20bother.%20Is%20this%20even%20a%20sensible%20strategy%20architecturally?%0A>%20%0A>%20>f+++++++%20pictures/digital%20camera/01/IMGP0013.JPG%0A>%20>f+++++++%20pictures/digital%20camera/01/IMGP0014.JPG%0A>%20rsync:%20writefd_unbuffered%20failed%20to%20write%204%20bytes:%20phase%20%22unknown%22%20%5Bsender%5D:%20Broken%20pipe%20(32)%0A>%20rsync:%20close%20failed%20on%20%22/mnt/ftp/pictures/digital%20camera/01/.IMGP0013.JPG.0lwcXm%22:%20Operation%20not%20permitted%20(1)%0A>%20rsync%20error:%20error%20in%20file%20IO%20(code%2011)%20at%20receiver.c(642)%0A>%20rsync:%20connection%20unexpectedly%20closed%20(538%20bytes%20received%20so%20far)%20%5Bgenerator%5D%0A>%20rsync%20error:%20error%20in%20rsync%20protocol%20data%20stream%20(code%2012)%20at%20io.c(420)%0A>%20rsync:%20connection%20unexpectedly%20closed%20(70%20bytes%20received%20so%20far)%20%5Bsender%5D%0A>%20rsync%20error:%20error%20in%20rsync%20protocol%20data%20stream%20(code%2012)%20at%20io.c(420)%0A>%20%0A>%20****************************************************%0A>%20output%20from%20curlftpfs%20running%20at%20the%20same%20time,%20started%20first!%0A>%20****************************************************%0A>%20sh-3.00#%20curlftpfs%20-V%0A>%20curlftpfs%200.9%20libcurl/7.15.4%20fuse/2.5%0A>%20sh-3.00#%20curlftpfs%20-d%20*******:%24%24%24%24%24%24%24@ftp.servage.net%20/mnt/ftp%0A>%20unique:%201,%20opcode:%20INIT%20(26),%20nodeid:%200,%20insize:%2056%0A>%20INIT:%207.7%0A>%20%20%20%20INIT:%207.5%0A>%20%20%20%20unique:%201,%20error:%200%20(Success),%20outsize:%2040%0A>%20unique:%202,%20opcode:%20STATFS%20(17),%20nodeid:%200,%20insize:%2040%0A>%20...%0A>%20FLUSH%5B134774616%5D%0A>%20%20%20%20unique:%20537,%20error:%20-1%20(Operation%20not%20permitted),%20outsize:%2016%0A>%20unique:%20538,%20opcode:%20RELEASE%20(18),%20nodeid:%205,%20insize:%2056%0A>%20RELEASE%5B134774616%5D%20flags:%200x8002%0A>%20%20%20%20unique:%20538,%20error:%200%20(Success),%20outsize:%2016%0A>%20unique:%20539,%20opcode:%20LOOKUP%20(1),%20nodeid:%201,%20insize:%2049%0A>%20LOOKUP%20/pictures%0A>%20%20%20%20NODEID:%202%0A>%20%20%20%20unique:%20539,%20error:%200%20(Success),%20outsize:%20136%0A>%20unique:%20540,%20opcode:%20LOOKUP%20(1),%20nodeid:%202,%20insize:%2055%0A>%20LOOKUP%20/pictures/digital%20camera%0A>%20%20%20%20NODEID:%203%0A>%20%20%20%20unique:%20540,%20error:%200%20(Success),%20outsize:%20136%0A>%20unique:%20541,%20opcode:%20LOOKUP%20(1),%20nodeid:%203,%20insize:%2043%0A>%20LOOKUP%20/pictures/digital%20camera/01%0A>%20%20%20%20NODEID:%204%0A>%20%20%20%20unique:%20541,%20error:%200%20(Success),%20outsize:%20136%0A>%20unique:%20542,%20opcode:%20LOOKUP%20(1),%20nodeid:%204,%20insize:%2061%0A>%20LOOKUP%20/pictures/digital%20camera/01/.IMGP0013.JPG.0lwcXm%0A>%20%20%20%20NODEID:%205%0A>%20%20%20%20unique:%20542,%20error:%200%20(Success),%20outsize:%20136%0A>%20unique:%20543,%20opcode:%20UNLINK%20(10),%20nodeid:%204,%20insize:%2061%0A>%20UNLINK%20/pictures/digital%20camera/01/.IMGP0013.JPG.0lwcXm%0A>%20%20%20%20unique:%20543,%20error:%200%20(Success),%20outsize:%2016%0A>%20unique:%20544,%20opcode:%20FORGET%20(2),%20nodeid:%205,%20insize:%2048%0A>%20FORGET%205/2%0A>%20delete:%205%0A>%20unique:%20545,%20opcode:%20STATFS%20(17),%20nodeid:%200,%20insize:%2040%0A>%20%20%20%20unique:%20545,%20error:%200%20(Success),%20outsize:%2096%0A>%20unique:%20546,%20opcode:%20STATFS%20(17),%20nodeid:%200,%20insize:%2040%0A>%20%20%20%20unique:%20546,%20error:%200%20(Success),%20outsize:%2096%0A>%20unique:%20547,%20opcode:%20STATFS%20(17),%20nodeid:%200,%20insize:%2040%0A>%20%20%20%20unique:%20547,%20error:%200%20(Success),%20outsize:%2096%0A>%20%0A>%20TIA%0A>%20_________________%0A>%20Will%0A>%20%0A>%20%0A>%20%0A>%20 "Reply to this message")
