Monthly Archives: November 2013

Segmented file transfer over ssh

Many people would say why don't you use rsync? rsync is, in deed, a wonderful little tool that has a lot of features, but it doesn't support segmented file transfer. Well, there are a lot of software applications out there that can handle segmented file transfers over FTP or HTTP protocol. One of them is prozilla. But, as rsync, prozilla doesn't support SFTP protocol. So, how can we handle this? The answer has four letters: lftp. Quote from lftp man page:

Gets  the specified file using several connections. This can speed up transfer, but loads the net and server heavily impacting other users. Use only if you really have to transfer the file ASAP

[codesyntax lang="bash"]

lftp sftp://user[:password]@host.ro[:port] -e "mirror -c --parallel=5 --use-pget-n=5 \"/path/to/folder/\""

[/codesyntax]

Very simple and effective, right?

(Very) Later edit:

[codesyntax lang="bash"]

echo 'set sftp:connect-program "ssh -a -x -i /full/path/to/the/ssh/private/key"' | tee ~/.lftprc
lftp -u user,xxx sftp://compxtreme.ro:6646 -e "mirror -c --parallel=10 --use-pget-n=10 \"/path/to/folder/\"; quit"

[/codesyntax]

 

Where:

  • user is your username on the system you connect to
  • xxx is just a junk password