lftp hangs with a 127 bytes file

Discussion:

SanskritFritz

2013-02-11 12:45:27 UTC

Hi
I'm using lftp to mirror my encfs encrypted pictures collection up to
an ftp server. The process was smooth for many months, uploading more
than 20 GB to the server. However since lftp version 4.4.1 lftp hangs
at a 127 byte file (or any other file for that matter).
I'm on archlinux, here is my upgrade log:
[2013-01-31 15:21] upgraded lftp (4.4.0-1 -> 4.4.1-1)
[2013-02-05 15:23] upgraded lftp (4.4.1-1 -> 4.4.2-1)
[2013-02-07 22:13] upgraded lftp (4.4.2-1 -> 4.4.3-1)

The last entry in my transfer.log is at 2013-01-31 18:38:09. Most
probably there was no reboot beween the first upgrade and this entry.

My lftp config is the following:

livedrive_sync.lftp:
open ftp://***@gmail.com:***@ftp.livedrive.com
mirror --continue --reverse --verbose=3 /Common/Pictures/.Family_encfs
/Backup/Common/Pictures/.Family_encfs
exit

the command I use from cron or manually:
lftp -f livedrive_sync.lftp
I also make sure from script that only one instance is running.

When I start the above command, this is the output:

Mirroring directory `,N,0y,npHM3DCiIaBMcM3dj6GZ-2uWp,oSa-gpikM4R5W0'
Mirroring directory `,XQ-VO6SpbTgVrGzdoBQwvoBw1lOn1flR7QhWn6HbAwT11'
...snip some more mirrorings...
Mirroring directory
`066pC4PWtiSpw17jUleYhDY2gEmwP7U9iq8,oPyTbYa4Xhfvm0pTCX1MMh97abSAlI1'
Old file `066pC4PWtiSpw17jUleYhDY2gEmwP7U9iq8,oPyTbYa4Xhfvm0pTCX1MMh97abSAlI1/PQ2dR3XG5bI3,cAL9v5-QZiC'
is not removed
Old file `066pC4PWtiSpw17jUleYhDY2gEmwP7U9iq8,oPyTbYa4Xhfvm0pTCX1MMh97abSAlI1/TGucn-6ukN9NT3-ane1QTO3e'
is not removed
Old file `066pC4PWtiSpw17jUleYhDY2gEmwP7U9iq8,oPyTbYa4Xhfvm0pTCX1MMh97abSAlI1/uM7OnzdOhlgquW0RXGBk6otW'
is not removed
Old file `066pC4PWtiSpw17jUleYhDY2gEmwP7U9iq8,oPyTbYa4Xhfvm0pTCX1MMh97abSAlI1/zGvI41nILsi,72OTPAaYhv1e'
is not removed
Mirroring directory `0EtqIOBrV4Av19qvpx4lwQJEULjnpOn9CFcHzhoNFOF810'
Mirroring directory
`0Fv-ZGvYTKKVTRMbHrD-OhKK63NCad-UQR0RbaITwt1ISzM2S1mqBMC72Ra,GdzMeGB'
...snip some more mirrorings...
Mirroring directory `43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1'
Transferring file
`43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1/5liCn6A-ZmHAynHQBW1mnNaj'

and there the process hangs. Nethogs shows zero network activity on lftp.
The size of the file in question is only 127 bytes:

ll /Common/Pictures/.Family_encfs/43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1/5liCn6A-ZmHAynHQBW1mnNaj
-rw-r--r-- 1 sans users 127 2010-09-10 23:32
/Common/Pictures/.Family_encfs/43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1/5liCn6A-ZmHAynHQBW1mnNaj

--log=FILE tells me that lftp is trying to execute this command:
get -O ftp://myusername%40gmail.com:***@ftp.livedrive.com/FrankBackup/Common/Pictures/.Family_encfs/43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1
file:/Common/Pictures/.Family_encfs/43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1/5liCn6A-ZmHAynHQBW1mnNaj

strace gives these 3 lines over and over again:
poll([{fd=6, events=POLLIN}], 1, 1000) = 0 (Timeout)
read(6, 0x2145e70, 65536) = -1 EAGAIN (Resource
temporarily unavailable)
read(6, 0x2145e70, 65536) = -1 EAGAIN (Resource
temporarily unavailable)

It is probably worth to mention that this file is very old, it should
have been already up on the server for a very long time, so here is
something I don't understand, either the file was deleted server side,
or it never got uploaded on the first hand. This theory is supported
by transfer.log never mentioning that file. The server side directory
is writable, I successfully uploaded the file manually through
curlftpfs.

Please help me finding the problem, I suspect also a server problem,
because curlftpfs also complains sometimes when I copy files. I'm
willing to debug, but it was a very long time ago when I last used a
debugger.
Thanks in advance.
SanskritFritz

SanskritFritz

2013-02-11 13:12:26 UTC

Permalink

Post by SanskritFritz
Hi
I'm using lftp to mirror my encfs encrypted pictures collection up to
an ftp server. The process was smooth for many months, uploading more
than 20 GB to the server. However since lftp version 4.4.1 lftp hangs
at a 127 byte file (or any other file for that matter).
[2013-01-31 15:21] upgraded lftp (4.4.0-1 -> 4.4.1-1)
[2013-02-05 15:23] upgraded lftp (4.4.1-1 -> 4.4.2-1)
[2013-02-07 22:13] upgraded lftp (4.4.2-1 -> 4.4.3-1)
The last entry in my transfer.log is at 2013-01-31 18:38:09. Most
probably there was no reboot beween the first upgrade and this entry.
mirror --continue --reverse --verbose=3 /Common/Pictures/.Family_encfs
/Backup/Common/Pictures/.Family_encfs
exit
lftp -f livedrive_sync.lftp
I also make sure from script that only one instance is running.
Mirroring directory `,N,0y,npHM3DCiIaBMcM3dj6GZ-2uWp,oSa-gpikM4R5W0'
Mirroring directory `,XQ-VO6SpbTgVrGzdoBQwvoBw1lOn1flR7QhWn6HbAwT11'
...snip some more mirrorings...
Mirroring directory
`066pC4PWtiSpw17jUleYhDY2gEmwP7U9iq8,oPyTbYa4Xhfvm0pTCX1MMh97abSAlI1'
Old file `066pC4PWtiSpw17jUleYhDY2gEmwP7U9iq8,oPyTbYa4Xhfvm0pTCX1MMh97abSAlI1/PQ2dR3XG5bI3,cAL9v5-QZiC'
is not removed
Old file `066pC4PWtiSpw17jUleYhDY2gEmwP7U9iq8,oPyTbYa4Xhfvm0pTCX1MMh97abSAlI1/TGucn-6ukN9NT3-ane1QTO3e'
is not removed
Old file `066pC4PWtiSpw17jUleYhDY2gEmwP7U9iq8,oPyTbYa4Xhfvm0pTCX1MMh97abSAlI1/uM7OnzdOhlgquW0RXGBk6otW'
is not removed
Old file `066pC4PWtiSpw17jUleYhDY2gEmwP7U9iq8,oPyTbYa4Xhfvm0pTCX1MMh97abSAlI1/zGvI41nILsi,72OTPAaYhv1e'
is not removed
Mirroring directory `0EtqIOBrV4Av19qvpx4lwQJEULjnpOn9CFcHzhoNFOF810'
Mirroring directory
`0Fv-ZGvYTKKVTRMbHrD-OhKK63NCad-UQR0RbaITwt1ISzM2S1mqBMC72Ra,GdzMeGB'
...snip some more mirrorings...
Mirroring directory `43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1'
Transferring file
`43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1/5liCn6A-ZmHAynHQBW1mnNaj'
and there the process hangs. Nethogs shows zero network activity on lftp.
ll /Common/Pictures/.Family_encfs/43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1/5liCn6A-ZmHAynHQBW1mnNaj
-rw-r--r-- 1 sans users 127 2010-09-10 23:32
/Common/Pictures/.Family_encfs/43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1/5liCn6A-ZmHAynHQBW1mnNaj
file:/Common/Pictures/.Family_encfs/43V2pRjgrVNw4JbCpvHXWiQrnB8dM0Rh2jETRsqdAv1ID1/5liCn6A-ZmHAynHQBW1mnNaj
poll([{fd=6, events=POLLIN}], 1, 1000) = 0 (Timeout)
read(6, 0x2145e70, 65536) = -1 EAGAIN (Resource
temporarily unavailable)
read(6, 0x2145e70, 65536) = -1 EAGAIN (Resource
temporarily unavailable)
It is probably worth to mention that this file is very old, it should
have been already up on the server for a very long time, so here is
something I don't understand, either the file was deleted server side,
or it never got uploaded on the first hand. This theory is supported
by transfer.log never mentioning that file. The server side directory
is writable, I successfully uploaded the file manually through
curlftpfs.
Please help me finding the problem, I suspect also a server problem,
because curlftpfs also complains sometimes when I copy files. I'm
willing to debug, but it was a very long time ago when I last used a
debugger.

Just a quick followup, I downgraded lftp to 4.4.0 and it works
perfectly. So I guess my comment about the old file not being uploaded
is not valid. Maybe I have renamed the folder, and so lftp reuploads
the whole directory.
Anyway, so version 4.4.0 works all other versions above that do not work here.

SanskritFritz

2013-03-05 15:46:47 UTC

Permalink

Post by SanskritFritz

Ok, besides of being offended that no one answered to my problem :) I
tried to localize the problem and to my amazement it was magically
solved. Ok, not so magically, I recalled that I didnt reload
everything after the upgrade, so I think there was some dependeny hell
going on. Anyways, thanks for lftp again, all is well on my side.
Am I invisible on this list? :p

Alexander V. Lukyanov

2013-03-06 05:15:38 UTC

Permalink

Post by SanskritFritz
Ok, besides of being offended that no one answered to my problem :) I

Sorry for not replying. I'm glad you've solved the problem.

Post by SanskritFritz
tried to localize the problem and to my amazement it was magically
solved. Ok, not so magically, I recalled that I didnt reload
everything after the upgrade, so I think there was some dependeny hell
going on. Anyways, thanks for lftp again, all is well on my side.
Am I invisible on this list? :p

You are certainly visible.

--
Alexander.

SanskritFritz

2013-03-06 14:38:30 UTC

Permalink

Post by Alexander V. Lukyanov

Post by SanskritFritz
Ok, besides of being offended that no one answered to my problem :) I

Sorry for not replying. I'm glad you've solved the problem.

An I'm sorry for the noise.

Post by Alexander V. Lukyanov

You are certainly visible.

Good :). I have to say lftp is an amazing piece of software, thank
you. I use in my hourly cron job, it saves all my data to the cloud.