Discussion:
possible bug in lftp mirror using xfer:use-temp-file option
akshay gupta
2014-09-30 08:18:15 UTC
Permalink
Hi,

I am trying to mirror files on Fuse mounted hdfs file system (
https://wiki.apache.org/hadoop/MountableHDFS). Lftp is creating some 0 byte
files.

Observations:
1. Files of size larger then 64k are getting transferred properly. Smaller
files have 0 byte file size.
2. Debug logs shows that lftp downloaded data for the files correctly. It
also renamed them.

Additional tests on fuse mounted hdfs.
- I created a file with a temp file name, written the data to it, then
renamed the file and finally closed the output stream. In this case it
creates a 0 byte file.
- I created a file with a temp file name, written the data to it, closed
the file and then renamed the file. In this case it creates the file
correctly.


Can someone please look into this issue, it might be possible that stream
is closed after the file is renamed.

Thanks,
Akshay
Alexander Lukyanov
2014-09-30 15:45:17 UTC
Permalink
In unix it is usually allowed to rename open files, so lftp does that. I'll
see if it is possible to reverse the order.
Post by akshay gupta
Hi,
I am trying to mirror files on Fuse mounted hdfs file system (
https://wiki.apache.org/hadoop/MountableHDFS). Lftp is creating some 0
byte files.
1. Files of size larger then 64k are getting transferred properly. Smaller
files have 0 byte file size.
2. Debug logs shows that lftp downloaded data for the files correctly. It
also renamed them.
Additional tests on fuse mounted hdfs.
- I created a file with a temp file name, written the data to it, then
renamed the file and finally closed the output stream. In this case it
creates a 0 byte file.
- I created a file with a temp file name, written the data to it, closed
the file and then renamed the file. In this case it creates the file
correctly.
Can someone please look into this issue, it might be possible that stream
is closed after the file is renamed.
Thanks,
Akshay
--
Alexander.
akshay gupta
2014-09-30 17:05:37 UTC
Permalink
Its causing unpredictable behaviour in case of fuse mounted file system.
Please look into it if it can be resolved.

Thanks for the quick response.

~Akshay
Post by Alexander Lukyanov
In unix it is usually allowed to rename open files, so lftp does that.
I'll see if it is possible to reverse the order.
Post by akshay gupta
Hi,
I am trying to mirror files on Fuse mounted hdfs file system (
https://wiki.apache.org/hadoop/MountableHDFS). Lftp is creating some 0
byte files.
1. Files of size larger then 64k are getting transferred properly.
Smaller files have 0 byte file size.
2. Debug logs shows that lftp downloaded data for the files correctly. It
also renamed them.
Additional tests on fuse mounted hdfs.
- I created a file with a temp file name, written the data to it, then
renamed the file and finally closed the output stream. In this case it
creates a 0 byte file.
- I created a file with a temp file name, written the data to it, closed
the file and then renamed the file. In this case it creates the file
correctly.
Can someone please look into this issue, it might be possible that stream
is closed after the file is renamed.
Thanks,
Akshay
--
Alexander.
Juan Simón
2014-09-30 19:38:48 UTC
Permalink
In which LFTP version exists this option (xfer:use-temp-file)?
Post by akshay gupta
Its causing unpredictable behaviour in case of fuse mounted file system.
Please look into it if it can be resolved.
Thanks for the quick response.
~Akshay
Post by Alexander Lukyanov
In unix it is usually allowed to rename open files, so lftp does that.
I'll see if it is possible to reverse the order.
Post by akshay gupta
Hi,
I am trying to mirror files on Fuse mounted hdfs file system (
https://wiki.apache.org/hadoop/MountableHDFS). Lftp is creating some 0
byte files.
1. Files of size larger then 64k are getting transferred properly.
Smaller files have 0 byte file size.
2. Debug logs shows that lftp downloaded data for the files correctly.
It also renamed them.
Additional tests on fuse mounted hdfs.
- I created a file with a temp file name, written the data to it, then
renamed the file and finally closed the output stream. In this case it
creates a 0 byte file.
- I created a file with a temp file name, written the data to it, closed
the file and then renamed the file. In this case it creates the file
correctly.
Can someone please look into this issue, it might be possible that
stream is closed after the file is renamed.
Thanks,
Akshay
--
Alexander.
_______________________________________________
lftp mailing list
http://univ.uniyar.ac.ru/mailman/listinfo/lftp
akshay gupta
2014-10-01 07:16:53 UTC
Permalink
This option (xfer:use-temp-file) is available in development snapshot
http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20140919.tar.gz.
Post by akshay gupta
Its causing unpredictable behaviour in case of fuse mounted file system.
Please look into it if it can be resolved.
Thanks for the quick response.
~Akshay
Post by Alexander Lukyanov
In unix it is usually allowed to rename open files, so lftp does that.
I'll see if it is possible to reverse the order.
Post by akshay gupta
Hi,
I am trying to mirror files on Fuse mounted hdfs file system (
https://wiki.apache.org/hadoop/MountableHDFS). Lftp is creating some 0
byte files.
1. Files of size larger then 64k are getting transferred properly.
Smaller files have 0 byte file size.
2. Debug logs shows that lftp downloaded data for the files correctly.
It also renamed them.
Additional tests on fuse mounted hdfs.
- I created a file with a temp file name, written the data to it, then
renamed the file and finally closed the output stream. In this case it
creates a 0 byte file.
- I created a file with a temp file name, written the data to it, closed
the file and then renamed the file. In this case it creates the file
correctly.
Can someone please look into this issue, it might be possible that
stream is closed after the file is renamed.
Thanks,
Akshay
--
Alexander.
akshay gupta
2014-10-01 07:24:56 UTC
Permalink
I am using this feature to know whether a file is completely transferred or
lftp is still downloading this file.
Post by akshay gupta
This option (xfer:use-temp-file) is available in development snapshot
http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20140919.tar.gz.
Post by akshay gupta
Its causing unpredictable behaviour in case of fuse mounted file system.
Please look into it if it can be resolved.
Thanks for the quick response.
~Akshay
Post by Alexander Lukyanov
In unix it is usually allowed to rename open files, so lftp does that.
I'll see if it is possible to reverse the order.
Post by akshay gupta
Hi,
I am trying to mirror files on Fuse mounted hdfs file system (
https://wiki.apache.org/hadoop/MountableHDFS). Lftp is creating some 0
byte files.
1. Files of size larger then 64k are getting transferred properly.
Smaller files have 0 byte file size.
2. Debug logs shows that lftp downloaded data for the files correctly.
It also renamed them.
Additional tests on fuse mounted hdfs.
- I created a file with a temp file name, written the data to it, then
renamed the file and finally closed the output stream. In this case it
creates a 0 byte file.
- I created a file with a temp file name, written the data to it,
closed the file and then renamed the file. In this case it creates the file
correctly.
Can someone please look into this issue, it might be possible that
stream is closed after the file is renamed.
Thanks,
Akshay
--
Alexander.
Alexander V. Lukyanov
2014-10-03 14:12:49 UTC
Permalink
Post by akshay gupta
Its causing unpredictable behaviour in case of fuse mounted file system.
Please look into it if it can be resolved.
Please test the new snapshot http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20141003.tar.gz
--
Alexander.
akshay gupta
2014-10-14 06:37:18 UTC
Permalink
Sorry for the delayed response,

It works!! I'll let you know if i find any more bugs :)

~ Akshay
Post by Alexander V. Lukyanov
Post by akshay gupta
Its causing unpredictable behaviour in case of fuse mounted file system.
Please look into it if it can be resolved.
Please test the new snapshot
http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20141003.tar.gz
--
Alexander.
Loading...