Mirror command and pureftpd max files

[Paul Barker]

Hi all,

I'm wondering if anyone knows how to work around the following issue.
I'm using the 'mirror -R --only-newer' command within lftp to upload
changed files to my webserver. One of the directories contains over
7000 files and it isn't possible to split these into separate
directories by hand as I'd need to re-write a lot of index files.

I've determined that the server is running PureFTPd, which has an
option to limit the maximum number of files it lists in response to an
'ls' command. Browsing the folder by HTTP shows all the files, running
'ls' in lftp only shows the first couple of thousand. This causes
'mirror' to think the remaining files aren't on the server and so they
get transferred again every time I run 'mirror'. This is frustrating
if there are just a couple of small new or updated files I want to
sync as it adds 100MB or so and about an hour to my upload.

Is there anything I can do about this from within lftp? The mirror is
recursive and this is a few directories down the hierarchy. If all
else fails I'll write a mirror script that transfers 'a*' then 'b*'
and so on for this directory, then handles other directories
separately...

[Alexander V. Lukyanov]

Post by Paul Barker
'ls' command. Browsing the folder by HTTP shows all the files, running
'ls' in lftp only shows the first couple of thousand. This causes

Can you upload the files using HTTP? If not, you can try this trick:
1. use "mirror --script-only=script.lftp" on http connection.
2. change http to ftp in the script using e.g. sed
3. run "lftp -f script.lftp"

Also, you can try some settings to change LIST to STAT or MLSD:
set ftp:use-mlsd yes
set ftp:use-stat-for-list yes

But I don't know if that would change pureftpd behaviour regarding huge
directories.

--
Alexander.

[Paul Barker]

Post by Alexander V. Lukyanov

Post by Paul Barker
'ls' command. Browsing the folder by HTTP shows all the files, running
'ls' in lftp only shows the first couple of thousand. This causes

1. use "mirror --script-only=script.lftp" on http connection.
2. change http to ftp in the script using e.g. sed
3. run "lftp -f script.lftp"

So I finally got round to giving this a try. I created
upload-files.lftp using the --script option of mirror then processed
with:

sed -i -e 's/^get -e/put/g' -e 's/^get/put/g' -e 's#http://<server
name>#<path of http root on ftp server>#g' upload-files.lftp

I then added the appropriate 'open' command to the beginning of the
script to specify username and password. Running the resulting lftp
script worked, saving me several hours of waiting :)

Post by Alexander V. Lukyanov
set ftp:use-mlsd yes
set ftp:use-stat-for-list yes
But I don't know if that would change pureftpd behaviour regarding huge
directories.

I tried these first and neither helped.

Many thanks for your assistance.