Tim Kelsch
2013-11-12 21:02:45 UTC
Hello,
I'm using LFTP v4.4.8 on rhel6u2 to reverse mirror thousands of files from
5 "client" hosts to 2 different to VSFTPD servers (2.2.2) on rhel6u2. We're
running anywhere between 10-40 LFTP clients on each client host. I'll hold
off on giving more details here as my first question has more to do with
something I saw in the documentation (http://lftp.yar.ru/lftp-man.html)
that alarmed me. In the Commands section, under the Mirror command:
-P, --parallel[=N] download N files in parallel
No problem there, but reading on in the Settings section:
*cmd:parallel* (number)
Number of jobs run in parallel in non-interactive mode.
For example, this may be useful for scripts with multiple `get'
commands. Note that setting this to a value greater than 1
changes conditional execution behaviour, basically makes it
inconsistent.
So I'm wondering if there is any possibility that the parallel
argument to the mirror command could cause "inconsistent" behavior.
My second question is "what exactly does inconsistent behavior mean"?
My third question is "what constitutes conditional execution"?
We are using commands such as:
mirror -v --use-cache --continue --parallel=10 --reverse
/file/system/1 /file/system/2
mirror -v --use-cache --continue --parallel=10 --reverse -x
".*log_.*$" --exclude "\.log$" -i "\.gz$" /file/system/1
/file/system/2
We're seeing very poor throughput between hosts and clients and
without getting into the specifics of it, without doing any real
research one of our Network admins is pointing to this "inconsistency"
and saying that LFTP is the wrong tool and that GridFTP will solve all
our problems.
command), that we're necessarily invoking conditional execution, or
that the type of inconsistency being referred to would necessarily
cause the type of issues we're seeing.
I can get deeper into the type of issues we're seeing but I'd like to
start there first.
Thanks,
Tim Kelsch
I'm using LFTP v4.4.8 on rhel6u2 to reverse mirror thousands of files from
5 "client" hosts to 2 different to VSFTPD servers (2.2.2) on rhel6u2. We're
running anywhere between 10-40 LFTP clients on each client host. I'll hold
off on giving more details here as my first question has more to do with
something I saw in the documentation (http://lftp.yar.ru/lftp-man.html)
that alarmed me. In the Commands section, under the Mirror command:
-P, --parallel[=N] download N files in parallel
No problem there, but reading on in the Settings section:
*cmd:parallel* (number)
Number of jobs run in parallel in non-interactive mode.
For example, this may be useful for scripts with multiple `get'
commands. Note that setting this to a value greater than 1
changes conditional execution behaviour, basically makes it
inconsistent.
So I'm wondering if there is any possibility that the parallel
argument to the mirror command could cause "inconsistent" behavior.
My second question is "what exactly does inconsistent behavior mean"?
My third question is "what constitutes conditional execution"?
We are using commands such as:
mirror -v --use-cache --continue --parallel=10 --reverse
/file/system/1 /file/system/2
mirror -v --use-cache --continue --parallel=10 --reverse -x
".*log_.*$" --exclude "\.log$" -i "\.gz$" /file/system/1
/file/system/2
We're seeing very poor throughput between hosts and clients and
without getting into the specifics of it, without doing any real
research one of our Network admins is pointing to this "inconsistency"
and saying that LFTP is the wrong tool and that GridFTP will solve all
our problems.
From reading the man page it doesn't seem obvious that we're using the
parallel setting (we are using the parallel argument to the mirrorcommand), that we're necessarily invoking conditional execution, or
that the type of inconsistency being referred to would necessarily
cause the type of issues we're seeing.
I can get deeper into the type of issues we're seeing but I'd like to
start there first.
Thanks,
Tim Kelsch