lftpd ls timeout

[one2]

Hi,

I am having problem on a certain ftp server with lftp.
The server runs ioftpd and I have no problems with ffxp or pftp.

The issue is the following: on directories that has complete release including
racestats and completed and other messages ( 0 size files) I get timeout when
trying to do a ls or dir.

`ls' at 0 [Waiting for response...]



I have tried all the following:

-set ftp:ssl-allow no: timeout
-set ftp:port-ipv4 IP and set ftp:port-range port1-portn: timeout
-set ftp:use-stat-for-list yes: it gets the list then timeout

with stat the last message is, then timeout: 212 End of Status or at the 2nd
attemp ---> STAT -a

som edditioanl log from pftp, it could be the section info is timing out.

Mon Mar 11 09:47:01 2013 [13512 ] SERVER LOG LINE : >>> LIST -la

Mon Mar 11 09:47:01 2013 [13512 ] in control buffer is = 0 bytes
Mon Mar 11 09:47:01 2013 [13512 ] wait for data and read will now select()
Mon Mar 11 09:47:01 2013 [13512 ] wait for data and read received 78 bytes,
already read was 0
Mon Mar 11 09:47:01 2013 [13512 ] SERVER LOG LINE : 150 Opening BINARY mode data
connection for directory listing using SSL/TLS.
Mon Mar 11 09:47:01 2013 [13512 ] in control buffer left = 0 bytes
Mon Mar 11 09:47:01 2013 [13512 ] list done
Mon Mar 11 09:47:01 2013 [13512 ] recieved filelist
Mon Mar 11 09:47:01 2013 [13512 ] in control buffer is = 0 bytes
Mon Mar 11 09:47:01 2013 [13512 ] wait for data and read will now select()
Mon Mar 11 09:47:01 2013 [13512 ] wait for data and read received 141 bytes,
already read was 0
Mon Mar 11 09:47:01 2013 [13512 ] SERVER LOG LINE : 226-[ Section: Default ] [
Ratio: Unlimited ] [ Credits: 0 MB ]
Mon Mar 11 09:47:01 2013 [13512 ] SERVER LOG LINE : 226 [ UL: 108 GB ] [ DL:
7.56 GB ] [ Speed: 2973.1 KB/Sec ] [ Free: 845 GB ]
Mon Mar 11 09:47:01 2013 [13512 ] in control buffer left = 0 bytes
Mon Mar 11 09:47:01 2013 [13512 ] SERVER LOG LINE : >>> TYPE I

Mon Mar 11 09:47:01 2013 [13512 ] in control buffer is = 0 bytes
Mon Mar 11 09:47:01 2013 [13512 ] wait for data and read will now select()

[Alexander V. Lukyanov]

Post by one2
The issue is the following: on directories that has complete release including
racestats and completed and other messages ( 0 size files) I get timeout when
trying to do a ls or dir.

I suspect the ftp protocol gets out-of-sync. Please send the complete debug
output to check that.

[one2]

Hi,

this time on x64, same timeout, tried active (with port forwarded) ssl-off,
none helped. Unfortunately debug did not say more about timeout reasons.

current: rc

debug
set ftp:ssl-auth SSL
set ftp:passive-mode true
set net:timeout 20
set net:max-retries 2
set ftp:ssl-allow true
set net:reconnect-interval-base 5

lftp :~> open xxx:444
---- Resolving host address...
---- 1 address found: xxx
lftp xxx:~> user xxx
Password:
lftp ***@xxx:~> cd /XVID/Y.N.You.Lie.You.Die.2012.DVDRip.XviD-aAF/
---- Connecting to xxx (xxx) port 444
<--- 220 FTP Server ready.
---> FEAT
<--- 211-Extensions supported:
<--- AUTH SSL
<--- AUTH TLS
<--- CLNT
<--- CPSV
<--- LIST -1aAdflLRsTUVZ
<--- MDTM
<--- MDTM YYYYMMDDHHMMSS filename
<--- PBSZ
<--- PROT
<--- REST STREAM
<--- SIZE
<--- SSCN
<--- STAT -1aAdflLRsTUVZ
<--- TVFS
<--- XCRC filename;start;end
<--- 211 END
---> AUTH SSL
<--- 234 AUTH SSL successful.
---> CLNT lftp/4.4.8
Certificate: CN=xxx
Issued by: CN=xxx
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘xxx’
<--- 530 Please log in first.
---> USER xxx
<--- 331 Password required for xxx.
---> PASS XXXX
<--- 230- _____
<--- 230- _____________ _________ ______|__ |____ __________ ________ ___
____
<--- 230- \ _ / _ / __ / | | _ / __ /
' /
<--- 230- \ \ / / / /____/ | | / / /____/|
___/
<--- 230- \________/____/ /______ /____|____|___/ /______ /| |
<---
230-.----------------/____/------|__/-------------/____/------|__/-|___|-----.
<--- 230-| There are currently no onelines.
|
<---
230-'------------------------------------------------------------------------'
<--- 230- '----( Type 'SITE ONEL <message>' to enter your message. )----'
<--- 230-User xxx from xxx, welcome to our FTP server.
<--- 230-
<--- 230-xxx activity:
<--- 230-
<--- 230- Users online : 9
<--- 230- Active transfers : 0
<--- 230- Uptime : 2 days, 16 hours, 45 mins, 7 secs
<--- 230-
<--- 230-Enjoy your stay.
<--- 230 User xxx logged in.
---> PWD
<--- 257 "/" is current directory.
---> PBSZ 0
---- CWD path to be sent is `/XVID/Y.N.You.Lie.You.Die.2012.DVDRip.XviD-aAF'
<--- 200 PBSZ 0 successful.
---> CWD /XVID/Y.N.You.Lie.You.Die.2012.DVDRip.XviD-aAF
<--- 250 CWD command successful.
<--- 250- .-----------------====-------------------------== ioNiNJA v1.0a ===--.
<--- 250- | |
<--- 250- |-=[ R A C E N F O ]=---------------. |
<--- 250- | `-----------------====----------|
<--- 250- | Release: Y.N.You.Lie.You.Die.2012.DVDRip.XviD-aAF |
<--- 250- | RaceNfo: 700MB in 49F @ 5.9MB/s in 1m 58s |
<--- 250- | Fastest: xxx @ 8.2MB/s |
<--- 250- | Slowest: xxx @ 2.1MB/s |
<--- 250- | |
<--- 250- |-=[ U S E R T O P ]=---------------. |
<--- 250- | `-----------------====----------|
<--- 250- | 1. xxx 485MB 34F 70% 8.2MB/s |
<--- 250- | 2. xxx 215MB 15F 30% 2.1MB/s |
<--- 250- |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - |
<--- 250- | 2. Total 700MB 49F 100% 5.9MB/s |
<--- 250- | |
<--- 250- |-=[ G R O U P T O P ]=-------------. |
<--- 250- | `--------------===--------------|
<--- 250- | 1. Traders 700MB 49F 100% 4.3MB/s |
<--- 250- |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - |
<--- 250- | 1. Total 700MB 49F 100% 5.9MB/s |
<--- 250- | |
<--- 250- |-=[ S A M P L E ]=-----------------. |
<--- 250- | `-----==-------------===--------|
<--- 250- | Type : XVID |
<--- 250- | Bitrate: 846 Kbps |
<--- 250- | AR : 1.857 @ 624x336 |
<--- 250- | FPS : 25.000 fps |
<--- 250- | Audio : MPEG Audio AudioBR: 123 Kbps |
<--- 250- | Mode : Variable Chans : 2 channels |
<--- 250- | Rate : 48.0 KHz Res : N/A |
<--- 250- | |
<--- 250- |-=[ I M D B N F O ]=---------------. |
<--- 250- | `-----==-------------===--------|
<--- 250- | Title : True Love (2012) |
<--- 250- | Genre : Thriller,IMDb.com, Inc. |
<--- 250- | Rating : 4.5 with 50 votes |
<--- 250- | Director : Enrico Clerico Nasino |
<--- 250- | Tagline : - |
<--- 250- | Country : USA,Italy |
<--- 250- | Language : English |
<--- 250- | Runtime : - |
<--- 250- | Screens : 0 and is STV |
<--- 250- | Type : STV |
<--- 250- | Company : Mercurio Domina |
<--- 250- | Url : http://www.imdb.com/title/tt1927172/ |
<--- 250- `-----================---------------------------==------------------'
cd ok, cwd=/XVID/Y.N.You.Lie.You.Die.2012.DVDRip.XviD-aAF
lftp ***@xxx:/XVID/Y.N.You.Lie.You.Die.2012.DVDRip.XviD-aAF> ls
---> PROT P
<--- 200 Protection set to: Private.
**** Timeout - reconnecting
---- Closing control socket
---- Connecting to xxx (xxx) port 444
<--- 220 FTP Server ready.
---> FEAT
<--- 211-Extensions supported:
<--- AUTH SSL
<--- AUTH TLS
<--- CLNT
<--- CPSV
<--- LIST -1aAdflLRsTUVZ
<--- MDTM
<--- MDTM YYYYMMDDHHMMSS filename
<--- PBSZ
<--- PROT
<--- REST STREAM
<--- SIZE
<--- SSCN
<--- STAT -1aAdflLRsTUVZ
<--- TVFS
<--- XCRC filename;start;end
<--- 211 END
---> AUTH SSL
<--- 234 AUTH SSL successful.
---> CLNT lftp/4.4.8
Certificate: CN=xxx
Issued by: CN=xxx
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘xxx’
<--- 530 Please log in first.
---> USER xxx
<--- 331 Password required for xxx.
---> PASS XXXX
---- CWD path to be sent is `/XVID/Y.N.You.Lie.You.Die.2012.DVDRip.XviD-aAF'
<--- 230- _____
<--- 230- _____________ _________ ______|__ |____ __________ ________ ___
____
<--- 230- \ _ / _ / __ / | | _ / __ /
' /
<--- 230- \ \ / / / /____/ | | / / /____/|
___/
<--- 230- \________/____/ /______ /____|____|___/ /______ /| |
<---
230-.----------------/____/------|__/-------------/____/------|__/-|___|-----.
<--- 230-| There are currently no onelines.
|
<---
230-'------------------------------------------------------------------------'
<--- 230- '----( Type 'SITE ONEL <message>' to enter your message. )----'
<--- 230-User xxx from xxx, welcome to our FTP server.
<--- 230-
<--- 230-xxx activity:
<--- 230-
<--- 230- Users online : 9
<--- 230- Active transfers : 0
<--- 230- Uptime : 2 days, 16 hours, 45 mins, 34 secs
<--- 230-
<--- 230-Enjoy your stay.
<--- 230 User xxx logged in.
---> PBSZ 0
<--- 200 PBSZ 0 successful.
---> CWD /XVID/Y.N.You.Lie.You.Die.2012.DVDRip.XviD-aAF
<--- 250 CWD command successful.
<--- 250- .-----------------====-------------------------== ioNiNJA v1.0a ===--.
<--- 250- | |
<--- 250- |-=[ R A C E N F O ]=---------------. |
<--- 250- | `-----------------====----------|
<--- 250- | Release: Y.N.You.Lie.You.Die.2012.DVDRip.XviD-aAF |
<--- 250- | RaceNfo: 700MB in 49F @ 5.9MB/s in 1m 58s |
<--- 250- | Fastest: xxx @ 8.2MB/s |
<--- 250- | Slowest: xxx @ 2.1MB/s |
<--- 250- | |
<--- 250- |-=[ U S E R T O P ]=---------------. |
<--- 250- | `-----------------====----------|
<--- 250- | 1. xxx 485MB 34F 70% 8.2MB/s |
<--- 250- | 2. xxx 215MB 15F 30% 2.1MB/s |
<--- 250- |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - |
<--- 250- | 2. Total 700MB 49F 100% 5.9MB/s |
<--- 250- | |
<--- 250- |-=[ G R O U P T O P ]=-------------. |
<--- 250- | `--------------===--------------|
<--- 250- | 1. Traders 700MB 49F 100% 4.3MB/s |
<--- 250- |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - |
<--- 250- | 1. Total 700MB 49F 100% 5.9MB/s |
<--- 250- | |
<--- 250- |-=[ S A M P L E ]=-----------------. |
<--- 250- | `-----==-------------===--------|
<--- 250- | Type : XVID |
<--- 250- | Bitrate: 846 Kbps |
<--- 250- | AR : 1.857 @ 624x336 |
<--- 250- | FPS : 25.000 fps |
<--- 250- | Audio : MPEG Audio AudioBR: 123 Kbps |
<--- 250- | Mode : Variable Chans : 2 channels |
<--- 250- | Rate : 48.0 KHz Res : N/A |
<--- 250- | |
<--- 250- |-=[ I M D B N F O ]=---------------. |
<--- 250- | `-----==-------------===--------|
<--- 250- | Title : True Love (2012) |
<--- 250- | Genre : Thriller,IMDb.com, Inc. |
<--- 250- | Rating : 4.5 with 50 votes |
<--- 250- | Director : Enrico Clerico Nasino |
<--- 250- | Tagline : - |
<--- 250- | Country : USA,Italy |
<--- 250- | Language : English |
<--- 250- | Runtime : - |
<--- 250- | Screens : 0 and is STV |
<--- 250- | Type : STV |
<--- 250- | Company : Mercurio Domina |
<--- 250- | Url : http://www.imdb.com/title/tt1927172/ |
<--- 250- `-----================---------------------------==------------------'
---> PROT P
<--- 200 Protection set to: Private.
`ls' at 0 [Waiting for response...]
**** Timeout - reconnecting
---- Closing control socket
ls: Fatal error: max-retries exceeded

[Alexander V. Lukyanov]

Post by one2
this time on x64, same timeout, tried active (with port forwarded) ssl-off,
none helped. Unfortunately debug did not say more about timeout reasons.

This server violates the ftp protocol:

<--- 250 CWD command successful.
<--- 250- .-----------------====-------------------------== ioNiNJA v1.0a ===--.

These lines are in reverse order.

See RFC 959 Paragraph 4.2:

Thus the format for multi-line replies is that the first line
will begin with the exact required reply code, followed
immediately by a Hyphen, "-" (also known as Minus), followed by
text. The last line will begin with the same code, followed
immediately by Space <SP>, optionally some text, and the Telnet
end-of-line code.

For example:
123-First line
Second line
234 A line beginning with numbers
123 The last line

The user-process then simply needs to search for the second
occurrence of the same reply code, followed by <SP> (Space), at
the beginning of a line, and ignore all intermediary lines. If
an intermediary line begins with a 3-digit number, the Server
must pad the front to avoid confusion.

[one2]

Hi,

I understand that, but the server is not under my control.
Windows clients (ftprush, flashfxp) work fine, also some old linux clients
like pftp works fine.
Tried up/downloading files with curl that worked fine too, but I'd rather
use the mirror function of lftp.
Even if it violates some parts as you can see from the log the timeout
occurs after ls and not after receiving those multiline messages.

Is there any chance you could get lftp working or I shall just try to use
some other way?

thx

o.

[Alexander V. Lukyanov]

Post by one2
I understand that, but the server is not under my control.
Windows clients (ftprush, flashfxp) work fine, also some old linux clients
like pftp works fine.
Tried up/downloading files with curl that worked fine too, but I'd rather
use the mirror function of lftp.
Even if it violates some parts as you can see from the log the timeout
occurs after ls and not after receiving those multiline messages.
Is there any chance you could get lftp working or I shall just try to use
some other way?

Please try this patch.

--
Alexander.

[one2]

Post by Alexander V. Lukyanov
Please try this patch.
--
Alexander.

Hi Alex,

it works nicely thank you for the patch.
I had this error on apply but I edited the files manually.


~/lftp-git$ git apply diff
diff:10: trailing whitespace.
if(conn->multiline_code && conn->multiline_code!=code
diff:11: trailing whitespace.
&& QueryBool("ftp:strict-multiline",closure))
diff:23: trailing whitespace.
{"ftp:strict-multiline", "off", ResMgr::BoolValidate,0},
error: patch failed: src/ftpclass.cc:2801
error: src/ftpclass.cc: patch does not apply
error: patch failed: src/resource.cc:213
error: src/resource.cc: patch does not apply

[one2]

this option solved it, so no need to edit manually:

git apply --whitespace=fix diff


thanks.

o.