mirror hangs itself / gets stuck

[Matthias Kauer]

Hi,

I am trying to sync my wordpress installation from my host's FTP to my home
computer. I have ssh access, but the server is very limited in what commands
it offers; in particular, there's no rsync.

My version is:
***@volcano:~$ lftp --version
LFTP | Version 4.3.8 | Copyright (c) 1996-2012 Alexander V. Lukyanov
on a ubuntu 13.04 computer.

Therefore, I tried to go ahead with something like this:
lftp -u A,B -e'mirror --parallel=10 httpdocs/
website_backup/matthiaskauer.com' matthiaskauer.com
This command got stuck after a while, only retrieving directory information
after a while and then reaching max retries after a really long time.

I figured this may be caused by symlinks on the server that create a loop.
I tested only synchronizing a subdirectory (about 46M) with less
parallelization and -n and it worked.
lftp -u A,B -e'mirror -n --parallel=4 --no-symlinks httpdocs/dokuwiki/
test3;exit' matthiaskauer.com

I added debugging and tried the same for the entire server structure again:
lftp -d -u A,B -e'mirror -n --parallel=4 --no-symlinks httpdocs/ test3;exit'
matthiaskauer.com 2> lftplog.txt

The messages I get in the command prompt are:
dokuwiki/lib/plugins/indexmenu/conf: Getting directory contents (0)
[Delaying before retry]
tt-rss-bef1.7.10/schema/versions/pgsql: Getting directory contents (0)
[Delaying before retry]
owncloud/owncloud/apps/files_trashbin: Getting directory contents (0)
[Delaying before retry]
dokuwiki/lib/plugins/indexmenu/conf: Getting directory contents (0)
[Connection idle]

The debug log is quite lengthy. At the end, the following seems to repeat
itself (pasted below)

Thanks for your help and best regards,
Matthias Kauer

---> PASV
---> PASV
<--- 227 Entering Passive Mode (212,224,89,152,153,160).
---- Connecting data socket to (212.224.89.152) port 39328
<--- 227 Entering Passive Mode (212,224,89,152,180,98).
---- Connecting data socket to (212.224.89.152) port 46178
---- Data connection established
---> LIST
---- Data connection established
---> LIST
<--- 150 Opening ASCII mode data connection for file list
<--- 150 Opening ASCII mode data connection for file list
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Panel,CN=Parallels Panel,EMAIL=***@parallels.com
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Panel,CN=Parallels Panel,EMAIL=***@parallels.com
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Panel,CN=Parallels Panel,EMAIL=***@parallels.com
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Panel,CN=Parallels Panel,EMAIL=***@parallels.com
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
---> PASV
---> PASV
<--- 227 Entering Passive Mode (212,224,89,152,160,245).
---- Connecting data socket to (212.224.89.152) port 41205
<--- 227 Entering Passive Mode (212,224,89,152,188,96).
---- Connecting data socket to (212.224.89.152) port 48224
---- Data connection established
---> LIST
---- Data connection established
---> LIST
<--- 150 Opening ASCII mode data connection for file list
<--- 150 Opening ASCII mode data connection for file list
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Panel,CN=Parallels Panel,EMAIL=***@parallels.com
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Panel,CN=Parallels Panel,EMAIL=***@parallels.com
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Panel,CN=Parallels Panel,EMAIL=***@parallels.com
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Panel,CN=Parallels Panel,EMAIL=***@parallels.com
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
---> PASV

---> PASV

[Alexander Lukyanov]

I'd recommend to use newest stable lftp version (4.4.14). Then try several
possibilities to access the server:
1. disable ssl (set ftp:ssl-allow no) or disable ssl for file listings
connections only (set ftp:ssl-protect-list no)
2. try to use fish or sftp if they are allowed (open fish://***@server)
--
Alexander.

Post by Matthias Kauer
Hi,
I am trying to sync my wordpress installation from my host's FTP to my home
computer. I have ssh access, but the server is very limited in what commands
it offers; in particular, there's no rsync.
LFTP | Version 4.3.8 | Copyright (c) 1996-2012 Alexander V. Lukyanov
on a ubuntu 13.04 computer.
lftp -u A,B -e'mirror --parallel=10 httpdocs/
website_backup/matthiaskauer.com' matthiaskauer.com
This command got stuck after a while, only retrieving directory information
after a while and then reaching max retries after a really long time.
I figured this may be caused by symlinks on the server that create a loop.
I tested only synchronizing a subdirectory (about 46M) with less
parallelization and -n and it worked.
lftp -u A,B -e'mirror -n --parallel=4 --no-symlinks httpdocs/dokuwiki/
test3;exit' matthiaskauer.com
lftp -d -u A,B -e'mirror -n --parallel=4 --no-symlinks httpdocs/ test3;exit'
matthiaskauer.com 2> lftplog.txt
dokuwiki/lib/plugins/indexmenu/conf: Getting directory contents (0)
[Delaying before retry]
tt-rss-bef1.7.10/schema/versions/pgsql: Getting directory contents (0)
[Delaying before retry]
owncloud/owncloud/apps/files_trashbin: Getting directory contents (0)
[Delaying before retry]
dokuwiki/lib/plugins/indexmenu/conf: Getting directory contents (0)
[Connection idle]
The debug log is quite lengthy. At the end, the following seems to repeat
itself (pasted below)
Thanks for your help and best regards,
Matthias Kauer
---> PASV
---> PASV
<--- 227 Entering Passive Mode (212,224,89,152,153,160).
---- Connecting data socket to (212.224.89.152) port 39328
<--- 227 Entering Passive Mode (212,224,89,152,180,98).
---- Connecting data socket to (212.224.89.152) port 46178
---- Data connection established
---> LIST
---- Data connection established
---> LIST
<--- 150 Opening ASCII mode data connection for file list
<--- 150 Opening ASCII mode data connection for file list
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
---> PASV
---> PASV
<--- 227 Entering Passive Mode (212,224,89,152,160,245).
---- Connecting data socket to (212.224.89.152) port 41205
<--- 227 Entering Passive Mode (212,224,89,152,188,96).
---- Connecting data socket to (212.224.89.152) port 48224
---- Data connection established
---> LIST
---- Data connection established
---> LIST
<--- 150 Opening ASCII mode data connection for file list
<--- 150 Opening ASCII mode data connection for file list
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
---> PASV
---> PASV
_______________________________________________
lftp mailing list
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

[Matthias Kauer]

Hi Alexander,
thanks for your quick reply.

I tested this before New Year's and then almost forgot to report that
this fixed my issues. To sum up.

Building the newest version from source failed for me (I tried
installing some additional packages to make .configure succeed, but
finally gave up when the compilation still failed), so I converted the
provided .rpm to a .deb instead with sudo alien packagename.rpm

Disabling ssl-allow made the command work successfully. Both with the
new version and with the old version that I had on my system.
I also tried using sftp and this worked as well (removing the ssl-allow
term from the ./lftp/rc). I guess I will therefore stick with:
lftp -u A,B-e'mirror -n --parallel=4 --no-symlinks httpdocs/ test3;exit'
sftp://matthiaskauer.com

Thanks again,
Matthias

Post by Alexander Lukyanov
I'd recommend to use newest stable lftp version (4.4.14). Then try
1. disable ssl (set ftp:ssl-allow no) or disable ssl for file listings
connections only (set ftp:ssl-protect-list no)
--
Alexander.
Hi,
I am trying to sync my wordpress installation from my host's FTP to my home
computer. I have ssh access, but the server is very limited in what commands
it offers; in particular, there's no rsync.
LFTP | Version 4.3.8 | Copyright (c) 1996-2012 Alexander V. Lukyanov
on a ubuntu 13.04 computer.
lftp -u A,B -e'mirror --parallel=10 httpdocs/
website_backup/matthiaskauer.com <http://matthiaskauer.com>'
matthiaskauer.com <http://matthiaskauer.com>
This command got stuck after a while, only retrieving directory information
after a while and then reaching max retries after a really long time.
I figured this may be caused by symlinks on the server that create a loop.
I tested only synchronizing a subdirectory (about 46M) with less
parallelization and -n and it worked.
lftp -u A,B -e'mirror -n --parallel=4 --no-symlinks httpdocs/dokuwiki/
test3;exit' matthiaskauer.com <http://matthiaskauer.com>
I added debugging and tried the same for the entire server
lftp -d -u A,B -e'mirror -n --parallel=4 --no-symlinks httpdocs/ test3;exit'
matthiaskauer.com <http://matthiaskauer.com> 2> lftplog.txt
dokuwiki/lib/plugins/indexmenu/conf: Getting directory contents (0)
[Delaying before retry]
tt-rss-bef1.7.10/schema/versions/pgsql: Getting directory
contents (0)
[Delaying before retry]
owncloud/owncloud/apps/files_trashbin: Getting directory contents (0)
[Delaying before retry]
dokuwiki/lib/plugins/indexmenu/conf: Getting directory contents (0)
[Connection idle]
The debug log is quite lengthy. At the end, the following seems to repeat
itself (pasted below)
Thanks for your help and best regards,
Matthias Kauer
---> PASV
---> PASV
<--- 227 Entering Passive Mode (212,224,89,152,153,160).
---- Connecting data socket to (212.224.89.152) port 39328
<--- 227 Entering Passive Mode (212,224,89,152,180,98).
---- Connecting data socket to (212.224.89.152) port 46178
---- Data connection established
---> LIST
---- Data connection established
---> LIST
<--- 150 Opening ASCII mode data connection for file list
<--- 150 Opening ASCII mode data connection for file list
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
---> PASV
---> PASV
<--- 227 Entering Passive Mode (212,224,89,152,160,245).
---- Connecting data socket to (212.224.89.152) port 41205
<--- 227 Entering Passive Mode (212,224,89,152,188,96).
---- Connecting data socket to (212.224.89.152) port 48224
---- Data connection established
---> LIST
---- Data connection established
---> LIST
<--- 150 Opening ASCII mode data connection for file list
<--- 150 Opening ASCII mode data connection for file list
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
Certificate: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
Issued by: C=US,ST=Virginia,L=Herndon,O=Parallels,OU=Parallels
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: certificate common name doesn't match
requested host name ‘matthiaskauer.com’
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
<--- 425 Unable to build data connection: Operation not permitted
---- Closing data socket
---> PASV
---> PASV
_______________________________________________
lftp mailing list
http://univ.uniyar.ac.ru/mailman/listinfo/lftp