lftp-4.3.4 coredump when using pget -n 10 on large (>150GB) file

[Justin Piszcz]

Hello,

When running pget -n 10 on a (vhd, windows backup file):

$ lftp --version
LFTP | Version 4.3.4 | Copyright (c) 1996-2011 Alexander V. Lukyanov

<--- 150 62912081.6 kbytes to download
`...ac-806e6f6e6963.vhd', got 86147617183 of 161054928896 (53%) 245.45M/s
eta:5
xrealloc: out of virtual memory when trying to get 18446744071562141696
bytes..
Segmentation fault (core dumped)

Here is the backtrace:

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `lftp'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000045a9b6 in ResMgr::str2bool (s=0x0) at ResMgr.cc:631
631 ResMgr.cc: No such file or directory.
(gdb) bt
#0 0x000000000045a9b6 in ResMgr::str2bool (s=0x0) at ResMgr.cc:631
#1 0x00000000004588d5 in ResValue::to_bool (this=0x7fff64a42f20)
at ResMgr.h:150
#2 0x000000000045bd69 in ResClient::QueryBool (this=0xc62ca0,
name=0x4f4b5d "use-abor", closure=0xc63140 "192.168.1.2") at
ResMgr.cc:991
#3 0x000000000049d2d0 in Ftp::DataAbort (this=0xc62c80) at ftpclass.cc:3041
#4 0x000000000049e5d9 in Ftp::Close (this=0xc62c80) at ftpclass.cc:3411
#5 0x000000000046358d in FileCopyPeerFA::PrepareToDie (this=0xc632b0)
at FileCopy.cc:1252
#6 0x0000000000451823 in SMTask::DeleteLater (this=0xc632b0) at
SMTask.cc:134
#7 0x0000000000451845 in SMTask::Delete (task=0xc632b0) at SMTask.cc:140
#8 0x0000000000409fd8 in SMTask::_DeleteRef (task=0xc632b0) at SMTask.h:89
#9 0x0000000000435a11 in SMTaskRef<FileCopyPeer>::~SMTaskRef
(this=0xc634a0,
__in_chrg=<optimized out>) at SMTask.h:127
#10 0x000000000045fc7b in FileCopy::~FileCopy (this=0xc63480,
__in_chrg=<optimized out>) at FileCopy.cc:429
#11 0x000000000045fcba in FileCopy::~FileCopy (this=0xc63480,
__in_chrg=<optimized out>) at FileCopy.cc:431
#12 0x0000000000451879 in SMTask::Delete (task=0xc63480) at SMTask.cc:143
#13 0x0000000000409fd8 in SMTask::_DeleteRef (task=0xc63480) at SMTask.h:89
#14 0x000000000045189a in SMTask::_SetRef (task=0xc63480, new_task=0x0)
at SMTask.cc:147
#15 0x00000000004336d6 in SMTaskRef<FileCopy>::operator= (this=0xc636b0,
p=0x0)
---Type <return> to continue, or q <return> to quit---
at SMTask.h:129
#16 0x0000000000432993 in CopyJob::PrepareToDie (this=0xc63630)
at CopyJob.cc:170
#17 0x0000000000451823 in SMTask::DeleteLater (this=0xc63630) at
SMTask.cc:134
#18 0x0000000000451845 in SMTask::Delete (task=0xc63630) at SMTask.cc:140
#19 0x0000000000409fd8 in SMTask::_DeleteRef (task=0xc63630) at SMTask.h:89
#20 0x000000000045189a in SMTask::_SetRef (task=0xc63630, new_task=0x0)
at SMTask.cc:147
#21 0x000000000042c3d0 in SMTaskRef<pgetJob::ChunkXfer>::operator= (
this=0xc64570, p=0x0) at SMTask.h:129
#22 0x000000000042c3a7 in SMTaskRef<pgetJob::ChunkXfer>::unset
(this=0xc64570)
at SMTask.h:130
#23 0x000000000042c312 in _RefArray<pgetJob::ChunkXfer,
SMTaskRef<pgetJob::ChunkXfer> >::dispose (this=0xc62800, i=0) at
xarray.h:138
#24 0x000000000042c33d in _RefArray<pgetJob::ChunkXfer,
SMTaskRef<pgetJob::ChunkXfer> >::dispose (this=0xc62800, i=1, j=9) at
xarray.h:139
#25 0x000000000042c248 in _RefArray<pgetJob::ChunkXfer,
SMTaskRef<pgetJob::ChunkXfer> >::unset (this=0xc62800) at xarray.h:151
#26 0x000000000042ac40 in pgetJob::free_chunks (this=0xc62750)
at pgetJob.cc:306
#27 0x000000000042aeca in pgetJob::PrepareToDie (this=0xc62750)
at pgetJob.cc:334
#28 0x0000000000451823 in SMTask::DeleteLater (this=0xc62750) at
SMTask.cc:134
---Type <return> to continue, or q <return> to quit---
#29 0x000000000040fec0 in Job::PrepareToDie (this=0xd8a320) at Job.cc:62
#30 0x0000000000451823 in SMTask::DeleteLater (this=0xd8a320) at
SMTask.cc:134
#31 0x000000000040fec0 in Job::PrepareToDie (this=0xbfdcf0) at Job.cc:62
#32 0x0000000000451823 in SMTask::DeleteLater (this=0xbfdcf0) at
SMTask.cc:134
#33 0x0000000000451845 in SMTask::Delete (task=0xbfdcf0) at SMTask.cc:140
#34 0x0000000000409fd8 in SMTask::_DeleteRef (task=0xbfdcf0) at SMTask.h:89
#35 0x0000000000417a07 in SMTaskRef<CmdExec>::~SMTaskRef (this=0x730c50,
__in_chrg=<optimized out>) at SMTask.h:127
#36 0x00007f963df75df2 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#37 0x00007f963df75e45 in exit () from /lib/x86_64-linux-gnu/libc.so.6
#38 0x0000000000467087 in memory_error_and_abort (fname=0x4eebe8 "xrealloc",

size=18446744071562141696) at xmalloc.cc:32
#39 0x0000000000467189 in xrealloc (pointer=0x7f9575702010,
bytes=18446744071562141696) at xmalloc.cc:69
#40 0x000000000046770d in xstring::get_space2 (this=0xc6c760,
s=18446744071562133504, g=8192) at xstring.cc:69
#41 0x000000000046ed68 in Buffer::Allocate (this=0xc6c740, size=65536)
at buffer.cc:91
#42 0x0000000000417353 in Buffer::GetSpace (this=0xc6c740, size=65536)
at buffer.h:89
#43 0x000000000047087e in IOBufferFDStream::Get_LL (this=0xc6c720,
size=65536)
at buffer.cc:622
#44 0x000000000046ffec in IOBuffer::Do (this=0xc6c720) at buffer.cc:440
---Type <return> to continue, or q <return> to quit---
#45 0x0000000000451be4 in SMTask::Schedule () at SMTask.cc:237
#46 0x0000000000411a26 in Job::WaitDone (this=0xbfdcf0) at Job.cc:525
#47 0x00000000004098e5 in main (argc=1, argv=0x7fff64a436c8) at lftp.cc:549
(gdb)

Justin.

[Alexander V. Lukyanov]

Post by Justin Piszcz
$ lftp --version
LFTP | Version 4.3.4 | Copyright (c) 1996-2011 Alexander V. Lukyanov
<--- 150 62912081.6 kbytes to download
`...ac-806e6f6e6963.vhd', got 86147617183 of 161054928896 (53%) 245.45M/s
eta:5
xrealloc: out of virtual memory when trying to get 18446744071562141696
bytes..

It could happen when a memory buffer grew to more than 2G. Maybe this
problem was fixed in version 4.3.5:
* fixed large memory consumption in ftp when target disk is full.

Can you re-check if the problem is fixed in the current lftp version?