your memory is corrupted!

[antonkulaga]

I failed to make UrQt work, I constantly get:

*** Error in `UrQt': malloc(): memory corruption: 0x0000000002312110 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f53332007e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7f533320b13e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f533320d184]
/lib/x86_64-linux-gnu/libz.so.1(+0xfe29)[0x7f5334020e29]
UrQt[0x4043d5]
UrQt[0x404aa3]
UrQt[0x407077]
UrQt[0x411756]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f53331a9830]
UrQt[0x4041f9]
======= Memory map: ========
00400000-0042c000 r-xp 00000000 00:57 62                                 /opt/UrQt/UrQt
0062b000-0062c000 r--p 0002b000 00:57 62                                 /opt/UrQt/UrQt
0062c000-0062d000 rw-p 0002c000 00:57 62                                 /opt/UrQt/UrQt
0062d000-00728000 rw-p 00000000 00:00 0 
02300000-02333000 rw-p 00000000 00:00 0                                  [heap]
7f532c000000-7f532c021000 rw-p 00000000 00:00 0 
7f532c021000-7f5330000000 ---p 00000000 00:00 0 
7f5333189000-7f5333349000 r-xp 00000000 00:57 38                         /lib/x86_64-linux-gnu/libc-2.23.so
7f5333349000-7f5333549000 ---p 001c0000 00:57 38                         /lib/x86_64-linux-gnu/libc-2.23.so
7f5333549000-7f533354d000 r--p 001c0000 00:57 38                         /lib/x86_64-linux-gnu/libc-2.23.so
7f533354d000-7f533354f000 rw-p 001c4000 00:57 38                         /lib/x86_64-linux-gnu/libc-2.23.so
7f533354f000-7f5333553000 rw-p 00000000 00:00 0 
7f5333553000-7f533356b000 r-xp 00000000 00:57 73                         /lib/x86_64-linux-gnu/libpthread-2.23.so
7f533356b000-7f533376a000 ---p 00018000 00:57 73                         /lib/x86_64-linux-gnu/libpthread-2.23.so
7f533376a000-7f533376b000 r--p 00017000 00:57 73                         /lib/x86_64-linux-gnu/libpthread-2.23.so
7f533376b000-7f533376c000 rw-p 00018000 00:57 73                         /lib/x86_64-linux-gnu/libpthread-2.23.so
7f533376c000-7f5333770000 rw-p 00000000 00:00 0 
7f5333770000-7f5333786000 r-xp 00000000 00:57 71                         /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5333786000-7f5333985000 ---p 00016000 00:57 71                         /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5333985000-7f5333986000 rw-p 00015000 00:57 71                         /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5333986000-7f5333a8e000 r-xp 00000000 00:57 70                         /lib/x86_64-linux-gnu/libm-2.23.so
7f5333a8e000-7f5333c8d000 ---p 00108000 00:57 70                         /lib/x86_64-linux-gnu/libm-2.23.so
7f5333c8d000-7f5333c8e000 r--p 00107000 00:57 70                         /lib/x86_64-linux-gnu/libm-2.23.so
7f5333c8e000-7f5333c8f000 rw-p 00108000 00:57 70                         /lib/x86_64-linux-gnu/libm-2.23.so
7f5333c8f000-7f5333e01000 r-xp 00000000 00:57 68                         /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f5333e01000-7f5334001000 ---p 00172000 00:57 68                         /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f5334001000-7f533400b000 r--p 00172000 00:57 68                         /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f533400b000-7f533400d000 rw-p 0017c000 00:57 68                         /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f533400d000-7f5334011000 rw-p 00000000 00:00 0 
7f5334011000-7f533402a000 r-xp 00000000 00:57 64                         /lib/x86_64-linux-gnu/libz.so.1.2.8
7f533402a000-7f5334229000 ---p 00019000 00:57 64                         /lib/x86_64-linux-gnu/libz.so.1.2.8
7f5334229000-7f533422a000 r--p 00018000 00:57 64                         /lib/x86_64-linux-gnu/libz.so.1.2.8
7f533422a000-7f533422b000 rw-p 00019000 00:57 64                         /lib/x86_64-linux-gnu/libz.so.1.2.8
7f533422b000-7f5334251000 r-xp 00000000 00:57 31                         /lib/x86_64-linux-gnu/ld-2.23.so
7f5334445000-7f533444b000 rw-p 00000000 00:00 0 
7f533444d000-7f5334450000 rw-p 00000000 00:00 0 
7f5334450000-7f5334451000 r--p 00025000 00:57 31                         /lib/x86_64-linux-gnu/ld-2.23.so
7f5334451000-7f5334452000 rw-p 00026000 00:57 31                         /lib/x86_64-linux-gnu/ld-2.23.so
7f5334452000-7f5334453000 rw-p 00000000 00:00 0 
7fff0174e000-7fff0184e000 rw-p 00000000 00:00 0                          [stack]
7fff01894000-7fff01896000 r--p 00000000 00:00 0                          [vvar]
7fff01896000-7fff01898000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
/cromwell-executions/quality_de_novo/2b78dd8c-2978-4ae7-ba16-181171aeaa5f/call-trimming_UrQt_pe/execution/script: line 15:    10 Aborted                 (core dumped) UrQt --in /cromwell-executions/quality_de_novo/2b78dd8c-2978-4ae7-ba16-181171aeaa5f/call-trimming_UrQt_pe/inputs/pipelines/cromwell-executions/quality_de_novo/2b78dd8c-2978-4ae7-ba16-181171aeaa5f/call-trimming_seqpurge/execution/GWliver_S1_L001_R1_001.fastq.gz --inpair /cromwell-executions/quality_de_novo/2b78dd8c-2978-4ae7-ba16-181171aeaa5f/call-trimming_UrQt_pe/inputs/pipelines/cromwell-executions/quality_de_novo/2b78dd8c-2978-4ae7-ba16-181171aeaa5f/call-trimming_seqpurge/execution/GWliver_S1_L001_R2_001.fastq.gz --out GWliver_S1_L001_R1_001_trimmed.fastq --outpair GWliver_S1_L001_R2_001_trimmed.fastq --t 24 --min_read_size 36 --gz -m 8

[l-modolo]

Hi @antonkulaga, there are currently two kinds of memory corruption that can happen with UrQt.

The first one is a bug that I am trying to fix (and which must have been introduced in the last commit...): UrQt don't handle any more reads of variable size

The second one is a problem happening with network volume: if there is a small error in I/O UrQt will crash. This is a trade-off between slow memory usage and speed. A quick fix is to copy the data locally.

Best,

[a-kroh]

Hi l-modolo,

I encountered the a similar issue as antonkulaga.

However, it clearly is not related to read length in my case - all reads are equal in length.

Interestingly, the crash happens only in certain cases and is neither related to file size (also happens for small subsets as small as just 1000 reads), number of cores used and many other parameters I tested.

I even moved a copy of UrQt to the data directory and ran it from there (to exclude the network volume issue you mentioned), but the error persists.

There clearly is some relation to the dataset used, because it persistently occurs for one (rarely in others) - I checked for corruption of the file, but this was not the case either. Also, if split into three batches UrQt does not generate an error (while it does for the 1000 read subset from which the three smaller subsets were created), so it does not seem the data that causes the error.

Below is the command I used and the error I got - would appreciate if you could look into it. I can provide the 1000 read subset if needed.

Kind regards Andreas

UrQt --in sub2_.1.fastq --inpair sub2_.2.fastq --out AZ028_R1_tx.fastq.gz --outpair AZ028_R2_tx.fastq.gz --t 28 --min_read_size 50 --gz --m 4` 

`processing:              sub2_.1.fastq
reads number:            1000`
`*** Error in `UrQt': free(): invalid next size (fast): 0x0000000001290e00 ***`
`======= Backtrace: =========
[0x4c0d36]
[0x478bdb]
[0x478dff]
[0x4880d4]
[0x40ea9d]
[0x40d9fb]
[0x41a960]
[0x4a15e4]
[0x4a1716]
[0x402166]
======= Memory map: ========
00400000-005d7000 r-xp 00000000 08:10 186392207                          /mnt/6459acde-2046-4261-b86a-021ad5beffd9/Andreas/AZ028/UrQt/UrQt
007d6000-007de000 rw-p 001d6000 08:10 186392207                          /mnt/6459acde-2046-4261-b86a-021ad5beffd9/Andreas/AZ028/UrQt/UrQt
007de000-008f4000 rw-p 00000000 00:00 0 
0126e000-012d5000 rw-p 00000000 00:00 0                                  [heap]
7faae8000000-7faae802a000 rw-p 00000000 00:00 0 
7faae802a000-7faaec000000 ---p 00000000 00:00 0 
7faaec000000-7faaec047000 rw-p 00000000 00:00 0 
7faaec047000-7faaf0000000 ---p 00000000 00:00 0 
7faaf0000000-7faaf0021000 rw-p 00000000 00:00 0 
7faaf0021000-7faaf4000000 ---p 00000000 00:00 0 
7faaf8000000-7faaf8021000 rw-p 00000000 00:00 0 
7faaf8021000-7faafc000000 ---p 00000000 00:00 0 
7faafc000000-7faafc021000 rw-p 00000000 00:00 0 
7faafc021000-7fab00000000 ---p 00000000 00:00 0 
7fab00000000-7fab00021000 rw-p 00000000 00:00 0 
7fab00021000-7fab04000000 ---p 00000000 00:00 0 
7fab04ec5000-7fab04ec6000 rw-p 00000000 00:00 0 
7fab04ec6000-7fab04ec7000 ---p 00000000 00:00 0 
7fab04ec7000-7fab056c7000 rw-p 00000000 00:00 0 
7fab056c7000-7fab056c8000 ---p 00000000 00:00 0 
7fab056c8000-7fab05ec8000 rw-p 00000000 00:00 0 
7fab05ec8000-7fab05ec9000 ---p 00000000 00:00 0 
7fab05ec9000-7fab066c9000 rw-p 00000000 00:00 0 
7fab066c9000-7fab066ca000 ---p 00000000 00:00 0 
7fab066ca000-7fab06eca000 rw-p 00000000 00:00 0 
7fab06eca000-7fab06ecb000 ---p 00000000 00:00 0 
7fab06ecb000-7fab076cc000 rw-p 00000000 00:00 0 
7ffdb6c32000-7ffdb6d31000 rw-p 00000000 00:00 0                          [stack]
7ffdb6d87000-7ffdb6d89000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

[l-modolo]

Hello @a-kroh, could you try from the commit d62c1f8 ?
I am aware of the bug and I will try to squeeze time to work on it soon.

Also if you have a small dataset on which the bug occurs 100% of the time that would be really helpful for me to track it down.

[l-modolo]

The procedure to build UrQt from the d62c1f8 commit is the following:

git clone https://github.com/l-modolo/UrQt.git
cd UrQt
git checkout d62c1f8
make

Thank you for the data

[a-kroh]

Hi again,
tried comit d62c1f8 now, but still crashes:

*** Error in `./UrQt': corrupted size vs. prev_size: 0x0000000001215620 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f5d4)[0x7fc9f4a3d5d4]
/lib64/libc.so.6(+0x82de4)[0x7fc9f4a40de4]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7fc9f4a431ac]
/lib64/libz.so.1(+0xdc8c)[0x7fc9f57d3c8c]
/lib64/libz.so.1(+0xdeb5)[0x7fc9f57d3eb5]
/lib64/libz.so.1(gzclose_w+0x32)[0x7fc9f57d45f2]
./UrQt[0x40417f]
./UrQt[0x404843]
./UrQt[0x410cc4]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fc9f49e03d5]
./UrQt[0x403f39]
======= Memory map: ========
00400000-0042a000 r-xp 00000000 08:10 186392248                          /mnt/6459acde-2046-4261-b86a-021ad5beffd9/Andreas/AZ028/UrQt_d62c1f8/UrQt
00629000-0062a000 r--p 00029000 08:10 186392248                          /mnt/6459acde-2046-4261-b86a-021ad5beffd9/Andreas/AZ028/UrQt_d62c1f8/UrQt
0062a000-0062b000 rw-p 0002a000 08:10 186392248                          /mnt/6459acde-2046-4261-b86a-021ad5beffd9/Andreas/AZ028/UrQt_d62c1f8/UrQt
0062b000-00726000 rw-p 00000000 00:00 0 
01215000-01257000 rw-p 00000000 00:00 0                                  [heap]
7fc9f0000000-7fc9f0021000 rw-p 00000000 00:00 0 
7fc9f0021000-7fc9f4000000 ---p 00000000 00:00 0 
7fc9f49be000-7fc9f4b80000 r-xp 00000000 fd:00 201774533                  /usr/lib64/libc-2.17.so
7fc9f4b80000-7fc9f4d80000 ---p 001c2000 fd:00 201774533                  /usr/lib64/libc-2.17.so
7fc9f4d80000-7fc9f4d84000 r--p 001c2000 fd:00 201774533                  /usr/lib64/libc-2.17.so
7fc9f4d84000-7fc9f4d86000 rw-p 001c6000 fd:00 201774533                  /usr/lib64/libc-2.17.so
7fc9f4d86000-7fc9f4d8b000 rw-p 00000000 00:00 0 
7fc9f4d8b000-7fc9f4da2000 r-xp 00000000 fd:00 202469285                  /usr/lib64/libpthread-2.17.so
7fc9f4da2000-7fc9f4fa1000 ---p 00017000 fd:00 202469285                  /usr/lib64/libpthread-2.17.so
7fc9f4fa1000-7fc9f4fa2000 r--p 00016000 fd:00 202469285                  /usr/lib64/libpthread-2.17.so
7fc9f4fa2000-7fc9f4fa3000 rw-p 00017000 fd:00 202469285                  /usr/lib64/libpthread-2.17.so
7fc9f4fa3000-7fc9f4fa7000 rw-p 00000000 00:00 0 
7fc9f4fa7000-7fc9f4fbc000 r-xp 00000000 fd:00 201878410                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fc9f4fbc000-7fc9f51bb000 ---p 00015000 fd:00 201878410                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fc9f51bb000-7fc9f51bc000 r--p 00014000 fd:00 201878410                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fc9f51bc000-7fc9f51bd000 rw-p 00015000 fd:00 201878410                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fc9f51bd000-7fc9f52be000 r-xp 00000000 fd:00 202342642                  /usr/lib64/libm-2.17.so
7fc9f52be000-7fc9f54bd000 ---p 00101000 fd:00 202342642                  /usr/lib64/libm-2.17.so
7fc9f54bd000-7fc9f54be000 r--p 00100000 fd:00 202342642                  /usr/lib64/libm-2.17.so
7fc9f54be000-7fc9f54bf000 rw-p 00101000 fd:00 202342642                  /usr/lib64/libm-2.17.so
7fc9f54bf000-7fc9f55a8000 r-xp 00000000 fd:00 202910626                  /usr/lib64/libstdc++.so.6.0.19
7fc9f55a8000-7fc9f57a7000 ---p 000e9000 fd:00 202910626                  /usr/lib64/libstdc++.so.6.0.19
7fc9f57a7000-7fc9f57af000 r--p 000e8000 fd:00 202910626                  /usr/lib64/libstdc++.so.6.0.19
7fc9f57af000-7fc9f57b1000 rw-p 000f0000 fd:00 202910626                  /usr/lib64/libstdc++.so.6.0.19
7fc9f57b1000-7fc9f57c6000 rw-p 00000000 00:00 0 
7fc9f57c6000-7fc9f57db000 r-xp 00000000 fd:00 201329266                  /usr/lib64/libz.so.1.2.7
7fc9f57db000-7fc9f59da000 ---p 00015000 fd:00 201329266                  /usr/lib64/libz.so.1.2.7
7fc9f59da000-7fc9f59db000 r--p 00014000 fd:00 201329266                  /usr/lib64/libz.so.1.2.7
7fc9f59db000-7fc9f59dc000 rw-p 00015000 fd:00 201329266                  /usr/lib64/libz.so.1.2.7
7fc9f59dc000-7fc9f59fe000 r-xp 00000000 fd:00 201328075                  /usr/lib64/ld-2.17.so
7fc9f5bd4000-7fc9f5bda000 rw-p 00000000 00:00 0 
7fc9f5bfb000-7fc9f5bfd000 rw-p 00000000 00:00 0 
7fc9f5bfd000-7fc9f5bfe000 r--p 00021000 fd:00 201328075                  /usr/lib64/ld-2.17.so
7fc9f5bfe000-7fc9f5bff000 rw-p 00022000 fd:00 201328075                  /usr/lib64/ld-2.17.so
7fc9f5bff000-7fc9f5c00000 rw-p 00000000 00:00 0 
7ffd087b9000-7ffd088b9000 rw-p 00000000 00:00 0                          [stack]
7ffd0891d000-7ffd0891f000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)