Assertion `mmap(__ag_high_shadow, (0xdfff0000fffULL), 0x1 | 0x2, 0x02 | 0x10 | 0x04000 | 0x20, -1, 0) != ((void *) -1)' failed.

[ohofox]

Thank you for this great project! Currently I'm trying to use afl++ in qemu mode. Everything is OK when I run it on Ubuntu 20.04. However, when I run qemu mode on my CentOS server, and when I set the AFL_USE_QASAN environment option to turn on QASAN module, the assertion error occurs. The assert triggered on line https://github.com/AFLplusplus/qemuafl/blob/master/qemuafl/asan-giovese-inl.h#L110
This is the configuration of my CentOS server:

# uname -a
Linux admin1 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
#lsb_release
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
# cat /proc/meminfo
MemTotal:       196503520 kB
MemFree:          707560 kB
MemAvailable:   161477888 kB
Buffers:          722940 kB
Cached:         154016044 kB
SwapCached:          624 kB
Active:         71413936 kB
Inactive:       91871344 kB
Active(anon):    7943012 kB
Inactive(anon):  4921596 kB
Active(file):   63470924 kB
Inactive(file): 86949748 kB
Unevictable:      136352 kB
Mlocked:        23091216 kB
SwapTotal:       8388604 kB
SwapFree:        8368380 kB
Dirty:               732 kB
Writeback:             0 kB
AnonPages:       8682552 kB
Mapped:           275244 kB
Shmem:           4312876 kB
Slab:           19835304 kB
SReclaimable:   11100756 kB
SUnreclaim:      8734548 kB
KernelStack:       78576 kB
PageTables:        60492 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    197031980 kB
Committed_AS:   43361680 kB
VmallocTotal:   34359738367 kB
VmallocUsed:     2332624 kB
VmallocChunk:   34256197332 kB
HardwareCorrupted:     0 kB
AnonHugePages:   7141376 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     1999244 kB
DirectMap2M:    60559360 kB
DirectMap1G:    139460608 kB

Also I write a simple mmap test program, the same assertion error occurs

#cat mmap.c
#include <sys/mman.h>
#include <stdio.h>
#include <assert.h>

#define handle_error(msg) \
    do { perror(msg); exit(EXIT_FAILURE); } while (0)

#define HIGH_SHADOW_ADDR ((void*)0x02008fff7000ULL)
#define HIGH_SHADOW_SIZE (0xdfff0000fffULL)

void* __ag_high_shadow = HIGH_SHADOW_ADDR;

int
main(int argc, char *argv[])
{
  printf("__ag_high_shadow=%p,HIGH_SHADOW_SIZE=%llu\n", __ag_high_shadow, HIGH_SHADOW_SIZE);
  void * ret = mmap(__ag_high_shadow, HIGH_SHADOW_SIZE, PROT_READ | PROT_WRITE,
              MAP_PRIVATE | MAP_FIXED | MAP_NORESERVE | MAP_ANON, -1, 0);
  if (ret == MAP_FAILED) perror("mmap");

  return 0;
}
# gcc mmap.c -o mmap & ./mmap
[1] 125079
__ag_high_shadow=0x2008fff7000,HIGH_SHADOW_SIZE=15392894357503
mmap: Cannot allocate memory

I am sure that there is enough memory space, but why the mmap with the options failed with Cannot allocate memory on the CentOS system?
Thanks a lot for any suggestions.

[vanhauser-thc]

what are you talking about? 15392894357503 byte of memory? that would be 15392894 GB ... impossible :)

[ohofox]

The size mapped is defined by HIGH_SHADOW_SIZE, from line https://github.com/AFLplusplus/qemuafl/blob/master/qemuafl/asan-giovese.h#L45.
Could anyone help to explain why it is so large?

[andreafioraldi]

https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm

Because it is a shadow memory

[ohofox]

Thank you for the explanation. So the shadow memory are calculated by the following equation:
Shadow = (Mem >> 3) + 0x7fff8000;
And the range of shadow memory are divided into 2 parts, LowMem and HighMem. And HIGH_SHADOW_SIZE is the size of HighShadow [0x02008fff7000, 0x10007fff7fff] .

[0x10007fff8000, 0x7fffffffffff] | HighMem
[0x02008fff7000, 0x10007fff7fff] | HighShadow
[0x00008fff7000, 0x02008fff6fff] | ShadowGap
[0x00007fff8000, 0x00008fff6fff] | LowShadow
[0x000000000000, 0x00007fff7fff] | LowMem

Back to this assertion error, the mmap is mapping HighShadow but failed.
Should I change any configuration of my CentOS system? Or maybe it is an flaw of mmap function on CentOS system? Could anyone offer help?

[andreafioraldi]

I don't really know what CentOS does, but you can check with /proc//maps what is overlapping the shadow memory regions before all the mmaps