Seg fault when trying to run Xilinx GoogLeNet on Ultra96 DietCHai

[pmousoul]

Hello,

I've built a custom platform for the Ultra96 board as it is shown in the CHaiDNN documentation.

After that, I build the SW and put everything on the SD.

When I try to run GoogLeNet, I get the following messages:

[INFOx] Network Path : models/GoogleNetWithoutLRN/8Bit
[PARSE] Parsing conv1/7x7_s2
[PARSE] Parsing conv1/relu_7x7
[PARSE] Parsing pool1/3x3_s2
[PARSE] Parsing conv2/3x3_reduce
[PARSE] Parsing conv2/relu_3x3_reduce
[PARSE] Parsing conv2/3x3
[PARSE] Parsing conv2/relu_3x3
[PARSE] Parsing pool2/3x3_s2
[PARSE] Parsing inception_3a/1x1
[PARSE] Parsing inception_3a/relu_1x1
[PARSE] Parsing inception_3a/3x3_reduce
[PARSE] Parsing inception_3a/relu_3x3_reduce
[PARSE] Parsing inception_3a/3x3
[PARSE] Parsing inception_3a/relu_3x3
[PARSE] Parsing inception_3a/5x5_reduce
[PARSE] Parsing inception_3a/relu_5x5_reduce
[PARSE] Parsing inception_3a/5x5
[PARSE] Parsing inception_3a/relu_5x5
[PARSE] Parsing inception_3a/pool
[PARSE] Parsing inception_3a/pool_proj
[PARSE] Parsing inception_3a/relu_pool_proj
[PARSE] Parsing inception_3a/output
[PARSE] Parsing inception_3b/1x1
[PARSE] Parsing inception_3b/relu_1x1
[PARSE] Parsing inception_3b/3x3_reduce
[PARSE] Parsing inception_3b/relu_3x3_reduce
[PARSE] Parsing inception_3b/3x3
[PARSE] Parsing inception_3b/relu_3x3
[PARSE] Parsing inception_3b/5x5_reduce
[PARSE] Parsing inception_3b/relu_5x5_reduce
[PARSE] Parsing inception_3b/5x5
[PARSE] Parsing inception_3b/relu_5x5
[PARSE] Parsing inception_3b/pool
[PARSE] Parsing inception_3b/pool_proj
[PARSE] Parsing inception_3b/relu_pool_proj
[PARSE] Parsing inception_3b/output
[PARSE] Parsing pool3/3x3_s2
[PARSE] Parsing inception_4a/1x1
[PARSE] Parsing inception_4a/relu_1x1
[PARSE] Parsing inception_4a/3x3_reduce
[PARSE] Parsing inception_4a/relu_3x3_reduce
[PARSE] Parsing inception_4a/3x3
[PARSE] Parsing inception_4a/relu_3x3
[PARSE] Parsing inception_4a/5x5_reduce
[PARSE] Parsing inception_4a/relu_5x5_reduce
[PARSE] Parsing inception_4a/5x5
[PARSE] Parsing inception_4a/relu_5x5
[PARSE] Parsing inception_4a/pool
[PARSE] Parsing inception_4a/pool_proj
[PARSE] Parsing inception_4a/relu_pool_proj
[PARSE] Parsing inception_4a/output
[PARSE] Parsing inception_4b/1x1
[PARSE] Parsing inception_4b/relu_1x1
[PARSE] Parsing inception_4b/3x3_reduce
[PARSE] Parsing inception_4b/relu_3x3_reduce
[PARSE] Parsing inception_4b/3x3
[PARSE] Parsing inception_4b/relu_3x3
[PARSE] Parsing inception_4b/5x5_reduce
[PARSE] Parsing inception_4b/relu_5x5_reduce
[PARSE] Parsing inception_4b/5x5
[PARSE] Parsing inception_4b/relu_5x5
[PARSE] Parsing inception_4b/pool
[PARSE] Parsing inception_4b/pool_proj
[PARSE] Parsing inception_4b/relu_pool_proj
[PARSE] Parsing inception_4b/output
[PARSE] Parsing inception_4c/1x1
[PARSE] Parsing inception_4c/relu_1x1
[PARSE] Parsing inception_4c/3x3_reduce
[PARSE] Parsing inception_4c/relu_3x3_reduce
[PARSE] Parsing inception_4c/3x3
[PARSE] Parsing inception_4c/relu_3x3
[PARSE] Parsing inception_4c/5x5_reduce
[PARSE] Parsing inception_4c/relu_5x5_reduce
[PARSE] Parsing inception_4c/5x5
[PARSE] Parsing inception_4c/relu_5x5
[PARSE] Parsing inception_4c/pool
[PARSE] Parsing inception_4c/pool_proj
[PARSE] Parsing inception_4c/relu_pool_proj
[PARSE] Parsing inception_4c/output
[PARSE] Parsing inception_4d/1x1
[PARSE] Parsing inception_4d/relu_1x1
[PARSE] Parsing inception_4d/3x3_reduce
[PARSE] Parsing inception_4d/relu_3x3_reduce
[PARSE] Parsing inception_4d/3x3
[PARSE] Parsing inception_4d/relu_3x3
[PARSE] Parsing inception_4d/5x5_reduce
[PARSE] Parsing inception_4d/relu_5x5_reduce
[PARSE] Parsing inception_4d/5x5
[PARSE] Parsing inception_4d/relu_5x5
[PARSE] Parsing inception_4d/pool
[PARSE] Parsing inception_4d/pool_proj
[PARSE] Parsing inception_4d/relu_pool_proj
[PARSE] Parsing inception_4d/output
[PARSE] Parsing inception_4e/1x1
[PARSE] Parsing inception_4e/relu_1x1
[PARSE] Parsing inception_4e/3x3_reduce
[PARSE] Parsing inception_4e/relu_3x3_reduce
[PARSE] Parsing inception_4e/3x3
[PARSE] Parsing inception_4e/relu_3x3
[PARSE] Parsing inception_4e/5x5_reduce
[PARSE] Parsing inception_4e/relu_5x5_reduce
[PARSE] Parsing inception_4e/5x5
[PARSE] Parsing inception_4e/relu_5x5
[PARSE] Parsing inception_4e/pool
[PARSE] Parsing inception_4e/pool_proj
[PARSE] Parsing inception_4e/relu_pool_proj
[PARSE] Parsing inception_4e/output
[PARSE] Parsing pool4/3x3_s2
[PARSE] Parsing inception_5a/1x1
[PARSE] Parsing inception_5a/relu_1x1
[PARSE] Parsing inception_5a/3x3_reduce
[PARSE] Parsing inception_5a/relu_3x3_reduce
[PARSE] Parsing inception_5a/3x3
[PARSE] Parsing inception_5a/relu_3x3
[PARSE] Parsing inception_5a/5x5_reduce
[PARSE] Parsing inception_5a/relu_5x5_reduce
[PARSE] Parsing inception_5a/5x5
[PARSE] Parsing inception_5a/relu_5x5
[PARSE] Parsing inception_5a/pool
[PARSE] Parsing inception_5a/pool_proj
[PARSE] Parsing inception_5a/relu_pool_proj
[PARSE] Parsing inception_5a/output
[PARSE] Parsing inception_5b/1x1
[PARSE] Parsing inception_5b/relu_1x1
[PARSE] Parsing inception_5b/3x3_reduce
[PARSE] Parsing inception_5b/relu_3x3_reduce
[PARSE] Parsing inception_5b/3x3
[PARSE] Parsing inception_5b/relu_3x3
[PARSE] Parsing inception_5b/5x5_reduce
[PARSE] Parsing inception_5b/relu_5x5_reduce
[PARSE] Parsing inception_5b/5x5
[PARSE] Parsing inception_5b/relu_5x5
[PARSE] Parsing inception_5b/pool
[PARSE] Parsing inception_5b/pool_proj
[PARSE] Parsing inception_5b/relu_pool_proj
[PARSE] Parsing inception_5b/output
[PARSE] Parsing pool5/7x7_s1
[PARSE] Parsing loss3/classifier
[PARSE] Parsing prob
[libprotobuf INFO google/protobuf/io/coded_stream.cc:610] Reading dangerously large protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 53535180
[IG001] Extracting conv1/7x7_s2 weights ... 
[IG001] Extracting conv1/7x7_s2 bias ... 
[IG001] Extracting conv2/3x3 weights ... 
[IG001] Extracting conv2/3x3 bias ... 
[IG001] Extracting conv2/3x3_reduce weights ... 
[IG001] Extracting conv2/3x3_reduce bias ... 
[IG001] Extracting inception_3a/1x1 weights ... 
[IG001] Extracting inception_3a/1x1 bias ... 
[IG001] Extracting inception_3a/3x3 weights ... 
[IG001] Extracting inception_3a/3x3 bias ... 
[IG001] Extracting inception_3a/3x3_reduce weights ... 
[IG001] Extracting inception_3a/3x3_reduce bias ... 
[IG001] Extracting inception_3a/5x5 weights ... 
[IG001] Extracting inception_3a/5x5 bias ... 
[IG001] Extracting inception_3a/5x5_reduce weights ... 
[IG001] Extracting inception_3a/5x5_reduce bias ... 
[IG001] Extracting inception_3a/pool_proj weights ... 
[IG001] Extracting inception_3a/pool_proj bias ... 
[IG001] Extracting inception_3b/1x1 weights ... 
[IG001] Extracting inception_3b/1x1 bias ... 
[IG001] Extracting inception_3b/3x3 weights ... 
[IG001] Extracting inception_3b/3x3 bias ... 
[IG001] Extracting inception_3b/3x3_reduce weights ... 
[IG001] Extracting inception_3b/3x3_reduce bias ... 
[IG001] Extracting inception_3b/5x5 weights ... 
[IG001] Extracting inception_3b/5x5 bias ... 
[IG001] Extracting inception_3b/5x5_reduce weights ... 
[IG001] Extracting inception_3b/5x5_reduce bias ... 
[IG001] Extracting inception_3b/pool_proj weights ... 
[IG001] Extracting inception_3b/pool_proj bias ... 
[IG001] Extracting inception_4a/1x1 weights ... 
[IG001] Extracting inception_4a/1x1 bias ... 
[IG001] Extracting inception_4a/3x3 weights ... 
[IG001] Extracting inception_4a/3x3 bias ... 
[IG001] Extracting inception_4a/3x3_reduce weights ... 
[IG001] Extracting inception_4a/3x3_reduce bias ... 
[IG001] Extracting inception_4a/5x5 weights ... 
[IG001] Extracting inception_4a/5x5 bias ... 
[IG001] Extracting inception_4a/5x5_reduce weights ... 
[IG001] Extracting inception_4a/5x5_reduce bias ... 
[IG001] Extracting inception_4a/pool_proj weights ... 
[IG001] Extracting inception_4a/pool_proj bias ... 
[IG001] Extracting inception_4b/1x1 weights ... 
[IG001] Extracting inception_4b/1x1 bias ... 
[IG001] Extracting inception_4b/3x3 weights ... 
[IG001] Extracting inception_4b/3x3 bias ... 
[IG001] Extracting inception_4b/3x3_reduce weights ... 
[IG001] Extracting inception_4b/3x3_reduce bias ... 
[IG001] Extracting inception_4b/5x5 weights ... 
[IG001] Extracting inception_4b/5x5 bias ... 
[IG001] Extracting inception_4b/5x5_reduce weights ... 
[IG001] Extracting inception_4b/5x5_reduce bias ... 
[IG001] Extracting inception_4b/pool_proj weights ... 
[IG001] Extracting inception_4b/pool_proj bias ... 
[IG001] Extracting inception_4c/1x1 weights ... 
[IG001] Extracting inception_4c/1x1 bias ... 
[IG001] Extracting inception_4c/3x3 weights ... 
[IG001] Extracting inception_4c/3x3 bias ... 
[IG001] Extracting inception_4c/3x3_reduce weights ... 
[IG001] Extracting inception_4c/3x3_reduce bias ... 
[IG001] Extracting inception_4c/5x5 weights ... 
[IG001] Extracting inception_4c/5x5 bias ... 
[IG001] Extracting inception_4c/5x5_reduce weights ... 
[IG001] Extracting inception_4c/5x5_reduce bias ... 
[IG001] Extracting inception_4c/pool_proj weights ... 
[IG001] Extracting inception_4c/pool_proj bias ... 
[IG001] Extracting inception_4d/1x1 weights ... 
[IG001] Extracting inception_4d/1x1 bias ... 
[IG001] Extracting inception_4d/3x3 weights ... 
[IG001] Extracting inception_4d/3x3 bias ... 
[IG001] Extracting inception_4d/3x3_reduce weights ... 
[IG001] Extracting inception_4d/3x3_reduce bias ... 
[IG001] Extracting inception_4d/5x5 weights ... 
[IG001] Extracting inception_4d/5x5 bias ... 
[IG001] Extracting inception_4d/5x5_reduce weights ... 
[IG001] Extracting inception_4d/5x5_reduce bias ... 
[IG001] Extracting inception_4d/pool_proj weights ... 
[IG001] Extracting inception_4d/pool_proj bias ... 
[IG001] Extracting inception_4e/1x1 weights ... 
[IG001] Extracting inception_4e/1x1 bias ... 
[IG001] Extracting inception_4e/3x3 weights ... 
[IG001] Extracting inception_4e/3x3 bias ... 
[IG001] Extracting inception_4e/3x3_reduce weights ... 
[IG001] Extracting inception_4e/3x3_reduce bias ... 
[IG001] Extracting inception_4e/5x5 weights ... 
[IG001] Extracting inception_4e/5x5 bias ... 
[IG001] Extracting inception_4e/5x5_reduce weights ... 
[IG001] Extracting inception_4e/5x5_reduce bias ... 
[IG001] Extracting inception_4e/pool_proj weights ... 
[IG001] Extracting inception_4e/pool_proj bias ... 
[IG001] Extracting inception_5a/1x1 weights ... 
[IG001] Extracting inception_5a/1x1 bias ... 
[IG001] Extracting inception_5a/3x3 weights ... 
[IG001] Extracting inception_5a/3x3 bias ... 
[IG001] Extracting inception_5a/3x3_reduce weights ... 
[IG001] Extracting inception_5a/3x3_reduce bias ... 
[IG001] Extracting inception_5a/5x5 weights ... 
[IG001] Extracting inception_5a/5x5 bias ... 
[IG001] Extracting inception_5a/5x5_reduce weights ... 
[IG001] Extracting inception_5a/5x5_reduce bias ... 
[IG001] Extracting inception_5a/pool_proj weights ... 
[IG001] Extracting inception_5a/pool_proj bias ... 
[IG001] Extracting inception_5b/1x1 weights ... 
[IG001] Extracting inception_5b/1x1 bias ... 
[IG001] Extracting inception_5b/3x3 weights ... 
[IG001] Extracting inception_5b/3x3 bias ... 
[IG001] Extracting inception_5b/3x3_reduce weights ... 
[IG001] Extracting inception_5b/3x3_reduce bias ... 
[IG001] Extracting inception_5b/5x5 weights ... 
[IG001] Extracting inception_5b/5x5 bias ... 
[IG001] Extracting inception_5b/5x5_reduce weights ... 
[IG001] Extracting inception_5b/5x5_reduce bias ... 
[IG001] Extracting inception_5b/pool_proj weights ... 
[IG001] Extracting inception_5b/pool_proj bias ... 
[INFOx] Graph generated
[INFOx] Generating JobQueue
[INFOx] JobQueue generated
[INFOx] Creating Memory
Failed to create memory!


[INFOx] Memory created
[INFOx] Network Path : models/GoogleNetWithoutLRN/8Bit

[INFOx] Init Start : This may take a while ...Seg fault @ (nil)
lib/libxlnxdnn.so(+0x258f8)[0x7f81c478f8]

What is it going wrong - where should I look for a solution?

Thanks,
Panos

[pmousoul]

some gdb output:

(gdb) run Xilinx 8 ./images/camel.jpg ./images/fish.jpg
Thread 1 "googlenet.elf" received signal SIGSEGV, Segmentation fault.
0x0000007fbf3d4680 in loadConvWgtsTXT(xChangeLayer*, float const*, float const*, int*, int, int, int, char*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from lib/libxstack.so

(gdb) bt
#0  0x0000007fbf3d4680 in loadConvWgtsTXT(xChangeLayer*, float const*, float const*, int*, int, int, int, char*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from lib/libxstack.so
#1  0x0000007fbf3d6ef4 in convInit(XlayerData*, xChangeLayer*, int, file_paths_*, int&, int) () from lib/libxstack.so
#2  0x0000007fbf3dc8f0 in initXChangeHost(char*, std::vector<XlayerData, std::allocator<XlayerData> >&, std::vector<xChangeLayer, std::allocator<xChangeLayer> > (&) [1], int) () from lib/libxstack.so
#3  0x0000007fbf3f9ea4 in xiInit(char*, char*, char*, _io_layer_info*, int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from lib/libxstack.so
#4  0x0000000000403e84 in main ()

[xlxxxxxl]

Hi @pmousoul
I have a similar problem with you. Have you solved this problem?
"Failed to create memory!"

Thanks,
xuelei

[pmousoul]

Hello xuelei,

the problem, at least in my case, is that sds_alloc() function cannot allocate memory returning a NULL pointer value.

I guess that this is caused because of bad kernel configuration when building PetaLinux for the specific platform. There is a kernel parameter which indicates how much memory can sds_alloc() allocate.

So, I think that the solution is to rebuild PetaLinux for your platform and take care the sds_alloc() related kernel parameters during kernel configuration.

Cheers,
Panos

[xlxxxxxl]

@pmousoul Thank you for your reply.

I set mega size to 1024 in the kernel, but this problem still occurs.
Later, when I changed all sys_port to AFI, I can allocate memory successfully. However, in this case, SDSOC automatically allocated HP ports. I did not know whether there was any performance loss.

In addition, I browsed a lot of issues. CHAIDNN originally planned to release the v3 version in 2018.12, and supported mobilenet and yolo. But they were not updated later. Do you know why?

Thanks,
xuelei