Refinedet Tensorrt prediction fails #1261

YaYaB · 2021-04-13T07:46:39Z

Configuration

Version of DeepDetect:
- Locally compiled on:
  - Ubuntu 18.04 LTS
  - Other:
- Docker CPU
- Docker GPU tensorrt
- Docker tensorrt
- Amazon AMI
Commit (shown by the server when starting):
I've tested the last two images and it fails for boths
74cf54c
4de3807

Your question / the problem you're facing:

I tried to make a prediction using a refinedet model with tensorrt backend. However I get an error and if I try to make a prediction two times it generated a core dump.

Error message (if any) / steps to reproduce the problem:

First, let us get one refinedet model (faces)

BASE_PATH="PATH_WANTED"
PATH_MODEL="${BASE_PATH}/face"
mkdir $PATH_MODEL && cd $PATH_MODEL
wget https://deepdetect.com/models/init/desktop/images/detection/faces_512.tar.gz
wget -O test_image.jpeg https://miro.medium.com/max/6528/1*DYUaxku5bfbZaDLW-4SyWg.jpeg
tar -xvf faces_512.tar.gz

Pull the last version (v0.15.0) of dd with tensorrt

docker pull jolibrain/deepdetect_gpu_tensorrt:v0.15.0

Launch DD's server

docker run --gpus 1 -d -u $(id -u ${USER}):$(id -g ${USER}) -p 8080:8080 -v $BASE_PATH:/opt/models/ jolibrain/deepdetect_gpu_tensorrt:v0.15.0

Launch the service

curl -X PUT "http://localhost:8080/services/imageserv" -d '{
"mllib":"tensorrt",
"description":"image classification service",
"type":"supervised",
"parameters":{
    "input":{
    "connector":"image",
    "width":512,
    "height":512,
    "mean": [104.146, 110.808, 119.856]
    },
    "mllib": {
       "datatype": "fp32",                                                                                                           
       "nclasses": 1,
       "maxBatchSize": 1,
       "maxWorkspaceSize": 8000,
       "gpuid":0
    }
},
"model":{
    "repository":"/opt/models/face"
}
}'

Output

[2021-04-13 07:28:54.342] [imageserv] [info] Registered plugin creator - ::SpecialSlice_TRT version 1
[2021-04-13 07:28:54.342] [imageserv] [info] setting max workspace size to 8388608000
[2021-04-13 07:28:54.342] [imageserv] [info] setting max batch size to 1
[2021-04-13 07:28:54.724] [imageserv] [info] trying to determine the input size...
[2021-04-13 07:28:54.739] [imageserv] [info] found 512x512 as input size
[2021-04-13 07:28:54.739] [api] [info] 172.17.0.1 "PUT /services/imageserv" 201 399ms

Launch prediction

curl -X POST "http://localhost:8080/predict" -d '{
      "service":"imageserv",                                                
      "parameters":{
        "input":{
          "width":512,
          "height":512
        },
        "output":{
          "confidence_threshold": 0.5,
          "bbox": true
        }
      },
      "data": ["https://icour.fr/ELeveSeconde/ajout/yann_lecum_vidal/images/yann_LeCun.jpg"]
    }'

Output using v 0.14.0

[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): conv5_3_norm_mbox_priorbox, Tactic: 0, conv5_3_norm[Float(512,32,32)], data[Float(3,512,512)] -> conv5_3_norm_mbox_priorbox[Float(2,12288,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): fc7_mbox_loc_ftune || fc7_mbox_conf_ftune, Tactic: 162, fc7[Float(1024,16,16)] -> fc7_mbox_loc_ftune || fc7_mbox_conf_ftune[Float(18,16,16)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): fc7_mbox_loc_perm_ftune + fc7_mbox_loc_flat_ftune, Tactic: 0, fc7_mbox_loc_ftune || fc7_mbox_conf_ftune[Float(12,16,16)] -> arm_loc_ftune[Float(3072,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): fc7_mbox_conf_perm_ftune + fc7_mbox_conf_flat_ftune, Tactic: 0, fc7_mbox_loc_ftune || fc7_mbox_conf_ftune[Float(6,16,16)] -> arm_conf_ftune[Float(1536,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): fc7_mbox_priorbox, Tactic: 0, fc7[Float(1024,16,16)], data[Float(3,512,512)] -> fc7_mbox_priorbox[Float(2,3072,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune, Tactic: 39, conv6_2[Float(512,8,8)] -> conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune[Float(18,8,8)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): conv6_2_mbox_loc_perm_ftune + conv6_2_mbox_loc_flat_ftune, Tactic: 0, conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune[Float(12,8,8)] -> arm_loc_ftune[Float(768,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): conv6_2_mbox_conf_perm_ftune + conv6_2_mbox_conf_flat_ftune, Tactic: 0, conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune[Float(6,8,8)] -> arm_conf_ftune[Float(384,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): conv6_2_mbox_priorbox, Tactic: 0, conv6_2[Float(512,8,8)], data[Float(3,512,512)] -> conv6_2_mbox_priorbox[Float(2,768,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): conv4_3_norm_mbox_priorbox copy, Tactic: 0, conv4_3_norm_mbox_priorbox[Float(2,49152,1)] -> arm_priorbox[Float(2,49152,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): conv5_3_norm_mbox_priorbox copy, Tactic: 0, conv5_3_norm_mbox_priorbox[Float(2,12288,1)] -> arm_priorbox[Float(2,12288,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): fc7_mbox_priorbox copy, Tactic: 0, fc7_mbox_priorbox[Float(2,3072,1)] -> arm_priorbox[Float(2,3072,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): conv6_2_mbox_priorbox copy, Tactic: 0, conv6_2_mbox_priorbox[Float(2,768,1)] -> arm_priorbox[Float(2,768,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(scudnn_winograd): P3_mbox_loc_ftune || P3_mbox_conf_ftune, Tactic: 3827454225649558724, P3[Float(256,64,64)] -> P3_mbox_loc_ftune || P3_mbox_conf_ftune[Float(18,64,64)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P3_mbox_loc_perm_ftune + P3_mbox_loc_flat_ftune, Tactic: 0, P3_mbox_loc_ftune || P3_mbox_conf_ftune[Float(12,64,64)] -> P3_mbox_loc_flat_ftune[Float(49152,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P3_mbox_conf_perm_ftune + P3_mbox_conf_flat_ftune, Tactic: 0, P3_mbox_loc_ftune || P3_mbox_conf_ftune[Float(6,64,64)] -> P3_mbox_conf_flat_ftune[Float(24576,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(scudnn_winograd): P4_mbox_loc_ftune || P4_mbox_conf_ftune, Tactic: 3827454225649558724, P4[Float(256,32,32)] -> P4_mbox_loc_ftune || P4_mbox_conf_ftune[Float(18,32,32)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P4_mbox_loc_perm_ftune + P4_mbox_loc_flat_ftune, Tactic: 0, P4_mbox_loc_ftune || P4_mbox_conf_ftune[Float(12,32,32)] -> P4_mbox_loc_flat_ftune[Float(12288,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P4_mbox_conf_perm_ftune + P4_mbox_conf_flat_ftune, Tactic: 0, P4_mbox_loc_ftune || P4_mbox_conf_ftune[Float(6,32,32)] -> P4_mbox_conf_flat_ftune[Float(6144,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): P5_mbox_loc_ftune || P5_mbox_conf_ftune, Tactic: 162, P5[Float(256,16,16)] -> P5_mbox_loc_ftune || P5_mbox_conf_ftune[Float(18,16,16)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P5_mbox_loc_perm_ftune + P5_mbox_loc_flat_ftune, Tactic: 0, P5_mbox_loc_ftune || P5_mbox_conf_ftune[Float(12,16,16)] -> P5_mbox_loc_flat_ftune[Float(3072,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P5_mbox_conf_perm_ftune + P5_mbox_conf_flat_ftune, Tactic: 0, P5_mbox_loc_ftune || P5_mbox_conf_ftune[Float(6,16,16)] -> P5_mbox_conf_flat_ftune[Float(1536,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune, Tactic: 39, P6[Float(256,8,8)] -> P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune[Float(18,8,8)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P6_mbox_loc_perm_ftune + P6_mbox_loc_flat_ftune, Tactic: 0, P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune[Float(12,8,8)] -> P6_mbox_loc_flat_ftune[Float(768,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P6_mbox_conf_perm_ftune + P6_mbox_conf_flat_ftune, Tactic: 0, P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune[Float(6,8,8)] -> P6_mbox_conf_flat_ftune[Float(384,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P3_mbox_loc_flat_ftune copy, Tactic: 0, P3_mbox_loc_flat_ftune[Float(49152,1,1)] -> odm_loc_ftune[Float(49152,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P4_mbox_loc_flat_ftune copy, Tactic: 0, P4_mbox_loc_flat_ftune[Float(12288,1,1)] -> odm_loc_ftune[Float(12288,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P5_mbox_loc_flat_ftune copy, Tactic: 0, P5_mbox_loc_flat_ftune[Float(3072,1,1)] -> odm_loc_ftune[Float(3072,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P6_mbox_loc_flat_ftune copy, Tactic: 0, P6_mbox_loc_flat_ftune[Float(768,1,1)] -> odm_loc_ftune[Float(768,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P3_mbox_conf_flat_ftune copy, Tactic: 0, P3_mbox_conf_flat_ftune[Float(24576,1,1)] -> odm_conf_ftune[Float(24576,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P4_mbox_conf_flat_ftune copy, Tactic: 0, P4_mbox_conf_flat_ftune[Float(6144,1,1)] -> odm_conf_ftune[Float(6144,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P5_mbox_conf_flat_ftune copy, Tactic: 0, P5_mbox_conf_flat_ftune[Float(1536,1,1)] -> odm_conf_ftune[Float(1536,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P6_mbox_conf_flat_ftune copy, Tactic: 0, P6_mbox_conf_flat_ftune[Float(384,1,1)] -> odm_conf_ftune[Float(384,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(ExtSoftMax): arm_conf_softmax, Tactic: 0, arm_conf_reshape_ftune[Float(16320,2)] -> arm_conf_softmax_ftune[Float(16320,2)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(ExtSoftMax): odm_conf_softmax, Tactic: 0, odm_conf_reshape_ftune[Float(16320,2)] -> odm_conf_softmax_ftune[Float(16320,2)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): detection_out, Tactic: 0, odm_loc_ftune[Float(65280,1,1)], odm_conf_flatten_ftune[Float(32640,1,1)], arm_priorbox[Float(2,65280,1)], arm_conf_flatten_ftune[Float(32640,1,1)], arm_loc_ftune[Float(65280,1,1)] -> detection_out[Float(1,500,7)], keep_count[Float(1,1,1)]
[2021-04-13 07:40:28.936] [imageserv] [warning] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
[2021-04-13 07:40:29.008] [imageserv] [error] ../rtSafe/cuda/caskConvolutionRunner.cpp (370) - Cuda Error in execute: 98 (invalid device function)
[2021-04-13 07:40:29.008] [imageserv] [error] FAILED_EXECUTION: std::exception
[2021-04-13 07:40:29.008] [imageserv] [error] Error while processing forward TRT pass, not enough memory ? Failed TRT enqueue call
[2021-04-13 07:40:29.008] [imageserv] [error] mllib internal error: Failed TRT enqueue call
[2021-04-13 07:40:29.008] [api] [error] 172.17.0.1 "POST /predict" 500 150564ms

Output using v 0.14.0, If the prediction is launched a second time we get the information

[2021-04-13 07:40:29.008] [api] [error] 172.17.0.1 "POST /predict" 500 150564ms
#assertion/opt/deepdetect/build/tensorrt-oss/src/tensorrt-oss/plugin/normalizePlugin/normalizePlugin.cpp,112
 0# dd::HttpJsonAPI::abort(int) at /opt/deepdetect/src/httpjsonapi.cc:626
 1# 0x00007F6259BB8F20 in /lib/x86_64-linux-gnu/libc.so.6
 2# 0x00007F6259BB8E97 in /lib/x86_64-linux-gnu/libc.so.6
 3# 0x00007F6259BBA801 in /lib/x86_64-linux-gnu/libc.so.6
 4# 0x00007F625E0A7348 in /opt/deepdetect/build/lib/libnvinfer_plugin.so.7
 5# 0x00007F625F9AD555 in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 6# 0x00007F625F726327 in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 7# 0x00007F625F7293E0 in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 8# dd::TensorRTLib<dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel>::predict(dd::APIData const&, dd::APIData&) at /opt/deepdetect/src/backends/tensorrt/tensorrtlib.cc:678
 9# mapbox::util::detail::dispatcher<dd::visitor_mllib::v_predict_job&, mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >, int, dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >::apply(mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >&, dd::visitor_mllib::v_predict_job&) at /usr/include/mapbox/variant.hpp:344
10# dd::Services::predict(dd::APIData const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, dd::APIData&, bool const&) at /opt/deepdetect/src/services.h:623
11# dd::JsonAPI::service_predict(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /opt/deepdetect/src/jsonapi.cc:1161
12# APIHandler::operator()(boost::network::http::basic_request<boost::network::http::tags::http_server> const&, boost::network::http::basic_response<boost::network::http::tags::http_server>&) at /opt/deepdetect/src/httpjsonapi.cc:465
13# boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>::handle_read_headers(boost::system::error_code const&, unsigned long) at /opt/deepdetect/build/cpp-netlib/src/cpp-netlib/boost/network/protocol/http/server/sync_connection.hpp:139
14# void boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long>&) at /usr/include/boost/asio/detail/impl/strand_service.hpp:91
15# boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) at /usr/include/boost/asio/detail/completion_handler.hpp:68
16# void boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >&) at /usr/include/boost/asio/detail/impl/strand_service.hpp:85
17# boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running> >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code
const&, unsigned long) at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:110
18# boost::asio::detail::epoll_reactor::descriptor_state::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:707
19# boost::network::http::sync_server_base<boost::network::http::tags::http_server, APIHandler>::run() at /opt/deepdetect/build/cpp-netlib/src/cpp-netlib/boost/network/protocol/http/server/sync_server.hpp:49
20# 0x00007F625A80D66F in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
21# 0x00007F625D9EA6DB in /lib/x86_64-linux-gnu/libpthread.so.0
22# 0x00007F6259C9B88F in /lib/x86_64-linux-gnu/libc.so.6

Aborted (core dumped)

It shows message saying "not enough memory" however I have a 1080Ti and enough gpu memory.

Output v0.15.0

etection_out reformatted input 4[Float(65280,1,1)] -> detection_out[Float(1,500,7)], keep_count[Float(1,1,1)]
[2021-04-13 07:37:01.040] [imageserv] [info] Allocated persistent device memory of size 289784320
[2021-04-13 07:37:01.040] [imageserv] [info] Allocated activation device memory of size 146426880
[2021-04-13 07:37:01.040] [imageserv] [info] Assigning persistent memory blocks for various profiles
#assertion/opt/deepdetect/build/tensorrt-oss/src/tensorrt-oss/plugin/normalizePlugin/normalizePlugin.cpp,112
 0# dd::OatppJsonAPI::abort(int) at /opt/deepdetect/src/oatppjsonapi.cc:253
 1# 0x00007F2C7EC30210 in /usr/lib/x86_64-linux-gnu/libc.so.6
 2# raise in /usr/lib/x86_64-linux-gnu/libc.so.6
 3# abort in /usr/lib/x86_64-linux-gnu/libc.so.6
 4# 0x00007F2C80685C3D in /opt/deepdetect/build/lib/libnvinfer_plugin.so.7
 5# nvinfer1::rt::cuda::PluginV2Runner::execute(nvinfer1::rt::CommonContext const&, nvinfer1::rt::ExecutionParameters const&) const in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 6# nvinfer1::rt::ExecutionContext::enqueueInternal(CUevent_st**) in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 7# nvinfer1::rt::ExecutionContext::enqueue(int, void**, CUstream_st*, CUevent_st**) in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 8# dd::TensorRTLib<dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel>::predict(dd::APIData const&, dd::APIData&) at /opt/deepdetect/src/backends/tensorrt/tensorrtlib.cc:678
 9# mapbox::util::detail::dispatcher<dd::visitor_mllib::v_predict_job&, mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >, int, dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >::apply(mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >&, dd::visitor_mllib::v_predict_job&) at /usr/include/mapbox/variant.hpp:348
10# dd::Services::predict(dd::APIData const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, dd::APIData&, bool const&) at /opt/deepdetect/src/services.h:623
11# dd::JsonAPI::service_predict(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /opt/deepdetect/src/jsonapi.cc:1161
12# DedeController::Z__PROXY_METHOD_predict(std::shared_ptr<oatpp::web::protocol::http::incoming::Request> const&) at /opt/deepdetect/src/http/controller.hpp:150
13# oatpp::web::server::api::ApiController::Handler<DedeController>::handle(std::shared_ptr<oatpp::web::protocol::http::incoming::Request> const&) at /opt/deepdetect/build/oatpp/src/oatpp/src/oatpp/web/server/api/ApiController.hpp:300
14# oatpp::web::server::HttpProcessor::processNextRequest(oatpp::web::server::HttpProcessor::ProcessingResources&, std::shared_ptr<oatpp::web::protocol::http::incoming::Request> const&, oatpp::web::protocol::http::utils::CommunicationUtils::ConnectionState&) in /opt/deepdetect/build/main/dede
15# oatpp::web::server::HttpProcessor::processNextRequest(oatpp::web::server::HttpProcessor::ProcessingResources&) in /opt/deepdetect/build/main/dede
16# oatpp::web::server::HttpProcessor::Task::run() in /opt/deepdetect/build/main/dede
17# 0x00007F2C7EF0FD84 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
18# start_thread in /usr/lib/x86_64-linux-gnu/libpthread.so.0
19# __clone in /usr/lib/x86_64-linux-gnu/libc.so.6

We already had a few issues with refinedet and you modified the CI but it does not seem to work well here.

Thanks in advance for your help

The text was updated successfully, but these errors were encountered:

beniz · 2021-04-13T08:01:52Z

Hi, can you try outside docker ? Thanks.

YaYaB · 2021-04-13T13:33:18Z

Honestly, it is kind of pain to install everything in a host as:

it takes time
it adds a lot of unecessary things in my host

I would rather avoid this except if it is absolutely needed.
Note that I gave you everything to test it in on your side.

fantes · 2021-04-13T14:51:26Z

Hi there. I cannot reproduce using fresh compiled master. @YaYaB could you try lower "maxWorkspaceSize" to 2000 ?

YaYaB · 2021-04-13T14:57:16Z

@fantes Yes I get a core dump as well with "maxWorkspaceSize" to 2000

fantes · 2021-04-13T15:04:22Z

thanks @YaYaB . Unfortunately, I cannot reproduce either using docker images, using your command lines (thanks for giving them , it helps a lot) . Maybe your BASE_PATH contains bad TRTengine_bs1 or bad net_tensorRT.proto ?

YaYaB · 2021-04-13T15:15:51Z

I make sure the directory does not contain any of those before creating the serving and launching the prediction. I'll try it in another machine to see If I get the same result

YaYaB · 2021-04-13T16:24:26Z

@fantes Ok my bad. After testing it in an another machine I found that I was using it with a Titan X (a bit older than 1080Ti). When I use a 1080Ti it works correctly. However it seems weird because I use classification models (and even ssd) with trt backend with Titan X. it may be the tensorrt_oss that makes the refinedet core dump. Is that possible?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refinedet Tensorrt prediction fails #1261

Refinedet Tensorrt prediction fails #1261

YaYaB commented Apr 13, 2021 •

edited

Loading

beniz commented Apr 13, 2021

YaYaB commented Apr 13, 2021 •

edited

Loading

fantes commented Apr 13, 2021

YaYaB commented Apr 13, 2021 •

edited

Loading

fantes commented Apr 13, 2021

YaYaB commented Apr 13, 2021

YaYaB commented Apr 13, 2021 •

edited

Loading

Refinedet Tensorrt prediction fails #1261

Refinedet Tensorrt prediction fails #1261

Comments

YaYaB commented Apr 13, 2021 • edited Loading

Configuration

Your question / the problem you're facing:

Error message (if any) / steps to reproduce the problem:

beniz commented Apr 13, 2021

YaYaB commented Apr 13, 2021 • edited Loading

fantes commented Apr 13, 2021

YaYaB commented Apr 13, 2021 • edited Loading

fantes commented Apr 13, 2021

YaYaB commented Apr 13, 2021

YaYaB commented Apr 13, 2021 • edited Loading

YaYaB commented Apr 13, 2021 •

edited

Loading

YaYaB commented Apr 13, 2021 •

edited

Loading

YaYaB commented Apr 13, 2021 •

edited

Loading

YaYaB commented Apr 13, 2021 •

edited

Loading