mxnet.base.MXNetError: Cannot find argument 'cudnn_algo_verbose'

[renganxu]

Description

The was error "mxnet.base.MXNetError: Cannot find argument 'cudnn_algo_verbose'" when I ran the resnet50 model from Image Classification in MLPerf benchmark.

Environment info (Required)

----------Python Info----------
Version      : 3.6.5
Compiler     : GCC 7.2.0
Build        : ('default', 'Apr 29 2018 16:14:56')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 19.0.1
Directory    : /home/rengan/miniconda3/lib/python3.6/site-packages/pip
----------MXNet Info-----------
Version      : 1.5.0
Directory    : /home/rengan/apps/mxnet/dev/lib/python3.6/site-packages/mxnet-1.5.0-py3.6.egg/mxnet
Hashtag not found. Not installed from pre-built package.
----------System Info----------
Platform     : Linux-3.10.0-862.el7.x86_64-x86_64-with-redhat-7.5-Maipo
system       : Linux
node         : node001
release      : 3.10.0-862.el7.x86_64
version      : #1 SMP Wed Mar 21 18:14:51 EDT 2018
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) list:   0-39
Thread(s) per core:    1
Core(s) per socket:    20
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping:              4
CPU MHz:               2400.000
BogoMIPS:              4800.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              28160K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt mba tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local ibpb ibrs stibp dtherm ida arat pln pts pku ospke spec_ctrl intel_stibp
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0371 sec, LOAD: 0.5303 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0976 sec, LOAD: 0.4910 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.1072 sec, LOAD: 0.4491 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0461 sec, LOAD: 0.2345 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0694 sec, LOAD: 2.2932 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0473 sec, LOAD: 0.4014 sec.

Package used (Python/R/Scala/Julia): Python

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio): gcc 7.2.0
MXNet commit hash: f95e794

Build config:

make -j 40 USE_OPENCV=1 \
          USE_BLAS=openblas USE_CUDA=1 \
          USE_CUDA_PATH=/cm/shared/apps/cuda10.0/toolkit/10.0.130 \
          USE_CUDNN=1 \
          USE_NCCL=1 \
          USE_NCCL_PATH=/home/rengan/apps/nccl/2.3.7

Error Message:

Traceback (most recent call last):
  File "train_imagenet.py", line 122, in <module>
    sym = net.get_symbol(**vars(args))
  File "/home/rengan/DL_benchmark/MLPerf/mlperf_results/v0.5.0/dellemc/submission/code/image_classification/mxnet/symbols/resnet-v1b-fl.py", line 370, in get_symbol
    use_dali          = use_dali)
  File "/home/rengan/DL_benchmark/MLPerf/mlperf_results/v0.5.0/dellemc/submission/code/image_classification/mxnet/symbols/resnet-v1b-fl.py", line 250, in resnet
    cudnn_tensor_core_only=force_tensor_core)
  File "<string>", line 179, in Convolution
  File "/home/rengan/apps/mxnet/dev/lib/python3.6/site-packages/mxnet-1.5.0-py3.6.egg/mxnet/_ctypes/symbol.py", line 125, in _symbol_creator
    ctypes.byref(sym_handle)))
  File "/home/rengan/apps/mxnet/dev/lib/python3.6/site-packages/mxnet-1.5.0-py3.6.egg/mxnet/base.py", line 252, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Cannot find argument 'cudnn_algo_verbose', Possible Arguments:
----------------
kernel : Shape(tuple), required
    Convolution kernel size: (w,), (h, w) or (d, h, w)
stride : Shape(tuple), optional, default=[]
    Convolution stride: (w,), (h, w) or (d, h, w). Defaults to 1 for each dimension.
dilate : Shape(tuple), optional, default=[]
    Convolution dilate: (w,), (h, w) or (d, h, w). Defaults to 1 for each dimension.
pad : Shape(tuple), optional, default=[]
    Zero pad for convolution: (w,), (h, w) or (d, h, w). Defaults to no padding.
num_filter : int (non-negative), required
    Convolution filter(channel) number
num_group : int (non-negative), optional, default=1
    Number of group partitions.
workspace : long (non-negative), optional, default=1024
    Maximum temporary workspace allowed (MB) in convolution.This parameter has two usages. When CUDNN is not used, it determines the effective batch size of the convolution kernel. When CUDNN is used, it controls the maximum temporary storage used for tuning the best CUDNN kernel when `limited_workspace` strategy is used.
no_bias : boolean, optional, default=0
    Whether to disable bias parameter.
cudnn_tune : {None, 'fastest', 'limited_workspace', 'off'},optional, default='None'
    Whether to pick convolution algo by running performance test.
cudnn_off : boolean, optional, default=0
    Turn off cudnn for this layer.
layout : {None, 'NCDHW', 'NCHW', 'NCW', 'NDHWC', 'NHWC'},optional, default='None'
    Set layout for input, output and weight. Empty for
    default layout: NCW for 1d, NCHW for 2d and NCDHW for 3d.NHWC and NDHWC are only supported on GPU.
, in operator Convolution(name="", layout="NHWC", no_bias="True", cudnn_algo_verbose="0", cudnn_algo_fwd="1", cudnn_algo_bwd_data="1", cudnn_algo_bwd_filter="1", stride="(2, 2)", cudnn_tensor_core_only="1", pad="(3, 3)", num_filter="64", kernel="(7, 7)", workspace="256")

Minimum reproducible example

The mxnet implementation of image classification model Resnet50 in MLPerf:
https://github.com/mlperf/results/tree/master/v0.5.0/nvidia/submission/code/image_classification/mxnet

Steps to reproduce

first install the dependence nvidia-dali:

$ pip install --extra-index-url https://developer.download.nvidia.com/compute/redist nvidia-dali

then run the benchmark:

$ python3.6 train_imagenet.py \
--gpus 0,1,2,3 \
--batch-size 1664 \
--kv-store device \
--lr 0.6 \
--lr-step-epochs 30,60,80 \
--warmup-epochs 5 \
--eval-period 4 \
--eval-offset 2 \
--optimizer sgd \
--network resnet-v1b-fl \
--num-layers 50 \
--num-epochs 100 \
--accuracy-threshold 0.749 \
--dtype float16 \
--use-dali \
--disp-batches 20 \
--image-shape 4,224,224 \
--fuse-bn-relu 1 \
--fuse-bn-add-relu 1 \
--min-random-area 0.05 \
--max-random-area 1.0 \
--conv-algo 1 \
--force-tensor-core 1 \
--input-layout NHWC \
--conv-layout NHWC \
--batchnorm-layout NHWC \
--pooling-layout NHWC \
--batchnorm-mom 0.9 \
--batchnorm-eps 1e-5 \
--data-train /mnt/isilon/DeepLearning/database/mlperf/ilsvrc2012_mxnet/train.rec \
--data-train-idx /mnt/isilon/DeepLearning/database/mlperf/ilsvrc2012_mxnet/train.idx \
--data-val /mnt/isilon/DeepLearning/database/mlperf/ilsvrc2012_mxnet/val.rec \
--data-val-idx /mnt/isilon/DeepLearning/database/mlperf/ilsvrc2012_mxnet/val.idx \
--dali-prefetch-queue 2 \
--dali-nvjpeg-memory-padding 64

What have you tried to solve it?

1. This error first appeared when I installed mxnet-cu100mkl using pip. Then I built from the source with the latest development code which has all output shown above. But it still has the same error.
2. Since this error is related to cudnn, I check whether the mxnet library is linked to libcudnn or not, and the result is yes:

[root@node001 ~]# ldd /home/rengan/apps/mxnet/dev/lib/libmxnet.so |grep cudnn
        libcudnn.so.7 => /home/rengan/apps/cudnn/7.4.1/lib64/libcudnn.so.7 (0x00002aaade829000)

[vdantu]

@mxnet-label-bot add [question]

[ptrendx]

The code from NVIDIA MLPerf submission assumes running inside the container from NGC (https://ngc.nvidia.com) and will not work (yet) in the upstream MXNet. Please consult https://ngc.nvidia.com/catalog/containers/nvidia:mxnet for information on how to download and run the NGC MXNet container.

[renganxu]

@ptrendx I know NGC container works, but I just want to run without container. Now I figured out NGC MXNet changed the Convolution operator by adding more parameters: cudnn_algo_verbose, cudnn_algo_fwd, cudnn_algo_bwd_data, cudnn_algo_bwd_filter, and cudnn_tensor_core_only. But they are not available in MXNet repo.

Do you know whether the MXNet repo has any plan to integrate these changes made by Nvidia? Thanks.

[ptrendx]

We (the DL Framework team at NVIDIA) are working to upstream all performance changes with multiple PRs already issued (and several more to go), see e.g. #13346 #13749 #13471 .
Those particular additional arguments are not really part of this upstreaming effort though (they are more of a performance debug tools for us rather than performance enablers).

[renganxu]

@ptrendx Thanks for these information. If those additional arguments are only for debug purpose, then I can remove them in the benchmark code to match the APIs in the existing MXNet repo.

[renganxu]

Hi @ptrendx Since I had performance issue when running on bare-metal, I also start to use the NGC container ngc18.11_mxnet to run the MLPerf mxnet resnet50 benchmark on our servers but it cannot converge. Each server has 4 V100-SXM2 32 GB. I ran the benchmark on two servers and set the parameters the same as DGX-1:

--batch-size=208
--kv-store=horovod
--lr=0.6
--warmup-epochs=5
--dali-prefetch-queue=2
--dali-nvjpeg-memory-padding=64

Here the batch size is 208 for each GPU, so the global batch size is 208*8=1664 which is the same batch size DGX-1 used in the MLPerf published result. But the model cannot reach the target accuracy 74.9% even with 100 epochs. The evaluation accuracy is 74.35% after 100 epochs (see the following figure). But DGX-1 reached 75.22% after only 62 epochs.

So could you give some guidance on how to choose the parameters to make the model converge and converge faster? Thanks.

[renganxu]

Hi @ptrendx, could you give some guidance on the question in my previous post? Also what is"--dali-nvjpeg-memory-padding" used for? Will it have impact to the model accuracy?

[ptrendx]

Oh, sorry, I completely missed that comment.

How do you prepare you training dataset? The way we did it in our submission is shown e.g. here: https://github.com/NVIDIA/DeepLearningExamples/tree/master/MxNet/Classification/RN50v1.5#prepare-dataset

I don't see anything obviously wrong with the options you set. How does it look like when running on single machine (basically halve the lr parameter and run on 4 GPUs)?

About the DALI options - they do not matter for convergence, they are there to avoid memory reallocations during training.

[renganxu]

Hi @ptrendx, I used the same commands as in nvidia github to create the training dataset, but still could not reproduce nvidia's results. The training on single node (4 GPUs) works well, where the model reached 75.21% accuracy with 62 epochs.

Is it possible to reproduce the same result as in nvidia's DGX-1? With single DGX-1, the --kv-store=device, but when multi-node is used, --kv-store=horovod and MPI is also used. The random seed is also different. If we want to reproduce DGX-1's result, should we use the same random seed as DGX-1 used for all 8 MPI processes (2 nodes)? But I know some cuDNN APIs also have non-deterministic results, so I am not sure whether this will work. But I will try.

So here the general question is how to reproduce the single-node result on multi-node? Here I mean almost exactly the same result, although small floating point difference is acceptable.

[ptrendx]

Hmm, this is strange. On Monday/Tuesday I am travelling, but I will try to reproduce your results on Wednesday and see what is the issue there. Horovod and device kvstore should give similar results (not exactly the same because of the addition order in reduction) and definitely should both converge.

[renganxu]

Thanks very much for helping check this issue! I really want to know why the result is not reproducible.

[ptrendx]

Hi @renganxu, could you post here exact command lines you tried (so the mpirun ... python train_imagenet.py ... line)? I did not yet have a chance to test on 2 nodes, but I run multiple single node 8 GPU jobs using horovod (so should be the same as your 2 node test) with the settings you posted and it worked as it should (convergence after 62 epochs). This is the exact line I used:

mpirun -N 8 --allow-run-as-root --tag-output --bind-to socket python train_imagenet.py --gpus 0,1,2,3,4,5,6,7 --batch-size 208 --kv-store horovod --lr 0.6 --lr-step-epochs 30,60,80 --warmup-epochs 5 --eval-period 4 --eval-offset 2 --optimizer sgd --network resnet-v1b-fl --num-layers 50 --num-epochs 100 --accuracy-threshold 0.749 --dtype float16 --use-dali --disp-batches 20 --image-shape 4,224,224 --fuse-bn-relu 1 --fuse-bn-add-relu 1 --min-random-area 0.05 --max-random-area 1.0 --conv-algo 1 --force-tensor-core 1 --input-layout NHWC --conv-layout NHWC --batchnorm-layout NHWC --pooling-layout NHWC --batchnorm-mom 0.9 --batchnorm-eps 1e-5 --data-train /data/imagenet/train-val-recordio-passthrough/train.rec --data-train-idx /data/imagenet/train-val-recordio-passthrough/train.idx --data-val /data/imagenet/train-val-recordio-passthrough/val.rec --data-val-idx /data/imagenet/train-val-recordio-passthrough/val.idx --dali-prefetch-queue 2 --dali-nvjpeg-memory-padding 64

[renganxu]

Hi @ptrendx, since my system could not install Docker, I converted the mxnet Docker container to Singularity container and then used Singularity container instead. My detailed command is:

mpirun --allow-run-as-root --tag-output -np 8 --map-by socket -report-bindings -mca pml ob1 -mca btl_openib_receive_queues X,4096,1024:X,12288,512:X,65536,512 \
singularity exec -B /cm/local/apps/cuda-driver/libs/410.66:/mnt/driver -B /mnt/isilon:/mnt/isilon sandbox-mxnet-ngc18.11 bash container_cmd.sh

The commands in the file container_cmd.sh is as follows:

export MXNET_UPDATE_ON_KVSTORE=0
export MXNET_CUDNN_AUTOTUNE_DEFAULT=0
export MXNET_EXEC_ENABLE_ADDTO=1
export MXNET_USE_TENSORRT=0
export MXNET_GPU_WORKER_NTHREADS=1
export MXNET_GPU_COPY_NTHREADS=1
export MXNET_CUDNN_AUTOTUNE_DEFAULT=0
export MXNET_OPTIMIZER_AGGREGATION_SIZE=54
export NCCL_SOCKET_IFNAME=ib0
export NCCL_BUFFSIZE=2097152
export NCCL_NET_GDR_READ=1
export HOROVOD_CYCLE_TIME=0.2
export HOROVOD_TWO_STAGE_LOOP=1
export HOROVOD_ALLREDUCE_MODE=1
export HOROVOD_FIXED_PAYLOAD=161

python train_imagenet.py \
--gpus 0,1,2,3 \
--batch-size 208 \
--kv-store horovod \
--lr 0.6 \
--lr-step-epochs 30,60,80,100 \
--warmup-epochs 5 \
--eval-period 4 \
--eval-offset 2      \
--optimizer sgd \
--network resnet-v1b-fl \
--num-layers 50 \
--num-epochs 121 \
--accuracy-threshold 0.749 \
--dtype float16 \
--use-dali \
--disp-batches 20 \
--image-shape 4,224,224 \
--fuse-bn-relu 1 \
--fuse-bn-add-relu 1 \
--min-random-area 0.05 \
--max-random-area 1.0 \
--conv-algo 1 \
--force-tensor-core 1 \
--input-layout NHWC \
--conv-layout NHWC \
--batchnorm-layout NHWC \
--pooling-layout NHWC \
--batchnorm-mom 0.9 \
--batchnorm-eps 1e-5 \
--data-train /mnt/isilon/DeepLearning/database/mlperf/ilsvrc12_recordio/train.rec \
--data-train-idx /mnt/isilon/DeepLearning/database/mlperf/ilsvrc12_recordio/train.idx \
--data-val /mnt/isilon/DeepLearning/database/mlperf/ilsvrc12_recordio/val.rec \
--data-val-idx /mnt/isilon/DeepLearning/database/mlperf/ilsvrc12_recordio/val.idx \
--dali-prefetch-queue 2 \
--dali-nvjpeg-memory-padding 64

The option "-mca btl_openib_receive_queues X,4096,1024:X,12288,512:X,65536,512" was added in MPI command because without it there will be error "A process failed to create a queue pair. This usually means either the device has run out of queue pairs (too many connections) or there are insufficient resources available to allocate a queue pair (out of memory). The latter can happen if either 1) insufficient memory is available, or 2) no more physical memory can be registered with the device."

Those environment variables were added because Nvidia set them when creating the Docker container.

I am also trying the code in https://github.com/NVIDIA/DeepLearningExamples/tree/master/MxNet/Classification/RN50v1.5. I noticed this implementation is slightly different than the implementation in MLPerf repo. I will let you know whether it works or not.

[ptrendx]

Hmmm, I'm not sure how singularity works, are you sure that horovod sees all 8 ranks there (since you do mpirun on singularity and not on the actual train_imagenet.py script)? Because I could easily see it not converging if e.g. horovod sees only a subset of ranks (which would make learning rate too high).

For example, how many copies of lines like this (with the same batch number) do you see?

[1,0]<stderr>:INFO:root:Epoch[61] Batch [100-120]       Speed: 9746.69 samples/sec      accuracy=0.730769

It should be only 1 copy of each such line (so only 1 Batch [0-20], Batch [20-40]) etc. per epoch.

[ptrendx]

Please print kv.num_workers after this line: https://github.com/mlperf/results/blob/master/v0.5.0/nvidia/submission/code/image_classification/mxnet/train_imagenet.py#L99

It should be 8.

[renganxu]

Hi @ptrendx , Horovod sees all 8 ranks there. Because with 1 node, the speed is ~5000 images/sec, and with 2 nodes, the speed becomes ~10000 images/sec. I added my log file in https://gist.github.com/renganxu/f68c4c680ad7e016bea8ee981f72c60c. Yes there is only one copy for those training steps, but there are two evaluation copies because two nodes are used. Could you help check what is the issue there?

I found the Resnet-50 implementation in Nvidia DeepLearningExamples can converge successfully on 2 nodes, and up to 8 nodes (32 V100). This implementation uses parameter-server distribution model. I can try to implement it with Horovod.

[ptrendx]

Hmmm, your accuracy seems low from the start, maybe there is something wrong with initial parameter broadcast? Could you set seeds to be the same (just hardcode them) for all ranks in https://github.com/mlperf/results/blob/master/v0.5.0/nvidia/submission/code/image_classification/mxnet/train_imagenet.py#L109-L114?