To identify number of Neuron Devices in a given instance use the neuron-ls
command.
$ neuron-ls
+--------+--------+--------+-----------+--------------+---------+---------+---------+
| NEURON | NEURON | NEURON | CONNECTED | PCI | RUNTIME | RUNTIME | RUNTIME |
| DEVICE | CORES | MEMORY | DEVICES | BDF | ADDRESS | PID | VERSION |
+--------+--------+--------+-----------+--------------+---------+---------+---------+
| 0 | 4 | 8 GB | 1 | 0000:00:1c.0 | NA | 12410 | NA |
| 1 | 4 | 8 GB | 2, 0 | 0000:00:1d.0 | NA | 12410 | NA |
| 2 | 4 | 8 GB | 3, 1 | 0000:00:1e.0 | NA | 12410 | NA |
| 3 | 4 | 8 GB | 2 | 0000:00:1f.0 | NA | 12410 | NA |
+--------+--------+--------+-----------+--------------+---------+---------+---------+
The above output is taken from an Inf1.6xlarge instance.
- NEURON DEVICE -> Logical ID assigned to the NeuronDevice. This id can be used when configuring multiple runtimes to use different NeuronDevices.
- NEURON CORES -> Number of NeuronCores present in the NeuronDevice.
- NEURON MEMORY -> Amount DRAM memory in NeuronDevice.
- CONNECTED DEVICES -> Shows other NeuronDevices connected to this NeuronDevice.
- PCI BDF -> PCI Bus Device Function (BDF) ID of the device.
- RUNTIME ADDRESS -> Shows address of runtime process using this NeuronDevice.
- RUNTIME PID -> Shows process id of runtime process using this NeuronDevice.
- RUNTIME VERSION -> Shows version of runtime process using this NeuronDevice.
Multiple NeuronCores(NC) can be combined to form a NeuronCore Group (NCG). Neuron framework layer will automatically create a default NeuronCore Group. To view list of available NCGs the following command can be used.
$ neuron-cli list-ncg
Device count 4 NC count 16
Found 4 NCG's
+--------+----------+--------------------+----------------+
| NCG ID | NC COUNT | DEVICE START INDEX | NC START INDEX |
+--------+----------+--------------------+----------------+
| 1 | 2 | 0 | 0 |
| 2 | 4 | 0 | 2 |
| 3 | 3 | 1 | 2 |
| 4 | 1 | 2 | 1 |
+--------+----------+--------------------+----------------+
The above examples shows there are 4 NCGs created on the system with the following grouping NCG ID 1: Device0:(Core0, Core1) NCG ID 2: Device0:(Core2, Core3), Device1:(Core0, Core1) NCG ID 3: Device1:(Core2, Core3), Device2:(Core0) NCG ID 2: Device1:(Core1)
Multiple models can be loaded into a single NCG, but only one can be in READY state at any given moment. Inference can be performed only on models in the READY state.
The neuron-cli list-model
command should be used to view all the models.
$ neuron-cli list-model
+----------------------------------------------+----------+--------------+---------------+-------+-------+----------------------+
| UUID | MODEL ID | MODEL STATUS | NEURON DEVICE | NC | NC | NAME |
| | | | START INDEX | INDEX | COUNT | |
+----------------------------------------------+----------+--------------+---------------+-------+-------+----------------------+
| 63c43dd60b0411eaa9160288cac7f65c30 | 10011 | STANDBY | 1 | 0 | 1 | test0_1_concat_multi |
| 63c43dd60b0411eaa9160288cac7f65c30330808637f | 10010 | STANDBY | 0 | 0 | 1 | test0_1_concat_multi |
| 63c43dd60b0411eaa9160288cac7f65ce078 | 10009 | READY | 1 | 1 | 1 | test0_1_concat_multi |
| 63c43dd60b0411eaa9160288cac7f65ca05f | 10008 | READY | 1 | 0 | 1 | test0_1_concat_multi |
| 6a9726 | 10007 | READY | 0 | 2 | 2 | onv_h1_2tpb_cpu_2tpb |
| 529c31da0b0411ea95730288cac7f65cb03b0afc627f | 10006 | READY | 0 | 0 | 0 | t-test0_5conv_h1_cpu |
+----------------------------------------------+----------+--------------+---------------+-------+-------+----------------------+
- UUID -> UUID generated for this model during compile time.
- MODEL ID -> Neuron Runtime identifier for this model.
- MODEL STATUS -> READY = The model is loaded on to the NeuronDevice and active on the NeuronCore. (Inference can be done only on models with READY state) STANDBY = The model is loaded on to the NeuronDevice but another model is currently active on the NeuronCore. (A model switch is needed to start inference)
Each model loaded consumes a different amount of memory (host and device), NeuronCore and CPU usage.
The neuron-top
command can be used to view the memory and NeuronCore usage.
$ neuron-top
neuron-top - 2020-02-12 23:03:15
NN Models: 2 total, 2 running
Number of VNCs tracked: 16
0000:00:1c.0 Utilizations: Neuron core0 0.00%, Neuron core1 0.00%, Neuron core2 0.00%, Neuron core3 0.00%,
0000:00:1e.0 Utilizations: Neuron core0 0.00%, Neuron core1 0.00%, Neuron core2 0.00%, Neuron core3 0.00%,
Model ID Model Name UUID Node ID Subgraph Exec. Unit Host Mem Device Mem Neuron core %
10018 1.0.6801.0-/home/ubuntu/benchmarking/compiler_workdir/rn50 d12cf238420d11ea8e270afe835c0a32 3 0 0000:00:1e.0:0 33554816 135290880 0.00
10017 1.0.6801.0-/home/ubuntu/benchmarking/compiler_workdir/rn50 d12cf238420d11ea8e270afe835c0a32 3 0 0000:00:1c.0:0 33554816 135290880 0.00
In the above output:
- Model ID -> Unique Identifier for models loaded in the NeuronDevice
- Model Name -> Neuron Compiler Version-compiler work directory/User defined model name
- Node ID -> For Internal use only
- UUID -> Unique Id assigned by the Neuron Compiler for a Model
- Exec. Unit -> BDF of Neuron Device followed by the Neuron Core ID, b:d:f.NC
- Host Mem -> Host memory consumed by the Model in bytes
- Device Mem -> NeuronDevice memory consumed by the Model in bytes
- Neuron Core % -> Utilization % of the neuron core at sample time. If there are no active inferences this value will be 0.
Please refer to Neuron Gatherinfo