Exception in thread "main" ai.djl.translate.TranslateException: ai.djl.engine.EngineException: MXNet engine call failed: MXNetError: Check failed: (*in_type)[i] == dtype (0 vs. 3) : This layer requires uniform type. Expected 'uint8' v.s. given 'float32' at 'weight'

[DongXiaona123]

I did a training like pikachu and saved the training parameters ".params" file. My purpose is to identify the driver in the picture. Now I use my local model and start inference.I override processInput and processOutput in Translate. My input data is an image. After rewriting the processInput method, an error occurs, as follows（I don't understand what causes this. please help me）:

Exception in thread "main" ai.djl.translate.TranslateException: ai.djl.engine.EngineException: MXNet engine call failed: MXNetError: Check failed: (*in_type)[i] == dtype (0 vs. 3) : This layer requires uniform type. Expected 'uint8' v.s. given 'float32' at 'weight'
Stack trace:
File "C:\source\mxnet\src\operator\nn\convolution.cc", line 301

at ai.djl.inference.Predictor.batchPredict(Predictor.java:191)
at ai.djl.inference.Predictor.predict(Predictor.java:128)
at ai.djl.driver.inference.DriverDetection.main(DriverDetection.java:102)

Caused by: ai.djl.engine.EngineException: MXNet engine call failed: MXNetError: Check failed: (*in_type)[i] == dtype (0 vs. 3) : This layer requires uniform type. Expected 'uint8' v.s. given 'float32' at 'weight'
Stack trace:
File "C:\source\mxnet\src\operator\nn\convolution.cc", line 301

at ai.djl.mxnet.jna.JnaUtils.checkCall(JnaUtils.java:1942)
at ai.djl.mxnet.jna.JnaUtils.imperativeInvoke(JnaUtils.java:521)
at ai.djl.mxnet.jna.FunctionInfo.invoke(FunctionInfo.java:75)
at ai.djl.mxnet.engine.MxNDManager.invoke(MxNDManager.java:346)
at ai.djl.mxnet.engine.MxNDArrayEx.convolution(MxNDArrayEx.java:567)
at ai.djl.nn.convolutional.Convolution.convolution(Convolution.java:285)
at ai.djl.nn.convolutional.Convolution.forwardInternal(Convolution.java:150)
at ai.djl.nn.AbstractBaseBlock.forward(AbstractBaseBlock.java:79)
at ai.djl.nn.Block.forward(Block.java:127)
at ai.djl.nn.SequentialBlock.forwardInternal(SequentialBlock.java:191)
at ai.djl.nn.AbstractBaseBlock.forward(AbstractBaseBlock.java:79)
at ai.djl.nn.Block.forward(Block.java:127)
at ai.djl.nn.SequentialBlock.forwardInternal(SequentialBlock.java:191)
at ai.djl.nn.AbstractBaseBlock.forward(AbstractBaseBlock.java:79)
at ai.djl.nn.Block.forward(Block.java:127)
at ai.djl.basicmodelzoo.cv.object_detection.ssd.SingleShotDetection.forwardInternal(SingleShotDetection.java:80)
at ai.djl.nn.AbstractBaseBlock.forward(AbstractBaseBlock.java:79)
at ai.djl.nn.Block.forward(Block.java:127)
at ai.djl.inference.Predictor.predictInternal(Predictor.java:142)
at ai.djl.inference.Predictor.batchPredict(Predictor.java:182)
... 2 more

Mycode:
public static void main(String[] args) throws IOException, ModelException, TranslateException {
Engine engine = Engine.getEngine("MXNet");
Path modelPath = Paths.get("build/model");
Model model = Model.newInstance("driver-ssd");
model.setBlock(TrainDriver.getSsdTrainBlock());
model.load(modelPath);
Path imageFile = Paths.get("D:\1.jpg");
Image img = ImageFactory.getInstance().fromFile(imageFile);
SingleShotDetectionTranslator.Builder builder = SingleShotDetectionTranslator.builder()
.addTransform(new ToTensor())
.optSynset(Collections.singletonList("driver"))
.optThreshold(0.7f);

    SingleShotDetectionTranslator translator = new SingleShotDetectionTranslator(builder) {

        @Override
        public NDList processInput(TranslatorContext translatorContext, Image image) {
            NDArray array = image.toNDArray(translatorContext.getNDManager());
            array = NDImageUtils.resize(array,3);
            array.toType(DataType.UINT8,true);
            NDList ndList = new NDList(array);
            return ndList;
        }

        @Override
        public DetectedObjects processOutput(TranslatorContext ctx, NDList list) {
            NDArray anchors = list.get(0);
            NDArray classPredictions = list.get(1).softmax(-1).transpose(0, 2, 1);
            NDArray boundingBoxPredictions = list.get(2);
            MultiBoxDetection multiBoxDetection =
                    MultiBoxDetection.builder().build();
            NDList detections =
                    multiBoxDetection.detection(
                            new NDList(
                                    classPredictions,
                                    boundingBoxPredictions,
                                    anchors));
            list = detections.singletonOrThrow().split(new long[]{1, 2}, 2);
            NDList output = new NDList(list.size());
            for (NDArray array : list) {
                output.add(array.squeeze(0)); // remove batch dim
            }
            return super.processOutput(ctx, output);
        }
    };

    try (Predictor<Image,DetectedObjects> predictor = model.newPredictor(translator)) {
        DetectedObjects detection = predictor.predict(img);
        saveBoundingBoxImage(img,detection);
        System.out.println(detection);
    }
}

[zachgk]

The issue should be that there is a difference between the inputs given during training and during inference. You can try changing the third line to array = array.toType(DataType.UINT8, false) in your processInput. It may also be worth using the implementation of processInput that is built-into the SingleShotDetectionTranslator (inherited from ObjectDetectionTranslator)

[DongXiaona123]

I am very happy to receive your answer. The processInput in the SingleShotDetectionTranslator worked, and there was no problem with the processInput, but there was a problem with the processOutput. The problem is as follows:

Exception in thread "main" ai.djl.translate.TranslateException: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
	at ai.djl.inference.Predictor.batchPredict(Predictor.java:191)
	at ai.djl.inference.Predictor.predict(Predictor.java:128)
	at ai.djl.driver.inference.DriverDetection.main(DriverDetection.java:55)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
	at ai.djl.modality.cv.translator.SingleShotDetectionTranslator.processOutput(SingleShotDetectionTranslator.java:64)
	at ai.djl.modality.cv.translator.SingleShotDetectionTranslator.processOutput(SingleShotDetectionTranslator.java:30)
	at ai.djl.inference.Predictor.processOutputs(Predictor.java:306)
	at ai.djl.inference.Predictor.batchPredict(Predictor.java:185)
	... 2 more

error location：

I found that the value of x is the same as the value of float classId. Did the NDArray boundingBoxes not get the correct value? Or there may be a problem with my training model. I don't understand where the problem is, please help me analyze it. Thank you.

My new Code:

public static void main(String[] args) throws IOException, ModelException, TranslateException {
        Engine engine = Engine.getEngine("MXNet");
        Path modelPath = Paths.get("build/model");
        Model model = Model.newInstance("driver-ssd");
        model.setBlock(TrainDriver.getSsdTrainBlock());
        model.load(modelPath);
        Path imageFile = Paths.get("D:\\1.jpg");
        Image img = ImageFactory.getInstance().fromFile(imageFile);
        SingleShotDetectionTranslator.Builder builder = SingleShotDetectionTranslator.builder()
                .addTransform(new ToTensor())
                .optSynset(Collections.singletonList("driver")) // there only one class in Pikachu SSD
                .optThreshold(0.7f);

        SingleShotDetectionTranslator translator = new SingleShotDetectionTranslator(builder);
        try (Predictor<Image,DetectedObjects> predictor = model.newPredictor(translator)) {
            DetectedObjects detection = predictor.predict(img);
            saveBoundingBoxImage(img,detection);
            System.out.println(detection);
        }
    }

[zachgk]

The pre-processing and post-processing depend on the specific model that you are using. Even with two different implementations of SSD, they may require different pre-processing and post-processing. It may be worth looking at how your specific model formats the output data, because it may not be the same as the one this post-processing is expecting. It may have the data in a different order, a different shape, maybe it will use different units (pixels vs fraction of image) for the bounding boxes, etc. Try printing out some of the output data from your model and the output shapes and see how it looks.

[DongXiaona123]

Thank you for your help!
I compared my output with that of other pre trained models and found that there was a problem with my output. The problem may arise on the model I trained.
my output：

Output of pre trained models：

By comparing the two outputs, my data should be incorrect.

Does it mean that the model I trained is incorrect?

Firstly, I need to confirm the dataset. Here are some data from my dataset:

{
        "image": "3.jpg",
        "annotations": [
            {
                "label": "driver",
                "coordinates": {
                    "x": 915.2471910112359,
                    "y": 426.0393258426966,
                    "width": 729.5056179775281,
                    "height": 587.9213483146068
                }
            }
        ]
    }

My image size is 1280 * 720 (width * height). Use labelImg to label and obtain the JSON above. I think this JSON is incorrect. If x=915 width=729, then x+width>1280.
May I ask if my understanding is correct? What should my dataset look like when I imitate the training done by Pikachu? Can you provide me with an example for reference?