About the methods of loading model

[freesia-sherry]

Currently, I find that the methods of loading model can only based on model's path. Such as below:

Criteria<NDArray, NDList> criteria = Criteria.builder()
                .setTypes(NDArray.class, NDList.class)
                .optModelPath(Paths.get("build/pytorch_models/tinyNet"))
                .optProgress(new ProgressBar())
                .optTranslator(translator)
                .build();

But I wonder that whether any methods to load model by streaming. For instance, the boosting model LightGBM can be converted to PMML format file, then be saved into Redis. In that case, java can get model from Redis and run it directly. So if any similar methods in DJL? Thank you!

[frankfliu]

@freesia-sherry

1. Criteria api support load model from url: Criteria.Builder.optModelUrls(). You can implement "redis://" Repository to support load model from Redis, see: https://docs.djl.ai/master/docs/load_model.html#implement-your-own-repository
2. We are considering add load model from InputStream, however there are many constraints:
   • Not all engines support load model from Stream, currently we only added InputStream support for PyTorch
   • Usually a model artifacts contains multiple files, InputStream API doesn't work for this case
   • It's hard for streaming api to handle the model file in a zip or tar stream

[freesia-sherry]

Thanks for the reply. I want to know the details about InputStream support for PyTorch. The load method whether can like below? Not from local, and not from the URL.

pytorch_model = getFromRedis(key)               // get pytorch model from Redis
Criteria.Builder.load_model(pytorch_model)  // load model based on object

the pytorch_model might be a bytes object or other objects can be saved into Redis. Because I worry about the performance of speed of loading model from URL or local. Thank you!

[frankfliu]

See: https://github.com/deepjavalibrary/djl/blob/master/engines/pytorch/pytorch-engine/src/main/java/ai/djl/pytorch/engine/PtModel.java#L119

Since we have not expose the API at DJL api level yet, you have to use PyTorch specific code: https://github.com/deepjavalibrary/djl/blob/master/engines/pytorch/pytorch-engine/src/test/java/ai/djl/pytorch/integration/PtModelTest.java#L45-L52

      try (PtModel model = (PtModel) Model.newInstance("test model", "PyTorch")) {
          model.load(Files.newInputStream(modelFile));
          try (Predictor<NDList, NDList> predictor = model.newPredictor(new NoopTranslator())) {
              NDArray array = model.getNDManager().ones(new Shape(1, 3, 224, 224));
              NDArray result = predictor.predict(new NDList(array)).singletonOrThrow();
              Assert.assertEquals(result.getShape(), new Shape(1, 1000));
          }
      }

[freesia-sherry]

Thanks, let me study it.