GSoc Project of 3D Jupyter notebooks and WebGL Integration of OpenVINO

[zhuo-yoyowz]

This is where we could show and discuss progress and challenges of GSoC on " 3D Jupyter notebooks and WebGL Integration of OpenVINO".

[spencergotowork]

Hi! Mentors , I worked with the OpenVino data processing as planned before.（human-pose-3D)

I am currently faced with the following problems:

1. Some Python modules are shared libraty, like *.so in linux or *.dll in windows write by C++, so the user had to compile them by their own. Because some data processing methods are from open_model_zoo (like pose extractor method). I communicated with open_model_zoo, they reply that ,if I use python, it will be very slow. Are there any good cross-platform solutions?(Or we build it for different systems)

2. The task I understood was a 3D engine, right? There are some another 3rd party 3D engines based on WebGL . If I can manage data processing and feed it to the 3rd party 3D engine? (As some packege, you have to use "pip" to download the package before you use them) Then add widgets to interact with them.

3. So where should the focus of the work be, the processing of data(According to every specific types of models) or the 3D drawing on Jupyter(new engines)

4. What is the final product form, a new package or a new Jupyter plugin? I know we can write some python code in "utils" directory in notebooks.

Thank you for your reading.

[raymondlo84]

#2 sounds like the best option moving forward. One goal for OpenVINO Notebooks is to reduce dependencies! So c++ and all dll issues shall be avoided at all cost.

You define the final product ;) I think as a developer (especially you are new to 3D world), let's see what is the easiest way to get something shown on screen! Often that is the #1 issue!

Also, i would recommend try building the first demo like... 3D human pose + 3D visualization with a webcam. That way you can see your results!

[zhuo-yoyowz]

June 30, Weekly meeting minutes:

1. Ran showed a live demo of running a Juptyer notebook for 3D pose estimation. Input is recorded video. Output is the edges and nodes to represent people's poses. A python 3D engine package is used.
2. Ran showed a notebook to map 2D to 3D, which could be used to map 2D coordinates of nodes and relations between nodes into 3D. Have some little bug in demo.
3. Ran will write a weekly blog on Ray suggested medium for progress so far, and what to do next.

[zhuo-yoyowz]

July 8th, Weekly meeting minutes:

Ran showed the progress on

1. Finished the debugging on data processing functions.
2. Started to include OpenCV functions in the demo to present the demo across different web browsers, without too much dependency. Current challenge is on wrong display effect. Needs debugging for that.
3. Discussed the possibility of 3D presentation in monodepth effect. Once the OpenCV functions are done, it may be easy to realize the monodepth presentation effect.

Ran will write a weekly blog on Ray suggested medium for progress so far, and what to do next.

[zhuo-yoyowz]

Aug 11th, Weekly meeting minutes:

Ran showed the progress on

• Modified code modifying where necessary
• Submitted a PR on this demo 406 Human Pose Estimation 3D #606
• Fixed some bugs in display effect with OpenCV
• Tested existing codes in Binder

Next step plan

• Test existing codes with different cameras, parameters, etc.
• Add another 3D demo to show the reusability of the notebook demo (considering 3D segmentation for now)

Ran will write a weekly blog on Ray suggested medium for progress so far, and what to do next.

[raymondlo84]

Looking forward to seeing his new blog :)

[zhuo-yoyowz]

Aug 25th, Weekly meeting minutes:

Ran showed two potential 3D models he was trying to use in his notebook. The 3D models belong the Media Pipe library, and could be referred to https://google.github.io/mediapipe/solutions/objectron.html
These 3D models could be exported to onnx format, then to IR format. Following work would be to draw lines to visualize the 3D inference results.
At the same time, Ran will continue to help with rewriting the pre-processing function of the dino map testing images.