InstantMeshes plugin development

[djoerg]

Hi,
first, thanks for trying to bring more features to MeshRoom. Highly appreciate it!

I'm curious if the obj format that InstantMeshes exports have ever worked on the Texturing node.
If've tried using InstantMeshes manualy, before i found your plugin, and the Texturing node always failed to load it. Sadly it's the same with your plugin.
After fiddling around a little bit, it seems to me that the Texturing node can't handle quad meshes. After converting the IM output to a pure triangle mesh via Meshlab and exporting it without normals or color, the Texturing node works. Including normals also leads to failure!
I've looked at the exported obj in an text editor and it turns out that IM is also exporting the mesh with normals!

How have you used your InstantMeshes node inside Meshroom?
Have you ever feed it directly into the Texturing node or have you put some other nodes in between?
Or, maybe it worked in an older version of MeshRoom? (I tried it with the 2019.2 version)

I really hope there's a solution for this. InstantMeshes works so well for automatic retopology, i would love to use it in a MeshRoom workflow.

Btw, i'm a software developer and i love the fact that it's so easy to implemet new nodes with python, so i would like to implement some own nodes or wrap some external CLI tools into nodes, but i have nearly no knowlegde about MeshRoom. It seems to me, that you have a fairly good understanding how MeshRoom works. Maybe we could work together (or share some knowledge) bringing more features to MeshRoom?

[natowi]

Hi,
first, thanks for trying to bring more features to MeshRoom. Highly appreciate it!

You are welcome.

I'm curious if the obj format that InstantMeshes exports have ever worked on the Texturing node.

No, at the moment there are no nodes compatible with the InstantMeshes quad mesh output.
It was based on this feature request alicevision/Meshroom#774
Support for texturing quad meshes would require changes on the Alicevision side of the texturing node. (Meshroom (python, qt) is the GUI for Alicevision (c++))
However it might be possible to use https://github.com/caosdoar/Fornos to bake the high resolution texture from the Texturing node to the low poly from the InstantMeshes node.

Btw, i'm a software developer and i love the fact that it's so easy to implement new nodes with python, so i would like to implement some own nodes or wrap some external CLI tools into nodes, but i have nearly no knowledge about Meshroom.

You are welcome to contribute. What do you have in mind?

Here are the basics on how to implement new nodes:

1. CommandLineNode: nodes run external cli programs - this is how most Meshroom nodes work.
   Simple example

 class Convert2MVE(desc.CommandLineNode):
     commandLine = 'aliceVision_exportMVE2 {allParams}' 

Runs aliceVision_exportMVE2.exe with all parameters. {allParams} is only supported by alicevision plugins.

For external programs cli parameters need to be adjusted (example):

class ImageMasking(desc.CommandLineNode):
    commandLine = 'mogrify -format png -path {outputValue} -type Grayscale -negate -fill black -fuzz {fuzzValue}% +opaque "#ffffff" -blur {radiusValue}x{sigmaValue} -type Bilevel -depth 1 {inputValue}/*jpg'

Runs mogrify.exe with defined parameters

2. Native Python nodes

It is possible to implement new nodes in python without running an external program from the cli.

Here is a good examples for pure python nodes: https://github.com/alicevision/meshroom/blob/develop/meshroom/nodes/aliceVision/SketchfabUpload.py
and https://github.com/alicevision/meshroom/pull/641/files (import modules)

References on different supported GUI elements for nodes can be found here

Maybe we could work together (or share some knowledge) bringing more features to MeshRoom?

I am happy to share some knowledge (although I am still learning myself and mainly work on the documentation). There is no lack of ideas for new features, but you are always welcome to implement your own projects.

Implementation notes:
It is easy to write a node for an existing cli tool, but if you want to contribute your node to the main Meshroom repository, a native Meshroom implementations is required. They can be pure python nodes or c++ implementations on the alicevision side with python cli nodes for Meshroom. This is to ensure multi platform compatibility and reduce dependencies on third parties. Licenses should be compatible with MPL2.

If you have any questions, feel free to ask.

[djoerg]

Wow thanks for the quick reply and all the information. I need some time to go through all of this.

I assumed that the Texturing node can't handle quads and i think it's not trivial to implement this.
But the problem with obj file including normals... should be relatively easy to implement a simple 'ignore normals' feature!?
The reasoning behind this is, that we could use InstantMeshes in pure triangle mode, but even this exports contains normals!
Do you think it's a valuable feature request?

You are absolutely right that implementing native stuff in c++ or python is the best way to go.
I've just started developing c++ a few weeks ago, so much to learn. My python skills are also very limited, just done some utility/helper stuff insde Blender.I'm comming from a PHP/Java/C# background and worked the last 10 years almost only in VisualStudio on C# projects.

So could you point me to some documentation on how to setup a development environment for Meshroom under Win7 with the VisualStudio Community Edition?
Is this possible or must i learn all this (for me very strange) cmake commandline stuff?

My goals/dreams with meshroom:

• long term goal: make it a full open source pipeline that is competitive with the commercial ones and is easaly useable by unexperienced users without much knowledge about photogrammetry
• better mesh quality (at the moment, every mesh i created contains hundreds of wrongly oriented polys that need a cleaup by hand, around 4-8 hours manual work per mesh, and then re-running the Texturing node)
• auto generation of low poly output (hence the intrest in InstantMeshes) with automatic generation of normal and/or displacement maps
• automatic (at least as much as possible) texture delightning
• using exposure brackets as input to generate HDR textures (to use as a 3D HDR Environment in VFX production. to circumvent the limitations of HDRi Environment maps)
• maybe an 'flat-only' workflow for pure texture generation (no mesh output) but with automatic normal/displacement maps (usefull for texture generation from walls, floors and other flat surfaces)
• and much more ;)

Btw, i'm a hobby photographer since 10 years, so i would happily share my photo/image processing skills to. I procduced many 360 HDRi envirment maps by myself and also coded an own HDR processing tool.

I could also imagine to contribute some of my own SFM image sets with a CC0 license.
(Sadly most of them are lost due to a hardrive crash, but i surely will shot more in the near future)

Phuu, enough for now. :)
I realy appreciate your help here, thank you so much

[natowi]

I assumed that the Texturing node can't handle quads and i think it's not trivial to implement this.
But the problem with obj file including normals... should be relatively easy to implement a simple 'ignore normals' feature!?
The reasoning behind this is, that we could use InstantMeshes in pure triangle mode, but even this exports contains normals!
Do you think it's a valuable feature request?

It might be possible to use https://github.com/caosdoar/Fornos to bake the high resolution texture from the Texturing node to the low poly from the InstantMeshes node.

Having a tight InstantMeshes plugin (maybe even with preview) would be interesting. This is something we could discuss with the main MR devs.

So could you point me to some documentation on how to setup a development environment for Meshroom under Win7 with the VisualStudio Community Edition?
Is this possible or must i learn all this (for me very strange) cmake commandline stuff?

Sure, I put this wiki together: https://github.com/natowi/meshroom_external_plugins/wiki
btw you should update to win10 if possible as many packages dropped win7 support or cause unexpected errors on win7

auto generation of low poly output (hence the intrest in InstantMeshes) with automatic generation of normal and/or displacement maps

Support for normals is wip alicevision/AliceVision#578

automatic (at least as much as possible) texture delightning

I think there is some basic support for this

using exposure brackets as input to generate HDR textures (to use as a 3D HDR Environment in VFX production. to circumvent the limitations of HDRi Environment maps)

Sounds useful and not too difficult to implement (especially since you have experience in this area)

There are some existing nodes related to HDR: LDRtoHDR and HDRI 360 stitching (will be available with the next release or when building from source)

maybe an 'flat-only' workflow for pure texture generation (no mesh output) but with automatic normal/displacement maps (usefull for texture generation from walls, floors and other flat surfaces)
and much more ;)

That is interesting, too

Btw, i'm a hobby photographer since 10 years, so i would happily share my photo/image processing skills to.

I am working on a guide "capturing images for photogrammetry" for the Meshroom documentation (to reduce bug reports from new users that are based on unsuitable image datasets and provide a general guideline). If you are interested, I am happy to add you to the project.

I could also imagine to contribute some of my own SFM image sets with a CC0 license.

A HDRI / exposure brackets sample dataset might be interesting.

[djoerg]

Sorry for comming back a little late, had not much time the last days.
But i managed to get the InstantMeshes node working with the Texturing node. 😄
InstantMeshes_0.3.py.zip
I used a rudimentary Load/Save OBJ script by James Gregson as a starting point to convert the IM output OBJ to a meshroom compliant format. I had to modify it a little bit and also removed the numpy dependency to keep it simple.
see: http://jamesgregson.ca/loadsave-wavefront-obj-files-in-python.html
(Haven't added any credits in the comments yet, think i should do that eventualy.)

I added some info to your original top-level comment, but i don't know if i have done it in a good way. Want to hear your thoughts an that!

For now, only the 'Triangle' mode works fine. The two Quad modes are working only sporadicaly. Seems there are problems with wrongly orientated and/or non-manifold faces. That would be the next thing i try to fix.

Added/changed some commandline params and the descriptions. Maybe you can review that too

A come back in a few hours to comment on your last post, i'm a little busy at the moment

Thanks for all your helpfull information

[natowi]

Good work. You are welcome to open a Pull Request for the InstantMeshes plugin to contribute your improvements.

The comments can be re-written later on.

Found out that the binary windows distribution of Meshroom can use uncompiled python source (*.py) files.

That is neat, nice find. I however did not yet get the IM node to work with the Texturing node.

A come back in a few hours to comment on your last post, i'm a little busy at the moment

There is no hurry, take your time.

[djoerg]

Good work. You are welcome to open a Pull Request for the InstantMeshes plugin to contribute your improvements.

Thanks. Unfortunatly, i'm not familiar with Git, used mostly TFS and SVN in the past. Must learn howto do a propper Pull request first. Don't want to mess up your repo.

I however did not yet get the IM node to work with the Texturing node.

Ohh, that's bad. Did my IM node caused problems or did the Texturing node throw any error? (With "Remesh mode=Triangle" it worked every time i tested it. I used only tiny meshes <50k)
Can you provide your input and output mesh-files (before/after my IM node) from the Meshroom-cache? Every error is good to make it more robust!
The Texturing-node log and the Meshroom output in the commandline window would also be helpfull.

[natowi]

Go to a file and click on the pen button to edit, this creates a new branch for you to edit. You can than save edits and later create a pull request. (#) No need to worry about messing up something.

I used this dataset for testing https://github.com/alicevision/dataset_monstree with three images
-however texturing works with six+ images.

The quad mesh option takes forever with Smoothing iterations at 2. Set to 0 for fast computation. Works with Texturing.

[djoerg]

Go to a file and click on the pen button to edit, this creates a new branch for you to edit. You can than save edits and later create a pull request. (#) No need to worry about messing up something.

Cool, will try that after getting it a little bit more robust. Maybe tomorrow.

I used this dataset for testing https://github.com/alicevision/dataset_monstree with three images
-however texturing works with six+ images.

I assume you keept all settings at default? The DepthMap with downscale=2? Or have you used a different resulotion?
To pin down these errors caused only by a few wrong faces, it's best to have the exact input mesh that produced the error. I'm not sure if the meshroom pipeline is fully deterministic! So even using the same settings as you can produce a slightly different mesh!!! (I have the gut feeling that the meshing isn't deterministic, cause i had errors on older projects that went away after executing the Meshing node a second time. Or maybe it was the MeshFiltering node, can't remeber exactly.)

Nevertheless, i think i found the main problem 😄

• InstantMeshes produces (more) self intersecting faces with smoothing, setting it to 0 worked most of the time
• The "Basic" unwrap method of the Texturing node seems more robust than the other two, "LSCM" definitly crashes on self-intersecting faces with "OpenNL assertion failed"! (Tested this by removing self-intersections with MeshLab, after that LSCM worked.)

I will try to find a way to remove the self-intersections, maybe using some third-party lib.

Hopefully, i come up tomorrow with a working Pull request 😉

[djoerg]

Short update:
After searching the web for about 5 hours, i couldn't find a lightweight, easy to use, portable, mesh library for python that could repair self-intersections. Found CGAL, LibIgl, and others but all have so much dependencies and aren't easy to install or have no longer maintained python wrappers, that i don't want to use them.

But i stumbled accross MeshFix, a simple (award winning) commandline tool to repair the common defects in 3D scanned meshes. Unfortunatly, it needs the mesh to be a single closed object and that's very rarly the case using photgrammetry, cause you need a closed scan with the bottom of the object and totaly separeted from the background. 😞 (AFAIK, nearly impossible to get in Meshroom without manual cleanup in an external tool like Meshlab.)
https://github.com/MarcoAttene/MeshFix-V2.1

Luckily, i found PyMeshFix, a python wrapper for MeshFix, that make it possible to run the various repair steps separetly. So i want to give it a try using ONLY the repair self-intersections step. 🙏
https://github.com/pyvista/pymeshfix

If everything runs smooth, i would come up with a PoC in a few hours.

[natowi]

Yes, I also found MeshFix, it produces good results but not on all meshes. It works best on mostly closed meshed.

There is also Meshlabserver that can be used to run Meshlab from cli. It might be interesting to write some nodes that access Meshlab. https://stackoverflow.com/questions/20797535/meshlab-invoke-meshlab-command-line-version

[djoerg]

Hmm, my 'few hours' statement was a little bit too optimistic. Had problems removing the (in our case) unnesessary UI dependecies in PyMeshFix. But akaszynski was so kind to deliver a new package without the dependencies. 😄
see: pyvista/pymeshfix#16

@akaszynski Thanks again for that

There is also Meshlabserver that can be used to run Meshlab from cli.

Yes, thought about that too. Used Meshlab sometimes in the past and i think i have a little bit of understanding of the various cleanup and remeshing options in Meshlab. Could possibly be usefull, albeit the functions in meshlab seem a little bit outdated compared to the more sophisticated algorithms in other tools.
For example: Removing self-intersections deletes ALL intersecting faces (without keeping the usefull ones) leaving holes in the mesh. Using the 'fill holes' function just uses the boundary vertecies without generating new topology and generating NEW self-intersections(!), so IMHO pretty useless. There are of course many other functions that could be usefull.

Btw, found a nice research paper for an algorithm removing self-intersections (and as a side effect also non-manifolds) without the need to have a closed mesh. (As far as i'm understanding it)
https://iopscience.iop.org/article/10.1088/1742-6596/1314/1/012149/pdf
But i found no tool that has implemented it yet. Could be a cool and educational project for the future to implement this algorithm be myself. 😉

Ok, now i try to integrate the PyMeshFix functions and come up with a new version the next days

[djoerg]

Quick update: With the new PyMeshFix version 0.13.4, i managed to call the clean self-intersections separetly which, luckily, did excactly what i hoped for: cleaning the mesh without generating a closed one. 😄

The downside is, we now have two dependencies, PyMeshfix and numpy, that need to be copied into the meshroom 'lib' folder. Maybe i could make this optional.

Now a i have to do some more tests with bigger/complexer meshes and cleanup the code a little bit (had to completly rewrite the OBJ load/save stuff).

I will definitly try to do a Pull request with the new 0.4 version today.
After that, i think i will make a pure MeshFix node also, with ALL clean methods to make it possible to generate fully closed (and hopefully watertight) single meshes.

[djoerg]

Ahh, one question:
The next bigger thing i will try is to automate the normal-map creation. I know, you mentioned that there is still WIP, but having more options is always better, i think.

So my quesition is, did you know any cli tools that can bake normals from a highpoly to a lowpoly mesh and be able to use obj as input/output?

[natowi]

Take a look at this list: https://gist.github.com/natowi/a8d6c1fffb91e58be8f10c45f21ba85f
Fornos is a good highpoly to lowpoly baking tool, but does not have cli support (could possibly be added to the cpp code)

[djoerg]

Hi, sorry for not replying so long. Bussy at work and a good friend visited me for a few days.

Yes, i've taken a look at the fornos source code and there is sadly no cli handling implemented. Maybe i could do that, but for a quick solution i would give xNormal a shot. It has also no 'real' cli handling, but it's possible to use a xml file to control the parameters. So that's the next thing i like to try.

Sooo, back to the InstantMeshes plugin. I've attached version 0.4 that now has detailed installation instructions included. I will do a pull request next.
Sadly, it's not working as i hopped. My first tests that resulted in a NOT closed mesh where missleading. The mesh keeps unchanged by the meshfix functiion if there are no invalid faces. If the mesh needs to be repaired, the result is always a closed, watertight mesh. So i'm still looking for another solution to repair self-intersections and nonmanifold faces. Maybe utilizing another graphics library like CGAL or libigl.

InstantMeshes_0.4.py.zip

[barefoots]

hello natowi & djoerg
Thankyou for working on the Instant Meshes node!

I got the new 0.4 version working with the texturing node using the 'outputMesh' output but not the 'outputInstantMeshes' output.

Is there something I am missing?
Thankyou.

[natowi]

@barefoots outputMesh is the Meshroom compatible obj output, outputInstantMeshes the original output from InstantMeshes.

[barefoots]

aaaaaaaaaaaahhhh thank-you!

[natowi]

@djoerg would you be interested in making a (WIP) PR in the main Meshroom Repository? Then we can clean up the code/comments and add all the dependencies to the requirements.

[TigerVersusT]

@djoerg Thanks for your contribution，it looks like you have so many ideas about improving this plugin ( and not finished ).I also want to do something to improve it, could you assign me some of your tasks? and if possible some suggestion about how to implement them well.

[TigerVersusT]

@natowi HI natowi, which python version are you uisng? I received error messages when runing " pip install pymeshfix ", it seems that there is no supported version of vtk for python 38. vtk from pypi:https://pypi.org/project/vtk/#files

[natowi]

@TigerVersusT take a look at the source code, in the comments there are install instructions. (line 55+)

[TigerVersusT]

@TigerVersusT take a look at the source code, in the comments there are install instructions. (line 55+)

oh, thanks!

[natowi]

@TigerVersusT sorry, I forgot to answer you. Did the respond by ChemicalXandco help you? Yes, it would be good to write the debug messages into a log file.

[TigerVersusT]

@natowi It's OK, and I will implement this feature as soon as possible

[TigerVersusT]

[

InstantMeshes_0.4.1.zip

](url)
@natowi Hi,natowi. I have done the job, my modification is based on the last file provided by @djoerg .And I added a new class inherited from the LogManager to handle our debug messages, all messages are stored in a new file called 'debug'. I think may be I can make the default instantMeshes log has the same format as other nodes provided in the AliceVisio framwork.

[natowi]

@TigerVersusT that´s great! You are welcome push this to the instant meshes branch

[TigerVersusT]

@natowi oh, I'll do this later, because I have some other features to implement for my bachelor’s degree

[TigerVersusT]

@natowi Hi,natowi! Can I ask you some other questions? I really need to freeze my meshroom into executeable , and I used command "python setup.py install" to do this but find out that there is no dlls for Alice Vision framework. Could you point out where am I wrong? And how can I release my version to others( just like the official prebuilt binary )? Thanks.

[natowi]

@TigerVersusT Meshroom is just the GUI for alicevision. Building Meshroom is tedious work, that´s why we are working on creating auto-builds using appveyor.
If you want o build Meshroom on your own, take a look at my quick guide here https://github.com/natowi/meshroom_external_plugins/wiki/3-Build

You can download the latest win binaries QTOIIO, QMLAlembic, Meshroom, but for Alicevision the auto-build is not yet complete (some files missing) but you can build it yourself or try my build from a few months ago. For QtAliceVision we do not have an auto build yet.

If you did not make any changes to Meshroom or alicevision, you do not need to build your own binaries. You can simply place your new nodes as py in the node folder and Meshroom will create pyc files for you. You can use the official binaries for that.

I have some other features to implement

That is interesting, what are your plans?

[TigerVersusT]

@natowi Thanks a lot for your information, may be I should reconsider my original plan of releasing my work to others. And the "Other features " currently is just 3D object compressing and decompressing( for test ) nodes using Draco, and for some reasons I also want to implement your original idea of "baking the high resolution texture from the Texturing node to the low poly from the InstantMeshes node" mentioned above。

[natowi]

@TigerVersusT

may be I should reconsider my original plan of releasing my work to others

No need to abandon the idea. I´d recommend you to fork meshroom and upload your changes to your fork. When you are done, we can see if we can include your work in the main project.

Sadly I have not yet found a suitable texture baking tool.

You can check out the wiki I linked to. There you can find helpful information and some interesting tools

Btw. what is the topic of your thesis?

[TigerVersusT]

Well , it.s "integration of 3D reconstruction software", and in my version of meshroom, except of instant-meshes node, I also changed some UI elements to call the python wrapped instant-meshes module and it is not suitable to be included

[TigerVersusT]

@natowi Hi! natowi, I've met a wired problem: I followed your guide to complie the latest meshroom source code, the execuatable did worked but crashed when executing the "feature extraction" node( however it works sometimes ). The source works well for the same dataset. Is there anyting I've missed? Or have you ever met such problem?

[natowi]

@TigerVersusT Yes, I also had some strange behaviour with feature extraction. I think there is already an issue for that.
If you share your additions to Meshroom somewhere, I can try to build it in my build environment.

[TigerVersusT]

@natowi Thanks a lot！this is my source code
modifications.zip and this is my wrapped module
instant-meshes-extension.zip. I write a README in the first file, for any question with my code, please contact me. And I think I should thank you in my graduation thesis for all your help 😄.

[natowi]

You are welcome. I´ll take a look at your code later today.

[natowi]

@TigerVersusT It worked (compiling)

[TigerVersusT]

@natowi Oh, how about the complied version? Does the feature extraction node works fine?

[natowi]

At the moment I only compiled Meshroom with your modifications

[TigerVersusT]

👍I'll wait for your good news

[natowi]

I need to recompile alicevision, as my last build is outdated

[natowi]

Ok, everything did compile, but your nodes throw errors:

ERROR:root:Error on node computation: Error on node "DracoEncoder_1":
could not find external_draco_encoder

Do the nodes work in your dev environment (starting meshroom from the uncompiled python code)? Or did I misplace some of your files?
Btw, how is the Retopology button supposed to work?
I guess I missed something. I´ll take another look at this within the next days.

However I did not experience any random crashes in FeatureExtraction, but it took a little longer than normal. Strange...

I´d recommend you to better separate gui/python code and c++ code, like it is being done with alicevision and meshroom. This simplifies the build process and results in a better code structure.

[TigerVersusT]

sorry for not repond immediately， it was too late for me yesterday.

Ok, everything did compile, but your nodes throw errors:__

That's because it's a CLI based nodes, you can test or use it if you like
external_draco_encoder.zip

external_draco_decoder.zip

Btw, how is the Retopology button supposed to work?

hope you don't laugh at me, this feature was designed before I know anything about meshroom pipline( so it disobey the priceples of SE ), it works by cliking the 'retopology' button after the defualt pipline is finished, then a specially named obj file will be found in the {cache}{texturing}{unique id} .In case you have interest in the retopology module, I just exploit source codes of intantMeshes, delete UIs and refactor them to fit my goal. There is a VS .sln file in the source folder you can test it if you like.

However I did not experience any random crashes in FeatureExtraction, but it took a little longer than normal. Strange...

Yes, I think my code works fine, but the compiled version always crashed at the second node, may be there is some enviromental problem because even the latest source code crashed after compilation.

I´d recommend you to better separate gui/python code and c++ code, like it is being done with alicevision and meshroom. This simplifies the build process and results in a better code structure.

Thanks, something in my code does disobey principles of SE ( especially the retopology module ), I will refactor it later.

[natowi]

but the compiled version always crashed at the second node, may be there is some enviromental problem

What are the versions of your environment modules?

I have: python 3.6.2 (64bit) and the exact versions specified in the requirements.

[TigerVersusT]

The only difference is my python version is 3.68 maybe I should change to that to compile again, btw, can you send me your compiled version? This is my email address: [email protected] or you can try [email protected]

[TigerVersusT]

I tried with python 3.62 it crashed again, maybe I should recompile the AliceVision

[natowi]

Ok, here are two builds you can try:

exe.win-amd64-3.6-allmods.zip (Includes all your modifications + numpy+pymeshfix)

exe.win-amd64-3.6-nodesonly-numpy-pymeshfix.zip (python nodes only + numpy + pymeshfix)

(The easiest solution would be to take the official 2019.2 build, add the numpy and pymeshfix folders to the lib folder, copy the python nodes to the meshroom node folder and copy the exe files into alicevsion. No building required)

[TigerVersusT]

cool! I'll try it now and reply you later.

[TigerVersusT]

@natowi ,I test with the 'allmodes' one, and I find out that alicevision in the official build is not compatible with your compilation. After changing that to my self-compiled alicevision, everthing works fine( tested for three times). Thanks a lot for your help these days, and I think I'm ready for the next feature( hope you haven't finished them all )😄.

[natowi]

Sorry, I already assumed you used your own compiled alicevision version.