Eval on IMC - I may help

[ducha-aiki]

Hi,

There is a sample repository for the submission to IMC
https://github.com/ducha-aiki/imc2021-sample-kornia-submission

If you want, I can adapt it for your toolbox. Caveat - it would be quite hard to do for the methods, not based on keypoints now.

[GrumpyZhou]

Hi @ducha-aiki ,

Actually, I had them roughly coded (80%) some time ago, but unfortunately i never get time to start testing them 😅😅😅.
How about i clean my local code first and push to another branch first? Probably this (latest next) weekend.
Then you can help me in testing them and improving, especially you seem really familiar with the benchmark.
And thanks for offering the help, that's definitely appreciated !

[ducha-aiki]

Sounds good! Just ping me here, or at email: ducha.aiki@gmail.com

[GrumpyZhou]

Hi @ducha-aiki ,

Sorry for the late update.. I finally got time to clean some code and push them.
Now they are in the new branch dev_imc.
I scratched a development log dev_imc.md.
I have wrote some of my ideas about how i want to organize the datasets, outputs. Feel free to change them or add comments there.
Anything regarding the data, code (namings, code lint, logic) are open for discussion and changes.
Just ping me if anything is unclear to you!
Regarding the time schedule, no pressure and no rush at all.
And thank you again for offering to improve this repo together :)

[ducha-aiki]

I have looked at the code structure and it seems to be very good. One thing, which I need to discuss with the other IMC orgs, is that for the stereo what your code does in order to generate "keypoints", violates the pairwise assumption. And if one does it anyway, then it would be only rational to perform some kind of optimization or bundle adjustment.

So maybe the correct way is for us to change the expected format to be suitable for pairwise methods.

I will update you soon.

[ducha-aiki]

OK, here is an update: I modified the benchmark code in the branch (not yet merged), which allows submission of "pairwise" keypoints, which are unique for every image pair.

ubc-vision/image-matching-benchmark#42

The caveat: this works only for the stereo.

For the multiview we are thinking about the following options: just concat the "pairwise" keypoints and resp. modify match indexes. However, this would rely on COLMAP ability to merge the duplicating disjoint keypoints.

The second option is to allow participants to merge/BA the keypoints themselves and submit - as usual - single keypoint list per image.

[ducha-aiki]

The template (with LoFTR) to generate such "pairwise" submission is here https://github.com/ducha-aiki/imc2021-sample-kornia-submission/blob/loftr/extract_loftr_matches.py

It does not include RANSAC though

[GrumpyZhou]

Hi @ducha-aiki ,

Thanks for the update, i think for stereo task it's good to have the support of pairwise matching.

Regarding multiview task, this repository already has the support for those methods that are not keypoint-based and see the implementation in localize_sfm_helper.py.
What i am doing there is:

1. Extract matches for all pairs
2. For each image, collect its set of keypoints by finding all of the points that are involved in the matches extracted from all image pairs.
3. Merge those keypoints by quantizing the very nearby keypoints w.r.t a quantize threshold.
   So i need to first extract all the matches, then i compute the keypoints.

This heuristic solution is validated on Patch2Pix to improve significantly the localization accuracy on Aachen Day and Night original version. I also applied this to Loftr, which also has this issue and results are: 89.1 / 94.9 / 98.3 87.8 / 93.9 / 100.0. If i just feed the raw keypoints that are concatenated, the performance will be really bad, which kinda refers that colmap is not handling those nearby keypoints well. While this solution is not perfect, it is rather handy since it doesn't bring much computation burden during evaluation.
So i hope this also applies to the multiview task in IMC.

Another thing, it seems i can not open the pairwise example: https://github.com/ducha-aiki/imc2021-sample-kornia-submission/blob/loftr/extract_loftr_matches.py.

[ducha-aiki]

Oh, so we basically can recommend your solution as an official one for the multiview. The remaining thing (for us at IMC) is to tell the code to take pairwise matches for stereo and per-image matching in mutiview, which we will do next week.
May ask then you to submit Patch2Pix entry to IMC, to make sure that everything is working is required?

Regarding the example - I have merged it to the master branch, sorry :)
Here it is https://github.com/ducha-aiki/imc2021-sample-kornia-submission/blob/master/extract_loftr_matches.py

[ducha-aiki]

Offtopic: We have recently added LoFTR to kornia
https://kornia.readthedocs.io/en/latest/_modules/kornia/feature/loftr/loftr.html#LoFTR

I think, it would be good for everyone to add Patch2pix there as well - for users it would be easy to use, for you - potentially more users. What do you think?

[GrumpyZhou]

Sure, i can submit Patch2Pix when you want to examine your updated benchmark.
One thing about the quantization solution you need to notice is that this sacrifice the accuracy during the quantization to have the repeatability, maybe when you write the documentation, you can inform the user what they should expect here.

When you say Patch2Pix, you mean the refinement part or the whole network when combined with NCNet or both?
I also think it's good to have it added to IMC. It might also help me to understand more where it works and where not.

[ducha-aiki]

I mean both - as a end-to-end matching module, and as a way to refine correspondences as well :)

[GrumpyZhou]

Sure 👍.. Just ping me when anything is needed to be done from my side :)