🐛Indexing duplications

[92amartins]

Describe the bug
I'm afraid in some cases pupyl is indexing the same image more than once.

To Reproduce

1. Collect few images (e.g. 10)
2. Run the example code from README.
3. See error

Expected behavior
On the screenshot below I expected cars to appear only once.

Screenshots

Desktop (please complete the following information):

• OS: macOS Big Sur
• Browser safari 14.0.1
• Python 3.8.6
• Pupyl Version 0.10.4

Additional context
Folder Structure:

.
├── images
│   ├── batman.jpg
│   ├── captain.jpg
│   ├── cars.jpg
│   ├── flash.jpg
│   ├── green.jpg
│   ├── inside_out.jpg
│   ├── monsters.jpg
│   ├── nemo.jpg
│   ├── superman.jpg
│   └── toy.jpg
└── images.py

# image.py
from pupyl.search import PupylImageSearch
from pupyl.web import interface


SAMPLES = 'images'

SEARCH = PupylImageSearch()

SEARCH.index(SAMPLES)

interface.serve()

[policratus]

First of all, thank you @92amartins for reporting this. A question: did you run the method PupylImageSearch.index() twice in the same source (in this case, a directory)?

[92amartins]

You're welcome.

Yes :(

In the first run, I had the images at the same directory as images.py. That caused pupyl to try to index the .py file raising FileIsNotImage exception. In spite of that, probably indexed a few files (alphabetically?) before raising the exception.

For the second run, I moved all the images to a separate folder, then ran the script with no issues.

I think that's why it got duplicated.

Maybe pupyl could have a mechanism for preventing indexing duplicated images, but as I'm not sure if this is a main case or edge case, I'll leave it up to you.

Feel free to close the issue, as I clearly made a misuse.

[policratus]

You're welcome.

Yes :(

In the first run, I had the images at the same directory as images.py. That caused pupyl to try to index the .py file raising FileIsNotImage exception. In spite of that, probably indexed a few files (alphabetically?) before raising the exception.

For the second run, I moved all the images to a separate folder, then ran the script with no issues.

I think that's why it got duplicated.

Maybe pupyl could have a mechanism for preventing indexing duplicated images, but as I'm not sure if this is a main case or edge case, I'll leave it up to you.

Feel free to close the issue, as I clearly made a misuse.

Well, we do have a bug here: pupyl should only try to index supported image files, not all files in a source. I'll open a issue to fix this bug.

The mechanism that you described seems pretty useful to avoid any duplications and it will be implemented as an optional parameter during an index process. This mechanism is nice but creates a little overhead during the index, so will be up to the user choosing it.

[policratus]

@92amartins, issues #66 and #67 created to fix or implement these bugs or ideias.

[policratus]

@92amartins, issue #67 resolved on pull request #68. Could you check if it works as intended now, when you have a chance?

[92amartins]

Yeah, sure! Can you provide an alpha version or instructions for building from source?

[policratus]

Should be a good idea having another branch with alpha releases, isn't it? I'll organize this too.

So @92amartins, to test the changes as it is, follow the steps:

1. Clone the branch:

# This case, the branch is check-duplicates
git clone -b check-duplicates [email protected]:policratus/pupyl.git

2. (Optional) Create a virtual environment: Using virtualenvwrapper, issue:

mkvirtualenv pupyl

3. Install dependencies: In the cloned directory, install the package (and its dependencies) with

cd pupyl && python setup.py install

4. (Optional) Compilation errors: if you see C/C++ compilation errors like error: command 'x86_64-linux-gnu-gcc' failed with exit status 1, go to TROUBLESHOOTING

5. (Optional) Create a directory with duplicated images: These images can be used to test:

Original	Subtle difference	Explicit difference

6. Execute the code: Open python and execute the following code snippet:

from pupyl.search import PupylImageSearch

# data_dir can be any created (if not, pupyl will create for you),
# or even you can leave it blank to save the assets to your
# temporary directory
SEARCH = PupylImageSearch(data_dir='/path/to/store/assets')

# Index a source (this case, a directory) that contains duplicates.
SEARCH.index('/path/with/image/duplicates', check_unique=True)

7. If everything works as intended, the following error must be raised:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pupyl/pupyl/search.py", line 170, in index
    index.append(
  File "pupyl/pupyl/indexer/facets.py", line 187, in append
    raise UniqueItemError
pupyl.indexer.exceptions.UniqueItemError

Any further information, please not hesitate to ask.

[policratus]

@92amartins, just opened #69 fixing #66.

[92amartins]

Your instructions are very straightforward, so maybe pupyl don't need the clutter of releasing alpha and beta versions right now.

I was able to reproduce the indexing process and raised an error when the check_unique flag was set to true 🎉

Thanks for the effort :)