Try using doc2vec as similarity algorithm

[marco-c]

After #637, we should try with doc2vec too and see what works better.

[Goku2699]

@marco-c is this issue resolved or still on? I'm new to open source so need your help to make some contributions.

[marco-c]

No, it's not resolved yet. It depends on #637, so that one must be fixed first before fixing this one.
Once that is fixed, something similar needs to be done but using doc2vec instead of word2vec.

[ashridh]

@Goku2699 #637 is done. Feel free to give this a try.

[probaku1234]

I'm trying to run the word2vec code, but bugzilla.get_bugs() in init function returns nothing and because of that error pops up.

[marco-c]

You should train a model first, so that the bugs DB will be downloaded.

python3 run.py --goal defect --train

[probaku1234]

I downloaded db with that code but still it gets nothing.

[marco-c]

Can you post the output of ls -al data/?
What error pops up?

Try to give as much info as possible, otherwise it'll be hard to help you :)

[probaku1234]

This is the result of ls

total 6278256
drwxr-xr-x  10 yunseoblee  staff         320 Jul  2 14:22 .
drwxr-xr-x  34 yunseoblee  staff        1088 Jul 12 22:25 ..
-rw-r--r--   1 yunseoblee  staff  1410490733 Jul  2 14:22 bugs.json
-rw-r--r--   1 yunseoblee  staff           1 Jun 21 15:53 bugs.json.version
-rw-r--r--   1 yunseoblee  staff   235073932 Jul  2 14:22 bugs.json.zst
-rw-r--r--   1 yunseoblee  staff          34 Jul  2 14:22 bugs.json.zst.etag
-rw-r--r--   1 yunseoblee  staff  1398900302 Jul  2 14:22 commits.json
-rw-r--r--   1 yunseoblee  staff           1 Jun 21 15:53 commits.json.version
-rw-r--r--   1 yunseoblee  staff   150652308 Jul  2 14:22 commits.json.zst
-rw-r--r--   1 yunseoblee  staff          34 Jul  2 14:22 commits.json.zst.etag

This is the error traceback

Traceback (most recent call last):
  File "/Users/yunseoblee/Desktop/bugbug/scripts/evaluate_similarity.py", line 38, in <module>
    main(parse_args(sys.argv[1:]))
  File "/Users/yunseoblee/Desktop/bugbug/scripts/evaluate_similarity.py", line 32, in main
    model = Word2VecWmdSimilarity()
  File "/Users/yunseoblee/Desktop/bugbug/bugbug/similarity.py", line 201, in __init__
    self.w2vmodel = Word2Vec(self.corpus, size=100, min_count=5)
  File "/Users/yunseoblee/Desktop/bugbug/venv/lib/python3.7/site-packages/gensim/models/word2vec.py", line 783, in __init__
    fast_version=FAST_VERSION)
  File "/Users/yunseoblee/Desktop/bugbug/venv/lib/python3.7/site-packages/gensim/models/base_any2vec.py", line 763, in __init__
    end_alpha=self.min_alpha, compute_loss=compute_loss)
  File "/Users/yunseoblee/Desktop/bugbug/venv/lib/python3.7/site-packages/gensim/models/word2vec.py", line 910, in train
    queue_factor=queue_factor, report_delay=report_delay, compute_loss=compute_loss, callbacks=callbacks)
  File "/Users/yunseoblee/Desktop/bugbug/venv/lib/python3.7/site-packages/gensim/models/base_any2vec.py", line 1081, in train
    **kwargs)
  File "/Users/yunseoblee/Desktop/bugbug/venv/lib/python3.7/site-packages/gensim/models/base_any2vec.py", line 536, in train
    total_words=total_words, **kwargs)
  File "/Users/yunseoblee/Desktop/bugbug/venv/lib/python3.7/site-packages/gensim/models/base_any2vec.py", line 1187, in _check_training_sanity
    raise RuntimeError("you must first build vocabulary before training the model")
RuntimeError: you must first build vocabulary before training the model

[ashridh]

@probaku1234 Are you in the correct directory? bugzilla.get_bugs() works only when you are in bugbug/. From the root directory of this project, run python scripts/evaluate_similarity.py --algorithm=word2vec_wmd.

[probaku1234]

If I run python scripts/evaluate_similarity.py --algorithm=word2vec_wmd or python3 scripts/evaluate_similarity.py --algorithm=word2vec_wmd, I get the error below

Traceback (most recent call last):
  File "scripts/evaluate_similarity.py", line 11, in <module>
    from bugbug.similarity import LSISimilarity, NeighborsSimilarity, Word2VecWmdSimilarity
ModuleNotFoundError: No module named 'bugbug'

[ashridh]

@probaku1234 Have you added bugbug to your pythonpath? If you're on linux, open the .bashrc file and add
export PYTHONPATH=path_to_bugbug_directory.

[probaku1234]

@ayush-1506 So....the error is gone now. But output on console shows below result and the program keep running.

INFO:gensim.summarization.textcleaner:'pattern' package not found; tag filters are not available for English
[nltk_data] Downloading package stopwords to
[nltk_data]     /Users/yunseoblee/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!

[ashridh]

@probaku1234 Yes! That's because the vocabulary is pretty big, so it takes time to build the word2vec model. For testing, you can use a smaller portion of the bug dataset.

[probaku1234]

@ayush-1506 Oh..I see. Then how can I set to use small portion of the dataset?