Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLEU scores too high in inference progess #27

Open
Shijie97 opened this issue Jun 28, 2020 · 2 comments
Open

BLEU scores too high in inference progess #27

Shijie97 opened this issue Jun 28, 2020 · 2 comments

Comments

@Shijie97
Copy link

Shijie97 commented Jun 28, 2020

Hi seanie!
Really appreciate for your code share, it does help me deepen my understanding of this paper.
But I have some trouble reproducing your code in the inference stage.
After inference stage, I moved the two files "generated.txt" and "golden.txt" to the "qgevalcap\result" folder, while there has been a "src.txt" in "qgevalcap\result" folder.
I have a few silly questions that may need to be answered.

  1. what on earth is the file "src.txt" for? Does it participate in the calculation of BLEU? Dost it exsit at the beginning?
  2. how is "golden.txt" generated? If this file is invariant, why not saving it as invariant file just like "scr.txt"?
  3. My result for BLEU is too high, it being shown below:
    Bleu_1: 0.65958
    Bleu_2: 0.43423
    Bleu_3: 0.30656
    Bleu_4: 0.22511
    ROUGE_L: 0.49419
    CIDEr: 0.56620

I wonder the reason, if you know what the problem is, may you please point it out? Thank you!

@seanie12
Copy link
Owner

  1. It is dummy file for indexing generated sentences
  2. golden.txt is the ground truth questions.
  3. Did you try my pretrained model?

@leesurelye
Copy link

same question, i trained the model at the beginging. the scores is :
Bleu_1: 0.51006
Bleu_2: 0.36335
Bleu_3: 0.27556
Bleu_4: 0.21658
METEOR: 0.22948
ROUGE_L: 0.52004
is there something wrong? the scores are too high

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants