paraphraser get stuck

[cecilialeiqi]

Hi,
Sometimes the paraphraser gets stuck and never finish, e.g.
when running:

from inference import *
source_sentence="That 's a potential nightmare scenario for the GOP establishment : a populist outsider with unlimited resources attacking their nominee from the right in the general election , raising hell -- and attracting votes -- with his rhetoric on issues like illegal immigration ."
paraphrases = paraphraser.sample_paraphrase(source_sentence, sampling_temp=0.75, how_many=1)

I tried changing the parameters which doesn't help either.
Do you have any idea how to fix this problem?

Thanks!

[vsuthichai]

Hi @cecilialeiqi would you be able to provide some information in regards to version of TF you're using, cuda and cudnn version, OS, if you're running on the gpu or cpu, etc? Thanks!

[cecilialeiqi]

Hi @vsuthichai , thanks for your reply! I'm running on a Ubuntu 16.04.3 LTS. I've tried tf 1.4.0 on gpu with CUDA 8.0.61, cuDNN v5, or tf 1.7.0 on cpu. It both gets stuck on the above sentence.
BTW, is there some special characters that should be avoided to use for the paraphraser?

[vsuthichai]

Hi @cecilialeiqi The padding character should be avoided but it's highly unlikely that it's within your sentence. I ended up retraining the model but haven't uploaded a new one yet. I experienced a bizarre freezing error as well. I suspect while upgrading my system, some incompatibility was introduced, but I'm uncertain. I didn't look into it too deeply and instead opted just to retrain. I can upload a new model if you like, you may have more success. It was trained on TF 1.6. I tried the sentence you entered above and I don't experience the hang. Let me know, I can upload or send it to you some other way.

[vsuthichai]

@cecilialeiqi Something I have noticed and I suspect this is an issue in Tensorflow is that the greedy sampling which works fine on the gpu but doesn't work on cpu. However, sampling with a temperature works fine on a cpu.

[cecilialeiqi]

It would be great if you could upload the new model. Thanks!

[cecilialeiqi]

Hi @vsuthichai, is it possible for you to upload the new pre-trained model? Many thanks.

[vsuthichai]

Yes! Sorry, I will upload it tonight. Been a bit busy.

[vsuthichai]

Hi @cecilialeiqi I've uploaded the new model, you can find it here https://drive.google.com/open?id=18uOQsosF4uVGvUgp6pB4BKrQZ1FktlmM

I hope you'll have success with this one. Please let me know if it works for you. Thanks!

[cecilialeiqi]

@vsuthichai Thank you so much for the new model! However, the new one still hangs at some different sentences, e.g. "Your solution is MORE POWER FOR THE SPEAKER?!?!?!"
“See original post for references 0 0 0 0 0 0”

Do you think there is someway to skip the paraphrasing automatically when we face a bizarre freeze?

[vsuthichai]

@cecilialeiqi Hmm, that is really strange, because I don't experience any hangs.. How frequent would you say that it hangs? I'm wondering if it's something else, and not the model. Could you try attaching strace to the process or tfgdb? Check if there is deadlock anywhere? Does it freeze during the session.run() of the inference or is it somewhere else? Wish I could actually be there to diagnose it in your environment. There have been some reports on tensorflow's github related to hangs. I know that's a bit of debug work there and I'm not sure how urgent it is that you need this model running. Is there any more information you could provide that might be useful? Apologies about the hangs.

[vsuthichai]

@cecilialeiqi I tried your sentence Your solution is MORE POWER FOR THE SPEAKER?!?!?! out numerous times and do not experience the hang. I've brought my demo service back online here at http://pair-a-phrase.it

If I may ask, did you have any specific need or purpose for the model? What will you be trying to accomplish? Thanks!

[cecilialeiqi]

Sorry for my bad, I mis-attached the first sentence. I also didn't experience hang for the first sentence now, but I do get hangs on the second one: "See original post for references 0 0 0 0 0 0", and also this one: "Support for the Euro Among the New Members of the Eurozone How has support for the euro evolved in the new member states that joined the euro after its physical introduction in January 2002, that is, in Cyprus, Estonia, Latvia, Lithuania, Malta, Slovakia and Slovenia?"
I see this hanging like one in 50 sentences, while I'm working on this fake or real news dataset:
https://github.com/GeorgeMcIntire/fake_real_news_dataset

I see it get's stuck at the line "predictions = self.sess.run(feeds, feed_dict)[0]"

I'm trying to use it for research purpose with its ddl in late May, and wish to get it running sooner. Sorry for taking up much of your time.

[vsuthichai]

Ah, I see now. Yes you're right it did hang for that sentence. Let me take a closer look, thanks for finding these issues btw!

Interestingly, it works ok for :
See original post for references
See original post for references 0
See original post for references 0 0
but does not work for See original post for references 0 0 0

[Raldir]

I was facing similar issues. Quite a bit of debugging finally led me to the problem: The dynamic decoding sometimes does not stop and keeps generating words as some sentences prevent the decoder from generating a sentence-end symbol. This also explains why See original post for references 0 0 works but See original post for references 0 0 0 doesn't. It keeps generating zeros and thus never comes to a halt.

Solving this was quite straight forward: Simply add the argument maximum_iterations to dynamic_decode in lstm_model.py with the maximum length of a sentence. This should solve the issue!