Twitter sentiment analysis - bottleneck

[Felixkruemel]

So we want to do some Twitter sentiment analysis for a bot.
The thing is that it takes too long on an Raspberry, like a lot too long.

So what is the thing which takes too long? Fetching the tweets from Twitter seems to go pretty quickly, but after that it's stuck for a while.
The "infer" command is way way faster which would work out fine.

So what can we do to get a good sentiment graph (even with less features like the one from Finbrain) from Twitter in a short amount of time?

[DidierRLopes]

Fetching tweets from Twitter seems to go pretty quickly, but that is 10 tweets per 1 hour and 240 per day. Which isn't much when you think about it. When I tested like 60 tweets per hour it took ages! We'll look into https://github.com/twintproject/twint in the future as there's no API key required and is much faster retrieving data.

I think the bottleneck may be on parsing each tweet, where we do:

• Cleaning tweet data to remove links, emojis, weird characters
• Predict the sentiment using this distilbert model
• Appending the sentiment score to a list

There should be easier ways to do this processing, but my goal at the beginning was to have something working and then we could focus on optimisations. As the saying goes: "Premature optimization is the root of all evil"

Plus, we are using a lighter version of Google's BERT model, which by itself should already make sentiment prediction faster.

[Felixkruemel]

@DidierRLopes
Be aware that Twitter has insane and stupid limitations for the API.
You can only query 500k Tweets per month which means not more than like 10 sentiment analysis per day. Additionally you have a restrictions on how much you can query in 15min. That's probably why 60tweets per hour take ages as the API just is forcing you to wait I guess. Twint would be way way better in every regard I think.

A faster sentiment prediction would be great :)
But not the highest priority in my opinion. A combination of sentiment analysis and chart prediction would be better :P

[DidierRLopes]

Agreed!

I think is just because of how many more tweets it is. I agree that Twint is a better tool. Although not a priority at the moment I'd say given that I think that new features would be preferred, rather than improving existing ones :)

Also agreed. But for a faster sentiment prediction, you risk using a less accurate model. There's a threshold there. In the future it would be nice to let the user select the model of their choice.