user_activity retrieves <500 posts, retrieving more requires logging in

[trislee]

Gettr seems to have changed its public API, and now only allows ~500 posts to be retrieved without logging in. This is also apparent when looking at a user's timeline in a web browser: scrolling to the bottom of a user timeline when you're not logged in shows ~500 user posts and says "END" when you reach the limit, while scrolling to the bottom of the timeline when you ARE logged in shows all user posts (see attached image)

Adding an X-App-Auth header parameter from the logged-in user containing the username and a generated token allows you to retrieve all of a user's posts, i.e.:

HEADERS = {
    'X-App-Auth': json.dumps({
        'user': '$MY_USERNAME', 
        'token': '$MY_TOKEN'}),
}
 resp = requests.get(
    url,
    params=params,
    timeout=10,
    headers=HEADERS,
)

Is implementing a login flow within the scope of this project? I could potentially take a crack at implementing it.

Might be related to issue #21

[trislee]

I implemented a quick workaround in commit bellingcat@0a698e5. Supplying the PrivateClient with a username and token, user_activity successfully retrieves all posts.

I'm not sure how frequently the token used in the X-App-Auth header changes though. Gettr's login flow seems to use recaptcha, so logging in via Requests and re-generating the token might be difficult to do without a recaptcha solver.

[milesmcc]

Thanks for the heads up, and for implementing a workaround. This prompts a larger question on the extent to which we're comfortable with GoGettr accessing "private" API endpoints. @lxcode, do you think we should incorporate a way to do private auth upstream?

[lxcode]

I don't see any problem with having an authenticated mode. I'd make it optional, since I assume other endpoints still allow pulling unauthed.

[KonradIT]

The artificially limited "end" response has results.aux.cursor set to 0, and so does a full result response (tested with a controlled account). Would be cool if the code, when running user_activity in unauthenticated mode would throw a warning to switch to authenticated when reaching the end, in case there are more posts (can't see a way to get total posts count from the api).