🐛 Jikan returns 404 for some recently added anime

[YoshiWalsh]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

For anime that's been very recently added to MAL, Jikan returns a 404 error. The response body for https://api.jikan.moe/v4/anime/55312/full says:

{
    "status": 404,
    "type": "BadResponseException",
    "message": "Resource does not exist",
    "error": "404 on https://myanimelist.net/anime/55312/"
}

But accessing https://myanimelist.net/anime/55312/ works, it returns 200. Strangely, show 55315 was added more recently and does not return 404. Until recently I was also getting this issue for anime 55302.

This is a bit of a problem because users can add the show to their animelist, but I can't retrieve any details about it.

Expected Behavior

Since the anime exists, Jikan should return 200 with the details.

Steps To Reproduce

Look at the recently added anime list: https://myanimelist.net/anime.php?o=9&c%5B0%5D=a&c%5B1%5D=d&cv=2&w=1

Try requesting details about recently added anime. Hopefully one of them allows you to repro the issue.

Environment

- OS: Windows 10, Amazon Linux 2 (Lambda)
- Browser: Postman, Marika

Anything else?

Is Jikan caching 404 errors to avoid unnecessary upstream requests?

How is Jikan getting a 404 error to cache in the first place? Are there circumstances were MAL will return 404 even though an anime already exists? (E.g. pending anime?)

Are there users of Jikan polling non-existent show IDs in order to find the newest added shows?

[pushrbx]

How is Jikan getting a 404 error to cache in the first place? Are there circumstances were MAL will return 404 even though an anime already exists? (E.g. pending anime?)

It is recommended to sanity check with: https://dbsentinel.sean.fish/
In case of MAL whenever an entry requested to be added (by a user), it gets an ID and is listed "on the website" - unlisted; at this point no one can add it to their list. If its approved it keeps that ID, else it becomes a 404. There can be also a use case where the moderators first unapprove an added entry, becomes 404, then approve it again. Jikan only updates itself daily, or if you directly request the ID again. When you directly ask for the ID, e.g. https://api.jikan.moe/v4/anime/55312/full then jikan check the cache, if empty, then sends a request to upstream. If upstream returns an error, then Jikan returns 404 response. 404 responses are not cached.

[pushrbx]

@irfan-dahir One preventative change I can think of to partially resolve this issue is to update the error message to be more informative as we can't do too much about the upstream responses.
Something along the lines of

Cache has expired or it was stale for the requested item. 
Tried to retrieve an update, but the upstream server has responded with a not successful status code. 
Please try again later.

[irfan-dahir]

This should be resolved now since we only return approved entries now by default. I can't reproduce it anymore either. We're also running sanity by removing deleted entries on a daily basis via mal-id-cache

Let me know if it still persists though, otherwise, I think we can go ahead and close this issue.

@JoshuaWalsh cc: @pushrbx

---

To answer some questions:

Is Jikan caching 404 errors to avoid unnecessary upstream requests?

Yes. 404 is cached on the master server for 5m where we also cache the HTTP response (before parsing) for 6 hours. I think we can reduce that 6 hours and probably prevent anything stale from being served at that level.

Are there users of Jikan polling non-existent show IDs in order to find the newest added shows?

Yes

[irfan-dahir]

Unfortunately, I was able to reproduce this recently. However in a slightly different manner.
Recently added entries initially show up as 404 but if I re-request it, it will show the appropriate response.

This is definitely a caching issue. I need to figure out where. Leaving the ticket open until then.

[irfan-dahir]

Updating caching on multiple levels. 404 is now only cached for 1 hour. Turned off proxy_cache_revalidate, serving of stale cache on 404, and proxy cache background updates on nginx - I think they were causing the issue as well. But let's see 🤞