Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various APOD issues/suggestions #42

Open
johnstorm-code opened this issue Jul 9, 2020 · 2 comments
Open

Various APOD issues/suggestions #42

johnstorm-code opened this issue Jul 9, 2020 · 2 comments
Labels
scrapper_difficulty_due_website_variance Probably issue with change in format on APOD website side of things.

Comments

@johnstorm-code
Copy link

These APODs are returning 'media_type' as 'other' but without any image/video content. The json response does not include a 'url' field, although some return a 'digg_url' field. I tested these through the https://apod.nasa.gov/ site as well and it seems these links may have broken content. Here is the complete list:

5/29/2006 - https://api.nasa.gov/planetary/apod?date=2006-5-29&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap060529.html - No digg_url - No image/video content at apod.nasa.gov
5/22/2007 - https://api.nasa.gov/planetary/apod?date=2007-5-22&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap070522.html - No digg_url - Flash content at apod.nasa.gov not working
12/31/2008 - https://api.nasa.gov/planetary/apod?date=2008-12-31&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap081231.html - Has digg_url - No image/video content at apod.nasa.gov
4/5/2009 - https://api.nasa.gov/planetary/apod?date=2009-4-5&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap090405.html - Has digg_url - Flash content at apod.nasa.gov not working
6/29/2009 - https://api.nasa.gov/planetary/apod?date=2009-6-29&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap090629.html - No digg_url - Youtube embedded content not working
5/10/2010 - https://api.nasa.gov/planetary/apod?date=2010-5-10&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap100510.html - No digg_url - Youtube embedded content not working
6/8/2010 - https://api.nasa.gov/planetary/apod?date=2010-6-8&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap100608.html - No digg_url - Youtube embedded content not working
8/25/2010 - https://api.nasa.gov/planetary/apod?date=2010-8-25&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap100825.html - No digg_url
12/15/2010 - https://api.nasa.gov/planetary/apod?date=2010-12-15&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap101215.html - No digg_url
1/23/2011 - https://api.nasa.gov/planetary/apod?date=2011-1-23&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap110123.html - No digg_url
2/1/2011 - https://api.nasa.gov/planetary/apod?date=2011-2-1&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap110201.html - No digg_url
2/22/2011 - https://api.nasa.gov/planetary/apod?date=2011-2-22&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap110222.html - No digg_url
3/7/2011 - https://api.nasa.gov/planetary/apod?date=2011-3-7&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap110307.html - No digg_url - Youtube embedded content at apod.nasa.gov not working

Also, these APODs return 'media_type' as 'other' but the content at apod.nasa.gov is to Youtube embedded videos. In consideration to consistency, is it possible to change these to 'media_type' 'video' and then include a 'url' field to the youtube url?

For example, APOD 7/1/2020 - https://api.nasa.gov/planetary/apod?date=2020-7-1&api_key=DEMO_KEY returns video and the URL to the Youtube content. While these do not:

8/10/2009 - API: https://api.nasa.gov/planetary/apod?date=2009-8-10&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap090810.html - Has digg_url
1/20/2010 - API: https://api.nasa.gov/planetary/apod?date=2010-1-20&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap100120.html - Has digg_url
1/24/2010 - API: https://api.nasa.gov/planetary/apod?date=2010-1-24&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap100124.html - Has digg_url
5/26/2010 - https://api.nasa.gov/planetary/apod?date=2010-5-26&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap100526.html - No digg_url
7/25/2010 - https://api.nasa.gov/planetary/apod?date=2010-7-25&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap100725.html No digg_url
8/25/2010 - https://api.nasa.gov/planetary/apod?date=2010-8-25&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap100825.html - No digg_url
12/15/2010 - https://api.nasa.gov/planetary/apod?date=2010-12-15&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap101215.html - No digg_url

Lastly, these are not issues but they feature Flash content. I wonder if it's possible to get only the flash embedded video?

4/13/2009 - API: https://api.nasa.gov/planetary/apod?date=2009-4-13&api_key=DEMO_KEY | https://apod.nasa.gov/apod/ap090413.html - Has digg_url

Thank you!!

@karthikmohan
Copy link

karthikmohan commented Jul 25, 2020

Also, this date: 4/19/2020 (A Vimeo Video)

This works:
https://api.nasa.gov/planetary/apod?api_key=hhLOEStXTcfad68lAeqOkEoVqhzNhTCyaPp5kaaO&date=2020-04-19

And this doesn't: (But it should)
https://api.nasa.gov/planetary/apod?api_key=hhLOEStXTcfad68lAeqOkEoVqhzNhTCyaPp5kaaO&date=2020-04-19&thumbs=true

Should show the thumbnail.
Instead shows error.

Something wrong with endpoint.

But works perfectly for YouTube videos.

@JustinGOSSES
Copy link
Contributor

Looks like the older ones have the youtube video inside of an object where as the newer one that "works" have the youtube video inside of an iframe.

The APOD API works by parsing the APOD website HTML. It doesn't have any special access.

Unfortunately the admin of this repository doesn't have time to investigate these edge cases right now. However, we would be happy to accept any pull request that investigates all the links above to confirm the patterns in the HTML and suggests code changes to the parser that return better data for those examples.

Thanks for your work in highlighting these issues.

@JustinGOSSES JustinGOSSES added the scrapper_difficulty_due_website_variance Probably issue with change in format on APOD website side of things. label Oct 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scrapper_difficulty_due_website_variance Probably issue with change in format on APOD website side of things.
Projects
None yet
Development

No branches or pull requests

3 participants