Find better solution / error page for feeds that are not scraped yet

[dewey]

Right now we just throw an error, this should probably be a "fetching in progress" message or page.

level=error ts=2018-08-23T22:33:58.515971321Z caller=service.go:40 msg="no value found for key 'rss_scmp'"
level=error ts=2018-08-23T22:34:12.749184206Z caller=service.go:40 msg="no value found for key 'json_scmp'"
level=error ts=2018-08-23T22:35:33.950560779Z caller=service.go:40 msg="no value found for key 'json_scmp'"

[tecufanujacu]

Hello there, I'm trying this project but unfortunately I'm not able to make it working.
I cloned the repository, I execute the bin /run_develop.sh but when I try to access an rss:
http://localhost:8080/feed/scmp/rss

I always have the error:
there was an error serving the feed

and in logs I can see:
level=error ts=2021-05-12T08:07:30.30856613Z caller=service.go:49 msg="no value found for key 'rss_scmp'"

I tried to leaving it working for several hours hoping it start fetching but without success.

If I try force the scariping:
curl -X POST http://localhost:8080/scmp/refresh?auth_token=feedbridge

I receive this error:
there was an error listing the plugins

And in the logs I can see:
level=info ts=2021-05-12T08:36:19.857660114Z caller=runner.go:114 plugin=scmp scrape_type=single msg="scrape started"

but nothing changes.

Can you help me?
What am I doing wrong?

[dewey]

Hey, I'm probably unable to investigate that issue because I haven't touched the project in a while. I'd guess that maybe the plugin is broken as the website changes and it never successfully scrapes the page.

[tecufanujacu]

Ah ok, so is there some difference between this repo and the live version at the address: https://feedbridge.notmyhostna.me/ ?
Because I see that the live version is working.

[dewey]

Can you please try again, I just pushed a small fix and that seems to resolve the issue. I also rebuild the Docker image and uploaded the new binaries.

[tecufanujacu]

Thank you, I'll try immediately. I don't use Docker and I can't user your binary because I'm on arm64. Just few moments and I'll make you know.

#EDIT1
Ok, just tested. Compiling my own bin I have the same situation with identical errors.

I noticed that you have also build a bin for arm64, I tested it and with your build I have a blank page in the browser and nothing in the logs.

#EDIT2
Wait, something has changed.
With my own build after forcing the scraping with the command:
curl -X POST http://localhost:8080/scmp/refresh?auth_token=feedbridge

this time I have a partial working rss:

<rss version="2.0">
<channel>
<title>South China Morning Post</title>
<link>https://www.scmp.com/topics/infographics-asia</link>
<description>
Your source for credible news and authoritative insights from Hong Kong, China and the world.
</description>
<managingEditor>[email protected] (SCMP)</managingEditor>
</channel>
</rss>

#EDIT3
racefansnet and roadsandkingdoms instead now work :-)
After forcing the scraping now I have a valid feed for them. Thanks a lot :-)

I would use this your great project for generating an rss feed for a site that doesn't provide any rss, do you have some suggestion?

[dewey]

With my own build after forcing the scraping with the command:

Yes you have to force the refresh or wait the defined interval for it to start the scrape automatically. Glad you got it working then!

I would use this your great project for generating an rss feed for a site that doesn't provide any rss, do you have some suggestion?

You'd have to write a custom plugin for that site similar to the other plugins that are already in the project and it should work. If you have any plugin related questions feel free to create a new issue or merge request and I'll try to assist. Good luck!