Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow to control headers on the url fetch GET request #192

Open
royasoolin opened this issue Nov 26, 2024 · 0 comments
Open

allow to control headers on the url fetch GET request #192

royasoolin opened this issue Nov 26, 2024 · 0 comments

Comments

@royasoolin
Copy link

Right now when i run for example:

python -m podcastfy.client --url https://www.reuters.com/article/business/key-dates-and-milestones-in-the-sp-500s-history-idUSBRE9450WL/ --tts-model geminimulti

i get the error:
2024-11-26 15:32:07,002 - main - ERROR - An error occurred in the process_content function: Failed to extract content from https://www.reuters.com/article/business/key-dates-and-milestones-in-the-sp-500s-history-idUSBRE9450WL/: 401 Client Error: HTTP Forbidden for url: https://www.reuters.com/article/business/key-dates-and-milestones-in-the-sp-500s-history-idUSBRE9450WL/

when i am trying to get the same url with curl - i also get the 401 error
(curl -w "\nStatus Code: %{http_code}\n" https://www.reuters.com/article/business/key-dates-and-milestones-in-the-sp-500s-history-idUSBRE9450WL)

when i set headers on the requests, in this case headers of:
curl https://www.reuters.com/article/business/key-dates-and-milestones-in-the-sp-500s-history-idUSBRE9450WL/ -H "Accept-Language: en-US,en;q=0.5" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" -H "Accept-Encoding: gzip, deflate, br" --compressed

i get valid response.

it will be better to implement this bypass default on the fetch request, or allow this as configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant