Skip to content

KeyError: 'status' when trying to schedule spider #156

Open
@kalessin

Description

@kalessin
Member

I am getting this error when trying to schedule a spider. This is happening with version 2.3.1

Traceback (most recent call last):
  File "/home/molveyra/.local/share/virtualenvs/mollie-AtuAN_AE/lib/python3.8/site-packages/scrapinghub/legacy.py", line 157, in _decode_response
    if data['status'] == 'ok':
KeyError: 'status'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/molveyra/.local/share/virtualenvs/mollie-AtuAN_AE/lib/python3.8/site-packages/scrapinghub/client/exceptions.py", line 69, in wrapped
    return method(*args, **kwargs)
  File "/home/molveyra/.local/share/virtualenvs/mollie-AtuAN_AE/lib/python3.8/site-packages/scrapinghub/client/__init__.py", line 19, in _request
    return super(Connection, self)._request(*args, **kwargs)
  File "/home/molveyra/.local/share/virtualenvs/mollie-AtuAN_AE/lib/python3.8/site-packages/scrapinghub/legacy.py", line 143, in _request
    return self._decode_response(response, format, raw)
  File "/home/molveyra/.local/share/virtualenvs/mollie-AtuAN_AE/lib/python3.8/site-packages/scrapinghub/legacy.py", line 169, in _decode_response
    raise APIError("JSON response does not contain status")
scrapinghub.legacy.APIError: JSON response does not contain status

Activity

vshlapakov

vshlapakov commented on Feb 26, 2021

@vshlapakov
Contributor

@kalessin Is it reproducible or transient? Could you share payload you're sending (w/o project id, etc)?

kalessin

kalessin commented on Feb 26, 2021

@kalessin
MemberAuthor

It is reproducible. It happens every time I try to schedule a spider. However, I just debugged it and the issue is that the response is

{'detail': 'Authentication credentials were not provided.'}

Still the issue is in the library, because it must handle this condition and print the correct failure.

vshlapakov

vshlapakov commented on Feb 26, 2021

@vshlapakov
Contributor

it must handle this condition and print the correct failure.

Agree, it should be addressed by the library

kalessin

kalessin commented on Feb 26, 2021

@kalessin
MemberAuthor

An update on this. Regardless the response is 'Authentication credentials were not provided.', this really happens when the apikey is wrong, so there is a server side issue too.

If I don't pass any apikey, I get correctly a runtime error:

RuntimeError: No API key provided and neither SH_APIKEY nor SHUB_JOBAUTH environment variables is set
vshlapakov

vshlapakov commented on Mar 1, 2021

@vshlapakov
Contributor

Regardless the response is 'Authentication credentials were not provided.', this really happens when the apikey is wrong, so there is a server side issue too.

I believe that's intentional - our server would react the same if credentials aren't provided or it's wrong, was done for security purpose. But I think we could handle it on the client side anyway to improve experience.

kalessin

kalessin commented on Mar 1, 2021

@kalessin
MemberAuthor

Right. Still the reaction is not the same. When you provide wrong apikey, then you should also get

RuntimeError: No API key provided and neither SH_APIKEY nor SHUB_JOBAUTH environment variables is set

instead of a meaningless KeyError exception.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @kalessin@vshlapakov@Gallaecio

        Issue actions

          KeyError: 'status' when trying to schedule spider · Issue #156 · scrapinghub/python-scrapinghub