Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

exception is too much verbose #68

Open
belforte opened this issue Mar 18, 2022 · 6 comments
Open

exception is too much verbose #68

belforte opened this issue Mar 18, 2022 · 6 comments

Comments

@belforte
Copy link
Member

I am using py3-dbs3-client/4.0.7 on python 3.8
I find that exception message from DBS error is way too much verbose to be usable/useful in logs.
Can something be done ?

Here's a quick example where I try to access a non-existing API [1].
Applications using DBS Client do not care to know the internal traceback (note in particular the During handling of the above exception, another exception occurred: In this case these two lines
would have been plenty.

RestClient.ErrorHandling.RestClientExceptions.HTTPError: HTTP Error 404: 
URL=https://cmsweb.cern.ch:8443/dbs/prod/global/BAD/serverinfo

[1]

>>> import dbs.apis.dbsClient
>>> dbs.apis.dbsClient.__file__
'/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-client/4.0.7/lib/python3.8/site-packages/dbs/apis/dbsClient.py'
>>> import sys
>>> sys.version
'3.8.2 (default, Jan 22 2021, 17:57:37) \n[GCC 6.3.0]'
>>> badApi = DbsApi(url='https://cmsweb.cern.ch/dbs/prod/global/BAD')
>>> badApi.serverinfo()
Traceback (most recent call last):
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-client/4.0.7/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 445, in __callServer
    self.http_response = method_func(self.url, method, params, data, request_headers)
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-pycurl/3.17.7-comp/lib/python3.8/site-packages/RestClient/RestApi.py", line 36, in get
    return http_request(self._curl)
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-pycurl/3.17.7-comp/lib/python3.8/site-packages/RestClient/RequestHandling/HTTPRequest.py", line 62, in __call__
    raise HTTPError(effective_url, http_code, http_response.msg, http_response.raw_header, http_response.body)
RestClient.ErrorHandling.RestClientExceptions.HTTPError: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-client/4.0.7/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 1838, in serverinfo
    return self.__callServer("serverinfo")
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-client/4.0.7/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 457, in __callServer
    self.__parseForException(data)
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-client/4.0.7/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 483, in __parseForException
    raise http_error
RestClient.ErrorHandling.RestClientExceptions.HTTPError: HTTP Error 404: 
URL=https://cmsweb.cern.ch:8443/dbs/prod/global/BAD/serverinfo
Code=404
Message=Not Found
Header=HTTP/1.1 404 Not Found
Date: Fri, 18 Mar 2022 13:57:40 GMT
Server: Apache
Content-Type: text/html
Content-Length: 146
CMS-Server-Time: D=11556 t=1647611860253268


Body=<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
@belforte
Copy link
Member Author

belforte commented Mar 18, 2022 •
edited
Loading

a related (although somehow independent) problem is that the above manages to utterly confuse WMCore's DBSReader wrapper producing a really awful ouput. @amaltaro

>>> from WMCore.Services.DBS.DBSReader import DBSReader
>>> myDBS=DBSReader('https://cmsweb.cern.ch/dbs/prod/global/BAD')
ERROR:root:DBSReaderError
Message: Can't contact DBS at https://cmsweb.cern.ch/dbs/prod/global/BAD, got errors Instantiating DBS3Reader failed with HTTP Error 503: 
URL=https://cmsweb-prod.cern.ch:8443/dbs/prod/global/BAD/serverinfo
Code=503
Message=Service Unavailable
Header=HTTP/1.1 503 Service Unavailable
Date: Fri, 18 Mar 2022 14:15:50 GMT
Server: Apache
Set-Cookie: cms-node=6234941600972e824cc69a3dcc1058a852e72918879184b240f209d2e673ea58c935cd1735d6b86ed48f22fe;path=/;secure;httponly
CMS-Server-Time: D=95612 t=1647612950611354
Connection: close
Content-Type: text/html


Body=<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
 <head>
 <title>CMSWEB Error: Service unavailable</title>
 <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
 <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" />
 <link rel="stylesheet" type="text/css" media="screen" href="/css/cmsweb.css" />
</head>
<body>
 <div id="main">
  <div id="top">
   <div class="boxTitle"><img src="/img/title.gif" alt="CMSWEB (title)" /></div>
  </div>
  <div id="middle">
   <div class="boxLinkContainer">
    <h2>Service unavailable</h2>
    <p>The site you requested is not available.</p><p>CMS <a href='https://twiki.cern.ch/twiki/bin/view/CMS/ScheduledInterventions'>scheduled interventions page</a> lists planned CMS and CERN service downtimes and known site-wide computing system incidents. Planned updates are notified on the <a href='https://hypernews.cern.ch/HyperNews/CMS/get/cernCompAnnounce.html'>CERN computing announcements</a> forum.</p><p>Would you please be so kind and <a href='https://ggus.eu/?mode=ticket_cms'>create a ticket</a> to"CMS Web Tools" support unit</a> if this interruption was unexpected?</p>
   </div>
  </div>
  <div id="bottom">
   <div class="boxFooter">&nbsp;</div>
  </div>
 </div>
</body>
</html>



	ClassName : None
	ModuleName : WMCore.Services.DBS.DBSErrors
	MethodName : __init__
	ClassInstance : None
	FileName : /data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/crabtaskworker/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/lib/python3.8/site-packages/WMCore/Services/DBS/DBSErrors.py
	LineNumber : 38
	ErrorNr : 1002

Traceback (most recent call last):
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-client/4.0.7/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 485, in __parseForException
    data = json.loads(data)
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/crabtaskworker/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/lib/python3.8/site-packages/WMCore/Services/DBS/DBSReader.py", line 19, in DBSReader
    dbs.dbs.serverinfo()
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-client/4.0.7/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 1838, in serverinfo
    return self.__callServer("serverinfo")
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-client/4.0.7/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 457, in __callServer
    self.__parseForException(data)
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/py3-dbs3-client/4.0.7/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 491, in __parseForException
    raise http_error
RestClient.ErrorHandling.RestClientExceptions.HTTPError: HTTP Error 503: 
URL=https://cmsweb-prod.cern.ch:8443/dbs/prod/global/BAD/serverinfo
Code=503
Message=Service Unavailable
Header=HTTP/1.1 503 Service Unavailable
Date: Fri, 18 Mar 2022 14:15:50 GMT
Server: Apache
Set-Cookie: cms-node=6234941600972e824cc69a3dcc1058a852e72918879184b240f209d2e673ea58c935cd1735d6b86ed48f22fe;path=/;secure;httponly
CMS-Server-Time: D=95612 t=1647612950611354
Connection: close
Content-Type: text/html


Body=<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
 <head>
 <title>CMSWEB Error: Service unavailable</title>
 <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
 <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" />
 <link rel="stylesheet" type="text/css" media="screen" href="/css/cmsweb.css" />
</head>
<body>
 <div id="main">
  <div id="top">
   <div class="boxTitle"><img src="/img/title.gif" alt="CMSWEB (title)" /></div>
  </div>
  <div id="middle">
   <div class="boxLinkContainer">
    <h2>Service unavailable</h2>
    <p>The site you requested is not available.</p><p>CMS <a href='https://twiki.cern.ch/twiki/bin/view/CMS/ScheduledInterventions'>scheduled interventions page</a> lists planned CMS and CERN service downtimes and known site-wide computing system incidents. Planned updates are notified on the <a href='https://hypernews.cern.ch/HyperNews/CMS/get/cernCompAnnounce.html'>CERN computing announcements</a> forum.</p><p>Would you please be so kind and <a href='https://ggus.eu/?mode=ticket_cms'>create a ticket</a> to"CMS Web Tools" support unit</a> if this interruption was unexpected?</p>
   </div>
  </div>
  <div id="bottom">
   <div class="boxFooter">&nbsp;</div>
  </div>
 </div>
</body>
</html>



During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/crabtaskworker/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/lib/python3.8/site-packages/WMCore/Services/DBS/DBSReader.py", line 23, in DBSReader
    raise DBSReaderError("Can't contact DBS at %s, got errors %s" % (endpoint, msg))
WMCore.Services.DBS.DBSErrors.DBSReaderError: DBSReaderError
Message: Can't contact DBS at https://cmsweb.cern.ch/dbs/prod/global/BAD, got errors Instantiating DBS3Reader failed with HTTP Error 503: 
URL=https://cmsweb-prod.cern.ch:8443/dbs/prod/global/BAD/serverinfo
Code=503
Message=Service Unavailable
Header=HTTP/1.1 503 Service Unavailable
Date: Fri, 18 Mar 2022 14:15:50 GMT
Server: Apache
Set-Cookie: cms-node=6234941600972e824cc69a3dcc1058a852e72918879184b240f209d2e673ea58c935cd1735d6b86ed48f22fe;path=/;secure;httponly
CMS-Server-Time: D=95612 t=1647612950611354
Connection: close
Content-Type: text/html


Body=<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
 <head>
 <title>CMSWEB Error: Service unavailable</title>
 <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
 <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" />
 <link rel="stylesheet" type="text/css" media="screen" href="/css/cmsweb.css" />
</head>
<body>
 <div id="main">
  <div id="top">
   <div class="boxTitle"><img src="/img/title.gif" alt="CMSWEB (title)" /></div>
  </div>
  <div id="middle">
   <div class="boxLinkContainer">
    <h2>Service unavailable</h2>
    <p>The site you requested is not available.</p><p>CMS <a href='https://twiki.cern.ch/twiki/bin/view/CMS/ScheduledInterventions'>scheduled interventions page</a> lists planned CMS and CERN service downtimes and known site-wide computing system incidents. Planned updates are notified on the <a href='https://hypernews.cern.ch/HyperNews/CMS/get/cernCompAnnounce.html'>CERN computing announcements</a> forum.</p><p>Would you please be so kind and <a href='https://ggus.eu/?mode=ticket_cms'>create a ticket</a> to"CMS Web Tools" support unit</a> if this interruption was unexpected?</p>
   </div>
  </div>
  <div id="bottom">
   <div class="boxFooter">&nbsp;</div>
  </div>
 </div>
</body>
</html>



	ClassName : None
	ModuleName : WMCore.Services.DBS.DBSErrors
	MethodName : __init__
	ClassInstance : None
	FileName : /data/srv/TaskManager/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/slc7_amd64_gcc630/cms/crabtaskworker/v3.220314patch1-3391c4ccadfae468fd63c78475580c2d/lib/python3.8/site-packages/WMCore/Services/DBS/DBSErrors.py
	LineNumber : 38
	ErrorNr : 1002

>>>

@vkuznet
Copy link
Contributor

vkuznet commented Mar 18, 2022

@belforte , here you hit several issues which bundle into single output:

  • nested exceptions in Python
  • full output (dump) of failed HTTP request
  • DBSClient and WMCore exception handling
    I don't really have clean recipe to fix this, but in fact you need all of these pieces when you debug the actual issue. For instance, without HTTP request dump you have no idea what happen during JSON parsing. I'm not sure if WMCore exception representation is useful here, but it certainly useful in other cases. The DBSClient handle both http exception and python one which may or may not be useful in different situations.

@vkuznet
Copy link
Contributor

vkuznet commented Mar 18, 2022

One specific suggestion may be useful (but may not be practical) is to avoid using WMCore DBS code and vice versa (using WMCore code within DBSClient) where it is not appropriate. At the end when to query DBS we only need to make HTTP calls, that's it, but it does not need WMCore internals (including exception handling, logging, etc.) And, as you can see there are so many wrappers like WMCore.Services.DBS.DBSErrors where they are may not be required.

@belforte
Copy link
Member Author

thanks @vkuznet . I agree that when something goes wrong details are needed.
I have always been tempted to skip WMCore/DBSReader layers and talk directly to DBS Client, maybe it is time to do it. Original developers of course had a point about code sharing across projects, but there's also something to say about useless wrappers :-(

I will figure out something

@vkuznet
Copy link
Contributor

vkuznet commented Mar 18, 2022

And, since we talk about wrappers. DBSClient is nothing else as yet another wrapper around DBS REST APIs. If you want to skip entire pipeline you may consider talk directly to DBS end-points. For instance DBS API call listDatasets() is nothing else as pure HTTP call to /dbs/.../datasets, etc. I understand the legacy but at the end clients should be aware of all of these intermediate steps and make conscious decision when it is useful or not. In my view we don't gain much going through client -> WMCore -> DBSClient -> DBS REST API, when we can replace entire pipeline with client -> DBS REST API, and our python codebase overhead is quite large in my view.

@belforte
Copy link
Member Author

well. I would not make direct calls to REST anyhow but would need some common code to wrap them to deal with common exceptions, config, and some peristent object in memory for connection reclycling etc.. so I am happy to use DBSClient which already takes care of some stuff I not comfortable with and returns well documented JSON ! It is not urgent, but sooner or later I will get rid of the wmcore wrapping

@belforte belforte mentioned this issue Mar 21, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vkuznet @belforte