Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust AlertManagerAPI to avoid using multiple spaces in various attibutes of alert #12237

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Adjust AlertManagerAPI to avoid using multiple spaces in various attibutes of alert #12237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/python/WMCore/MicroService/MSTransferor/MSTransferor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -603,7 +603,7 @@ def alertTransferCouchDBError(self, workflowName):
workflowName)
alertSeverity = "high"
alertSummary = "[MSTransferor] Transfer document could not be created in CouchDB."
alertDescription = "Workflow: {}, failed request due to error posting to CouchDB".format(workflowName)
alertDescription = "Workflow: {}, failed request due to error posting to CouchDB".format(workflowName)
self.sendAlert(alertName, alertSeverity, alertSummary, alertDescription,
self.alertServiceName)
self.logger.warning(alertDescription)
Expand Down
20 changes: 17 additions & 3 deletions src/python/WMCore/Services/AlertManager/AlertManagerAPI.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,11 @@
import socket
import json
import logging
import re

from WMCore.Services.pycurl_manager import RequestHandler
from Utils.Timers import LocalTimezone
from WMCore.Services.UUIDLib import makeUUID


class AlertManagerAPI(object):
Expand Down Expand Up @@ -72,13 +74,14 @@ def sendAlert(self, alertName, severity, summary, description, service, tag="wmc
labels["severity"] = severity
labels["tag"] = tag
labels["service"] = service
labels["uuid"] = makeUUID()
alert["labels"] = labels

# add annotations
annotations["hostname"] = self.hostname
annotations["summary"] = summary
annotations["description"] = description
alert["annotations"] = annotations
annotations["summary"] = normalize_spaces(summary)
annotations["description"] = normalize_spaces(description)
alert["annotations"] = normalize_spaces(annotations)

# In python3 we won't need the LocalTimezone class
# Will change to d = datetime.now().astimezone() + timedelta(seconds=endSecs)
Expand All @@ -90,7 +93,9 @@ def sendAlert(self, alertName, severity, summary, description, service, tag="wmc
# need to do this because pycurl_manager only accepts dict and encoded strings type
params = json.dumps(request)

# provide dump of alert send to AM which will allow to match it in WM logs
res = self.mgr.getdata(self.alertManagerUrl, params=params, headers=self.headers, verb='POST')
self.logger.info("ALERT: %s, status=%s", params, res)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might not be a good idea, as some alerts might be very large (e.g., with a list of block names).
In addition, there is no way to control the log level of this class, so I am inclined to keep it out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alan, this is valid concern, but to make things traceable (i.e. match alert with WM logs we need something to present in WM logs). I'm changing this to alert UUID which will be sufficient to make such cross-check.


return res

Expand All @@ -104,3 +109,12 @@ def _isValidSeverity(self, severity):
logging.critical("Alert submitted to AlertManagerAPI with invalid severity: %s", severity)
return False
return True

def normalize_spaces(text):
vkuznet marked this conversation as resolved.
Show resolved Hide resolved
"""
Helper function to remove any number of empty spaces within given text and replace
then with single space.
:param text: string
:return: normalized string
"""
return re.sub(r'\s+', ' ', text).strip()