Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Add datapoints.delete method for improved dps delete support #1084

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ Changes are grouped as follows
- `Fixed` for any bug fixes.
- `Security` in case of vulnerabilities.

## [4.11.2] - 2022-11-08
### Fixed
- Add support for deleting single datapoint with datapoints.delete_point.

## [4.11.1] - 2022-11-08
### Changed
- Update doc for Vision extract method
Expand Down
61 changes: 55 additions & 6 deletions cognite/client/_api/datapoints.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import math
import re as regexp
from datetime import datetime
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set, Tuple, Union, cast
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Sequence, Set, Tuple, Union, cast, overload

import cognite.client.utils._time
from cognite.client import utils
Expand Down Expand Up @@ -322,6 +322,55 @@ def insert_multiple(self, datapoints: List[Dict[str, Union[str, int, List]]]) ->
dps_poster = DatapointsPoster(self)
dps_poster.insert(datapoints)

@overload
def delete_point(self, point: Dict[str, Union[int, str, datetime]]) -> None:
...

@overload
def delete_point(self, point: Sequence[Dict[str, Union[int, str, datetime]]]) -> None:
...

def delete_point(
self, point: Union[Dict[str, Union[int, str, datetime]], Sequence[Dict[str, Union[int, str, datetime]]]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it doesn't make much sense to allow str here; as these are all the "unprecise" or relative time offsets like "an hour ago" or "now".

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. But this dict also contains the externslId, which has a string value.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...what I would like to see though was Datapoint and Datapoints supported!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! That's probably much more useful.

) -> None:
"""Delete a single datapoint from one or more time series.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a suggestion as the current sound like "delete t=200 from ts=a,b and c":

Suggested change
"""Delete a single datapoint from one or more time series.
"""Delete individual datapoints from one or more time series.


Args:
point (Union[Dict[str, Union[int, str, datetime]], Sequence[Dict[str, Union[int, str, datetime]]]]): The point or points to delete

Returns:
None

Examples:

Deleting a single datapoint from a single time series::

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> c.datapoints.delete_point(point={"id": 1, "timestamp": 12345})

Deleting a single datapoint from multiple time series::

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> c.datapoints.delete_point(point=[{"id": 1, "timestamp": 123}, {"externalId": "abc", "timestamp": datetime(year=1992)}])
"""
point_list = point if isinstance(point, Sequence) else [point]
valid_ranges = []
for point in point_list:
for key in point:
if key not in ("id", "externalId", "timestamp"):
raise AssertionError(
f"Invalid key '{key}' in point. Must contain 'timestamp', and 'id' or 'externalId"
Comment on lines +363 to +364
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyError or ValueError?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. Some fire-from-the-hip copy-pasting here 😁

)
Comment on lines +360 to +365
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I've seen this type of logic in multiple places in the SDK, could be split out to utils.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do 👍

id = cast(Optional[int], point.get("id"))
external_id = cast(Optional[str], point.get("externalId"))
valid_range = Identifier.of_either(id, external_id).as_dict()
start = utils._time.timestamp_to_ms(point["timestamp"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May raise KeyError. Add a must-be-present check in loop above?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check above already ensures the existence of this key. Or did you mean smth else?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah no it doesn't, it just checks that no other keys are present. I'll make a utility for doing both of these types of checks for dicts.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch!

valid_range.update({"start": start, "end": start + 1})
valid_ranges.append(valid_range)
self.delete_ranges(ranges=valid_ranges)

def delete_range(
self, start: Union[int, str, datetime], end: Union[int, str, datetime], id: int = None, external_id: str = None
) -> None:
Expand All @@ -346,17 +395,17 @@ def delete_range(
"""
start = utils._time.timestamp_to_ms(start)
end = utils._time.timestamp_to_ms(end)
assert end > start, "end must be larger than start"
assert end >= start, "end must be larger than start"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: revert this. Having them equal doesn't make sense.


delete_dps_object = Identifier.of_either(id, external_id).as_dict()
delete_dps_object.update({"inclusiveBegin": start, "exclusiveEnd": end})
self._delete_datapoints_ranges([delete_dps_object])

def delete_ranges(self, ranges: List[Dict[str, Any]]) -> None:
def delete_ranges(self, ranges: List[Dict[str, Union[int, str, datetime]]]) -> None:
"""`Delete a range of datapoints from multiple time series. <https://docs.cognite.com/api/v1/#operation/deleteDatapoints>`_

Args:
ranges (List[Dict[str, Any]]): The list of datapoint ids along with time range to delete. See examples below.
ranges (List[Dict[str, Union[int, str, datetime]): The list of datapoint ids along with time range to delete. See examples below.

Returns:
None
Expand All @@ -378,8 +427,8 @@ def delete_ranges(self, ranges: List[Dict[str, Any]]) -> None:
raise AssertionError(
"Invalid key '{}' in range. Must contain 'start', 'end', and 'id' or 'externalId".format(key)
)
id = range.get("id")
external_id = range.get("externalId")
id = cast(Optional[int], range.get("id"))
external_id = cast(Optional[str], range.get("externalId"))
valid_range = Identifier.of_either(id, external_id).as_dict()
start = utils._time.timestamp_to_ms(range["start"])
end = utils._time.timestamp_to_ms(range["end"])
Expand Down
2 changes: 1 addition & 1 deletion cognite/client/_version.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
__version__ = "4.11.1"
__version__ = "4.11.2"
__api_subversion__ = "V20220125"
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[tool.poetry]
name = "cognite-sdk"

version = "4.11.1"
version = "4.11.2"

description = "Cognite Python SDK"
readme = "README.md"
Expand Down
19 changes: 19 additions & 0 deletions tests/tests_unit/test_api/test_datapoints.py
Original file line number Diff line number Diff line change
Expand Up @@ -703,6 +703,25 @@ def mock_delete_datapoints(rsps, cognite_client):


class TestDeleteDatapoints:
def test_delete_point(self, cognite_client, mock_delete_datapoints):
res = cognite_client.datapoints.delete_point({"id": 1, "timestamp": 1})
assert res is None
assert {"items": [{"id": 1, "inclusiveBegin": 1, "exclusiveEnd": 2}]} == jsgz_load(
mock_delete_datapoints.calls[0].request.body
)

def test_delete_points(self, cognite_client, mock_delete_datapoints):
res = cognite_client.datapoints.delete_point(
[{"id": 1, "timestamp": 1}, {"externalId": "abc", "timestamp": 10}]
)
assert res is None
assert {
"items": [
{"id": 1, "inclusiveBegin": 1, "exclusiveEnd": 2},
{"externalId": "abc", "inclusiveBegin": 10, "exclusiveEnd": 11},
]
} == jsgz_load(mock_delete_datapoints.calls[0].request.body)

def test_delete_range(self, cognite_client, mock_delete_datapoints):
res = cognite_client.datapoints.delete_range(start=datetime(2018, 1, 1), end=datetime(2018, 1, 2), id=1)
assert res is None
Expand Down