Skip to content

ExportPartition - TTL after completion, not creation #1280

@arthurpassos

Description

@arthurpassos

As of now, each partition export task has a create_time and a ttl_in_seconds field. Whenever current_time >= create_time + ttl AND (status == COMPLETED or status == FAILED or status == KILLED) is satisfied, the entry is removed from zookeeper.

Depending on the TTL / part size, there is a chance the entry is removed from zookeeper as soon as it finishes. This might hurt observability.

One approach to this problem is creating a new field completion_time and use this field instead of create_time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions