Skip to content

First wave of fixes for remote-run #4033

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 27, 2025
Merged

First wave of fixes for remote-run #4033

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d1421ae
describe 403 code for remote-run apis
piax93 Mar 26, 2025
3a0e02e
add more output to remote-run cli
piax93 Mar 26, 2025
175b153
fix wrong kwarg in paasta_cleanup_remote_run_resources
piax93 Mar 26, 2025
bfef60c
drop unneed pod options for remote-run jobs
piax93 Mar 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion paasta_tools/api/api_docs/oapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
openapi: 3.0.0
info:
title: Paasta API
version: 1.1.0
version: 1.1.1
servers:
- url: "{scheme}://{host}/{basePath}"
variables:
Expand Down Expand Up @@ -1216,6 +1216,8 @@ paths:
schema:
$ref: '#/components/schemas/RemoteRunOutcome'
description: Successfully started remote-run sandbox
"403":
description: Unauthorized to remote-run this service
"404":
description: Service instance not found
"500":
Expand Down Expand Up @@ -1252,6 +1254,8 @@ paths:
schema:
$ref: '#/components/schemas/RemoteRunOutcome'
description: Remote run pod stopped
"403":
description: Unauthorized to remote-run this service
"404":
description: Service instance not found
"500":
Expand Down Expand Up @@ -1288,6 +1292,8 @@ paths:
schema:
$ref: '#/components/schemas/RemoteRunOutcome'
description: Pod status information
"403":
description: Unauthorized to remote-run this service
"404":
description: Service instance not found
"500":
Expand Down Expand Up @@ -1324,6 +1330,8 @@ paths:
schema:
$ref: '#/components/schemas/RemoteRunToken'
description: Token generated successfully
"403":
description: Unauthorized to remote-run this service
"404":
description: Service instance not found
"500":
Expand Down
14 changes: 13 additions & 1 deletion paasta_tools/api/api_docs/swagger.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"swagger": "2.0",
"info": {
"title": "Paasta API",
"version": "1.1.0"
"version": "1.1.1"
},
"basePath": "/v1",
"schemes": [
Expand Down Expand Up @@ -856,6 +856,9 @@
"$ref": "#/definitions/RemoteRunOutcome"
}
},
"403": {
"description": "Unauthorized to remote-run this service"
},
"404": {
"description": "Deployment key not found"
},
Expand Down Expand Up @@ -904,6 +907,9 @@
"$ref": "#/definitions/RemoteRunOutcome"
}
},
"403": {
"description": "Unauthorized to remote-run this service"
},
"404": {
"description": "Service instance not found"
},
Expand Down Expand Up @@ -952,6 +958,9 @@
"$ref": "#/definitions/RemoteRunOutcome"
}
},
"403": {
"description": "Unauthorized to remote-run this service"
},
"404": {
"description": "Service instance not found"
},
Expand Down Expand Up @@ -998,6 +1007,9 @@
"$ref": "#/definitions/RemoteRunToken"
}
},
"403": {
"description": "Unauthorized to remote-run this service"
},
"404": {
"description": "Service instance not found"
},
Expand Down
7 changes: 7 additions & 0 deletions paasta_tools/cli/cmds/remote_run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ def paasta_remote_run_start(
print(f"Error from PaaSTA APIs while starting job: {start_response.message}")
return 1

print(
f"Triggered remote-run job for {args.service}. Waiting for pod to come online..."
)
start_time = time.time()
while time.time() - start_time < args.timeout:
poll_response = client.remote_run.remote_run_poll(
Expand All @@ -69,7 +72,9 @@ def paasta_remote_run_start(
start_response.job_name,
)
if poll_response.status == 200:
print("")
break
print(f"\rStatus: {poll_response.message}", end="")
time.sleep(10)
else:
print("Timed out while waiting for job to start")
Expand All @@ -79,6 +84,8 @@ def paasta_remote_run_start(
print("Successfully started remote-run job")
return 0

print("Pod ready, establishing interactive session...")

token_response = client.remote_run.remote_run_token(
args.service, args.instance, user
)
Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/kubernetes/bin/paasta_cleanup_remote_run_resources.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def parse_args() -> argparse.Namespace:
def main():
args = parse_args()
kube_client = KubeClient()
age_limit = datetime.now(tzinfo=timezone.utc) - timedelta(seconds=args.max_age)
age_limit = datetime.now(tz=timezone.utc) - timedelta(seconds=args.max_age)
for namespace in get_all_managed_namespaces(kube_client):
clean_namespace(kube_client, namespace, age_limit)

Expand Down
28 changes: 21 additions & 7 deletions paasta_tools/kubernetes_tools.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1436,6 +1436,7 @@ def get_kubernetes_containers(
aws_ebs_volumes: Sequence[AwsEbsVolume],
secret_volumes: Sequence[SecretVolume],
service_namespace_config: ServiceNamespaceConfig,
include_sidecars: bool = True,
) -> Sequence[V1Container]:
ports = [self.get_container_port()]
# MONK-1130
Expand Down Expand Up @@ -1471,11 +1472,13 @@ def get_kubernetes_containers(
projected_sa_volumes=self.get_projected_sa_volumes(),
),
)
containers = [service_container] + self.get_sidecar_containers( # type: ignore
system_paasta_config=system_paasta_config,
service_namespace_config=service_namespace_config,
hacheck_sidecar_volumes=hacheck_sidecar_volumes,
)
containers = [service_container]
if include_sidecars:
containers += self.get_sidecar_containers( # type: ignore
system_paasta_config=system_paasta_config,
service_namespace_config=service_namespace_config,
hacheck_sidecar_volumes=hacheck_sidecar_volumes,
)
return containers

def get_readiness_probe(
Expand Down Expand Up @@ -2052,11 +2055,15 @@ def format_kubernetes_job(
self,
job_label: str,
deadline_seconds: int = 3600,
keep_routable_ip: bool = False,
include_sidecars: bool = False,
) -> V1Job:
"""Create the config for launching the deployment as a Job

:param str job_label: value to set for the "job type" label
:param int deadline_seconds: maximum allowed duration for the job
:param bool keep_routable_ip: maintain routable IP annotation in pod template
:param bool include_sidecars: do not discard sidecar containers when building pod spec
:return: job object
"""
try:
Expand All @@ -2073,6 +2080,8 @@ def format_kubernetes_job(
git_sha=git_sha,
system_paasta_config=system_paasta_config,
restart_on_failure=False,
include_sidecars=include_sidecars,
force_no_routable_ip=not keep_routable_ip,
),
),
)
Expand Down Expand Up @@ -2205,6 +2214,8 @@ def get_pod_template_spec(
git_sha: str,
system_paasta_config: SystemPaastaConfig,
restart_on_failure: bool = True,
include_sidecars: bool = True,
force_no_routable_ip: bool = False,
) -> V1PodTemplateSpec:
service_namespace_config = load_service_namespace_config(
service=self.service, namespace=self.get_nerve_namespace()
Expand All @@ -2214,8 +2225,10 @@ def get_pod_template_spec(
)

hacheck_sidecar_volumes = system_paasta_config.get_hacheck_sidecar_volumes()
has_routable_ip = self.has_routable_ip(
service_namespace_config, system_paasta_config
has_routable_ip = (
"false"
if force_no_routable_ip
else self.has_routable_ip(service_namespace_config, system_paasta_config)
Comment on lines +2228 to +2231
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one day we'll update things so that the default is no routable ip #dream

)
annotations: KubePodAnnotations = {
"smartstack_registrations": json.dumps(self.get_registrations()),
Expand All @@ -2239,6 +2252,7 @@ def get_pod_template_spec(
secret_volumes=self.get_secret_volumes(),
system_paasta_config=system_paasta_config,
service_namespace_config=service_namespace_config,
include_sidecars=include_sidecars,
),
share_process_namespace=True,
node_selector=self.get_node_selector(),
Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/api/autoscaler_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/api/default_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/api/remote_run_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/api/resources_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/api/service_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/api_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
4 changes: 2 additions & 2 deletions paasta_tools/paastaapi/configuration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down Expand Up @@ -370,7 +370,7 @@ def to_debug_report(self):
return "Python SDK Debug Report:\n"\
"OS: {env}\n"\
"Python Version: {pyversion}\n"\
"Version of the API: 1.1.0\n"\
"Version of the API: 1.1.1\n"\
"SDK Package Version: 1.0.0".\
format(env=sys.platform, pyversion=sys.version)

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/exceptions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/model/adhoc_launch_history.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/model/autoscaler_count_msg.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/model/deploy_queue.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/model/deploy_queue_service_instance.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/model/envoy_backend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/model/envoy_location.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/model/envoy_status.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/model/flink_cluster_overview.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/model/flink_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
2 changes: 1 addition & 1 deletion paasta_tools/paastaapi/model/flink_job.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501

The version of the OpenAPI document: 1.1.0
The version of the OpenAPI document: 1.1.1
Generated by: https://openapi-generator.tech
"""

Expand Down
Loading
Loading