-
Notifications
You must be signed in to change notification settings - Fork 201
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge changes from local fork into ASWF repo --------- Signed-off-by: Diego Tavares <[email protected]>
1 parent
5d49cfa
commit 1219377
Showing
24 changed files
with
410 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,6 +9,7 @@ venv*/ | |
.coverage | ||
coverage.xml | ||
htmlcov/ | ||
rest_gateway/*.tar\.gz | ||
/.env | ||
.envrc | ||
.vscode | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
0.34 | ||
0.35 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
# Build from project root | ||
FROM rockylinux:9.3 as rocky-golang | ||
|
||
WORKDIR /src/go | ||
RUN dnf install -y 'dnf-command(config-manager)' && dnf config-manager --set-enabled crb | ||
|
||
# WARN: Download do tarball and update file path accordingly | ||
COPY rest_gateway/Go_1.22.2_Linux_ARM64.tar.gz golang.tar.gz | ||
|
||
RUN tar -C /usr/local -xzf golang.tar.gz | ||
ENV PATH="$PATH:/usr/local/go/bin" | ||
|
||
# WARN: Uncoment if your environment requires a proxy | ||
#ENV GOPROXY=artifactory.yourcompany.com/go-proxy | ||
ENV GO111MODULE=on | ||
ENV GOSUMDB=off | ||
|
||
RUN go version | ||
|
||
RUN rm -rf /src/go | ||
|
||
FROM rocky-golang AS build | ||
|
||
RUN dnf install -y \ | ||
git \ | ||
protobuf-compiler \ | ||
&& dnf clean all | ||
WORKDIR /app | ||
ENV PATH=$PATH:/root/go/bin:/opt/protobuf3/usr/bin/ | ||
|
||
# Copy all of the staged files (protos plus go source) | ||
COPY ./proto /app/proto | ||
COPY ./rest_gateway/opencue_gateway /app/opencue_gateway | ||
# COPY ./lib /app/lib | ||
|
||
|
||
WORKDIR /app/opencue_gateway | ||
RUN go mod init opencue_gateway && go mod tidy | ||
|
||
RUN go install \ | ||
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway \ | ||
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2 \ | ||
google.golang.org/protobuf/cmd/protoc-gen-go \ | ||
google.golang.org/grpc/cmd/protoc-gen-go-grpc | ||
|
||
# Generate go grpc code | ||
RUN mkdir -p gen/go && \ | ||
protoc -I ../proto/ \ | ||
--go_out ./gen/go/ \ | ||
--go_opt paths=source_relative \ | ||
--go-grpc_out ./gen/go/ \ | ||
--go-grpc_opt paths=source_relative \ | ||
../proto/*.proto | ||
|
||
# Generate grpc-gateway handlers | ||
RUN protoc -I ../proto/ --grpc-gateway_out ./gen/go \ | ||
--grpc-gateway_opt paths=source_relative \ | ||
--grpc-gateway_opt generate_unbound_methods=true \ | ||
../proto/*.proto | ||
|
||
# Build project | ||
RUN go build -o grpc_gateway main.go | ||
|
||
FROM rockylinux:9.3 | ||
COPY --from=build /app/opencue_gateway/grpc_gateway /app/ | ||
|
||
EXPOSE 8448 | ||
ENTRYPOINT ["/app/grpc_gateway"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,163 @@ | ||
# Opencue Rest Gateway | ||
|
||
A gateway to provide a REST endpoint to opencue gRPC API. | ||
|
||
## How does it work | ||
|
||
This is a go serviced based on the official [grpc-gateway project](https://github.com/grpc-ecosystem/grpc-gateway) | ||
that compiles opencue's proto files into a go service that provides a REST interface and redirect calls to the | ||
grpc endpoint. | ||
|
||
## Running the service | ||
|
||
Running the service is very simple: | ||
* Read and modify the rest_gateway/Dockerfile according to your environment and build the gateway image using docker. | ||
* Run the image providing the environment variable `CUEBOT_ENDPOINT=your.cuebot.server:8443` | ||
|
||
## REST interface | ||
|
||
All service rpc calls are accessible: | ||
* HTTP method is POST | ||
* URI path is built from the service’s name and method: /<fully qualified service name>/<method name> (e.g.: /show.ShowInterface/FindShow) | ||
* HTTP body is a JSON with the request object: e.g.: | ||
```proto | ||
message ShowFindShowRequest { | ||
string name = 1; | ||
} | ||
``` | ||
becomes: | ||
```json | ||
{ | ||
"name": "value for name" | ||
} | ||
``` | ||
* HTTP response is a JSON object with the formatted response | ||
### Example (getting a show): | ||
show.proto: | ||
```proto | ||
service ShowInterface { | ||
// Find a show with the specified name. | ||
rpc FindShow(ShowFindShowRequest) returns (ShowFindShowResponse); | ||
} | ||
message ShowFindShowRequest { | ||
string name = 1; | ||
} | ||
message ShowFindShowResponse { | ||
Show show = 1; | ||
} | ||
message Show { | ||
string id = 1; | ||
string name = 2; | ||
float default_min_cores = 3; | ||
float default_max_cores = 4; | ||
string comment_email = 5; | ||
bool booking_enabled = 6; | ||
bool dispatch_enabled = 7; | ||
bool active = 8; | ||
ShowStats show_stats = 9; | ||
float default_min_gpus = 10; | ||
float default_max_gpus = 11; | ||
} | ||
``` | ||
request (gateway running on `http://opencue-gateway.apps.com`): | ||
```bash | ||
curl -i -X POST http://opencue-gateway.apps.com/show.ShowInterface/FindShow -d '{"name": "ashow"}` | ||
``` | ||
response | ||
```bash | ||
HTTP/1.1 200 OK | ||
Content-Type: application/json | ||
Grpc-Metadata-Content-Type: application/grpc | ||
Grpc-Metadata-Grpc-Accept-Encoding: gzip | ||
Date: Tue, 12 Dec 2023 18:05:18 GMT | ||
Content-Length: 501 | ||
{"show":{"id":"00000000-0000-0000-0000-99999999999999","name":"ashow","defaultMinCores":1,"defaultMaxCores":10,"commentEmail":"[email protected]","bookingEnabled":true,"dispatchEnabled":true,"active":true,"showStats":{"runningFrames":75,"deadFrames":14,"pendingFrames":1814,"pendingJobs":175,"createdJobCount":"2353643","createdFrameCount":"10344702","renderedFrameCount":"9733366","failedFrameCount":"1096394","reservedCores":252,"reservedGpus":0},"defaultMinGpus":100,"defaultMaxGpus":100000}} | ||
``` | ||
### Example (getting frames for a job): | ||
job.proto: | ||
```proto | ||
service JobInterface { | ||
// Returns all frame objects that match FrameSearchCriteria | ||
rpc GetFrames(JobGetFramesRequest) returns (JobGetFramesResponse); | ||
} | ||
message JobGetFramesRequest { | ||
Job job = 1; | ||
FrameSearchCriteria req = 2; | ||
} | ||
message Job { | ||
string id = 1; | ||
JobState state = 2; | ||
string name = 3; | ||
string shot = 4; | ||
string show = 5; | ||
string user = 6; | ||
string group = 7; | ||
string facility = 8; | ||
string os = 9; | ||
oneof uid_optional { | ||
int32 uid = 10; | ||
} | ||
int32 priority = 11; | ||
float min_cores = 12; | ||
float max_cores = 13; | ||
string log_dir = 14; | ||
bool is_paused = 15; | ||
bool has_comment = 16; | ||
bool auto_eat = 17; | ||
int32 start_time = 18; | ||
int32 stop_time = 19; | ||
JobStats job_stats = 20; | ||
float min_gpus = 21; | ||
float max_gpus = 22; | ||
} | ||
// Object for frame searching | ||
message FrameSearchCriteria { | ||
repeated string ids = 1; | ||
repeated string frames = 2; | ||
repeated string layers = 3; | ||
FrameStateSeq states = 4; | ||
string frame_range = 5; | ||
string memory_range = 6; | ||
string duration_range = 7; | ||
int32 page = 8; | ||
int32 limit = 9; | ||
int32 change_date = 10; | ||
int32 max_results = 11; | ||
int32 offset = 12; | ||
bool include_finished = 13; | ||
} | ||
message JobGetFramesResponse { | ||
FrameSeq frames = 1; | ||
} | ||
// A sequence of Frames | ||
message FrameSeq { | ||
repeated Frame frames = 1; | ||
} | ||
``` | ||
request (gateway running on `http://opencue-gateway.apps.com`): | ||
Note: it is important to include 'page' and 'limit' when getting frames for a job. | ||
```bash | ||
curl -i -X POST http://opencue-gateway.apps.com/job.JobInterface/GetFrames -d '{"job":{"id":"9999999999-b8d7-9999-a29c-99999999999999"}, "req": {"include_finished":true,"page":1,"limit":100}}' | ||
``` | ||
response | ||
```bash | ||
HTTP/1.1 200 OK | ||
content-type: application/json | ||
grpc-metadata-content-type: application/grpc | ||
grpc-metadata-grpc-accept-encoding: gzip | ||
date: Tue, 13 Feb 2024 17:15:49 GMT | ||
transfer-encoding: chunked | ||
set-cookie: 3d3a38cc45d028e42e93031e0ccc9b1e=534d34fde72242856a7fdadc27260929; path=/; HttpOnly | ||
{"frames":{"frames":[{"id":"9999999", "name":"0001-some_frame_0999990", "layerName":"h", "number":1, "state":"WAITING", "retryCount":0, "exitStatus":-1, "dispatchOrder":0, "startTime":0, "stopTime":0, "maxRss":"0", "usedMemory":"0", "reservedMemory":"0", "reservedGpuMemory":"0", "lastResource":"/0.00/0", "checkpointState":"DISABLED", "checkpointCount":0, "totalCoreTime":0, "lluTime":1707842141, "totalGpuTime":0, "maxGpuMemory":"0", "usedGpuMemory":"0", "frameStateDisplayOverride":null}, {"id":"10fa17d4-9313-4924-86f5-380c5b2a25d8", "name":"0002-some_frame_0999990", "layerName":"some_frame_0999990", "number":2, "state":"WAITING", "retryCount":0, "exitStatus":-1, "dispatchOrder":1, "startTime":0, "stopTime":0, "maxRss":"0", "usedMemory":"0", "reservedMemory":"0", "reservedGpuMemory":"0", "lastResource":"/0.00/0", "checkpointState":"DISABLED", "checkpointCount":0, "totalCoreTime":0, "lluTime":1707842141, "totalGpuTime":0, "maxGpuMemory":"0", "usedGpuMemory":"0", "frameStateDisplayOverride":null}, {"id":"8d39c602-0b27-4b1e-a09b-4fa35db40e55", "name":"0003-some_frame_0999990", "layerName":"some_frame_0999990", "number":3, "state":"WAITING", "retryCount":0, "exitStatus":-1, "dispatchOrder":2, "startTime":0, "stopTime":0, "maxRss":"0", "usedMemory":"0", "reservedMemory":"0", "reservedGpuMemory":"0", "lastResource":"/0.00/0", "checkpointState":"DISABLED", "checkpointCount":0, "totalCoreTime":0, "lluTime":1707842141, "totalGpuTime":0, "maxGpuMemory":"0", "usedGpuMemory":"0", "frameStateDisplayOverride":null}, {"id":"d418a837-e974-4716-9105-296f495bc407", "name":"0004-some_frame_0999990", "layerName":"some_frame_0999990", "number":4, "state":"WAITING", "retryCount":0, "exitStatus":-1, "dispatchOrder":3, "startTime":0, "stopTime":0, "maxRss":"0", "usedMemory":"0", "reservedMemory":"0", "reservedGpuMemory":"0", "lastResource":"/0.00/0", "checkpointState":"DISABLED", "checkpointCount":0, "totalCoreTime":0, "lluTime":1707842141, "totalGpuTime":0, "maxGpuMemory":"0", "usedGpuMemory":"0", "frameStateDisplayOverride":null}, {"id":"d2113372-99999-4c05-8100-9999999", "name":"0005-some_frame_0999990", "layerName":"some_frame_0999990", "number":5, "state":"WAITING", "retryCount":0, "exitStatus":-1, "dispatchOrder":4, "startTime":0, "stopTime":0, "maxRss":"0", "usedMemory":"0", "reservedMemory":"0", "reservedGpuMemory":"0", "lastResource":"/0.00/0", "checkpointState":"DISABLED", "checkpointCount":0, "totalCoreTime":0, "lluTime":1707842141, "totalGpuTime":0, "maxGpuMemory":"0", "usedGpuMemory":"0", "frameStateDisplayOverride":null}]}} | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
package tools | ||
|
||
import ( | ||
_ "github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway" | ||
_ "github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2" | ||
_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc" | ||
_ "google.golang.org/protobuf/cmd/protoc-gen-go" | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
package main | ||
import ( | ||
"context" | ||
"flag" | ||
"net/http" | ||
"os" | ||
|
||
"github.com/grpc-ecosystem/grpc-gateway/v2/runtime" | ||
"google.golang.org/grpc" | ||
"google.golang.org/grpc/credentials/insecure" | ||
"google.golang.org/grpc/grpclog" | ||
|
||
gw "opencue_gateway/gen/go" // Update | ||
) | ||
|
||
func getEnv(key, fallback string) string { | ||
if value, ok := os.LookupEnv(key); ok { | ||
return value | ||
} | ||
return fallback | ||
} | ||
|
||
func run() error { | ||
grpcServerEndpoint := getEnv("CUEBOT_ENDPOINT", "opencuetest01.your.test.server:8443") | ||
port := getEnv("REST_PORT", "8448") | ||
|
||
ctx := context.Background() | ||
ctx, cancel := context.WithCancel(ctx) | ||
defer cancel() | ||
|
||
// Register gRPC server endpoint | ||
// Note: Make sure the gRPC server is running properly and accessible | ||
mux := runtime.NewServeMux() | ||
opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())} | ||
|
||
// show.proto | ||
errShow := gw.RegisterShowInterfaceHandlerFromEndpoint(ctx, mux, grpcServerEndpoint, opts) | ||
if errShow != nil { | ||
return errShow | ||
} | ||
|
||
errFrame := gw.RegisterFrameInterfaceHandlerFromEndpoint(ctx, mux, grpcServerEndpoint, opts) | ||
if errFrame != nil { | ||
return errFrame | ||
} | ||
errGroup := gw.RegisterGroupInterfaceHandlerFromEndpoint(ctx, mux, grpcServerEndpoint, opts) | ||
if errGroup != nil { | ||
return errGroup | ||
} | ||
errJob := gw.RegisterJobInterfaceHandlerFromEndpoint(ctx, mux, grpcServerEndpoint, opts) | ||
if errJob != nil { | ||
return errJob | ||
} | ||
errLayer := gw.RegisterLayerInterfaceHandlerFromEndpoint(ctx, mux, grpcServerEndpoint, opts) | ||
if errLayer != nil { | ||
return errLayer | ||
} | ||
|
||
// host.proto | ||
errDeed := gw.RegisterDeedInterfaceHandlerFromEndpoint(ctx, mux, grpcServerEndpoint, opts) | ||
if errDeed != nil { | ||
return errDeed | ||
} | ||
errHost := gw.RegisterHostInterfaceHandlerFromEndpoint(ctx, mux, grpcServerEndpoint, opts) | ||
if errHost != nil { | ||
return errHost | ||
} | ||
errOwner := gw.RegisterOwnerInterfaceHandlerFromEndpoint(ctx, mux, grpcServerEndpoint, opts) | ||
if errOwner != nil { | ||
return errOwner | ||
} | ||
errProc := gw.RegisterProcInterfaceHandlerFromEndpoint(ctx, mux, grpcServerEndpoint, opts) | ||
if errProc != nil { | ||
return errProc | ||
} | ||
|
||
|
||
// Start HTTP server (and proxy calls to gRPC server endpoint) | ||
return http.ListenAndServe(":" + port, mux) | ||
} | ||
|
||
func main() { | ||
flag.Parse() | ||
|
||
if err := run(); err != nil { | ||
grpclog.Fatal(err) | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
kind: Deployment | ||
apiVersion: apps/v1 | ||
metadata: | ||
name: opencue-rest-gateway | ||
namespace: opencue | ||
labels: | ||
app: opencue | ||
component: rest-gateway | ||
spec: | ||
replicas: 1 | ||
selector: | ||
matchLabels: | ||
app: opencue | ||
component: rest-gateway | ||
template: | ||
metadata: | ||
creationTimestamp: null | ||
labels: | ||
app: opencue | ||
component: rest-gateway | ||
spec: | ||
containers: | ||
- name: rest-gateway | ||
imagePullPolicy: Always | ||
terminationMessagePolicy: File | ||
ports: | ||
- containerPort: 8448 | ||
image: docker-local.artifactory.spimageworks.com/gitlab/spi/dev/infrastructure/api/opencue/rest_gateway:latest | ||
env: | ||
- name: CUEBOT_ENDPOINT | ||
values: cuebot-grpc.opencue.svc.cluster.local:8443 | ||
resources: | ||
limits: | ||
cpu: "1000m" | ||
memory: "2Gi" | ||
requests: | ||
cpu: "200m" | ||
memory: "1Gi" | ||
restartPolicy: Always | ||
terminationGracePeriodSeconds: 10 | ||
--- | ||
apiVersion: v1 | ||
kind: Service | ||
metadata: | ||
name: rest-gateway-service | ||
namespace: opencue | ||
spec: | ||
selector: | ||
app: opencue | ||
component: rest-gateway | ||
ports: | ||
- protocol: TCP | ||
port: 8448 | ||
targetPort: 8448 |