-
Notifications
You must be signed in to change notification settings - Fork 80
Contributor meetings
The Trino Gateway project organizes regular meetings with contributors to improve collaboration with maintainers and among contributors. This space acts is used to plan upcoming meetings and provide meeting minutes after the event.
- Virtual meetings are held every second Wednesday, 6am PST.
- Anyone can attend.
- Contact Manfred Moser on Trino slack for an invite.
- Meeting details are also announced on slack in the trino-gateway-dev channel
- Topics can be suggested prior to the event by updating this page, message on the slack channel, or during the meeting
Attendees
mosabua, Chaho12, willmostly, vishalya, andythsu, oneonestar, wendigo, Simian You (Doordash)
Topics, notes, action items, decisions
- mosabua
- Standardizing on a better config file name name than gateway-ha-config.yml
- for example config.yml only, or trino-gateway-config.yml
- https://github.com/trinodb/trino-gateway/pull/586
- Discussed and agreed on
config.yaml
- https://github.com/trinodb/trino-gateway/issues/598
- wendigo
- proposal to use launcher from airlift
- discussed and found that due to the yaml usage for config that is currently not possible
- we would also have to discuss benefits
- vishalya
- Filter proposal at https://github.com/trinodb/trino-gateway/issues/591
- wendigo very knowledgeable
- In principal we want to go ahead
- At least sample implementation and docs
- HTTP filter, already possible as part of airlift, use jakarta filters
- willmostly
- heads up - some db changes are on his horizon,
- storing rules in the db - new table,
- some configs to become specific per Trino cluster, such as the xforwarded header
- Discussed flyway setup, willmostly to check but we should NOT have automatic downgrades, if wrong schema , nodes should fail to start
- oneonestar
- health check API and PR to load all modules by default, need review
- Team agreed in principal, reviews to follow
- https://github.com/trinodb/trino-gateway/pull/597
- https://github.com/trinodb/trino-gateway/pull/595
- andythsu
- testing of routing rules in CI or in running application, upgrade broke some routing
- Simian
- will ask for approval for Doordash to be added to users page
Attendees
mosabua, Chaho12, willmostly, vishalya, andythsu, oneonestar
Topics, notes, action items, decisions
- mosabua
- Presentation from Trino Summit is live, with slides and video, 200 views so far
- https://trinodb.github.io/trino-gateway/resources/#opening-up-the-trino-gateway
- https://trinodb.github.io/trino-gateway/users/
- UX issues found - filed https://github.com/trinodb/trino-gateway/issues/585
- PR and issue backlog work
- Working towards 14 after round of reviews and merges
- Chat about Netflix presentation
- Scale and such
- Aiming to look at Trino Gateway
- vishalya and mosabua to invite speaker to join dev sync to chat
- Chaho12
- external routing needs a timeout option
- had a hard time debugging issue where another service used in the external routing server had timing issues
- Created https://github.com/trinodb/trino-gateway/pull/580
- request to make client timeouts consistent - mosabua to initiate and coordinate
- willmostly
- easyrules removal PR is ready again, looking for final review
- also working on JMX and router rules UI PRs
- We need to add some more monitoring/metrics, for example response time overall, discussion with others
- vishal
- looking at more capabilities to validate headers some more, protecting endpoints
- still in design, in Trino Gateway or not?,
- team requested more details in an issue
- andythsu
- how to instrument jetty for more metrics
- willmostly and mosabua - lots already done in airlift, ask wendigo for more info if needed
- some info in https://github.com/trinodb/trino/pull/21089 should also apply in gateway
- oneonestar
- Had some issues with k8s and the fact that we have no healthcheck endpoint
- team agreed we need one or two
- should replace using the UI as healtcheck as currently done in chart
- oneonestar to send PR
Attendees
nineinchnick, mosabua, Chaho12, willmostly, vishalya, Anna Schibli
Topics, notes, action items, decisions
- Trino Gateway 13 release is out
- No additional issues so far
- Helm chart improvement yet to be done and merged, will be 1.13.1
- Create separate page and explain versioning and such - mosabua
- Database init
- mentioned by willmostly
- we agreed to adopt flyway or something to automatically init the db
- to be done as part of app, not init container or chart, so it works with any deployment
- need upgrade test
- needs docs about required permissions (write and ddl)
- Final review and editing of slide deck for Trino Summit
- Editing completed
- We decided to record live during the event
Attendees
mosabua, Chaho12, andythsu, willmostly, oneonestar, alaturqua
Topics, notes, action items, decisions
- Trino Gateway 12 is out, plan for 13
- Presentation at Trino Summit
- Chart changes - remove chart in main repo, update docs, change release notes and link to charts, further changes
- JMX PR close to ready
Attendees
willmostly and others.
Topics, notes, action items, decisions
none available.
Attendees
mosabua, Chaho12, andythsu, vishalya, willmostly, oneonestar
Topics, notes, action items, decisions
- andythsu - waiting for 12 release, done a bunch of testing and deployment, all good
- willmostly - kill query PR approved and merged, ready for 12
- mosabua
- get 464 PR merged before 12, after 464 release
- trino summit talk submitted - to organize slide deck and so on
- vishalya and others
- easy-rules not maintained - wrapper around mvel, no plan yet, routing module with hardcoded custom rules maybe?, we all chat about routing rules and different implementation, will - also need to maintain ease of use, potentially plug in some scripting language
- oneonestar - coordinatorid PR, talking with martint and wendigo, also looking towards transaction support,
- over time we will need to look at issuing query id placeholder from the Trino Gateway and translate towards actual trino query id, so that we can have a queue and provision clusters behind Trino Gateway...
- vishalya - using ansible for trino gateway and trino, tied together
- jaehoo - with clause PR, almost finished, ideally go in with 12,
- cli and extra headers, https://github.com/trinodb/trino/pull/15826, mosabua to follow up
Attendees
mosabua, Chaho12, andythsu, vishalya, willmostly, oneonestar, nineinchnick, rdsarvar, dprophet, dancodeluppi
Topics, notes, action items, decisions
- Discuss approaches for managing and publishing the Helm chart
- nineinchnick will help
- aim to allow
helm repo add
usage instead of current requirement to check out tagged source - https://github.com/trinodb/trino-gateway/issues/391
- decided to use gh pages of "charts" repo
- decided to leave code in trino-gateway repo for now since we believe that things will continue to evolve in parallel
- also easier for subproject maintainers
- Trino Gateway release flow will publish into gh pages of charts repo
- Trino Gateway docs will include docs and point to reference docs in chart repo
- Trino Gateway 12
- mosabua looking for review on https://github.com/trinodb/trino-gateway/pull/525
- mosabua looking for review on https://github.com/trinodb/trino-gateway/pull/518
- willmostly looking for review on https://github.com/trinodb/trino-gateway/pull/505, done by mosabua now
- mosabua looking for release notes review https://github.com/trinodb/trino-gateway/pull/473
- willmostly close to merge with https://github.com/trinodb/trino-gateway/pull/425
- Secrets, Helm and more
- rdsarvar and others discussed
- originally no init container in helm charts to keep it simpler
- might want to add after all
- support secrets from env variable, injected by secrets manager
- we will continue to work towards both
- Trino and airlift are getting improved secrets mgt, Trino Gateway will follow
- https://github.com/trinodb/trino/pull/22633
- https://github.com/airlift/airlift/pull/1217
- env vars and keystores
- SQL transactions support
- brought up by willmostly
- created https://github.com/trinodb/trino-gateway/issues/526
- Trino Gateway expose to public internet?
- dprophet
- nobody in the meeting does that currently
- seems risky, discussed potential exploit approaches
- https://github.com/j-easy/easy-rules is unmaintained
- has remote code execution CVE
- https://nvd.nist.gov/vuln/detail/CVE-2023-50571
- oneonestar has removed easyrules on his fork
- created follow up issue https://github.com/trinodb/trino-gateway/issues/527
Attendees
mosabua, Chaho12, andythsu, willmostly, oneonestar, xkrogen,
Topics, notes, action items, decisions
- mosabua
- working on release notes and PR reviews to get version 12 out soon
- mysql PR currently stale
- docker docs still tbd
- Maybe update runner class soon
- Presentation at OpenSourceDataSummit with will mostly about Trino Gateway
- Slide deck on the website, recording to come
- willmostly
- https://github.com/trinodb/trino-gateway/pull/425
- kill query PR getting close, final changes
- to be included in 12
- andythsu
- https://github.com/trinodb/trino-gateway/pull/360
- pending state PR in final stage
- needs rebase but then probably ready to go
- to be included in 12
- oneonestar
- PR template update from Trino
- checkboxes create tasklist- mosabua to revert in Trino
- PRs from Ben ( @choiwaiyiu)
- JMX for cluster stats https://github.com/trinodb/trino-gateway/pull/502
- Cache for backend db https://github.com/trinodb/trino-gateway/pull/501
- Both would be good for 12
- Chaho12
- https://github.com/trinodb/trino-gateway/pull/497/files
- agreed on renaming variables in meeting, to be updated and then ready
- xkrogen
- Switch from Presto Gateway fork coming soon
- They will contribute missing features and other improvements to Trino Gateway
- Will work on features related to health check and monitoring, failure rates
- Potentially have issue for design discussion by next meeting
- Has to figure out how to use non-LTS Java
Attendees
mosabua, Chaho12, andythsu, vishalya, willmostly
Topics, notes, action items, decisions
- Trino Gateway 11 shipped, so far no issues
- monitoring via JMX, https://github.com/trinodb/trino-gateway/pull/472
- should be fine, but not ideal, beans are not guaranteed to be stable across Trino releases
- JMX not really faster as protocol
- willmostly and mosabua to chase down health check API in trino and JMX PR from Isa
- also help with resource groups
- Projects for intern via vishalya
- backend state db improvements
- willmostly also looking to put rules into DB, also to help with UI PR https://github.com/trinodb/trino-gateway/pull/433
- also could help on docs and UI
- Chat about sticky routing xforwarded header and routing based on FQDN/URL
- Discussion about resource groups
- use same db for all trino cluters
- need to test and maybe fix and improve
- probably not remove code unless its badly broken and nobody wants it, need to also check with @xkrogen
- Helm chart
- mosabua to look at improved publishing and docs
- External routing server - ideally get an example impl from Chaho12, also would be good for testing
Attendees
mosabua, Chaho12, andythsu, vishalya, oneonestar
Topics, notes, action items, decisions
- zoom background
- mosabua to add to presentations repo, https://github.com/trinodb/presentations/pull/27
- Release note suggestions
- mosabua explains and shows
- see also https://github.com/trinodb/trino.io/pull/702
- Squash and merge process
- mosabua explains and shows how to han
- Docs testing locally
- mosabua shows docs
- oneonestar mentions docker command in PR
- mosabua to add to docs, see https://github.com/trinodb/trino-gateway/pull/462
- Releasing Trino Gateway 11
- mosabua requests reviews on https://github.com/trinodb/trino-gateway/pull/326 to merge for 11
- chaho12 requesting review https://github.com/trinodb/trino-gateway/pull/423, ideally getting into 11
- HTTP/2 discussion
- raised by oneonestar
- mosabua to clarify status more
- currently HTTP/2 only relevant for Trino internal communication
- not yet for client connections, therefore also not yet for Trino Gateway
- Secrets
- oneonestar raises issue that secrets from env variables are not support for Trino Gateway config
- various approaches and issues are discussed
- oneonestar might look at implementing simple search and replace in config file after loading and before parsing
- vishalya interested to be reviewer
- vishal using ansible, env variables would work for him as well
- Stability with database failures
- or connection problems
- currently database down = Trino Gateway down
- not good, hard dependency on database being up and running and reachable
- oneonestar has workaround - copies state to memory and maintains in regular interval, going to send PR, vishalya interested to review
- Monitoring
- vishalya raised question about monitoring, as result from db discussion
- oneonestar mentions we added OpenMetrics
- oneonestar planning to use Prometheus
- chaho12 using https://victoriametrics.com/ for better scalability
- oneonestar uses compression with Prometheus
- mosabua - we will probably also implement opentelemetry over time, with usage of jaeger or datadog, but lots of data
Attendees
mosabua, Chaho12, andythsu, vishalya, willmostly, oneonestar
Topics, notes, action items, decisions
- Helm chart versioning (https://github.com/trinodb/trino-gateway/pull/438
- discussed and approved
- mosabua to send release process doc PR - done with https://github.com/trinodb/trino-gateway/pull/442
-
https://github.com/trinodb/trino-gateway/pull/417
- mosabua to review
-
https://github.com/trinodb/trino-gateway/pull/431
- probably to merge
-
https://github.com/trinodb/trino-gateway/pull/424
- needs a review, specifically from andythsu but also others, vishalya will follow up
-
https://github.com/trinodb/trino-gateway/pull/326
- we should get in, mosabua and others to review
-
https://github.com/trinodb/trino-gateway/pull/425
- just a simple test PR, mosabua and others to review
- vishalya is looking into Trino cluster scale up and down from Trino Gateway
- via interface that can then have an implmentation that uses a k8s operator
- mosabua still needs to organize k8s operator sync in larger trino community
- more complex operations like new clusters and such will be more complicated
- we will discuss use cases and more
-
https://github.com/trinodb/trino-gateway/pull/423
- from Chaho12
- waiting for other PR to merge from willmostly, and then review
- Release process
- mosabua will look at automation from wendigo in airlift and maybe adapt
- Java 22
- oneonestar would like to upgrade
- no objections from team
- mosabua - we should add PR to test with 21 and 22 first, same as in trino, mosabua submitted https://github.com/trinodb/trino-gateway/pull/443
- 22 would unblock update of Trino dependencies
- Routing based on domain name
- oneonestar has use case of routing based on domain name used for cluster groups,
- all behind one Trino Gateway
- issues with OAUTH and Trino web interface
- would be a welcome additional feature
- team provided some help, oneonestar will test and check
- Javascript and UI work
- mosabua mentioned looking for JS devs at Starburst and elsewhere
- for Trino Gateway, Trino Preview Web UI and js client
- mosabua will loop in prakhar for next syncs
mosabua unavailable, team skipped sync
Attendees
mosabua, Chaho12, andythsu, vishalya, willmostly, oneonestar, xkrogen, wendigo, dprophet
Topics, notes, action items, decisions
Release 10 status:
- merged release blockers
- release notes ready
- hopefully today or very soon, mosabua to organize with Martin
- doc change needed for process forwarded header, done via #418
Query rewriting discussion:
- Discussed need and use cases for query rewriting, brought by dprophet
- Question if this should be done in Trino Gateway or Trino, or a mixed approach of both
- All companies involved have need for this, and specific examples were shared
- willmostly shared his experience on how query rewriting can be very brittle and complex
- Xkrogen and others shared that metadata is crucial
- Also related to better plugin framework for gateway that allows service calls as part of routing
- service calls could go to separate service but also a Trino cluster, also for example a small cluster for metadata and explain plan generation only
- trino-lb already calls out to service so we know it works
- InQuery fork of Trino Gateway also used this approach successfully
- Group agreed that we want to move forward with adding support for query rewriting as well as custom plugins for such purposes, it is understood that this might require compromises and only work for specific implementation for each company for their use case.
- Group agreed we dont know what is going to work, we will make Trino Gateway pluggable so that different approaches can tried
- Group will collaborate on framework based on design from star
- Different groups will test offloading to other trino server, other rest service for metadata, and others
- Group agreed that over time Trino Gateway might take some workload and concerns from Trino coordinator, overlap and split is to be determined, depends on outcome and practical experience with different approaches
- Group agreed to collaborate on all that
Cross-cluster resource and quota management:
- Brought up as topic by xkrogen
- Desire to do load based routing to cluster
- Hence also need resource and quota management across cluster
- Need higher abstraction in Trino Gateway
- Group agreed with this need
- mosabua expressed how the current resource groups setup is just lifted from Trino and always felt architecturally wrong and weird since it forces same setup across clusters and at the same time has no knowledge of clusters and Trino Gateway
- aim is to be able to enforce resource groups and quota management in a global way
- current approach is leaky and allows resource overuse since multiple clusters are involved and enforcement is separate
- Agreed that this triggers the need for query queueing in Trino Gateway
- Discussed how the need for these features also brings us to the next step of actively scaling clusters or adding more clusters under control from Trino Gateway
- Approach is implemented and proven as workable in stackable trino-lb and their operator
- mosabua mentioned that collab on Trino operator has yet to be started, mosabua is working towards that
- All agreed that this is a valid use case and requirement and we desire to drive the project to support this
Other items:
- we will need to figure out how to get contributors or maintainers who can help with the javascript UI
- x-forwarded-prefix should allow same domain name for multiple clusters, filed as #419
- star requests that everyone adds use cases into the routing framework design issue #409
- central consistent storage of health check info should be moved forward, also necessary for resource groups and other efforts, call for more input and work on #222
- Trino API for cluster health, work needs to be kicked off, mosabua to file issue -> https://github.com/trinodb/trino/issues/22802
Attendees
mosabua, Chaho12, andythsu, vishalya, willmostly
Topics, notes, action items, decisions
- oneonestar new subproject maintainer
- airlift migration done
- new web interface context at /trino-gateway/ui
- Aim to release 10 soon
- mosabua will work on release notes
- helm chart PR needs to be merged before (https://github.com/trinodb/trino-gateway/pull/399)
- @willmostly aims to get process request PR merged before 10 https://github.com/trinodb/trino-gateway/pull/325
- @willmostly making slow progress on #326, does not have to hit before 10
- @vishalya asking for review and merge of #406, review done now by @mosabua
- @Chaho12 would like to get #398 merged before 10, longer discussion around approach ended up with decision to refactor and discuss more on slack
Attendees
mosabua, Chaho12, andythsu, vishalya, oneonestar, choiwaiyiu (Ben), xkrogen
Topics, notes, action items, decisions
- oneonestar introduces choiwaiyiu as future contributor and collaborator
- mosabua and others - Recap from Trino Fest and Trino Contributor Congregation
- vishalya - we need docs about properties that are supported in the new airlift setup, mosabua to follow up over time
- Talked about info about cluster health end point, auth is required, we need to implememnt new endpoint in Trino, potentially specific user,
- Trino UI endpoints considered unstable
- SQL parser integration in testing at Bloomberg (https://github.com/trinodb/trino-gateway/pull/325), looking good, we will merge PR after airlift completion
- Main uses cases at Bloomberg are federation and smart load balancing
- Discussion around parsing for smart load balancing vs explain plan running on Trino cluster, we will probably do both over time
- xkrogen had research project around predicting load based on SQL text only, also needs explain data, with model about past query history, will work on this again in the future
- scriptable routing rules in the works by star
- oneonestar shows his proposal on how to set up route modules sequence
- Chaho12 is currently building a new module, working with Vishal on how to implement, he will demo/document
- Discussion around reuse trino libraries like auth as dependencies
- Easier to use coordinator as service than pulling out dependencies
- Trino is not a library is a common quote
- xkrogen is potentially going to work on auth refacotr for trino and trino gateway to consolidate
- Talk about client protocol and compatibility with different Trino version, including Project swift
- will emerge over time but we plan on wider compatibility across Trino versions
- andythsu asked how to use
log.annotation-file
in Trino Gateway - later answered on slack - Plan for version 10 release
- Get https://github.com/trinodb/trino-gateway/pull/382 merged
- Get https://github.com/trinodb/trino-gateway/pull/325 merged
- Merge other PRs as needed
- Update docs
- Release hopefully before next sync
Attendees
mosabua, Chaho12, andythsu, vishalya, dprophet, oneonestar, rdsarvar, avinashdesireddy
Topics, notes, action items, decisions
- Review and discussion of the slide deck for TCC - all finalized and ready
- Updated slides a bit more around questions and future plans
6am PST
Attendees
mosabua, Chaho12, andythsu, willmostly, vishalya, oneonestar, rdsarvar
Topics, notes, action items, decisions
- Release 10 planning
- airlift refactor ideally hit before
- Slide deck for Trino Contributor Congregation
- Manfred created deck, team will edit together and review in next sync
- ping Manfred for access
- Discussed the issue around HTTPS and separate port https://github.com/trinodb/trino-gateway/pull/263
- mentioned by Will
- decided to close in favour of using only one HTTP port in airlift refactor and hence close this PR.
- Manfred has done that now
- Basically will be fixed in release 10
- Discussed how we have context paths for Trino and Trino Gateway that overlap
- Explained paths and extrawhitelist paths configuration
- Decided to move to specific context for Trino Gateway in the longer run
- For example /tgw/ui and /tgw/api and so on
- And then assume all others have to be passed through
- But that is a larger refactor for later (after airlift refactor and such)
- Vishal reports that release 9 is working well in production
- Decided to file issues for UI problems
- so we can track and work on the
- and also ping Peng in these issues
- Andy and Jaeho to work on pending state PR for review and merge
- Vishal and Will to work on Process Request PR
- manfred to add discussion topic for TCC
- unauthenticated Trino rest endpoint with number of workers and active queries - done
- Star will work on airlift removal soon
- requesting merge hold
- hopefully ready for review next week
6am PST
Attendees
mosabua, Chaho12, andythsu, willmostly, vishalya, oneonestar, rdsarvar
Topics, notes, action items, decisions
- Trino Gateway 8 and 9 releases, plan towards 10
- Documentation improvements
- Discussed #222 Trino gateway health state design
- Decided to proceed with separate in-memory status per instance for now, andythsu will work on PR
- Second step to add to UI
- Also add JMX mbean
- Openmetrics to follow when airlift refactor allows openmetrics easily, will need config like in Trino
- Debate about terms "Backends" vs "Trino clusters"
- Decided as team to refactor over time to use the term "Trino clusters" instead of "Backends"
- We will implement this in the codebase and the docs
- Query details extraction PR, willmostly
- Trino 445 and airlift upgrade caused issues due to transient jetty upgrade, will back out and do that separately later
- General advice from mosabua - keep PRs smaller to enable easier reviews and prevent scope creep
- willmostly asked about helm chart PR - mosabua already reviewed and approved
- Docs update will proceed with focus on installation and cleanup, mosabua to help
- vishalya working towards getting 9 release into prod
- vishalya reviewing sql parsing PR from willmostly
- vishalya mentioned small UI bugs
- we need to file issues and then also learn more react and familiarize ourselves with codebase
- andythsu and mosabua to look a bit
- mosabua to also pull in Peng (orig author of UI)
- Trino Contributor Congregration - team to collaborate on overview and roadmap deck
- Manfred to start deck
- oneonestar found some oauth/oidc security issue and created fix PRs, mosabua to review and work towards merge before 10 release, further testing can be done by others after merge easier
- oneonestar mentioned issue that currently two ports are used for UI and API separately, airlift does not support that
- team agreed that collapsing it all into one port is fine
- reason is most likely historical to separate security concerns, but we now have authentication so no problem
Cancelled due to availability issues
6am PST
Attendees
mosabua, Chaho12, andythsu, willmostly, dprophet, vishalya
Topics, notes, action items, decisions
- Discussion around Trino Gateway 8 release
- Decided to release asap even if helm chart and other changes don't make it in
- mosabua to update release notes and get things ready to work with martint
- Airlift as base
- work ongoing
- required to get plugin system off the ground
- also required for OpenTelemetry integration
- Discussion around smarter routing based on query and security info
- definitely something we are working towards,
- first effort in https://github.com/trinodb/trino-gateway/pull/325
- more to come, also including info of queried catalogs, schemas, and more
- in future potentially around query plan data including cost and query shape
- also discussed trino2trino connector and concluded that the two separate efforts are related and complimentary, so we will pursue both
- Cluster status
- Related to https://github.com/trinodb/trino-gateway/pull/163
- Need to support multiple Trino Gateway instances
- Store status in db and cache on nodes
- willmorrison walked through example guava cache code
- andythsu will create issue and close existing PR
- mosabua to potentially replace https://github.com/trinodb/trino-gateway/pull/263
- Docs site still in progress with mosabua
- we are looking to get more details for deployment advice
6am PST
Attendees
mosabua, Chaho12, andythsu, willmostly, vishalya,
Topics, notes, action items, decisions
- Trino Gateway 7 release done, working towards 8 soon
- HTTPS issue, agreed to merge before 8
- Client host redirect PR in Trino - oneonestar mentioned that this is not yet ready, mosabua mentioned it might therefore not make it for Trino 444, @electrum will review
- Agreed to merge router module PR, and get separate PR to follow up on code style, done by mosabua
- JDBI refactor - reviewed now, agreed to merge after router module
- mosabua to drive HTTPS PR to merge https://github.com/trinodb/trino-gateway/pull/263
- mosabua to replace https://github.com/trinodb/trino-gateway/pull/217/ with Trino config tips section, done in https://github.com/trinodb/trino-gateway/pull/297
- andythsu will get release 7 in prod soon and share info
- Chaho12 updated quickstart PR, mosabua to finalize https://github.com/trinodb/trino-gateway/pull/247, done
6am PST
Attendees
mosabua, vishalya, Chaho12, andythsu, oneonestar, wendigo, nineinchnick
Topics, notes, action items, decisions
- Trino Gateway 7 release about to be cut, release notes and codebase are ready
- Agreed to upgrade to Java 21 after 7 release - https://github.com/trinodb/trino-gateway/pull/225
- Doc content and site use has no progress, but work will continue soon https://github.com/trinodb/trino-gateway/pull/254
- Email notification PR needs update and review - https://github.com/trinodb/trino-gateway/pull/244
- Once new release is available, andythsu will provide feedback about production usage
- Agreed to get router module PR merged right after 7 is cut - https://github.com/trinodb/trino-gateway/pull/98
- Agreed to get ActiveJDBC removal/JDBI refactor merged right after 7 is cut - https://github.com/trinodb/trino-gateway/pull/284 and https://github.com/trinodb/trino-gateway/pull/287
- mosabua to set up topics in docs for production deployment requirements and tips
- vishalya to start rate limiting high level discussion
- mosabua to follow up about https://github.com/trinodb/trino/issues/21026 at contributor call and in general
- mosabua to plan for faster follow up Trino Gateway 8 release and higher cadence going forward
6am PST
Attendees
mosabua, willmostly, vishalya, Chaho12, andythsu, oneonestar, [alaturqua])(https://github.com/alaturqua)
Topics, notes, action items, decisions
- Discussed UI PR, decided to potentially merge after final review, probably ready to merge
- Discussed v1/info PR, mosabua to review and potentially merge, also discussed potential changes in Trino itself for that end point and wish for more info from a Trino endpoint such as GC pauses, memory and CPU pressure, nubmer of workers and more, mosabua to talk to others
- Query Id extraction PR is ready, willmostly will add javadoc and merge
- Cookie based routing needs reviews, mosabua will look
- Configurable routing modules - vishalya will update PR to get ready
- mosabua to start doc to assemble ideas for cluster health data
- oneonestar lead discussion around support for certificate auth, mosabua and others to chime in on issue
- future feature "query retry" needs design doc or ticket and more, needs to be initiated
- sql rewriting comes up as feature request, potentially good to expose SPI for others to add whatever rewriting they need, mosabua to start issue for gathering use cases and more
- Topic of preprocessing of query for better routing rules was discussed, definitely interesting future feature, potentially share parse tree
- Topic of result caching with redis and/or object storage as cache was discussed, mosabua and willmostly to continue discussion with other
6am PST
Attendees
mosabua, willmostly, dprophet, vishalya, Chaho12, sbernauer, rdsarvar
Notes, action items, decisions
- Trino Gateway 6 is out, including first docker container
- Congrats to our new subproject maintainers willmostly, vishalya, and Chaho12
- Report from PoC with MkDocs
- Chat about about Trino healthcheck REST endpoind, mosabua to follow up with Trino maintainers
- Decided to put focus on pluggable policies and the PR from vishalya before we implement further routing policies
- New UI testing went well as reported by Chaho12 and vishalya, decided to work with contributor to get to merge, authentication will have to become optional
- Discussed impact on Oracle support (and in general) of the refactor from ActiveJDBC to JDBI, will continue refactor and figure out specific solution
- State management for multipole instances - to be determined but probably via database, maybe add Redis as supported backend, no current task
- Discussion about automatic retry of queries in the future, will need storage of submitted queries and state, and more, will work on that at a later stage
- Trino LB demo and lots of good discussion with sbernauer, for example scripted polices via some JVM scripting support
6am PST
Proposed topics
- Documentation setup (MkDocs, Docusaurus, ..?)
- PR reviews and status
- Docker setup and container image
- Next release
Attendees
mosabua, willmostly, dprophet, andythsu, vishalya, Chaho12, sbernauer, soenkeliebau, rdsarvar
Notes, action items, decisions
- Round of introductions from everyone including current status/tasks
- Discussion of current ongoing PRs, various reviews and rebases are in progress, we will try to get as many done as possible before we cut new release
- Java 21 discussion - decided to cut at least version 6 and 7 before we go to Java 21, no current benefit to move
- New UI - decided to ensure testing of the new UI is done by a number of contributors before we merge
- Trino Gateway Healthcheck - filed issue, we should add Dropwizard standard healthcheck and then use it for container start
- Trino healthcheck and cluster status - discussed approach since a healthy cluster must have at least one worker and the current REST endpoints dont show that info, mosabua to talk with Trino maintainers about potentially adding another public endpoint or expand /v1/status or /v1/info with worker count or so, for now we will potentially have to use UI endpoint (and need to figure out authentication with that) or use JDBC or something else
- Plan for new release after some more merges, definitely before next sync
6am PST
Proposed topics
- Recent code changes towards version 5 release
- Onboarding new maintainers
- New UI
- Docker container work
- Java 17 or migrating to Java 21
- Trino cluster healthcheck design
- Improve Trino JDBC driver usage
- Dependency cleanup
Attendees
mosabua, willmostly, dprophet, andythsu, vishalya, Chaho12, nil1729, sajjoseph
Notes, action items, decisions
- Round of introductions from everyone including current status/tasks
- Discussed plans towards next releases
- Reached agreement to merge OAuth2 PR and then cut release 5
- Docker, new UI, Oracle and others all after the release
- Onboarding for subproject maintainers in progress
- Agreed to move to Java 21 soon, need to verify dependencies and functionality, and update deps
- Discussed plans for docs to move to sphinx and github pages over time
- Agreed to look at new UI and move towards it after review and some testing
- Agreed to work towards pluggable router policies
- Did not find time to talk about cluster healthcheck design and distributed healthchecks
- Agreed to use github issues for roadmapping and discussion about design for new features
- Agreed to have separate meetings and discussion for specific topics as needed
- Agreed to use trino-gateway-dev channel for communication
- Agreed that meeting time and schedule every two weeks is good
- Project will work towards smarter routing with explain plan/cost, client tags and other approaches, specific cluster only routing for oauth
- Discussed need for metadata caching to tackle traffic very noisy BI tools, considering this to also be in scope of the project
- Discussed deployment modes with cold-standby vs multiple stateless nodes and will work on details more
Prior events include various calls and meeting in the creation and start of the project.