feat: Add application to schema and API

[markstory]

To better support multiple client applications using a single taskbroker instance, we need to introduce the concept of application. An application represents a logical client or consumer. Applications contain many namespaces, and namespaces can be duplicated across applications.

Currently, clients can consume from all namespaces, or a single namespace. There isn't a way for a worker to request tasks from multiple namespaces, and more importantly there isn't a way to exclude namespaces. By adding application to activations & taskbroker, we'll be able to have clients do any of the following:

1. Request tasks from any namespace for a specific application.
2. Request tasks from a specific namespace + application.
3. Request tasks from a given namespace across any application. This is no longer allowed.
4. Request tasks from any application and any namespace.

Scenario 4 will be deprecated in the future with runtime enforcement. Once sentry is updated, this scenario will return errors/no-results as well. Changing the protobufs is a 'breaking change', so requiring application will be a runtime constraint until we can revise protobufs.

Refs STREAM-603

[linear]

STREAM-603 Update storage schema to contain application

[evanh]

LGTM.

[fpacifici]

Applications contain many namespaces, and namespaces can be duplicated across applications.

When we say that namespaces can be duplicated across applications does it mean that there are going to be two independent namespaces with the same name in two applications or that the namespace is shared and messages for a namespace can be consumed by either of the two applications ?

Let's say a producer issues a task for namespace ingest-transaction. The application is sentry. Does the producer specify the application or is it only sentry that can produce with the sentry application ? Let's say I have to workers now from two different applications that consume the ingest-transaction namespace, only the sentry one would receive the task. In such case what would a valid use case be to share the ingest-transactions namespace ?

Request tasks from a given namespace across any application.

Can we just disallow this from the beginning ?

[fpacifici]

Who decides which application a task belong when issuing the task itself?

• is a task definition statically defined as part of an application ? So the application is automatically assigned when a task is issued ?
• Can the client pick the application ?
• is the client automatically part of an application ? (so no cross application tasks)

[markstory]

When we say that namespaces can be duplicated across applications does it mean that there are going to be two independent namespaces with the same name in two applications or that the namespace is shared and messages for a namespace can be consumed by either of the two applications ?

A namespace may be duplicated by multiple applications (as they are just strings at the end of the day). My plan is to end up in a state where a namespace value could be used by multiple applications, but each copy of the namespace is consumed by separate workers in separate applications.

Let's say a producer issues a task for namespace ingest-transaction. The application is sentry. Does the producer specify the application or is it only sentry that can produce with the sentry application ? Let's say I have to workers now from two different applications that consume the ingest-transaction namespace, only the sentry one would receive the task. In such case what would a valid use case be to share the ingest-transactions namespace ?

The producer of an activation is responsible for choosing the right namespace and application values.

In such case what would a valid use case be to share the ingest-transactions namespace ?

I can't think of any scenarios where we would intentionally create overlapping namespace names in different applications, but it could happen.

Can we just disallow this from the beginning ?

Yes. I can make requests that specify namespace & omit application invalid. We'll need to support the scenario where a worker provides neither an application or namespace for a short period of time to handle deploy boundaries between sentry and taskbroker. Once sentry has been updated to produce application in its activations, my plan is to make application required for all activations that are ingested and worker requests.

[markstory]

Who decides which application a task belong when issuing the task itself?

• is a task definition statically defined as part of an application ? So the application is automatically assigned when a task is issued ?
• Can the client pick the application ?
• is the client automatically part of an application ? (so no cross application tasks)

The producer will have the final say on which application a task belongs. My plan is to have the application defined statically in the application code where namespaces are defined, and tasks will inherit config from the namespace.

... is the client automatically part of an application ? (so no cross application tasks)

Yes, this is where I'd like to end up. Clients have to define their application label statically, and only in scenarios where we have cross-application messaging would a producer use a different application value.