Add endpoints for just updating specific portions of entityInfo

[alechenninger]

It's a bit stressful updating entityInfo because if anything is messed up it can be fatal, and often you just want to update one small thing, like hook configuration.

It'd be great if you could just hit an endpoint like PUT /metadata/user/hooks/notificationHook/ with notification hook configuration JSON.

[alechenninger]

@bserdar @jewzaam Would you be opposed to something like this? I'd be willing to work on a pull request this week if you don't see any issues. It wouldn't be comprehensive for all of entityInfo, but would set a pattern for other bits of entityInfo to follow.

Specifically I would like to implement something like PUT /metadata/{entityName]/entityInfo/hooks/{hookName} as this would help simplify our release process and reduce barrier to entry for folks still new to lightblue metadata management. We could also more simply automate hook config updates potentially (we already produce the config json on build, but actually getting this into lightblue is still a manual, copy+paste, error-prone endeavor).

In the future, you could imagine additions like PUT ../entityInfo/indexes or defaultVersion, etc.

It could be implemented purely in lightblue-rest as sugar for getEntityInfo -> updateEntityInfo where only relevant bit is replaced. Or, we could add additional API's to Metadata in core, implement them in mongo, and expose them in rest. This is more complicated, but could make the operations atomic, so that an update to >1 piece of entity info at the same time wouldn't risk losing one or more updates to unrelated pieces. That being said, I don't think that would be all that hard given the relative simplicity of metadata operations.

What do you guys think?

[bserdar]

+1 for implementing this in lightblue rest. Adding functionality to core
would be difficult, as it does not specify how actually the metadata is
stored and managed.

On Sat, Jun 11, 2016 at 8:24 AM, Alec Henninger [email protected]
wrote:

@bserdar https://github.com/bserdar @jewzaam
https://github.com/jewzaam Would you be opposed to something like this?
I'd be willing to work on a pull request this week if you don't see any
issues. It wouldn't be comprehensive for all of entityInfo, but would set a
pattern for other bits of entityInfo to follow.

Specifically I would like to implement something like PUT
/metadata/{entityName]/entityInfo/hooks/{hookName} as this would help
simplify our release process and reduce barrier to entry for folks still
new to lightblue metadata management. We could also more simply automate
hook config updates potentially (we already produce the config json on
build
https://github.com/esbtools/lightblue-notification-hook/tree/master/config-generation,
but actually getting this into lightblue is still a manual, copy+paste,
error-prone endeavor).

In the future, you could imagine additions like PUT ../entityInfo/indexes
or defaultVersion, etc.

It could be implemented purely in lightblue-rest as sugar for
getEntityInfo -> updateEntityInfo where only relevant bit is replaced. Or,
we could add additional API's to Metadata in core, implement them in mongo,
and expose them in rest. This is more complicated, but could make the
operations atomic, so that an update to >1 piece of entity info at the same
time wouldn't risk losing one or more updates to unrelated pieces. That
being said, I don't think that would be all that hard given the relative
simplicity of metadata operations.

What do you guys think?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#235 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADgDDcglHYwEgk8ivCWA1jmS1pLNdgO4ks5qKsU1gaJpZM4Igt6T
.

[alechenninger]

as it does not specify how actually the metadata is stored and managed.

Naively, couldn't we add abstract methods such as updateHookConfig(String, HookConfiguration) and let lightblue-mongo figure out how to store it?

I guess core would have to be able to parse the hook configuration, but I think it can do this?

I'm mostly curious, I don't have an issue with implementing it only in lightblue-rest.

[bserdar]

It will be defined in core, but won't be used there. It will be like
defining a function needed and used in one layer somewhere else just
because there happens to be an interface where it fits.

Using the same logic., it doesn't make sense to have createMetadata,
createSchema, etc. in core as well, and we should remove them from metadata
interface. We can define a separate metadata management interface, and put
these functions there. It could be defined in lightblue-mongo.

On Mon, Jun 13, 2016 at 6:23 AM, Alec Henninger [email protected]
wrote:

as it does not specify how actually the metadata is stored and managed.

Naively, couldn't we add abstract methods such as updateHookConfig(String,
HookConfiguration) and let lightblue-mongo figure out how to store it?

I guess core would have to be able to parse the hook configuration, but I
think it can do this?

I'm mostly curious, I don't have an issue with implementing it only in
lightblue-rest.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#235 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADgDDSgJu5o1diDG-8UYmQ208J3NgMzDks5qLUvfgaJpZM4Igt6T
.

[alechenninger]

But putting that in core decouples you from the backend right? I guess I
understood the Metadata interface is like a Mediator but for crud on
metadata instead of crud on entities. It abstracts the backend.

Again, just exploring this with you :). I'd like to understand the bits and
pieces better.

On Mon, Jun 13, 2016, 8:52 AM Burak Serdar [email protected] wrote:

It will be defined in core, but won't be used there. It will be like
defining a function needed and used in one layer somewhere else just
because there happens to be an interface where it fits.

Using the same logic., it doesn't make sense to have createMetadata,
createSchema, etc. in core as well, and we should remove them from metadata
interface. We can define a separate metadata management interface, and put
these functions there. It could be defined in lightblue-mongo.

On Mon, Jun 13, 2016 at 6:23 AM, Alec Henninger [email protected]
wrote:

as it does not specify how actually the metadata is stored and managed.

Naively, couldn't we add abstract methods such as
updateHookConfig(String,
HookConfiguration) and let lightblue-mongo figure out how to store it?

I guess core would have to be able to parse the hook configuration, but I
think it can do this?

I'm mostly curious, I don't have an issue with implementing it only in
lightblue-rest.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
#235 (comment)
,
or mute the thread
<
https://github.com/notifications/unsubscribe/ADgDDSgJu5o1diDG-8UYmQ208J3NgMzDks5qLUvfgaJpZM4Igt6T

.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#235 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADVaby2XOhPSDtXoqfLqxMmXOZ4X3-y3ks5qLVJ4gaJpZM4Igt6T
.

[bserdar]

I don't think putting that in core decouples front-end and back-end. I
think it couples them to core unnecessarily.

core is at the bottom of the stack, and has a very limited view of the
world. It mainly deals with crud orchestration. All it needs in terms of
metadata is to retrieve it in a structure it can interpret.The actual
metadata management is done in mongo, and that's only one of the possible
metadata implementations. Rest layer has to know the capabilities of the
implementation so it can expose the apis.

So ideally, all metadata management operations should be done in mongo, and
exposed there, and then the rest layer can downcast that metadata
implementation and expose its apis.

On Wed, Jun 15, 2016 at 5:23 AM, Alec Henninger [email protected]
wrote:

But putting that in core decouples you from the backend right? I guess I
understood the Metadata interface is like a Mediator but for crud on
metadata instead of crud on entities. It abstracts the backend.

Again, just exploring this with you :). I'd like to understand the bits and
pieces better.

On Mon, Jun 13, 2016, 8:52 AM Burak Serdar [email protected]
wrote:

It will be defined in core, but won't be used there. It will be like
defining a function needed and used in one layer somewhere else just
because there happens to be an interface where it fits.

Using the same logic., it doesn't make sense to have createMetadata,
createSchema, etc. in core as well, and we should remove them from
metadata
interface. We can define a separate metadata management interface, and
put
these functions there. It could be defined in lightblue-mongo.

On Mon, Jun 13, 2016 at 6:23 AM, Alec Henninger <
[email protected]>
wrote:

as it does not specify how actually the metadata is stored and managed.

Naively, couldn't we add abstract methods such as
updateHookConfig(String,
HookConfiguration) and let lightblue-mongo figure out how to store it?

I guess core would have to be able to parse the hook configuration,
but I
think it can do this?

I'm mostly curious, I don't have an issue with implementing it only in
lightblue-rest.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<

#235 (comment)

,
or mute the thread
<

https://github.com/notifications/unsubscribe/ADgDDSgJu5o1diDG-8UYmQ208J3NgMzDks5qLUvfgaJpZM4Igt6T

.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<
#235 (comment)
,
or mute the thread
<
https://github.com/notifications/unsubscribe/ADVaby2XOhPSDtXoqfLqxMmXOZ4X3-y3ks5qLVJ4gaJpZM4Igt6T

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#235 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADgDDZGmj3OyVOYc8SEYW47_WsMbwI7mks5qL-DLgaJpZM4Igt6T
.

[alechenninger]

Interesting, makes more sense now, thank you. Do you think in the future there would be a good place other than mongo and core to add an interface to abstract working with metadata? (That is, capability we would expect of any metadata implementation, some of which may not be relevant to core)

[bserdar]

Can't say. It is possible to write a completely separate webapp that deals
with metadata maintenance. That eliminates all the need to involve core or
mongo in metadata maintenance. All we really care about is some way to get
it.

On Wed, Jun 15, 2016 at 8:29 AM, Alec Henninger [email protected]
wrote:

Interesting, makes more sense now, thank you. Do you think in the future
there would be a good place other than mongo and core to add an interface
to abstract working with metadata?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#235 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADgDDX9NSN2K2VxgjM7SJq4fGOuiP68kks5qMAw3gaJpZM4Igt6T
.

[bserdar]

Here's a proposal:

Entity info has these components:

• indexes[]
• hooks[]
• enums[]
• datastore

For the arrays, we can add a an add api that gets a json document:
{
indexes: [
...
]
}
The api adds the index/hook/enum to metadata.

A remove api, something like
DELETE .../indexes=[,...]

And a modify api that gets the same json doc as add function.

For datasource, we'd only have a modify api.

[jewzaam]

So the full entityInfo is in the request but use query param to indicate which subset to operate against?

[bserdar]

No, only the relevant part is in the request. Maybe something like a PATCH request containing an object with pieces of entityInfo, and it'll be patched to the stored entityInfo.

[jewzaam]

That makes sense. Could we have a quick spec on what is expected? Sample metadata and how to make a request to operate against an index, assuming it's the same for hooks and enums. How to delete, add, and update.

[bserdar]

Here's a first attempt:

PATCH rest/metadata/entityName

{
datastore: {...},
indexes: [ ... ],
enums: [... ],
hooks: [ ...],
}

If a PATCHed object exists in entityInfo any field included in the patch
replaces the existing value.

indexes/enums/hooks are matched by name. If an array element is not in the
entityInfo, it is added. If it is there, it is PATCHed.

Deleting: Two options:

1. Separate delete API, or
2. {
   indexes: [
   {name: indexName, delete:true}
   ]
   }

On Thu, Jun 23, 2016 at 11:40 AM, Naveen Malik [email protected]
wrote:

That makes sense. Could we have a quick spec on what is expected? Sample
metadata and how to make a request to operate against an index, assuming
it's the same for hooks and enums. How to delete, add, and update.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#235 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADgDDSEYUFJR2U6vTdxOFt5lMGKaFWCNks5qOsT3gaJpZM4Igt6T
.