How to stop users from creating non-indexed metadata

[marianne013]

Hi,

We ran into the problem that users who try to use metadata will create non-indexed metadata fields, typically by using e.g.
meta set their_file their_key their_value
without realising that in order to make it searchable they would have to do:
meta index -f their_key keytype first (similarly for the API).
Ideally we'd like to be able to prevent this from happening by blocking the creation of non-indexed fields, as they do not behave in the way our users expect and we don't like surprised users.
Is there an option for this ?
If not, what would be the best strategy to code it ?

Daniela

[andresailer]

Hi Daniela,
You can restrict the use of the FileCatalog Service function "setMetadata" (which is what meta set calls) to the admin group, or whomever.
"meta index" corresponds to addMetadataField
E.g. I did this for "reasons" (before reasons happened)
(path like, Systems/DataManagement/Production/Services/FileCatalog/, you know)

          Authorization
          {
            Default = authorized
            deleteMetadataField = FileCatalogManagement
            addMetadataField = FileCatalogManagement
          }

[marianne013]

Hi Andre,

There were two people who got that wrong:
a) The production manager of a VO (who should have been able to make create these indices), but didn't know about having to create the index first
b) me, while trying to type up an example and making a typo (creating an index called "testcases" and then assigning metadata called "testcase" to some files).
So I would really like to able able to turn this "feature" (of unindexed metadata) off (or remove it)
Does anyone actually use non-indexed metadata ?

Daniela

[andresailer]

This doesn't work to make it not work?

          Authorization
          {
            Default = authorized
            deleteMetadataField = AbsolutelyPositivelyNoone
            addMetadataField = AbsolutelyPositivelyNoone
          }

Yes we use non-indexed metadata

[marianne013]

Restricting the group of users who is allowed to add or delete metadata is one of the things we want to do. But here what we really want to do is something like this:
i.e. not allowing the creation of unindexed metadata at all (possibly on a VO by VO basis)
We reckon something along those lines would probably do it:
851f9d3

Does this look sensible to you ?

[andresailer]

This doesn't work ?

          Authorization
          {
            Default = authorized
            deleteMetadataField = vo:vo_that_knows_how_to_deal_with_non_indexed_meta_data
            addMetadataField = vo:vo_that_knows_how_to_deal_with_non_indexed_meta_data
          }

(https://dirac.readthedocs.io/en/latest/AdministratorGuide/Configuration/generalConfigOptions.html)
Though, to be fair, I don't know how it is interpreted, if a second VO is in hat list (@atsareg ???)

[marianne013]

@andresailer Different problem: As far as @sfayer and I can tell, your suggestion doesn't prevent the creation of non-indexed metadata.

[andresailer]

What exactly did you do?

[andresailer]

Pretty sure what I originally proposed (#6029 (comment)) will actually prevent the setting of any metadata indexed or not T_T

[sfayer]

Hi,

I thought that the client call sequence for creating non-indexed metadata was just "setMetadata" with a new field name that has never been indexed before (and that addMetadataField isn't used at all in that case as that's the call specifically for creating the index). Have I misunderstood?

Regards,
Simon

[andresailer]

yeah, and if you prevent setMetadata from being used (which can be done via Authorization, tested on lbcertifdirac70), then also indexed meta data cannot be set, you can create an index, but not set any values for this key either.

[marianne013]

I've turned that commit into a pull request (#6113). As it preserves the current default of allowing both indexed and unindexed metadata, it should hopefully be uncontroversial, but we'd really like our database to be safe before the mess gets to be too big to be fixed easily :-)