Being more restrictive on Arctic Library Names?

[yschimke]

For discussion, should we allow things like quotes in Arctic library names?

[richardbounds]

Ha ha ha ha. No. 😀

[yschimke]

I'm asking because we had an internal bug that seemed to be as a result of this happening, so glad you agree :)

[dimosped]

Do we have any proposals for naming when using initialize_library?
e.g.

[A-Za-z0-9_]+

[richardbounds]

That sounds fair to me. Maybe worth checking whether it would break anything.

[dimosped]

@bmoscon @yschimke @richardbounds @jamesblackburn
It could be then:

'^([A-Za-z0-9_]+)(\.[A-Za-z0-9_]+)?$'

For new libraries don't allow:

• more than one '.'
• lib name ending with '.'
• '-'
• quotes (single/double)
• spaces
• any other special character

For the number of '.' in the library name, while mongoDB perfectly allows NS coll name nesting, bear in mind that:

In[4]: arctic.arctic.ArcticLibraryBinding._parse_db_lib('abbbb.cccc')
Out[4]: ('arctic_abbbb', 'cccc')
In[5]: arctic.arctic.ArcticLibraryBinding._parse_db_lib('abbbb')
Out[5]: ('arctic', 'abbbb')
In[6]: arctic.arctic.ArcticLibraryBinding._parse_db_lib('abbbb.sdfsdf.asdfsdf')
Out[6]: ('arctic', 'abbbb.sdfsdf.asdfsdf')

[micheartin]

I have spent some hours to figure out that the dots . within my library name was the origin of the following exception:

arctic.exceptions.LibraryNotFoundException: Library arctic.MARKET.T.DIG was not correctly initialized in <Arctic at 0x7f8bd6492e10, connected to MongoClient(host=['127.0.0.1:27017'], ...

[rob256]

@micheartin How did you initialise your library? And how are you attempting to access it?
It looks like your library name would have been arctic.MARKET.T.DIG?

[micheartin]

Hi @rob256

Yes the final MongoDB library name was arctic.MARKET.T.DIG and currently is arctic.MARKET-T-DIG. FYI, in my current code (simplified extract below), I have replaced dots . by dashes -.

Sorry if my previous post was short and unclear. I just wanted to say that the LibraryNotFoundException exception text was missing some explanation in order to help the user to fix the issue. If I have the time I may send a PR...

I do not yet know if my below simplified snippet is correct because I am learning Arctic and I have some issues when storing an order book (ValueError: Column of type 'mixed' cannot be converted to string). Moreover, I am not sure I am using pandas.DataFrame in the best way...

class Library:
    def __init__(self, name, lib):
        self.name = name
        self.lib = lib
    def write(self, ns, **kwargs)
        df = pd.DataFrame({key: [value] for key, value in kwargs.items()})
        df['date'] = pd.to_datetime(ns, unit="ns")
        df.set_index("date", inplace=True)
        df.index = df.index.tz_localize(None)  # timestamp in UTC
        self.lib.append(pair, df, upsert=True, chunk_size="T")  # once per minute

class LibraryBuilder:
    def __init__(self, host="127.0.0.1", lib_type=arctic.CHUNK_STORE):
        self.con = arctic.Arctic(host)
        self.lib_type = lib_type
    def get_library(self, market, key) -> Library:
        name = f"{market}-{key}"
        lib_name = f"{name}-DIG"
        if lib_name not in self.con.list_libraries():
            self.con.initialize_library(lib_name, lib_type=self.lib_type)
        return Library(name, self.con[lib_name])