Open Metadata and Governance Standards
The Data Governance PMC is working to increase adoption in data tools and technology that supports the open metadata and governance standards. These standards provide APIs and message exchange protocols to allow tools from different vendors to share metadata. The definition of the open APIs and event formats, along with an embeddable reference implementation of the associated protocols is available under the Apache 2.0 license from the Apache Atlas project. The implementation in Apache Atlas is organized as shown in Figure 1:
Figure 1: Structure of Open Metadata Implementation in the Apache Atlas project
At the center of the standards are the REST APIs and event formats (payload structures) used to manage, exchange and query metadata. A vendor may choose to take these standards and create their own implementation of the interfaces and protocols. However, this is not recommended. Past experience has shown that each implementation of an API or protocol has subtle differences that create integration errors. There is also a significant cost to a vendor to implement the API/protocol. In open metadata we are looking to minimize cost and speed up time to market for vendors. Therefore, Apache Atlas provides embeddable libraries that implement the protocols and frameworks to help a vendor rapidly implement the standards. These are shown in yellow in figure 1.
Next there is a server (called the OMAG Server) that includes the implementation of the open metadata APIs, protocols and frameworks and an administration interface to configure the open metadata capabilities in the embeddable libraries. This server does not have an ability to store metadata. It can be used by a vendor to extend, or create a new metadata server. Finally, the Apache Atlas project offers the Atlas Server. This is a server with all of the open metadata function plus a metadata repository and additional features to support management and governance.
The APIs and events for open metadata support low level integration between metadata repositories (called the open metadata repository services) and higher level interfaces and events for data tools/engines/platforms to integrate with. The higher level services are called the open metadata access services. Again the higher level access services interfaces aim to minimize the effort needed for vendors to integrate into the open metadata ecosystem.
The use and capability of the higher level interfaces is demonstrated by the open metadata UI. This UI may be used by an organization to manage metadata, but it is expected that vendors will provide more functional UIs on top of the access services. Figure 2 shows data tools, repositories and the UI connecting into the open metadata APIs and event formats.
Figure 2: Connecting to open metadata
Future releases of Apache Atlas will add tow new open metadata and governance services along with their corresponding standards and frameworks. Figure 3 shows these servers connecting into the open metadata APIs and events.
Figure 3: Adding the discovery and stewardship servers