Skip to content

Commit

Permalink
Merge pull request #1160 from mike-wade/feature/1097-cosmos-db-gremli…
Browse files Browse the repository at this point in the history 
…n-support

#1097 - Gremlin graph support to CosmosDb
  • Loading branch information
@ninjarobot
ninjarobot authored Dec 6, 2024
2 parents 77273c7 + e2b51e0 commit e0b126a
Show file tree
Hide file tree
Showing 5 changed files with 193 additions and 63 deletions.
3 changes: 3 additions & 0 deletions RELEASE_NOTES.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ Release Notes
## 1.9.7
- AKS Cluster: support for Sku and Tier. Support for pod subnet in agent pool config. Support for node pool autoscaling

## 1.9.7
* CosmosDB: Add support for Gremlin Graphs

## 1.9.6
- Network Interface: Support for adding Network Security Group (NSG) to Network Interface (NIC)

Expand Down
44 changes: 25 additions & 19 deletions docs/content/api-overview/resources/cosmos-db.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,35 +12,39 @@ The CosmosDb package containers two builders, used to create *databases* and *co
* CosmosDB SQL (`Microsoft.DocumentDB/databaseAccounts/sqlDatabases`)
* CosmosDB MongoDB (`Microsoft.DocumentDB/databaseAccounts/mongodbDatabases`)
* CosmosDB SQL Container (`Microsoft.DocumentDb/databaseAccounts/sqlDatabases/containers`)
* CosmosDB Graph databases (`Microsoft.DocumentDb/databaseAccounts/gremlinDatabases`)
* CosmosDB Graph containers (`Microsoft.DocumentDb/databaseAccounts/gremlinDatabases/graphs`)

> There is currently only support for document databases (the so-called "SQL API"), with support for Gremlin, Table and Cassandra data models planned.
> There is currently support for document databases (the so-called "SQL API") and Gremlin graphs. Support for Table and Cassandra data models planned.
#### Cosmos DB Builder
The CosmosDB builder abstracts the idea of an account and database into one. If you wish to "re-use" an already-created Cosmos DB account, use the `link_to_account` keyword - no account will be created and the database will be attached to the existing one.

| Applies To | Keyword | Purpose |
|-|-|-|
| Database | name | Sets the name of the database. |
| Database | link_to_account | Instructs Farmer to link this database to an existing Cosmos DB account rather than creating a new one. |
| Database | throughput | Sets the throughput with either "provisioned throughput" or "serverless". |
| Database | add_containers | Adds a list of containers to the database. |
| Account | account_name | Sets the name of the CosmosDB account. |
| Account | api (not yet implemented) | Sets the API and data model to use -- currently defaults to "Core (SQL)". |
| Account | enable_public_network_access | Enables public network access for the account. |
| Applies To | Keyword | Purpose |
|-|-------------------------------|-|
| Database | name | Sets the name of the database. |
| Database | link_to_account | Instructs Farmer to link this database to an existing Cosmos DB account rather than creating a new one. |
| Database | throughput | Sets the throughput with either "provisioned throughput" or "serverless". |
| Database | add_containers | Adds a list of containers to the database. |
| Account | account_name | Sets the name of the CosmosDB account. |
| Account | kind | Sets the API and data model to use -- currently defaults to "Core (SQL)". |
| Account | enable_public_network_access | Enables public network access for the account. |
| Account | disable_public_network_access | Disables public network access for the account. |
| Account | consistency_policy | Sets the consistency policy of the database. |
| Account | failover_policy | Sets the failover policy of the database. |
| Account | free_tier | Registers this server with the free pricing tier, if supported and allowed by Azure. |
| Account | consistency_policy | Sets the consistency policy of the database. |
| Account | failover_policy | Sets the failover policy of the database. |
| Account | free_tier | Registers this server with the free pricing tier, if supported and allowed by Azure. |

#### Cosmos Container Builder
The container builder allows you to create and configure a specific container that is attached to a cosmos database.

| Keyword | Purpose |
|-|-|
| name | Sets the name of the container. |
| partition_key | Sets the partition key of the container. |
| add_index | Adds an index to the container. |
| exclude_path | Excludes a path from the container index. |
| Keyword | Purpose |
|-|--------------------------------------------------------------------------------------------|
| name | Sets the name of the container. |
| partition_key | Sets the partition key of the container. |
| add_index | Adds an index to the container. |
| exclude_path | Excludes a path from the container index. |
| gremlin_graph | Marks the container as graph (must be used with an account of `kind DatabaseKind.Gremlin`) |


#### Example
```fsharp
Expand All @@ -53,12 +57,14 @@ let myCosmosDb = cosmosDb {
throughput 400<CosmosDb.RU> // or throughput Serverless
failover_policy CosmosDb.NoFailover
consistency_policy (CosmosDb.BoundedStaleness(500, 1000))
//kind DatabaseKind.Gremlin //Create a gremlin enabled account
add_containers [
cosmosContainer {
name "myContainer"
partition_key [ "/id" ] CosmosDb.Hash
add_index "/path" [ CosmosDb.Number, CosmosDb.Hash ]
exclude_path "/excluded/*"
//gremlin_graph //Mark this container to be a graph
}
]
}
Expand Down
117 changes: 75 additions & 42 deletions src/Farmer/Arm/DocumentDb.fs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,24 @@ let mongoDatabases =
let databaseAccounts =
ResourceType("Microsoft.DocumentDb/databaseAccounts", "2021-04-15")

let gremlinDatabases =
ResourceType("Microsoft.DocumentDb/databaseAccounts/gremlinDatabases", "2022-05-15")

let graphs =
ResourceType("Microsoft.DocumentDb/databaseAccounts/gremlinDatabases/graphs", "2022-05-15")

type DatabaseKind =
| Document
| Mongo
| Gremlin

module DatabaseAccounts =
module SqlDatabases =
module Containers =
type Container = {
Name: ResourceName
Account: ResourceName
Database: ResourceName
Kind: DatabaseKind
PartitionKey: {|
Paths: string list
Kind: IndexKind
Expand All @@ -45,44 +53,56 @@ module DatabaseAccounts =

interface IArmResource with
member this.ResourceId =
containers.resourceId (this.Account / this.Database / this.Name)
match this.Kind with
| Gremlin -> graphs.resourceId (this.Account / this.Database / this.Name)
| _ -> containers.resourceId (this.Account / this.Database / this.Name)

member this.JsonModel = {|
containers.Create(
this.Account / this.Database / this.Name,
dependsOn = [ sqlDatabases.resourceId (this.Account, this.Database) ]
) with
properties = {|
resource = {|
id = this.Name.Value
partitionKey = {|
paths = this.PartitionKey.Paths
kind = string this.PartitionKey.Kind
|}
uniqueKeyPolicy = {|
uniqueKeys =
this.UniqueKeyPolicy.UniqueKeys |> Set.map (fun k -> {| paths = k.Paths |})
|}
indexingPolicy = {|
indexingMode = "consistent"
includedPaths =
this.IndexingPolicy.IncludedPaths
|> List.map (fun p -> {|
path = p.Path
indexes =
p.Indexes
|> List.map (fun (dataType, kind) -> {|
kind = string kind
dataType = dataType.ToString().ToLower()
precision = -1
|})
|})
excludedPaths =
this.IndexingPolicy.ExcludedPaths |> List.map (fun p -> {| path = p |})
member this.JsonModel =
let resourceType =
match this.Kind with
| Gremlin -> graphs
| _ -> containers

{|
resourceType.Create(
this.Account / this.Database / this.Name,
dependsOn = [
match this.Kind with
| Gremlin -> gremlinDatabases.resourceId (this.Account, this.Database)
| _ -> sqlDatabases.resourceId (this.Account, this.Database)
]
) with
properties = {|
resource = {|
id = this.Name.Value
partitionKey = {|
paths = this.PartitionKey.Paths
kind = string this.PartitionKey.Kind
|}
uniqueKeyPolicy = {|
uniqueKeys =
this.UniqueKeyPolicy.UniqueKeys |> Set.map (fun k -> {| paths = k.Paths |})
|}
indexingPolicy = {|
indexingMode = "consistent"
includedPaths =
this.IndexingPolicy.IncludedPaths
|> List.map (fun p -> {|
path = p.Path
indexes =
p.Indexes
|> List.map (fun (dataType, kind) -> {|
kind = string kind
dataType = dataType.ToString().ToLower()
precision = -1
|})
|})
excludedPaths =
this.IndexingPolicy.ExcludedPaths |> List.map (fun p -> {| path = p |})
|}
|}
|}
|}
|}
|}

type SqlDatabase = {
Name: ResourceName
Expand All @@ -92,13 +112,17 @@ module DatabaseAccounts =
} with

interface IArmResource with
member this.ResourceId = sqlDatabases.resourceId (this.Account / this.Name)
member this.ResourceId =
match this.Kind with
| Gremlin -> gremlinDatabases.resourceId (this.Account / this.Name)
| _ -> sqlDatabases.resourceId (this.Account / this.Name)

member this.JsonModel =
let resource =
match this.Kind with
| Document -> sqlDatabases
| Mongo -> mongoDatabases
| Gremlin -> gremlinDatabases

{|
resource.Create(this.Account / this.Name, dependsOn = [ databaseAccounts.resourceId this.Account ]) with
Expand Down Expand Up @@ -176,6 +200,7 @@ type DatabaseAccount = {
match this.Kind with
| Document -> "GlobalDocumentDB"
| Mongo -> "MongoDB"
| Gremlin -> "GlobalDocumentDB"
properties =
{|
consistencyPolicy = {|
Expand All @@ -193,22 +218,30 @@ type DatabaseAccount = {
enableAutomaticFailover = this.EnableAutomaticFailover |> Option.toNullable
enableMultipleWriteLocations = this.EnableMultipleWriteLocations |> Option.toNullable
locations =
match this.FailoverLocations, this.Serverless with
| [], Enabled ->
// Locations has to be specified for the account to be gremlin enabled.
// Otherwise graph database provisioning fails.
match this.FailoverLocations, (this.Serverless = Enabled || this.Kind = Gremlin) with
| [], true ->
box [
{|
locationName = this.Location.ArmValue
|}
]
| [], Disabled -> null
| [], false -> null
| locations, _ -> box locations
publicNetworkAccess = string this.PublicNetworkAccess
enableFreeTier = this.FreeTier
capabilities =
if this.Serverless = Enabled then
box [ {| name = "EnableServerless" |} ]
if this.Serverless = Enabled || this.Kind = Gremlin then
box [
if this.Serverless = Enabled then
{| name = "EnableServerless" |}
if this.Kind = Gremlin then
{| name = "EnableGremlin" |}
]
else
null

|}
|> box
|}
38 changes: 36 additions & 2 deletions src/Farmer/Builders/Builders.Cosmos.fs
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
[<AutoOpen>]
module Farmer.Builders.CosmosDb

open System
open Farmer
open Farmer.Arm
open Farmer.CosmosDb
open Farmer.Arm.DocumentDb
open DatabaseAccounts
open SqlDatabases
open Containers

type KeyType =
| PrimaryKey
Expand Down Expand Up @@ -62,6 +63,7 @@ type CosmosDbContainerConfig = {
Indexes: (string * (IndexDataType * IndexKind) list) list
UniqueKeys: Set<string list>
ExcludedPaths: string list
Kind: DatabaseKind
}

type CosmosDbConfig = {
Expand Down Expand Up @@ -138,6 +140,7 @@ type CosmosDbConfig = {
Name = container.Name
Account = this.AccountResourceId.Name
Database = this.DbName
Kind = container.Kind
PartitionKey = {|
Paths = fst container.PartitionKey
Kind = snd container.PartitionKey
Expand All @@ -160,6 +163,7 @@ type CosmosDbConfig = {
type CosmosDbContainerBuilder() =
member _.Yield _ = {
Name = ResourceName ""
Kind = DatabaseKind.Document
PartitionKey = [], Hash
Indexes = []
UniqueKeys = Set.empty
Expand All @@ -186,6 +190,13 @@ type CosmosDbContainerBuilder() =
[<CustomOperation "name">]
member _.Name(state: CosmosDbContainerConfig, name) = { state with Name = ResourceName name }

/// Sets the container kind of the container.
[<CustomOperation "gremlin_graph">]
member _.Graph(state: CosmosDbContainerConfig) = {
state with
Kind = DatabaseKind.Gremlin
}

/// Sets the partition key of the container.
[<CustomOperation "partition_key">]
member _.PartitionKey(state: CosmosDbContainerConfig, partitions, indexKind) = {
Expand Down Expand Up @@ -239,6 +250,29 @@ type CosmosDbBuilder() =
Kind = DatabaseKind.Document
}

static member ValidateContainers(state: CosmosDbConfig) =
let validateContainerAndAccountConfig (container: CosmosDbContainerConfig, accountKind: DatabaseKind) =
if container.Kind = accountKind then
Ok container
else
Error $"Container {container.Name.Value} must be of {state.Kind} kind"

state.Containers
|> List.map (fun container -> validateContainerAndAccountConfig (container, state.Kind))

member _.Run state =
let errors =
CosmosDbBuilder.ValidateContainers(state)
|> List.choose (fun r ->
match r with
| Error e -> Some(e)
| Ok _ -> None)

if errors.Length > 0 then
errors |> String.concat Environment.NewLine |> raiseFarmer

state

/// Sets the name of the CosmosDB server.
[<CustomOperation "account_name">]
member _.AccountName(state: CosmosDbConfig, accountName: ResourceName) = {
Expand Down
Loading

0 comments on commit e0b126a

Please sign in to comment.