diff --git a/Directory.Build.props b/Directory.Build.props index bebac5421..5b9b3b1cd 100644 --- a/Directory.Build.props +++ b/Directory.Build.props @@ -20,8 +20,9 @@ false - - $(NoWarn);FS2003;NU5105 + + + $(NoWarn);FS2003;NU5105;FS0988 diff --git a/README.md b/README.md index 3c9fc509f..dbbf241b8 100644 --- a/README.md +++ b/README.md @@ -26,14 +26,14 @@ _If you're looking to learn more about and/or discuss Event Sourcing and it's my - Logging is both high performance and pluggable (using [Serilog](https://github.com/serilog/serilog) to your hosting context (we feed log info to Splunk and the metrics embedded in the LogEvent Properties to Prometheus; see relevant tests for examples) - Extracted from working software; currently used for all data storage within Jet's API gateway and Cart processing. - Significant test coverage for core facilities, and per Storage system. -- **`Equinox.EventStore` Transactionally-consistent Rolling Snapshots**: Command processing can be optimized by employing in-stream 'compaction' events in service of the following ends: +- **`Equinox.EventStore` In-stream Rolling Snapshots**: Command processing can be optimized by employing 'compaction' events in service of the following ends: - no additional roundtrips to the store needed at either the Load or Sync points in the flow - support, (via `UnionContractEncoder`) for the maintenance of multiple co-existing compaction schemas in a given stream (A snapshot isa Event) - compaction events typically do not get deleted (consistent with how EventStore works), although it is safe to do so in concept - NB while this works well, and can deliver excellent performance (especially when allied with the Cache), [it's not a panacea, as noted in this excellent EventStore article on the topic](https://eventstore.org/docs/event-sourcing-basics/rolling-snapshots/index.html) - **`Equinox.Cosmos` 'Tip with Unfolds' schema**: In contrast to `Equinox.EventStore`'s `Access.RollingSnapshots`, when using `Equinox.Cosmos`, optimized command processing is managed via the `Tip`; a document per stream with a well-known identity enabling syncs via point-reads by virtue of the fact that the document maintains: a) the present Position of the stream - i.e. the index at which the next events will be appended - b) (compressed) [_unfolds_]((https://github.com/jet/equinox/wiki/Cosmos-Storage-Model) + b) (compressed) [_unfolds_](https://github.com/jet/equinox/wiki/Cosmos-Storage-Model) c) (optionally) events since those unfolded events ([presently removed](https://github.com/jet/equinox/pull/58), but [should return](https://github.com/jet/equinox/wiki/Roadmap)) This yields many of the benefits of the in-stream Rolling Snapshots approach while reducing latency, RU provisioning requirement, and Request Charges:- @@ -48,21 +48,21 @@ _If you're looking to learn more about and/or discuss Event Sourcing and it's my The Equinox components within this repository are delivered as a series of multi-targeted Nuget packages targeting `net461` (F# 3.1+) and `netstandard2.0` (F# 4.5+) profiles; each of the constituent elements is designed to be easily swappable as dictated by the task at hand. Each of the components can be inlined or customized easily:- - `Equinox.Handler` (Nuget: `Equinox`, depends on `Serilog` (but no specific Serilog sinks, i.e. you can forward to `NLog` etc)): Store-agnostic decision flow runner that manages the optimistic concurrency protocol -- `Equinox.Codec` (Nuget: `Equinox.Codec`, depends on `TypeShape`, (`Newtonsoft.Json` `>= 10.0.3` on `net461`, `>= 11.0.2` on `netstandard2.0` but can support any serializer): [a scheme for the serializing Events modelled as an F# Discriminated Union with the following capabilities](https://eiriktsarpalis.wordpress.com/2018/10/30/a-contract-pattern-for-schemaless-datastores/): +- `Equinox.Codec` (Nuget: `Equinox.Codec`, depends on `TypeShape`, `Newtonsoft.Json` (`>= 10.0.3` on `net461`, `>= 11.0.2` on `netstandard2.0` but can support any serializer): [a scheme for the serializing Events modelled as an F# Discriminated Union with the following capabilities](https://eiriktsarpalis.wordpress.com/2018/10/30/a-contract-pattern-for-schemaless-datastores/): - independent of any specific serializer - - allows tagging of Discriminated Union cases in a versionable manner with low-dependency `DataMember(Name=` tags using [TypeShape](https://github.com/eiriktsarpalis/TypeShape)'s [`UnionContractEncoder`](https://github.com/eiriktsarpalis/TypeShape/blob/master/tests/TypeShape.Tests/UnionContractTests.fs) -- `Equinox.Cosmos` (Nuget: `Equinox.Cosmos`, depends on `System.Runtime.Caching`, `FSharp.Control.AsyncSeq`, `TypeShape`, `Microsoft.Azure.DocumentDb[.Core]`): Production-strength Azure CosmosDb Adapter with integrated transactionally-consistent snapshotting, facilitating optimal read performance in terms of latency and RU costs, instrumented to the degree necessitated by Jet's production monitoring requirements. -- `Equinox.EventStore` (Nuget: `Equinox.EventStore`, depends on `EventStore.Client[Api.NetCore] >= 4`, `System.Runtime.Caching`, `FSharp.Control.AsyncSeq`, `TypeShape`): Production-strength [EventStore](https://eventstore.org/) Adapter instrumented to the degree necessitated by Jet's production monitoring requirements + - allows tagging of F# Discriminated Union cases in a versionable manner with low-dependency `DataMember(Name=` tags using [TypeShape](https://github.com/eiriktsarpalis/TypeShape)'s [`UnionContractEncoder`](https://github.com/eiriktsarpalis/TypeShape/blob/master/tests/TypeShape.Tests/UnionContractTests.fs) +- `Equinox.Tool` (Nuget: `dotnet tool install Equinox.Tool -g`): Tool incorporating a benchmark scenario runner, facilitating running representative load tests composed of transactions in `samples/Store` and `samples/TodoBackend` against any nominated store; this allows perf tuning and measurement in terms of both latency and transaction charge aspects. - `Equinox.MemoryStore` (Nuget: `Equinox.MemoryStore`): In-memory store for integration testing/performance baselining/providing out-of-the-box zero dependency storage for examples. +- `Equinox.EventStore` (Nuget: `Equinox.EventStore`, depends on `EventStore.Client[Api.NetCore] >= 4`, `System.Runtime.Caching`, `FSharp.Control.AsyncSeq`): Production-strength [EventStore](https://eventstore.org/) Adapter instrumented to the degree necessitated by Jet's production monitoring requirements +- `Equinox.Cosmos` (Nuget: `Equinox.Cosmos`, depends on `System.Runtime.Caching`, `FSharp.Control.AsyncSeq`, `Microsoft.Azure.DocumentDb[.Core]`): Production-strength Azure CosmosDb Adapter with integrated 'unfolds' feature, facilitating optimal read performance in terms of latency and RU costs, instrumented to the degree necessitated by Jet's production monitoring requirements. - `samples/Store` (in this repo): Example domain types reflecting examples of how one applies Equinox to a diverse set of stream-based models - `samples/TodoBackend` (in this repo): Standard https://todobackend.com compliant backend -- `Equinox.Tool` (Nuget: `dotnet tool install Equinox.Tool -g`): Tool incorporating a benchmark scenario runner, facilitating running representative load tests composed of transactions in `samples/Store` and `samples/TodoBackend` against any nominated store; this allows perf tuning and measurement in terms of both latency and transaction charge aspects. ## Versioning ## About Versioning -The repo is versioned based on [SemVer 2.0](https://semver.org/spec/v2.0.0.html) using the tiny-but-mighty [MinVer](https://github.com/adamralph/minver) from @adamralph. [See here](https://github.com/adamralph/minver#how-it-works) for more information on how it works. +The repo is versioned based on [SemVer 2.0](https://semver.org/spec/v2.0.0.html) using the tiny-but-mighty [MinVer](https://github.com/adamralph/minver) from [@adamralph](https://github.com/adamralph). [See here](https://github.com/adamralph/minver#how-it-works) for more information on how it works. ## CONTRIBUTING @@ -174,7 +174,6 @@ The CLI can drive the Store and TodoBackend samples in the `samples/Web` ASP.NET eqx run -t saveforlater -f 200 web ### run CosmosDb benchmark (when provisioned) - $env:EQUINOX_COSMOS_CONNECTION="AccountEndpoint=https://....;AccountKey=....=;" $env:EQUINOX_COSMOS_DATABASE="equinox-test" $env:EQUINOX_COSMOS_COLLECTION="equinox-test" diff --git a/build.proj b/build.proj index fd1713fa0..96eeef436 100644 --- a/build.proj +++ b/build.proj @@ -15,11 +15,11 @@ - - - - - + + + + + diff --git a/build.ps1 b/build.ps1 index 49d61f363..54842ad50 100644 --- a/build.ps1 +++ b/build.ps1 @@ -21,8 +21,8 @@ $env:EQUINOX_INTEGRATION_SKIP_EVENTSTORE=[string]$skipEs if ($skipEs) { warn "Skipping EventStore tests" } function cliCosmos($arghs) { - Write-Host "dotnet run cli/Equinox.Cli -- $arghs cosmos -s -d $cosmosDatabase -c $cosmosCollection" - dotnet run -p cli/Equinox.Cli -f netcoreapp2.1 -- @arghs cosmos -s $cosmosServer -d $cosmosDatabase -c $cosmosCollection + Write-Host "dotnet run tools/Equinox.Tool -- $arghs cosmos -s -d $cosmosDatabase -c $cosmosCollection" + dotnet run -p tools/Equinox.Tool -f netcoreapp2.1 -- @arghs cosmos -s $cosmosServer -d $cosmosDatabase -c $cosmosCollection } if ($skipCosmos) { diff --git a/samples/Infrastructure/Storage.fs b/samples/Infrastructure/Storage.fs index 873d829df..cabf2bdf2 100644 --- a/samples/Infrastructure/Storage.fs +++ b/samples/Infrastructure/Storage.fs @@ -9,7 +9,7 @@ type [] MemArguments = interface IArgParserTemplate with member a.Usage = a |> function | VerboseStore -> "Include low level Store logging." -type [] EsArguments = +and [] EsArguments = | [] VerboseStore | [] Timeout of float | [] Retries of int @@ -73,7 +73,7 @@ module EventStore = heartbeatTimeout=heartbeatTimeout, concurrentOperationsLimit = col, log=(if log.IsEnabled(Serilog.Events.LogEventLevel.Debug) then Logger.SerilogVerbose log else Logger.SerilogNormal log), tags=["M", Environment.MachineName; "I", Guid.NewGuid() |> string]) - .Establish("equinox-tool", Discovery.GossipDns dnsQuery, ConnectionStrategy.ClusterTwinPreferSlaveReads) + .Establish("equinox-samples", Discovery.GossipDns dnsQuery, ConnectionStrategy.ClusterTwinPreferSlaveReads) let private createGateway connection batchSize = GesGateway(connection, GesBatchingPolicy(maxBatchSize = batchSize)) let config (log: ILogger, storeLog) (cache, unfolds) (sargs : ParseResults) = let host = sargs.GetResult(Host,"localhost") @@ -89,7 +89,7 @@ module EventStore = let conn = connect storeLog (host, heartbeatTimeout, concurrentOperationsLimit) creds operationThrottling |> Async.RunSynchronously let cacheStrategy = if cache then - let c = Caching.Cache("Cli", sizeMb = 50) + let c = Caching.Cache("equinox-samples", sizeMb = 50) CachingStrategy.SlidingWindow (c, TimeSpan.FromMinutes 20.) |> Some else None StorageConfig.Es ((createGateway conn defaultBatchSize), cacheStrategy, unfolds) @@ -99,11 +99,11 @@ module Cosmos = /// Standing up an Equinox instance is necessary to run for test purposes; You'll need to either: /// 1) replace connection below with a connection string or Uri+Key for an initialized Equinox instance with a database and collection named "equinox-test" - /// 2) Set the 3x environment variables and create a local Equinox using cli/Equinox.cli/bin/Release/net461/Equinox.Cli ` + /// 2) Set the 3x environment variables and create a local Equinox using tools/Equinox.Tool/bin/Release/net461/eqx.exe ` /// cosmos -s $env:EQUINOX_COSMOS_CONNECTION -d $env:EQUINOX_COSMOS_DATABASE -c $env:EQUINOX_COSMOS_COLLECTION provision -ru 1000 let private connect (log: ILogger) mode discovery operationTimeout (maxRetryForThrottling, maxRetryWaitTime) = EqxConnector(log=log, mode=mode, requestTimeout=operationTimeout, maxRetryAttemptsOnThrottledRequests=maxRetryForThrottling, maxRetryWaitTimeInSeconds=maxRetryWaitTime) - .Connect("equinox-cli", discovery) + .Connect("equinox-samples", discovery) let private createGateway connection (maxItems,maxEvents) = EqxGateway(connection, EqxBatchingPolicy(defaultMaxItems=maxItems, maxEventsPerSlice=maxEvents)) let conn (log: ILogger, storeLog) (sargs : ParseResults) = let read key = Environment.GetEnvironmentVariable key |> Option.ofObj @@ -126,7 +126,7 @@ module Cosmos = let dbName, collName, pageSize, conn = conn (log, storeLog) sargs let cacheStrategy = if cache then - let c = Caching.Cache("Cli", sizeMb = 50) + let c = Caching.Cache("equinox-tool", sizeMb = 50) CachingStrategy.SlidingWindow (c, TimeSpan.FromMinutes 20.) |> Some else None StorageConfig.Cosmos (createGateway conn (defaultBatchSize,pageSize), cacheStrategy, unfolds, dbName, collName) \ No newline at end of file diff --git a/samples/Store/Integration/CartIntegration.fs b/samples/Store/Integration/CartIntegration.fs index 291feb431..16c451eee 100644 --- a/samples/Store/Integration/CartIntegration.fs +++ b/samples/Store/Integration/CartIntegration.fs @@ -5,6 +5,7 @@ open Equinox.Cosmos.Integration open Equinox.EventStore open Equinox.MemoryStore open Swensen.Unquote +open Xunit #nowarn "1182" // From hereon in, we may have some 'unused' privates (the tests) @@ -47,6 +48,9 @@ type Tests(testOutputHelper) = } [] +#if NET461 + [] // Likely due to net461 not having consistent json.net refs and no binding redirects +#endif let ``Can roundtrip in Memory, correctly folding the events`` args = Async.RunSynchronously <| async { let log, store = createLog (), createMemoryStore () let service = createServiceMem log store diff --git a/samples/Store/Integration/CodecIntegration.fs b/samples/Store/Integration/CodecIntegration.fs index 89eea866e..9fa894a87 100644 --- a/samples/Store/Integration/CodecIntegration.fs +++ b/samples/Store/Integration/CodecIntegration.fs @@ -4,6 +4,7 @@ module Samples.Store.Integration.CodecIntegration open Domain open Swensen.Unquote open TypeShape.UnionContract +open Xunit let serializationSettings = Newtonsoft.Json.Converters.FSharp.Settings.CreateCorrect(converters= @@ -42,6 +43,9 @@ let render = function let codec = genCodec() [] +#if NET461 + [] // Likely due to net461 not having consistent json.net refs and no binding redirects +#endif let ``Can roundtrip, rendering correctly`` (x: SimpleDu) = let serialized = codec.Encode x render x =! System.Text.Encoding.UTF8.GetString(serialized.payload) diff --git a/samples/Store/Integration/ContactPreferencesIntegration.fs b/samples/Store/Integration/ContactPreferencesIntegration.fs index 571d079bb..91a527301 100644 --- a/samples/Store/Integration/ContactPreferencesIntegration.fs +++ b/samples/Store/Integration/ContactPreferencesIntegration.fs @@ -5,6 +5,7 @@ open Equinox.Cosmos.Integration open Equinox.EventStore open Equinox.MemoryStore open Swensen.Unquote +open Xunit #nowarn "1182" // From hereon in, we may have some 'unused' privates (the tests) @@ -38,6 +39,9 @@ type Tests(testOutputHelper) = test <@ value = actual @> } [] +#if NET461 + [] // Likely due to net461 not having consistent json.net refs and no binding redirects +#endif let ``Can roundtrip in Memory, correctly folding the events`` args = Async.RunSynchronously <| async { let service = let log, store = createLog (), createMemoryStore () in createServiceMem log store do! act service args diff --git a/samples/Store/Integration/FavoritesIntegration.fs b/samples/Store/Integration/FavoritesIntegration.fs index c3a382278..bbf603124 100644 --- a/samples/Store/Integration/FavoritesIntegration.fs +++ b/samples/Store/Integration/FavoritesIntegration.fs @@ -5,6 +5,7 @@ open Equinox.Cosmos.Integration open Equinox.EventStore open Equinox.MemoryStore open Swensen.Unquote +open Xunit #nowarn "1182" // From hereon in, we may have some 'unused' privates (the tests) @@ -40,6 +41,9 @@ type Tests(testOutputHelper) = test <@ Array.isEmpty items @> } [] +#if NET461 + [] // Likely due to net461 not having consistent json.net refs and no binding redirects +#endif let ``Can roundtrip in Memory, correctly folding the events`` args = Async.RunSynchronously <| async { let store = createMemoryStore () let service = let log = createLog () in createServiceMem log store diff --git a/src/Equinox.Cosmos/Equinox.Cosmos.fsproj b/src/Equinox.Cosmos/Equinox.Cosmos.fsproj index 5377af83e..4572b39b5 100644 --- a/src/Equinox.Cosmos/Equinox.Cosmos.fsproj +++ b/src/Equinox.Cosmos/Equinox.Cosmos.fsproj @@ -23,6 +23,8 @@ + + diff --git a/src/Equinox.EventStore/Equinox.EventStore.fsproj b/src/Equinox.EventStore/Equinox.EventStore.fsproj index 92af6061c..27f1a0810 100644 --- a/src/Equinox.EventStore/Equinox.EventStore.fsproj +++ b/src/Equinox.EventStore/Equinox.EventStore.fsproj @@ -7,7 +7,7 @@ false true true - NET461 + $(DefineConstants);NET461 diff --git a/tests/Equinox.Cosmos.Integration/CosmosIntegration.fs b/tests/Equinox.Cosmos.Integration/CosmosIntegration.fs index 23db4fa03..e2742db14 100644 --- a/tests/Equinox.Cosmos.Integration/CosmosIntegration.fs +++ b/tests/Equinox.Cosmos.Integration/CosmosIntegration.fs @@ -12,7 +12,8 @@ let genCodec<'Union when 'Union :> TypeShape.UnionContract.IUnionContract>() = Equinox.UnionCodec.JsonUtf8.Create<'Union>(serializationSettings) module Cart = - let fold, initial, snapshot = Domain.Cart.Folds.fold, Domain.Cart.Folds.initial, Domain.Cart.Folds.snapshot + let fold, initial = Domain.Cart.Folds.fold, Domain.Cart.Folds.initial + let snapshot = Domain.Cart.Folds.isOrigin, Domain.Cart.Folds.compact let codec = genCodec() let createServiceWithoutOptimization connection batchSize log = let store = createEqxStore connection batchSize