Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for postgres-xl table distribution #1697

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
using JetBrains.Annotations;
using Microsoft.EntityFrameworkCore.Infrastructure;
using Microsoft.EntityFrameworkCore.Metadata.Builders;
using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata;
using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata.Internal;
using Npgsql.EntityFrameworkCore.PostgreSQL.Utilities;
using NpgsqlTypes;
Expand Down Expand Up @@ -321,6 +322,106 @@ public static EntityTypeBuilder<TEntity> UseCockroachDbInterleaveInParent<TEntit

#endregion CockroachDB Interleave-in-parent

#region Postgres-xl Distribute By

public static EntityTypeBuilder UsePostgresXlDistributedBy(
Jmorjsm marked this conversation as resolved.
Show resolved Hide resolved
[NotNull] this EntityTypeBuilder entityTypeBuilder,
PostgresXlDistributeByStrategy distributeByStrategy)
{
switch (distributeByStrategy)
{
case PostgresXlDistributeByStrategy.Replication:
case PostgresXlDistributeByStrategy.Roundrobin:
case PostgresXlDistributeByStrategy.Randomly:
var distribute = entityTypeBuilder.Metadata.GetPostgresXlDistributeBy();
distribute.DistributionStrategy = distributeByStrategy;
break;
}

return entityTypeBuilder;
}

public static EntityTypeBuilder<TEntity> UsePostgresXlDistributedBy<TEntity>(
[NotNull] this EntityTypeBuilder<TEntity> entityTypeBuilder,
PostgresXlDistributeByStrategy distributeByStrategy)
where TEntity : class
=> (EntityTypeBuilder<TEntity>)UsePostgresXlDistributedBy((EntityTypeBuilder)entityTypeBuilder, distributeByStrategy);

public static EntityTypeBuilder UsePostgresXlDistributedBy(
[NotNull] this EntityTypeBuilder entityTypeBuilder,
[NotNull] string columnName,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The preferred way to continue a column is via Expression<Func<TEntity,object>>, which allows a more fluent and strongly-typed approach. Take a look at how IncludeProperties is implemented on indexes.

Another thing that's important, is that the metadata and builder API does not deal with columns - it deals with (conceptual) properties. This is important since the property can have a different name than the column it's mapped to. Once again, IncludeProperties should provide a similar scenario to follow.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would an overload on these methods be sufficient?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah - we need overloads for both Expression<Func<..>> (for regular strongly-typed usage) and string (for weakly-typed usage), and also make sure the string parameter references the property, and not its column. Basically use IncludeProperties in NpgsqlIndexBuilderExtensions as your implementation model.

PostgresXlDistributeByColumnFunction distributeByColumnFunction = PostgresXlDistributeByColumnFunction.None)

{
Check.NotNull(entityTypeBuilder, nameof(entityTypeBuilder));
Check.NotEmpty(columnName, nameof(columnName));

var distribute = entityTypeBuilder.Metadata.GetPostgresXlDistributeBy();

distribute.DistributionStrategy = PostgresXlDistributeByStrategy.None;
distribute.DistributeByColumnFunction = PostgresXlDistributeByColumnFunction.None;
distribute.DistributeByColumnName = columnName;

return entityTypeBuilder;
}

public static EntityTypeBuilder<TEntity> UsePostgresXlDistributedBy<TEntity>(
[NotNull] this EntityTypeBuilder<TEntity> entityTypeBuilder,
[NotNull] string columnName,
PostgresXlDistributeByColumnFunction distributeByColumnFunction = PostgresXlDistributeByColumnFunction.None)
where TEntity : class
=> (EntityTypeBuilder<TEntity>)UsePostgresXlDistributedBy(
(EntityTypeBuilder)entityTypeBuilder, columnName, distributeByColumnFunction);

public static EntityTypeBuilder UsePostgresXlDistributedRandomly(
[NotNull] this EntityTypeBuilder entityTypeBuilder)
{
Check.NotNull(entityTypeBuilder, nameof(entityTypeBuilder));

var distribute = entityTypeBuilder.Metadata.GetPostgresXlDistributeBy();
distribute.DistributionStrategy = PostgresXlDistributeByStrategy.Randomly;

return entityTypeBuilder;
}

public static EntityTypeBuilder<TEntity> UsePostgresXlDistributedRandomly<TEntity>(
[NotNull] this EntityTypeBuilder<TEntity> entityTypeBuilder)
where TEntity : class
=> (EntityTypeBuilder<TEntity>)UsePostgresXlDistributedRandomly((EntityTypeBuilder)entityTypeBuilder);

public static EntityTypeBuilder UsePostgresXlDistributionStyle(
[NotNull] this EntityTypeBuilder entityTypeBuilder,
PostgresXlDistributionStyle distributionStyle,
[NotNull] string distributionKey)
{
Check.NotNull(entityTypeBuilder, nameof(entityTypeBuilder));
Check.NotEmpty(distributionKey, nameof(distributionKey));

var distribute = entityTypeBuilder.Metadata.GetPostgresXlDistributeBy();

switch (distributionStyle)
{
case PostgresXlDistributionStyle.Even:
case PostgresXlDistributionStyle.Key:
case PostgresXlDistributionStyle.All:
distribute.DistributionStyle = distributionStyle;
distribute.DistributeByColumnName = distributionKey;
return entityTypeBuilder;

default:
throw new ArgumentOutOfRangeException(nameof(distributionStyle), distributionStyle, $@"Invalid {nameof(PostgresXlDistributionStyle)} provided.");
}
}

public static EntityTypeBuilder<TEntity> UsePostgresXlDistributionStyle<TEntity>(
[NotNull] this EntityTypeBuilder<TEntity> entityTypeBuilder,
PostgresXlDistributionStyle distributionStyle,
[NotNull] string distributionKey)
where TEntity : class
=> (EntityTypeBuilder<TEntity>)UsePostgresXlDistributionStyle((EntityTypeBuilder)entityTypeBuilder, distributionStyle, distributionKey);

#endregion Postgres-xl Distribute By

#region Obsolete

/// <summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -94,5 +94,12 @@ public static CockroachDbInterleaveInParent GetCockroachDbInterleaveInParent([No
=> new(entityType);

#endregion CockroachDb interleave in parent

#region Postgres-xl Distribute By

public static PostgresXlDistributeBy GetPostgresXlDistributeBy([NotNull] this IReadOnlyEntityType entityType)
=> new(entityType);

#endregion Postgres-xl Distribute By
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

namespace Npgsql.EntityFrameworkCore.PostgreSQL.Metadata.Internal
{
internal static class PostgresXlDistributeByAnnotationNames
{

Jmorjsm marked this conversation as resolved.
Show resolved Hide resolved
public const string Prefix = NpgsqlAnnotationNames.Prefix + "PostgresXL:";

public const string DistributeBy = Prefix + "DistributeBy";

}
}
145 changes: 145 additions & 0 deletions src/EFCore.PG/Metadata/PostgresXlDistributeBy.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.
using System;
using System.Text;
using JetBrains.Annotations;
using Microsoft.EntityFrameworkCore.Infrastructure;
using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata.Internal;

namespace Npgsql.EntityFrameworkCore.PostgreSQL.Metadata
Jmorjsm marked this conversation as resolved.
Show resolved Hide resolved
{
public class PostgresXlDistributeBy
{
private const string AnnotationName = PostgresXlDistributeByAnnotationNames.DistributeBy;

readonly IReadOnlyAnnotatable _annotatable;
Jmorjsm marked this conversation as resolved.
Show resolved Hide resolved

public virtual Annotatable Annotatable
=> (Annotatable)_annotatable;

Jmorjsm marked this conversation as resolved.
Show resolved Hide resolved

public PostgresXlDistributeBy([NotNull] IReadOnlyAnnotatable annotatable)
=> _annotatable = annotatable;

public virtual PostgresXlDistributeByStrategy DistributionStrategy
{
get => GetData().DistributionStrategy;
set
{
(_, var distributeByColumnFunction, var distributionStyle, var columnName) = GetData();
Jmorjsm marked this conversation as resolved.
Show resolved Hide resolved
SetData(value, distributeByColumnFunction, distributionStyle, columnName);
}
}

public virtual PostgresXlDistributeByColumnFunction DistributeByColumnFunction
{
get => GetData().DistributeByColumnFunction;
set
{
(var distributionStrategy, _, var distributionStyle, var columnName) = GetData();
SetData(distributionStrategy, value, distributionStyle, columnName);
}
}

public virtual PostgresXlDistributionStyle DistributionStyle
{
get => GetData().DistributionStyle;
set
{
(var distributionStrategy, var distributeByColumnFunction, _, var columnName) = GetData();
SetData(distributionStrategy, distributeByColumnFunction, value, columnName);
}
}

public virtual string DistributeByColumnName
{
get => GetData().ColumnName;
[param:NotNull] set
{
(var distributionStrategy, var distributeByColumnFunction, var distributionStyle, _) = GetData();
SetData(distributionStrategy, distributeByColumnFunction, distributionStyle, value);
}
}

private (PostgresXlDistributeByStrategy DistributionStrategy, PostgresXlDistributeByColumnFunction DistributeByColumnFunction, PostgresXlDistributionStyle DistributionStyle, string ColumnName) GetData()
{
var str = Annotatable[AnnotationName] as string;
return str == null
? (0, 0, 0, null)
: Deserialize(str);
}

private void SetData(
PostgresXlDistributeByStrategy distributionStrategy,
PostgresXlDistributeByColumnFunction distributeByColumnFunction,
PostgresXlDistributionStyle postgresXlDistributionStyle,
string distributeByColumnName)
{
Annotatable[AnnotationName] = Serialize(distributionStrategy, distributeByColumnFunction, postgresXlDistributionStyle, distributeByColumnName);
}

private string Serialize(
PostgresXlDistributeByStrategy distributionStrategy,
PostgresXlDistributeByColumnFunction distributeByColumnFunction,
PostgresXlDistributionStyle postgresXlDistributionStyle,
string distributeByColumnName)
{
var stringBuilder = new StringBuilder();

EscapeAndQuote(stringBuilder, distributionStrategy);
stringBuilder.Append(",");
EscapeAndQuote(stringBuilder, distributeByColumnFunction);
stringBuilder.Append(",");
EscapeAndQuote(stringBuilder, postgresXlDistributionStyle);
stringBuilder.Append(",");
EscapeAndQuote(stringBuilder, distributeByColumnName);

return stringBuilder.ToString();
}

private (PostgresXlDistributeByStrategy DistributionStrategy,
PostgresXlDistributeByColumnFunction DistributeByColumnFunction,
PostgresXlDistributionStyle DistributionStyle,
string ColumnName)
Deserialize(string str)
{
var position = 0;
var distributionStrategy = Enum.Parse<PostgresXlDistributeByStrategy>(ExtractValue(str, ref position));
var distributeByColumnFunction = Enum.Parse<PostgresXlDistributeByColumnFunction>(ExtractValue(str, ref position));
var distributionStyle = Enum.Parse<PostgresXlDistributionStyle>(ExtractValue(str, ref position));
var columnName = ExtractValue(str, ref position);

return (distributionStrategy, distributeByColumnFunction, distributionStyle, columnName);
}

private static void EscapeAndQuote(StringBuilder builder, object value)
{
builder.Append("'");

if (value != null)
{
builder.Append(value.ToString().Replace("'", "''"));
}

builder.Append("'");
}

private static string ExtractValue(string value, ref int position)
{
position = value.IndexOf('\'', position) + 1;

var end = value.IndexOf('\'', position);

while (end + 1 < value.Length
&& value[end + 1] == '\'')
{
end = value.IndexOf('\'', end + 2);
}

var extracted = value.Substring(position, end - position).Replace("''", "'");
position = end + 1;

return extracted.Length == 0 ? null : extracted;
}
}
}
12 changes: 12 additions & 0 deletions src/EFCore.PG/Metadata/PostgresXlDistributeByColumnFunction.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

namespace Npgsql.EntityFrameworkCore.PostgreSQL.Metadata
Jmorjsm marked this conversation as resolved.
Show resolved Hide resolved
{
public enum PostgresXlDistributeByColumnFunction
{
None = 0,
Hash = 1,
Modulo = 2,
}
}
13 changes: 13 additions & 0 deletions src/EFCore.PG/Metadata/PostgresXlDistributeByStrategy.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

namespace Npgsql.EntityFrameworkCore.PostgreSQL.Metadata
{
public enum PostgresXlDistributeByStrategy
{
None = 0,
Replication = 1,
Roundrobin = 2,
Jmorjsm marked this conversation as resolved.
Show resolved Hide resolved
Randomly = 3,
}
}
13 changes: 13 additions & 0 deletions src/EFCore.PG/Metadata/PostgresXlDistributionStyle.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

namespace Npgsql.EntityFrameworkCore.PostgreSQL.Metadata
Jmorjsm marked this conversation as resolved.
Show resolved Hide resolved
{
public enum PostgresXlDistributionStyle
{
None = 0,
Even = 1,
Key = 2,
All = 3,
}
}
Loading