How does the Entity Framework core map dynamic models?

Preface

In this paper, we will discuss several ways of mapping dynamic model , I believe that some children's shoes projects have such demand , Like every day / Generate a table every hour , This kind of dynamic model mapping is very common , After my groping , Here is a detailed idea for each step , Hope to help children's shoes without any clue , This article takes .NET Core 3.1 Console , At the same time SQL Server Database as an example （ Other databases are copied in the same way ）, Because of the built-in APi, Due to different versions, the constructor may need to be slightly adjusted . notes ： Although it's sample code , But I've encapsulated it as an actual project , Basically universal . This article is a little longer , Please be patient .

Dynamic mapping model introduces premise

First, we give the required features and the corresponding enumeration , Just look at the notes

public enum CustomTableFormat
{
    /// <summary>
    ///  Every day ,（yyyyMMdd）
    /// </summary>
    [Description(" Every day ")]
    DAY,
    /// <summary>
    ///  Every hour ,（yyyyMMddHH）
    /// </summary>
    [Description(" Every hour ")]
    HOUR,
    /// <summary>
    ///  Every minute （yyyyMMddHHmm）
    /// </summary>
    [Description(" Every minute ")]
    MINUTE
}

[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public class EfEntityAttribute : Attribute
{
    /// <summary>
    ///  Whether to enable dynamic table generation 
    /// </summary>
    public bool EnableCustomTable { get; set; } = false;
    /// <summary>
    ///  Dynamically generate table prefixes 
    /// </summary>
    public string Prefix { get; set; }
    /// <summary>
    ///  Table generation rules 
    /// </summary>
    public CustomTableFormat Format { get; set; } = CustomTableFormat.DAY;

    public override string ToString()
    {
        if (EnableCustomTable)
        {
            return string.IsNullOrEmpty(Prefix) ? Format.FormatToDate() : $"{Prefix}{Format.FormatToDate()}";
        }
        return base.ToString();
    }
}

public static class CustomTableFormatExetension
{
    public static string FormatToDate(this CustomTableFormat tableFormat)
    {
        return tableFormat switch
        {
            CustomTableFormat.DAY => DateTime.Now.ToString("yyyyMMdd"),
            CustomTableFormat.HOUR => DateTime.Now.ToString("yyyyMMddHH"),
            CustomTableFormat.MINUTE => DateTime.Now.ToString("yyyyMMddHHmm"),
            _ => DateTime.Now.ToString("yyyyMMdd"),
        };
    }
}

By defining features , The main starting point is based on two considerations ： firstly ： Inject the model from the outside instead of writing it dead DbSet Attribute access 、 second ： Each model can define dynamic mapping table rules

Dynamic mapping model mode （ One ）

First, we give the context we need to use , For the convenience of demonstration, let's take the automatic map per minute model as an example

public class EfDbContext : DbContext
{
    public string Date { get; set; } = CustomTableFormat.MINUTE.FormatToDate();
    public EfDbContext(DbContextOptions<EfDbContext> options) : base(options)
    {

    }
}

Dynamic model refers to different table names , For example, we realize that every day / Every hour / Dynamically map the model and generate a table every minute . In the following interface, we need to generate a table format every minute , So define the per minute attribute in context . The first way is through IModelCacheKeyFactory Interface , This interface caches all model table names in the specified context , So we can change it according to the dynamic model table name , as follows ：

public class CustomModelCacheKeyFactory : IModelCacheKeyFactory
{
    public object Create(DbContext context)
    {
        var efDbContext = context as EfDbContext;
        if (efDbContext != null)
        {
            return (context.GetType(), efDbContext.Date);
        }
        return context.GetType();
    }
}

The above implementation seems to feel a little incomprehensible , This is mainly to directly implement the interface in one step , The underlying essence is to call an extra instance of a cache key class , We will change the above to the following two steps to make it clear at a glance

public class CustomModelCacheKeyFactory : ModelCacheKeyFactory
{
    private string _date;
    public CustomModelCacheKeyFactory(ModelCacheKeyFactoryDependencies dependencies)
        : base(dependencies)
    {

    }
    public override object Create(DbContext context)
    {
        if (context is EfDbContext efDbContext)
        {
            _date = efDbContext.Date;
        }

        return new CustomModelCacheKey(_date, context);
    }
}

public class CustomModelCacheKey : ModelCacheKey
{
    private readonly Type _contextType;
    private readonly string _date;
    public CustomModelCacheKey(string date, DbContext context) : base(context)
    {
        _date = date;
        _contextType = context.GetType();
    }

    public virtual bool Equals(CustomModelCacheKey other)
      => _contextType == other._contextType && _date == other._date;

    public override bool Equals(object obj)
      => (obj is CustomModelCacheKey otherAsKey) && Equals(otherAsKey);

    public override int GetHashCode() => _date.GetHashCode();
}

And then in OnModelCreating Method to scan, identify and register the model , as follows ：

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    var entityMethod = typeof(ModelBuilder).GetMethod(nameof(modelBuilder.Entity), 
    new Type[] { });
    
    var assembly = Assembly.GetExecutingAssembly();

    //【1】 Use Entity Methods registration 
    foreach (var type in assembly.ExportedTypes)
    {
        if (!(type.GetCustomAttribute(typeof(EfEntityAttribute)) is EfEntityAttribute attribute))
        {
            continue;
        }

        if (type.IsNotPublic || type.IsAbstract || type.IsSealed
            || type.IsGenericType
            || type.ContainsGenericParameters)
        {
            continue;
        }

        entityMethod.MakeGenericMethod(type)
                .Invoke(modelBuilder, new object[] { });
    }

    //【2】 Use IEntityTypeConfiguration<T> register 
    modelBuilder.ApplyConfigurationsFromAssembly(assembly);
    
    base.OnModelCreating(modelBuilder);
}

The first way is to register the model through reflection , Its essence is to call modeBuilder.Entity Method , If we use annotations on the model , It will also be applied accordingly

 

But annotations are not flexible enough , For example, to identify the union primary key , Can only be used Fluent APi, So we do it externally IEntityTypeConfiguration To register , then EF Core Provides registration for this interface assembly , Its underlying nature is also scanning assemblies , Both methods support , Don't worry about external model registration

 

Then we give the test model , The table name is current minute , Table names cannot be annotated （ Value must be constant ）, So we use the second mapping model as follows

[EfEntity(EnableCustomTable = true, Format = CustomTableFormat.MINUTE)]
public class Test
{
    [Table(DateTime.Now.ToString("yyyyMMdd"))]
    public int Id { get; set; }
    public string Name { get; set; }
}

public class TestEntityTypeConfiguration : IEntityTypeConfiguration<Test>
{
    public void Configure(EntityTypeBuilder<Test> builder)
    {
        builder.ToTable(DateTime.Now.ToString("yyyyMMddHHmm"));
    }
}

The second configuration mentioned above is possible , But we still have a more concise one-step operation , So delete here The second way mentioned above , Because in OnModelCreating Method inside , We reflected and called Entity Method , So we call reflection directly Entity Method is cast to EntityTypeBuilder, On the existing basis , The code makes a key mark

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    var entityMethod = typeof(ModelBuilder).GetMethod(nameof(modelBuilder.Entity), new Type[] { });
    var assembly = Assembly.GetExecutingAssembly();

    //【1】 Use Entity Methods registration 
    foreach (var type in assembly.ExportedTypes)
    {
        if (!(type.GetCustomAttribute(typeof(EfEntityAttribute)) is EfEntityAttribute attribute))
        {
            continue;
        }

        if (type.IsNotPublic || type.IsAbstract || type.IsSealed
            || type.IsGenericType
            || type.ContainsGenericParameters)
        {
            continue;
        }

        //  Cast to EntityTypeBuilder
        var entityBuilder = (EntityTypeBuilder)entityMethod.MakeGenericMethod(type)
               .Invoke(modelBuilder, new object[] { });

        if (attribute.EnableCustomTable)
        {
            entityBuilder.ToTable(attribute.ToString());
        }
    }

    //【2】 Use IEntityTypeConfiguration<T> register 
    modelBuilder.ApplyConfigurationsFromAssembly(assembly);

    base.OnModelCreating(modelBuilder);
}

Finally, context Injection , Here we distinguish the internal and external containers （EF Core Why is it divided into internal containers , Please refer to the article for specific reasons 《EntityFramework Core 3.x Context constructors can inject instances ？》）

 

Because in a real project, the context may need to inject other interfaces into the context constructor , For example, we may inject an interface into the context constructor to change the table schema or different table name rules according to the specific interface implementation

static IServiceProvider Initialize()
{
    var services = new ServiceCollection();

    services.AddEntityFrameworkSqlServer()
        .AddDbContext<EfDbContext>(
            (serviceProvider, options) =>
                options.UseSqlServer("server=.;database=efcore;uid=sa;pwd=sa123;")
                .UseInternalServiceProvider(serviceProvider));

    services.Replace(ServiceDescriptor.Singleton<IModelCacheKeyFactory, CustomModelCacheKeyFactory>());

    return services.BuildServiceProvider();
}

Because we have distinguished EF Core Internal and external containers , So when replacing the custom cache key factory , It can no longer be called directly as follows ReplaceService Methods to replace , It's bound to throw an exception

options.UseSqlServer("server=.;database=efcore;uid=sa;pwd=sa123;")
                        .ReplaceService<IModelCacheKeyFactory, CustomModelCacheKeyFactory>()

At the same time, keep in mind that Web Project utilization EF Core Always use scope (scope) To release the context , Unlike Web Can be based on HTTP Ask to act as scope, Finally, we test as follows

using (var scope1 = ServiceProvider.CreateScope())
{
    var context1 = scope1.ServiceProvider.GetService<EfDbContext>();

    context1.Database.EnsureCreated();

    var type = context1.Model.FindEntityType(typeof(Test));

    Console.WriteLine(type?.GetTableName());

    var tests = context1.Set<Test>().ToList();
}

Thread.Sleep(60000);

using (var scope2 = ServiceProvider.CreateScope())
{
    var context2 = scope2.ServiceProvider.GetService<EfDbContext>();

    context2.Database.EnsureCreated();

    var type = context2.Model.FindEntityType(typeof(Test));

    Console.WriteLine(type?.GetTableName());

    var tests1 = context2.Set<Test>().ToList();
}

For the convenience of seeing the actual effect , We build two scope, Then sleep for a minute , Print out the table name on the interface , If the name of the printed table is inconsistent after two minutes , It shows that we have achieved our expectation

Dynamic mapping model mode （ Two ）

On Let's use the rule per minute dynamic mapping table , At the same time, different models can have their own rules （ Prefix , Every hour or every day ） wait , This is the first way

 

If you understand the first way completely , There may be doubts , Because the interface life cycle of the first method is singleton , If not, all models in the context will be cached

 

call OnModelCreating The method is just model building , But now we call the built-in APi To use all the models manually , There will be no caching at this point , So you don't need it anymore IModelCacheKeyFactory Interface

 

Yes EF Core If you know a little bit , We know OnModelCreating Method will only be called once , We use and dispose of all models manually , In other words, each request uses a new model , Said so much , So what are we going to do ？

 

If you've seen my previous principle analysis , Probably know EntityFramework Core For model processing （ Except for the default model cache ） There are three steps , Except for the model cache ： Build the model , Using the model , Disposal model .

 

We will OnModelCreating The method code is copied directly , It's just three more steps , In our case ModelBuilder when , We need to provide the default contract for the corresponding database , Then use the model 、 Disposal model , It turns out like this

 services.AddEntityFrameworkSqlServer()
      .AddDbContext<EfDbContext>(
          (serviceProvider, options) => {
          
            options.UseSqlServer("server=.;database=efcore;uid=sa;pwd=sa123;")
               .UseInternalServiceProvider(serviceProvider);

            var conventionSet = SqlServerConventionSetBuilder.Build();

            var modelBuilder = new ModelBuilder(conventionSet);

            // OnModelCreating Method , Code duplication 

            options.UseModel(modelBuilder.Model);

            modelBuilder.FinalizeModel();               
  )};

Run the first way to test the code , And then there's no problem

  The problem is coming. , If there are multiple databases , Don't we all have to do it again like the above ？ The above implementation essentially builds and reuses a new model each time a context is constructed , So we put it in context constructors , Then write an extension method to build the model , as follows ：

public static class ModelBuilderExetension
{
    public static ModelBuilder BuildModel(this ModelBuilder modelBuilder)
    {

        var entityMethod = typeof(ModelBuilder).GetMethod(nameof(modelBuilder.Entity), new Type[] { });
        var assembly = Assembly.GetExecutingAssembly();

        //【1】 Use Entity Methods registration 
        foreach (var type in assembly.ExportedTypes)
        {
            if (!(type.GetCustomAttribute(typeof(EfEntityAttribute)) is EfEntityAttribute attribute))
            {
                continue;
            }

            if (type.IsNotPublic || type.IsAbstract || type.IsSealed
                || type.IsGenericType
                || type.ContainsGenericParameters)
            {
                continue;
            }

            var entityBuilder = (EntityTypeBuilder)entityMethod.MakeGenericMethod(type)
                   .Invoke(modelBuilder, new object[] { });

            if (attribute.EnableCustomTable)
            {
                entityBuilder.ToTable(attribute.ToString());
            }
        }

        //【2】 Use IEntityTypeConfiguration<T> register 
        modelBuilder.ApplyConfigurationsFromAssembly(assembly);

        return modelBuilder;
    }
}

Finally, in the context constructor , Simple call , as follows ：

public class EfDbContext : DbContext
{
    public string Date { get; set; } = CustomTableFormat.MINUTE.FormatToDate();
    public EfDbContext(DbContextOptions<EfDbContext> options) : base(options)
    {
        // Provide different database default conventions 
        ConventionSet conventionSet = null;

        if (Database.ProviderName == "Microsoft.EntityFrameworkCore.SqlServer")
        {
            conventionSet = SqlServerConventionSetBuilder.Build();
        }
        else if (Database.ProviderName == "Microsoft.EntityFrameworkCore.Sqllite")
        {
            conventionSet = SqliteConventionSetBuilder.Build();
        }
        else if (Database.ProviderName == "Microsoft.EntityFrameworkCore.MySql")
        {
            conventionSet = MySqlConventionSetBuilder.Build();
        }

        var modelBuilder = new ModelBuilder(conventionSet);

        var optionBuilder = new DbContextOptionsBuilder(options);

        // Using the model 
        optionBuilder.UseModel(modelBuilder.Model);

        // Disposal model 
        modelBuilder.FinalizeModel();
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        // Build the model 
        modelBuilder.BuildModel();

        base.OnModelCreating(modelBuilder);
    }
}

Dynamic mapping model table generation

See here , Careful, you don't know if you have found , How did my printout work , No exception was thrown , The reality is that an exception must be thrown , Because we only do model dynamic mapping , But table auto generation, which I've ignored before , as follows ：

 

  How to generate this table also depends on the actual situation , such as SQL Server Write a job, automatically generate a table every day, etc , To be compatible with multiple databases , I'm afraid it's a little bit of trouble

 

I didn't spend much time looking at the source code , A little look at , If you take a chance, you may be able to directly find the interface implementation to create tables based on the model , It doesn't seem to be , Even if there is one, it's troublesome , Then we will Build... Manually SQL Statement or pass lambda Construction can also

 

We can get the model that needs to be dynamically generated to implement its features in context , Then set up a timer to generate the corresponding table every minute , For different database types , We can get it through the following properties （ It's the same name as the package ）

//  such as SQL Server：Microsoft.EntityFrameworkCore.SqlServer
context.Database.ProviderName

Here I use SQL Server Database, for example , Other databases like MySqL、Sqlite The only difference is that self growth settings and column types are different , Create table , It's made up of five parts ： Does the table exist , Table name , Primary key , All columns , constraint . We define it as follows ：

internal sealed class CustomTableModel
{
    public CustomEntityType CustomEntityType { get; set; }

    public string TableName { get; set; } = string.Empty;
    public string CheckTable { get; set; } = string.Empty;
    public string PrimaryKey { get; set; } = string.Empty;
    public string Columns { get; set; } = string.Empty;
    public string Constraint { get; set; } = string.Empty;

    public override string ToString()
    {
        var placeHolder = $"{CheckTable} create table {TableName} ({PrimaryKey} {Columns}";

        placeHolder = string.IsNullOrEmpty(Constraint) ? $"{placeHolder.TrimEnd(',')})" : $"{placeHolder}{Constraint})";

        return placeHolder.Replace("@placeholder_table_name", CustomEntityType.ToString());
    }
}

Because each generation only has a different table name , So we cache the entire table data structure , Just replace the table name inside . The whole implementation logic is as follows ：

public static void Execute()
{
    using var scope = Program.ServiceProvider.CreateScope();
    var context = scope.ServiceProvider.GetService<EfDbContext>();

    context.Database.EnsureCreated();

    var cache = scope.ServiceProvider.GetService<IMemoryCache>();

    var cacheKey = context.GetType().FullName;

    if (!cache.TryGetValue(cacheKey, out List<CustomTableModel> models))
    {
        lock (_syncObject)
        {
            if (!cache.TryGetValue(cacheKey, out models))
            {
                models = CreateModels(context);

                models = cache.Set(cacheKey, models, new MemoryCacheEntryOptions { Size = 100, Priority = CacheItemPriority.High });
            }
        }
    }

    Create(context, models);
}

private static void Create(EfDbContext context, List<CustomTableModel> models)
{
    foreach (var m in models)
    {
        context.Execute(m.ToString());
    }
}

internal static void CreateEntityTypes(CustomEntityType customEntityType)
{
    EntityTypes.Add(customEntityType);
}

The red part above is very important , Why? ？ Let's do it first OnModelCreating Method , That is to say, we have to make sure that all models have been built , We can get all the model metadata in context

 

And then there's OnModeCreating In the method , On the basis of starting the automatic mapping model , Add the following code （ Of course, you also need to check whether the table name is duplicate ）：

 if (attribute.EnableCustomTable)
  {
      entityBuilder.ToTable(attribute.ToString());

      var customType = new CustomEntityType()
      {
          ClrType = type,
          Prefix = attribute.Prefix,
          Format = attribute.Format
      };

      var existTable = CreateCustomTable.EntityTypes.FirstOrDefault(c => c.ToString() == customType.ToString());

      if (existTable != null)
      {
          throw new ArgumentNullException($"Cannot use table '{customType}' for entity type '{type.Name}' since it is being used for entity type '{existTable.ClrType.Name}' ");
      }

      CreateCustomTable.CreateEntityTypes(customType);
  }

Believe in building SQL I don't care about the sentence , No more , Children's shoes in real need , But I don't know , If there are more people , I will be compatible with different databases SQL Statement construction will be placed in github Up , The console entry method is called as follows ：

private const int TIME_INTERVAL_IN_MILLISECONDS = 60000;
private static Timer _timer { get; set; }
public static IServiceProvider ServiceProvider { get; set; }
static void Main(string[] args)
{
    ServiceProvider = Initialize();

    // Check once during initialization 
    CreateCustomTable.Execute();

    // Timing check 
    _timer = new Timer(TimerCallback, null, TIME_INTERVAL_IN_MILLISECONDS, Timeout.Infinite);

    using (var scope1 = ServiceProvider.CreateScope())
    {
        var context1 = scope1.ServiceProvider.GetService<EfDbContext>();

        context1.Database.EnsureCreated();

        var type = context1.Model.FindEntityType(typeof(Test1));

        Console.WriteLine(type?.GetTableName());

        var tests = context1.Set<Test1>().ToList();
    }

    Thread.Sleep(60000);

    using (var scope2 = ServiceProvider.CreateScope())
    {
        var context2 = scope2.ServiceProvider.GetService<EfDbContext>();

        context2.Database.EnsureCreated();

        var type = context2.Model.FindEntityType(typeof(Test2));

        Console.WriteLine(type?.GetTableName());

        var tests1 = context2.Set<Test2>().ToList();
    }

    Console.ReadKey();

}

The next step is to define the timer , Callbacks call the above Execute Method , as follows ：

static void TimerCallback(object state)
{
      var watch = new Stopwatch();

      watch.Start();

      CreateCustomTable.Execute();

      _timer.Change(Math.Max(0, TIME_INTERVAL_IN_MILLISECONDS - watch.ElapsedMilliseconds), Timeout.Infinite);
 }

Finally, let's test the actual effect of two models

[EfEntity(EnableCustomTable = true, Prefix = "test1", Format = CustomTableFormat.MINUTE)]
public class Test1
{
    public int Id { get; set; }
    public int UserId { get; set; }
    public string Name { get; set; }
}

public class Test1EntityTypeConfiguration : IEntityTypeConfiguration<Test1>
{
    public void Configure(EntityTypeBuilder<Test1> builder)
    {
        builder.HasKey(k => new { k.Id, k.UserId });
    }
}


[EfEntity(EnableCustomTable = true, Prefix = "test2", Format = CustomTableFormat.MINUTE)]
public class Test2
{
    public int Id { get; set; }
    public int UserId { get; set; }
    public string Name { get; set; }
}

public class Test2EntityTypeConfiguration : IEntityTypeConfiguration<Test2>
{
    public void Configure(EntityTypeBuilder<Test2> builder)
    {
        builder.HasKey(k => new { k.Id, k.UserId });
    }
}

summary

Last, last , Old rules , There are two ways to implement dynamic mapping model , By manually building SQL Statement and cache , Summarized below ！

   Use IModelCacheKeyFactory

 

  Using the model manually 、 Disposal model

 

    Compatible with different databases , Build... Manually SQL Statement and cache