Documentation

fsds/db-manager-implementation-plan.md

Unified DbManager Component Plan

Problem Summary

  • Issue: "Aggregate deployment is too large" error when running azd up
  • Root Cause: 13 separate DbMigrator projects each deploy as individual Azure Container Apps
  • Solution: Consolidate into single db-manager component that handles all database migrations

Design Decisions (User Choices)

Decision Choice
Discovery method Manifest declaration (HasDatabase = true)
Type safety Compile-time (source generator)
Migration ordering Reuse existing EngineManifest.Dependencies
Component name db-manager

Architecture Overview

Current State (13 DbMigrators)

AppHost → CoreData-DbMigrator → Database
       → Prism-DbMigrator → Database (waits for CoreData-DbMigrator)
       → Identity-DbMigrator → Database (waits for Prism-DbMigrator)
       → ... 10 more migrators

Target State (1 DbManager)

AppHost → DbManager → Database
                   ↓
          Runs migrations for all components in dependency order

Implementation Stages

Stage 1: Extend EngineManifest Pattern

Goal: Add database declaration to engine manifests

Update each EngineManifest.cs (13 engines with databases):

// engines/catalog/src/Acsis.Dynaplex.Engines.Catalog/EngineManifest.cs
public static class EngineManifest {
    public const string Name = "catalog";
    public const string Prefix = "ctlg";
    public static readonly string[] Dependencies = ["prism", "core-data", "identity", "spatial"];

    // NEW: Database declaration
    public const bool HasDatabase = true;
    public const string Schema = "catalog";
}

Components with HasDatabase = true (13):

  • core-data, prism, identity, spatial, catalog, events, file-service, printing, transport, workflow, iot, bbu, system-environment

Components with HasDatabase = false (or omitted):

  • importer, intelligence

Stage 2: Update Source Generator

File: strata/source-generators/src/Acsis.Dynaplex.Strata.SourceGenerators/ComponentIndexGenerator.cs

Changes:

  1. Parse HasDatabase and Schema fields from EngineManifest
  2. Update EngineMetadata record to include database info
  3. Generate DatabaseComponents array with topological ordering

Generated output enhancement:

// ComponentIndex.g.cs (enhanced)
public record ComponentMetadata(
    string Name,
    string Prefix,
    string[] Dependencies,
    bool HasDatabase,
    string? Schema
) : IResourceAnnotation;

public static partial class ComponentIndex {
    public static class Catalog {
        public const string Name = "catalog";
        public const string Prefix = "ctlg";
        public const bool HasDatabase = true;
        public const string Schema = "catalog";

        public static readonly ComponentMetadata Metadata = new(
            Name: "catalog",
            Prefix: "ctlg",
            Dependencies: new[] { "prism", "core-data", "identity", "spatial" },
            HasDatabase: true,
            Schema: "catalog"
        );
    }

    // Topologically sorted list of components with databases
    public static readonly string[] DatabaseComponentsInOrder = new[] {
        "core-data",       // Level 0: no deps
        "events",          // Level 0: no deps
        "prism",           // Level 1: depends on core-data
        "system-environment", // Level 1
        "file-service",    // Level 1
        "identity",        // Level 2: depends on core-data, prism
        "spatial",         // Level 3: depends on prism, core-data, identity
        "printing",        // Level 3
        "transport",       // Level 3
        "catalog",         // Level 4: depends on prism, core-data, identity, spatial
        "iot",             // Level 4
        "workflow",        // Level 5: depends on prism, core-data, catalog
        "bbu"              // Level 5: depends on catalog
    };
}

Stage 3: Create DbManager Component

Location: engines/db-manager/src/Acsis.Dynaplex.Engines.DbManager/

Project structure:

engines/db-manager/
├── src/
│   └── Acsis.Dynaplex.Engines.DbManager/
│       ├── Acsis.Dynaplex.Engines.DbManager.csproj
│       ├── EngineManifest.cs
│       ├── Program.cs
│       └── MigrationWorker.cs

DbManager.csproj - References all Database projects:

<Project Sdk="Microsoft.NET.Sdk.Worker">
  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="$(ServiceDefaultsProject)" />
    <!-- All Database projects -->
    <ProjectReference Include="$(CoreDataDatabaseProject)" />
    <ProjectReference Include="$(PrismDatabaseProject)" />
    <ProjectReference Include="$(IdentityDatabaseProject)" />
    <!-- ... all 13 Database projects ... -->
  </ItemGroup>
</Project>

MigrationWorker.cs - Core orchestration logic:

public class MigrationWorker(
    IServiceProvider serviceProvider,
    IHostApplicationLifetime hostApplicationLifetime,
    ILogger<MigrationWorker> logger
) : BackgroundService {

    protected override async Task ExecuteAsync(CancellationToken ct) {
        // Migrations run in topological order (from ComponentIndex.DatabaseComponentsInOrder)
        foreach (var componentName in ComponentIndex.DatabaseComponentsInOrder) {
            await MigrateComponentAsync(componentName, ct);
        }

        hostApplicationLifetime.StopApplication();
    }

    private async Task MigrateComponentAsync(string componentName, CancellationToken ct) {
        logger.LogInformation("Migrating {Component}...", componentName);

        using var scope = serviceProvider.CreateScope();

        // Get the appropriate DbContext based on component name
        var dbContext = GetDbContextForComponent(scope.ServiceProvider, componentName);

        await EnsureSchemaAsync(dbContext, ct);
        await dbContext.Database.MigrateAsync(ct);
        await RunSeederIfExistsAsync(scope.ServiceProvider, componentName, ct);

        logger.LogInformation("Completed {Component}", componentName);
    }

    private DynaplexDbContext GetDbContextForComponent(IServiceProvider sp, string name) {
        return name switch {
            "core-data" => sp.GetRequiredService<CoreDataDb>(),
            "prism" => sp.GetRequiredService<PrismDb>(),
            "identity" => sp.GetRequiredService<IdentityDb>(),
            // ... etc
            _ => throw new ArgumentException($"Unknown component: {name}")
        };
    }
}

Program.cs:

var builder = Host.CreateApplicationBuilder(args);
builder.ConfigureOpenTelemetry();
builder.Services.AddServiceDiscovery();
builder.Services.AddHostedService<MigrationWorker>();

// Register ALL DbContexts
builder.AddAcsisDbContext<CoreDataDb>(CoreDataDb.SCHEMA);
builder.AddAcsisDbContext<PrismDb>(PrismDb.SCHEMA);
builder.AddAcsisDbContext<IdentityDb>(IdentityDb.SCHEMA);
// ... all 13 DbContexts

var host = builder.Build();
host.Run();

Stage 4: Update Aspire Orchestration

File: strata/orchestration/src/Acsis.Dynaplex.Strata.Orchestration/DynaplexInfrastructureExtensions.cs

Add new extension method:

public static IDistributedApplicationBuilder AddDynaplexDbManager(
    this IDistributedApplicationBuilder builder)
{
    var context = builder.GetDynaplexContext();

    if (context.Database == null) {
        throw new InvalidOperationException("Call AddDynaplexDatabase() first.");
    }

    context.DbManager = builder.AddProject<Acsis_Components_DbManager>("db-manager")
        .WithReference(context.Database)
        .WaitFor(context.Database)
        .PublishAsAzureContainerApp((infra, app) => {
            app.Name = $"ca-{context.GroupIdentifier}-dbm";
        });

    return builder;
}

Update DynaplexContext.cs:

public IResourceBuilder<ProjectResource>? DbManager { get; set; }

Update AddComponent to wait for DbManager:

// In DynaplexComponentExtensions.cs
public static IResourceBuilder<ProjectResource> AddComponent<TProject>(...)
{
    // ... existing code ...

    // If component has database, wait for the unified DbManager
    if (metadata.HasDatabase && context.DbManager != null) {
        component.WaitFor(context.DbManager);
    }

    return component;
}

Deprecate WithMigrator:

[Obsolete("Use AddDynaplexDbManager() instead of per-component migrators")]
public static IResourceBuilder<ProjectResource> WithMigrator<TMigrator>(...)
{
    // Log warning or throw during transition period
}

Stage 5: Update AppHost

File: projects/bbu-rfid/src/Acsis.Dynaplex.Projects.BbuRfid/AppHost.cs

Before:

builder.AddComponent<Acsis_Components_CoreData>(ComponentIndex.CoreData.Metadata)
    .WithMigrator<Acsis_Components_CoreData_DbMigrator>();
builder.AddComponent<Acsis_Components_Prism>(ComponentIndex.Prism.Metadata)
    .WithMigrator<Acsis_Components_Prism_DbMigrator>();
// ... 11 more WithMigrator calls

After:

builder.AddDynaplexDatabase();
builder.AddDynaplexDbManager();  // Single unified migrator

builder.AddComponent<Acsis_Components_CoreData>(ComponentIndex.CoreData.Metadata);
builder.AddComponent<Acsis_Components_Prism>(ComponentIndex.Prism.Metadata);
// No more .WithMigrator<>() calls!

Stage 6: Cleanup Old DbMigrators

Remove these 13 projects:

engines/bbu/src/Acsis.Dynaplex.Engines.Bbu.DbMigrator/
engines/catalog/src/Acsis.Dynaplex.Engines.Catalog.DbMigrator/
engines/core-data/src/Acsis.Dynaplex.Engines.CoreData.DbMigrator/
engines/events/src/Acsis.Dynaplex.Engines.Events.DbMigrator/
engines/file-service/src/Acsis.Dynaplex.Engines.FileService.DbMigrator/
engines/identity/src/Acsis.Dynaplex.Engines.Identity.DbMigrator/
engines/iot/src/Acsis.Dynaplex.Engines.Iot.DbMigrator/
engines/printing/src/Acsis.Dynaplex.Engines.Printing.DbMigrator/
engines/prism/src/Acsis.Dynaplex.Engines.Prism.DbMigrator/
engines/spatial/src/Acsis.Dynaplex.Engines.Spatial.DbMigrator/
engines/system-environment/src/Acsis.Dynaplex.Engines.SystemEnvironment.DbMigrator/
engines/transport/src/Acsis.Dynaplex.Engines.Transport.DbMigrator/
engines/workflow/src/Acsis.Dynaplex.Engines.Workflow.DbMigrator/

Also remove:

  • strata/project-templates/DbMigrator/

Critical Files to Modify

File Changes
EngineIndexGenerator.cs Parse HasDatabase/Schema, generate DatabaseEnginesInOrder
DynaplexInfrastructureExtensions.cs Add AddDynaplexDbManager()
DynaplexEngineExtensions.cs Update AddEngine to wait for DbManager, deprecate WithMigrator
DynaplexContext.cs Add DbManager property
AppHost.cs (bbu-rfid) Remove WithMigrator calls, add AddDynaplexDbManager
13x EngineManifest.cs Add HasDatabase and Schema fields
Directory.Build.props Add MSBuild properties for Database project paths

New Files to Create

File Purpose
engines/db-manager/src/Acsis.Dynaplex.Engines.DbManager/ New component directory
Acsis.Dynaplex.Engines.DbManager.csproj Project file with all Database references
EngineManifest.cs Declares db-manager component
Program.cs Registers all DbContexts
MigrationWorker.cs Orchestrates migrations in dependency order

Benefits

  1. Solves deployment size issue - 1 container app instead of 13
  2. Architectural improvement - Engines declare database needs via manifest
  3. Simpler AppHost - No more .WithMigrator<>() boilerplate
  4. Consistent with manifest pattern - Same pattern as permissions, dependencies
  5. Type-safe - Source generator ensures compile-time correctness
  6. Maintainable - Single place for all migration logic

Risks & Mitigations

Risk Mitigation
Migration order bugs Use topological sort; existing Dependencies already define order
Seeders fail Seeders are already idempotent; DbManager can run them after migrations
Rollback needed Keep old DbMigrator projects until unified approach is proven