Documentation

adrs/059-sap-integration-architecture.md

ADR-059: SAP Integration Architecture

Status

Accepted (Stages 1-5 complete)

Context

Acsis operates in warehouse and logistics environments where SAP is the dominant ERP system. The erp-integration strata library currently supports Oracle LogFire (OBLPN XML parsing), consumed by the BBU engine's OracleProcessor. We need to extend this to support SAP integration as a first-class capability.

Business Requirements

  1. Raw data ingestion — Receive IDocs from SAP and store them in raw tables for data lake, audit, and backup purposes (following the IoT engine's proven raw data capture pattern).
  2. Standard IDoc models — Comprehensive C# models for common logistics IDoc types, enabling type-safe parsing and processing.
  3. Configurable mapping — A per-tenant mapping framework that transforms SAP data into Acsis domain objects (Catalog items, Transport shipments, Spatial locations, etc.), with customer-specific customization.
  4. Workflow triggers — When specific IDocs arrive, automatically trigger workflows. This establishes the event-driven workflow pattern beyond the current time-based/expiry triggers.

Current State

  • strata/erp-integration/ — LogFire-only: OBLPN XML models + parser. No SAP code.
  • engines/bbu/ — OracleProcessor (3600+ lines) handles LogFire → domain transformation, tightly coupled to BBU business logic.
  • engines/iot/ — Proven raw data capture pattern: raw tables with RawJson columns, capture services, replay/processing flags, tenant isolation.
  • engines/workflow/ — Three predefined event types (Move, Status Change, Arrival), time-based trigger configuration, no event subscription/dispatch mechanism yet.

SAP Integration Patterns

SAP systems exchange data via several mechanisms:

Method Format Use Case SAP Dependency
IDoc (file) XML or flat-file Batch integration, most common None — just file parsing
IDoc (tRFC) Binary/XML over RFC Real-time push SAP NCo library (not redistributable)
BAPI/RFC Function calls Direct read/write to SAP SAP NCo library
OData JSON/XML over HTTP S/4HANA modern API None — standard HTTP

Decision: Start with IDoc file parsing (zero SAP dependencies, same pattern as LogFire). OData support is a natural second phase. RFC/BAPI requires the SAP NCo library and customer-specific setup, so it's a future phase.

Architecture

Layer Overview

┌─────────────────────────────────────────────────────────────┐
│  Engines (consumers)                                         │
│  ┌──────────┐ ┌───────────┐ ┌─────────┐ ┌────────────────┐ │
│  │ Catalog  │ │ Transport │ │ Spatial │ │ Workflow       │ │
│  │ (items)  │ │ (ships)   │ │ (locs)  │ │ (triggers)     │ │
│  └────▲─────┘ └─────▲─────┘ └────▲────┘ └──────▲─────────┘ │
│       │             │            │              │           │
│  ┌────┴─────────────┴────────────┴──────────────┴─────────┐ │
│  │  Spile Engine (new)                                      │ │
│  │  - Raw IDoc storage (data lake)                         │ │
│  │  - IDoc processing pipeline                             │ │
│  │  - Mapping execution                                    │ │
│  │  - Event dispatch (→ workflow triggers)                  │ │
│  └────▲────────────────────────────────────────────────────┘ │
└───────┼─────────────────────────────────────────────────────┘
        │
┌───────┼─────────────────────────────────────────────────────┐
│  Strata (foundations)                                        │
│  ┌────┴────────────────────────────────────────────────────┐ │
│  │  erp-integration library                                │ │
│  │  ┌──────────────┐ ┌────────────┐ ┌───────────────────┐ │ │
│  │  │ IDoc Models  │ │ Parsers    │ │ Mapping Framework │ │ │
│  │  │ (SAP types)  │ │ (XML/flat) │ │ (definitions +    │ │ │
│  │  │              │ │            │ │  transforms)      │ │ │
│  │  └──────────────┘ └────────────┘ └───────────────────┘ │ │
│  │  ┌──────────────────────────────────────────────────┐   │ │
│  │  │ LogFire Models + Parser (existing)               │   │ │
│  │  └──────────────────────────────────────────────────┘   │ │
│  └─────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘

IDoc Structure

Every SAP IDoc has the same envelope:

┌─────────────────────────┐
│ Control Record (EDI_DC)  │  — Metadata: sender, receiver, message type, IDoc type
├─────────────────────────┤
│ Data Records (EDI_DD)    │  — Hierarchical segments specific to the IDoc type
│   Segment E1EDL20        │    (e.g., delivery header)
│     Segment E1EDL24      │    (e.g., delivery item)
│       Segment E1EDL41    │    (e.g., partner data)
│   ...                    │
├─────────────────────────┤
│ Status Records (EDI_DS)  │  — Processing status history
└─────────────────────────┘

Raw Storage Model (follows IoT pattern)

// Raw IDoc table — stores everything that arrives, unmodified
public class RawIdoc
{
    public Guid Id { get; set; }
    public Guid TenantId { get; set; }

    // Normalized fields for indexing
    public string MessageType { get; set; }     // e.g., "DESADV", "MATMAS"
    public string IDocType { get; set; }        // e.g., "DELVRY05", "MATMAS05"
    public string IDocNumber { get; set; }      // SAP IDoc number
    public string SenderSystem { get; set; }    // SAP system ID
    public string SenderPartner { get; set; }   // SAP partner number

    // Complete original payload
    public string RawContent { get; set; }      // Full XML or flat-file content
    public string ContentFormat { get; set; }   // "xml" or "flat"

    // Timestamps
    public DateTime SapCreatedAt { get; set; }  // When SAP created the IDoc
    public DateTime CapturedAt { get; set; }    // When we received it

    // Processing status (IoT replay pattern)
    public DateTime? ProcessedAt { get; set; }  // null = not yet processed
    public string? ProcessingError { get; set; } // Error details if processing failed
}

Mapping Framework

The mapping system allows per-tenant configuration of how SAP fields map to Acsis domain objects.

// Stored per-tenant mapping configuration
public class ErpMappingDefinition
{
    public Guid Id { get; set; }
    public Guid TenantId { get; set; }
    public string SourceSystem { get; set; }    // "SAP", "LogFire", etc.
    public string MessageType { get; set; }     // "DESADV", "MATMAS", etc.
    public string TargetDomain { get; set; }    // "Catalog.Item", "Transport.Shipment", etc.
    public string MappingRulesJson { get; set; } // JSON mapping rules
    public bool IsActive { get; set; }
}

// Example mapping rules (JSON):
// {
//   "fields": [
//     { "source": "E1MARAM.MATNR", "target": "ItemType.Name", "transform": "trim" },
//     { "source": "E1MARAM.MAKTX", "target": "ItemType.Description" },
//     { "source": "E1MARAM.MEINS", "target": "ItemType.BaseUom", "transform": "sapUomToAcsis" }
//   ],
//   "defaults": {
//     "ItemCategory": "SAP Materials"
//   },
//   "filters": {
//     "E1MARAM.MTART": ["FERT", "HALB", "ROH"]  // Only these material types
//   }
// }

Workflow Trigger Integration

New workflow event type: ERP IDoc Received. When an IDoc is processed, the Spile dispatches an event that the workflow engine can subscribe to.

IDoc arrives → Raw storage → Parse → Map → Domain updates
                                  └──→ Dispatch workflow event
                                        (message type, IDoc type, tenant, mapped entity IDs)

This establishes the pattern for event-driven workflow triggers that can be extended to other event sources beyond SAP.

Implementation Plan

Stage 1: IDoc Foundation (strata/erp-integration) ✓

Goal: IDoc data structures and parsing infrastructure.

Status: Complete

Deliverables:

  • ✓ IDoc envelope models (control record, data record, status record)
  • ✓ Generic IDoc parser interface (IIdocParser) with XML implementation
  • ✓ XML IDoc parser implementation (IdocXmlParser)
  • ✓ First IDoc type models: DESADV/DELVRY (delivery) — most relevant for logistics
  • ✓ Unit tests with sample IDoc XML files

Success criteria: Can parse a DESADV IDoc XML file into strongly-typed C# objects.

Stage 2: Standard IDoc Type Library (strata/erp-integration) ✓

Goal: Comprehensive models for common logistics IDoc types.

Status: Complete

Deliverables:

  • ✓ MATMAS (Material Master) — 9 segment models with SAP metadata attributes
  • ✓ ORDERS (Purchase/Sales Orders) — 9 segment models
  • ✓ WMMBXY (Goods Movements / Inventory) — 4 segment models
  • ✓ DESADV (Delivery — extended from Stage 1) — 9 segment models (header, items, partners, addresses, serial numbers, deadlines, handling units, text headers)
  • ✓ LOIPRO (Production Orders) — 4 segment models
  • ✓ DEBMAS (Customer Master) — 3 segment models
  • ✓ CREMAS (Vendor Master) — 3 segment models
  • ✓ SHPMNT (Shipment) — 5 segment models
  • ✓ SAP metadata attributes: [SapIdocType], [SapSegmentName], [SapFieldName] on all models
  • ✓ Test data XML files for each IDoc type
  • ✓ 296 unit tests covering all IDoc types

Success criteria: All listed IDoc types can be parsed from XML format into strongly-typed C# models with full SAP metadata attribution.

Stage 3: Raw Storage Engine (new engine: spile) ✓

Goal: Receive and store IDocs in raw tables, following the IoT raw data capture pattern.

Status: Complete

Deliverables:

  • ✓ New engine: engines/spile/ with standard Dynaplex component structure (4 projects: main, abstractions, database, API client)
  • ✓ Database schema: RawIdoc table with normalized fields (message_type, idoc_type, document_number, sender_system, sender_partner) + raw_content text column
  • ✓ Indexes: tenant+message_type+captured_at (query), tenant+document_number (lookup), tenant+processed_at filtered (replay), unique tenant+document_number+sender_system (dedup)
  • ✓ SpileService — captures raw IDocs, parses control record metadata via IIdocParser, queries by type/status/document number, supports mark-processed and reset-processing for replay
  • ✓ API endpoints: POST /idocs (capture), GET /idocs (list with filtering), GET /idocs/, GET /idocs/by-docnum/, POST /idocs//mark-processed, POST /idocs//reset-processing
  • ✓ Engine manifest, Aspire AppHost registration, db-manager integration
  • ✓ EF Core Initial migration
  • ✓ 20 unit tests covering capture, query, filtering, mark-processed, reset, edge cases
  • ✓ Tenant isolation via DynaplexDbContext.SetTenantId() pattern

Success criteria: Can POST an IDoc to the API, see it stored in the raw table with normalized fields, and query it back.

Stage 4: Mapping Framework (strata/erp-integration + spile engine) ✓

Goal: Configurable per-tenant mapping from SAP IDoc segments to Acsis domain objects.

Status: Complete

Deliverables:

Mapping Framework (strata/erp-integration):

  • ✓ Mapping rule models: MappingRuleSet, FieldMapping, MappingFilter, MappingResult — JSON-serializable rule definitions with source→target field mapping, optional transforms, default values, and pre-processing filters
  • ✓ MappingEngine — core engine that resolves SEGMENT.FIELD dot-notation paths against IdocDocument segments and applies mapping rules (filters → defaults → field mappings with transforms)
  • ✓ IMappingTransform interface + MappingTransformRegistry — extensible transform pipeline with auto-registered built-in transforms
  • ✓ Built-in transforms: trim (strip leading zeros + whitespace), trimWhitespace, sapDate (YYYYMMDD → yyyy-MM-dd), sapNumber (SAP decimal normalization), uppercase, lowercase
  • ✓ DefaultMappingRules — static default rule sets for MATMAS→Catalog.ItemType (9 fields, filter on FERT/HALB/ROH), DESADV→Transport.Shipment (11 fields), ORDERS→Order (5 fields), WMMBXY→InventoryMovement (8 fields)
  • ✓ 27 unit tests covering field mapping, transforms, defaults, filters, default rules, and JSON serialization

Mapping Definitions (Spile engine):

  • ✓ MappingDefinition entity with tenant isolation, source system, message type, target domain, JSON mapping rules, active flag
  • ✓ EF Core migration: mapping_definitions table in spile schema with composite indexes
  • ✓ MappingDefinitionService — CRUD + active mapping lookup with automatic fallback to DefaultMappingRules (works out-of-the-box without DB configuration)
  • ✓ Admin API endpoints: GET/POST/PUT/DELETE /mappings, POST /mappings/from-default (bootstrap from defaults)

Processing Pipeline (Spile engine):

  • ✓ IdocProcessingService — batch/single IDoc processing: fetch unprocessed → parse via IIdocParser → find mapping (tenant-specific then default fallback) → apply rules via MappingEngine → mark processed with status tracking
  • ✓ Processing API endpoints: POST /processing/batch, POST /processing/
  • ✓ Processing status tracking: ProcessedAt and ProcessingError on RawIdoc
  • ✓ 11 unit tests covering batch processing, single processing, tenant override, filtered IDocs, parse failures, missing mappings
  • ✓ Kiota API client auto-generated with mapping and processing endpoints

Success criteria: A MATMAS IDoc arrives, gets stored raw, then automatically creates/updates item types in Catalog via configured mapping rules. Different tenants can have different mappings for the same IDoc type.

Stage 5: MATMAS → Catalog ItemType Sync PoC (spile engine) ✓

Goal: End-to-end proof-of-concept: MATMAS IDocs flow through the mapping pipeline and create/update Catalog item types, with Planned Delivery Time stored as a Prism quantity attribute.

Status: Complete

Deliverables:

Sync API Ports (testability layer):

  • ✓ ICatalogSyncPort, IPrismSyncPort, ICoreDataSyncPort — mockable interfaces for external API operations, decoupling domain logic from Kiota-generated clients
  • ✓ KiotaCatalogSyncAdapter, KiotaPrismSyncAdapter, KiotaCoreDataSyncAdapter — thin Kiota-backed implementations registered via DI

Material Sync Service:

  • ✓ MaterialSyncService — orchestrates MATMAS → Catalog ItemType synchronization:
    • Lookup existing item type by name (trimmed MATNR, case-insensitive match)
    • Upsert item type via Catalog API: Name (MATNR), Description (MAKTX), ERPMaterialID (MATNR → triggers Prism identifier registration), ERPType (MTART), IsActive
    • Set Planned Delivery Time (PLIFZ from E1MARCM) as a quantity-type Prism attribute using Duration/Day UoM from CoreData
    • Idempotent attribute definition creation with duplicate keySlug recovery via effective attribute scanning
    • Thread-safe static caching of attribute definition ID and UoM references (SemaphoreSlim)
  • ✓ MaterialSyncResult — typed result with ItemTypeId, Created/Updated flags, AttributeSet, Error

Mapping Rules (strata/erp-integration):

  • ✓ Extended MATMAS default mapping with E1MARAM.MATNR → ERPMaterialID (trim) and E1MARCM.PLIFZ → PlannedDeliveryTime (sapNumber)
  • ✓ Cross-segment resolution: E1MARCM.PLIFZ resolves from child segment of E1MARAM (plant-level data)

Pipeline Wiring:

  • ✓ IdocProcessingService dispatches to MaterialSyncService when targetDomain is Catalog.ItemType and messageType is MATMAS
  • ✓ EngineManifest.Dependencies updated to ["catalog", "prism", "core-data"] (auto-wires .WithReference() in AppHost)
  • ✓ API client registrations: AddAcsisApiClient<CatalogApiClient/PrismApiClient/CoreDataApiClient>()

Tests:

  • ✓ 12 MaterialSyncService unit tests: create/update item types, missing name, planned delivery time set/skipped/non-numeric, field mapping verification, lookup by name (found/not found/case insensitive)
  • ✓ Updated IdocProcessingService tests (4 tests updated for sync dispatch, 43 total Spile tests passing)
  • ✓ 1 new mapping engine test for E1MARCM plant data cross-segment extraction (28 total mapping tests, 324 total ERP integration tests)

Deferred: SAP Standard Price (STPRS from E1MBEWM) attribute — requires currency/monetary quantity support in UoM system (tracked as acsis-core-zvs4).

Success criteria: A MATMAS IDoc arrives, gets parsed and mapped, then creates/updates a Catalog item type with MATNR as identifier and Planned Delivery Time as a Prism attribute.

Stage 6: Workflow Triggers (workflow engine + spile)

Goal: Event-driven workflow triggers when IDocs are processed.

Deliverables:

  • New workflow event type: "ERP IDoc Received" (or more granular per message type)
  • Event dispatch from spile processing pipeline
  • Workflow trigger configuration: "when DESADV arrives for item type X, trigger workflow Y"
  • Workflow subscription mechanism (extends current WorkflowEvent infrastructure)
  • This establishes the general event-driven trigger pattern for the workflow engine

Success criteria: A DESADV IDoc arrives, creates a shipment in Transport, and automatically triggers a configured "Inbound Shipment Processing" workflow.

Stage 7: Flat-File IDoc Parser (strata/erp-integration)

Goal: Parse IDocs in SAP's native flat-file format (fixed-width, segment hierarchy encoded via position).

Deliverables:

  • Flat-file IDoc parser implementation (IIdocParser for flat-file format)
  • Support for SAP's standard fixed-width field layout (segment type, data record format)
  • Flat-file to IdocDocument conversion (reuses all existing typed models)
  • Unit tests with sample flat-file IDoc data

Success criteria: All IDoc types from Stage 2 can be parsed from flat-file format into the same IdocDocument model, producing identical typed objects as XML parsing.

Note: This was moved from Stage 2 to its own stage because flat-file parsing adds complexity without blocking the primary XML-based integration path. Most modern SAP integrations use XML IDocs; flat-file support is needed for legacy file-based interfaces.

Risks and Mitigations

Risk Mitigation
IDoc models may not match customer's SAP version Default models cover standard segments; mapping framework allows field-level customization
SAP flat-file format is complex (fixed-width, segment hierarchy) Start with XML format; flat-file parser deferred to Stage 6
Mapping framework could become over-engineered Start with simple field mapping + transforms; add complexity only when needed
Workflow event dispatch needs reliable delivery Use the same patterns as existing workflow infrastructure; add retry/dead-letter later
No real SAP test data available Build from SAP documentation; flag areas that need customer validation

Future Phases (Not In Scope)

  • OData integration for S/4HANA direct API communication
  • RFC/BAPI via SAP NCo library (requires customer SAP licensing)
  • Outbound IDocs — sending data back to SAP
  • Real-time tRFC — receiving IDocs via live RFC connection
  • LogFire migration — refactoring OracleProcessor to use the new mapping framework

Decision

Proceed with the seven-stage plan. Start with IDoc parsing in the strata library (zero risk, zero dependencies), build upward to raw storage and mapping, and use the SAP integration as the proving ground for event-driven workflow triggers. Flat-file parsing (Stage 7) is deferred until needed, as XML is the dominant format in modern SAP integrations.

Stages 1-5 are complete. The foundation is solid: 8 IDoc type models with 50+ segment models and full SAP metadata attribution (Stages 1-2), a fully functional Spile engine for raw IDoc storage with API endpoints, dedup, replay support, and 20 unit tests (Stage 3), a configurable per-tenant mapping framework with a processing pipeline, 6 built-in transforms, default mapping rules for 4 IDoc types, and 38 additional unit tests (Stage 4), and an end-to-end MATMAS → Catalog ItemType sync PoC with Planned Delivery Time as a Prism quantity attribute, inter-service API wiring, mockable port interfaces for testability, and 12 new unit tests (Stage 5). Next up is Stage 6 (workflow triggers) which will establish event-driven workflow triggers when IDocs are processed.