Documentation

fsds/bbu-rfid-azure-deployment-auth-fix-1.md

Quick Operational Fix Plan: Azure Authentication Failures

Problem Summary

Two critical authentication failures are preventing the BBU RFID deployment from functioning:

Problem 1: IoT Component - Blob Storage 403 Forbidden

  • Error: Azure.RequestFailedException: This request is not authorized to perform this operation. ErrorCode: AuthorizationFailure
  • Location: IoT component DatalogicBlobProcessor trying to access storage account saacsiscorpbbuatdev
  • Root Cause: IoT component using hardcoded connection string with an expired/rotated storage account key
  • Impact: Datalogic camera blob processor crashes on startup, entire IoT service fails

Problem 2: BBU Component - PostgreSQL Authentication Failed

  • Error: Npgsql.PostgresException: 28P01: password authentication failed for user "bbu_identity-aszxig73dxx3m"
  • Location: BBU component trying to connect to PostgreSQL
  • Root Cause: Connection string in Key Vault uses managed identity name instead of admin username
  • Impact: BBU service cannot connect to database, MQTT processing fails

Quick Fix Strategy

This plan focuses on immediate operational fixes with minimal code changes. The approach is:

  1. Fix configuration issues directly in Azure Portal / Key Vault
  2. Update hardcoded credentials in appsettings.json files
  3. Restart affected services
  4. Verify functionality

Problem 1 Fix: IoT Blob Storage 403 Forbidden

Diagnostic Steps

Step 1.1: Identify the actual storage account

  • Check Azure Portal resource group rg-elbbudev
  • Find storage account (likely named storage<uniquestring> based on Bicep)
  • Verify this is NOT saacsiscorpbbuatdev (which appears to be dev/test account)

Step 1.2: Verify IoT managed identity has storage roles

  • Find managed identity: iot_identity-<uniquestring>
  • Check role assignments on the actual storage account
  • Should have: Storage Blob Data Contributor role
  • NOTE: There is NO iot-roles-storage module in the infrastructure, which means IoT identity does NOT have storage access!

Step 1.3: Check if storage account key access is disabled

  • Storage account property: allowSharedKeyAccess: false (from storage.module.bicep line 13)
  • This means connection strings will NOT work - managed identity is required!

Root Cause Analysis

The IoT component is configured to use a hardcoded connection string pointing to saacsiscorpbbuatdev, but:

  1. The actual deployed storage account is different (generated name)
  2. Storage account has allowSharedKeyAccess: false, so connection strings don't work
  3. IoT managed identity has NO storage role assignments (missing infrastructure module)
  4. The storage account key in the hardcoded string may also be expired/rotated

Fix Options (Fastest to Slowest)

OPTION A: Enable Aspire Blob Storage Integration (RECOMMENDED - 5 minutes)

  • IoT Program.cs already has commented-out Aspire blob integration (lines 27-32)
  • IoT Bicep MISSING ConnectionStrings__blobs environment variable
  • Quick fix: Add storage endpoint to IoT container app environment

OPTION B: Rotate Storage Key and Update appsettings.json (10 minutes)

  • Get valid storage account key from Azure Portal
  • Update hardcoded connection string in IoT appsettings.json
  • Requires re-enabling allowSharedKeyAccess: true on storage account
  • NOT RECOMMENDED: Goes against infrastructure-as-code principles

OPTION C: Create IoT Storage Role Assignment (30 minutes + redeploy)

  • Create iot-roles-storage Bicep module (copy from bbu-roles-storage)
  • Add module reference to main.bicep
  • Redeploy infrastructure
  • Uncomment Aspire blob client code in IoT Program.cs
  • BETTER LONG-TERM but requires full redeploy

Step 1.4: Add blob storage connection to IoT container app

Edit file: /Users/dcastonguay/source/acsis/portfolio/acsis-core/projects/bbu-rfid/src/Acsis.Dynaplex.Projects.BbuRfid/infra/iot/iot.module.bicep

Add parameter (after line 24):

param storage_outputs_blobendpoint string

Add to IoT container environment variables (after line 120):

{
  name: 'ConnectionStrings__blobs'
  value: storage_outputs_blobendpoint
}

Update iot.tmpl.bicepparam:

param storage_outputs_blobendpoint = '{{ .Env.STORAGE_BLOBENDPOINT }}'

Update main.bicep iot module call to include storage parameter:

storage_outputs_blobendpoint: storage.outputs.blobEndpoint

Step 1.5: Create iot-roles-storage module

Create file: projects/bbu-rfid/src/Acsis.Dynaplex.Projects.BbuRfid/infra/iot-roles-storage/iot-roles-storage.module.bicep

Copy from bbu-roles-storage.module.bicep and change bbu to iot.

Add to main.bicep (after line 77):

module iot_roles_storage 'iot-roles-storage/iot-roles-storage.module.bicep' = {
  name: 'iot-roles-storage'
  scope: rg
  params: {
    location: location
    principalId: iot_identity.outputs.principalId
    storage_outputs_name: storage.outputs.name
  }
}

Step 1.6: Remove hardcoded connection string from appsettings.json

Edit: /Users/dcastonguay/source/acsis/portfolio/acsis-core/engines/iot/src/Acsis.Dynaplex.Engines.Iot/appsettings.json

Change line 98:

"BlobStorageConnectionString": null,

Or comment it out to use Aspire-injected BlobServiceClient.

Step 1.7: Uncomment Aspire blob client in Program.cs

Edit: /Users/dcastonguay/source/acsis/portfolio/acsis-core/engines/iot/src/Acsis.Dynaplex.Engines.Iot/Program.cs

Uncomment lines 30-32:

if(!isOpenApiBuild) {
	builder.AddAzureBlobServiceClient("blobs");
}

Step 1.8: Redeploy

cd /Users/dcastonguay/source/acsis/portfolio/acsis-core/projects/bbu-rfid/src/Acsis.Dynaplex.Projects.BbuRfid
azd deploy

Problem 2 Fix: BBU PostgreSQL Authentication Failed

Diagnostic Steps

Step 2.1: Check Key Vault secret value

Azure Portal:

  • Navigate to Key Vault (name from POSTGRES_KV_NAME output)
  • Check secret connectionstrings--acsis
  • Current value likely: Host=psql-elbbudev-db.postgres.database.azure.com;Username=bbu_identity-aszxig73dxx3m;Password=...;Database=acsis
  • Problem: Username is managed identity name, NOT admin username

Step 2.2: Get correct admin username

From Azure deployment parameters:

  • Check main.parameters.json or Azure Portal PostgreSQL server
  • Parameter: postgres_admin_user (passed to postgres.module.bicep)
  • Likely value: acsis_admin or postgres_admin or similar

Step 2.3: Get current admin password

Password is stored as secure parameter in deployment.

  • Check Azure Key Vault for postgres-admin-password secret (if stored)
  • Or retrieve from azd env get-values output
  • Or from deployment parameters that were used

Root Cause Analysis

The Bicep template at postgres/postgres.module.bicep line 64 correctly creates the connection string with admin credentials:

value: 'Host=${postgres.properties.fullyQualifiedDomainName};Username=${administratorLogin};Password=${administratorLoginPassword}'

However, the BBU logs show connection attempts using username bbu_identity-aszxig73dxx3m (the managed identity name).

This means EITHER:

  1. The Key Vault secret was manually edited and is wrong
  2. The deployment parameter administratorLogin was set to the managed identity name by mistake
  3. The connection string was overridden somewhere in the deployment process

Fix Options (Fastest to Slowest)

OPTION A: Update Key Vault Secret Directly (2 minutes) - RECOMMENDED FOR IMMEDIATE FIX

  • Get correct admin username and password from deployment parameters
  • Update connectionstrings--acsis secret in Key Vault with correct values
  • Restart BBU container app
  • FASTEST but doesn't fix root cause if deployment is wrong

OPTION B: Redeploy with Correct Parameters (10 minutes)

  • Verify main.parameters.json has correct postgres_admin_user value
  • Re-run azd deploy to regenerate secrets
  • More complete fix but takes longer

Step 2.4: Get admin credentials

From terminal:

cd /Users/dcastonguay/source/acsis/portfolio/acsis-core/projects/bbu-rfid/src/Acsis.Dynaplex.Projects.BbuRfid
azd env get-values | grep -i postgres

Look for:

  • POSTGRES_ADMIN_USER or similar
  • Admin password (might be in secure values)

OR check Azure Portal:

  • PostgreSQL Flexible Server: psql-elbbudev-db
  • Settings → "Server parameters" shows admin username

Step 2.5: Update Key Vault secret

Azure Portal:

  1. Navigate to Key Vault (check POSTGRES_KV_NAME output from deployment)
  2. Secrets → connectionstrings--acsis
  3. Click "+ New Version"
  4. Set value to:
Host=psql-elbbudev-db.postgres.database.azure.com;Username=<CORRECT_ADMIN_USER>;Password=<ADMIN_PASSWORD>;Database=acsis
  1. Save

Step 2.6: Restart BBU container app

Azure Portal:

  1. Navigate to Container App: ca-elbbudev-bbu-com
  2. Click "Restart"
  3. Or via CLI:
az containerapp restart --name ca-elbbudev-bbu-com --resource-group rg-elbbudev

Step 2.7: Verify connection

Check container app logs:

az containerapp logs show --name ca-elbbudev-bbu-com --resource-group rg-elbbudev --follow

Should see BBU initialization succeed without password authentication errors.

Problem 2 Alternative: Check Deployment Parameters

If Key Vault secret is correct but issue persists, the deployment parameters may be wrong.

Step 2.8: Verify deployment parameters

Check file: /Users/dcastonguay/source/acsis/portfolio/acsis-core/projects/bbu-rfid/src/Acsis.Dynaplex.Projects.BbuRfid/infra/main.parameters.json

Look for:

{
  "postgres_admin_user": {
    "value": "???"
  },
  "postgres_admin_password": {
    "value": "???" // or reference to key vault
  }
}

If postgres_admin_user is set to bbu_identity-aszxig73dxx3m or similar, that's the problem!

Fix: Change to proper admin username (e.g., acsis_admin) and redeploy.

Verification Steps

Verify IoT Component

  1. Check IoT container logs for successful startup:
az containerapp logs show --name ca-elbbudev-iot-com --resource-group rg-elbbudev --follow

Expected: Datalogic Camera Blob Processor starting with polling interval 15s
No 403 errors!

  1. Verify blob storage role assignment:
az role assignment list --assignee <iot-identity-principal-id> --scope /subscriptions/<sub>/resourceGroups/rg-elbbudev/providers/Microsoft.Storage/storageAccounts/<storage-name>

Should show "Storage Blob Data Contributor" role.

  1. Test blob upload manually:
  • Upload test XML file to storage container bbu-image
  • Check IoT logs for processing messages

Verify BBU Component

  1. Check BBU container logs for successful database connection:
az containerapp logs show --name ca-elbbudev-bbu-com --resource-group rg-elbbudev --follow

Expected: BBU initialization completed successfully
No password authentication errors!

  1. Test MQTT processing:
  • Send test MQTT message to broker
  • Verify BBU processes and saves to database
  • Check bbu.zebra_rfid_reads table for new records

Post-Fix: Remove Hardcoded Credentials

Update BBU appsettings.json

File: /Users/dcastonguay/source/acsis/portfolio/acsis-core/engines/bbu/src/Acsis.Dynaplex.Engines.Bbu/appsettings.json

Line 26 has same hardcoded blob storage connection string:

"BlobStorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=saacsiscorpbbuatdev;AccountKey=..."

This should be removed since BBU already uses Aspire blob integration (Program.cs line 27).

Change to:

"BlobStorageConnectionString": null,

Or remove the property entirely (OracleProcessor will use injected BlobServiceClient).

Estimated Time to Fix

  • Problem 1 (IoT Blob Storage): 15-20 minutes

    • Edit 4 Bicep files
    • Edit 2 C# files
    • Redeploy infrastructure
    • Verify

  • Problem 2 (BBU PostgreSQL): 5 minutes

    • Get correct credentials
    • Update Key Vault secret
    • Restart container app
    • Verify

Total Quick Fix Time: ~25 minutes

Risk Assessment

Low Risk

  • Updating Key Vault secrets (non-destructive, can revert)
  • Restarting container apps (quick recovery)
  • Adding storage role assignments (additive, doesn't break existing)

Medium Risk

  • Removing hardcoded connection strings (requires code redeploy)
  • Uncommenting Aspire blob client (minor code change)

High Risk

  • None (this is operational fix, not structural changes)

Success Criteria

  • IoT container starts without 403 Forbidden errors
  • IoT DatalogicBlobProcessor successfully polls blob storage
  • BBU container connects to PostgreSQL successfully
  • BBU initialization completes without password errors
  • MQTT messages are processed and saved to database
  • No hardcoded credentials remain in appsettings.json files
  • All managed identity role assignments are in place

Follow-Up Actions (Not Part of Quick Fix)

  1. Remove all hardcoded credentials from source control

    • IoT appsettings.json line 98
    • BBU appsettings.json line 26
    • Commit with sanitized values

  2. Add deployment validation

    • Script to check Key Vault secret format before deployment
    • Validate username is NOT a managed identity name

  3. Document storage account architecture

    • Clarify what saacsiscorpbbuatdev is used for
    • Document proper storage account for each environment

  4. Improve error messages

    • Add better logging in DatalogicBlobProcessor to show which storage account is being used
    • Log connection string format (without password) in BBU startup