Audissey Infrastructure & Information Flow

Overview

This document provides a comprehensive view of the Audissey platform infrastructure, covering:

• Mobile app CI/CD pipelines (Flutter)
• Backend services (API, Worker, ML Service)
• Azure infrastructure across environments
• External service integrations
• Data flow and security architecture

Architecture Diagram

Infrastructure Components

1. Developer Workflow & Source Control

Repositories:

• audissey_app - Flutter mobile application (iOS & Android)
• Audissey - Backend services (NestJS API, Worker, ML Service)

Development Flow:

• Developers push code to feature branches
• Push to dev branch triggers CI/CD pipelines
• Push to main branch triggers staging and production releases

2. CI/CD Pipelines (Azure DevOps)

Mobile Pipelines

Flutter CI (azure-pipelines-flutter-ci.yml)

• Trigger: Push to dev branch with changes to .env.*, lib/**, test/**, pubspec.yaml, android/**, or ios/**
• Actions: Code analysis (flutter analyze), unit tests (flutter test)
• Agent: Self-hosted Linux with Docker support

Release Tag (azure-pipelines-release-tag.yml)

• Trigger: Push to dev or main branch
• Actions:
  • Creates environment-specific git tags based on pubspec.yaml version
  • dev branch → audissey_app-dev-v{version}-{shortsha} (unique per commit)
  • main branch → audissey_app-staging-v{version} and audissey_app-prod-v{version}
• Note: Tags trigger downstream CD pipelines

Android/iOS CD Pipelines (dev, staging, prod)

• Trigger: Git tags matching audissey_app-{env}-v*
• Actions:
  • Build signed APK/AAB (Android) or IPA (iOS)
  • Distribute to Firebase App Distribution (dev/staging) or App Stores (prod)
• Agents:
  • Android: Self-hosted Linux (x86_64) with Docker
  • iOS: Self-hosted macOS with Xcode

Backend Pipelines

API Build & Release (azure-pipelines-api-build.yml, azure-pipelines-api-release.yml)

• Trigger: Push to monitored branches
• Actions:
  • Build Docker image (job timeout 90 min, as the worker: a package-lock.json change misses the layer cache and the build plus cache push ran past the old 30 min on 25/10/2026)
  • Deploy to Azure App Service (progressive: dev → staging → prod)
  • Validate Key Vault access via Managed Identity
  • Smoke test /api/health endpoint

Worker Build & Release (azure-pipelines-worker-build.yml, azure-pipelines-worker-release.yml)

• Trigger: Push to monitored branches
• Actions: Build Docker image, smoke test Chromium in it, push to ACR, deploy to Azure Container Apps (dev/staging/prod)
  • Build can take >30 minutes (Nx build + Docker build/push); pipeline job timeout is configured accordingly
  • A failed Chromium smoke test (scripts/worker-chromium-smoke.js) stops the build before the push, so nothing is published and the release does not run
  • Release pipeline uses deterministic image tags from build runID
  • Preflight checks validate managed identity, env vars (MONGO_URI, REDIS_HOST, API_URL, ML_SERVICE_URL, APPLICATIONINSIGHTS_CONNECTION_STRING)
  • Health polling with 20 retries after deployment (expects GET /api/health to return 200)

ML Service Build & Release (azure-pipelines-ml-build.yml, azure-pipelines-ml-release.yml)

• Trigger: Push to monitored branches
• Actions: Build and deploy ML service containers

3. Mobile Application (Flutter)

Flavors & Environments:

• dev (com.audissey.audisseyapp.dev) - .env.dev
• staging (com.audissey.audisseyapp.staging) - .env.staging
• prod (com.audissey.audisseyapp) - .env.prod

Configuration:

• Environment-specific .env.* files (tracked in git, containing safe client IDs)
• Per-flavor Facebook App IDs and Google OAuth Client IDs
• API base URLs pointing to corresponding backend environments

Distribution:

• Dev/Staging: Firebase App Distribution (internal testers)
• Production: Apple App Store and Google Play Store

Social Authentication:

• Facebook Login with flavor-specific App IDs
• Google Sign-In with OAuth 2.0 ID token flow
• Client sends ID token to backend for verification

4. Azure Infrastructure

Region: West Europe

Environments:

• audissey-dev-rg (Development)
• audissey-staging-rg (Staging)
• audissey-prod-rg (Production)
• audissey-shared-rg (Shared services: ACR, Redis, Cosmos DB)
• audissey-rg (Legacy — hosts App Service Plans)

App Service Plans

Per-Environment Resources

App Service (API)

• Service: Azure App Service (Linux, Docker container)
• Names: audissey-api-dev, audissey-api-staging, audissey-api (prod)
• Plans: dev on audissey-plan-dev-ws (B1), staging on audissey-plan-staging (P0v3), prod on audissey-plan-prod-ws (B1)
• Runtime: Node.js NestJS application
• Authentication: Managed Identity for Key Vault access
• VNet Integration: Connected to private VNet for Key Vault access

Worker Service

• Service: Azure Container Apps (Docker container)
• Names: audissey-worker-dev, audissey-worker-staging, audissey-worker (prod)
• Container Apps Environment: audissey-cae-northeu in audissey-staging-rg (North Europe). All 3 environments share one CAE — prod is temporary here while West Europe AKS capacity is exhausted.
• Purpose: Background job processing (BullMQ queues via Redis)
• Identity: SystemAssigned Managed Identity (used for ACR image pulls)
• Ingress: Internal only (not publicly accessible); the worker is a background queue consumer with no inbound traffic requirements
• Scaling: minReplicas=1, maxReplicas=1 per environment
• Queue: Connected to Redis via REDIS_HOST / REDIS_PASSWORD env vars (sourced from Key Vault via pipeline)
• ML service: ML_SERVICE_URL env var — URL of the ML service Container App (e.g. https://audissey-ml-dev.salmonglacier-eb702166.northeurope.azurecontainerapps.io, https://audissey-ml-staging.salmonglacier-eb702166.northeurope.azurecontainerapps.io, https://audissey-ml-prod.salmonglacier-eb702166.northeurope.azurecontainerapps.io) for HTML cleaning and schedule extraction fallback. Required for the scraping pipeline to work; without it the worker defaults to http://localhost:5000 and scraping silently fails. The worker release pipeline validates this env var is present but does not set it — ensure it is configured in the Container App environment variables (set via variable group or manual config).
• API connectivity: Workers enqueue results directly to Redis (BullMQ); the API picks them up. No direct HTTP calls from Worker→API.
• BullMQ config: Shared ioredis instance for all Queue/Worker clients (reduces connections from ~48 to ~17 per process). Options: enableReadyCheck: false, maxRetriesPerRequest: null, lazyConnect: true. See jobs.module.ts.

Admin Dashboard

• Service: Azure App Service (Linux, Docker container)
• Names: audissey-admin-dev, audissey-admin-staging, audissey-admin (prod)
• Purpose: Internal admin panel for managing scraping, radios, and monitoring
• Plans: dev on audissey-plan-dev-ws (B1), staging on audissey-plan-staging (P0v3), prod on audissey-plan-prod-ws (B1)

ML Service (Python)

• Service: Azure Container Apps (shared CAE audissey-cae-northeu in audissey-staging-rg)
• Names: audissey-ml-dev, audissey-ml-staging, audissey-ml-prod
• FQDNs: audissey-ml-dev.salmonglacier-eb702166.northeurope.azurecontainerapps.io, audissey-ml-staging.salmonglacier-eb702166.northeurope.azurecontainerapps.io, audissey-ml-prod.salmonglacier-eb702166.northeurope.azurecontainerapps.io
• Purpose: HTML cleaning, schedule extraction, and NLP processing for the scraping pipeline
• Resources: dev/staging: 1.0 vCPU / 2Gi; prod: 2.0 vCPU / 4Gi
• Scale: minReplicas=0, maxReplicas=2 (KEDA HTTP scaler, concurrentRequests=10)
• Identity: SystemAssigned Managed Identity (used for ACR image pulls)
• Called by: Worker service via ML_SERVICE_URL env var

Key Vault

• Names: audissey-kv-dev, audissey-kv-staging, audissey-kv-production
• Secrets Stored:
  • facebook-app-secret - Facebook App Secret (per environment)
  • facebook-app-id - Facebook App ID (per environment)
  • google-oauth-client-ids - Comma-separated list of OAuth Client IDs
  • google-server-client-id - Server-side Google OAuth Client ID
  • jwt-secret - JWT signing key
  • mongodb-uri - MongoDB connection string
  • Additional service-specific secrets (Podcast Index, etc.)
• Access Control:
  • Network: PublicNetworkAccess=Disabled (all environments), defaultAction=Deny, bypass=None
  • Purge Protection: Enabled on production; staging/dev have soft-delete only
  • Private Endpoint: Accessible only via VNet Private Endpoint
  • Authentication: App Service Managed Identity with Secret Get permissions
• Integration: Secrets loaded at API startup via keyvault.service.ts

Networking

• Per-environment VNets:
  • audissey-vnet-dev (audissey-dev-rg, West Europe)
  • audissey-vnet-staging (audissey-staging-rg, West Europe)
  • audissey-vnet-prod (audissey-prod-rg, West Europe)
• Workers VNet: audissey-vnet-workers-we (audissey-staging-rg, West Europe)
  • Subnet private-endpoints (10.251.2.0/24): private endpoint NICs
  • Subnet appsvc-integration (10.251.3.0/24, Microsoft.Web/serverFarms delegation): for App Service VNet Regional Integration (requires S1+ plan)
  • Subnet containerapps-infra (10.251.0.0/23, Microsoft.App/environments): hosts the Container Apps Environment
• Private DNS Zones: For Key Vault, Redis, and Cosmos DB private link resolution
• Private Endpoints:
  • pe-workers-kv-dev/staging/prod: Key Vault access from VNet
  • pe-workers-redis-we: Redis access from VNet
  • pe-workers-cosmos-mongo: Cosmos DB access from VNet
• Note: API/ML App Services on B1 plans access Redis/MongoDB over the public internet (restricted by firewall rules). Container Apps workers run in the containerapps-infra subnet within the VNet.

5. Data Stores

MongoDB (Cosmos DB MongoDB API)

• Service: Azure Cosmos DB for MongoDB (audissey-cosmos, shared across environments)
• Resource Group: audissey-shared-rg
• Databases: audissey-dev, audissey-staging, audissey-prod
• Purpose: Primary application database
• Connection: Via MONGODB_URI secret from Key Vault
• Access: From API services across all environments
• CosmosDB Limitations (important for schema design):
  • Sparse unique indexes are NOT properly supported. CosmosDB treats null/missing values as
    equal for unique index enforcement, so { unique: true, sparse: true } on optional fields
    (e.g., social auth IDs) will cause E11000 duplicate-key errors when more than one document
    lacks the field. Use non-unique indexes ({ index: true }) on such fields and enforce
    uniqueness at application level instead.
  • Unique indexes cannot be modified on non-empty collections. To fix index definitions,
    the collection must be dropped and recreated (backup data first).
  • Do NOT combine $exists with $nin in the same sub-document filter. The pattern
    { field: { $exists: true, $nin: [null, ''] } } returns 0 results on CosmosDB even when
    matching documents exist. Use explicit $and with separate conditions instead:

    {
      $and: [{ field: { $exists: true } }, { field: { $ne: null } }, { field: { $ne: '' } }];
    }
    

Redis

• Service: Azure Cache for Redis (audissey-redis-we, Standard C0, shared across all 3 environments)
• Version: 6.0 (recommend upgrading to 7.2 for BullMQ ≥ 6.2 compatibility)
• Max clients: 256 (Standard C0 limit)
• Host: audissey-redis-we.redis.cache.windows.net:6380 (TLS 1.2 only, non-SSL port disabled)
• Resource Group: audissey-shared-rg
• Purpose: BullMQ job queue (API enqueues jobs, Container Apps workers consume them) + recommendations cache
• Connection: REDIS_HOST / REDIS_PASSWORD env vars on workers; REDIS_CONNECTION_STRING Key Vault secret on APIs
• Connection sharing: Both API and Worker processes use a single shared ioredis instance for all BullMQ Queue clients. Workers duplicate the shared connection for blocking operations. This reduces per-process connections from ~48 (worker) / ~18 (API) to ~17 / ~2. Total across 3 envs: ~57 connections (vs. 256 limit).
• Network Access:
  • publicNetworkAccess=Enabled with firewall rules (NOT open to all IPs)
  • Firewall rules restrict access to App Service outbound IPs and Container Apps outbound IPs
  • Private endpoint: pe-workers-redis-we in audissey-vnet-workers-we/private-endpoints
  • Important: If App Service plans change or scale, outbound IPs may change and firewall rules need updating.
• BullMQ non-blocking startup: Workers use enableReadyCheck: false + lazyConnect: true + retryStrategy so NestJS completes bootstrap even when Redis is temporarily unreachable

6. External Service Integrations

Facebook Login API

• Purpose: Social authentication
• Configuration:
  • Separate Facebook App per environment (dev, staging, prod)
  • App IDs stored in client .env.* files
  • App Secrets stored in Key Vault facebook-app-secret
• Flow:
  • Client obtains Facebook token
  • Token sent to backend /auth/social/facebook
  • Backend validates via Graph API debug_token or JWT (Limited Login)
  • Returns user data and app JWT

Google OAuth

• Purpose: Social authentication
• Configuration:
  • OAuth Client IDs per platform (Android, iOS, Web)
  • Client IDs list stored in Key Vault google-oauth-client-ids
• Flow:
  • Client obtains Google ID token via google_sign_in
  • Token sent to backend /auth/social/google
  • Backend validates via google-auth-library verifyIdToken
  • Returns user data and app JWT

Podcast Index API

• Purpose: Podcast content and metadata
• Configuration: API key and secret in Key Vault
• Usage: Backend fetches podcast data for app consumption

Sender

• Purpose: Transactional email (passwordless login links, health alerts, schedule-change digest)
• Configuration: SENDER_API_TOKEN secret in the Azure DevOps Variable Groups (applied to the API and Worker Container Apps by the release pipelines)
• Usage: API and Worker call POST https://api.sender.net/v2/message/send through worker-shared/src/services/sender-email.client.ts (see Email com Sender)

Security Architecture

Secret Management

Principles:

• No secrets in source code
• Client applications contain only non-sensitive public IDs
• Server secrets stored exclusively in Azure Key Vault
• Environment-specific secrets per deployment stage

Key Vault Access:

• Network isolation via Private Endpoints (all environments)
• Managed Identity authentication (no credentials in code)
• Principle of least privilege (Secret Get permissions only)

App Service Hardening:

• httpsOnly=true on all App Services (HTTP requests redirect to HTTPS)
• ftpsState=Disabled on all App Services (no FTP/FTPS access; Docker/ACR deployment only)
• minTlsVersion=1.2 on all App Services
• Container Apps ingress set to internal (no public endpoints for workers)
• ACR admin user disabled; pulls via Managed Identity

Pipeline Security:

• Secure Files for signing certificates (Android keystore, iOS provisioning profiles)
• Secret variables in Azure DevOps Variable Groups
• Validation steps ensure required secrets exist before deployment

Authentication Flow

Mobile App → Backend:

1. User authenticates with social provider (Facebook/Google)
2. Client receives provider token/ID token
3. Client sends token to backend /auth/social/{provider}
4. Backend validates token with provider API
5. Backend creates or updates user in MongoDB
6. Backend generates app-specific JWT
7. Client stores JWT for subsequent API requests

Backend → External Services:

• Secrets retrieved from Key Vault on application startup
• Cached securely in memory
• Used for provider API authentication

Deployment Flow

Mobile Deployment

1. Developer pushes to dev or main
2. Flutter CI validates code (if path matches)
3. Release Tag pipeline creates git tags (per environment)
4. CD pipelines triggered by tags:
   • Build signed artifacts
   • Run tests
   • Distribute to Firebase or Stores
5. Testers receive notification (Firebase) or users update via stores

Backend Deployment

API & ML Service (App Service):

1. Developer pushes code
2. Build pipeline creates Docker image, pushes to ACR
3. Release pipeline deploys progressively:
   • Deploy to Dev → smoke test
   • Deploy to Staging → smoke test
   • Manual approval gate
   • Deploy to Production → smoke test
4. App Service pulls new container image from ACR (Managed Identity)
5. Key Vault secrets loaded on startup
6. Health checks validate deployment

Worker (Container Apps):

1. Developer pushes code
2. Build pipeline creates Docker image, pushes to ACR with deterministic tag
3. Release pipeline deploys via az containerapp update:
   • Preflight: validate managed identity + required env vars
   • Deploy to Dev (on dev push) / Staging (on main push) / Prod (manual approval)
   • Ensure KEDA scaler: run scripts/ensure-worker-keda.sh to enforce scale-to-zero + Redis-based autoscaling (idempotent)
   • Health polling with 20 retries after each deployment
4. Container App pulls image from ACR via SystemAssigned Managed Identity
5. KEDA monitors BullMQ wait queues in Redis; scales 0→N replicas on demand

Worker health endpoints (Container Apps):

• GET / (startup probe compatibility; returns 200)
• GET /api/health (standard health endpoint; used by alerts/pipeline polling)
• GET /api/health/alive (minimal liveness probe)

Troubleshooting

Key Vault Access Issues

Symptom: API fails to load secrets from Key Vault
Causes:

• Network: Ensure App Service VNet integration and Private Endpoint are configured
• Identity: Verify Managed Identity is enabled and has Secret Get permissions
• DNS: Check Private DNS Zone resolution for *.vault.azure.net

Resolution:

• Pipeline preflight steps validate Key Vault access before deployment
• Use Azure CLI from VPN/jumpbox for manual Key Vault operations
• Review App Service logs for Key Vault connection errors

Social Login Failures

Facebook "Invalid key hash":

• Ensure correct signing certificate SHA-1 is registered in Facebook App Settings
• Use keytool or aapt dump to extract actual certificate hash
• Dev/staging use CI signing certificate, not local debug certificate

Google "Login canceled":

• Verify OAuth Android client in Google Cloud Console includes:
  • Correct package name (com.audissey.audisseyapp.{dev|staging})
  • CI signing certificate SHA-1
• Ensure backend google-oauth-client-ids includes all platform client IDs

Backend token validation errors:

• Verify FACEBOOK_APP_ID and FACEBOOK_APP_SECRET match the client's Facebook App
• Ensure GOOGLE_OAUTH_CLIENT_IDS includes the requesting client's OAuth Client ID
• Check API logs for specific provider error messages

Pipeline Failures

Flutter CI not triggering:

• Verify push includes files matching trigger.paths.include (.env.*, lib/**, etc.)
• Check Azure DevOps pipeline settings: "Override YAML continuous integration trigger" disabled

CD not triggering:

• Confirm Release Tag pipeline ran and pushed tags
• Verify tag name matches CD pipeline trigger.tags.include pattern
• Check self-hosted agent is online and meets demands

Build failures:

• Review pipeline logs for specific error messages
• Validate secure files and secret variables are configured
• Ensure self-hosted agents have required tools (Flutter, Docker, Xcode)

Related Documentation

• Developer API Overview
• Git Workflow
• Email com Sender
• Scripts Reference