Ideal Architecture — Event-Driven Now-Playing Pipeline

Status: Proposal (not implemented)
Date: 2026-04-06
Context: Documents the optimal architecture for the ICY collection pipeline if rebuilt from scratch, without cost constraints. Serves as a north-star reference for future refactoring decisions.

⚠️ Historical note: CosmosDB was decommissioned in April 2026. The database now runs on MongoDB Atlas. Cost comparisons and Cosmos DB references below reflect the pre-migration state.

Problem Statement

The current architecture couples ICY probing, DB writes, enrichment, and API reads in two processes (API + Worker) sharing a single Cosmos DB RU budget. This causes:

The fundamental issue is structural: backpressure must be architectural (queue-based), not hand-coded (semaphores).

Target Architecture

High-Level Data Flow

Design Principles

1. Separation of concerns — Each component has a single responsibility: probe, persist, enrich, push, or serve.
2. Structural backpressure — Queue depth regulates throughput. No hand-coded semaphores.
3. Fault isolation — A component failure doesn't cascade. Messages stay in the queue until the consumer recovers.
4. Independent scaling — Each component scales by its own bottleneck (radio count, queue depth, client connections, request rate).
5. At-least-once delivery — Every write is a queue consumer with automatic retry. No fire-and-forget.

Component Specifications

A. ICY Probe Service

Event schema:

interface NowPlayingProbed {
  radioId: string;
  streamUrl: string;
  rawIcyTitle: string; // original ICY metadata string
  parsedArtist?: string;
  parsedTrack?: string;
  station?: string;
  genre?: string;
  bitrate?: number;
  probedAt: Date; // UTC timestamp of probe
  probeLatencyMs: number; // network round-trip time
}

Behavior:

• Probes all assigned radios every ~60 seconds
• Concurrency: 50 per replica (configurable)
• Timeout: 8 seconds per stream
• Emits one event per radio per cycle (including "no change" marker for monitoring)
• Does NOT perform change detection — that's the writer's job

B. Azure Service Bus (Event Bus)

Why Service Bus over BullMQ/Redis:

• Dead-letter queues with inspection (no lost messages)
• Message sessions (ordered processing per radio)
• Built-in exponential retry
• Competing consumers (multiple workers on one subscription)
• No single-point-of-failure Redis
• KEDA scaler integration

C. Write Workers

Behavior:

1. Receive NowPlayingProbed message
2. Compare with cached previous track in Redis (now-playing:{radioId})
3. If unchanged → ack message, update now_playing_updated_at timestamp only
4. If changed:
   • Radio.updateOne() — set now_playing, now_playing_updated_at
   • IcyHistory.create() — append to history
   • Update Redis cache with new track
5. Ack message after successful write

Backpressure: Service Bus prefetch count = write concurrency. If Cosmos is slow (429), worker naturally slows → messages buffer in the queue. No burst, no data loss.

D. Enrichment Workers

Behavior:

1. Receive NowPlayingProbed message (only on track change — use SQL filter on subscription)
2. Look up MusicLink by (artist, track):
   • If cached in DB → skip
   • If not → query Spotify API → upsert MusicLink
3. Increment TrackRadioStats for the radio+track pair
4. Rate-limit Spotify calls: token bucket at 2 req/s per worker

Completely decoupled: If Spotify is down, messages retry in Service Bus with exponential backoff. Zero impact on now-playing freshness.

E. Real-Time Push Service

Behavior:

1. Receive NowPlayingProbed message (track change only — SQL filter)
2. Broadcast to SignalR group radio:{radioId}
3. Connected clients receive update immediately

Replaces: Current polling model where the app hits /api/radios/:id/now-playing every 30–60 seconds.

F. API (Read Gateway)

Key difference from current: The API has zero write contention with the collector. Reads go to a read replica or Redis cache.

Database Architecture

Option 1: Cosmos DB with Dedicated Throughput per Container

Advantage: Each container has its own RU budget. Writes to icy-history cannot starve reads on radios.

Option 2: PostgreSQL (Recommended)

Advantages over Cosmos DB:

• No RU budgeting — throughput scales with compute tier
• Connection pooling eliminates connection storms
• Read replicas for zero-contention reads
• Familiar SQL, better tooling, simpler operations
• Significantly lower cost at this scale
• No 429 errors — ever

Scaling Model

How This Solves Current Problems

Comparison: Current vs Ideal

Migration Path (Incremental)

If pursuing this architecture, a phased migration avoids a risky rewrite:

Phase 1 — Extract probe from write (lowest risk)

• Move ICY stream probing into a standalone Container App
• Probe emits events to Service Bus instead of writing to DB directly
• Writer still runs in the current Worker process, consuming from Service Bus
• Benefit: Probing is decoupled. Probe latency drops. DB writes become queue-driven.

Phase 2 — Separate enrichment

• Move MusicLink/Spotify enrichment to a separate Service Bus subscription + worker
• Benefit: Enrichment failures/slowness no longer affect collection freshness.

Phase 3 — Add real-time push

• Add SignalR integration consuming a realtime subscription
• Update Flutter app to receive push updates instead of polling
• Benefit: Now-playing latency drops from ~60s to ~2s.

Phase 4 — Split read/write DB paths

• Add Cosmos read replica (or migrate to PostgreSQL)
• Route API reads to replica/separate compute
• Benefit: Zero read/write contention. No more API 500s during collection.

Phase 5 — Remove legacy code

• Delete semaphore logic, BullMQ ICY queue, inline write paths
• Benefit: Simpler codebase, easier to reason about.

Technology Choices

Appendix: Event Flow Timeline

A typical 60-second cycle with ~2000 radios:

Key timing: Client sees track change at T+16s (vs T+60–120s in current architecture).

Cost Analysis by Environment

All prices are Azure West Europe retail pricing (April 2026). Costs include compute, messaging, database, cache, monitoring, and networking. Prices exclude tax and reserved-instance discounts.

Pricing Assumptions

Dev Environment

Minimal footprint. Service Bus Standard (not Premium). Database scaled down. SignalR free tier. Probe runs every 5 min. Single replica of everything.

Staging Environment

Full architecture with reduced scale. Service Bus Standard (Premium sessions not needed for testing). One read replica (PostgreSQL) or per-container throughput (Cosmos). Probe at production frequency (every 1 min).

Production Environment

Full redundancy. Service Bus Premium (message sessions for per-radio ordering, higher throughput). Multi-replica probe. DB sized for 2000+ radios × 1 min cycle. SignalR scaled for concurrent users.

Cost Summary: All Environments

Cost vs Current: Where the Money Goes

Cost Optimization Levers

If the ideal architecture is too expensive, these levers can reduce cost while keeping the benefits: