SRS: UC_6.1.1 — CME Reference Data Ingestion & Dynamic Timekeeper

---

Note (split, 2026-09-06): This UC was previously published together with UC_6.1.2 and UC_6.1.3 in a single combined file. Per BA instruction, the three UCs now live in three separate files, one per UC. Round-2 QnA (QnA_init_docs.md) and the UC_6.1.1 readiness audit remain in /docs/BA/UC_6.1-6.3/UC_6.1.1-6.1.3/.

UC_6.1.1 — CME Reference Data Ingestion & Dynamic Timekeeper

1. Overview

No wireframe — this UC is a backend cron job with no UI screen of its own. Its output (CME_Instrument_Details) is consumed by the Market Status Widget UI, specified separately in UC_4.1.3.

---

2. Trigger

Cron step: Scheduled — Daily @ 01:00 UTC. No user or UI trigger — pure background job. By this time, CME has already published the following day's trading schedule.

There is currently no mechanism to trigger an intraday re-run — if CME announces an urgent mid-day calendar change (unscheduled holiday, emergency halt), the platform continues operating on the previous day's ingested data until the next scheduled 01:00 UTC run (confirmed limitation).

Dynamic Timekeeper step (same flow, continuous phase): Not itself cron-triggered — it is the middleware's continuously-running timeline-authority engine, not a scheduled job. It is set in motion by this same flow's own cron step, immediately after that step's database commit: once a given trading day's CME_Instrument_Details row for an asset class is written/updated, the middleware (re)computes that asset class's independent timeline thread for the day from the newly-committed Official_Close_UTC/Halt_Start_UTC/Halt_End_UTC/Session_Open_UTC values. Each whitelisted asset class runs its own timeline thread in parallel, since CME's schedule defines distinct closing boundaries per asset class — see BR_6.1.1.9.

---

3. Pre-conditions

For the cron step:

For the Dynamic Timekeeper step (same flow, per asset class):

---

4. Post-conditions

Cron step

Success (full run):

• CME_Instrument_Details is updated for every symbol in CME_Whitelist where Enabled = TRUE, with both schedule fields (Step A) and margin values (Step B) committed in the same transaction (Step C).
• The stored window is trimmed to exactly TODAY → TODAY + 14 days; rows outside this range are discarded.
• Rolling backup data (maintained by a separate job — see BR_6.1.1.2) is now current as of this run.

Partial failure (some symbols only):

• Symbols that failed to resolve (schedule and/or margin) after retries retain their previous day's stored values — not nulled out. See Alternative Flow Alt-1 / Alt-2.

Full failure (an entire CME service unreachable):

• CME_Instrument_Details is not overwritten for the affected data type this run. The platform continues operating on the existing rolling-window backup — no downtime. Fresh data arrives on the next successful 01:00 UTC run. See Exceptional Flow Exc-1.

Dynamic Timekeeper step (same flow)

Success (timeline generated):

• For each whitelisted asset class with a valid cron-committed row for the current trading day, an independent timeline thread exists holding the 6 sequential milestone timestamps — T-3min, T-2min, T-30sec, Halt_Start_UTC, Halt_End_UTC, Session_Open_UTC — derived from that row. See BR_6.1.1.9.
• Each milestone is exposed for UC_6.1.2 — Margin Enforcement to subscribe/react to when the clock reaches it. The middleware itself does not perform the enforcement action — it only signals that the milestone has been reached.

Degraded (missing/stale reference data for an asset class):

• If the cron step has no valid row for an asset class on the current trading day (e.g. a symbol was skipped by Alt-1/Alt-2, or the whole run hit Exc-1), the middleware falls back to the previous day's already-computed timeline for that asset class rather than generating no timeline at all. See Exceptional Flow Exc-4.

---

5. Basic Flow

This flow follows the exact narrative order of RFQ_ Stack Trading Prop Tech V7.pdf's "Dynamic Timekeeper and Ingestion Architecture" passage (lines 1657-1688): the middleware first establishes itself as the timeline authority via a daily cron task (steps 1-6 below), then continuously derives each asset class's timeline from what that cron just committed (steps 7-11 below). It is presented as one continuous flow, not two separate flows — steps 1-6 are the cron phase, steps 7-11 are the continuous Dynamic Timekeeper phase that the cron phase feeds into.

1. The middleware is the timeline authority. Per RFQ V7, the Node.js middleware "functions as the timeline authority" for all overnight risk enforcement — everything below (the cron acquiring data, and the continuous engine deriving milestones from it) is one mechanism serving that single role, not two independent mechanisms.

2. Prerequisite — read whitelist. The cron reads the static whitelist table:

   SELECT Symbol, Exchange, Category FROM CME_Whitelist WHERE Enabled = TRUE;

   Returns ~31 whitelisted futures symbols. Mid-day edits to this table (Ops enabling/disabling a symbol) only take effect at the next 01:00 UTC run — see BR_6.1.1.3.

3. Cron wakes — 01:00 UTC daily. CME has already published the next trading day's schedule by this time. RFQ V7 describes this as "a single daily cron task that extracts schedules from the CME Reference Data API and commits them directly to the CME_Instrument_Details database table" — the sub-steps below (4a-4c) are where CR-20260827-001's confirmed logic supersedes that single-endpoint description [CHR-76], expanding it into a 3-step worker without changing its role in the overall narrative.

4. Cron task body — acquire and commit CME reference data (CR-20260827-001 confirmed logic [CHR-76], superseding RFQ V7's single-endpoint description at this step only):

   1. Step A — Schedules. [CHR-76] 4a.1. Call GET /v3/tradingHours once — returns the full trading-schedule dataset for all quadrants (e.g. "CME Equities", "CBOT Interest Rates", "NYMEX Energy", "COMEX Metals") and all schedules within each quadrant. There is no filter parameter — the response comes back full and must be trimmed client-side. Each entry contains: quadrant, scheduleNames[] (index-aligned schedule IDs, e.g. {id: 1, name: "RTH"}, {id: 2, name: "ETH"}), schedules (per-day session open/close times and maintenance halts), groupCodes[] (every globexGroupCode that follows this schedule). 4a.2. Trim the response to the window TODAY → TODAY + 14 days. Discard everything outside this range. 4a.3. For each whitelisted symbol (from step 2), call GET /v3/productDetails?masterSymbol={SYMBOL} — one call per symbol, run sequentially, not in parallel. Extract the globexGroupCode field — the sole link between a symbol and its trading schedule. 4a.4. Match: for each symbol, search the trimmed step 4a.2 data for the quadrant whose groupCodes[] array contains that symbol's globexGroupCode. The symbol belongs to that quadrant and follows its schedule.

      • If no quadrant's groupCodes[] contains the symbol's code → see Alternative Flow Alt-2.

   2. Step B — Margins. [CHR-76] For each whitelisted symbol, query the CME Margin Service (MarginServiceApi / RealTimeMargin) for the latest outright maintenance/overnight margin requirement. Same per-symbol, sequential execution pattern as step 4a.3.

   3. Step C — Database Commit. [CHR-76] Within a single database transaction, UPSERT both the resolved schedule timestamps (Step A) and the margin value (Step B) for every successfully-processed symbol into CME_Instrument_Details. See BR_6.1.1.4 (storage shape) and BR_6.1.1.6 (atomicity). Only the fields downstream services actually consume are stored — see BR_6.1.1.7.

5. Cron task ends at commit. Once Step C commits, the newly-written/updated Official_Close_UTC/Halt_Start_UTC/Halt_End_UTC/Session_Open_UTC values for each asset class are what the continuous engine (steps 7-11 below) reads. The cron's own responsibility ends here — it does not itself compute or track any T-3min/T-2min/T-30sec milestone.

6. Continuous phase begins. From this commit onward, the middleware's timeline-authority role continues in an always-running form (not a one-shot batch step, and it does not "complete" the way the cron does) — described by RFQ V7 as: "the enforcement engine continuously reads the Official_Close_UTC, Halt_Start_UTC, Halt_End_UTC, and Session_Open_UTC fields for each whitelisted asset family to map out its execution timeline dynamically."

7. Read committed reference data. For each asset class in CME_Whitelist where Enabled = TRUE, the middleware reads that asset class's current-day row from CME_Instrument_Details — specifically the four fields committed in step 4 (sub-steps 1-3: Step A/B/C). See BR_6.1.1.9.

8. Maintain one independent timeline thread per asset class. Per RFQ V7: "Because the CME schedule dictates distinct closing boundaries per asset class, the middleware manages independent timeline threads for each category." The middleware does not run a single global clock — it manages a separate timeline thread per whitelisted asset class, each computed from that asset class's own row. A change to one asset class's schedule (e.g. a holiday affecting only Energy) does not shift any other asset class's timeline. See BR_6.1.1.9.

9. Derive the 6 sequential milestones for the asset class's timeline thread, all computed as fixed offsets from the 3 reference timestamps read in step 7:

The offset milestones (T-3min/T-2min/T-30sec) are anchored to Official_Close_UTC only — they are not separately derived from Halt_Start_UTC.

10. Signal each milestone as the clock reaches it. As real time crosses each derived timestamp for a given asset class, the middleware signals that milestone (e.g. via an internal event/callback) so that UC_6.1.2's enforcement logic can react. The middleware does not itself perform the enforcement action at this point (it does not freeze order routing, does not run the compliance audit, does not trigger the native failsafe) — it only generates and signals the timeline. What happens at each milestone is UC_6.1.2's responsibility, not this UC's — see BR_6.1.1.10.

11. Re-derive on next cron commit. Each time the cron (step 4) commits a fresh row for an asset class (normally once daily at/after 01:00 UTC), the middleware re-derives that asset class's timeline thread for the new trading day from the updated fields. Until the next commit, the previous day's derived timeline for that asset class remains in effect if still relevant (e.g. late in the current session) — see Exceptional Flow Exc-4.

12. Downstream consumption (out of scope for this UC — cross-reference only). The milestones this flow generates, and the reference data this flow relies on, are consumed by:

    • UC_6.1.2 — Margin Enforcement — owns the T-3min pre-close lockout, the T-2min Universal Drawdown Safety Gate + margin compliance audit, and the Halt/Session Open handling this flow signals. UC_6.1.3 — TRIM_TO_COMPLIANCE owns the T-120s trim loop and T-30s Node.js Active Failsafe backstop that UC_6.1.2 dispatches into when a margin breach is found (per CR-20260906-001, superseding the old T-30sec Native Broker Failsafe milestone label).
    • UC_4.1.3 — Market Status Widget — GET /system/market-status reads CME_Instrument_Details (this flow's cron-committed output) directly to compute the current market phase (Open/Lockout/Closed/Halted) per asset class; it does not depend on this flow's in-memory timeline threads.

---

6. Alternative Flows

Alternative flows represent paths where the ingestion/timekeeping goal is still substantially achieved, through a degraded/partial path rather than the full happy path.

Cron step

• Alt-1 — Per-symbol call failure (Step A.3.3 productDetails or Step B margin call).

  • Context: an individual productDetails or margin-service call fails (timeout, non-200) for one symbol out of the whitelist.
  • The system retries the failed call — a retry mechanism exists, but this rule does not fix an exact retry count or backoff schedule.
  • The run is not aborted — all other symbols continue processing normally through Step C.

• Alt-2 — No matching quadrant found (Step A.3.4).

  • Context: a whitelisted symbol's globexGroupCode does not appear in any quadrant's groupCodes[] array (CME has not yet published the mapping, or the whitelist entry is misconfigured).
  • The system retains that symbol's previous day's schedule mapping (does not null it out) and logs the mismatch. Exception: if that same symbol also had no resolved mapping on the previous day (i.e. there is no valid prior value to retain), the mapping is set to null instead.
  • The run continues normally for all other symbols.

Dynamic Timekeeper step

• Alt-3 — Asset class's reference data unchanged from the previous run.

  • Context: the cron step commits a row for an asset class whose schedule fields happen to be identical to the previous trading day (no holiday, no halt change).
  • The middleware re-derives the same 6 milestones as before — this is not an error path, just a no-op recompute. No alert is needed.

---

7. Exceptional Flows

Exceptional flows represent paths where the ingestion/timekeeping goal is not achieved for the affected scope — the system falls back to stored/previous data rather than enforcing on missing/stale data.

Cron step

• Exc-1 — Full failure of either CME endpoint/service (the entire 01:00 UTC run cannot reach tradingHours, or cannot reach the CME Margin Service at all).

  • The system does not overwrite CME_Instrument_Details for the affected data type this run.
  • The system reads from the rolling 14-day backup already stored in the database — kept current by a separate maintenance job (see BR_6.1.1.2), independent of this cron's own daily run.
  • The platform continues operating on this stored backup data — no downtime. Fresh data arrives automatically on the next successful 01:00 UTC run.

• Exc-2 — Partial failure affecting one or more (but not all) whitelisted symbols.

  • See Alternative Flow Alt-1 — handled as a per-symbol degradation, not a full-run exception.

• Exc-3 — Urgent mid-day CME calendar change (unscheduled holiday, emergency halt announced after this run already completed).

  • No automated intraday update mechanism exists. The platform continues operating on the data ingested at the most recent 01:00 UTC run until the next scheduled run. (Confirmed limitation.)

Dynamic Timekeeper step

• Exc-4 — No valid cron-committed row exists for an asset class on the current trading day (the cron step hit Exc-1/Exc-2/Exc-3 for that asset class, or this is the very first run with no prior data at all).

  • The middleware does not skip generating a timeline for that asset class outright. It falls back to the previous day's already-derived timeline (same milestone offsets, reusing the last known good Official_Close_UTC/Halt_Start_UTC/Halt_End_UTC/Session_Open_UTC) so that enforcement (UC_6.1.2) still has some timeline to act on rather than none.
  • If there is no prior day's timeline either (cold start, brand-new whitelisted asset class), no timeline thread is created for that asset class until the cron step successfully commits at least one row for it — logged and alerted to Slack #risk-ops.

---

8. Business Rules

Cron step — CME Reference Data Ingestion

BR_6.1.1.1: Unified single cron — 3 sequential steps, not two separate schedules

The 01:00 UTC CME Reference Data Ingestion job is a single, unified multi-step worker — not two (or three) separate cron schedules. It executes sequentially within the same daily run: Step A (Schedules) → Step B (Margins) → Step C (Database Commit). This supersedes any reading of the ingestion logic as split cron jobs by data type.

(Source: CR-20260827-001 [CHR-76] — overrides the single-endpoint description in RFQ_ Stack Trading Prop Tech V7.pdf §3.3, lines 3164-3175)

---

BR_6.1.1.2: Rolling backup window — 14 days, maintained by a separate job

The trim/failover window for this ingestion cron is 14 days (TODAY → TODAY + 14). This is the confirmed rule for this UC's ingestion/failover logic specifically — it does not retroactively change any other context that separately cited a 7-day window prior to this change.

A separate maintenance job (independent of the 01:00 UTC ingestion cron) keeps this rolling 14-day window of market open/close dates always current in the database, so that Exceptional Flow Exc-1 always has fresh-enough fallback data to read from.

(Source: CR-20260827-001 [CHR-76] — supersedes the 7-day figure in QnA STAGE2-033 pt.3, References/QnA from clients/03_STAGE2_DASHBOARD_EVALUATION/QnA_STAGE2_DASHBOARD_EVALUATION.md, confirmed 2026-08-05, for this UC's logic)

---

BR_6.1.1.3: Whitelist scope and edit latency

Only symbols in CME_Whitelist with Enabled = TRUE are processed. CME_Whitelist is a plain database table; Ops edits it directly via the Admin/BPS UI or SQL — no deployment is required. A change made mid-day only takes effect starting at the next 01:00 UTC run; there is no real-time propagation mechanism.

---

BR_6.1.1.4: Schedule-symbol matching algorithm

Each whitelisted symbol is matched to exactly one trading schedule via its globexGroupCode (resolved in Step A.3.3): the system searches the trimmed tradingHours data (Step A.3.2) for the quadrant whose groupCodes[] array contains that globexGroupCode. The symbol is assigned to that quadrant's schedule. Example mapping:

CME_Instrument_Details stores one row per (Symbol, Calendar_Date), denormalizing the schedule fields (Session_Open_UTC, Official_Close_UTC, Halt_Start_UTC, Halt_End_UTC, is_modified_schedule) from the symbol's matched quadrant/schedule, alongside the symbol's own margin value from Step B — this keeps the table flat for simple per-symbol lookups by downstream services, matching the existing table shape already read by UC_4.1.3 and UC_4.10.1/UC_4.15.1. No hierarchical/quadrant-nested object is persisted downstream — any such structure is intermediate processing shape only, flattened before Step C commits.

is_modified_schedule is populated directly from the equivalent flag/diff the CME Reference Data API returns per day within schedules — the backend performs a 1:1 field mapping and does not independently compute a diff against a locally-derived baseline.

(Source: CR-20260827-001 [CHR-76]; CME_Instrument_Details table name per RFQ_ Stack Trading Prop Tech V7.pdf §3.3)

---

BR_6.1.1.5: Margin ingestion is mandatory, same worker

Step B (margin ingestion via the CME Margin Service — MarginServiceApi/RealTimeMargin) is a mandatory part of this same cron, not an optional or separately-scheduled task. It retrieves the latest outright maintenance/overnight margin requirement for every whitelisted symbol, feeding UC_6.1.2's dual-constraint Required Margin calculation (CME Overnight Margin × Contract Count, compared against 25% of position notional value — see BR_6.1.2.4).

(Source: CR-20260827-001 [CHR-76]; reaffirms the margin-ingestion requirement in RFQ_ Stack Trading Prop Tech V7.pdf §3.3, lines 3171-3172, now formalized as an explicit named step)

---

BR_6.1.1.6: Atomic commit — Step C

Step C commits both the schedule timestamps (Step A) and the margin values (Step B) into CME_Instrument_Details within a single database transaction. Downstream readers (Margin Enforcement, Market Status Widget) must never observe a partially-updated table mid-run — either the full batch for a given run is visible, or none of it is.

(Source: CR-20260827-001 [CHR-76])

---

BR_6.1.1.7: UTC timestamp handling — Futures timestamps arrive pre-resolved, no system-side conversion

For Futures, the CME Reference Data API returns Session_Open_UTC, Official_Close_UTC, Halt_Start_UTC, and Halt_End_UTC already as correct UTC instants — DST shifts and holiday-hours adjustments are handled upstream, at CME's data source. This cron does not perform any active local-time-to-UTC conversion; it stores the timestamps as received. This reaffirms BR_4.1.3.5 (UC_4.1.3) without modification — no update to that rule is required by this UC.

All stored timestamp fields follow CR-08 (UTC / ISO 8601 / millisecond precision), including Last_Ingested_UTC, which §8.3 of that rule already lists as an affected field.

(Source: CR-20260827-001 [CHR-76])

---

BR_6.1.1.8: Strict field trimming

Only the fields downstream services actually consume are persisted to CME_Instrument_Details — the full CME API response payload is not stored verbatim. Fields kept: Symbol, Calendar_Date, Session_Open_UTC, Official_Close_UTC, Halt_Start_UTC, Halt_End_UTC, is_modified_schedule, and the margin value from Step B. Intermediate fields used only during matching (e.g. raw globexGroupCode, quadrant, scheduleNames[]) are not persisted as separate columns once the match is resolved.

(Source: client-supplied CME Reference Data Ingestion logic, BA session 2026-08-27 — "Strict rule: Trim all excess fields. There is absolutely no need to store every field the API returns.")

---

Dynamic Timekeeper step (same flow, continuous phase)

BR_6.1.1.9: Independent timeline thread per asset class, sourced from ingested reference data only

The Dynamic Timekeeper phase is the continuous side of the same middleware that runs the ingestion cron — not a scheduled batch job — and it maintains one independent timeline thread per whitelisted asset class (quadrant), because CME's trading calendar gives each asset class its own distinct closing/halt boundaries. Each thread is derived exclusively from the 4 fields the cron step has already committed to CME_Instrument_Details for that asset class: Official_Close_UTC, Halt_Start_UTC, Halt_End_UTC, Session_Open_UTC. This phase never calls the CME Reference Data API or the CME Margin Service directly — that is the cron step's exclusive responsibility (see BR_6.1.1.1). A schedule change affecting one asset class (e.g. an Energy-only holiday) does not shift any other asset class's timeline.

(Source: RFQ_ Stack Trading Prop Tech V7.pdf §Risk Enforcement Endpoints — "Enforce Overnight Margin" → "Dynamic Timekeeper and Ingestion Architecture", lines 1657-1688 — "the middleware manages independent timeline threads for each category")

---

BR_6.1.1.10: Scope boundary — this UC generates the timeline, it does not enforce

The Dynamic Timekeeper phase's responsibility ends at signaling that a milestone has been reached. It does not itself: freeze/unfreeze order routing at Halt Start/End, run the Universal Drawdown Safety Gate or margin compliance audit at T-2min, run the trim loop or Active Failsafe at T-30sec, or restore day margin at Session Open. Those enforcement actions, their business logic, and their payload construction belong to UC_6.1.2 — Margin Enforcement and UC_6.1.3 — TRIM_TO_COMPLIANCE, and are intentionally not duplicated here — this UC only defines the timeline-generation mechanism. See QnA_init_docs.md A-20 → A-27 (Vòng 2) for the enforcement-side scope.

(Source: BA scope decision 2026-08-27/2026-09-03 — QnA_init_docs.md Vòng 2 scope note, applying Single Source of Truth for Content per global-rules.md)

---

9. Update History

---