TurgonHCT · Solution Architecture · v1.0HCT Data Modernization — The move to an AI-ready enterprise.
HCT's modernization is not blocked by strategy or tooling spend. It is blocked at the pipeline layer: five parallel ETL patterns, three disconnected orchestrators, ~100% traditional E-T-L with 70%+ on a 15-year-old SAP Data Services stack, and lineage that does not function because IDMC cannot read the legacy tools. This document defines the future-state architecture that fixes it.
The move to an AI-ready enterprise.
HCT's own maturity assessment scores governance highest (2.83) and execution lowest — MDM & RDM 1.60, Data Quality 1.67, Data Engineering 1.75. The thinking is done; this architecture supplies the execution layer.
Consolidate onto one pipeline platform
Five parallel ETL patterns and three disconnected orchestrators replaced by one governed pipeline layer of agent-generated, human-approved, git-versioned code.
Land in a governed medallion lakehouse
Bronze / Silver / Gold on an open table format — the single analytical store, with SCD-2 native, DQ-gated silver and reverse-engineered gold KPIs.
Governance executed, not documented
TEOS builds dictionary, ontology, glossary, column-level lineage and master-data control — every agent conclusion validated by a named human before it goes live.
Coexist, then retire
SAP DS, Informatica BDM and SSIS are retired per-pipeline behind parity evidence — never big-bang. Existing dashboards keep running throughout.
What this document defines — and what it does not.
The future-state architecture, the migration path from the current estate, and the technical operating model that sustains it.
- Ingestion, transformation, orchestration, storage, DQ, metadata / governance and consumption architecture for HCT's analytical estate.
- The agent platform and its human-validation model.
- Migration architecture: SAP DS (.atl), Informatica BDM and SSIS logic → open, licence-free code.
- Governance substrate: data dictionary, ontology / knowledge graph, business glossary, end-to-end lineage, master-data control.
- Security, access, residency; environments, observability and capacity.
- Delivery waves and the technical operating model.
- Source-system (OLTP) re-architecture — Banner, Oracle Fusion and SAP remain unchanged at source.
- Application-layer decisions on end-of-life systems — data-layer implication captured in 08.6.4; the application decision is HCT's.
- Detailed network / infrastructure build — dependent on the platform decision (011, ADR-007).
Why HCT must modernise its data core now.
This section sets out the reasons and criticality of modernising the underlying IT infrastructure — the forces reshaping how HCT sources, governs and consumes data.
No trusted single source of truth
Five parallel ETL patterns produce the same fact differently; no MDM platform; duplication is uncontrolled.
Inaccurate data reaches external regulators
Reporting to PMO / TDRA / Ministry of Higher Education on data leadership cannot vouch for — a real institutional exposure.
Licence exit
HCT does not intend to renew SAP Data Services or Informatica. One pipeline platform, licence-free at the core.
Inconsistent business definitions
"Enrolled student" and "full-time-equivalent student" mean different things in different reports. Semantic drift corrupts even correct pipelines.
Tool sprawl → unification
10+ overlapping tools. HCT wants eventual unification with minimum friction — stepwise, not disruptive.
Execution deficit
Platforms procured, governance drafted — but the build capability is absent. HCT's own deck names this precisely.
Sustained operations, not a hand-off
HCT wants a data-as-a-service model post-go-live: L1 / L2 / L3, new pipelines and reports within a retainer.
The current state — as it actually is.
Every number below is taken from HCT's own DA1.1 deck. Nothing here is inferred.
- CDC — absent · manual watermarks in shell scripts
- Native SCD-2 — absent · nightly batch only
- Lineage — IDMC cannot read SAP DS / BDM · impact analysis non-functional
- MDM — not procured or operationalised
- Fine-grained RBAC — inconsistent across 1000+ users
- Regulatory path — CHEDS via .NET fn + GSB / APIM relay
HCT Data and IT Maturity Self Assessment.
The institution's strengths are governance and strategy; its weaknesses are precisely the execution domains — MDM, data quality, data engineering. This architecture supplies the execution layer; HCT retains governance ownership.
- Governance forums and policy foundation
- Conceptual clarity on lake / warehouse paradigms
- Dimensional modelling in use
- Power BI in place and wanted
- Dataiku procured, 3 AI/ML use cases live
- SSO / AD standardisation
- API documentation exists
- IDMC provides DQ configuration capability
Coexist, then retire — never rip-and-replace (AP-1, AP-6).
From five parallel ETL patterns to one governed pipeline layer.
Same sources. Same consumers. Radically different middle.
Fragmented estate
- Overlapping ETL stacks, licence sprawl
- Silent data-quality failures at source
- No lineage, tribal knowledge
- Batch-only, reconciled manually
Agent-driven lakehouse
- One lakehouse, medallion‑zoned
- Rule‑based flagging at ingest
- End‑to‑end lineage & catalog
- Agents run, monitor, self‑heal
Where accuracy actually breaks — and the response to each.
Stale or wrong data enters the estate
Including external partner feeds. Cannot be fixed at source.
Detect and flag at ingestion with a rule-based engine; surface to data owners; quarantine on contract violation.
ETL corrupts or mis-maps data en route
Broken lineage hides the cause. Impact analysis is non-functional today.
Rebuild pipelines with in-line DQ gates and end-to-end column-level lineage — every defect attributable to a transform.
"Enrolled student" means different things
Full-time-equivalent, active, registered — the divergence only becomes visible at the dashboard.
Business glossary reconciled once with data owners and enforced inside silver transforms. Definitions become code.
The principles we are following in our transformation and design decisions.
Keep the lights on
No rip-and-replace. Existing dashboards continue to run through every cutover.
One pipeline platform; open, licence-free core
Transformation logic lands in DBT / SQL and Python — versioned, portable, readable by HCT engineers.
Agents come to the data
Agents execute inside HCT's environment against scoped credentials. Data does not leave the tenant.
Human-in-the-loop is mandatory
Every agent conclusion — dictionary, mapping, DQ rule, KPI — validated by a named human before it goes live.
Contract preservation
Migrated pipelines produce the same tables, at the same grain, on the same schedule — until HCT chooses otherwise.
Coexist, then retire
Legacy platforms are retired per-pipeline behind parity evidence; never big-bang.
Governance is executed, not documented
Glossary, lineage and quality rules are enforced by the pipeline at runtime, not stored beside it.
Tool-agnostic at the edges
HCT chooses its BI / consumption tools; Turgon brings no proprietary front-end lock-in.
The future state.
Sources are unchanged. One governed pipeline layer replaces five parallel ETL patterns and three orchestrators. The medallion lakehouse is the single governed store. TEOS governance and the managed service span every layer — everything executes inside HCT's trust boundary.
config-driven connectors · zero watermarks · schema-drift captured at landing
dbt tests + expectations · failures quarantine · attributed source | pipeline | semantic
open code · CI-tested · native SCD-2 · the licence exit
one control plane · retries · SLA alerting · replaces 3 disconnected engines
every layer versioned · corrections re-run only affected stages
SAP Data Services (.atl) · Informatica BDM · SSIS · high-watermark shell scripts · 3 schedulers
IDMC → retire at renewal OR feed it lineage (ADR-008)
immutable · full history · schema-drift captured · the audit floor
SCD-2 · DQ-gated · PII tagged · glossary-ENFORCED definitions
Student · Academic · Finance · Regulatory · KPIs reverse-engineered from reports
gold → materialised views back · dashboards keep their connection
dim_student parity = the visible test
semantic models on gold · RLS + PII masking via AD
phased retire — reports follow pipelines
consolidates into Power BI (HCT's plan)
retained by choice — reads governed silver/gold
context layer — deliberately last
governed, audited pipelines replace the .NET/APIM relay · every figure traceable to a source record
all sources — vs Banner-only today
reconciled with data owners · ENFORCED in silver transforms
column-level, ingestion → BI · impact analysis finally works
student · course · campus · golden records — MDM buy avoided
HCT SME + Turgon validator sign off every agent conclusion before it is live
- Azure AD / SSO · scoped service accounts
- ONE RBAC plane — replaces fragmented BO / Power BI / Tableau (1000+ users)
- PII catalogue + masking on student data
- Full audit trail for regulatory traceability
- All compute + storage in-region (ADR-007)
- Offline mode: schema export where no live path
- Drift detection on every source (schema · volume · semantic)
- Self-healing pipeline agents
- Versioned runbooks · SLA reporting (L1 / L2 / L3)
- Live Workflow Board — HCT watches sources move bronze → silver → gold past each validation gate, in real time
Figure 1 · Sources unchanged · one governed pipeline layer replaces 5 ETL patterns & 3 orchestrators · medallion lakehouse = single governed store · TEOS + AMS span every layer.
Ingestion
Config-driven CDC across Banner, Fusion, SAP, MS SQL and app feeds. Schema drift captured at landing; watermarks retired.
Transformation
DBT + Python as the transformation core — open, portable, CI-tested. Native SCD-2 replaces nightly batch.
Orchestration
One control plane with retries, SLA alerting and versioned runbooks — replacing three disconnected engines.
Storage — medallion lakehouse
Bronze immutable raw · Silver conformed & DQ-gated · Gold report-ready domains. Open table format, one RBAC plane.
Data quality
Expectations + dbt tests inline in the pipeline. Failures quarantine and are attributed to source, pipeline or semantic layer.
16 functional. 10 non-functional. 8 constraints.
Agent execution model.
parses source metadata, .atl/DSDL, existing reports · builds a working model of the estate
drafts CDC connectors, landing schemas, DQ tests
reads legacy reports · reconstructs metric definitions into the semantic layer
conforms entities · SCD-2 · glossary-enforced definitions · PII tagged
domain marts · dashboards · APIs · reverse-materialised to Exadata for coexistence
paired agents draft and adversarially test every artefact before it reaches a human reviewer.
no pipeline, KPI or schema promotes to production without an explicit human sign-off recorded in git.
Understand → (ingest ∥ reverse-engineer the KPIs behind existing reports) → bridge via silver → consume. Each logical step is a swarm of designer + validator agents running in parallel, and every stage terminates in a mandatory human gate (AP-4) — nothing reaches production unvalidated.
Every artefact is agent-drafted, validator-checked, human-approved before promotion.
Governance executed by the pipeline, not stored beside it.
Data dictionary + ontology
Automated capture across all sources — replacing Banner-only coverage today. Ontology becomes the durable domain memory.
Business glossary
Reconciled with data owners once and enforced inside silver transforms — definitions become code, not documentation.
Automated lineage
Column-level, ingestion → BI. Impact analysis works because the lineage graph is generated by the pipeline itself.
Master-data control
Golden records for student, course and campus derived from the ontology — avoiding a separate MDM procurement.
Human-in-the-loop gate
Every agent output — mapping, DQ rule, KPI — signed off by a named HCT SME and a Turgon validator before promotion.
Security, observability and capacity — in brief.
Security, access & residency
SSO via Azure AD with scoped service accounts. One RBAC plane replaces fragmented BO / Power BI / Tableau access. PII catalogue with masking on student data. Full audit trail for regulator traceability. All compute and storage remain in-region.
Observability & operations
Drift detection on every source — schema, volume and semantic. Self-healing pipeline agents with versioned runbooks. Live Workflow Board makes bronze → silver → gold movement visible in real time, with SLA reporting across L1 / L2 / L3.
Capacity & performance
Consumption-based compute right-sized to workload; near-real-time capable where the source supports CDC. Schedule parity preserved through migration, then re-baselined from measured throughput.
Dev → QA / Parity → Approval → Prod → DR.
Four environments with a controlled promotion path. Dev is where agents author and iterate. QA / Parity dual-runs the new pipeline against the incumbent until outputs match at agreed tolerance. An HCT approval gate precedes Prod promotion. DR mirrors Prod in-region and is exercised on schedule — matching or exceeding today's resilience posture.
agents draft here · fast iteration · synthetic + sampled data
dual-run vs legacy · DQ tests · schema-drift detection · load & SLA rehearsal
human sign-off · change record · reversible plan · recorded in git
single orchestrator · SLA-monitored · versioned refreshes
in-region warm mirror · storage-replicated · rehearsed cut-over runbook
artefacts move only via CI · no manual edits in higher environments · rollback is a redeploy, not a hotfix.
all four environments run inside HCT's tenant, in-region · no data leaves the trust boundary.
The QA/Parity environment is where the migration is actually proven: legacy and rebuilt pipelines dual-run the same window and the parity harness compares row counts, checksums and business aggregates. No code — agent-written or not — reaches Prod without passing CI, parity and a named human approval. The 36-server footprint is retired; environment discipline is preserved.
Four isolated environments · every promotion is gated · DR is a warm mirror, not an afterthought.
Read → Normalise → Rebuild → Validate → Cut over.
ingest legacy artefacts — SAP DS .atl, Informatica DSDL, SSIS packages, shell watermarks
map each dialect into a single intermediate representation the agents can reason over
regenerate as DBT + Python · CI-tested · native SCD-2 · git-versioned
parity harness runs old and new side-by-side · row-, column- and metric-level diffs must pass
traffic swings to the new pipeline · legacy tool decommissioned at explicit licence gate
dual-run gate is mandatory — a pipeline only cuts over when its outputs match the legacy result within tolerance.
SAP DS, Informatica BDM and SSIS renewals are retired only after their pipelines pass parity and dashboards land on the new stack.
The proprietary .atl/DSDL problem is solved by parsing legacy logic once into a normalised mapping IR held in the knowledge graph, regenerating open DBT/Python code against the same output contracts, and cutting over only behind a passed parity gate. SAP DS, Informatica BDM and SSIS renewals are retired per-pipeline, not big-bang.
Legacy ETL is not lifted — it is read, re-expressed as open code, proven at parity, then retired.
Validation slices → Foundation → Consolidation → Quality / Semantics / AI.
narrow, end-to-end proofs on real data · establish parity harness, DQ gates and the promotion path
W1 Student/Academic (Banner — the parity test) · lakehouse, orchestrator, governance stood up
W2 Regulatory (CHEDS/TDRA) · W3 Finance/HR (Fusion, SAP) · licence gates retire SAP DS, Informatica, SSIS
W4 Engagement/Ops · semantic layer hardened · AI/agent features on governed gold
spans every phase from the first production pipeline onwards — the platform is operated, not handed over.
Validation slices (3–4 weeks) prove access, the engine and the licence exit on a hard pipeline. Phase 1 stands up the lakehouse, governance and first sources. Phase 2 migrates P01/P02/P04 with per-domain parity cutovers behind explicit SAP DS, Informatica and SSIS licence gates. Phase 3 lands golden records, CHEDS submissions and chat-with-data on governed gold — with Living Infrastructure (AMS) running from the first production pipeline onward.
Waves land in risk-and-value order · legacy licences retire only at proven parity.
Named risks and their mitigations.
Higher-education domain onboarding
Domain-acquisition mechanism (documentation ingestion + structured SME sessions + the ontology as durable capture), demonstrated on Banner in Slice 1 rather than asserted.
Unquantified accuracy expectations (NFR-09)
Close OQ-01 before the POC starts; adopt ADR-012 — parity = hard gate; precision = measured and reported.
Access / NDA delay
Issue the prerequisite list with the scenarios; offer the offline schema-export mode for blocked sources.
Incumbent platform dependencies
Coexist-then-retire: nothing is removed until its replacement is proven and its dashboards are live.
.atl conversion harder than modelled
Prove on a representative slice in POC-2 before committing sequence dates; the IR approach (09) must be demonstrated, not asserted.
Dual-run cost and effort
Scope dual-run per pipeline with a fixed window; retire promptly after sign-off.
Residency ruling forces on-prem (OQ-02)
Architecture is platform-portable — open code, open table format; ADR-002 / 007 deliberately decoupled.
Scope creep in the CHEDS slice
Fix the slice tightly: one submission, one window, one parity pack.
Tribal knowledge / documentation deficit
SME sessions are a scheduled delivery input, not an ad-hoc favour; the ontology captures the knowledge permanently.
Sizing unknown (OQ-10)
Request with POC prerequisites; size conservatively for validation slices, then re-baseline from measured throughput.
Agent-generated code without modelling standards
08.11 fixes conventions up front; CI enforces them; every model ships with tests.
Transformation begins here.
One governed pipeline layer. One medallion lakehouse. Agents that build, humans who approve, licences that retire on evidence. HCT's data core — rebuilt, not renewed.