On this page
Summary
This note defines the transitional production architecture for a single OpenShift AI cluster in DC-A (AM1), created to deliver production value quickly by reusing selected IST (legacy) services. It is intentionally outside the long-term DC 3.0 SOLL target and is treated as a time-boxed exception with explicit caveats, controls, and exit criteria.
The lane runs as a standalone bootstrap-installed cluster (no ACM lifecycle management) with a fixed hardware profile of three bare-metal control-plane nodes and six GPU worker nodes. Engineering delivery is anchored in Workpackage 21, while governance and exception boundaries are anchored in ADR 022. It uses a production Ceph cluster in AM1, created in line with the DC 3.0 Storage Architecture.
| Aspect | Decision |
|---|---|
| Lane type | Transitional production exception in DC-A |
| Delivery anchor | Workpackage 21 |
| Governance anchor | ADR 022 |
| Service model | Single tier production with NaaS and MaaS |
| IAM | AD/ADFS OIDC, curated allowlist, break-glass controls |
| Registry | Harbor pull-through cache primary, Red Hat direct exception |
| Model governance | LLMeval/LMjudge stretch goal; if enabled, advisory and non-blocking |
| Storage baseline | Production Ceph in AM1 aligned to DC 3.0 Storage Architecture |
| Exit trigger | DC-B (AM4) proven production under DC 3.0 readiness gates |
Applicable Principles
- Automation First — changes and evidence remain Git-governed where possible.
- Security by Design — AI workloads inherit platform security baselines.
- Observability by Default — evaluation pipelines produce audit evidence.
- Reliability and Resilience — AI services align with availability targets.
- Data Protection — model and dataset handling follows protection policy.
Architecture
Decision snapshot
- Why this lane exists: deliver production value quickly while DC 3.0 target-state is still being built.
- What is explicitly constrained: single cluster in DC-A, no broad self-service expansion, no target-state precedent.
- How delivery is governed: mandatory control catalog with named owners and evidence artifacts.
- How the lane exits: migrate to DC-B after readiness gates; rebuild DC-A as DC 3.0 production site.
Physical architecture
OpenShift AI - Prod - Physical
Logical architecture
View navigation
- Architecture intent and controls: this note.
- Implementation baseline for engineering: Workpackage 21.
- Exception governance decision: ADR 022.
Intent and positioning
- This cluster exists to accelerate production delivery and management confidence while full DC 3.0 platform capabilities are built in Location B.
- The lane is deliberately transitional: production use is allowed, but capability growth is constrained to avoid creating a second permanent platform baseline.
- The long-term target architecture remains openshift-concept-overview and associated ADRs.
Scope lock (hard boundaries)
- Exactly one OpenShift AI cluster in DC-A.
- Standalone cluster lifecycle for this lane (no ACM-managed deployment path).
- No broad multi-tenant platform onboarding in this lane.
- No expansion into a generic CaaS/VMaaS platform role.
- Full BIO2 compliance and formal BIO2 certification are not in scope for this transitional lane due to delivery timeline constraints.
- No policy exception may be treated as precedent for DC 3.0 target-state design.
Cluster role and service intent
- Deliver internal DevOps acceleration capabilities using coding-agent workflows.
- Production-facing service set:
- Notebooks-as-a-Service (Jupyter-based workflows).
- Models-as-a-Service (LLM inference endpoints).
- Target additional platform usage as an LLM backend for OpenShift Lightspeed, serving both this DC-A OpenShift AI cluster and DC 3.0 development activities in DC-B (AM4).
- Evaluation remains a stretch-goal capability in this lane; if implemented, outputs are advisory (non-blocking).
Dependency and integration baseline (DC-A transitional)
| Capability area | Current DC-A transitional posture | Long-term direction |
|---|---|---|
| Network lane | Single ACI pod, AI-only scope (adr-019-hybrid-network-rollout-aci-a-ai-only-evpn-b-target) | EVPN/VXLAN target lane in Location B |
| Connectivity profile | Restricted network baseline (adr-011-connected-vs-disconnected) | Target controls in DC 3.0 platform lanes |
| External DNS/DHCP/NTP | IST dependency via infoblox integration contracts | Standardized platform-managed dependency contracts |
| North-south load balancing | External ADC dependency via netscaler-blx lane constraints | Datacenter Services baseline per adr-005-datacenter-services-baseline |
| IAM and authN/authZ | Direct integration with existing enterprise Active Directory (AD) for this transitional lane; no Keycloak in DC-A lane | Consolidated IAM model via adr-012-single-sign-on and adr-015-keycloak-identity-access-management |
| Registry path | Existing Harbor registry is used in this transitional lane; no Quay in DC-A lane | Tiered Quay target architecture |
| Storage | External Ceph via ODF integration (RBD/CephFS; object path by service need) | Same backend pattern with broader DC 3.0 lifecycle controls |
| Backup/restore | OADP plus Cohesity policy model per topic-openshift-backup-and-restore | Same model with full DR orchestration maturity |
In-cluster implementation stack (initial posture)
- Platform core: OpenShift plus OpenShift AI, Git-governed configuration, and controlled day-2 operations.
- Installation posture: bootstrap-based installation flow for a standalone cluster.
- Hardware footprint: bare-metal topology with 3 control-plane nodes and 6 GPU worker nodes.
- GPU runtime: GPU worker pools with explicit capacity boundaries for inference and evaluation workloads.
- Storage integration: PVC profiles mapped to workload classes and Ceph-backed storage classes.
- Security and policy controls: baseline IAM/RBAC, namespace boundaries, and admission policy controls aligned with platform defaults.
- Observability controls: capture evaluation evidence, serving telemetry, and operational events required for release and audit.
Operating profiles (DC-A transitional)
The sections below define the normative operating baseline for this lane.
Service profile
| Aspect | Profile detail | Owner | Evidence |
|---|---|---|---|
| Baseline | Single service tier production with NaaS and MaaS | Platform Ops | Service readiness checks |
| Support window | Business-hours support for standard operations; major incidents follow escalation process | Platform Ops | Incident response records |
| SLO posture | Business-hours response SLO applies in this phase | Platform Ops | Service-level reports |
| Boundary | No managed training service and no broad multi-tenant self-service in this phase | Platform Ops | Scope compliance review |
| Onboarding gate | Technical readiness and architecture sign-off are both required | Platform Ops | Approved onboarding records |
IAM profile
| Aspect | Profile detail | Owner | Evidence |
|---|---|---|---|
| Baseline | OIDC via ADFS with curated AD allowlist mapped to OpenShift RBAC | Platform Ops | OAuth and login validation output |
| Access model | platform-admin is cluster-admin; service-consumer is consume-only | Platform Ops | Group-to-role mapping records |
| Outage posture | Fail closed for normal SSO access; break-glass accounts only during AD/ADFS outage | Platform Ops | Outage and access audit trail |
| Break-glass controls | Primary/backup local emergency admins; two-person approval; post-incident credential rotation | Platform Ops | Break-glass approvals and rotation logs |
| Boundary | No Keycloak capabilities in this lane; AD group naming finalized with IAM team during implementation | Platform Ops | IAM alignment notes |
Registry profile
| Aspect | Profile detail | Owner | Evidence |
|---|---|---|---|
| Baseline | Harbor pull-through cache is primary image path; Red Hat registries are direct-access exception | Platform Ops | Image source inventory and pull tests |
| Boundary | Additional direct pulls are operationally controlled; shared pull-secret model is accepted in this phase | Platform Ops | Runtime access configuration snapshots |
| Exception record posture | No formal exception record is mandatory for additional direct pulls in this phase | Platform Ops | Operational control evidence |
| Control posture | No hard image scanning/signing gate at day-1 onboarding | Platform Ops | Policy configuration review |
Backup and recovery profile
| Aspect | Profile detail | Owner | Evidence |
|---|---|---|---|
| Baseline | Backup scope includes AI namespaces and platform configuration | Backup Ops | Backup configuration and run reports |
| Drill posture | Monthly restore drills run in original namespace with metadata plus sample PVC data validation | Backup Ops | Restore drill records |
| Failure policy | Remediation after failed drills is case-by-case and does not block releases | Backup Ops | Remediation tracking notes |
| Evidence retention | Restore-drill evidence is retained for 12 months | Backup Ops | Retention audit records |
Observability profile
| Aspect | Profile detail | Owner | Evidence |
|---|---|---|---|
| Baseline | Service-impact paging uses ticket and chat channels | Platform Ops | Paging route tests and sample alerts |
| Boundary | Warning alerts are signal-only by default and do not page | Platform Ops | Alert policy configuration |
Model governance profile
| Aspect | Profile detail | Owner | Evidence |
|---|---|---|---|
| Baseline | LLMeval/LMjudge is a stretch-goal capability | AI Platform Team | Implementation or defer decision record |
| If implemented | Execution is advisory and non-blocking with no fixed numeric thresholds in this phase | AI Platform Team | Gate policy record |
| Cadence (if implemented) | Run on each release candidate and weekly scheduled regressions | AI Platform Team | Execution logs and scorecards |
| Poor-result handling | Formal acceptance records are optional and recommended for significant regressions | AI Platform Team | Acceptance records when used |
Continuous evaluation workflow (LLMeval + LMjudge)
This workflow is a stretch-goal capability in the DC-A transitional lane. When implemented, it remains advisory and non-blocking.
- Candidate model is promoted to an evaluation namespace and registered in the model registry.
- LLMeval executes coding-agent benchmark suites (pipeline generation, IaC updates, code-review tasks) and records quality and runtime metrics.
- LMjudge applies rubric scoring for correctness, safety, and policy compliance.
- Metrics and artifacts are retained in approved storage targets with traceable release decisions.
- Release workflows consume scorecards to inform promotion decisions and risk discussions.
Production caveats (explicit)
- This lane is production but not full DC 3.0 target-state parity.
- Legacy dependency reuse increases operational variance versus target-state automation.
- Networking for this lane uses IST-provided ACI as-is, with no platform-side network automation in scope.
- IAM for this lane depends on existing AD connectivity and does not include Keycloak capabilities.
- Registry supply chain for this lane depends on existing Harbor and does not include Quay controls.
- The dedicated AI operations team assignment is still pending for this transitional phase.
- No BIO2 certification is planned for this transitional lane.
- Some controls are transitional and require additional guardrails until cutover parity is reached.
- Service catalog breadth is intentionally constrained to prevent uncontrolled scope expansion.
Mandatory production control catalog (DC-A transitional)
All controls below are mandatory before and during production onboarding in this lane.
| Control family | Mandatory control | Enforcement posture | Evidence artifact | Owner |
|---|---|---|---|---|
| Access and identity | OIDC via ADFS for cluster authentication | Mandatory now | OAuth configuration snapshot and auth test evidence | Platform Ops |
| Access and identity | Curated AD group allowlist to OpenShift RBAC mappings | Mandatory now | Approved group-to-role mapping record | Platform Ops |
| Access and identity | Break-glass flow uses primary/backup accounts with two-person approval and post-incident rotation | Mandatory now | Break-glass runbook, dual-approval records, audit event trail, and rotation evidence | Platform Ops |
| Image and supply chain | Image pulls use Harbor pull-through cache as default path | Mandatory now | Workload image source inventory report | Platform Ops |
| Image and supply chain | Direct path is limited to Red Hat registries; additional direct external pulls are operationally controlled | Mandatory now | Runtime access control configuration evidence | Platform Ops |
| Image and supply chain | Shared cluster-wide pull secret lifecycle is controlled | Mandatory now | Secret rotation and access review records | Platform Ops |
| Network and egress | Restricted-network egress allowlist is enforced | Mandatory now | Firewall and proxy allowlist evidence | Platform Ops |
| Network and egress | DNS/DHCP/NTP dependencies are defined and tested | Mandatory now | Dependency contract checklist and test logs | Platform Ops |
| Data and storage | Storage classes for NaaS/MaaS are approved and documented | Mandatory now | Storage profile matrix and approval record | Platform Ops |
| Data and storage | Dataset and artifact retention classes are defined | Mandatory now | Data retention policy mapping for AI workloads | Platform Ops |
| Backup and recovery | OADP schedules are active for AI namespaces plus platform configuration scope | Mandatory now | OADP configuration and backup run reports | Backup Ops |
| Backup and recovery | Monthly restore drills run in original namespace with metadata plus sample PVC data | Mandatory now | Restore drill results, remediation tracking records, and retained evidence (12 months) | Backup Ops |
| Observability and operations | Logs, metrics, and alerts are wired to approved platforms | Mandatory now | Monitoring and alert route validation evidence | Platform Ops |
| Observability and operations | Incident and escalation runbooks are approved | Mandatory now | Runbook approval records and drill evidence | Platform Ops |
| Model governance | LLMeval and LMjudge operate as advisory stretch-goal capability (if enabled) | Stretch goal | Gate execution reports and scorecard evidence, or defer rationale | AI Platform Team |
| Model governance | Poor-result acceptance record process exists (optional use in this phase) | Stretch goal | Acceptance record template and any captured approvals | AI Platform Team |
| Platform change control | Production changes use approved Git workflow | Mandatory now | PR history and deployment traceability | Platform Ops |
| Platform change control | Emergency change path is controlled and reviewed | Mandatory now | Emergency change records and postmortems | Platform Ops |
Expansion automation gate (pilot to broader onboarding)
- Expansion beyond pilot users requires core controls to be automated.
- Core controls are tracked as mandatory automation outcomes, without a fixed hardcoded checklist in this phase.
Exit and retirement criteria
- Primary transition trigger is met: DC 3.0 target-state in DC-B is proven in production with approved readiness evidence.
- AI services are migrated from DC-A to DC-B after production-proof gate approval.
- Migration (if needed) is executed as a joint effort by platform and customer teams, in planned service windows with best-effort downtime posture.
- DC-A transitional lane is sunset after service migration completion.
- DC-A site is rebuilt to DC 3.0 production-ready target-state after migration.
Control retirement mapping (separate from control catalog)
- Retirement mapping is maintained as a separate migration artifact that links each transitional control to its DC 3.0 target-state replacement pattern.
- Retirement mapping includes transition trigger, dependency, migration owner, and validation evidence.
- Retirement mapping must explicitly track AD/ADFS and Harbor transitional dependencies until replaced by target-state patterns.
Execution focus
This architecture is implemented through Workpackage 21, which contains the handoff-ready engineering package, workstreams, acceptance criteria, and evidence requirements.
Decisions
- ADR-004 Single-active phased rollout (draft) - single-active rollout framing for Location A and B.
- ADR-019 Hybrid network rollout: ACI-A AI-only, EVPN-B target (accepted) - ACI AI-only lane boundary in Location A.
- ADR-011 Connected vs disconnected (accepted) - restricted-network baseline for phase-1 cluster operation.
- ADR-022 Transitional production exception for OpenShift AI in DC-A (draft) - transitional production exception and sunset controls.
Delivery Context
- Standalone delivery stream: Workpackage 21.
Future Work
- Finalize the detailed installation runbook sequence, including persistent bootstrap host responsibilities and Ceph automation flow from bootstrap.
- Keep service boundary, control catalog, and OpenShift Lightspeed backend usage boundaries synchronized with implemented operations.
- Complete the operating model for AI operations ownership and handoff between Platform Ops and the dedicated AI operations function.
- Add a linked evidence profile for release gates, caveat controls, and override approvals, and keep Workpackage 21 implementation content current.
Open Questions
- Which retirement trigger and sequence applies for AD- and Harbor-specific transitional dependencies?
- Which minimum hard-gate controls are required at scale-up checkpoints, given the transitional posture and no BIO2 certification target?
- Which party owns OpenShift Lightspeed LLM-backend lifecycle operations in each phase (platform, AI operations, and customer teams)?
- Which migration decision gates and readiness checks determine whether migration to DC-B is required versus continued operation in DC-A?