On this page
- Summary
- Applicable Principles
- Architecture
- Visual Architecture Views
- Platform Role and Boundaries
- Deployment and Platform Topology
- Baseline Service Topology
- Data Service Model
- CRUSH and failure-domain strategy
- Network and Security Baseline
- Access, prerequisites, and integration contracts
- Scope boundaries from CER design
Summary
This note defines the current draft architecture baseline for the dedicated external Ceph platform used by OpenShift through ODF external mode. It consolidates accepted storage ADR direction with the detailed Red Hat engagement design input so the team can move from high-level decisions to an implementable storage architecture.
Applicable Principles
- Data Protection - Ceph design must enforce encryption, backup, and isolation expectations.
- Reliability and Resilience - Ceph service layout and failure domains must support platform availability targets.
- Security by Design - Access boundaries, key handling, and network separation are built into the baseline.
- Observability by Default - Ceph and ODF health, capacity, and performance must be operationally visible.
- Automation First - Deployment and day-2 operations are executed through approved automation.
Architecture
Visual Architecture Views
Ceph and ODF integration boundaries
CSI and operator integration] Apps --> ODF end subgraph CEPH[External RHCS or Ceph platform] MON[🧭 MON quorum] MGR[⚙️ MGR services] OSD[💽 OSD data plane] MDS[📁 MDS for CephFS] RGW[🪣 RGW object endpoints] OBS[📈 Prometheus, Grafana, exporters] MON --> OSD MGR --> OSD MDS --> OSD RGW --> OSD OBS --> MGR end ODF -->|RBD and CephFS consumption| OSD ODF -->|Object access where required| RGW classDef app fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20,stroke-width:1px; classDef integration fill:#E3F2FD,stroke:#1565C0,color:#0D47A1,stroke-width:1px; classDef cephCore fill:#FFF3E0,stroke:#EF6C00,color:#BF360C,stroke-width:1px; classDef observability fill:#F3E5F5,stroke:#6A1B9A,color:#4A148C,stroke-width:1px; class Apps app; class ODF integration; class MON,MGR,OSD,MDS,RGW cephCore; class OBS observability;
Logical storage isolation model
Network segmentation intent
Failure-domain and blast-radius intent
failure domain rack, device class ssd] POOL[📦 Replicated pool baseline
size 3, min_size 2] CLIENTS[🧩 OpenShift consumers] RULE --> POOL POOL --> A1 POOL --> B1 POOL --> C1 CLIENTS --> POOL IMPACT[⚠️ Single host failure: tolerated
⚠️ Single rack failure: degraded but available] POOL --> IMPACT classDef rack fill:#ECEFF1,stroke:#455A64,color:#263238,stroke-width:1px; classDef osd fill:#FFF3E0,stroke:#EF6C00,color:#BF360C,stroke-width:1px; classDef policy fill:#E3F2FD,stroke:#1565C0,color:#0D47A1,stroke-width:1px; classDef consumer fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20,stroke-width:1px; classDef impact fill:#FCE4EC,stroke:#C2185B,color:#880E4F,stroke-width:1px; class RackA,RackB,RackC rack; class A1,A2,B1,B2,C1,C2 osd; class RULE,POOL policy; class CLIENTS consumer; class IMPACT impact;
Platform Role and Boundaries
Ceph is the shared distributed storage platform for OpenShift clusters and is consumed through ODF external mode. ODF provides the OpenShift-side integration layer (CSI and operator flows), while external RHCS/Ceph provides storage services and data durability. Lifecycle ownership is intentionally split: OpenShift teams own ODF integration behavior and storage teams own external Ceph lifecycle and health.
Deployment and Platform Topology
Deployment follows a baseline-first model from the CER: start with stock Ceph deployment, then apply controlled tuning after health and interoperability validation.
| Profile item | CER-aligned baseline | Architecture intent |
|---|---|---|
| OS and Ceph stack | Preferred RHEL 10 + RHCS 9; fallback RHCS 8 when ODF compatibility requires it | Keep supportability-first upgrade path with explicit compatibility gates |
| Host placement model | Rack-aware, label-driven placement | Avoid ad-hoc daemon spread and preserve predictable failure domains |
| Cluster posture | Dedicated external Ceph consumed by ODF external mode | Shared storage platform for multiple OpenShift consumers |
Baseline Service Topology
Service topology reflects the CER design profile and keeps a clear distinction between data-plane services, control-plane services, and observability services.
| Service area | Baseline profile | Notes |
|---|---|---|
| Core quorum and control | MON=3, MGR=3 | Quorum and management spread over labeled hosts |
| Data services | OSD on dedicated storage hosts; MDS=2; RGW=2 | Supports block, file, and object paths |
| Observability | Grafana=1, Prometheus=1, host-wide node-exporter, crash, ceph-exporter | Baseline visibility for storage and node health |
| Placement controls | Host labels plus magic labels _admin, _no_schedule, _no_autotune_memory | Deterministic scheduling and guarded day-2 operations |
The CER host model is a mixed-role pattern rather than strict node-type silos. Some hosts carry control-plane roles (mon, mgr) plus OSD duties, while designated hosts carry data-plane and edge roles (mds, rgw, monitoring). This keeps the initial footprint efficient, but requires explicit placement policy so role co-location remains intentional and supportable.
Data Service Model
The CER design is multi-consumer by default, but isolated by namespace, subvolume, and realm boundaries. Block and file are shared platforms with per-client logical segregation; object is segmented through RGW realm and zone boundaries.
| Data path | Baseline isolation model | Pool behavior |
|---|---|---|
| RBD block | Shared RBD pool with per-client RADOS namespaces | Replicated pool, compressed data path |
| CephFS file | Shared filesystem with per-client subvolumes (namespace isolated) | Data pool compressed, metadata uncompressed |
| RGW object | Realm and zone per trust-bubble pattern; avoid realm name default | Realm and zone scoped pools; bucket data compressed |
Pool and compression baseline (CER-aligned)
Compression profile follows CER recommendations and favors lz4 as the default balance between CPU overhead and throughput.
| Pool class | Replication baseline | Compression baseline |
|---|---|---|
RBD (rbd.ocp.normal) | size=3, min_size=2 | aggressive, lz4, ratio baseline 0.85 |
| CephFS data pool | size=3, min_size=2 | aggressive, lz4, ratio baseline 0.85 |
| CephFS metadata pool | size=3, min_size=2 | none |
| RGW bucket data pool | size=3, min_size=2 | aggressive, lz4, ratio baseline 0.85 |
| RGW control/meta/index pools | size=3, min_size=2 | none |
Replica-1 pools remain outside the baseline. The CER explicitly calls out the operational and failure-recovery complexity; these pools require separate risk acceptance and stricter day-2 procedures.
CRUSH and failure-domain strategy
The baseline avoids the default host failure domain for production pools. The CER design recommends a custom rack-scoped replicated rule (for example rep-rack) and explicit device_class=ssd targeting so data does not silently spread to slower media when future classes are introduced.
Network and Security Baseline
Network and security controls are defined as architecture constraints, not implementation options.
| Control area | Baseline design | Why it matters |
|---|---|---|
| Traffic segmentation | Public/client, cluster replication, optional RGW network | Limits blast radius and isolates traffic classes |
| Network interface policy | Bonded uplinks, 802.3ad, Layer3+4; jumbo frames only with end-to-end validation | Stabilizes throughput and hashing behavior under OSD parallelism |
| Time sync | Shared Chrony sources with strict skew control | Protects quorum stability and certificate validity |
| Encryption at rest | OSD disk encryption with LUKS2 | Meets baseline at-rest controls |
| Encryption policy mapping | ADR-014 workload-class mapping through StorageClass to Ceph path | Keeps controls auditable and workload-aware |
Access, prerequisites, and integration contracts
Before deployment, repository and registry access for Ceph nodes are hard prerequisites. ODF external-mode integration also requires explicit connectivity and credential lifecycle controls between OpenShift and external Ceph. RGW admin-ops credentials are treated as platform secrets and scoped to tenant trust boundaries.
| Prerequisite class | CER-aligned requirement | Architecture interpretation |
|---|---|---|
| RHEL repositories | BaseOS and AppStream plus RHCS tool repositories for target stack | Treat repository reachability as a hard day-0 gate |
| Container registry | Access to required Ceph images (direct or mirrored) | Keep disconnected-readiness path open via mirroring strategy |
| Time source | Shared Chrony/NTP sources across all Ceph hosts | Make time drift an operational SLO, not a best effort |
| ODF integration path | Version and connectivity compatibility between ODF and RHCS | Promote only with explicit interoperability evidence |
Scope boundaries from CER design
The CER design is explicit about boundaries that this architecture keeps intact. Server-side encryption for RGW is a future capability, not a baseline control. Advanced NVMe namespace slicing is treated as a performance-optimization track, not a deployment prerequisite. Replica-2 and replica-1 pool variants remain exception patterns that require separate SOP and risk handling.
Operations, Monitoring, and Lifecycle
Operationally, this architecture assumes a strong observability baseline and explicit ownership handoffs. Monitoring includes Ceph-native telemetry plus ODF integration signals. Grafana TLS hardening is part of day-2 baseline (replace self-signed defaults with managed certificates).
| Lifecycle checkpoint | Required evidence |
|---|---|
| ODF to Ceph interoperability | Compatibility and connectivity validation for target versions |
| Day-2 readiness | Host replacement, pool onboarding, namespace onboarding, and DR check runbooks |
| Security hardening | Managed cert rollout for monitoring endpoints and secret ownership boundaries |
Performance and Capacity Baseline
Performance baseline is RBD-centric and uses FIO for both single-client and concurrent-client profiles. The throughput-performance tuning profile is the default operating profile; custom tuning requires explicit profiling and control.
| Dimension | Baseline stance |
|---|---|
| Test method | FIO on RBD (single and concurrent clients) |
| Isolation mechanism | Pool design, namespace/subvolume boundaries, CRUSH and device-class targeting |
| Replica-1 usage | Out of baseline; only through explicit exception and operational risk controls |
Scaling caveat from CER design
The current CER object-storage model (realm per client trust bubble) improves administrative isolation, but it also increases pool count and PG pressure as client count grows. The architecture therefore treats RGW realm strategy as a growth gate and keeps this caveat explicit until the referenced product enhancement path is resolved.
Delivery alignment
| Workpackage | Relevance to this architecture | Current status |
|---|---|---|
| WP-04 DC3.0 Storage | Primary implementation path for Ceph and ODF external integration, including DR layering and storage profiles | planned |
| WP-21 OpenShift AI DC-A Transitional Production Lane | Transitional lane that explicitly depends on Ceph automation readiness and ODF external integration gates | active delivery lane |
Readiness gates for WP-04
| Readiness gate | Current state | Assessment |
|---|---|---|
| Architecture baseline | Ceph architecture baseline is documented and trace-linked to ADRs and storage topics | green |
| Workpackage linkage | WP-04 and WP-21 now reference Ceph architecture explicitly | green |
| Product compatibility baseline | Preferred stack is clear, but final ODF to RHCS compatibility decision at go-live is still open | amber |
| Network definition | VLAN IDs, CIDRs, gateway behavior, and routability decisions are not yet fixed in architecture artifacts | red |
| Placement and host mapping | Label and role model is defined, but final per-site host inventory and rack mapping constraints are not fixed | amber |
| Security and PKI model | At-rest and secret boundaries are clear, but CA and certificate operating model for Ceph endpoints is not finalized | amber |
| SLO and evidence thresholds | Direction exists, but measurable SLO targets and acceptance thresholds are not defined in WP-04 | red |
| Runbook and ownership model | Ownership split is clear at architecture level, but certified day-2 runbooks and escalation contracts are not yet tied to workpackage exit criteria | amber |
Overall WP-04 readiness is amber: architecture direction is strong, but execution gating is blocked by unresolved network definitions and missing measurable acceptance thresholds.
Decisions
- adr-001-storage-backend - Dedicated external Ceph platform selected as the primary storage backend through ODF external mode.
- adr-009-odf-structure - External ODF model is the production target; hyperconverged mode is non-production only.
- adr-014-encryption-at-rest - Encryption controls are policy-driven by workload class and mapped to ODF/Ceph implementation paths.
- Working implementation baseline is aligned to the Red Hat Consulting Engagement Report draft (2026-02-06), Chapter 8 (Ceph), including service specs, network model, pool strategy, CRUSH intent, and performance baseline.
Open Questions
- Which RHCS release baseline is final for first production rollout, based on target ODF interoperability at go-live?
- What are the final VLAN IDs, CIDR allocations, gateway patterns, and routability boundaries for Ceph public, cluster, and RGW networks per site?
- What are the final rack-to-host mappings and service placement constraints for each site and hardware batch, including role co-location limits?
- Which certificate authority and certificate lifecycle process will be used for Ceph service endpoints (for example Grafana and gateway surfaces)?
- Is a support exception for multi-service node placement required, and what policy guardrails apply if granted?
- Which RGW realm strategy is approved for multi-tenant growth before and after the current RFE dependency is resolved?
- Which measurable SLOs and alert thresholds are mandatory for Ceph and ODF integration (latency, capacity, recovery, and error budgets)?
- Which DR scope is committed at storage layer (Ceph replication) versus application failover orchestration (ODF DR and platform workflows)?