Theme: From deployed models to trustworthy systems
Why This Guide Exists
The CDI Data Science pathway has moved progressively from understanding data to building predictive models and making those models available through software:
Data Science Foundations → Advanced Data Science → Machine Learning → Model Deployment → From Models to Systems
The previous guide, Model Deployment, ended with a model that had been packaged, validated, tested, and containerized as a prediction service. A successful request could enter the service, pass validation, reach the model pipeline, and return a prediction.
That achievement establishes that the model can be served reliably. It does not establish that the surrounding system will remain useful, safe, scalable, or trustworthy in the real world.
This guide begins at that boundary.
The Starting Point
We assume that a deployable model service already exists. The starting artifact is not a notebook or an experimental model. It is a tested service with:
a saved preprocessing and prediction pipeline;
a defined request and response contract;
input validation and error handling;
automated tests; and
a container image that can run consistently across environments.
From this point onward, the central question changes:
How do we operate a deployed model as part of a dependable, observable, and governable system?
A Deployed Model Is Only One Component
Once a model is used in practice, it becomes one component in a larger sociotechnical system. That system includes software, infrastructure, data pipelines, monitoring, people, policies, and the decisions influenced by model outputs.
A system can fail even when its model remains statistically accurate. For example:
the service may become unavailable under increased demand;
upstream data may change without warning;
predictions may arrive too late to support a decision;
feedback may be incomplete, delayed, or systematically biased;
an update may improve an aggregate metric while harming an important subgroup; or
people may misunderstand, ignore, or over-trust the output.
Production success therefore requires more than predictive performance. It requires reliable operations, evidence-based updates, clear ownership, and human judgment.
The Models-to-Systems Loop
This guide uses the following loop to connect technical operation with real-world decisions:
Code
flowchart TD A[Operate] --> B[Observe] B --> C[Evaluate] C --> D{Decision gate} D -->|Continue| A D -->|Improve| E[Validate change] D -->|Pause or retire| F[Human review] E --> G[Release safely] G --> A F --> C classDef stage fill:#f4f8ff,stroke:#036281,stroke-width:2px,color:#0f172a; classDef decision fill:#fff7ed,stroke:#f59e0b,stroke-width:2px,color:#7c2d12; classDef review fill:#f8fafc,stroke:#64748b,stroke-width:2px,color:#334155; class A,B,C,E,G stage; class D decision; class F review;
flowchart TD
A[Operate] --> B[Observe]
B --> C[Evaluate]
C --> D{Decision gate}
D -->|Continue| A
D -->|Improve| E[Validate change]
D -->|Pause or retire| F[Human review]
E --> G[Release safely]
G --> A
F --> C
classDef stage fill:#f4f8ff,stroke:#036281,stroke-width:2px,color:#0f172a;
classDef decision fill:#fff7ed,stroke:#f59e0b,stroke-width:2px,color:#7c2d12;
classDef review fill:#f8fafc,stroke:#64748b,stroke-width:2px,color:#334155;
class A,B,C,E,G stage;
class D decision;
class F review;
The loop is deliberately controlled. Observation does not trigger an automatic update. Evidence is evaluated at a decision gate before the system continues, changes, pauses, or is retired.
A Governing Principle
A trustworthy system does not learn from every signal. It changes only when evidence has been evaluated and the proposed change has been validated.
Feedback can be noisy, incomplete, delayed, biased, manipulated, or unrelated to model quality. Treating every signal as training data can reinforce errors rather than correct them. The decision gate makes improvement an accountable process rather than an automatic reaction.
One Continuous Case Study
The guide develops one practical case study from beginning to end. The same deployed prediction service is carried through operational planning, automation, monitoring, scaling, feedback, retraining, governance, and human decision-making.
Using one continuous system makes the dependencies visible. A monitoring metric affects an alert; an alert initiates investigation; an approved change enters a delivery pipeline; a new release is validated; and its consequences return through feedback and review.
The case study is a teaching system rather than a claim that one architecture fits every organization. Its components are intentionally small enough to run locally while preserving the reasoning used in larger production environments.
What This Guide Covers
The chapters move beyond deployment into:
DevOps foundations: environments, version control, automation, and repeatable delivery;
CI/CD: testing, building, releasing, and safely rolling back system changes;
MLOps: model and data lineage, experiment tracking, model versioning, and reproducible retraining;
observability: logs, metrics, traces, service health, and actionable alerts;
production monitoring: data quality, drift, prediction behavior, outcomes, and performance degradation;
reliability and scaling: capacity, resilience, failure recovery, and service-level objectives;
feedback and retraining: collecting evidence, validating labels, and deciding when change is justified;
governance: ownership, documentation, auditability, risk controls, and approval gates; and
human decision systems: placing model outputs within workflows where people remain responsible for consequential decisions.
What This Guide Does Not Repeat
This guide does not return to model selection, basic evaluation, API construction, request validation, or containerizing a model service. Those foundations belong to the earlier guides in the pathway.
Here, the model service is the starting component. The system around it is the subject.
How to Use This Guide
Work through the chapters in order because the case study accumulates operational components. You should:
run the provided scripts inside the repository-specific .venv;
inspect logs, metrics, test results, and generated artifacts rather than accepting successful commands at face value;
introduce controlled failures and observe how the system responds;
compare automated signals with the evidence required for a human decision; and
record assumptions, thresholds, owners, and rollback conditions as part of the system—not as afterthoughts.
Executable programs will live in scripts/python/, orchestration commands in scripts/bash/, and generated outputs in the appropriate results/, reports/, or data directories. Short code fragments in the chapters illustrate concepts without replacing the complete scripts.
By the End of the Guide
You should be able to explain and demonstrate how to:
operate a deployed model service through repeatable engineering workflows;
observe both software behavior and model behavior;
distinguish technical incidents from data or model degradation;
design safe release, rollback, feedback, and retraining processes;
apply governance and human review at meaningful decision points; and
evaluate the complete system by its reliability, consequences, and fitness for purpose—not only by a model metric.
CDI Perspective
Models produce estimates. Systems turn those estimates into actions and consequences.
The goal is therefore not merely to keep a model running. It is to build an operational process in which evidence is observable, changes are controlled, responsibilities are explicit, and human decisions remain informed and accountable.
What Comes Next
The next chapter establishes the working environment and the baseline deployed service that will anchor the case study. From there, each chapter adds one part of the operational system around it.
The model has been deployed. Now we build the system that makes it dependable.