Security, Governance, and Risk

Learning objectives

By the end of this chapter, you will be able to:

• distinguish security, governance, and risk management in a model system;
• identify threats across data, model, service, and human decision boundaries;
• apply layered controls instead of relying on a single approval or technical safeguard;
• define ownership, evidence, and escalation for important system risks;
• implement policy gates that fail safely and leave an auditable record; and
• evaluate residual risk before a system is released or allowed to continue operating.

From reliable operation to responsible control

The preceding chapters developed the operational capabilities needed to deliver, observe, scale, monitor, retrain, and version a model system. Those capabilities answer questions such as: Is the service available? Is its behavior changing? Can the team reproduce the deployed model? Security, governance, and risk management add a different set of questions:

• Who is allowed to use, modify, approve, and inspect the system?
• Which data and actions must never be exposed to an unauthorized party?
• What evidence demonstrates that a release satisfied policy?
• Who accepts the remaining risk, and under which conditions?
• What happens when a control fails or the system leaves its approved operating envelope?

A technically reliable service can still be unsafe. It may expose sensitive inputs in logs, use an artifact from an untrusted source, silently serve an expired model, or automate a high-impact decision without an accountable reviewer. Conversely, a policy document does not make a system governed unless its requirements are translated into owners, controls, evidence, and enforceable decisions.

Security, governance, and risk management therefore operate as a connected system:

Discipline	Primary question	Typical mechanisms	Evidence
Security	How can assets and operations be protected?	authentication, authorization, encryption, isolation, validation, secret management	access logs, scan results, key records, incident records
Governance	Who may decide what, under which rules?	roles, approvals, policies, review boards, model cards, change records	signed approvals, decision logs, lineage, exceptions
Risk management	What can go wrong, how serious is it, and what remains?	risk identification, scoring, treatment, monitoring, acceptance	risk register, control tests, residual-risk decisions

The three disciplines overlap, but they are not interchangeable. Security controls reduce particular threats. Governance makes authority and accountability explicit. Risk management helps the organization prioritize controls and decide whether the remaining exposure is acceptable.

Protect the complete model system

The attack surface extends beyond the prediction endpoint. A production model system includes source code, dependencies, training data, feature pipelines, model artifacts, container images, deployment configuration, credentials, telemetry, feedback, and human workflows. A useful review follows information and authority through the complete system rather than inspecting only the API.

Assets, actors, and trust boundaries

Start by identifying three things:

1. Assets are items whose confidentiality, integrity, availability, or permitted use matters. Examples include personal data, labels, model artifacts, signing keys, predictions, audit logs, and approval records.
2. Actors are people or services that interact with those assets. Examples include developers, reviewers, operators, clients, automated retraining jobs, and external vendors.
3. Trust boundaries are places where data or authority crosses between environments, identities, teams, or organizations.

For the continuing decision-service case study, important trust boundaries include the client-to-API boundary, the service-to-feature-store boundary, the CI pipeline-to-registry boundary, and the model recommendation-to-human-review boundary. Each crossing should have an explicit identity, allowed action, validation rule, and audit event.

Threats across the lifecycle

Threat modelling is most useful when performed before implementation and revisited after architectural change. The team should ask how an attacker, faulty component, or authorized-but-mistaken user could cause harm.

System area	Example threat	Consequence	Example controls
Data intake	malformed or poisoned inputs	incorrect predictions or service failure	schema checks, range checks, provenance, quarantine
Training pipeline	unauthorized data or code change	compromised model behavior	protected branches, reviewed changes, signed runs, lineage
Dependencies	vulnerable or substituted package	code execution or data exposure	lock files, scanning, trusted registries, minimal images
Model registry	artifact replacement	unreviewed model reaches production	immutable versions, checksums, signatures, promotion roles
Prediction API	stolen credential or excessive access	unauthorized inference or extraction	strong identity, scoped authorization, rate limits, rotation
Logging	raw sensitive fields recorded	privacy breach	data minimization, redaction, access controls, retention limits
Feedback loop	manipulated outcomes	biased or degraded retraining	source validation, delayed labels, review, anomaly detection
Human workflow	recommendation treated as final decision	unaccountable high-impact action	role separation, explanations, appeal and override paths

This review should include accidental failure and misuse, not only deliberate attack. A permissive service account, an incorrectly configured storage bucket, or an operator bypassing a review step can be as damaging as an external intrusion.

Layered security controls

No single control should carry the full protection burden. Layered controls reduce the chance that one failure becomes a system-level incident.

Identity and least privilege

Authentication establishes identity; authorization determines what that identity may do. Every human and workload identity should receive only the permissions required for its task, for the shortest practical duration.

Useful separations include:

• training jobs may read approved training data but cannot promote models;
• deployment jobs may retrieve an approved artifact but cannot modify its lineage;
• application services may make predictions but cannot read signing keys;
• reviewers may approve a release but cannot rewrite its test evidence; and
• emergency access is time-limited, logged, and reviewed after use.

Avoid shared credentials. Short-lived workload identities and role-based permissions make attribution and revocation easier. Authorization should be enforced by the receiving component, not merely assumed because the request originated inside a private network.

Secrets, encryption, and sensitive data

Secrets must not be embedded in source code, container images, notebooks, or model artifacts. Store them in an approved secret manager, inject them at runtime, rotate them, and monitor access. Encrypt sensitive data in transit and at rest, while recognizing that encryption does not correct excessive collection or overly broad access.

Data minimization is often the strongest privacy control: do not collect, transmit, log, or retain a field unless the system has a defined need for it. When sensitive inputs are required, define their purpose, allowed users, retention period, deletion process, and whether they may be used for monitoring or retraining.

Software and artifact integrity

Model systems inherit software supply-chain risk. Dependencies, base images, build actions, and downloaded artifacts should be pinned or otherwise constrained to reviewed versions. Build outputs should be immutable and traceable to source, tests, configuration, and data lineage.

A secure promotion process should verify at least:

• the source revision was reviewed;
• required tests and scans completed successfully;
• the model artifact checksum matches the registered version;
• the container image is the same candidate tested in staging;
• the deployer is authorized for the target environment; and
• rollback points and incident ownership are known.

These checks belong in automation where possible. Manual judgment remains necessary for context-dependent risk, but repeated mechanical checks should not depend on memory.

Safe interfaces and observable failures

External inputs are untrusted. Validate types, required fields, ranges, sizes, encodings, and allowed values before they reach model code. Apply request limits and timeouts, and return errors that are useful without exposing internal paths, stack traces, secrets, or sensitive records.

Security-relevant events should be observable: failed authorization, unusual request rates, artifact verification failures, policy overrides, changes to privileged roles, and access to sensitive datasets. Logs themselves require protection because they may contain identities, system structure, and decision history.

Governance as an operating system

Governance is not a committee added after deployment. It is the set of decision rights and evidence requirements embedded throughout the lifecycle.

Assign decision rights

For every material change, distinguish who proposes, reviews, approves, executes, and monitors it. Small teams may combine roles, but the responsibilities should remain explicit. High-risk changes benefit from separation between the person who builds the change and the person who authorizes production use.

Decision	Accountable role	Required evidence
approve a new data source	data owner	purpose, provenance, quality, permission, retention
register a model version	model owner	training run, evaluation, limitations, lineage
promote to production	service owner or release authority	test results, security checks, rollback plan, approvals
change a decision threshold	policy owner	impact analysis, validation, affected groups, rationale
approve a temporary exception	risk owner	scope, compensating control, expiry, remediation owner
retire a model	model and service owners	replacement or shutdown plan, record retention, client notice

Approval should apply to a precise object: a data snapshot, code revision, model version, image digest, configuration, and target environment. An approval detached from versions cannot prove what was actually deployed.

Produce durable evidence

Governed systems create evidence as work occurs. Useful records include:

• intended use, excluded use, known limitations, and model owner;
• data provenance, consent or legal basis, quality checks, and retention;
• experiment lineage and evaluation results;
• review comments, approval identity, timestamp, and approved versions;
• deployments, configuration changes, rollbacks, and overrides;
• monitoring alerts, incidents, corrective actions, and risk acceptance; and
• user notices, appeal outcomes, and recurring governance reviews.

Evidence must be protected from unauthorized alteration and retained for a period aligned with operational, contractual, and legal needs. Retaining everything indefinitely is not governance; it can create unnecessary privacy and security exposure.

Manage exceptions explicitly

Real systems sometimes operate under temporary exceptions. A good exception identifies the policy being waived, why the normal control cannot be met, the affected scope, the compensating control, the approving risk owner, and an expiry date. Expired exceptions should fail closed or trigger escalation rather than becoming permanent through silence.

Risk assessment and treatment

A risk statement should connect a cause, an uncertain event, and an impact. For example: Because production logs may contain raw request fields, unauthorized log access could expose sensitive client data, causing privacy harm and loss of trust. This is more actionable than a label such as “privacy risk.”

Score inherent and residual risk

A simple risk matrix can help prioritize work:

[ = ]

If likelihood and impact each use a 1–5 scale, scores range from 1 to 25. The score before controls is inherent risk. The score after considering the design and effectiveness of controls is residual risk.

The arithmetic is a decision aid, not an objective truth. Two risks with the same score may require different responses, and rare catastrophic harms should not disappear inside an average. Record the rationale, uncertainty, affected stakeholders, and control effectiveness alongside the number.

Choose a treatment

Each material risk needs an owner and one of four deliberate treatments:

• avoid the activity that creates the risk;
• reduce likelihood or impact through controls;
• transfer part of the exposure contractually or through insurance, without transferring accountability; or
• accept the residual risk through an authorized, time-bounded decision.

A risk register is useful only when it drives action. Each entry should include the risk statement, owner, inherent rating, controls, residual rating, treatment, review date, and status. Controls should have test procedures and evidence, not just names.

Connect thresholds to action

Policy gates turn governance requirements into repeatable release or runtime decisions. A gate can return:

• allow when all mandatory controls pass;
• review when operation may continue only with authorized judgment; or
• block when a prohibited condition exists.

Fail-safe design gives the most restrictive appropriate outcome when critical evidence is missing. For example, a missing authorization decision should not be interpreted as approval. At the same time, a blanket block for every anomaly can make the service unusable and encourage bypasses. The response should match the risk: reject an unauthorized request, quarantine an invalid record, route an uncertain high-impact recommendation to review, or stop promotion when artifact integrity cannot be verified.

Case-study implementation

The companion program converts a synthetic stream of decision-service requests into governance outcomes. Each request may present one or more control conditions:

• authorization failure;
• missing permission for sensitive processing;
• invalid input range;
• a critical integrity event;
• an expired model approval;
• elevated drift requiring review; or
• a high-impact recommendation requiring human review.

The policy engine uses an ordered set of rules. Conditions involving identity, permission, input validity, integrity, or approval status block automated processing. Drift and high-impact recommendations are routed to review. Only requests that pass all mandatory controls are allowed automatically.

Run the implementation from the repository root:

bash scripts/bash/11-run-governance-risk-assessment.sh

The script creates:

• results/11-governance-decision-summary.csv, a summary by policy outcome;
• results/11-risk-register.csv, a compact risk register with inherent and residual scores;
• results/11-governance-assessment.json, machine-readable metrics and gate status; and
• results/figures/11-security-governance-risk.png, a three-panel governance report.

The implementation is intentionally transparent. Production policy engines may be more sophisticated, but their rules should remain versioned, testable, reviewable, and explainable to the people accountable for their consequences.

Figure 12.1: Governance outcomes, policy triggers, and risk reduction in the case-study simulation.

Interpret the evidence

The figure should be read as a control report rather than a model-performance report.

• The policy outcomes panel shows how much traffic is allowed, routed to review, or blocked.
• The control triggers panel shows which rules create those outcomes. A rising trigger count may represent attack, upstream quality failure, policy change, or an instrumentation change; investigation requires context.
• The risk register panel compares inherent and residual scores. A lower residual score indicates expected risk reduction, not proof that the control always works.

The generated JSON includes an overall gate status. A critical integrity event produces a blocked gate even if aggregate rates appear acceptable. This prevents a low average from concealing a severe event.

Operational checklist

Before release, confirm that:

• system assets, actors, trust boundaries, and credible misuse cases are documented;
• human and workload identities use least privilege and are independently attributable;
• secrets are managed outside source code and build artifacts;
• sensitive data is minimized, protected, and governed by defined retention rules;
• dependencies, model artifacts, and container images are traceable and integrity checked;
• policy gates refer to immutable versions and fail safely when critical evidence is absent;
• material risks have owners, treatments, control tests, and review dates;
• approvals, exceptions, overrides, and deployments create durable audit evidence;
• monitoring covers both technical failures and violations of approved use; and
• incident response includes containment, stakeholder communication, recovery, and learning.

Chapter summary

A production model system must be secure enough to resist misuse and failure, governed enough to make authority and evidence visible, and risk-managed enough to prioritize action and justify residual exposure. These outcomes come from layered technical and organizational controls across the complete lifecycle—not from a final compliance review.

The practical pattern is consistent: identify assets and threats, assign accountable owners, implement controls, test their effectiveness, record evidence, and connect policy thresholds to explicit allow, review, or block actions. The next chapter extends this foundation into human decision systems, where recommendations, judgment, overrides, explanations, and appeals must work together as part of the system design.