AI & Algorithmic Systems

SECTION 01

Domain overview and system boundary

Define the automated decision before defining the model. A classifier used to suggest human review and the same classifier used to deny access have different consequence pathways. Record the target population, the action, the decision owner and the evidence available at the moment of action. Include upstream data collection and downstream appeals in the system boundary. This prevents a narrow accuracy metric from substituting for an assessment of the whole service. Model documentation can organise the intended-use claim, while independent local evaluation determines whether that claim is supported in the learning case. [A05] [A18]

AI and algorithmic systems convert observations into predictions, rankings, generated content or actions. The relevant system includes training data, target definitions, model parameters, interfaces, human decision makers and the institution that acts on the output. A high-performing benchmark model can fail in practice when its target is a poor proxy, its deployment population differs from the evaluation sample or its recommendations change the environment from which future data are collected. The analytical boundary should therefore include the decision and its consequences, not just the model endpoint.

This domain examines how errors become consequential and how an organisation can preserve a justified ability to detect, contest, constrain and reverse automated decisions. Its scope includes conventional scoring, computer vision, language models, recommender systems and autonomous agents. Scientific evaluation separates predictive accuracy, calibration, distributional performance, security, human factors and the reversibility of downstream actions. A c-ECO learning case should identify the affected people or processes, the decision owner, the time available for review and the evidence needed before automation can resume.

A domain label is not yet a case boundary. Separate the condition of a resource or asset from the function delivered to users. A nominally available installation may fail to deliver usable service, and a system-wide average can conceal localized loss. Preserve the original physical or institutional unit alongside any normalized representation. The boundary should make it possible to ask what enters, what changes, what leaves and who experiences the consequences.

Section readings [A05; A18]

SECTION 02

Historical development and changing conditions

Algorithmic systems have evolved from fixed scoring rules to learned predictors, foundation models and tools that act across external services. This history changes the location of uncertainty: an explicit rule may be inspectable but encode an invalid construct, while a flexible model may fit data well but behave unpredictably outside evaluation conditions. Do not equate greater complexity with greater capability or risk in every case. Build a timeline of changes to data, task, deployment scale and permissions. Historical benchmarks become relevant when their assumptions match the decision being examined, rather than because the technology names resemble one another. [A01] [A10]

A historical baseline must be examined for changes in exposure, technology, operating practice and observation. Compare episodes using their mechanisms and evidence, rather than treating all events with a similar label as exchangeable observations. A reconstruction should show what was known at each time; later explanations should not be presented as information available to the original decision maker. Recovery and long-term change may require a separate timeline from the initiating incident.

Section readings [A01; A10]

SECTION 03

Actors, institutions and decision responsibilities

Institutional responsibility remains distributed even when a model is supplied through one endpoint. Dataset custodians determine collection and permissions; developers select targets and evaluation procedures; deployers configure workflows; professional decision makers interpret outputs; affected people observe consequences that may be invisible in logs. Map these roles and their ability to change or stop the system. A procurement contract does not automatically supply evaluation evidence or an effective remedy. Identify who receives model-update notices, who retains decision records and who can authorise resumption after an incident. Lifecycle auditing provides a useful structure for this responsibility map. [A08] [A18]

Map actors by the function they perform and the responsibility they actually hold in the case. The same organization may observe a condition, supply a service and participate in a decision; those roles should remain distinguishable. Examine information access, resources, expertise, affected users and accountability together. A list of institution names is not an operating governance arrangement. Record the evidence for each responsibility and the process for resolving a disagreement or correcting a record.

Section readings [A08; A18; C07]

SECTION 04

Primary systemic risks

A primary risk is a mechanism connecting an input or design choice to a consequential error. Proxy mismatch, distribution shift, privacy leakage and unsafe actuation should be separately analysed even when the same model exhibits several of them. For each, identify the observable condition and the affected decision. An impressive held-out score cannot establish that the target represents the intended social or operational objective. Nor does the absence of observed harm establish safety when labels arrive late or affected people lack a reporting channel. The register should preserve these distinct pathways and their evidence requirements. [A06] [A09] [A13]

The inventory is a structured set of research hypotheses. It separates a threatening condition from the mechanism of loss and from its consequences. The proposed TFP association indicates which question to investigate; it is not evidence that a particular indicator predicts the event. Risk importance depends on the defined function, exposure, duration and usable alternatives. Do not infer frequency, severity or priority from the position of a row in the table.

Section readings [A06; A09; A13]

SECTION 05

Amplification mechanisms and secondary risks

Secondary risks often arise from the way people and institutions respond to automation. A recommendation can change access to services, which changes the population whose outcomes are later observed. Reviewers may accept confident outputs too readily, while a large alert volume can consume the capacity needed to detect genuine errors. Analyse the feedback loop explicitly: decision, changed behaviour, changed observation and retraining. The response should target the loop that amplifies the problem. A better classifier alone may leave the selection mechanism unchanged and therefore preserve the same blind spots in future data. [A07] [A01]

An amplifier changes the consequences of an initiating condition without necessarily causing the initiating event. Useful counterfactuals hold the threat constant and vary concentration, maintenance, access, information or response capacity. Document the mechanism and the observation that would distinguish it from an alternative explanation. Several amplifiers may share the same upstream cause; counting each one as independent evidence exaggerates confidence.

Section readings [A07; A01]

SECTION 06

Emerging risks and research horizons

Horizon scanning should distinguish a new capability from a demonstrated failure mechanism. Tool-using agents increase the importance of permission boundaries and action sequencing; synthetic training data raise questions about provenance and information retention; multimodal systems require tests for conflicting evidence. For each item, identify a concrete exposure, a measurable signal and a bounded experiment. Avoid a single speculative risk score for all emerging AI. Existing methods for reference standards, change control and deployment evaluation remain relevant, while the action space and evidence requirements must be updated for the particular system. [A16] [A19] [A22]

A research horizon is tied to the time needed to investigate, prepare or change a system. It is not a date at which an emerging threat is predicted to occur. Record the reason for concern, the strength of evidence, the exposed function and the observation that would change the priority. Distinguish a newly observed mechanism from a known mechanism whose exposure, scale or institutional context is changing.

Section readings [A16; A19; A22]

SECTION 07

Cascades and cross-domain dependencies

Algorithmic cascades can propagate through shared models, shared labels or coordinated actions. Multiple organisations may deploy the same provider model and therefore make correlated errors even when their local infrastructure is independent. A recommendation can also alter a physical or financial system, whose changed state becomes a new model input. Label the causal edge: data dependence, model dependence, behavioural feedback or direct actuation. Then identify the delay and the ability to interrupt the edge. A graph of software calls alone will miss the institutional decision and human response that often determine whether an error becomes systemic. [A01] [A07]

Proxy error occurs when the optimised label represents an administratively convenient variable rather than the outcome the decision is meant to improve. More data and a better fit to that label can strengthen the wrong allocation rule.

Distribution shift changes the relationship between evaluation evidence and deployment. Geographic, temporal, institutional and demographic shifts require distinct tests; a single pooled score can conceal failure in the context that matters.

Feedback can make prediction performative: a recommendation changes access, behaviour or exposure, and those altered outcomes become future training data. A retraining loop may reinforce its own selection effects.

For each directed dependency, specify the function supplied, the destination, the propagation time and the substitute. Shared-cause failure, interacting threats and cascading propagation are different structures. A directed arrow becomes an explanatory claim only when the timing and mechanism are supported. Test apparent redundancy by removing the common access route, platform, energy supply or specialist on which both alternatives depend.

Section readings [A01; A07; C01; C02]

SECTION 08

Readiness and response protocols

Response protocols should be tied to a defined consequence pathway. A rise in unsupported generated claims may justify restricting an advisory feature, whereas an unauthorised tool action may require immediate credential revocation and a review of external state changes. Specify the action owner, evidence threshold, scope and acceptance test. Suspending the model does not repair decisions already made. The protocol should therefore maintain separate technical containment and consequence-remedy tasks. Risk-management guidance helps organise these tasks, but the actual authority to stop or correct a service belongs to the responsible deploying institution. [A18] [A19]

Preparedness connects a signal to a responsible actor, an available measure and a condition of completion. This learning architecture does not grant emergency command or operational authority to the c-ECO Institute or a course participant. A useful protocol distinguishes checking the signal, protecting exposed functions and verifying recovery. A paper plan contributes to usable capacity only when its relevant assumptions have been tested.

Section readings [A18; A19]

SECTION 09

Scientific review and decision records

Independent review should challenge the target, the deployment population and the intervention design, rather than inspect only the final model score. Reviewers need access to evaluation definitions, subgroup denominators, version history and the workflow in which outputs are used. A disagreement about the legitimacy of a target cannot be resolved by adding another decimal place to accuracy. Record which claims are empirical, which are normative and which depend on a legal or professional authority. The review should also identify the observations that would change the decision to deploy, pause or restrict automation. [A08] [A10]

A review record separates observation, inference and decision. Record the data version, methods, alternative explanations, relevant expertise, conflicts of interest and unresolved disagreement. Review should be matched to the question and its consequences. Scientific analysis can inform a competent actor without replacing that actor’s mandate. No membership size, quorum or institutional power is established by this teaching library.

Section readings [A08; A10; C05; C07]

SECTION 10

Communication and observation continuity

Communication should describe the decision that is affected, the limits of present evidence and the route to human review. A generic statement that a system uses AI gives little operational information. Explain whether outputs are advisory, whether a person can override them and how an affected person can correct inaccurate inputs. During an incident, distinguish model withdrawal from correction of downstream consequences. Public claims about fairness or safety should identify the evaluation scope and period. Model cards can improve reporting, but accessible communication must also fit the language, literacy and circumstances of the people using the service. [A05] [A21]

Continuity of observation should be tested alongside continuity of the function itself. Losing data may increase uncertainty before the physical process visibly changes. Redundancy must be demonstrated against common causes, including power, location, communications, credentials and specialist availability. During disconnected operation, preserve the last reliable observation, record the uncertainty that follows and reconcile timestamps and authorship when information paths return.

Section readings [A05; A21; C01]

SECTION 11

International instruments and technical references

AI governance combines technical evidence, institutional responsibilities and applicable legal duties. A voluntary framework can organise analysis but cannot establish that a deployment is lawful or certified. Identify the jurisdiction, sector, decision type and affected population before checking requirements. Separate data permissions, model-development practices and the authority to act on an output. The case should retain a record of applicable instruments and the person responsible for their interpretation. Scientific evidence can test performance and expose a harmful proxy, while legal and ethical evaluation determines which uses and remedies are acceptable within the setting. [A18] [A08]

Classify each instrument before using it. A scholarly article supports an inference, a technical guide describes a method, a standard has a defined scope, and a legal instrument may create obligations only where applicable. The case record should identify the specific provision or method, its version and the activity to which it relates. A source’s institutional reputation cannot by itself establish applicability or demonstrate compliance.

Section readings [A18; A08]

SECTION 12

Jurisdiction-specific responsibilities and requirements

The deployment baseline should specify the exact model, prompt, retrieval collection, thresholds, interface and human workflow. A provider name is insufficient when behaviour changes across versions or configurations. Record the intended population and the reference standard used for evaluation. Preserve the date range and the cases excluded from testing. A local baseline should include a conventional or manual comparator so that automation is assessed against the actual alternative. Where a model is externally hosted, identify the evidence available after an update and the contractual process for learning about material changes. [A05] [A10]

The register should cover the actual activity and location, including interfaces between organizations and jurisdictions. Separate responsibility for observing, supervising, deciding and executing. Retain the official source, version consulted and evidence needed to assess a condition. Conflicting definitions and overlapping mandates are analytical findings to investigate. A general library cannot create a universal legal threshold for all settings.

Section readings [A05; A10]

SECTION 13

Causation, accountability and cascading consequences

Responsibility for an error cannot be assigned solely by identifying who trained the model. Target selection, procurement, interface design, staffing and the decision to automate can all contribute. A traceable record should connect the model output with the actual action and subsequent review. Preserve the distinction between an allegation, an investigation finding and a settlement obligation when studying historical cases. The purpose of accountability analysis is to identify who can prevent recurrence and deliver remedy, not merely to label the system opaque. An audit becomes useful when its findings have an identified decision owner and a follow-up record. [A08] [A21]

A technical chain of consequences is not automatically a legal allocation of liability. Separate evidence of causation from the duties, control and decisions relevant to each actor. A case may contain direct damage, dependent service interruption and persistent environmental or social consequences on different timescales. The analysis should show which links are documented and which need further investigation, without presuming an institutional restoration fund, mandatory levy or insurance mechanism.

Section readings [A08; A21]

SECTION 14

Cross-boundary coordination

Cross-border deployment changes the evidence needed for a justified use claim. A model evaluated in one language, institution or population may face different data collection, outcome prevalence and service constraints elsewhere. Translation can alter both inputs and explanations. Test the local decision process rather than assuming that a global model implies global validity. Data transfer permissions are separate from predictive performance. For an international learning case, retain the origin of evaluation data and identify which conclusions are portable, which need recalibration and which require a new reference standard or institutional agreement. [A09] [A17]

Processes and dependencies often cross administrative, organizational or national boundaries. Coordination requires compatible definitions, useful timing, authorized exchange and an identifiable decision interface. Sharing data does not by itself resolve disagreements over objectives, risk allocation or resources. An exercise should expose which decision can be coordinated under existing arrangements and which would require an additional agreement or competent determination.

Section readings [A09; A17; C07]

SECTION 15

Position (P): candidate observations

A candidate P construct should describe the remaining capacity to make a justified and correctable decision under the case conditions. It should not be reduced to model accuracy. Relevant observations may include local error, calibration, subgroup performance, review capacity and the availability of a safe fallback. Preserve each component in its original unit before considering a composite. An improvement in average accuracy cannot compensate automatically for a failure in a safety-critical group or an irreversible action. Any normalisation must state the decision tolerance and the reason that the selected metric represents it. [A02] [A03]

P represents position relative to a specified loss of function. The raw observation, reference condition, direction of deterioration and boundary of failure must remain explicit. A candidate monotone transformation can help compare states within a defined case, but it does not establish the threshold or validate an aggregate index. Capacity, delivered performance and exposure may require different indicators even when they concern the same installation.

A didactic monotone transformation for an indicator that worsens as it rises is p = clip[(x − x_ref)/(x_failure − x_ref), 0, 1]. Reverse the direction for a reserve that deteriorates as it falls. The expression makes the reference points visible; it does not choose those points, prove comparability or establish an operational aggregate. Retain x in its original unit and justify the function separately for each case.

Section readings [A02; A03]

SECTION 16

Velocity (ΔV): rates and trajectory

Velocity connects changing performance with the time available to intervene. Track error growth, review-backlog accumulation, model-update frequency and the speed of automated actions separately. A drift statistic can change before labels confirm an error, but it is not itself proof of harmful failure. Compare its trajectory with the delay required to obtain labels, evaluate a change and stop the affected workflow. Mark changes in population and evaluation procedure on the time series. Otherwise an apparent trend can reflect a new measurement process rather than a change in the deployed model. [A09] [A07]

ΔV describes direction and rate of change over a declared interval. Two systems at the same apparent position may have different reaction times because their trajectories differ. Compare rates using consistent units, observation windows and reference systems. Check seasonality, missing periods, reporting changes and process acceleration before extrapolating. A time-to-boundary calculation is conditional on its trajectory assumption; it is not a general forecast.

Section readings [A09; A07]

SECTION 17

Uncertainty (σ): evidence and model limits

Uncertainty includes sampling error, reference-standard disagreement, missing counterfactual outcomes, unknown deployment shift and model-selection variation. A confidence interval around pooled accuracy captures only part of this problem. Report subgroup sample sizes and explain where outcomes are unavailable because prior decisions prevented observation. Sensitivity to random seed or model choice is a separate issue from uncertainty about whether the target is valid. The case should map each uncertainty to the decision it could change. This prevents a precise statistical estimate from conveying unwarranted confidence in a poorly defined or incompletely observed decision process. [A03] [A10]

Uncertainty should be recorded by object and source. Measurement uncertainty, sampling limitations, parameter uncertainty, model structure, future scenarios and institutional ambiguity cannot be replaced by one unsupported percentage. More simulation samples can reduce numerical sampling error without resolving a missing mechanism. Identify which uncertainty changes the decision and which further observation, comparison or test could reduce it.

Section readings [A03; A10; C06]

SECTION 18

Reversibility liquidity (Lr): usable recovery resources

Recoverability concerns both the technical system and the people or processes affected by its actions. A prior model version may restore familiar behaviour while leaving erroneous denials, alerts or records in place. Inventory the ability to identify affected decisions, obtain human review and correct consequences. Some actions can be reversed quickly; others require compensation, notification or a physical intervention outside the model team’s control. Measure the tested throughput and quality of the fallback process. A stored checkpoint is a recovery resource, but it does not establish that the full decision service can resume safely. [A08] [A20]

Lr examines resources that can preserve, contain or recover the defined function within the relevant time. A resource listed in an inventory is different from a resource that is accessible, compatible, staffed and mobilizable. Record capacity, useful arrival time, duration, dependencies and evidence of testing. Avoid averaging away an essential bottleneck. Recovery of a service does not necessarily restore ecological condition, accumulated losses or public trust.

Section readings [A08; A20]

SECTION 19

Stress scenarios and comparative strategies

Scenarios should vary the causal mechanism and the level of automation. Compare a subgroup distribution shift, a corrupted data feed, a provider model update, a prompt-injection attempt and a surge that overwhelms human review. Include a case in which the model remains accurate but the target no longer reflects the intended objective. For each scenario, identify which observations remain available and which actions can be stopped or reversed. Historical audits supply plausible mechanisms, but their measured error rates should not be copied into another population as if they were universal parameters. [A04] [A06] [A19]

A stress scenario states the initiating condition, prior state, time sequence, exposed function, shared dependencies and feasible responses. Its purpose is to reveal when a strategy loses usefulness and why. Compare alternatives across several conditions, including absent substitutes and simultaneous demand. Scenarios should lead to observations, tests and resource questions; assigning a precise event probability requires a separate justified model and evidence.

Section readings [A04; A06; A19]

SECTION 20

Reproducible simulation and sensitivity analysis

A quantitative exercise should distinguish uncertainty in performance from uncertainty in consequences. Sampling an error rate does not establish how many people experience harm unless the deployment volume, decision pathway and remedy process are also defined. Correlated errors from a shared model or data source should not be simulated as independent events. State the population, observation delay, action rule and intervention capacity before calculating outcomes. Compare alternative assumptions about missing labels and human review effectiveness. The resulting distribution is conditional on the exercise design and should be used to identify sensitive assumptions, not to claim a validated field probability. [A09] [A10]

Simulation is an explicit transformation of assumptions. Input distributions, dependencies, parameter estimation, output definition and testing must be documented. Numerical convergence does not establish model validity. Examine sensitivity to the function used, the weights selected, dependence assumptions and conditions outside the fitting data. The reproducible laboratory below is a synthetic exercise, separated from SPS and from any claim about real-event frequency.

The factor loadings define association on the latent scale. Correlations after the nonlinear transform should be estimated from the simulated samples. Q is a teaching index and is not SPS. Its exceedance fraction concerns the generated sample under these assumptions; it is not the probability of a real event or a forecast over a stated number of years. A temporal forecast would require a separately specified process and observation model.

from random import Random
from math import log, exp, sqrt
from statistics import mean
rng = Random(854081170)
m = [.72, .62, .40, .38]
s = [.40, .45, .30, .40]
a = [.65, .55, .35, -.45]
q = []
for _ in range(10000):
    g = rng.gauss(0, 1)
    x = []
    for mj, sj, aj in zip(m, s, a):
        z = aj*g + sqrt(1-aj*aj)*rng.gauss(0, 1)
        x.append(1/(1+exp(-(log(mj/(1-mj))+sj*z))))
    p, v, u, lr = x
    q.append(.35*p+.25*v+.15*u+.25*(1-lr))
q.sort()
print(mean(q), q[499], (q[4999]+q[5000])/2, q[9499])
print(sum(x >= .70 for x in q)/len(q))

Sensitivity exercise: remove the common factor, vary one dispersion, change the weight of recovery resources, and compare the ranking of the selected strategies. Then state which domain observations would be needed to replace each illustrative assumption. Use a suitable estimator when inputs are dependent; the common-factor experiment is a dependence check, not a claim that a particular Sobol estimator applies unchanged.

Section readings [A09; A10; C03; C06]

SECTION 21

Restoration and performance verification

A restoration exercise should begin with a precise stopping condition and a record of external actions already taken. Revoke unnecessary tool permissions, preserve the disputed configuration and route cases to a tested alternative. Then verify the proposed fix on a clean local evaluation set and on the failure pathway that triggered the incident. Include subgroup and workflow checks before resuming use. A model rollback can fail if preprocessing, retrieval or client interfaces changed at the same time. The final acceptance test should cover the complete decision service and the remaining remedy backlog. [A01] [A08]

Restoration follows prerequisites and verification conditions rather than a universal fixed clock. Tasks may proceed in parallel only where they do not compete for the same people, access, equipment or information. Protect essential functions while the critical path is addressed. Define what is measured before returning service, who can make that determination and which effects require continued observation after a technical restart.

Section readings [A01; A08]

SECTION 22

Recovery provider capability

An AI assurance or restoration provider should demonstrate competence in the task, population and failure mechanism. General machine-learning expertise does not establish clinical judgement, legal authority or the ability to conduct a representative subgroup evaluation. Ask for reproducible methods, independence, data-access safeguards and a clear treatment of uncertainty. Separate the organisation that supplies the model from the party that accepts the validation result when conflicts could affect judgement. The engagement should define whether the work tests predictive performance, security, human workflow or remedy. A broad assurance label is insufficient without those bounded claims and evidence. [A08] [A18]

Capability assessment must be specific to the task, place and stress conditions. A training certificate cannot demonstrate that a team can mobilize and deliver the required function. Inspect competence, safety, compatibility, relevant experience and actual availability. Define the verification method and limits of recognition. This teaching library does not establish a c-ECO provider certification scheme or substitute for professional or regulatory requirements.

Section readings [A08; A18]

SECTION 23

Recovery sequencing and time constraints

Recovery sequencing should preserve evidence, constrain actions, identify affected decisions, establish a safe alternative and only then evaluate a corrected system. These steps may run partly in parallel, but each has a different owner and completion criterion. A security fix does not establish that the target is valid, and a new evaluation score does not establish that affected people have received remedy. Track technical recovery and consequence correction separately. Before expanding automation again, verify the original failure mechanism and the operational ability to detect a recurrence within the relevant action window. [A13] [A20] [A21]

A recovery time objective is a decision criterion to test against real capacity. Distinguish mobilization, work, verification and decision time, and identify dependencies between tasks. Shared resources can prevent apparently parallel activities from proceeding together. Repeating the sequence with unavailable access, a specialist or a verification service reveals where a target relies on an untested assumption.

Section readings [A13; A20; A21]

SECTION 24

Equipment, inputs and replenishment

The inventory should contain reproducible model artefacts, clean data snapshots, evaluation definitions, reference labels, access controls and qualified human capacity. A model checkpoint without its preprocessing and decision thresholds may not reproduce a disputed output. A manual fallback without staffing estimates may fail under incident volume. Record provenance, permissions, custody and the latest demonstrated test for each resource. Include the records needed to locate affected decisions without exposing unnecessary personal information. The inventory becomes useful when it is linked to a recovery task and reveals a missing dependency or an unavailable authorised reviewer. [A05] [A08]

Inventory analysis connects each item to a protected function. Use condition, compatibility, accessibility and consumption under stress rather than a nominal stock count alone. Replenishment time must include procurement, transport, installation and verification where relevant. A substitute that depends on the same supplier or platform may disappear in the same event. Keep the last inspection date and the evidence supporting usable availability.

Section readings [A05; A08]

SECTION 25

Mutual aid and cooperation arrangements

Mutual assistance should be bounded by expertise, access and authority. An external evaluator may help test distribution shift without receiving unrestricted access to identifiable records or production tools. Predefine confidentiality, permitted use, evidence retention and the process for reporting a serious finding. Verify that apparently independent evaluators do not rely on the same provider or benchmark without recognising the dependence. Exercises should test the delivery of assistance within the case’s decision window. Cooperation is a resource only when the relevant skills, data access and acceptance responsibility are available under the actual scenario conditions. [A18] [A08]

Cooperation contributes to recovery when its scope, activation, resources and practical availability are demonstrated. Record who requests support, who authorizes it, the useful arrival time, responsibility for the work and conditions of return. Consider simultaneous demand across participants. A reference to an organization or network in the literature does not imply that c-ECO or a case partner has joined it or secured a deployment commitment.

Section readings [A18; A08]

SECTION 26

Source classification and evidence quality

Classify evidence by what it establishes. A benchmark paper reports performance within a defined evaluation; a field study observes deployment conditions; an official investigation documents a particular incident; a reporting framework proposes a structure. These sources should not be collapsed into one authority score. Record version, population, task and study period. A historical audit of commercial systems does not establish the current performance of their successors. Conversely, a new product score does not erase the mechanism identified by an earlier study. The library should preserve both the bounded finding and the analytical question that remains relevant. [A04] [A17]

Source classes organize origin and function; they do not form an automatic hierarchy of evidential strength. An official statistic, an operational record and a causal study answer different questions. Classify the product, not only the organization. Detect shared data origins so that a dashboard, article and secondary report based on one observation are not counted as three independent confirmations.

Section readings [A04; A17; C04; C05]

SECTION 27

Institutional, regulatory and technical sources

Official technical resources help connect governance objectives with activities such as mapping use, measuring performance and managing incidents. Their role is to structure questions and evidence, not to certify the model automatically. For the learning case, select the objectives that address a documented failure pathway and identify how implementation would be demonstrated. A generative-AI profile can reveal risks involving content, privacy or tool use, while a sector-specific authority may impose different professional requirements. Record the edition and scope of the resource and avoid presenting a voluntary framework as a universal legal obligation. [A18] [A19]

Institutional sources should be selected by product, method and scale. Record whether the source defines a requirement, observes a condition, reports an investigation or synthesizes a literature. A current portal may contain several editions and methods; cite the actual version used. Public access does not establish accuracy for a different question, and a restricted source can still be used only under its authorized conditions.

Section readings [A18; A19]

SECTION 28

Operational observations and monitoring products

Operational evidence includes input distributions, output records, version history, action logs, review decisions and reported consequences. Each has a different observation bias. Logs may omit people who abandon a service, while appeals may overrepresent those able to navigate the review process. Align records with a common decision identifier and retain the transformations used for analysis. A provider’s model card is an intended-use statement, whereas local monitoring tests actual deployment conditions. Compare these layers rather than treating them as substitutes. Evidence retention should be proportionate to the evaluation need and governed by appropriate access safeguards. [A05] [A08]

Operational observations should show what the system delivered, not merely that an asset was declared available. Inspect the acquisition process, sensor or reporting changes, gaps, coverage and quality controls. Compare records with independent observations where feasible. Aggregated performance, a status dashboard and user experience can diverge because they concern different units, locations, periods or definitions of availability.

Section readings [A05; A08]

SECTION 29

Academic research and independent evidence

Research papers are most useful when their assumptions are translated into case-specific tests. Calibration research motivates checking probability reliability; distribution-shift benchmarks motivate evaluating local contexts; performative prediction motivates tracing feedback. No single paper supplies a universal checklist for every model. Read the task, dataset, comparison and limitations before importing a method. A local explanation can describe a surrogate around one prediction without establishing the causal reason for an outcome. The learning exercise should ask what each method reveals, what it leaves unidentified and which additional evidence would resolve the decision-relevant gap. [A03] [A07] [A11]

Read the research design before transferring a conclusion. Identify the question, unit of analysis, method, comparison, result, limitations and conditions that matter for the case. Foundational concepts, empirical studies, critical replications and technical syntheses serve different functions. A literature-supported mechanism may justify a hypothesis without validating a local threshold, forecasting performance or a complete TFP application.

Section readings [A03; A07; A11]

SECTION 30

Source–variable correspondence

A source-to-variable matrix should identify the observation, transformation and denominator behind every metric. An error rate requires a reference outcome; an abstention rate requires the full set of cases presented; a remedy metric requires a defined population of affected decisions. Separate measured performance from expert judgement about consequence severity. Retain model and dataset versions so that a changed result can be attributed to its cause. When labels arrive late, distinguish provisional monitoring signals from confirmed outcomes. This traceability makes re-evaluation possible without silently rewriting what was known at the time of the original decision. [A05] [A08]

The source–variable matrix exposes the path from an observation to an analytical representation. Preserve the raw unit, time, location, method, transformation and uncertainty. Multiple derived variables may share one observation and therefore share its limitations. Record the code or reproducible procedure, authorized access and version. A score should remain inspectable through its originating evidence rather than replacing that evidence.

Section readings [A05; A08; C04; C05]

SECTION 31

Source–risk correspondence

A source-to-risk mapping should show how evidence supports a particular failure mechanism. A drift detector can identify changed inputs but cannot prove target invalidity. A subgroup audit can reveal unequal errors while leaving the reason for the disparity unresolved. An incident complaint can identify a concern without establishing every alleged fact. For each risk, record the confirming evidence, relevant counterevidence and unobserved conditions. Reusing one benchmark across several risks does not create independent support. The matrix should guide the next investigation step rather than turn the presence of a citation into a binary validation label. [A04] [A09] [A21]

A source–risk matrix should cover the threat, the observed state, the mechanism and the consequences separately. Evidence that an event occurred does not alone show why a function failed or who was affected. Use a chronology and alternative explanations to test the chain. A citation becomes useful when the specific part of the claim it supports and its limits are visible.

Section readings [A04; A09; A21]

SECTION 32

Monitoring gaps and research priorities

Monitoring gaps include unlabelled cases, small subgroups, excluded languages, delayed outcomes and consequences outside the deploying organisation’s records. These gaps are not random by default. Build a register that states which inference is weakened and what alternative evidence could help. A low appeal rate can indicate low harm, but it can also indicate an inaccessible remedy process. A falling error rate can reflect selective case exclusion. Track changes to the monitored population and distinguish missing data from a negative finding. The purpose is to make uncertainty actionable rather than hide it behind a single aggregate confidence score. [A06] [A17]

Prioritize a gap by its consequence for the decision. The next observation may be a targeted measurement, an independent comparison, a capability test or a correction of incompatible definitions. More data of the same kind may not resolve structural uncertainty. Specify the question, required observation, responsible custodian, time of collection and condition under which the gap is considered sufficiently addressed.

Section readings [A06; A17; C03]

SECTION 33

Dependency matrix and propagation tests

Cross-domain analysis should follow the action, not merely the model output. An inaccurate water forecast becomes consequential through an operational decision; a health score through allocation; a financial alert through review or restriction. Map the receiving institution, the time tolerance, the independent safety barrier and the route for correction. Shared models can introduce correlated judgement across otherwise separate sectors. The dependency register should therefore distinguish common infrastructure, common evidence and common decision rules. This gives reviewers a concrete basis for testing whether a local error could propagate and where a human or technical barrier can interrupt it. [A01] [A07]

Underspecification arises when several models satisfy the same training and validation criteria but differ on deployment-relevant behaviour. Development randomness and untested features can then matter more than the headline score suggests.

Confidence is a model output, not an automatic probability of correctness. Calibration depends on the evaluation population, binning and task; it can deteriorate after a distribution shift.

Human oversight works only if reviewers have time, information, authority and a usable alternative. A nominal approval button may introduce delay without providing effective detection or reversal.

A dependency matrix should be directional and time-specific. Record the supplying function, receiving function, mechanism, useful delay, substitute and evidence. Test the destination when the origin becomes unavailable for different periods. Then remove a common prerequisite of the substitute. The resulting structure describes the case; it does not establish all connections in a global architecture or validate an edge weight.

For each relationship, add the measured delay, duration of tolerable interruption, substitute and observation supporting the directed edge. Repeat the test for short, intermediate and extended unavailability. A generic list of linked sectors is not an estimated network, and no edge weight is validated by inclusion in this table.

Section readings [A01; A07; C01; C02]

SECTION 34

Documented cases, near misses and recovery

Historical cases provide different forms of learning. Gender Shades and the health-need algorithm study are empirical audits; the Thailand work is a field evaluation; the Tempe report is an accident investigation; the Rite Aid record contains regulatory allegations and legal outcomes. Compare the mechanism, evidence type and decision pathway rather than aggregating these records into a generic AI incident rate. Ask which failure assumption was exposed and which observation could have changed the deployment decision. Keep the original scope visible, especially when the evaluated model, population or institutional setting differs from the present case. [A04] [A06] [A17] [A20] [A21]

Historical analysis distinguishes observed loss, protective response, near miss and recovery. Several records can concern one event without becoming independent events. Preserve what the source actually investigated and avoid adding unsupported casualty, cost or restoration-time estimates. Compare cases by mechanism and evidence quality; the fact that one response was associated with an outcome does not alone establish a general causal effect.

Section readings [A04; A06; A17; A20; A21]

SECTION 35

Integrated review, transfer and learning portfolio

AI can help organise SIL evidence, but generated summaries and classifications require source-level verification. A fluent answer may omit a qualification, cite an irrelevant document or fail to use evidence buried in a long context. Preserve the underlying source, the claim being supported and the human review record. Treat model-generated analytical suggestions as candidates for investigation rather than validated domain relationships. Cross-domain use is strongest when automation improves traceability and coverage while domain experts retain responsibility for causal interpretation, measurement quality and the consequences of a decision. [A14] [A15] [A18]

The integrated portfolio should allow a reviewer to move from the question to observations, transformations, mechanism tests, scenarios and proposed monitoring. Make the strongest competing explanation and the most consequential unresolved gap visible. Transfer requires checking mechanisms, exposure and recovery resources in the new setting. Formation, optional credentialing, submission permission and licensing remain distinct processes; no case or data contribution enters the SIL without an associated fellow and applicable review.

Section readings [A14; A15; A18; C04; C05]