Appendix

Published

Aug 2026

  • ID: ADS-APP
  • Type: Reference
  • Audience: Intermediate to Advanced
  • Theme: A defensible analysis is reproducible from question to decision

This appendix consolidates the reusable structures, commands, modelling patterns, and review checks developed throughout Advanced Data Science. Use it as an operational reference—not as a substitute for the reasoning in the main chapters.

Reproducible Project Structure

applied-data-science/
├── data/
│   ├── raw/
│   ├── processed/
│   └── reference/
├── models/
├── notebooks/
├── reports/
├── results/
│   ├── figures/
│   └── tables/
├── scripts/
│   ├── bash/
│   └── python/
├── .gitignore
├── _quarto.yml
├── requirements.txt
└── requirements-lock.txt
  • Preserve data/raw/ as the unchanged source layer.
  • Write derived datasets only to data/processed/.
  • Keep reusable analysis and figure generation in chapter-prefixed scripts.
  • Separate models, figures, tables, and narrative reports.
  • Record direct dependencies in requirements.txt and, when needed, freeze the complete working environment in requirements-lock.txt.

This separation makes computational steps easier to inspect, rerun, and revise. Reproducibility requires more than sharing code: the data, software environment, execution order, and analytical decisions must also be sufficiently documented (Peng 2011). Data and metadata intended for reuse should additionally follow the FAIR principles of findability, accessibility, interoperability, and reusability (Wilkinson et al. 2016).

Essential Commands

Activate the project environment

source .venv/bin/activate
which python
python --version

Install or reproduce dependencies

python -m pip install -r requirements.txt
python -m pip freeze > requirements-lock.txt

Run a chapter script

python scripts/python/09-generate-model-evaluation-figures.py

Render the complete guide

quarto render

Leave the environment

deactivate

Always run project scripts from the repository root unless a script explicitly states otherwise.

End-to-End Analytical Workflow

  1. Frame the question. Define the population, outcome, unit of analysis, time horizon, and intended decision.
  2. Audit the data-generating process. Identify selection, measurement, missingness, timing, and access constraints.
  3. Define the evaluation design. Choose the split strategy, baselines, metrics, and acceptance criteria before model comparison.
  4. Prepare data within the design. Fit imputation, scaling, encoding, feature selection, and resampling using training data only.
  5. Establish a baseline. Compare sophisticated candidates with a simple, reproducible reference.
  6. Develop candidates. Change one justified component at a time and evaluate it consistently.
  7. Validate performance. Examine discrimination or error, calibration where relevant, uncertainty, stability, and subgroup results.
  8. Interpret cautiously. Distinguish model behaviour from causal explanation and test whether conclusions survive plausible alternatives.
  9. Connect evidence to action. Evaluate thresholds, costs, benefits, capacity, and consequences of errors.
  10. Communicate and document. Report methods, results, limitations, and decision conditions clearly enough for review.

Leakage-Safe Modelling Pattern

Scikit-learn provides a consistent interface for preprocessing, modelling, validation, and pipelines (Pedregosa et al. 2011). The following pattern keeps learned preprocessing inside cross-validation.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric_pipeline = make_pipeline(
    SimpleImputer(strategy="median"),
    StandardScaler(),
)

categorical_pipeline = make_pipeline(
    SimpleImputer(strategy="most_frequent"),
    OneHotEncoder(handle_unknown="ignore"),
)

preprocessor = ColumnTransformer(
    transformers=[
        ("numeric", numeric_pipeline, numeric_features),
        ("categorical", categorical_pipeline, categorical_features),
    ]
)

model = make_pipeline(
    preprocessor,
    LogisticRegression(max_iter=2_000),
)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_validate(
    model,
    X_train,
    y_train,
    cv=cv,
    scoring={
        "roc_auc": "roc_auc",
        "average_precision": "average_precision",
        "neg_log_loss": "neg_log_loss",
    },
    return_train_score=True,
)

The pipeline must include every operation that learns from data. If the observations are grouped, spatial, or time ordered, replace ordinary random folds with a split strategy that respects that dependence.

Evaluation Quick Reference

Task Useful evidence Common mistake
Regression MAE or RMSE, residual patterns, uncertainty, subgroup error Reporting one average error without scale or context
Balanced classification Confusion matrix, sensitivity, specificity, ROC AUC, calibration Treating accuracy as sufficient
Imbalanced classification Precision, recall, average precision, calibration, threshold consequences Relying on ROC AUC alone
Probability estimation Log loss, Brier score, calibration curve Assuming good ranking means reliable probabilities
Unsupervised learning Compactness, separation, stability, sensitivity to preprocessing Treating clusters as discovered truth
Decision support Expected benefit or cost, capacity, threshold sensitivity Selecting a threshold from statistical convenience

Compare validation results with a meaningful baseline, report variation across folds or repeated samples, and reserve the test set for the final selected procedure. Transparent prediction-model reporting should describe the data, participants or observations, predictors, outcomes, missing-data handling, validation design, performance measures, and limitations (Collins et al. 2015).

Interpretation Guardrails

  • Association is not causation. Predictive importance can arise from correlation, proxies, confounding, measurement practices, or leakage.
  • Global and local explanations answer different questions. A global summary describes average model behaviour; a local explanation concerns one prediction.
  • Importance is conditional on the fitted model and available features. Correlated variables can share, mask, or redistribute apparent importance.
  • Explanations do not validate a model. Performance, calibration, robustness, and fitness for the intended use require separate evidence.
  • Subgroup results need adequate sample sizes and uncertainty. Small differences may be unstable, while aggregate performance can conceal important failures.

SHAP is one principled framework for assigning feature contributions to individual predictions, but its outputs remain explanations of a fitted model rather than causal effects (Lundberg and Lee 2017).

Decision and Responsible-Use Checklist

Before recommending or using an analytical result, ask:

  • What decision will this result inform, and who is affected?
  • Are the development data representative of the intended setting?
  • Which errors are most harmful, and to whom?
  • Is the chosen threshold consistent with costs, benefits, and operational capacity?
  • Are probability estimates sufficiently calibrated for the decision?
  • Does performance remain acceptable across relevant groups and conditions?
  • What happens when data quality deteriorates or the population changes?
  • Can a person question, override, or appeal a consequential output?
  • Who monitors the system, and what triggers review, retraining, or withdrawal?
  • Are privacy, security, legal, and domain-specific requirements satisfied?

Responsible analytical practice is a lifecycle activity. The NIST AI Risk Management Framework organizes this work around governance, context mapping, measurement, and risk management rather than treating responsibility as a final compliance check (National Institute of Standards and Technology 2023).

Final Reproducibility Audit

Before publishing or handing off the work, confirm that:

  • the raw data remain unchanged and provenance is recorded;
  • all paths are project-relative;
  • random seeds are fixed where stochastic behaviour should be reproducible;
  • transformations and feature definitions are explicit;
  • preprocessing is learned only from training data;
  • the split strategy matches the structure and intended use of the data;
  • baselines, metrics, thresholds, and selection rules are documented;
  • tables and figures can be regenerated from scripts;
  • package requirements are recorded;
  • results have been rerun from a clean session or environment;
  • limitations and unresolved risks are visible; and
  • another analyst could trace the final claim back to data and code.

Core Software

The software implements the workflow; the credibility of the result still depends on the question, data, design, validation, interpretation, and decision context.