Setting Up a Reproducible Analytical Environment

Published

Aug 2026

  • ID: ADS-L01
  • Type: Applied Data Science Foundation
  • Audience: Intermediate to Advanced
  • Theme: Reproducibility and structure support reliable analytical work

In Data Science Foundations, you learned how to inspect, clean, summarize, and visualize data.

In this guide, you will extend those skills into applied workflows that must remain:

This requires a more deliberate setup.

An environment is not just where code runs. It is the foundation of the analytical workflow you are building.

Why Setup Matters

A data science project may begin as a notebook, a script, or a small experiment. As soon as the work becomes important, however, its setup also becomes important.

A reliable project should make it clear:

  • where data is stored
  • where code lives
  • where outputs are written
  • which dependencies are required
  • how the project can be rebuilt
  • how another person can reproduce the work

Without this structure, an analysis becomes difficult to trust, review, and reuse.

The goal of this lesson is to create a stable project foundation before moving into feature engineering, model building, model evaluation, interpretation, and deployment.

What This Setup Supports

By the end of this lesson, you should have an environment that supports:

  • consistent execution across sessions
  • controlled Python dependencies
  • structured project organization
  • reproducible checks and builds
  • reusable scripts
  • preparation for modeling, pipelines, and deployment

The setup is intentionally simple. The goal is to make the project understandable, repeatable, and ready to grow without over-engineering it.

Project Structure

A recommended structure for this guide is shown below.

project-root/
│
├── data/
│   ├── raw/
│   ├── processed/
│   └── reference/
│
├── scripts/
│   ├── bash/
│   └── python/
│
├── notebooks/
├── models/
├── reports/
├── docs/
│
├── requirements.txt
├── requirements-lock.txt
├── _quarto.yml
│
├── index.qmd
├── 00-preface.qmd
├── 01-setting-up-environment.qmd
├── 02-feature-engineering.qmd
├── 03-model-building.qmd
├── 04-model-evaluation.qmd
└── ...

Each folder or file has a clear role.

Folder or file Purpose
data/raw/ Original input data that should not be edited manually
data/processed/ Cleaned or transformed data used for analysis
data/reference/ Lookup tables, metadata, labels, or supporting files
scripts/bash/ Shell scripts for setup, checks, and project builds
scripts/python/ Python scripts for preparation, modeling, and validation
notebooks/ Exploratory notebooks, when needed
models/ Saved models and model-related outputs
reports/ Analytical summaries, tables, and generated outputs
docs/ Rendered Quarto website output
requirements.txt Direct Python dependencies needed by the project
requirements-lock.txt Snapshot of all packages installed in the active environment
_quarto.yml Quarto project configuration

A clear structure reduces confusion and makes the project easier to review, teach, publish, and extend.

Work from the Project Root

Run the commands in this lesson from the project root: the directory containing _quarto.yml and requirements.txt.

Check your current location:

pwd

On Windows PowerShell, use:

Get-Location

The reusable scripts later in this lesson can locate the project root from their own file locations. Even so, working from the project root is a useful and consistent project habit.

Create the Project Folders

From the project root, create the core folders.

mkdir -p data/raw data/processed data/reference
mkdir -p scripts/bash scripts/python
mkdir -p notebooks models reports docs

Check that the folders were created.

find . -maxdepth 2 -type d | sort

You should see the main project directories in the output.

Create a Python Environment

A virtual environment keeps project dependencies separate from packages installed elsewhere on your computer.

Create an environment named .venv:

python -m venv .venv

Activate it on macOS or Linux:

source .venv/bin/activate

Activate it on Windows PowerShell:

.venv\Scripts\Activate.ps1

Verify the Python interpreter.

python --version
python -c "import sys; print(sys.executable)"

When the environment is active, the executable path should point inside the project’s .venv directory.

Note

The scripts in this lesson require Python 3.10 or later. Python 3.12 is a suitable choice for this guide.

Control Project Dependencies

Dependencies should be recorded so the project can be rebuilt later.

Create requirements.txt and add the direct packages used by the guide.

pandas
numpy
scikit-learn
matplotlib
joblib
fastapi
uvicorn
jupyterlab

Install them through the Python interpreter in the active environment.

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

After installation, record the exact package versions in the current environment.

python -m pip freeze > requirements-lock.txt

The two files serve different purposes:

  • requirements.txt lists the direct dependencies intended for the project.
  • requirements-lock.txt records a complete snapshot of packages installed in the active environment, including indirect dependencies.
Important

pip freeze creates an environment snapshot, not a fully portable lock file. Package availability and compatibility may still vary by operating system, Python version, and hardware architecture. Record the Python version alongside the snapshot and test the environment when moving the project to another system.

Check the Quarto Project

This guide uses Quarto for reproducible reporting and publishing.

Check that Quarto is available.

quarto --version

During development, preview the guide with live updates.

quarto preview

When you are ready to generate the complete output, render the project.

quarto render

If rendering succeeds, Quarto writes the website to the output directory configured in _quarto.yml, usually docs/.

Reusable Project Scripts

The following scripts turn the manual setup steps into repeatable project commands. Save them in the paths shown and run them from the project root.

Script 01A — Create the Project Structure

Create:

scripts/bash/01a-create-project-structure.sh

Add:

#!/usr/bin/env bash

set -euo pipefail

script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
project_root="$(cd -- "$script_dir/../.." && pwd)"
cd "$project_root"

echo "Creating Applied Data Science project structure..."

mkdir -p data/raw data/processed data/reference
mkdir -p scripts/bash scripts/python
mkdir -p notebooks models reports docs

echo "Project folders created."
echo
echo "Current project structure:"
find . -maxdepth 2 -type d | sort

Run it:

bash scripts/bash/01a-create-project-structure.sh

The script first identifies the project root from its own location. It therefore behaves consistently even if it is launched from another directory.

Script 01B — Check the Environment

Create:

scripts/python/01b_check_environment.py

Add:

from importlib.util import find_spec
from pathlib import Path
import shutil
import sys


PROJECT_ROOT = Path(__file__).resolve().parents[2]
MINIMUM_PYTHON = (3, 10)

REQUIRED_DIRS = [
    "data/raw",
    "data/processed",
    "data/reference",
    "scripts/bash",
    "scripts/python",
    "notebooks",
    "models",
    "reports",
    "docs",
]

REQUIRED_FILES = [
    "requirements.txt",
    "_quarto.yml",
]

REQUIRED_PACKAGES = [
    "pandas",
    "numpy",
    "sklearn",
    "matplotlib",
    "joblib",
    "fastapi",
    "uvicorn",
    "jupyterlab",
]


def display_paths(title: str, paths: list[str]) -> None:
    if paths:
        print(title)
        for path in paths:
            print(f"- {path}")
        print()


expected_venv = (PROJECT_ROOT / ".venv").resolve()
active_prefix = Path(sys.prefix).resolve()

missing_dirs = [
    path for path in REQUIRED_DIRS
    if not (PROJECT_ROOT / path).is_dir()
]
missing_files = [
    path for path in REQUIRED_FILES
    if not (PROJECT_ROOT / path).is_file()
]
missing_packages = [
    package for package in REQUIRED_PACKAGES
    if find_spec(package) is None
]

python_supported = sys.version_info >= MINIMUM_PYTHON
venv_active = sys.prefix != sys.base_prefix
project_venv_active = venv_active and active_prefix == expected_venv
quarto_available = shutil.which("quarto") is not None

print(f"Project root: {PROJECT_ROOT}")
print(f"Python executable: {sys.executable}")
print(f"Python version: {sys.version.split()[0]}")
print(f"Project .venv active: {'yes' if project_venv_active else 'no'}")
print(f"Quarto available: {'yes' if quarto_available else 'no'}")
print()

display_paths("Missing directories:", missing_dirs)
display_paths("Missing files:", missing_files)
display_paths("Missing Python packages:", missing_packages)

errors = []

if not python_supported:
    errors.append("Python 3.10 or later is required.")
if not project_venv_active:
    errors.append("Activate the project's .venv environment.")
if not quarto_available:
    errors.append("Install Quarto or add it to PATH.")
if missing_dirs or missing_files or missing_packages:
    errors.append("Create or install the missing project requirements listed above.")

if errors:
    for error in errors:
        print(f"ERROR: {error}")
    raise SystemExit(1)

print("Environment check passed.")

Run it:

python scripts/python/01b_check_environment.py

This check distinguishes directories from files, verifies Python 3.10 or later, confirms that the project’s .venv is active, checks core package imports, and confirms that Quarto is available.

Note

The import name for scikit-learn is sklearn. This is why the environment check uses sklearn even though requirements.txt uses scikit-learn.

Script 01C — Check and Build the Project

Create:

scripts/bash/01c-build-project.sh

Add:

#!/usr/bin/env bash

set -euo pipefail

script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
project_root="$(cd -- "$script_dir/../.." && pwd)"
cd "$project_root"

if ! command -v python >/dev/null 2>&1; then
  echo "ERROR: Python is not available on PATH." >&2
  exit 1
fi

if ! command -v quarto >/dev/null 2>&1; then
  echo "ERROR: Quarto is not available on PATH." >&2
  exit 1
fi

echo "Checking the project environment..."
python scripts/python/01b_check_environment.py

echo
echo "Rendering the Quarto project..."
quarto render

echo
echo "Build complete."

Run it:

bash scripts/bash/01c-build-project.sh

The build stops if Python, Quarto, or any checked project requirement is unavailable. Quarto renders only after the environment check passes.

Script 01D — Record Dependency Versions

Create:

scripts/bash/01d-freeze-dependencies.sh

Add:

#!/usr/bin/env bash

set -euo pipefail

script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
project_root="$(cd -- "$script_dir/../.." && pwd)"
cd "$project_root"

if ! command -v python >/dev/null 2>&1; then
  echo "ERROR: Python is not available on PATH." >&2
  exit 1
fi

expected_prefix="$project_root/.venv"
active_prefix="$(python -c 'import sys; print(sys.prefix)')"

if [[ "$active_prefix" != "$expected_prefix" ]]; then
  echo "ERROR: Activate the project's .venv before freezing dependencies." >&2
  echo "Expected: $expected_prefix" >&2
  echo "Active:   $active_prefix" >&2
  exit 1
fi

echo "Recording Python dependency versions..."
python -m pip freeze > requirements-lock.txt

echo "Dependency snapshot written to requirements-lock.txt."

Run it only after activating the project’s .venv:

bash scripts/bash/01d-freeze-dependencies.sh

The script refuses to create the snapshot from a global environment or a different virtual environment. This prevents unrelated packages from being recorded accidentally.

Common Setup Problems

Python does not point inside .venv

Activate the project environment and check again.

source .venv/bin/activate
python -c "import sys; print(sys.executable)"

A required package cannot be imported

Install the declared dependencies through the active Python interpreter.

python -m pip install -r requirements.txt

Quarto is not found

Install Quarto using the official installer for your operating system, then open a new terminal and run:

quarto --version

The project renders from one location but not another

Run commands from the project root and avoid hard-coded absolute paths in analytical code. Build file paths from a known project location instead.

Completion Check

Before continuing, confirm that:

  • the project folders exist
  • .venv is created and active
  • Python 3.10 or later is available
  • requirements.txt contains the direct dependencies
  • the required packages are installed
  • Quarto is available
  • the environment check passes
  • quarto preview opens the guide
  • the full project renders successfully
  • requirements-lock.txt was created from the project environment

At this point, the project has a reproducible foundation for the applied workflow developed in the remaining chapters.

Next Step

The next chapter moves from project setup to feature engineering: transforming raw variables into useful, well-defined inputs for modeling while protecting the validity of the analytical workflow.