Packaging and Distribution: Ship Your Python

TIP

Learning Objectives: After this lesson, you'll understand modern Python packaging with pyproject.toml, manage dependencies effectively, create and publish packages to PyPI, and structure professional Python projects.

TIP

A note on this lesson. Packaging is a command-line and filesystem workflow — you build, you twine upload, you pip install. None of that runs inside a browser sandbox, so this lesson is a guided reference: read the config and code as formatted text, follow the diagrams, and run the commands in your own terminal. Every snippet below is copy-paste ready.

Modern Python Packaging

Python packaging has evolved significantly. Let's understand the modern approach.

The packaging landscape

The old way (2000s–2010s) spread a project's configuration across several files: setup.py (executable config, run with distutils/setuptools), requirements.txt for dependencies, and MANIFEST.in for file inclusion. It worked, but every tool invented its own conventions.

The modern way (2020s+) collapses all of that into a single declarative file — pyproject.toml (standardized by PEP 517, 518, and 621). One source of truth, build-system agnostic, with far better tooling support.

You'll choose three kinds of tools:

TIP

In 2024, the simplest modern default is uv + hatchling: fast, standards-based, and one tool for environments, dependencies, and Python versions. See Further Reading below.

pyproject.toml

The single configuration file that describes your package — its metadata, dependencies, entry points, and which build backend to use:

COPY
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "my-awesome-package"
version = "0.1.0"
description = "A fantastic Python package"
readme = "README.md"
license = {text = "MIT"}
requires-python = ">=3.9"
authors = [
    {name = "Your Name", email = "you@example.com"}
]
keywords = ["python", "package", "example"]
classifiers = [
    "Development Status :: 3 - Alpha",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
]

dependencies = [
    "requests>=2.28.0",
    "click>=8.0.0",
    "rich>=13.0.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "mypy>=1.0.0",
    "ruff>=0.1.0",
]
docs = [
    "mkdocs>=1.5.0",
    "mkdocs-material>=9.0.0",
]

[project.scripts]
my-cli = "mypackage.cli:main"

[project.urls]
Homepage = "https://github.com/user/my-awesome-package"
Documentation = "https://my-awesome-package.readthedocs.io"
Repository = "https://github.com/user/my-awesome-package"

[tool.hatch.build.targets.wheel]
packages = ["src/mypackage"]

Read it top to bottom: [build-system] says how to build, [project] is the metadata PyPI shows, dependencies is what gets installed alongside your package, [project.optional-dependencies] are extras users opt into (pip install mypackage[dev]), and [project.scripts] wires a terminal command to a Python function.

Project Structure

A professional layout separates configuration, source, tests, and docs:

COPY
my-project/
├── pyproject.toml          # Package configuration
├── README.md               # Project documentation
├── LICENSE                 # License file
├── .gitignore
│
├── src/                    # Source layout (recommended)
│   └── mypackage/
│       ├── __init__.py     # Package root
│       ├── __main__.py     # python -m mypackage
│       ├── core.py         # Core functionality
│       ├── utils.py        # Utilities
│       └── cli.py          # CLI commands
│
├── tests/                  # Test files
│   ├── __init__.py
│   ├── conftest.py         # pytest fixtures
│   ├── test_core.py
│   └── test_cli.py
│
├── docs/                   # Documentation
│   ├── index.md
│   └── api.md
│
└── examples/               # Example scripts
    └── basic_usage.py

Why the src/ layout? Putting your package one directory down, under src/, has four concrete benefits:

1. It prevents accidental imports from the project root — your tests can't import the source by accident.
2. Tests import the installed package, exactly as users will, catching packaging bugs early.
3. It cleanly separates source code from configuration and metadata.
4. It's the industry standard for libraries of any real size.

Package initialization

The __init__.py file is your package's front door. It defines what import mypackage exposes, holds package metadata, and curates the public API via __all__:

COPY
# src/mypackage/__init__.py
"""
MyPackage - A fantastic Python library.

Example usage:
    >>> from mypackage import process_data
    >>> result = process_data("input.txt")
"""

from mypackage.core import process_data, DataProcessor
from mypackage.utils import format_output

# Package metadata
__version__ = "0.1.0"
__author__ = "Your Name"
__all__ = [
    "process_data",
    "DataProcessor",
    "format_output",
]

A __main__.py makes the package runnable as a module with python -m mypackage:

COPY
# src/mypackage/__main__.py
"""Allow running with: python -m mypackage"""

from mypackage.cli import main

if __name__ == "__main__":
    main()

Dependency Management

Understanding dependencies

Dependencies come in two flavours, declared in pyproject.toml:

Runtime dependencies are required to run your package — pip installs them automatically when a user installs you:

COPY
dependencies = [
    "requests>=2.28.0",
    "click>=8.0.0,<9.0.0",
]

Optional dependencies power extra features and are opted into with pip install mypackage[dev,docs]:

COPY
[project.optional-dependencies]
dev = ["pytest", "mypy"]
docs = ["mkdocs"]

The string after each package name is a version specifier:

Best practices: specify minimum versions rather than exact ones; use upper bounds sparingly (they cause conflicts downstream); pin exact versions in requirements.txt for applications but not for libraries; and re-test against the latest releases regularly.

Lock files

pyproject.toml specifies version ranges. A lock file records the exact resolved version of every dependency — including transitive ones — so an install is byte-for-byte reproducible across machines and time.

Different tools generate them: pip-tools (pip-compile), Poetry (poetry.lock), PDM (pdm.lock), and uv (uv.lock).

With pip-tools, you write the loose requirements:

COPY
# requirements.in (what you write)
requests>=2.28.0
click>=8.0.0

…and compile them into a fully pinned lock:

COPY
# requirements.txt (generated)
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
idna==3.6
requests==2.31.0
urllib3==2.2.0

The workflow:

COPY
pip install pip-tools
pip-compile requirements.in            # Generate the lock
pip-compile --upgrade requirements.in  # Upgrade pinned versions
pip-sync requirements.txt              # Install exactly what's locked

# For development dependencies:
pip-compile requirements-dev.in --output-file requirements-dev.txt
pip-sync requirements.txt requirements-dev.txt

Virtual Environments

A virtual environment isolates one project's dependencies from every other project and from system Python. That isolation buys you reproducibility and saves you from version conflicts — never install project packages globally.

Create one with the built-in venv:

COPY
python -m venv .venv           # Create
source .venv/bin/activate      # Activate (macOS / Linux)
.venv\Scripts\activate.bat     # Activate (Windows)
deactivate                     # Deactivate

Or with uv, which is dramatically faster:

COPY
uv venv                              # Create .venv
source .venv/bin/activate            # Same activation
uv pip install -r requirements.txt   # Much faster than pip

The standard flow when you start work on a project is: clone, create the environment, then install the package in editable mode (-e) so your source edits take effect without reinstalling:

COPY
git clone https://github.com/user/project.git
cd project
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"        # Editable install with dev deps

Building and Publishing

Building your package

Install the build tool and run it:

COPY
pip install build
python -m build

This produces two artifacts in dist/:

• mypackage-0.1.0.tar.gz — the source distribution (sdist)
• mypackage-0.1.0-py3-none-any.whl — the wheel (a pre-built, fast-to-install archive)

A wheel's filename encodes its compatibility. Read the tags left to right:

• Pure Python — mypackage-0.1.0-py3-none-any.whl: py3 (Python 3) · none (no compiled ABI) · any (any platform). Installs everywhere.
• Platform-specific — mypackage-0.1.0-cp311-cp311-linux_x86_64.whl: cp311 (CPython 3.11) · linux_x86_64 (Linux 64-bit). Built because the package ships compiled code.

Verify the build before publishing:

COPY
pip install dist/mypackage-0.1.0-py3-none-any.whl
python -c "import mypackage; print(mypackage.__version__)"

Publishing to PyPI

1. Create accounts on test.pypi.org (for testing) and pypi.org (production).

2. Configure credentials with API tokens in ~/.pypirc:

   COPY
   [pypi]
   username = __token__
   password = pypi-xxxxx...
   
   [testpypi]
   username = __token__
   password = pypi-xxxxx...

3. Install twine and build:

   COPY
   pip install twine
   python -m build

4. Upload to TestPyPI first, then verify the install works:

   COPY
   twine upload --repository testpypi dist/*
   pip install --index-url https://test.pypi.org/simple/ mypackage

5. Upload to production PyPI:

   COPY
   twine upload dist/*

Best practices: use API tokens (never passwords), always test on TestPyPI first, follow semantic versioning, include a changelog, and consider signing releases.

Most teams automate this with GitHub Actions so publishing happens on every release:

COPY
# .github/workflows/publish.yml
name: Publish to PyPI

on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install build twine

      - name: Build package
        run: python -m build

      - name: Publish to PyPI
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
        run: twine upload dist/*

CLI Applications

Many packages ship a terminal command. Click is the standard library for building one — it turns decorated functions into a polished command group, and Rich makes the output beautiful:

COPY
# src/mypackage/cli.py
"""Command-line interface for mypackage."""

import click
from rich.console import Console
from rich.table import Table

console = Console()

@click.group()
@click.version_option(version="0.1.0")
def cli():
    """MyPackage - A fantastic CLI tool."""
    pass

@cli.command()
@click.argument("name")
@click.option("--count", "-c", default=1, help="Number of greetings")
@click.option("--formal", is_flag=True, help="Use formal greeting")
def greet(name: str, count: int, formal: bool):
    """Greet someone nicely."""
    greeting = "Good day" if formal else "Hello"
    for _ in range(count):
        console.print(f"{greeting}, [bold]{name}[/bold]!")

@cli.command()
@click.option("--format", "-f", type=click.Choice(["table", "json"]), default="table")
def list_items(format: str):
    """List available items."""
    items = [
        {"name": "Item 1", "price": 10.00},
        {"name": "Item 2", "price": 25.00},
    ]

    if format == "json":
        import json
        console.print_json(json.dumps(items))
    else:
        table = Table(title="Items")
        table.add_column("Name")
        table.add_column("Price", justify="right")
        for item in items:
            table.add_row(item["name"], f"${item['price']:.2f}")
        console.print(table)

@cli.command()
@click.argument("input_file", type=click.Path(exists=True))
@click.argument("output_file", type=click.Path())
@click.option("--verbose", "-v", is_flag=True)
def process(input_file: str, output_file: str, verbose: bool):
    """Process a file."""
    if verbose:
        console.print(f"Processing [cyan]{input_file}[/cyan]...")

    with open(input_file) as f:
        data = f.read()

    with open(output_file, "w") as f:
        f.write(data.upper())

    console.print(f"✓ Written to [green]{output_file}[/green]")

def main():
    cli()

if __name__ == "__main__":
    main()

Wire the command into pyproject.toml with an entry point, and pip will install a mypackage executable onto the user's PATH:

COPY
# In pyproject.toml
[project.scripts]
mypackage = "mypackage.cli:main"

COPY
# After installation:
mypackage greet Alice
mypackage --help
mypackage list-items --format json

Configuration Management

pyproject.toml isn't only for packaging — it's also where modern tools read their settings, so your whole toolchain configures from one file:

COPY
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --cov=mypackage"
python_files = "test_*.py"

[tool.mypy]
python_version = "3.11"
strict = true
ignore_missing_imports = true

[tool.ruff]
line-length = 88
select = ["E", "F", "I", "N", "W"]
ignore = ["E501"]

[tool.ruff.isort]
known-first-party = ["mypackage"]

[tool.coverage.run]
source = ["src/mypackage"]
branch = true

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "if TYPE_CHECKING:",
]

For your application's runtime configuration (as opposed to tooling config), three patterns cover almost everything — environment variables, a TOML config file, and typed settings via Pydantic:

COPY
# 1. Environment variables
import os

DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///db.sqlite")
DEBUG = os.getenv("DEBUG", "false").lower() == "true"

# 2. Configuration file (config.toml)
import tomllib  # Python 3.11+

with open("config.toml", "rb") as f:
    config = tomllib.load(f)

# 3. Pydantic settings — typed, validated, .env-aware
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    database_url: str = "sqlite:///db.sqlite"
    debug: bool = False
    api_key: str

    class Config:
        env_file = ".env"

settings = Settings()

Best Practices

1. Project structure — use the src/ layout for libraries, separate code/tests/docs, and include a py.typed marker so consumers get your type hints.
2. Version management — semantic versioning (MAJOR.MINOR.PATCH), a single source of truth for the version, and a CHANGELOG.md.
3. Dependencies — keep runtime dependencies minimal, specify ranges (not exact pins) in libraries, and test across dependency versions.
4. Documentation — a comprehensive README.md, docstring-driven API docs, and runnable examples.
5. Testing — test against every supported Python version in CI, and keep coverage high.
6. Publishing — test on TestPyPI first, automate with GitHub Actions, and tag releases in git.
7. Maintenance — respond to issues, keep dependencies current, and document breaking changes.

A pre-release checklist worth keeping:

• Version bumped appropriately
• CHANGELOG.md updated
• All tests passing
• Documentation updated
• README accurate and current
• License file present
• Build succeeds locally
• Works when installed from TestPyPI
• Git tag created
• Release notes written

Practice Exercise

Complete package setup

Putting it together — this script scaffolds a complete, publishable package from scratch. Run it in your own terminal, then read each generated file to see how the pieces connect:

COPY
#!/bin/bash

# Create project structure
mkdir -p my-new-package/src/mypackage
mkdir -p my-new-package/tests
cd my-new-package

# Create pyproject.toml
cat > pyproject.toml << 'EOF'
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "my-new-package"
version = "0.1.0"
description = "A new Python package"
readme = "README.md"
requires-python = ">=3.9"
dependencies = []

[project.optional-dependencies]
dev = ["pytest>=7.0", "ruff>=0.1"]

[tool.hatch.build.targets.wheel]
packages = ["src/mypackage"]

[tool.pytest.ini_options]
testpaths = ["tests"]

[tool.ruff]
line-length = 88
select = ["E", "F", "I"]
EOF

# Create package files
cat > src/mypackage/__init__.py << 'EOF'
"""My new package."""
__version__ = "0.1.0"
EOF

cat > src/mypackage/core.py << 'EOF'
"""Core functionality."""

def greet(name: str) -> str:
    """Return a greeting."""
    return f"Hello, {name}!"
EOF

# Create test file
cat > tests/test_core.py << 'EOF'
from mypackage.core import greet

def test_greet():
    assert greet("World") == "Hello, World!"
EOF

# Create README
cat > README.md << 'EOF'
# My New Package

A fantastic new Python package.
EOF

# Setup and verify
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"   # Editable install with dev deps
pytest                    # Run the tests
python -m build           # Build the wheel + sdist

echo "Package created successfully!"

Try it, then experiment: add a dependency to pyproject.toml, write a second function in core.py with a matching test, and rebuild.

Key Takeaways

Essential Commands

COPY
# Virtual environment
python -m venv .venv
source .venv/bin/activate

# Install in development mode
pip install -e ".[dev]"

# Build package
python -m build

# Upload to PyPI
twine upload dist/*

# Check package
pip install --index-url https://test.pypi.org/simple/ mypackage

Next Steps

In the next and final lesson, we'll bring everything together in a Capstone Project: Building a Production-Ready CLI Tool—applying all concepts from the course in a comprehensive project.

---

Ready for the final project? Let's build something real!

---

Further Reading

The Modern Python Tooling Stack (2024+)

The packaging ecosystem has consolidated dramatically. If you learned Python before 2023, much of what you know is now legacy. Use these:

• uv — Astral's all-in-one project + dependency + Python-version manager (Rust-backed, ~10× faster than pip). Replaces pip, pip-tools, pyenv, virtualenv, poetry, pipx in one tool. The 2024 default.
• hatch — modern PyPA-blessed build backend + project manager. Strong alternative to uv if you prefer PyPA tooling.
• ruff — formatter + linter that replaces black, flake8, isort, and most of pylint.

Standards & Specs

• Python Packaging User Guide (PyPA) — the canonical reference.
• PEP 621 — pyproject.toml metadata — modern package metadata.
• PEP 660 — Editable installs via pyproject.toml — pip install -e . properly.
• PEP 668 — Externally Managed Environments — why Linux distros block pip install system-wide now.

Tutorials

• uv — Working on projects — the modern equivalent of "how do I start a Python project?"
• Real Python — How to Publish a Python Package to PyPI — still excellent walkthrough.
• Hynek Schlawack — Python Application Management — historically influential, still relevant on the dependency-pinning question.

When You Need More

• scikit-build-core — the modern way to build C/C++/Rust extensions.
• maturin — build PyO3-based Rust extensions; what polars, pydantic-core, and ruff use.
• cibuildwheel — build wheels for every platform/Python version on CI.

Books

• Book: Publishing Python Packages — Dane Hillard (Manning, 2022). Whole book on pyproject.toml + PyPI workflows.