Data Input/Output: Loading and Saving Data

TIP

Learning Objectives: After this lesson, you'll master reading and writing data in various formats—CSV, JSON, Excel, and more. You'll also learn to fetch data from web APIs and handle different file encodings.

The Data Lifecycle

Data analysis starts with loading data and ends with saving results. Pandas makes this seamless across formats.

Reading CSV Files

CSV (Comma-Separated Values) is the most common data format. Pandas handles it effortlessly.

Basic CSV Reading

Common CSV Options

Handling Large Files

Writing CSV Files

Working with JSON

JSON is common for web APIs and nested data structures.

Reading JSON

Writing JSON

Excel Files

Pandas can read and write Excel files (requires openpyxl for .xlsx).

Working with SQL Databases

Pandas integrates seamlessly with SQL databases.

Fetching Data from Web APIs

APIs return data (usually JSON) that can be converted to DataFrames.

Pagination and Multiple API Calls

Handling File Encodings

Different files use different character encodings. Understanding this prevents errors.

Binary Formats: Parquet and Pickle

For performance, consider binary formats.

Practical Example: Data Pipeline

Key Takeaways

✅ CSV is universal—use pd.read_csv() and df.to_csv() with appropriate options

✅ JSON handles nested data—use pd.json_normalize() for complex structures

✅ Excel requires openpyxl—supports multiple sheets with ExcelWriter

✅ SQL integration is seamless—use pd.read_sql() and df.to_sql()

✅ APIs return JSON—convert to DataFrames after parsing

✅ Encodings matter—specify encoding for international data

✅ Binary formats (Parquet, Pickle) are faster for large data

Connections: Data I/O in Practice

🔗 Connection to Data Engineering

🔗 Connection to Machine Learning

Loading data is the first step in any ML workflow:

COPY
# Typical ML data loading
train = pd.read_csv('train.csv')
X = train.drop('target', axis=1)
y = train['target']

Practice Exercises

Exercise 1: Multi-Format Pipeline

Next Steps

Now that you can load and save data, you're ready for Data Visualization with Matplotlib—turning your data into compelling visual stories.

Ready to visualize your data? Let's create beautiful charts!

Further Reading

Official Docs

• Pandas — IO Tools — every read/write format pandas supports, in one table.
• Pandas — read_csv parameters — parse_dates, dtype, chunksize, usecols, na_values. Worth bookmarking.
• Apache Parquet documentation — the modern columnar format. Use it instead of CSV for anything > 1MB.
• Apache Arrow Python — the in-memory columnar standard underlying modern fast IO.

Tutorials

• Real Python — Reading and Writing CSV Files — covers stdlib csv + pandas patterns.
• Real Python — Reading Excel Files in Python — openpyxl, pandas.read_excel, multi-sheet workflows.

Choose the Right Format

• CSV: human-readable, universal — but slow, type-ambiguous, no compression by default.
• Parquet: columnar, typed, compressed. Use for everything that isn't shared with non-tech humans.
• JSON Lines: one JSON object per line. Ideal for log/streaming data.
• Feather / Arrow IPC: fastest pandas-to-pandas roundtrip.

APIs & Web Scraping

• httpx — modern, sync + async, drop-in requests replacement.
• Beautiful Soup — HTML parsing. Pair with requests/httpx for scraping.
• Playwright Python — for JavaScript-heavy sites that need a real browser.

Books

• Book: Python for Data Analysis (3rd ed.) — Wes McKinney. Chapter 6 ("Data Loading, Storage, and File Formats").