Most AI systems are not just model calls. A large part of the work is moving data into the right shape: loading documents, normalizing JSON, reading logs, saving evaluation results, calling APIs, extracting PDF text, and keeping enough metadata to debug the result later.

This chapter covers the file and data-handling patterns that show up in practical AI work: pathlib, context managers, JSON and JSONL, CSV, HTTP clients, PDF extraction, HTML parsing, and large-file processing. The focus is simple: write code that behaves predictably when files are large, text is multilingual, network calls fail, and someone needs to trace an answer back to its source.

Python's os.path functions like os.path.join() and os.path.exists() still work. But pathlib is usually clearer, because a path becomes an object with methods and properties instead of a string passed through helper functions.

pathlib, available in the standard library, gives you Path objects that work across operating systems and compose naturally.

Creating and Joining Paths

That / operator is not doing division here. pathlib uses it for path joining, which reads more clearly than os.path.join("data", "documents"). It also handles the right path separator for the operating system.

Useful Path Properties

Every Path object gives you easy access to parts of the path:

These are properties, not method calls, so there are no parentheses. .stem and .suffix are useful when processing batches of files. For example, you can turn report.pdf into report.txt after extracting text.

Reading and Writing Files

For small files, pathlib gives you one-liner convenience methods to work with files:

These methods handle opening and closing the file for you. They are a good fit for small files such as configuration files, prompts, and short outputs. For large files, stream with open() instead of reading the whole file into memory.

Checking Existence and Creating Directories

The parents=True flag creates missing parent directories too. Without it, Python raises an error if a parent directory does not exist. The exist_ok=True flag prevents an error when the directory is already there.

Finding Files with glob

When you need to find files matching a pattern, pathlib has .glob() and .rglob() built in:

The difference between .glob() and .rglob() is that .rglob() searches subdirectories recursively. Use .rglob() when a document loader needs to find files inside nested folders.

pathlib's .read_text() and .write_text() are useful for small files. When you need more control, such as reading line by line, appending to a file, or working with binary data, use open(). Whenever you use open(), use a context manager.

Here is the fragile approach to reading a file:

This works only if nothing goes wrong. If f.read() raises an exception, f.close() never runs. The file handle stays open. In a long-running service or batch job, repeated leaks can hit the operating system's limit on open file descriptors.

The with statement fixes this by guaranteeing cleanup:

No matter what happens inside the with block, whether the code succeeds, raises an exception, or returns early, the file gets closed. Use context managers for resources that need cleanup.

Writing Your Own Context Manager

You may want the same guarantee for your own resources, such as a database connection, a temporary directory, or a timer. Python's contextlib module makes this straightforward:

Everything before yield runs when entering the with block. Everything after it runs when exiting. Wrapping yield in try/finally guarantees the cleanup code runs even if the block raises an exception. This same idea shows up in AI code when you manage temporary files, experiment runs, model clients, or expensive local resources.

JSON is the common interchange format for AI systems. Provider APIs, tool calls, configuration files, evaluation sets, and logs often use JSON or JSONL, where each line is a separate JSON object.

The Basics: load, dump, loads, dumps

Python's json module has four core functions. The names are easy to mix up at first:

The mnemonic: loads and dumps work with strings. load and dump work with files.

Pretty Printing

Compact JSON is good for machines, but not always pleasant for people to read:

Use indent=2 for config files, debugging output, and evaluation results that humans will inspect. Skip it for high-volume machine-only data, since extra whitespace adds up.

Handling Encoding Issues

When working with multilingual data or special characters, encoding defaults can surprise you. Specify UTF-8 explicitly:

The ensure_ascii=False flag tells json.dumps to write Unicode characters directly instead of escaping them as \uXXXX. This keeps the file readable for humans.

JSONL: JSON Lines

AI workflows often use the JSONL format, where each line is a separate JSON object. This format is common for datasets, evaluation sets, and log files:

JSONL has a practical advantage over one large JSON array: you can append new records without rewriting the whole file, and you can process records one at a time. That matters when a dataset has millions of rows.

CSV files show up less often than JSON in many AI systems, but they are still common for tabular datasets, evaluation metrics, labeling exports, and spreadsheet data.

DictReader and DictWriter

The csv module's DictReader is often easier to use than the basic reader, because it gives you dictionaries keyed by column names instead of positional lists:

One thing to watch: DictReader returns all values as strings, even numbers. Cast them yourself, for example float(row["accuracy"]), if you need to do math. Libraries like pandas can infer types for you, but for simple jobs, the built-in csv module is lighter and has no extra dependency.

Most AI applications call external APIs: model providers, embedding services, vector databases, observability systems, and internal data services. The requests library is a common choice for synchronous HTTP code.

GET and POST Requests

The json= parameter serializes your dictionary to JSON and sends the request with a JSON content type. You could also use data=json.dumps(payload), but json= is clearer for normal API calls.

Response Handling

Every response object gives you several ways to access the data:

The .raise_for_status() method is important in real code. Without it, a failed API call still returns a response object, and your program may not fail until later when it tries to use missing or malformed data. Also remember that .json() can raise an error if the response body is not valid JSON.

Timeouts: Do Not Skip These

Every HTTP request should have a timeout. Without one, a slow or unreachable server can leave your program waiting indefinitely:

In AI applications, long model responses can legitimately take a while, so set the read timeout based on the operation. The connect timeout should usually be shorter, often 5 to 10 seconds, because a server that cannot be reached rarely becomes reachable by waiting much longer on the same attempt.

requests is mature and reliable for synchronous code. If one workflow needs to make several network calls concurrently, such as calling a model API and a vector database at the same time, async support becomes useful.

That is where httpx comes in.

The synchronous API looks similar to requests, but the main reason many teams choose httpx is async support:

For now, remember the decision point: requests is fine for straightforward synchronous scripts. httpx.AsyncClient is a better fit when one workflow needs many concurrent network calls.

PDFs are one of the most common document types in RAG pipelines: company reports, research papers, contracts, policies, and technical manuals. Extracting useful text from them is a recurring task, but PDFs are not always easy to parse cleanly.

PyMuPDF (fitz)

PyMuPDF (imported as fitz) is fast and handles many text-based PDFs well:

The page-by-page extraction is deliberate. In RAG systems, you usually want to know which page a chunk of text came from, so you can cite sources and debug retrieval results. Dumping the entire PDF into one string loses that information.

Handling Scanned PDFs

Not all PDFs contain selectable text. Scanned documents are often page images wrapped in a PDF container, and get_text() may return little or no text for those. For scanned PDFs, you need OCR, which stands for Optical Character Recognition.

Libraries like pytesseract or cloud OCR services can handle this, but OCR brings its own trade-offs around accuracy, cost, latency, and privacy. Always check whether extraction returned meaningful text before feeding it into a pipeline.

HTML parsing is useful when you need to ingest web pages into a search index or RAG knowledge base. Respect robots.txt, licenses, authentication boundaries, and site terms. BeautifulSoup is a common library for parsing HTML in Python.

The get_text(separator="\n", strip=True) call is useful because it extracts text from an element and its children, joins pieces with newlines, and strips extra whitespace. It will not recover the article structure from every page, but it gives you a reasonable starting point before sending content to an LLM or embedding model.

Extracting Links and Metadata

A common pattern in AI data collection is to start with a list of URLs, fetch each allowed page, extract text and metadata, and save the results as JSONL for later ingestion. The combination of requests, BeautifulSoup, json, and pathlib covers that basic workflow.

Some examples above load whole files into memory. That is fine for a small config file or a short document, but AI datasets can be much larger. A JSONL file with millions of examples might be many gigabytes. Reading that into memory all at once can slow your machine down or crash the process.

Line-by-Line Reading

The simplest approach for large text files is to read them line by line:

Python's file iterator reads one line at a time. The entire file is never loaded into memory. The enumerate call lets you track progress, which helps when processing millions of records.

Chunked Reading for Binary Files

For binary files or when you need to process data in fixed-size chunks:

This generator pattern is memory-efficient because it yields one chunk at a time. The same approach shows up in model downloads, dataset streaming, and file upload utilities.

Now put the pieces together. The following diagram shows a typical data ingestion pipeline for a RAG system or dataset preparation job.

Every source type has its own parser, but they all converge on the same validation and chunking step. The output is a uniform format, often JSONL, that downstream components can consume without caring where the data originally came from.

Here is a simplified version of that pipeline in code:

This is a starting point. A production pipeline would usually add retries, structured logging, deduplication, text chunking, metadata validation, and better error reporting. The core pattern stays the same: find files, parse them, normalize the output, and write a format downstream systems can consume.

References