pickle is Python's built-in serializer for Python objects. Where JSON converts to a small fixed set of types that any language can read, pickle captures Python objects as Python objects: lists with nested dicts, custom classes, complex numbers, datetime values, even most of your own instances, all in one call. The price is that the result is Python-only and that loading an untrusted pickle can run arbitrary code on your machine. This lesson covers what pickle does, the four functions used most often, what's safe to pickle and what isn't, the five protocols, the safer alternatives, and the security model that matters before exposing pickle to anything you don't control.

Pickle takes a Python object and produces a byte stream that can rebuild an equivalent object later. The conversion goes both ways with two pairs of functions: dumps/loads for in-memory bytes, and dump/load for files.

The output is binary, not text. The first byte (\x80) is the pickle protocol marker, followed by opcodes that the decoder interprets to rebuild the object. The bytes aren't meant to be read by eye; pass them back to pickle.loads. The restored object is structurally equal to the original, including the nested list of dicts.

The encoder walks the object graph, emitting opcodes that describe types, attributes, and references. The decoder reads those opcodes back and runs them to reconstruct the graph. The decode side is what makes pickle dangerous; the security section covers that.

The convention matches json: s at the end means "string" (well, bytes here), no s means "file". The file-based versions always take a file opened in binary mode ("rb" or "wb"), because pickle data is binary.

dump is more efficient than dumps followed by f.write, because it doesn't build the full byte string in memory before writing. For small objects the difference is invisible; for large ones it matters.

pickle.load and pickle.loads allocate the full object graph in memory at once. A pickle representing a 2 GB dataframe needs at least 2 GB of memory to load, plus overhead. For pickling large datasets, look at pickle.HIGHEST_PROTOCOL (smaller output) or formats designed for the job, like Parquet or HDF5.

Pickle handles most things in the Python standard library, plus most user-defined classes, with no special effort on your part.

Pickles cleanly without setup:

Decimal and datetime round-trip with full precision. No custom encoder, no string conversion, no type tags. The Order instance comes back as the same class, with the same attributes, ready to use.

Tuples and sets round-trip exactly, which is one of the main advantages over JSON:

JSON would turn both into lists. Pickle preserves the distinction because it has the type information natively.

Pickle can't capture an object whose state depends on resources outside the running process. Common examples:

The fix is to redesign so the unpicklable thing isn't part of the state you're trying to save. For an object that holds a connection, store the connection parameters, not the connection. For a callback, store its name or use a registry; don't pickle a lambda.

For more control over what gets pickled, classes can define __getstate__ (returns the state dict to pickle) and __setstate__ (rebuilds from that state). That lets you exclude unpicklable attributes during dump and recreate them on load:

__getstate__ and __setstate__ are the standard extension points. The same hooks are useful for a compact representation, migrating old pickles into a new schema, or scrubbing credentials before serialization.

A pickle starts with a one-byte protocol marker that tells the decoder which set of opcodes to expect. Python ships with six protocol versions, numbered 0 through 5. Each one is a strict superset of the previous, with smaller and faster encodings.

pickle.dumps accepts an optional protocol argument. The two relevant defaults are pickle.DEFAULT_PROTOCOL (a sensible balance for the current Python version) and pickle.HIGHEST_PROTOCOL (the newest format the interpreter supports).

The numbers shift around depending on the object, but the trend is consistent: protocol 0 is much larger than the binary protocols. For interoperability between Python versions, use the lowest protocol both sides understand. For storage and speed, use the highest one the writer supports and accept that older readers can't decode it.

For pickles meant to be read by a different Python version, pin the protocol explicitly:

For most code, the default protocol is fine. Pin it only when there's a known compatibility constraint.

pickle.load is not a parser, it's an interpreter. It runs opcodes from the byte stream, and those opcodes can call almost any Python function, including os.system, subprocess.Popen, eval, exec, or anything else importable. A malicious pickle is essentially a script that runs on import.

A concrete example: a pickle that runs a shell command when loaded.

The __reduce__ method is the official extension point for telling pickle "to rebuild me, call this function with these arguments". The same mechanism that lets a class control its own serialization is what makes loading untrusted pickles dangerous. The payload doesn't have to be os.system; it can be anything reachable through the import system.

This isn't theoretical. Pickle has been used in attacks against ML model registries (where downloaded models are pickled .pt or .pkl files), task queue messages, and cache stores.

The rule, with no exceptions:

If you must accept pickles from a partially trusted source (an internal service, a colleague's data export), the safest approach is to load in a sandboxed subprocess and validate the result before using it. Even then, treat pickle as a last resort.

The standard library ships a pickle.Unpickler subclass mechanism (find_class) that lets you restrict which classes can be reconstructed. It's not a security boundary to rely on; it raises the attack cost slightly but doesn't eliminate it. Treat it as a hardening measure, not a fix.

The custom unpickler refuses to resolve os.system, so the malicious pickle fails. This is a whitelist, and getting whitelists right against a determined attacker is hard. The right default is "don't load untrusted pickles".

The two formats look comparable on the surface; both serialize Python data structures and read them back. The differences are large and shape when you should use each.

The decision rule:

When in doubt, start with JSON. Move to pickle only when the type mismatch becomes a real problem and you can guarantee the data won't cross a trust boundary.

Situations where pickle is the appropriate tool:

Caching Computed Objects

A function that does expensive computation can cache its result on disk and reload it on subsequent runs. Pickle handles whatever object the function returns without requiring the cache layer to understand its shape.

The second call hits the cache and skips the expensive computation. The cache path is fully under the program's control, so the security concern doesn't apply.

Multiprocessing IPC

The multiprocessing module uses pickle to send objects between processes. When you call Pool.map(fn, items), Python pickles each item, sends the bytes to a worker, unpickles them there, runs the function, and pickles the result back. The pickle calls are implicit in the API.

Pool.map works on the items because each dict of primitives pickles cleanly. The function total_for_order works because it's defined at module level (and so is importable by name on the worker side). A lambda passed in its place would fail with a pickling error. That's the most common multiprocessing trap.

Snapshots for Debugging

When a bug appears intermittently in production data, pickling the relevant in-memory state before the crash gives you something to reload in a debugger later. The state captures everything, custom classes and all, so the failing logic can be re-run in isolation.

Loading crash-state.pkl later in a REPL provides the exact dict that triggered the failure, with full type fidelity. JSON would lose the distinction between sets, tuples, and custom classes that may matter for reproducing the bug.

Pickle isn't the only serializer in town. Each alternative trades a feature in exchange for a different property.

The trade-off: pickle is the most convenient and the least safe. JSON is the safest and the most limited. msgpack and protobuf sit between them, trading some convenience for portability and (in protobuf's case) a schema you can evolve.

For new code that doesn't have a strong reason to be Python-only, prefer one of the cross-language formats. Use pickle when the data is Python-shaped and the trust model is "I wrote both sides".

A small caching helper that handles the common cases: it pickles a value, writes it to disk through a temporary file (so a crash mid-write doesn't corrupt the cache), and reloads it on demand.

HIGHEST_PROTOCOL keeps the output compact. Writing through .tmp and os.replace means a reader either sees the old file or the new one, never a half-written file. os.fsync is the extra step that forces the data to physical disk before the rename; for caches it can be dropped, for anything that has to survive a power loss, keep it.

This is roughly the shape of a production-grade pickle write: atomic, with an explicit protocol, and confined to a path the program controls.