json.Marshal and json.Unmarshal work on a single []byte that holds the entire payload. That's fine for small request bodies and config files, but it falls apart the moment the data is big, slow to arrive, or doesn't fit in memory at all. The json package solves this with json.Encoder and json.Decoder, two types that wrap any io.Writer or io.Reader and process JSON one value at a time. This chapter covers when and how to use them, the small set of configuration knobs they expose, and the patterns that come up in HTTP handlers and log processors.

The convenience pair, json.Marshal and json.Unmarshal, has a shape problem. Marshal(v) returns a []byte that holds the entire encoded value. Unmarshal(data, &v) requires the whole data slice in memory before it parses anything. For a 200-byte customer payload, neither of these costs anything. For a 2 GB order-event log file or an HTTP body that arrives in chunks over a slow network, both fail in different ways: they either allocate huge byte slices, or they force you to read the entire stream before any work can begin.

json.Encoder and json.Decoder flip the model. Instead of "give me the bytes, then I'll parse them," they say "give me a reader, and I'll pull bytes from it as I need them." Instead of "I'll return all the bytes when I'm done," they say "give me a writer, and I'll push bytes to it as I produce them." The payload never has to exist as one big slice. This is the same buffered, pull-as-you-go pattern that io.Reader and io.Writer use, applied to JSON.

There are three concrete situations where streaming pays off:

The picture below shows the shape of the difference. With Marshal, the whole value materializes as a []byte before the writer ever sees it. With Encoder, each Encode call writes a chunk directly to the underlying writer.

The red node on top, the big intermediate buffer, is the cost you pay for Marshal. The streaming path on the bottom never holds the whole payload at once. For small data the buffer is invisible; for big data it's the difference between a happy server and an OOM kill.

json.NewEncoder(w io.Writer) returns an *json.Encoder bound to w. Each call to enc.Encode(v) writes the JSON encoding of v to w, followed by a newline. That trailing newline is a small but consequential detail: it makes a stream of Encode calls automatically produce NDJSON (one JSON value per line), which is the dominant format for log files and event streams.

Here's the smallest useful example, writing two order records to standard output:

Each call to Encode produces one complete JSON object plus a newline. There's no comma between them and no surrounding array brackets. That's the NDJSON convention: each line is independently parseable, and a tool like grep, head, or tail can scan the file without understanding JSON at all.

If you want pretty-printed JSON instead of the compact form, configure the encoder with SetIndent before calling Encode. The first argument is a prefix added before every line; the second is the per-level indent string.

Pretty printing is a debugging and human-readable-file feature, not a wire-format choice. Don't ship indented JSON in API responses; it wastes bandwidth and slows down parsing on the client side. Use it for config files, test fixtures, and any file a person will read.

The other knob worth knowing is SetEscapeHTML(false). By default, the encoder escapes <, >, and & as <, >, and & so the output is safe to drop into an HTML page without script-injection risk. For pure machine-to-machine data (log files, internal APIs, queues), the escaping is noise. Turning it off keeps strings readable.

Same value, two outputs. The first is safe for embedding in HTML; the second is more readable but unsafe to inject into a web page without further escaping. For order events, audit logs, and most server-to-server traffic, prefer the second form.

json.NewDecoder(r io.Reader) returns an *json.Decoder that reads from r. Each call to dec.Decode(&v) parses exactly one JSON value from the stream and writes it into v. After a successful Decode, the decoder is positioned just after that value, ready to read the next one.

The simplest case is a file or HTTP body containing a single JSON object. The decoder reads it once and returns:

This looks identical to json.Unmarshal, and for a single small value the difference is invisible. The payoff shows up when the stream contains many values, when the values are large, or when the source is genuinely streaming (an HTTP body, a pipe, a slow network connection).

For an NDJSON source, you call Decode in a loop. Each call eats one line and one trailing newline. The decoder happily skips whitespace (including newlines) between values, so it doesn't care whether the values are on separate lines or run together:

The loop reads each event, prints it, and breaks when the decoder reports io.EOF. Notice the use of errors.Is(err, io.EOF) instead of err == io.EOF. Both work in practice for this case (the decoder returns io.EOF unwrapped), but errors.Is is the safe default for any sentinel comparison since Go 1.13.

Memory stays small no matter how long the file is. The decoder keeps an internal read buffer (a few KB by default) and a single OrderEvent value. A 10 GB log file processes with the same memory footprint as a 10-line one.

NDJSON is the friendliest shape for a streaming decoder. A real JSON array, like [{...},{...},{...}], is harder because it has surrounding [ and ] brackets plus commas between elements. Decode reads one full JSON value at a time, and the array as a whole is one value. If you call Decode on a 10 GB array, the decoder still tries to read the whole thing into a single slice in memory, which defeats the point.

The Token method exists for this case. It reads the next JSON token (a delimiter, a string, a number, a boolean, or null) without consuming a whole value. Combined with dec.More(), which reports whether the current array or object has more elements, you can walk through a giant array element by element.

The pattern is always the same:

The decoder reads one product at a time. The full array never exists as a single slice in memory; only the current Product value does. For a feed with a million products, the peak memory is one product plus the read buffer, regardless of file size.

Token returns a json.Token, which is an interface holding one of: json.Delim (the four delimiters [, ], {, }), bool, float64, json.Number (with UseNumber), string, or nil. The two delimiter tokens you care about for array streaming are json.Delim('[') and json.Delim(']'). The More method handles the comma-skipping for you, so you don't have to read or check the commas yourself.

The diagram below shows the relationship between the source, the decoder's internal buffer, and the consumer that pulls values one at a time. The decoder reads ahead just enough to identify the next value, hands it to the application, and reads more bytes from the source when its buffer is empty.

The arrows form a back-and-forth loop. The application asks for the next value, the parser reads tokens until it has one, the read buffer refills from the source as needed. At any moment, only the current value and a small read buffer are live. This is the structural reason streaming is cheap in memory.

The decoder has two settings that affect how it interprets the stream. Both are off by default, and both are worth knowing.

DisallowUnknownFields() tells the decoder to fail when the JSON contains a field that doesn't map to any field in the destination struct. Without it, unknown fields are silently ignored. This is the right behavior for forward-compatible APIs (servers add fields, old clients keep working), but it's the wrong behavior for strict input validation. If you're decoding a config file or a request body and you want to catch typos like "qauntity": 5, turn the check on.

The typo is caught at the source. Without DisallowUnknownFields, the decoder would have happily produced Item{Product: "notebook", Quantity: 0} and the bug would surface later as "why is every order showing quantity zero?"

UseNumber() is the other useful knob. By default, the decoder turns any JSON number into a float64 when the destination is any (or interface{}). For prices and totals that involve cents, float64 loses precision: the value 0.1 is not exactly representable in binary floating point, and operations on currency values accumulate rounding errors. UseNumber makes the decoder produce json.Number instead, which is a string internally that preserves the original digits exactly. You then convert to int64, float64, or *big.Float based on what the field actually means.

total and count come through as json.Number instead of float64. To use them, call methods on the type:

Without UseNumber, count (a value larger than float64 can represent exactly as an integer) would silently lose its lower digits. With UseNumber, the digit string is preserved and the caller decides how to convert.

The textbook use case for the streaming API is an HTTP handler. The request body is an io.ReadCloser, the response writer is an io.Writer, and the handler's job is to decode one value and encode another. Doing this with Marshal/Unmarshal works for small payloads but wastes a io.ReadAll buffer on the way in and a Marshal buffer on the way out. The streaming form is just as easy to write and uses dramatically less memory for big payloads.

A complete handler that accepts an order, looks up its total, and returns an acknowledgement:

A curl request like curl -X POST localhost:8080/order -d '{"order_id":"1001","customer":"Priya","total":124.95}' produces a JSON response with the order ID, status, and message fields populated.

A few details in this handler are worth calling out:

The decoder also has one subtle pitfall on the request side: it doesn't drain the body if there's extra junk after the JSON value. For HTTP/1.1 with keep-alive, an undrained body can prevent the connection from being reused. The fix is straightforward but easy to forget:

For most handlers, the body contains exactly one JSON value followed by nothing, so the drain is a no-op. For handlers that decode only the prefix of a body (rare, but it happens with multipart-like protocols), the drain prevents connection pool starvation.

The other big practical use is processing a file that's too large to fit in memory. Order event logs, customer activity logs, and product feed snapshots are commonly stored as NDJSON because each line is independently parseable. The decoder pattern is identical to the in-memory NDJSON example earlier; the only thing that changes is the source.

Imagine a file orders.log that grows by a few million lines a day, with one event per line. A program that scans the file and counts events per status would look like this:

Two observations:

The *os.File can be replaced by any io.Reader, including a network connection, a gzip.Reader, or a pipe. Reading a gzip-compressed log file directly is one extra line:

Same loop, same memory profile, decompression on the fly.

A few things go wrong often enough to be worth listing explicitly.

Forgetting to close the body. A decoder reads from http.Request.Body until it has one JSON value, then stops. The body still owns OS-level resources (a TCP connection's read side, an SSL session) that need to be released. defer r.Body.Close() immediately after the body is available is the safest habit. The decoder doesn't and shouldn't close the underlying reader; that's the caller's job.

Mixing the encoder's newline with framed protocols. Encode always appends a newline. If you're writing to a protocol that expects exact byte counts (a binary frame format, a CSV column), the newline corrupts the output. For NDJSON, log files, and HTTP responses, the newline is harmless or helpful. For anything else, write the encoded bytes manually with json.Marshal plus your own framing.

Sharing a decoder across goroutines. json.Decoder is not safe for concurrent use. If you have a single source feeding multiple consumers, either decode in one goroutine and fan out the values over a channel, or give each consumer its own reader and decoder. The same rule applies to json.Encoder: one writer, one encoder, no shared state.

Trying to seek backwards. Once Decode has advanced past a value, you can't go back. The decoder doesn't keep the source bytes after parsing. If you need to look at a value twice, decode it into a struct and inspect the struct, or buffer the raw bytes upstream with bytes.Buffer and create a fresh decoder for the replay.

Assuming `io.EOF` is a failure. io.EOF is the normal end-of-stream signal. For json.Decoder.Decode, it means "the stream finished after the last value." Compare with errors.Is(err, io.EOF) and treat it as a clean termination, not an error to log and re-raise.

Forgetting that `Decode` can return non-EOF errors that still leave the stream in a usable state. If a value mid-stream is malformed JSON, Decode returns the parse error and the decoder is positioned at the failure point. You can sometimes recover by calling Token to skip ahead, but in practice abort the loop and log the bad input. A corrupted record usually means upstream truncation, and continuing tends to hide the real problem.

Here's a quick reference table summarizing which approach fits each scenario:

Marshal and Unmarshal are not obsolete; they're the right answer for small fixed-size payloads where the convenience of a single []byte outweighs the buffer cost. Use the streaming API when the payload is big, when it's coming from a stream that's already there (a body, a file), or when you're producing a stream of values rather than a single one.