The bufio package wraps a file (or any reader/writer) in an in-memory buffer so a program doesn't pay the cost of a syscall on every byte. It also provides higher-level helpers like Scanner, which is the idiomatic way to read a file line by line. This lesson covers when buffering matters, the three main types (Reader, Writer, Scanner), the methods you'll use most, and two common mistakes: forgetting to flush, and Scanner's silent line-length limit.

Every call to Read or Write on an *os.File is a syscall. The CPU has to drop out of user mode, jump into the kernel, copy bytes between user space and kernel space, and return. A single syscall is fast in absolute terms, around a microsecond on a modern Linux box, but writing a 100 MB log one byte at a time would mean 100 million of them. The work the program actually wanted to do (push bytes to disk) is dwarfed by the cost of asking the kernel a hundred million times.

A buffer fixes this. Instead of handing each byte to the kernel as soon as the program produces it, bufio.Writer collects bytes in a memory slab (4 KB by default) and hands the whole slab to the kernel in one syscall when it fills up. A 100 MB write becomes roughly 25,000 syscalls instead of 100 million. The CPU spends its time copying bytes inside a tight loop, which is what CPUs are good at, rather than crossing the user/kernel boundary on every iteration.

bufio.Reader does the same trick in the other direction. It asks the kernel for 4 KB at a time and serves your ReadByte or ReadString calls out of that local copy until it runs dry, then refills.

The diagram makes the cost story concrete. Without a buffer, every write is a syscall. With a buffer, most writes are a memcpy into a slice and only the buffer-full events cross into the kernel. The savings scale with how small the writes are: writing 4 KB at a time directly to a file is already efficient, but writing one rune at a time is roughly a thousand times faster through a buffered writer.

bufio doesn't open files. It wraps something that already implements the io.Reader or io.Writer interface, typically an *os.File. That separation is important: the same buffered reader works over a file, a network connection, or os.Stdin.

bufio.NewReader(r io.Reader) wraps any reader and returns a *bufio.Reader with a 4096-byte buffer. bufio.NewReaderSize(r, size) lets you pick the size (with a minimum of 16 bytes). The wrapped reader exposes methods that operate on the buffered bytes: Read, ReadByte, ReadRune, ReadString, ReadBytes, ReadLine, Peek, UnreadByte, and UnreadRune.

The most common pattern is reading a text file one line at a time. An e-commerce platform might dump every order placed in a day to a flat file, one JSON record per line, and a nightly job reads that file to build reports.

ReadString('\n') returns everything up to and including the next newline. When the file ends without a trailing newline, it returns the leftover bytes along with io.EOF in the same call, which is why the code prints line before checking for EOF. Forgetting that detail is a common bug: if you skip a line != "" check, you drop the last line whenever the file doesn't end with \n.

ReadBytes('\n') is the byte-slice version of the same method. Use it when you want a []byte directly (to avoid the allocation of converting string back to []byte for further parsing).

ReadByte and ReadRune pull single bytes or single UTF-8 code points. Both advance the read position. Their unread counterparts (UnreadByte, UnreadRune) push the last one back so the next call returns it again. Unread only works for the single most recent byte or rune; you can't unread two in a row.

Peek(n) returns the next n bytes without advancing the position. It's useful when you need to look ahead, for example to check whether a file begins with a magic number before deciding how to parse it.

Peek did not advance the reader, so the subsequent ReadString('\n') still sees the header line at the start of the buffer.

ReadLine vs ReadString

ReadLine is the method that confuses people. Its signature is ReadLine() (line []byte, isPrefix bool, err error). It returns a single line without the trailing \n or \r\n. The isPrefix bool is true when the line was longer than the buffer, meaning you got only a partial line and need to call ReadLine again to get the rest.

The Go standard library documentation says it plainly: "ReadLine is a low-level line-reading primitive. Most callers should use ReadBytes('\n') or ReadString('\n') instead, or use a Scanner." Two reasons. First, ReadLine doesn't return the terminator, so you lose information about whether the last line ended with one. Second, the isPrefix machinery is awkward; you have to loop to reassemble a long line. ReadString('\n') returns the terminator and gives you the whole line in one call, even if the line is bigger than the buffer (it grows internally).

If you can read all your lines into memory, ReadString('\n') is simpler. If you want the cleanest API for line-by-line processing, bufio.Scanner (covered below) is even better. Use ReadLine only when you specifically need the bool-driven partial-line semantics, which is rare.

bufio.NewWriter(w io.Writer) wraps a writer in a 4096-byte buffer. bufio.NewWriterSize(w, size) picks a custom size. The wrapped writer exposes Write, WriteString, WriteByte, WriteRune, Flush, Available, and Buffered.

Consider an export job that writes a CSV product list for a partner integration. The naive approach (write directly to the file) issues one syscall per call to Fprintf. The buffered approach collects the bytes in memory and writes them out in 4 KB chunks.

Five writes (one header, four product lines) but only one syscall when Flush runs (the total is well under 4 KB). For a bigger export with 100,000 rows, the savings are dramatic.

Available() reports how many bytes the buffer has left before it fills up and flushes automatically. Buffered() reports how many bytes are currently sitting in the buffer waiting to be flushed. These are mostly useful for diagnostics or for code that wants to flush at a specific boundary.

The Flush Rule

You must call `Flush` before the underlying writer is closed. Otherwise, bytes still sitting in the buffer never reach the file.

Here's the bug:

The 22 bytes written to w sat in the buffer when the program exited. f.Close() flushed the kernel's view of the file (nothing was there), but the buffer in user space was never drained. The file ends up empty.

The fix is to flush explicitly, ideally with defer so it runs even if a later error returns from the function. But the defer order matters: defers run in LIFO order, so defer w.Flush() must come after defer f.Close() in the source, so it runs before the close.

The ReadFile inside main still sees an empty file because the deferred Flush hasn't fired yet. But once main returns, the defers run in reverse order: w.Flush() first (draining the buffer into the file), then f.Close() (releasing the file descriptor). After the program exits, the file on disk contains the line.

A more defensive pattern checks the error from Flush:

The deferred closures capture errors from Flush and Close and assign them to the named return err only when no earlier error is set. This is the pattern Go's standard library uses for any function that writes to a file and wants to surface I/O errors cleanly.

bufio.Scanner is the highest-level type in the package. It wraps a reader and lets you iterate over tokens (by default, lines) with a clean three-method API: Scan, Text, Err. For line-by-line file reading, Scanner is what most idiomatic Go uses.

The loop reads cleanly. scanner.Scan() returns true while there's another token, and false at end-of-file or when an error occurs. scanner.Text() returns the current token as a string, stripped of its trailing newline. After the loop, scanner.Err() reports any error other than EOF (EOF is silent and ends the loop normally).

scanner.Bytes() returns the same token as a []byte. The bytes are only valid until the next call to Scan, because the scanner reuses its internal buffer. If you need to keep them across calls, copy with append([]byte(nil), scanner.Bytes()...) or just use Text(), which returns a fresh string.

Split Functions

Scanner has a pluggable split function that decides what a token is. The default is bufio.ScanLines, which splits on \n (and handles a trailing \r\n on Windows files). Three other built-ins are useful:

You install a split function with scanner.Split(splitFunc) before the first call to Scan. Calling it after scanning has started will panic.

ScanWords strips whitespace and yields each word as a token. This is the easy way to count word frequencies, build a simple search index over a review corpus, or tokenize input for further processing.

Custom split functions are also possible. The signature is func(data []byte, atEOF bool) (advance int, token []byte, err error). You write one when you need to split on a delimiter the standard package doesn't cover, for example, on every | character in a pipe-delimited file. They're an advanced feature; lean on the built-ins until you have a clear reason to write your own.

The Long-Line Limit

Scanner has a default maximum token size of 64 KB (bufio.MaxScanTokenSize). If any single line is longer than that, Scan returns false and Err returns bufio.ErrTooLong. A common mistake is forgetting to check Err after the loop, so the iteration stops early unnoticed.

Here's a program that hits the limit:

The third line is never reached. The scanner saw the 70,000-byte second line, failed to fit it in its 64 KB buffer, and stopped. Without the scanner.Err() check, a developer would see only the first line and assume the file has one line.

The fix is to grow the buffer with scanner.Buffer(buf, max). The first argument is the initial buffer (often nil so the scanner allocates); the second is the maximum size it's allowed to grow to.

All three lines now appear. The second argument to Buffer (here 1 MB) caps how big the buffer can grow; lines bigger than that still trip the same error. Pick a max based on the largest line you actually expect, with some headroom.

If lines could be arbitrarily large and you don't want to commit to a max upfront, drop Scanner and use bufio.Reader.ReadString('\n') instead. ReadString has no built-in size cap; it grows as needed until it either finds the delimiter or runs out of memory.

The three options stack from highest-level to lowest:

A nightly job that scans a 2 GB order log line by line, where each line is under 1 KB: Scanner. A program that has to handle log lines of unknown maximum size: Reader with ReadString. A program that reads a fixed-record binary file 1 KB at a time: raw os.File.Read (the kernel's page cache already does the buffering for you at that granularity, and adding bufio on top buys nothing).

The default choice for text files is Scanner. For binary files where you read in known chunks, use raw os.File. Use bufio.Reader directly when you need delimiter-driven reading without Scanner's cap.

A common e-commerce task is filtering a big log to extract only the records that match some condition. Here's a program that reads an order log line by line, extracts orders over $100, and writes them to a new file. Both ends are buffered, and the writer uses the defer pattern correctly.

Three things to note. First, the scanner and writer are both buffered, so the program reads and writes in 4 KB chunks regardless of how small each line is. Second, defer writer.Flush() is declared after defer out.Close(), so it runs first when main returns. Third, the explicit writer.Flush() before the ReadFile is what makes the in-program read see the bytes; without it, the file on disk is still empty until main exits.

This pattern (scan, filter, write) scales to gigabyte-sized files because memory usage stays constant. Only one line is in memory at a time on the read side, and only up to 4 KB of pending bytes on the write side.