Python's standard library ships urllib.request, a no-dependency way to call HTTP services. The third-party requests library is what most code uses in practice, but seeing the manual form first makes the convenience of requests clear. This lesson walks through the anatomy of a request, the standard library tools to build one, how to read the response back, and the common headers and encodings.

A single HTTP request has four parts:

GET and DELETE typically have no body. POST, PUT, and PATCH usually do. The headers describe both the body (its format and size) and the request itself (who's calling, what they accept, what auth they bring).

When building a request with urllib.request, set these four pieces explicitly. The library wraps them in the matching text format and sends them over a TCP connection to the server.

The diagram shows the layers. The Python code creates a request object. The library serialises it into the HTTP text format. That text travels over a TCP (or TLS) connection to the server. The server processes it and writes back response bytes that travel the same path in reverse. urllib.request handles the middle two boxes; the request object is the caller's responsibility.

The smallest useful request is a GET for a single URL. urllib.request.urlopen takes a URL string or a Request object and returns a response.

urlopen opens the connection, sends the request, and returns a response object that works like a file (so it's wrapped in with). response.status is the HTTP status code. response.headers behaves like a case-insensitive dict of response headers. response.read() gives the raw bytes of the body, decoded to a string with the response's character encoding. httpbin.org is a public test server that echoes back the request details, useful for examples.

Calling urlopen with only a URL defaults to a GET request. There's no body, no extra headers beyond what the library adds, and no auth. For trying out a public endpoint, this one line is enough.

urlopen opens a fresh TCP (and TLS) connection on every call. Calling it in a tight loop pays the connection setup cost every time. For more than a handful of requests, the requests library's Session reuses the connection.

For anything more than a default GET, build a Request object explicitly and pass it to urlopen. The Request constructor takes a URL, an optional body (data), a headers dict, and the HTTP method.

This is the same call as before, but now with explicit headers. Accept: application/json tells the server "give me JSON if you have it." User-Agent identifies the calling program. Servers sometimes log this, and some block requests with the default Python-urllib/3.x user agent because it's a common scraping signature.

The method argument is optional. If data is provided, the method defaults to POST. Otherwise, it defaults to GET. Either can be overridden by setting method explicitly.

The object urlopen returns has a small but useful interface. The common methods and attributes:

The body is always returned as bytes. For text content, decode it with the matching encoding. Most JSON APIs use UTF-8, so response.read().decode("utf-8") is the default move.

response.read() returns bytes, and json.loads accepts bytes (since Python 3.6) and figures out the encoding from the JSON itself. The end result is a Python dict ready to index into. httpbin.org/get echoes back the request, so the Accept header is visible in the response.

To check the encoding the server reported, look at Content-Type:

The Content-Type header sometimes carries a charset= parameter. get_content_charset() extracts it. When the server doesn't say, UTF-8 is a safe default for modern text APIs.

A GET request often carries parameters in the URL's query string. The query can be written manually, but anything with special characters needs to be percent-encoded. Use urllib.parse.urlencode for that.

urlencode takes a dict and produces a query string with the correct escaping. The space in "audio gear" became a + (the form-encoding shorthand for space), and the dict's keys got joined with &. For a list value ({"tag": ["sale", "new"]}), pass doseq=True so it expands to tag=sale&tag=new instead of stringifying the list.

For percent-encoding a single value (a path segment, not a query), use urllib.parse.quote.

quote percent-encodes everything that isn't a "safe" URL character. The accented é became %C3%A9 (its UTF-8 bytes), and the spaces became %20. There's a quote_plus variant that encodes spaces as + instead of %20, which is what urlencode uses internally.

For methods that carry a body (POST, PUT, PATCH), the body's format depends on the API. Three encodings cover almost everything:

Form-Encoded

This is the format an HTML <form> sends by default. The body looks like a query string.

The body has to be bytes, not a string, which is why .encode("utf-8") follows urlencode. The Content-Type header tells the server how to parse it. urlencode and the matching content type together imitate a browser form submission.

JSON

Modern APIs almost always want JSON. The pattern is the same shape: serialise the body, set the content type, send.

json.dumps produces a string, .encode("utf-8") turns it into bytes for the body, and Content-Type: application/json tells the server how to parse it. The server (in this case httpbin.org/post) echoes the parsed body back under the json key, confirming it arrived intact.

This is the most common API call shape. The requests library has a one-liner for it (requests.post(url, json=payload)), but the underlying mechanics are what's shown above.

Multipart (File Uploads)

When the body needs to include both fields and files (a product photo upload, for example), the format is multipart/form-data. Building this by hand is painful with urllib.request, which is one of the strongest reasons to use the requests library. multipart/form-data is the matching content type for uploading a file alongside form fields.

Three headers carry most of the weight on real API calls:

Setting them is a matter of passing a headers dict to Request. There are no special methods; headers are arbitrary string key/value pairs.

The server echoes the headers it received. Accept-Encoding: identity was added by urllib.request itself; the library disables compression by default unless asked. The other three were set in code, exactly as configured.

A note on case: HTTP header names are case-insensitive, and most servers normalise them. Python's urllib.request stores them with the case passed in, but the server doesn't distinguish between Authorization and authorization. Pick one style and stick with it.

Two auth flavours cover most APIs called from Python.

Basic auth sends a username and password on every request, base64-encoded in the Authorization header. It's the oldest scheme, still used by some internal tools and CI systems.

The username and password are joined with a colon, base64-encoded, and prefixed with Basic . Base64 is encoding, not encryption: anyone who sees the header can decode it back to the original. Basic auth must always travel over HTTPS, or those credentials are visible to anyone on the network.

Bearer auth sends a token issued by the server. Most modern APIs use this, often with OAuth or a similar system handing out the tokens.

The structure is similar: Authorization: Bearer <token>. The token itself is opaque to the client; the server knows what it means. Like Basic, Bearer relies on HTTPS to keep the token from leaking in transit.

Logging a request that includes the Authorization header is a common way tokens leak into log files and bug reports. Strip or redact that header before logging request details.

urlopen raises an exception when the request fails. There are two main exception types in urllib.error:

HTTPError is a subclass of URLError, so catching URLError catches both. Handling them separately is often appropriate because they mean different things.

A 404 raised HTTPError, not a returned response object. That's a quirk of urllib.request: any 4xx or 5xx is an exception, not a value. The requests library is more forgiving here, returning the response and leaving status checking to the caller.

A connection-level failure is different:

The DNS lookup failed, so the request never reached a server. That's URLError, not HTTPError. Distinguishing the two matters for retry logic: a 503 might warrant a retry, but a DNS failure suggests checking the URL, not retrying.

urlopen accepts a timeout argument in seconds. Without one, a request will hang forever if the server stops responding, which is rarely the intended behaviour.

/delay/5 makes the server wait 5 seconds before responding. The 2-second timeout cuts the connection first, and urlopen raises URLError with a "timed out" reason. In modern Python the timeout shows up wrapped as URLError, though socket.timeout is the underlying exception type.

Always set a timeout on real code. The matching value depends on the API; somewhere between 5 and 30 seconds is typical for normal APIs, with shorter values (1 to 2 seconds) for fast internal services.

urlopen follows HTTP redirects automatically. If a request to /old-product returns 301 Moved Permanently with Location: /products/101, the library issues a second request to /products/101 and returns that response.

response.status is 200 (the final response, not the 302 that triggered the redirect). response.url is the URL the client ended up at, which differs from the URL originally requested. That's a useful debugging hint: an unexpected URL in response.url often means a redirect happened.

Most of the time this automatic behaviour is fine. For some uses (auditing, security), the redirect itself needs to be visible. Disabling it requires installing a custom redirect handler, which is one place urllib.request shows its age. The requests library has a one-line allow_redirects=False, another nudge toward it for non-trivial work.

urllib.request is in the standard library, which means no dependencies. That matters when writing a small script, a constrained-deployment environment, or a library that doesn't want to drag dependencies onto its users. For those cases, it does the job.

For most real application code, the third-party requests library is what people use. It cleans up almost every wart this lesson has touched:

The mechanics covered here carry over to the requests library. A requests.post(url, json=payload) ultimately produces a request with Content-Type: application/json, a JSON body, and a POST method. Knowing what's happening underneath makes the higher-level library make sense.