Python http.client

HTTP, or Hypertext Transfer Protocol, is the foundation of data communication on the World Wide Web. It is a protocol used for transferring hypertext (text with links) and other resources between clients (like web browsers) and servers.

Key Features of HTTP

Request-Response Model: HTTP operates on a request-response model, where a client sends a request to a server, and the server responds with the requested resource or an error message.

Stateless: Each HTTP request is independent. The server does not retain information about previous requests, which can lead to a more efficient design but requires additional mechanisms for maintaining state (like cookies).

Methods: HTTP defines several methods to specify the action to be performed on the resource. Common methods include:

GET: Retrieve data from the server.
POST: Send data to the server (often used for form submissions).
PUT: Update a resource on the server.
DELETE: Remove a resource from the server.
Headers: HTTP requests and responses can include headers that provide additional context, such as content type, authentication tokens, and caching information.

Status Codes: The server's response includes a status code indicating the result of the request. Common status codes include:

200 OK: The request was successful.
404 Not Found: The requested resource could not be found.
500 Internal Server Error: The server encountered an error.

Python http.client Module

The http.client module enables sending HTTP requests and handling responses. It's a lower-level library than popular libraries like requests, providing more control over the HTTP request and response cycle.

1. Making a Basic GET Request

The HTTPConnection class is used to establish a connection to the server. It allows you to send requests and receive responses.

import http.client

# Create a connection to a host
conn = http.client.HTTPConnection("www.example.com")

# Make a GET request
conn.request("GET", "/")

# Get the response
response = conn.getresponse()
print("Status:", response.status)
print("Reason:", response.reason)
print("Headers:", response.getheaders())

# Read and print the response body
body = response.read()
print("Body:", body.decode())

# Close the connection
conn.close()

Output:

Status: 200
Reason: OK
Headers: [(Header key-value pairs)]
Body: ...

Explanation: We create a connection to "www.example.com", make a GET request, and print the status, reason, headers, and body of the response.

2. Making a POST Request with Headers and Body

The request() method also supports the POST method. We can include headers and a body with the request.

import http.client
import json

# Establish the connection
conn = http.client.HTTPConnection("jsonplaceholder.typicode.com")

# Define the headers and body
headers = {
    "Content-type": "application/json"
}
body = json.dumps({
    "title": "foo",
    "body": "bar",
    "userId": 1
})

# Make the POST request
conn.request("POST", "/posts", body, headers)

# Get and print the response
response = conn.getresponse()
print("Status:", response.status)
print("Reason:", response.reason)
print("Headers:", response.getheaders())
response_body = response.read().decode()
print("Body:", response_body)

# Close the connection
conn.close()

Output:

Status: 201
Reason: Created
Headers: [(Header key-value pairs)]
Body: {"title": "foo", "body": "bar", "userId": 1, "id": 101}

Explanation: The POST request sends JSON data to the server, and the server responds with status code 201 (Created), confirming the successful creation of a resource.

3. Using HTTPS with HTTPSConnection

To connect securely, we use HTTPSConnection, which establishes a secure SSL/TLS connection.

import http.client

# Create a secure HTTPS connection
conn = http.client.HTTPSConnection("www.example.com")

# Make a GET request
conn.request("GET", "/")

# Print response details
response = conn.getresponse()
print("Status:", response.status)
print("Reason:", response.reason)
print("Headers:", response.getheaders())
body = response.read().decode()
print("Body:", body[:100])  # Print first 100 characters for brevity

# Close the connection
conn.close()

Output:

Status: 200
Reason: OK
Headers: [(Header key-value pairs)]
Body: ... (truncated)

Explanation: The HTTPS connection provides encrypted communication with the server. The response output is similar to that of HTTP but over a secure channel.

4. Handling Response Status and Errors

The http.client module provides exceptions like HTTPException for handling errors in communication. This example catches and displays HTTP errors.

import http.client

try:
    conn = http.client.HTTPConnection("nonexistentwebsite.com")
    conn.request("GET", "/")
    response = conn.getresponse()
except http.client.HTTPException as e:
    print("HTTPException occurred:", e)
except Exception as e:
    print("An error occurred:", e)
finally:
    conn.close()

Output:

An error occurred: [Errno -2] Name or service not known

Explanation: In this example, a request to a nonexistent server raises an error, which we handle using try-except to prevent the program from crashing.

5. Handling Chunked Responses

Some servers respond in "chunked" transfer mode, where data arrives in segments. The read() method can handle this.

import http.client

# Connect to a host that uses chunked transfer
conn = http.client.HTTPConnection("www.httpbin.org")
conn.request("GET", "/stream-bytes/1024")  # Endpoint that streams data

# Process the response
response = conn.getresponse()
print("Status:", response.status)
print("Transfer-Encoding:", response.getheader("Transfer-Encoding"))

# Read and print the first 100 bytes of the response
body = response.read(100)
print("Chunked Response:", body.decode())
conn.close()

Output:

Status: 200
Transfer-Encoding: chunked
Chunked Response: <first 100 bytes of streamed data>

Explanation: For endpoints that return chunked responses, this method reads the data until the connection closes or the buffer limit is reached.

6. Adding Query Parameters

To add query parameters, use urllib.parse to construct the URL.

import http.client
from urllib.parse import urlencode

# Create connection
conn = http.client.HTTPConnection("jsonplaceholder.typicode.com")

# Define parameters
params = urlencode({"userId": 1})
url = f"/posts?{params}"

# Send GET request
conn.request("GET", url)
response = conn.getresponse()

# Print the response
print("Status:", response.status)
print("Body:", response.read().decode())
conn.close()

Output:

Status: 200
Body: [{"userId": 1, "id": 1, ...}]

Explanation: Query parameters, like ?userId=1, allow more specific data requests without changing the endpoint URL.

7. Setting and Retrieving Custom Headers

Adding headers to a request is essential when interacting with APIs that require authentication tokens or specific content types.

import http.client

# Create connection
conn = http.client.HTTPConnection("jsonplaceholder.typicode.com")

# Define custom headers
headers = {
    "User-Agent": "Python-http.client",
    "Accept": "application/json"
}

# Send GET request with headers
conn.request("GET", "/posts/1", headers=headers)
response = conn.getresponse()

# Print the response headers and body
print("Response Headers:", response.getheaders())
print("Body:", response.read().decode())
conn.close()

Output:

Response Headers: [(Header key-value pairs)]
Body: {"userId": 1, "id": 1, ...}

Explanation: Custom headers ensure that the request is identified properly by the server or follows certain specifications.

8. HTTP Client Context Management

Using with statements can simplify the process of closing connections.

import http.client

# Use with statement to manage the connection context
with http.client.HTTPConnection("www.example.com") as conn:
    conn.request("GET", "/")
    response = conn.getresponse()
    print("Status:", response.status)
    print("Body:", response.read(100).decode())

Output:

Status: 200
Body: <first 100 bytes of response>

Explanation: The with statement manages resources by automatically closing the connection once the block completes.

Summary

The http.client module provides a powerful tool for handling HTTP requests in Python with methods for GET, POST, headers, error handling, secure connections, and chunked transfers. Though it requires more code than higher-level libraries, it offers greater control over HTTP communication and error management.