Getting a raw, unparsed HTTP response

Question

Are there any straightforward ways to make a HTTP request and get at the raw, unparsed response (specifically the headers)?

Answer 1

Using the socket module directly:

import socket

CRLF = "\r\n"

request = [
    "GET / HTTP/1.1",
    "Host: www.example.com",
    "Connection: Close",
    "",
    "",
]

# Connect to the server
s = socket.socket()
s.connect(('www.example.com', 80))

# Send an HTTP request
s.send(CRLF.join(request))

# Get the response (in several parts, if necessary)
response = ''
buffer = s.recv(4096)
while buffer:
    response += buffer
    buffer = s.recv(4096)

# HTTP headers will be separated from the body by an empty line
header_data, _, body = response.partition(CRLF + CRLF)

print header_data

HTTP/1.0 302 Found
Location: http://www.iana.org/domains/example/
Server: BigIP
Connection: Keep-Alive
Content-Length: 0