Offensive Security Researcher & Quantum Cryptography Analyst

Building an HTTP client with urllib.request module in python

Abdul Wahab Junaid
Building an HTTP client with urllib.request module in python

The urllib.request module in Python provides a higher-level interface for fetching URLs (Uniform Resource Locators) than the http module. It offers a simpler and more convenient way to make HTTP requests, handle redirections, and deal with common HTTP features like cookies and authentication.

Key features of the urllib.request module:

Make a POST request using the urllib.request module. It sends data to the “http://httpbin.org/post” URL and prints the response.

Import Statements

Python
import urllib.request
import urllib.parse

Data Preparation for POST Request

Python
data_dictionary = {"id": "0123456789"}
data = urllib.parse.urlencode(data_dictionary)
data = data.encode('ascii')
  1. Data Dictionary: Creates a dictionary containing data for the POST request.
  2. URL Encoding: Uses urllib.parse.urlencode to convert the dictionary into a URL-encoded string.
  3. Byte Encoding: Converts the string into bytes using encode('ascii') to prepare it for sending in the POST request.

Making the POST Request

Python
with urllib.request.urlopen("http://httpbin.org/post", data) as response:
    print(response.read().decode('utf-8'))
  1. urlopen Method: Opens the specified URL (“http://httpbin.org/post”) and sends the prepared data as a POST request.
  2. Response Handling:
Python
import urllib.request
import urllib.parse

#POST request

data_dictionary = {"id": "0123456789"}
data = urllib.parse.urlencode(data_dictionary)
data = data.encode('ascii')

with urllib.request.urlopen("http://httpbin.org/post", data) as response:
	print(response.read().decode('utf-8'))

The script demonstrates how to make a POST request using the urllib.request module. It prepares data in the form of a dictionary, encodes it, sends it to the specified URL, and prints the response received from the server. The http://httpbin.org/post URL is a testing service that echoes back the data it receives, making it useful for testing and debugging HTTP requests.

How to make an HTTP GET request to the “http://httpbin.org/get” URL using the urllib.request module. It then processes the JSON response and prints the decoded data.

Import Statements

Python
import urllib.request
import json

URL for GET Request

Python
url = "http://httpbin.org/get"

Making the GET Request

Python
with urllib.request.urlopen(url) as response_json:
    data_json = json.loads(response_json.read().decode("utf-8"))
    print(data_json)
  1. urlopen Method: Opens the specified URL (“http://httpbin.org/get”) and sends a GET request.
  2. Response Handling:

The script makes an HTTP GET request to “http://httpbin.org/get” using the urllib.request module. It reads the response, decodes it as a UTF-8 string, and then uses the json module to parse the JSON-formatted string into a Python dictionary. Finally, it prints the decoded JSON data, demonstrating a simple way to interact with a web API and handle JSON responses in Python.

Get Response and Request Header


In the context of HTTP (Hypertext Transfer Protocol), a request header and a response header are additional pieces of information that accompany an HTTP request and response, respectively. They provide metadata about the request or response, enabling better communication between the client (web browser) and the server.

Request Headers

Request headers are sent by the client to the server along with the HTTP request. They provide information about the client’s capabilities, preferences, and the context of the request. Some common request headers include:

Response Headers

Response headers are sent by the server to the client along with the HTTP response. They provide information about the response, such as the status code, content type, and server details. Some common response headers include:

Significance of Request and Response Headers

Request and response headers play a crucial role in the efficient and effective communication between the client and server. They provide valuable information that enables:

Request and response headers are vital components of HTTP communication, providing contextual information that enhances the interaction between clients and servers. They facilitate content negotiation, authentication, caching, error handling, and debugging, ensuring seamless and efficient data exchange on the World Wide Web.

How to set a custom user agent (in this case, mimicking the Chrome browser’s user agent) for an HTTP request using the urllib.request module. It then prints the response headers received from the server and the request headers sent to the server.

Import Statements

Python
import urllib.request
from urllib.request import Request

URL and User Agent Definition

Python
url = "https://awjunaid.com"
USER_AGENT = 'Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.101 Mobile Safari/537.36'

Function to Set Chrome User Agent

Python
def chrome_user_agent():
    opener = urllib.request.build_opener()
    opener.addheaders = [('User-agent', USER_AGENT)]
    urllib.request.install_opener(opener)
    response = urllib.request.urlopen(url)
  1. Opener Creation: Creates an opener using urllib.request.build_opener() to set a custom user agent.
  2. Setting User Agent: Adds the custom user agent to the opener’s headers.
  3. Installing Opener: Installs the custom opener.
  4. HTTP Request: Opens the URL using urllib.request.urlopen().

Printing Response Headers

Python
    print("Response headers")
    print("--------------------")
    for header, value in response.getheaders():
        print(header + ":" + value)

Creating Request with Custom User Agent

Python
    request = Request(url)
    request.add_header('User-agent', USER_AGENT)
  1. Request Creation: Creates a new request using urllib.request.Request() for the same URL.
  2. Setting User Agent in Request: Adds the custom user agent to the request headers.

Printing Request Headers

Python
    print("\nRequest headers")
    print("--------------------")
    for header, value in request.header_items():
        print(header + ":" + value)

Main Block

Python
if __name__ == '__main__':
    chrome_user_agent()
Python
#!/usr/bin/env python3

import urllib.request
from urllib.request import Request

url="https://awjunaid.com"
USER_AGENT = 'Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.101 Mobile Safari/537.36'

def chrome_user_agent():
    opener = urllib.request.build_opener()
    opener.addheaders = [('User-agent', USER_AGENT)]
    urllib.request.install_opener(opener)
    response = urllib.request.urlopen(url)
    print("Response headers")
    print("--------------------")
    for header,value in response.getheaders():
        print(header + ":" + value)

    request = Request(url)
    request.add_header('User-agent', USER_AGENT)
    print("\nRequest headers")
    print("--------------------")
    for header,value in request.header_items():
	    print(header + ":" + value)

if __name__ == '__main__':
    chrome_user_agent()

Output

Bash
Response headers
--------------------
date:Thu, 16 Nov 2023 01:42:50 GMT
content-type:text/html; charset=UTF-8
transfer-encoding:chunked
vary:Accept-Encoding
cache-control:public, s-maxage=216000
server:Apache
x-content-security-policy:default-src 'self'; script-src 'self' https://hcaptcha.com https://*.hcaptcha.com; frame-src 'self' https://hcaptcha.com https://*.hcaptcha.com; style-src 'self' https://hcaptcha.com https://*.hcaptcha.com; connect-src 'self' https://hcaptcha.com https://*.hcaptcha.com; unsafe-eval 'self' https://hcaptcha.com https://*.hcaptcha.com; unsafe-inline 'self' https://hcaptcha.com https://*.hcaptcha.com;
link:<https://awjunaid.com/wp-json/>; rel="https://api.w.org/", <https://awjunaid.com/wp-json/wp/v2/pages/2172>; rel="alternate"; type="application/json", <https://wp.me/PeTdSC-z2>; rel=shortlink
x-stackcache-cacheable:yes
x-cache-enabled:true
x-provided-by:StackCDN
x-origin-cache-status:HIT
x-cdn-cache-status:HIT
x-via:MAD1
connection:close

Request headers
--------------------
User-agent:Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.101 Mobile Safari/537.36

The script demonstrates how to set a custom user agent for an HTTP request using urllib.request. It mimics the Chrome browser’s user agent, makes a request to “http://python.org,” and prints both the response headers received from the server and the request headers sent to the server. The user agent can be crucial for web scraping or making requests that mimic different browsers or devices.

Extracting Emails from a URL with urllib.request

Fetch and extract email addresses from the HTML content of a given URL. It sets a custom user agent, makes an HTTP request to the specified URL, retrieves the HTML content, and then uses a regular expression to find and print email addresses present in the HTML.

Import Statements

Python
import urllib.request
import re

User Agent and URL Input

Python
USER_AGENT = 'Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.101 Mobile Safari/537.36'

url =  input("Enter url:http://")

Custom Opener with User Agent

Python
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', USER_AGENT)]
urllib.request.install_opener(opener)

HTTP Request to the Given URL

Python
response = urllib.request.urlopen('http://'+url)
html_content= response.read()

Regular Expression Pattern

Python
pattern = re.compile("[-a-zA-Z0-9._]+[-a-zA-Z0-9._]+@[-a-zA-Z0-9_]+.[a-zA-Z0-9_.]+")
mails = re.findall(pattern, str(html_content))

Printing Extracted Email Addresses

Python
print(mails)
Python
import urllib.request
import re

USER_AGENT = 'Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.101 Mobile Safari/537.36'

url =  input("Enter url:http://")

https://awjunaid.com/terms-conditions/

opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', USER_AGENT)]
urllib.request.install_opener(opener)

response = urllib.request.urlopen('http://'+url)
html_content= response.read()
pattern = re.compile("[-a-zA-Z0-9._]+[-a-zA-Z0-9._]+@[-a-zA-Z0-9_]+.[a-zA-Z0-9_.]+")
mails = re.findall(pattern,str(html_content))
print(mails)

This script is designed to extract email addresses from the HTML content of a given URL. It sets a custom user agent, performs an HTTP request, retrieves the HTML, applies a regular expression pattern to identify email addresses, and finally prints the extracted email addresses. Keep in mind that using regular expressions for parsing HTML can be fragile, and more robust solutions like HTML parsers are recommended for complex scenarios.

Download Files with urllib.request

Download an image file (Python logo) from a given URL using two different methods: urllib.request.urlretrieve and urllib.request.urlopen. It saves the downloaded image as “python.png.”

Import Statement

Python
import urllib.request

Starting the Download

Python
print("starting download....")

URL for Image Download

Python
url = "https://www.python.org/static/img/python-logo.png"

Downloading with urllib.request.urlretrieve

Python
# download file with urlretrieve
urllib.request.urlretrieve(url, "python.png")

Downloading with urllib.request.urlopen

Python
# download file with urlopen
with urllib.request.urlopen(url) as response:
    print("Status:", response.status)
    print("Downloading python.png")
    with open("python.png", "wb") as image:
        image.write(response.read())
Python
#!/usr/bin/python

import urllib.request

print("starting download....")

url="https://www.python.org/static/img/python-logo.png"

#download file with urlretrieve
urllib.request.urlretrieve(url, "python.png")

#download file with urlopen
with urllib.request.urlopen(url) as response:
    print("Status:", response.status)
    print( "Downloading python.png")
    with open("python.png", "wb" ) as image:
        image.write(response.read())

The script demonstrates two methods for downloading an image file from a URL using the urllib.request module. It first uses urlretrieve to directly download and save the image, and then it uses urlopen to open the URL, read the content, and save it to a local file. The script is a basic example of downloading files from the web in Python.

Handling Exceptions with urllib.request

Defines a function, count_words_file(url), which attempts to open and read the content of a file from a given URL. It then counts the number of words in the content using the split() method. The code also includes error handling to catch URLError exceptions and print information about the error.

Import Statements

Python
import urllib.request
import urllib.error

Function for Counting Words in a File

Python
def count_words_file(url):
    try:
        file_response = urllib.request.urlopen(url)
    except urllib.error.URLError as error:
        print('Exception', error)
        print('Reason:', error.reason)
    else:
        content = file_response.read()
        return len(content.split())

Example Usage

Python
print(count_words_file('https://www.gutenberg.org/cache/epub/2000/pg2000.txt'))
Python
count_words_file('https://not-exists.txt')
Python
#!/usr/bin/env python3

import urllib.request
import urllib.error

def count_words_file(url):
    try:
        file_response = urllib.request.urlopen(url)
    except urllib.error.URLError as error:
        print('Exception', error)
        print('reason', error.reason)
    else:
        content = file_response.read()
        return len(content.split())


print(count_words_file('https://www.gutenberg.org/cache/epub/2000/pg2000.txt'))

count_words_file('https://not-exists.txt')

The script defines a function to count words in the content of a file retrieved from a given URL. It demonstrates error handling for URLError exceptions when attempting to open a URL. The examples showcase the function’s usage with both a valid URL and a non-existent URL, highlighting the error-handling mechanism.

Exit mobile version