Building an HTTP client with Request and RESTful API in python

Abdul Wahab Junaid

3 years ago

Building an HTTP client with Request and RESTful API in python

The requests module in Python is a popular library for making HTTP requests, and it is commonly used for working with RESTful APIs. RESTful APIs (Representational State Transfer APIs) are web services that adhere to the principles of REST and allow clients to interact with server resources over HTTP.Overview of using the requests module to work with RESTful APIs in Python:

Installing the requests module To use the requests module, you’ll need to install it using pip:

Bash

pip install requests

Making a GET request To make a GET request to an API endpoint, you can use the requests.get() method. For example, to get a list of users from a JSON API, you would do the following:

Python

import requests

response = requests.get('https://jsonplaceholder.typicode.com/users')

The response object contains the HTTP response from the API. You can access the status code of the response using the status_code property:

Python

status_code = response.status_code
print(status_code)

If the request was successful (status code 200), you can access the JSON data using the json() method:

Python

data = response.json()
print(data)

Making a POST request To make a POST request to an API endpoint, you can use the requests.post() method. For example, to create a new user in a JSON API, you would do the following:

Python

import requests

data = {
    "name": "John Doe",
    "email": "johndoe@example.com"
}

response = requests.post('https://jsonplaceholder.typicode.com/users', json=data)

The json=data parameter tells the requests module to encode the data dictionary as JSON and send it as the request body. You can access the status code of the response using the status_code property:

Python

status_code = response.status_code
print(status_code)

If the request was successful (status code 201), you can access the JSON data using the json() method:

Python

data = response.json()
print(data)

Making a PUT request To make a PUT request to an API endpoint, you can use the requests.put() method. For example, to update an existing user in a JSON API, you would do the following:

Python

import requests

data = {
    "id": 1,
    "name": "Jane Doe",
    "email": "janedoe@example.com"
}

response = requests.put('https://jsonplaceholder.typicode.com/users/1', json=data)

Python

status_code = response.status_code
print(status_code)

If the request was successful (status code 200), you can access the JSON data using the json() method:

Python

data = response.json()
print(data)

Making a DELETE request To make a DELETE request to an API endpoint, you can use the requests.delete() method. For example, to delete an existing user in a JSON API, you would do the following:

Python

import requests

response = requests.delete('https://jsonplaceholder.typicode.com/users/1')

You can access the status code of the response using the status_code property:

Python

status_code = response.status_code
print(status_code)

If the request was successful (status code 204), there will be no JSON data in the response.

The Requests module provides a wide range of other features for interacting with RESTful APIs, such as:

Adding headers to requests
Handling authentication
Following redirects
Uploading files
Debugging requests

For more information on using the Requests module, please refer to the official documentation: https://requests.readthedocs.io/en/latest/

The requests module to send a GET request to a specified URL (hostname) and then prints information about the response, including the JSON content, status code, response headers, and request headers.

Import Statements

Bash

import requests
import json

Import Statements: Import the requests module for making HTTP requests and the json module for handling JSON data.

User Input

Bash

domain = input("Enter the hostname http://")

User Input: Takes user input for the hostname (URL) with a prefix of “http://”.

Sending a GET Request

Bash

response = requests.get("http://" + domain)

GET Request: Uses the requests.get method to send a GET request to the specified URL.

Printing JSON Content

Bash

print(response.json)

Printing JSON Content: Attempts to print the JSON content of the response. However, there is a mistake in the code (response.json should be response.json()), and this part may not work correctly.

Printing Status Code

Bash

print("Status code: " + str(response.status_code))

Printing Status Code: Prints the HTTP status code of the response.

Printing Response Headers

Bash

print("Headers response: ")
for header, value in response.headers.items():
    print(header, '-->', value)

Printing Response Headers: Loops through the headers of the response and prints each header along with its value.

Printing Request Headers

Bash

print("Headers request : ")
for header, value in response.request.headers.items():
    print(header, '-->', value)

Printing Request Headers: Loops through the headers of the original request and prints each header along with its value.

Bash

#!/usr/bin/env python3

import requests, json


domain = input("Enter the hostname http://")

response = requests.get("http://"+domain)

print(response.json)

print("Status code: "+str(response.status_code))

print("Headers response: ")
for header, value in response.headers.items():
  print(header, '-->', value)
  
print("Headers request : ")
for header, value in response.request.headers.items():
  print(header, '-->', value)

Obtaining only keys from header

Bash

import requests

if __name__ == "__main__":
    domain = input("Enter the hostname http://")
    response = requests.get("http://"+domain)
    for header in response.headers.keys():
        print(header  + ":" + response.headers[header])

This script takes user input for a hostname, sends a GET request to the specified URL using the requests module, and prints information about the response, including the JSON content (if corrected), status code, response headers, and request headers. The code provides insights into the communication between the client and the server for the given URL.

Getting images and links from a URL with request

Extract and display images and links from the HTML content of a given URL. It uses the requests module to fetch the HTML content and regular expressions to identify and extract image sources (<img>) and links (<a>).

Import Statements

Bash

import requests
import re

Import Statements: Import the requests module for making HTTP requests and the re module for regular expressions.

User Input for URL

Bash

url = input("Enter URL > ")

User Input: Takes user input for the URL.

Sending a GET Request and Retrieving HTML Content

Bash

var = requests.get(url).text

GET Request: Uses the requests.get method to send a GET request to the specified URL and retrieves the HTML content.

Extracting and Printing Images

Bash

print("Images:")
print("#########################")
for image in re.findall("<img (.*)>", var):
    for images in image.split():
        if re.findall("src=(.*)", images):
            image = images[:-1].replace("src=\"", "")
            if image.startswith("http"):
                print(image)
            else:
                print(url + image)

Image Extraction:
Uses a regular expression to find <img> tags in the HTML content.
Iterates through the attributes of each <img> tag.
Extracts the src attribute (image source).
Prints the full image URL, appending the base URL if the source is relative.

Extracting and Printing Links

Bash

print("#########################")
print("Links:")
print("#########################")
for link, name in re.findall("<a (.*)>(.*)</a>", var):
    for a in link.split():
        if re.findall("href=(.*)", a):
            url_image = a[0:-1].replace("href=\"", "")
            if url_image.startswith("http"):
                print(url_image)
            else:
                print(url + url_image)

Link Extraction:
Uses a regular expression to find <a> tags in the HTML content.
Iterates through the attributes of each <a> tag.
Extracts the href attribute (link).
Prints the full link URL, appending the base URL if the link is relative.

Bash

#!/usr/bin/env python3
    
import requests
import re

url = input("Enter URL > ")
var = requests.get(url).text

print("Images:")
print("#########################")
for image in re.findall("<img (.*)>",var):
    for images in image.split():
        if re.findall("src=(.*)",images):
            image = images[:-1].replace("src=\"","")
            if(image.startswith("http")):
                print(image)
            else:
                print(url+image)

print("#########################")
print("Links:")
print("#########################")
for link,name in re.findall("<a (.*)>(.*)</a>",var):
    for a in link.split():
        if re.findall("href=(.*)",a):
            url_image = a[0:-1].replace("href=\"","")
            if(url_image.startswith("http")):
                print(url_image)
            else:
                print(url+url_image)

Bash

Enter URL > https://www.awjunaid.com

The script takes user input for a URL, fetches the HTML content using the requests module, and then uses regular expressions to extract and print images and links found in the HTML. The script provides a basic way to analyze the structure of a webpage and identify images and links. Keep in mind that using regular expressions for parsing HTML can be fragile, and more robust solutions like HTML parsers are recommended for complex scenarios.

Making GET request with the REST API

The requests module to make an HTTP GET request to “http://httpbin.org/get.” It then prints information about the response, including the HTTP status code, response headers, and specific details obtained from the response content (in JSON format).

Import Statements

Python

import requests
import json

Import Statements: Import the requests module for making HTTP requests and the json module for handling JSON data.

Making an HTTP GET Request

Python

response = requests.get("http://httpbin.org/get", timeout=5)

GET Request: Uses the requests.get method to send an HTTP GET request to “http://httpbin.org/get” with a timeout of 5 seconds.

Printing HTTP Status Code

Python

print("HTTP Status Code: " + str(response.status_code))

Printing Status Code: Prints the HTTP status code of the response.

Printing Response Headers

Python

print(response.headers)

Printing Response Headers: Prints the entire dictionary of response headers.

Checking Status Code and Processing JSON Content

Python

if response.status_code == 200:
    results = response.json()
    for result in results.items():
        print(result)

Checking Status Code and Processing JSON:
Checks if the HTTP status code is 200 (OK).
Parses the JSON content of the response and prints each key-value pair.

Printing Specific Response Headers

Python

print("Headers response: ")
for header, value in response.headers.items():
    print(header, '-->', value)

Printing Specific Response Headers: Loops through the response headers and prints each header along with its value.

Printing Request Headers

Python

print("Headers request : ")
for header, value in response.request.headers.items():
    print(header, '-->', value)

Printing Request Headers: Loops through the headers of the original request and prints each header along with its value.

Printing Server Information

Python

print("Server:" + response.headers['server'])

Printing Server Information: Prints the value of the ‘server’ header from the response.

Handling Error Responses

Python

else:
    print("Error code %s" % response.status_code)

Handling Error Responses: Prints an error message if the HTTP status code is not 200.

Python

import requests, json

response = requests.get("http://httpbin.org/get",timeout=5)

print("HTTP Status Code: " + str(response.status_code))
print(response.headers)

if response.status_code == 200:

	results = response.json()
	for result in results.items():
		print(result)

	print("Headers response: ")
	for header, value in response.headers.items():
		print(header, '-->', value)

	print("Headers request : ")
	for header, value in response.request.headers.items():
		print(header, '-->', value)

	print("Server:" + response.headers['server'])
else:
	print("Error code %s" % response.status_code)

This script sends an HTTP GET request to “http://httpbin.org/get,” retrieves and processes the response. It prints information such as the HTTP status code, response headers, specific details from the JSON content, and server information if the response is successful (status code 200). If there is an error, it prints an error message with the status code.

Making POST request with the REST API

Sending an HTTP POST request to “http://httpbin.org/post” with JSON data. It utilizes the requests module to make the request and includes headers for specifying the content type and expected response type.

Import Statements

Python

import requests
import json

Import Statements: Import the requests module for making HTTP requests and the json module for handling JSON data.

POST Request with JSON Data

Python

data_dictionary = {"id": "0123456789"}
headers = {"Content-Type": "application/json", "Accept": "application/json"}
response = requests.post("http://httpbin.org/post", data=data_dictionary, headers=headers, json=data_dictionary)

POST Request:
Uses the requests.post method to send an HTTP POST request to “http://httpbin.org/post.”
Provides JSON data in the data parameter and sets headers for content type and expected response type.

Printing HTTP Status Code

Python

print("HTTP Status Code: " + str(response.status_code))

Printing Status Code: Prints the HTTP status code of the response.

Printing Response Headers

Python

print(response.headers)

Printing Response Headers: Prints the entire dictionary of response headers.

Checking Status Code and Processing JSON Content

Python

if response.status_code == 200:
    results = response.json()
    for result in results.items():
        print(result)

Checking Status Code and Processing JSON:
Checks if the HTTP status code is 200 (OK).
Parses the JSON content of the response and prints each key-value pair.

Printing Specific Response Headers

Python

print("Headers response: ")
for header, value in response.headers.items():
    print(header, '-->', value)

Printing Specific Response Headers: Loops through the response headers and prints each header along with its value.

Printing Request Headers

Python

print("Headers request : ")
for header, value in response.request.headers.items():
    print(header, '-->', value)

Printing Request Headers: Loops through the headers of the original request and prints each header along with its value.

Printing Server Information

Python

print("Server:" + response.headers['server'])

Printing Server Information: Prints the value of the ‘server’ header from the response.

Handling Error Responses

Python

else:
    print("Error code %s" % response.status_code)

Handling Error Responses: Prints an error message if the HTTP status code is not 200.

Python

#!/usr/bin/env python3

import requests,json
data_dictionary = {"id": "0123456789"}
headers = {"Content-Type" : "application/json","Accept":"application/json"}
response = requests.post("http://httpbin.org/post",data=data_dictionary,headers=headers,json=data_dictionary)
print("HTTP Status Code: " + str(response.status_code))

print(response.headers)

if response.status_code == 200:

	results = response.json()
	for result in results.items():
		print(result)

	print("Headers response: ")
	for header, value in response.headers.items():
		print(header, '-->', value)

	print("Headers request : ")
	for header, value in response.request.headers.items():
		print(header, '-->', value)

	print("Server:" + response.headers['server'])
else:
	print("Error code %s" % response.status_code)

This script sends an HTTP POST request to “http://httpbin.org/post” with JSON data, retrieves and processes the response. It prints information such as the HTTP status code, response headers, specific details from the JSON content, and server information if the response is successful (status code 200). If there is an error, it prints an error message with the status code.

Managing a proxy with requests

An intresitng feature offered by the requsest module is the option to make the request through a proxy or intermediate machine between our internal network and the external network.

Python

import requests

proxy_url = "http://proxy.example.com:8080"
auth = ("username", "password")

response = requests.get('https://api.example.com/data', proxies={"http": proxy_url, "https": proxy_url}, auth=auth)

The proxies dictionary defines the proxy URLs for both HTTP and HTTPS requests. The auth parameter provides basic authentication credentials for the proxy server.

When working with the requests library in Python, you may need to use a proxy for making HTTP requests. A proxy acts as an intermediary between your client and the destination server. Here’s how you can manage a proxy with requests:

Using a Proxy

To use a proxy with requests, you can pass the proxies parameter to the request methods. The proxies parameter should be a dictionary with the protocol (http, https, etc.) as the key and the proxy URL as the value.

Python

import requests

# Define the proxy URL
proxy_url = "http://your_proxy_url"

# Specify the proxy in the request
proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

# Make a request using the proxy
response = requests.get("http://example.com", proxies=proxies)

# Print the response
print(response.text)

Proxy with Authentication

If your proxy requires authentication, you can include the username and password in the proxy URL. The format is http://username:password@proxy_url.

Python

import requests

# Define the proxy URL with authentication
proxy_url = "http://username:password@your_proxy_url"

# Specify the proxy in the request
proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

# Make a request using the proxy
response = requests.get("http://example.com", proxies=proxies)

# Print the response
print(response.text)

Exceptions and Timeouts with Proxies

When using proxies, you may encounter exceptions or longer response times. It’s a good practice to handle these situations gracefully.

Python

import requests

# Define the proxy URL
proxy_url = "http://your_proxy_url"

# Specify the proxy in the request
proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

try:
    # Make a request using the proxy with a timeout
    response = requests.get("http://example.com", proxies=proxies, timeout=10)

    # Raise an exception for bad status codes
    response.raise_for_status()

    # Print the response
    print(response.text)

except requests.exceptions.RequestException as e:
    print(f"Error: {e}")

In this example:

The timeout parameter sets a maximum time for the request to wait for a response.
raise_for_status() checks for bad status codes and raises an exception if the response is not successful.

Remember to replace "http://your_proxy_url" with the actual URL of your proxy. If the proxy requires authentication, include the username and password in the URL as demonstrated.

Using proxies can be helpful for various purposes, such as:

Anonymizing your web browsing activity
Accessing geo-restricted content
Scraping websites that block requests from certain IP addresses
Bypassing censorship

However, it’s important to note that using proxies can also slow down your requests and introduce potential security risks. It’s essential to choose a reputable proxy provider and ensure that your proxy settings are secure.