Working with the requests library

/ Python / By

The Python requests library is a library used for sending HTTP requests using Python. In this article, I’ll show you how to use the most basic features of requests such as sending GET & POST requests and how to handle responses.

HTTP

Before diving into how to use requests, let’s talk about HTTP. HTTP stands for Hypertext Transfer Protocol, it is the network protocol used to deliver virtually all data and files on the World Wide Web. HTTP is what allows clients such as browsers to communicate with servers. HTTP communication follows a request -> response style. This means that a client(browser or other computer program) sends a request for information to a server and the server receives and processes the request and sends a response back to the client.

There are seven different HTTP requests that clients can make to a server. In this article I will discuss two popular ones, namely GET and POST. GET is the most common HTTP request and as it’s name suggests, it is used to request data from a server. POST is used to send data to a server in order to create or update a resource.

The requests library abstracts much of the complexities of working with HTTP in an easy to use API.

Using requests

To install requests run:

pip install requests

Once you have installed requests, import it into your program:

import requests

The next step is to create requests. An HTTP request is a message sent from the client to the server to retrieve some information. In the next example, you’ll see how to request a webpage using a GET request:

>>> import requests
>>> response = requests.get('http://example.com/')                              
>>> response                                                                 
<Response [200]>
>>> response.status_code
200
>>> print(response.text)                                                        
<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    ...
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>

The code above grabs the content found at http://example.com/. Running .get() returns a response object. The response object contains the server’s response to the request we sent. The response contains useful information such as a status code and the text from the webpage. A status code is a code issued by the server in response to a client’s request for information. There are five categories of status codes. The first digit in the status code determines what class it belongs to:

  • 1xx — Informational response
  • 2xx — Success
  • 3xx — Redirection
  • 4xx — Client Error
  • 5xx — Server Error

In the code example above,you saw how to make and retrieve as simple response from a server. The response returned a 200 status code which means that the request was successful. Another common response status code you have probably come across is 404 which means that a page cannot be found.response.text contains the full text of the web page.

Working with JSON

Some servers provide responses in JSON. JSON is a file format used to interchange data. It is a collection of key/value pairs like a dictionary in Python. I am a fan of Star Wars so, from this point, I’ll use examples from the Star Wars API to illustrate concepts.

To get a list of all the characters in the Star Wars films:

>>> response = requests.get('https://swapi.co/api/people/')

The Star Wars API returns data in JSON format. To retrieve this data call .json():

>>> response.json()                                                             
{'count': 87, 'next': 'https://swapi.co/api/people/?page=2', 'previous': None, 'results': [{'name': 'Luke Skywalker', 'height': '172', 'mass': '77', 'hair_color': 'blond', 'skin_color': 'fair', 'eye_color': 'blue', 'birth_year': '19BBY', 'gender': 'male', 'homeworld': 'https://swapi.co/api/planets/1/', 'films': ['https://swapi.co/api/films/2/', 'https://swapi.co/api/films/6/', 'https://swapi.co/api/films/3/', 'https://swapi.co/api/films/1/', 'https://swapi.co/api/films/7/'], 'species': ['https://swapi.co/api/species/1/'], 'vehicles': ['https://swapi.co/api/vehicles/14/', 'https://swapi.co/api/vehicles/30/'], 'starships': ['https://swapi.co/api/starships/12/', 'https://swapi.co/api/starships/22/'], 'created': '2014-12-09T13:50:51.644000Z', 'edited': '2014-12-20T21:17:56.891000Z', 'url': 'https://swapi.co/api/people/1/'}, {'name': 'C-3PO', 'height': '167', 'mass': '75', 'hair_color': 'n/a', 'skin_color': 'gold', 'eye_color': 'yellow', 'birth_year': '112BBY', 'gender': 'n/a', 'homeworld': 'https://swapi.co/api/planets/1/', 'films': ['https://swapi.co/api/films/2/', 'https://swapi.co/api/films/5/', 'https://swapi.co/api/films/4/', 'https://swapi.co/api/films/6/', 'https://swapi.co/api/films/3/', 'https://swapi.co/api/films/1/'], 'species': ['https://swapi.co/api/species/2/'], 'vehicles': [], 'starships': [], 'created': '2014-12-10T15:10:51.357000Z', 'edited': '2014-12-20T21:17:50.309000Z', 'url': 'https://swapi.co/api/people/2/'}, {'name': 'R2-D2', 'height': '96', 'mass': '32', 'hair_color': 'n/a', 'skin_color': 'white, blue', 'eye_color': 'red', 'birth_year': '33BBY', 'gender': 'n/a', 'homeworld': 'https://swapi.co/api/planets/8/', 'films': ['https://swapi.co/api/films/2/', 'https://swapi.co/api/films/5/', 'https://swapi.co/api/films/4/', 'https://swapi.co/api/films/6/', 'https://swapi.co/api/films/3/', 'https://swapi.co/api/films/1/', 'https://swapi.co/api/films/7/'], 'species': ['https://swapi.co/api/species/2/'], 'vehicles': [], 'starships': [], 'created': '2014-12-10T15:11:50.376000Z', 'edited': '2014-12-20T21:17:50.311000Z', 'url': 'https://swapi.co/api/people/3/'}, {'name': 'Darth Vader', 'height': '202', 'mass': '136', 'hair_color': 'none', 'skin_color': 'white', 'eye_color': 'yellow', 'birth_year': '41.9BBY', 'gender': 'male', 'homeworld': 'https://swapi.co/api/planets/1/', 'films': ['https://swapi.co/api/films/2/', 'https://swapi.co/api/films/6/', 'https://swapi.co/api/films/3/', 'https://swapi.co/api/films/1/'], 'species': ['https://swapi.co/api/species/1/'], 'vehicles': [], 'starships': ['https://swapi.co/api/starships/13/'], 'created': '2014-12-10T15:18:20.704000Z', 'edited': '2014-12-20T21:17:50.313000Z', 'url': 'https://swapi.co/api/people/4/'}, }

requests can do more than just retrieve status codes and message bodies. You can also use it to inspect responses, pass authentication info, handle SSL certificates, and manage sessions. I will not cover these advanced uses in this article.

Query strings

HTTP allows you to pass information to the server in a couple of ways. For sending data that isn’t sensitive, you can use a query string. A query string is the part of the URL where data or information is passed to the web application. For example, take a look at this URL:

https://www.google.com/search?q=trees

The part of the URL in bold is the query string. In this case, we’re telling Google to show us results about trees. Query strings are useful for retrieving specific data or filtering the results you receive from a web application.

The previous request you sent to the Star Wars API retrieved a list of all the Star Wars characters. Let’s filter the results to search for a single person. To do this, you must pass query strings to the API URL:

>>> response = requests.get('https://swapi.co/api/people/', params={"search":'Darth Vader'})
>>> response.status_code
200
>>> response.json()                                                                              
{'count': 1, 'next': None, 'previous': None, 'results': [{'name': 'Darth Vader', 'height': '202', 'mass': '136', 'hair_color': 'none', 'skin_color': 'white', 'eye_color': 'yellow', 'birth_year': '41.9BBY', 'gender': 'male', 'homeworld': 'https://swapi.co/api/planets/1/', 'films': ['https://swapi.co/api/films/2/', 'https://swapi.co/api/films/6/', 'https://swapi.co/api/films/3/', 'https://swapi.co/api/films/1/'], 'species': ['https://swapi.co/api/species/1/'], 'vehicles': [], 'starships': ['https://swapi.co/api/starships/13/'], 'created': '2014-12-10T15:18:20.704000Z', 'edited': '2014-12-20T21:17:50.313000Z', 'url': 'https://swapi.co/api/people/4/'}]}

Now, the response only contains one result.

POST Requests

Passing information via a URL is great for passing little bits of information to a server in order to restrict or limit data in the server’s response. This is not a very secure way of transferring data however. The HTTP specification includes the POST method which is used for creating or updating resources on a server. POST sends data to the server through its message body as opposed to appending the request data to the URL. Whenever you sign up, login or fill out some sort of form in a website, chances are that data is sent to the server via POST.

requests has a post() method for making POST requests. To send a POST request, you call it and pass a payload (information you’re sending to the server) to it’s data parameter. 

>>> payload = {"name": "Luke Skywalker"}
>>> response = requests.post("https://httpbin.org/post", data=payload)
>>> response.status_code
200
>>> print(response.text)
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "name": "Luke Skywalker"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "19", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.20.1"
  }, 
  "json": null, 
  ...
}

This is for sending form data. For this example, I used httpbin.org, a service for testing HTTP requests and responses. I should note here, that the parameter used for passing a payload to .post() depends on the type of content you’re sending. Below are two examples showing how to send different types of payloads. The first one shows how to send JSON data:

>>> url = "https://httpbin.org/post"
>>> payload = {"name": "Luke Skywalker"}

>>> response = requests.post(url, json=payload)

To upload a file using POST:

>>> url = "https://httpbin.org/post"
>>> files = {'file': open('document.xls', 'rb')}

>>> response = requests.post(url, files=files)

Conclusion

You have learned how to install and use the Python requests library. You saw how to use requests to retrieve resources using .get(), how to send query strings in urls and, how to send POST data.