Using the request module in python, you can exchange data on the web as it allows you to send HTTP requests. HTTP serves as a system of data exchange between the client which could be your browser and the server which is the system that hosts the data you need access to.

There are two common python-requests methods that you will come across when it comes to sending a request to a server.

They are:

• GET which is used to request data from a server
• POST which is used to submit data to a server for processing

The python request module has an easy-to-use API that allows you to handle your requests adequately. Features that are at your disposal include adding headers, custom headers, and others.

Interesting Read : Proxy Servers Explained

Before you begin using the requests module in python, you would need to install it. Type the following code in python to install the request module:

$ pip install requests

If you want to use the python packaging tool, Pipenv in installing the requests module, use this code instead:

$ pipenv install requests

After the installation is complete, you can now import it for use in your programs using the following code:

Import requests

GET Request

GET request is one of the most commonly used HTTP methods in the python request module. It is used when you are trying to get data from a web server and the basic syntax used is as follows:

Requests.get(url, params={key: value}, args)

In that format, the URL is that of the target website. The params represent the list of tuples that would be used to send the query string. Args can be any of the named arguments the GET method offers and it can be any of the following:

• Proxies – its default value is none, and it’s the dictionary of the protocol in use for the proxy URL.
• Headers – this is the dictionary that contains HTTP headers that would be sent to the target URL.
• Cookies – This is a dictionary of cookies that are intended for a URL.
• Allow_redirects – this Boolean value is used to allow or prevent redirection.
• Cert – it could either be a string or a tuple for mentioning a key or cert file
• Auth – it’s the tuple that is used to enable an HTTP authentication
• Verify – this string of Boolean values shows the server’s TLS certificate verification.
• Stream – this is a string of Boolean values that may be true or false, but shows if the response should be streamed or downloaded immediately. If it’s to be streamed, it would be True, and if it’s to be downloaded immediately, it's false.

After the GET request has been sent successfully, a request would be returned called the response object. It stores the response to the request as gotten from the server and the result can be stored in a variable. From there you can examine the details of the response and the properties that will be useful are:

• Response.cookies – response.cookies are used to get a cookie jar object with all the cookies that are gotten from the server.
• Response.content – this provides you with the content of the response you got from the server
• Response.status_code – this lets you know the status of your sent request. Getting a response of 200 OK means your request was successful. 400 means the sent request is bad and it comes up when the server doesn’t understand the client. 401 comes up when credentials are required and it stands for ‘Unauthorized’. 403 request is ‘Forbidden’ and it comes up when the server won't fulfill the request by the client even though it understands it. A 404 Not Found response means that your sent request could not locate the resource you are looking for.

POST Request

A python POST request is the second most popular HTTP method that is used. It’s used to create a resource on a target server with specific data. The following is the format to send a POST request:

Response = requests.post(‘https://httpbin.org/post’, data = {‘key’:’value’})

When HTTP methods are used to send requests to the web server, responses come in a format that can be read. For POST requests, they can be read just as with GET.

Print(response.text)

POST requests contain information about the sent request to the server. Sometimes, being specific about the data in the POST request may not be adequate and arguments from dictionary objects are then utilized to send more detailed and advanced information.

Payload = {‘key1’: ‘value1 ‘, ‘key2’: ‘value2’}

Response = requests.post(‘https://httpbin.org/post’ , data = payload)

The new request would then send the payload object to the target server. Sending JSON POST requests can also be needed and requests have a useful feature that converts POST requests into JSON automatically.

Import requests

Payload = {‘key1’: ‘value1’, ‘key2’: ‘value2’}

Response = requests.post(‘https://httpbin.org/post’,json = payload)

Print(response.json())

The JSON library can also be used to convert dictionaries into JSON objects. For this to be done, a new import would need to change the object type:

Import JSON

Import requests

Payload = {

‘key’: ‘value1’,

‘key2’: ‘value2’}

JasonData = json.dumps(payload)

Response = requests.post(‘https:httpbin.org/post’, data = jsonData)

Print(response.json())

If either data or files are used, the JSON argument would be overridden. Only if the three would be accepted in a POST request.

Making A POST Request

It’s an equally easy process to make a POST request just as it is to make a GET request. Just replace the get() function with post().

This will come in handy when you are submitting forms and an example can be seen below as the following code will download the entire Wikipedia page and save it to your device.

1 import requests

2 req = requests.post('https://en.wikipedia.org/w/index.php', data = {'search':'Nanotechnology'})

3 req.raise_for_status()

4 with open('Nanotechnology.html', 'wb') as fd:

5 for chunk in req.iter_content(chunk_size=50000):

6 fd.write(chunk)

Apart from GET and POST HTTP requests, there are other types of requests methods you can use to interact with the servers using HTTP:

• DELETE – this requests method removes all the representations of the target resource provided by the URL.
• HEAD – HEAD is akin to GET and it transfers the status and header part
• PATCH – this request method applies modifications to a specified resource.
• OPTIONS – The OPTIONS method describes the communication options for the target resource
• PUT – this method replaces all current representations of the target resource using the uploaded content
• TRACE – it sends an echo of the requests back to their origin

The HTTP methods that have been listed are not usually used outside of the server administration, debugging, and also web development. It’s not available for use by an average internet user as methods such as DELETE and PUT need permissions which the users won’t have to have performed on any website.

You can also use some HTTP methods for website testing and it’s also outside the interest of a regular internet user.

Inserting Headers and Cookies in A Request

You can gain access to the cookies and headers that the target server sends to you using req.cookies and req.headers. You can also utilize requests to send your cookies and headers in a request.

Pass the HTTP headers in a dict to the headers parameter to add HTTP headers to a request. You can also send your custom cookies to a server with a dict to the cookies parameter.

1 import requests

2

3 url = 'http://some-domain.com/set/cookies/headers'

4

5 headers = {'user-agent': 'your-own-user-agent/0.0.1'}

6 cookies = {'visit-month': 'February'}

7

8 req = requests.get(url, headers=headers, cookies=cookies)

Cookie jars can be used to store cookies and they provide a complete interface that allows the use of cookies over multiple paths.

1 import requests

2

3 jar = requests.cookies.RequestsCookieJar()

4 jar.set('first_cookie', 'first', domain='httpbin.org', path='/cookies')

5 jar.set('second_cookie', 'second', domain='httpbin.org', path='/extra')

6 jar.set('third_cookie', 'third', domain='httpbin.org', path='/cookies')

7

8 url = 'http://httpbin.org/cookies'

9 req = requests.get(url, cookies=jar)

10

11 req.text

12

13 # returns '{ "cookies": { "first_cookie": "first", "third_cookie": "third" }}'

You would need to create a variable to allow you to view the python requests response object sent by the GET request. For easy understanding we would call it ‘response’

Response = requests.get(‘http://httpbin.org/’)

The timeout value for python-requests is set by default and so if a response isn’t gotten, the application will hang.

The status code can now be accessed without the need for the console. You can do this by printing out a specific section:

Print(response.status_code)

The output obtained from this will be the same as that obtained in <response (200)>. The status codes are attached to Boolean values; 200 to 400 is true, above 400 is false.

It’s an advantage to use response codes as Boolean values for several reasons such as to check if the response was successful.

To read the content of the response, you will need to get access to the text part using response.text. You can get the entire response into the python debugger window by printing out the output.

Print(response.text)

It’s important to provide a value as requests automatically guess at the encoding using the HTTP header as a reference. You may need to change the encoding sometimes and it can be done by giving a specific value to the response encoding. This value would then be used whenever a call is made.

You can decode responses to the JSON format. Note that HTTPbin doesn’t send requests that need to be decoded into JSON, so any attempt at doing so will raise an exception.

Response = requests.get(‘http://api.github.com’)

Print(response.json())

With the JSON(), a dictionary object is returned and it can be searched and accessed.

In this guide, we would try downloading an image of a forest from Pixabay using request modules.

To download the image, you will need the following code:

1 import requests

2 req = requests.get('path/to/forest.jpg', stream=True)

3 req.raise_for_status()

4 with open('Forest.jpg', 'wb') as fd:

5 for chunk in req.iter_content(chunk_size=50000):

6 print('Received a Chunk')

7 fd.write(chunk)

The ‘path/to/forest.jpg is the image URL and you can input any URL here to start the download process for that image. This is just used as a sample guide and since the size of the image is 185kb, you need to set chunk_size to 50,000 bytes.

The use of requests also allows you to pass parameters in a URL. It can come in handy when you are searching for results on a web page such as a specific image. Provide the query strings as a dictionary of strings using the params keyword in the GET request.

1 import requests

2

3 query = {'q': 'Forest', 'order': 'popular', 'min_width': '800', 'min_height': '600'}

4 req = requests.get('https://pixabay.com/en/photos/', params=query)

5

6 req.url

7 # returns 'https://pixabay.com/en/photos/?order=popular_height=600&q=Forest&min_width=800'

Response headers form an important part of python-requests and even though they don’t contain any part of the original request, they have some important data about the response like server information, encoding, date, etc. you can get every detail from the initial response:

Print(response.headers)

Just as with the .json()call, headers also create an accessible dictionary type object. Including parameters to the call will list out some part of the response:

Print(response.headers[‘date’])

This function will now print the date that is stored in the response header.

Custom python request headers can also be sent and here, dictionary type objects are used too. To check that the request header is sent successfully, the response.request.headers call needs to be made:

Import requests

Headers = {‘user-agent’: ‘my-agent/1.0.1’}

Response = requests.get(‘http://httpbin.org/’, headers=headers)

Print(response.request.headers)