In this web scraping tutorial, you will need two pieces of software:

1. Any code editor you are comfortable with
2. Node.js which comes with the Node.js Package Manager

You need to know at this point that Node.js is a runtime framework. This means that JavaScript code doesn’t necessarily need a browser to run. Node.js is available for download for Windows, Mac OS, and Linux and you can download here.

Interesting Read : What You Need To Know Now About Encryption

Setting It Up

Before you begin writing your code to scrape using Node.js, create a folder to store JavaScript files. These are the files where all the code you require for web scraping would be stored. After creating the folder, navigate to it and run the initialization command:

npm init –y

Doing this will create a package.json file in the directory that will contain all information about the packages that have been installed in the folder. The next thing to do is the installation of the Node.js Packages.

The Available Packages

To carry out web scraping using Node.js, certain packages referred to as libraries have to be used. These libraries are prepackaged codes that can be used more than once. You can download and reinstall them using the npm install command (the Node.js Package Manager).

To install a package, run npm install . for instance, to install Axios, run the following on your terminal:

npm install Axios

this code also supports the installation of more than one package. To install all the packages that would be used in this tutorial, run the following command:

npm install Axios cheerio json2csv

running this command will download the packages in the node_modules directory, thereby updating the package.json file.

Cheerio

Cheerio is an efficient library that gives you the power of JQUERY API on the server side. It takes out all the inconsistencies of DOM, and removes all browser related features as well so that you are left with an efficient API.

Cheerio is different from a web browser as it doesn’t execute JavaScript, render any parsed or manipulated elements, or loads any external resource. So if your target website is JavaScript heavy, then you should consider using another library.

JSDOM

JSDOM is a web scraping library that parsed and interacts with parsed HTML just as a browser would do. Since it acts like a browser, it allows you to interact with your target web page on a programming level so that you can perform functions like button clicks.

Puppeteer

Just as the name suggests, the puppeteer library allows you to manipulate your browser on a programming level. This is possible by providing you with an advanced API that allows you to take automatic control of a Headless version of chrome browser, and you can also tweak it to be non-headless.

Puppeteer is so flexible it allows you to have your go at some features like taking a screenshot of a web page. This comes in handy for example when you want to have a copy of your competitors’ products pages.

Puppeteer is preferred to other tools because of its interaction with the browser. It crawls the web page in a human-like manner making it even more difficult for your bot to be detected and blocked.

Other features and possibilities that puppeteer brings are an automated user interaction with the browser such as form submissions and page navigations, web page PDF generation, and web page crawling to generate prerendered content.

You can install puppeteer by running the following command: npm install puppeteer

Nightmare

Nightmare is a great alternative to puppeteer if for some reason you need to use another automation library other than puppeteer. You can start nightmare by the following command: npm install nightmare

Regular Expressions

Using Regular Expressions makes it simple to begin web scraping without much dependencies. This simplicity comes with a big con; loss of flexibility as users find it difficult to write the correct expressions most times.

A very common example of JavaScript web scraping is to scrape eCommerce stores, you can start with the fictional book store: http://books.toscrape.com. It’s similar to the real eCommerce store, but it’s made for web scraping learning.

Create The Selectors

Creating selectors is the first step to take before you begin JavaScript web scraping. This is to identify the specific elements that would be queried.

To begin, open the URL using Chrome or Firefox: http://books.toscrape.com/catalogue/category/books/mystery_3index.html

Once the page opens, right-click on the title of the relevant genre ‘Mystery’, and then select inspect. Doing this should open the Developer Tools with

mystery

You can simply create the selector by right-clicking the h1 tag in the Developer Tools, go to copy, and click on copy selector. It will lead to something like this:

#content_inner > article > div.row > div.col-sm-6.product_main > h1

This is a valid selector and would work well, but the pitfall of this method is the long selector it creates. So it can be difficult to understand the code and maintain it.

After staying on the page a while longer, you will notice that there is only one h1 tag on the page. By this, you can easily create a very short selector unlike the previous:

h1

you can use another option like the Selector Gadget extension for chrome to quickly create selectors. It’s a useful tool to scrape the web using JavaScript.

Note that even though this works most times, there are some exceptions where it will fail.

Scrape The Genre Of Interest

In scraping the genre, the first thing is to define the constants that will be referenced to Axios and Cheerio.

Const cheerio = require(“cheerio”);

Const Axios = require(“Axios”);

Save the address of the page to be scraped in the variable URL for readability

Const url = “http://books.toscrape.com/catalogue/category/books/mystery_3index.html”

Axios has a method get() that sends the HTTP GET request and it’s an asynchronous method, thus will need to await prefix:

Const response = await Axios.get (url) ;

If you need to pass more headers like User-Agent, this can be sent as the second parameter:

const response = await axios.get(url, {

\ headers:

\ {

\ "User-Agent": "custom-user-agent string",

\ }

\ });

This site doesn’t need any unique header, making learning easier.

Axios is compatible with both the Promise pattern and the async-await pattern. In this tutorial, we will focus on the async-await pattern. Its response has some characteristics like headers, data, and so on. The HTML we need is in the data attribute and can be loaded into an object to be queried using cheerio.load() method.

Const $ = cheerio.load(response.data) ;

The cheerio’s load method returns a reference to the document to be stored as a constant under any name. To enhance the look of our web scraping code so it can be more like jQuery, a $ can be used rather than a name.

This specific element can be found easily within the document just by writing $(“”) and in this case, it would be $(“h1”).

The method text() would be used generally when writing the scraping code with java as it can get the text inside any required element. This can be extracted and saved in a local variable:

Const genre = $(“h1”) . text() ;

Lastly, the console.log() will print the value of the variable on the console

Console.log(genre) ;

For the avoidance of errors, the code will have a try-catch block around it. Remember that it's good practice to use console.error for any errors, and console.log for other messages.

Below is the complete code put together. Save the code as genre.js in the folder you created at the beginning, where you ran the command npm init.

const cheerio = require("cheerio");

const axios = require("axios");

const url = "http://books.toscrape.com/catalogue/category/books/mystery_3/index.html";

async function getGenre() {

try {

\ const response = await axios.get(url);

\ const document = cheerio.load(response.data);

\ const genre = document("h1").text();

\ console.log(genre);

} catch (error) {

\ console.error(error);

}

}

getGenre();

Finally, you will need to run this JavaScript web scraping using Node.js. open the terminal and run the following command:

Node genre.js

The code output will be the genre name:

Mystery

Scrape Target Book Listings

Let’s try scraping book listings from the web page of the mystery genre:

http://books.toscrape.com/catalogue/category/books/mystery_3/index.html

the first thing to do is to analyze the page and understand the HTML structure that is used. Open the page using a chrome browser and tap on F2 to examine the elements.

Each book is in

    title = $(this).find("h3 a").text(); //extracting book title

    console.log(title);//print the book title

        });

Axios

Axios is a client that runs both in the browser and NodeJS. It comes with promise support and is straightforward when making HTTP requests.

Download Axios on GitHub and install using the following command; npm install axios

Request

The Request client is one of the most popular HTTP clients due to its features and simplicity. It’s available at GitHub and can be installed using the following command: npm install request

Superagent

This is another HTTP client that assists web scraping. It supports promises and also a sync/await syntax sugar and even though it’s fairly straightforward, it’s not as proper as the others.

It’s available on GitHub and you can install it with the following command: npm install superagent

Interesting Read : The Internet of Things – Technology of the future

Most JavaScript web scraping or Node.js web scraping would be made of three basic steps:

• The HTTP request would be sent
• The HTTP response would be parsed and data extracted
• The resulting data would be saved in a persistent storage such as a database

We will now see how Axios can be used to send the HTTP request, Cheerio to parse the response and for data extraction, and then saving the extracted data to CSV with json2csv.

Send The HTTP Request

JavaScript web scraping begins with finding the package that can send the HTTP request and also return the response. The request and response promise of a package used to be popular, but is now being discredited.

Various examples and old codes use these packages, and Axios is a good alternative. It’s compatible with promise syntax and async-await syntax.

Parse The HTTP Response with Cheerio

You also get the Cheerio package from Node.js. Its conversion of raw HTML data format as captured by Axios into something that can be worked with using jQuery-like syntax is what makes it valuable.

Since most JavaScript developers are familiar with jQuery, it makes Cheerio a very good choice in data extraction from HTML.