Scrape Google Maps To Extract Businesses Using Nightmare In Node JS

Get title, address and rating of each business

 November 14, 2019  (Updated November 18, 2019)  |  6 minutes  |  1138 words  |  Talha Awan

Google Maps is a rich source of information when it comes to finding businesses or places worldwide. Most of the data contained in the maps is provided by Google through their Places API, which is the official and recommended way to fetch any kind of data off Google. [Read More]
nodejs  google-maps  scraping 

How to Run Nightmare JS On Remote Linux Server

 November 13, 2019  |  1 minutes  |  132 words  |  Talha Awan

Nightmare js is a browser automation tool. It works perfectly on local machine and even has an option to view the browser in action by passing { show: true } option when initializing Nightmare. On remote linux machine, however, it won’t work without installing and including an extra package xvfb. [Read More]
nodejs  nightmarejs 

How to Deal With Nightmare JS Zombie Electron Processes

Nightmare JS leaves in its wake zombie electron processes and memory leaks

 November 12, 2019  |  3 minutes  |  545 words  |  Talha Awan

Nightmare JS is a great browser automation library, especially effective for scrapping websites. It has two faults though: memory leaks and zombie electron processes. But with a duct-tape hacks we can easily overcome both. For memory leak, which is minor but builds up over time, daily restart of the node script or app is pretty effective and avoids solutions that are overkill. [Read More]
nodejs  nightmarejs 

Restart Pm2 Process Every Day Using A Node Script

A brute force yet effective way to restart scripts that need it

 November 12, 2019  |  3 minutes  |  618 words  |  Talha Awan

To run node js scripts and apps constantly we usually use pm2 (some prefer forever). However, some scripts require a brute force restart from time to time as they don’t fare better in uninterrupted longer runs. Mostly, the reason is memory leaks of packages you’re bound to use but have no control over. [Read More]
nodejs  pm2 

Trace Redirect Urls NPM Request

 November 12, 2019  |  2 minutes  |  227 words  |  Talha Awan

With NPM request module it’s quite easy to observe and trace all the intermediate urls before the final url is reached. We can do that using followRedirect option, which takes either true or false, or a function that should return a boolean value. [Read More]
nodejs  request 

Build Queue For Processing Webhooks With Node Js And MongoDB

Manage The Flood Of Webhooks With A Database Queue

 April 24, 2019  |  4 minutes  |  815 words  |  Talha Awan

Many of the third party services we integrate in our applications use webhooks to communicate events to us. Responding to the incoming webhook with success response (status 200) is all that’s required to acknowledge the receipt of the webhook. To process that webhook immediately upon receiving or entering it in a queue to be processed later is up to you. [Read More]
nodejs  mongodb 

Intercept Node Js Console Log And Insert In Mongodb

Easily Manageable And Searchable Application Logs Stored In Database

 April 23, 2019  |  4 minutes  |  821 words  |  Talha Awan

Your node js application is running on production, leaving hard-to-search ever-increasing log files in its wake. You want all your logs in a centralized, easily searchable place. For that, you have two options: 1 - Buy a subscription of some cloud log management service, such as papertrail. [Read More]
nodejs  mongodb 

Scrape Website Data Without Headless Browser, Using Node Js

Not Every Website Scraping Needs Headless Browsers; Request And Cheerio Js Do Just Fine

 April 22, 2019  |  4 minutes  |  734 words  |  Talha Awan

During the course of web development, we occasionally need to scrape a website. Not just one page but many. On such requirement, mostly because the host site doesn’t allow otherwise, we naturally tend to favor the use of headless browsers such as phantomjs/casperjs. [Read More]
nodejs  scraping  request  cheerio