How to Run Nightmare JS On Remote Linux Server

Nightmare JS is a browser automation tool. It works perfectly on local machine and even has an option to view the browser in action by passing { show: true } option when initializing Nightmare. On remote linux machine, however, it will not work without installing and including an extra package xvfb. [Read More]

How to Deal With Nightmare JS Zombie Electron Processes

Nightmare JS leaves in its wake zombie electron processes and memory leaks

Nightmare JS is a great browser automation library, especially effective for scrapping websites. It has two faults though: memory leaks and zombie electron processes. But with a duct-tape hacks we can easily overcome both. For memory leak, which is minor but builds up over time, daily restart of the node script or app is pretty effective and avoids solutions that are overkill. [Read More]

Restart Pm2 Process Every Day Using A Node Script

A brute force yet effective way to restart scripts that need it

To run NodeJS scripts and apps constantly, we usually use pm2 (some prefer forever). However, some scripts require a brute force restart from time to time as they do not fare better in uninterrupted longer runs. Mostly, the reason is memory leaks of packages you are bound to use but have no control over. [Read More]
nodejs  pm2  script 

Trace Redirect Urls NPM Request

With NPM request module it is quite easy to observe and trace all the intermediate urls before the final url is reached. We can do that using followRedirect option, which takes either true or false, or a function that should return a boolean value. [Read More]

Clean Up MongoDB Of Old Inactive Users And Their Data With Node Js Script

Reduce Expensive Database Space By Pruning Never-To-Be-Used-Again Data

Databases are not cheap, especially when your application is just taking off and you are low on budget. It is then that freeing up retrievable and precious space from the database is a good way to save some dollars for a while. [Read More]

Build Queue For Processing Webhooks With Node Js And MongoDB

Manage The Flood Of Webhooks With A Database Queue

Many of the third party services we integrate in our applications use webhooks to communicate events to us. Responding to the incoming webhook with success response (status 200) is all that is required to acknowledge the receipt of the webhook. To process that webhook immediately upon receiving or entering it in a queue to be processed later is up to you.

[Read More]

Intercept Node Js Console Log And Insert In Mongodb

Easily Manageable And Searchable Application Logs Stored In Database

Your node js application is running on production, leaving hard-to-search ever-increasing log files in its wake. You want all your logs in a centralized, easily searchable place. For that, you have two options: 1 - Buy a subscription of some cloud log management service, such as papertrail. [Read More]

Scrape Website Data Without Headless Browser, Using Node Js

Not Every Website Scraping Needs Headless Browsers; Request And Cheerio Js Do Just Fine

During the course of web development, we occasionally need to scrape a website. Not just one page but many. On such requirement, mostly because the host site does not allow otherwise, we naturally tend to favor the use of headless browsers such as phantomjs or casperjs. [Read More]

Confirm If A Chargebee Customer Has Already Purchased An AddOn

A workaround to overcome Chargebee's less-than-ideal querying options and non-existing email uniqueness constraint

I got to work with Chargebee integration recently, and my overall experience was good. However, I felt two obvious things were missing. First, email is not unique in Chargebee. You can add multiple customers with same email. Though it could be a well thought out decision on Chargebee’s part, I can’t imagine an application where you can have as many users as you like against a single email. [Read More]

Export MongoDB Collection Data In CSV From Node Express Mongoose Application

Suppose a student collection in mongoDB have some data that we want to export in a CSV file from a Mongoose, Node, and Express app. For simplicity, let us assume only two fields in Mongoose schema: lastName and firstName. To export a CSV file, install moment, json2csv, and mongo-date-query (optional) npm packages: [Read More]