TecHighness

Clean Up MongoDB Of Old Inactive Users And Their Data With Node Js Script

Reduce Expensive Database Space By Pruning Never-To-Be-Used-Again Data

Talha Awan | April 25, 2019 | 3 minutes (499 words)

Databases are not cheap, especially when your application is just taking off and you are low on budget. It is then that freeing up retrievable and precious space from the database is a good way to save some bucks for a while. For that, you have to set up a criteria based on which pruning to happen. Pruning Criteria The focal point of majority apps is user, and most of the other data in the database is usually related to it. [Read More]

Build Queue For Processing Webhooks With Node Js And MongoDB

Manage The Flood Of Webhooks With A Database Queue

Talha Awan | April 24, 2019 | 4 minutes (815 words)

Many of the third party services we integrate in our applications use webhooks to communicate events to us. Responding to the incoming webhook with success response (status 200) is all that’s required to acknowledge the receipt of the webhook. To process that webhook immediately upon receiving or entering it in a queue to be processed later is up to you. In this post I am sharing a way to build and process a mongodb based webhook queue for node/express applications. [Read More]

Intercept Node Js Console Log And Insert In Mongodb

Easily Manageable And Searchable Application Logs Stored In Database

Talha Awan | April 23, 2019 | 4 minutes (818 words)

Your node js application is running on production, leaving hard-to-search ever-increasing log files in its wake. You want all your logs in a centralized, easily searchable place. For that, you have two options: 1 - Buy a subscription of some cloud log management service, such as papertrail. 2 - Spin up your own log database and dump all your application logs in it. Option 1 is the way to go if you want all the features it offers, and most importantly you can afford the price tag. [Read More]

Scrape Website Data Without Headless Browser, Using Node Js

Not Every Website Scraping Needs Headless Browsers; Request And Cheerio Js Do Just Fine

Talha Awan | April 22, 2019 | 4 minutes (734 words)

During the course of web development, we occasionally need to scrape a website. Not just one page but many. On such requirement, mostly because the host site doesn’t allow otherwise, we naturally tend to favor the use of headless browsers such as phantomjs/casperjs. Apparently the choice is obvious, but is it? Do we really need a headless browser to scrape all websites? How To Confirm The Website Needs Headless Browser Or Not? [Read More]

With Update Method, Toggle Boolean Field In All Documents Of A MongoDB Collection

Talha Awan | October 27, 2018 | 4 minutes (706 words)

Recently I got a situation in which I had to update all user documents in mongoDB to toggle a boolean field. I was looking for single update query that could take care of it, but being a rare need, the problem had not many helpful solutions. I finally resorted to old-school brute force technique of using “temporary” field. This didn’t result in a single query though, but three queries, which was still a quick and acceptable solution for my one time need. [Read More]

Confirm If A Chargebee Customer Has Already Purchased An AddOn

A workaround to overcome Chargebee's less-than-ideal querying options and non-existing email uniqueness constraint

Talha Awan | September 3, 2018 | 2 minutes (324 words)

I got to work with Chargebee integration recently, and my overall experience was good. However, I felt two obvious things were missing. First, email is not unique in Chargebee. You can add multiple customers with same email. Though it could be a well thought out decision on Chargebee’s part, I can’t imagine an application where you can have as many users as you like against a single email. Just to handle this unnatural scenario, the developer has to add an extra field in database user to keep Chargebee’s “unique” id, which could easily have been managed with already existing email. [Read More]

Export Mongodb Collection Data In CSV From Node Express Mongoose Application

Talha Awan | May 26, 2018 (Updated August 25, 2018) | 2 minutes (360 words)

Suppose a student collection in mongodb have some data that we want to export in a csv file from a Mongoose and Node/Express app. For simplicity, let’s assume only two fields in Mongoose schema: lastName and firstName. To export a CSV file, install moment, json2csv, and mongo-date-query (optional) npm packages: npm install moment json2csv mongo-date-query --save And the required code is as follows: 'use strict'; const fs = require('fs'); const moment = require('moment'); const mdq = require('mongo-date-query'); const json2csv = require('json2csv'). [Read More]

Validate Uploaded CSV Data In Node Express

Confirm existence and data type of all columns within each row

Talha Awan | May 25, 2018 | 3 minutes (554 words)

Continuing from earlier post Node Express Endpoint to Upload and Process CSV File, in this post we will add validation to the uploaded CSV data. With the same index.js file and settings as the earlier post, we only need to add moment library for date validation. So install and save it: npm install moment --save. The file should look like this: 'use strict'; const http = require('http'); const fs = require('fs'); const express = require('express'); const multer = require('multer'); const csv = require('fast-csv'); const moment = require('moment'); const Router = express. [Read More]

Integrate Polling Based API In Node/Express Js

Talha Awan | May 18, 2018 | 5 minutes (1041 words)

At times we need to integrate in our Node/Express application third party APIs that have implemented polling mechanism i.e. they only send back results once available; the client must keep on attempting until the service has put together all the results to return. Usually data providers of hotel, travel, tourism, property have polling mechanism in place for their APIs. In Node js application it’s fairly easy to implement. There might be multiple ways but here I’ll be using async library and its whilst control flow method. [Read More]

Nosql Many-to-Many Relationship Examples; Bound And Unbound Cases

Right Approach To Modeling Many To Many Relations And Use Of Associative Collection

Talha Awan | March 24, 2018 | 3 minutes (513 words)

In NoSql data modeling I analyzed NoSql data modeling and ways to model many to many relationships. In another post I added a few examples of 1-to-many bound and unbound cases. Presently, I’ll illustrate bound and unbound cases for many-to-many. Plus I’ll identify an instance where use of associative collection becomes necessary. Many To Many Bound Let’s take a practical example of Cab is assigned to many Drivers and Driver is assigned to many Cabs. [Read More]