Introduction

It’s 2024 and we have a lot of help in programming from AI. Despite that, building a full saas application (even for MVP scale), it is a lot of efforts. Surely, AI will reduce this work, but putting together all the pieces to make a functioning app, is still where you will need experts. In this post, I try to uncover all the things that as an engineer you need to build a web application in 2024.

Depending on your MVP requirements, you can either remove of some these tech from your overall tech stack.

Frontend and Backend

Let’s start with the backbone of the application. Frontend and Backend. Recent progress in frontend frameworks have challenged the usual separation of frontend and backend.

Framework like Remix can be used to build a full stack web application. But of course, you can choose these frameworks based on what you are good at.

There are number of backend frameworks available.

• Python has Django OR Flask.
• Java has Spring Boot.
• Typescript has NestJS or Hapi.

I built Xpenses using NextJS for frontend and NestJS for backend.

A simple tip here is choose the framework that you are comfortable with. This allows the focus is on delivering business value.

Do not get caught up in what framework to use. Once you keep using one techstack, it makes easier and faster to build apps. You can always learn to use different recipes in that techstack.

Load Balancer

A load balancer manages load to the application. Usually request comes to frontend and then frontend routes that traffic to backend through a load balancer. Load balancer acts as a traffic controller.

Load balancer also improves reliability of the application. Most load balancers are prepared to handle spike in traffic. So, they play a key role in a web application.

There are number of options available from cloud providers to free ones. At times, you can also question if you need a load balancer for your MVP application.

But it is always worth to prepare.

• AWS ALB – Application load balancer. If you are using AWS Cloud infrastructure for your backend application, then it makes sense to use AWS ALB.
• NGINX – Free reverse proxy load balancer. It’s lightweight and fast and can be easily configured. The most suitable for MVP application.
• Cloudflare – Cloudflare is easy to set up with DDoS attack protection option. If your application starts to see traffic, this is a great option since it offers security protection as well global traffic CDN network.

Authentication and Authorization

Even when building a MVP app, you will need authentication mechanism. As an engineer, you should take into account basic security measures when building a MVP app.

There are number of options available for login.

1. Single Sign On with Gmail
2. Auth0 has free plan for SSO. You can always leverage SAML or OAuth protocols.
3. Depending on your application, you can use Github OR LinkedIn as signin options.
4. Username and Password.
5. Some applications offer Magic Link, but I am not a big fan of magic link as it can be easily abused.

On the other hand, there should also be safeguards using permissions. And authorization plays a part. Your backend resources should be protected. A client calling backend APIs need to prove who they are. Most frontend calls backend APIs with expiring Json Web Token (JWT).

CI/CD

Whether you want CI/CD for your MVP or not, is up-to-you. You might not even be writing tests, but if you are then it makes sense to have continuous integration (CI). But you will need continuous deployment (CD) to make it easier to deploy changes as soon as you merge your pull request (PR).

The easiest way to build continuous deployment is to use Github actions. When building xpenses, I used Github action and it made building an app and deploying to remote server easier.

Redis

Using redis will depend on if you need cache OR any redis-backed queuing mechanism. I frequently use Bull Queues for NestJS applications.

Redis is also a good option for rate-limiting mechanism.

There are multiple options available where you set up Redis. Railway platform offers standalone app for redis that you can deploy. Fly.io has an option to set up redis when you are setting up server. You can create a digital ocean droplet and you can set up an redis on that server as well.

I have used Digital Ocean droplet for redis for smaller projects. Depending on where you are running your main app, you will need to figure out the networking part between your application connecting to Redis.

Email

Email usage can be contextual for the application. But one way or other, you will have to use email. Email is the standard way to create user account in the application. Whether you use email as username for login will depend on if you use username/password crendetial flow. It’s not the most secure flow, but allows anyone to create account.

Other scenarios where you might need email

• User confirmation
• User notification for various activities within the app
• Sales OR Support questions

Here are few options that I know for setting up email.

• Resend
• Mailtrap
• Mailgun

Handling Bots and Spams

The easiest way to tackle bots is to make a paid app. Once you wall to pay for the app to use it, then free users or bots will be less of a worry. That does not mean you completely block them, but no random user is trying to access your app.

Cloudflare offers an option of bot management.

As far as Spams are concerns, you can embed Recaptcha OR Cloudflare’s turnstile in your public facing forms.

Database

I like relational databases and of course, that comes with my knowledge of SQL. But if you have used NoSQL databases like Mongo, it can also be handy to build a complete web app. Depending on what database you are using, you have multiple options to set up and many of these options have free tier OR free plan. As you start to see the traction for your application, you will need to switch to paid plan.

What database (Relational or non-Relational) to use will depend on what kind of application you are building. In most cases, one type of database will be enough. Postgres supports JSON like objects as well so it can do the same role as Mongo database.

If you need strong consistency for your data, then relational database makes sense. Databases like MySQL and Postgres have been there for more than few decades now, so they are well tested and documented.

Storage Configuration

Some point you will need to store some files or even some data in files. Of course, there can be applications without file storage.  If you need file storage, there are a few good options for storage.

• AWS S3 is the most famous one with a neat SDK to use in your code. SDK have well abstracted methods to create/write/read/delete files.
• Google Cloud Storage is also popular choice as it provides resumable upload capabilities.
• Supabase along with database, provides a storage option. It is mostly wrapper over AWS S3.

Pricing and Payments

One of the challenges of building saas application is to figure out the right pricing plan. And even then, there are multiple payment providers to work with. The most popular one is Stripe. Stripe provides its SDK to use for both frontend and backend. Based on your techstack, you will have to use their SDK library in your code.

Whether you are charging one-time payment OR recurring payment, you have an option to use Stripe, Lemon Sequeezy OR Paddle.

Servers

Of course building an app does not come without the infrastructure. The infrastructure for servers. Servers will host your app (for both frontend and backend). I haven’t mostly talked about domain configuration in this post, but many of these services need some tinkering with domain name servers to work correctly. It can be overwhelming if you look at number of services we are using.

In a traditional sense, it would have been easier if you hosted everything on a single or fleet of servers yourself instead of using all these services. You would have needed a server for app, server for database, server for email and then figure out the communication between these servers. One way or other it is challenging.

There are few out of the box products available that make server deploying easier.

• Railway – I really like their product as it comes some of the in-built marketplace to re-use existing infrastructure pieces.
• Digital Ocean – They have one of the cheapest option with droplet for as minimum as $4 per month.
• Fly.io
• Render

Conclusion

Building a Saas in 2024 might not be straight forward, but considering how many services and products we have, it has definitely become cheaper to host. If you are an indie-hacker or bootstrapper who wants to build your own app, this is the best time to build. With AI tools on the rise, building will only get easier. The challenge still remains for distribution.

Introduction

It’s naïve to think of software as a static system. Static systems are those that do not change in response to external factors. However, I argue that truly static systems are exceedingly rare. Consider furniture, books, or other objects in your home—they might seem static at first glance. But one factor changes them all: time.

Time is the universal force that alters everything. Imagine a piece of furniture crafted today. How will it look 100 years from now? Or 200 years? We see many antiquated buildings that have endured for decades, even centuries, but they are not immune to change. Over time, cracks may form in the walls, and pipes installed a century ago may deteriorate. Time is a relentless and influential external factor, and no system, no matter how static it seems, is exempt.

The same applies to software. Whether you make code changes or not, the passage of time affects software. Systems degrade, dependencies become outdated, and performance may decline. Time alone transforms even seemingly unchanging software into a dynamic system.

Dynamic systems are inherently subject to change. They evolve in response to external variables, internal behaviors, and time.

Given that systems are rarely static, how should we approach thinking about them?

To understand systems effectively, you need to consider both the big picture and the small picture—often referred to as the macro view and the micro view.

Developing a mindset for system thinking takes experience, and often, the best way to learn is by building a system yourself. Every system exhibits certain properties, and the manifestation of these properties determines how the system will behave.

System Thinking in Software

Usually, when it comes to software, there can be two types of feedback loops. One that is automatic and other that is manual. Automatic is mostly when the system adapts based on the scenarios it is in and how it has acted based on the input. With the advancement of technology, we have seen a self-healing systems and that is mostly automatic feedback loop. The example that comes to mind is kubernetes auto scaling feature.

Kubernetes auto scales up the pod based on the load it sees on the system and scales down when the load on the system goes down.

On the other hand, software serves a customer and customer provides a feedback about what the software is doing and what not. Based on that feedback, an engineer can tweak the system to behave differently for the customer. That’s manual feedback.

System thinking often involves observing events or data to identify patterns of behavior over time. This will surface the underlying structure that triggers those events.

Conclusion

In this post, I covered some thoughts about system thinking. System thinking in software is even more critical when building a distributed system. An experienced engineer will always look at macro view while making decisions for micro.

Introduction

Streams in NodeJS can be one of the best features as well as the most misunderstood features at the same time. And a lot of time, the confusion stems from number of options that are available within npm ecosystem. Streaming is a general data handling technique allows to process the data sequentially at a controlled speed without overwhelming the memory or CPU.

When processing a large set of data, especially from files, it could be challenging to read all the data in memory and process it. This can create high memory usage as well CPU usage. In turn, it can cause backend service to fail.

Streaming in NodeJS

Streaming is an old concept. It has been popularized when we started building a lot of data-intensive applications. The most popular being Netflix or Youtube. Idea of stream is to take small set of data (a character or a byte) and process it and continue the process till the we have completely read all the data.

There are mainly two types of streams – readable and writable. There are also duplex that does both reading and writing.

• Readable stream (Input Stream) is where you read the data from.
• Writeable stream (Output Stream) is where you write the data into.

Files, Database, Console can be considered for readable stream while they can be considered for writable stream as well. Readable stream can be combined with writable stream to make processing easier. This is also considered as piping. Piping has been there from the time of unix invention. If you have used pipe in unix where you can combine more than one commands, theoretically, it is the same concept when combining two streams.

Transform

NodeJS offers steams features with a number of powerful concepts. And one of them is transform. We just not only transfer the data between readable and writeable streams, but we can also do transformation on this data as it becomes available through readable stream.

const { Transform } = require('stream');

Piping

As previously said, piping comes from Unix. But within NodeJS stream, we can also use pipe to combine multiple streams. This allows data to flow from one stream to another as it gets processed.
In data intensive applications, we will come across scenarios where we will have to perform various operations on data at different stages. In such scenarios, piping allows to combine transform and pass data between transform.

Here is an example of how piping can be implemented

const { pipeline } = require('stream');
const { promisify } = require('util');
const pipelineAsync = promisify(pipeline);

await pipelineAsync(
        buildSellObjectTransform,
        splitUserDataTransform,
        enqueueDataTransform
      );

Backpressure

So far, I have mentioned that how strong stream as a feature is from NodeJS. But it comes with its own set of concerns/issues. Processing large set of data can still pose challenges with stream.

At the end, it depends on how fast the stream is processing and how fast output is handling this data.

Look at example from above where I have a pipeline. And I will explain why I used pipelineAsync instead of just pipeline.

When there is a constraint on resources, we want to make sure that we don’t overwhelm the downstream services. Input stream will keep sending data till it reaches the end of it, but transform OR other downstream services that are handling that data needs to match the same speed as input stream.

Streams are fire and forget . Once they start sending data, they don’t care on how other services are handling that data. This creates an issue.

This is what NodeJS documentation describes –
There is a general problem that occurs during data handling called backpressure and describes a buildup of data behind a buffer during data transfer. When the receiving end of the transfer has complex operations, or is slower for whatever reason, there is a tendency for data from the incoming source to accumulate, like a clog.

If you look at below picture, you can see a faucet releasing the water at force and if we don’t apply backpressure on the faucet, water can overflow. Backpressure is the same idea in nodejs stream.

By applying some backpressure, stream only release certain size of data and wait for receiving service to process it before releasing more data. There are various ways to solve the problem for backpressure.

• Piping is one of the solutions to handle backpressure. If you use pipe() from nodejs, this can handle backpressure. Sometimes, you have to explicitly set the value for highWaterMark on what data size to handle.
• pipelineAsync/pipeline handles backpressure automatically without having to set highWaterMark.

Error Handling

We have talked about reading data, handling data. But what happens to streams or pipeline if there is a corrupt data or some data processing function failed either through system error or custom error.
In cases when there is an error in transform stream, you can use callback(error) to catch the error in your calling function. And on error, you can either destroy or drain the stream so the objects get cleaned up and the stream doesn’t end up occupying memory waiting for garbage collection.

async _transform(chunk, encoding, callback) {
    try {
      const sellItem = this.buildSellObject(chunk, this.customFields);
      if (this.push(sellItem)) {
        callback();
      } else {
        this.once('drain', callback);
      }
    } catch (error) {
      this.logger.error(error);
      callback(error);
    }
  }

And the calling function

    try {
      await pipelineAsync(
        buildSellObjectTransform,
        splitUserDataTransform,
        enqueueDataTransform
      );
    } catch (error) {
      this.logger.error(`${logPrefix} Error in processing file data: ${error}`);
      throw error;
    } finally {
      this.logger.info(`${logPrefix} ends file processing`);
      fileStream.destroy();
    }

Conclusion

In this post, I shared the details on how powerful the feature of streams from nodejs is. With transform and piping, the stream can be used in various use cases of large data processing.

References

• Backpressure in Streams

Introduction

This is a simple idea and I want to show case the scaling power of bull queue. One thing I really like about bull jobs is that they are performant and easy to scale. The jobs also give an option to do asynchronous processing. Keep in mind that if you have a CPU intensive task, you can still use Bull queue. I have covered how to use Bull Queue for asynchronous processing.

As part of this post, we will upload a large file. The controller stores the file on disk, but adds a job to bull queue. The worker for this job will then read the file data and add each record to another bull queue.

We will have multiple workers to process file data. Each worker will process a separate job that is there in the bull queue.

Adding a single job

Let’s start with a single job first. I will not be covering any fundamentals about Bull Queues and workers. I previously wrote about worker pattern.

Nevertheless, we have a NestJS application with an API to upload a file. This API will create the first job in the queue file-upload-queue as follows:

  @Post('/uploadFile')
  @UseInterceptors(FileInterceptor("csv", {
    storage: diskStorage({
      destination: './csv',
      fileName: (req, file, cb) => {
        const randomName = Array(32).fill(null).map(() => (Math.round(Math.random() * cb(null, `${randomName}${extname(file.originalname)}`))))
      }
    })
  }))
  async uploadLargeCsvFile(@UploadedFile() file): Promise {
    const job = await this.fileQueue.add('process-file', {file: file});
    console.log(`created job ${ job.id}`);
    await this.fileQueue.close();
  }

Above code basically adds a job to file-upload-queue to process that file.

Adding multiple jobs

One of the features that bull library offers to add multiple jobs to a queue. In our example, we read the data from file and we add each record as a job to another bull queue. This allows us to add multiple jobs to queue file-data-queue.

import { InjectQueue, Process, Processor } from "@nestjs/bull";
import { Job, Queue } from "bull";

const csv = require('csvtojson');

@Processor('file-upload-queue')
export class FileUploadProcessor{

    constructor(@InjectQueue('file-data-queue') private fileDataQueue: Queue) {}
    
    @Process('process-file')
    async processFile(job: Job) {
        const file = job.data.file;
        const filePath = file.path;
        const userData = await csv().fromFile(filePath);

        await this.fileDataQueue.addBulk(userData.map(user => ({
            name: 'process-data',
            data: user
        })));

        console.log('file uploaded successfully');
    }
    
}

We use addBulk functionality to add all the records from the file to queue file-data-queue.

Worker

Creating a worker through NestJS framework is simple. NestJS has a feature to run standalone application. We will use the same to run our workers while creating a separate module to process jobs from file-data-queue.

Our separate module FileDataModule will have a processor to process each record from the file.

import { Process, Processor } from "@nestjs/bull";
import { Job } from "bull";


@Processor('file-data-queue')
export class FileDataProcessor{

    
    @Process('process-data')
    async processFile(job: Job) {
        const data = job.data;

        console.log('processing data for a single user');
        console.log(data);

        // To-Do add some processing like inserting this data in DB
    }
    
}

We will use createApplicationContext to create a worker for FileDataModule like below:

import { NestFactory } from "@nestjs/core";
import { FileDataModule } from "src/file-read/file-data.module";

async function main() {
    const app = await NestFactory.createApplicationContext(FileDataModule, 
        {
            bufferLogs: true,
            abortOnError: false,
        }
    );
    app.enableShutdownHooks();
    await app.init();
    console.log(`Worker started`);

    process.on('SIGINT', async () => {
        console.log(`SIGINT signal received`);
        try {
            console.log('closing app...');
            await app.close();
            console.log(`Worker stopped`);
        } catch (error) {
            console.error(`Error during shutdown: ${error.message}`);

        } finally {
            console.log('exiting...');
            process.exit(0);
        }
    });
}

main();

This worker basically starts the application and waits for SIGINT signal to be terminated. Considering this worker is create the application context for FileDataModule, it uses the processor FileDataProcessor to process data from the queue.

Scaling Workers

We will run two instances of the worker we created above. We will also be running our NestJS Application and if we have imported FileDataModule in our main application module, we will have three instances of FileDataProcessor running to process the jobs from the bull queue file-data-queue.

There are two concepts to understand in Bull Queue since bull can offer either.

Parallelism

In Parallelism, two or more tasks run in parallel, independent of each other. The simplest way to understand this is when you have multiple machines running and each performing its own task.

Concurrency

Two or more tasks runs at the same time while diving the available CPU so that all the tasks can advance in their processing.

Concurrency might not increase the throughout, but parallelism will. Parallelism also scales linearly that means if you add more workers, more jobs will get processed.

Bull offers configuration for concurrency. But in this demo, we are focusing on parallelism.

Demo

We have already described the scenario. We have a NestJS application with an API to upload a file is running in one terminal. We have two workers running in two other terminals.

We upload the file through Postman with our API. This API will create the first job. Processor for this job then adds multiple jobs to another queue file-data-queue.

The NestJS application and the two workers then process jobs from this queue in parallel. The three screenshots below show the application, the worker 1 and the worker 2.

Conclusion

In this post, I showed how to scale bull job workers horizontally. This allows us to process jobs with high throughput. The code for this post is available here.