Author Archives: yogesh.mali@gmail.com

Background Job Scheduling with Pg-Boss and NestJS

In a distributed system, we need to automate various workflows and there are ways to automate these workflows. One of the ways that I have used frequently is, background jobs. And if you noticed, even my last post was about using background jobs with trigger.dev. In this post, I will share a new way for background job scheduling with Pg-Boss and NestJS.  pg-boss is a job queue based on postgres database. How cool is that? And if it is not clear, I love Postgres database.

What is Pg-Boss?

pg-boss is a job queue library for node.js application and uses Postgres for persistence. It uses reliability of Postgres for storing, scheduling jobs. Over last 10 years, Postgres as database had made changes and pg-boss as a feature for background job queuing is one of them. In short, pg-boss reduces one more infrastructure item while building a distributed system application.

Pg-Boss vs Bull Queues

You might have noticed, I have written extensively about Bull Queues. Bull queues are another way for job queue system based on redis. The question that might come immediately, then why use pg-boss? Bull queues have been great and easy to use as well. There has been a huge community support for bull queues. Then what is so special about pg-boss?

I get it.

There is no right or wrong answer when choosing pg-boss or bull queues. It all depends on the system you are building.

Then why choose pg-boss?

If you are already using Postgres as a database for your application, then it is a no-brainer to use pg-boss instead of bull queues which are based on Redis.  Redis is an additional infrastructure overhead if you are not using cache for any other purposes.

Since pg-boss is based on Postgres and Postgres is really good at transactional support, locking and handles the queue-based mechanism really well.

Nevertheless, choose the queueing mechanism that fits your needs right. There is no trade-off if you choose Bull queues over pg-boss.

Fundamentals of Pg-Boss

Pg-Boss is a queue system and it is very similar to various other queue systems like Bull, AWS SQS.

Jobs in Pg-Boss queue are a state machine. All jobs start with created state and then move to active when picked up for processing.

When the worker picks up the job to process and completes it successfully, it moves to completed state.

If the job fails, it moves to failed state and can be moved to retry state if retries options are configured. If this job is retried, it can move back into active state.

active job takes too long to process, then it moves into expired state.

Any job that is either in created or active state, you can cancel them with cancel and then the job will move into cancelled state.

All jobs that are completed, failed, cancelled or expired, can move to archive state.

Pg-Boss and NestJS

In this post, we will show a simple example of how to use pg-boss queue with NestJS application and how we can schedule a background job with Pg-Boss.

We will use @apricote/nest-pg-boss module in our application to integrate pg-boss.

I will not go over setting up a sample NestJS application. In this sample application, I am using Prisma ORM and Postgres database.

Let’s start with installing this module dependency in your NestJS Application.

npm install @apricote/nest-pg-boss

Once the dependency is installed, we will set up the PGBossModule module in our top main module.

Add the following module in your app.module.ts file


    PGBossModule.forRootAsync({
      application_name: 'default',
      useFactory: (config: ConfigService) => ({
        host: config.get('DB_HOST'),
        user: config.get('DB_USERNAME'),
        password: config.get("DB_PASSWORD"),
        database: config.get("DB_DATABASE"),
        schema: "public",
        max: config.get("DB_POOL_MAX"),
      }),
      inject: [ConfigService],
    }),

As you can see there are few environment variables are involved, make sure you have those configured for your database.

Once you start the application, you will notice few things about the Pg-Boss. It will initialize that module and if you have created any jobs, it will display the corresponding workers.

Interestingly, when the Pg-Boss module is initialized, it will also create these tables in your Postgres database.
archive
job
schedule
subscription
version
Pg-Boss module uses these tables to persist the jobs and their states. It also allows us to schedule these jobs at particular frequency. In short, a nice replacement for cron jobs.

In the next section, I will show how to create a job, schedule it and run it with an handler.

Scheduling Job with Pg-Boss

Let’s create our first job for Pg-Boss queue. @apricote/nest-pg-boss library offers an handy method createJob .

I created a simple interface for job data that I will pass for my job. This job will send a welcome email to user after they sign up for my application.

import { createJob } from "@apricote/nest-pg-boss";

export interface UserJobData {
    email: string;
    firstName: string;
}

export const userCreatedJob = createJob('user-signup-job');

And here we created a simple job user-signup-job.

And now include this job in your module so it will be created on the application start up. Basically, it will create a worker to process the job when the application will trigger this job.

@Module({
  imports: [DBAccessModule, CompanyApiModule, UsersModule, PGBossModule.forJobs([userCreatedJob])],
  controllers: [UsersController],
  providers: [UsersService, UserJobService],
  exports: [UsersService]
})
export class UsersApiModule {}

In my controller for sign-up, application saves the user information in database. Once that is complete, it will send this job in the queue.


    async createUser(user: CreateUserDto, company: Company) {
        const hashedPassword = await bcrypt.hash(user.password, 12);

        const userToBeCreated = User.createNewUser({            
            firstName: user.firstName,
            lastName: user.lastName,
            email: user.email,
            companyId: company.id,
            password: hashedPassword,
        });
        const savedUser = await this.userRepository.save(userToBeCreated);

        await this.userCreatedJobService.send({email: savedUser.email, firstName: savedUser.firstName}, {});

        return savedUser;
    }

In the next section, we will see how to run this job with an handler.

Running Job with Handlers

In my previous createUser method, we sent a job to worker for processing. I have a service UserCreatedJobService and that is a worker to handle this job.


import { Injectable } from "@nestjs/common";
import { UserJobData, userCreatedJob } from "common/jobs";
import { Job } from "pg-boss";

@Injectable()
export class UserJobService {
    
    @userCreatedJob.Handle()
    async handleJob(job: Job) {
        console.log(`sending an email to user with email ${job.data.email} and name ${job.data.firstName}`);
    }
}

The decorate @userCreatedJob.Handle() allows us to process this job added in the queue.

More about Job-Options

We showed a simple job in this post. But, in a real production application, you might need to create complex jobs depending on your requirements. Good thing is that Pg-Boss offers variety of options to configure these jobs.

  • priority – You can configure priority for the job with a number. For higher priority jobs, use the bigger number.
  • retry options – There are few options like retryLimit, retryDelay and retryBackoff available to retry the jobs.
  • expiration options – If you have a long running jobs, you probably need to set expireInSeconds option. There is also expireInMinutes OR expireInHours option.
  • retention options – This option allows to retain the job in a particular state before it can be archived. retentionSeconds, retentionMinutes.
  • deferred job – The option startAfter allows to specify when to start the job.
  • unique jobs – singletonKey OR useSingletonQueue allow a unique key for the job and that ensures only one job with that key is processed.

Conclusion

In this post, I shared the details of Pg-Boss queue library and how to use it for background job scheduling in a NestJS application.

Long Running Background Jobs With Trigger Dev and NestJS

In this post, I show how to create long-running background jobs using Trigger.dev and NestJS framework.

I have been playing around on how to write cron jobs that listen and do things on certain events. You can write a simple cron job that runs on a schedule to check something.

What if you want to trigger some scheduled work on some event or some webhook trigger? Can we have a long-running background job for that? Serverless frameworks like Vercel offer edge functions, but they are limited by timeout, so they don’t serve the purpose of long-running. Let’s dive deep into building a background job with trigger dev and NestJS.

Set up NestJS Application

Here are some of my previous posts where I have covered about setting up a NestJS application.

To demonstrate this particular post, I have a simple NestJS application running.

Configure Trigger.dev

Let’s set up trigger.dev for a long-running background job. We will need trigger.dev sdk in our NestJS project.

npm install @trigger.dev/sdk @trigger.dev/nestjs @nestjs/config

If you have not created an account on trigger.dev, you can create one now. They offer a hobby plan that does not charge anything. You can also self-host trigger.dev.

Once you have signed up, you will need to get API KEY and API URL.

Set the environment variable in your NestJS project

TRIGGER_API_KEY=api_key_value_from_trigger_dev

TRIGGER_API_URL=https://cloud.trigger.dev

Create a Trigger Module in NestJS

In your main module, add a new module to configure trigger with API and URL.


TriggerDevModule.registerAsync({
    inject: [ConfigService],
    useFactory: (config: ConfigService) => ({
      id: 'betterjavacode',
      apiKey: config.getOrThrow('TRIGGER_API_KEY'),
      apiUrl: config.getOrThrow('TRIGGER_API_URL'),
      verbose: false,
      ioLogLocalEnabled: true,
    }),
  }),

Define a job

Once we have the trigger module set up, we will define a background job. To be able to set up a job, we will create a new controller job.controller.ts.

 


import { Controller, Get } from "@nestjs/common";
import { InjectTriggerDevClient } from "@trigger.dev/nestjs";
import { TriggerClient, eventTrigger } from "@trigger.dev/sdk";


@Controller()
export class JobController {
    constructor(
        @InjectTriggerDevClient() private readonly client: TriggerClient
    ) {
        this.client.defineJob({
            id: 'test-job-one',
            name: 'Sample Job One',
            version: '0.0.1',
            trigger: eventTrigger({
                name: 'test.event',
            }),
            run: async (payload, io, ctx) => {
                await io.logger.info('Hello world!', { payload });

                return {
                    message: `Hello world - ${payload.name}!`,
                };
            }
        });
    }

    @Get('/get-simple-job-trigger')
    getSimpleJobTrigger(): string {
        return `running trigger.dev with client-id ${this.client.id}`;
    }
}

Anyhow, this might be a lot to understand. So, I will go over this controller in steps.

With trigger.dev, one can define a job. The job contains a trigger and a task.

Trigger is either an event, a scheduler, or a manual that triggers the job.

A task is what the job executes on triggered. This is the basic block of run defined within the job.

As shown in the code, I have defined a job with id test-job-one that gets triggered with an event trigger test.event and when triggered, it runs a job that returns Hello world - ${payload.name}.

Now, to be able to create this job and run this job on a server, we have to create this job in trigger.dev app. One way to define is to configure this in package.json

 


"trigger.dev": {
    "endpointId": "betterjavacode"
  }

Once you run the app, you will see the job created in trigger.dev portal

TriggerDev Job and NestJS Application

Running the app

While running the app, make sure to run both nestjs app and trigger.dev.

npm run start – to start the nestjs application

npx @trigger.dev/cli@latest dev --port 3001 – to start trigger dev app

How to trigger the job

trigger.dev provides a test run option. In trigger.dev portal, go to your job and see the section for test and you will see an option to pass payload to your job. And then press Run button.

Test Run with Trigger.Dev

I like the flexibility of trigger.dev on defining the job. Another feature I love about trigger.dev is that they have various integrations available like OpenAI, Linear, Slack etc. If you need to develop some automated workflow, trigger.dev is a great choice to build that workflow.

Conclusion

In this post, I showed how to use trigger.dev to create a long-running background job with NestJS application. Hopefully, I will cover a real example of a workflow with trigger.dev in upcoming posts.

7 Key Characteristics of a Good Staff Engineer

If you are wondering, what does a staff engineer do? OR what exactly are different areas of work that staff engineers can contribute to? OR what are the top characteristics of a good staff engineer? Then you are at the right place.

In this post, I cover the 7 key characteristics of a good staff engineer. If you are a senior engineer and wondering how you can get promoted to a staff engineer, I share some details that can help you in your next promotion plan.

What is a Staff Engineer?

Different companies have different ladders for software engineers. In my experience, I have seen

  • software engineer => senior software engineer => principal software engineer => architect
  • software engineer => senior software engineer => staff software engineer => principal software engineer

Staff Engineer is an individual contribution ladder role. On the management path, it can resemble to Engineering Manager. However, the duties of a staff engineer and that of an engineering manager are different.

Irrespective of what the ladder is, a staff engineer is a senior role where an engineer helps a team and other engineers to lead on various projects.

The difference between a senior engineer and a staff engineer lies in the impact they create on the team and projects. A senior engineer can lead a single project while a staff engineer can be part of multiple projects.

Let’s dive into the 7 key characteristics of a good staff engineer.

7 Characteristics of Staff Engineer

1. Stay Curious

Irrespective of your impact or role, you should always stay curious. With evolving technology and stacks, it helps to stay curious and find different solutions to either the existing problems or to the new problems.

2. Always Be Learning

Stay humble. Irrespective of how much experience you have, you should always be learning. Learn from your seniors, from your juniors, from peers, and other engineers. Learn from every source of information you read.

The quickest way to grow is to keep learning. And the quickest way to stop learning is to think that you have all the answers.

Not knowing something is a strength and that keeps the door open to learn that thing. The smartest staff engineers are always learning. They are aware that technology will keep changing and they have to keep learning.

3. Company First, Team First

Staff Engineers can have a huge impact on a company’s growth. In early-stage startups, staff engineers form the engineering culture, create and build processes, and build the knowledge base.

It would be selfish of engineers to think for only their careers. Nevertheless, this is especially important for Staff Engineers. They have to set up the team for success. They have to set up the company for success.

4. Know the system

A good staff engineer will know the big picture of the system they are operating in. They will have end-to-end knowledge of the services they are working or managing. System design should be one of the strengths of a staff engineer. You can argue that staff engineers need to know the entire architecture of the platform that the company runs. But, it is not necessary. Having a big picture of the platform is good enough to figure out risks.

A good staff engineer can also foresee the possible risks as the system goes through various stages of scale and growth.

5. Provide honest feedback

Working with different skillset engineers will help staff engineers figure out the strengths of those engineers. In return, staff engineers have to provide honest feedback to those engineers to help them accelerate their careers.

The staff engineer is an ally for the manager and the rest of the team. Staff engineer fosters a bond between them and sets both parties up for success.

6. Make your team 10x

Delegation is one of the hardest skills to learn for an engineer when they know they can build something. But it is also equally important to help other engineers grow. A good staff engineer not only helps in various project activities but also helps the rest of the team to 10x their output.

Providing knowledge, training, skills, and feedback, a good staff engineer helps the team to do better and better.

7. Action First

In every project, there will be a time when there is not enough clarity about requirements. And even if there is, engineers who are coordinating on the project are stuck at making a decision. Everybody is still figuring out the trade-offs. In such cases, a good staff engineer leads the team for action.

A good staff engineer prioritizes the action for the project so the team can keep moving and not get stuck around various decisions to be made. A good staff engineer needs to make some bold calls to keep moving the needle and in the process lead the team for further progress.

Conclusion

In this post, I showed the 7 key characteristics a staff engineer can have. Not every staff engineer will have all these characteristics.

Figuring out your strengths as a staff engineer is what you need to do as a staff engineer. Most staff engineers I know have a combination of these characteristics and skills.

Three Dimensions of Product Building

In this post, I want to talk about 3 dimensions every engineer can consider while building a good product.

If a product exists, it must solve a pain, or a problem OR at least elevate the experience of the user if there was never a problem before. A good example of this is the iPod.

In short, a product must do

  • save time for a user
  • save money for a user
  • make money for a user

Let’s look at these dimensions in detail now

1. Save time

Most users will pay for your product if it will save them time. Time is money and it plays a role when it comes to software products. If a user needs a solution for a problem, they can look at various aspects. But they also look at if the solution can save them time. That’s why we have reduced the usage of Mainframe around the world. With advanced hardware, building performant software has become easier.

Depending on the problem a consumer is facing, the product can also improve the user experience. From a suite of productivity apps to convenience apps, all help us to save time. Uber helps us get a taxi in a short time without leaving our house. Doordash delivers food without wasting our time in commute.

Time is money. Save time, save money.

2. Save Money

If you exclude luxury items, most people love to save money on many things they buy. A great example is Amazon Shopping. You can buy anything on Amazon and there are multiple options available to users. Users can choose any option that can save money to them.

Cheap flights, cheap tickets for the show, cheap houses. You name it. People love to save money. With the current market of inflation, everyone would love to save some money.

Black Friday is a shopping holiday and people spend thousands of dollars to save some money on things they want.

If your product can save money for your users, and for your company, then it is worth building that product.

You can use this dimension to decide what features to build, and what bugs to fix.

3. Make Money

I intentionally kept this dimension of product building at last. At the end, who does not like to make more money? If your product can make money for others, you will be at the top of a pyramid of wanted products.

This dimension has two aspects to it. One to make money for your users and one to make money for yourself if you are the product builder.

If you build a product that solves a pain for users, they will pay you to use that product. Can your product also make money for your users in the process? Can you leverage your product in a way that can open another stream of income for yourself other than subscription users?

A lot of fintech products become custodians of their customers’ money and solve various problems for customers from payment processors, invoicing, taxes, purchase parity, etc. But they also make money by keeping that money for their customers. Stripe is a good example of this.

Other Aspects of Product Building

What about user experience? What about user ease of product usage? And what about the security of the product?

Yes, yes and yes. They are all important and necessary to build over some time. If you ask the question Why are you building something or why are you fixing something, you will find an answer that aligns with these 3 dimensions of product building.

User experience will enhance product usage for users and in turn, users can pay money to use the product.

Security will help to avoid future risks from hackers and save you time and money.

Next time, you build a feature, a fix or an entire product, look at every task with these 3 dimensions in mind. If you can’t find the right reason immediately, you can put that feature or fix it on the back burner.

Conclusion

In this post, I shared the three dimensions of product building. These 3 dimensions are fundamental block for products. As a product engineer, if you look at these fundamental blocks and solve your next problem, you will be able to a build a product that people want.

How To Use Event Emitter Technique in NestJS App

In this post, I demonstrate how to use an event-emitter technique in a NestJS application. This particular technique is part of the NestJS framework. But you can learn about the fundamentals of event-driven architecture and event-sourcing in microservices.

Introduction

Events-based architecture is helping to build scalable applications. The major advantage of this architecture pattern is there is a source and destination of events, aka publisher and subscriber. Subscriber processes the events asynchronously.  Frameworks like NestJS provide techniques for emitting the event without any other overhead. Let’s dive into this event-emitter technique.

Event Emitter Technique

NestJS Documentation says “Event Emitter package (@nestjs/event-emitter) provides a simple observer implementation, allowing you to subscribe and listen for various events that occur in your application”

Using this technique will allow you to publish an event within your service/application and let another part of the service use that event for further processing. In this post, I will demonstrate this

  • client uploads a file
  • an event is emitted for a file uploaded
  • another service processes that file based on that event.

Now, the question arises when you want to use this technique and if there are any drawbacks to using this technique. Let’s look into that further.

When To Use

In an application, if you want to do some CPU-heavy OR data-intensive work, you can think of doing that asynchronously.  How do we start this asynchronous work though? That’s when event-emitter comes into the picture.

One part of your application will emit an event and another part will listen to that event. The listener will process that event and perform the downstream work.

To understand this better, let’s look at an example.

Demo

Controller –

We have a simple REST controller to upload a file.

@Controller('/v1/api/fileUpload')
export class FileController {
    constructor(private fileUploadService: FileUploadService) {}

    @Post()
    @UseInterceptors(FileInterceptor('file'))
    async uploadFile(@UploadedFile() file: Express.Multer.File): Promise {
        const uploadedFile = await this.fileUploadService.uploadFile(file.buffer, file.originalname);
        console.log('File has been uploaded,', uploadedFile.fileName);        
    }

}

This controller uses another NestJS technique Interceptor that we have previously seen in this post.

Register the Event Emitter Module –

To use an event emitter, we first register the module for the same in our application module. It will look like below:

EventEmitterModule.forRoot()

Service –

Our controller uses a fileUploadService to upload the file to AWS S3 bucket.  Nevertheless, this service will also emit the event after the file has been uploaded.


@Injectable()
export class FileUploadService {
    constructor(private prismaService: PrismaService,
        private readonly configService: ConfigService,
        private eventEmitter: EventEmitter2){}
    
    async uploadFile(dataBuffer: Buffer, fileName: string): Promise {
        const s3 = new S3();
        const uploadResult = await s3.upload({
            Bucket: this.configService.get('AWS_BUCKET_NAME'),
            Body: dataBuffer,
            Key: `${uuid()}-${fileName}`,
        }).promise();

        const fileStorageInDB = ({
            fileName: fileName,
            fileUrl: uploadResult.Location,
            key: uploadResult.Key,
        });

        const filestored = await this.prismaService.fileEntity.create({
            data: fileStorageInDB
        });

        const fileUploadedEvent = new FileUploadedEvent();
        fileUploadedEvent.fileEntityId = filestored.id;
        fileUploadedEvent.fileName = filestored.fileName;
        fileUploadedEvent.fileUrl = filestored.fileUrl;

        const internalEventData = ({
            eventName: 'user.fileUploaded',
            eventStatus: EventStatus.PENDING,
            eventPayload: JSON.stringify(fileUploadedEvent),
        });

        const internalEventCreated = await this.prismaService.internalEvent.create({
            data: internalEventData
        });
        fileUploadedEvent.id = internalEventCreated.id;

        if (internalEventCreated) {
            console.log('Publishing an internal event');
            const emitted = this.eventEmitter.emit(
                'user.fileUploaded',
                fileUploadedEvent
            );
            if (emitted) {
                console.log('Event emitted');
            }
        }

        return filestored;
    }
}

This service code is doing a few things.

  • Upload a file to AWS S3
  • Store the internal event user.fileUploaded
  • Emit that event

One reason we are storing that event in the database is to know exactly when the event was emitted and if we ever want to reprocess the same event if we can emit that.

Anyhow, we are using EventEmitter2 class in our constructor. To be able to use this class, we should make sure we install @nestjs/event-emitter dependency in our Nest project.

Event Listener –

We have a service that is emitting the event, but we will also need a listener service that will process the emitted event. So let’s write our listener service.

Our event class looks like this:

export class FileUploadedEvent {
    id: string;
    fileName: string;
    fileUrl: string;
    fileEntityId: number;
}

And our listener class will look like below:

@Injectable()
export class FileUploadedListener {

  constructor(private prismaService: PrismaService){}

  @OnEvent('user.fileUploaded')
  async handleFileUploadedEvent(event: FileUploadedEvent) {
    
    console.log('File has been uploaded');
    console.log(event);
    await this.prismaService.internalEvent.update(
        {
            where: {
                id: event.id,
            },
            data: {
                eventStatus: EventStatus.PROCESSED,
            }
        }
    );
    console.log('File will get processed');

  }
}

To be able to listen to emitted events, we use annotation @OnEvent. Just like this listener, we can add more listeners for the same event and each listener might do its own set of work.

Now, if we run our application and upload a file,  an event will be emitted and listened on. We can see the database entries for the event.

Event Emitter NestJS Application

Conclusion

In this post, I showed how to use the Event Emitter technique in a NestJS Application.