In this post, I will show how to use Keycloak as an identity broker. If you want to learn more, you can follow the guide on how to use keycloak with spring boot application.

What you will need

• Keycloak
• Postgres Database
• Service Provider Application (We are using a spring boot application)
• Identity Provider

What is Identity Broker?

An identity broker is a service that integrates two other services. An identity broker usually acts as an intermediary between a service provider and another identity provider. The identity broker creates a relationship with a third-party identity provider to use the provider’s identities to access the service provider application.

In this post, we will show how you can configure Keycloak as an identity broker between a spring boot application and an identity provider.

Start Keycloak with Postgres Database

If you download and run Keycloak as a standalone service, it will H2 (In-Memory) database. But we plan to use the Postgres database. This will also give a lot of clarity on how to use Keycloak with a relational database in an actual production environment.

• Install Postgres (I am using version 13.11)
• Download Postgres JDBC driver
• Create a database called keycloak in Postgres once you set up the database server locally.

To make Keycloak to use the Postgres database, we will need to update some configuration in the Keycloak installation directory. For this demo, I am using Keycloak version 11.0.1. If you are using the latest version, I will share the information below on how to integrate that with the Postgres database.

• • Go to Keycloak_installation/standalone/configuration directory
  • Open standalone.xml file in a XML editor
  • Add the below XML tags in datasources

<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true"
				statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}" >
	<connection-url>jdbc:postgresql://localhost:5432/keycloak </connection-url>
	<driver> postgres </driver>
	<security>
		<user-name>username </user-name>
		<password>password </password>
	</security>
</datasource>
<drivers>
	<driver name="postgres" module="com.postgres">
	<driver-class>org.postgresql.Driver </driver-class>
        <xa-datasource-class>org.postgresql.xa.PGXADataSource </xa-datasource-class>
	</driver>
</drivers>

• • We also need to update bindings in subsystem tags in the same standalone.xml file as below

<default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/KeycloakDS" 
managed-executor-service="java:jboss/ee/concurrency/executor/default" 
managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" 
managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>

If you are using the latest version of Keycloak, all you have to do is

• • Edit the file conf/keycloak.conf and set the following properties

db=postgresql
db-username=username
db-password=password
db-url=jdbc:postgresql://localhost:5432/keycloak

Once you set the above configuration, now we can start the Keycloak. If you are using the older version like me, you can use the following command:

standalone.bat -Djboss.socket.binding.port-offset=100

OR if you are using the latest version of Keycloak, you can use

kc.bat start-dev

Both of these commands are from a Windows environment perspective. Once the Keycloak server starts, access the Keycloak administration UI in the browser http://localhost:8180

Configure Keycloak with a realm

Let’s create a realm in Keycloak once you login as an administrator.

Once we create a realm, we don’t need to do anything yet. We will use Realm to set up our service provider later.

In the next step, we will configure another identity provider.

Configure Identity Provider in Keycloak

You can use Keycloak as an identity provider for SSO. But in this demo, Keycloak is acting as an identity broker. We will use saml-idp as an identity provider. This is a simple node-based SAML identity provider for development purposes. You can read about it more here.

npm install -g saml-idp.

Use the below command to start this identity provider. It will run on port 7000 by default.

saml-idp --acsUrl {POST URL} --audience {audience}

Now if we access http://localhost:7000 , the identity provider will be running and we can see the metadata in the browser. We can download this metadata to configure in Keycloak.

Let’s create an identity provider (SAML Version 2.0).

Enter an alias and Redirect URI

Let’s configure the SAML Identity provider’s details in Keycloak.

SSO URL – http://localhost:7000/saml/sso

Save the configuration. Now you should be able to download Keycloak’s metadata that you can import into the identity provider (saml-idp).

ACS URL = http://localhost:8180/auth/realms/ToDoListSAMLApp/broker/Saml-IDP-Test/endpoint

Audience = http://localhost:8180/auth/realms/ToDoListSAMLApp

Now, we can run our identity provider with the right ACS URL and Audience.

saml-idp --acsUrl http://localhost:8180/auth/realms/ToDoListSAMLApp/broker/Saml-IDP-Test/endpoint --audience http://localhost:8180/auth/realms/ToDoListSAMLApp

Next, we configure the service provider application.

Configure Service Provider in Keycloak

We already have written a spring boot application that we will use as a service provider. If you want to learn more about the Spring Boot application, you can follow this previous post.

1. Create a new client under our realm in Keycloak
2. Provide a client id as https://localhost:8743/saml2/service-provider-metadata/keycloak
3. Enter Name and Description for the client
4. Select SAMLas a protocol under Client Protocol
5. Enable include authnstatements, sign documents, and sign assertions
6. Select RSA_SHA256 as Signature Algorithm
7. username as Name ID Format
8. Root URL – https://localhost:8743
9. Valid Redirect URLs – https://localhost:8743/*
10. Master SAML Processing URL – https://localhost:8743/saml2/service-provider-metadata/keycloak
11. ACS POST/REDIRECT Binding URL – https://localhost:8743/login/saml2/sso/keycloak

And that’s all for Keycloak configuration is concerned.

Since we are using Spring Boot Application, we have configured the following properties in application.properties

# ====================================================================================
## Keycloak and SAML SP Properties
# ====================================================================================
spring:
    security:
        saml2:
            relyingparty:
                registration:
                    keycloak:
                        identityprovider:
                            entity-id: http://localhost:8180/auth/realms/ToDoListSAMLApp
                            singlesignon:
                                sign-request: false
                                url: http://localhost:8180/auth/realms/ToDoListSAMLApp/protocol/saml
                            verification:
                                credentials:
                                - certificate-location: classpath:credentials/idppostgres.cert

Start the spring boot application once you configure the properties.

Demo of Keycloak as Identity Broker

Assuming we are running Spring Boot Application, Keycloak, and another identity provider (saml-idp), now we can run the demo by accessing the Spring Boot Application.

You can select the identity provider Saml-IDP-Test and it will redirect to that identity provider.

Once you select Sign In option, it will redirect you back to service provider application if the user already exists in Keycloak. If the user does not exist in Keycloak, keycloak will ask to enter minimal attributes and it will create that user dynamically (JIT Provisioning).

And this way, the user was able to log in successfully.

Conclusion

In this post, I showed how to use Keycloak as an identity broker with another identity provider. Keycloak is a very well-documented SSO provider. If you have any questions or feedback, please do send me questions.

With this post, we will start a series of posts to build a full-stack application using NestJS. NestJS is a popular framework to build scalable and efficient Node.JS server-side applications.

Techstack

To build this application, we will be using multiple different frameworks.

• Backend – NestJS with Typescript
• Frontend – NextJS with Typescript
• Database – MySQL
• ORM – Prisma
• Cloud Provider – Amazon Web Services (AWS)

Application Ideation

Everything starts with an idea and execution. The idea for this application is to build an applicant tracking system for recruiting agencies.

There are two types of users in this application. A recruiter creates the job and an applicant who applies for that job. Once the applicants apply for the job, the recruiter reviews the application, contacts applicants, and invites them for an interview. Once the interview is complete, the result is updated in the system so that an applicant can view it at any time.

There are various stages that a job application goes through.

For recruiter – Job creation -> Interview in progress -> Candidate selected -> Job complete -> Job expired.

For candidate – Job application -> Interview if selected -> Result.

There will be a bunch of automation involved with respect to notification through the life-cycle of the job.

NestJS Setup

We will be using the NestJS framework with typescript to build the backend of this application. Additionally, we will use ReactJS with typescript for the front end of this application.

Nevertheless, we will set up our NestJS framework on our development machine.

Create a new app using NestJS with the following command:

nest new staffingapp

Just like in previous articles, we will use prisma as our ORM for creating this nestjs application.

• Install Prisma dependencies

npm install prisma --save-dev

npm install @prisma/client

• Create initial Prisma setup

npx prisma init

If you are using a Windows environment, you might run into an error while running the above command. So set environment variables

set PRISMA_CLI_QUERY_ENGINE_TYPE=binary

set PRISMA_CLIENT_ENGINE_TYPE=binary

This completes our initial backend app setup. Let’s create a data model for this application further.

Data Model for Staffing App

We have three main data objects.

• User
• Job
• Company

There are more that we will require for a functioning app, but for now we will look at these three only.

These models will contain the following fields:

enum ROLE {
  CANDIDATE
  RECRUITER
  ADMIN
}

model User {
  id         String @id @default(uuid())
  email      String @unique
  first_name String 
  last_name  String 
  password   String 
  company  Company @relation(fields: [company_id], references: [id])
  company_id  String
  role       ROLE
  created_at DateTime @default(now())
  updated_at DateTime @default(now())  
}

model Company {
  id         String  @id @default(uuid())
  name       String  
  type       String 
  user       User[]
  job        Job[]
  created_at DateTime @default(now())
  updated_at DateTime @default(now())
}
enum JobType {
  FULLTIME
  PARTTIME
  CONTRACTOR
  INTERN
}
model Job {
  id         String   @id @default(uuid())
  job_type    JobType 
  job_name    String 
  job_role    String
  description String 
  job_details JobDetails @relation(fields: [job_details_id], references: [id])
  job_details_id String @unique
  company    Company @relation(fields: [company_id], references: [id])
  company_id  String
  expires_at DateTime
  created_at DateTime @default(now())
  updated_at DateTime @default(now())
}

model JobDetails {
  id              String @id @default(uuid())
  salary_details  String
  benefit_details String
  job             Job?
  created_at      DateTime @default(now())
  updated_at      DateTime @default(now())
}

With the above data model, we have set up the initial database and application. In the next post, we will create backend APIs and integrate them with the database.

Conclusion

In this post, we started building a full-stack application using NestJS framework. We set up the initial application and database along with Prisma ORM.

In this post, I will show how to use the new feature of Flows with Bull Queue using NestJS application. If you are new to using Bull Queue, here is my previous post about using bull queue in NestJS application.

What are Flows?

Flows are the new feature from Bull Queue. With this feature, we can establish a parent-child relationship between jobs. A job in a queue can create a number of children jobs and push them into another queue for processing.

In real-world applications, you will need to split large images, video files, or even text files into smaller chunks, process them and bind them together again. Flow is an ideal candidate to implement splitting and binding operations and processing those chunks in parallel.

Divide and Conquer OR Map-Reduce are very well-known programming techniques. Flows follow the same pattern when processing CPU-intensive jobs.

File Validation And Processing

Now, let’s look at a real example to understand how the Flows work. As part of this example, we will be processing a large file. This large fill will go through file validation and valid records will be processed further or merged into another file OR stored in the database. To understand this better, look at the diagram below:

Flows with Bull Queue

I will not cover the basics to get started with NestJS Application, but you can look at this previous post to get started.

BullMQ offers Flows as a feature currently. One key thing to note is that BullMQ and Bull are different open-source libraries supporting the same set of features. And both libraries work with NestJs as well. Bull does not offer Flows as a feature yet.

Install bullmq as the library for your application.

npm i  bullmq

npm i @nestjs/bullmq

Splitting a large file

To understand Flows better, let’s create our design flow first.

• Upload a large file.
• A bull job will split the file into chunks.
• We will create a Flow to process each of these chunks.
• The parent job of the flow will merge the result of all chunks into another file.

We are using a NestJS application as part of this demo and we have a controller to upload the file.

Nevertheless, we will first configure our Bull Queues with Redis Connection.

import { Module } from '@nestjs/common';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { AppController } from './app.controller';
import { BullBoardController } from './bull-board-controller';
import { FileUploadProcessor } from './file-upload.processor';
import { PrismaService } from './prisma.service';
import { UserService } from './user.service';
import { TransformFileProcessor } from './transform-file-processor';
import { SplitFileProcessor } from './split-file.processor';
import { MergeDataProcessor } from './merge-data.processor';
import { BullModule } from '@nestjs/bullmq';

@Module({
  imports: [    
    BullModule.forRootAsync({
      imports: [ConfigModule],
      useFactory: async (configService: ConfigService) => ({
        connection: {
          host: configService.get('REDIS_HOST'),
          port: Number(configService.get('REDIS_PORT')),
        }
      }),
      inject: [ConfigService],
    }),
    BullModule.registerQueue({
      name: 'file-upload-queue'
    },
    {
      name: 'split-file-queue',
    },
    {
      name: 'transform-file-queue',
    },
    {
      name: 'merge-data-queue'
    }),
    BullModule.registerFlowProducer({
      name: 'merge-all-files',
    }),
  ],
  controllers: [AppController, BullBoardController],
  providers: [UserService, PrismaService, FileUploadProcessor, 
           TransformFileProcessor, 
           SplitFileProcessor, 
           MergeDataProcessor],
})
export class AppModule {}

We have configured Bull Queues and registered those queues and the flow.

The controller to upload a file is a REST API.

  @Post('/uploadALargeFile')
  @UseInterceptors(FileInterceptor("csv", {
    storage: diskStorage({
      destination: './csv',
      fileName: (req, file, cb) => {
        const randomName = Array(32).fill(null).map(() => (Math.round(Math.random() * cb(null, `${randomName}${extname(file.originalname)}`))))
      }
    })
  }))
  async uploadLargeCsvFile(@UploadedFile() file): Promise {
    const job = await this.splitFileQueue.add('split', {file: file});
    console.log(`created job ${ job.id}`);
    await this.splitFileQueue.close();
  }

We use FileInterceptor to upload the file on a local disk and add the file to the Bull Queue job. This will job will process this file further.

split-file-queue will process the job. It will read the file and split the large file into chunks of 500 rows. Then it will add all these chunks to another queue to transform each file.

    async process(job: Job) {        
        const file  = job.data.file;        
        const filePath = file.path;
        const chunksOfInputFile = await this.splitInChunks(filePath);
        console.log(chunksOfInputFile);
        await this.addChunksToQueue(chunksOfInputFile);
    }

Interestingly, we add chunks to the queue through a Flow. The feature that Bull Queue offers for a parent-children relationship with jobs.

    async addChunksToQueue(files: string[]) {
        const flowProducer = new FlowProducer();

        try {
            return await flowProducer.add({
                name: 'merge-all-files',
                queueName: 'merge-data-queue',
                children: files.map((file) => ({
                    name: 'transform-file',
                    queueName: 'transform-file-queue',
                    data: { file: file},
                })),            
            });
        } catch (err) {
            console.log (`Error adding flow ${err}`);
        }

    }

In the above code, we can see that we created a flow merge-all-files. The parent job is going to get processed in merge-data-queue. transform-file-queue will process each file that we split previously.

Transforming the file

In transform-file-queue, we read the file and validate each record. As part of this transformation, we store error records separately from valid records. Valid records from each file job are then stored in a transformed file.

@Processor('transform-file-queue')
export class TransformFileProcessor extends WorkerHost{
    
    async process(job: Job): Promise {
        const file  = job.data.file;

        console.log('Validating the file', file);

        return await this.validateAndWriteFile(file, job.id!); 
    }

    async validateAndWriteFile(file: string, jobId: string) {
        console.log('transforming the file');
        const srcName = basename(file);            
        const output = `./output/transformed-${srcName}`;

        const validatedData: string[] = [];
        const errorData: string[] = [];
        const validate = new Promise<{ errorData: string[], validatedData: string[] }>(function(resolve, reject){
            fs.createReadStream(file)
            .pipe(csvparser())        
            .on('data', (data) => {                               
                const regExp = /[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,3}$/
                if (data['Name'] === '') {
                    errorData.push(`${data['Name']},${data['Email']},${data['EmployeeId']}`);
                } else if (!regExp.test(data['Email'])) {
                    errorData.push(`${data['Name']},${data['Email']},${data['EmployeeId']}`);
                } else {
                    validatedData.push(`${data['Name']},${data['Email']},${data['EmployeeId']}`);
                }
            })
            .on('end', () => {
                resolve({
                    errorData,
                    validatedData,
                });
            });
        });

        const result: {
            errorData: string[],
            validatedData: string[],
        } = await validate;

        console.log('Any invalid data ', result?.errorData);

        const csvData = validatedData.map((e) => {
            return e.replace(/;/g, ",");
        });
        fs.writeFile(output, csvData.join("\r\n"), (err) => {
            console.log(err || 'done')
        });
        return output;
    }
}

Once we complete processing all the file jobs, the parent job of merging these files will start the process.

The one key benefit of Flow is that it will be in a waiting-children state for children jobs to complete. Once all the children are done, it will be moved into the wait state and further, it will start the processing.

Merging the files

Another advantage of Flow is that parent jobs can access the result of all the children jobs. In this case, we will be processing all the transformed files and merging them into a single file. The resulting file will contain all the valid data that we can use for further processing.

@Processor('merge-data-queue')
export class MergeDataProcessor extends WorkerHost {    
        
    async process(job: Job): Promise {        
        const transformedChunks = await job.getChildrenValues();
        const files = Object.values(transformedChunks).sort();

        console.log('Start merging data into a single file', files);

        await this.mergeFiles(
            job.id, 
            files,
            `./output/merged-${job.id}.csv`
        );
    }

    async mergeFiles(
        jobId: string,
        files: string[],
        finalOutputFileName: string,
    ): Promise {

        const data = [];
        files.forEach( (file) => {
            const fileData = fs.readFileSync(file);
            data.push(fileData);
        });
        fs.writeFile(finalOutputFileName, data.join("\r\n"), (err) => {
            console.log(err || 'done')
        });
    }
}

Flows are a really great feature if you are processing a large set of data and consuming CPU.

Conclusion

In this post, I showed how to use Bull Queue Flows in a NestJS application. One can easily, use this feature for various tasks where you need some kind of parent-child relationship. The code for this demo is available bull queue github repository.

As more and more sensitive information is being shared online, it is becoming increasingly important to have strong security measures in place to protect against unauthorized access. One way to enhance the security of web applications is to implement two-factor authentication (2FA) in addition to traditional username and password authentication. In this article, we’ll explore best practices for securing Spring Security applications with 2FA.

What is Two-Factor Authentication?

Two-factor authentication (2FA) is a security process that requires users to provide two different authentication factors to access an application. These factors typically fall into three categories:

1. Something the user knows, such as a password or PIN
2. Something the user has, such as a smart card or mobile device
3. Something the user is, such as a biometric identifier like a fingerprint or facial recognition

By requiring two different authentication factors, 2FA adds an extra layer of security to traditional username and password authentication.

Why Use Two-Factor Authentication in Spring Security Applications?

Implementing 2FA in Spring Security applications can greatly enhance the security of your application. Here are some reasons why:

1. Protection against password guessing attacks: Attackers may use automated scripts to guess passwords until they gain access to an account. 2FA adds an extra layer of security, making it more difficult for attackers to gain access.
2. Compliance with security regulations: Many industries have regulations that require 2FA, such as the Payment Card Industry Data Security Standard (PCI DSS).
3. Enhanced user trust: By providing an additional layer of security, users will have more trust in your application, which can lead to increased usage and adoption.

Best Practices for Implementing Two-Factor Authentication in Spring Security Applications

Here are some best practices for implementing 2FA in Spring Security applications:

1. Choose a secure 2FA method

When implementing 2FA, it is important to choose a secure method for the second factor. Some common 2FA methods include:

• SMS authentication: Sends a unique code to the user’s mobile phone via SMS
• Mobile authentication app: Users install an app on their mobile device to generate a unique code
• Email authentication: Sends a unique code to the user’s email address

Each of these methods has its pros and cons. It is important to choose a method that is secure and convenient for your users.

2. Use a separate authentication provider

When implementing 2FA, it is recommended to use a separate authentication provider for the second factor. This provider should be separate from the authentication provider used for the initial login. This helps prevent attackers from gaining access to both factors.

3. Store sensitive data securely

Sensitive data, such as user passwords and 2FA codes, should be stored securely. This includes hashing and salting passwords and encrypting 2FA codes.

4. Provide clear instructions to users

Users may be unfamiliar with 2FA and may need clear instructions on how to use it. It is important to provide clear and concise instructions to users to help them understand how to use 2FA.

5. Test thoroughly

Before implementing 2FA in a production environment, it is important to test thoroughly to ensure that it is working correctly. This includes testing for usability, security, and performance.

Conclusion

In this post, I shared some of the best practices that one can follow while implementing two-factor authentication (2FA) in a Spring Security application. You can learn more about Spring Security in my book Simplifying Spring Security.