Author Archives: yogesh.mali@gmail.com

Spring Retry vs Resilience4j Retry

In this post, I will show the comparison of the two retries – Spring Retry vs Resilience4j Retry. Usually, you can combine retry with a circuit breaker when implementing to make your application more robust. I already covered the circuit breaker demo. Also, I have updated my book Simplifying Spring Security with Okta Demo if you are interested to learn more about Spring Security.

Spring Retry vs Resilience4j Retry

Spring Retry allows applications to retry a failed operation automatically. In most cases, if your service is calling another service and another service is not responding for some reason, you can use Spring Retry to retry the same operation. This provides another way to make your service more available.

Retry makes your application more robust and less prone to failures. You can either configure Spring Retry on a method that you think can fail or you can configure a RetryTemplate. The ease of configuration makes Spring Retry an easier choice when writing code.

On other hand, the Resilience4j Retry module offers an equally easier configuration – either through code or through properties.

In this post, I will show how to use Spring Retry and Resilience4j Retry modules when calling any methods or services.

When to use Retry?

Usually, you should consider Retry operation in certain scenarios.

  1. HTTP call to a REST Endpoint
  2. Sending or retrieving messages from SQS
  3. Remote Procedure call or a web service
  4. Fetching or storing data from databases

In such cases, we can either throw an error if we fail to do the operation successfully. But with the availability of applications becoming more important, most of the time, these errors are trivial and most services come back online within a few milliseconds to seconds.

Therefore, it makes sense to apply retry. You must be careful that the operation that you are applying retry with must be idempotent. Suppose, your application sent a request and the target service received the request, but in between something happened and your target service couldn’t respond in time. Then, with retry, the target service should not treat the retry attempt as a separate or new request. This makes your system more resilient.

Spring Retry

In this section, I will show various ways to use Spring Retry. To start with, we will have a simple Spring Boot REST application to retrieve a list of companies from the database. As usual, I will not show how to build a Spring Boot application.

Gradle Dependencies

To use Spring Retry, we need two dependencies in our configuration.

        implementation 'org.springframework.retry:spring-retry:1.3.1'
	implementation 'org.springframework:spring-aspects:5.3.5'

EnableRetry Annotation

Once, we have spring-retry dependency, we will be able to annotate our main class with annotation @EnableRetry as follows:


package com.betterjavacode.retrydemo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.retry.annotation.EnableRetry;
import org.springframework.retry.backoff.FixedBackOffPolicy;
import org.springframework.retry.policy.SimpleRetryPolicy;
import org.springframework.retry.support.RetryTemplate;


@SpringBootApplication
@EnableRetry
@EnableJpaRepositories(basePackages = "com.betterjavacode.retrydemo.daos")
public class RetrydemoApplication {

	public static void main(String[] args) {
		SpringApplication.run(RetrydemoApplication.class, args);
	}

	@Bean
	public RetryTemplate retryTemplate()
	{
		RetryTemplate retryTemplate = new RetryTemplate();

		FixedBackOffPolicy backOffPolicy = new FixedBackOffPolicy();
		backOffPolicy.setBackOffPeriod(100);

		SimpleRetryPolicy simpleRetryPolicy = new SimpleRetryPolicy();
		simpleRetryPolicy.setMaxAttempts(2);

		retryTemplate.setRetryPolicy(simpleRetryPolicy);
		retryTemplate.setBackOffPolicy(backOffPolicy);
		return retryTemplate;
	}


}

I will explain the rest of the code as we go along, but just note here the annotation @EnableRetry. This will enable the retry in our application.

REST Controller

We will show Spring Retry in two different ways.

  1. Using @Retryable annotation
  2. Using RetryTemplate

Our REST Controller will fetch us a list of companies, a company by id, or a list of companies by name. It will look like below:


package com.betterjavacode.retrydemo.controllers;

import com.betterjavacode.retrydemo.dtos.CompanyDto;
import com.betterjavacode.retrydemo.service.CompanyService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.util.List;

@RestController
@RequestMapping("/v1/betterjavacode/companies")
public class CompanyController
{
    @Autowired
    CompanyService companyService;

    @GetMapping
    public ResponseEntity<List> getAllCompanies()
    {
        List companyDtos = companyService.getAllCompanies();

        if(companyDtos.isEmpty())
        {
            return new ResponseEntity<>(HttpStatus.NO_CONTENT);
        }

        return new ResponseEntity<>(companyDtos, HttpStatus.OK);
    }

    @GetMapping("/{id}")
    public ResponseEntity getCompanyById(@PathVariable("id") long id)
    {
        CompanyDto companyDto = companyService.getCompany(id);
        if(companyDto == null)
        {
            return new ResponseEntity<>(HttpStatus.NO_CONTENT);
        }
        return new ResponseEntity<>(companyDto, HttpStatus.OK);
    }

    @GetMapping("/")
    public ResponseEntity<List> searchCompanies(@RequestParam("name") String companyName)
    {
        List companyDtos = companyService.searchCompanyByName(companyName);
        if(companyDtos.isEmpty())
        {
            return new ResponseEntity<>(HttpStatus.NO_CONTENT);
        }

        return new ResponseEntity<>(companyDtos, HttpStatus.OK);
    }
}

In our controller, we are using a @Service object called CompanyService. This service object provides us with a way to implement our methods to fetch company data.

Service with various Retry Configuration

So, we will see how we can use annotation @Retryable:


    @Retryable(value = SQLException.class, maxAttempts = 2, backoff = @Backoff(delay = 100))
    public List getAllCompanies()
    {
        List companies =  companyRepository.findAll();
        List companyDtos = new ArrayList<>();
        for(Company company : companies)
        {
            CompanyDto companyDto = new CompanyDto(company.getName(), company.getType(),
                    company.getCity(), company.getState(), company.getDescription());
            companyDtos.add(companyDto);
        }

        return companyDtos;
    }

In the above code, we are fetching a list of companies. If this method fails to fetch the result with any exception related to SQLException, we will retry the fetching. We will retry this twice as configured with maxAttempts. Between each attempt, there will be a delay of 100 milliseconds. Now if we run our application and call this method, we will see how this retry works.

To simulate the error, I will stop SQL Service from Windows Services. I will show a successful response and a retried response below:

Spring Retry vs Resilience4j Retry - RetryableAs you can see in the above screenshot, there were two attempts to retry. In each retry, it tried to connect to MySQL server thrice.

What is Spring Boot Retry Template?

Similarly, we can also use retry template that Spring-Retry offers. In the following code, I show a method that I have added in CompanyService to get company data for an id.


    public CompanyDto getCompany(long id)
    {
        CompanyDto companyDto = retryTemplate.execute(rt -> {
           Company company = companyRepository.findById(id).get();
           CompanyDto localCompanyDto = new CompanyDto(company.getName(), company.getType(),
                   company.getCity(),
                   company.getState(), company.getDescription());
           return localCompanyDto;
        });

        return companyDto;
    }

This retryTemplate bean is configured with simpleRetryPolicy with 2 attempts and 100 milliseconds delay between each attempt. Nevertheless, if I try to execute this method the same way I did for @Retryable, we will see the below output:

As mentioned above, all I am doing is stopping my MySQL service from windows services and it allows my method to get executed to retry.

Is Retry Template Thread Safe?

Retry Template class is thread-safe. It allows concurrent access. In return, one can execute multiple operations.

Resilience4j Retry

While using resilience4j-retry library, you can register a custom global RetryConfig with a RetryRegistry builder. Use this registry to build a Retry. In our demo to fetch company data, we added a new method to retrieve companies by name.

This method will look like below:


    public List searchCompanyByName(String name)
    {
        LOGGER.info("Search for company = {}", name);

        RetryConfig retryConfig =
                RetryConfig.custom().maxAttempts(4).waitDuration(Duration.of(2, SECONDS)).build();         

        RetryRegistry retryRegistry = RetryRegistry.of(retryConfig);

        Retry retryConfiguration = retryRegistry.retry("companySearchService", retryConfig);

        Supplier<List> companiesSupplier = () -> companyRepository.findAllByName(name);

        Supplier<List> retryingCompaniesSearch =
                Retry.decorateSupplier(retryConfiguration, companiesSupplier);

        List companyDtos = new ArrayList<>();
        List companies = retryingCompaniesSearch.get();
        LOGGER.info("Retrying..");
        for(Company company : companies)
        {
            CompanyDto companyDto = new CompanyDto(company.getName(), company.getType(),
                    company.getCity(), company.getState(), company.getDescription());
            companyDtos.add(companyDto);
        }

        return companyDtos;
    }

In the above method, we first create RetryConfig. We create a RetryRegistry and add RetryConfig in this registry. Then when we create our call to fetch a list of companies. We decorate this call with retryConfiguration.

Customizations with Resilience4j-Retry

RetryConfig offers different customization:

  1. maxAttempts – 3 is the default number of attempts for retries.
  2. waitDuration – a fixed wait duration between each retry attempt.
  3. intervalFunction – a function to modify the waiting interval after a failure.
  4. retryOnResultPredicate – configures a predicate that evaluates if a result should be retried.
  5. retryExceptions – Configures a list of throwable classes that are used for retrying
  6. ignoreExceptions – Configures a list of throwable classes that are ignored
  7. failAfterMaxRetries – A boolean to enable or disable throwing of MaxRetriesExceededException when the Retry has reached the configured maxAttempts

Demo

Now, let’s look at what happens when we execute this method with resilience4j-retry. The following screenshot shows the successful response when SQL service is still running.

If I stop SQL service, we will see the retry attempts 4 times as we have configured it for 4.

Code

The code for this demo can be found in my github repository.

Conclusion

In this post, I showed the comparison between Spring Retry vs Resilience4j Retry. When to use either of these libraries depends on your scenario. Usually, Resilience4j Retry goes well if you also plan to resilience4j circuit breaker module. Spring Retry can be handy with various configurations as well using RetryTemplate.

If you enjoyed this post, please subscribe to my blog here.

How to Deploy Spring Boot Application on AWS ECS

In this post, I will show how we can deploy a spring boot application on AWS ECS (Elastic Container Service). ECS is one of the amazon web services that is mainly used to run the applications. It is an alternative to use EC2 instances directly.

What is ECS?

ECS is a container orchestration service. ECS allows you to run your container. You can also easily build ECS cluster using AWS Farget. Farget removes the need to provision and manages a server. The advantage of using ECS is that you don’t have to decide which server EC2 instance to use, the service does that for you. It also improves security through application isolation.

As part of this demo, I will show step by step how you can deploy your spring boot application on the ECS cluster with AWS Fargate.

Spring Boot Application

I will not be showing you how to build a Spring Boot application, but you can visit my other posts about it. In this application, we will have a controller that will show a to-do list. We will also have form-based authentication using the MySQL database. The config method in security config will look like below:


    @Override
    protected void configure(HttpSecurity httpSecurity) throws Exception
    {
        httpSecurity
                .authorizeRequests()
                .antMatchers("/js/**","/css/**","/img/**").permitAll()
                .antMatchers("/signup","/forgotpassword")
                .permitAll()
                .anyRequest().authenticated()
                .and()
                .formLogin()
                .loginPage("/login")
                .permitAll()
                .and()
                .logout()
                .logoutUrl("/logout")
                .logoutSuccessHandler(logoutSuccessHandler)
                .deleteCookies("JSESSIONID")
                .permitAll()
                .and()
                .csrf();

    }

So, as you see above, we will allow anyone to access signup and forgotpassword pages.  Equally, anyone can access the login page. We will use our custom login form located on the login page.

If the user successfully logs in, the user will see a static TO-DO list page showing TO-DO list.

As part of this demo, we will also be creating a MySQL database in AWS RDS. Our application running in AWS Fargate ECS cluster will access this MySQL database for storing users and user authentication.

Building a docker container for Spring Boot Application

We will create a docker file.

FROM openjdk:8-jdk-alpine
COPY ./build/libs/todolist-0.0.1-SNAPSHOT.war todolist-0.0.1-SNAPSHOT.war
ENTRYPOINT ["java", "-jar","todolist-0.0.1-SNAPSHOT.war"]

Basically, we pull Java 8 image and copy the war file from our project into the docker instance. We also define the entry-point.

Command to create a docker image

docker build -t todolist .

If you want to run your application through docker locally, you can use the following command:

docker run -p 8743:8743 todolist

So to deploy our application on ECS, we will need to push this docker image to the elastic container repository (ECR).

Usually building a docker image and pushing it to ECR is all part of CI/CD. I will not be covering CI/CD in this post.

To push this docker image to ECR,

  1. You will need aws-cli tools installed on your machine.
  2. Create a repository in ECR

Now on the command line, execute this command to authenticate docker client to ECR repository

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin XXXXXXX.dkr.ecr.us-east-1.amazonaws.com

Once we are authenticated, we can tag and push docker images to the ECR repository.

  1. Tag the docker image – docker tag IMAGEID XXXXXX.dkr.ecr.us-east-1.amazonaws.com/todolist
  2. Push the docker image – docker push XXXXX.dkr.ecr.us-east-1.amazonaws.com/todolist

Now our docker image is in ECR.

Deploying docker image in ECS

Before we can deploy our docker image in ECS, we will have to execute three steps.

  1. Create a database instance of MySQL in AWS RDS. The configuration to create this instance will look like below:
  2. Once you create the database, you will have database instance server and port available that you can use to connect to database server either through Database Administration UI or command line. Create a database simplifyingspringsecurity and database table users.
  3. Now let’s move to ECS service and create a task definition for AWS Fargate. It will look like below: Also remember to create a IAM role beforehand which should have a permission policy `AmazonECSTaskExecutionRolePolicy` . In the same task definition, we will have to add our container and the properties for this container. They will look like below: We have mapped two ports 8743 and 80 from our host to the container. In the same container, also add environment properties if you want to override properties for your data source. These properties include spring.datasource.url, spring.datasource.username, spring.datasource.password, spring.datasource.driver-class-name, and spring.jpa.properties.hibernate.dialect.  That should cover our configuration for the task.
  4. Ultimately, we will create an ECS cluster which we will use to run our task. As you see, we have chosen a AWS Fargate based cluster. On the next step you provide a name for the cluster and click “Create” and that should create a cluster.
  5. Once the cluster is configured and running, we can go on the tab “Tasks” and click “Run new task”. Here we will provide few more configurations that will allow us to run our task in a way so that we can access the application once the task starts running. We have security groups defined here, but it also provides the option to create a new security group when running. In this security group, you should choose which type of protocol and ports you want to allow for access. In this case, we want our port 8743 for internet and TCP. Now click ‘Run Task’ and that should start the task. Once the task starts, we will be able to see the logs from cloud watch as below: On this basis, if you look at logs, now our application is running successfully on port 8743. Now we can access the application using the public IP that the Fargate task provided. Here we go, we have our application up and running.

Learnings

ECS can be complicated with the number of configurations a user can do when deploying an application. To keep this simple, just think this way AWS Cloud -> VPC -> Subnet -> ECS -> Fargate -> Task.

Conclusion

In this post, I showed how to deploy a Spring Boot Application on AWS ECS. As much as it seems straightforward, the issue comes up with the combination of configurations you can do in AWS ECS. Once you understand how the cloud handles private networks, it becomes simpler.

User Management with Okta SDK and Spring Boot

In this post, I will show how we can build user management and authentication with Okta SDK and Spring Boot.

Introduction

As part of any application, developers have to be careful how they build authentication. Despite we are using Form-Based authentication for a long time, it is not the most secure one. In this post, I plan to show Form-Based authentication where users are not necessarily authenticated by validating their encrypted password against the password stored in a database. If you want to learn more about Spring Security with different authentication flows, I recently released a book Simplifying Spring Security. You can buy the book here.

Okta is an identity provider. It’s an application that provides user management and authentication with different protocols.

Okta SDK APIs

Okta offers two libraries okta-sdk-java and okta-auth-java for user management APIs and authentication.

Are these libraries right for you? This depends on your use case. Okta also offers okta-spring-boot-starter library to use okta for different OAuth flows in your Spring Boot Application. We will not be using this library in this demo.

You can find more details about these libraries here and here.

Include these libraries in your project as follows:

 


	implementation 'com.okta.authn.sdk:okta-authn-sdk-api:2.0.1'
	runtimeOnly 'com.okta.authn.sdk:okta-authn-sdk-impl:2.0.1'
	runtimeOnly 'com.okta.sdk:okta-sdk-httpclient:3.0.1'

User Management with Okta SDK in Spring Boot Application

In this demo, I have a sample application of To-Do List. When a user launches the application, user will see a login screen. It has Sign up option. If the user doesn’t exist in the application, the user will have to create an account.

On Sign-up page, when a user enters the “Submit” button, we will save the user in our database and then call Okta SDK API to create the user on Okta side.

To achieve this, we need Okta Client.


    @Bean
    public Client client()
    {

        Client clientConfig =
                Clients.builder().setOrgUrl("https://oktadomainurl").setClientCredentials(new TokenClientCredentials(secret))
                        .build();


        return clientConfig;

    }

As you see above, we are creating a client that we will use to call Okta API. The `secret` is the API token you will be able to find in Okta admin UI. If you don’t find it, either you don’t have admin privileges or you have not created the token yet. There is another way to create this client with an access token.


    @Bean
    public Client client()
    {

        Client clientConfig =
                Clients.builder().setOrgUrl("https://oktadomainurl")
                      .setAuthorizationMode(AuthorizationMode.PRIVATE_KEY).setClientId("{clientId}")
                      .setScopes(new HashSet<>(Arrays.asList("okta.users.read", "okta.apps.read")))
                      .setPrivateKey("/path/to/yourPrivateKey.pem")


        return clientConfig;

    }

The advantage of this Client configuration is that you don’t need to know API access token created based on admin privileges.

Now on my Controller side, I will use this client to create user in Okta as below:

 


        UserDto userDto = new UserDto();
        userDto.setEmail(email);
        userDto.setFirstName(firstname);
        userDto.setLastName(lastname);
        userDto.setPassword(encodedPassword);
        userDto.setRole("ADMIN");
        userDto.setEnabled(true);

        UserDto returnedUser = usersManager.createUser(userDto);

        LOGGER.info("Create the user in Okta");

        User oktaUser = UserBuilder.instance().setEmail(returnedUser.getEmail())
                .setFirstName(returnedUser.getFirstName())
                .setLastName(returnedUser.getLastName())
                .buildAndCreate(client);

That covers the user management part. You can similarly call GET or DELETE API to manage users.

User Authentication

Now comes the critical part of authentication. In many enterprise applications, when using third-party identity provides, the trouble always comes with user data synchronization. Both applications need to store user data.

For authentication, we will need authenticationClient bean. This client will allow us to call Okta API for authentication.


    @Bean
    public AuthenticationClient authenticationClient()
    {
        AuthenticationClient authenticationClient =
                AuthenticationClients.builder()
                        .setOrgUrl("https://oktadomainurl")
                        .build();

        return authenticationClient;
    }

In our security config, I will override the form-based login with a custom login page.



    @Autowired
    private CustomAuthenticationProvider customAuthenticationProvider;

    @Bean(BeanIds.AUTHENTICATION_MANAGER)
    @Override
    public AuthenticationManager authenticationManagerBean() throws Exception
    {
        return super.authenticationManagerBean();
    }


    @Override
    protected void configure(HttpSecurity httpSecurity) throws Exception
    {

        httpSecurity.authorizeRequests()
                .antMatchers("/js/**","/css/**","/img/**").permitAll()
                .antMatchers("/signup","/forgotpassword").permitAll()
                .anyRequest().authenticated()
                .and()
                .formLogin()
                .loginPage("/login")
                .permitAll();

    }

As you see in above code, I am using customAuthenticationProvider, this provider will use authenticationClient to authentcate with Okta. This AuthenticationProvider will look like below:


package com.betterjavacode.sss.todolist.clients;

import com.betterjavacode.sss.todolist.security.AuthenticationStateHandler;
import com.okta.authn.sdk.client.AuthenticationClient;
import com.okta.authn.sdk.resource.AuthenticationResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.security.authentication.AuthenticationProvider;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.AuthenticationException;
import org.springframework.security.core.GrantedAuthority;
import org.springframework.security.core.authority.SimpleGrantedAuthority;
import org.springframework.security.core.userdetails.User;
import org.springframework.security.core.userdetails.UserDetails;
import org.springframework.stereotype.Component;

import java.util.ArrayList;
import java.util.List;

@Component
public class CustomAuthenticationProvider implements AuthenticationProvider
{

    private static final Logger LOGGER = LoggerFactory.getLogger(CustomAuthenticationProvider.class);

    @Autowired
    private AuthenticationClient authenticationClient;

    @Autowired
    private AuthenticationStateHandler authenticationStateHandler;

    @Override
    public Authentication authenticate (Authentication authentication) throws AuthenticationException
    {
        String username = authentication.getName();
        String password = authentication.getCredentials().toString();
        String relayState = "/index";
        AuthenticationResponse authnResponse = null;
        try
        {
            LOGGER.info("Going to connect to Okta");
            authnResponse = authenticationClient.authenticate(username, password.toCharArray(),
                    relayState,
                    authenticationStateHandler);
        }
        catch(com.okta.authn.sdk.AuthenticationException e)
        {
            LOGGER.error("Unable to authentcate the user", e);
        }

        if(authnResponse != null)
        {
            final List grantedAuths = new ArrayList<>();
            grantedAuths.add(new SimpleGrantedAuthority("ROLE_ADMIN"));
            final UserDetails principal = new User(username, password, grantedAuths);
            final Authentication authen = new UsernamePasswordAuthenticationToken(principal,
                    password, grantedAuths);
            return authen;
        }
        else
        {
            LOGGER.info("Unable to authenticate");
            return null;
        }

    }

    @Override
    public boolean supports (Class<?> authentication)
    {
        return true;
    }
}

We use authenticationClient to call authenticate method. AuthenticationStateHandler basically handles the status authentication. The implementation of this handle is as below:


package com.betterjavacode.sss.todolist.security;

import com.okta.authn.sdk.AuthenticationStateHandlerAdapter;
import com.okta.authn.sdk.resource.AuthenticationResponse;
import com.okta.commons.lang.Strings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

@Component
public class AuthenticationStateHandler extends AuthenticationStateHandlerAdapter
{
    private static final Logger LOGGER = LoggerFactory.getLogger(AuthenticationStateHandler.class);

    @Override
    public void handleUnknown (AuthenticationResponse unknownResponse)
    {
        // TO DO
    }

    @Override
    public void handleSuccess (AuthenticationResponse successResponse)
    {
        if (Strings.hasLength(successResponse.getSessionToken()))
        {
            LOGGER.info("Login successful");
            String relayState = successResponse.getRelayState();
            String dest = relayState != null ? relayState : "/";

        }
    }
}

That’s all. This covers user authentication. Remember this is still form based authentication where you are entering user credentials on your custom login page and behind the screen calling Okta API to authenticate.

In my book, Simplifying Spring Security, I have also added the demo for login with Okta OAuth.

Conclusion

In this post, I showed how to use Okta SDK for user management and authentication with the Spring Boot application. If you have any questions, feel free to send me an email by subscribing to my blog here.

 

How to Use Circuit Breaker in Spring Boot Application

In this post, I will show how we can use the Circuit Breaker pattern in a Spring Boot Application. When I say Circuit Breaker pattern, it is an architectural pattern. Netflix had published a library Hysterix for handling circuit breakers.  As part of this post, I will show how we can use a circuit breaker pattern using the resilence4j  library in a Spring Boot Application.

In other news, I recently released my book Simplifying Spring Security. If you are interested to learn about Spring Security, you can buy it here.

Image from Pixabay – By Jürgen Diermaier

What is Circuit Breaker?

The concept of Circuit Breaker comes from Electrical Engineering. In most electricity networks, circuit breakers are switches that protect the network from damage caused by an overload of current or short circuits.

Similarly, in software, a circuit breaker stops the call to a remote service if we know the call to that remote service is either going to fail or time out. The advantage of this is to save resources and be proactive in our troubleshooting of the remote procedure calls.

The circuit breaker makes the decision of stopping the call based on the previous history of the calls. But there are alternative ways how it can handle the calls. Usually, it will keep track of previous calls. Suppose 4 out of 5 calls have failed or timed out, then the next call will fail. This helps to be more proactive in handling the errors with the calling service and the caller service can handle the response in a different way, allowing users to experience the application differently than an error page.

Another way a circuit breaker can act is if calls to remote service are failing in particular time duration.  A circuit breaker will open and will not allow the next call till remote service improves on error.

Resilience4J Library

We have our code which we call remote service. The circuit breaker module from resilience4j library will have a lambda expression for a call to remote service OR a supplier to retrieve values from the remote service call. I will show this as part of the example. The circuit breaker decorates this remote service call in such a way that it can keep track of responses and switch states.

Different configurations of Resilience4j Library

To understand the circuit breaker concept, we will look at different configurations this library offers.

slidingWindowType() – This configuration basically helps in making a decision on how the circuit breaker will operate. There are two types COUNT_BASED and TIME_BASED. COUNT_BASED circuit breaker sliding window will take into account the number of calls to remote service while TIME_BASED circuit breaker sliding window will take into account the calls to remote service in certain time duration.

failureRateThreshold() – This configures the failure rate threshold in percentage. If x percentage of calls are failing, then the circuit breaker will open.

slidingWindowSize() – This setting helps in deciding the number of calls to take into account when closing a circuit breaker.

slowCallRateThreshold() – This configures the slow call rate threshold in percentage. If x percentage of calls are slow, then the circuit breaker will open.

slowCallDurationThreshold – Time duration threshold about which calls are considered slow.

minimumNumberOfCalls() – A minimum number of calls required before which circuit breaker can calculate the error rate.

ignoreException() – This setting allows you to configure an exception that a circuit breaker can ignore and will not count towards the success or failure of a call of remote service.

waitDurationInOpenState() – Duration for which the circuit breaker should remain in the open state before transitioning into a half-open state.  The default value is 60 seconds.

Count-Based Circuit Breaker

While using resilience4j library, one can always use the default configurations that the circuit breaker offers. Default configurations are based on the COUNT-BASED sliding window type.

So how do we create a circuit breaker for the COUNT-BASED sliding window type?


      CircuitBreakerConfig circuitBreakerConfig = CircuitBreakerConfig.custom()
                .slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
                .slidingWindowSize(10)
                .slowCallRateThreshold(65.0f)
                .slowCallDurationThreshold(Duration.ofSeconds(3))
                .build();

        CircuitBreakerRegistry circuitBreakerRegistry =
                CircuitBreakerRegistry.of(circuitBreakerConfig);

        CircuitBreaker cb = circuitBreakerRegistry.circuitBreaker("BooksSearchServiceBasedOnCount");

In the above example, we are creating a circuit breaker configuration that includes a sliding window of type COUNT_BASED. This circuit breaker will record the outcome of 10 calls to switch the circuit-breaker to the closed state.  If 65 percent of calls are slow with slow being of a duration of more than 3 seconds, the circuit breaker will open.

CircuitBreakerRegistry is a factory to create a circuit breaker.

Time-Based Circuit Breaker

Now on Time-Based circuit breaker.


       CircuitBreakerConfig circuitBreakerConfig = CircuitBreakerConfig.custom()
                .slidingWindowType(CircuitBreakerConfig.SlidingWindowType.TIME_BASED)
                .minimumNumberOfCalls(3)
                .slidingWindowSize(10)
                .failureRateThreshold(70.0f)
                .build();

        CircuitBreakerRegistry circuitBreakerRegistry =
                CircuitBreakerRegistry.of(circuitBreakerConfig);

        CircuitBreaker cb = circuitBreakerRegistry.circuitBreaker("BookSearchServiceBasedOnTime");

In the above example, we are creating a circuit breaker configuration that includes a sliding window of type TIME_BASED. Circuit breaker will record the failure of calls after a minimum of 3 calls.  If 70 percent of calls fail, the circuit breaker will open.

Example of Circuit Breaker in Spring Boot Application

We have covered the required concepts about the circuit breaker. Now, I will show we can use a circuit breaker in a Spring Boot application.

On one side, we have a REST application BooksApplication that basically stores details of library books. On the other side, we have an application Circuitbreakerdemo that calls the REST application using RestTemplate. We will decorate our REST call through the circuit breaker.

BooksApplication stores information about books in a MySQL database table librarybooks. The REST Controller for this application has GET and POST methods.


package com.betterjavacode.books.controllers;

import com.betterjavacode.books.daos.BookDao;
import com.betterjavacode.books.models.Book;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

@CrossOrigin("https://localhost:8443")
@RestController
@RequestMapping("/v1/library")
public class BookController
{
    @Autowired
    BookDao bookDao;

    @GetMapping("/books")
    public ResponseEntity<List> getAllBooks(@RequestParam(required = false) String bookTitle)
    {
        try
        {
            List listOfBooks = new ArrayList<>();
            if(bookTitle == null || bookTitle.isEmpty())
            {
                bookDao.findAll().forEach(listOfBooks::add);
            }
            else
            {
                bookDao.findByTitleContaining(bookTitle).forEach(listOfBooks::add);
            }

            if(listOfBooks.isEmpty())
            {
                return new ResponseEntity<>(HttpStatus.NO_CONTENT);
            }

            return new ResponseEntity<>(listOfBooks, HttpStatus.OK);
        }
        catch (Exception e)
        {
            return new ResponseEntity<>(null, HttpStatus.INTERNAL_SERVER_ERROR);
        }
    }

    @GetMapping("/books/{id}")
    public ResponseEntity getBookById(@PathVariable("id") long id)
    {
        try
        {
            Optional bookOptional = bookDao.findById(id);

            return new ResponseEntity<>(bookOptional.get(), HttpStatus.OK);
        }
        catch (Exception e)
        {
            return new ResponseEntity<>(null, HttpStatus.INTERNAL_SERVER_ERROR);
        }
    }

    @PostMapping("/books")
    public ResponseEntity addABookToLibrary(@RequestBody Book book)
    {
        try
        {
            Book createdBook = bookDao.save(new Book(book.getTitle(), book.getAuthor(),
                    book.getIsbn()));
            return new ResponseEntity<>(createdBook, HttpStatus.CREATED);
        }
        catch (Exception e)
        {
            return new ResponseEntity<>(null, HttpStatus.INTERNAL_SERVER_ERROR);
        }
    }

    @PutMapping("/books/{id}")
    public ResponseEntity updateABook(@PathVariable("id") long id, @RequestBody Book book)
    {
        Optional bookOptional = bookDao.findById(id);

        if(bookOptional.isPresent())
        {
            Book updatedBook = bookOptional.get();
            updatedBook.setTitle(book.getTitle());
            updatedBook.setAuthor(book.getAuthor());
            updatedBook.setIsbn(book.getIsbn());
            return new ResponseEntity<>(bookDao.save(updatedBook), HttpStatus.OK);
        }
        else
        {
            return new ResponseEntity<>(HttpStatus.NOT_FOUND);
        }
    }

    @DeleteMapping("/books/{id}")
    public ResponseEntity deleteABook(@PathVariable("id") long id)
    {
        try
        {
            bookDao.deleteById(id);
            return new ResponseEntity<>(HttpStatus.NO_CONTENT);
        }
        catch (Exception e)
        {
            return new ResponseEntity<>(HttpStatus.INTERNAL_SERVER_ERROR);
        }
    }
}

On the other side, our application Circuitbreakerdemo has a controller with thymeleaf template so a user can access the application in a browser.

For the demo purpose, I have defined CircuitBreaker in a separate bean that I will use in my service class.


    @Bean
    public CircuitBreaker countCircuitBreaker()
    {
        CircuitBreakerConfig circuitBreakerConfig = CircuitBreakerConfig.custom()
                .slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
                .slidingWindowSize(10)
                .slowCallRateThreshold(65.0f)
                .slowCallDurationThreshold(Duration.ofSeconds(3))
                .build();

        CircuitBreakerRegistry circuitBreakerRegistry =
                CircuitBreakerRegistry.of(circuitBreakerConfig);

        CircuitBreaker cb = circuitBreakerRegistry.circuitBreaker("BooksSearchServiceBasedOnCount");

        return cb;
    }

    @Bean
    public CircuitBreaker timeCircuitBreaker()
    {
        CircuitBreakerConfig circuitBreakerConfig = CircuitBreakerConfig.custom()
                .slidingWindowType(CircuitBreakerConfig.SlidingWindowType.TIME_BASED)
                .minimumNumberOfCalls(3)
                .slidingWindowSize(10)
                .failureRateThreshold(70.0f)
                .build();

        CircuitBreakerRegistry circuitBreakerRegistry =
                CircuitBreakerRegistry.of(circuitBreakerConfig);

        CircuitBreaker cb = circuitBreakerRegistry.circuitBreaker("BookSearchServiceBasedOnTime");
        return cb;
    }

I have defined two beans one for the count-based circuit breaker and another one for time-based.

The BookStoreService will contain a calling BooksApplication and show books that are available. This service will look like below:


@Controller
public class BookStoreService
{

    private static final Logger LOGGER = LoggerFactory.getLogger(BookStoreService.class);

    @Autowired
    public BookManager bookManager;

    @Autowired
    private CircuitBreaker countCircuitBreaker;

    @RequestMapping(value = "/home", method= RequestMethod.GET)
    public String home(HttpServletRequest request, Model model)
    {
        return "home";
    }

    @RequestMapping(value = "/books", method=RequestMethod.GET)
    public String books(HttpServletRequest request, Model model)
    {
        Supplier<List> booksSupplier =
                countCircuitBreaker.decorateSupplier(() -> bookManager.getAllBooksFromLibrary());

        LOGGER.info("Going to start calling the REST service with Circuit Breaker");
        List books = null;
        for(int i = 0; i < 15; i++)
        {
            try
            {
                LOGGER.info("Retrieving books from returned supplier");
                books = booksSupplier.get();
            }
            catch(Exception e)
            {
                LOGGER.error("Could not retrieve books from supplier", e);
            }
        }
        model.addAttribute("books", books);

        return "books";
    }
}

So when the user clicks on the books page, we retrieve books from our BooksApplication REST Service.

I have autowired the bean for countCircuitBreaker. For demo purposes – I will be calling the REST service 15 times in a loop to get all the books. This way, I can simulate interruption on my REST service side.

Our circuit breaker decorates a supplier that does REST call to remote service and the supplier stores the result of our remote service call.

In this demo, we are calling our REST service in a sequential manner, but remote service calls can happen parallelly also. The circuit breaker will still keep track of results irrespective of sequential or parallel calls.

Demo

Let’s look at how the circuit breaker will function in a live demo now. My REST service is running on port 8443 and my Circuitbreakerdemo application is running on port 8743.

Initially, I start both of the applications and access the home page of Circuitbreakerdemo application. The home page contains the link for viewing all the books from the store.

Now to simulate some errors, I have added the following code in my RestTemplate call that basically sleeps for 3 seconds before returning the result of the REST call.


    public List getAllBooksFromLibrary ()
    {
        HttpHeaders httpHeaders = new HttpHeaders();
        httpHeaders.setContentType(MediaType.APPLICATION_JSON);

        ResponseEntity<List> responseEntity;
        long startTime = System.currentTimeMillis();
        LOGGER.info("Start time = {}", startTime);
        try
        {
            responseEntity= restTemplate.exchange(buildUrl(),
                    HttpMethod.GET, null, new ParameterizedTypeReference<List>()
                    {});
            if(responseEntity != null && responseEntity.hasBody())
            {
                Thread.sleep(3000);
                LOGGER.info("Total time to retrieve results = {}",
                        System.currentTimeMillis() - startTime);
                return responseEntity.getBody();
            }
        }
        catch (URISyntaxException | InterruptedException e)
        {
            LOGGER.error("URI has a wrong syntax", e);
        }

        LOGGER.info("No result found, returning an empty list");
        return new ArrayList<>();
    }

In short, my circuit breaker loop will call the service enough times to pass the threshold of 65 percent of slow calls that are of duration more than 3 seconds. Once I click on the link for here, I will receive the result, but my circuit breaker will be open and will not allow future calls till it is in either half-open state or closed state.

You will notice that we started getting an exception CallNotPermittedException when the circuit breaker was in the OPEN state. Also, the circuit breaker was opened when the 10 calls were performed. This is because our sliding window size is 10.

Another way, I can simulate the error by shutting down my REST service or database service. That way REST calls can take longer than required.

Now, let’s switch the COUNT_BASED circuit breaker to TIME_BASED circuit breaker. In TIME_BASED circuit breaker, we will switch off our REST service after a second, and then we will click on here link from the home page. If 70 percent of calls in the last 10 seconds fail, our circuit breaker will open.

Since REST Service is closed, we will see the following errors in Circuitbreakdemo application

We will see the number of errors before the circuit breaker will be in OPEN state.

One configuration we can always add how long we want to keep the circuit breaker in the open state.  For the demo, I have added the circuit breaker will be in an open state for 10 seconds.

How to handle OPEN circuit breakers?

One question arises, how do you handle OPEN circuit breakers? Luckily, resilience4j offers a fallback configuration with Decorators utility. In most cases, you can always configure this to get the result from previous successful results so that users can still work with the application.

Conclusion

In this post, I have covered how to use a circuit breaker in a Spring Boot application. The code for this demo is available here.

In this demo, I have not covered how to monitor these circuit breaker events as resilience4j the library allows storing these events with metrics that one can monitor with a monitoring system.

If you enjoyed this post, consider subscribing to my blog here.

References

  1. Resilience4J Library – Resilience4J
  2. Circuit Breaker with Resilience4j – Circuit Breaker

Step by Step Spring Batch Tutorial

In this post, I want to show how you can use Spring Batch. This is a step by step Spring Batch Tutorial.

In enterprise applications, batch processing is common. But with data becoming more prevalent on the internet, it has also become important how we process this data. There are multiple solutions available. Apache Storm or Apache Spark helps with processing and transforming the data in the required format. In this post, we will be looking at Spring Batch more closely.

What is Spring Batch?

Spring Batch is a lightweight framework designed to facilitate batch processing. It allows developers to create batch applications. In turn, these batch applications process the incoming data and transform it for further usage.

Another big advantage of using the Spring Batch is that it allows for high-performance processing of this data. The applications that rely upon data heavily, it is of utmost importance that data becomes instantly available.

Spring Batch allows a developer to use POJO based approach. In this approach, a developer can transform the batch-processed data into data models that she can further use for application business logic.

In this post, I will cover an example where we will batch process a data-intensive CSV file for employee records and transform, and validate that data to load into our database.

What is Batch Processing?

Batch processing is a data processing mode. It involves consuming all the data, processing that data, transforming it, and then sending it to another data source. Usually, this is done through an automated job. Either a triggering system or a user triggers a job and that job processes the job definition. Job definition will be about consuming the data from its source.

The key advantage of batch processing is it handles a large volume of data. Nevertheless, this operation can be asynchronous. Most applications perform batch processing separately from real-time user interaction.

Next, we will learn about the Spring Batch framework and what it comprises.

Spring Batch Framework

The following architecture shows the components of the Spring Batch framework.

First, the batch process involves a job. User schedules a job to be run at a certain time or based on a certain condition. This can also involve a job trigger.

Spring Batch framework also includes

  • logging and tracing
  • transaction management
  • job processing statistics
  • job restart
  • resource management

Usually, when you configure a job, it will be saved in the job repository. Job Repository keeps the metadata information of all the jobs. A trigger starts these jobs at their scheduled time.

A job launcher is an interface to launch a job or runs a job when the jobs’ scheduled time arrives.

Job is defined with job parameters. When a job starts, a job instance runs for that job. Every execution of job instance has job execution and it keeps track status of the job. A job can have multiple steps.

Step is an independent phase of a job. A job can be comprised of more than one step. Similar to the job, each step has step execution that executes the step and keeps track of the status of the step.

Each step has an item reader that basically reads the input data, an item processor that processes the data and transforms it, and an item writer that takes the processed data and output it.

Now, let’s see all these components in our demo.

Step by Step Spring Batch Tutorial with an example

As part of the demo, we will be uploading a csv file through Spring Batch Framework. So to start with, create the spring project and add the following dependency:

implementation 'org.springframework.boot:spring-boot-starter-batch'

This is the main dependency of our project. Also our main application will look like below:


package com.betterjavacode.springbatchdemo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;


@SpringBootApplication
public class SpringbatchdemoApplication
{

	public static void main(String[] args)
	{
		SpringApplication.run(SpringbatchdemoApplication.class, args);
	}

}

Create DTO Object

I will be uploading employee data through a CSV file, so I will have my DTO object for Employee created as below:


package com.betterjavacode.springbatchdemo.dtos;

import com.betterjavacode.springbatchdemo.models.Company;
import com.betterjavacode.springbatchdemo.models.Employee;
import com.betterjavacode.springbatchdemo.repositories.CompanyRepository;
import org.springframework.beans.factory.annotation.Autowired;


import java.io.Serializable;

public class EmployeeDto implements Serializable
{
    private static final long serialVersionUID = 710566148641281929L;

    @Autowired
    public CompanyRepository companyRepository;

    private int employeeId;
    private int companyId;
    private String firstName;
    private String lastName;
    private String email;
    private String jobTitle;

    public EmployeeDto()
    {

    }

    public EmployeeDto(int employeeId, String firstName, String lastName, String email,
                        String jobTitle, int companyId)
    {
        this.employeeId = employeeId;
        this.firstName = firstName;
        this.lastName = lastName;
        this.email = email;
        this.jobTitle = jobTitle;
        this.companyId = companyId;
    }

    public Employee employeeDtoToEmployee()
    {
        Employee employee = new Employee();
        employee.setEmployeeId(this.employeeId);
        employee.setFirstName(this.firstName);
        employee.setLastName(this.lastName);
        employee.setEmail(this.email);
        Company company = companyRepository.findById(this.companyId).get();
        employee.setCompany(company);
        employee.setJobTitle(this.jobTitle);
        return employee;
    }

    public int getEmployeeId ()
    {
        return employeeId;
    }

    public void setEmployeeId (int employeeId)
    {
        this.employeeId = employeeId;
    }

    public int getCompanyId ()
    {
        return companyId;
    }

    public void setCompanyId (int companyId)
    {
        this.companyId = companyId;
    }

    public String getFirstName ()
    {
        return firstName;
    }

    public void setFirstName (String firstName)
    {
        this.firstName = firstName;
    }

    public String getLastName ()
    {
        return lastName;
    }

    public void setLastName (String lastName)
    {
        this.lastName = lastName;
    }

    public String getEmail ()
    {
        return email;
    }

    public void setEmail (String email)
    {
        this.email = email;
    }

    public String getJobTitle ()
    {
        return jobTitle;
    }

    public void setJobTitle (String jobTitle)
    {
        this.jobTitle = jobTitle;
    }
}

This DTO class also uses a repository CompanyRepository to get a company object and convert DTO to a database object.

Setting up Spring Batch Configuration

Now, we will set up a batch configuration for our job that will run to upload a CSV file into the database. Our class BatchConfig contain an annotation @EnableBatchProcessing. This annotation enables Spring Batch features and provides a base configuration to set up batch jobs in a @Configuration class.


@Configuration
@EnableBatchProcessing
public class BatchConfig
{

}

This Batch Configuration will include a definition of our job, steps involved in the job. It will also include how we want to read our file data and process it further.


    @Bean
    public Job processJob(Step step)
    {
        return jobBuilderFactory.get("processJob")
                .incrementer(new RunIdIncrementer())
                .listener(listener())
                .flow(step).end().build();
    }

    @Bean
    public Step orderStep1(JdbcBatchItemWriter writer)
    {
        return stepBuilderFactory.get("orderStep1").<EmployeeDto, EmployeeDto> chunk(10)
                .reader(flatFileItemReader())
                .processor(employeeItemProcessor())
                .writer(writer).build();
    }

Above bean declares the job processJob. incrementer adds job parameters. listener will listen to job and handle job status. The bean for listener will handle job completion or job failure notification. As discussed in Spring Batch architecture, every job includes more than one step.

@Bean for step uses stepBuilderFactory to create a step. This step processes a chunk of data in a size of 10. It has a Flat File Reader flatFileItemReader(). A processor employeeItemReader will process the data that has been read by Flat File Item Reader.


    @Bean
    public FlatFileItemReader flatFileItemReader()
    {
        return new FlatFileItemReaderBuilder()
                .name("flatFileItemReader")
                .resource(new ClassPathResource("input/employeedata.csv"))
                .delimited()
                .names(format)
                .linesToSkip(1)
                .lineMapper(lineMapper())
                .fieldSetMapper(new BeanWrapperFieldSetMapper(){{
                    setTargetType(EmployeeDto.class);
                }})
                .build();
    }

    @Bean
    public LineMapper lineMapper()
    {
        final DefaultLineMapper defaultLineMapper = new DefaultLineMapper<>();
        final DelimitedLineTokenizer delimitedLineTokenizer = new DelimitedLineTokenizer();
        delimitedLineTokenizer.setDelimiter(",");
        delimitedLineTokenizer.setStrict(false);
        delimitedLineTokenizer.setNames(format);

        defaultLineMapper.setLineTokenizer(delimitedLineTokenizer);
        defaultLineMapper.setFieldSetMapper(employeeDtoFieldSetMapper);

        return defaultLineMapper;
    }

    @Bean
    public EmployeeItemProcessor employeeItemProcessor()
    {
        return new EmployeeItemProcessor();
    }

    @Bean
    public JobExecutionListener listener()
    {
        return new JobCompletionListener();
    }

    @Bean
    public JdbcBatchItemWriter writer(final DataSource dataSource)
    {
        return new JdbcBatchItemWriterBuilder()
                .itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
                .sql("INSERT INTO employee(employeeId, firstName, lastName, jobTitle, email, " +
                        "companyId) VALUES(:employeeId, :firstName, :lastName, :jobTitle, :email," +
                        " " +
                        ":companyId)")
                .dataSource(dataSource)
                .build();
    }

We will take a look at each of these beans now.

FlatFileItemReader will read the data from the flat file. We are using a FlatFileItemReaderBuilder to create a FlatFileItemReader of type EmployeeDto.

resource indicates the location of the file.

delimited – This builds a delimited tokenizer.

names – will show the order of fields in the file.

lineMapper is an interface to map lines from file to domain object.

fieldSetMapper will map the data from fieldset to an object.

lineMapper bean needs tokenizer and fieldsetmapper.

employeeDtoFieldSetMapper is another bean that we have autowired in this class.

package com.betterjavacode.springbatchdemo.configurations.processor;

import com.betterjavacode.springbatchdemo.dtos.EmployeeDto;
import org.springframework.batch.item.file.mapping.FieldSetMapper;
import org.springframework.batch.item.file.transform.FieldSet;
import org.springframework.stereotype.Component;
import org.springframework.validation.BindException;

@Component
public class EmployeeDtoFieldSetMapper implements FieldSetMapper
{

    @Override
    public EmployeeDto mapFieldSet (FieldSet fieldSet) throws BindException
    {
        int employeeId = fieldSet.readInt("employeeId");
        String firstName = fieldSet.readRawString("firstName");
        String lastName = fieldSet.readRawString("lastName");
        String jobTitle = fieldSet.readRawString("jobTitle");
        String email = fieldSet.readRawString("email");
        int companyId = fieldSet.readInt("companyId");

        return new EmployeeDto(employeeId, firstName, lastName, jobTitle, email, companyId);
    }
}

As you can see, this FieldSetMapper maps fields to individual objects to create an EmployeeDto.

EmployeeItemProcessor implements the interface ItemProcessor. Basically in this class, we validate EmployeeDto data to verify if the company, the employee belongs to, exists.

JobCompletionListener  checks for job completion status.


    @Override
    public void afterJob(JobExecution jobExecution)
    {
        if (jobExecution.getStatus() == BatchStatus.COMPLETED)
        {
            // Log statement
            System.out.println("BATCH JOB COMPLETED SUCCESSFULLY");
        }
    }

Now, let’s look at ItemWriter. This bean basically uses JdbcBatchItemWriter  . JdbcBatchItemWriter uses INSERT sql statement to insert processed EmployeeDto data into the configured data source.

Configuring Application Properties

Before we run our application to process a file, let’s look at application.properties.


spring.datasource.url=jdbc:mysql://127.0.0.1/springbatchdemo?autoReconnect=true&useSSL=false
spring.datasource.username = root
spring.datasource.password=*******
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
spring.datasource.hikari.connection-test-query=SELECT 1
spring.batch.initialize-schema=ALWAYS

Other than regular data source properties, we should understand the property spring.batch.initialize-schema=ALWAYS.  If we don’t use this property and start the application, the application will complain Table batch_job_instance doesn't exist.

To avoid this error, we are basically telling to create batch job-related metadata during startup. This property will create additional database tables in your database like batch_job_execution, batch_job_execution_context, batch_job_execution_params, batch_job_instance etc.

Demo

Now if I execute my Spring Boot Application, it will run and execute the job. There are different ways to trigger a job. In an enterprise application, you will receive a file or data in some kind of storage place (S3 or Amazon SNS-SQS), and you will have a job that will be monitoring this location to trigger the file loading Spring Batch job.

You can see in the execution a message about job completion – “BATCH JOB COMPLETED SUCCESSFULLY“. If we check our database table, we will see the data loaded.

You can download the code for this demo from my github repository.

What more?

I have covered a Spring Batch tutorial here, but this is not all. There is more to Spring Batch than this introductory part. You can have different input data sources or you can also load the data from file to file with various data processing rules.

There are also ways to automate these jobs and process a high volume of data in a performant manner.

Conclusion

In this post, I showed a step by step Spring Batch Tutorial. There are many ways to handle batch jobs, but Spring Batch has made this very easy.

In other news, I recently released my new book – Simplifying Spring Security. If you are looking to learn about Spring Security, you can buy the book here. Accompany this book with this post of  Spring Boot Interview questions and you will be ready for your next job interview.