Building a Kafka Consumer and Producer with Spring Boot: Step-by-Step Example

Sure! Here’s a step-by-step example of setting up a Kafka consumer and producer using Spring Boot:

Step 1: Set up a Spring Boot Project
Create a new Spring Boot project or use an existing one. Include the necessary dependencies in your pom.xml file:

<dependencies>
    <!-- Kafka dependencies -->
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>

    <!-- Spring Boot dependencies -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
</dependencies>

Step 2: Configure Kafka Properties
Add the Kafka broker properties to the application.properties file:

spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=my-group

Step 3: Create a Kafka Consumer
Create a Kafka consumer class that listens for messages on a specific topic:

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class KafkaConsumer {

    @KafkaListener(topics = "my-topic")
    public void consume(String message) {
        System.out.println("Received message: " + message);
        // Add your business logic here
    }
}

Step 4: Create a Kafka Producer
Create a Kafka producer class that sends messages to a specific topic:

import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;

@Component
public class KafkaProducer {

    private final KafkaTemplate<String, String> kafkaTemplate;

    public KafkaProducer(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void produce(String message) {
        kafkaTemplate.send("my-topic", message);
        System.out.println("Sent message: " + message);
    }
}

Step 5: Create a REST Controller
Create a REST controller to expose an API endpoint for producing messages:

import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class MessageController {

    private final KafkaProducer kafkaProducer;

    public MessageController(KafkaProducer kafkaProducer) {
        this.kafkaProducer = kafkaProducer;
    }

    @PostMapping("/messages")
    public void sendMessage(@RequestBody String message) {
        kafkaProducer.produce(message);
    }
}

Step 6: Start the Application
Start your Spring Boot application. It will initialize the Kafka consumer and listen for messages on the specified topic. You can now send messages to the Kafka topic using the API endpoint /messages.

That’s it! You have successfully set up a Kafka consumer and producer using Spring Boot. When a message is sent to the /messages endpoint, it will be produced to the Kafka topic, and the consumer will receive and process the message.

Make sure you have a running Kafka broker on localhost:9092, and create a topic named my-topic before running the application.

Remember to customize the consumer and producer logic according to your requirements. You can add error handling, implement custom serialization/deserialization, and utilize additional Kafka features provided by Spring Kafka.