Upgrade Kafka from the current version 1.1.1 to version 3.2.1 and to provide guided steps below. For detailed information from the Apache Kafka project, refer to the Upgrading section of the Kafka documentation.

1. Syndeia Cloud

   1. All services are stopped.

   2. Check using the below command.

      systemctl status sc-*

   3. If not stopped, then stop using the below command.

      systemctl stop sc-*

   Since no one can use SC anymore, we are making sure that no new events reach the running Kafka.

2. Ensure previous stages have been completed without error.

3. Make sure that the Zookeeper service is running.

   1. Check using the below command.

      systemctl status zookeeper

      You can verify that it is running if, Active: active (running) shows up in the output.

      ● zookeeper.service Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2022-09-09 09:10:07 EDT; 7min ago Main PID: 11943 (java)

4. Make sure that the Kafka service is running.

   1. Check using the below command.

      sudo systemctl status kafka

   2. If stopped, start it by using the following command.

      sudo systemctl start kafka

Step 1 - Before we stop Kafka 1.1.1, we need to collect some information.

1. Navigate to the bin directory within the Kafka install directory.

   cd /opt/kafka-current/bin

2. Run the following two commands to get & save the count of the store and auth events.

   1. Store-Events

      store_event_count_v111=$(./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic store-StoreEvent)

   2. Auth-Events

      auth_event_count_v111=$(./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic auth-AuthEvent)

3. Let’s view these two different counts, ex:

   echo $store_event_count_v111 echo $auth_event_count_v111

   Store output, ex:

   store-StoreEvent:0:132 ...

   Auth output, ex:

   auth-AuthEvent:0:2 ...

4. Note down the location of the Kafka log directory by searching for the value of log.dirs in /opt/kafka-current/config/server.properties file.

Step 2 - Stop Kafka 1.1.1 and Zookeeper 3.6.3.

1. Stop the kafka service.

   sudo systemctl stop kafka

2. Stop the zookeeper service.

   sudo systemctl stop zookeeper

Step 3 - Extract kafka files from the downloaded media (Stage 3)

1. Extract the Kafka package to /opt.

   sudo tar -xzvf kafka_2.13-3.2.1.tgz -C /opt

Step 4 - Copy the /opt/kafka-current/config/server.properties file from the old kafka install directory to the new kafka install directory.

Step 5 - Update the symlink to point to kafka_2.13-3.2.1.

Step 6 - Take ownership of the extracted folder & the symlink.

Step 7 -Update the /opt/kafka-current/config/server.properties file.

1. Add the following lines to the end of the file.

   ############################# Upgrading Kafka ############################# inter.broker.protocol.version=1.1-IV0

2. Look for the log.dirs parameter and update it as follows.

   log.dirs = /opt/kafka-current/logs

Step 8 - Now, we’ll copy the old Kafka logs to the logs directory of the new Kafka installation.

Step 9 - Restart Zookeeper 3.6.3 and then start Kafka 3.2.1.

1. Restart the zookeeper service.

   sudo systemctl restart zookeeper

2. Restart the kafka service.

   sudo systemctl restart kafka

Step 10 - Now we’ll try to gather the same metrics for Kafka 3.2.1 as we did for the older version.  So let’s navigate to the bin directory of the newer Kafka install directory.

Now run the same commands as mentioned in Step 1. So let’s say -

Step 11 - Now, store_event_count_v321 and auth_event_count_v321 above should match the counts of the store and auth events from Step 1c, so the below should be true.

Step 12 - If the counts match, then we can be sure that all the old events are visible to the newly installed Kafka version and we can proceed safely with the rest of the SC upgrade process.

Step 13 - Once the cluster's behavior and performance has been verified, navigate to the config/server.properties file in the Kafka 3.2.1 install directory and update the inter.broker.protocol.version to 3.2-IV0 (as shown below).

Step 14 - Restart Kafka 3.2.1.