Ansible Kafka Admin

Ansible Kafka Admin is an Ansible library written in Python that can be used to administer any Kafka server using Ansible.

Contrary to some alternatives, this library does not use the scripts provided with any Kafka installation. Instead, it uses the Kafka protocol to directly talk with the Kafka server. This means no SSH connection is actually needed to the Kafka server, just a direct connection, that you can secure in multiple ways. The library handles any Kafka cluster topology (1 vs multiple brokers, zookeeper vs kraft), taking care of talking with the good coordinator when needed. Every major Kafka version is tested via the CI.

Generally speaking, this library is very handy to perform some not-that-easy administrative tasks like partition increase for a topic, which is not handled by default by the “official” Kafka scripts. For this task, the library will take care of generating new partitions while trying to keep the current ones on their assigned brokers to avoid moving too much data. Obviously, decreasing a number of partitions for a topic is not possible, as it would mean losing data.

Many modules are available for performing tasks against several Kafka resources (topic, users, ACLs, quotas, etc.). Here is an example of a playbook that will created a topic in a Kafka server available at localhost:9092:

---
- name: Example | Topic creation
  hosts: 127.0.0.1
  roles:
    - name: kafka_lib
  post_tasks:
    - name: "Create topic 'test-topic-creation'"
      kafka_topic:
        api_version: "2.6.0"
        name: "test-topic-creation"
        partitions: 2
        replica_factor: 1
        bootstrap_servers: "localhost:9092"
        state: "present"

The only weird thing is the need to specify the kafka_lib role. This is not ideal but, as far as I know, there is no real workaround to avoid it.

Testing is made using molecule which is very handy to create some specific stacks and run tests against it. In our case, we try to only have 1 stack per major Kafka version (0, 1, 2, 3) and run the tests for each of the modules. We found it better than having one scenario per module (which would be a natural way to do it), as it would increase the number of tests to perform meaning it would take more time (and maybe cost money, our CI is using GitHub actions) without having real gains.

For any readers using k8s, the library can also be used with the operator SDK to build a Kafka administration operator for k8s. This is not something we plan to do for now, but who knows what can happen in the future?