Confluent was founded on open source principles and has consistently contributed to some of the world’s most popular open source software, including Apache Kafka®. We are excited to reaffirm our commitment to the open source community by announcing that the Confluent CLI is now open source on GitHub!
Our open source announcement coincides with our latest major version release, confluent CLI v3, which ushers in a host of new features and UX improvements. Notable features include 1) full adoption of Kafka REST for all confluent kafka commands, 2) acknowledgement prompts to prevent the accidental deletion of Cloud resources, and 3) increased command syntax standardization and command discoverability. This release is the capstone of more than a year’s worth of work. We hope the latest features in v3 and the open source CLI improve and facilitate more communication and collaboration between customers, developers, and contributors.
This post covers our design objectives, details the effort required to open source the CLI, and provides an introduction on how to make an open source contribution.
Why we are open sourcing the confluent CLI
To become the de facto CLI tool for Kafka, ksqlDB, and more
Our users rely on Confluent, Kafka, and ksqlDB every day. To provide an integrated user experience, we built a comprehensive command-line tool that can facilitate all use cases. Our CLI is a feature-complete interface that supports tasks like organization and user management, cluster provisioning, and producing/consuming messages. We are actively making investments to ensure the confluent CLI provides features that parallel the functionality of Kafka and ksqlDB bash scripts. The open source confluent CLI helps to accelerate our mission to make a unified command-line tool for all data streaming workflows and to become the recommended CLI for any OSS project using technologies like Kafka or ksqlDB.
To empower users to contribute and provide rich feedback
The CLI is our most popular programmatic interface by active user count. It works as both a learning tool and a tried-and-true component of automated production workflows. When the CLI was closed source, collecting user feedback required users to sign up for a support plan and open tickets through a support portal. Because the majority of our CLI users are developers, GitHub Issues is a much more familiar and lower-friction interface for reporting bugs and requesting features. Another benefit is that all GitHub issues are accessible to everyone, which, over time, will result in an extensive library of suggestions and contributions. Users are also encouraged to open pull requests (PRs) in our repository, following the contribution guide.
To build credibility through transparency
Thousands of enterprises rely on Confluent, and a worldwide community of Kafka users support our codebase. As such, we value making the confluent CLI more reliable. We want to empower our users to actively drive CLI development, and we want to build trust and confidence with our users and community through the transparency of our logic, code, and documentation.
High-level design of the confluent CLI
In a previous blog post, we covered the design of the confluent CLI and highlighted some challenges of shipping a cross-platform tool that communicates with a variety of APIs both on-prem and in the cloud. To recap, our CLI is designed to be:
Powerful: Users can fully manage their Confluent deployment (Cloud and/or Platform) and modern data flow through any of our developer interfaces.
Stable: Functionality is reliable. Breaking changes are minimal, infrequent, and clearly communicated in advance.
Intuitive: Syntax is consistent and predictable, conforming to users’ expectations. Users can easily discover the functionality of the platform and quickly develop production-ready, automated workflows.
For example, to create your first fully managed Kafka cluster in Confluent Cloud, you can run:
$ confluent kafka cluster create my-cluster --cloud aws --region us-east-1
To produce a message to your Kafka topic “test” with an Avro schema with ID 100001:
$ confluent kafka topic produce test --value-format avro --schema-id 100001 --schema-registry-api-secret
Starting Kafka Producer. Use Ctrl-C or Ctrl-D to exit.
{"name":"hello","amount":1.00}
And to consume a message from offset 0:
$ confluent kafka topic consume test --value-format avro --offset 0
Starting Kafka Consumer. Use Ctrl-C to exit.
{"name":"hello","amount":1}
What happens when a command executes?
We chose to write the confluent CLI in Go because it has great support for cross-platform compilation and lets us easily bundle all of our dependencies into a single binary. Go also allows us to build our CLI with the widely adopted Cobra CLI framework, which provides handy functionality such as parsing arguments and flags, creating commands, and executing them as needed.
When a user executes a command, the CLI client generates a Cobra command that holds all arguments, flags, and a series of Run() functions that do the execution job. Before the actual work function runs, the CLI sets up API clients as needed, ready to communicate with the backend services and dependent libraries. This includes confluent-kafka-go, an open source Go client that enables quick and efficient producing to and consuming from Kafka.
In a Run() function, the CLI reads arguments and flags from the command and prepares a request to do the job. The request is then sent to an SDK layer, where the communication between the CLI client and the actual API is made possible. Communication with Kafka brokers is also done in this layer. The CLI takes backend responses, handles errors and exceptions, and provides users with structured, informative output.
Supporting cross-compilation and many OS targets
The confluent CLI supports native development on Darwin, Linux, and Windows machines, whether using ARM or AMD CPUs. It’s also possible to cross-compile the CLI from macOS to Linux and Windows with CGO enabled and the right cross compilers: aarch64-linux-musl-gcc, x86_64-w64-mingw32-gcc, and x86_64-linux-musl-gcc. We take advantage of this when preparing a release of the CLI. Building for glibc-based Linux distributions is more complicated: we use a Docker container to create the native environment, install all the essential tools, build the binary file in it, then copy it back to the release environment. With all these techniques, we are able to release and ship weekly versions of the CLI with ease. We have a pre-built Docker image ready for use, and a well-structured release script, both of which help us construct an automated release process.
Make your first contribution to the confluent CLI
Our team mainly develops on darwin/amd64. However, the CLI supports more operating systems and architectures, as long as you have the necessary tools and environment setup. We invite the community to explore the CLI codebase, build local copies of the CLI, and open PRs to request new flags and features that you find interesting—or simply report bugs! Check out our contribution guide for more information.
Plugins: Add your own CLI commands!
We also want to take this opportunity to share a powerful new feature that was introduced last year and comes packaged with all confluent v3 binaries: plugins! Similar to tools like git and kubectl, you can add your own custom command to confluent by placing an executable file prefixed with “confluent-” anywhere in your $PATH. For more information, see our plugin guide.
Enabling the open source CLI
Over the past year, we’ve made many architectural changes to the CLI to make it open source. This work included:
Migrating to new public APIs. Confluent recently rolled out a new API framework, requiring all feature teams to release new APIs for their respective backend features. We needed to completely migrate the CLI to use the new public and well-documented APIs, which required complex cross-team collaboration and dependency tracking over multiple quarters. An example of one of these new APIs is the Cluster Management for Kafka (CMK) API.
Hardening the CLI. Security is a big consideration for open sourcing the CLI, and we worked diligently to prevent secret leaks and arbitrary code execution in our build pipeline. Rounds of reviews were done for our CI system to ensure compliance. We set up Dependabot to ensure all dependencies remain up to date and free of insecure versions, and we implemented other technical requirements such as updating versions of security protocols to ensure a safe and secure CLI.
Eliminating private dependencies. To make the CLI open source we needed to also open source several dependencies. To do so, we applied two different strategies to our dependencies: 1) for the Confluent Cloud SDK (ccloud-sdk-go-v2), we migrated to our new public v2 APIs; 2) for some SDKs like kafka-rest-sdk-go, no new implementation was required, but we worked to ensure everything was publicly documented, properly licensed, and secure.
Conclusion
The confluent CLI evolved from a hallway conversation in early 2019 and has since grown to support thousands of enterprises and use cases ranging from local development to mission-critical applications. Our CLI has been used all around the world—including on airplanes and on ships in international waters—and we can’t wait to see the innovative use cases from our users in the future (especially with new features like the plugin framework). We are thrilled to give back to the open source community, which has meaningfully benefited CLI development over the years (from cobra to goreleaser and many packages in between). Our open source v3 CLI release is a significant milestone, but we believe this is just the beginning of a new level of development and engagement with the community. We are excited to directly engage with the community and partner with you to build the best version of the confluent CLI.
Get in touch
Our team wants your feedback! Send us your suggestions, bug reports, feature requests, and general feedback. You can open a GitHub Issue, make a PR, send us an email at cl******@*******nt.io, or file a ticket through Confluent Support. We look forward to hearing from you!
Read More
Yuri Lupo