Addressing the scaling issue
Helsing.ai, a leading software company based in Munich, has announced the open-sourcing of Buffrs. This package manager aims to simplify inter-service API design and implementation through protocol buffers and gRPC. Previously, the company had faced scaling challenges with gRPC APIs. The limitations largely came down to inadequate tooling for protocol buffer definitions and a lack of best practices for API decomposition and reuse.
To tackle these issues, Buffrs allows engineers to package protocol buffer definitions, publish them to a shared registry, and easily integrate them into their projects. This new tool embraces versioned API dependencies and offers seamless build tool integration.
Design goals and principles
Buffrs is informed by a set of guiding design principles. Firstly, it includes a versioning scheme for APIs. This feature makes it easier for developers to manage both backwards-compatible and breaking changes. Secondly, the system strives to make wire-format compatible APIs also source-code compatible. This eases minor version upgrades and offers automation capabilities.
Moreover, Buffrs addresses the issues of diamond dependencies and Rust’s orphan rule, two issues that have plagued protocol buffer-based systems. The design emphasises consumer-side code generation, allowing different consumers to tailor code generation to their needs.
Composition and discoverability
Another major goal of Buffrs is to encourage code reuse and composition. The package manager facilitates the use of base messages and data types across various APIs. The introduction of a central registry enables better discoverability, enhancing engineering productivity and allowing easier tracking of API evolution.
Lessons from existing solutions
Before developing Buffrs, Helsing.ai had reviewed existing tools like Buf, Bazel, and Git. Although these tools offer some useful features, they each had drawbacks. For instance, Buf wasn’t suitable for Helsing due to data residency requirements, and its approach to distributing compiled language bindings conflicted with Helsing’s focus on source compatibility. Bazel lacked features like explicit versioning and a central registry. Git, although a quick way to start, proved cumbersome for API evolution and offered inadequate monitoring options.
Architecture and workflow
Buffrs has three core components: the Buffrs CLI, a Buffrs Registry, and local code generators. The CLI allows declaring, fetching, and publishing of protocol buffer packages. The registry hosts these published packages, and the local code generators compile protocol buffers into language-specific bindings. Buffrs differentiates between two types of packages: Libraries and APIs, each designed to serve specific needs in reusable code and service APIs, respectively.
Towards a sustainable ecosystem
While the Buffrs project is still in its early stages, Helsing.ai believes it has the potential to establish a reliable open-source package registry for protocol buffer libraries. By emphasising best practices, code reuse, and discoverability, Buffrs aims to simplify and streamline API design and implementation in complex, scaling environments.
—
If you are looking to build revolutionary software products, you need the right team. Speak to a PL Talents recruitment expert today.