Build systems allow for the automation of various software development tasks that need to be done daily. With build systems, all we have to care about is coding software. They'll compile the code for us, and they’ll have all the tests run automatically. Furthermore, they’ll address dependencies, they’ll adjust our code for performance, and they’ll bundle everything up into a package. Build systems can even produce documentation based on what’s in our code.
But when it comes to using build systems for monorepos — a single repository that stores all the code and assets from multiple projects — there are different factors to consider than with several small repos. The complexity of a monorepo requires tools that can scale accordingly and build, test, and package huge amounts of cross-language code simultaneously.
In this post, we'll explain what monorepos and build systems are, we'll understand the difference between imperative and declarative build systems, and, finally, we'll introduce several build systems for monorepos.
Related: Pros and Cons of Using Monorepos
Imperative vs. Declarative Build Systems
As previously mentioned, build systems enable developers to automate a few important, repetitive, problematic, and otherwise annoying tasks. But before examining specific build systems for monorepos, it’s good to understand two concepts that define the way build systems operate.
Imperative and declarative are two of the most common programming paradigms.
One of the best ways to define and understand the difference between imperative and declarative build systems is by examining who handles the control flow of the build. We define control flow as the tasks that need to be executed and the dependencies that they have.
In imperative systems, the person who writes the build script is the one who controls the flow. When we describe our build, we have to analytically write all the commands which make up our build and define which ones should run. In other words, we have to tell the build system exactly what it has to do.
On the other hand, in declarative build systems, the person who writes the build script doesn’t have to analytically define the tasks that need to get executed. We can simply make a statement about our project — for example, we have a web application. From there, without having to specify what tasks to execute, they're automatically added to our build.
Of course, to achieve this level of automation, declarative systems require a convention. This convention describes where the files are stored, where the dependencies are, and where to store the artifacts. With the declarative approach, we don’t have to define the structure for every project.
Declarative build systems are less flexible than imperative ones since they don’t allow writing out build steps as arbitrary code. This additional structure makes builds much easier to parallelize or cache, however, which allows them to scale to support much large code repositories (such as most monorepos).
More on Monorepos
Many large tech companies (like Google, Facebook, and Uber) use monorepos. Instead of having each product or library stored in its own source control, they're stored in the same repository. This has the benefit of enabling organizations to develop, test, and deploy all projects and their dependencies at the same time.
One of the main pros of monorepos is that every change can be done as one atomic commit. When someone has access to the code, they can see all the changes and the relationship between projects and can create updates on them without having to first check in or get access. Also, with monorepos, finding dependencies is easy because everyone is following the same structure.
To better understand monorepos, let’s imagine we want to create a new product. The product depends on an npm package for some of its functionality. Later, we need to create a new feature for the product and the npm package needs to update. Monorepos help us make all the necessary changes at the same time and test them instantly. This allows us to find potential bugs early. At last, when the changes are finished, we can release the product and the package simultaneously.
Top Build Systems for Monorepos
We mentioned earlier that not all build systems can tackle the complexity that comes with a monorepo. Of course, as monorepos become more prevalent, the number of available tools is growing. Here are three of the most popular.
Bazel Build System
Bazel is an open source, declarative build system created by Google. It can run on Windows, Linux, and macOS, and it supports cross-platform languages like Java, C++, and Python. Bazel allows us to have a huge project written in multiple languages and only use one build system instead of mixing. It performs very well for big enterprise projects, and a lot of firms, like Adobe, Huawei, Dropbox, and many more, use it.
Bazel can support a codebase of any size, in a huge monorepo or across many repositories. It allows projects to scale, as it can cut build times and support the growth of the project.
Furthermore, with Bazel, we have faster build processes, as it only rebuilds what is needed. It can achieve this with its built-in support for local and distributed caching of both builds and tests, along with the optimized dependency analysis and parallel execution of Bazel.
Finally, although Bazel already supports a plethora of platforms and languages, we can extend it to support any other framework or language we need.
Buck Build System
Buck is a build system created by Facebook especially for supporting monorepos. Like Bazel, it’s open source and declarative. It supports multiple programming languages across many platforms, like iOS, Android, and .NET.
Buck can achieve fast iteration that allows developers to compile and run their changes quickly. It's optimized for incremental builds. This means that only files that were changed get recompiled while other artifacts remain intact.
Buck simplifies the build process since developers only have to rebuild what they're currently working with. This results in improved productivity and increased velocity.
Buck provides the buck query feature. With it, we can better understand our dependencies and what we need to build our product. Lastly, with buck project, our IDE can better understand our project, which increases our integration with it.
Pants Build System
Pants is a declarative build system created by Twitter. It’s open source, but unlike the other two systems, it can only run on Linux or macOS and currently supports projects written in Python.
Pants works well with monorepos. It's great for managing dependencies both inside and outside the repository, and it's optimized for build speed.
Other Pants features like fine-grained invalidation, shared result caching, concurrent execution, and remote execution don’t require manual intervention from the build author. Finally, Pants offers a unified interface for multiple tools and languages and extensibility and customizability via a plugin API.
Monorepo Build Systems: The Bottom Line
Although monorepos aren't the best solution for all organizations, they can provide multiple benefits to companies that choose them as a development strategy. And, build systems like Bazel, Bucks, and Pants help users tackle the complexities of monorepos and get more value from them.
A software composition analysis tool like FOSSA can also go a long way toward helping organizations manage monorepos. FOSSA identifies dependencies within your codebase and alerts you to any security or licensing issues before they become problematic — even on monorepo projects that may contain hundreds of gigabytes of code. Get in touch today if you’d like more information on bringing FOSSA to your team.
