Composable Infrastructure-as-Code (IaC) with Independent Bit Components

Build reusable and modular Infrastructure-as-Code (IaC) using Bit, Pulumi and AWS

Infrastructure-as-Code (IaC) is the modern standard of provisioning infrastructure. Unlike the old days when cloud infrastructure was set manually, with IaC, we can version control the infrastructure code, providing visibility into the changes we make over time.

Another key aspect of IaC is that it is composable. Each infra module or component can depend on other modules and be used by others. For example, an AWS Lambda component can have a business logic component as a dependency but can also serve as a dependency of an AWS API Gateway component.

A graph of IaC (independent) Bit components

This modular approach enables you to build complex infrastructure by composing simple modules. However, a composable infrastructure's benefits are lost when maintaining it as a monolith. It is hard to reuse IaC components, and the dependencies between components are not easily visible.

This monolithic approach can make the infrastructure hard to understand and maintain, with a high potential for dependency issues like circular dependencies or dependency mismatches.

In this blog post, we’ll see the benefits of using Bit, an open-source toolchain for the development of composable software, for the development of composable infrastructure as code.

See the independent IaC components in their Bit scope

To get started quickly with a demo repository, see:

GitHub - bitdev-community/bit-pulumi: IaC Components using Bit, Pulumi and AWS

Keep in mind that independent Bit components are not coupled to any repository. That means you can directly fork any component in this demo Bit scope.

IaC History

If we go back to history, many cloud providers introduced domain-specific languages to define their infrastructure as code. For example, AWS introduced CloudFormation; Azure introduced ARM scripts, etc. Vendor-neutral technologies like Terraform also emerged, supporting multiple cloud providers.

Yet, there was a significant drawback with all these. Each technology requires you to learn a new programming language or a modal.

Fast forward to today, many cloud providers are now moving towards popular languages like JavaScript (TypeScript), Python, and .NET for defining infrastructure. In addition to that, we can find vendor-neutral providers, like Pulumi, that do the same.

So, if you ask the question, can we write our IaC using TypeScript, the answer is Yes!

Quick Example (Using Pulumi)

Take a look at the following code block:

import * as aws from "@pulumi/aws";

// Create our bucket using infrastructure as code.
const imageBucket = new aws.s3.Bucket("image-bucket");

// Create an AWS Lambda event handler on the bucket using magic functions.
imageBucket.onObjectCreated("imageHandler", (e) => {
    // your lambda code goes here
    // e.g You can create thumbnails of the image uploaded
});

The code is self-explanatory and as straightforward as it gets when defining this infrastructure. Looking from the outside, it seems like all you’re doing is creating an S3 Bucket and a Lambda Function, then assigning it to handle events for the S3 Bucket.

But when you take a closer look, there’s actually more to it...

After creating the Lambda Function, its code must be uploaded with its dependencies (e.g. Node.js libraries). It also needs permission to access the S3 Bucket. Besides, the bucket needs to be set up with specific access control rules. If this sounds complicated, don’t worry. Pulumi takes care of all these details behind the scenes.

The Challenge of IaC Reusability

So, one problem is solved when we have a solid and straightforward programming model to define our infrastructure. But, the real challenge comes with the code itself when we want to follow best practices for modularity and reusability.

Let me give a simple example using the above code block. Just think about the below scenarios;

1. You plan to reuse the image thumbnail creation logic for two different buckets.
2. Enforce S3 bucket policies across multiple buckets.
3. Create a CloudFront (CDN) to serve images and thumbnails.
4. Reuse these constructs across multiple applications.

Looking at these scenarios, each needs a different solution. For example, we may need to find a way to reuse logic across two Lambda Functions. You may think putting the logic into an NPM library, and reuse across Lambda may solve it. Then, you encounter another challenge of maintaining that code and its lifecycle. Enforcing S3 Bucket policies may need creating the bucket policy as a separate construct and extend simpler S3 Bucket creation.

However, upon deeper examination, the issue fundamentally resides in our approach to structuring the code. Merely partitioning the code into multiple files or stacks isn’t the solution.

Even though the infrastructure is spread out, the Infrastructure as Code (IaC) remains monolithic.

Imagine the possibilities if we could actually divide the code into independent, modular blocks or components that could be developed and reused without hard dependencies.

The solution lies with the Bit Composable Architecture.

Using Bit Composable Architecture for IaC

If you are new to the concept, the Bit Composable Architecture is an approach to design software in a component-driven way, as a collection of loosely coupled components that work together.

Bit Components are versioned with their dependencies and reusable even across multiple applications. Bit provides the open-source toolchain for TypeScript and JavaScript to implement this architecture.

Let’s see a quick demo of implementing Bit Components for IaC:

We created an S3 Bucket, S3 Object (HTML file), API Gateway, and a Lambda Function in this demo. Each one of these is created as an independent Bit Component.

S3 Bucket Policy Component

This way, for example, if you decide to reuse the S3 bucket policy across multiple applications, you can do so either by;

1. Using the S3 bucket policy component as a NPM package.
2. Importing the component into your application (It’s a new concept of reusing code across projects, for more details refer ).

Testing your Infrastructure Code at Build Time

Component Status

When you define your infrastructure as a collection of independent Bit Components, you can also compile (bit compile) and build (bit build) to validate them against any build errors. You can identify the modified components, and how it propagates across dependent components.

It provides an additional safety net before you run pulumi up to see the final impact against the existing cloud infrastructure.

Structuring your APIs and Lambda Functions

Lambda function code

Another unique advantage you get is that you can now design your Lambda code in a modular and reusable way. For instance, if your Lambda Function is supposed to update the database and trigger an SNS event, you can break your Lambda logic into two components used within your Lambda handler.

This way, you can reuse the SNS trigger component across multiple other Lambda Functions. The same goes for the update database component; if the need comes, you can easily reuse it without duplicating the code.

This is massive in every way, where the architecture becomes flexible to design from Lambda monoliths to Lambda Microservices.

Getting Started

Overall, IaC approach with composable architecture provides a huge potential for defining and managing modular infrastructure at scale.

You can start experimenting on the concepts by creating a simple node example using Bit. To create infrastructure with Pulumi, use the demo project shown here as the starting point, since it addresses some of the challenging areas when combining Pulumi with Bit.

I hope you got a good understanding on composable architecture for infrastructure components. If you have any questions, post them in the comments below.

Thanks for reading! Cheers!