Writing your first contribution to Azure CAF Terraform module

Microsoft Cloud Adoption Framework for Azure Terraform edition, helps you deploy resources on Azure in a structured way, it come with various components that you can leverage together or you can pick based on where you are on the cloud journey and based on your needs.

Today we are going to review how to contribute to the Cloud Adoption Framework for Azure Terraform module Open Source project.

The CAF module

Almost a year ago we switched to a one-module approach that allows high speed of evolution (we know how time-consuming it is to update a set of small modules). We have implemented that with conditional submodules feature introduced in Terraform 0.13 and we have been running that way up to Terraform 1.0.0 with excellent feedbacks.

Having one single module makes the development more difficult for the developer, as it requires working on the iterative structures and junction between the different objects – One could argue this is the job of anyone trying to put more than two Terraform modules together. On the other hand, having the single module integrated allows composition and configuration way faster for the DevOps engineers and architects putting together a configuration.

This module can be leveraged from any Terraform code via the Terraform registry, using the following syntax:

module "caf" {
  source  = "aztfmod/caf/azurerm"
  version = "5.3.11"
  # insert the 6 required variables here
}



You can find many examples on the module GitHub repository here.

The power of the community

With the speed of evolution of Terraform and the Azure provider, inner-sourcing components for enterprises is a pragmatic option that allows DevOps engineers to focus on delivering value.

Adding a new capability

Let’s say you are reviewing the CAF module capabilities and realise it’s missing a feature, and you want to write it and contribute to the community.

We will do that with a simple example: someone in the community opened an issue for Azure Dedicated Hosts, which allows you to deploy a dedicated Hypervisor for your Virtual Machines. You can find all the details about it here:

Basically we will create two objects:

  • the hosts groups: a set of machines in a particular region with a set of capabilities like availability sones and fault domains:
  • the hosts: the hypervisor itself, which will be hosting the Virtual Machines.

And we can foresee that we will need to change the virtual machines module capabilities to declare that a particular VM will be deployed on a dedicated host.

The toolset

In order to get things started, I will get the following development environment:

Aligning all side-by-side, and lets go for the coding!

For the rest, let’s do it in a video – full screen required :):


Refactoring and dependencies

It’s very rare that we get things right from the beginning, so after a review, I detect that I can improve the following ones:

Object syntax for the VM

In my initial implementation, I did the following syntax to reference the Dedicated Host to my virtual machine:

 dedicated_host_group = {
        dedicated_host_group_key = "dh1"
        # lz_key = "remoteLZKey"
        # id = "full resource ID"
    }

But I actually don’t the need group but the host, so I refactor the object and standardize on our pattern that allow using the object from:

  • resource ID of the object deployed outside of this module
  • key of an object deployed locally
  • key of an object from another landing zone (via the lz_key attribute)
  dedicated_host = {
        key = "dh1"
        # lz_key = "remoteLZKey"
        # id = "full resource ID"
    }

Composition patterns to retrieve the Dedicated VM dedicated

At first the pattern to locate the dedicated host_id in the VM object was the following:

dedicated_host_id = try(try(each.value.dedicated_host_group.id, var.dedicated_hosts[try(each.value.dedicated_host_group.lz_key, var.client_config.landingzone_key)][each.value.dedicated_host_group.dedicated_host_group_key].id), null)

It worked, but did not align with updated patterns that offer better visibility and troubleshooting, so I just updated it into:

dedicated_host_id = try(coalesce(
    try(each.value.dedicated_host.id, null),
    var.dedicated_hosts[try(each.value.dedicated_host.lz_key, var.client_config.landingzone_key)][each.value.dedicated_host.key].id,
    ),
    null
  )

Dependencies

There are also dependencies that I managed in the meantime, like the name convention provider. I checked at the naming reference here.

The CAF Terraform name provider can be found on the Terraform registry here

And we load it this way:

terraform {
  required_providers {
    azurecaf = {
      source  = "aztfmod/azurecaf"
      version = "~> 1.2.0"
    }
}

It was my first PR to the name provider so I was quite surprised of the easiness of contribution, the most complicated part was to find the right regex to match the supported characters for Azure Dedicated Hosts.

You can review the PR here and contribution guide.

Now it’s your turn

Let’s get started! Happy to hear your thoughts and learn from your feedbacks, also if you identify a missing feature and you want to give it a try, lets:

  1. In the GitHub Issues, verify if there is an Epic covering the module you are describing.
  2. If the change you are proposing is a sub-feature of an epic, please open an issue describing your changes in details and the reasons for the change with an example.
  3. On submitting the PR, please mention the Issue related to the PR.

Also before submitting the PR, make sure:

  1. Make sure you are using the Visual Studio Dev environment with pre-commit hooks effective.
  2. Matching with coding conventions and common engineering criteria described here.
  3. Provide examples including the main scenarios the module is supposed to achieve.
  4. Include integration tests for your examples.

Thanks

Arnaud

Understanding landing zones for Azure Cloud Adoption Framework – Terraform edition

Microsoft Cloud Adoption Framework (http://aka.ms/caf) is the one Microsoft guidance to adopt Azure in entreprise. One of the fastest way to put the framework in action and deploy an environment fast and good, is to use CAF landing zones for
Terraform.

CAF landing zones are an open source project living on GitHub and we welcome any contribution: https://aka.ms/tf-landingzones

In this article, you will find a set of videos and articles that should help you understanding the landing zones architectures and how to start crafting your own.

Cloud Adoption Framework: Introduction landing zones for Terraform


Cloud Adoption Framework: Deep-dive on landing zones for Terraform


First Deployment:


Setup Environment:

Cloud Adoption Framework landing zones with Terraform

Official Cloud Adoption Framework landing zones:

https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/expanded-scope/terraform-landing-zone

If you are getting started with Terraform on Azure:

Operating governance on cloud native technologies:

Cloud Adoption Framework landing zones with Terraform

At Microsoft Ignite 2019, we announced that Cloud Adoption Framework for Azure now has a Terraform edition of it’s landing zones. We see landing zones as a great way to enable value quicker on Azure, and as we see tremendous excitment and growth in this area, we are evolving the experience to make it simpler and more powerful.

Adopting CAF landing zones with Terraform, customer consistenly realize they can:

  • deploy foundations of cloud environments fast
  • easily deploy governance and policies as code (with ongoing compliance enforcement and remediation)
  • have consistent ways to deploy innovation (infra as code, policies as code or application code are using the same mechanisms).
  • and many more.

This overall enables them to: optimize their operations, innovate faster and cheaper, but we will come back to that in another article.

For now, let’s get you started with landing zones development and deployment!

 

Setting up development environment

When deploying Infra-As-Code environments, we found it extremely important to have:

  • strong versioning (accross dev environment, DevOps pipelines, etc.).
  • short feedback loops: ability to innovate fast on your development, test new code without having to go through the whole pull-request and so mechanisms.
  • accomodate diversity of environments (running Windows, Mac, Linux).
  • strong identity consistency and no secret in any code.

In order to enable that, we use Docker containers as development (on your laptop) and deployment (CI/CD pipelines) environement. The toolset is packaged inside what we called the rover and in order to get started – whichever your platform is, just:

Once its done, make sure Docker is up and running, then Open Visual Studio Code:

We need to install “Remote Development” extension, so click the Extensions icon and then find and install “Remote Development” from Microsoft:

Screenshot 2019-11-15 at 10.10.05 AM.png

To make your life easier, we are going to leverage Visual Studio Development Containers, which allows you to seamlessly work on your machine and execute your code in the rover container.:

68747470733a2f2f636f64652e76697375616c73747564696f2e636f6d2f6173736574732f646f63732f72656d6f74652f636f6e7461696e6572732f6172636869746563747572652d636f6e7461696e6572732e706e67

So on your machine, you will have your Terraform files, with your Git connection and other tools, but when you run the code, its seemlessly integrated with the Rover running in the container, so you can develop, test, run the complete environment.

 

Deploying your first landing zone

Once you have deployed the prerequisites, you can just clone (with Git, or GitHub Desktop) from our repository: http://aka.ms/tf-landingzones and open it in Visual Studio.

If you look at the files, it’s classic Terraform scripts except for the  .devcontainer folder where we have two files:

  • devcontainer.json: describes how to connect to the container, the mounting points and extensions.
  • docker-compose.yml: describes the container configuration needed, especially where to get the rover image (tip: we put it on the Docker registry so you dont have to manage it)

Screenshot 2019-11-15 at 10.18.53 AM.png

You can then go to the bottom left, green part of Visual Studio Code and select Screenshot 2019-11-15 at 10.21.51 AM.png

Open Remote-Containers: Open Workspace in Container, then select the default workspace we created for you (workspace.code-workspace). You will see a new window opening and you can see the details of the Docker container operations:

  • downloading the container from registry
  • mounting the volumes
  • creating the network
  • installing VSCode drivers in the containers

Screenshot 2019-11-15 at 10.24.33 AM.png

You are now ready to start hacking, you can start by launching:

  1. rover login 
  2. rover /tf/caf/landingzones/landingzone_caf_foundations plan 
  3. rover /tf/caf/landingzones/landingzone_caf_foundations apply Screenshot 2019-11-15 at 10.29.00 AM

So let the fun begin!

 

Demo

TLDR; Here is a video with the different steps I described:

 

Happy hacking, let me know your feedbacks and lets connect!

@arnaudlheureux

Azure landing zones using Terraform: Getting started

This article describes the old experience for Azure CAF landing zones on Terraform, please refer to this article for the new updated experience:

https://arnaud-web.azurewebsites.net/2019/11/15/cloud-adoption-framework-landing-zones-with-terraform/

 

Cloud Adoption Framework for Azure has a great set of recommendations to accelerate deployment of Azure for entreprises who seek to consolidate their IT environment in the cloud and innovate on their applications.

CAF introduces landing zones as a concept that describes all the elements that must be in place to deploy a production-grade quality deployment (ie, that includes a minimal set of auditing, controls, policies, etc).

In order to accelerate that, let’s review how to deploy our first landing zone for Azure based on Terraform! We assume that you know Terraform and Azure already, if thats not the case, spend some time on my previous post.

We have published and will keep on updating a repository on GitHub: http://aka.ms/tf-landingzones

Getting started

The fastest way is to use Azure Cloud Shell:

  1. Open https://shell.azure.com
  2. Go to the clouddrive directory: cd clouddrive
  3. Clone the GitHub repo from http://aka.ms/tf-landingzones git clone https://github.com/aztfmod/blueprints.git
  4. Initialize the environment – this will create the fundamentals for the Terraform state, like Storage Account, Azure Key Vault, and the managed identities: ./launchpad.sh
  5. Deploy your first tranquility blueprint: ./launchpad.sh landingzone_vdc_level1 plan
  6. Review the configuration and if you are ok with it, deploy it by running: ./launchpad.sh landingzone_vdc_level1 apply

Below is a quick demo of doing that:

Customization

In order to ease your first contact with the landing zone, we created a sample configuration file proto.landing_zone_vdc_level1.auto.tfvars. As any .auto.tfvars file, it is going to be picked up automatically by Terraform when running. This file automatically configures all variables needed to get started:

  • Retention period for Activity logs, and operations logs.
  • Names of the resource groups to be created.
  • Location of the resources to be deployed.
  • Tags for the resources.
  • Name of log analytics and list of solutions to be deployed.
  • Security center contact details.

You can tune it to match your criteria, we hope the syntax is self explanatory, feel free to provide feedback on it!

Screenshot 2019-09-17 at 3.09.49 PM

Quick tour of the architecture:

Currently the solution is composed of 3 main components:

1. Launchpad.sh

Is a shell script, it initiates the Terraform state locally, uploads it to the Azure storage account, manages the environment variables and communication accross landing zones and other components.

2. Level0_launchpad

Creates a Key Vault, an Azure managed identity a storage account that is used to create store the Terraform state of our environment. It also creates a serie of service principals to be used to access the Terraform state and to integrate with Azure DevOps (in a future release). For full documentation, refer to the readme file.

3. Landingzone_vdc_level1

This vdc level1 landing zones leverages one blueprint (called tranquility) that sets the foundation for everything accounting and operations logs in your Azure subscription:

  • Resource groups
  • Activity Logging
  • Diagnostics Logging
  • Log Analytics
  • Security Center

Each of those features deployments are accomplished by a respective module, which is also on the GitHub, which Terraform download when it needs it (at the plan stage).

A diagram of landingzone_vdc_level1 would look something like that:

Screenshot 2019-09-17 at 3.32.02 PM.png

For full documentation, refer to the readme file.

That’s it for now, it should be enough for you to get started on the Azure Landing Zones using Terraform. Keep in mind there is much more to come on the Landing Zones, we will keep on publishing more and we will have a couple of webinars to go deep dive on the subject!

Happy landing landing zones crafting!

Arnaud Lheureux