Add native Terraform integration

[jaketf]

Description

Create a terraform integration for apache airflow.

Use case / motivation

Use terraform to manage ephemeral infrastructure used in airflow DAGs taking advantage of it's "drift" detection features and wide array of existing integrations. For teams who use terraform this could replace tasks like create / delete dataproc cluster operator. This could be really interesting for automating nightly large scale e2e integration tests of your terraform and data pipelines code bases (terraform apply >> run data pipelines >> terraform destory.

Related Issues

Inspired by this discussion in #9593

cc: @brandonjbjelland @potiuk

Brandon and I will discuss a design for this next week.

[potiuk]

Looking forward to it!

[kaxil]

I would definitely recommend using BashOperator for this :) . Just to clear the air I am not against TerraformOperator for sure but I have used a lot of terraform with Airflow in the past and BashOperator has just worked perfectly without having to care about different TF versions

[potiuk]

I think it would be nice to have a Hook/Operator that could manage installing Terraform automatically and expose a python API. There is the https://pypi.org/project/python-terraform/ wrapper and installing terraform is basically downloading the right binary from https://www.terraform.io/downloads.html (you can even specify version of terraform). This way you could use the power of terraform without worrying about having it installed at your worker.

I think just mentioning that there is a "Terraform" hook/operaotr is something that can make more Airflow users more aware that they can actually use Terraform rather than dedicated actions. Plus I think often terraform scripts are rather complex - usually they are stored somewhere in repository - not necessarily in the DAG's folder, so it would be great to have an option to somehow download (git-sync?) a specified set of terraform scripts. Or maybe we can think out some more "airflow-y" way of distributing such scripts.

[kaxil]

Just an FYI: https://pypi.org/project/python-terraform/ (https://github.com/beelit94/python-terraform) hasn't been actively maintained so if we someone wants to work on this one, try finding another library

[jaketf]

@kaxil I'd love to hear more about your use of terraform in airflow.
Did you bake terraform installation into your airflow image?
I agree w/ @potiuk that we want to make this easier to install

without having to care about different TF versions
How does using bash operator eliminate the need to care about terraform version? Did your bash script install terraform each time?

I think something that comes to mind for me is that beyond terraform binaries sometimes, terraform scripts depend on other binaries e.g. custom providers or to shell out and do hacky things from null provider.

The first thing that came to mind was KubernetesPodOperator. I think if we made it easy to "bring your own terraform image or use this default image for this version (pulled from docker hub)"

[kaxil]

Hey Jacob, in our Setup the DAG Author had the first task as Setup the environment where they downloaded necessary binaries from internal Nexus which would have multiple versions of binaries too as the different team relies on different versions of Terraform and Ansible. These binaries were stored in a temporary location and were cleared by the task at the end of the DAG.

We had bash scripts to run Terraform and Ansible as there were clients who would run these things on ad-hoc basis via their local Machine or through Jenkins. Running things via BashOperator was an easy way for us without maintenance over-head of a Bash script and a Custom Operator. But it would be different for different teams with different use-cases I suppose :)

[jaketf]

@kaxil How would you pull your terraform configuration source?
in the bash operator or the setup environment or was this very small terraform configurations embeded in your DAG code?

Our idea would be to provide some abstraction of that setup the environment step to make it easy to have a terraform binary running in your airflow execution environment.

My first thought was to essentially create a subclass of KubernetesPodOperator with a git-sync initialization container and a terraform container.
The idea being we could provide reasonable defaults for terraform image (e.g. official image from docker hub) or the user could override this to be a container w/ additional binaries (gcloud, providers, etc). In theory the user could also specify an image for popular wrappers like terragrunt

This would give a lot of flexibility to the advanced user and takes care of a lot of boilerplate.

tf_task = TerraformOperator(
  command='terraform apply -auto-approve',
  git_ssh_key_secret_name='my_tf_git_sync',
  sub_path='terraform/my_dir',
  terraform_image='gcr.io/my/terragrunt-image', # this would default to ''hashicorp/terraform:latest',
  gcp_secret_name='gcp-terraform-key',
  aws_secret_path=None,
  azure_secrete_path=None,
)

The drawback naturally is for non-k8s based airflow deployments.

I think to make this really useful at most enterprises we need to think about how to best handle secrets.
Terraform often needs a lot of permissions so if there was an opportunity to mange the secret for terraform's credentials outside of airflow this would be ideal (so not every DAG can bootstrap the god-like permissions for terraform).

The idea here is that the user manages a k8s secret for the god-like terraform credentials and just tells our operator this secret name so we can mount it as a volume in our pod definition. Alternatively these secret names could be omitted and we could fall back on provider specific magic like workload identity.

[jaketf]

I think the approach of subclassing k8s pod operator is far from how we typically build airflow integrations and the heavy reliance on k8s secrets as opposed to leaning in on airflow secrets managers might be overly opinionated.

I guess I'd like to ask does this seem like something that belongs in airflow core or something that belongs as an opinionated plugin released elsewhere?

Due to the wide variety of binaries folks end up relying on in their terraform scripts (even perhaps different terraform version in different DAGs), the large permissions footprint required I think it is difficult to develop a very general integration that gives the user enough flexibility to meet their security needs.

Taking a step back:
I am a concerned that I'm sort of making up these requirements based on my own speculation (I have no real connection to hashi and haven't worked w/ a gcp customer who directly asked for this feature but rather it evolved from this discussion).
Is there a good way to crowdsource from the community what their interest / requirements are for an airflow/terraform integration are? Is this best done in the dev-list? user-list? slack sig?
Is there a good way for airflow community to engage a provider like hashi in helping design / develop this based on what they might hear from their customers?

[kaxil]

How would you pull your terraform configuration source?
in the bash operator or the setup environment or was this very small terraform configurations embedded in your DAG code?

We had everything in the bash script, our terraform modules were on a private Gitlab repo but the ssh key of our Airflow Box was added to Gitlab and the native Terraform + git integration worked fine when running terraform init for us.

Although I definitely feel there are more users out there who would be happy with your solution too :) so yes please feel free to PR the TerraformOperator

[marshall7m]

Just an FYI: https://pypi.org/project/python-terraform/ (https://github.com/beelit94/python-terraform) hasn't been actively maintained so if we someone wants to work on this one, try finding another library

tftest is an alternative that is actively maintained

[marshall7m]

Hey all, just wondering if there's potential for a Terraform/Terragrunt operator that's simply for returning Terraform output values to be used for downstream tasks. Since this operator will be scoped to only the terraform output command, the operator would only need a minimal amount of dependencies and probably wouldn't require setting up a new environment. The operator would only need to know the: terraform/terragrunt version, infrastructure repository URL, directory to run the output command, and the associated read credentials needed to access the remote tfstate file. I think this would be ideal for users who have CI/CD pipelines already in place for running terraform apply and they just need a way to sync their Terraform resources to their DAG tasks.

The operator would look something like this:

terra_output = TerraOutputOperator(
  binary='terraform',
  repo_url="https://company/infra-live",
  sub_path='infra/region/project',
  tf_version='1.3.5',
  tg_version="32.0.0", # only used if `binary` arg is set to `terragrunt`
  gcp_iam_role=None,
  aws_iam_role="arn:aws:iam::role"
)

some_task = SomeCloudOperator(
    task_id="some_task",
    instance_arn=terra_output["instance_arn"]
)

within the execute() of the TerraOutputOperator, the tftest PyPi package can be used to parse the Terraform output into a Python dictionary that can be accessed by downstream tasks. The operator execute() function could look like something along the lines of:

from tftest import TerraformTest, TerragruntTest

def execute(self, context: Context):
    terra = TerraformHook()

    # install repo containing IaC files to local tmp directory
    tfdir = terra.get_source(self.repo_url)
    # install terraform binary if not already installed and sets the version in PATH
    # could use terraform version manager like tfenv
    terra.set_terraform_version(self.tf_version)
    
    if self.binary == "terraform":
        terra = TerraformTest(tfdir)
    elif self.binary == "terragrunt":
        # install terragrunt binary if not already installed and sets the version in PATH
        # could use terragrunt version manager like tgswitch
        terra.set_terragrunt_version(self.tg_version)
        terra = TerragruntTest(tfdir)
    
    # removes tfdir directory and could remove terragrunt/terraform if needed
    terra.cleanup()

    # parses terraform/terragrunt output results into python dictionary
    return terra.output()

If this is within the scope of Airflow's built-in operators/hooks, I'd be happy to make a PR!