Asset Bundles Sync Include/Exclude Precedence

[danricbede]

We have custom yaml configuration files for each environment. Our local directory looks as follows:

- config
  - target-a
    - config.yml
  - target-b
    - config.yml

I was hoping to be able to use the include/exclude items to only sync the directory for the active target. For example, given a deployment to target-a, I'd want the workspace to look as follows:

- config
  - target-a
    - config.yml

It seems that the wildcards for the exclude override the explicit include - as this doesn't work:

sync:
  include:
    - "config/${bundle.target}/*"
  exclude:
    - "config/*/*"

But I can achieve the opposite by doing:

sync:
  include:
    - "config/*"
  exclude:
    - "config/${bundle.target}/*"

I can't seem to find a way to achieve the desired behaviour. Only things I can think of would be changing the precedence behaviour according to order in which they're defined, or supporting a not operator. Hopefully I'm just missing something - in general DABs look super powerful.

[andrewnester]

@danricbede could you please provide your use case why do you need to deploy a different set of files to different targets?

[danricbede]

@andrewnester Our setup is more complex than Dev > QA > Prod. We have multiple QAs and Prods terraformed for each of our clients, each with their own configurations. We split these configurations across different files.

[andrewnester]

@danricbede is having a separate bundle (bundle configuration + set of files) an option?
cc @pietern

[danricbede]

I mocked up this scenario - it could work. We'd probably reduce repetition/redundancy through use of includes, though this might get a bit messy in the future when we want to deviate from defaults, as either we'll need to manually pick the includes or inline things instead. We would still have the same problem in Issue 815 though, where we're bringing in everything by default rather than cherry picking individual files/dirs - unless this is addressed by the "set of files" you mentioned.

It seems a shame though, the behaviour is so close with bundle variables.

Edit: It occured to me that we might want to leave the tests in place for some envs (e.g. dev) - with this in mind multiple bundles might be the only way, unless you can think of something better.

[andrewnester]

The fix for #815 was just merged and it will be release in the next CLI release so it should unblock you

[danricbede]

I made an assumption that you could pass the bundle file as a parameter on deploy, but having re-read the docs this isn't the case:

A bundle configuration file must be expressed in YAML format and must contain at minimum the top-level bundle mapping. Each bundle must contain at minimum one (and only one) bundle configuration file named databricks.yml. If there are multiple bundle configuration files, they must be referenced by the databricks.yml file.

I can't quite figure out how to include or exclude based on target (original problem) - even with #815 being fixed. sync and include aren't overridable in the target section, and you can't seem to use variables to dynamically pick the directory/file in the include section. Can you provide an example of how this might be possible?

E.g.

• On target "dev", sync "tests" dir or include specific config
• On other targets, don't sync "tests" dir, or don't include specific config

[andrewnester]

Indeed, sync overrides are not currently supported. This should be fixed by this PR #856

[danricbede]

I took the latest snapshot build and tried some deployments - there is definite progress, but doesn't fully solve the underlying problem (it's merging default and target-level definitions rather than "overriding").

Is it feasible to have an option on sync to "include none" by default instead of "include all" (current behaviour)? This would allow each target to explicitly define what it wanted. Something like:

sync:
  # This is the suggested new flag:
  include_all_by_default: False

  include:
    - config/${bundle.target}/**
    - assets/**

target:
  dev:
    sync:
      include:
        - tests/**

  client_b:
    sync:
      include:
        - specific_stuff/**

Otherwise, I can't see a way around it without pre-processing the yaml file or performing some post-processing of the workspace, both of which I'd like to avoid 🙏

[andrewnester]

Append behaviour is consistent with other resource and configuration for targets behaviour so we stick to this.

If I understand correctly the desired behaviour you want to achieve is

1. By default: sync config/${bundle.target}/**, assets/**
2. If dev target: sync only tests/**
3. If client_b target: sync only specific_stuff/**

If so, then what you can do is to specify it like this

targets:
  default:
    default: true
    sync:
      include:
         - config/${bundle.target}/**
         - assets/**
  dev:
    sync:
      include:
        - tests/**

  client_b:
    sync:
      include:
        - specific_stuff/**

[danricbede]

The understanding is not quite right - it is:

1. Always: sync config/${bundle.target}/, assets/
2. If dev target: sync option 1 and tests/**
3. If client_b target: sync option 1 and specific_stuff/**

Applying a target of default does not achieve this - it would ideally look something like:

sync:
  include:
    - config/${bundle.target}/**
    - assets/**

target:
  dev:
    sync:
      include:
        - tests/**

  client_b:
    sync:
      include:
        - specific_stuff/**

The behavior I am seeing is that everything is included by default - specificying additional includes has no impact. No matter what target is selected, the entire projects files are synced.

Append behaviour is ok - I am not suggesting this changes - I am suggesting that we have an option to start from no files, rather than everything. This could be defaulted to the current behaviour, but allowed to be set False so nothing is included.