JSON Linter in Variables

[selvakumar-arumugapandian]

Description

JSON Linter in Variables

When user adds an object (JSON format) as variable, there should be option enabled to validate the object or validate automatically.

For example:

I have a variable named "my_var_group", with the following value

{
"dag_prefix": "my-prefix",
"dag_retry": 5
}

Imagine, I am using this variable "my_var_group" in 500+ DAGs and any mistake in the JSON fails to load all the 500+ DAGs.

So, A feature which does automatic linting of JSON object (By throwing error during variable creation/update) shall ensure that "error" prone JSON is not added.

Use case/motivation

Real-time experience in an environment where we are running more than 1000+ DAGs.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[bbovenzi]

Good idea. Feel free to copy over the JSON linter we use for a dag run config: https://github.com/apache/airflow/blob/main/airflow/www/templates/airflow/trigger.html

[selvakumar-arumugapandian]

@bbovenzi , Thank you. I shall check this out.

[obulat]

We recently ran into this issue in Openverse. We have an array of objects in a JSON variable to determine whether to send a Slack notification or not. When updating the value using the UI, we accidentally left the trailing comma ([{"object1", "value1"},]). Because of this, json.loads threw an error, and the DAG failed. It is very easy to make such an mistake when setting or updating a JSON variable using the UI, without a linter.

I would love to contribute here, but I'm not sure how. @bbovenzi, where would adding a check make sense in the new UI code? There is no indication if the variable will be parsed as JSON, or as string, in the Add Variable Form.

There could be a check to see if the value contains { or [ symbols, and show a toast before saving if the value is not a valid JSON. This way, the user can still save the value if it's not actually a JSON value, and will be aware that the value needs to be fixed if it is JSON.

[pierrejeambrun]

Indeed we do not have any backend information on whether or not this is meant to be used as a json or a plain string.

For now the easiest solution would be what you are suggesting @obulat, doing that check in the front-end and showing an error message if the json is not valid, but without preventing the user from saving.

I would love to contribute here, but I'm not sure how

I think that an error message in the AddVariableForm is a good start :)

The only thing is to find a good and reliable way of knowing if we have to validate the json or not. Looking for { or [ is a first idea, another would be a checkbox or a button that the user needs to click lint json or any other user opt-in action (not sure that's better, just mentioning it). Maybe others will have a better idea.

[bbovenzi]

I think a simple warning message without preventing submission is a good first step!