Allow variables to be printed to STDOUT with airflow variables export. by vedantlodha · Pull Request #31854 · apache/airflow

vedantlodha · 2023-06-12T09:48:42Z

This change allows airflow cli to export variables to stdout similar to connections using airflow variables export -.

Fixes #31851
^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

pankajastro

wondering if it makes sense to use AirflowConsole, it would allow print in multiple formats.

vedantlodha · 2023-06-12T11:03:59Z

If by console, you mean the UI, that makes sense, however this allows the cli to do so as well, similar to connections.

In the past i have had to switch between the cli and the UI for connections and variables.

pankajastro · 2023-06-12T13:09:47Z

If by console, you mean the UI, that makes sense, however this allows the cli to do so as well, similar to connections.

In the past i have had to switch between the cli and the UI for connections and variables.

In stdout only. I was thinking if we could print in different formats like json/yaml etc

But I see we do print in different formats for connection export like connection export - --file-format yaml but not for variables export command. so maybe it ok for now.

utkarsharma2 · 2023-06-13T02:07:58Z

Can we please use more meaningful variable names instead of qry, var, val?

Fair point. I just refactored the helper function into this. Will update the patch.

utkarsharma2 · 2023-06-13T02:11:59Z

Suggested change

except Exception:

except json.decoder.JSONDecodeError:

It's always a good idea to handle specific exceptions as close to the source as possible.

utkarsharma2 · 2023-06-13T02:15:55Z

This method doesn't really add much value, I would directly use fileio.name == "<stdout>" in line 96. Saves one lookup to a different method. :)

Sure, this was just to maintain some consistency with connection_command.py's export. Will update the patch

vedantlodha · 2023-06-13T06:31:41Z

Thanks for the reviews @utkarsharma2. Updated the patch. Please take a look when you have some time.

utkarsharma2 · 2023-06-14T00:55:49Z

Suggested change

query = session.query(Variable).all()

data = json.JSONDecoder()

for variable in query:

try:

value = data.decode(variable.val)

except json.decoder.JSONDecodeError:

value = variable.val

var_dict[variable.key] = value

rows = session.query(Variable).all()

data = json.JSONDecoder()

for row in rows:

try:

value = data.decode(row.val)

except json.decoder.JSONDecodeError:

value = row.val

var_dict[row.key] = value

Maybe something like this?

I think we can drop the all() as well since it’s only iterated once. Also why is the decoder called the data?

Apologies for the delay, had been unwell lately.

Thanks for the inputs. Ive updated the patch.

Similar to connections, this change allows the cli to export the variables to the stdout using 'airflow variables export -' Fixes apache#31851

- handle JSONDecodeError instead of Exception

uranusjr · 2023-06-20T09:46:56Z

+            var_dict[row.key] = value
+
+    with args.file as f:
+        f.write(json.dumps(var_dict, sort_keys=True, indent=4))


Suggested change

f.write(json.dumps(var_dict, sort_keys=True, indent=4))

json.dump(f, var_dict, sort_keys=True, indent=4)

github-actions · 2023-08-04T18:53:32Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

boring-cyborg Bot added the area:CLI label Jun 12, 2023

vedantlodha force-pushed the 31851 branch from b900f92 to 22c9d3b Compare June 12, 2023 09:51

pankajastro reviewed Jun 12, 2023

View reviewed changes

vedantlodha requested a review from pankajastro June 12, 2023 11:51

utkarsharma2 reviewed Jun 13, 2023

View reviewed changes

vedantlodha force-pushed the 31851 branch from 21e896c to 7e79824 Compare June 13, 2023 06:31

vedantlodha requested a review from utkarsharma2 June 13, 2023 06:31

utkarsharma2 reviewed Jun 14, 2023

View reviewed changes

vedantlodha added 6 commits June 20, 2023 14:56

Allow variables to be printed to STDOUT with airflow variables export.

c3294af

Similar to connections, this change allows the cli to export the variables to the stdout using 'airflow variables export -' Fixes apache#31851

run pre-commit on the changes.

f632a1a

- improve variable naming

dc60166

- handle JSONDecodeError instead of Exception

remove helper method file_is_stdout

b24e0f8

run pre-commit check

52da697

improve variable naming

99e26ef

vedantlodha force-pushed the 31851 branch from 320a4ff to 99e26ef Compare June 20, 2023 09:27

uranusjr reviewed Jun 20, 2023

View reviewed changes

Comment thread airflow/cli/commands/variable_command.py Outdated

Simplify variable reference

7eb0a4c

uranusjr reviewed Jun 20, 2023

View reviewed changes

github-actions Bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Aug 4, 2023

github-actions Bot closed this Aug 10, 2023

uranusjr mentioned this pull request Aug 10, 2023

Allow 'airflow variables export' to print to stdout #33279

Merged

-        query = session.query(Variable).all()
-        data = json.JSONDecoder()
-        for variable in query:
-            try:
-                value = data.decode(variable.val)
-            except json.decoder.JSONDecodeError:
-                value = variable.val
-            var_dict[variable.key] = value
+        rows = session.query(Variable).all()
+        data = json.JSONDecoder()
+        for row in rows:
+            try:
+                value = data.decode(row.val)
+            except json.decoder.JSONDecodeError:
+                value = row.val
+            var_dict[row.key] = value

	f.write(json.dumps(var_dict, sort_keys=True, indent=4))
	json.dump(f, var_dict, sort_keys=True, indent=4)

Uh oh!

Conversation

vedantlodha commented Jun 12, 2023

Uh oh!

pankajastro left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vedantlodha commented Jun 12, 2023

Uh oh!

pankajastro commented Jun 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

utkarsharma2 Jun 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vedantlodha commented Jun 13, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uranusjr Jun 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pankajastro left a comment •

edited

Loading

pankajastro commented Jun 12, 2023 •

edited

Loading

utkarsharma2 Jun 13, 2023 •

edited

Loading

uranusjr Jun 14, 2023 •

edited

Loading