Skip to content

feat: AJDA-2445 add command to migrate data-apps orchestrator/flow tasks to data-app-control#101

Open
sykora-ji wants to merge 4 commits into
mainfrom
sykorajiri-AJDA-2445
Open

feat: AJDA-2445 add command to migrate data-apps orchestrator/flow tasks to data-app-control#101
sykora-ji wants to merge 4 commits into
mainfrom
sykorajiri-AJDA-2445

Conversation

@sykora-ji

@sykora-ji sykora-ji commented Jul 9, 2026

Copy link
Copy Markdown

Link to issue

Description

Adds a new manage:migrate-data-apps-orchestrator-tasks command to cli-utils that rewrites orchestration/flow tasks referencing the legacy keboola.data-apps component so they use keboola.data-app-control instead.

  • DataAppOrchestratorTaskMigrator (new): given a Storage API Components client, lists all configurations of both keboola.orchestrator (legacy orchestrations) and keboola.flow (conditional flows — verified on a live stack to be a distinct component, not a variant of keboola.orchestrator), scans each configuration's tasks[] for tasks pointing at keboola.data-apps, resolves the app ID from either inline configData or a referenced configId, and rewrites the task's componentId to keboola.data-app-control with parameters.appId set — while preserving every other field on the task object (type, mode, delay, retry, variableOverrides) unchanged, since the keboola.flow schema requires type alongside componentId/mode. Tasks whose keboola.data-apps parameters carry an unsupported task value (anything other than the implicit/app-start/start variants used to launch an app) are left untouched and reported separately, so unrelated task types (create/delete/terminate/restore/...) are never silently mistransformed. Already-migrated tasks are skipped, making re-runs idempotent.
  • MigrateDataAppsOrchestratorTasks (new command): accepts a Manage API token, stack URL, and either a comma-separated list of project IDs or all. For each target project it mints a short-lived, scoped Storage API token via createProjectStorageToken() and runs the migrator against it; failures on one project are caught and logged without aborting the run. Defaults to dry-run (only reports what would change); --force/-f performs the actual updateConfiguration calls with a changeDescription. Prints a summary of projects/configurations/tasks scanned, migrated, and skipped.
  • Registered in cli.php and documented in README.md under "Project manipulation".

Manually verified end-to-end against a test flow on the canary-orion stack (dry-run, force run, idempotent re-run, and schema validity after the rewrite) before opening this PR.

Release Notes

  • Justification
    • The platform is moving data-app orchestration/flow tasks from the legacy keboola.data-apps component to the newer keboola.data-app-control component. Orchestrations and conditional flows created before this change still reference the legacy component, so a one-off bulk-migration tool is needed to move them over before the legacy component/UI path can be retired.
    • No behavior change for end users of Keboola Connection — this is an internal operations tool used by the AJDA team to update stored configurations; the new component performs the same "start app" action as the old one.
  • Plans for Customer Communication
    • No customer communication needed. This is an internal maintenance tool, not a customer-facing feature or behavioral change.
  • Impact Analysis
    • The tool only mutates orchestrator/flow configurations that already reference keboola.data-apps; everything else is left untouched. Tasks with an unrecognized keboola.data-apps parameter shape are skipped rather than guessed at, to avoid silently breaking a flow.
    • Not behind a feature flag — this is a manually-triggered CLI command run against a chosen project or stack, not part of the platform's request path, so there is no partial rollout state. Single-tenant setups are affected the same way as any other project: only if they have an orchestration/flow with a keboola.data-apps task.
  • Deployment Plan
    • No special rollout — merging deploys the updated cli-utils Docker image via the existing CI pipeline (tag push). The command itself is then run manually by an engineer, per project or per stack, as needed; it is not invoked automatically.
  • Rollback Plan
    • The tool change itself can be reverted/redeployed like any other cli-utils change. The configuration rewrites it performs are not automatically reversible, but each affected orchestrator/flow configuration keeps its Storage API version history (with the changeDescription set by this command), so any specific config can be rolled back to a prior version via Storage API if needed.
  • Post-Release Support Plan
    • No automated monitoring needed since the command is run on demand, and merging this PR does not itself run it against any project. Support does not need to be notified for the merge/deploy. Before the AJDA team actually executes --force against production projects/stacks, they will run a dry-run first to produce a report of every affected project/configuration, and track that list in the AJDA-2445 Linear issue so the rollout itself is auditable.

…sks to data-app-control

Adds manage:migrate-data-apps-orchestrator-tasks, which rewrites orchestration
and conditional-flow tasks pointing at the legacy keboola.data-apps component
so they use keboola.data-app-control instead. Supports a single project or an
entire stack via a Manage token, defaults to dry-run, and is idempotent.
@linear-code

linear-code Bot commented Jul 9, 2026

Copy link
Copy Markdown

AJDA-2445

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an operational CLI command to bulk-migrate orchestrator/flow tasks from the legacy keboola.data-apps component to keboola.data-app-control, with a dry-run default and an opt-in force mode to persist changes via Storage API.

Changes:

  • Introduces manage:migrate-data-apps-orchestrator-tasks command to iterate selected or all projects, mint temporary Storage tokens, and run the migration (dry-run by default; --force updates configs).
  • Adds DataAppOrchestratorTaskMigrator to scan keboola.orchestrator and keboola.flow configurations, rewrite eligible tasks, and summarize migrated vs skipped tasks.
  • Registers the command in cli.php and documents usage/behavior in README.md.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/Keboola/Console/Command/MigrateDataAppsOrchestratorTasks.php New Symfony Console command wiring Manage API + Storage API access, project selection, dry-run/force execution, and summary output.
src/Keboola/Console/Command/DataAppOrchestratorTaskMigrator.php Implements the config/task scanning and rewrite logic for orchestrator/flow configurations.
README.md Documents the new command under “Project manipulation”, including args/options and behavior.
cli.php Registers the new command in the CLI application bootstrap.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Keboola/Console/Command/MigrateDataAppsOrchestratorTasks.php
Comment thread src/Keboola/Console/Command/DataAppOrchestratorTaskMigrator.php
Comment thread src/Keboola/Console/Command/MigrateDataAppsOrchestratorTasks.php Outdated
Validate the projects argument strictly instead of silently dropping
invalid IDs, catch Storage API errors when resolving a configId reference
so one bad reference doesn't abort the whole project, cache resolved
configIds to avoid repeated lookups, and fix the "Checked N projects"
summary to include disabled/errored projects.
@sykora-ji sykora-ji requested a review from MiroCillik July 9, 2026 11:11
@sykora-ji sykora-ji marked this pull request as ready for review July 9, 2026 11:11

@MiroCillik MiroCillik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Solid, careful PR — dry-run by default, idempotent re-runs, short-lived scoped tokens, and per-project error isolation. I found no blocking correctness bugs. I confirmed via the Storage API SDK that listComponentConfigurations returns the full configuration payload by default, so the scan reads real task data.

Main non-blocking suggestions (see inline comments):

  1. Split the "couldn't resolve app id" case out of the generic tasksSkippedUnsupported counter — for an ops tool, an unresolvable/inaccessible referenced config needs human follow-up, whereas create/delete tasks are deliberately skipped; lumping them can hide a real problem.
  2. Consider a confirmation prompt for the --force all combination (stack-wide config mutation with no prompt; sibling commands use ConfirmationQuestion). Dry-run default mitigates this.
  3. The full replacement of configData deliberately drops any other task-level params — worth an explicit comment since it's the highest-impact behavioral assumption.

Nits: no unit tests for the very testable pure helpers (extractAppId/resolveAppId) — consistent with the repo having none, but this rewrites stored customer configs, so a couple of tests would be cheap insurance.

Also: the PR body still has an unresolved [TODO: confirm whether Support needs to be notified...] in the Post-Release Support Plan — worth resolving before merge.

Nothing here blocks merge. 👍

Comment thread src/Keboola/Console/Command/DataAppOrchestratorTaskMigrator.php
Comment thread src/Keboola/Console/Command/DataAppOrchestratorTaskMigrator.php Outdated
Comment thread src/Keboola/Console/Command/MigrateDataAppsOrchestratorTasks.php
Comment thread src/Keboola/Console/Command/MigrateDataAppsOrchestratorTasks.php
Jiří Novák added 2 commits July 10, 2026 14:07
Distinguish deliberately-skipped tasks (unsupported keboola.data-apps
task variants like create/delete/terminate) from tasks that should have
migrated but whose app id couldn't be resolved (missing/inaccessible
configId reference, malformed parameters) - the latter is now a separate
counter/message so it can't hide behind "unsupported shape" and needs
manual follow-up. Add an interactive confirmation guard for the
"all" + "--force" combination as a safety net against an accidental
stack-wide run. Document that the configData replacement on migrated
tasks is deliberate, not an oversight.
Adds phpunit/phpunit as a dev dependency and a small test suite covering
DataAppOrchestratorTaskMigrator (inline configData migration, unsupported
vs. unresolvable skip reasons, configId caching, phases/other task fields
preserved, scans both keboola.orchestrator and keboola.flow, idempotency)
and MigrateDataAppsOrchestratorTasks::parseProjectIds validation. Wires
the suite into CI. This command rewrites stored project configurations,
so the extra coverage is worth the one-off cost of introducing a test
harness to a repo that previously had none.
@sykora-ji

Copy link
Copy Markdown
Author

Addressed the nits from the review summary:

  • Added a PHPUnit test suite (tests/DataAppOrchestratorTaskMigratorTest.php, tests/MigrateDataAppsOrchestratorTasksTest.php) covering the migration/skip-reason logic, configId caching, idempotency, and parseProjectIds validation, wired into CI (composer tests) — given this rewrites stored project configs, the one-off cost of introducing a test harness to the repo felt justified.
  • Resolved the Post-Release Support Plan TODO in the PR description: merging doesn't execute anything against real projects; before the AJDA team runs --force against production, they'll dry-run first and track the affected-project list in AJDA-2445.

🤖 Addressed by Claude Code

@MiroCillik MiroCillik self-requested a review July 10, 2026 12:48

@MiroCillik MiroCillik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, muze se mergnout, ja to pak jeste nekde vyzkousim

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants