Default to latest logs and fix scroll bar jump performance #54174 by sujitha-saranam · Pull Request #54447 · apache/airflow

sujitha-saranam · 2025-08-13T09:49:23Z

Overview

This PR addresses #54174. It fixes the default scroll to the most recent logs and ensures that jump-to-start/end buttons scroll instantly.

Root Cause:

Jumping to the end is slow because the virtualizer has to compute/measure positions for a large number of rows before it can accurately place the scroll. For very long logs, this causes the temporary slow scroll effect.

Solution

Implemented precise offset calculation for hash-based navigation to jump directly to a target log.
Enhanced jump-to-start and jump-to-end logic to bypass gradual virtualizer measurement, allowing instant scrolling.
Updated highlight calculation to work correctly with the reversed order, ensuring the selected log is always visually marked.

Changes Made

Updated useLayoutEffect to compute exact scroll offsets based on virtual item positions.
Modified handleScrollTo to combine scrollToIndex with scrollTop via requestAnimationFrame for instant jumps.
Adjusted the bgColor calculation for virtualized items to correctly highlight the log matching the hash.
Tweak Code and VStack container styling to match virtualized content height and improve scroll behavior.

closes: #54174
related: #54174

closes: #51788

pierrejeambrun

Thank you for the pull request.

I wouldn't reverse the logs in the frontend because it will cause many different problem, maybe we can just scroll to bottom on loading. (log reverse should be done in the backend, because if we start paginating / streaming logs reversing just in the UI will not work)

With the current implementation I see a few issues:
Groups are weird

First line is truncated

Once I click on the "scroll to bottom" button, I cannot scroll up.

Screen.Recording.2025-08-18.at.18.07.14.mov

sujitha-saranam · 2025-08-20T13:32:49Z

Hi @pierrejeambrun

Screencast.from.2025-08-20.18-59-02.mp4

I have fixed the logs reversal from the backend and also addressed the UI issues. I’ve attached a screen recording of my local setup showing these updates. Could you please check and let me know if everything looks fine, or if there are any external changes I should also make?

Thanks!

…while jumping to the end of the logs

This reverts commit 5af219a.

pierrejeambrun

Why do we need to update the backend ordering there? Also I didn't take a closer look but we most probably can't do that because that should be at the reader lvl to reverse order. (start from the end of the file and cursor back to the beginning)

jason810496

Even we have the another UI endpoint for this feature, the builit-in reversed will still cause the OOM problem for API server when the log is large.

jscheffl · 2025-08-25T19:00:03Z

Iactually would not like to read logs in reverse order. I this is made then it would need to be an optional feature, should not require all users to read logs in reverse order.

uranusjr · 2025-08-26T08:46:18Z

-        log_stream = task_log_reader.read_log_stream(ti, try_number, metadata)  # type: ignore[arg-type]
+        ndjson_log_stream = list(task_log_reader.read_log_stream(ti, try_number, metadata))  # type: ignore[arg-type]
+        reversed_stream = reversed(ndjson_log_stream)
+        log_stream = (line for line in reversed_stream)


Isn’t this just iter(reversed_stream)?

sujitha-saranam · 2025-08-26T11:17:22Z

I am reverting the back-end changes as they were optional. Please let me know if any other changes i should do.

pierrejeambrun

Overall looks good. Tested locally and working.

A few adjustments/cleaning/simplification suggestions before we can merge.

pierrejeambrun · 2025-09-02T12:07:30Z

+      return;
+    }
+
+    const targetIndex = Math.max(0, Math.min(parsedLogs.length - 1, Number(hash) || 0));
+    const el = parentRef.current;
+
+    const total = rowVirtualizer.getTotalSize();
+    const clientH = el?.clientHeight ?? 0;


This breaks the direct link to lines. We used to add 5 more lines so it wouldn't be on the edge of the container, f screenshot.

Both clicking on a log line, or refreshing the page with a URL with a line number specified will not land where we want. http://localhost:28080/dags/big_logs/runs/manual__2025-09-02T12:02:44.060539+00:00/tasks/my_task?try_number=1#4806

Can we use something like https://developer.mozilla.org/en-US/docs/Web/CSS/scroll-margin ?

pierrejeambrun · 2025-09-02T12:07:47Z

+      return;
+    }
+
+    // === bottom === (instant jump even for huge lists)


There's a comment for bottom, but not for top. Remove it

Suggested change

// === bottom === (instant jump even for huge lists)

pierrejeambrun · 2025-09-02T12:09:57Z

+      try {
+        rowVirtualizer.scrollToOffset(offset);
+      } catch {
+        rowVirtualizer.scrollToIndex(parsedLogs.length - 1, { align: "end" });
+      }


This should be enough. We can remove the typeof check. (undefined is not callable) (same above)

pierrejeambrun · 2025-09-02T12:14:18Z

+    requestAnimationFrame(() => {
+      el.scrollTop = offset;
+      requestAnimationFrame(() => {
+        el.scrollTop = offset;
+        const lastItem = el.querySelector<HTMLElement>(
+          `[data-testid="virtualized-item-${parsedLogs.length - 1}"]`,
+        );
+
+        if (lastItem) {
+          lastItem.scrollIntoView({ behavior: "auto", block: "end" });
+        }
+      });
+    });


What are those requestAnimationFrame achieving? I Tried without them and it still look fine.

eladkal · 2025-09-15T03:50:42Z

@sujitha-saranam can you address above comments?

github-actions · 2025-11-20T00:18:17Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

sujitha-saranam requested review from bbovenzi, jscheffl, pierrejeambrun, ryanahamilton and shubhamraj-git as code owners August 13, 2025 09:49

boring-cyborg Bot added the area:UI Related to UI/UX. For Frontend Developers. label Aug 13, 2025

sujitha-saranam mentioned this pull request Aug 13, 2025

Fix jump to most recent logs and jump by default #54174

Closed

1 task

pierrejeambrun reviewed Aug 18, 2025

View reviewed changes

sujitha-saranam added 2 commits August 20, 2025 20:06

updated logs to newest on the top and also fixed scroll bar fucntion …

423405a

…while jumping to the end of the logs

Added reversal logs logic from backend and updated ui code

ac154bb

sujitha-saranam force-pushed the 54174-fix-large-logs-scrolling branch from 639c813 to ac154bb Compare August 20, 2025 14:50

sujitha-saranam requested review from bugraoz93, ephraimbuddy, jason810496 and rawwar as code owners August 20, 2025 14:50

Merge branch 'main' into 54174-fix-large-logs-scrolling

9cc119a

sujitha-saranam marked this pull request as draft August 20, 2025 17:02

sujitha-saranam marked this pull request as ready for review August 21, 2025 08:03

Merge branch 'main' into 54174-fix-large-logs-scrolling

c6a63cb

sujitha-saranam marked this pull request as draft August 21, 2025 08:04

updated reverse logs in ndjson response

43c02f4

sujitha-saranam marked this pull request as ready for review August 21, 2025 08:30

sujitha-saranam added 5 commits August 21, 2025 14:01

Merge branch 'main' into 54174-fix-large-logs-scrolling

b2e1c8a

Updated the response

5af219a

Merge branch 'main' into 54174-fix-large-logs-scrolling

23832d8

Revert "Updated the response"

7784f43

This reverts commit 5af219a.

fixed the test case changes as the response is now reverse

2c70a60

pierrejeambrun reviewed Aug 22, 2025

View reviewed changes

Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/TaskLogContent.tsx Outdated

Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/TaskLogContent.tsx

jason810496 reviewed Aug 25, 2025

View reviewed changes

Comment thread airflow-core/src/airflow/api_fastapi/core_api/routes/public/log.py Outdated

Comment thread airflow-core/src/airflow/api_fastapi/core_api/routes/public/log.py Outdated

uranusjr reviewed Aug 26, 2025

View reviewed changes

sujitha-saranam marked this pull request as draft August 26, 2025 09:14

sujitha-saranam marked this pull request as ready for review August 26, 2025 10:56

sujitha-saranam added 2 commits August 26, 2025 17:29

reverted backend changes, test log changes were also reverted

d15ee49

Merge branch 'main' into 54174-fix-large-logs-scrolling

4e5c4cb

pierrejeambrun added the backport-to-v3-0-test label Sep 2, 2025

pierrejeambrun added this to the Airflow 3.0.7 milestone Sep 2, 2025

pierrejeambrun reviewed Sep 2, 2025

View reviewed changes

kaxil modified the milestones: Airflow 3.0.7, Airflow 3.1.0, Airflow 3.2.0 Sep 13, 2025

kaxil removed the backport-to-v3-0-test label Sep 14, 2025

pierrejeambrun mentioned this pull request Sep 16, 2025

Scrolling in logs does not work properly. #51788

Closed

2 tasks

github-actions Bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 20, 2025

github-actions Bot closed this Nov 26, 2025

guan404ming mentioned this pull request Jan 20, 2026

Fix slow log scrolling for large task logs #60806

Merged

1 task

Conversation

sujitha-saranam commented Aug 13, 2025 • edited by pierrejeambrun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Root Cause:

Solution

Changes Made

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sujitha-saranam commented Aug 20, 2025

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jscheffl commented Aug 25, 2025

Uh oh!

uranusjr Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

sujitha-saranam commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

ashb Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

eladkal commented Sep 15, 2025

Uh oh!

github-actions Bot commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

sujitha-saranam commented Aug 13, 2025 •

edited by pierrejeambrun

Loading

pierrejeambrun left a comment •

edited

Loading

sujitha-saranam commented Aug 26, 2025 •

edited

Loading

pierrejeambrun left a comment •

edited

Loading