Fix performance regression in Coordinates.to_index#11306
Conversation
The codes passed to pd.MultiIndex were being converted from cache-friendly ndarrays into Python lists to silence a mypy arg-type error introduced in pydata#10694. The extra per-element conversion dominates runtime for large indexes (~13s on a 100x2000x300 array). Pass the ndarrays directly and suppress the type error the same way as for `levels` just above. Fixes pydata#11305 Co-authored-by: Claude <noreply@anthropic.com>
|
Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient. |
|
cc @max-sixty (introduced the [This is Claude Code on behalf of Tim Hodson] |
|
@ianhi , I ran the script to verify the performance improvement. Since this is a triage one-liner, I'm going to open the PR. Thanks for your feedback. |
|
Looks like merging is blocked on the failing docs build (unrelated failure) - can you just push one more commit or force push to re-trigger the build @thodson-usgs |
Co-authored-by: Claude <noreply@anthropic.com>
Last login: Sat Apr 18 10:01:59 on ttys007
c% thodson@IGSASC167LTTOH2 spherely % cd ~/Desktop/research/manuscripts/adaptive-rating/package
thodson@IGSASC167LTTOH2 package % git status
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
(use "git push" to publish your local commits)
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: src/adaptive_rating/kalman.py
modified: src/adaptive_rating/model.py
modified: src/adaptive_rating/spline.py
modified: tests/test_model.py
Untracked files:
(use "git add <file>..." to include in what will be committed)
camels_name.txt
rating_demo_multisite.png
rating_demo_single_10154200.png
scripts/ablation_provo.py
scripts/compare_usgs_03339000.py
scripts/log_transform_comparison.py
scripts/monospline_test.png
scripts/rating_gp_demo.py
scripts/stress_test_camels.py
scripts/stress_test_control.py
scripts/stress_test_direct.py
scripts/stress_test_final.py
scripts/stress_test_v2.py
scripts/stress_test_v3.py
scripts/stress_test_v4.py
src/adaptive_rating/preprocessing.py
stress_test_best.png
stress_test_worst.png
tests/test_preprocessing.py
usgs_03339000_obs.png
usgs_03339000_slices.png
usgs_03339000_winter.png
no changes added to commit (use "git add" and/or "git commit -a")
thodson@IGSASC167LTTOH2 package % git add src/adaptive_rating/*.py
thodson@IGSASC167LTTOH2 package % git commit -m"WIP"
[main 58e17d4] WIP
4 files changed, 391 insertions(+), 140 deletions(-)
create mode 100644 src/adaptive_rating/preprocessing.py
thodson@IGSASC167LTTOH2 package % git push
Enumerating objects: 22, done.
Counting objects: 100% (22/22), done.
Delta compression using up to 12 threads
Compressing objects: 100% (12/12), done.
Writing objects: 100% (14/14), 8.61 KiB | 4.30 MiB/s, done.
Total 14 (delta 7), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (7/7), completed with 6 local objects.
To nodohs:nodohs/adaptive-rating.git
8719958..58e17d4 main -> main
thodson@IGSASC167LTTOH2 package % git branch
jump-process
* main
thodson@IGSASC167LTTOH2 package % cd ~/Desktop/dev/software/xarray
thodson@IGSASC167LTTOH2 xarray % claude
▗ ▗ ▖ ▖ Claude Code v2.1.114
Opus 4.7 (1M context) with xhigh effort · Claude Max
▘▘ ▝▝ ~/Desktop/dev/software/xarray
❯ I started https://github.com/pydata/xarray/pull/11306 . Can you think about this change one more time and whether it might introduce any
other bugs or failure modes. Thank you
⏺ Bash(git log -1 --stat f647a5cb)
⎿ commit f647a5cb27a65b4be18df6b6724bc5caa7ce148a
Author: thodson-usgs <thodson@usgs.gov>
Date: Sat Apr 18 09:42:37 2026 -0500
… +16 lines (ctrl+o to expand)
⏺ Bash(git show f647a5cb -- xarray/core/coordinates.py)
⎿ commit f647a5cb27a65b4be18df6b6724bc5caa7ce148a
Author: thodson-usgs <thodson@usgs.gov>
Date: Sat Apr 18 09:42:37 2026 -0500
… +25 lines (ctrl+o to expand)
Read 1 file (ctrl+o to expand)
⏺ Bash(git log --all --oneline -- xarray/core/coordinates.py | head -15)
⎿ f647a5cb Fix Coordinates.to_index performance regression
97254417 Add an `arithmetic_compat` option to `xr.set_options` (#10943)
8bed33fe Fix mypy errors with newer pandas-stubs (#11066)
… +12 lines (ctrl+o to expand)
⏺ Bash(git show a13a2556 -- xarray/core/coordinates.py | head -40)
⎿ commit a13a2556a29b3c5ba342a402b2598bab42939b46
Author: Maximilian Roos <5635139+max-sixty@users.noreply.github.com>
Date: Wed Sep 3 19:48:28 2025 -0700
… +34 lines (ctrl+o to expand)
Searched for 2 patterns (ctrl+o to expand)
⏺ Bash(uv run pytest xarray/tests/test_dataarray.py xarray/tests/test_dataset.py -k "to_index or stack or multi" -x -q 2>&1 | tail -30)
⎿ pyenv: uv: command not found
The `uv' command exists in these Python versions:
… +4 lines (ctrl+o to expand)
⏺ Bash(pyenv shell 3.13.3 && uv run pytest xarray/tests/test_dataarray.py xarray/tests/test_dataset.py -k "to_index or stack" -x -q 2>&1 |
tail -15)
⎿ ............................... [100%]
31 passed, 1039 deselected in 2.43s
⏺ Reviewing the one-line change:
codes=code_list, # was [list(c) for c in code_list]
Where code_list is list[np.ndarray], each element freshly built via np.tile(np.repeat(code, …)).
Failure modes I checked:
1. Value correctness — identical. list(ndarray) just produced numpy scalars that pandas re-packed; no data change.
2. Dtype — subtle behavioral shift: list(c) effectively widened codes to int64 inside pandas, whereas ndarrays keep their original dtype
(int8/16/32 from MultiIndex.codes, int64 from pd.factorize). pandas' MultiIndex.__new__ accepts any integer dtype and fits them to the
level size anyway, so this is a memory win, not a bug.
3. Empty cartesian product (cumprod_lengths[-1] == 0) — np.repeat(code, 0) → empty ndarray → np.tile preserves empty. pandas handles empty
codes; the old [list(c) …] path hit the same branch.
4. NaN / -1 sentinels from pd.factorize — preserved either way.
5. Aliasing — each ndarray is freshly constructed inside the list-comp, nothing external holds a reference, so handing them to pandas is
safe.
6. MultiIndex validation (verify_integrity=True default) — same codes, same levels, same result.
Pre-existing, unrelated to this PR: repeat_counts is float64 from the cumprod_lengths[-1] / cumprod_lengths division, and np.repeat(code,
float) will eventually be rejected by numpy. Worth noting but not in scope.
Tests pass (-k "to_index or stack", 31 passed). I don't see a new failure mode — the fix is tight.
✻ Crunched for 1m 51s
※ recap: You're shipping PR #11306 to fix the Coordinates.to_index performance regression, and asked me to sanity-check the one-line change
for new bugs. I reviewed it and found none; next step is yours to merge or request more checks. (disable recaps in /config)
❯ and you verified the timing improvement is real?
Read 1 file (ctrl+o to expand)
⏺ Bash(pyenv shell 3.13.3 && uv run python -c "
import timeit, numpy as np, xarray as xr…)
⎿ NEW (codes=code_list): min=0.224s, median=0.230s
⏺ Update(xarray/core/coordinates.py)
⎿ Added 1 line, removed 1 line
194
195 return pd.MultiIndex(
196 levels=level_list, # type: ignore[arg-type,unused-ignore]
197 - codes=code_list, # type: ignore[arg-type,unused-ignore]
197 + codes=[list(c) for c in code_list],
198 names=names,
199 )
200
⏺ Bash(pyenv shell 3.13.3 && uv run python -c "
import timeit, numpy as np, xarray as xr…)
⎿ OLD (list comprehension): min=9.575s, median=9.686s
⏺ Update(xarray/core/coordinates.py)
⎿ Added 1 line, removed 1 line
194
195 return pd.MultiIndex(
196 levels=level_list, # type: ignore[arg-type,unused-ignore]
197 - codes=[list(c) for c in code_list],
197 + codes=code_list, # type: ignore[arg-type,unused-ignore]
198 names=names,
199 )
200
⏺ Yes, just verified on a 100×2000×300 DataArray (to_dataframe):
- OLD ([list(c) for c in code_list]): median 9.69s
- NEW (codes=code_list): median 0.23s
~42× speedup on this shape — consistent with the ~13s → sub-second figure in the commit message. Restored the file to the fixed version
afterward.
✻ Churned for 1m 9s
※ recap: You're reviewing PR #11306 (fix for Coordinates.to_index perf regression) for any new bugs. I found none and confirmed a ~42×
speedup benchmark. Next: decide whether the PR is ready to merge. (disable recaps in /config)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯ /exit
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
/exit Exit the CLI
/extra-usage Configure extra usage to keep working when limits are hit
/context Visualize current context usage as a colored grid
/claude-api Build, debug, and optimize Claude API / Anthropic SDK apps. Apps built with this skill should include pro…
/memory Edit Claude memory files
/ultraplan a few minutes · Claude Code on the web drafts a plan you can edit and approve. See https://code.claude.co…
Resume this session with:
claude --resume 4ee8f3cf-e067-490b-8b95-a006572fecab
thodson@IGSASC167LTTOH2 xarray % cd ..
thodson@IGSASC167LTTOH2 software % ls
claude-code dataretrieval-python fluegg gumbridge ratingcurve spherely
cubed DataRetrieval.jl fluegg_prev hytest schwarz-bridge xarray
cubed-clone discontinuum frmc_mobile lithops schwarz-bridge-gpt xarray-regrid
cubed-xarray discontinuum-delete frmc_python model_evaluation_kit schwarz-demo xdscore
dataRetrieval earth2mip gplm_python numcodecs scratch zarr-python
thodson@IGSASC167LTTOH2 software % cd dataretrieval-python
thodson@IGSASC167LTTOH2 dataretrieval-python % git pull
Already up to date.
thodson@IGSASC167LTTOH2 dataretrieval-python % cd ../xarray
thodson@IGSASC167LTTOH2 xarray % ls
__pycache__ CODE_OF_CONDUCT.md HOW_TO_RELEASE.md script.py
_stubtest conftest.py LICENSE uv.lock
AI_POLICY.md CONTRIBUTING.md licenses xarray
asv_bench CORE_TEAM_GUIDE.md pixi.toml xarray.egg-info
ci DATATREE_MIGRATION_GUIDE.md properties
CITATION.cff design_notes pyproject.toml
CLAUDE.md doc README.md
thodson@IGSASC167LTTOH2 xarray % claude
▗ ▗ ▖ ▖ Claude Code v2.1.116
Opus 4.7 (1M context) with xhigh effort · Claude Max
▘▘ ▝▝ ~/Desktop/dev/software/xarray
! git branch
⎿ * fix/11305-to-index-perf-regression
main
! git checkout fix-sortby-tuple-4821
⎿ Switched to a new branch 'fix-sortby-tuple-4821'
branch 'fix-sortby-tuple-4821' set up to track 'origin/fix-sortby-tuple-4821'.
❯ Refer to https://github.com/pydata/xarray/pull/11297 I' submitted a bugfix to pytest-mypy-plugins, can you run the mypy tests locally (after
updating the environment) and see if that fixes the failure.
⏺ Bash(gh pr view 11297 --json title,body,files,state,headRefName,statusCheckRollup)
⎿ Error: Exit code 1
GraphQL: Could not resolve to a PullRequest with the number of 11297. (repository.pullRequest)
⏺ Bash(gh pr view 11297 --repo pydata/xarray --json title,body,files,state,headRefName,statusCheckRollup)
⎿ {"body":"## Summary\n\n- Fixes `da.sortby(da.dims)` raising `KeyError` when `da.dims` is a tuple (e.g. `('x', 'y')`)\n- Tuples are now
treated as iterables of variable names, consistent with the `str | Iterable[Hashable]` convention established in #6142\n\nCloses #482
1\n\n### Usage example\n\n```python\n>>> import xarray as xr\n>>> da = xr.DataArray(\n... [[1, 2], [3, 4], [5, 6]],\n... coord
… +101 lines (ctrl+o to expand)
⏺ Let me get the details on the failing mypy test.
⏺ Bash(gh run view 24522563933 --repo pydata/xarray --log-failed 2>&1 | head -200)
⎿ ubuntu-latest | test-py311-with-typing (mypy) UNKNOWN STEP 2026-04-16T16:46:15.1164683Z Current runner version: '2.333.1'
ubuntu-latest | test-py311-with-typing (mypy) UNKNOWN STEP 2026-04-16T16:46:15.1196068Z ##[group]Runner Image Provisioner
ubuntu-latest | test-py311-with-typing (mypy) UNKNOWN STEP 2026-04-16T16:46:15.1197310Z Hosted Compute Agent
… +211 lines (ctrl+o to expand)
⏺ Bash(gh run view 24522563933 --repo pydata/xarray --log-failed 2>&1 | grep -iE "(error|fail|mypy)" | head -50)
⎿ ubuntu-latest | test-py311-with-typing (mypy) UNKNOWN STEP 2026-04-16T16:46:15.1164683Z Current runner version: '2.333.1'
ubuntu-latest | test-py311-with-typing (mypy) UNKNOWN STEP 2026-04-16T16:46:15.1196068Z ##[group]Runner Image Provisioner
ubuntu-latest | test-py311-with-typing (mypy) UNKNOWN STEP 2026-04-16T16:46:15.1197310Z Hosted Compute Agent
… +47 lines (ctrl+o to expand)
⏺ Bash(gh run view 24522563933 --repo pydata/xarray --log-failed --job 71684084342 2>&1 > /tmp/mypy_fail_log.txt; wc -l /tmp/mypy_fail_log.txt)
⎿ 4173 /tmp/mypy_fail_log.txt
Searched for 1 pattern, read 1 file (ctrl+o to expand)
⏺ Confirmed the CI failure: pytest_mypy_plugins/item.py:404 store.close() -> AttributeError: 'FilesystemMetadataStore' object has no attribute
'close' with pytest-mypy-plugins 4.0.1. Let me find how to run these tests locally.
Searched for 4 patterns, listed 1 directory (ctrl+o to expand)
⏺ Bash(uv pip show pytest-mypy-plugins 2>&1 | head -20)
⎿ pyenv: uv: command not found
The `uv' command exists in these Python versions:
… +4 lines (ctrl+o to expand)
⏺ Bash(which -a uv python3; pyenv versions 2>/dev/null | head)
⎿ /Users/thodson/.pyenv/shims/uv
/Users/thodson/.pyenv/shims/python3
/opt/homebrew/bin/python3
… +6 lines (ctrl+o to expand)
⏺ Bash(ls /Users/thodson/.pyenv/versions/3.13.3/bin/uv 2>/dev/null; /Users/thodson/.pyenv/versions/3.13.3/bin/uv --version)
⎿ /Users/thodson/.pyenv/versions/3.13.3/bin/uv
uv 0.11.7 (9d177269e 2026-04-15 aarch64-apple-darwin)
⏺ Bash(/Users/thodson/.pyenv/versions/3.13.3/bin/uv pip show pytest-mypy-plugins 2>&1 | head -20)
⎿ Name: pytest-mypy-plugins
Version: 4.0.2
Location: /Users/thodson/Desktop/dev/software/xarray/.venv/lib/python3.13/site-packages
… +2 lines (ctrl+o to expand)
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv sync 2>&1 | tail -30)
⎿ Resolved 151 packages in 3ms
Checked 145 packages in 274ms
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv pip install --upgrade pytest-mypy-plugins 2>&1 | tail -20)
⎿ Resolved 22 packages in 260ms
Downloading mypy (12.7MiB)
Downloaded mypy
… +5 lines (ctrl+o to expand)
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv pip show pytest-mypy-plugins | head -5)
⎿ Name: pytest-mypy-plugins
Version: 4.0.2
Location: /Users/thodson/Desktop/dev/software/xarray/.venv/lib/python3.13/site-packages
… +2 lines (ctrl+o to expand)
Read 1 file (ctrl+o to expand)
⏺ The fix is already in 4.0.2 (getattr(store, "close", None) guard at line 406). Let me run the mypy tests locally now.
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv run pytest xarray/tests/test_dataarray_typing.yml
xarray/tests/test_dataset_typing.yml xarray/…)
⎿ Uninstalled 1 package in 176ms
Installed 1 package in 37ms
============================= test session starts ==============================
… +30 lines (ctrl+o to expand)
⎿ (timeout 10m)
⏺ The tests are all skipped. Let me find out why.
Searched for 1 pattern (ctrl+o to expand)
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv run pytest xarray/tests/test_dataarray_typing.yml
xarray/tests/test_dataset_typing.yml xarray/…)
⎿ E pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output:
E Actual:
E ../../../../../../Users/thodson/Desktop/dev/software/xarray/xarray/core/utils:117: error: Unused "type: ignore" comment [unused
… +36 lines (ctrl+o to expand)
⎿ (timeout 10m)
⏺ The original FilesystemMetadataStore.close bug is gone — the fix is in 4.0.2. The new failures are unrelated (Unused "type: ignore") — they're
caused by mypy 1.20.1 vs the CI-pinned 1.19.1. Let me downgrade mypy to match CI.
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv pip install "mypy==1.19.1" 2>&1 | tail -10)
⎿ Checked 1 package in 39ms
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv run --no-sync pytest xarray/tests/test_dataarray_typing.yml
xarray/tests/test_dataset_typing.y…)
⎿ FAILED xarray/tests/test_dataset_typing.yml::test_mypy_pipe_function_missing_kwarg
FAILED xarray/tests/test_dataset_typing.yml::test_mypy_pipe_function_missing_keyword
FAILED xarray/tests/test_dataset_typing.yml::test_mypy_pipe_tuple_return_type_dataset
… +17 lines (ctrl+o to expand)
⎿ (timeout 10m)
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv run --no-sync pytest
xarray/tests/test_dataarray_typing.yml::test_mypy_pipe_lambda_noarg_retur…)
⎿ ============================= test session starts ==============================
platform darwin -- Python 3.13.3, pytest-9.0.3, pluggy-1.6.0
rootdir: /Users/thodson/Desktop/dev/software/xarray
… +24 lines (ctrl+o to expand)
⎿ (timeout 2m)
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv pip show mypy | head -3)
⎿ Name: mypy
Version: 1.19.1
Location: /Users/thodson/Desktop/dev/software/xarray/.venv/lib/python3.13/site-packages
Read 2 files (ctrl+o to expand)
⏺ Let me verify the Unused "type: ignore" errors are a pre-existing local env issue (pandas-stubs mismatch) and not caused by the sortby PR.
⏺ Bash(git stash -u 2>&1 | head -3; git checkout main 2>&1 | tail -3)
⎿ Saved working directory and index state WIP on fix-sortby-tuple-4821: 704014dc Fix type hints
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv run --no-sync pytest
xarray/tests/test_dataarray_typing.yml::test_mypy_pipe_lambda_noarg_retur…)
⎿ xarray/tests/test_dataarray_typing.yml F [100%]
=================================== FAILURES ===================================
… +18 lines (ctrl+o to expand)
⎿ (timeout 2m)
⏺ Same failure on main — so the Unused "type: ignore" noise is a pre-existing local env issue (pandas-stubs mismatch with CI's pixi env), not
caused by the sortby PR. Restoring the branch.
⏺ Bash(git checkout fix-sortby-tuple-4821 2>&1 | tail -3; git stash pop 2>&1 | tail -5)
⎿ Switched to branch 'fix-sortby-tuple-4821'
Your branch is up to date with 'origin/fix-sortby-tuple-4821'.
(use "git add <file>..." to include in what will be committed)
… +4 lines (ctrl+o to expand)
⏺ Here's what I found:
Yes, your pytest-mypy-plugins bugfix resolves the CI failure.
The CI error was AttributeError: 'FilesystemMetadataStore' object has no attribute 'close' at pytest_mypy_plugins/item.py:404 store.close().
CI was pinned to pytest-mypy-plugins==4.0.1.
Locally with pytest-mypy-plugins==4.0.2 (which uv sync picks up — pyproject.toml already pins >=4.0.0), that same file now wraps the call in a
getattr(store, "close", None) guard at item.py:406-408. Running the 39 typing tests against your branch no longer crashes at teardown — so
the FilesystemMetadataStore.close failure is gone.
However, the tests still fail locally with a different, unrelated issue:
xarray/core/utils:117: error: Unused "type: ignore" comment [unused-ignore]
xarray/core/variable:73: error: Unused "type: ignore" comment [unused-ignore]
I confirmed this happens on main too (with the fixed plugin), so it's noise from my local env (pandas-stubs divergence from CI's pixi
test-py311-with-typing env, which pins mypy==1.19.1 plus specific stubs) — not caused by your sortby PR.
Recommendation: once CI picks up pytest-mypy-plugins>=4.0.2 (either by re-resolving the pixi lockfile or bumping the floor in
pixi.toml/pyproject.toml), the test-py311-with-typing (mypy) job should run the 39 pipe typing cases instead of crashing in teardown — so the
sortby PR should go green without any change to the PR itself.
✻ Baked for 9m 34s
※ recap: Verifying whether your pytest-mypy-plugins bugfix resolves the mypy CI failure on PR 11297; confirmed it does (4.0.2 removes the
FilesystemMetadataStore.close crash). Next: bump the pinned floor to pytest-mypy-plugins>=4.0.2 in pixi.toml so CI picks it up. (disable
recaps in /config)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
? for shortcuts control this session from your phone · /remote-control
⎿ Switched to branch 'fix-sortby-tuple-4821'
Your branch is up to date with 'origin/fix-sortby-tuple-4821'.
(use "git add <file>..." to include in what will be committed)
… +4 lines (ctrl+o to expand)
⏺ Here's what I found:
Yes, your pytest-mypy-plugins bugfix resolves the CI failure.
The CI error was AttributeError: 'FilesystemMetadataStore' object has no attribute 'close' at pytest_mypy_plugins/item.py:404
store.close(). CI was pinned to pytest-mypy-plugins==4.0.1.
Locally with pytest-mypy-plugins==4.0.2 (which uv sync picks up — pyproject.toml already pins >=4.0.0), that same file now wraps the
call in a getattr(store, "close", None) guard at item.py:406-408. Running the 39 typing tests against your branch no longer crashes at
teardown — so the FilesystemMetadataStore.close failure is gone.
However, the tests still fail locally with a different, unrelated issue:
xarray/core/utils:117: error: Unused "type: ignore" comment [unused-ignore]
xarray/core/variable:73: error: Unused "type: ignore" comment [unused-ignore]
I confirmed this happens on main too (with the fixed plugin), so it's noise from my local env (pandas-stubs divergence from CI's pixi
test-py311-with-typing env, which pins mypy==1.19.1 plus specific stubs) — not caused by your sortby PR.
Recommendation: once CI picks up pytest-mypy-plugins>=4.0.2 (either by re-resolving the pixi lockfile or bumping the floor in
pixi.toml/pyproject.toml), the test-py311-with-typing (mypy) job should run the 39 pipe typing cases instead of crashing in teardown —
so the sortby PR should go green without any change to the PR itself.
✻ Baked for 9m 34s
※ recap: Verifying whether your pytest-mypy-plugins bugfix resolves the mypy CI failure on PR 11297; confirmed it does (4.0.2 removes
the FilesystemMetadataStore.close crash). Next: bump the pinned floor to pytest-mypy-plugins>=4.0.2 in pixi.toml so CI picks it up.
(disable recaps in /config)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
? for shortcuts control this session from your phone · /remote-control
(use "git add <file>..." to include in what will be committed)
… +4 lines (ctrl+o to expand)
⏺ Here's what I found:
Yes, your pytest-mypy-plugins bugfix resolves the CI failure.
The CI error was AttributeError: 'FilesystemMetadataStore' object has no attribute 'close' at pytest_mypy_plugins/item.py:404 store.close().
CI was pinned to pytest-mypy-plugins==4.0.1.
Locally with pytest-mypy-plugins==4.0.2 (which uv sync picks up — pyproject.toml already pins >=4.0.0), that same file now wraps the call in a
getattr(store, "close", None) guard at item.py:406-408. Running the 39 typing tests against your branch no longer crashes at teardown — so
the FilesystemMetadataStore.close failure is gone.
However, the tests still fail locally with a different, unrelated issue:
xarray/core/utils:117: error: Unused "type: ignore" comment [unused-ignore]
xarray/core/variable:73: error: Unused "type: ignore" comment [unused-ignore]
I confirmed this happens on main too (with the fixed plugin), so it's noise from my local env (pandas-stubs divergence from CI's pixi
test-py311-with-typing env, which pins mypy==1.19.1 plus specific stubs) — not caused by your sortby PR.
Recommendation: once CI picks up pytest-mypy-plugins>=4.0.2 (either by re-resolving the pixi lockfile or bumping the floor in
pixi.toml/pyproject.toml), the test-py311-with-typing (mypy) job should run the 39 pipe typing cases instead of crashing in teardown — so the
sortby PR should go green without any change to the PR itself.
✻ Baked for 9m 34s
※ recap: Verifying whether your pytest-mypy-plugins bugfix resolves the mypy CI failure on PR 11297; confirmed it does (4.0.2 removes the
FilesystemMetadataStore.close crash). Next: bump the pinned floor to pytest-mypy-plugins>=4.0.2 in pixi.toml so CI picks it up. (disable
recaps in /config)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
? for shortcuts control this session from your phone · /remote-control
⎿ (timeout 2m)
⏺ Same failure on main — so the Unused "type: ignore" noise is a pre-existing local env issue (pandas-stubs mismatch with CI's pixi
env), not caused by the sortby PR. Restoring the branch.
⏺ Bash(git checkout fix-sortby-tuple-4821 2>&1 | tail -3; git stash pop 2>&1 | tail -5)
⎿ Switched to branch 'fix-sortby-tuple-4821'
Your branch is up to date with 'origin/fix-sortby-tuple-4821'.
(use "git add <file>..." to include in what will be committed)
… +4 lines (ctrl+o to expand)
⏺ Here's what I found:
Yes, your pytest-mypy-plugins bugfix resolves the CI failure.
The CI error was AttributeError: 'FilesystemMetadataStore' object has no attribute 'close' at pytest_mypy_plugins/item.py:404
store.close(). CI was pinned to pytest-mypy-plugins==4.0.1.
Locally with pytest-mypy-plugins==4.0.2 (which uv sync picks up — pyproject.toml already pins >=4.0.0), that same file now wraps the
call in a getattr(store, "close", None) guard at item.py:406-408. Running the 39 typing tests against your branch no longer crashes at
teardown — so the FilesystemMetadataStore.close failure is gone.
However, the tests still fail locally with a different, unrelated issue:
xarray/core/utils:117: error: Unused "type: ignore" comment [unused-ignore]
xarray/core/variable:73: error: Unused "type: ignore" comment [unused-ignore]
I confirmed this happens on main too (with the fixed plugin), so it's noise from my local env (pandas-stubs divergence from CI's pixi
test-py311-with-typing env, which pins mypy==1.19.1 plus specific stubs) — not caused by your sortby PR.
Recommendation: once CI picks up pytest-mypy-plugins>=4.0.2 (either by re-resolving the pixi lockfile or bumping the floor in
pixi.toml/pyproject.toml), the test-py311-with-typing (mypy) job should run the 39 pipe typing cases instead of crashing in teardown —
so the sortby PR should go green without any change to the PR itself.
✻ Baked for 9m 34s
※ recap: Verifying whether your pytest-mypy-plugins bugfix resolves the mypy CI failure on PR 11297; confirmed it does (4.0.2 removes
the FilesystemMetadataStore.close crash). Next: bump the pinned floor to pytest-mypy-plugins>=4.0.2 in pixi.toml so CI picks it up.
(disable recaps in /config)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
? for shortcuts control this session from your phone · /remote-control
(use "git add <file>..." to include in what will be committed)
… +4 lines (ctrl+o to expand)
⏺ Here's what I found:
Yes, your pytest-mypy-plugins bugfix resolves the CI failure.
The CI error was AttributeError: 'FilesystemMetadataStore' object has no attribute 'close' at pytest_mypy_plugins/item.py:404 store.close().
CI was pinned to pytest-mypy-plugins==4.0.1.
Locally with pytest-mypy-plugins==4.0.2 (which uv sync picks up — pyproject.toml already pins >=4.0.0), that same file now wraps the call in a
getattr(store, "close", None) guard at item.py:406-408. Running the 39 typing tests against your branch no longer crashes at teardown — so
the FilesystemMetadataStore.close failure is gone.
However, the tests still fail locally with a different, unrelated issue:
xarray/core/utils:117: error: Unused "type: ignore" comment [unused-ignore]
xarray/core/variable:73: error: Unused "type: ignore" comment [unused-ignore]
I confirmed this happens on main too (with the fixed plugin), so it's noise from my local env (pandas-stubs divergence from CI's pixi
test-py311-with-typing env, which pins mypy==1.19.1 plus specific stubs) — not caused by your sortby PR.
Recommendation: once CI picks up pytest-mypy-plugins>=4.0.2 (either by re-resolving the pixi lockfile or bumping the floor in
pixi.toml/pyproject.toml), the test-py311-with-typing (mypy) job should run the 39 pipe typing cases instead of crashing in teardown — so the
sortby PR should go green without any change to the PR itself.
✻ Baked for 9m 34s
※ recap: Verifying whether your pytest-mypy-plugins bugfix resolves the mypy CI failure on PR 11297; confirmed it does (4.0.2 removes the
FilesystemMetadataStore.close crash). Next: bump the pinned floor to pytest-mypy-plugins>=4.0.2 in pixi.toml so CI picks it up. (disable
recaps in /config)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯ /exit
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
/exit Exit the CLI
/extra-usage Configure extra usage to keep working when limits are hit
/context Visualize current context usage as a colored grid
/claude-api Build, debug, and optimize Claude API / Anthropic SDK apps. Apps built with this skill should include prompt …
/memory Edit Claude memory files
/ultraplan a few minutes · Claude Code on the web drafts a plan you can edit and approve. See https://code.claude.com/do…
Resume this session with:
claude --resume d7e3e421-faa6-4d82-bc55-0dc4683f21d7
thodson@IGSASC167LTTOH2 xarray % cd ..
thodson@IGSASC167LTTOH2 software % ls
claude-code dataretrieval-python fluegg gumbridge ratingcurve spherely
cubed DataRetrieval.jl fluegg_prev hytest schwarz-bridge xarray
cubed-clone discontinuum frmc_mobile lithops schwarz-bridge-gpt xarray-regrid
cubed-xarray discontinuum-delete frmc_python model_evaluation_kit schwarz-demo xdscore
dataRetrieval earth2mip gplm_python numcodecs scratch zarr-python
thodson@IGSASC167LTTOH2 software % git clone --depth 10 git@github.com:thodson-usgs/sparse.git
Cloning into 'sparse'...
remote: Enumerating objects: 206, done.
remote: Counting objects: 100% (206/206), done.
remote: Compressing objects: 100% (186/186), done.
remote: Total 206 (delta 40), reused 92 (delta 11), pack-reused 0 (from 0)
Receiving objects: 100% (206/206), 748.15 KiB | 5.12 MiB/s, done.
Resolving deltas: 100% (40/40), done.
thodson@IGSASC167LTTOH2 software % cd sparse
thodson@IGSASC167LTTOH2 sparse % ls
benchmarks docs mkdocs.yml pyproject.toml release-procedure.md sparse
benchmarks_original examples pixi.lock pytest.ini scripts tox.ini
conftest.py LICENSE pixi.toml README.md setup.cfg xp-tests
thodson@IGSASC167LTTOH2 sparse % claude
╭─── Claude Code v2.1.116 ───────────────────────────────────────────────────────────────────────────────────────────────╮
│ │ Tips for getting started │
│ Welcome back Timothy! │ Run /init to create a CLAUDE.md file with instructions for Claude │
│ │ ───────────────────────────────────────────────────────────────── │
│ ▗ ▗ ▖ ▖ │ Recent activity │
│ │ No recent activity │
│ ▘▘ ▝▝ │ │
│ Opus 4.7 (1M context) with xh… · Claude Max · │ │
│ tohodson@gmail.com's Organization │ │
│ ~/Desktop/dev/software/sparse │ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
❯ /init and review the contribution guidelines and dev environment config and set up a local dev environment for this repo
Read 6 files, listed 4 directories (ctrl+o to expand)
Read 7 files, listed 1 directory (ctrl+o to expand)
⏺ Bash(which pixi 2>/dev/null; pixi --version 2>/dev/null || echo "pixi not installed")
⎿ Running…
4 tasks (0 done, 1 in progress, 3 open)
◼ Explore repo architecture and dev docs
◻ Review contribution guidelines
◻ Set up local dev environment
◻ Write CLAUDE.md
──────────────────────────────────────────────────────────────────────────────────────…

Summary
Fixes #11305.
The
codespassed topd.MultiIndexinCoordinates.to_indexwere being converted from the cache-friendly ndarrays produced bynp.tile/np.repeatinto Pythonlists. That conversion was introduced in #10694 to silence a mypyarg-typeerror (pandas-stubs declarescodes: Sequence[Sequence[int]]). The per-element materialisation dominates runtime for large indexes.This PR passes
code_listdirectly and suppresses the mypy complaint the same way we already do forlevelson the line above (# type: ignore[arg-type,unused-ignore]).Verified that
pd.MultiIndexwith ndarraycodesproduces a result that is.identical()to the list-of-int version (same internal code dtype, same values), so the change is purely a performance fix.MVCE (from #11305)
Run with
uv run script.py. Local results on my machine:to_dataframemain(before)~40× speed-up, matching the ~10s reporter saw.
Test plan
Ran locally with
uv run pytest -n auto:xarray/tests/test_coordinates.py— 27 passedxarray/tests/test_variable.py,test_concat.py,test_missing.py— 827 passed, 69 skipped, 9 xfailed, 3 xpassedxarray/tests/test_dataset.py,test_dataarray.py,test_groupby.py— 1458 passed, 91 skipped, 5 xfailed, 2 xpassedpre-commitclean on touched files, no new mypy errors introduced onxarray/core/coordinates.pyI grepped for callers of
Coordinates.to_index(xarray/core/{dataset,dataarray,variable,indexes,missing,accessor_dt}.py,xarray/plot/facetgrid.py,xarray/groupers.py) — all paths produce apd.MultiIndexwhose consumers rely on.values/.get_loc/ iteration, which are unaffected by whethercodeswas built from ndarrays or lists.[This is Claude Code on behalf of Tim Hodson]