Skip to content

Fix performance regression in Coordinates.to_index#11306

Merged
ianhi merged 2 commits intopydata:mainfrom
thodson-usgs:fix/11305-to-index-perf-regression
Apr 22, 2026
Merged

Fix performance regression in Coordinates.to_index#11306
ianhi merged 2 commits intopydata:mainfrom
thodson-usgs:fix/11305-to-index-perf-regression

Conversation

@thodson-usgs
Copy link
Copy Markdown
Contributor

@thodson-usgs thodson-usgs commented Apr 18, 2026

Summary

Fixes #11305.

The codes passed to pd.MultiIndex in Coordinates.to_index were being converted from the cache-friendly ndarrays produced by np.tile/np.repeat into Python lists. That conversion was introduced in #10694 to silence a mypy arg-type error (pandas-stubs declares codes: Sequence[Sequence[int]]). The per-element materialisation dominates runtime for large indexes.

This PR passes code_list directly and suppresses the mypy complaint the same way we already do for levels on the line above (# type: ignore[arg-type,unused-ignore]).

Verified that pd.MultiIndex with ndarray codes produces a result that is .identical() to the list-of-int version (same internal code dtype, same values), so the change is purely a performance fix.

MVCE (from #11305)

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "xarray[complete]@git+https://github.com/pydata/xarray.git@refs/pull/11306/head",
# ]
# ///
import time
import numpy as np
import xarray as xr

da = xr.DataArray(np.arange(100 * 2000 * 300).reshape(100, 2000, 300), name="foo")
t = time.perf_counter()
da.to_dataframe()
print(f"to_dataframe: {time.perf_counter() - t:.3f}s")

Run with uv run script.py. Local results on my machine:

to_dataframe
main (before) 10.33s
this PR (after) 0.26s

~40× speed-up, matching the ~10s reporter saw.

Test plan

Ran locally with uv run pytest -n auto:

  • xarray/tests/test_coordinates.py — 27 passed
  • xarray/tests/test_variable.py, test_concat.py, test_missing.py — 827 passed, 69 skipped, 9 xfailed, 3 xpassed
  • xarray/tests/test_dataset.py, test_dataarray.py, test_groupby.py — 1458 passed, 91 skipped, 5 xfailed, 2 xpassed
  • pre-commit clean on touched files, no new mypy errors introduced on xarray/core/coordinates.py

I grepped for callers of Coordinates.to_index (xarray/core/{dataset,dataarray,variable,indexes,missing,accessor_dt}.py, xarray/plot/facetgrid.py, xarray/groupers.py) — all paths produce a pd.MultiIndex whose consumers rely on .values / .get_loc / iteration, which are unaffected by whether codes was built from ndarrays or lists.


[This is Claude Code on behalf of Tim Hodson]

The codes passed to pd.MultiIndex were being converted from cache-friendly
ndarrays into Python lists to silence a mypy arg-type error introduced in
pydata#10694. The extra per-element conversion dominates runtime for large indexes
(~13s on a 100x2000x300 array). Pass the ndarrays directly and suppress the
type error the same way as for `levels` just above.

Fixes pydata#11305

Co-authored-by: Claude <noreply@anthropic.com>
@welcome
Copy link
Copy Markdown

welcome Bot commented Apr 18, 2026

Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient.
If you have questions, some answers may be found in our contributing guidelines.

@thodson-usgs
Copy link
Copy Markdown
Contributor Author

cc @max-sixty (introduced the codes=[list(c) for c in code_list] conversion in #10694 to silence a mypy arg-type error) and @ianhi (reverted the analogous levels conversion in #10705 for dtype reasons, left the codes one in place) — curious if either of you has context on why the list(...) conversion was needed for codes specifically beyond the mypy stub, or any concern with the suppress-via-type-ignore approach used here.

[This is Claude Code on behalf of Tim Hodson]

@thodson-usgs thodson-usgs marked this pull request as ready for review April 18, 2026 15:12
@thodson-usgs thodson-usgs marked this pull request as draft April 18, 2026 16:03
@ianhi
Copy link
Copy Markdown
Collaborator

ianhi commented Apr 20, 2026

and @ianhi (reverted the analogous levels conversion in #10705 for dtype reasons, left the codes one in place) —

I strongly suspect this was an oversight on my part - or rather that I wasn't seeing an immediate error from it so I didn't fix it

@thodson-usgs
Copy link
Copy Markdown
Contributor Author

@ianhi , I ran the script to verify the performance improvement. Since this is a triage one-liner, I'm going to open the PR. Thanks for your feedback.

@thodson-usgs thodson-usgs marked this pull request as ready for review April 20, 2026 16:43
@ianhi
Copy link
Copy Markdown
Collaborator

ianhi commented Apr 22, 2026

Looks like merging is blocked on the failing docs build (unrelated failure) - can you just push one more commit or force push to re-trigger the build @thodson-usgs

Co-authored-by: Claude <noreply@anthropic.com>
@ianhi ianhi merged commit b8bfeca into pydata:main Apr 22, 2026
43 checks passed
@welcome
Copy link
Copy Markdown

welcome Bot commented Apr 22, 2026

Congratulations on completing your first pull request! Welcome to Xarray! We are proud of you, and hope to see you again! celebration gif

thodson-usgs added a commit to DOI-USGS/dataretrieval-python that referenced this pull request Apr 23, 2026
Last login: Sat Apr 18 10:01:59 on ttys007
c%                                                                                                                                          thodson@IGSASC167LTTOH2 spherely % cd ~/Desktop/research/manuscripts/adaptive-rating/package 
thodson@IGSASC167LTTOH2 package % git status
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   src/adaptive_rating/kalman.py
	modified:   src/adaptive_rating/model.py
	modified:   src/adaptive_rating/spline.py
	modified:   tests/test_model.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	camels_name.txt
	rating_demo_multisite.png
	rating_demo_single_10154200.png
	scripts/ablation_provo.py
	scripts/compare_usgs_03339000.py
	scripts/log_transform_comparison.py
	scripts/monospline_test.png
	scripts/rating_gp_demo.py
	scripts/stress_test_camels.py
	scripts/stress_test_control.py
	scripts/stress_test_direct.py
	scripts/stress_test_final.py
	scripts/stress_test_v2.py
	scripts/stress_test_v3.py
	scripts/stress_test_v4.py
	src/adaptive_rating/preprocessing.py
	stress_test_best.png
	stress_test_worst.png
	tests/test_preprocessing.py
	usgs_03339000_obs.png
	usgs_03339000_slices.png
	usgs_03339000_winter.png

no changes added to commit (use "git add" and/or "git commit -a")
thodson@IGSASC167LTTOH2 package % git add src/adaptive_rating/*.py
thodson@IGSASC167LTTOH2 package % git commit -m"WIP"
[main 58e17d4] WIP
 4 files changed, 391 insertions(+), 140 deletions(-)
 create mode 100644 src/adaptive_rating/preprocessing.py
thodson@IGSASC167LTTOH2 package % git push
Enumerating objects: 22, done.
Counting objects: 100% (22/22), done.
Delta compression using up to 12 threads
Compressing objects: 100% (12/12), done.
Writing objects: 100% (14/14), 8.61 KiB | 4.30 MiB/s, done.
Total 14 (delta 7), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (7/7), completed with 6 local objects.
To nodohs:nodohs/adaptive-rating.git
   8719958..58e17d4  main -> main
thodson@IGSASC167LTTOH2 package % git branch
  jump-process
* main
thodson@IGSASC167LTTOH2 package % cd ~/Desktop/dev/software/xarray
thodson@IGSASC167LTTOH2 xarray % claude
▗ ▗   ▖ ▖  Claude Code v2.1.114
           Opus 4.7 (1M context) with xhigh effort · Claude Max
  ▘▘ ▝▝    ~/Desktop/dev/software/xarray

❯ I started https://github.com/pydata/xarray/pull/11306 . Can you think about this change one more time and whether it might introduce any  
  other bugs or failure modes. Thank you                                                                                                    
                                                                                                                                            
⏺ Bash(git log -1 --stat f647a5cb)                                                                                                          
  ⎿  commit f647a5cb27a65b4be18df6b6724bc5caa7ce148a
     Author: thodson-usgs <thodson@usgs.gov>                                                                                                
     Date:   Sat Apr 18 09:42:37 2026 -0500
     … +16 lines (ctrl+o to expand)                                                                                                         
                                                                                                                                            
⏺ Bash(git show f647a5cb -- xarray/core/coordinates.py)
  ⎿  commit f647a5cb27a65b4be18df6b6724bc5caa7ce148a                                                                                        
     Author: thodson-usgs <thodson@usgs.gov>                                                                                                
     Date:   Sat Apr 18 09:42:37 2026 -0500
     … +25 lines (ctrl+o to expand)                                                                                                         
                                                                                                                                            
  Read 1 file (ctrl+o to expand)        
                                                                                                                                            
⏺ Bash(git log --all --oneline -- xarray/core/coordinates.py | head -15)                                                                    
  ⎿  f647a5cb Fix Coordinates.to_index performance regression           
     97254417 Add an `arithmetic_compat` option to `xr.set_options` (#10943)                                                                
     8bed33fe Fix mypy errors with newer pandas-stubs (#11066)                                                                              
     … +12 lines (ctrl+o to expand)                                                                                                         
                                                                                                                                            
⏺ Bash(git show a13a2556 -- xarray/core/coordinates.py | head -40)                                                                          
  ⎿  commit a13a2556a29b3c5ba342a402b2598bab42939b46                                                                                        
     Author: Maximilian Roos <5635139+max-sixty@users.noreply.github.com>                                                                   
     Date:   Wed Sep 3 19:48:28 2025 -0700                                                                                                  
     … +34 lines (ctrl+o to expand)                                                                                                         
                                                                                                                                            
  Searched for 2 patterns (ctrl+o to expand)      
                                                                                                                                            
⏺ Bash(uv run pytest xarray/tests/test_dataarray.py xarray/tests/test_dataset.py -k "to_index or stack or multi" -x -q 2>&1 | tail -30)     
  ⎿  pyenv: uv: command not found                                                                                                      
                                                                                                                                            
     The `uv' command exists in these Python versions:                                                                                      
     … +4 lines (ctrl+o to expand)                                                                                                          
                                                                                                                                            
⏺ Bash(pyenv shell 3.13.3 && uv run pytest xarray/tests/test_dataarray.py xarray/tests/test_dataset.py -k "to_index or stack" -x -q 2>&1 |  
      tail -15)                                                                                                                             
  ⎿  ...............................                                          [100%]                                                        
     31 passed, 1039 deselected in 2.43s                                                                                                    
                                                                                                                                            
⏺ Reviewing the one-line change:                                
                                                                                                                                            
  codes=code_list,  # was [list(c) for c in code_list]                                                                                      
                                                                                                                                            
  Where code_list is list[np.ndarray], each element freshly built via np.tile(np.repeat(code, …)).                                          
                                                                                                                                            
  Failure modes I checked:                                                                                                                  
                                                                                                                                            
  1. Value correctness — identical. list(ndarray) just produced numpy scalars that pandas re-packed; no data change.                        
  2. Dtype — subtle behavioral shift: list(c) effectively widened codes to int64 inside pandas, whereas ndarrays keep their original dtype
  (int8/16/32 from MultiIndex.codes, int64 from pd.factorize). pandas' MultiIndex.__new__ accepts any integer dtype and fits them to the    
  level size anyway, so this is a memory win, not a bug.
  3. Empty cartesian product (cumprod_lengths[-1] == 0) — np.repeat(code, 0) → empty ndarray → np.tile preserves empty. pandas handles empty
   codes; the old [list(c) …] path hit the same branch.                                                                                     
  4. NaN / -1 sentinels from pd.factorize — preserved either way.
  5. Aliasing — each ndarray is freshly constructed inside the list-comp, nothing external holds a reference, so handing them to pandas is  
  safe.                                                                                                                                     
  6. MultiIndex validation (verify_integrity=True default) — same codes, same levels, same result.                                          
                                                                                                                                            
  Pre-existing, unrelated to this PR: repeat_counts is float64 from the cumprod_lengths[-1] / cumprod_lengths division, and np.repeat(code, 
  float) will eventually be rejected by numpy. Worth noting but not in scope.                                                               
                                                                                                                                            
  Tests pass (-k "to_index or stack", 31 passed). I don't see a new failure mode — the fix is tight.                                        
  
✻ Crunched for 1m 51s                                                                                                                       
                  
※ recap: You're shipping PR #11306 to fix the Coordinates.to_index performance regression, and asked me to sanity-check the one-line change 
  for new bugs. I reviewed it and found none; next step is yours to merge or request more checks. (disable recaps in /config)
                                                                                                                                            
❯ and you verified the timing improvement is real?                                                                                          
                                                   
  Read 1 file (ctrl+o to expand)                                                                                                            
                                                                                                                                            
⏺ Bash(pyenv shell 3.13.3 && uv run python -c "
      import timeit, numpy as np, xarray as xr…)                                                                                            
  ⎿  NEW (codes=code_list): min=0.224s, median=0.230s               
                                                                                                                                            
⏺ Update(xarray/core/coordinates.py)                                                                                                        
  ⎿  Added 1 line, removed 1 line                                                                                                           
      194                                                                                                                                   
      195          return pd.MultiIndex(                                                                                                    
      196              levels=level_list,  # type: ignore[arg-type,unused-ignore]                                                           
      197 -            codes=code_list,  # type: ignore[arg-type,unused-ignore]                                                             
      197 +            codes=[list(c) for c in code_list],                                                                                  
      198              names=names,                                                                                                         
      199          )                                                                                                                      
      200                                                                                                                                   
                                                                                                                                            
⏺ Bash(pyenv shell 3.13.3 && uv run python -c "                                                                                             
      import timeit, numpy as np, xarray as xr…)                                                                                            
  ⎿  OLD (list comprehension): min=9.575s, median=9.686s                                                                                    
                                                                                                                                            
⏺ Update(xarray/core/coordinates.py)                                                                                                        
  ⎿  Added 1 line, removed 1 line                                                                                                           
      194                                                                                                                                   
      195          return pd.MultiIndex(                                                                                                    
      196              levels=level_list,  # type: ignore[arg-type,unused-ignore]                                                           
      197 -            codes=[list(c) for c in code_list],                                                                                  
      197 +            codes=code_list,  # type: ignore[arg-type,unused-ignore]                                                             
      198              names=names,                                                                                                         
      199          )                                        
      200  

⏺ Yes, just verified on a 100×2000×300 DataArray (to_dataframe):
                                                                    
  - OLD ([list(c) for c in code_list]): median 9.69s                                                                                        
  - NEW (codes=code_list): median 0.23s                                                                                                     
                                                                                                                                            
  ~42× speedup on this shape — consistent with the ~13s → sub-second figure in the commit message. Restored the file to the fixed version   
  afterward.                                                                                                                                
                                                                                                                                            
✻ Churned for 1m 9s                                         

※ recap: You're reviewing PR #11306 (fix for Coordinates.to_index perf regression) for any new bugs. I found none and confirmed a ~42×      
  speedup benchmark. Next: decide whether the PR is ready to merge. (disable recaps in /config)
                                                                                                                                            
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯ /exit 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
/exit                         Exit the CLI                                                                      
/extra-usage                  Configure extra usage to keep working when limits are hit                                                 
/context                      Visualize current context usage as a colored grid          
/claude-api                   Build, debug, and optimize Claude API / Anthropic SDK apps. Apps built with this skill should include pro…
/memory                       Edit Claude memory files                                                                                  
/ultraplan                    a few minutes · Claude Code on the web drafts a plan you can edit and approve. See https://code.claude.co…

Resume this session with:
claude --resume 4ee8f3cf-e067-490b-8b95-a006572fecab
thodson@IGSASC167LTTOH2 xarray % cd ..
thodson@IGSASC167LTTOH2 software % ls
claude-code		dataretrieval-python	fluegg			gumbridge		ratingcurve		spherely
cubed			DataRetrieval.jl	fluegg_prev		hytest			schwarz-bridge		xarray
cubed-clone		discontinuum		frmc_mobile		lithops			schwarz-bridge-gpt	xarray-regrid
cubed-xarray		discontinuum-delete	frmc_python		model_evaluation_kit	schwarz-demo		xdscore
dataRetrieval		earth2mip		gplm_python		numcodecs		scratch			zarr-python
thodson@IGSASC167LTTOH2 software % cd dataretrieval-python 
thodson@IGSASC167LTTOH2 dataretrieval-python % git pull
Already up to date.
thodson@IGSASC167LTTOH2 dataretrieval-python % cd ../xarray
thodson@IGSASC167LTTOH2 xarray % ls
__pycache__			CODE_OF_CONDUCT.md		HOW_TO_RELEASE.md		script.py
_stubtest			conftest.py			LICENSE				uv.lock
AI_POLICY.md			CONTRIBUTING.md			licenses			xarray
asv_bench			CORE_TEAM_GUIDE.md		pixi.toml			xarray.egg-info
ci				DATATREE_MIGRATION_GUIDE.md	properties
CITATION.cff			design_notes			pyproject.toml
CLAUDE.md			doc				README.md
thodson@IGSASC167LTTOH2 xarray % claude
▗ ▗   ▖ ▖  Claude Code v2.1.116                                                                                                                 
           Opus 4.7 (1M context) with xhigh effort · Claude Max                                                                                 
  ▘▘ ▝▝    ~/Desktop/dev/software/xarray                                                                                                        
                                                                                                                                                
! git branch                                                                                                                                    
  ⎿  * fix/11305-to-index-perf-regression                                                                                                       
       main                                                                                                                                     
                                                                                                                                                
! git checkout fix-sortby-tuple-4821                                                                                                            
  ⎿  Switched to a new branch 'fix-sortby-tuple-4821'                                                                                           
     branch 'fix-sortby-tuple-4821' set up to track 'origin/fix-sortby-tuple-4821'.                                                             
                                                                                                                                                
❯ Refer to https://github.com/pydata/xarray/pull/11297  I' submitted a bugfix to pytest-mypy-plugins, can you run the mypy tests locally (after 
   updating the environment) and see if that fixes the failure.                                                                                 
                                                                                                                                                
⏺ Bash(gh pr view 11297 --json title,body,files,state,headRefName,statusCheckRollup)                                                            
  ⎿  Error: Exit code 1                                                      
     GraphQL: Could not resolve to a PullRequest with the number of 11297. (repository.pullRequest)
                                                                                                                                                
⏺ Bash(gh pr view 11297 --repo pydata/xarray --json title,body,files,state,headRefName,statusCheckRollup)
  ⎿  {"body":"## Summary\n\n- Fixes `da.sortby(da.dims)` raising `KeyError` when `da.dims` is a tuple (e.g. `('x', 'y')`)\n- Tuples are now     
      treated as iterables of variable names, consistent with the `str | Iterable[Hashable]` convention established in #6142\n\nCloses #482     
     1\n\n### Usage example\n\n```python\n>>> import xarray as xr\n>>> da = xr.DataArray(\n...     [[1, 2], [3, 4], [5, 6]],\n...     coord     
     … +101 lines (ctrl+o to expand)                                                                                                            
                                                                                                                                                
⏺ Let me get the details on the failing mypy test.                                                                                              
                                                                             
⏺ Bash(gh run view 24522563933 --repo pydata/xarray --log-failed 2>&1 | head -200)                                                              
  ⎿  ubuntu-latest | test-py311-with-typing (mypy)      UNKNOWN STEP    2026-04-16T16:46:15.1164683Z Current runner version: '2.333.1'          
     ubuntu-latest | test-py311-with-typing (mypy)      UNKNOWN STEP    2026-04-16T16:46:15.1196068Z ##[group]Runner Image Provisioner
     ubuntu-latest | test-py311-with-typing (mypy)      UNKNOWN STEP    2026-04-16T16:46:15.1197310Z Hosted Compute Agent                       
     … +211 lines (ctrl+o to expand)                                         
                                                                                                                                                
⏺ Bash(gh run view 24522563933 --repo pydata/xarray --log-failed 2>&1 | grep -iE "(error|fail|mypy)" | head -50)                                
  ⎿  ubuntu-latest | test-py311-with-typing (mypy)      UNKNOWN STEP    2026-04-16T16:46:15.1164683Z Current runner version: '2.333.1'          
     ubuntu-latest | test-py311-with-typing (mypy)      UNKNOWN STEP    2026-04-16T16:46:15.1196068Z ##[group]Runner Image Provisioner          
     ubuntu-latest | test-py311-with-typing (mypy)      UNKNOWN STEP    2026-04-16T16:46:15.1197310Z Hosted Compute Agent                       
     … +47 lines (ctrl+o to expand)                                                                                                             
                                                                                                                                                
⏺ Bash(gh run view 24522563933 --repo pydata/xarray --log-failed --job 71684084342 2>&1 > /tmp/mypy_fail_log.txt; wc -l /tmp/mypy_fail_log.txt) 
  ⎿      4173 /tmp/mypy_fail_log.txt                                                                                                            
                                                                                                                                                
  Searched for 1 pattern, read 1 file (ctrl+o to expand)                                                                                        
                                                                             
⏺ Confirmed the CI failure: pytest_mypy_plugins/item.py:404 store.close() -> AttributeError: 'FilesystemMetadataStore' object has no attribute  
  'close' with pytest-mypy-plugins 4.0.1. Let me find how to run these tests locally.                                                           
                                                                             
  Searched for 4 patterns, listed 1 directory (ctrl+o to expand)                                                                                
                                                                                                                                                
⏺ Bash(uv pip show pytest-mypy-plugins 2>&1 | head -20)                                                                                         
  ⎿  pyenv: uv: command not found                                                                                                               
                                                                                                                                                
     The `uv' command exists in these Python versions:                                                                                          
     … +4 lines (ctrl+o to expand)                                                                                                              
                                                                                                                                                
⏺ Bash(which -a uv python3; pyenv versions 2>/dev/null | head)                                                                                  
  ⎿  /Users/thodson/.pyenv/shims/uv                                                                                                             
     /Users/thodson/.pyenv/shims/python3                                                                                                        
     /opt/homebrew/bin/python3                                                                                                                  
     … +6 lines (ctrl+o to expand)                                                                                                              
                                                                                                                                                
⏺ Bash(ls /Users/thodson/.pyenv/versions/3.13.3/bin/uv 2>/dev/null; /Users/thodson/.pyenv/versions/3.13.3/bin/uv --version)                     
  ⎿  /Users/thodson/.pyenv/versions/3.13.3/bin/uv                                                                                               
     uv 0.11.7 (9d177269e 2026-04-15 aarch64-apple-darwin)                                                                                      
                                                                                                                                                
⏺ Bash(/Users/thodson/.pyenv/versions/3.13.3/bin/uv pip show pytest-mypy-plugins 2>&1 | head -20)                                               
  ⎿  Name: pytest-mypy-plugins                                                                                                                  
     Version: 4.0.2                                                                                                                             
     Location: /Users/thodson/Desktop/dev/software/xarray/.venv/lib/python3.13/site-packages                                                    
     … +2 lines (ctrl+o to expand)                                                                                                              
                                                                                                                                                
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv sync 2>&1 | tail -30)                                                  
  ⎿  Resolved 151 packages in 3ms                                                                                                               
     Checked 145 packages in 274ms                                                                                                              
                                                                                                                                                
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv pip install --upgrade pytest-mypy-plugins 2>&1 | tail -20)             
  ⎿  Resolved 22 packages in 260ms                                                                                                              
     Downloading mypy (12.7MiB)                                                                                                                 
      Downloaded mypy                                                                                                                           
     … +5 lines (ctrl+o to expand)                                                                                                              
                                                                                                                                                
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv pip show pytest-mypy-plugins | head -5)                                
  ⎿  Name: pytest-mypy-plugins                                                                                                                  
     Version: 4.0.2                                                                                                                             
     Location: /Users/thodson/Desktop/dev/software/xarray/.venv/lib/python3.13/site-packages                                                    
     … +2 lines (ctrl+o to expand)                                                                                                              
                                                                                                                                                
  Read 1 file (ctrl+o to expand)                                                                                                                
                                                                             
⏺ The fix is already in 4.0.2 (getattr(store, "close", None) guard at line 406). Let me run the mypy tests locally now.                         
                                                                                                                                                
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv run pytest xarray/tests/test_dataarray_typing.yml
      xarray/tests/test_dataset_typing.yml xarray/…)                                                                                            
  ⎿  Uninstalled 1 package in 176ms                                                                                                             
     Installed 1 package in 37ms                                                                                                                
     ============================= test session starts ==============================                                                           
     … +30 lines (ctrl+o to expand)                                                                                                             
  ⎿  (timeout 10m)                                                           
                                                                                                                                                
⏺ The tests are all skipped. Let me find out why.                                                                                               
                                                                                                                                                
  Searched for 1 pattern (ctrl+o to expand)                                                                                                     
                                                                                                                                                
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv run pytest xarray/tests/test_dataarray_typing.yml
      xarray/tests/test_dataset_typing.yml xarray/…)                                                                                            
  ⎿  E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output:
     E   Actual:                                                                                                                                
     E     ../../../../../../Users/thodson/Desktop/dev/software/xarray/xarray/core/utils:117: error: Unused "type: ignore" comment  [unused
     … +36 lines (ctrl+o to expand)                                                                                                             
  ⎿  (timeout 10m)                                                           
                                                                                                                                                
⏺ The original FilesystemMetadataStore.close bug is gone — the fix is in 4.0.2. The new failures are unrelated (Unused "type: ignore") — they're
   caused by mypy 1.20.1 vs the CI-pinned 1.19.1. Let me downgrade mypy to match CI.                                                            
                                                                                                                                                
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv pip install "mypy==1.19.1" 2>&1 | tail -10)                            
  ⎿  Checked 1 package in 39ms                                               
                                                                                                                                                
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv run --no-sync pytest xarray/tests/test_dataarray_typing.yml            
      xarray/tests/test_dataset_typing.y…)                                   
  ⎿  FAILED xarray/tests/test_dataset_typing.yml::test_mypy_pipe_function_missing_kwarg                                                         
     FAILED xarray/tests/test_dataset_typing.yml::test_mypy_pipe_function_missing_keyword                                                       
     FAILED xarray/tests/test_dataset_typing.yml::test_mypy_pipe_tuple_return_type_dataset
     … +17 lines (ctrl+o to expand)                                                                                                             
  ⎿  (timeout 10m)                                                           
                                                                                                                                                
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv run --no-sync pytest                                                   
      xarray/tests/test_dataarray_typing.yml::test_mypy_pipe_lambda_noarg_retur…)                                                               
  ⎿  ============================= test session starts ==============================                                                           
     platform darwin -- Python 3.13.3, pytest-9.0.3, pluggy-1.6.0                                                                               
     rootdir: /Users/thodson/Desktop/dev/software/xarray                     
     … +24 lines (ctrl+o to expand)                                                                                                             
  ⎿  (timeout 2m)                                                            
                                                                                                                                                
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv pip show mypy | head -3)                                               
  ⎿  Name: mypy                                                                                                                                 
     Version: 1.19.1                                                                                                                            
     Location: /Users/thodson/Desktop/dev/software/xarray/.venv/lib/python3.13/site-packages                                                    
                                                                             
  Read 2 files (ctrl+o to expand)                                                                                                               
                                                                                                                                                
⏺ Let me verify the Unused "type: ignore" errors are a pre-existing local env issue (pandas-stubs mismatch) and not caused by the sortby PR.
                                                                                                                                                
⏺ Bash(git stash -u 2>&1 | head -3; git checkout main 2>&1 | tail -3)        
  ⎿  Saved working directory and index state WIP on fix-sortby-tuple-4821: 704014dc Fix type hints                                              
     Switched to branch 'main'                                                                                                                  
     Your branch is up to date with 'origin/main'.                                                                                              
                                                                                                                                                
⏺ Bash(export PATH="/Users/thodson/.pyenv/versions/3.13.3/bin:$PATH"; uv run --no-sync pytest                                                   
      xarray/tests/test_dataarray_typing.yml::test_mypy_pipe_lambda_noarg_retur…)                                                               
  ⎿  xarray/tests/test_dataarray_typing.yml F                                 [100%]
                                                                                                                                                
     =================================== FAILURES ===================================
     … +18 lines (ctrl+o to expand)                                                                                                             
  ⎿  (timeout 2m)                                                            
                                                                                                                                                
⏺ Same failure on main — so the Unused "type: ignore" noise is a pre-existing local env issue (pandas-stubs mismatch with CI's pixi env), not   
  caused by the sortby PR. Restoring the branch.                                                                                                
                                                                                                                                                
⏺ Bash(git checkout fix-sortby-tuple-4821 2>&1 | tail -3; git stash pop 2>&1 | tail -5)                                                         
  ⎿  Switched to branch 'fix-sortby-tuple-4821'                              
     Your branch is up to date with 'origin/fix-sortby-tuple-4821'.                                                                             
       (use "git add <file>..." to include in what will be committed)                                                                           
     … +4 lines (ctrl+o to expand)                                           
                                                                                                                                                
⏺ Here's what I found:                                                                                                                          
                                                                             
  Yes, your pytest-mypy-plugins bugfix resolves the CI failure.                                                                                 
                                                                                                                                                
  The CI error was AttributeError: 'FilesystemMetadataStore' object has no attribute 'close' at pytest_mypy_plugins/item.py:404 store.close().  
  CI was pinned to pytest-mypy-plugins==4.0.1.                                                                                                  
                                                                                                                                                
  Locally with pytest-mypy-plugins==4.0.2 (which uv sync picks up — pyproject.toml already pins >=4.0.0), that same file now wraps the call in a
   getattr(store, "close", None) guard at item.py:406-408. Running the 39 typing tests against your branch no longer crashes at teardown — so
  the FilesystemMetadataStore.close failure is gone.                                                                                            
                  
  However, the tests still fail locally with a different, unrelated issue:                                                                      
  xarray/core/utils:117: error: Unused "type: ignore" comment  [unused-ignore]
  xarray/core/variable:73: error: Unused "type: ignore" comment  [unused-ignore]                                                                
  I confirmed this happens on main too (with the fixed plugin), so it's noise from my local env (pandas-stubs divergence from CI's pixi         
  test-py311-with-typing env, which pins mypy==1.19.1 plus specific stubs) — not caused by your sortby PR.                                      
                                                                                                                                                
  Recommendation: once CI picks up pytest-mypy-plugins>=4.0.2 (either by re-resolving the pixi lockfile or bumping the floor in                 
  pixi.toml/pyproject.toml), the test-py311-with-typing (mypy) job should run the 39 pipe typing cases instead of crashing in teardown — so the 
  sortby PR should go green without any change to the PR itself.
                                                                                                                                                
✻ Baked for 9m 34s                  
                                                                             
※ recap: Verifying whether your pytest-mypy-plugins bugfix resolves the mypy CI failure on PR 11297; confirmed it does (4.0.2 removes the       
  FilesystemMetadataStore.close crash). Next: bump the pinned floor to pytest-mypy-plugins>=4.0.2 in pixi.toml so CI picks it up. (disable      
  recaps in /config)                                                                                                                            
                                                                                                                                                
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯  
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ? for shortcuts                                                                       control this session from your phone · /remote-control
                                                                                                                                                

  ⎿  Switched to branch 'fix-sortby-tuple-4821'
     Your branch is up to date with 'origin/fix-sortby-tuple-4821'.
       (use "git add <file>..." to include in what will be committed)
     … +4 lines (ctrl+o to expand)

⏺ Here's what I found:

  Yes, your pytest-mypy-plugins bugfix resolves the CI failure.

  The CI error was AttributeError: 'FilesystemMetadataStore' object has no attribute 'close' at pytest_mypy_plugins/item.py:404
  store.close(). CI was pinned to pytest-mypy-plugins==4.0.1.

  Locally with pytest-mypy-plugins==4.0.2 (which uv sync picks up — pyproject.toml already pins >=4.0.0), that same file now wraps the
  call in a getattr(store, "close", None) guard at item.py:406-408. Running the 39 typing tests against your branch no longer crashes at
   teardown — so the FilesystemMetadataStore.close failure is gone.

  However, the tests still fail locally with a different, unrelated issue:
  xarray/core/utils:117: error: Unused "type: ignore" comment  [unused-ignore]
  xarray/core/variable:73: error: Unused "type: ignore" comment  [unused-ignore]
  I confirmed this happens on main too (with the fixed plugin), so it's noise from my local env (pandas-stubs divergence from CI's pixi
  test-py311-with-typing env, which pins mypy==1.19.1 plus specific stubs) — not caused by your sortby PR.

  Recommendation: once CI picks up pytest-mypy-plugins>=4.0.2 (either by re-resolving the pixi lockfile or bumping the floor in
  pixi.toml/pyproject.toml), the test-py311-with-typing (mypy) job should run the 39 pipe typing cases instead of crashing in teardown —
   so the sortby PR should go green without any change to the PR itself.

✻ Baked for 9m 34s

※ recap: Verifying whether your pytest-mypy-plugins bugfix resolves the mypy CI failure on PR 11297; confirmed it does (4.0.2 removes
  the FilesystemMetadataStore.close crash). Next: bump the pinned floor to pytest-mypy-plugins>=4.0.2 in pixi.toml so CI picks it up.
  (disable recaps in /config)

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯ 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ? for shortcuts                                                               control this session from your phone · /remote-control

       (use "git add <file>..." to include in what will be committed)
     … +4 lines (ctrl+o to expand)

⏺ Here's what I found:

  Yes, your pytest-mypy-plugins bugfix resolves the CI failure.

  The CI error was AttributeError: 'FilesystemMetadataStore' object has no attribute 'close' at pytest_mypy_plugins/item.py:404 store.close().
  CI was pinned to pytest-mypy-plugins==4.0.1.

  Locally with pytest-mypy-plugins==4.0.2 (which uv sync picks up — pyproject.toml already pins >=4.0.0), that same file now wraps the call in a
   getattr(store, "close", None) guard at item.py:406-408. Running the 39 typing tests against your branch no longer crashes at teardown — so
  the FilesystemMetadataStore.close failure is gone.

  However, the tests still fail locally with a different, unrelated issue:
  xarray/core/utils:117: error: Unused "type: ignore" comment  [unused-ignore]
  xarray/core/variable:73: error: Unused "type: ignore" comment  [unused-ignore]
  I confirmed this happens on main too (with the fixed plugin), so it's noise from my local env (pandas-stubs divergence from CI's pixi
  test-py311-with-typing env, which pins mypy==1.19.1 plus specific stubs) — not caused by your sortby PR.

  Recommendation: once CI picks up pytest-mypy-plugins>=4.0.2 (either by re-resolving the pixi lockfile or bumping the floor in
  pixi.toml/pyproject.toml), the test-py311-with-typing (mypy) job should run the 39 pipe typing cases instead of crashing in teardown — so the
  sortby PR should go green without any change to the PR itself.

✻ Baked for 9m 34s

※ recap: Verifying whether your pytest-mypy-plugins bugfix resolves the mypy CI failure on PR 11297; confirmed it does (4.0.2 removes the
  FilesystemMetadataStore.close crash). Next: bump the pinned floor to pytest-mypy-plugins>=4.0.2 in pixi.toml so CI picks it up. (disable
  recaps in /config)

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯ 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ? for shortcuts                                                                       control this session from your phone · /remote-control
                                                                                                                                                

  ⎿  (timeout 2m)

⏺ Same failure on main — so the Unused "type: ignore" noise is a pre-existing local env issue (pandas-stubs mismatch with CI's pixi
  env), not caused by the sortby PR. Restoring the branch.

⏺ Bash(git checkout fix-sortby-tuple-4821 2>&1 | tail -3; git stash pop 2>&1 | tail -5)
  ⎿  Switched to branch 'fix-sortby-tuple-4821'
     Your branch is up to date with 'origin/fix-sortby-tuple-4821'.
       (use "git add <file>..." to include in what will be committed)
     … +4 lines (ctrl+o to expand)

⏺ Here's what I found:

  Yes, your pytest-mypy-plugins bugfix resolves the CI failure.

  The CI error was AttributeError: 'FilesystemMetadataStore' object has no attribute 'close' at pytest_mypy_plugins/item.py:404
  store.close(). CI was pinned to pytest-mypy-plugins==4.0.1.

  Locally with pytest-mypy-plugins==4.0.2 (which uv sync picks up — pyproject.toml already pins >=4.0.0), that same file now wraps the
  call in a getattr(store, "close", None) guard at item.py:406-408. Running the 39 typing tests against your branch no longer crashes at
   teardown — so the FilesystemMetadataStore.close failure is gone.

  However, the tests still fail locally with a different, unrelated issue:
  xarray/core/utils:117: error: Unused "type: ignore" comment  [unused-ignore]
  xarray/core/variable:73: error: Unused "type: ignore" comment  [unused-ignore]
  I confirmed this happens on main too (with the fixed plugin), so it's noise from my local env (pandas-stubs divergence from CI's pixi
  test-py311-with-typing env, which pins mypy==1.19.1 plus specific stubs) — not caused by your sortby PR.

  Recommendation: once CI picks up pytest-mypy-plugins>=4.0.2 (either by re-resolving the pixi lockfile or bumping the floor in
  pixi.toml/pyproject.toml), the test-py311-with-typing (mypy) job should run the 39 pipe typing cases instead of crashing in teardown —
   so the sortby PR should go green without any change to the PR itself.

✻ Baked for 9m 34s

※ recap: Verifying whether your pytest-mypy-plugins bugfix resolves the mypy CI failure on PR 11297; confirmed it does (4.0.2 removes
  the FilesystemMetadataStore.close crash). Next: bump the pinned floor to pytest-mypy-plugins>=4.0.2 in pixi.toml so CI picks it up.
  (disable recaps in /config)

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯ 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ? for shortcuts                                                               control this session from your phone · /remote-control

       (use "git add <file>..." to include in what will be committed)
     … +4 lines (ctrl+o to expand)

⏺ Here's what I found:

  Yes, your pytest-mypy-plugins bugfix resolves the CI failure.

  The CI error was AttributeError: 'FilesystemMetadataStore' object has no attribute 'close' at pytest_mypy_plugins/item.py:404 store.close().
  CI was pinned to pytest-mypy-plugins==4.0.1.

  Locally with pytest-mypy-plugins==4.0.2 (which uv sync picks up — pyproject.toml already pins >=4.0.0), that same file now wraps the call in a
   getattr(store, "close", None) guard at item.py:406-408. Running the 39 typing tests against your branch no longer crashes at teardown — so
  the FilesystemMetadataStore.close failure is gone.

  However, the tests still fail locally with a different, unrelated issue:
  xarray/core/utils:117: error: Unused "type: ignore" comment  [unused-ignore]
  xarray/core/variable:73: error: Unused "type: ignore" comment  [unused-ignore]
  I confirmed this happens on main too (with the fixed plugin), so it's noise from my local env (pandas-stubs divergence from CI's pixi
  test-py311-with-typing env, which pins mypy==1.19.1 plus specific stubs) — not caused by your sortby PR.

  Recommendation: once CI picks up pytest-mypy-plugins>=4.0.2 (either by re-resolving the pixi lockfile or bumping the floor in
  pixi.toml/pyproject.toml), the test-py311-with-typing (mypy) job should run the 39 pipe typing cases instead of crashing in teardown — so the
  sortby PR should go green without any change to the PR itself.

✻ Baked for 9m 34s

※ recap: Verifying whether your pytest-mypy-plugins bugfix resolves the mypy CI failure on PR 11297; confirmed it does (4.0.2 removes the
  FilesystemMetadataStore.close crash). Next: bump the pinned floor to pytest-mypy-plugins>=4.0.2 in pixi.toml so CI picks it up. (disable
  recaps in /config)

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯ /exit 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
/exit                         Exit the CLI                                                                                                    
/extra-usage                  Configure extra usage to keep working when limits are hit                                                         
/context                      Visualize current context usage as a colored grid          
/claude-api                   Build, debug, and optimize Claude API / Anthropic SDK apps. Apps built with this skill should include prompt …
/memory                       Edit Claude memory files                                                                                      
/ultraplan                    a few minutes · Claude Code on the web drafts a plan you can edit and approve. See https://code.claude.com/do…

Resume this session with:
claude --resume d7e3e421-faa6-4d82-bc55-0dc4683f21d7
thodson@IGSASC167LTTOH2 xarray % cd ..
thodson@IGSASC167LTTOH2 software % ls
claude-code		dataretrieval-python	fluegg			gumbridge		ratingcurve		spherely
cubed			DataRetrieval.jl	fluegg_prev		hytest			schwarz-bridge		xarray
cubed-clone		discontinuum		frmc_mobile		lithops			schwarz-bridge-gpt	xarray-regrid
cubed-xarray		discontinuum-delete	frmc_python		model_evaluation_kit	schwarz-demo		xdscore
dataRetrieval		earth2mip		gplm_python		numcodecs		scratch			zarr-python
thodson@IGSASC167LTTOH2 software % git clone --depth 10 git@github.com:thodson-usgs/sparse.git
Cloning into 'sparse'...
remote: Enumerating objects: 206, done.
remote: Counting objects: 100% (206/206), done.
remote: Compressing objects: 100% (186/186), done.
remote: Total 206 (delta 40), reused 92 (delta 11), pack-reused 0 (from 0)
Receiving objects: 100% (206/206), 748.15 KiB | 5.12 MiB/s, done.
Resolving deltas: 100% (40/40), done.
thodson@IGSASC167LTTOH2 software % cd sparse 
thodson@IGSASC167LTTOH2 sparse % ls
benchmarks		docs			mkdocs.yml		pyproject.toml		release-procedure.md	sparse
benchmarks_original	examples		pixi.lock		pytest.ini		scripts			tox.ini
conftest.py		LICENSE			pixi.toml		README.md		setup.cfg		xp-tests
thodson@IGSASC167LTTOH2 sparse % claude
╭─── Claude Code v2.1.116 ───────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                    │ Tips for getting started                                          │
│                Welcome back Timothy!               │ Run /init to create a CLAUDE.md file with instructions for Claude │
│                                                    │ ───────────────────────────────────────────────────────────────── │
│                      ▗ ▗   ▖ ▖                     │ Recent activity                                                   │
│                                                    │ No recent activity                                                │
│                        ▘▘ ▝▝                       │                                                                   │
│   Opus 4.7 (1M context) with xh… · Claude Max ·    │                                                                   │
│   tohodson@gmail.com's Organization                │                                                                   │
│           ~/Desktop/dev/software/sparse            │                                                                   │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

❯ /init and review the contribution guidelines and dev environment config and set up a local dev environment for this repo                      
                                                                                                                           
  Read 6 files, listed 4 directories (ctrl+o to expand)                                                                                         
                                                                                                                                                
  Read 7 files, listed 1 directory (ctrl+o to expand)                                                                                           
                                                                                                                                                
⏺ Bash(which pixi 2>/dev/null; pixi --version 2>/dev/null || echo "pixi not installed")                                                         
  ⎿  Running…                                                                                                                                   
                                                                                                                                                
  4 tasks (0 done, 1 in progress, 3 open)                                                                                                       
  ◼ Explore repo architecture and dev docs                                                                                                      
  ◻ Review contribution guidelines                                                                                                              
  ◻ Set up local dev environment                                                                                                                
  ◻ Write CLAUDE.md                                                                                                                             
                                                                                                                                                
──────────────────────────────────────────────────────────────────────────────────────…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Major performance regression in Coordinates.to_index()

3 participants