concat automagically outer-joins coordinates

[j08lue]

I would like to concatenate two netCDF files that have float64 coordinate variables. I thought the coordinate values were the same, but in fact they differ by something in the order of 1e-14.

Using open_mfdataset or concat to merge the datasets, I get a completely different output shape than the two input files have.

This is because concat is somewhere performing an outer join on the coordinates. Now I am wondering where else in my workflows this might happen without my notice...

It would be awesome if there was an option to change this behaviour on concat, open_mfdataset, and auto_combine. I would actually rather make these functions fail if any dimension other than the concatenation dimension differs.

Note: This could also be a special case, because

>>> (ds1.lon == ds2.lon).all()
<xarray.DataArray 'lon' ()>
array(True, dtype=bool)

while

>>> (ds1.lon.values == ds2.lon.values).all()
False

Maybe an interface change could be considered together with that discussed in #1340?

[shoyer]

This only happens in concat.

merge, which happens automatically in many operations, e.g., to decide which coordinates propagate, has different behavior: it automatically drops conflicting coordinate values.

Anyways, I agree: we should try to make this more explicit. We might still keep the "magical" version of concat around, but we should certainly expose the fully explicit version as public API.

[shoyer]

#1330 will help, in so far as it adds a join='exact' option that can disable automatic alignment. We would just need to make the join argument used internally in align inside xarray.concat part of the public API for concat.

[wqshen]

I also have this problem.

I have a list of dataset with all same shaped array at different time, while each have a different coordinate. (Each dataset is centered in the tropical cyclone center, with a same radius width buffer.)

I use xr.concat to concat them along time dimension, this function auto outer-joins coordinates, which makes the final dataset have a larger-shape, but i just want the original dataset shape.

Any solution ?

[darothen]

@wqshen, a workaround until a more complete modification to align is available would be to explicitly copy/set the coordinate values on your arrays before using xr.concat(). Alternatively, if it's as simple as stacking along a new tailing axis, you could stack via dask/numpy and then construct a new DataArray passing the coordinates explicitly.

[shoyer]

I think the right solution here is to relabel the coordinates on each of the datasets that you are concatenating. The alternative behavior here is xarray raising an error in the case of conflicting coordinate labels.

[stale]

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

[j08lue]

Thanks for reminding me, stalte bot, this is still relevant to me.

I think your proposed solution @shoyer would solve this issue perfectly: expose align's join parameter in the concat function - and please also in open_mfdataset. Would you like a PR for this?

[shoyer]

Sure, we would welcome a PRs for this.

…

On Wed, Jun 5, 2019 at 11:33 PM Jonas ***@***.***> wrote: Thanks for reminding me, stalte bot, this is still relevant to me. I think your proposed solution @shoyer <https://github.com/shoyer> would solve this issue perfectly: expose align's join parameter in the concat function - and please also in open_mfdataset. Would you like a PR for this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1354?email_source=notifications&email_token=AAJJFVWVBBRU3HO3PZDGCO3PZCVUTA5CNFSM4DGS3QR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXB36MQ#issuecomment-499367730>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJJFVQ4YEERVTH2WFN77OLPZCVUTANCNFSM4DGS3QRQ> .