[C++/Python] Add option to Take kernel to interpret negative indices as indexing from the right

[asfimport]

Currently negative integers are explicitly forbidding in the Take kernel. It would be nice to have the option to treat negative integers as "indices from the right" instead.

Reporter: Uwe Korn / @xhochy

Related issues:

• [C++/Python] Support necessary functionality to have an Arrow-string type in pandas (is depended upon by)

_{Note: This issue was originally created as ARROW-9432. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
In R, negative indices means to take everything except those indices. So maybe these behaviors should be handled in Python/R, translating the requested indices to the "normal" form that the C++ library expects?

[asfimport]

Wes McKinney / @wesm:
We should not alter the existing Take implementation hot path. I agree with Neal that it would be better (when "wraparound" is required) to do a vectorized check for negative indices, and if any are encountered, to transform to all non-negative indices and then invoke the usual Take code path

[asfimport]

Uwe Korn / @xhochy:

In R, negative indices means to take everything except those indices. So maybe these behaviors should be handled in Python/R, translating the requested indices to the "normal" form that the C++ library expects?

Uh, how would that work? I thought we have a 1:1 mapping of index in the array to a resulting row.

We should not alter the existing Take implementation hot path.

I would rather have a second kernel implementation that is called if one requests the treatment of negative integers in this way. As one can see, I have opened two of these "behave-different-on-negative-indices" issues and R seems to have a third implementation. This shows that these kernels are useful but that there is no consensus on what they should do.

[asfimport]

Antoine Pitrou / @pitrou:
While dedicated kernel variants would be faster, they would also increase the code size and build times substantially. This would have to be checked when drafting the implementation.

[asfimport]

Wes McKinney / @wesm:
It might be acceptable to have a slower, but smaller (in code size) implementation. For example, simply transposing the negative indices to be positive and then invoking the non-negative-Take implementation

[asfimport]

Eduardo Ponce / @edponce:
I think the core issue here is to support in a consistent manner, non-negative indices across compute kernels.

Given that there are multiple kernels that "behave-different-on-negative-indices" and there are different semantics of what a negative index entails, this can be handled in Arrow C++ by having a utility function with an option to dictate on how to interpret negative indices and always resolve to a set of nonnegative indices. For kernels that have index parameters and where negative indices are meaningful, we could have an option to specify the desired behavior and set the default value based on the language's semantics. These kernels would then resolve the negative indices before processing any data.

The drawback from this approach is having an additional (non-familiar) option for these kernels, so maybe resolving the negative indices in the particular language bindings might be a better choice.

[thisisnic]

This issue hasn't had activity in a long time. If it's still being worked on, please leave a comment. Otherwise, it will be closed on 23rd June.

Labelled Status: Stale-Warning for tracking.