from_frame(df_with_mixed_str_number_multi_index) broken

[gdementen]

The numbers are converted to strings. I noticed initially that that arrays with mixed-type labels do not roundtrip correctly, but this is the root issue. Also adding an explicit test for the roundtrip to HDF would be a good idea though. The bug is probably also present for simple (non-multi) index dataframes and maybe in the sorted case too (I need to test the status of the 4 combinations of simple/multi and sort/no sort).

I discovered this when preparing the 0.35.1 release because I fixed it alongside another issue but I reintroduced the bug on purpose because the demo model relies on the broken behavior and I do not want to knowingly break models in a bugfix release. See inout/pandas.py:simple_index_to_labels()

>>> df = pd.DataFrame([['s0', 'o0', 0], ['s1', 1, 1]], columns=['str', 'obj', 'value']).set_index(['str', 'obj'])
>>> df.index.unique(1)
Index(['o0', 1], dtype='object', name='obj')
>>> from_frame(df).obj.dtype
dtype('<U21')

[gdementen]

Ouch. This is even uglier than I thought. This issue leads to silent data loss (the cells corresponding to the int labels which are converted to string) in some cases because the reindex does not find the new (bad) string labels in the old (mixed type) index.

>>> df = pd.DataFrame([['s0', 'o0', 0], ['s0', 1, 1]], columns=['str', 'obj', 'value']).set_index(['str', 'obj'])
>>> from_frame(df)
str  obj\{2}  value
 s0       o0    0.0
 s0        1    nan

There was no data loss in the users code because that code path (H5 save+load) use cartesian_product=False, which produces the bad labels but does not use reindex.

I cannot leave a silent data loss issue open like that but fixing it breaks users code 😭. Outputting a warning without fixing the result would have been a nice tradeoff but I hate warnings a user cannot hide via better code. I guess I need to break users code after all and explain them how they need to change their code to get the old behavior.

[gdementen]

Well, I changed my mind again. We are lacking many tests concerning mixed-type labels. I just fixed #1194 but there are probably other issues and I cannot invest that time now. I think the best course of action is to avoid fixing this issue until we do an overall "mixed-type (labels) push" (#910, #989, #882, #911, ...). Let's see if I can fix/warn on the dataloss issue in from_frame without fixing this issue.

[gdementen]

In the end I decided the lesser-evil was to fix this issue anyway but implement an exception for H5 code where I know it breaks a least one model.