I'm confused about how to preset min_itemsizes for appending to an HDFStore. Say DataFrame a and b in the MWE below is user-provided, so it can contain any character and the encoding is unknown. Appending a works, but appending b fails even though:

In [4]: len('香')
Out[4]: 1

So far I simply used str.len().max() on the string columns to the the numbers for min_itemsize, but this does not work in the example here. This MWE is of course simplified, but I guess I'm wondering:

• how does pytables come up with the string length?
• how should I determine the string length? Considering the encoding is unknown, but pytables assumes some encoding / pytables converts the strings to some other object?

In this toy example I could encode the string as utf-8 to get the correct length, but this isn't a general approach:

In [5]: len('香'.encode('utf-8'))
Out[5]: 3

MWE:

import pandas as pd
                                                      
a = pd.DataFrame([['a', 'b']], columns = ['A', 'B'])
b = pd.DataFrame([['香', 'b']], columns = ['A', 'B'])

store = pd.HDFStore('/tmp/tmpstore')

store.append('df', a, min_itemsizes={'A': 1, 'B': 1})
store.append('df', b, min_itemsizes={'A': 1, 'B': 1}) # fails

Expected Output

ValueError: Trying to store a string with len [3] in [values_block_0] column but
this column has a limit of [1]!
Consider using min_itemsize to preset the sizes on these columns
Closing remaining open files:/tmp/tmpstore...done

Output of pd.show_versions()