pandas3 DataFrames with text Series (pyarrow)

[PhilipYip1988]

Currently a Series in a DataFrame that has text has the dtype object:

import numpy as np
import pandas as pd
# pd.options.future.infer_string = True

df = pd.DataFrame({'text1': ['a', 'b', 'c', 'd', 'e'],
                   'text2': ['aa', 'bb', 'cc', 'dd', 'ee']})

And Spyder opens them as expected:

Using an dtype of object is problematic as the Series can contain str and numeric values and a TypeError occurs essentially with operations like '2'+ 2. In the future (pandas 3) Series in DataFrames with text will default to the dtype string [pyarrow_numpy]:

import numpy as np
import pandas as pd
pd.options.future.infer_string = True

df = pd.DataFrame({'text1': ['a', 'b', 'c', 'd', 'e'],
                   'text2': ['aa', 'bb', 'cc', 'dd', 'ee']})

Unfortunately when this option is enabled in Spyder 6.0.4 (standalone) with a Miniforge conda environment:

conda create -n spyder-env spyder-kernels python seaborn scikit-learn pyarrow sympy openpyxl xlrd xlsxwriter lxml sqlalchemy tabulate pyqt ffmpeg ruff

And the DataFrame is opened, there is an error:

Spyder was unable to retrieve the value of this variable from the console.

The error message was:

The 'pyarrow' package is required to open this variable. Unfortunately its not part of our installer, which means your variable can't be displayed by Spyder.

Note: Please don't report this problem on GitHub, there's nothing to do about it.

Although it says not to report, given this is going to be the future default behaviour of every Series with text. It is probably worthwhile adding pyarrow to the standalone installer and displaying the DataFrame (or Series or Index) in the same way as a DataFrame (or Series or Index) with dtype of object displays. Otherwise this error message will frequently occur when pandas version 3 is released.