Proposal: Improve division of functionality between build_lookup_from_csv, read_csv_to_dataframe, pandas.read_csv

[emlys]

read_csv_to_dataframe is a wrapper around pandas.read_csv that adds this functionality:

• Strip whitespace from column names
• Convert column names to lowercase
• Strip whitespace from table values
• Expand relative paths

build_lookup_from_csv reformats the table as a dictionary indexed by a particular column, and also adds this functionality:

• Convert table values to lowercase
• Only use certain columns
• Drop empty rows
• Replace NA values with an empty string

It would make more sense if the additional functionality of build_lookup_from_csv was moved to read_csv_to_dataframe. Particularly the lowercase conversion options are confusing: the to_lower arg of build_lookup_from_csv affects column names AND values. the to_lower arg of read_csv_to_dataframe affects column names only.

Pseudocode outlining the proposed division of functionality:

def build_lookup_from_csv(path, index_col, **kwargs):
    df = read_csv_to_dataframe(path, **kwargs)
    return df reformatted as a dictionary indexed by index_col

def read_csv_to_dataframe(path, cols_to_lower=True, vals_to_lower=True, 
        expand_path_cols=[], column_list=None, sep=None, engine='python', encoding='utf-8-sig', **kwargs):
    df = pandas.read_csv(path, **kwargs)
    use only the columns in column_list
    drop empty rows
    strip whitespace from column names and values
    expand relative paths in expand_path_cols
    replace NA values with an empty string
    if cols_to_lower:
        convert column names to lowercase
    if vals_to_lower:
        convert values to lowercase
    return df

The data processing that can be done with arguments to pandas.read_csv (such as selecting columns, dropping empty rows, and applying converters to column values) should be moved there to avoid duplication.

[emlys]

From 6/15 coffee call: Once this is implemented, we might not need build_lookup_from_csv at all. If it's reduced to basically just a call to df.to_dict, we can replace build_lookup_from_csv with read_csv_to_dataframe().to_dict() everywhere.