Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Remove direct pandas import #6476

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open

Conversation

hoxbro
Copy link
Member

@hoxbro hoxbro commented Dec 13, 2024

This PR introduces the utilities _lazy_module and gen_types. Both are meant to depend on 3rd party libraries without using direct imports.

These have been used throughout the code, so pandas is no longer imported directly; this has significantly improved import time and, at the same time, set up the codebase to handle a world where pandas is not the only DataFrame library. Some functionality can likely be rewritten to not depend on pandas, but I will do that in future PRs.

The _lazy_module has also "sneaked" into the Ibis data interface, which, in my opinion, makes it cleaner by not having all methods start with import ibis.

Lastly, I have also removed the use of lazy_isinstance and deprecated it as the functionality can be done with _lazy_module and a bool check. This is done in holoviews/operation/datashader.py.

Some statics:

Hyperfine

Main (0ba7398)

❯ hyperfine "python -c ''" "python -c 'import holoviews'" --warmup 5
Benchmark 1: python -c ''
  Time (mean ± σ):      16.0 ms ±   0.6 ms    [User: 12.0 ms, System: 3.9 ms]
  Range (min … max):    14.9 ms …  17.6 ms    171 runs

Benchmark 2: python -c 'import holoviews'
  Time (mean ± σ):     367.3 ms ±  11.7 ms    [User: 1938.8 ms, System: 57.8 ms]
  Range (min … max):   356.4 ms … 391.3 ms    10 runs

Summary
  python -c '' ran
   22.90 ± 1.10 times faster than python -c 'import holoviews'

This PR (18f4db4)

❯ hyperfine "python -c ''" "python -c 'import holoviews'" --warmup 5
Benchmark 1: python -c ''
  Time (mean ± σ):      16.2 ms ±   0.6 ms    [User: 12.0 ms, System: 4.0 ms]
  Range (min … max):    14.9 ms …  19.1 ms    179 runs

Benchmark 2: python -c 'import holoviews'
  Time (mean ± σ):     189.7 ms ±   9.2 ms    [User: 1781.3 ms, System: 29.2 ms]
  Range (min … max):   182.5 ms … 216.8 ms    15 runs

Summary
  python -c '' ran
   11.74 ± 0.71 times faster than python -c 'import holoviews'

Tuna

Running python -X importtime -c 'import holoviews' 2> tuna.log && tuna tuna.log

Main (0ba7398)

image

This PR (18f4db4)
image

Output of hv.show_versions
Python              :  3.13.2 | packaged by conda-forge | (main, Feb 17 2025, 14:10:22) [GCC 13.3.0]
Operating system    :  Linux-6.13.5-arch1-1-x86_64-with-glibc2.41
Panel comms         :  default

holoviews           :  1.20.1.post1.dev7+g95d3af8b2

bokeh               :  3.6.3
colorcet            :  3.1.0
cudf                :  -
cupy                :  13.4.0
dask                :  2025.2.0
dask-expr           :  -
datashader          :  0.17.0.post1.dev5+gc3e4b58
geoviews            :  1.14.0.post1.dev2+gd89e2f7
hvplot              :  0.11.3.dev9+g9700bec.d20250301
ibis-framework      :  10.1.0
IPython             :  8.32.0
ipywidgets-bokeh    :  1.6.0
jupyter-bokeh       :  4.0.5
jupyterlab          :  4.3.5
matplotlib          :  3.10.1
networkx            :  3.4.2
notebook            :  -
numba               :  0.61.0
numpy               :  2.1.3
pandas              :  2.2.3
panel               :  1.6.1.post1.dev4+gf5170c481.d20250301
param               :  2.2.0.post1.dev22+gee19961
pillow              :  11.1.0
plotly              :  6.0.0
pyarrow             :  19.0.1
pyviz-comms         :  3.0.4
scikit-image        :  0.25.2
scipy               :  1.15.2
spatialpandas       :  0.5.0
streamz             :  -
tsdownsample        :  0.1.4.1
xarray              :  2025.1.2

@hoxbro hoxbro force-pushed the remove_direct_pandas branch from 7fc6d1d to dc063df Compare December 13, 2024 16:26
@hoxbro hoxbro force-pushed the remove_direct_pandas branch from 0fa4a05 to af07d2f Compare February 21, 2025 13:09
@hoxbro hoxbro force-pushed the remove_direct_pandas branch from 0e81703 to dee9c00 Compare February 21, 2025 13:37
Copy link

codecov bot commented Feb 21, 2025

Codecov Report

Attention: Patch coverage is 86.55462% with 32 lines in your changes missing coverage. Please review.

Project coverage is 88.77%. Comparing base (2465ad1) to head (fe84295).

Files with missing lines Patch % Lines
holoviews/core/util/dependencies.py 79.66% 12 Missing ⚠️
holoviews/core/util/types.py 92.64% 5 Missing ⚠️
holoviews/core/util/__init__.py 87.50% 4 Missing ⚠️
holoviews/core/data/cudf.py 0.00% 3 Missing ⚠️
holoviews/core/data/pandas.py 86.66% 2 Missing ⚠️
holoviews/element/util.py 66.66% 2 Missing ⚠️
holoviews/core/data/ibis.py 85.71% 1 Missing ⚠️
holoviews/element/graphs.py 50.00% 1 Missing ⚠️
holoviews/operation/datashader.py 90.00% 1 Missing ⚠️
holoviews/streams.py 91.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6476      +/-   ##
==========================================
- Coverage   88.79%   88.77%   -0.03%     
==========================================
  Files         323      325       +2     
  Lines       68949    69090     +141     
==========================================
+ Hits        61225    61333     +108     
- Misses       7724     7757      +33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@hoxbro hoxbro marked this pull request as ready for review February 21, 2025 15:45
@hoxbro hoxbro force-pushed the remove_direct_pandas branch from b02a1a2 to 98d307c Compare February 24, 2025 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant