Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: try load page indexes #1425

Merged
merged 3 commits into from
Jan 4, 2024
Merged

feat: try load page indexes #1425

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7240bc3
feat: try load page indexes
jiacai2050 Jan 4, 2024
e9cae33
fix link
jiacai2050 Jan 4, 2024
cf5c032
fix CR
jiacai2050 Jan 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions analytic_engine/src/sst/parquet/async_reader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ use datafusion::{
};
use futures::{Stream, StreamExt};
use generic_error::{BoxError, GenericResult};
use logger::{debug, error};
use logger::{debug, error, warn};
use object_store::{ObjectStoreRef, Path};
use parquet::{
arrow::{arrow_reader::RowSelection, ParquetRecordBatchStreamBuilder, ProjectionMask},
Expand Down Expand Up @@ -397,7 +397,27 @@ impl<'a> Reader<'a> {
file_path: self.path.to_string(),
})?;

// TODO: Support page index until https://github.com/CeresDB/ceresdb/issues/1040 is fixed.
let mut parquet_meta_data = Arc::new(parquet_meta_data);
let object_store_reader = parquet_ext::reader::ObjectStoreReader::new(
self.store.clone(),
self.path.clone(),
parquet_meta_data.clone(),
);

if let Ok(meta_data) = parquet_ext::meta_data::meta_with_page_indexes(object_store_reader)
.await
.map_err(|e| {
// When loading page indexes failed, we just log the error and continue querying
// TODO: Fix this in stream. https://github.com/apache/incubator-horaedb/issues/1040
warn!(
"Fail to load page indexes, path:{}, err:{:?}.",
self.path, e
);
e
})
{
parquet_meta_data = meta_data;
}

MetaData::try_new(&parquet_meta_data, ignore_sst_filter, self.store.clone())
.await
Expand Down