Performance regression when reading scoring files during matching

[nebfield]

pygscatalog/pgscatalog.match/src/pgscatalog/match/lib/_arrow.py

Lines 6 to 21 in e88b41f

	def loose(record_batches, schema, tmpdir=None):
	"""Loose an arrow :) Stream compressed text files into temporary arrow files
	polars + arrow = very fast reading and processing
	"""
	if tmpdir is None:
	tmpdir = tempfile.mkdtemp()
	arrowpath = tempfile.NamedTemporaryFile(dir=tmpdir, delete=False)
	with pa.OSFile(arrowpath.name, "wb") as sink:
	with pa.RecordBatchFileWriter(sink=sink, schema=schema) as writer:
	for batch in record_batches:
	writer.write(batch)
	return arrowpath

We used to parse CSV files with polars and save IPC files, it's super fast 🚀

Streaming pyarrow batches is terribly slow in comparison (when working on UK Biobank). I think i was worried about memory usage when I wrote this.

This might also drop the pyarrow dependency in pgscatalog.core