Well, there are spreadsheet backends using Apache Arrow for storage. You could use something like pola.rs or Arrow DataFusion + Arrow Ballista for distributed processing of the dataset.