How to use Pandas HDF5 as a Database in Python

Question

Pandas is not a database, right?

Is there a way to integrate the analysis power of pandas into a flat HDF5 file database? I know that unfortunately HDF5 is not designed to deal natively with concurrency.

Also been looking for inspiration into parallel HDF5, flat file database managers or multiprocessing but I still lack an idea. Help?

Answer 1

HDF5 works fine for concurrent read only access.

For concurrent write access you either have to use parallel HDF5 or have a worker process that takes care of writing to an HDF5 store.

I recommend to use a hybrid approach and expose it via a RESTful API. You can store meta-information in a SQL/NoSQL database and keep the raw data (time series data) in one or multiple HDF5 files.

There is one public REST API to access the data and the user doesn't have to care what happens behind the curtains.