SQLAlchemy ORM for 12 duplicate tables on each of 6 shard databases

427 views Asked by At

The situation:

I have a set of 12 tables of the same structure (representing data by month) that exist on each of 6 shards across different databases. I need to get a sample set of data across any of these databases for any given set of months.

The database is determined by the record id, the table is determined by the month of a date value in the record.

The question:

What is the best way to set up a SQLAlchemy ORM for this?

Initially I only needed to query for data one month at a time so I set up a ShardedSession mapping my Data class to one of the twelve data tables, ignoring the existence of the ones I didn't need. Now I need to select random sets of data across 3 different months at any given time. Is there a way I can map the same class onto 3 different tables of the same structure so they are all treated as one? So that I can get a random selection of records from these 3 months.

I've previously asked a similar question and posted the solution I came up with here: SQLAlchemy Classical Mapping Model to sharded Postgres databases I am using the Classical Mapping Model, I'm not sure if that is best but it was the only way I could figure out how to make things work.

I've read a little bit about Concrete Table Inheritance and Entity Name but last time I tried I couldn't get the Entity Name solution to work.

Any pointers would be greatly appreciated! Thanks!

0

There are 0 answers