Am trying to make enterprise solution architecture using Datamesh architecture and stuck in a place like how common data across different domains gets ingested and where does it stores ? Any realworld implementation example will be helpful.
Thanks!
Architectural solution for my question
Let me give you an example:
A distributed query engine can access data from different, distributed data sources. Each domain can have an "endpoint", that provides the data for the outside world. This could be the server for mysql, other would have AWS s3 in case of parquet files, and another one csv.
In a query engine like Trino (Open Source, we are using Trino in some of the projects), you can define all those data sources and put them together in one query.
The data is stored in the domains (we are avoiding moving the data to different place like a centralised Data Lake), each domain has it's own way of storage, can be a file-based storage or a relational db. They just have to make it available for the "outside world" so that it's usable and can be combined with the data from another domains.
There are some different aspects of DataMesh like the self-service platform (we need to somehow find the data and understand what it represents), access rights management (table masking, row masking, making the data not being able to out of some region) etc.
Most important, Data Mesh is a change in a company culture, to not work in silos anymore but rather cooperate with each other, take responsibility for the data as a domain owner, make it available (and usable) for others within the company.