Data Mesh

Decentralized Data Architecture

1. Overview: A Domain-Distributed Data Architecture That Overcomes the Bottleneck of Centralized Data Management

    flowchart LR
    A["Centralized data-team<br/>bottleneck<br/>(data swamp)"] --"Distributed domain<br/>ownership & productization"--> B["Domain-autonomous<br/>data product management"] --"Self-service platform +<br/>federated governance"--> C["Data democratization &<br/>accelerated value realization"]

    style A fill:#FFEBEE,stroke:#D32F2F,color:#000
    style B fill:#E3F2FD,stroke:#1976D2,color:#000
    style C fill:#E8F5E9,stroke:#388E3C,color:#000

Definition: A decentralized data architecture in which, instead of a single central team managing data, each business domain team directly owns and manages its own data, serving it as a data product, and operates enterprise-wide data on top of a self-service platform and federated governance.

Characteristics: (Domain-Driven Design applied to data) Applies DDD principles to data management, reflecting business reality in the data structure. (Autonomous production/consumption) Domain teams produce and consume data autonomously, without a central data team as a bottleneck. (Four core principles) Implemented on four principles: domain ownership, data as a product, self-service, and federated governance.

2. Core Components of Data Mesh

A. The Four Core Principles

    flowchart TD
    subgraph R1["　"]
        direction LR
        P1["Domain-oriented<br/>decentralized ownership<br/>The business domain directly<br/>owns and manages its data"]
        P2["Data as a Product (DaaP)<br/>Consumer-centric<br/>quality assurance<br/>Discoverable and trustworthy"]
    end
    subgraph R2["　"]
        direction LR
        P3["Self-serve<br/>data platform<br/>Abstracts infrastructure complexity<br/>Supports domain autonomy"]
        P4["Federated<br/>computational governance<br/>Global standards + local autonomy<br/>Guarantees interoperability"]
    end

    style P1 fill:#E3F2FD,stroke:#1976D2,color:#000
    style P2 fill:#F3E5F5,stroke:#7B1FA2,color:#000
    style P3 fill:#FFF3E0,stroke:#F57C00,color:#000
    style P4 fill:#E8F5E9,stroke:#388E3C,color:#000
    style R1 fill:none,stroke:none
    style R2 fill:none,stroke:none

Principle	Core Content	Difference from Centralized Management
Domain ownership	The business unit that knows the data best owns, manages, and is accountable for it	Responsibility shifts from a central data team to domain teams
Data as a Product	Data is developed, deployed, and maintained like a product for internal consumers	Pipeline output becomes a discoverable, trustworthy, usable product
Self-serve platform	Domain teams use data infrastructure without needing deep technical expertise	Reliance on central engineers is replaced by platform abstraction
Federated governance	Enterprise-wide common standards are maintained while preserving domain autonomy	Central control is replaced by autonomous, standards-based governance

B. Data Product Architecture

    flowchart LR
    subgraph DP["Data Product"]
        direction TB
        IN["Input port<br/>Source systems, events, APIs"]
        TR["Transformation logic<br/>Cleansing, aggregation, feature generation"]
        OUT["Output port<br/>Query API, stream, files"]
        IN --> TR --> OUT
    end

    subgraph PLANE["Mesh Platform Layer"]
        direction TB
        EXP["Experience plane<br/>Data catalog, search, access management"]
        INF["Infrastructure plane<br/>Storage, compute, CI/CD, security"]
    end

    DP <--> PLANE

    style DP    fill:#E3F2FD,stroke:#1976D2,color:#1E3A5F
    style PLANE fill:#E8F5E9,stroke:#388E3C,color:#1B5E20

Component	Role	Key Technology
Input port	Receives data from source systems, events, or other data products	Kafka, CDC, REST API
Transformation logic	Cleansing, aggregation, feature generation per domain business rules	dbt, Spark, SQL
Output port	The interface consumers access (query, stream, or file)	REST API, Snowflake, S3
Experience plane	Supports discovery and access management for data products	Datahub, Collibra, Apache Atlas
Infrastructure plane	Common infrastructure for domain teams to build and deploy data products	Terraform, Kubernetes, Airflow

3. Expected Benefits and Practical Application of Data Mesh

Category	Key Expected Benefit	Practical Application
Scalability	Data processing capacity grows in proportion to the number of domains	Form independent data-product teams per domain and roll out incrementally
Speed to value	Faster data analysis without dependence on a central team	Build a business-led, self-service analytics environment
Data quality	Direct data management by source-system experts	Operate SLA-based data-product quality metrics (freshness, completeness)
AI/ML acceleration	A feature store built on high-quality domain data products	Use data products as inputs to the ML feature pipeline

Data Lake Architecture NoSQL / CAP Theorem