Data Lake Architecture

1. Overview: An Integrated Data Platform That Centrally Stores Raw Data for Multi-Purpose Use

    flowchart LR
    A["Data silos<br/>(scattered, non-standard,<br/>heterogeneous stores)"] --"Centrally collect and<br/>store raw data"--> B["Data lake<br/>(single repository)"] --"Process/analyze<br/>per use case"--> C["Multi-purpose use:<br/>BI · ML · AI"]

    style A fill:#FFEBEE,stroke:#D32F2F,color:#000
    style B fill:#E3F2FD,stroke:#1976D2,color:#000
    style C fill:#E8F5E9,stroke:#388E3C,color:#000

Definition: A scalable data platform architecture that centrally loads structured (DB/DW), semi-structured (JSON/XML/logs), and unstructured (image/video/text) data in raw format into large-scale distributed storage, then processes and analyzes it for specific purposes as needed.

Characteristics: (Schema-on-Read) No schema is defined at ingestion time; structure is applied at read time (in contrast to a DW’s Schema-on-Write). (Horizontally scalable) A horizontally scalable structure built on low-cost object storage such as S3, HDFS, and Azure ADLS. (Risk of a “data swamp”) Without proper management, the absence of data quality and governance can turn it into a data swamp.

2. Core Components of Data Lake Architecture

A. The Four-Layer Architecture

    flowchart TD
    subgraph ING["Ingestion Layer"]
        direction LR
        I1["Batch ingestion<br/>Sqoop/Flume/ETL"]
        I2["Streaming ingestion<br/>Kafka/Kinesis/Fluentd"]
        I3["API/CDC ingestion<br/>REST/Debezium"]
    end

    subgraph STR["Storage Layer"]
        direction LR
        S1["Raw Zone<br/>(raw data)"]
        S2["Curated Zone<br/>(cleansed data)"]
        S3["Analytics Zone<br/>(analysis-ready data)"]
    end

    subgraph PROC["Processing Layer"]
        direction LR
        P1["Batch processing<br/>Spark/Hive"]
        P2["Stream processing<br/>Flink/Spark Streaming"]
        P3["Interactive query<br/>Presto/Athena"]
    end

    subgraph SRV["Serving Layer"]
        direction LR
        V1["BI/dashboards<br/>Tableau/Power BI"]
        V2["ML/AI platform<br/>SageMaker/MLflow"]
        V3["API/application services<br/>REST/GraphQL"]
    end

    ING --> STR --> PROC --> SRV

    style ING  fill:#E3F2FD,stroke:#1976D2,color:#1E3A5F
    style STR  fill:#F3E5F5,stroke:#7B1FA2,color:#4A148C
    style PROC fill:#FFF3E0,stroke:#F57C00,color:#E65100
    style SRV  fill:#E8F5E9,stroke:#388E3C,color:#1B5E20

Layer	Role	Key Technology
Ingestion Layer	Collects raw data from diverse sources via batch, streaming, or API	Kafka, Kinesis, Flume, Sqoop, Debezium
Storage Layer	Manages data maturity across Raw → Curated → Analytics zones	HDFS, AWS S3, Azure ADLS, Delta Lake
Processing Layer	Large-scale data processing and transformation — batch, streaming, or interactive	Apache Spark, Flink, Hive, Presto
Serving Layer	Delivers data to end consumers as analytics, ML, or an API	Tableau, SageMaker, REST API

Storage Zone Design

Zone	Data State	Retention Policy
Raw Zone (Landing)	Raw data kept unmodified	Long-term retention, audit baseline
Curated Zone (Silver)	Cleansed, standardized, partitioned	Loaded after quality validation
Analytics Zone (Gold)	Business-logic aggregation and feature generation complete	Reserved for BI/ML consumption

B. Evolution to the Data Lakehouse and Governance

    flowchart LR
    subgraph R1["　"]
        direction LR
        DW["Data Warehouse<br/>Structured data, high-performance queries<br/>High cost, low flexibility<br/>Schema-on-Write"]
        DL["Data Lake<br/>Stores data of any form<br/>Low cost, high flexibility<br/>Weak governance"]
    end
    subgraph R2["　"]
        direction LR
        DLH["Data Lakehouse<br/>DW + DL combined<br/>ACID transaction support<br/>Delta Lake/Iceberg/Hudi"]
        DM["Data Mesh<br/>Distributed management by domain<br/>Data-as-a-product<br/>Federated governance"]
    end

    style DW  fill:#E3F2FD,stroke:#1976D2,color:#000
    style DL  fill:#F3E5F5,stroke:#7B1FA2,color:#000
    style DLH fill:#1E3A5F,stroke:#1E3A5F,color:#fff
    style DM  fill:#E8F5E9,stroke:#388E3C,color:#000
    style R1 fill:none,stroke:none
    style R2 fill:none,stroke:none

Key Data Lake Governance Challenges

Challenge	Problem	Resolution
Data quality	Errors and duplicates left unaddressed when raw data is loaded → a data swamp	DQ validation gate at ingestion, apply Great Expectations
Metadata management	Location, meaning, and owner of data unknown → data cannot be discovered	Integrate a data catalog (Datahub, AWS Glue)
Access control	Unrestricted access to all data → privacy violations	Column/row-level access control, data masking
ACID guarantees	Inconsistency from concurrent writes/updates	Adopt Delta Lake, Apache Iceberg, or Apache Hudi
Cost management	Uncontrolled storage growth → storage cost explosion	Data lifecycle policy (hot/warm/cold tiering)

3. Expected Benefits and Practical Application of Data Lake Architecture

Category	Key Expected Benefit	Practical Application
Cost reduction	Tens-fold savings from low-cost object storage compared to a DW	Move cold data to S3/GCS, keep only hot data in the DW
Flexibility	Accommodates unstructured/semi-structured data, enabling AI/ML data	Build training data pipelines for AI models based on image/text/log data
Scalability	Handles petabyte-scale data through horizontal scaling	Leverage on-demand compute resources via a cloud-native architecture
Lakehouse transition	ACID support unifies DW functionality with data-lake flexibility	Adopt Delta Lake for real-time updates and time-travel capability

Big Data 5V Data Mesh