Big Data 5V

Volume · Velocity · Variety · Veracity · Value

1. Overview: An Analysis Framework for Creating Value from the Five Attributes of Big Data (5V)

    flowchart LR
    A["The 3V era<br/>(Volume, Velocity, Variety)<br/>Focused on quantitative traits"] --"Extend with<br/>Veracity & Value"--> B["Integrated 5V framework<br/>Combining quantitative & qualitative traits"] --"Architecture<br/>design criteria"--> C["Business value creation<br/>from big data"]

    style A fill:#E3F2FD,stroke:#1976D2,color:#000
    style B fill:#1E3A5F,stroke:#1E3A5F,color:#fff
    style C fill:#E8F5E9,stroke:#388E3C,color:#000

Definition: A framework that defines big data through five attributes — Volume, Velocity, Variety, Veracity, and Value — and designs collection, storage, processing, and analysis architectures suited to each attribute in order to create business value from data.

Characteristics: (The 5V extended model) An extension of the original 3V (Volume, Velocity, Variety) that adds Veracity and Value. (Integrated quantitative/qualitative management) Manages both quantitative traits (volume, velocity, variety) and qualitative traits (veracity, value) together. (Architecture selection criterion) Used as a criterion for selecting big-data processing architectures such as the Lambda Architecture and Kappa Architecture.

2. Core Components of Big Data 5V

A. Volume, Velocity, Variety, Veracity, Value

    flowchart LR
    V1["Volume<br/>Scale<br/>TB/PB class<br/>Massive datasets"]
    V2["Velocity<br/>Speed<br/>Real-time/streaming<br/>Millions of events per second"]
    V3["Variety<br/>Diversity<br/>Structured, semi-structured,<br/>and unstructured mixed together"]
    V4["Veracity<br/>Accuracy<br/>Quality/reliability<br/>Noise removal"]
    V5["Value<br/>Value<br/>Deriving insight<br/>Supporting decisions"]

    V1 --> V5
    V2 --> V5
    V3 --> V5
    V4 --> V5

    style V1 fill:#E3F2FD,stroke:#1976D2,color:#000
    style V2 fill:#F3E5F5,stroke:#7B1FA2,color:#000
    style V3 fill:#FFF3E0,stroke:#F57C00,color:#000
    style V4 fill:#FFEBEE,stroke:#D32F2F,color:#000
    style V5 fill:#1E3A5F,stroke:#1E3A5F,color:#fff

Attribute	Definition	Core Challenge	Enabling Technology
Volume	Tens of terabytes to petabytes or more of data	Distributed storage and parallel processing	HDFS, S3, data lake
Velocity	Real-time data generated at millions of events per second	Low-latency stream processing	Kafka, Spark Streaming, Flink
Variety	Structured (DB), semi-structured (JSON/XML), and unstructured (image/text) data	Integrating diverse formats	Spark, Hive, NoSQL
Veracity	Degraded quality, noise, and incomplete data mixed in	Quality validation and cleansing	DQ tools, data profiling
Value	Deriving meaningful insight from large volumes of data	Developing and deploying analytical models	ML/AI platforms, BI tools

B. Big Data Analytics Architectures

Lambda Architecture (batch + stream hybrid)

    flowchart TD
    SRC["Source data<br/>(events, logs, transactions)"]

    subgraph BATCH["Batch Layer"]
        direction LR
        B1["Store all data<br/>HDFS / S3"]
        B2["Batch processing/aggregation<br/>Spark / Hive"]
    end

    subgraph SPEED["Speed Layer"]
        direction LR
        S1["Real-time stream ingestion<br/>Kafka / Kinesis"]
        S2["Stream processing<br/>Flink / Storm"]
    end

    subgraph SERVING["Serving Layer"]
        direction LR
        SV1["Batch view"]
        SV2["Real-time view"]
    end

    APP["Application services<br/>BI · ML · API"]

    SRC --> BATCH
    SRC --> SPEED
    BATCH --> SERVING
    SPEED --> SERVING
    SERVING --> APP

    style BATCH   fill:#E3F2FD,stroke:#1976D2,color:#1E3A5F
    style SPEED   fill:#FFF3E0,stroke:#F57C00,color:#7C3700
    style SERVING fill:#E8F5E9,stroke:#388E3C,color:#1B5E20
    style APP     fill:#1E3A5F,stroke:#1E3A5F,color:#fff

Layer	Role	Characteristics	Key Technology
Batch Layer	Stores all source data and periodically batch-processes it	High accuracy, but processing latency exists	HDFS, Spark, Hive
Speed Layer	Processes real-time streaming data immediately to produce an up-to-date view	Low latency, processes partial data	Kafka, Flink, Storm
Serving Layer	Merges the batch view and real-time view to serve query responses	Fast reads, combines up-to-date and accurate data	HBase, Cassandra, Druid

Kappa Architecture: A simplified model that handles both batch and stream processing through the Speed Layer alone. Implemented on Kafka Streams or Flink, with lower operational complexity than Lambda.

3. Expected Benefits and Practical Application of the Big Data 5V Framework

Category	Key Expected Benefit	Practical Application
Architecture design	Choosing the optimal processing architecture based on 5V characteristics	Strengthen the Speed Layer for real-time needs, the Batch Layer for accuracy needs
Quality assurance	Improved analytical confidence through Veracity management	Automate profiling/cleansing within the data quality pipeline
Cost optimization	Reduced storage cost through tiering by use case	Apply hot/warm/cold tiering and compression/partitioning strategies
AI/ML integration	Improved AI model training quality from high-quality, high-volume data	Build a feature store and integrate with the MLOps pipeline

MDM (Master Data Management)Data Lake Architecture