Data Quality Management (DQC)

Data Quality Management

1. Overview of DQC — A Quality Diagnosis and Improvement System for Continuously Securing Trustworthy Data

    flowchart LR
    A["Data quality problems<br/>(errors, missing values,<br/>inconsistency)"] --"Define and measure<br/>quality criteria"--> B["Diagnose and analyze<br/>current quality state"] --"Remediation<br/>and monitoring"--> C["Secure trustworthy<br/>data assets"]

    style A fill:#FFEBEE,stroke:#D32F2F,color:#000
    style B fill:#E3F2FD,stroke:#1976D2,color:#000
    style C fill:#E8F5E9,stroke:#388E3C,color:#000

Definition: A data management system that measures and diagnoses the quality of an organization’s structured and unstructured data against criteria such as completeness, accuracy, consistency, and validity, and that systematically remediates and monitors any quality issues found.

Characteristics: (Six quality dimensions) Defines and measures data quality across six dimensions: completeness, accuracy, consistency, validity, timeliness, and uniqueness. (Integrated quality management) Covers not only structured data (databases, data warehouses) but also unstructured data (text, images, logs) in an integrated quality-management approach. (Shift to prevention) Shifts the center of gravity from post-hoc cleansing toward prevention and continuous monitoring.

2. Core Structure of DQC

A. Structured and Unstructured Data Quality

    flowchart TD
    subgraph R1["　"]
        direction LR
        Q1["Completeness<br/>Are required values<br/>filled in without gaps?"]
        Q2["Accuracy<br/>Does the data match<br/>reality?"]
        Q3["Consistency<br/>Is the value the same<br/>across systems?"]
    end
    subgraph R2["　"]
        direction LR
        Q4["Validity<br/>Does it comply with the<br/>defined format/rules?"]
        Q5["Timeliness<br/>Is it up to date<br/>when needed?"]
        Q6["Uniqueness<br/>Is it a single record<br/>with no duplicates?"]
    end

    style Q1 fill:#E3F2FD,stroke:#1976D2,color:#000
    style Q2 fill:#F3E5F5,stroke:#7B1FA2,color:#000
    style Q3 fill:#FFF3E0,stroke:#F57C00,color:#000
    style Q4 fill:#FFEBEE,stroke:#D32F2F,color:#000
    style Q5 fill:#E8F5E9,stroke:#388E3C,color:#000
    style Q6 fill:#E0F2F1,stroke:#00796B,color:#000
    style R1 fill:none,stroke:none
    style R2 fill:none,stroke:none

Dimension	Structured Data Quality Issue	Unstructured Data Quality Issue
Completeness	NULL values in required columns, missing records	Blank text, corrupted images, missing log entries
Accuracy	Incorrect code values, date errors	Typos, mislabeling, sensor outliers
Consistency	Code mismatches between systems, mixed units	Inconsistent language/expression, duplicate documents
Validity	Format violations (phone number, email)	Disallowed characters, format non-compliance
Timeliness	Delayed batch loads, expired data	Stale content, delayed real-time feeds
Uniqueness	Duplicate keys, duplicate customer records	Duplicate storage of the same document, duplicate images

B. Quality Diagnosis and Improvement Process

    flowchart LR
    PRF["Profiling<br/>Analyze current-state statistics<br/>Detect anomalous patterns"]
    DIA["Diagnosis<br/>Root-cause analysis<br/>Classify quality issues"]
    CLN["Cleansing<br/>Correct errors, standardize<br/>Handle missing values"]
    ENR["Enrichment<br/>Combine external data<br/>Add derived attributes"]
    MON["Monitoring<br/>Track quality metrics<br/>Threshold alerts"]

    PRF --> DIA --> CLN --> ENR --> MON
    MON -->|"Rediagnose<br/>on recurrence"| PRF

    style PRF fill:#E3F2FD,stroke:#1976D2,color:#000
    style DIA fill:#F3E5F5,stroke:#7B1FA2,color:#000
    style CLN fill:#FFF3E0,stroke:#F57C00,color:#000
    style ENR fill:#FFEBEE,stroke:#D32F2F,color:#000
    style MON fill:#E8F5E9,stroke:#388E3C,color:#000

Stage	Key Activities	Core Tools/Techniques
Profiling	Statistical analysis of data distribution/format/patterns, detecting anomalies	SQL statistical queries, Talend, Informatica DQ
Diagnosis	Classifying the root cause of quality issues (business-rule violation, system error, input error)	Quality-error classification scheme, root cause analysis (RCA)
Cleansing	Correcting erroneous data, standardization, imputing missing values, deduplication	Match & merge, rule-based cleansing, ML-based correction
Enrichment	Combining external reference data, generating derived attributes, adding metadata	Reference data mapping, MDM integration
Monitoring	Operating quality-metric dashboards, alerting and auto-remediation on threshold breach	Great Expectations, dbt tests, Grafana

3. Expected Benefits and Application of DQC

Category	Key Benefit	Application in Practice
Decision-making confidence	Fewer management errors from reliance on accurate data	Introduce data-quality grading for KPI calculation and track issues
AI model quality	Improved model accuracy from higher-quality training data	Apply a DQ gate in the training-data pipeline
Regulatory compliance	Fulfilling data-accuracy obligations under privacy law and GDPR	Use data quality reports as audit evidence
Operational efficiency	Reduced rework and incident cost from faulty data	Minimize downstream impact through early detection of quality issues

NoSQL / CAP Theorem CRISP-DM