DATA INTEGRATED. DECISIONS ELEVATED.
GCP Migration and AI Enablement
Overview
D&I Integrators is conducting a large scale data migration for a home improvement retailer, moving its Oracle based Retail Data Warehouse to Google Cloud Platform to unlock modern analytics and AI capabilities for its merchandising organization. The project involves migrating key dimension and fact tables containing billions of rows of historical retail data.
The Challenge
The retailer's Retail Data Warehouse ran on an on premise Oracle system that limited modern analytics and made AI capabilities difficult to build for the merchandising organization. Migrating billions of rows of historical data to BigQuery meant resolving significant type compatibility gaps between Oracle and BigQuery, along with infrastructure constraints and sheer data scale.
the solution
Three distinct ingestion patterns were built to match table size and automation needs: manual Python CSV extraction for smaller dimension tables, custom Parquet chunking for a 9.7 billion row inventory fact table, and fully automated Apache Beam pipelines on Dataflow for end to end extraction, transformation, and load.
01
Oracle Schema Analysis and BigQuery Design
Extracted Oracle source schemas and mapped all column types to BigQuery compatible equivalents, with special handling for Oracle DATE fields, high precision NUMBER fields, and columns prone to autodetect misclassification.
03
BigQuery Load and Data Validation
Loaded all tables into BigQuery using explicit schema definitions, with time partitioning applied to large fact tables, and validated row counts and schema fidelity across all target datasets.
02
Data Extraction and Staged Ingestion to GCS
Developed Python extraction scripts to pull Oracle data in chunked, date partitioned batches, exporting to Parquet before landing in Google Cloud Storage, with parallel workstreams to accelerate large fact table loads.
04
Dataflow Automation via Airflow CI/CD
Built Apache Beam YAML pipelines and Airflow DAGs to automate the full Oracle to GCS to BigQuery flow, version controlled and automatically deployed to Cloud Composer.
Why It Matters
The migration gives the retailer's merchandising organization a modern, scalable cloud foundation, replacing rigid on premise infrastructure with a platform built for both current reporting needs and future AI capabilities.