DATA INTEGRATED. DECISIONS ELEVATED.
Snowflake and dbt Assessment and Optimization
Overview
D&I Integrators conducted a thorough current state assessment of a major apparel retailer's Snowflake and dbt environments, analyzing 54.5 TB of data, 2,500 plus tables, and 428 dbt models processing 40 to 50 TB daily. The engagement identified critical performance bottlenecks, developed a detailed optimization plan, and led to ongoing staff augmentation of data engineering services.
The Challenge
The environment had grown to significant scale, over 2,500 tables and hundreds of dbt models processing tens of terabytes daily, without performance tuning to match. Undersized BI warehouses, aggressive auto suspend settings, and poor model partitioning were slowing reports and driving up compute costs, with no clear roadmap for fixing the underlying causes.
the solution
01
Infrastructure Evaluation
Comprehensive analysis of the data environment architecture, reviewing naming conventions, medallion architecture implementation, and schema structures.
03
Workflow Optimization
Developed strategic remediation approaches, including query performance improvements through partition pruning and incremental load fixes, upsized key warehouses, and reviewed queries scanning more than 50 percent of partitions.
02
Performance Analysis
Evaluated current state performance across warehouses, queries, and data pipelines to identify suboptimal outcomes, including severely undersized BI warehouses, aggressive auto suspend settings, and poor model partitioning.
04
Financial Impact and Implementation Roadmap
Calculated projected annual savings of $50K plus and created a detailed phased implementation plan, including a new optimization pattern library, intelligent caching, clustering on large fact tables, and automated monitoring for ongoing maintenance.
Why It Matters
By cleaning up the data model at every layer, from Snowflake schemas to ThoughtSpot logic to the liveboards themselves, the retailer's distribution center analytics platform is faster, easier to maintain, and built on assets the broader organization can reuse with confidence.