Category: Data Warehouse ETL
-
Complex Data Integration: Best Practices
With 25 years plus experience as a Data Architect in both Fortune 500 and start up business I’ve seen a lot of pain points of complex data integration. Let’s list out typical issues facing data integration; (Note: This article is a work in progress) Poorly designed data models (This deserves 5 bullet points) Large and/or…
-
Complex Data Integration: Best Practices
With 25 years plus experience as a Data Architect in both Fortune 500 and start up business I’ve seen a lot of pain points of complex data integration. Let’s list out typical issues facing data integration; Poorly designed data models (This deserves 5 bullet points) Large and/or complex schema Poorly structured flat files Large SQL…
-
Use Case: Automatic ETL schema generation
Semistructured source data with automatic ETL schema generation at SQL destination Problem:A Software as a Service (SaaS) company stores OLTP data in a mixed format of relational and semi-structured data. This multi-tenant data consists of workflow, form data and documents. The company must export the data for their clients in a tabular form by pivoting the…
-
ETL Data Lake vs Data Warehouse
Rules Based/Metadata ETL has Lowered the LOE of ETL Typically Associated with a Data Warehouse. METL makes a fundamental change of the cost calculus between a Data Lake vs the IDW. A Data Lake is loosely integrated data typically placed in Hadoop. An IDW has tightly integrated data stored in either a relational database and/or…