Category: Data Warehouse ETL

  • Complex Data Integration: Best Practices

    With 25 years plus experience as a Data Architect in both Fortune 500 and start up business I’ve seen a lot of pain points of complex data integration. Let’s list out typical issues facing data integration; (Note: This article is a work in progress) Poorly designed data models (This deserves 5 bullet points) Large and/or…

  • Complex Data Integration: Best Practices

    With 25 years plus experience as a Data Architect in both Fortune 500 and start up business I’ve seen a lot of pain points of complex data integration. Let’s list out typical issues facing data integration; Poorly designed data models (This deserves 5 bullet points) Large and/or complex schema Poorly structured flat files Large SQL…

  • Use Case: Automatic ETL schema generation

    Semistructured source data with automatic ETL schema generation¬†at SQL destination Problem:A Software as a Service (SaaS) company stores OLTP data in a mixed format of relational and semi-structured data. This multi-tenant data consists of workflow, form data and documents. The company must export the data for their clients in a tabular form by pivoting the…

  • ETL Data Lake vs Data Warehouse

    Rules Based/Metadata ETL has Lowered the LOE of ETL Typically Associated with a Data Warehouse. METL makes a fundamental change of the cost calculus between a Data Lake vs the IDW. A Data Lake is loosely integrated data typically placed in Hadoop. An IDW has tightly integrated data stored in either a relational database and/or…