Mindblown: a blog about philosophy.
-
Complex Data Integration: Best Practices
With 25 years plus experience as a Data Architect in both Fortune 500 and start up business I’ve seen a lot of pain points of complex data integration. Let’s list out typical issues facing data integration; (Note: This article is a work in progress) Poorly designed data models (This deserves 5 bullet points) Large and/or…
-
ETL MongoDB and SQL
ETL MongoDB to Oracle, SQL Server, MySQL & Postgres Note: This article is a work in progress. MongoDB is a Json based NoSQL database that stores your hierarchical data in a collection (equivalent to a table in a SQL database). MongoDB is schema-less because it’s Json schema can be modified without executing explicit commands. SQL…
-
ETL of Loan Boarding / Mortgage Servicing
Problems: ETL of Boarding Loans ETL of loan boarding is a complex data integration process that is a significant pain point for the mortgage servicing industry. Mortgage transfers contain a lot of files, requiring a lot of data mappings to be managed. Compound that with every mortgage servicer have slightly different file format standards so…
-
Complex Data Integration: Best Practices
With 25 years plus experience as a Data Architect in both Fortune 500 and start up business I’ve seen a lot of pain points of complex data integration. Let’s list out typical issues facing data integration; Poorly designed data models (This deserves 5 bullet points) Large and/or complex schema Poorly structured flat files Large SQL…
-
Balancing End User Needs with Flexible Data Architectures
By Jared DeckerExpertAnalytics.com To accommodate the massive influx and wide variety of data that organizations are capturing and storing for analytical and strategic value, approaches that accommodate flexible data structures and file types (such as data lakes) are increasingly common. These architectures require few accommodations when bringing data in but can involve significant processing…
-
Debunking the Visual vs Tabular Preference Dichotomy in Data Analysis Presentations
By Jared DeckerExpertAnalytics.com We often come across people that self-identify as fitting into one of two categories: visual people or numbers people. This discussion often comes up when determining how data should be presented to different business audiences using business intelligence and analytics tools that have been purchased by the company or custom-developed in-house. Often times we…
-
ETL SQL into Elasticsearch
Low latency, complex data model SQL data synchronization with Elasticsearch Elasticsearch (ES) is a NoSQL databases optimized for the fuzzy logic of text search. SQL databases tend to struggle with text search. They are CPU and IO intensive and not as flexibility with the search logic. So there is a strong need for a SQL…
-
Declarative ETL Cost Savings
Legacy ETL solutions are a procedural programming exercise. An ETL process starts with a source query pulling data, then through some workflow steps the data transformations are executed with data finally applied at the destination. Intelligent Integration’s Json document framework fundamentally changes the traditional paradigm from procedural programming to declarative. In declarative programming the goal…
-
ETL NoSQL Best Practices
Mongo, Redis, Elasticsearch or Couch Legacy ETL tools are problematic for NoSQL databases. They assume tabular data and are not well suited for hierarchical Json data typically used in NoSQL databases. They also assume batch processing and JDBC style connection. ETL for NoSQL database use cases likely require a restful interface and streaming for low…
-
Use Case: Automatic ETL schema generation
Semistructured source data with automatic ETL schema generation at SQL destination Problem:A Software as a Service (SaaS) company stores OLTP data in a mixed format of relational and semi-structured data. This multi-tenant data consists of workflow, form data and documents. The company must export the data for their clients in a tabular form by pivoting the…
Got any book recommendations?