Contact Us
Semantic ETL Completes Your Agile Technology Stack
Semiotics' metadata-managed ETL is similar in concept to the semantic layers of ORM or Business Intelligence. It provides a declarative low or no-code solution that speeds development. When adding our rules engine this can reach the level of being automated. Semiotic is unopinionated and completely configurable to your business and technical requirements. Of course, we have pre-configured rules for turn-key setup.
How Automated ETL Management increased efficiency by 90%
Our customer imports and verifies a copious amount of financial data from third parties. Their main software platform is a SaaS vendor's BPM app (Business Process Management) with workflows and dynamic forms. The BPM app stores data in a MySQL database using a hybrid of SQL and a complex JSON data model. The workflows and forms are customized and are constantly changing. Semiotic automatically, based on the customer's defined rules, normalizes the data (including JSON Arrays) into a reporting database. Form data, each with its own JSON schema, are split into tables with the form name becoming the table name. JSON arrays are also split into tables with foreign keys and surrogate keys automatically added. Considering source data is from third parties, data cleansing is required, which is also automated. Semiotic creates integrated, properly data-typed, and cleansed data models that the Looker and PowerBI analytics team can easily query. Schema changes, both SQL and JSON, are automatically detected and propagated. Semiotic's quality assurance tools have various data checks and monitoring that provide assurance the ETL is working correctly. If the QA check does detect an irregularity, most scenarios allow remediation with just a few clicks.
The Customer's Win
Our customer can create and deploy workflows and form changes without any ETL cost or delays. The analytics team can query and create BI models against SQL schema without any JSON manipulations. Additionally, data is properly data-typed and cleansed. The business has continuous monitoring of ETL. The system is fully documented and has searchable metadata. Data management costs and LOE is a fraction of other solutions.
Use Cases
Semantic ETL combines the best of both worlds between the Integrated Data Warehouse (IDW) vs the Data Lake (DL). It maintains a managed schema of a IDW while keeping the agility of a DL. You can also consider a hybrid between managed and unmanaged schema, so parts of source data map to SQL while other remain Json.
Note that Semiotic is unopinionated in regards to your data architecture. We handle SQL or Json and hierarchical data natively. It's all in your configuration and rules engine choices.
Manage large schemas with vastly reduced effort compared to ETL programming.
Don't let ETL slow your team down. Automatic, rules based schema change propagation from your OLTP database (SQL or NoSQL) to destination datastores. Plus we handle Json and dynamic forms.
Workflows and dynamic forms typical use some sort of hybrid sql, json or nosql. Semiotic can automatically normalized, cleanse and homogenize data into a new data model so it's ready for analytics or data feeds.
Sweep a file directory, ingest multiple files, read column headers, sample data, detect schema changes then create or update metadata and optionally the destination schema.
Native Json processing and metadata that supports Json Object and Json Arrays allows Semiotic to effortlessly integrate SQL, NoSQL and REST. We can automatically normalize Json hierarchies, including Json arrays, into separate tables including adding surrogate keys etc.
Allow your customers, vendors or business users to define and manage their own data feeds via a web portal. Manual or automated subscriptions. Authentication, auditing, logging and creating rules.
Microservices interface allows easy integration. Add real-time/codeless data integration to your app. OEM licensing available.
Seamlessly handle SQL and semi-structured data. Pivot/unpivot and normalize Json automatically.
Archive databases and files and store the data in low-cost cloud storage. Capture metadata and index it in a search engine like Elasticsearch.
Semiotic empowers your business and data analyst to be more productive without needing deep programming skills.
Scale up and/or scale out. Parallel processing, Kubernetes and Kafka queues.
Contact Us
Learn More
Pain Points of Data Integration
Problems
Resolution
Page [tcb_pagination_current_page] of [tcb_pagination_total_pages]