What is the P in "ELTP Process"?

Adding P to ELT
If you have worked with a data engineer, you might have heard ETL or ELT.
- E for Extracting data from the source.
- T for Transforming from raw data to clean and useful information.
- L for Loading to the destination data lake or data warehouse.
The trend is to transform the data after loading it to a powerful modern data warehouse such as Google BigQuery, Snowflake, AWS Redshift, or Microsoft Synapse. So, ELT rather than ETL is increasingly used to refer the data processing.
I’m adding a P to ELT, and it’s "Publishing." It’s the process of publishing the post-transformation data to exactly where people or another program will consume them. We call this "frontline application".
ELT does not solve the "data last mile problem."
Consider this business scenario: Product Qualified Lead scoring (PQL scoring).
Suppose your company is a Software-as-a-Service (SaaS) company. The basic features are free. Your potential customer finds your service through an online ad. Sign up is free, but the user must enter their work email and business information.
If you are using online marketing automation tools such as Marketo or Pardot, you can capture the prospective customer’s product discovery channel and their email at this point. But marketing processes typically disconnected from usage statistics. It’s because usage data is typically sitting in a production database.
Or you may have gone one step further to replicate the data from the production database to the data warehouse through an ELT process. You may be a data scientist who came up with a formula or machine learning algorithm to compute a PQL score to indicate which free-tier users would likely convert to paying customers.
But as long as the data is sitting in the warehouse, it’s not going to be used. A PSQL score should be published to Marketo, Pardot, or Salesforce because that is where the sales and marketing staff do their job. They are too busy to open a business intelligence tool or run queries to find out which prospects should be prioritized.

Publish: Push the data out of the warehouse
The importance of publishing the metrics to frontline applications is critical beyond product marketing use cases. Another compelling case for a SaaS business is customer success. For a subscription-based service, it is crucial to track the health of each subscriber account. Especially for complex business applications, the customer may give up on the product before seeing the value. Are your customers taking the right steps towards getting value after signing up, or are they having difficulties starting?
Enterprise SaaS companies typically have a customer success function to help new accounts in the onboarding process and beyond. Product usage statistics and the account’s health score would be beneficial if only information were available right where they do their work, such as Zendesk.
ELTP’s P takes care of the last mile problem of information delivery to make business operations smart, efficient, and lean.
ELTP automation
Over the last years, the ELT business grew. There are so many services to automatically move data from various online applications to a data warehouse. But there are few resources and services available to automate data publishing, much less provide no-code solutions.
Lack of a no-code solution does not have to stop a business from taking advantage of the powerful ELTP process. A little bit of investment in data engineering will pay dividends for the entire business operation.
One of the popular Open Source ELT frameworks is singer.io. The singer.io community builds data extractors called "tap" and data loader called "target". Singer.io’s specification helps data engineers mix-and-match taps and targets to create a source-destination combo for each business use case. In a typical ELT framework, cloud applications such as Salesforce and Marketo are the data sources (taps), and the data warehouses are the destinations (targets).
When we engage in the P of "ELTP", we reversed the designation: For example, we developed a tap program for BigQuery to extract product usage metrics and developed a target program for Pardot. By running this tap-target combination, we automated the process of publishing the product usage data from BigQuery to Pardot so that our clients’ marketing and sales teams can fully use the PQL metrics with no need to manually move data around.
Future of ELTP
Data publishing is not limited to human consumption. The computed metrics can be replicated back to the production data store or caching layer to make the product’s user experience more optimized and personalized. The metrics could be based on simple statistics or the result of more complex computation by machine-learning. By taking care of the last-mile problem and ensuring the valuable signal is delivered right where it matters, we can unlock unseen potential.
There will be more new businesses providing no-code solutions and services to close the loop in the near future. In the meantime, we will help businesses with our custom ELTP solution and make more success stories in sales, marketing, and customer successful use cases.
A fun demonstration
Here is a fun demonstration of solving the "last mile problem." These GIF animations are created and posted on Twitter automatically at specified intervals. We extract the data from the sources (geological and financial), transform the data (including the part to produce the animation), and deliver it to where it matters (social media).


About the author
Daigo Tanaka is the CEO and data scientist at ANELEN Co., LLC, a boutique consulting firm focused on data engineering, Analytics, and data science. ANELEN’s mission is to help innovative businesses make smarter decisions with data science.
Originally published at https://articles.anelen.co on February 13, 2021.