The term was a pet name for what we envisioned the end product to be, a universe of metadata and relationships between all the artifacts and concepts in the Pentaho stack. You may see the term 'metaverse' listed throughout the code and documentation (including the project name itself). The code for the current data lineage capability is entirely open source and is available on GitHub here. So in that sense you can follow your data all the way through your PDI process. When jobs or transformations call other jobs or transformations, that relationship is also captured. over the course of a transformation or job. Basically we keep track of all fields as they are created, split, aggregated, transformed, etc.

Data lineage is an oft-overloaded term, but for the purposes of this blog I will be talking about the flow of data from external sources into steps/entries and possibly out to other external targets. : $.In Pentaho Data Integration 6.0, we released a great new capability to collect data lineage for PDI transformations and jobs. #!/bin/bash # based on set -e if then # checking if env vars get passed down # echo '$KETTLE_HOME/' # echo '$PENTAHO_HOME' if then # Set variables to defaults if they are not already set

If you are in a rush and just want to quickly start up some Docker containers, use aloysius-lim’s image: The second one offers a few more options than the first one.Īs a learning experience, we will create one ourselves, basically by picking the best bits from the above Dockerfiles and trying to add a few improvements.

To set up several instances, or in other words, to create our application, use docker-compose, which will allow you to spin up several containers based on that one image.Īs we do not want to reinvent the wheel, we perform a search on Google and see what’s already out there: There are at least two good examples out there on how to build Docker images for Pentaho Data Integration:.While this article is not intended to be an introduction to Docker, we will be looking at creating a Dockerfile for Pentaho Data Integration from scratch, which should help you to learn the essentials. It is an invaluable tool to automate the setup and configuration of a given environment. Over the last few years Docker has enjoyed an enormous surge in popularity.

Pentaho data integration variables