Parameters Metadata for Data Engineering
A data engineering design pattern
One Cool Thing About Design Patterns
Design patterns help enterprises improve quality while reducing development time (and costs because time is money). Starting the development phase of solution delivery with a collection of tried-and-true design patterns, familiar to the developer(s) or team, drips with efficiency.
The Value of Parameters (get it?)
Parameters are useful for managing application and execution values at enterprise, solution, execution, and application scopes. Data engineering parameters promote code reuse because a tested and proven pipeline, script, notebook, or package - when parameterized - may be used to perform similar orchestration or data loads.
Other pro’s of parameterization? Fewer data engineering artifacts means:
Lower development cost
Fewer points of failure
Higher confidence
Plus, parameters may be updated without recompiling code.
The con’s of parameterization?
With great flexibility comes increased complexity.
