KRkrawczstef·Aug 31, 2022
Yes! As one of the creators of https://github.com/stitchfix/hamilton this was one of the aims. Simplifying the cognitive burden for those developing and managing data transforms over the course of years, and in particular for ones they didn't write!
For example in Hamilton -- we force people to write "declarative functions" which then are stitched together to create a dataflow.
E.g. example function -- my guess is that you can read and understand/guess what it does very easily.
# in client_features.py
@tag(owner='Data-Science', pii='False')
@check_output(data_type=np.float64, range=(-5.0, 5.0), allow_nans=False)
def height_zero_mean_unit_variance(height_zero_mean: pd.Series,
height_std_dev: pd.Series) -> pd.Series:
"""Zero mean unit variance value of height"""
return height_zero_mean / height_std
Read the rest
_dev
To use `height_zero_mean_unit_variance` somewhere else, you'd then find it as a parameter to some other function, and that's the basic gist of how you'd develop a dataflow. To then execute the dataflow, you'd then in a separate file, decoupled from your transform logic, write some code to (1) create the DAG, and then (2) request what you want executed from it.
In terms of reducing cognitive burden for maintainers, by forcing things into a function there's all these nice properties to be had for maintenance:
- unit testing is always possible
- integration testing is super easy, since you can add code, and then _only_ test that path without having to run your entire transform workflow.
- documentation has a natural place & and we can build a DAG visualizing the dataflow that you're looking at
- debugging is methodical since it's straightforward to map an output to a function, check the logic there, and then methodically traverse its dependencies because they are declared.
- you can wrap functions/inject things at run time easily