Do you know that feeling when you had planned your data flow to be nice and tidy and all you got was your favourite dish of pasta: spaghetti code? We know too! Initially it tastes good, but at some point you might want to share your code with another project or there is some infrastructure change (Hey we got a new cluster!). At that point you wish you said "No thanks, no more spaghetti for me!".
I will show how explicit state, essentially pure functions and declarative approaches can help to keep up your good intentions from the beginning. Sticking to these principles not only makes sharing and structuring your code easier, but gives benefits in a lot of areas: From testing to performance optimisation - we observe a reduction of complexity in almost each case where we apply these principles. Finally, I will show how we realise these ideas and patterns in our day to day work at Blue Yonder, leading us to composable and understandable data pipelines from explorative analysis to production systems.
