I think that every client wants a validation process, but showing lineage is a approach that they are not asking, and may not be the right way to prove it.


On Tue, Apr 4, 2017 at 4:19 AM, kant kodali <kanth909@gmail.com> wrote:
Hi All,

I am wondering if there a way to persist the lineages generated by spark underneath? Some of our clients want us to prove if the result of the computation that we are showing on a dashboard is correct and for that If we can show the lineage of transformations that are executed to get to the result then that can be the Q.E.D moment but I am not even sure if this is even possible with spark?