spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chandan prakash <>
Subject structured streaming: how to keep counter of error records in log running streaming application
Date Thu, 05 Jul 2018 06:19:20 GMT
I am writing a structured streaming application, where I process records
post some validation (lets say , not null).
Want to keep a counter of invalid records in the long running streaming
application while other valid records get processed.
How can I achieve it ?

First thought was using LongAccumulator but it seems like it is per batch
and not for the life of the application.

Any other way or workaround to achieve this, please share.
Thanks in advance.

Chandan Prakash

View raw message