nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shashi Vishwakarma <>
Subject Spark Streaming with Nifi
Date Sun, 04 Jun 2017 02:43:33 GMT

I am looking for way where I can make use of spark streaming in Nifi. I see
couple of post where SiteToSite tcp connection is used for spark streaming
application but I thinking it will be good If I can launch Spark streaming
from Nifi custom processor.

PublishKafka will publish message into Kafka followed by Nifi Spark
streaming processor will read from Kafka Topic.

I can launch Spark streaming application from custom Nifi processor using
Spark Streaming launcher API but biggest challenge is that it will create
spark streaming context for each flow file which can be costly operation.

Does any one suggest storing spark streaming context  in controller service
? or any better approach for running spark streaming application with Nifi ?

Thanks and Regards,

View raw message