spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshay Bhardwaj <akshay.bhardwaj1...@gmail.com>
Subject Spark 2.3 | Structured Streaming | Metric for numInputRows
Date Wed, 27 Feb 2019 06:06:35 GMT
Hi Experts,

I have a structured streaming query running on spark 2.3 over yarn cluster,
with below features:

   - Reading JSON messages from Kafka topic with:
      - maxOffsetsPerTrigger as 5000
      - trigger interval of my writeStream task is 500ms.
      - streaming dataset is defined as events with  fields: id, name,
      refid, createdTime
   - A cached dataset of CSV file read from HDFS, such that, the CSV file
   contains a list of prohibited events refid
   - I have defined an intermediate dataset with the following query, which
   filters out prohibited events from the streaming data
      - select * from events where event.refid NOT IN (select refid from
      CSVData)


The query progress from StreamingQuery object, it shows metrics as
numInputRows, inputRowsPerSecond and processedRowsPerSecond as 0, although
my query is executing with an execution time of ~400ms. And I can see that
the query does take records from kafka and writes the processed data to the
output database.

If I remove the event filtering tasks, then all the metrics are displayed
properly.

Can anyone please point out why this behaviour is observed and how to
gather metrics like numInputRows, etc while also filtering events fetched
from CSV file?
I am also open to suggestions if there is a better way of filtering out the
prohibited events in structured streaming.

Thanks in advance.

Akshay Bhardwaj
+91-97111-33849

Mime
View raw message