spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reminia Scarlet <reminia.scar...@gmail.com>
Subject SparkStreming logical plan leaf nodes is not equal pysical plan leaf nodes and streaming metrics cannot be reported.
Date Wed, 23 Oct 2019 12:57:04 GMT
Hi all:
 I use StreamingQueryListener to report batch inputRecordsNum as metrics.
 But the numInputRows is aways 0. And the debug log  in
MicroBatchExecution.scala said:

 2019-10-23 06:56:05 WARN  MicroBatchExecution:66 - Could not report
metrics as number leaves in trigger logical plan did not match that of
the execution plan:

 And this causes num input rows by sources always 0 from below codes
in ProgressReporter.scala when number of leaves size not matches in
logical plan and execution plan.

[image: image.png]
Attached the output logical plan && physical plan leaves. I think
there might be some bugs. Seems LogicalRDD is duplicate as Relation in
the logical plan.
And counting twice as leaf.If we remove the LogcialRDD, leave size
should be the same.

[image: image.png]
[image: image.png]

Can anyone help? Thx very much.

Mime
View raw message