spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <>
Subject [SS] How to create a streaming DataFrame (for a custom Source in Spark 2.4.4 / MicroBatch / DSv1)?
Date Tue, 01 Oct 2019 18:31:55 GMT

I think I've got stuck and without your help I won't move any further.
Please help.

I'm with Spark 2.4.4 and am developing a streaming Source (DSv1,
MicroBatch) and in getBatch phase when requested for a DataFrame, there is
this assert [1] I can't seem to go past with any DataFrame I managed to
create as it's not streaming.

            s"DataFrame returned by getBatch from $source did not have
isStreaming=true\n" +


All I could find is private[sql],
e.g. SQLContext.internalCreateDataFrame(..., isStreaming = true) [2] or [3]


Jacek Laskowski
The Internals of Spark SQL
The Internals of Spark Structured Streaming
The Internals of Apache Kafka
Follow me at

View raw message