spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: [KafkaSourceProvider] Why topic option and column without reverting to path as the least priority?
Date Mon, 01 May 2017 15:43:59 GMT
I'm confused about what you're suggesting.  Are you saying that a
Kafka sink should take a filesystem path as an option?

On Mon, May 1, 2017 at 8:52 AM, Jacek Laskowski <jacek@japila.pl> wrote:
> Hi,
>
> I've just found out that KafkaSourceProvider supports topic option
> that sets the Kafka topic to save a DataFrame to.
>
> You can also use topic column to assign rows to topics.
>
> Given the features, I've been wondering why "path" option is not
> supported (even of least precedence) so when no topic column or option
> are defined, save(path: String) would be the least priority.
>
> WDYT?
>
> It looks pretty trivial to support --> see KafkaSourceProvider at
> lines [1] and [2] if I'm not mistaken.
>
> [1] https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L145
> [2] https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L163
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message