spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject [KafkaSourceProvider] Why topic option and column without reverting to path as the least priority?
Date Mon, 01 May 2017 13:52:23 GMT
Hi,

I've just found out that KafkaSourceProvider supports topic option
that sets the Kafka topic to save a DataFrame to.

You can also use topic column to assign rows to topics.

Given the features, I've been wondering why "path" option is not
supported (even of least precedence) so when no topic column or option
are defined, save(path: String) would be the least priority.

WDYT?

It looks pretty trivial to support --> see KafkaSourceProvider at
lines [1] and [2] if I'm not mistaken.

[1] https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L145
[2] https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L163

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message