sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "richard zhou" <richard.z...@intel.com>
Subject Re: Review Request 35591: Sqoop-1853: Kafka connector supporting FROM direction
Date Fri, 10 Jul 2015 08:58:41 GMT


> On June 19, 2015, 12:37 a.m., Abraham Elmahrek wrote:
> > connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaExtractor.java,
line 71
> > <https://reviews.apache.org/r/35591/diff/1/?file=986676#file986676line71>
> >
> >     Schemaless is assumed here. Maybe the user could provie a schema? If not, maybe
a follow up Jira?

This patch is for CSV (schemaless) format. The following JIRA will handle schema and CDC.


> On June 19, 2015, 12:37 a.m., Abraham Elmahrek wrote:
> > connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaPartitioner.java,
line 42
> > <https://reviews.apache.org/r/35591/diff/1/?file=986681#file986681line42>
> >
> >     Topic not being set. Also, I think you can partition based on the number of
partitions in a kafka topic?

Actually, I guess topic should be the job_from_config. Like the import data is from Kafka,
which topic is "report", instead of import all data from Kafka and use topic as partition.
The reason is that for the Kafka connector (To Side), all data is exported to one topic only.
What do you think?


- richard


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35591/#review88473
-----------------------------------------------------------


On July 10, 2015, 8:52 a.m., richard zhou wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35591/
> -----------------------------------------------------------
> 
> (Updated July 10, 2015, 8:52 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> This is an initial patch. It may not work currently. Please check whether this design
of framework is fine or not.
> 
> 
> Diffs
> -----
> 
>   connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConnector.java
e3eafe87f7376031d79fec76739fd9e648df872b 
>   connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaExtractor.java
PRE-CREATION 
>   connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaFromDestroyer.java
PRE-CREATION 
>   connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaFromInitializer.java
PRE-CREATION 
>   connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaLoader.java
ad9ec34755e36985078e880a0a076c46337ef411 
>   connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaPartition.java
PRE-CREATION 
>   connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaPartitioner.java
PRE-CREATION 
>   connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaUtils.java
PRE-CREATION 
>   connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/FromJobConfig.java
PRE-CREATION 
>   connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/FromJobConfiguration.java
PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/35591/diff/
> 
> 
> Testing
> -------
> 
> Passed compile, but it still needs to be tested in the real cluster.
> 
> 
> Thanks,
> 
> richard zhou
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message