sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brenden Cobb <Brenden.C...@humedica.com>
Subject Re: Export only newly inserted values
Date Thu, 17 Jul 2014 17:39:15 GMT
Hey- sorry I misunderstood your question.

I don't think you're going to find an out-of-the-box solution for doing this with Sqoop. My
suggestion is you'll need to schedule some ETL work in Map reduce, Pig, or Hive in order to
separate out only the records your want to export, then call Sqoop to move the new file. I
have a similar situation and Sqoop exports are really just a transport mechanism to move static
data out of HDFS. Shaping that data is left to other tools.



From: Muthu Pandi <muthu1086@gmail.com<mailto:muthu1086@gmail.com>>
Reply-To: "user@sqoop.apache.org<mailto:user@sqoop.apache.org>" <user@sqoop.apache.org<mailto:user@sqoop.apache.org>>
Date: Wednesday, July 16, 2014 12:14 AM
To: "user@sqoop.apache.org<mailto:user@sqoop.apache.org>" <user@sqoop.apache.org<mailto:user@sqoop.apache.org>>
Subject: Re: Export only newly inserted values

Yes i can use Oozie for the schedule but while export the whole data will be exported again
and again. let say i had 10000 rows at 4:00 and i do export 10000 data will be exported after
15 Min data will be increased by 10500 so while exporting at 4:15 again 10500 data will be
exported i need to export only the new data that is 500 data.


Regards
Muthupandi.K
9894594112

[Picture (Device Independent Bitmap)]


On Tue, Jul 15, 2014 at 10:06 PM, Brenden Cobb <Brenden.Cobb@humedica.com<mailto:Brenden.Cobb@humedica.com>>
wrote:
You might want to look into creating an Oozie job to handle your periodic exports.



From: Muthu Pandi <muthu1086@gmail.com<mailto:muthu1086@gmail.com>>
Reply-To: "user@sqoop.apache.org<mailto:user@sqoop.apache.org>" <user@sqoop.apache.org<mailto:user@sqoop.apache.org>>
Date: Monday, July 14, 2014 12:25 AM
To: "user@sqoop.apache.org<mailto:user@sqoop.apache.org>" <user@sqoop.apache.org<mailto:user@sqoop.apache.org>>
Subject: Re: Export only newly inserted values

Yes but i have to export periodically say 15 mins for the same day


Regards
Muthupandi.K
9894594112

[Picture (Device Independent Bitmap)]


On Sun, Jul 13, 2014 at 6:58 AM, Venkat Ranganathan <vranganathan@hortonworks.com<mailto:vranganathan@hortonworks.com>>
wrote:
If your hive table is partitioned by date (say year/month/day), then you can export data from
a specific partition using hcatalog

Thanks

Venkat


On Thu, Jul 10, 2014 at 11:05 PM, Muthu Pandi <muthu1086@gmail.com<mailto:muthu1086@gmail.com>>
wrote:
Dear All

        Am in an analytic project where all the data are fed into hive warehouse and am in
need of exporting those values to mysql since the analytic on hive is too slow. Data will
be coming to hive in timely fashion so i need to export only the new data not all the data
using sqoop export kindly point me the direction.

Regards
Muthupandi.K

[Picture (Device Independent Bitmap)]


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.



Mime
View raw message