sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kate Ting <k...@cloudera.com>
Subject Re: Map/Reduce being idempotent for export
Date Wed, 30 Nov 2011 04:52:48 GMT
Sharad,

Sqoop handles this by the following arguments:

--staging-table <staging-table-name>
The table in which data will be staged before being inserted into the
destination table.

--clear-staging-table
Indicates that any data present in the staging table can be deleted.

For more information:
http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html#_syntax_3

Note that only some connectors support staging tables.

Regards, Kate

On Tue, Nov 29, 2011 at 8:29 PM, Sharad Agarwal <sharad.apache@gmail.com>wrote:

> How do we ensure that map/reduce tasks are idempotent. To be more clear: if
> there are more than 1 map/reduce attempts for an export job, how do we make
> sure that data from only single task attempt is committed ?
>
> Normally this is handled by OutputCommitter, however I see
> NullOutputCommitter which doesn't do anything.
>
> Thanks
> Sharad
>

Mime
View raw message