sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Cecho" <jar...@apache.org>
Subject Re: Review Request 14240: SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function
Date Tue, 08 Oct 2013 23:40:54 GMT


> On Oct. 1, 2013, 7:49 p.m., Jarek Cecho wrote:
> > Thank you Alexandre for taking up this ticket!
> > 
> > Would you mind adding automated tests to ensure that the functionality is indeed
working?
> 
> Alexandre Normand wrote:
>     I'd like to be able to integration test bulk load but this is still not a possibility
in non-distributed mode. We currently have the same problems in some of our map/reduce jobs
that use bulk loading. The way we've approached it with those is that we have two modes for
each job: bulk load enabled/bulk load disabled. The integration tests always run with bulk
load disabled. The "bulk load mode" is tested on a real cluster with automated tests that
run from jenkins. 
>     
>     For what it's worth, I've done the manual testing on our cluster for this patch.

Thank you for the feedback Alexandre! Do you happen to know if there is already a JIRA for
adding the bulk load mode into the HBase MiniCluster? I tried to search for now, but didn't
found it.


> On Oct. 1, 2013, 7:49 p.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java, lines 149-152
> > <https://reviews.apache.org/r/14240/diff/5/?file=359511#file359511line149>
> >
> >     This seems to be very dangerous as at some point everyone can read the files,
wouldn't be much secure to simply change the owner to the hbase user?
> 
> Alexandre Normand wrote:
>     It would be more secure but it doesn't work. I just tried and I get:
>     Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
Non-super user cannot change owner
>     
>     The actual alternative would be to use the new approach enabled by https://issues.apache.org/jira/browse/HBASE-5498
and described in https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load
>     
>     But sqoop can't assume this patch is available, right?

Yeah, we can't assume that the HBASE-5498 is available at this point. Fair enough then.


- Jarek


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14240/#review26576
-----------------------------------------------------------


On Oct. 4, 2013, 5:45 p.m., Alexandre Normand wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14240/
> -----------------------------------------------------------
> 
> (Updated Oct. 4, 2013, 5:45 p.m.)
> 
> 
> Review request for Sqoop, Jarek Cecho, Jean-Marc Spaggiari, and Vasanth kumar RJ.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> This is the patch to address some of the comments on Zhancheng Deng's review request
(https://reviews.apache.org/r/13052/) which was basically the change to add the --bulk-load-dir
option to support the HBase doBulkLoad function.
> 
> 
> Diffs
> -----
> 
>   src/java/org/apache/sqoop/SqoopOptions.java 01805f9 
>   src/java/org/apache/sqoop/hbase/HBasePutProcessor.java 9ceb5bd 
>   src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java 5ccf311 
>   src/java/org/apache/sqoop/manager/SqlManager.java 2a4992d 
>   src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/ImportJobBase.java ab7f21e 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java 0eca991 
>   src/test/com/cloudera/sqoop/TestSqoopOptions.java 03e2504 
> 
> Diff: https://reviews.apache.org/r/14240/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Alexandre Normand
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message