sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suraj Nayak <snay...@gmail.com>
Subject Re: Binary data transfer using Sqoop
Date Thu, 16 Jul 2015 09:51:27 GMT
Hi Abe,

Thanks for your input. With --inline-lob-limit 0, am able to get the data
as BLOB in external store
*_lob/large_obj_task_1436909150341_0086_m_0000000.lob*. The record
contain *externalLob(lf,_lob/large_obj_task_1436909150341_0086_m_0000000.lob,284,18)
*as data in the column. How can I read this?

Also, is there a way to get the actual data instead of this kind of
external reference? The size of the encrypted column is very small. I think
should not be more than 128 bytes.

Thanks
Suraj

On Wed, Jul 15, 2015 at 10:02 PM, Abraham Elmahrek <abe@cloudera.com> wrote:

> http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_large_objects
>
> use --inline-lob-limit 0.
>
> -Abe
>
> On Wed, Jul 15, 2015 at 9:22 PM, Suraj Nayak <snayakm@gmail.com> wrote:
>
>> Hi All,
>>
>> I tried --map-column-java com.cloudera.sqoop.lib.BlobRef by using
>> --query "select CAST(email as BLOB) from table_name where \$CONDITIONS",
>> and --map-column-java org.apache.hadoop.io.BytesWritable. Still the same
>> issue.
>>
>>
>> Sqoop Output: 20 16 3a 34 fa 40 f6 fc 24 3d 5f b9 c6 63 b5 5f 54 77
>> Required output : [image: Inline image 1]
>>
>>
>>
>>
>>
>> On Wed, Jul 15, 2015 at 9:04 PM, Suraj Nayak <snayakm@gmail.com> wrote:
>>
>>> One correction in Sqoop version. Its 1.4.4 not 1.4.5.
>>>
>>> On Wed, Jul 15, 2015 at 8:56 PM, Suraj Nayak <snayakm@gmail.com> wrote:
>>>
>>>> Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
>>>>
>>>> On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <snayakm@gmail.com> wrote:
>>>>
>>>>> Hi Abe,
>>>>>
>>>>> I think the 1st option suits my use case best. But, what is the
>>>>> method/way to tell Sqoop to use Large Object Loading?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <abe@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Hey man,
>>>>>>
>>>>>> I think Sqoop does this intentionally:
>>>>>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>>>>>> .
>>>>>>
>>>>>> I see two things that may get you beyond this point:
>>>>>>
>>>>>>    1. You can try type casting to a BLOB and tell Sqoop to use Large
>>>>>>    Object Loading.
>>>>>>    2. Try the --as-avrodatafile option and write as Avro.
>>>>>>
>>>>>> -Abe
>>>>>>
>>>>>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <snayakm@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Abe,
>>>>>>>
>>>>>>> Thanks for highlighting missing required info quickly. Below
are the
>>>>>>> details:
>>>>>>>
>>>>>>>    - *Version:* Sqoop 1.4.5
>>>>>>>    - *Sqoop Command: *sqoop import --connect
>>>>>>>    jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username
sqoop_usr
>>>>>>>    --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by
\\001 -m
>>>>>>>    1 --target-dir /tmp/ENC_TBL --connection-manager
>>>>>>>    "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>>>>>>    com.teradata.jdbc.TeraDriver
>>>>>>>    - *Table structure:* id:varchar, count:int, first_name:binary,
>>>>>>>    email:binary, column5:varchar.
>>>>>>>
>>>>>>> Binary is used as the data is encrypted.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <abe@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey man,
>>>>>>>>
>>>>>>>> Need some details to help:
>>>>>>>>
>>>>>>>>    - What version of Sqoop?
>>>>>>>>    - Sqoop command?
>>>>>>>>    - Database table structure (preferably a describe on the
>>>>>>>> database)
>>>>>>>>
>>>>>>>> -Abe
>>>>>>>>
>>>>>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <snayakm@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> > Hi Sqoop Users and Developers,
>>>>>>>> >
>>>>>>>> > How can i import a Binary data column in a table into
HDFS without
>>>>>>>> > converting it into String.
>>>>>>>> >
>>>>>>>> > I have encrypted data in RDBMS, I need to import this
column as
>>>>>>>> is without
>>>>>>>> > converting it into string. As of now, Sqoop is typecasting
the
>>>>>>>> data into
>>>>>>>> > String/text and decryption is failing in Hadoop.
>>>>>>>> >
>>>>>>>> > Can someone provide pointers to solve this? Any workaround?
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > Thanks
>>>>>>>> > Suraj Nayak M
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks
>>>>>>> Suraj Nayak M
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks
>>>>> Suraj Nayak M
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Suraj Nayak M
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks
>>> Suraj Nayak M
>>>
>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>


-- 
Thanks
Suraj Nayak M

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message