sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Kemper <mar...@cloudera.com>
Subject Re: 回复: How to limit the number of rows to export data when using sqoopto export data from hdfs to oracle?
Date Fri, 01 Dec 2017 13:44:17 GMT
Hey Attila,

See https://issues.apache.org/jira/browse/SQOOP-3027

Great to hear from you!


Markus Kemper
Customer Operations Engineer
[image: www.cloudera.com] <http://www.cloudera.com>


On Fri, Dec 1, 2017 at 8:26 AM, Attila Szabó <maugli@apache.org> wrote:

> Hey Markus!
>
> Many thanks for confirming my hcatalog concerns!
>
> Always the heavy super users got the solution, not the devs ;-)
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_-6988492243282966611_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Fri, Dec 1, 2017 at 2:24 PM, Markus Kemper <markus@cloudera.com> wrote:
>
>> Hey Team,
>>
>> Not sure a VIEW with export will work.  If I recall export with
>> --hcatalog is not aware of HMS VIEWs, need to test.
>>
>>
>> Markus Kemper
>> Customer Operations Engineer
>> [image: www.cloudera.com] <http://www.cloudera.com>
>>
>>
>> On Fri, Dec 1, 2017 at 8:00 AM, Attila Szabó <maugli@apache.org> wrote:
>>
>>> Hey,
>>>
>>> IMHO it's CTAS which stands for "Create Table As Select".
>>>
>>> On the front of views :
>>> The question is more than valid if Sqoop supports it or not. I do
>>> remember a problem we've faced 1-1.5 year ago in connection with Hive +
>>> Hcatalog + Sqoop,  and that was not supported because of some missing Hive
>>> Serde implementation. I'm not sure if this problem exists with standard
>>> Hive views and Sqoop export command, but you should give a try.
>>>
>>> The thing what could be problematic with views:
>>> A View is a result of a select statement by design. But in Hive every
>>> HiveQL command is translated to a map/reduce job. But Sqoop also works with
>>> map/reduce jobs and thus it tries to read files from the HDFS ( because of
>>> data locality and things), so this might be a clashing problem here, but as
>>> I've advised you should give it a try.
>>>
>>> With CTAS:
>>> It should definitely work, because in this case Hive will store the
>>> filtered results in a different Hive table ( == HDFS directory)  and thus
>>> export dir is your friend. :)
>>>
>>> Were I able to clarify everything or do you have further questions?
>>>
>>> Cheers,
>>> Attila
>>>
>>> On Dec 1, 2017 1:33 PM, "qq" <987626311@qq.com> wrote:
>>>
>>>> Hello:
>>>>       First of all, thank you very much for your answer, I just started
>>>> to touch sqoop, there are many do not understand, you can explain in detail
>>>> about the operation steps of sqoop export work with views and the steps of
>>>> exact dataset to sqoop using a CAST?
>>>>      thinks.
>>>>      I am looking forward to your reply!
>>>>
>>>>
>>>> ------------------ 原始邮件 ------------------
>>>> *发件人:* "Per Ullberg";<per.ullberg@klarna.com>;
>>>> *发送时间:* 2017年12月1日(星期五) 晚上6:21
>>>> *收件人:* "user"<user@sqoop.apache.org>;
>>>> *主题:* Re: How to limit the number of rows to export data when using
>>>> sqoopto export data from hdfs to oracle?
>>>>
>>>> Does Sqoop export work with Views? If not, you'll have to materialise
>>>> the exact dataset you want to sqoop using a CTAS.
>>>>
>>>> regards
>>>> /Pelle
>>>>
>>>> On Fri, Dec 1, 2017 at 11:08 AM, Attila Szabó <maugli@apache.org>
>>>> wrote:
>>>>
>>>>> Hey,
>>>>>
>>>>> If you're trying to export from Hive into RDBMS I would suggest
>>>>> creating a Hive view and only export the content of the view. Thus you
>>>>> could directly control the data quantity by the underlying HiveQL query.
>>>>>
>>>>> My 2cents,
>>>>> Attila
>>>>>
>>>>>
>>>>> On Dec 1, 2017 10:54 AM, "qq" <987626311@qq.com> wrote:
>>>>>
>>>>> Hello:
>>>>>
>>>>>       I have a question on the export of sqoop need your help, the
>>>>> problem is as follows:
>>>>>       How to limit the number of rows that need to be exported when
>>>>> exporting data from hdfs to oracle using sqoop?
>>>>>       For example: hive data stored in the hdfs 100 lines, just want
>>>>> the first 10 lines of data through sqoop exported to the oracle table,
how
>>>>> to achieve through sqoop?
>>>>>       I am looking forward to your reply!
>>>>>       thinks.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *Per Ullberg*
>>>> Datavault Tech Lead
>>>> Odin (Uppsala)
>>>>
>>>> Klarna Bank AB (publ)
>>>> Sveavägen 46, 111 34 Stockholm
>>>> <https://maps.google.com/?q=Sveav%C3%A4gen+46,+111+34+Stockholm&entry=gmail&source=g>
>>>> Tel: +46 8 120 120 00
>>>> Reg no: 556737-0431
>>>> klarna.com
>>>>
>>>>
>>
>

Mime
View raw message