sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Attila Szabó <mau...@apache.org>
Subject Re: 回复: How to limit the number of rows to export data when using sqoopto export data from hdfs to oracle?
Date Fri, 01 Dec 2017 13:47:01 GMT
Haha,

Shame on me. It's been committed by me, and I do not remember! :-)

Thank you Markus once more!

BTW:
What would you advise in this case? Would you go into the CTAS direction,
or do you have some other best practice for limiting the output?

Cheers,
Attila

On Fri, Dec 1, 2017 at 2:44 PM, Markus Kemper <markus@cloudera.com> wrote:

> Hey Attila,
>
> See https://issues.apache.org/jira/browse/SQOOP-3027
>
> Great to hear from you!
>
>
> Markus Kemper
> Customer Operations Engineer
> [image: www.cloudera.com] <http://www.cloudera.com>
>
>
> On Fri, Dec 1, 2017 at 8:26 AM, Attila Szabó <maugli@apache.org> wrote:
>
>> Hey Markus!
>>
>> Many thanks for confirming my hcatalog concerns!
>>
>> Always the heavy super users got the solution, not the devs ;-)
>>
>>
>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
>> www.avg.com
>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>> <#m_8642581050548475250_m_-6988492243282966611_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>> On Fri, Dec 1, 2017 at 2:24 PM, Markus Kemper <markus@cloudera.com>
>> wrote:
>>
>>> Hey Team,
>>>
>>> Not sure a VIEW with export will work.  If I recall export with
>>> --hcatalog is not aware of HMS VIEWs, need to test.
>>>
>>>
>>> Markus Kemper
>>> Customer Operations Engineer
>>> [image: www.cloudera.com] <http://www.cloudera.com>
>>>
>>>
>>> On Fri, Dec 1, 2017 at 8:00 AM, Attila Szabó <maugli@apache.org> wrote:
>>>
>>>> Hey,
>>>>
>>>> IMHO it's CTAS which stands for "Create Table As Select".
>>>>
>>>> On the front of views :
>>>> The question is more than valid if Sqoop supports it or not. I do
>>>> remember a problem we've faced 1-1.5 year ago in connection with Hive +
>>>> Hcatalog + Sqoop,  and that was not supported because of some missing Hive
>>>> Serde implementation. I'm not sure if this problem exists with standard
>>>> Hive views and Sqoop export command, but you should give a try.
>>>>
>>>> The thing what could be problematic with views:
>>>> A View is a result of a select statement by design. But in Hive every
>>>> HiveQL command is translated to a map/reduce job. But Sqoop also works with
>>>> map/reduce jobs and thus it tries to read files from the HDFS ( because of
>>>> data locality and things), so this might be a clashing problem here, but
as
>>>> I've advised you should give it a try.
>>>>
>>>> With CTAS:
>>>> It should definitely work, because in this case Hive will store the
>>>> filtered results in a different Hive table ( == HDFS directory)  and thus
>>>> export dir is your friend. :)
>>>>
>>>> Were I able to clarify everything or do you have further questions?
>>>>
>>>> Cheers,
>>>> Attila
>>>>
>>>> On Dec 1, 2017 1:33 PM, "qq" <987626311@qq.com> wrote:
>>>>
>>>>> Hello:
>>>>>       First of all, thank you very much for your answer, I just
>>>>> started to touch sqoop, there are many do not understand, you can explain
>>>>> in detail about the operation steps of sqoop export work with views and
the
>>>>> steps of exact dataset to sqoop using a CAST?
>>>>>      thinks.
>>>>>      I am looking forward to your reply!
>>>>>
>>>>>
>>>>> ------------------ 原始邮件 ------------------
>>>>> *发件人:* "Per Ullberg";<per.ullberg@klarna.com>;
>>>>> *发送时间:* 2017年12月1日(星期五) 晚上6:21
>>>>> *收件人:* "user"<user@sqoop.apache.org>;
>>>>> *主题:* Re: How to limit the number of rows to export data when using
>>>>> sqoopto export data from hdfs to oracle?
>>>>>
>>>>> Does Sqoop export work with Views? If not, you'll have to materialise
>>>>> the exact dataset you want to sqoop using a CTAS.
>>>>>
>>>>> regards
>>>>> /Pelle
>>>>>
>>>>> On Fri, Dec 1, 2017 at 11:08 AM, Attila Szabó <maugli@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hey,
>>>>>>
>>>>>> If you're trying to export from Hive into RDBMS I would suggest
>>>>>> creating a Hive view and only export the content of the view. Thus
you
>>>>>> could directly control the data quantity by the underlying HiveQL
query.
>>>>>>
>>>>>> My 2cents,
>>>>>> Attila
>>>>>>
>>>>>>
>>>>>> On Dec 1, 2017 10:54 AM, "qq" <987626311@qq.com> wrote:
>>>>>>
>>>>>> Hello:
>>>>>>
>>>>>>       I have a question on the export of sqoop need your help, the
>>>>>> problem is as follows:
>>>>>>       How to limit the number of rows that need to be exported when
>>>>>> exporting data from hdfs to oracle using sqoop?
>>>>>>       For example: hive data stored in the hdfs 100 lines, just want
>>>>>> the first 10 lines of data through sqoop exported to the oracle table,
how
>>>>>> to achieve through sqoop?
>>>>>>       I am looking forward to your reply!
>>>>>>       thinks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *Per Ullberg*
>>>>> Datavault Tech Lead
>>>>> Odin (Uppsala)
>>>>>
>>>>> Klarna Bank AB (publ)
>>>>> Sveavägen 46, 111 34 Stockholm
>>>>> <https://maps.google.com/?q=Sveav%C3%A4gen+46,+111+34+Stockholm&entry=gmail&source=g>
>>>>> Tel: +46 8 120 120 00
>>>>> Reg no: 556737-0431
>>>>> klarna.com
>>>>>
>>>>>
>>>
>>
>

Mime
View raw message