nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jorge Machado <jom...@me.com>
Subject Re: Stackoverflow question: Moving data from one RDB to another through NiFi
Date Thu, 22 Mar 2018 08:28:12 GMT
That would be probably the fastest way. 

Jorge Machado
www.jmachado.me





> On 22 Mar 2018, at 08:38, Brett Ryan <brett.ryan@gmail.com> wrote:
> 
> Hmmm, now I’m doubting myself. It’s possible we sqoop to hdfs then sqoop
> out, will have to look, sorry if I am wrong.
> 
> On Thu, 22 Mar 2018 at 18:27, Jorge Machado <jomach@me.com> wrote:
> 
>> Hi Bryan Sure ? Database to Database ? Or  with a step in between ? Can
>> you past the command that you use ? That would be new to me and I would be
>> interested.
>> 
>> Jorge Machado
>> 
>> 
>> 
>> 
>> 
>>> On 22 Mar 2018, at 08:22, Brett Ryan <brett.ryan@gmail.com> wrote:
>>> 
>>> Sure it does, I’m using it for postgres and MariaDB (which is
>> essentially MySQL).
>>> 
>>>> On 22 Mar 2018, at 18:18, Jorge Machado <jomach@me.com> wrote:
>>>> 
>>>> Sqoop does not import into a mySql database. Just into Hive if you tell
>> him to do so.
>>>> You could use Nifi but if you have a lot of data may be you should try
>> Spark.  which reads and writes in Parallel.
>>>> Using Nifi would work to but you have the overhead of pumping the data
>> over “insert” unless you copy the files into the server and on the server
>> then use some import bulk…..
>>>> 
>>>> Jorge Machado
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On 22 Mar 2018, at 08:13, Brett Ryan <brett.ryan@gmail.com> wrote:
>>>>> 
>>>>> Could Sqoop [1] be an option?
>>>>> 
>>>>> [1]: http://sqoop.apache.org/
>>>>> 
>>>>>> On 22 Mar 2018, at 16:33, Sivaprasanna <sivaprasanna246@gmail.com>
>> wrote:
>>>>>> 
>>>>>> I had a chance to attempt a question raised on stackoverflow regarding
>>>>>> moving data from SQL Server to MySQL using NiFi. The user is using
>>>>>> GenerateTableFetch to read data from SQL Server and then try to use
>> LOAD
>>>>>> DATA command in ExecuteSQL but this involves writing the read SQL
>> Server
>>>>>> data to filesystem and then load it, which is a performance hit,
I
>>>>>> suggested the user to try PutDatabaseRecord but I have never tried
the
>>>>>> approach myself and going by the docs, I think it won't show any
>>>>>> performance benefit than LOAD DATA because the former reads from
file
>> and
>>>>>> inserts at a high speed while the latter reads content and parses
it
>>>>>> according to the configured Record Reader and insert the rows as
a
>> single
>>>>>> batch. Confused, I wanted to get the community's opinion/thoughts
on
>> this.
>>>>>> Please attempt the questions, if you have better suggestions.
>>>>>> 
>>>>>> Links:
>>>>>> 
>>>>>> -
>>>>>> 
>> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>>>>>> -
>>>>>> 
>> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Sivaprasanna
>>>> 
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message