nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Ryan <brett.r...@gmail.com>
Subject Re: Stackoverflow question: Moving data from one RDB to another through NiFi
Date Thu, 22 Mar 2018 07:38:33 GMT
Hmmm, now I’m doubting myself. It’s possible we sqoop to hdfs then sqoop
out, will have to look, sorry if I am wrong.

On Thu, 22 Mar 2018 at 18:27, Jorge Machado <jomach@me.com> wrote:

> Hi Bryan Sure ? Database to Database ? Or  with a step in between ? Can
> you past the command that you use ? That would be new to me and I would be
> interested.
>
> Jorge Machado
>
>
>
>
>
> > On 22 Mar 2018, at 08:22, Brett Ryan <brett.ryan@gmail.com> wrote:
> >
> > Sure it does, I’m using it for postgres and MariaDB (which is
> essentially MySQL).
> >
> >> On 22 Mar 2018, at 18:18, Jorge Machado <jomach@me.com> wrote:
> >>
> >> Sqoop does not import into a mySql database. Just into Hive if you tell
> him to do so.
> >> You could use Nifi but if you have a lot of data may be you should try
> Spark.  which reads and writes in Parallel.
> >> Using Nifi would work to but you have the overhead of pumping the data
> over “insert” unless you copy the files into the server and on the server
> then use some import bulk…..
> >>
> >> Jorge Machado
> >>
> >>
> >>
> >>
> >>
> >>> On 22 Mar 2018, at 08:13, Brett Ryan <brett.ryan@gmail.com> wrote:
> >>>
> >>> Could Sqoop [1] be an option?
> >>>
> >>> [1]: http://sqoop.apache.org/
> >>>
> >>>> On 22 Mar 2018, at 16:33, Sivaprasanna <sivaprasanna246@gmail.com>
> wrote:
> >>>>
> >>>> I had a chance to attempt a question raised on stackoverflow regarding
> >>>> moving data from SQL Server to MySQL using NiFi. The user is using
> >>>> GenerateTableFetch to read data from SQL Server and then try to use
> LOAD
> >>>> DATA command in ExecuteSQL but this involves writing the read SQL
> Server
> >>>> data to filesystem and then load it, which is a performance hit, I
> >>>> suggested the user to try PutDatabaseRecord but I have never tried the
> >>>> approach myself and going by the docs, I think it won't show any
> >>>> performance benefit than LOAD DATA because the former reads from file
> and
> >>>> inserts at a high speed while the latter reads content and parses it
> >>>> according to the configured Record Reader and insert the rows as a
> single
> >>>> batch. Confused, I wanted to get the community's opinion/thoughts on
> this.
> >>>> Please attempt the questions, if you have better suggestions.
> >>>>
> >>>> Links:
> >>>>
> >>>> -
> >>>>
> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
> >>>> -
> >>>>
> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Sivaprasanna
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message