From dev-return-16742-apmail-nifi-dev-archive=nifi.apache.org@nifi.apache.org Thu Mar 22 07:38:51 2018 Return-Path: X-Original-To: apmail-nifi-dev-archive@minotaur.apache.org Delivered-To: apmail-nifi-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BFF6417D69 for ; Thu, 22 Mar 2018 07:38:51 +0000 (UTC) Received: (qmail 53776 invoked by uid 500); 22 Mar 2018 07:38:51 -0000 Delivered-To: apmail-nifi-dev-archive@nifi.apache.org Received: (qmail 53711 invoked by uid 500); 22 Mar 2018 07:38:51 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 53689 invoked by uid 99); 22 Mar 2018 07:38:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Mar 2018 07:38:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id A345E180917 for ; Thu, 22 Mar 2018 07:38:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id s15U12czSYUO for ; Thu, 22 Mar 2018 07:38:45 +0000 (UTC) Received: from mail-ot0-f171.google.com (mail-ot0-f171.google.com [74.125.82.171]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 0CB2A5F2A9 for ; Thu, 22 Mar 2018 07:38:45 +0000 (UTC) Received: by mail-ot0-f171.google.com with SMTP id 108-v6so8449933otv.3 for ; Thu, 22 Mar 2018 00:38:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=pLCWXCWSCzyEmxQYuZXf+T7GW/a7V2IxZD5JAqJedos=; b=DqLLfiFEAoIbweWfaKTSswNT/gPhEJ2YG2O+FLiJb59qQePLSUObmpdJTjbzzOcNTe dOg3RYqXUAta5GFGgJ60u6DQEuBduTCFAstpac2VSmpPe6y9poUuGiUr5VkCwi/uihJL kGIIg6rFYZMFmHQ9cNN+N2d8eGqtkMLZiYCMfof+PvaUFcubUKAlfwSTP+lB2n80IZ3r iNMitEccJ9Ug1LW1izkXQOMtQbHySaIsuaDagysDtSMwf2abyK3VjDEUclG8d9eyYT79 hWfgp9mEn5Jd80t+8InXwi/XAlAbmsB59YBsPub9u11asGNertL6bP0OolrsxicuoVBO Yl0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=pLCWXCWSCzyEmxQYuZXf+T7GW/a7V2IxZD5JAqJedos=; b=iHss/pdP87427YvzdkPrwHaKxti9rfhWm1dRqaRqPtnTPBZuOLsIfWR+Yf0me360AG eMoeTjlwwOLGSRdAyAArdmW8wq54O3D8FX/hTetJ1VUvL9kHJbM5ljkOgWz20rRT2iZI weccWGIHRWfTs6hozlkQTILSkhRl6IO5a/CumkcSsWVS9n4zgwB4vt2F0NkE08o8teSm yGOX4H/6eYsJpKO3E4Kd7getkZ3hno4VkvXkvlKxXOnetrX8IEZHTSfazDFULeB8ULhw VQ9Ie65E9mv/dpyrGaZIiKwwTWmYfYb0Tb4/czETBRa6wUkzaKEA8/kxrQXger8FCvm/ oVTg== X-Gm-Message-State: AElRT7FQu7dcifLQWDwLqcskDrlONd1DbwQjBgPM6nqtxF8eWf1kwQHi jwWDcQZ8ssqQd3ndUS5NGUZqQNd3n7GYAF3V/8or7Q== X-Google-Smtp-Source: AIpwx4/yBPX/qHV8W8y4xlho+88bYQXqJwQmO06fyRMeEefY4nxwzSPLcIT6jho7h09zHpVDRt3fB6xILObAZ+ll/bA= X-Received: by 2002:a9d:d88:: with SMTP id 8-v6mr6983807ots.297.1521704323579; Thu, 22 Mar 2018 00:38:43 -0700 (PDT) MIME-Version: 1.0 References: <01BF34C7-FF35-4780-A9BD-1B724DD2DADC@gmail.com> <148D0081-D5BB-4EB5-9D77-29E88309F9EB@gmail.com> In-Reply-To: From: Brett Ryan Date: Thu, 22 Mar 2018 07:38:33 +0000 Message-ID: Subject: Re: Stackoverflow question: Moving data from one RDB to another through NiFi To: dev@nifi.apache.org Content-Type: multipart/alternative; boundary="000000000000a903470567fb651f" --000000000000a903470567fb651f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hmmm, now I=E2=80=99m doubting myself. It=E2=80=99s possible we sqoop to hd= fs then sqoop out, will have to look, sorry if I am wrong. On Thu, 22 Mar 2018 at 18:27, Jorge Machado wrote: > Hi Bryan Sure ? Database to Database ? Or with a step in between ? Can > you past the command that you use ? That would be new to me and I would b= e > interested. > > Jorge Machado > > > > > > > On 22 Mar 2018, at 08:22, Brett Ryan wrote: > > > > Sure it does, I=E2=80=99m using it for postgres and MariaDB (which is > essentially MySQL). > > > >> On 22 Mar 2018, at 18:18, Jorge Machado wrote: > >> > >> Sqoop does not import into a mySql database. Just into Hive if you tel= l > him to do so. > >> You could use Nifi but if you have a lot of data may be you should try > Spark. which reads and writes in Parallel. > >> Using Nifi would work to but you have the overhead of pumping the data > over =E2=80=9Cinsert=E2=80=9D unless you copy the files into the server a= nd on the server > then use some import bulk=E2=80=A6.. > >> > >> Jorge Machado > >> > >> > >> > >> > >> > >>> On 22 Mar 2018, at 08:13, Brett Ryan wrote: > >>> > >>> Could Sqoop [1] be an option? > >>> > >>> [1]: http://sqoop.apache.org/ > >>> > >>>> On 22 Mar 2018, at 16:33, Sivaprasanna > wrote: > >>>> > >>>> I had a chance to attempt a question raised on stackoverflow regardi= ng > >>>> moving data from SQL Server to MySQL using NiFi. The user is using > >>>> GenerateTableFetch to read data from SQL Server and then try to use > LOAD > >>>> DATA command in ExecuteSQL but this involves writing the read SQL > Server > >>>> data to filesystem and then load it, which is a performance hit, I > >>>> suggested the user to try PutDatabaseRecord but I have never tried t= he > >>>> approach myself and going by the docs, I think it won't show any > >>>> performance benefit than LOAD DATA because the former reads from fil= e > and > >>>> inserts at a high speed while the latter reads content and parses it > >>>> according to the configured Record Reader and insert the rows as a > single > >>>> batch. Confused, I wanted to get the community's opinion/thoughts on > this. > >>>> Please attempt the questions, if you have better suggestions. > >>>> > >>>> Links: > >>>> > >>>> - > >>>> > https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-in= to-mysql-apache-nifi?noredirect=3D1#comment85843021_49400447 > >>>> - > >>>> > https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/= 49398500?noredirect=3D1#comment85805848_49398500 > >>>> > >>>> Thanks, > >>>> > >>>> Sivaprasanna > >> > > --000000000000a903470567fb651f--