nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Percivall <joeperciv...@yahoo.com>
Subject Re: Nifi Kite-dataset URI issue
Date Mon, 21 Dec 2015 14:38:21 GMT
Hello Chandra,
Are you still running into issues?
If so, can someone with more experience with Kite and Hadoop chime in?
Joe- - - - - - Joseph Percivalllinkedin.com/in/Percivalle: joepercivall@yahoo.com
 

    On Wednesday, December 16, 2015 2:08 PM, Chandu Koripella <ckoripel@starbucks.com>
wrote:
 

 #yiv9849974592 #yiv9849974592 -- _filtered #yiv9849974592 {font-family:Wingdings;panose-1:5
0 0 0 0 0 0 0 0 0;} _filtered #yiv9849974592 {font-family:Wingdings;panose-1:5 0 0 0 0 0 0
0 0 0;} _filtered #yiv9849974592 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered
#yiv9849974592 {font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;}#yiv9849974592 #yiv9849974592
p.yiv9849974592MsoNormal, #yiv9849974592 li.yiv9849974592MsoNormal, #yiv9849974592 div.yiv9849974592MsoNormal
{margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv9849974592 a:link, #yiv9849974592
span.yiv9849974592MsoHyperlink {color:blue;text-decoration:underline;}#yiv9849974592 a:visited,
#yiv9849974592 span.yiv9849974592MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv9849974592
p {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv9849974592 p.yiv9849974592MsoAcetate,
#yiv9849974592 li.yiv9849974592MsoAcetate, #yiv9849974592 div.yiv9849974592MsoAcetate {margin:0in;margin-bottom:.0001pt;font-size:8.0pt;}#yiv9849974592
p.yiv9849974592MsoListParagraph, #yiv9849974592 li.yiv9849974592MsoListParagraph, #yiv9849974592
div.yiv9849974592MsoListParagraph {margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv9849974592
span.yiv9849974592BalloonTextChar {}#yiv9849974592 span.yiv9849974592EmailStyle20 {color:#1F497D;}#yiv9849974592
span.yiv9849974592EmailStyle21 {color:#1F497D;}#yiv9849974592 .yiv9849974592MsoChpDefault
{font-size:10.0pt;} _filtered #yiv9849974592 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv9849974592
div.yiv9849974592WordSection1 {}#yiv9849974592 _filtered #yiv9849974592 {} _filtered #yiv9849974592
{font-family:Symbol;} _filtered #yiv9849974592 {} _filtered #yiv9849974592 {font-family:Wingdings;}
_filtered #yiv9849974592 {font-family:Symbol;} _filtered #yiv9849974592 {} _filtered #yiv9849974592
{font-family:Wingdings;} _filtered #yiv9849974592 {font-family:Symbol;} _filtered #yiv9849974592
{} _filtered #yiv9849974592 {font-family:Wingdings;} _filtered #yiv9849974592 {} _filtered
#yiv9849974592 {font-family:Symbol;} _filtered #yiv9849974592 {} _filtered #yiv9849974592
{font-family:Wingdings;} _filtered #yiv9849974592 {font-family:Symbol;} _filtered #yiv9849974592
{} _filtered #yiv9849974592 {font-family:Wingdings;} _filtered #yiv9849974592 {font-family:Symbol;}
_filtered #yiv9849974592 {} _filtered #yiv9849974592 {font-family:Wingdings;} _filtered #yiv9849974592
{} _filtered #yiv9849974592 {font-family:Symbol;} _filtered #yiv9849974592 {} _filtered #yiv9849974592
{font-family:Wingdings;} _filtered #yiv9849974592 {font-family:Symbol;} _filtered #yiv9849974592
{} _filtered #yiv9849974592 {font-family:Wingdings;} _filtered #yiv9849974592 {font-family:Symbol;}
_filtered #yiv9849974592 {} _filtered #yiv9849974592 {font-family:Wingdings;}#yiv9849974592
ol {margin-bottom:0in;}#yiv9849974592 ul {margin-bottom:0in;}#yiv9849974592 HI,    I am running
into another issue. I really appreciate your help. My apology if I am asking very basic question.
I am first time user for both Nifi, Kite dataset and Hadoop    My Nifi is service is running
as HDFS user.    ·       Create a dataset as HDFS and I could able to import just
fine as HDFS        ·       Permissions:           ·       URI Validates
just fine in NIfi          ·       Hadoop Configuration files: /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-site.xml
   Any help is appreciated.    Thanks, Chandra    From: Chandu Koripella [mailto:ckoripel@starbucks.com]
Sent: Monday, December 14, 2015 11:49 AM
To: users@nifi.apache.org
Subject: RE: Nifi Kite-dataset URI issue    You are right. It works just fine by changing
the keyword to hdfs. Kit-dataset accepting the both hdfs and hive           Thanks, Chandu
      From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Monday, December 14, 2015 11:37 AM
To: users@nifi.apache.org
Subject: Re: Nifi Kite-dataset URI issue    the property to focus on is 'Target Dataset URI'
   I'm no expert on Kite but a quick check of their docs makes me think the current URI being
used really is not valid.    Try dataset:hdfs:/default/ctest1       On Mon, Dec 14, 2015
at 2:33 PM, Chandu Koripella <ckoripel@starbucks.com> wrote: HI Alan,   Thanks for
responding. Hive dataset has 777 permissions. I am not sure why it still unable to validate
it. I can load the data just fine if execute kite-dataset command on the server. can you please
take a look?      /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-exec-log4j.properties,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-exec-log4j.properties,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-log4j.properties
  Here is last few lines from nifi logs.   2015-12-14 11:07:26,671 INFO [pool-18-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with
0 records in 27 milliseconds 2015-12-14 11:09:16,643 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false 2015-12-14 11:09:26,672 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository 2015-12-14 11:09:26,700 INFO [pool-18-thread-1]
org.wali.MinimalLockingWriteAheadLogorg.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed
with 0 Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 12 milliseconds,
Clear Edit Logs time = 13 millis), max Transaction ID -1 2015-12-14 11:09:26,700 INFO [pool-18-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with
0 records in 28 milliseconds 2015-12-14 11:09:42,649 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false 2015-12-14 11:10:33,658 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false 2015-12-14 11:11:26,700 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository 2015-12-14 11:11:26,740 INFO [pool-18-thread-1]
org.wali.MinimalLockingWriteAheadLogorg.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed
with 0 Records and 0 Swap Files in 39 milliseconds (Stop-the-world time = 12 milliseconds,
Clear Edit Logs time = 24 millis), max Transaction ID -1 2015-12-14 11:11:26,740 INFO [pool-18-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with
0 records in 39 milliseconds 2015-12-14 11:13:26,740 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository 2015-12-14 11:13:26,769 INFO [pool-18-thread-1]
org.wali.MinimalLockingWriteAheadLogorg.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed
with 0 Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 12 milliseconds,
Clear Edit Logs time = 13 millis), max Transaction ID -1 2015-12-14 11:13:26,769 INFO [pool-18-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with
0 records in 28 milliseconds 2015-12-14 11:15:21,314 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false 2015-12-14 11:15:26,769 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository 2015-12-14 11:15:26,796 INFO [pool-18-thread-1]
org.wali.MinimalLockingWriteAheadLogorg.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed
with 0 Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 11 milliseconds,
Clear Edit Logs time = 12 millis), max Transaction ID -1 2015-12-14 11:15:26,796 INFO [pool-18-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with
0 records in 27 milliseconds 2015-12-14 11:15:47,820 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false 2015-12-14 11:16:28,328 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false 2015-12-14 11:17:26,797 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository 2015-12-14 11:17:26,824 INFO [pool-18-thread-1]
org.wali.MinimalLockingWriteAheadLogorg.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed
with 0 Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 10 milliseconds,
Clear Edit Logs time = 13 millis), max Transaction ID -1 2015-12-14 11:17:26,825 INFO [pool-18-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with
0 records in 27 milliseconds 2015-12-14 11:19:26,825 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository 2015-12-14 11:19:26,900 INFO [pool-18-thread-1]
org.wali.MinimalLockingWriteAheadLogorg.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed
with 0 Records and 0 Swap Files in 74 milliseconds (Stop-the-world time = 59 milliseconds,
Clear Edit Logs time = 13 millis), max Transaction ID -1 2015-12-14 11:19:26,900 INFO [pool-18-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with
0 records in 75 milliseconds 2015-12-14 11:21:26,900 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository 2015-12-14 11:21:26,928 INFO [pool-18-thread-1]
org.wali.MinimalLockingWriteAheadLogorg.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed
with 0 Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 12 milliseconds,
Clear Edit Logs time = 13 millis), max Transaction ID -1 2015-12-14 11:21:26,928 INFO [pool-18-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with
0 record         From: Alan Jackoway [mailto:alanj@cloudera.com]
Sent: Monday, December 14, 2015 11:05 AM 
To: users@nifi.apache.org
Subject: Re: Nifi Kite-dataset URI issue   Try adding the path to hive-site.xml.   Did it
give you any error message beyond just "cannot validate url"? Is there anything interesting
in the nifi-app.log?   On Mon, Dec 14, 2015 at 12:54 PM, Chandu Koripella <ckoripel@starbucks.com>
wrote: It would be great help if someone can take a look at this.   Thanks, Chandu   From:
Chandu Koripella
Sent: Friday, December 11, 2015 11:44 AM
To: 'users@nifi.apache.org'
Subject: RE: Nifi Kite-dataset URI issue   Hi,   This is final step in implementing first
NIfi job. I really appreciate any quick tips to resolve this this.    Thanks, Chandu    
From: Chandu Koripella
Sent: Friday, December 11, 2015 7:03 AM
To: users@nifi.apache.org
Subject: RE: Nifi Kite-dataset URI issue   Hi Allen,   Thanks for helping me with this issue.
Here is the full value Hadoop config property.   /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml
     This works just fine with putHDFS processor      From: Alan Jackoway [mailto:alanj@cloudera.com]
Sent: Friday, December 11, 2015 5:42 AM
To: users@nifi.apache.org
Subject: Re: Nifi Kite-dataset URI issue   Can you give the full value of that hadoop configuration
files property? On Dec 10, 2015 11:23 PM, "Chandu Koripella" <ckoripel@starbucks.com>
wrote: Hi,   I am trying to load data into a hive external table. Nifi  is unable to validate
the URI. It is working just fine when I use kite-dataset. Can you please advice if am missing
any parameters here?         Thanks, Chandu      

  
Mime
View raw message