nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chandu Koripella <ckori...@starbucks.com>
Subject RE: Nifi Kite-dataset URI issue
Date Wed, 16 Dec 2015 19:07:35 GMT
HI,

I am running into another issue. I really appreciate your help. My apology if I am asking
very basic question. I am first time user for both Nifi, Kite dataset and Hadoop

My Nifi is service is running as HDFS user.


·        Create a dataset as HDFS and I could able to import just fine as HDFS

[cid:image001.png@01D137F0.62C659A0]


·        Permissions:

[cid:image003.png@01D137F1.F3FC4FA0]



·        URI Validates just fine in NIfi
[cid:image002.png@01D137F1.62345D60]

[cid:image007.png@01D137F0.07E03D30]


·        Hadoop Configuration files: /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-site.xml

Any help is appreciated.

Thanks,
Chandra

From: Chandu Koripella [mailto:ckoripel@starbucks.com]
Sent: Monday, December 14, 2015 11:49 AM
To: users@nifi.apache.org
Subject: RE: Nifi Kite-dataset URI issue

You are right. It works just fine by changing the keyword to hdfs. Kit-dataset accepting the
both hdfs and hive


[cid:image008.png@01D137F0.07E03D30]

Thanks,
Chandu


From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Monday, December 14, 2015 11:37 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Nifi Kite-dataset URI issue

the property to focus on is 'Target Dataset URI'

I'm no expert on Kite but a quick check of their docs makes me think the current URI being
used really is not valid.

Try dataset:hdfs:/default/ctest1


On Mon, Dec 14, 2015 at 2:33 PM, Chandu Koripella <ckoripel@starbucks.com<mailto:ckoripel@starbucks.com>>
wrote:
HI Alan,

Thanks for responding. Hive dataset has 777 permissions. I am not sure why it still unable
to validate it. I can load the data just fine if execute kite-dataset command on the server.
can you please take a look?

[cid:image009.png@01D137F0.07E03D30]

/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-exec-log4j.properties,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-exec-log4j.properties,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-log4j.properties

Here is last few lines from nifi logs.

2015-12-14 11:07:26,671 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully
checkpointed FlowFile Repository with 0 records in 27 milliseconds
2015-12-14 11:09:16,643 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d<mailto:org.apache.nifi.controller.FlowController@66f2383d>
// Another save pending = false
2015-12-14 11:09:26,672 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating
checkpoint of FlowFile Repository
2015-12-14 11:09:26,700 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@75a9cc24<mailto:org.wali.MinimalLockingWriteAheadLog@75a9cc24>
checkpointed with 0 Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 12
milliseconds, Clear Edit Logs time = 13 millis), max Transaction ID -1
2015-12-14 11:09:26,700 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully
checkpointed FlowFile Repository with 0 records in 28 milliseconds
2015-12-14 11:09:42,649 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d<mailto:org.apache.nifi.controller.FlowController@66f2383d>
// Another save pending = false
2015-12-14 11:10:33,658 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d<mailto:org.apache.nifi.controller.FlowController@66f2383d>
// Another save pending = false
2015-12-14 11:11:26,700 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating
checkpoint of FlowFile Repository
2015-12-14 11:11:26,740 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@75a9cc24<mailto:org.wali.MinimalLockingWriteAheadLog@75a9cc24>
checkpointed with 0 Records and 0 Swap Files in 39 milliseconds (Stop-the-world time = 12
milliseconds, Clear Edit Logs time = 24 millis), max Transaction ID -1
2015-12-14 11:11:26,740 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully
checkpointed FlowFile Repository with 0 records in 39 milliseconds
2015-12-14 11:13:26,740 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating
checkpoint of FlowFile Repository
2015-12-14 11:13:26,769 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@75a9cc24<mailto:org.wali.MinimalLockingWriteAheadLog@75a9cc24>
checkpointed with 0 Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 12
milliseconds, Clear Edit Logs time = 13 millis), max Transaction ID -1
2015-12-14 11:13:26,769 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully
checkpointed FlowFile Repository with 0 records in 28 milliseconds
2015-12-14 11:15:21,314 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d<mailto:org.apache.nifi.controller.FlowController@66f2383d>
// Another save pending = false
2015-12-14 11:15:26,769 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating
checkpoint of FlowFile Repository
2015-12-14 11:15:26,796 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@75a9cc24<mailto:org.wali.MinimalLockingWriteAheadLog@75a9cc24>
checkpointed with 0 Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 11
milliseconds, Clear Edit Logs time = 12 millis), max Transaction ID -1
2015-12-14 11:15:26,796 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully
checkpointed FlowFile Repository with 0 records in 27 milliseconds
2015-12-14 11:15:47,820 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d<mailto:org.apache.nifi.controller.FlowController@66f2383d>
// Another save pending = false
2015-12-14 11:16:28,328 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d<mailto:org.apache.nifi.controller.FlowController@66f2383d>
// Another save pending = false
2015-12-14 11:17:26,797 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating
checkpoint of FlowFile Repository
2015-12-14 11:17:26,824 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@75a9cc24<mailto:org.wali.MinimalLockingWriteAheadLog@75a9cc24>
checkpointed with 0 Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 10
milliseconds, Clear Edit Logs time = 13 millis), max Transaction ID -1
2015-12-14 11:17:26,825 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully
checkpointed FlowFile Repository with 0 records in 27 milliseconds
2015-12-14 11:19:26,825 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating
checkpoint of FlowFile Repository
2015-12-14 11:19:26,900 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@75a9cc24<mailto:org.wali.MinimalLockingWriteAheadLog@75a9cc24>
checkpointed with 0 Records and 0 Swap Files in 74 milliseconds (Stop-the-world time = 59
milliseconds, Clear Edit Logs time = 13 millis), max Transaction ID -1
2015-12-14 11:19:26,900 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully
checkpointed FlowFile Repository with 0 records in 75 milliseconds
2015-12-14 11:21:26,900 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating
checkpoint of FlowFile Repository
2015-12-14 11:21:26,928 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@75a9cc24<mailto:org.wali.MinimalLockingWriteAheadLog@75a9cc24>
checkpointed with 0 Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 12
milliseconds, Clear Edit Logs time = 13 millis), max Transaction ID -1
2015-12-14 11:21:26,928 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully
checkpointed FlowFile Repository with 0 record




From: Alan Jackoway [mailto:alanj@cloudera.com<mailto:alanj@cloudera.com>]
Sent: Monday, December 14, 2015 11:05 AM

To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Nifi Kite-dataset URI issue

Try adding the path to hive-site.xml.

Did it give you any error message beyond just "cannot validate url"? Is there anything interesting
in the nifi-app.log?

On Mon, Dec 14, 2015 at 12:54 PM, Chandu Koripella <ckoripel@starbucks.com<mailto:ckoripel@starbucks.com>>
wrote:
It would be great help if someone can take a look at this.

Thanks,
Chandu

From: Chandu Koripella
Sent: Friday, December 11, 2015 11:44 AM
To: 'users@nifi.apache.org<mailto:users@nifi.apache.org>'
Subject: RE: Nifi Kite-dataset URI issue

Hi,

This is final step in implementing first NIfi job. I really appreciate any quick tips to resolve
this this.

Thanks,
Chandu


From: Chandu Koripella
Sent: Friday, December 11, 2015 7:03 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: Nifi Kite-dataset URI issue

Hi Allen,

Thanks for helping me with this issue. Here is the full value Hadoop config property.

/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml

[cid:image010.png@01D137F0.07E03D30]

This works just fine with putHDFS processor

[cid:image011.png@01D137F0.07E03D30]

From: Alan Jackoway [mailto:alanj@cloudera.com]
Sent: Friday, December 11, 2015 5:42 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Nifi Kite-dataset URI issue


Can you give the full value of that hadoop configuration files property?
On Dec 10, 2015 11:23 PM, "Chandu Koripella" <ckoripel@starbucks.com<mailto:ckoripel@starbucks.com>>
wrote:
Hi,

I am trying to load data into a hive external table. Nifi  is unable to validate the URI.
It is working just fine when I use kite-dataset.
Can you please advice if am missing any parameters here?

[cid:image012.png@01D137F0.07E03D30]

[cid:image013.png@01D137F0.07E03D30]

Thanks,
Chandu


Mime
View raw message