nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Agarwal, Kirti (MIND)" <Kirti.Agar...@mind-infotech.com>
Subject Getting error in Kafka Consumer
Date Mon, 07 May 2018 07:37:53 GMT
Hi,

we have produced some data in kafka in Avro format with embedded schema.

We are able to consume this data using the storm topology code at our end. Also, tested the
same data with Streamsets and it is parsed successfully.

But, when we tried to read same data through "ConsumeKafkaRecord_0_10" in Apache NiFi, it
is giving an error as below:

Failed to properly receive messages from Kafka. Will roll back session and any un-committed
offsets from Kafka.: org.apache.nifi.processor.exception.ProcessException: Could not determine
the Avro Schema to use for writing the content

Please help, and let me know in case of more details required.

Complete Stack Trace is as below:
2018-05-04 07:11:19,796 ERROR [Timer-Driven Process Thread-1] o.a.n.p.k.pubsub.ConsumeKafkaRecord_0_10
ConsumeKafkaRecord_0_10[id=a68b13d0-1001-1163-c2e2-c460278f7145] Failed to properly receive
messages from Kafka. Will roll back session and any un-committed offsets from Kafka.: org.apache.nifi.processor.exception.ProcessException:
Could not determine the Avro Schema to use for writing the content
org.apache.nifi.processor.exception.ProcessException: Could not determine the Avro Schema
to use for writing the content
at org.apache.nifi.avro.AvroRecordSetWriter.createWriter(AvroRecordSetWriter.java:109)
at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy164.createWriter(Unknown Source)
at org.apache.nifi.processors.kafka.pubsub.ConsumerLease.writeRecordData(ConsumerLease.java:510)
at org.apache.nifi.processors.kafka.pubsub.ConsumerLease.lambda$processRecords$2(ConsumerLease.java:322)
at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at org.apache.nifi.processors.kafka.pubsub.ConsumerLease.processRecords(ConsumerLease.java:309)
at org.apache.nifi.processors.kafka.pubsub.ConsumerLease.poll(ConsumerLease.java:170)
at org.apache.nifi.processors.kafka.pubsub.ConsumeKafkaRecord_0_10.onTrigger(ConsumeKafkaRecord_0_10.java:328)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.nifi.schema.access.SchemaNotFoundException: Failed to compile Avro Schema
at org.apache.nifi.avro.AvroRecordSetWriter.createWriter(AvroRecordSetWriter.java:100)
... 23 common frames omitted
Caused by: org.apache.avro.SchemaParseException: Illegal character in: p_AUX Code
at org.apache.avro.Schema.validateName(Schema.java:1151)
at org.apache.avro.Schema.access$200(Schema.java:81)
at org.apache.avro.Schema$Field.<init>(Schema.java:403)
at org.apache.avro.Schema.parse(Schema.java:1281)
at org.apache.avro.Schema$Parser.parse(Schema.java:1032)
at org.apache.avro.Schema$Parser.parse(Schema.java:1020)
at org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(AvroRecordSetWriter.java:131)
at org.apache.nifi.avro.AvroRecordSetWriter.createWriter(AvroRecordSetWriter.java:92)
... 23 common frames omitted


Regards,
Kirti Agarwal


________________________________

The information contained in this electronic message and any attachments to this message are
intended for the exclusive use of the addressee(s) and may contain proprietary, confidential
or privileged information. If you are not the intended recipient, you should not disseminate,
distribute or copy this e-mail. Please notify the sender immediately and destroy all copies
of this message and any attachments. WARNING: Computer viruses can be transmitted via email.
The recipient should check this email and any attachments for the presence of viruses. The
company accepts no liability for any damage caused by any virus/trojan/worms/malicious code
transmitted by this email. www.motherson.com

Mime
View raw message