nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@apache.org>
Subject Re: PutElasticsearchHttp error behaviour
Date Mon, 07 Nov 2016 19:53:24 GMT
I agree, this should be caught and the flow file routed to failure. Do
you mind filing a Jira for this?

Thanks,
Matt

On Mon, Nov 7, 2016 at 2:40 PM, Gaspar, Carson <gaspar@deshaw.com> wrote:
> Yes, I know why it failed, and how to fix it. What I don’t understand is why
> it was retried forever, causing backpressure and stopping other updates.
> That’s really unacceptable.
>
>
>
> From: Pierre Villard [mailto:pierre.villard.fr@gmail.com]
> Sent: Monday, November 07, 2016 1:15 PM
>
>
> To: users@nifi.apache.org
> Subject: Re: PutElasticsearchHttp error behaviour
>
>
>
> Hi Gaspar,
>
> I believe this is because 4294967295 is a too big value that cannot be
> contained in an integer. I am not a ES expert but you should look at your
> applicationId type if you can change it.
>
> Hope this helps.
>
>
>
> 2016-11-07 19:09 GMT+01:00 Gaspar, Carson <gaspar@deshaw.com>:
>
> It stays in the queue on a field type error, e.g.:
> [2016-11-07 12:37:17,263][DEBUG][action.bulk              ] [Cybele]
> [estreamer_index_2016.11.07][0] failed to execute bulk item (index) index
> {[estreamer_index_2016.11.07][estreamer_doc][AVg_3ZmnIxO_esWdRpuM], source
> ...data...
> MapperParsingException[failed to parse [applicationId]]; nested:
> NumberFormatException[For input string: "4294967295"];
>         at
> org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:329)
>         at
> org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:311)
>         at
> org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:438)
>         at
> org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:264)
>         at
> org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:124)
>         at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309)
>         at
> org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:529)
>         at
> org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:506)
>         at
> org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:214)
>         at
> org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:223)
>         at
> org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:327)
>         at
> org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:120)
>         at
> org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
>         at
> org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:648)
>         at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
>         at
> org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
>         at
> org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
>         at
> org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
>         at
> org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
>         at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NumberFormatException: For input string: "4294967295"
>         at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>         at java.lang.Integer.parseInt(Integer.java:583)
>         at java.lang.Integer.parseInt(Integer.java:615)
>         at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.intValue(AbstractXContentParser.java:127)
>         at
> org.elasticsearch.index.mapper.core.IntegerFieldMapper.innerParseCreateField(IntegerFieldMapper.java:293)
>         at
> org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:241)
>         at
> org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:321)
>         ... 22 more
>
>
> -----Original Message-----
> From: Koji Kawamura [mailto:ijokarumawak@gmail.com]
> Sent: Monday, November 07, 2016 12:10 AM
> To: users@nifi.apache.org
> Subject: Re: PutElasticsearchHttp error behaviour
>
> Hello Gaspar,
>
> I looked at the PutElasticsearchHttp code, it distinguishes following two
> types of error:
>
> 1. Error occurs when the processor tries to connect Elasticsearch, but it
> couldn't. E.g. the specified Elasticsearch URL is not correct and
> java.net.ConnectException is thrown.
> When this happens, the processor throws ProcessException, and NiFi framework
> keeps the FlowFile in the queue, so that it can be retried later.
> This indicates that DataFlowManager(user) has to update the processor
> configuration to make it work properly.
>
> 2. The processor successfully sends a request to Elasticsearch, but
> Elasticsearch returns error response. This happens with operations such as
> updating with a document Id that doesn't exist in the target index.
> When this happens, the processor routes the incoming FlowFile which caused
> the error, to failure relationship.
>
> I think the error you encountered the former type of error above (#1).
> If the error was returned from Elasticsearch, it should be routed to failure
> relationship as you suspected. Would you please let us know about the
> specific error you got? A StackTrace would be helpful.
>
> BTW, PutElasticsearchHttp also has REL_RETRY relationship, but it seems it's
> not used currently.
>
> Thanks!
> Koji
>
> On Thu, Nov 3, 2016 at 5:47 AM, Gaspar, Carson <gaspar@deshaw.com> wrote:
>> When PutElasticsearchHttp gets an error from Elasticsearch, it
>> _appears_ to never dequeue the problem flow file and just try it over
>> and over again. It certainly doesn’t route the problem flow file to
>> the retry or failure queues. This seems wrong to me – am I missing
>> something?
>>
>> Nifi is from the Hortonworks
>> nifi_2_0_0_0_579-1.0.0.2.0.0.0-579.el6.x86_64
>> RPM.
>>
>>
>
>

Mime
View raw message