nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gaspar, Carson" <gas...@deshaw.com>
Subject RE: PutElasticsearchHttp error behaviour
Date Tue, 22 Nov 2016 16:07:44 GMT
Created https://issues.apache.org/jira/browse/NIFI-3082


-----Original Message-----
From: Matt Burgess [mailto:mattyb149@apache.org] 
Sent: Monday, November 07, 2016 2:53 PM
To: users@nifi.apache.org
Subject: Re: PutElasticsearchHttp error behaviour

I agree, this should be caught and the flow file routed to failure. Do you mind filing a Jira
for this?

Thanks,
Matt

On Mon, Nov 7, 2016 at 2:40 PM, Gaspar, Carson <gaspar@deshaw.com> wrote:
> Yes, I know why it failed, and how to fix it. What I don’t understand 
> is why it was retried forever, causing backpressure and stopping other updates.
> That’s really unacceptable.
>
>
>
> From: Pierre Villard [mailto:pierre.villard.fr@gmail.com]
> Sent: Monday, November 07, 2016 1:15 PM
>
>
> To: users@nifi.apache.org
> Subject: Re: PutElasticsearchHttp error behaviour
>
>
>
> Hi Gaspar,
>
> I believe this is because 4294967295 is a too big value that cannot be 
> contained in an integer. I am not a ES expert but you should look at 
> your applicationId type if you can change it.
>
> Hope this helps.
>
>
>
> 2016-11-07 19:09 GMT+01:00 Gaspar, Carson <gaspar@deshaw.com>:
>
> It stays in the queue on a field type error, e.g.:
> [2016-11-07 12:37:17,263][DEBUG][action.bulk              ] [Cybele]
> [estreamer_index_2016.11.07][0] failed to execute bulk item (index) 
> index 
> {[estreamer_index_2016.11.07][estreamer_doc][AVg_3ZmnIxO_esWdRpuM], source ...data...
> MapperParsingException[failed to parse [applicationId]]; nested:
> NumberFormatException[For input string: "4294967295"];
>         at
> org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:329)
>         at
> org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:311)
>         at
> org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:438)
>         at
> org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:264)
>         at
> org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:124)
>         at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309)
>         at
> org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:529)
>         at
> org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:506)
>         at
> org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:214)
>         at
> org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:223)
>         at
> org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:327)
>         at
> org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:120)
>         at
> org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
>         at
> org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:648)
>         at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
>         at
> org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
>         at
> org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
>         at
> org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
>         at
> org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
>         at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NumberFormatException: For input string: "4294967295"
>         at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>         at java.lang.Integer.parseInt(Integer.java:583)
>         at java.lang.Integer.parseInt(Integer.java:615)
>         at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.intValue(AbstractXContentParser.java:127)
>         at
> org.elasticsearch.index.mapper.core.IntegerFieldMapper.innerParseCreateField(IntegerFieldMapper.java:293)
>         at
> org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:241)
>         at
> org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:321)
>         ... 22 more
>
>
> -----Original Message-----
> From: Koji Kawamura [mailto:ijokarumawak@gmail.com]
> Sent: Monday, November 07, 2016 12:10 AM
> To: users@nifi.apache.org
> Subject: Re: PutElasticsearchHttp error behaviour
>
> Hello Gaspar,
>
> I looked at the PutElasticsearchHttp code, it distinguishes following 
> two types of error:
>
> 1. Error occurs when the processor tries to connect Elasticsearch, but 
> it couldn't. E.g. the specified Elasticsearch URL is not correct and 
> java.net.ConnectException is thrown.
> When this happens, the processor throws ProcessException, and NiFi 
> framework keeps the FlowFile in the queue, so that it can be retried later.
> This indicates that DataFlowManager(user) has to update the processor 
> configuration to make it work properly.
>
> 2. The processor successfully sends a request to Elasticsearch, but 
> Elasticsearch returns error response. This happens with operations 
> such as updating with a document Id that doesn't exist in the target index.
> When this happens, the processor routes the incoming FlowFile which 
> caused the error, to failure relationship.
>
> I think the error you encountered the former type of error above (#1).
> If the error was returned from Elasticsearch, it should be routed to 
> failure relationship as you suspected. Would you please let us know 
> about the specific error you got? A StackTrace would be helpful.
>
> BTW, PutElasticsearchHttp also has REL_RETRY relationship, but it 
> seems it's not used currently.
>
> Thanks!
> Koji
>
> On Thu, Nov 3, 2016 at 5:47 AM, Gaspar, Carson <gaspar@deshaw.com> wrote:
>> When PutElasticsearchHttp gets an error from Elasticsearch, it 
>> _appears_ to never dequeue the problem flow file and just try it over 
>> and over again. It certainly doesn’t route the problem flow file to 
>> the retry or failure queues. This seems wrong to me – am I missing 
>> something?
>>
>> Nifi is from the Hortonworks
>> nifi_2_0_0_0_579-1.0.0.2.0.0.0-579.el6.x86_64
>> RPM.
>>
>>
>
>
Mime
View raw message