lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anatharaman, Srinatha (Contractor)" <Srinatha_Ananthara...@comcast.com>
Subject RE: Flume SolrSink No Error but not Indexing
Date Wed, 08 Feb 2017 01:49:41 GMT
M, Thank you for your suggestion

I kept just one file to index. It is very small file with 3 lines and 3 characters on each
line
I see the content in flume log file and an entry in Solr but not sure why it is not going
to Solr

-----Original Message-----
From: Markus Jelsma [mailto:markus.jelsma@openindex.io] 
Sent: Tuesday, February 07, 2017 6:04 PM
To: solr-user@lucene.apache.org
Subject: RE: Flume SolrSink No Error but not Indexing

Hi - are the channels filling up? If not, events can be silently dropped by custom interceptors,
but you don 't seem to have any. We're not using that sink you have, it might also silenently
drop events. At least check you channel size, it goes up in most cases, meaning all your events
are still stored and ready to be shipped to the sink.

M.
 
 
-----Original message-----
> From:Anatharaman, Srinatha (Contractor) 
> <Srinatha_Anantharaman@comcast.com>
> Sent: Tuesday 7th February 2017 22:07
> To: solr-user@lucene.apache.org
> Subject: Flume SolrSink No Error but not Indexing
> 
> Hi,
> 
> 
> I am indexing text document using Flume, I do not see any error or 
> warning message but data is not getting ingested to Solr Log level for 
> both Solr and Flume is set to TRACE, ALL
> 
> Flume version : 1.5.2.2.3
> Solr Version : 5.5
> Config files are as below
> Flume Config :
> agent.sources = SpoolDirSrc
> agent.channels = FileChannel
> agent.sinks = SolrSink
> 
> # Configure Source
> agent.sources.SpoolDirSrc.channels = fileChannel 
> agent.sources.SpoolDirSrc.type = spooldir 
> agent.sources.SpoolDirSrc.spoolDir = /home/flume/source_emails 
> agent.sources.SpoolDirSrc.basenameHeader = true 
> agent.sources.SpoolDirSrc.fileHeader = true 
> #agent.sources.src1.fileSuffix = .COMPLETED 
> agent.sources.SpoolDirSrc.deserializer = 
> org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder
> # Use a channel that buffers events in memory 
> agent.channels.FileChannel.type = file 
> agent.channels.FileChannel.capacity = 10000 
> #agent.channels.FileChannel.transactionCapacity = 10000 # Configure 
> Solr Sink agent.sinks.SolrSink.type = 
> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
> agent.sinks.SolrSink.morphlineFile = /etc/flume/conf/morphline.conf 
> agent.sinks.SolrSink.batchsize = 1000 
> agent.sinks.SolrSink.batchDurationMillis = 2500 
> agent.sinks.SolrSink.channel = fileChannel 
> agent.sinks.SolrSink.morphlineId = morphline1 
> agent.sources.SpoolDirSrc.channels = FileChannel 
> agent.sinks.SolrSink.channel = FileChannel
> 
> Morphline Config
> solrLocator: {
> collection : gsearch
> #zkHost : "127.0.0.1:9983"
> zkHost : "codesolr-as-r3p:21810,codesolr-as-r3p:21811,codesolr-as-r3p:21812"
> }
> morphlines :
> [
>   {
>     id : morphline1
>     importCommands : ["org.kitesdk.**", "org.apache.solr.**"]
>     commands :
>     [
>       { detectMimeType { includeDefaultMimeTypes : true } }
>       {
>         solrCell {
>           solrLocator : ${solrLocator}
>           captureAttr : true
>           lowernames : true
>           capture : [_attachment_body, _attachment_mimetype, basename, content, content_encoding,
content_type, file, meta]
>           parsers : [ { parser : org.apache.tika.parser.txt.TXTParser } ]
>          }
>       }
>       { generateUUID { field : id } }
>       { sanitizeUnknownSolrFields { solrLocator : ${solrLocator} } }
>       { logDebug { format : "output record: {}", args : ["@{}"] } }
>       { loadSolr: { solrLocator : ${solrLocator} } }
>     ]
>   }
> ]
> 
> Please help me what could be the issue Regards, ~Sri
> 
> 

Mime
View raw message