lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject RE: Flume SolrSink No Error but not Indexing
Date Tue, 07 Feb 2017 23:04:14 GMT
Hi - are the channels filling up? If not, events can be silently dropped by custom interceptors,
but you don 't seem to have any. We're not using that sink you have, it might also silenently
drop events. At least check you channel size, it goes up in most cases, meaning all your events
are still stored and ready to be shipped to the sink.

M.
 
 
-----Original message-----
> From:Anatharaman, Srinatha (Contractor) <Srinatha_Anantharaman@comcast.com>
> Sent: Tuesday 7th February 2017 22:07
> To: solr-user@lucene.apache.org
> Subject: Flume SolrSink No Error but not Indexing
> 
> Hi,
> 
> 
> I am indexing text document using Flume,
> I do not see any error or warning message but data is not getting ingested to Solr
> Log level for both Solr and Flume is set to TRACE, ALL
> 
> Flume version : 1.5.2.2.3
> Solr Version : 5.5
> Config files are as below
> Flume Config :
> agent.sources = SpoolDirSrc
> agent.channels = FileChannel
> agent.sinks = SolrSink
> 
> # Configure Source
> agent.sources.SpoolDirSrc.channels = fileChannel
> agent.sources.SpoolDirSrc.type = spooldir
> agent.sources.SpoolDirSrc.spoolDir = /home/flume/source_emails
> agent.sources.SpoolDirSrc.basenameHeader = true
> agent.sources.SpoolDirSrc.fileHeader = true
> #agent.sources.src1.fileSuffix = .COMPLETED
> agent.sources.SpoolDirSrc.deserializer = org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder
> # Use a channel that buffers events in memory
> agent.channels.FileChannel.type = file
> agent.channels.FileChannel.capacity = 10000
> #agent.channels.FileChannel.transactionCapacity = 10000
> # Configure Solr Sink
> agent.sinks.SolrSink.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
> agent.sinks.SolrSink.morphlineFile = /etc/flume/conf/morphline.conf
> agent.sinks.SolrSink.batchsize = 1000
> agent.sinks.SolrSink.batchDurationMillis = 2500
> agent.sinks.SolrSink.channel = fileChannel
> agent.sinks.SolrSink.morphlineId = morphline1
> agent.sources.SpoolDirSrc.channels = FileChannel
> agent.sinks.SolrSink.channel = FileChannel
> 
> Morphline Config
> solrLocator: {
> collection : gsearch
> #zkHost : "127.0.0.1:9983"
> zkHost : "codesolr-as-r3p:21810,codesolr-as-r3p:21811,codesolr-as-r3p:21812"
> }
> morphlines :
> [
>   {
>     id : morphline1
>     importCommands : ["org.kitesdk.**", "org.apache.solr.**"]
>     commands :
>     [
>       { detectMimeType { includeDefaultMimeTypes : true } }
>       {
>         solrCell {
>           solrLocator : ${solrLocator}
>           captureAttr : true
>           lowernames : true
>           capture : [_attachment_body, _attachment_mimetype, basename, content, content_encoding,
content_type, file, meta]
>           parsers : [ { parser : org.apache.tika.parser.txt.TXTParser } ]
>          }
>       }
>       { generateUUID { field : id } }
>       { sanitizeUnknownSolrFields { solrLocator : ${solrLocator} } }
>       { logDebug { format : "output record: {}", args : ["@{}"] } }
>       { loadSolr: { solrLocator : ${solrLocator} } }
>     ]
>   }
> ]
> 
> Please help me what could be the issue
> Regards,
> ~Sri
> 
> 

Mime
View raw message