lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: PDF search functionality using Solr
Date Tue, 06 Jan 2015 18:51:18 GMT
Seconding Jürgen's comment. 4G docs are almost, but not quite totally
useless to search How many JIRA's each? That's _one_ document unless
you do some fancy dancing. Pulling the data directly using the JIRA
API sounds far superior.

If you _must_ use the JIRA->PDF->Solr option, consider the following:
Use Tika on the client to parse the doc and taking control of the
mapping of the meta-data
and, probably, breaking thins up into individual document, one Solr
document per JIRA.

That'll give you a chance to deal with charset issues and the like.
Here's an example:

https://lucidworks.com/blog/indexing-with-solrj/

That one has both Tika and database connectivity but should be pretty
straight-forward to adapt, just pull the database junk out.

Best,
Erick

On Tue, Jan 6, 2015 at 9:55 AM, "Jürgen Wagner (DVT)"
<juergen.wagner@devoteam.com> wrote:
> Hello,
>   no matter which search platform you will use, this will pose two
> challenges:
>
> - The size of the documents will render search less and less useful as the
> likelihood of matches increases with document size. So, without a proper
> semantic extraction (e.g., using decent NER or relationship extraction with
> a commercial text mining product), I doubt you will get the required
> precision to make this overly usefiul.
>
> - PDFs can have their own character sets based on the characters actually
> used. Such file-specific character sets are almost impossible to parse,
> i.e., if your PDFs happen to use this "feature" of the PDF format, you won't
> be lucky getting any meaningful text out of them.
>
> My suggestion is to use the Jira REST API to collect all necessary documents
> and index the resulting XML or attachment formats. As the REST API provides
> filtering capabilities, you could easily create incremental feeds to avoid
> humongous indexing every time there's new information in Jira. Dumping Jira
> stuff as PDF seems to me to be the least suitable way of handling this.
>
> Best regards,
> --Jürgen
>
>
>
> On 06.01.2015 18:30, Ganesh.Yadav@sungard.com wrote:
>
> Hello Solr-users and developers,
> Can you please suggest,
>
> 1.       What I should do to index PDF content information column wise?
>
> 2.       Do I need to extract the contents using one of the Analyzer,
> Tokenize and Filter combination and then add it to Index? How can test the
> results on command prompt? I do not know the selection of specific Analyzer,
> Tokenizer and Filter for this purpose
>
> 3.       How can I verify that the needed column info is extracted out of
> PDF and is indexed?
>
> 4.       So for example How to verify Ticket number is extracted in
> Ticket_number tag and is indexed?
>
> 5.       Is it ok to post 4 GB worth of PDF to be imported and indexed by
> Solr? I think I saw some posts complaining on how large size that can be
> posted ?
>
> 6.       What will enable Solr to search in any PDF out of many, with
> different words such as "Runtime" "Error" "XXXX" and result will provide the
> link to the PDF
>
> My PDFs are nothing but Jira ticket system.
> PDF has info on
> Ticket Number:
> Desc:
> Client:
> Status:
> Submitter:
> And so on:
>
>
> 1.       I imported PDF document in Solr and it does the necessary searching
> and I can test some of it using the browse client interface provided.
>
> 2.       I have 80 GB worth of PDFs.
>
> 3.       Total number of PDFs are about 200
>
> 4.       Many PDFs are of size 4 GB
>
> 5.       What do you suggest me to import such a large PDFs? What tools can
> you suggest to extract PDF contents first in some XML format and later Post
> that XML to be indexed by Solr.?
>
>
>
>
>
>
>
> Your early response is much appreciated.
>
>
>
> Thanks
>
> G
>
>
>
>
> --
>
> Mit freundlichen Grüßen/Kind regards/Cordialement vôtre/Atentamente/С
> уважением
> i.A. Jürgen Wagner
> Head of Competence Center "Intelligence"
> & Senior Cloud Consultant
>
> Devoteam GmbH, Industriestr. 3, 70565 Stuttgart, Germany
> Phone: +49 6151 868-8725, Fax: +49 711 13353-53, Mobile: +49 171 864 1543
> E-Mail: juergen.wagner@devoteam.com, URL: www.devoteam.de
>
> ________________________________
> Managing Board: Jürgen Hatzipantelis (CEO)
> Address of Record: 64331 Weiterstadt, Germany; Commercial Register:
> Amtsgericht Darmstadt HRB 6450; Tax Number: DE 172 993 071
>
>

Mime
View raw message