tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TIKA-2762) Capture short fields (<150 chars) in EnviParserHeader Metadata
Date Wed, 24 Oct 2018 06:44:00 GMT

    [ https://issues.apache.org/jira/browse/TIKA-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661807#comment-16661807
] 

ASF GitHub Bot commented on TIKA-2762:
--------------------------------------

lewismc commented on issue #251: TIKA-2762 Capture short fields (<150 chars) in EnviParserHeader
Metadata
URL: https://github.com/apache/tika/pull/251#issuecomment-432532425
 
 
   Hi @tballison I think it's an excellent idea. I've prefixed any key with ```envi.``` and
also aligned the replacement of space ```" "``` with period ```.```
   
   Additionally, I am still having issues actually detecting the envi header files as such,
they seem to be detected as .txt files unless additional header information is provided. This
justifies the additional tests I added to detection logic.
   
   An example as follows
   
   **Using tika-app**
   Incorrectly detected and minimal metadata extracted
   ```
   java -jar tika-app-2.0.0-SNAPSHOT.jar /usr/local/pycoal_data/avng.jpl.nasa.gov/AVNG_2015_data_distribution/L2/ang20150420t182050_rfl_v1e/ang20150420t182050_corr_v1e_img.hdr
   Oct 23, 2018 11:24:00 PM org.apache.tika.config.InitializableProblemHandler$3 handleInitializableProblem
   WARNING: J2KImageReader not loaded. JPEG2000 files will not be processed.
   See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io
   for optional dependencies.
   
   Oct 23, 2018 11:24:00 PM org.apache.tika.config.InitializableProblemHandler$3 handleInitializableProblem
   WARNING: org.xerial's sqlite-jdbc is not loaded.
   Please provide the jar on your classpath to parse sqlite files.
   See tika-parsers/pom.xml for the correct version.
   <?xml version="1.0" encoding="UTF-8"?><html xmlns="http://www.w3.org/1999/xhtml">
   <head>
   <meta name="X-Parsed-By" content="org.apache.tika.parser.DefaultParser"/>
   <meta name="X-Parsed-By" content="org.apache.tika.parser.txt.TXTParser"/>
   <meta name="Content-Encoding" content="ISO-8859-1"/>
   <meta name="resourceName" content="ang20150420t182050_corr_v1e_img.hdr"/>
   <meta name="Content-Length" content="20731"/>
   <meta name="Content-Type" content="text/plain; charset=ISO-8859-1"/>
   <title/>
   </head>
   ```
   **Using tika-server with additional header information**
   Correct metadata extracted 
   ```
   curl -T ang20150420t182050_corr_v1e_img.hdr http://localhost:9998/meta --header "Content-Type:
application/envi.hdr"
   "envi.byte.order","0"
   "X-Parsed-By","org.apache.tika.parser.DefaultParser","org.apache.tika.parser.envi.EnviHeaderParser"
   "envi.sensor.type","Unknown"
   "envi.bands","432"
   "envi.header.offset","0"
   "language","en"
   "envi.interleave","bil"
   "envi.wavelength.units","Nanometers"
   "envi.lines","14674"
   "envi.file.type","ENVI Standard"
   "envi.samples","739"
   "Content-Encoding","ISO-8859-1"
   "envi.data.type","4"
   "Content-Type","application/envi.hdr"
   ```
   **Using tika-server with NO additional header information**
   Again, same result as tika-app, detection logic is not working...
   ```
   curl -T ang20150420t182050_corr_v1e_img.hdr http://localhost:9998/meta
   "X-Parsed-By","org.apache.tika.parser.DefaultParser","org.apache.tika.parser.txt.TXTParser"
   "Content-Encoding","ISO-8859-1"
   "language","en"
   "Content-Type","text/plain; charset=ISO-8859-1"
   ```
   There is something up with detection logic for application/envi.hdr files with .hdr file
suffix even though this information is included within tika-mimetypes.xml

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Capture short fields (<150 chars) in EnviParserHeader Metadata
> --------------------------------------------------------------
>
>                 Key: TIKA-2762
>                 URL: https://issues.apache.org/jira/browse/TIKA-2762
>             Project: Tika
>          Issue Type: Improvement
>          Components: parser
>    Affects Versions: 1.19.1
>            Reporter: Lewis John McGibbney
>            Assignee: Lewis John McGibbney
>            Priority: Major
>             Fix For: 1.20
>
>
> I have always wanted to capture more metadata for the EnviHeader files. Right now everything
is shoved into the records content and I think we could improve it.
> I've implemented a rudimentary parser improvement with essentially captures any reasonably
sized lines items (<150 chars) which can then be populated up to Metadata level making
faceted search over ENVI .hdr documents a much easier task.
> PR coming up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message