tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick C (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (TIKA-1513) Add mime detection and parsing for dbf files
Date Mon, 11 Apr 2016 23:11:25 GMT

    [ https://issues.apache.org/jira/browse/TIKA-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236171#comment-15236171
] 

Nick C edited comment on TIKA-1513 at 4/11/16 11:10 PM:
--------------------------------------------------------

Some of my checks maybe a little strict because you can have extra bytes at the end of the
file and after the field headers (I haven't personally seen any files like that though) I
figure in those cases hopefully the file extension glob matches. I put some TODOs that can
be changed to call jdbf for validating the DBF file type and field type. Feel free to do what
you want with the code
https://gist.github.com/fxfixer/e54f86095a548cbfb8aeb948ff77a41b

I used the jdbf v3 branch and here are the bugs I noticed. If [~iryndin] is interested I'll
create a pull request.
Calls to input.read(byte[]…) should use IOUtils.readFully. (Sometimes if the dbf is in a
zip file, the read call returns less than the requested bytes)
DBFMetadataReader.readHeader()
	- Needs to call IOUtils.readFully when reading headerBytes
	- NPE if DbfFileTypeEnum.fromInt returns null (Maybe throw an unsupported exception?)
	- Reads record count as int instead of unsigned int
	
DBFRecordIterator
	- Unnecessary call to Arrays.fill to set byte[] bytes to 0 (Not really a bug)
	- Needs to call IOUtils.readFully when reading recordBuffer;
	
Some encoding names are not correct in CharsetHelper.getCharsetByByte
936 = cp936 // Chinese (PRC, Singapore) Windows
932 = cp932 // Japanese Windows
1255 = Windows-1255 // Hebrew Windows
1256 = Windows-1256 // Arabic Windows
1250 = Windows-1250 // Eastern European Windows
1251 = Windows-1251 // Russian Windows
1254 = Windows-1254 // Turkish Windows
1253 = Windows-1253 // Greek Windows


was (Author: nicholasc):
Some of my checks maybe a little strict because you can have extra bytes at the end of file
and after the field headers (I haven't personally seen any files like that though) I figure
in those cases hopefully the file extension glob matches. I put some TODOs that can be changed
to call jdbf for validating the DBF file type and field type. Feel free to do what you want
with the code
https://gist.github.com/fxfixer/e54f86095a548cbfb8aeb948ff77a41b

I used the jdbf v3 branch and here are the bugs I noticed. If [~iryndin] is interested I'll
create a pull request.
Calls to input.read(byte[]…) should use IOUtils.readFully. (Sometimes if the dbf is in a
zip file, the read call returns less than the requested bytes)
DBFMetadataReader.readHeader()
	- Needs to call IOUtils.readFully when reading headerBytes
	- NPE if DbfFileTypeEnum.fromInt returns null (Maybe throw an unsupported exception?)
	- Reads record count as int instead of unsigned int
	
DBFRecordIterator
	- Unnecessary call to Arrays.fill to set byte[] bytes to 0 (Not really a bug)
	- Needs to call IOUtils.readFully when reading recordBuffer;
	
Some encoding names are not correct in CharsetHelper.getCharsetByByte
936 = cp936 // Chinese (PRC, Singapore) Windows
932 = cp932 // Japanese Windows
1255 = Windows-1255 // Hebrew Windows
1256 = Windows-1256 // Arabic Windows
1250 = Windows-1250 // Eastern European Windows
1251 = Windows-1251 // Russian Windows
1254 = Windows-1254 // Turkish Windows
1253 = Windows-1253 // Greek Windows

> Add mime detection and parsing for dbf files
> --------------------------------------------
>
>                 Key: TIKA-1513
>                 URL: https://issues.apache.org/jira/browse/TIKA-1513
>             Project: Tika
>          Issue Type: Improvement
>            Reporter: Tim Allison
>            Priority: Minor
>             Fix For: 1.13
>
>
> I just came across an Apache licensed dbf parser that is available on [maven|https://repo1.maven.org/maven2/org/jamel/dbf/dbf-reader/0.1.0/dbf-reader-0.1.0.pom].
> Let's add dbf parsing to Tika.
> Any other recommendations for alternate parsers?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message