flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johannes (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2984) Support lenient parsing of SVMLight input files
Date Sat, 07 Nov 2015 20:29:14 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14995398#comment-14995398
] 

Johannes commented on FLINK-2984:
---------------------------------

Good question about the widespread mis/use of this changed format.

I would go with the setup : whatever other tools are able to read should work here as well.

If you have a look at for example [liblinear|https://github.com/ninjin/liblinear/blob/master/train.c#L318]
^1^ they use the following method to split label and features

{code}
char *p = strtok(line," \t"); // label
{code}

Where {{strtok}} (arguably not the best function) treats multiple delimiters as one.

1. [https://github.com/ninjin/liblinear/blob/master/train.c#L318]

> Support lenient parsing of SVMLight input files
> -----------------------------------------------
>
>                 Key: FLINK-2984
>                 URL: https://issues.apache.org/jira/browse/FLINK-2984
>             Project: Flink
>          Issue Type: Improvement
>          Components: Machine Learning Library
>    Affects Versions: 0.9.1
>            Reporter: Johannes
>            Priority: Trivial
>              Labels: easyfix
>
> The current implementation for the reader assumes that the format follows the exact specification.
> The [splice-site Dataset| https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html#splice-site]
dataset is formatted slightly different
> Example
> {noformat}
> -1  1:0.381846 2:0.163648 3:0.245472 4:0.627318
> {noformat}
> note the two spaces after the label.
> Currently MLUtils.scala splits on single spaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message