spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lizhengbing (bing, BIPA)" <zhengbing...@huawei.com>
Subject Could the function MLUtils.loadLibSVMFile be modified to support zero-based-index data?
Date Tue, 08 Jul 2014 06:29:55 GMT

1)  I download the imdb data from http://komarix.org/ac/ds/Blanc__Mel.txt.bz2 and use this
data to test LBFGS
When I run examples referencing http://spark.apache.org/docs/latest/mllib-optimization.html,
 an error occus.
4/07/07 08:37:27 ERROR Executor: Exception in task ID 2
java.lang.ArrayIndexOutOfBoundsException: -1
         at breeze.linalg.operators.DenseVector_SparseVector_Ops$$anon$129.apply(SparseVectorOps.scala:231)
         at breeze.linalg.operators.DenseVector_SparseVector_Ops$$anon$129.apply(SparseVectorOps.scala:216)
         at breeze.linalg.operators.BinaryRegistry$class.apply(BinaryOp.scala:60)
         at breeze.linalg.VectorOps$$anon$178.apply(Vector.scala:391)
         at breeze.linalg.NumericOps$class.dot(NumericOps.scala:83)
         at breeze.linalg.DenseVector.dot(DenseVector.scala:47)
..................

2)  I find the imdb data are zero-based-index data
0 0:1 3:1 6208:1 8936:1 8959:1 16434:1 29840:1 29843:1 30274:1 32092:1 63727:1 109302:1 114311:1
114336:1 119637:1 121867:1 143744:1 145106:1 186951:1 216401:1 228548:1 248919:1 251691:1
294713:1 302316:1 307685:1 316421:1 316556:1 317062:1 321771:1 327174:1 364381:1 384514:1
404531:1 414947:1 434235:1 434250:1 462625:1 471013:1 503923:1 511725:1 514582:1 514635:1
519251:1 524274:1 540734:1 556018:1 559036:1 559037:1 559039:1 559341:1 609032:1 644534:1
650763:1 659114:1 666864:1 669778:1 669783:1 669787:1 673083:1

3) If change code "val index = indexAndValue(0).toInt - 1" to "val index = indexAndValue(0).toInt
- offset" (offset equals 0 or 1 based on user's selection), then MLUtils.loadLibSVMFile will
support both zero-based-index data and one-based-index data.
  That also means the interface of MLUtils.loadLibSVMFile will be changed



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message