mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Angelo Immediata <angelo...@gmail.com>
Subject Re: Info about KMEans clustering
Date Thu, 28 Nov 2013 10:42:52 GMT
Hi

sorry..I got it...I hadn't to close the writer and reader in the finally
(or at least I had to close them in the right time)

Now it seems to be working

Thank you
Angelo


2013/11/28 Angelo Immediata <angeloimm@gmail.com>

> Hi Suneel
> First of all thank you for your support
> As I'm using maven and I chenged the HADOOP version in my POM by no more
> excluding the HADOOP version related to the mahout pom (it should be the
> 1.1.2); now I have this dependency tree:
> [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @
> clusteranalysis ---
> [INFO] it.eng.tz.pinf:clusteranalysis:jar:0.0.1-SNAPSHOT
> [INFO] +- org.apache.mahout:mahout-core:jar:0.8:compile
> [INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.12:compile
> [INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.12:compile
> [INFO] |  +- org.slf4j:slf4j-api:jar:1.7.5:compile
> [INFO] |  +- org.apache.commons:commons-lang3:jar:3.1:compile
> [INFO] |  +- commons-io:commons-io:jar:2.4:compile
> [INFO] |  +- com.thoughtworks.xstream:xstream:jar:1.4.4:compile
> [INFO] |  |  +- xmlpull:xmlpull:jar:1.1.3.1:compile
> [INFO] |  |  \- xpp3:xpp3_min:jar:1.1.4c:compile
> [INFO] |  +- org.apache.lucene:lucene-core:jar:4.3.0:compile
> [INFO] |  +- org.apache.lucene:lucene-analyzers-common:jar:4.3.0:compile
> [INFO] |  +- org.apache.mahout.commons:commons-cli:jar:2.0-mahout:compile
> [INFO] |  +- org.apache.commons:commons-math3:jar:3.2:compile
> [INFO] |  +- org.apache.mrunit:mrunit:jar:hadoop1:1.0.0:compile
> [INFO] |  |  \- org.mockito:mockito-all:jar:1.8.5:compile
> [INFO] |  \- org.apache.hadoop:hadoop-core:jar:1.1.2:compile
> [INFO] |     +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] |     +- com.sun.jersey:jersey-core:jar:1.8:compile
> [INFO] |     +- com.sun.jersey:jersey-json:jar:1.8:compile
> [INFO] |     |  +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] |     |  |  \- stax:stax-api:jar:1.0.1:compile
> [INFO] |     |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] |     |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
> [INFO] |     |  |     +- javax.xml.stream:stax-api:jar:1.0-2:compile
> [INFO] |     |  |     \- javax.activation:activation:jar:1.1:compile
> [INFO] |     |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.7.1:compile
> [INFO] |     |  \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile
> [INFO] |     +- com.sun.jersey:jersey-server:jar:1.8:compile
> [INFO] |     |  \- asm:asm:jar:3.1:compile
> [INFO] |     +- commons-httpclient:commons-httpclient:jar:3.0.1:compile
> [INFO] |     +- commons-codec:commons-codec:jar:1.4:compile
> [INFO] |     +- org.apache.commons:commons-math:jar:2.1:compile
> [INFO] |     +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] |     |  +-
> commons-collections:commons-collections:jar:3.2.1:compile
> [INFO] |     |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] |     |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] |     |  \-
> commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO] |     +- commons-net:commons-net:jar:1.4.1:compile
> [INFO] |     \- commons-el:commons-el:jar:1.0:compile
> [INFO] +- org.apache.mahout:mahout-math:jar:0.8:compile
> [INFO] |  \- com.google.guava:guava:jar:14.0.1:compile
> [INFO] +- org.apache.mahout:mahout-collections:jar:1.0:compile
> [INFO] +- org.apache.mahout:mahout-utils:jar:0.5:compile
> [INFO] |  +- org.apache.solr:solr-commons-csv:jar:3.1.0:compile
> [INFO] |  \- org.slf4j:slf4j-jcl:jar:1.6.0:compile
> [INFO] +- org.apache.mahout:mahout-integration:jar:0.8:compile
> [INFO] |  +- commons-dbcp:commons-dbcp:jar:1.4:compile
> [INFO] |  +- commons-pool:commons-pool:jar:1.6:compile
> [INFO] |  +- org.apache.lucene:lucene-benchmark:jar:4.3.0:compile
> [INFO] |  |  +- org.apache.lucene:lucene-highlighter:jar:4.3.0:compile
> [INFO] |  |  |  \- org.apache.lucene:lucene-queries:jar:4.3.0:compile
> [INFO] |  |  +- org.apache.lucene:lucene-memory:jar:4.3.0:compile
> [INFO] |  |  +- org.apache.lucene:lucene-queryparser:jar:4.3.0:compile
> [INFO] |  |  |  \- org.apache.lucene:lucene-sandbox:jar:4.3.0:compile
> [INFO] |  |  |     \- jakarta-regexp:jakarta-regexp:jar:1.4:compile
> [INFO] |  |  +- org.apache.lucene:lucene-facet:jar:4.3.0:compile
> [INFO] |  |  +- com.ibm.icu:icu4j:jar:49.1:compile
> [INFO] |  |  +- net.sourceforge.nekohtml:nekohtml:jar:1.9.17:compile
> [INFO] |  |  +- org.apache.commons:commons-compress:jar:1.4.1:compile
> [INFO] |  |  \- xerces:xercesImpl:jar:2.9.1:compile
> [INFO] |  +- org.mongodb:mongo-java-driver:jar:2.11.1:compile
> [INFO] |  +- org.mongodb:bson:jar:2.11.1:compile
> [INFO] |  +- org.apache.cassandra:cassandra-all:jar:1.2.5:compile
> [INFO] |  |  +- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
> [INFO] |  |  +- net.jpountz.lz4:lz4:jar:1.1.0:compile
> [INFO] |  |  +- com.ning:compress-lzf:jar:0.8.4:compile
> [INFO] |  |  +- commons-lang:commons-lang:jar:2.4:compile
> [INFO] |  |  +-
> com.googlecode.concurrentlinkedhashmap:concurrentlinkedhashmap-lru:jar:1.3:compile
> [INFO] |  |  +- org.antlr:antlr:jar:3.2:compile
> [INFO] |  |  |  \- org.antlr:antlr-runtime:jar:3.2:compile
> [INFO] |  |  |     \- org.antlr:stringtemplate:jar:3.2:compile
> [INFO] |  |  |        \- antlr:antlr:jar:2.7.7:compile
> [INFO] |  |  +-
> org.apache.cassandra.deps:avro:jar:1.4.0-cassandra-1:compile
> [INFO] |  |  |  \- org.mortbay.jetty:jetty:jar:6.1.22:compile
> [INFO] |  |  |     \-
> org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
> [INFO] |  |  +- jline:jline:jar:1.0:compile
> [INFO] |  |  +- com.googlecode.json-simple:json-simple:jar:1.1:compile
> [INFO] |  |  +-
> com.github.stephenc.high-scale-lib:high-scale-lib:jar:1.1.2:compile
> [INFO] |  |  +- org.yaml:snakeyaml:jar:1.6:compile
> [INFO] |  |  +- edu.stanford.ppl:snaptree:jar:0.1:compile
> [INFO] |  |  +- org.mindrot:jbcrypt:jar:0.3m:compile
> [INFO] |  |  +- com.yammer.metrics:metrics-core:jar:2.0.3:compile
> [INFO] |  |  +- org.apache.thrift:libthrift:jar:0.7.0:compile
> [INFO] |  |  |  +- javax.servlet:servlet-api:jar:2.5:compile
> [INFO] |  |  |  \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
> [INFO] |  |  |     \- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
> [INFO] |  |  +- org.apache.cassandra:cassandra-thrift:jar:1.2.5:compile
> [INFO] |  |  +- com.github.stephenc:jamm:jar:0.2.5:compile
> [INFO] |  |  \- io.netty:netty:jar:3.5.9.Final:compile
> [INFO] |  +- org.apache.hbase:hbase-client:jar:0.95.0:compile
> [INFO] |  |  +- org.apache.hbase:hbase-common:jar:0.95.0:compile
> [INFO] |  |  +- org.apache.hbase:hbase-protocol:jar:0.95.0:compile
> [INFO] |  |  +- com.google.protobuf:protobuf-java:jar:2.4.1:compile
> [INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
> [INFO] |  |  +- org.cloudera.htrace:htrace:jar:1.50:compile
> [INFO] |  |  |  \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [INFO] |  |  \-
> com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
> [INFO] |  \- me.prettyprint:hector-core:jar:1.0-5:compile
> [INFO] |     +- com.github.stephenc.eaio-uuid:uuid:jar:3.2.0:compile
> [INFO] |     \- com.ecyrd.speed4j:speed4j:jar:0.9:compile
> [INFO] +- org.apache.mahout:mahout-examples:jar:0.8:compile
> [INFO] +- commons-logging:commons-logging:jar:1.1.3:compile
> [INFO] +- log4j:log4j:jar:1.2.17:compile
> [INFO] \- junit:junit:jar:4.11:test
> [INFO]    \- org.hamcrest:hamcrest-core:jar:1.3:test
>
> As we can see I have the hadoop-core version 1.1.2 in my classpath but I'm
> still having that error...I can't figure what I'm wrong
> The error is always:
> java.lang.IllegalStateException:
> file:/root/Scrivania/testKmean/input/testdata/clusters/part-00000
>  at
> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:129)
> at
> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:120)
>  at com.google.common.collect.Iterators$8.transform(Iterators.java:860)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
>  at com.google.common.collect.Iterators$5.hasNext(Iterators.java:597)
> at
> com.google.common.collect.ForwardingIterator.hasNext(ForwardingIterator.java:43)
>  at
> org.apache.mahout.clustering.kmeans.KMeansUtil.configureWithClusterInfo(KMeansUtil.java:51)
> at
> org.apache.mahout.clustering.kmeans.KMeansDriver.buildClusters(KMeansDriver.java:209)
>  at
> org.apache.mahout.clustering.kmeans.KMeansDriver.run(KMeansDriver.java:143)
> at
> it.eng.tz.pinf.clustering.ClusterAnalysisModule.executeClusterAnalysis(ClusterAnalysisModule.java:84)
>  at
> it.eng.tz.pinf.clustering.KMeansDriverClusterTest.kmeansClusteringTest(KMeansDriverClusterTest.java:24)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>  at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>  at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>  at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>  at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>  at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>  at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
>  at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1512)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1490)
>  at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
>  at
> org.apache.mahout.common.iterator.sequencefile.SequenceFileValueIterator.<init>(SequenceFileValueIterator.java:56)
> at
> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:124)
>  ... 33 more
>
>
> Am I missing any other passage? Should I activate HADOOP daemons? Should I
> put my input and output directories in specific points?
>
> Thank you
> Angelo
>
>
>
> 2013/11/28 Suneel Marthi <suneel_marthi@yahoo.com>
>
>> This is not an issue with Mahout and more to do with ur environment. U
>> seem to be missing Hadoop in it path,
>>
>> Also mahout 0.8 is officially not supported on Hadoop 2.2.
>>
>> Sent from my iPhone
>>
>> > On Nov 28, 2013, at 4:39 AM, Angelo Immediata <angeloimm@gmail.com>
>> wrote:
>> >
>> > Hi all
>> > I'm pretty new to mahout and I don't know if this is the right place
>> where
>> > to post this questions, so pardon me if I'm wrong :)
>> > I'm using apache mahout 0.8, apache hadoop 2.2.0; I wanted to test this
>> > class:
>> > public class ClusterAnalysisModule {
>> >
>> > public static final double[][] points = { { 1, 1 }, { 2, 1 }, { 1, 2 },
>> {
>> > 2, 2 }, { 3, 3 }, { 8, 8 }, { 9, 8 }, { 8, 9 }, { 9, 9 } };
>> > private static final Log LOG =
>> > LogFactory.getLog(ClusterAnalysisModule.class.getName());
>> > private void writePointsToFile(List<Vector> points, String fileName,
>> > FileSystem fs, Configuration conf) throws IOException {
>> > Path path = new Path(fileName);
>> > Option fileOption = SequenceFile.Writer.file(path);
>> > Option keyClassOption =
>> SequenceFile.Writer.keyClass(LongWritable.class);
>> > Option valueClassOption =
>> > SequenceFile.Writer.valueClass(VectorWritable.class);
>> > SequenceFile.Writer writer = SequenceFile.createWriter(conf, fileOption,
>> > keyClassOption, valueClassOption);
>> > long recNum = 0;
>> > VectorWritable vec = new VectorWritable();
>> > for (Vector point : points) {
>> > vec.set(point);
>> > writer.append(new LongWritable(recNum++), vec);
>> > }
>> > writer.close();
>> > }
>> >
>> > private List<Vector> getPoints(double[][] raw) {
>> > List<Vector> points = new ArrayList<Vector>();
>> > for (int i = 0; i < raw.length; i++) {
>> > double[] fr = raw[i];
>> > Vector vec = new RandomAccessSparseVector(fr.length);
>> > vec.assign(fr);
>> > points.add(vec);
>> > }
>> > return points;
>> > }
>> >
>> > public void executeClusterAnalysis() throws Exception {
>> > SequenceFile.Writer writer = null;
>> > SequenceFile.Reader reader = null;
>> > try{
>> > // Numero dei cluster in uscita
>> > int k = 2;
>> > List<Vector> vectors = getPoints(points); // Input data
>> > Configuration conf = new Configuration();
>> > FileSystem fs = FileSystem.get(conf);
>> > writePointsToFile(vectors, "/root/Scrivania/testKmean/dati/file1", fs,
>> > conf); // Scrivo
>> > // i
>> > // punti
>> > // centrali
>> > // iniziali
>> > Path path = new
>> > Path("/root/Scrivania/testKmean/input/testdata/clusters/part-00000");
>> > Option fileOption = SequenceFile.Writer.file(path);
>> > Option keyClassOption = SequenceFile.Writer.keyClass(Text.class);
>> > Option valueClassOption = SequenceFile.Writer.valueClass(Kluster.class);
>> > writer = SequenceFile.createWriter(conf, fileOption, keyClassOption,
>> > valueClassOption);
>> > for (int i = 0; i < k; i++) {
>> > Vector vec = vectors.get(i);
>> > // Cluster cluster = new Canopy(vec, i, new
>> > // EuclideanDistanceMeasure());
>> > Kluster cluster = new Kluster(vec, i, new EuclideanDistanceMeasure());
>> > writer.append(new Text((cluster).getIdentifier()), cluster);
>> > }
>> > KMeansDriver.run(conf, new Path("/root/Scrivania/testKmean/dati/"),
>> > new Path("/root/Scrivania/testKmean/input/testdata/clusters"),
>> > new Path("/root/Scrivania/testKmean/output"), new
>> > EuclideanDistanceMeasure(), 0.001, 10, true, 10, false);
>> > // Path readerPath = new Path("/root/Scrivania/testKmean/output/" +
>> > Cluster.CLUSTERED_POINTS_DIR + "/part-m-00000");
>> > // org.apache.hadoop.io.SequenceFile.Reader.Option optReadPath =
>> > org.apache.hadoop.io.SequenceFile.Reader.file(readerPath);
>> > // reader = SequenceFile.createWriter(conf, optReadPath);
>> > reader = new SequenceFile.Reader(fs, new
>> > Path("/root/Scrivania/testKmean/output/" + Cluster.CLUSTERED_POINTS_DIR
>> +
>> > "/part-m-00000"), conf);
>> > IntWritable key = new IntWritable();
>> > WeightedVectorWritable value = new WeightedVectorWritable();
>> > while (reader.next(key, value)) {
>> > LOG.info(value.toString() + " belongs to cluster " + key.toString());
>> > }
>> > } catch(Exception e){
>> >
>> > LOG.fatal("Errore", e);
>> > throw new IllegalStateException(e);
>> > } finally {
>> > if( writer != null ){
>> >
>> > writer.close();
>> > }
>> > if( reader != null ){
>> >
>> > reader.close();
>> > }
>> > }
>> > }
>> > }
>> >
>> > But when I execute it I get this error:
>> > 10:39:14,083 DEBUG [MutableMetricsFactory] field
>> > org.apache.hadoop.metrics2.lib.MutableRate
>> > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
>> > with annotation
>> > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time,
>> value=[Rate
>> > of successful kerberos logins and latency (milliseconds)], about=,
>> > type=DEFAULT, always=false, sampleName=Ops)
>> > 10:39:14,097 DEBUG [MutableMetricsFactory] field
>> > org.apache.hadoop.metrics2.lib.MutableRate
>> > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure
>> > with annotation
>> > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time,
>> value=[Rate
>> > of failed kerberos logins and latency (milliseconds)], about=,
>> > type=DEFAULT, always=false, sampleName=Ops)
>> > 10:39:14,099 DEBUG [MetricsSystemImpl] UgiMetrics, User and group
>> related
>> > metrics
>> > SLF4J: Class path contains multiple SLF4J bindings.
>> > SLF4J: Found binding in
>> >
>> [jar:file:/root/.m2/repository/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: Found binding in
>> >
>> [jar:file:/root/.m2/repository/org/slf4j/slf4j-jcl/1.6.0/slf4j-jcl-1.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> > explanation.
>> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>> > 10:39:14,355 DEBUG [KerberosName] Kerberos krb5 configuration not found,
>> > setting default realm to empty
>> > 10:39:14,358 DEBUG [Groups]  Creating new Groups object
>> > 10:39:14,361 DEBUG [NativeCodeLoader] Trying to load the custom-built
>> > native-hadoop library...
>> > 10:39:14,362 DEBUG [NativeCodeLoader] Failed to load native-hadoop with
>> > error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
>> > 10:39:14,362 DEBUG [NativeCodeLoader]
>> >
>> java.library.path=/usr/lib/jvm/jdk1.7.0_45/jre/lib/i386/server:/usr/lib/jvm/jdk1.7.0_45/jre/lib/i386:/usr/lib/jvm/jdk1.7.0_45/jre/../lib/i386:/usr/lib/jvm/jdk1.7.0_45/jre/lib/i386/client:/usr/lib/jvm/jdk1.7.0_45/jre/lib/i386::/usr/java/packages/lib/i386:/lib:/usr/lib
>> > 10:39:14,362 WARN  [NativeCodeLoader] Unable to load native-hadoop
>> library
>> > for your platform... using builtin-java classes where applicable
>> > 10:39:14,362 DEBUG [JniBasedUnixGroupsMappingWithFallback] Falling back
>> to
>> > shell based
>> > 10:39:14,363 DEBUG [JniBasedUnixGroupsMappingWithFallback] Group mapping
>> > impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
>> > 10:39:14,495 DEBUG [Groups] Group mapping
>> > impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback;
>> > cacheTimeout=300000
>> > 10:39:14,516 DEBUG [UserGroupInformation] hadoop login
>> > 10:39:14,517 DEBUG [UserGroupInformation] hadoop login commit
>> > 10:39:14,521 DEBUG [UserGroupInformation] using local
>> user:UnixPrincipal:
>> > root
>> > 10:39:14,527 DEBUG [UserGroupInformation] UGI loginUser:root
>> (auth:SIMPLE)
>> > 10:39:14,828 DEBUG [Shell] setsid exited with exit code 0
>> > 10:39:14,917 INFO  [CodecPool] Got brand-new compressor [.deflate]
>> > 10:39:14,945 DEBUG [CodecPool] Got recycled compressor
>> > 10:39:14,959 INFO  [KMeansDriver] Input: /root/Scrivania/testKmean/dati
>> > Clusters In: /root/Scrivania/testKmean/input/testdata/clusters Out:
>> > /root/Scrivania/testKmean/output Distance:
>> > org.apache.mahout.common.distance.EuclideanDistanceMeasure
>> > 10:39:14,959 INFO  [KMeansDriver] convergence: 0.001 max Iterations: 10
>> > 10:39:14,987 WARN  [FSInputChecker] Problem opening checksum file:
>> > file:/root/Scrivania/testKmean/input/testdata/clusters/part-00000.
>> > Ignoring exception:
>> > java.io.EOFException
>> > at java.io.DataInputStream.readFully(DataInputStream.java:197)
>> > at java.io.DataInputStream.readFully(DataInputStream.java:169)
>> > at
>> >
>> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:146)
>> > at
>> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339)
>> > at
>> org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1832)
>> > at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1752)
>> > at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileValueIterator.<init>(SequenceFileValueIterator.java:56)
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:124)
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:120)
>> > at com.google.common.collect.Iterators$8.next(Iterators.java:812)
>> > at com.google.common.collect.Iterators$5.hasNext(Iterators.java:544)
>> > at
>> >
>> com.google.common.collect.ForwardingIterator.hasNext(ForwardingIterator.java:43)
>> > at
>> >
>> org.apache.mahout.clustering.kmeans.KMeansUtil.configureWithClusterInfo(KMeansUtil.java:51)
>> > at
>> >
>> org.apache.mahout.clustering.kmeans.KMeansDriver.buildClusters(KMeansDriver.java:209)
>> > at
>> >
>> org.apache.mahout.clustering.kmeans.KMeansDriver.run(KMeansDriver.java:143)
>> > at
>> >
>> it.eng.tz.pinf.clustering.ClusterAnalysisModule.executeClusterAnalysis(ClusterAnalysisModule.java:84)
>> > at
>> >
>> it.eng.tz.pinf.clustering.KMeansDriverClusterTest.kmeansClusteringTest(KMeansDriverClusterTest.java:24)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:606)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>> > at
>> >
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>> > at
>> >
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>> > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>> > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>> > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>> > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>> > at
>> >
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>> > 10:39:14,993 FATAL [ClusterAnalysisModule] Errore
>> > java.lang.IllegalStateException:
>> > file:/root/Scrivania/testKmean/input/testdata/clusters/part-00000
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:129)
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:120)
>> > at com.google.common.collect.Iterators$8.next(Iterators.java:812)
>> > at com.google.common.collect.Iterators$5.hasNext(Iterators.java:544)
>> > at
>> >
>> com.google.common.collect.ForwardingIterator.hasNext(ForwardingIterator.java:43)
>> > at
>> >
>> org.apache.mahout.clustering.kmeans.KMeansUtil.configureWithClusterInfo(KMeansUtil.java:51)
>> > at
>> >
>> org.apache.mahout.clustering.kmeans.KMeansDriver.buildClusters(KMeansDriver.java:209)
>> > at
>> >
>> org.apache.mahout.clustering.kmeans.KMeansDriver.run(KMeansDriver.java:143)
>> > at
>> >
>> it.eng.tz.pinf.clustering.ClusterAnalysisModule.executeClusterAnalysis(ClusterAnalysisModule.java:84)
>> > at
>> >
>> it.eng.tz.pinf.clustering.KMeansDriverClusterTest.kmeansClusteringTest(KMeansDriverClusterTest.java:24)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:606)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>> > at
>> >
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>> > at
>> >
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>> > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>> > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>> > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>> > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>> > at
>> >
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>> > Caused by: java.io.EOFException
>> > at java.io.DataInputStream.readFully(DataInputStream.java:197)
>> > at java.io.DataInputStream.readFully(DataInputStream.java:169)
>> > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1845)
>> > at
>> >
>> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1810)
>> > at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1759)
>> > at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileValueIterator.<init>(SequenceFileValueIterator.java:56)
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:124)
>> > ... 32 more
>> > 10:39:14,996 FATAL [KMeansDriverClusterTest] Errore durante il test del
>> > clustering KMeans; messaggio errore: java.lang.IllegalStateException:
>> > file:/root/Scrivania/testKmean/input/testdata/clusters/part-00000
>> > java.lang.IllegalStateException: java.lang.IllegalStateException:
>> > file:/root/Scrivania/testKmean/input/testdata/clusters/part-00000
>> > at
>> >
>> it.eng.tz.pinf.clustering.ClusterAnalysisModule.executeClusterAnalysis(ClusterAnalysisModule.java:99)
>> > at
>> >
>> it.eng.tz.pinf.clustering.KMeansDriverClusterTest.kmeansClusteringTest(KMeansDriverClusterTest.java:24)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:606)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>> > at
>> >
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>> > at
>> >
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>> > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>> > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>> > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>> > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>> > at
>> >
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> > at
>> >
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>> > Caused by: java.lang.IllegalStateException:
>> > file:/root/Scrivania/testKmean/input/testdata/clusters/part-00000
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:129)
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:120)
>> > at com.google.common.collect.Iterators$8.next(Iterators.java:812)
>> > at com.google.common.collect.Iterators$5.hasNext(Iterators.java:544)
>> > at
>> >
>> com.google.common.collect.ForwardingIterator.hasNext(ForwardingIterator.java:43)
>> > at
>> >
>> org.apache.mahout.clustering.kmeans.KMeansUtil.configureWithClusterInfo(KMeansUtil.java:51)
>> > at
>> >
>> org.apache.mahout.clustering.kmeans.KMeansDriver.buildClusters(KMeansDriver.java:209)
>> > at
>> >
>> org.apache.mahout.clustering.kmeans.KMeansDriver.run(KMeansDriver.java:143)
>> > at
>> >
>> it.eng.tz.pinf.clustering.ClusterAnalysisModule.executeClusterAnalysis(ClusterAnalysisModule.java:84)
>> > ... 24 more
>> > Caused by: java.io.EOFException
>> > at java.io.DataInputStream.readFully(DataInputStream.java:197)
>> > at java.io.DataInputStream.readFully(DataInputStream.java:169)
>> > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1845)
>> > at
>> >
>> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1810)
>> > at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1759)
>> > at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileValueIterator.<init>(SequenceFileValueIterator.java:56)
>> > at
>> >
>> org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator$1.apply(SequenceFileDirValueIterator.java:124)
>> > ... 32 more
>> >
>> > May anybody tell me where I'm wrong?
>> >
>> > Thank you
>> > Angelo
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message