samoa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From edi-bice <...@git.apache.org>
Subject [GitHub] incubator-samoa pull request: Changes required to work on newer ve...
Date Fri, 22 Jan 2016 22:03:25 GMT
GitHub user edi-bice opened a pull request:

    https://github.com/apache/incubator-samoa/pull/47

    Changes required to work on newer versions of Samza (0.10) and YARN (2.6.0)

    One important issue remains however. It seems to not pick up the yarn-site.xml resourcemanager.address
and defaults to 0.0.0.0:8032 which is not the case for my cluster so connection times out.

    
    bash-4.1$ YARN_HOME=/usr/hdp/current/hadoop-yarn-client/etc/hadoop bin/samoa samza target/SAMOA-Samza-0.4.0-incubating-SNAPSHOT.jar
"PrequentialEvaluation -l classifiers.ensemble.Bagging -s (ArffFileStream -f covtypeNorm.arff)
-f 1000"
    bin/samoa
    Deploying to SAMZA
    Command line string =  PrequentialEvaluation -l classifiers.ensemble.Bagging -s (ArffFileStream
-f covtypeNorm.arff) -f 1000
    2016-01-22 21:49:16,860 [main] INFO  org.apache.samoa.SamzaDoTask (SamzaDoTask.java:103)
- Sucessfully instantiating org.apache.samoa.tasks.PrequentialEvaluation
    2016-01-22 21:49:17,023 [main] INFO  org.apache.samoa.utils.SystemsUtils (SystemsUtils.java:114)
- Hadoop config home:/usr/hdp/current/hadoop-yarn-client/etc/hadoop
    2016-01-22 21:49:17,338 [main] INFO  org.apache.samoa.utils.SystemsUtils (SystemsUtils.java:172)
- Filesystem name:hdfs://lvsdevfrsa01.gspt.net:8020
    2016-01-22 21:49:21,684 [main] INFO  org.apache.samoa.utils.SystemsUtils (SystemsUtils.java:114)
- Hadoop config home:/usr/hdp/current/hadoop-yarn-client/etc/hadoop
    2016-01-22 21:49:21,924 [main] INFO  org.apache.samoa.utils.SystemsUtils (SystemsUtils.java:172)
- Filesystem name:hdfs://lvsdevfrsa01.gspt.net:8020
    2016-01-22 21:49:23,689 [main] INFO  org.apache.samoa.topology.impl.SamzaEngine (SamzaEngine.java:106)
- Config:{yarn.container.count=1, yarn.container.memory.mb=1024, yarn.config.home=/usr/hdp/current/hadoop-yarn-client/etc/hadoop,
systems.kafka0.consumer.zookeeper.connect=lvsdevfrsa01.gspt.net:2181, serializers.registry.kryo.class=org.apache.samoa.utils.SamzaKryoSerdeFactory,
systems.kafka0.producer.bootstrap.servers=lvsdevfrsa01.gspt.net:6667,lvsdevfrsa02.gspt.net:6667,lvsdevfrsa03.gspt.net:6667,
systems.samoa.samza.factory=org.apache.samoa.topology.impl.SamoaSystemFactory, kryo.register=org.apache.samoa.learners.classifiers.rules.common.Perceptron:org.apache.samoa.learners.classifiers.rules.common.Perceptron?PerceptronSerializer,org.apache.samoa.learners.classifiers.trees.ComputeContentEvent:org.apache.samoa.learners.classifiers.trees.ComputeContentEvent?ComputeCEFullPrecSerializer,org.apache.samoa.moa.classifiers.core.AttributeSplitSuggestion:org.apache.samoa.utils.Serializa
 bleSerializer,org.apache.samoa.learners.classifiers.trees.AttributeContentEvent:org.apache.samoa.learners.classifiers.trees.AttributeContentEvent?AttributeCEFullPrecSerializer,org.apache.samoa.learners.classifiers.rules.common.TargetMean:org.apache.samoa.learners.classifiers.rules.common.TargetMean?TargetMeanSerializer,
systems.kafka0.producer.producer.type=sync, yarn.am.container.memory.mb=1024, job.coordinator.replication.factor=2,
task.processor.filesystem=hdfs, job.name=Prequential_20160122214916-0, task.processor.file=hdfs://lvsdevfrsa01.gspt.net:8020/user/ebice/.samoa/dat/Prequential_20160122214916.dat,
job.coordinator.system=kafka0, systems.kafka0.samza.msg.serde=kryo, task.inputs=samoa.Prequential_20160122214916-0,
job.factory.class=org.apache.samza.job.yarn.YarnJobFactory, yarn.package.path=hdfs://lvsdevfrsa01.gspt.net:8020/user/ebice/.samoa/SAMOA-Samza-0.4.0-incubating-SNAPSHOT.jar,
systems.kafka0.producer.batch.num.messages=1, task.class=org.apache.samoa.topology.impl.Sam
 zaEntranceProcessingItem, systems.kafka0.samza.offset.default=oldest, task.opts=-Xmx768M
-XX:+PrintGCDateStamps, systems.kafka0.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory}

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/edi-bice/incubator-samoa master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-samoa/pull/47.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #47
    
----
commit 6d57f0409b1a4bd909c768ae36541eff77c9bf82
Author: edi_bice <edi_bice@yahoo.com>
Date:   2016-01-21T16:31:03Z

    See this for details
    
    http://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file

commit 3ff4d05e9f027254d8e891695cc0d723245b5c58
Author: edi_bice <edi_bice@yahoo.com>
Date:   2016-01-21T20:51:13Z

    Upgraded to Samza 0.10

commit 956b01645b0b5d5b39ebce2f98b75df5024b7ecb
Author: edi_bice <edi_bice@yahoo.com>
Date:   2016-01-21T21:02:14Z

    Job coordinator config required

commit c3a9dcd348cf4e2447c4c04e267d88524aa798d7
Author: edi_bice <edi_bice@yahoo.com>
Date:   2016-01-21T21:11:07Z

    still complaining about missing coordinator config

commit a309024392302cc6120c57f17c314920a41d2828
Author: edi_bice <edi_bice@yahoo.com>
Date:   2016-01-21T21:17:00Z

    key was changed between 0.7 and 0.10

commit b82726d6c01d3f098f43ec60525e635c0c9ac1e3
Author: edi_bice <edi_bice@yahoo.com>
Date:   2016-01-22T21:44:21Z

    key was changed between 0.7 and 0.10

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message