hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Fenchel <jfen...@gmail.com>
Subject Re: Using hama with Mesos 0.20 and CDH5.1.2
Date Wed, 19 Nov 2014 18:47:53 GMT
Thanks for jumping in here. Any improvements to the wiki are greatly
appreciated.

-Jeff

On Thu, Nov 13, 2014 at 2:07 AM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> Done!
>
> On Thu, Nov 13, 2014 at 6:05 AM, Stratos Dimopoulos
> <stratos.dimopoulos@gmail.com> wrote:
>> Thanks Edward!
>> My wiki login name is StratosDimopoulos - I'll edit the page and add a link
>> to the blog as soon as I have edit permission.
>>
>> cheers,
>> Stratos
>>
>> On Wed, Nov 12, 2014 at 1:06 AM, Edward J. Yoon <edwardyoon@apache.org>
>> wrote:
>>
>>> Awesome! Please feel free to link to your blog post on
>>> GettingStartedMesos wiki page, or update it. If you need edit
>>> permission let us know your wiki ID.
>>>
>>> On Sat, Nov 8, 2014 at 6:51 PM, Stratos Dimopoulos
>>> <stratos.dimopoulos@gmail.com> wrote:
>>> > Hi Jeff,
>>> >
>>> > Just wanted to let you know that your patch worked on my setup. Since
>>> > lately I am doing a lot of installations I was keeping some notes and
>>> now I
>>> > polished them and published a blog post (
>>> > https://strat0sphere.wordpress.com/2014/11/08/apache-hama-on-mesos/) on
>>> > which I am including some errors I encountered on the process as a
>>> > non-expert mesos/hama user and the configuration I did to make it work.
>>> > Also, if you think it would be useful to update the documentation (
>>> > http://wiki.apache.org/hama/GettingStartedMesos) on the Hama website,
>>> > mainly the hama-site.xml file, please let me know and I'll be happy doing
>>> > it as my first tiny contribution to the project.
>>> >
>>> > I also noticed that all the links provided on the website are for the
>>> older
>>> > 0.6.4 Hama version, which doesn't support Mesos. Not sure if this is
>>> > intentional or just a mistake.
>>> >
>>> > Again, I really appreciated your help and thanks a ton for doing the fix
>>> in
>>> > such a timely manner!
>>> >
>>> > cheers,
>>> > Stratos
>>> >
>>> > On Mon, Oct 27, 2014 at 12:35 AM, Stratos Dimopoulos <
>>> > stratos.dimopoulos@gmail.com> wrote:
>>> >
>>> >> Great thank you! Looking forward for the patch!
>>> >> On Oct 26, 2014 10:30 PM, "Jeff Fenchel" <jfenc91@gmail.com> wrote:
>>> >>
>>> >>> Stratos,
>>> >>>
>>> >>> Thanks for sending me the stack trace. I was able to replicate your
>>> >>> error. So, it definitely appears to be a bug in Hama. I will open
up a
>>> >>> ticket and see if I can put together a patch for this sometime this
>>> >>> week. In the mean time, Hama without mesos seems to work fine with
>>> >>> cdh5.
>>> >>>
>>> >>> -Jeff
>>> >>>
>>> >>> On Fri, Oct 24, 2014 at 8:20 PM, Stratos Dimopoulos
>>> >>> <stratos.dimopoulos@gmail.com> wrote:
>>> >>> > And when you added the list's user email you had a typo which
I copy
>>> >>> pasted
>>> >>> > and the email never reached the list - So I am just forwarding
again
>>> to
>>> >>> the
>>> >>> > correct address this time! :-)
>>> >>> >
>>> >>> >
>>> >>> > On Thu, Oct 23, 2014 at 5:36 PM, Stratos Dimopoulos
>>> >>> > <stratos.dimopoulos@gmail.com> wrote:
>>> >>> >>
>>> >>> >> Yes Jeff correct - this is the way I compile. Here is the
stack
>>> trace
>>> >>> >> close to the error on the executor - I don't think the
log file on
>>> the
>>> >>> BSP
>>> >>> >> master would help you at all but let me know if you need
to send you
>>> >>> >> anything else:
>>> >>> >>
>>> >>> >> I1022 17:25:31.828336 22440 exec.cpp:132] Version: 0.20.0
>>> >>> >> I1022 17:25:31.838382 22454 exec.cpp:206] Executor registered
on
>>> slave
>>> >>> >> 20141019-011259-1257767434-5050-4023-1
>>> >>> >> 14/10/22 17:25:31 INFO bsp.MesosExecutor: Executor registered
with
>>> the
>>> >>> >> slave
>>> >>> >> 14/10/22 17:25:31 INFO bsp.MesosExecutor: Launching task
: Task_0
>>> >>> >> 14/10/22 17:25:32 INFO bsp.GroomServer: groom start
>>> >>> >> 14/10/22 17:25:32 INFO bsp.GroomServer:
>>> >>> >>
>>> >>>
>>> /mnt/mesos-work-dir/slaves/20141019-011259-1257767434-5050-4023-1/frameworks/20141019-011259-1257767434-5050-4023-0031/executors/executor_Task_0/runs/d69f21af-9180-4c0c-9263-82038ae81937/hama-0.7.0-SNAPSHOT/bsp/local
>>> >>> >> 14/10/22 17:25:32 WARN util.NativeCodeLoader: Unable to
load
>>> >>> native-hadoop
>>> >>> >> library for your platform... using builtin-java classes
where
>>> >>> applicable
>>> >>> >> 14/10/22 17:25:33 ERROR bsp.MesosExecutor: Caught exception,
>>> committing
>>> >>> >> suicide.
>>> >>> >> java.util.ServiceConfigurationError:
>>> org.apache.hadoop.fs.FileSystem:
>>> >>> >> Provider org.apache.hadoop.fs.LocalFileSystem not found
>>> >>> >>         at java.util.ServiceLoader.fail(ServiceLoader.java:231)
>>> >>> >>         at
>>> java.util.ServiceLoader.access$300(ServiceLoader.java:181)
>>> >>> >>         at
>>> >>> >> java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
>>> >>> >>         at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
>>> >>> >>         at
>>> >>> >>
>>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2364)
>>> >>> >>         at
>>> >>> >>
>>> >>>
>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2375)
>>> >>> >>         at
>>> >>> >>
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392)
>>> >>> >>         at
>>> >>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
>>> >>> >>         at
>>> >>> >>
>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
>>> >>> >>         at
>>> >>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
>>> >>> >>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
>>> >>> >>         at
>>> >>> org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:339)
>>> >>> >>         at
>>> >>> >>
>>> org.apache.hama.bsp.GroomServer.deleteLocalFiles(GroomServer.java:483)
>>> >>> >>         at
>>> >>> >> org.apache.hama.bsp.GroomServer.initialize(GroomServer.java:321)
>>> >>> >>         at org.apache.hama.bsp.GroomServer.run(GroomServer.java:860)
>>> >>> >>         at
>>> >>> org.apache.hama.bsp.MesosExecutor$1.run(MesosExecutor.java:92)
>>> >>> >>
>>> >>> >> thanks,
>>> >>> >> Stratos
>>> >>> >>
>>> >>> >> On Thu, Oct 23, 2014 at 2:51 PM, Jeff Fenchel <jfenc91@gmail.com>
>>> >>> wrote:
>>> >>> >>>
>>> >>> >>> Fyi... Sorry I forgot to add your email to list. Please
add
>>> >>> >>> user@hama.apach.org to your reply.
>>> >>> >>>
>>> >>> >>> Thanks,
>>> >>> >>> Jeff
>>> >>> >>>
>>> >>> >>> ---------- Forwarded message ----------
>>> >>> >>> From: Jeff Fenchel <jfenc91@gmail.com>
>>> >>> >>> Date: Thursday, October 23, 2014
>>> >>> >>> Subject: Using hama with Mesos 0.20 and CDH5.1.2
>>> >>> >>> To: user@hama.apache.org
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> Hey Stratos,
>>> >>> >>>
>>> >>> >>> Thanks for giving Hama on Mesos a try. I do not think
it has ever
>>> been
>>> >>> >>> tested with hadoop 2, but I see no reason why it shouldn't
work.
>>> So,
>>> >>> I am
>>> >>> >>>
>>> >>> >>> guessing you compiled with something like the following?
>>> >>> >>>
>>> >>> >>> mvn clean install -Phadoop2 -Dhadoop.version=2.3.0-cdh5.1.2
>>> >>> >>> -Dmesos.version=0.20.0 -DskipTests
>>> >>> >>>
>>> >>> >>> If so, can you please send us the complete stack trace?
It does
>>> look
>>> >>> >>> like this might be a bug in hama.
>>> >>> >>>
>>> >>> >>> Thanks,
>>> >>> >>> -Jeff
>>> >>> >>>
>>> >>> >>>
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> On Thu, Oct 23, 2014 at 1:13 AM, Stratos Dimopoulos
>>> >>> >>> <stratos.dimopoulos@gmail.com> wrote:
>>> >>> >>> > Hi,
>>> >>> >>> >
>>> >>> >>> > I cloned Hama from Git and installed in a Eucalyptus
cloud with
>>> >>> Mesos
>>> >>> >>> > 0.20
>>> >>> >>> > and CDH5.1.2 HDFS. It is also configured with
zookeeper but my
>>> >>> >>> > zookeeper at
>>> >>> >>> > the moment just runs in one master. I followed
the instructions
>>> >>> here:
>>> >>> >>> > http://wiki.apache.org/hama/GettingStartedMesos
- that are for
>>> an
>>> >>> older
>>> >>> >>> > version and additionally to that I added some
more properties to
>>> >>> >>> > overcome
>>> >>> >>> > some runtime problems and until Mesos was able
to recognize Hama
>>> as
>>> >>> a
>>> >>> >>> > framework. Overall the properties I have set are
the following:
>>> >>> >>> >
>>> >>> >>> > bsp.master.address
>>> >>> >>> > bsp.master.port
>>> >>> >>> > bsp.master.TaskWorkerManager.class
>>> >>> >>> > fs.default.name (I don't think it needs to be
here but in an
>>> >>> effort to
>>> >>> >>> > fix
>>> >>> >>> > the issues I had)
>>> >>> >>> > hama.mesos.executor.uri
>>> >>> >>> > bsp.tasks.maximum.total
>>> >>> >>> > hama.mesos.master
>>> >>> >>> > bsp.child.java.opts
>>> >>> >>> > bsp.system.dir
>>> >>> >>> > bsp.local.dir
>>> >>> >>> > bsp.tmp.dir
>>> >>> >>> > bsp.disk.queue.dir
>>> >>> >>> > hama.zookeeper.quorum
>>> >>> >>> > hama.zookeeper.property.clientPort
>>> >>> >>> >
>>> >>> >>> > When I run the example the executor is running
but the task
>>> fails.
>>> >>> The
>>> >>> >>> > error output on the executors log is: ERROR bsp.MesosExecutor:
>>> >>> Caught
>>> >>> >>> > exception, committing suicide.
>>> >>> >>> > java.util.ServiceConfigurationError:
>>> >>> org.apache.hadoop.fs.FileSystem:
>>> >>> >>> > Provider org.apache.hadoop.fs.LocalFileSystem
not found
>>> >>> >>> >
>>> >>> >>> > I did some research on this error message but
I cannot find
>>> >>> something
>>> >>> >>> > that
>>> >>> >>> > really helps. It seems to me as a configuration
issue but I don't
>>> >>> know
>>> >>> >>> > how
>>> >>> >>> > to solve it.
>>> >>> >>> > Also note that I have verified that my HDFS works
properly. I am
>>> >>> >>> > actually
>>> >>> >>> > already running Spark and Hadoop over Mesos without
problems.
>>> >>> >>> >
>>> >>> >>> > I hope somebody can help!
>>> >>> >>> >
>>> >>> >>> > thanks,
>>> >>> >>> > Stratos
>>> >>> >>>
>>> >>> >>
>>> >>> >
>>> >>>
>>> >>
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> CEO at DataSayer Co., Ltd.
>>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.

Mime
View raw message