whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrei Savu <savu.and...@gmail.com>
Subject Re: Java and Hadoop on Ubuntu 11.10 AMI
Date Sun, 04 Dec 2011 21:26:25 GMT
I am happy to hear that!! Ping the email list any time you have questions.

On Sun, Dec 4, 2011 at 11:14 PM, Periya.Data <periya.data@gmail.com> wrote:

> Hi Andrei,
>    Many thanks again. Will see those links and try to create a cluster
> with the tools I need.
>
> Will be happy to contribute in any way I can. Will see the Wiki on how to
> contribute and will proceed from there.
>
> /PD.
>
>
> On Sun, Dec 4, 2011 at 1:10 PM, Andrei Savu <savu.andrei@gmail.com> wrote:
>
>> The best way would be to define a new service for each one with new
>> install / configure scripts.
>>
>> See this presentation about creating a new Whirr service:
>> http://www.oscon.com/oscon2011/public/schedule/detail/19214
>>
>> For Flume the following repo contains some working code:
>> https://github.com/tomwhite/whirr-service-example
>>
>> For Hive we are tracking progress here:
>> https://issues.apache.org/jira/browse/WHIRR-343
>>
>> Currently there is no open issue for adding Sqoop as a service.
>>
>> I strongly recommend you to submit patches / pull requests as early as
>> possible and I will give as
>> much assistance as needed. See the following wiki page to learn more
>> about how to contribute:
>> https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute
>>
>> On Sun, Dec 4, 2011 at 11:00 PM, Periya.Data <periya.data@gmail.com>wrote:
>>
>>> Thanks much. Will follow your suggestions. A quick question:
>>>
>>> If I have to install Hive, Sqoop and Flume on the my cluster nodes, what
>>> is the suggested way?
>>> --> Log into each cluster node and install them manually?
>>> --> Maybe the I have to hack the install script
>>> (/whirr/services/cdh/src/main/resources/functions) or create a new one for
>>> each of the tools?
>>>
>>> Thanks,
>>> PD.
>>>
>>>
>>> On Sun, Dec 4, 2011 at 12:44 PM, Andrei Savu <savu.andrei@gmail.com>wrote:
>>>
>>>> Here are a few advices:
>>>>
>>>> * use Ubuntu 10.04 LTS 64bit on the cluster nodes - this is the OS
>>>> version we are testing against before making a release
>>>> * whirr.hadoop.tarball.url is irrelevant if you specify
>>>> whirr.hadoop.install-function. Same is true for whirr.hadoop.version.
>>>> * install_cdh_hadoop will install by default the latest release.
>>>> We've implemented a way of selecting the version in 0.7.0:
>>>> https://issues.apache.org/jira/browse/WHIRR-415
>>>> * use something bigger than t1.micro - this is just too small to run
>>>> Hadoop
>>>>
>>>> Let me know if you are still having problems.
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>>
>>>> On Sun, Dec 4, 2011 at 9:44 PM, Periya.Data <periya.data@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>    I am trying Whirr to spin-up EC2 clusters. Here are the specs:
>>>>>
>>>>>
>>>>>    - Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2
>>>>>    - EC2 Whirr config :
>>>>>    -
>>>>>       - whirr.hadoop.install-function=install_cdh_hadoop
>>>>>    - whirr.hadoop.configure-function=configure_cdh_hadoop
>>>>>       - whirr.hadoop.version=0.20.2
>>>>>       - whirr.hadoop.tarball.url=
>>>>>       http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz
>>>>>       - whirr.hardware-id=t1.micro
>>>>>                 # Ubuntu 11.10 Oneiric EBS boot from alestic
>>>>>    - whirr.image-id=us-east-1/ami-a7f539ce
>>>>>       - whirr.location-id=us-east-1
>>>>>
>>>>>
>>>>> *Problem:* The cluster seems to be up and running. But, cannot see
>>>>> Java and Hadoop on the instance.
>>>>>
>>>>> *Question:* Is this a problem or should I clearly mention in my Whirr
>>>>> config/properties file what version of JDK to be installed and its download
>>>>> URL?
>>>>>
>>>>>
>>>>> *Some logs:* (I ssh-ed into one of the nodes and tried to see the
>>>>> logs)
>>>>>
>>>>> sri@domU-12-31-39-02-21-20:/tmp$ ls -al
>>>>> total 40
>>>>> drwxrwxrwt  4 root root  4096 2011-12-04 06:25 .
>>>>> drwxr-xr-x 22 root root  4096 2011-12-04 05:46 ..
>>>>> -rw-r--r--  1 root root   938 2011-12-04 05:46 core-site.xml
>>>>> -rw-r--r--  1 root root   701 2011-12-04 05:46 hdfs-site.xml
>>>>> drwxr-xr-x  2 root root  4096 2011-12-04 05:46
>>>>> jclouds-script-1322977554131
>>>>> drwxr-xr-x  2 root root  4096 2011-12-04 05:44 logs
>>>>> -rw-r--r--  1 root root  1447 2011-12-04 05:46 mapred-site.xml
>>>>> -rwxr--r--  1 root root 11212 2011-12-04 05:44 setup-sri.sh
>>>>> sri@domU-12-31-39-02-21-20:/tmp$
>>>>> sri@domU-12-31-39-02-21-20:/tmp$
>>>>> sri@domU-12-31-39-02-21-20:/tmp$
>>>>> sri@domU-12-31-39-02-21-20:/tmp$ cd logs/
>>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l
>>>>> total 12
>>>>> -rw-r--r-- 1 root root  117 2011-12-04 05:45 stderr.log
>>>>> -rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log
>>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log
>>>>> E: Package 'sun-java6-jdk' has no installation candidate
>>>>> /tmp/setup-sri.sh: line 161: java: command not found
>>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>>>
>>>>> ----------
>>>>>
>>>>> Get:36 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe
i386 Packages [57.2 kB]
>>>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse
i386 Packages [4396 B]
>>>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>>> TranslationIndex [73 B]
>>>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse
TranslationIndex [72 B
>>>>> ]
>>>>> Get:40 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe
TranslationIndex [73 B]
>>>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
>>>>> Translation-en [701 kB]
>>>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
>>>>> Translation-en
>>>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
>>>>> Translation-en [3165 kB]
>>>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>>> Translation-en [99.2 kB]
>>>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse
Translation-en [2419 B
>>>>> ]
>>>>> Get:45 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe
Translation-en [35.4 kB]
>>>>> Fetched 16.2 MB in 12s (1290 kB/s)
>>>>> Reading package lists...
>>>>> Reading package lists...
>>>>> Building dependency tree...
>>>>> Reading state information...
>>>>> Package sun-java6-jdk is not available, but is referred to by another
>>>>> package.
>>>>> This may mean that the package is missing, has been obsoleted, or
>>>>> is only available from another source
>>>>>
>>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>>> Get:36 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe
i386 Packages [57.2 kB]
>>>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse
i386 Packages [4396 B]
>>>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>>> TranslationIndex [73 B]
>>>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse
TranslationIndex [72 B]
>>>>> Get:40 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe
TranslationIndex [73 B]
>>>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
>>>>> Translation-en [701 kB]
>>>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
>>>>> Translation-en
>>>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
>>>>> Translation-en [3165 kB]
>>>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>>> Translation-en [99.2 kB]
>>>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse
Translation-en [2419 B]
>>>>> Get:45 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe
Translation-en [35.4 kB]
>>>>> Fetched 16.2 MB in 12s (1290 kB/s)
>>>>> Reading package lists...
>>>>> Reading package lists...
>>>>> Building dependency tree...
>>>>> Reading state information...
>>>>> Package sun-java6-jdk is not available, but is referred to by another
>>>>> package.
>>>>> This may mean that the package is missing, has been obsoleted, or
>>>>> is only available from another source
>>>>>
>>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>>>
>>>>> *[On a different cluster spun-up at a later time]*
>>>>>
>>>>> sri@domU-12-31-39-10-6D-38:~$ java
>>>>> The program 'java' can be found in the following packages:
>>>>>  * gcj-4.4-jre-headless
>>>>>  * gcj-4.6-jre-headless
>>>>>  * openjdk-6-jre-headless
>>>>>  * gcj-4.5-jre-headless
>>>>>  * openjdk-7-jre-headless
>>>>> Ask your administrator to install one of them
>>>>> sri@domU-12-31-39-10-6D-38:~$ java -version
>>>>> The program 'java' can be found in the following packages:
>>>>>  * gcj-4.4-jre-headless
>>>>>  * gcj-4.6-jre-headless
>>>>>  * openjdk-6-jre-headless
>>>>>  * gcj-4.5-jre-headless
>>>>>  * openjdk-7-jre-headless
>>>>> Ask your administrator to install one of them
>>>>> sri@domU-12-31-39-10-6D-38:~$
>>>>>
>>>>> sri@domU-12-31-39-10-6D-38:~$ hadoop version
>>>>> hadoop: command not found
>>>>> sri@domU-12-31-39-10-6D-38:~$
>>>>>
>>>>> ===========================================
>>>>>
>>>>> Finally, a quick Googling pointed to a JIRA issue: https://issues.apache.org/jira/browse/WHIRR-435
>>>>>
>>>>>
>>>>> I am seeing the same stuff. Any updates? If this is a real issues, I
>>>>> do not have any problems in using Ubuntu 11.04 AMI for now.
>>>>>
>>>>>
>>>>> Your help is very much appreciated.
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> PD.
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message