hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6108) Add support for EBS storage on EC2
Date Wed, 18 Nov 2009 23:57:39 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tom White updated HADOOP-6108:

    Attachment: HADOOP-6108.patch

Thanks for the really detailed review, Aaron! Responses inline below.

> README.txt
> 34] you seem to pick an arbitrary AMI. How's this chosen? Is there a list of appropriate
AMIS somewhere?

I think listing the AMIs on a wiki page is probably best, since these may be updated from
time to time. AMIs simply need to run scripts passed as (compressed) user data and have Java
installed. It would be a good follow up issue to include scripts to create these AMIs. 

> 58] "with with"


> integration-test/transient-cluster.sh]
> 1) /bin/bash -> /usr/bin/env bash


> 25) Is it really necessary to source ~/.bashrc? The /bin/bash on the shebang line should
take care of this.

Only for interactive shells. I've run these scripts using Hudson, and this is then needed.

> persistent-cluster.sh]
> same two concerns as above.

> 34) the `sleep 5` can be pushed above the if and used only once.


> teststorage.py]
> TestJsonVolumeSpecManager really needs some comments; there's a number of multiply-nested
arrays being dereferenced and it's not obvious what these prove. Is this just testing that
the "spec" record earlier in the file was parsed correctly?

Yes. I've addressed this by improving the naming in the test.

> 171) given EBS replication, is it ok to go to dfs rep 2 here?

Possibly, although I would like to be conservative, and have users pro-actively reduce it.

> VERSION ] should this file match the version associated with the hadoop-common project
at large?


> cli.py]

> instead of a million 'from.... import' commands, consider 'import hadoop.cloud.commands
as commands', then use commands.MASTER, commands.attach_storage(), etc.


> 254) _prompt consider being case-insensitive here.


> 291) in the event of a timeout, should you still print the master url? Is there definitely
a master? Please add a comment saying why further error handling is not needed. A message
suggesting follow-up action items for the user (e.g. "check hadoop-ec2 list to make sure it
really booted") would be helpful too.

I've improved the message. Printing the master address is useful for diagnosing the problem.

> 329) ditto.


> 346) since we're already parsing a config file, can the proxy port be configurable? What
if the user's got two clusters running simultaneously?

I agree that this could be a limitation for some users. I'd like to tackle this in a follow-up

> 461) Add a message saying how to get usage/help text out of the app?

Changed to print usage.

> cluster.py]

> get_cluster() 33) unpythonic key check; this should throw KeyError if provider is not


> Cluster stub functions ) For mandatory methods (e.g. get_provider_code), how about raise
Exception("Unimplemented") instead of pass, or maybe add UnimplementedException / NotYetImplemented
/ Something similar?


> commands.py]

> 36) Maybe add a comment describing high-level purpose of ROLES, and examples of potential
future use (e.g., "HBase support might add new roles here, as would divorced JT/NN, etc.")


> launch_master() 83) better error msg / comment explaining how this timeout is handled
in the rest of the method's logic?

> launch_slaves() 190) ditto.

Improved message, and changed to exit on timeout.

> 231) Since we support both Hadoop 0.18 and 0.20 in the documentation, is there additional
logic required to gracefully support both cluster versions in this code here?

This code supports both Hadoop 0.18 and 0.20, so no further changes should be needed.


> JsonVolumeManager._load() ) why not just percolate the IOError? Do we want file errors
to silently initialize an empty vol manager? (Is this file something the user specifies, or
an optional file? If the former, this should fail hard. If the latter, silent continue seems

We want the latter case.

> class Storage ) same comment as above re. NotYetImplemented vs. pass


> util.py ]

> bash_quote() ) do you need to double-escape backslashes? The unit tests also don't cover
internal backslash characters.

The double backslash is a python-escaped backslash. I've added a test for an internal backslash.

> ec2.py ]
> authorize_role() 150) Why not just catch InvalidPermission.Duplicate?

I couldn't see a clean way of catching this error and ignoring other exceptions.

> Add support for EBS storage on EC2
> ----------------------------------
>                 Key: HADOOP-6108
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6108
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/ec2
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6108.patch, HADOOP-6108.patch, HADOOP-6108.patch
> By using EBS for namenode and datanode storage we can have persistent, restartable Hadoop
clusters running on EC2.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message