hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6541) An Interactive Hadoop FS shell
Date Mon, 08 Feb 2010 19:33:28 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831074#action_12831074

Aaron Kimball commented on HADOOP-6541:


You can indeed shell escape with {{! command}}. No notion of a local working dir, unfortunately.
(Next task: lcd and lls commands!)

Currently you can't launch a Hadoop job within this shell. I think that's another patch /
separate issue -- dealing with the classloader for a jar has enough subtleties associated
with it that I punted for the first version. Eventually this shell should be able to do all
the {{hadoop jar}}, {{hadoop job}}, {{mradmin/dfsadmin}} etc commands too, but there's a lot
of functionality to implement here. All separate tasks in my mind.

In order to establish this code base, my first cut of this system is just focused on implementing
filesystem access.

You're right that 'rmr' is a hack. A better argument parsing system is probably necessary
moving forward to support things like 'rm -[rf]', 'test', etc. Right now all arguments are
treated as filenames in commands like rm; supporting various flags would require using CommandLineParser
and other fun. Not impossible, but not first-cut material. Also, jline's tab-completor makes
it a bit tricky to handle command-specific {{-arguments}}. I'll need to poke around more to
figure out if/how that can be done.

As for cd taking a URL -- technically it already does. But the current design initializes
a FileContext and uses its setWorkingDirectory() method to move around, and then uses FsShell
to do some operations like deletes (which require globbing). FsShell doesn't allow reinitialization
of the FileSystem, so this code checks for changes in the default FS and reinitializes the
structures it uses. We should probably refactor FsShell soon to allow it to work on an arbitrary
FileContext instead of the one it instantiates at construction time (or instantiate a new
FsShell for each operation? That seems unnecessary). But again, refactoring FsShell I think
is beyond the scope of this issue. 

All of these are great ideas but I think that they deserve separate tasks. There are enough
discussion points as to how to implement each of these that I think a single thread would
be confusing. Also getting this committed first would allow different people to attack these
problems in parallel if they'd like. I'll file these follow-up tasks after this is resolved.
Of course, feel free to file your own in the meantime if you'd like :)

Thanks for the suggestions!

> An Interactive Hadoop FS shell
> ------------------------------
>                 Key: HADOOP-6541
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6541
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: HADOOP-6541.2.patch, HADOOP-6541.3.patch, HADOOP-6541.patch
> A shell that allows the user to execute multiple filesystem operations in a single JVM
instance at a prompt.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message