Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Cassandra Wiki" for change notification.
The "HadoopSupport" page has been changed by jeremyhanna.
The comment on this change is: Consolidating the cluster configuration stuff..
http://wiki.apache.org/cassandra/HadoopSupport?action=diff&rev1=16&rev2=17
--------------------------------------------------
SlicePredicate predicate = new SlicePredicate().setColumn_names(Arrays.asList(columnName.getBytes()));
ConfigHelper.setSlicePredicate(job.getConfiguration(), predicate);
}}}
- Cassandra's splits are location-aware (this is the nature of the Hadoop [[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/InputSplit.html|InputSplit]]
design). Cassandra gives the Hadoop [[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/JobTracker.html|JobTracker]]
a list of locations with each split of data. That way, the !JobTracker can try to preserve
data locality when assigning tasks to [[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TaskTracker.html|TaskTracker]]s.
Therefore, when using Hadoop alongside Cassandra, it is best to have a !TaskTracker running
on each Cassandra node.
- As of 0.7, configuration for Hadoop no longer resides in your job's specific storage-conf.xml.
See the READMEs in the word_count and pig contrib modules for more details.
+ As of 0.7, configuration for Hadoop no longer resides in your job's specific storage-conf.xml.
See the `README` in the word_count and pig contrib modules for more details.
==== Output To Cassandra ====
@@ -78, +77 @@
== Cluster Configuration ==
- If you would like to configure a Cassandra cluster so that Hadoop may operate over its data,
it's best to overlay a Hadoop cluster over your Cassandra nodes. You'll want to have a separate
server for your Hadoop `namenode`/`jobtracker`. Then install Hadoop `tasktracker`s on each
of your Cassandra nodes. That will allow the `jobtracker` to assign tasks to the Cassandra
nodes that contain data for those tasks. At least one node in your cluster will also need
to be a `datanode`. That's because Hadoop uses HDFS to store information like jar dependencies
for your job, static data (like stop words for a word count), and things like that - it's
the distributed cache. It's a very small amount of data but the Hadoop cluster needs it to
run properly.
+ If you would like to configure a Cassandra cluster so that Hadoop may operate over its data,
it's best to overlay a Hadoop cluster over your Cassandra nodes. You'll want to have a separate
server for your Hadoop namenode/`JobTracker`. Then install a Hadoop `TaskTracker` on each
of your Cassandra nodes. That will allow the `Jobtracker` to assign tasks to the Cassandra
nodes that contain data for those tasks. At least one node in your cluster will also need
to be a datanode. That's because Hadoop uses HDFS to store information like jar dependencies
for your job, static data (like stop words for a word count), and things like that - it's
the distributed cache. It's a very small amount of data but the Hadoop cluster needs it to
run properly.
- The nice thing about having `tasktracker`s on every node is that 1, you get data locality
and 2, your analytics engine scales with your data.
+ The nice thing about having a `TaskTracker` on every node is that you get data locality
and your analytics engine scales with your data.
[[#Top|Top]]
|