cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Morton (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-2981) Provide Hadoop read access to Counter Columns.
Date Mon, 01 Aug 2011 11:13:11 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron Morton updated CASSANDRA-2981:
------------------------------------

    Attachment: 0001-2981-hadoop-counters-input.patch

0001-2981-hadoop-counters-input.patch modifies the CFRR to turn CounterColumns returned through
the thrift API into o.a.c.db.Column instances. 

Could not use the CounterColumn as the CounterContext needs to read the node ID, and this
requires the StorageService to be running and access to cassandra.yaml.

It's not great, but the full CounterColumn should not be needed as Hadoop is read only access.
Let me know it's too hacky.

Also added another test to the hadoop_word_count example that sums the counter columns in
a row.  

> Provide Hadoop read access to Counter Columns.
> ----------------------------------------------
>
>                 Key: CASSANDRA-2981
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2981
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Hadoop
>    Affects Versions: 0.8.2
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>            Priority: Minor
>         Attachments: 0001-2981-hadoop-counters-input.patch
>
>
> o.a.c.Hadoop.ColumnFamilyRecordReader does not test for counter columns, which are different
objects in the ColumnOrSuperColumn struct. Currently it raises an error as it thinks it's
a super column 
> {code:java}
> 2011-07-26 17:23:34,376 ERROR CliDriver (SessionState.java:printError(343)) - Failed
with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> 	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:341)
> 	at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:133)
> 	at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1114)
> 	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftifySuper(ColumnFamilyRecordReader.java:303)
> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftify(ColumnFamilyRecordReader.java:297)
> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:288)
> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:177)
> 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:136)
> 	at org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:153)
> 	at org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:111)
> 	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:326)
> 	... 10 more
> {code}
> My plan is to return o.a.c.db.CounterColumn objects just like the o.a.c.db.Column and
SuperColumn that are returned.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message