chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lewis John McGibbney (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CHUKWA-734) Gora Storage System for Chuckwa Logs
Date Mon, 23 Feb 2015 05:03:12 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14332963#comment-14332963
] 

Lewis John McGibbney commented on CHUKWA-734:
---------------------------------------------

Hi [~eyang]
bq. I got an error for running TestHBaseWriter unit test:
My tests hang on {code}Running org.apache.hadoop.chukwa.datacollection.sender.TestAcksOnFailure{code},
however when I run just the HBaseWriter test, I also get an error
{code}
  5 testWriters(org.apache.hadoop.chukwa.datacollection.writer.TestHBaseWriter)  Time elapsed:
0.026 sec  <<< ERROR!
  6 java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/Stoppable
  7         at java.lang.ClassLoader.defineClass1(Native Method)
  8         at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
  9         at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
 10         at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
 11         at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
 12         at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
 13         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
 14         at java.security.AccessController.doPrivileged(Native Method)
 15         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
 16         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
 17         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
 18         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
 19         at java.lang.ClassLoader.defineClass1(Native Method)
 20         at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
 21         at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
 22         at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
 23         at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
 24         at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
 25         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
 26         at java.security.AccessController.doPrivileged(Native Method)
 27         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
 28         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
 29         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
 30         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
 31         at java.lang.ClassLoader.defineClass1(Native Method)
 32         at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
 33         at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
 34         at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
 35         at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
 36         at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
 37         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
 38         at java.security.AccessController.doPrivileged(Native Method)
 39         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
 40         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
 41         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
 42         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
 43         at org.apache.hadoop.chukwa.datacollection.writer.TestHBaseWriter.setUp(TestHBaseWriter.java:65)
 44         at junit.framework.TestCase.runBare(TestCase.java:132)
 45         at junit.framework.TestResult$1.protect(TestResult.java:110)
 46         at junit.framework.TestResult.runProtected(TestResult.java:128)
 47         at junit.framework.TestResult.run(TestResult.java:113)
 48         at junit.framework.TestCase.run(TestCase.java:124)
 49         at junit.framework.TestSuite.runTest(TestSuite.java:243)
 50         at junit.framework.TestSuite.run(TestSuite.java:238)
 51         at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
 52         at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
 53         at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
 54         at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
 55         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}

bq. Does Gora support Hadoop1?
Yes we have shim layer support for Hadoop 1.2.1 and 2.5.2. We just need to configure it properly.

bq. We probably need to setup another profile for enabling Hadoop 1 vs Hadoop 2.
Most likely Eric. I will be looking in to this. I see that there are existing profiles for
0.96.2-hadoop1 and 0.94.9
I am therefore thinking that we could potentially set up profiles for Gora-supported backends
for HBase 0.98.8-hadoop2, Cassandra 2.0.2, Solr 4.10.3, Mongodb 2.6 and Accumulo 1.5.1. Before
I do this right now however I need clarification on 
 * what version of HBase is currently activated by default in Gora?
 * how and where is this defined as default within pom.xml?

bq. Can Gora map sequence ID value to column name in HBase?
Mmmmmm.... I am not sure about this. Reasoning is as follows: currently we define AHEAD OF
MAPPING
 * a table name
 * columns within this table (columns can also have [optional params like compression, bloom
filters, etc|http://gora.apache.org/current/gora-hbase.html#gora-hbase-mappings])
 * we then map out data beans to these definitions

It would appear to me that for us to be able to map sequenceID to a column name, we would
1) want to dynamically create many many columns over time directly dependent on the number
of data chunks we get, is this correct? 2) once we get a new incoming data chunk we would
wish to dynamically generate a new column within the existing table with the sequenceID as
the column name, is this correct?

Thanks


> Gora Storage System for Chuckwa Logs
> ------------------------------------
>
>                 Key: CHUKWA-734
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-734
>             Project: Chukwa
>          Issue Type: New Feature
>          Components: Data Collection
>    Affects Versions: 0.6.0
>            Reporter: Lewis John McGibbney
>             Fix For: 0.6.0
>
>         Attachments: CHUKWA-734.patch
>
>   Original Estimate: 5h
>  Remaining Estimate: 5h
>
> I would like to build a Gora-backed log-to-datastore module for Chuckwa. I am going to
work on this today.
> Gora is an in-memory data modeling and storage abstraction 
> http://gora.apache.org
> Gora powers the Apache Nutch 2.X software which generates a bunch of log data. Having
a Chuckwa monitoring tool for Nutch would be grand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message