hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billy Pearson" <sa...@pearsonwholesale.com>
Subject Re: Help with Map/Reduce program
Date Thu, 11 Jun 2009 18:43:52 GMT
That might be a good idea but you might be able to redesign you layout of 
the table
using a different key then the current one worth barnstorming.

Billy



"llpind" <sonny_heer@hotmail.com> wrote in 
message news:23975432.post@talk.nabble.com...

Sorry I forgot to mention the overflow then overflows into new row keys per
10,000 column entries (or some other split number).



llpind wrote:
>
>
> When is the plan for releasing .20?  This particular issue is really
> important to us.
>
> Stack, I also have another question: The problem we are trying to solve
> doesn't really need the extra layer present in HBase (BigTable) structure
> (RowResult holds row key and a HashMap of column name, value). What we
> really need is a row key which simply holds a set of values.  Essentially
> this is a many-to-many.  I wanted your thoughts on how we can go about
> solving this problem (we can start another post for this if you’d like).
> Is this something HBase can solve, or something that could potentially be
> a HBase fork?  Right now we are still in test mode, and only having to
> deal with millions of columns, but in production (if the company sticks
> with HBase) the columns could be in the billions.  One idea we came up
> with is to have an overflow table… e.g.
>
> For a given row key we list the first 10,000 columns (values in our case),
> and after that we create a column with an overflow id pointing an overflow
> table which is keyed on this id.
>
> This appears it may work, but isn’t the most elegant solution.  I’d
> appreciate input from anyone on this issue.   Please, let me know if you
> need me explain our problem in more detail.
>
>
>
> stack-3 wrote:
>>
>> On Wed, Jun 10, 2009 at 4:52 PM, llpind 
>> <sonny_heer@hotmail.com> wrote:
>>
>>>
>>> Thanks.  I think the problem is I have potentially millions of columns.
>>>
>>
>>> where a given RowResult can hold millions of columns to values.   Thats
>>> why
>>> Map/Reduce is having problems as well (Java Heap exception).  I've upped
>>> mapred.child.java.opts, but problem presists.
>>>
>>
>> See also HBASE-867: https://issues.apache.org/jira/browse/HBASE-867
>> St.Ack
>>
>>
>
>

-- 
View this message in context: 
http://www.nabble.com/Help-with-Map-Reduce-program-tp23952252p23975432.html
Sent from the HBase User mailing list archive at Nabble.com.




Mime
View raw message