cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dominic Williams (JIRA)" <>
Subject [jira] Commented: (CASSANDRA-1423) Cassandra's internal state broke by getting column slices. Error org.apache.thrift.TApplicationException: Internal error processing get_slice afterwards.
Date Mon, 23 Aug 2010 22:25:33 GMT


Dominic Williams commented on CASSANDRA-1423:

Looks like this bug might hopefully be a straightforward case of not cleaning up file handles
(or at least, always opening files afresh, and then not handling the error when you can't
open any more). I should have checked logs. 

>From system.log:

ERROR [pool-1-thread-74] 2010-08-23 23:18:34,749 (line 2651) Internal error
processing get_slice
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
/var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db (Too many open
	at org.apache.cassandra.service.StorageProxy.weakRead(
	at org.apache.cassandra.service.StorageProxy.readProtocol(
	at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(
	at org.apache.cassandra.thrift.CassandraServer.getSlice(
	at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(
	at org.apache.cassandra.thrift.CassandraServer.get_slice(
	at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(
	at org.apache.cassandra.thrift.Cassandra$Processor.process(
	at org.apache.cassandra.thrift.CustomTThreadPoolServer$
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
	at java.util.concurrent.ThreadPoolExecutor$
Caused by: java.util.concurrent.ExecutionException:
/var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db (Too many open
	at java.util.concurrent.FutureTask$Sync.innerGet(
	at java.util.concurrent.FutureTask.get(
	at org.apache.cassandra.service.StorageProxy.weakRead(
	... 11 more
Caused by: /var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db
(Too many open files)
	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(
	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(
	at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(
	at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(
	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(
	at org.apache.cassandra.db.Table.getRow(
	at org.apache.cassandra.db.SliceFromReadCommand.getRow(
	at org.apache.cassandra.service.StorageProxy$
	at java.util.concurrent.FutureTask$Sync.innerRun(
	... 3 more
Caused by: /var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db
(Too many open files)
	at Method)
	... 16 more

> Cassandra's internal state broke by getting column slices. Error org.apache.thrift.TApplicationException:
Internal error processing get_slice afterwards.
> ---------------------------------------------------------------------------------------------------------------------------------------------------------
>                 Key: CASSANDRA-1423
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7 beta 1
>         Environment: Ubuntu 9.04
>            Reporter: Dominic Williams
>            Priority: Critical
>             Fix For: 0.7 beta 2
>         Attachments:
> The attached program (with data that it can import) causes large column slices to be
requested from Cassandra. 
> The program itself uploads school address data to a Cassyndex full text index, and then
allows you to search that. The program simulates someone typing a search into an active search
box, which shows you the matches for the current term as you type. Thus when you enter a search
term such as "cherwell school oxford" actually it performs the searches "c", "ch", "che",
"cher", "cherw" etc
> You can configure the delay between the "keystrokes". If your delay allows the searches
to complete sequentially, you are ok. But if you have a short delay, and searches are created
in parallel, pretty quickly this error will arise - "org.apache.thrift.TApplicationException:
Internal error processing get_slice".
> Once this has occurred all future attempts and getting slices of columns will return
the same error, and your'e only option is to restart Cassandra.
> This looks like some kind of concurrency edge condition bug caused by requesting sufficiently
large intersecting slices in parallel. It may be in other versions too.
> I've been testing on 0.7 B1 using an RP cluster. 
> The attached maven project should pull down the scale7 libraries but if interested you
can find the sources at

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message