gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renato Javier Marroquín Mogrovejo (JIRA) <j...@apache.org>
Subject [jira] [Commented] (GORA-210) thread safety
Date Tue, 26 Feb 2013 16:24:12 GMT

    [ https://issues.apache.org/jira/browse/GORA-210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587231#comment-13587231
] 

Renato Javier Marroquín Mogrovejo commented on GORA-210:
--------------------------------------------------------

Hi Roland,

Thank you very much for tracking these errors down! they are difficult ones indeed! (:
So you are saying that we won't have to synchronize? I thought we would still need to do something
like this because now that the buffer is now a synchronized map, we can "synchronize" the
access to it. Please correct me if I am misunderstanding [1] ):

{code}
    synchronized(this.buffer) {
        Set<K> keys = this.buffer.keySet();

       // this duplicates memory footprint
       K[] keyArray = (K[]) keys.toArray();
    }
{code}
                
> thread safety
> -------------
>
>                 Key: GORA-210
>                 URL: https://issues.apache.org/jira/browse/GORA-210
>             Project: Apache Gora
>          Issue Type: Bug
>          Components: storage-cassandra
>    Affects Versions: 0.2
>         Environment: nutch 2.1 / cassandra 1.2.1 / gora-cassandra 0.2 / gora-core 0.2.1
> running fetch with parse=true
> fetcher.threads.per.queue>1
>            Reporter: Roland
>            Priority: Critical
>              Labels: patch
>         Attachments: GORA-210.patch
>
>
> This is the result of debugging one of my issues described in NUTCH-1534.
> I think there is a wrong assumpation about thread safety of LinkedHashMap, it is not
enough to not iterate over the buffer (which is a LinkedHashMap).
> My patch fixes this error for me:
> java.util.ConcurrentModificationException
>         at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
>         at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:405)
>         at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
>         at org.apache.gora.cassandra.store.CassandraStore.flush(CassandraStore.java:200)
>         at org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:65)
>         at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:587)
>         at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>         at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.output(FetcherReducer.java:664)
>         at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:534)
> It may not be perfect from a performance point of view...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message