cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ignace Desimpel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-14164) Calling StorageService.loadNewSSTables function results in deadlock with compaction background task
Date Tue, 16 Jan 2018 08:56:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-14164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326897#comment-16326897
] 

Ignace Desimpel commented on CASSANDRA-14164:
---------------------------------------------

I was testing the an automated restore of datafiles of a keyspace in an embedded cassandra
application. Nothing else was running.

Based on what you said about initCf : Before restoring the datafiles, an embedded call to Keyspace.clear()
was made to avoid any open handles, and then to delete and replace all files. That will be
the reason why it is opened the first time. 

I probably can find a way arround it, leaving the theoretical locking issue.

> Calling StorageService.loadNewSSTables function results in deadlock with compaction background
task
> ---------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-14164
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14164
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction, Tools
>         Environment: code
>            Reporter: Ignace Desimpel
>            Assignee: Marcus Eriksson
>            Priority: Major
>             Fix For: 2.2.x, 3.0.x
>
>         Attachments: Stack1.txt
>
>
> Tested on version 2.2.11 (but seems like trunck 3.x is still the same for the related
code path), using nodetool refresh for restoring a snapshot
> Calling StorageService.loadNewSSTables function results in deadlock with compaction background
task.
> because  : 
> From StorageService class , function public void loadNewSSTables(String ksName, String
cfName) a call is made to ColumnFamilyStore class , function public static synchronized void
loadNewSSTables(String ksName, String cfName) and then a call to Keyspace class, function
public static Keyspace open(String keyspaceName)
> getting to the function private static Keyspace open(String keyspaceName, Schema schema,
boolean loadSSTables)
> finally trying to get a lock by synchronized (Keyspace.class)
> So inside the ColumnFamilyStore class lock, there is an attempt to get the lock on the
Keyspace.class
> Now at the same time I have the thread OptionalTasks executing the ColumnFamilyStore.getBackgroundCompactionTaskSubmitter()
task.
> The thread task is also calling Keyspace.open function, already progressed as far as
getting the lock on Keyspace class.
> But then the call also initializes the column families and thus is calling on class ColumnFamilyStore
the public static synchronized ColumnFamilyStore createColumnFamilyStore ...
> Result : the external call on loadNewSSTables blocks the internal compaction background
task.
> So function 1 locks A and then B
> And function 2 locks B and then A
> leading to deadlock (due to incorrect order of locking objects)
> Regards,
> Ignace



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message