sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Chitre (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SQOOP-1301) 'run' method in Sqoop.java is not thread-safe
Date Thu, 27 Mar 2014 01:02:04 GMT
Ajay Chitre created SQOOP-1301:

             Summary: 'run' method in Sqoop.java is not thread-safe
                 Key: SQOOP-1301
                 URL: https://issues.apache.org/jira/browse/SQOOP-1301
             Project: Sqoop
          Issue Type: Bug
    Affects Versions: 1.4.4
            Reporter: Ajay Chitre
             Fix For: 1.4.5

It seems the ‘run’ method in Sqoop is not thread-safe.  When multiple Sqoop jobs are triggered
exactly at the same time, they end up stepping on each other.  Here’s what seems to be the
problem:  The ‘run’ method calls tool.parseArguments which in turn calls ConfigurationHelper.parseGenericOptions
which in turn creates a new instance of GenericOptionsParser.  The constructor for GenericOptionsParser
is not thread-safe because it ends up calling ‘buildGeneralOptions’ which uses OptionBuilder.withArgName.
 This method uses instance variables thereby making it thread unsafe.

The way we’ve got around it is by creating a ‘Lock’ object.  This seems to be working.
 If there’s a better way, please let us know.  If not, please consider adding this feature.
 We can create a patch if there’s an interest.  Thanks.

Ajay Chitre (achitre@cisco.com)
Virendra Singh (virsingh@cisco.com)

Anyway, here’s what we’ve done:

1)  Created a class that extends Sqoop.java
public class DlSqoop extends Sqoop {

2)  Created a Lock object:

    private static Lock monitor = new ReentrantLock();

3)  Overridden ‘run’ method as follows:

  public int run(String [] args) {
    if (options.getConf() == null) {
      // Configuration wasn't initialized until after the ToolRunner
      // got us to this point. ToolRunner gave Sqoop itself a Conf
      // though.

    try {
      options = tool.parseArguments(args, null, options, false);
    } catch (Exception e) {
      // Couldn't parse arguments.
      // Log the stack trace for this exception
      LOG.debug(e.getMessage(), e);
      // Print exception message.
      // Print the tool usage message and exit.
      ToolOptions toolOpts = new ToolOptions();
      return 1; // Exit on exception here.
       } finally {
    return tool.run(options);

This message was sent by Atlassian JIRA

View raw message