crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Igor Bernstein (JIRA)" <>
Subject [jira] [Created] (CRUNCH-600) Credentials are lost in local runner mode
Date Sun, 10 Apr 2016 19:41:25 GMT
Igor Bernstein created CRUNCH-600:

             Summary: Credentials are lost in local runner mode
                 Key: CRUNCH-600
             Project: Crunch
          Issue Type: Bug
            Reporter: Igor Bernstein
            Priority: Minor

The credentials attached to a job are lost when CrunchOutputs builds a new Job in getJob.
For example, the following scenario fails:

class MyTarget implements MapReduceTarget {
  void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String name)
    job.getCredentials().addSecretKey("myPassword", "secret".getBytes());
    CrunchOutputs.addNamedOutput(job, name, FormatBundle.forOutput(MyOutputFormat.class),

class MyOutputFormat extends OutputFormat<Object,Object> {
  public RecordWriter<Object, Object> getRecordWriter(TaskAttemptContext context) throws
IOException, InterruptedException {
    Assert.equals("secret".getBytes(), context.getCredentials().getSecretKey("myPassword"));

This happens because CrunchOutputs.getJob copies the baseConf using the base Configuration
class and the base Configuration class doesn't have a notion of Credentials, so the credentials
get dropped.

This is issue only manifests itself when using the local runner. In an actual hadoop cluster
the credentials will be inherited from UserGroupInformation.

This message was sent by Atlassian JIRA

View raw message