sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veena Basavaraj (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SQOOP-1762) Unit tests in Sqoop should use MRUnit for testing the execution engine
Date Tue, 18 Nov 2014 23:00:34 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Veena Basavaraj updated SQOOP-1762:
-----------------------------------
    Description: 
Running unit tests {code}TestMatching{code} and {code}TestMapreduce{code} tries to run the
local hadoop runner environment.

Unit tests should not rely on external dependencies such as MR as well, Either we mock or
use a {code} MRUnit {code} to test the MR execution code

The following exception was generated when trying to run unit tests from outside of the cloudera
VPN.

{code}
499  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Picked org.apache.hadoop.mapred.LocalClientProtocolProvider
as the ClientProtocolProvider

499  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:vbasavaraj
(auth:SIMPLE) from:org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:161)

500  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:vbasavaraj
(auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)

508  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedActionException
as:vbasavaraj (auth:SIMPLE) cause:java.net.UnknownHostException: vbasavaraj.att.net: vbasavaraj.att.net:
nodename nor servname provided, or not known

Testing with Schemas

FROM: Schema{name=FROM-1,columns=[

FixedPoint{name=1,nullable=true,type=FIXED_POINT,byteSize=null,unsigned=null},

FloatingPoint{name=2,nullable=true,type=FLOATING_POINT,byteSize=null},

Text{name=3,nullable=true,type=TEXT,charSize=null}]}

TO: Schema{name=TO-2,columns=[

FixedPoint{name=1,nullable=true,type=FIXED_POINT,byteSize=null,unsigned=null},

FloatingPoint{name=2,nullable=true,type=FLOATING_POINT,byteSize=null}]}

532  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:vbasavaraj
(auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.connect(Job.java:1250)

533  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Trying ClientProtocolProvider : org.apache.hadoop.mapred.YarnClientProtocolProvider

533  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Cannot pick org.apache.hadoop.mapred.YarnClientProtocolProvider
as the ClientProtocolProvider - returned null protocol

533  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Trying ClientProtocolProvider : org.apache.hadoop.mapred.LocalClientProtocolProvider

534  [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics  - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

535  [main] INFO  org.apache.hadoop.metrics.MetricsUtil  - Unable to obtain hostName

java.net.UnknownHostException: vbasavaraj.att.net: vbasavaraj.att.net: nodename nor servname
provided, or not known

at java.net.InetAddress.getLocalHost(InetAddress.java:1473)

at org.apache.hadoop.metrics.MetricsUtil.getHostName(MetricsUtil.java:95)

at org.apache.hadoop.metrics.MetricsUtil.createRecord(MetricsUtil.java:84)

at org.apache.hadoop.mapred.LocalJobRunnerMetrics.<init>(LocalJobRunnerMetrics.java:44)

at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:714)

at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:707)

at org.apache.hadoop.mapred.LocalClientProtocolProvider.create(LocalClientProtocolProvider.java:42)

at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)

at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)

at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)

at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1255)

at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1251)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)

at org.apache.hadoop.mapreduce.Job.connect(Job.java:1250)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:1279)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)

at org.apache.sqoop.job.util.MRJobTestUtil.runJob(MRJobTestUtil.java:60)

at org.apache.sqoop.job.TestMatching.testSchemaMatching(TestMatching.java:137)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)

at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

at org.junit.runners.Suite.runChild(Suite.java:127)

at org.junit.runners.Suite.runChild(Suite.java:26)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)

at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)

at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

Caused by: java.net.UnknownHostException: vbasavaraj.att.net: nodename nor servname provided,
or not known

at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)

at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)

at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)

at java.net.InetAddress.getLocalHost(InetAddress.java:1469)

... 50 more

536  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Picked org.apache.hadoop.mapred.LocalClientProtocolProvider
as the ClientProtocolProvider

536  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:vbasavaraj
(auth:SIMPLE) from:org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:161)

537  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:vbasavaraj
(auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)


{code}

> Unit tests  in Sqoop should use MRUnit for testing the execution engine
> -----------------------------------------------------------------------
>
>                 Key: SQOOP-1762
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1762
>             Project: Sqoop
>          Issue Type: Sub-task
>          Components: test
>            Reporter: Veena Basavaraj
>             Fix For: 2.0.0
>
>
> Running unit tests {code}TestMatching{code} and {code}TestMapreduce{code} tries to run
the local hadoop runner environment.
> Unit tests should not rely on external dependencies such as MR as well, Either we mock
or use a {code} MRUnit {code} to test the MR execution code
> The following exception was generated when trying to run unit tests from outside of the
cloudera VPN.
> {code}
> 499  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Picked org.apache.hadoop.mapred.LocalClientProtocolProvider
as the ClientProtocolProvider
> 499  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction
as:vbasavaraj (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:161)
> 500  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction
as:vbasavaraj (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> 508  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedActionException
as:vbasavaraj (auth:SIMPLE) cause:java.net.UnknownHostException: vbasavaraj.att.net: vbasavaraj.att.net:
nodename nor servname provided, or not known
> Testing with Schemas
> FROM: Schema{name=FROM-1,columns=[
> FixedPoint{name=1,nullable=true,type=FIXED_POINT,byteSize=null,unsigned=null},
> FloatingPoint{name=2,nullable=true,type=FLOATING_POINT,byteSize=null},
> Text{name=3,nullable=true,type=TEXT,charSize=null}]}
> TO: Schema{name=TO-2,columns=[
> FixedPoint{name=1,nullable=true,type=FIXED_POINT,byteSize=null,unsigned=null},
> FloatingPoint{name=2,nullable=true,type=FLOATING_POINT,byteSize=null}]}
> 532  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction
as:vbasavaraj (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.connect(Job.java:1250)
> 533  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Trying ClientProtocolProvider
: org.apache.hadoop.mapred.YarnClientProtocolProvider
> 533  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Cannot pick org.apache.hadoop.mapred.YarnClientProtocolProvider
as the ClientProtocolProvider - returned null protocol
> 533  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Trying ClientProtocolProvider
: org.apache.hadoop.mapred.LocalClientProtocolProvider
> 534  [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics  - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized
> 535  [main] INFO  org.apache.hadoop.metrics.MetricsUtil  - Unable to obtain hostName
> java.net.UnknownHostException: vbasavaraj.att.net: vbasavaraj.att.net: nodename nor servname
provided, or not known
> at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
> at org.apache.hadoop.metrics.MetricsUtil.getHostName(MetricsUtil.java:95)
> at org.apache.hadoop.metrics.MetricsUtil.createRecord(MetricsUtil.java:84)
> at org.apache.hadoop.mapred.LocalJobRunnerMetrics.<init>(LocalJobRunnerMetrics.java:44)
> at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:714)
> at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:707)
> at org.apache.hadoop.mapred.LocalClientProtocolProvider.create(LocalClientProtocolProvider.java:42)
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
> at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
> at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
> at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1255)
> at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1251)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.connect(Job.java:1250)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1279)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.sqoop.job.util.MRJobTestUtil.runJob(MRJobTestUtil.java:60)
> at org.apache.sqoop.job.TestMatching.testSchemaMatching(TestMatching.java:137)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
> Caused by: java.net.UnknownHostException: vbasavaraj.att.net: nodename nor servname provided,
or not known
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
> at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
> ... 50 more
> 536  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Picked org.apache.hadoop.mapred.LocalClientProtocolProvider
as the ClientProtocolProvider
> 536  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction
as:vbasavaraj (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:161)
> 537  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction
as:vbasavaraj (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message