sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bejoy ks <bejo...@gmail.com>
Subject Re: Sqoop import failed
Date Thu, 13 Mar 2014 03:00:09 GMT
Hi Krishnan

I second Venkat's thoughts here. The TT nodes can't make a connection to DB
server due to some firewall blockage.

Gateway node is more like a client  that has access to the cluster. You log
on to this machine from your dev environment, to submit jobs into cluster
as your dev environment doesn't have direct access to cluster.

When the TaskTracker nodes/map tasks makes a db connection, it is more or
less of a direct connection that is made. That connection won't take an
extra hop of TT->GateWay->DBServer.

Sqoop to work as desired the db server should be reachable from all nodes
in the cluster. You need to touch base with your admins and make some
exception to the firewall may be to permit this connection alone.


On Wed, Mar 12, 2014 at 6:55 PM, Krishnan Narayanan <
krishnan.smile@gmail.com> wrote:

> hi Venkat,
>
> Thanks for your reply, it is the gateway node for which we have access to
> the DB host machine.So I believe all the other nodes will also have access
> through gateway node.
>
> Thx
> Krishnan
>
>
> On Wed, Mar 12, 2014 at 5:43 PM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
>
>> Krishnan
>>
>> It looks like you have access to the Sql Server from the client
>> machine(s) only (where you run the Sqoop command).   Sqoop launches mappers
>> that run in the task nodes and you have to enable DB access from those
>> hosts also.
>>
>> Thanks
>>
>> Venkat
>>
>>
>> On Wed, Mar 12, 2014 at 3:27 PM, Krishnan Narayanan <
>> krishnan.smile@gmail.com> wrote:
>>
>>> syslog logs
>>>
>>> 2014-03-11 14:08:55,398 WARN mapreduce.Counters: Group
>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
>>> org.apache.hadoop.mapreduce.TaskCounter instead
>>> 2014-03-11 14:08:55,971 WARN org.apache.hadoop.conf.Configuration:
>>> session.id is deprecated. Instead, use dfs.metrics.session-id
>>> 2014-03-11 14:08:55,972 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>>> Initializing JVM Metrics with processName=MAP, sessionId=
>>> 2014-03-11 14:08:56,369 INFO org.apache.hadoop.util.ProcessTree: setsid
>>> exited with exit code 0
>>> 2014-03-11 14:08:56,373 INFO org.apache.hadoop.mapred.Task:  Using
>>> ResourceCalculatorPlugin :
>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7c3ded93
>>> 2014-03-11 14:09:11,760 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>> 2014-03-11 14:09:11,763 WARN org.apache.hadoop.mapred.Child: Error
>>> running child
>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>> com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to
>>> the host QYSPRDESTSQL06.net, port 20101 has failed. Error: "connect timed
>>> out. Verify the connection properties. Make sure that an instance of SQL
>>> Server is running on the host and accepting TCP/IP connections at the port.
>>> Make sure that TCP connections to the port are not blocked by a firewall.".
>>>  at
>>> org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
>>> at
>>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
>>>  at
>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:635)
>>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>>  at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>  at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>> Caused by: java.lang.RuntimeException:
>>> com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to
>>> the host QYSPRDESTSQL36.ie..net, port 20101 has failed. Error: "connect
>>> timed out. Verify the connection properties. Make sure that an instance of
>>> SQL Server is running on the host and accepting TCP/IP connections at the
>>> port. Make sure that TCP connections to the port are not blocked by a
>>> firewall.".
>>>  at
>>> org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:193)
>>> at
>>> org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:162)
>>>  ... 9 more
>>> Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP
>>> connection to the host QYSPRDESTSQL36.ie..net, port 20101 has failed.
>>> Error: "connect timed out. Verify the connection properties. Make sure that
>>> an instance of SQL Server is running on the host and accepting TCP/IP
>>> connections at the port. Make sure that TCP connections to the port are not
>>> blocked by a firewall.".
>>>  at
>>> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:190)
>>> at
>>> com.microsoft.sqlserver.jdbc.SQLServerException.ConvertConnectExceptionToSQLServerException(SQLServerException.java:241)
>>>  at
>>> com.microsoft.sqlserver.jdbc.SocketFinder.findSocket(IOBuffer.java:2243)
>>> at com.microsoft.sqlserver.jdbc.TDSChannel.open(IOBuffer.java:491)
>>>  at
>>> com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1309)
>>> at
>>> com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:991)
>>>  at
>>> com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:827)
>>> at
>>> com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1012)
>>>  at java.sql.DriverManager.getConnection(DriverManager.java:571)
>>> at java.sql.DriverManager.getConnection(DriverManager.java:215)
>>>  at
>>> org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:294)
>>> at
>>> org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:187)
>>>  ... 10 more
>>> 2014-03-11 14:09:11,770 INFO org.apache.hadoop.mapred.Task: Runnning
>>> cleanup for the task
>>> 2014-03-11 14:09:11,782 ERROR
>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>> as:sys_apd_sl (auth:SIMPLE)
>>> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>>> Operation category READ is not supported in state standby
>>> 2014-03-11 14:09:11,783 WARN org.apache.hadoop.ipc.Client: Exception
>>> encountered while connecting to the server :
>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>>> Operation category READ is not supported in state standby
>>> 2014-03-11 14:09:11,783 ERROR
>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>> as:sys_apd_sl (auth:SIMPLE)
>>> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>>> Operation category READ is not supported in state standby
>>>
>>>
>>>
>>> On Wed, Mar 12, 2014 at 3:20 PM, bejoy ks <bejoyks@gmail.com> wrote:
>>>
>>>> Hi Krishnan
>>>>
>>>> Can you please post the complete verbose logs on sqoop command
>>>> execution?
>>>>
>>>> In the mean time you can verify whether this SQL Server is reachable
>>>> from all nodes(Task Trackers) in your cluster.
>>>>
>>>>
>>>> On Wed, Mar 12, 2014 at 10:36 AM, Krishnan Narayanan <
>>>> krishnan.smile@gmail.com> wrote:
>>>>
>>>>> Hi ,
>>>>>
>>>>> I am using cloudera and sqoop 1.4.3, I trying to connect mssql server
>>>>> database for getting the data.I have the sqljdbc4.jar file in lib.
>>>>> I am able to connect and view the tables using sqoop list-tables but
>>>>> unable to import.
>>>>> I found that the sqoop 1.4.1 has this bug and fixed in the next
>>>>> versions. can someone help me in this.
>>>>>
>>>>> The below code works and list all the table in DB.
>>>>>
>>>>> sqoop list-tables -Dmapred.job.queue.name=foundation  --connect
>>>>> "jdbc:sqlserver://QYSPRDESTSQL06.net:20101;database=profiledb" \
>>>>> --username "rpuser" \
>>>>> --password "xxxxxxxxxxxx"
>>>>>
>>>>> The below code throws error. TCP/IP error unable to connect host
>>>>> machine check network setting
>>>>>
>>>>> sqoop import -libjars /usr/lib/sqoop/lib/sqljdbc4.jar -
>>>>> Dmapred.job.queue.name=foundation --connect
>>>>> "jdbc:sqlserver://QYSPRDESTSQL06.net:20101;database=profiledb" \
>>>>> --username rpuser \
>>>>> --password xxxxxxxxxxxxx \
>>>>> --fields-terminated-by 'รพ' \
>>>>> --target-dir /app/apd/sys_apd_sl/test/mytable \
>>>>> --table mytable
>>>>>
>>>>> thanks
>>>>> Krishnan
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards
>>>>        Bejoy
>>>>
>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>


-- 
Regards
       Bejoy

Mime
View raw message