whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joris Poort <gpo...@gmail.com>
Subject Re: EC2 Custom AMI's connection issue
Date Thu, 25 Aug 2011 05:13:47 GMT
Interestingly, as mentioned before I can ssh in using my keypair used
to create the custom AMI.  So I'm thinking that it still has to do
with whirr not being able to get the right keypair access with the
local id_rsa.

Thanks again for any help,

Joris

On Wed, Aug 24, 2011 at 10:11 PM, Joris Poort <gpoort@gmail.com> wrote:
> Thanks guys.  I guess I was using "ubuntu" as user incorrectly, now
> adjusted to local-user "gpoort" I'm still not getting through.  See
> output below from -vv (sorry about the long email).
>
> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
> ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: Applying options for *
> debug2: ssh_connect: needpriv 0
> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
> [184.72.86.108] port 22.
> debug1: Connection established.
> debug2: key_type_from_name: unknown key type '-----BEGIN'
> debug2: key_type_from_name: unknown key type '-----END'
> debug1: identity file .ssh/id_rsa type 1
> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
> debug1: identity file .ssh/id_rsa-cert type -1
> debug1: Remote protocol version 2.0, remote software version
> OpenSSH_5.3p1 Debian-3ubuntu7
> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
> debug1: Enabling compatibility mode for protocol 2.0
> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
> debug2: fd 3 setting O_NONBLOCK
> debug1: SSH2_MSG_KEXINIT sent
> debug1: SSH2_MSG_KEXINIT received
> debug2: kex_parse_kexinit:
> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
> debug2: kex_parse_kexinit:
> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit: first_kex_follows 0
> debug2: kex_parse_kexinit: reserved 0
> debug2: kex_parse_kexinit:
> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit: none,zlib@openssh.com
> debug2: kex_parse_kexinit: none,zlib@openssh.com
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit: first_kex_follows 0
> debug2: kex_parse_kexinit: reserved 0
> debug2: mac_setup: found hmac-md5
> debug1: kex: server->client aes128-ctr hmac-md5 none
> debug2: mac_setup: found hmac-md5
> debug1: kex: client->server aes128-ctr hmac-md5 none
> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
> debug2: dh_gen_key: priv key bits set: 138/256
> debug2: bits set: 502/1024
> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
> matches the RSA host key.
> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
> debug2: bits set: 497/1024
> debug1: ssh_rsa_verify: signature correct
> debug2: kex_derive_keys
> debug2: set_newkeys: mode 1
> debug1: SSH2_MSG_NEWKEYS sent
> debug1: expecting SSH2_MSG_NEWKEYS
> debug2: set_newkeys: mode 0
> debug1: SSH2_MSG_NEWKEYS received
> debug1: Roaming not allowed by server
> debug1: SSH2_MSG_SERVICE_REQUEST sent
> debug2: service_accept: ssh-userauth
> debug1: SSH2_MSG_SERVICE_ACCEPT received
> debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
> debug1: Authentications that can continue: publickey
> debug1: Next authentication method: publickey
> debug1: Offering RSA public key: .ssh/id_rsa
> debug2: we sent a publickey packet, wait for reply
> debug1: Authentications that can continue: publickey
> debug2: we did not send a packet, disable method
> debug1: No more authentication methods to try.
> Permission denied (publickey).
> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
> gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: Applying options for *
> debug2: ssh_connect: needpriv 0
> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
> [184.72.86.108] port 22.
> debug1: Connection established.
> debug2: key_type_from_name: unknown key type '-----BEGIN'
> debug2: key_type_from_name: unknown key type '-----END'
> debug1: identity file .ssh/id_rsa type 1
> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
> debug1: identity file .ssh/id_rsa-cert type -1
> debug1: Remote protocol version 2.0, remote software version
> OpenSSH_5.3p1 Debian-3ubuntu7
> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
> debug1: Enabling compatibility mode for protocol 2.0
> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
> debug2: fd 3 setting O_NONBLOCK
> debug1: SSH2_MSG_KEXINIT sent
> debug1: SSH2_MSG_KEXINIT received
> debug2: kex_parse_kexinit:
> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
> debug2: kex_parse_kexinit:
> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit: first_kex_follows 0
> debug2: kex_parse_kexinit: reserved 0
> debug2: kex_parse_kexinit:
> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit: none,zlib@openssh.com
> debug2: kex_parse_kexinit: none,zlib@openssh.com
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit: first_kex_follows 0
> debug2: kex_parse_kexinit: reserved 0
> debug2: mac_setup: found hmac-md5
> debug1: kex: server->client aes128-ctr hmac-md5 none
> debug2: mac_setup: found hmac-md5
> debug1: kex: client->server aes128-ctr hmac-md5 none
> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
> debug2: dh_gen_key: priv key bits set: 145/256
> debug2: bits set: 505/1024
> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
> matches the RSA host key.
> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
> debug2: bits set: 495/1024
> debug1: ssh_rsa_verify: signature correct
> debug2: kex_derive_keys
> debug2: set_newkeys: mode 1
> debug1: SSH2_MSG_NEWKEYS sent
> debug1: expecting SSH2_MSG_NEWKEYS
> debug2: set_newkeys: mode 0
> debug1: SSH2_MSG_NEWKEYS received
> debug1: Roaming not allowed by server
> debug1: SSH2_MSG_SERVICE_REQUEST sent
> debug2: service_accept: ssh-userauth
> debug1: SSH2_MSG_SERVICE_ACCEPT received
> debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
> debug1: Authentications that can continue: publickey
> debug1: Next authentication method: publickey
> debug1: Offering RSA public key: .ssh/id_rsa
> debug2: we sent a publickey packet, wait for reply
> debug1: Authentications that can continue: publickey
> debug2: we did not send a packet, disable method
> debug1: No more authentication methods to try.
> Permission denied (publickey).
>
> On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <savu.andrei@gmail.com> wrote:
>> You should be able to login user the same user that is running Whirr.
>>
>> ssh -i ~/.ssh/id_rsa local-user@server
>>
>> Permission denied most of the time means invalid key, user pair.
>>
>> It should be ok to use the keys generate by default.
>>
>> -- Andrei Savu / andreisavu.ro
>>
>> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gpoort@gmail.com> wrote:
>>> I'm just using the defaults.  But you may be onto the problem here,
>>> when I try to ssh into the node using:
>>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>>
>>> I get a permission denied.
>>>
>>> Any idea how to fix this?  Should I create my own set of SSH key pairs?
>>>
>>> Thanks again,
>>>
>>> Joris
>>>
>>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <savu.andrei@gmail.com> wrote:
>>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>>
>>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa &
id_rsa.pub.
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gpoort@gmail.com> wrote:
>>>>> I think you're probably right its an auth issue - although I was
>>>>> expecting a more direct/clear error message if the keypair wasn't
>>>>> working.
>>>>>
>>>>> I created the AMI by taking an EBS snapshot then converting to
>>>>> instance-store.  I've tried both the ebs back ami and instance-store
>>>>> with the same results.  My understanding is that the keypair used to
>>>>> create the AMI is generally one of the accepted keys in addition to
>>>>> the key pair used to launch the instance created by jclouds.  I'm not
>>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>>> anywhere that can be used to test this?
>>>>>
>>>>> Thanks again for your help,
>>>>>
>>>>> Joris
>>>>>
>>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <savu.andrei@gmail.com>
wrote:
>>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates
it's
>>>>>> own key pair using the local SSH keys as specified in the properties
>>>>>> file.
>>>>>>
>>>>>> You've created the custom ami by taking an EBS snapshot? Can you
use
>>>>>> that custom ami with a different key pair?
>>>>>>
>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gpoort@gmail.com>
wrote:
>>>>>>> Andrei - thanks for the response!
>>>>>>>
>>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>>> machine (I'm executing whirr via ubuntu virtual machine).  I've
tried
>>>>>>> both spot instances and regular instances and am getting the
same
>>>>>>> behavior.
>>>>>>>
>>>>>>> Full output on terminal looks like (lines between "Starting 1
node"
>>>>>>> and "Dying because" are not always there):
>>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException:
Broken
>>>>>>> transport; encountered EOF
>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException:
Broken
>>>>>>> transport; encountered EOF
>>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>>> Broken transport; encountered EOF
>>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered
EOF
>>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport;
encountered EOF
>>>>>>>        at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>
>>>>>>> Last few entries on whirr.log:
>>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2)
>>
>>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>>> securityGroupIds=[],
>>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4)
<<
>>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2)
<<
>>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4)
<<
>>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2)
<<
>>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread
8) >>
>>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000
seconds
>>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread
7) >>
>>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>>> seconds
>>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread
7) <<
>>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread
8) <<
>>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>>
>>>>>>> After ssh onto node,  didn't find any logs in /tmp.
>>>>>>>
>>>>>>> Thanks again for any help on this!
>>>>>>>
>>>>>>> Joris
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <savu.andrei@gmail.com>
wrote:
>>>>>>>> I suspect this is an authentication issue. How do you login
to the custom AMI?
>>>>>>>>
>>>>>>>> Also check whirr.log for more details and on the remote machines
look
>>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>>
>>>>>>>> I know from IRC that you are using spot instances. Are you
seeing the
>>>>>>>> same behavior with regular ones?
>>>>>>>>
>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gpoort@gmail.com>
wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm new to whirr and I'm running custom AMI configuration
(application
>>>>>>>>> installed on working canonical image).  Executing with
whirr 0.6.0
>>>>>>>>> everything executes fine until I get the following error:
>>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read
timed out"
>>>>>>>>>
>>>>>>>>> The instances are running fine, I can ssh into them,
but the whirr
>>>>>>>>> code stalls and I get the above error (2x number of instances),
no
>>>>>>>>> proxy shell is created.  If I run the exact same code
with vanilla
>>>>>>>>> canonical images I don't have any issues.
>>>>>>>>>
>>>>>>>>> Anyone have any ideas on things to test, debug or work
around this?
>>>>>>>>> Would really appreciate it!
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Joris
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message