metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laurens Vets <laur...@daemon.be>
Subject Re: AWS single_node_vm
Date Mon, 05 Jun 2017 16:20:11 GMT
Hi David,

That works, thanks! Now trying to get this down even more (4 sounds like 
a nice number :p)

On 2017-05-30 17:05, David Lyle wrote:
> This one should do the trick for you.
> 
> Due to how smal_cluster.yml is structured (designed for 10+ nodes)- 
> you'll
> only get 2 data nodes, so you'll have to reduce replication to 2 if you
> don't want to see under-replicated block warnings.
> 
>     - include: tasks/create-hosts.yml host_count=1
> host_type=sensors,ambari_master,ec2,monit
>     - include: tasks/create-hosts.yml host_count=2
> host_type=ambari_slave,search,ec2
>     - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,ec2
>     - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>     - include: tasks/create-hosts.yml host_count=1
> host_type=pcap_server,monit,ambari_slave,web,ec2
> 
> The reason the first one didn't work is because order matters. In the 
> that
> example, the host created by
> 
>     - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
> 
> was ending up in 2 separate hostgroups, which is not allowed. 
> Relocating
> that line solves the problem. I also relocated the web hosts (ES Master 
> and
> Kibana) since there were fewer hosts.
> 
> 
> -D...
> 
> 
> On Tue, May 30, 2017 at 12:49 PM, Laurens Vets <laurens@daemon.be> 
> wrote:
> 
>> I put in what you said previously :)
>> 
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=sensors,ambari_master,ec2,monit
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=pcap_server,monit,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>> - include: tasks/create-hosts.yml host_count=2
>> host_type=ambari_slave,search,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,web,ec2
>> 
>> On 2017-05-30 08:53, David Lyle wrote:
>> 
>>> That's actually a different error than you posted before. That error 
>>> is
>>> Ambari telling you that the blueprint that Ansible submitted is 
>>> invalid.
>>> Likely a result of a required hostgroup not being defined. Can you 
>>> (re?)
>>> post what you've put in amazon-ec2/playbook.yml?
>>> 
>>> -D...
>>> 
>>> 
>>> On Mon, May 29, 2017 at 12:02 PM, Laurens Vets <laurens@daemon.be> 
>>> wrote:
>>> 
>>> Same problem as all the previous ones...
>>>> 
>>>> TASK [ambari_config : Install python-requests]
>>>> *********************************
>>>> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => 
>>>> {"attempts":
>>>> 1, "changed": false, "msg": "", "rc": 0, "results":
>>>> ["python-requests-2.6.0-3.el6.noarch providing python-requests is
>>>> already
>>>> installed"]}
>>>> 
>>>> TASK [ambari_config : check if ambari-server is up on
>>>> ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
>>>> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => 
>>>> {"changed":
>>>> false, "elapsed": 120, "path": null, "port": 8080, "search_regex": 
>>>> null,
>>>> "state": "started"}
>>>> 
>>>> TASK [ambari_config : Deploy cluster with Ambari;
>>>> http://ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
>>>> fatal: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com]: FAILED! 
>>>> =>
>>>> {"changed": false, "failed": true, "msg": "Ambari client exception
>>>> occurred: Could not create cluster: request code 400,
>>>>  request message {\n  \"status\" : 400,\n  \"message\" : \"Topology
>>>> validation failed: org.apache.ambari.server.topol
>>>> ogy.InvalidTopologyException:
>>>> The following hosts are mapped to multiple host groups: [
>>>> ec2-34-208-74-17.us-west-2.compute.amazonaws.com]. Be aware that 
>>>> host
>>>> names are converted to lowercase, case differences do not matter in
>>>> Ambari
>>>> deployments.\"\n}"}
>>>>         to retry, use: --limit @/root/metron/metron-deploymen
>>>> t/amazon-ec2/playbook.retry
>>>> 
>>>> PLAY RECAP 
>>>> ************************************************************
>>>> *********
>>>> ec2-34-208-74-17.us-west-2.compute.amazonaws.com : ok=47   
>>>> changed=31
>>>>  unreachable=0    failed=0
>>>> ec2-34-209-23-182.us-west-2.compute.amazonaws.com : ok=48   
>>>> changed=28
>>>>  unreachable=0    failed=1
>>>> ec2-35-163-138-4.us-west-2.compute.amazonaws.com : ok=41   
>>>> changed=27
>>>>  unreachable=0    failed=0
>>>> ec2-35-163-150-6.us-west-2.compute.amazonaws.com : ok=41   
>>>> changed=27
>>>>  unreachable=0    failed=0
>>>> ec2-35-167-207-80.us-west-2.compute.amazonaws.com : ok=17   
>>>> changed=8
>>>> unreachable=0    failed=0
>>>> ec2-54-70-75-61.us-west-2.compute.amazonaws.com : ok=41   changed=27
>>>>  unreachable=0    failed=0
>>>> localhost                  : ok=18   changed=13   unreachable=0
>>>> failed=0
>>>> 
>>>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>>> 
>>>> 
>>>> On 2017-05-27 04:45, David Lyle wrote:
>>>> 
>>>> For some reason, kibana_hosts is undefined, so the curl command that
>>>>> loads
>>>>> the Kibana dashboard doesn't contain the URL for Kibana. If you 
>>>>> look at
>>>>> small_cluster.yml, you'll see that the Kibana should have been 
>>>>> installed
>>>>> on
>>>>> the first host in the "web" hostgroup. Verify that you placed a 
>>>>> host in
>>>>> that hostgroup (it's the 5th line in Mike's example). If you did, 
>>>>> check
>>>>> that host on Ambari and verify that Kibana was installed, if not, 
>>>>> you'll
>>>>> want to dig through the install log to find out why.
>>>>> 
>>>>> Also- now that I've looked at the example carefully, I noticed that 
>>>>> the
>>>>> Zeppelin hostgroup was duplicated. That's not a problem, because 
>>>>> the 2nd
>>>>> host will be ignored, maybe you caught it when you created your 
>>>>> own.
>>>>> 
>>>>> I would try something like (for a 6 node host cluster):
>>>>> 
>>>>>     - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=sensors,ambari_
>>>>> master,ec2,monit
>>>>>     - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=pcap_server,monit,ec2
>>>>>     - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=ambari_slave,
>>>>> enrichment,metron,ec2,zeppelin
>>>>>     - include: tasks/create-hosts.yml host_count=2
>>>>> host_type=ambari_slave,search,ec2
>>>>>     - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=ambari_slave,web,ec2
>>>>> 
>>>>> -D...
>>>>> 
>>>>> 
>>>>> On Fri, May 26, 2017 at 8:15 PM, Laurens Vets <laurens@daemon.be>
>>>>> wrote:
>>>>> 
>>>>> Different error than before, but it still fails:
>>>>> 
>>>>>> 
>>>>>> TASK [ambari_gather_facts : set_fact] 
>>>>>> ******************************
>>>>>> ************
>>>>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>>>>> {"ansible_facts": {"kibana_hosts": []}, "changed": false}
>>>>>> 
>>>>>> TASK [ambari_gather_facts : debug] ******************************
>>>>>> ***************
>>>>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] => {
>>>>>>     "msg": "zookeeper_port = 2181, zookeeper_hosts = [u'
>>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com'], zookeeper_url

>>>>>> =
>>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com:2181,
>>>>>> kafka_broker_port
>>>>>> = 6667, kafka_broker_hosts = [u'ec2-34-210-18-244.us-west-2
>>>>>> .compute.amazonaws.com', u'ec2-35-164-236-192.us-west-2
>>>>>> .compute.amazonaws.com', u'ec2-52-41-30-101.us-west-2.c
>>>>>> ompute.amazonaws.com'], kafka_broker_url =
>>>>>> ec2-34-210-18-244.us-west-2.co
>>>>>> mpute.amazonaws.com:6667,ec2-35-164-236-192.us-west-2.compu
>>>>>> te.amazonaws.com:6667,ec2-52-41-30-101.us-west-2.compute.
>>>>>> amazonaws.com:6667, metron_hosts = [u'ec2-35-164-236-192.us-west-
>>>>>> 2.compute.amazonaws.com']"
>>>>>> }
>>>>>> 
>>>>>> TASK [load_web_templates : Load ES Templates]
>>>>>> **********************************
>>>>>> changed: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>>>>> {"changed": true, "cmd": ["curl", "-s", "-w", "%{http_code}", 
>>>>>> "-u",
>>>>>> "admin:admin", "-H", "X-Requested-By: ambari", "-X", "POST", "-d",

>>>>>> "{
>>>>>> \"RequestInfo\": { \"context\": \"Install ES Template from REST\",
>>>>>> \"command\": 
>>>>>> \"ELASTICSEARCH_TEMPLATE_INSTALL\"},\"Requests/resource_
>>>>>> filters\":
>>>>>> [{\"service_name\": \"METRON\",\"component_name\":
>>>>>> \"METRON_INDEXING\",\"hosts\" : \"ec2-35-164-236-192.us-west-2
>>>>>> .compute.amazonaws.com\"}]}", "http://ec2-34-210-207-190.us-
>>>>>> west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests"],
>>>>>> "delta": "0:00:00.049367", "end": "2017-05-26 21:33:34.767106",
>>>>>> "failed":
>>>>>> false, "failed_when_result": false, "rc": 0, "start": "2017-05-26
>>>>>> 21:33:34.717739", "stderr": "", "stdout": "{\n  \"href\" : \"
>>>>>> http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:
>>>>>> 8080/api/v1/clusters/metron/requests/14\",\n  \"Requests\" : {\n
>>>>>> \"id\" : 14,\n    \"status\" : \"Accepted\"\n  }\n}202",
>>>>>> "stdout_lines":
>>>>>> ["{", "  \"href\" : \"http://ec2-34-210-207-190.us
>>>>>> -west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\
>>>>>> <http://west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14%5C>
>>>>>> <http://west-2.compute.amazonaws.com:8080/api/v1/clusters/
>>>>>> metron/requests/14%5C>
>>>>>> 
>>>>>> ",",
>>>>>> "  \"Requests\" : {", "    \"id\" : 14,", "    \"status\" :
>>>>>> \"Accepted\"",
>>>>>> "  }", "}202"], "warnings": []}
>>>>>> 
>>>>>> TASK [load_web_templates : Load Kibana Dashboard]
>>>>>> ******************************
>>>>>> fatal: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com]: 
>>>>>> FAILED!
>>>>>> =>
>>>>>> {"failed": true, "msg": "the field 'args' has an invalid value, 
>>>>>> which
>>>>>> appears to include a variable that is undefined. The error was: 
>>>>>> list
>>>>>> object
>>>>>> has no element 0\n\nThe error appears to have been in
>>>>>> '/root/metron/metron-deployment/roles/load_web_templates/
>>>>>> tasks/main.yml':
>>>>>> line 26, column 3, but may\nbe elsewhere in the file depending on

>>>>>> the
>>>>>> exact
>>>>>> syntax problem.\n\nThe offending line appears to be:\n\n\n- name:

>>>>>> Load
>>>>>> Kibana Dashboard\n  ^ here\n"}
>>>>>>         to retry, use: --limit @/root/metron/metron-deploymen
>>>>>> t/amazon-ec2/playbook.retry
>>>>>> 
>>>>>> PLAY RECAP ******************************
>>>>>> ******************************
>>>>>> *********
>>>>>> ec2-34-210-18-244.us-west-2.compute.amazonaws.com : ok=41   
>>>>>> changed=27
>>>>>>  unreachable=0    failed=0
>>>>>> ec2-34-210-207-190.us-west-2.compute.amazonaws.com : ok=80
>>>>>>  changed=31
>>>>>>  unreachable=0    failed=1
>>>>>> ec2-34-211-19-168.us-west-2.compute.amazonaws.com : ok=41   
>>>>>> changed=27
>>>>>>  unreachable=0    failed=0
>>>>>> ec2-35-164-236-192.us-west-2.compute.amazonaws.com : ok=47
>>>>>>  changed=31
>>>>>>  unreachable=0    failed=0
>>>>>> ec2-52-41-30-101.us-west-2.compute.amazonaws.com : ok=41   
>>>>>> changed=27
>>>>>>  unreachable=0    failed=0
>>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com : ok=41   
>>>>>> changed=27
>>>>>>  unreachable=0    failed=0
>>>>>> localhost                  : ok=16   changed=11   unreachable=0
>>>>>> failed=0
>>>>>> 
>>>>>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>>>>> 
>>>>>> 
>>>>>> On 2017-05-26 11:59, Michael Miklavcic wrote:
>>>>>> 
>>>>>> Maybe you could re-purpose the playbook.yml for the 10-node 
>>>>>> install -
>>>>>> 
>>>>>>> comment out some of the host creation lines and move the host

>>>>>>> types
>>>>>>> where
>>>>>>> you want them installed, e.g.
>>>>>>> 
>>>>>>> From this
>>>>>>> 
>>>>>>>   tasks:
>>>>>>>     - include: tasks/create-keypair.yml
>>>>>>>     - include: tasks/create-vpc.yml
>>>>>>>     - include: tasks/create-open-inbound-security-group.yml
>>>>>>>     - include: tasks/create-open-outbound-security-group.yml
>>>>>>>     - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=sensors,ambari_master,ec2,monit
>>>>>>>     - include: tasks/create-hosts.yml host_count=4
>>>>>>> host_type=ambari_slave,ec2
>>>>>>>     - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=pcap_server,monit,ec2
>>>>>>>     - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>>>     - include: tasks/create-hosts.yml host_count=2
>>>>>>> host_type=ambari_slave,search,ec2
>>>>>>>     - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=ambari_slave,web,ec2
>>>>>>> 
>>>>>>> to this (6-node)
>>>>>>> 
>>>>>>>   tasks:
>>>>>>>     - include: tasks/create-keypair.yml
>>>>>>>     - include: tasks/create-vpc.yml
>>>>>>>     - include: tasks/create-open-inbound-security-group.yml
>>>>>>>     - include: tasks/create-open-outbound-security-group.yml
>>>>>>>     - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
>>>>>>>     - include: tasks/create-hosts.yml host_count=3
>>>>>>> host_type=ambari_slave,ec2
>>>>>>>     - include: tasks/create-hosts.yml host_count=1 host_type=
>>>>>>> ambari_slave
>>>>>>> ,pcap_server,monit,ec2
>>>>>>>     - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>>> 
>>>>>>> Of course, this is just to demonstrate re-locating items, not
>>>>>>> necessarily
>>>>>>> how you would want to lay them out.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Mike Miklavcic
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, May 26, 2017 at 11:28 AM, Laurens Vets 
>>>>>>> <laurens@daemon.be>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> I'm trying to install Metron in AWS but with fewer hosts than
the
>>>>>>> default
>>>>>>> 
>>>>>>> 10 node cluster install... So far no luck. Whatever I do, it

>>>>>>> always
>>>>>>>> ends
>>>>>>>> in
>>>>>>>> errors...
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 2017-05-26 10:16, David Lyle wrote:
>>>>>>>> 
>>>>>>>> I'm gonna to venture a 'no' simply because I don't know what

>>>>>>>> that is.
>>>>>>>> 
>>>>>>>> Afaik, single_node_vm.yml was never used with AWS. What are
you
>>>>>>>>> trying
>>>>>>>>> to
>>>>>>>>> do?
>>>>>>>>> 
>>>>>>>>> -D...
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets 
>>>>>>>>> <laurens@daemon.be>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Is AWS single_node_vm still supported?

Mime
View raw message