metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmed Shah <AhmedS...@cmail.carleton.ca>
Subject Re: Ability to deploy metron full dev as a single node to aws with vagrant
Date Tue, 30 Jan 2018 09:41:26 GMT
Hello,

I have made the following pull request for two new ways to deploy to AWS.
https://github.com/apache/metron/pull/916

i) AWS Single Node Cluster Deployment Using Vagrant
I was able to deploy Metron as a Single node to AWS using the latest Metron code in git as
of 2018-01-30 8AM UTC. However, after a while Ambari shows "Metron REST" as red and had to
restart a few components. See pic for what Kibana looks like.

ii) AWS Single Node Cluster Deployment Using an AMI
In addition to including instructions for deploying 0.4.3 using Vagrant I also included instructions
for deploying to AWS using a preexisting AMI (I made the following AMI public as a proof-of-concept:
"GCR-Xetron Demo"/"ami-93cb4ff7"). This might be the quickest option for deployment since
no pre-install requirements would be needed. However, Ambari shows a few components as red
and had to restart them.

-Ahmed
_______________________________________________________________
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php>


________________________________
From: Ahmed Shah <AhmedShah@cmail.carleton.ca>
Sent: January 25, 2018 2:17 PM
To: Otto Fowler; user@metron.apache.org
Subject: Ability to deploy metron full dev as a single node to aws with vagrant


Thanks,


The old script worked for 0.4.1, but it seems to fail with 0.4.2.

(I put the 0.4.2 code I'm working on here for now:

https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/aws-vagrant/Vagrantfile
)



It fails with 0.4.2 here:  ansible.log

2018-01-25 13:36:38,321 p=10334 u=csocadmin |  TASK [ambari_gather_facts : Ask Ambari: namenode_host]
*************************

2018-01-25 13:36:38,742 p=10334 u=csocadmin |  ^[[0;31mfatal: [node1]: FAILED! => {"changed":
false, "failed": true, "msg": "Unable to resolve the host name given."}^[[0m

2018-01-25 13:36:38,752 p=10334 u=csocadmin |  PLAY ***************************************************************************

2018-01-25 13:36:38,795 p=10334 u=csocadmin |  TASK [setup] *******************************************************************

2018-01-25 13:36:39,263 p=10334 u=csocadmin |  ^[[0;32mok: [node1]^[[0m

2018-01-25 13:36:39,267 p=10334 u=csocadmin |  TASK [epel : Install EPEL repository] ******************************************

.

.

.

2018-01-25 13:36:50,369 p=10334 u=csocadmin |  TASK [ambari_gather_facts : set_fact] ******************************************

2018-01-25 13:36:50,400 p=10334 u=csocadmin |  ^[[0;36mskipping: [node1]^[[0m

2018-01-25 13:36:50,404 p=10334 u=csocadmin |  TASK [ambari_gather_facts : Ask Ambari: namenode_host]
*************************

2018-01-25 13:36:50,872 p=10334 u=csocadmin |  ^[[0;31mfatal: [node1]: FAILED! => {"changed":
false, "failed": true, "msg": "Unable to resolve the host name given."}^[[0m

2018-01-25 13:36:50,882 p=10334 u=csocadmin |  PLAY ***************************************************************************

2018-01-25 13:36:50,942 p=10334 u=csocadmin |  TASK [setup] *******************************************************************

2018-01-25 13:36:51,482 p=10334 u=csocadmin |  ^[[0;32mok: [node1]^[[0m

2018-01-25 13:36:51,486 p=10334 u=csocadmin |  TASK [epel : Install EPEL repository] ******************************************

2018-01-25 13:37:01,684 p=10334 u=csocadmin |  ^[[0;32mok: [node1]^[[0m

2018-01-25 13:37:01,687 p=10334 u=csocadmin |  TASK [python-pip : Install python-pip] *****************************************

2018-01-25 13:37:02,301 p=10334 u=csocadmin |  ^[[0;32mok: [node1]^[[0m

2018-01-25 13:37:02,304 p=10334 u=csocadmin |  TASK [httplib2 : Install python httplib2 dependency]
***************************

2018-01-25 13:37:03,013 p=10334 u=csocadmin |  ^[[0;32mok: [node1]^[[0m

2018-01-25 13:37:03,017 p=10334 u=csocadmin |  TASK [ambari_gather_facts : Ask Ambari: cluster_name]
**************************

2018-01-25 13:37:03,086 p=10334 u=csocadmin |  ^[[0;36mskipping: [node1]^[[0m

2018-01-25 13:37:03,089 p=10334 u=csocadmin |  TASK [ambari_gather_facts : set_fact] ******************************************

2018-01-25 13:37:03,147 p=10334 u=csocadmin |  ^[[0;36mskipping: [node1]^[[0m

2018-01-25 13:37:03,150 p=10334 u=csocadmin |  TASK [ambari_gather_facts : Ask Ambari: namenode_host]
*************************

2018-01-25 13:37:03,814 p=10334 u=csocadmin |  ^[[0;31mfatal: [node1]: FAILED! => {"changed":
false, "failed": true, "msg": "Unable to resolve the host name given."}^[[0m

2018-01-25 13:37:03,815 p=10334 u=csocadmin |  PLAY RECAP *********************************************************************

2018-01-25 13:37:03,815 p=10334 u=csocadmin |  127.0.0.1                  : ok=0    changed=0
   unreachable=1    failed=0

2018-01-25 13:37:03,816 p=10334 u=csocadmin |  node1                      : ok=103  changed=48
  unreachable=0    failed=5




When it works for 0.4.2 I will make the pull request into Metron.


I'm running the vagrant file on a Mac.

The /etc/hosts was updated to point node1 to the elastic_ip in the Mac.


Any idea what the issue might be?




-Ahmed
_______________________________________________________________
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php>


________________________________
From: Otto Fowler <ottobackwards@gmail.com>
Sent: January 10, 2018 10:27 AM
To: Ahmed Shah; user@metron.apache.org
Subject: Re: Intro & Question

So, what would work would be:

1.  Create a jira like “Ability to deploy metron full dev to aws with vagrant”
With a description of the use case, and how the vagrant file will fill it.
2. create a pr, with the new file in metron-deployment/vagrant/aws
3. update the readme

I think



On January 10, 2018 at 10:16:03, Ahmed Shah (ahmedshah@cmail.carleton.ca<mailto:ahmedshah@cmail.carleton.ca>)
wrote:

Would be glad to.

Where in github should I put it?


-Ahmed
_______________________________________________________________
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php>


________________________________
From: Otto Fowler <ottobackwards@gmail.com<mailto:ottobackwards@gmail.com>>
Sent: January 9, 2018 11:51 AM
To: Ahmed Shah; user@metron.apache.org<mailto:user@metron.apache.org>
Subject: Re: Intro & Question

Any interest in submitting this?



On January 9, 2018 at 10:42:08, Ahmed Shah (ahmedshah@cmail.carleton.ca<mailto:ahmedshah@cmail.carleton.ca>)
wrote:

Hello Srikanth,


Our team adapted the Metron 0.4.1 Single Node VM install (Original Code Here: https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform)
 to deploy a single node to AWS.

Our Vagrent file is here:

https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/Vagrantfile


You can define your AWS Elastic IP,  Subnet ID, VPC, and Security Group ID before running
the file.


Hope it helps.



-Ahmed
_______________________________________________________________
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php>


________________________________
From: Srikanth Nagarajan <sri@gandivanetworks.com<mailto:sri@gandivanetworks.com>>
Sent: January 9, 2018 2:39 AM
To: user@metron.apache.org<mailto:user@metron.apache.org>
Subject: Intro & Question


Hi

My name is Srikanth and work for a Cyber Security firm.   We are building Metron to test in
our lab environment using AWS.

1. Is there a single VM version for Cloud install available ?   If yes, please share procedure.

2. During the Amazon-Ec2 install for the multi node version provided in the metron git-hub
docs

https://github.com/apache/metron/tree/master/metron-deployment/amazon-ec2  get an error

[WARNING]:  * Failed to parse /Users/sri/metron/metron-deployment/amazon-ec2/ec2.py with script
plugin: Inventory script (/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py) had an execution

error: ERROR: "Forbidden", while: getting ElastiCache clusters

Any assistance would be appreciated.

Thanks

Srikanth

______________________

Srikanth Nagarajan
Principal

Gandiva Networks Inc

732.690.1884 Mobile

sri@gandivanetworks.com<mailto:sri@gandivanetworks.com>

www.gandivanetworks.com<http://www.gandivanetworks.com/>

Please consider the environment before printing this. NOTICE: The information contained in
this e-mail message is intended for addressee(s) only. If you have received this message in
error please notify the sender.

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message