karaf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofre ...@nanthrax.net>
Subject Re: KARAF DECANTER Elastic Search Collector configuration for Kibana
Date Wed, 30 Sep 2020 16:21:18 GMT
Hi Mark,

See my comments inline:
> 
> 1. Karaf container version: 
> karaf@root()> version
> 4.1.6
> I will repeat my tests for version 4.2.8, but need to be able to add activemq 5.15 and
camel to this container, as well as the decanter features. Have had problems finding just
the right version for them all to install fine.

To be honest, I didn’t test Decanter with Karaf 4.1.x (it’s pretty old).
As Decanter uses SCR, it should work but I would rather use Karaf 4.2.x (as it’s the version
used in Decanter tests).

> 
> 2. My use case is to have an activemq / camel container able to report on its own metrics
with kibana using the kibana feature,  and forward its metrics to an external elastic search.
> So my questions are:
> 1) do i need to have a an elastic search collector at all? the only only data I need
the /kibana context to display here is locally gathered jmx metrics.

No need of elasticsearch collector. Again, the collector is there to get data from elasticsearch.
In your case, it’s the opposite: you have to send data to elasticsearch and then you plug
a kibana / grafana on this elasticsearch.

> 2) can kibana feature even work without a locally embedded elastic search, without installing
the elastic search feature? it seems it cannot: kibana cannot connect and does not start,
complaining always of localhost:9200 not connecting. 
That’s actually recommended to NOT USE embedded kibana and elasticsearch instance.


> 
> 3. More details regarding the collector and appender config: I was able to test these
files and indeed I see that setting the collector addresses to a non local host is picked
up (I see it in the output of config:list. But the logs would show that collector would always
read from localhost:9200, as if it were cached or hardcoded somewhere. I tested with the elasticsearch
feature running and also stopped
> 
> 4. For today I would like not even to load elastic feature, not even see the elasticsearch.yml
config, and still be able to get kibana to display my locally collected jmx data. Failing
that, I will install only the local elasticsearch to get kibana to work, and install the elastic
appender as well as the prometheus appended (always works great).

So, what you need is this:

1. Start elasticsearch instance (standalone) wherever you want, for instance on host1

  $ bin/elasticsearch

2. Start kibana to point to host:9200

3. In Karaf, just do:

$ feature:repo-add decanter 2.5.0
$ feature:install decanter-appender-elasticsearch
// configure etc/org.apache.karaf.decanter.collector.elasticsearch.cfg to point to host1:9200
in addresses
$ feature:install decanter-collector-log
$ feature:install decanter-collector-jmx
$ feature:install decanter-collector-jmx-activemq
$ feature:install decanter-collector-jmx-camel

After a minute or so, you will see the data coming in elasticsearch, and so in kibana

Regards
JB


> 
>> On Sep 29, 2020, at 20:30, Jean-Baptiste Onofre <jb@nanthrax.net> wrote:
>> 
>> Hi Mark,
>> 
>> Elasticsearch collector means that you get data from elasticsearch. Is it what you
want or more pushing data to elasticsearch (and in that case, it’s the elasticsearch appender).
>> 
>> Anyway, I just double checked and it works fine. Here’s what I did:
>> 
>> $ feature:repo-add decanter 2.5.0
>> 
>> # for the decanter elasticsearch collector (getting data from elasticsearch)
>> $ feature:install decanter-collector-elasticsearch
>> # edit etc/org.apache.karaf.decanter.collector.elasticsearch.cfg to set addresses
>> # check in component config
>> $ scr:info org.apache.karaf.decanter.collector.elasticsearch
>> …
>> addresses<String> = http://foo:9201 <http://foo:9201/>
>> ...
>> # in the log, I see
>> ...
>> Caused by: java.net.UnknownHostException: foo: nodename nor servname provided, or
not known
>> ...
>> So it’s correct ;)
>> 
>> # for the decanter elasticsearch appender (pushing data to elasticsearch)
>> $ feature:install decanter-appender-elasticsearch
>> # edit etc/org.apache.karaf.decanter.appender.elasticsearch.cfg to set addresses
>> # check in component config
>> $ scr:info org.apache.karaf.decanter.appender.elasticsearch
>> …
>> addresses<String> = http://foo:9201 <http://foo:9201/>
>> ...
>> # to test it I’m adding the log collector for instance (just to generate data)
>> $ feature:install decanter-collector-log
>> # in the log I see:
>> …
>> Caused by: java.net.UnknownHostException: foo
>> …
>> 
>> So, both works fine.
>> 
>> What Karaf version are you using ? If you use 4.2.9, can you please try with 4.2.8
?
>> 
>> Thanks,
>> Regards
>> JB
>> 
>>> Le 30 sept. 2020 à 00:49, Mark Bordelon <mbordelon@savoirtech.com> a écrit
:
>>> 
>>> Hello Karaf experts.
>>> 
>>> I have followed the documentation http://karaf.apache.org/manual/decanter/latest-2/html/#_prometheus
<http://karaf.apache.org/manual/decanter/latest-2/html/#_prometheus> while going through
JBO’s blog http://blog.nanthrax.net/?p=715 <http://blog.nanthrax.net/?p=715> with
his quick start.
>>> 
>>> Unfortunately, try as I might, I cannot get the elastic search collector to read
the host/port values in my config file org.apache.karaf.decanter.collector.elasticsearch.cfg.
The logs show only that localhost:9200 is being read from, and since there is no local elastic
search, it fails and doesnt start kibana.
>>> 
>>> Here is the important exceprt content of my config file:
>>> #addresses=http://elasticsearch-vela-1.sit.etrade.com:9200 addresses=http://ceng4w100m3.etrade.com:7070/logsearch-sit
>>> 
>>> Is a local elastic search absolutely necessary to run kibana on the karaf container?
 
>>> 
>>> Thanks,
>>> Mark Bordelon
>> 
> 


Mime
View raw message