stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Fremantle <p...@wso2.com>
Subject Re: Messaging protocol with Stratos python cartridge agent
Date Tue, 02 Sep 2014 10:46:36 GMT
Gayan

Normally you would structure your topic hierarchy to be capture the
structure of events and the schema

e.g.
topic
/metrics/datacentre1/system1/cpu
message
{ json format for cpu data }

Paul


On 2 September 2014 11:40, Gayan Gunarathne <gayang@wso2.com> wrote:

> Hi Paul,
>
> AFAIU with the existing message design and with your points with the
> Pub/Sub ,+1 for improve our Startos messaging model as far as we can as it
> will improve the performance as well as standards.As far as I understood
> the current messaging model , we can thinking about following two
> suggestions.
>
> *Suggestion 1 - *
> We can used separate topic for the every event type.But AFAIN there are
> around 30 events(same schema or shape) around in the current scenario
> and I supposed it is growing as well.Please correct me if I am wrong
> here.So we have to maintain separate topics for the each events and then
> easily we can distinguished the messages based on the topic.
>
> *Suggestion 2 -*
> With the current implementation we used the "event-class-name" value to
> distinguished the messages.IMO also We don't need to distinguished the
> messages based on the header and we can easily include that in the message
> body.But still I guess there will be a overhead is that every time we need
> to deserialize  the JSON message and see what is the message class name
> is.WDYT?
>
> Please share your thoughts on this and also it is really appreciate if you
> can recommend any other suggestion in your mind to improve the current
> messaging model.
>
>
> Thanks,
> Gayan
>
>
>
> On Tue, Sep 2, 2014 at 2:24 PM, Paul Fremantle <paul@wso2.com> wrote:
>
>> Gayan
>>
>> I don't understand this. I think this is a mistake on how to use Pub/Sub.
>>
>> The headers timestamp and expiry should probably be in the message (not
>> sure)
>> The destination is part of the pub/sub network
>> The event-class should be implicitly based on the topic. In other words
>> in a pub/sub network, every message in a specific topic should have the
>> same schema or shape. You shouldn't need some extra header to know which
>> schema it is.
>>
>> Paul
>>
>>
>>
>> On 2 September 2014 09:44, Gayan Gunarathne <gayang@wso2.com> wrote:
>>
>>> Hi Paul,
>>>
>>> Please find the example of existing message header and body.AFAIK
>>> "event-class-name" is the header value used for distinguish the messages.
>>>
>>> header: key expires , value 0
>>> header: key timestamp , value 1409646637558
>>> header: key destination , value /topic/topology
>>> header: key persistent , value true
>>> header: key priority , value 4
>>> header: key event-class-name , value
>>> org.apache.stratos.messaging.event.topology.CompleteTopologyEvent
>>>
>>> Message body:
>>>
>>>  {"topology":{"serviceMap":{"gateway":{"serviceName":"gateway","serviceType":"SingleTenant","clusterIdClusterMap":{"gatewaygateway.am.wso2.com.dom":{"serviceName":"gateway","clusterId":"gatewaygateway.am.wso2.com.dom","autoscalePolicyName":"auto-scl-pol1","deploymentPolicyName":"
>>> dep-pol1","hostNames":["foo.org"],"tenantRange":"1-*","isLbCluster":false,"memberMap":{"gatewaygateway.am.wso2.com.dom-member-2":{"serviceName":"gateway","clusterId":"gatewaygateway.am.wso2.com.dom","networkPartitionId":"network-partition1","partitionId":"cloud-partition1","memberId":"gatewaygateway.am.wso2.com.dom-member-2","portMap":{"9080":{"protocol":"http","value":8080,"proxy":9080}},"memberPublicIp":"127.0.0.1
>>> public","status":"Activated","memberIp":"127.0.0.1
>>> private","properties":{"mykey1xxxxxxxxx":"myvalweccccccccccccccc"}},"gatewaygateway.am.wso2.com.dom-member-1":{"serviceName":"gateway","clusterId":"gatewaygateway.am.wso2.com.dom","networkPartitionId":"network-partition1","partitionId":"cloud-partition1","memberId":"gatewaygateway.am.wso2.com.dom-member-1","portMap":{"9080":{"protocol":"http","value":8080,"proxy":9080}},"memberPublicIp":"127.0.0.1
>>> public","status":"Activated","memberIp":"127.0.0.1
>>> private","properties":{"mykey1xxxxxxxxx":"myvalweccccccccccccccc"}},"gatewaygateway.am.wso2.com.dom5863cf8d-2746-4be1-91e9-da98c2bcb5a9":{"serviceName":"gateway","clusterId":"gatewaygateway.am.wso2.com.dom","networkPartitionId":"network-partition1","partitionId":"cloud-partition1","memberId":"gatewaygateway.am.wso2.com.dom5863cf8d-2746-4be1-91e9-da98c2bcb5a9","portMap":{},"memberPublicIp":"public
>>> IP","memberIp":"private
>>> IP"},"lb-member-id-1":{"serviceName":"lb","clusterId":"lbisuruh.lk.domain","networkPartitionId":"network-partition1","partitionId":"cloud-partition1","memberId":"lb-member-id-1","portMap":{},"memberPublicIp":"public
>>> IP of apistore","memberIp":"private IP of
>>> apistore"}},"properties":{"mykey1weeee":"myvalweeeeeeeee1","mykey1tttttttt":"myvalweetttttttttttt1"}}},"portMap":{"9080":{"protocol":"http","value":8080,"proxy":9080}},"properties":{"mykey1weeee":"myvalweeeeeeeee1","mykey1tttttttt":"myvalweetttttttttttt1"}},"apistore":{"serviceName":"apistore","serviceType":"SingleTenant","clusterIdClusterMap":{"apistorestore.am.wso2.com.doma":{"serviceName":"apistore","clusterId":"apistorestore.am.wso2.com.doma","autoscalePolicyName":"auto-scl-pol1","deploymentPolicyName":"
>>> dep-pol1","hostNames":["foo.org"],"tenantRange":"1-*","isLbCluster":false,"memberMap":{"apistorestore.am.wso2.com.doma298e2959-a3b4-45a0-a11f-7671606158ed":{"serviceName":"apistore","clusterId":"apistorestore.am.wso2.com.doma","networkPartitionId":"network-partition1","partitionId":"cloud-partition1","memberId":"apistorestore.am.wso2.com.doma298e2959-a3b4-45a0-a11f-7671606158ed","portMap":{},"memberPublicIp":"pub
>>> ip","status":"Activated","memberIp":"ip","lbClusterId":"lbmongolb594224131.mongolb.dom"}}}},"portMap":{"9080":{"protocol":"http","value":8080,"proxy":9080}}},"lb":{"serviceName":"lb","serviceType":"SingleTenant","clusterIdClusterMap":{"lbisuruh.lk.domain":{"serviceName":"lb","clusterId":"lbisuruh.lk.domain","autoscalePolicyName":"auto-scl-pol1","deploymentPolicyName":"
>>> dep-pol1","hostNames":["foo.org
>>> "],"tenantRange":"1-*","isLbCluster":false,"memberMap":{}}},"portMap":{"9080":{"protocol":"http","value":8080,"proxy":9080}}}},"initialized":false}}
>>>
>>>
>>> Thanks,
>>> Gayan
>>>
>>>
>>>
>>> On Tue, Sep 2, 2014 at 1:49 PM, Paul Fremantle <paul@wso2.com> wrote:
>>>
>>>> Can you post an example of the headers and body?
>>>>
>>>> Paul
>>>>
>>>>
>>>> On 2 September 2014 09:17, Gayan Gunarathne <gayang@wso2.com> wrote:
>>>>
>>>>> Hi Paul,
>>>>>
>>>>> Currently we have existing java based message publisher with the
>>>>> headers. Here we try to use those published messages with the python
>>>>> client. If we use JSON structured message with  headers enclosed we have
to
>>>>> change all other current implementations as well. AFAIK We will
>>>>> distinguished the messages with their header value.So my point is if
we
>>>>> used this sort of JSON structure for messages, we need to change all
the
>>>>> current messaging model.
>>>>>
>>>>> OTOH with the Stomp protocol with python client we were able to get
>>>>> those header values and used those messages as it is.
>>>>>
>>>>> Thanks,
>>>>> Gayan
>>>>>
>>>>>
>>>>> On Tue, Sep 2, 2014 at 1:04 PM, Paul Fremantle <paul@wso2.com>
wrote:
>>>>>
>>>>>> What headers do you need to pass?
>>>>>>
>>>>>> The way of doing headers with MQTT is to use a structured message
>>>>>> (e.g. JSON) and put some headers into the JSON.
>>>>>>
>>>>>> Paul
>>>>>>
>>>>>>
>>>>>> On 2 September 2014 07:58, Gayan Gunarathne <gayang@wso2.com>
wrote:
>>>>>>
>>>>>>>
>>>>>>> We did some analysis regarding the MQTT protocol as well.Following
>>>>>>> are our findings regarding the MQTT.
>>>>>>>
>>>>>>> MQTT is low bandwidth consumption messaging protocol which
>>>>>>> supports pub/sub protocol with the following limitations.
>>>>>>>
>>>>>>>    - It has fixed header (2 bytes) . Specifications mentioning
that
>>>>>>>    it can be extended.But AFAIU clients[1] didn't support for
this header
>>>>>>>    extension. We have checked the MQTT with the messages published
with
>>>>>>>    headers and we didn't receive the header on those.Only receive
the message
>>>>>>>    body.
>>>>>>>    - The payload size for a MQTT message is 256MB.I guess design
>>>>>>>    point for MQTT usage is targeted at more constrained devices.[3].
MQTT is
>>>>>>>    better at a high volume of low size messages.
>>>>>>>    - It is supports only topic.This will not be a issue currently
>>>>>>>    in our scenario as we used only topic currently AFAIK.[4]
>>>>>>>
>>>>>>>
>>>>>>> I guess it build with above limitations and it is very good choice
>>>>>>> for the mobile and small application as it is very lightweight
and
>>>>>>> low power.[5] WDYT?
>>>>>>>
>>>>>>> Please share your thoughts on this.
>>>>>>>
>>>>>>> [1] https://pypi.python.org/pypi/paho-mqtt
>>>>>>> [2[ https://github.com/iwanbk/nyamuk
>>>>>>> [3] http://stephendnicholas.com/archives/1217
>>>>>>> [4]
>>>>>>> http://www.mqtechconference.com/sessions_v2013/MQTC_v2013_MQTT.pdf
>>>>>>> [5]
>>>>>>> http://blogs.vmware.com/vfabric/2013/02/choosing-your-messaging-protocol-amqp-mqtt-or-stomp.html
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Gayan
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 1, 2014 at 4:32 PM, Gayan Gunarathne <gayang@wso2.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Paul.
>>>>>>>>
>>>>>>>> We will check on this paho-mgtt python library as well.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Gayan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 1, 2014 at 4:17 PM, Paul Fremantle <paul@wso2.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Python has excellent pure python MQTT support.
>>>>>>>>>
>>>>>>>>> > pip install paho-mqtt
>>>>>>>>>
>>>>>>>>> https://pypi.python.org/pypi/paho-mqtt
>>>>>>>>>
>>>>>>>>> Paul
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 1 September 2014 11:40, Gayan Gunarathne <gayang@wso2.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Paul,
>>>>>>>>>>
>>>>>>>>>> Here we try to connect the our existing Apache ActiveMQ
with the
>>>>>>>>>> pure python.(We try to replace our java cartridge
agent with the
>>>>>>>>>> python as after that we can remove the java dependency
from the cartridge
>>>>>>>>>> itself).
>>>>>>>>>> In the initial research we didn't notice such a pure
python
>>>>>>>>>> libraries with the MQTT  which can connect with the
ActiveMQ. Also with the
>>>>>>>>>> initial reading I guess they are recommend it for
the mobile applications
>>>>>>>>>> as its low bandwidth consumption.Also I am noticed
lot of web socket based
>>>>>>>>>> applications are developed through MQTT.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Gayan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 1, 2014 at 3:42 PM, Paul Fremantle <paul@wso2.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Have you looked at mqtt?
>>>>>>>>>>>
>>>>>>>>>>> Paul
>>>>>>>>>>> On 1 Sep 2014 12:10, "Gayan Gunarathne" <gayang@wso2.com>
wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> As per our initial research on the python
based messaging
>>>>>>>>>>>> implementation of python cartridge agent
and discussion[1] we can summarize
>>>>>>>>>>>> our findings as following.
>>>>>>>>>>>>
>>>>>>>>>>>> *1.AMQP*
>>>>>>>>>>>>
>>>>>>>>>>>> Apache ActiveMQ used the AMQP 1.0 protocol.
We found C based
>>>>>>>>>>>> client library (proton) [2] which can be
used with python.
>>>>>>>>>>>> But it is a platform dependent library and
every time we need
>>>>>>>>>>>> to install the libraries based on the architecture
and OS version.IMO this
>>>>>>>>>>>> will be difficult us to manage as we need
to install the libraries based on
>>>>>>>>>>>> the platform.
>>>>>>>>>>>>
>>>>>>>>>>>> Other python libraries(txAMQP ,pika etc)
supports AMQP 0.9
>>>>>>>>>>>> which fails to connect with the ActiveMQ.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *2.STOMP*
>>>>>>>>>>>>
>>>>>>>>>>>> We can connect to the ActiveMQ with the STOMP
protocol by using
>>>>>>>>>>>> the STOMP python client library[3].IMO we
can used STOMP with ActiveMQ
>>>>>>>>>>>> without worry about the platform based library
installation.By considering
>>>>>>>>>>>> our scope of the cartridge agent I think
it is better to have a platform
>>>>>>>>>>>> independence cartridge as far as we can.
Then, IMO STOMP based messaging
>>>>>>>>>>>> queue implementation  will be more manageable
when we increase our number
>>>>>>>>>>>> of platform supported.
>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>
>>>>>>>>>>>> When comparing the performance of theses
two protocols it looks
>>>>>>>>>>>> like there are no such huge difference[4].I
guess the performance also
>>>>>>>>>>>> tightly coupled with the message broker you
are using as well.OTOH Apache
>>>>>>>>>>>> ActiveMQ recommended way also to used the
STOMP with the python[5].
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Please share your thoughts on this.
>>>>>>>>>>>>
>>>>>>>>>>>> [1] Stratos Python Cartridge Agent
>>>>>>>>>>>> [2] https://github.com/apache/qpid-proton
>>>>>>>>>>>> [3] https://github.com/jasonrbriggs/stomp.py
>>>>>>>>>>>> [4]
>>>>>>>>>>>> http://blogs.vmware.com/vfabric/2013/02/choosing-your-messaging-protocol-amqp-mqtt-or-stomp.html
>>>>>>>>>>>> [5] http://activemq.apache.org/python.html
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Gayan
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>>> Technical Lead
>>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>>> email  : gayang@wso2.com  | mobile : +94
766819985
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards,
>>>>>>>>>>
>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>> Technical Lead
>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>  email  : gayang@wso2.com  | mobile : +94 766819985
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Paul Fremantle
>>>>>>>>> CTO and Co-Founder, WSO2
>>>>>>>>> OASIS WS-RX TC Co-chair, Apache Member
>>>>>>>>>
>>>>>>>>> UK: +44 207 096 0336
>>>>>>>>>
>>>>>>>>> blog: http://pzf.fremantle.org
>>>>>>>>> twitter.com/pzfreo
>>>>>>>>> paul@wso2.com
>>>>>>>>>
>>>>>>>>> wso2.com Lean Enterprise Middleware
>>>>>>>>>
>>>>>>>>> Disclaimer: This communication may contain privileged
or other
>>>>>>>>> confidential information and is intended exclusively
for the addressee/s.
>>>>>>>>> If you are not the intended recipient/s, or believe that
you may have
>>>>>>>>> received this communication in error, please reply to
the sender indicating
>>>>>>>>> that fact and delete the copy you received and in addition,
you should not
>>>>>>>>> print, copy, retransmit, disseminate, or otherwise use
the information
>>>>>>>>> contained in this communication. Internet communications
cannot be
>>>>>>>>> guaranteed to be timely, secure, error or virus-free.
The sender does not
>>>>>>>>> accept liability for any errors or omissions.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>>
>>>>>>>> Gayan Gunarathne
>>>>>>>> Technical Lead
>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>>
>>>>>>> Gayan Gunarathne
>>>>>>> Technical Lead
>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Paul Fremantle
>>>>>> CTO and Co-Founder, WSO2
>>>>>> OASIS WS-RX TC Co-chair, Apache Member
>>>>>>
>>>>>> UK: +44 207 096 0336
>>>>>>
>>>>>> blog: http://pzf.fremantle.org
>>>>>> twitter.com/pzfreo
>>>>>> paul@wso2.com
>>>>>>
>>>>>> wso2.com Lean Enterprise Middleware
>>>>>>
>>>>>> Disclaimer: This communication may contain privileged or other
>>>>>> confidential information and is intended exclusively for the addressee/s.
>>>>>> If you are not the intended recipient/s, or believe that you may
have
>>>>>> received this communication in error, please reply to the sender
indicating
>>>>>> that fact and delete the copy you received and in addition, you should
not
>>>>>> print, copy, retransmit, disseminate, or otherwise use the information
>>>>>> contained in this communication. Internet communications cannot be
>>>>>> guaranteed to be timely, secure, error or virus-free. The sender
does not
>>>>>> accept liability for any errors or omissions.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>>
>>>>> Gayan Gunarathne
>>>>> Technical Lead
>>>>> WSO2 Inc. (http://wso2.com)
>>>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Paul Fremantle
>>>> CTO and Co-Founder, WSO2
>>>> OASIS WS-RX TC Co-chair, Apache Member
>>>>
>>>> UK: +44 207 096 0336
>>>>
>>>> blog: http://pzf.fremantle.org
>>>> twitter.com/pzfreo
>>>> paul@wso2.com
>>>>
>>>> wso2.com Lean Enterprise Middleware
>>>>
>>>> Disclaimer: This communication may contain privileged or other
>>>> confidential information and is intended exclusively for the addressee/s.
>>>> If you are not the intended recipient/s, or believe that you may have
>>>> received this communication in error, please reply to the sender indicating
>>>> that fact and delete the copy you received and in addition, you should not
>>>> print, copy, retransmit, disseminate, or otherwise use the information
>>>> contained in this communication. Internet communications cannot be
>>>> guaranteed to be timely, secure, error or virus-free. The sender does not
>>>> accept liability for any errors or omissions.
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>>
>>> Gayan Gunarathne
>>> Technical Lead
>>> WSO2 Inc. (http://wso2.com)
>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>
>>>
>>
>>
>>
>> --
>> Paul Fremantle
>> CTO and Co-Founder, WSO2
>> OASIS WS-RX TC Co-chair, Apache Member
>>
>> UK: +44 207 096 0336
>>
>> blog: http://pzf.fremantle.org
>> twitter.com/pzfreo
>> paul@wso2.com
>>
>> wso2.com Lean Enterprise Middleware
>>
>> Disclaimer: This communication may contain privileged or other
>> confidential information and is intended exclusively for the addressee/s.
>> If you are not the intended recipient/s, or believe that you may have
>> received this communication in error, please reply to the sender indicating
>> that fact and delete the copy you received and in addition, you should not
>> print, copy, retransmit, disseminate, or otherwise use the information
>> contained in this communication. Internet communications cannot be
>> guaranteed to be timely, secure, error or virus-free. The sender does not
>> accept liability for any errors or omissions.
>>
>
>
>
> --
> Best Regards,
>
> Gayan Gunarathne
> Technical Lead
> WSO2 Inc. (http://wso2.com)
> email  : gayang@wso2.com  | mobile : +94 766819985
>
>



-- 
Paul Fremantle
CTO and Co-Founder, WSO2
OASIS WS-RX TC Co-chair, Apache Member

UK: +44 207 096 0336

blog: http://pzf.fremantle.org
twitter.com/pzfreo
paul@wso2.com

wso2.com Lean Enterprise Middleware

Disclaimer: This communication may contain privileged or other confidential
information and is intended exclusively for the addressee/s. If you are not
the intended recipient/s, or believe that you may have received this
communication in error, please reply to the sender indicating that fact and
delete the copy you received and in addition, you should not print, copy,
retransmit, disseminate, or otherwise use the information contained in this
communication. Internet communications cannot be guaranteed to be timely,
secure, error or virus-free. The sender does not accept liability for any
errors or omissions.

Mime
View raw message