ws-savan-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanka Samaranayake <ssa...@gmail.com>
Subject Re: [GSoC Project Proposal] Distributed TCP Monitor
Date Thu, 01 Apr 2010 01:28:33 GMT
Hi Samisa,

I think the this scope of this project would be to provide a
light-weight tool which allows to monitor the message flows, execution
in each node in response to those message flows to a given input
message primarily for debugging purposes . (Just like the way tcpmon
is used in developing Web services)

And I guess now the question is whether such a tool is useful ..

Anyone cares to share their thoughts ?

Nilupa

On Thu, Apr 1, 2010 at 2:51 AM, Samisa Abeysinghe
<samisa.abeysinghe@gmail.com> wrote:
>
>
> On Thu, Apr 1, 2010 at 4:53 AM, nilupa bandara <nilupas@gmail.com> wrote:
>>
>> I was thinking about Java Swing based UI
>
> Swing is a good start. But if you look at some of the requirements like
> monitoring multiple instances and correlate them, discussed in this thread,
> it sounds as if we would be better off having a Web interface, rather than a
> desktop client.
>
>>
>> because the client may have
>> to cache whatever messages it already has and will receive,
>
> If you are looking to support message-correlation, as discussed in some
> replies in this thread above, you need to have a persistence storage (a DB
> or something like that) rather than a memory cache.
> Because, some message sequences, like those involved in an async
> invocations, need to have a way to keep the messages, in there, longer, and
> may be there is a server re-start, in the middle.
> Of course, this requirements spend on the scope of this effort.
> Plus, if you want to deal with historic data, and look at what was happening
> to message sequences over time, you got to use a persistence layer. Again
> this might be out of scope for this project, but if this is to be extended
> to support these, you better think of this right now.
>
>>
>> build the
>> message flow and update the UI accordingly. Perhaps there is a better
>> way ?
>
> We already do what you are trying to do, and much more with WSO2 BAM. I am
> tempted to say that Shinding based gadget dashboard is really a compelling
> UI for this.
> I am not trying to take away you project or trying to hinder this effort at
> all.
> I think it will be a good idea for you to have a look at BAM and see what
> the missing pieces are and figure out the right model for this tool.
> Also, having had worked on WSO2 BAM project, I also have some understanding
> on sort of problems that you might run into, when doing this. For e.g., if
> you want to correlate message sequences in a high volume setup, you will be
> swamped by the volume of messages, sooner than later. So correlation for the
> purpose of monitoring is not an easy thing to do. Correlation, for the
> purpose of debugging will not run into this problem, if you run this on a
> controlled environment. However, what you really want to debug is the deal
> clustered setup, and not a staged dev setup, the problem again surfaces.
> You might want to list the objectives and then break them down
> into monitoring and debugging spaces.
> Samisa...
>
>>
>> Thanks
>> Nilupa.
>>
>> On Wed, Mar 31, 2010 at 4:12 PM, Samisa Abeysinghe
>> <samisa.abeysinghe@gmail.com> wrote:
>> > What UI technology would be used for this?
>> >
>> > Samisa...
>> >
>> > On Wed, Mar 31, 2010 at 4:26 AM, Andreas Veithen
>> > <andreas.veithen@gmail.com> wrote:
>> >> I was thinking more about something like ITCAM for SOA.
>> >>
>> >> On Tue, Mar 30, 2010 at 13:40, Sanjiva Weerawarana
>> >> <sanjiva@opensource.lk> wrote:
>> >>> That runtime monitoring / correlation is what the WSO2 Business
>> >>> Activity
>> >>> Monitor does / is for ...
>> >>> See: http://wso2.com/products/business-activity-monitor/
>> >>> Sanjiva.
>> >>>
>> >>> On Tue, Mar 30, 2010 at 2:27 PM, Andreas Veithen
>> >>> <andreas.veithen@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> I was actually referring to scenarios where a service may call other
>> >>>> services, which in turn may call again other services. In these
>> >>>> scenarios, it is not sufficient to simply collect received/sent
>> >>>> messages on different hosts, unless the system is isolated
>> >>>> (development environment) or the request rate is sufficiently low
so
>> >>>> that messages can be correlated based on time. Here is a concrete
>> >>>> scenario from a project in my company:
>> >>>>
>> >>>> - Request comes in on an ESB that does security and validation.
>> >>>> - Request is processed by an application server which persists the
>> >>>> received information and publishes a JMS message (pub/sub events).
>> >>>> - The event is consumed by one or more components that may in turn
>> >>>> interact with other services.
>> >>>>
>> >>>> If you want track this flow, it is not sufficient to intercept the
>> >>>> messages on the different hosts: in addition, you need to instrument
>> >>>> the services so that the outgoing requests are correlated with the
>> >>>> incoming requests (by adding SOAP and/or transport headers). What
I
>> >>>> would like to be able to do in the above scenario is to enable
>> >>>> end-to-end monitoring on a per-message basis: I add a special SOAP
or
>> >>>> HTTP header to the initial request to enable logging and this
>> >>>> information is propagated along the chain. Every service/component
>> >>>> then sends a copy of the incoming/outgoing requests/responses to
a
>> >>>> central place where the sequence of events is reconstructed.
>> >>>>
>> >>>> One way this could be achieved is with a tool that postprocesses
the
>> >>>> artifacts from the build process. For each artifact, the tool would
>> >>>> disassemble the artifact, instrument the code using a set of AspectJ
>> >>>> aspects and reassemble the artifact for deployment. The
>> >>>> responsibility
>> >>>> of the aspects would be to intercept messages, log them and modify
>> >>>> them to ensure proper correlation. Of course this only works if
the
>> >>>> artifacts are J2EE deployables (WARs or EARs). Note that we already
>> >>>> use AspectJ for a similar use case (although at a much smaller scale)
>> >>>> in the transport test suites [1]. Interestingly, this approach would
>> >>>> allow to cover interactions that are not SOAP based, e.g. one could
>> >>>> even intercept the database queries executed during the flow.
>> >>>>
>> >>>> Andreas
>> >>>>
>> >>>> [1]
>> >>>>
>> >>>> https://svn.apache.org/repos/asf/axis/axis2/java/transports/trunk/modules/jms/src/test/java/org/apache/axis2/transport/jms/LogAspect.java
>> >>>>
>> >>>> On Tue, Mar 30, 2010 at 09:11, Srinath Perera <hemapani@gmail.com>
>> >>>> wrote:
>> >>>> > Hi Nilupa;
>> >>>> >
>> >>>> > When we collect messages from a one location by installing
proper
>> >>>> > handler that will intercept and send messages to to that one
>> >>>> > location
>> >>>> > (this one location can be a single server, pub/sub channel
etc).
>> >>>> > There
>> >>>> > is many ways to make sense of those collected messages. What
>> >>>> > Andreas
>> >>>> > mentioned (following complete transaction) is a one possibility.
>> >>>> >
>> >>>> > I think you should come up with few scenarios on how you would
make
>> >>>> > sense of the message.
>> >>>> >
>> >>>> > Thanks
>> >>>> > Srinath
>> >>>> >
>> >>>> > On Sun, Mar 28, 2010 at 9:46 PM, nilupa bandara <nilupas@gmail.com>
>> >>>> > wrote:
>> >>>> >> Hi,
>> >>>> >>
>> >>>> >> First let me thank you for commenting.
>> >>>> >>
>> >>>> >> As far as I understood, what you would like to see from
the
>> >>>> >> proposed
>> >>>> >> tool is to view set of messages that are exchanged in reponse
to a
>> >>>> >> particular input message. With the understanding that I
am having
>> >>>> >> at
>> >>>> >> the momnet, one way to do it is to filter out the central
>> >>>> >> repository
>> >>>> >> of messages based on 'To' , 'From' headers and try to contruct
the
>> >>>> >> message chain from it. We can allow the client GUI wich
connects
>> >>>> >> to
>> >>>> >> the central repository to provide the paramenters (For
instance
>> >>>> >> the
>> >>>> >> value of 'To' header) from which an intelligent filtering
can be
>> >>>> >> done
>> >>>> >> for the set of messages avialable at the central repository.
>> >>>> >>
>> >>>> >> Perhaps someone has an idea of a better way of doing it
and is
>> >>>> >> willing
>> >>>> >> to share it with us. It would be really nice to hear from
them.
>> >>>> >>
>> >>>> >> Thanks in advance ..!!
>> >>>> >>
>> >>>> >> Best Regards,
>> >>>> >> Nilupa
>> >>>> >>
>> >>>> >> On Sat, Mar 27, 2010 at 6:40 PM, Andreas Veithen
>> >>>> >> <andreas.veithen@gmail.com> wrote:
>> >>>> >>> Personally, I think that the added value of extending
the SOAP
>> >>>> >>> monitor
>> >>>> >>> module to collect messages in a central place is too
limited to
>> >>>> >>> attract enough interest from the user community, so
that it will
>> >>>> >>> be
>> >>>> >>> difficult to ensure the evolution of such a project
in the
>> >>>> >>> future.
>> >>>> >>> However, what many people are looking for is a tool
that allows
>> >>>> >>> to
>> >>>> >>> track the flow of a message through a distributed system,
or more
>> >>>> >>> generally to track the sequence of events triggered
by a given
>> >>>> >>> input
>> >>>> >>> message (sort of end-to-end transaction monitoring).
>> >>>> >>>
>> >>>> >>> On Wed, Mar 24, 2010 at 23:03, nilupa bandara <nilupas@gmail.com>
>> >>>> >>> wrote:
>> >>>> >>>> Hello,
>> >>>> >>>>
>> >>>> >>>> I am graduate student at Politecnica de Madrid
and I am thinking
>> >>>> >>>> of
>> >>>> >>>> proposing a GSoC project this summer. I would like
to take
>> >>>> >>>> project
>> >>>> >>>> "Distribute TCP Monitor" descried[1] which is very
interesting
>> >>>> >>>> and
>> >>>> >>>> also should be helpful to any Java developer using
Apache Axis2
>> >>>> >>>> Web
>> >>>> >>>> services middleware. I will submit more detailed
proposal later.
>> >>>> >>>> I
>> >>>> >>>> would really like to hear any feedback from you.
Any suggestions
>> >>>> >>>> or
>> >>>> >>>> features that you would like to see in a "Distributed
TCP
>> >>>> >>>> Monitor"
>> >>>> >>>> are
>> >>>> >>>> most welcome.
>> >>>> >>>>
>> >>>> >>>> Best Regards,
>> >>>> >>>> Nilupa Bandara
>> >>>> >>>>
>> >>>> >>>> [1]
>> >>>> >>>>
>> >>>> >>>> Distributed TCP monitor
>> >>>> >>>>
>> >>>> >>>> To use TCP monitor we have to tweak the endpoints,
which is bit
>> >>>> >>>> hard
>> >>>> >>>> sometimes (e.g. two channels Async case for Axis2).
We can solve
>> >>>> >>>> the
>> >>>> >>>> problem by writing a Axis2 module (Handler) that
intercept
>> >>>> >>>> messages
>> >>>> >>>> comes in and goes out of a Axis2 server and send
to a UI (via a
>> >>>> >>>> pub/sub channel or a message Box e.g. ) or record
them so user
>> >>>> >>>> can
>> >>>> >>>> pull those messages. Also, we can do something
to turn the
>> >>>> >>>> module on
>> >>>> >>>> and off remotely, so users can have his module
deployed, but
>> >>>> >>>> turn it
>> >>>> >>>> on only when they want to debug the system.
>> >>>> >>>>
>> >>>> >>>> Then we can take this to next level by adding this
module to all
>> >>>> >>>> services in a system , configuring modules to send
all collected
>> >>>> >>>> messages to a pub/sub channel, and subscribing
to those messages
>> >>>> >>>> via
>> >>>> >>>> a
>> >>>> >>>> UI, and depicting those messages through a UI.
Then users have a
>> >>>> >>>> TCPMon for a whole system.
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>> ---------------------------------------------------------------------
>> >>>> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>> >>>> >>>> For additional commands, e-mail: java-dev-help@axis.apache.org
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>> ---------------------------------------------------------------------
>> >>>> >>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>> >>>> >>> For additional commands, e-mail: java-dev-help@axis.apache.org
>> >>>> >>>
>> >>>> >>>
>> >>>> >>
>> >>>> >>
>> >>>> >> ---------------------------------------------------------------------
>> >>>> >> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>> >>>> >> For additional commands, e-mail: java-dev-help@axis.apache.org
>> >>>> >>
>> >>>> >>
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > ============================
>> >>>> > Srinath Perera, Ph.D.
>> >>>> >   WSO2 Inc. http://wso2.com
>> >>>> >   Blog: http://srinathsview.blogspot.com/
>> >>>> >
>> >>>> >
>> >>>> > ---------------------------------------------------------------------
>> >>>> > To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>> >>>> > For additional commands, e-mail: java-dev-help@axis.apache.org
>> >>>> >
>> >>>> >
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>> >>>> For additional commands, e-mail: java-dev-help@axis.apache.org
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Sanjiva Weerawarana, Ph.D.
>> >>> Founder, Director & Chief Scientist; Lanka Software Foundation;
>> >>> http://www.opensource.lk/
>> >>> Founder, Chairman & CEO; WSO2, Inc.; http://www.wso2.com/
>> >>> Member; Apache Software Foundation; http://www.apache.org/
>> >>> Director; Sahana Software Foundation; http://www.sahanafoundation.org/
>> >>> Visiting Lecturer; University of Moratuwa; http://www.cse.mrt.ac.lk/
>> >>>
>> >>> Blog: http://sanjiva.weerawarana.org/
>> >>>
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>> >> For additional commands, e-mail: java-dev-help@axis.apache.org
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>> > For additional commands, e-mail: java-dev-help@axis.apache.org
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>> For additional commands, e-mail: java-dev-help@axis.apache.org
>>
> Samisa...
> --
> blog: http://samisa-abeysinghe.blogspot.com/
>



-- 
Sanka Samaranayake

http://sankas.blogspot.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org


Mime
View raw message