synapse-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Veithen <andreas.veit...@gmail.com>
Subject Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
Date Mon, 16 Sep 2013 16:00:15 GMT
On Fri, Aug 9, 2013 at 9:26 PM, Rajika Kumarasiri
<rajika.kumarasiri@gmail.com> wrote:
> Yeah, actually I have started working on a new syslog consumer
> implementation. As part of this I am also planning to write brand new TCP
> and UDP transports(for high volume syslog message deployments). I will
> report any progress as I go forward.

Why would the current UDP transport not be suitable for high volume messages?

Andreas

> Rajika
>
>
> On Fri, Aug 9, 2013 at 3:15 PM, Hiranya Jayathilaka <hiranya911@gmail.com>wrote:
>
>>
>> On Aug 9, 2013, at 4:57 AM, "Wright, Peter" <Peter.Wright@six-group.com>
>> wrote:
>>
>> > Hi Hiranja,
>> >
>> > Great, thanks a lot, now appears to be working correctly!!
>>
>> That's great.
>>
>> I think it would be good to have some syslog mediation support built into
>> Synapse. The correct solution is probably to implement a syslog message
>> builder and a formatter. There's a long standing jura issue regarding this
>> as well. We'll consider this feature for a future release.
>>
>> Thanks,
>> Hiranya
>>
>> > I changed my class mediator to generate the XML as you suggested, and
>> then had to
>> > include the namespace in the iterate definition and include it in the
>> xpath expression
>> > before it worked correctly.
>> >
>> > Best regards,
>> > Peter
>> >
>> > =================================
>> >
>> > 1. XML message returned by my class mediator
>> >
>> > <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">This is msg number 1</text><text
>> xmlns="http://ws.apache.org/commons/ns/payload">This is msg number
>> 2</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is
msg
>> number 3</text></soapenv:Body></soapenv:Envelope>
>> >
>> > 2. synapse.xml fragment (iterator definition)
>> >
>> > <iterate id="syslogInterator" preservePayload="false" sequential="true"
>> xmlns:m0="http://ws.apache.org/commons/ns/payload" expression="//m0:text">
>> >  <target>
>> >    <sequence>
>> >      <send>
>> >       <endpoint>
>> >          <address .../>
>> >        </endpoint>
>> >      </send>
>> >    </sequence>
>> >  </target>
>> > </iterate>
>> >
>> > -----Original Message-----
>> > From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
>> > Sent: Freitag, 9. August 2013 01:54
>> > To: user@synapse.apache.org
>> > Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages
>> in one TCP message?
>> >
>> >
>> > On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <Peter.Wright@six-group.com>
>> wrote:
>> >
>> >> Hi Hiranja,
>> >>
>> >> I have now implemented a class mediator that parses the (multiple)
>> >> syslog plain text messages and makes an XML payload from them.
>> >> I've also got an iterate mediator that splits the messages and
>> >> sends them individually as JMS messages to the messaging provider.
>> >> So far so good...
>> >>
>> >> However, the individual JMS messages are now in XML form rather
>> >> than plain text (with all the XML headers that I don't want).
>> >> How can I get just the plain text payload to be sent?
>> >>
>> >> Thanks
>> >> Peter
>> >>
>> >> =====================
>> >>
>> >> 1. Input: Dummy (syslog) message sent to TCP listener (individual
>> messages separated by NL)
>> >> 20 This is msg number 1
>> >> 20 This is msg number 2
>> >> 20 This is msg number 3
>> >>
>> >> 2. SOAP message logged by synapse
>> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">20 This is msg number 1
>> >> 20 This is msg number 2
>> >> 20 This is msg number 3
>> >> </text></soapenv:Body></soapenv:Envelope>
>> >>
>> >> 3. XML message returned by my class mediator
>> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This
>> is msg number 1</msg><msg>This is msg number 2</msg><msg>This
is msg number
>> 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
>> >
>> > This should create a message with multiple <text> elements. See the
>> output from step 2 above. It has a single <text> element with all the
>> content. You should break it down to multiple <text> elements. The plain
>> text serializer of Axis2 looks for this specific <text> element when
>> sending plain text messages.
>> >
>> > Pay attention to the namespace (http://ws.apache.org/commons/ns/payload)
>> of the <text> element too. It's also required.
>> >
>> > Thanks,
>> > Hiranya
>> >
>> >>
>> >> 4. JMS messages sent to the messaging provider
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This
is msg
>> number 1</msg></soapenv:Body></soapenv:Envelope>
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This
is msg
>> number 2</msg></soapenv:Body></soapenv:Envelope>
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This
is msg
>> number 3</msg></soapenv:Body></soapenv:Envelope>
>> >>
>> >> 5. JMS messages that I want to be sent
>> >> This is msg number 1
>> >> This is msg number 2
>> >> This is msg number 3
>> >>
>> >> 6. synapse.xml
>> >> <definitions xmlns="http://ws.apache.org/ns/synapse">
>> >>   <proxy name="proxyTcp2Jms" transports="tcp">
>> >>       <target>
>> >>           <inSequence>
>> >>               <!-- Define TCP listener expects plain text (syslog)
>> messages -->
>> >>               <property name="messageType" value="text/plain"
>> scope="axis2"/>
>> >>               <property name="OUT_ONLY" value="true"/>
>> >>               <property name="TRANSPORT_HEADERS" scope="axis2"
>> action="remove"/>
>> >>               <log level="full"/>
>> >>
>> >>               <!-- Split any multiple syslog messages for the iterator
>> -->
>> >>               <!-- and return as XML with child messages
>> -->
>> >>               <class
>> name="com.sixtelekurs.classMediators.syslogMsgBuilder">
>> >>                   <log level="full"/>
>> >>               </class>
>> >>
>> >>               <!-- Iterate over any multiple messages -->
>> >>               <iterate id="syslogInterator" preservePayload="false"
>> sequential="true" expression="//msg">
>> >>                   <log level="full"/>
>> >>                   <target>
>> >>                       <sequence>
>> >>                           <send>
>> >>                               <endpoint>
>> >>                                   <address
>> uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&amp;java.naming.provider.url=MY_LDAP_URL&amp;transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&amp;transport.jms.DestinationType=topic&amp;java.naming.security.principal=MY_DN&amp;java.naming.security.credentials=MY_PW"/>
>> >>                               </endpoint>
>> >>                           </send>
>> >>                       </sequence>
>> >>                   </target>
>> >>               </iterate>
>> >>           </inSequence>
>> >>           <outSequence/>
>> >>           <faultSequence>
>> >>               <log level="full" category="ERROR" separator=","/>
>> >>           </faultSequence>
>> >>       </target>
>> >>       <parameter name="transport.tcp.port">6060</parameter>
>> >>       <parameter name="transport.tcp.contentType">text/plain</parameter>
>> >>   </proxy>
>> >>
>> >> </definitions>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
>> >> Sent: Freitag, 26. Juli 2013 20:05
>> >> To: user@synapse.apache.org
>> >> Subject: Re: Synapse: tcp-listener: handling of multiple syslog
>> messages in one TCP message?
>> >>
>> >> You should probably apply some kind of a content transformation to
>> transform this payload into a XML payload with multiple "text" elements
>> (one per each syslog message). But I'm not sure whether traditional
>> techniques like XSLT and XQuery support that kind of transformations. After
>> all you're trying to transform based on XML text and not the XML nodes.
>> That's almost always very difficult.
>> >>
>> >> IMO you're better off writing a custom mediator to do this
>> transformation. Then you can use the iterate mediator to split the message
>> into its constituents and send to the JMS queue. An even more elegant
>> solution would be to write a custom message builder that knows how to parse
>> the incoming batch requests and construct a proper XML payload out of it.
>> >>
>> >> Thanks,
>> >> Hiranya
>> >>
>> >> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <
>> Peter.Wright@six-group.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I got synapse working with a TCP-Listener accepting plain text messages
>> >>> as input, and the messages being sent as JMS messages to IBM Websphere
>> MQ.
>> >>>
>> >>> Now we're in the test phase with "real" syslog (RFC-5424) data, and
>> I'm having a problem
>> >>> because the TCP listener is receiving multiple syslog messages in one
>> TCP message.
>> >>> The multiple messages are being sent on as one JMS message. The
>> individual
>> >>> syslog messages are separated by NL and have the message length at the
>> start.
>> >>>
>> >>> Here an example message from my log file with 8 syslog messages:
>> >>>
>> >>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To:
,
>> WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID:
>> urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
>> >>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">
>> >>> 186 &lt;174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD
12355 -
>> [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
>> >>> 400 &lt;170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD
12355 -
>> [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New"
>> BC="462" BK="842" Occured="1374760680" Reported="1374761340"
>> Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)"
>> Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured:
>> 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
>> >>> 186 &lt;174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD
12355 -
>> [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
>> >>> 186 &lt;174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD
12355 -
>> [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
>> >>> 119 &lt;174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD
24438 -
>> [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
>> >>> 362 &lt;170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD
12355 -
>> [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462"
>> BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem
>> with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"
>> Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD
>> on mdzhldos02)"]
>> >>> 186 &lt;174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD
12355 -
>> [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
>> >>> 186 &lt;174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD
12355 -
>> [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
>> >>> </text></soapenv:Body></soapenv:Envelope>
>> >>>
>> >>> What is the recommended way of handling this? Use the Iterate-Mediator
>> to
>> >>> split the messages and send them individually? Do I have to use Xpath
>> here,
>> >>> or is it possible to use e.g. javascript? Any examples would be
>> appreciated.
>> >>>
>> >>> Thanks for any help,
>> >>> Peter
>> >>>
>> >>>
>> >>> The content of this e-mail is intended only for the confidential use
>> of the person addressed.
>> >>> If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> >>> Thank you.
>> >>
>> >> --
>> >> Hiranya Jayathilaka
>> >> Mayhem Lab/RACE Lab;
>> >> Dept. of Computer Science, UCSB;  http://cs.ucsb.edu
>> >> E-mail: hiranya@cs.ucsb.edu;  Mobile: +1 (805) 895-7443
>> >> Blog: http://techfeast-hiranya.blogspot.com
>> >>
>> >> The content of this e-mail is intended only for the confidential use of
>> the person addressed.
>> >> If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> >> Thank you.
>> >
>> > --
>> > Hiranya Jayathilaka
>> > Mayhem Lab/RACE Lab;
>> > Dept. of Computer Science, UCSB;  http://cs.ucsb.edu
>> > E-mail: hiranya@cs.ucsb.edu;  Mobile: +1 (805) 895-7443
>> > Blog: http://techfeast-hiranya.blogspot.com
>> >
>> > The content of this e-mail is intended only for the confidential use of
>> the person addressed.
>> > If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> > Thank you.
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB;  http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu;  Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org


Mime
View raw message