ws-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Hoegg <rho...@isisnetworks.net>
Subject Re: Really large amounts of data, was Re: XML-RPC for Database Access
Date Sat, 22 Nov 2003 03:12:45 GMT
I had a similar issue because the server was using HTTP 1.1 and the 
client was requesting HTTP 1.1 but only accepting HTTP 1.0.  HTTP 1.1 
uses chunking, which messes up the XML returned by the server if it is 
interpreted as if it was not chunked.

You can fix it by
a) changing the server to use HTTP 1.0
b) making the client request HTTP 1.0
c) making the client understand HTTP 1.1

I opted for b first, then c.  If you use the code in CVS HEAD, you can 
plug in a CommonsXmlRpcClient, which supports HTTP 1.1

--
Ryan Hoegg
ISIS Networks
http://www.isisnetworks.net

Greg Barnett wrote:

> Bruce Van Horn wrote:
>
>> about a year and a half ago I built  a transaction server that took a 
>> series of queries in an xml document via xmlrpc.
>>
>> It works great as long as you don't try to move really large amounts 
>> of data or BLOBs.
>
>
> I'm currently writing a script that needs to retrieve a large amount 
> of data via XML-RPC.  When the server returns a smallish amount of 
> data, everything works correctly.  When a large amount of data is 
> returned, it looks like not all of the data is getting sent to the SAX 
> parser.  I have tested the server with other clients, and they are 
> able to receive all of the data just fine.
>
> Here is the relevant portion of my code (if it looks a bit odd, it is 
> because it is written in Jython).
>
> ############################################################
> from org.apache.xmlrpc import XmlRpcClient
> from org.apache.xmlrpc import XmlRpc
>
> def xmlrpc(method, params, username, password):
>     #XmlRpc.setDriver('org.apache.xerces.parsers.SAXParser')
>     XmlRpc.setDebug(true)
>     client = XmlRpcClient('https://host/cgi-bin/file.py')
>     client.setBasicAuthentication(username, password)
>     return client.execute(method, params)
> ############################################################
>
> The server returns a list of dictionaries.  I've had the client 
> successfully retrieve up to about 10,000 dictionaries in the list. 
> Typically, it seems to have problems with more than a few hundred 
> dictionaries.
>
> My original thought was that it was the default SAX parser, but 
> switching to xerces just gave a slightly different error message.
>
> Is there any way to make sure that all of the data gets read by the 
> parser?
>
> Greg Barnett
> firstname dot lastname at crowncollege dot edu
>
>
> This is with the xerces SAX parser:
>
> Created client to url space https://host/cgi-bin/file.py
> Client calling procedure 'foo.bar' with parameters [spam , eggs]
> Beginning parsing XML input stream
> startElement: methodResponse
> startElement: params
> startElement: param
> startElement: value
> startElement: array
> startElement: data
> startElement: value
> startElement: struct
> startElement: member
> startElement: name
> endElement: name
> startElement: value
> startElement: string
> endElement: string
> endElement: value
> endElement: member
> startElement: member
> startElement: name
> [ 2700 lines snipped ]
> startElement: value
> startElement: struct
> startElement: member
> startElement: name
> Fatal error parsing XML: org.xml.sax.SAXParseException: The element 
> type "name" must be terminated by the matching end-tag "</name>".
> org.xml.sax.SAXParseException: The element type "name" must be 
> terminated by the matching end-tag "</name>".
>     at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
>     at org.apache.xmlrpc.XmlRpc.parse(XmlRpc.java:419)
>     at 
> org.apache.xmlrpc.XmlRpcClient$Worker.execute(XmlRpcClient.java:436)
>     at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:163)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>     at java.lang.reflect.Method.invoke(Unknown Source)
>     at 
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
>     at org.python.core.PyMethod.__call__(PyMethod.java)
>     at org.python.core.PyObject.__call__(PyObject.java)
>     at org.python.core.PyInstance.invoke(PyInstance.java)
>     at 
> org.python.pycode._pyx0.xmlrpc$3(h:\cvswork\ccdb\client\JythonClient.py:52) 
>
>     at 
> org.python.pycode._pyx0.call_function(h:\cvswork\ccdb\client\JythonClient.py) 
>
>     at org.python.core.PyTableCode.call(PyTableCode.java)
>     at org.python.core.PyTableCode.call(PyTableCode.java)
>     at org.python.core.PyFunction.__call__(PyFunction.java)
>     at org.python.core.PyInstance.invoke(PyInstance.java)
>     at 
> org.python.pycode._pyx0.updateTest$4(h:\cvswork\ccdb\client\JythonClient.py:81) 
>
>     at 
> org.python.pycode._pyx0.call_function(h:\cvswork\ccdb\client\JythonClient.py) 
>
>     at org.python.core.PyTableCode.call(PyTableCode.java)
>     at org.python.core.PyTableCode.call(PyTableCode.java)
>     at org.python.core.PyTableCode.call(PyTableCode.java)
>     at org.python.core.PyFunction.__call__(PyFunction.java)
>     at org.python.core.PyMethod.__call__(PyMethod.java)
>     at 
> org.python.core.PyCompoundCallable.__call__(PyCompoundCallable.java)
>     at org.python.core.PyObject.__call__(PyObject.java)
>     at org.python.core.PyObject._jcallexc(PyObject.java)
>     at org.python.core.PyObject._jcall(PyObject.java)
>     at 
> org.python.proxies.java.awt.event.ActionListener$Adapter.actionPerformed(Unknown 
> Source)
>     at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
>     at 
> javax.swing.AbstractButton$ForwardActionEvents.actionPerformed(Unknown 
> Source)
>     at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
>     at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
>     at 
> javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
>     at java.awt.Component.processMouseEvent(Unknown Source)
>     at java.awt.Component.processEvent(Unknown Source)
>     at java.awt.Container.processEvent(Unknown Source)
>     at java.awt.Component.dispatchEventImpl(Unknown Source)
>     at java.awt.Container.dispatchEventImpl(Unknown Source)
>     at java.awt.Component.dispatchEvent(Unknown Source)
>     at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
>     at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
>     at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
>     at java.awt.Container.dispatchEventImpl(Unknown Source)
>     at java.awt.Window.dispatchEventImpl(Unknown Source)
>     at java.awt.Component.dispatchEvent(Unknown Source)
>     at java.awt.EventQueue.dispatchEvent(Unknown Source)
>     at java.awt.EventDispatchThread.pumpOneEventForHierarchy(Unknown 
> Source)
>     at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown 
> Source)
>     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
>     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
>     at java.awt.EventDispatchThread.run(Unknown Source)
> Traceback (innermost last):
>   (no code object) at line 0
>
>
>
>
>
>
>
>
> this is with the default parser:
>
> Created client to url space https://host/cgi-bin/file.py
> Client calling procedure 'foo.bar' with parameters [spam, eggs]
> Beginning parsing XML input stream
> startElement: methodResponse
> startElement: params
> startElement: param
> startElement: value
> startElement: array
> startElement: data
> startElement: value
> startElement: struct
> startElement: member
> startElement: name
> endElement: name
> startElement: value
> startElement: string
> endElement: string
> endElement: value
> endElement: member
> startElement: member
> startElement: name
> <snip>
> endElement: string
> endElement: value
> endElement: member
> startElement: member
> startElement: name
> endElement: name
> startElement: value
> startElement: string
> endElement: string
> endElement: value
> endElement: member
> endElement: struct
> endElement: value
> startElement: value
> startElement: struct
> startElement: member
> startElement: name
> Fatal error parsing XML: org.xml.sax.SAXParseException: unexpected end 
> of file found
> Spent 401 millis parsing
> Spent 10325 in request
>
>
>
>



Mime
View raw message