ws-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Barnett <gr...@crowncollege.edu>
Subject Really large amounts of data, was Re: XML-RPC for Database Access
Date Sat, 22 Nov 2003 02:14:23 GMT
Bruce Van Horn wrote:
> about a year and a half ago I built  a transaction server that took a 
> series of queries in an xml document via xmlrpc.
> 
> It works great as long as you don't try to move really large amounts of 
> data or BLOBs.

I'm currently writing a script that needs to retrieve a large amount of 
data via XML-RPC.  When the server returns a smallish amount of data, 
everything works correctly.  When a large amount of data is returned, it 
looks like not all of the data is getting sent to the SAX parser.  I 
have tested the server with other clients, and they are able to receive 
all of the data just fine.

Here is the relevant portion of my code (if it looks a bit odd, it is 
because it is written in Jython).

############################################################
from org.apache.xmlrpc import XmlRpcClient
from org.apache.xmlrpc import XmlRpc

def xmlrpc(method, params, username, password):
     #XmlRpc.setDriver('org.apache.xerces.parsers.SAXParser')
     XmlRpc.setDebug(true)
     client = XmlRpcClient('https://host/cgi-bin/file.py')
     client.setBasicAuthentication(username, password)
     return client.execute(method, params)
############################################################

The server returns a list of dictionaries.  I've had the client 
successfully retrieve up to about 10,000 dictionaries in the list. 
Typically, it seems to have problems with more than a few hundred 
dictionaries.

My original thought was that it was the default SAX parser, but 
switching to xerces just gave a slightly different error message.

Is there any way to make sure that all of the data gets read by the parser?

Greg Barnett
firstname dot lastname at crowncollege dot edu


This is with the xerces SAX parser:

Created client to url space https://host/cgi-bin/file.py
Client calling procedure 'foo.bar' with parameters [spam , eggs]
Beginning parsing XML input stream
startElement: methodResponse
startElement: params
startElement: param
startElement: value
startElement: array
startElement: data
startElement: value
startElement: struct
startElement: member
startElement: name
endElement: name
startElement: value
startElement: string
endElement: string
endElement: value
endElement: member
startElement: member
startElement: name
[ 2700 lines snipped ]
startElement: value
startElement: struct
startElement: member
startElement: name
Fatal error parsing XML: org.xml.sax.SAXParseException: The element type 
"name" must be terminated by the matching end-tag "</name>".
org.xml.sax.SAXParseException: The element type "name" must be 
terminated by the matching end-tag "</name>".
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at org.apache.xmlrpc.XmlRpc.parse(XmlRpc.java:419)
	at org.apache.xmlrpc.XmlRpcClient$Worker.execute(XmlRpcClient.java:436)
	at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:163)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
	at org.python.core.PyMethod.__call__(PyMethod.java)
	at org.python.core.PyObject.__call__(PyObject.java)
	at org.python.core.PyInstance.invoke(PyInstance.java)
	at 
org.python.pycode._pyx0.xmlrpc$3(h:\cvswork\ccdb\client\JythonClient.py:52)
	at 
org.python.pycode._pyx0.call_function(h:\cvswork\ccdb\client\JythonClient.py)
	at org.python.core.PyTableCode.call(PyTableCode.java)
	at org.python.core.PyTableCode.call(PyTableCode.java)
	at org.python.core.PyFunction.__call__(PyFunction.java)
	at org.python.core.PyInstance.invoke(PyInstance.java)
	at 
org.python.pycode._pyx0.updateTest$4(h:\cvswork\ccdb\client\JythonClient.py:81)
	at 
org.python.pycode._pyx0.call_function(h:\cvswork\ccdb\client\JythonClient.py)
	at org.python.core.PyTableCode.call(PyTableCode.java)
	at org.python.core.PyTableCode.call(PyTableCode.java)
	at org.python.core.PyTableCode.call(PyTableCode.java)
	at org.python.core.PyFunction.__call__(PyFunction.java)
	at org.python.core.PyMethod.__call__(PyMethod.java)
	at org.python.core.PyCompoundCallable.__call__(PyCompoundCallable.java)
	at org.python.core.PyObject.__call__(PyObject.java)
	at org.python.core.PyObject._jcallexc(PyObject.java)
	at org.python.core.PyObject._jcall(PyObject.java)
	at 
org.python.proxies.java.awt.event.ActionListener$Adapter.actionPerformed(Unknown 
Source)
	at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
	at 
javax.swing.AbstractButton$ForwardActionEvents.actionPerformed(Unknown 
Source)
	at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
	at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
	at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
	at java.awt.Component.processMouseEvent(Unknown Source)
	at java.awt.Component.processEvent(Unknown Source)
	at java.awt.Container.processEvent(Unknown Source)
	at java.awt.Component.dispatchEventImpl(Unknown Source)
	at java.awt.Container.dispatchEventImpl(Unknown Source)
	at java.awt.Component.dispatchEvent(Unknown Source)
	at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
	at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
	at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
	at java.awt.Container.dispatchEventImpl(Unknown Source)
	at java.awt.Window.dispatchEventImpl(Unknown Source)
	at java.awt.Component.dispatchEvent(Unknown Source)
	at java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.awt.EventDispatchThread.pumpOneEventForHierarchy(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.run(Unknown Source)
Traceback (innermost last):
   (no code object) at line 0








this is with the default parser:

Created client to url space https://host/cgi-bin/file.py
Client calling procedure 'foo.bar' with parameters [spam, eggs]
Beginning parsing XML input stream
startElement: methodResponse
startElement: params
startElement: param
startElement: value
startElement: array
startElement: data
startElement: value
startElement: struct
startElement: member
startElement: name
endElement: name
startElement: value
startElement: string
endElement: string
endElement: value
endElement: member
startElement: member
startElement: name
<snip>
endElement: string
endElement: value
endElement: member
startElement: member
startElement: name
endElement: name
startElement: value
startElement: string
endElement: string
endElement: value
endElement: member
endElement: struct
endElement: value
startElement: value
startElement: struct
startElement: member
startElement: name
Fatal error parsing XML: org.xml.sax.SAXParseException: unexpected end 
of file found
Spent 401 millis parsing
Spent 10325 in request





Mime
View raw message