spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Darin McBeath <>
Subject Re: Turning off DTD Validation using XML Utils package - Spark
Date Fri, 04 Dec 2015 11:45:11 GMT
ok, a new capability has been added to spark-xml-utils (1.3.0) to address this request.

Essentially, the capability to specify 'processor' features has been added (through a new
getInstance function).  Here is a list of  the features that can be set (
 Since we are leveraging the s9apiProcessor under the covers, features relevant to that are
the only ones that would make sense to use.

To address your request of completely ignoring the Doctype declaration in the xml, you would
need to do the following:

import net.sf.saxon.lib.FeatureKeys;
HashMap<String,Object> featureMap = new HashMap<String,Object>();
featureMap.put(FeatureKeys.ENTITY_RESOLVER_CLASS, "com.somepackage.IgnoreDoctype");
// The first parameter is the xpath expression
// The second parameter is the hashmap for the namespace mappings (in this case there are
// The third parameter is the hashmap for the processor features
XPathProcessor proc = XPathProcessor.getInstance("/books/book",null,featureMap);

The following evaluation should now work ok.

proc.evaluateString("<?xml version=\"1.0\" encoding=\"UTF-8\"?><!DOCTYPE books SYSTEM
\"sample.dtd\"><books><book><title lang=\"en\">Some Book</title><author>Some
} catch (XPathException e) 

You then would define the following class (and make sure it is included in your application)

package com.somepackage;
import org.xml.sax.EntityResolver;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;

public class IgnoreDoctype implements EntityResolver {

public InputSource resolveEntity(java.lang.String publicId, java.lang.String systemId)
             throws SAXException,
          // Ignore everything
          return new InputSource(new ByteArrayInputStream("<?xml version='1.0' encoding='UTF-8'?>".getBytes()));



Lastly, you will need to include the saxon9he jar file (open source version).

This would work for XPath, XQuery, and XSLT.

Hope this helps.  When I get a chance, I will update the spark-xml-utils github site with
details on the new getInstance function and some more information on the various features.

From: Darin McBeath <>
To: "" <> 
Sent: Tuesday, December 1, 2015 11:51 AM
Subject: Re: Turning off DTD Validation using XML Utils package - Spark

The problem isn't really with DTD validation (by default validation is disabled).  The underlying
problem is that the DTD can't be found (which is indicated in your stack trace below).  The
underlying parser will try and retrieve the DTD (regardless of  validation) because things
such as entities could be expressed in the DTD.

I will explore providing access to some of the underlying 'processor' configurations.  For
example, you could provide your own EntityResolver class that could either completely ignore
the Doctype declaration (return a 'dummy' DTD that is completely empty) or you could have
it find 'local' versions (on the workers or in S3 and then cache them locally for performance).

I will post an update when the code has been adjusted.


----- Original Message -----
From: Shivalik <>
Sent: Tuesday, December 1, 2015 8:15 AM
Subject: Turning off DTD Validation using XML Utils package - Spark

Hi Team,

I've been using XML Utils library 
( to parse
XML using XPath in a spark job. One problem I am facing is with the DTDs. My
XML file, has a doctype tag included in it.

I want to turn off DTD validation using this library since I don't have
access to DTD file. Has someone faced this problem before. Please help.

The exception I am getting it is as below:

stage 0.0 (TID 0, localhost):
com.elsevier.spark_xml_utils.xpath.XPathException: I/O error reported by XML
parser processing null: <path>/filename.dtd (No such file or directory)



at com.thomsonreuters.xmlutils.XMLParser.lambda$0(

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message