spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong(Ryan) Zhu" <shixi...@databricks.com>
Subject Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)
Date Tue, 03 Jul 2018 20:22:26 GMT
-dev +user

Hey Peter,

You should upgrade "org.apache.spark_spark-sql-kafka-0-10_2.11-2.2.0.jar"
to 2.3.1. The doc link you posted is for DStream. You should take a look at
https://spark.apache.org/docs/2.3.1/structured-streaming-kafka-integration.html.
You don't need "spark-streaming-kafka-0-10" at all if you are using
Structured Streaming. This jar is for DStream.



Best Regards,
Ryan

On Tue, Jul 3, 2018 at 8:55 AM, Peter Liu <peter.pliu@gmail.com> wrote:

> Hi Sean,
>
> thanks for your reply (sorry for the late response - it was a long weekend
> here).
>
> do you happen to have any suggestion for any specific area or jars I
> should take a close look? (i have the spark environment page attached
> below).
>
> This is very much appreciated!
>
> Peter
>
> Environment
> Runtime Information
> NameValue
> Java Home /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
> Java Version 1.8.0-internal (Oracle Corporation)
> Scala Version version 2.11.8 Spark Properties
> NameValue
> spark.serializer org.apache.spark.serializer.KryoSerializer
> spark.executor.extraJavaOptions -XX:ParallelGCThreads=4
> spark.driver.host isvhdp11lab.spark.ibm.com
> spark.history.fs.logDirectory hdfs://isvhdp7lab/sparkLogs
> spark.eventLog.enabled true
> spark.executor.heartbeatInterval 60s
> spark.ui.port 0
> spark.driver.port 39641
> spark.yarn.app.id application_1530116533665_0008
> spark.app.name Consumer Cfg: Stream_Dur300000_RecPerSec60000_camp100_
> recGenNumP10
> spark.network.timeout 1200
> spark.scheduler.mode FIFO
> spark.driver.memory 4g
> spark.executor.instances 60
> spark.yarn.secondary.jars org.apache.kafka_kafka-clients-0.10.0.1.jar,org.
> apache.spark_spark-sql-kafka-0-10_2.11-2.2.0.jar,org.spark-
> project.spark_unused-1.0.0.jar,net.jpountz.lz4_lz4-1.3.0.
> jar,org.xerial.snappy_snappy-java-1.1.2.6.jar,org.slf4j_
> slf4j-api-1.7.16.jar
> spark.executor.id driver
> spark.streaming.stopGracefullyOnShutdown true
> spark.yarn.app.container.log.dir *********(redacted)
> spark.submit.deployMode cluster
> spark.master yarn
> spark.ui.filters org.apache.hadoop.yarn.server.
> webproxy.amfilter.AmIpFilter
> spark.executor.memory 3g
> spark.eventLog.dir hdfs://isvhdp7lab/sparkLogs
> spark.executor.cores 1
> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.
> AmIpFilter.param.PROXY_HOSTS isvhdp7.aus.stglabs.ibm.com
> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.
> AmIpFilter.param.PROXY_URI_BASES http://isvhdp7.aus.stglabs.
> ibm.com:8088/proxy/application_1530116533665_0008
> spark.yarn.dist.jars file:///home/pgl/SparkStreamingBenchmark-
> RemoteConsumer/org.apache.kafka_kafka-clients-0.10.0.1.
> jar,file:///home/pgl/.ivy2/jars/org.apache.spark_spark-
> sql-kafka-0-10_2.11-2.2.0.jar,file:///home/pgl/.ivy2/jars/
> org.apache.kafka_kafka-clients-0.10.0.1.jar,file:///
> home/pgl/.ivy2/jars/org.spark-project.spark_unused-1.0.0.
> jar,file:///home/pgl/.ivy2/jars/net.jpountz.lz4_lz4-1.3.
> 0.jar,file:///home/pgl/.ivy2/jars/org.xerial.snappy_snappy-
> java-1.1.2.6.jar,file:///home/pgl/.ivy2/jars/org.slf4j_
> slf4j-api-1.7.16.jar
> spark.app.id application_1530116533665_0008
> spark.sql.shuffle.partitions 40 System Properties
> NameValue
> java.io.tmpdir /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/tmp
> line.separator
> path.separator :
> sun.management.compiler HotSpot 64-Bit Tiered Compilers
> sun.cpu.endian little
> java.specification.version 1.8
> java.vm.specification.name Java Virtual Machine Specification
> java.vendor Oracle Corporation
> java.vm.specification.version 1.8
> user.home /home/pgl
> file.encoding.pkg sun.io
> sun.nio.ch.bugLevel
> sun.arch.data.model 64
> sun.boot.library.path /home/pgl/root-dir/jdk8_
> trlopt_55cf01d/jre/lib/ppc64le
> user.dir /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001
> java.library.path /home/maha/hadoop-2.9.0/lib/
> native/:/home/maha/hadoop-2.9.0/lib/native/:/home/pgl/
> hadoop-2.9.0/lib/native::/usr/java/packages/lib/ppc64le:/
> usr/lib64:/lib64:/lib:/usr/lib
> sun.cpu.isalist
> os.arch
> ppc64le
> java.vm.version 25.71-b00
> java.endorsed.dirs /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/endorsed
> java.runtime.version 1.8.0-internal-root_2018_04_19_13_32-b00
> java.vm.info mixed mode
> java.ext.dirs /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/ext:/
> usr/java/packages/lib/ext
> java.runtime.name OpenJDK Runtime Environment
> file.separator /
> java.class.version 52.0
> java.specification.name Java Platform API Specification
> sun.boot.class.path /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/
> resources.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/
> lib/rt.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/
> sunrsasign.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/
> lib/jsse.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/
> lib/jce.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/
> lib/charsets.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/
> jre/lib/jfr.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/classes
> file.encoding UTF-8
> user.timezone America/Chicago
> java.specification.vendor Oracle Corporation
> sun.java.launcher SUN_STANDARD
> java.vm.compressedOopsMode Zero based
> os.version 4.14.0-81.el7a.ppc64le
> sun.os.patch.level unknown
> java.vm.specification.vendor Oracle Corporation
> user.country US
> sun.jnu.encoding UTF-8
> user.language en
> java.vendor.url http://java.oracle.com/
> java.awt.printerjob sun.print.PSPrinterJob
> java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
> awt.toolkit sun.awt.X11.XToolkit
> os.name Linux
> java.vm.vendor Oracle Corporation
> java.vendor.url.bug http://bugreport.sun.com/bugreport/
> user.name pgl
> java.vm.name OpenJDK 64-Bit Server VM
> sun.java.command org.apache.spark.deploy.yarn.ApplicationMaster --class
> SparkStreamingTest --jar file:/home/pgl/SparkStreamingBenchmark-
> RemoteConsumer/target/scala-2.11/sparkstreamingbenchmark-
> remoteconsumer_2.11-0.1-SNAPSHOT.jar --arg 300000 --arg 60000 --arg 20
> --arg 10 --arg 100 --arg 80 --arg 80 --arg 80 --arg Stream_Dur300000_
> RecPerSec60000_camp100_recGenNumP10 --properties-file
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_conf__/__spark_conf__.properties
> java.home /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
> java.version 1.8.0-internal
> sun.io.unicode.encoding UnicodeLittle Classpath Entries
> ResourceSource
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax-api-1.0-2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-beanutils-core-1.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guava-11.0.2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/datanucleus-api-jdo-3.2.6.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/guava-11.0.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-util-1.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-hive_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javax.inject-1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-recipes-2.7.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/libthrift-0.9.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/okio-1.13.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-cli-1.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-shuffle-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-3.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/leveldbjni-all-1.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-network-common_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-nodemanager-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/okhttp-3.8.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/junit-4.11.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/json4s-ast_2.11-3.2.11.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-configuration-1.6.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/avro-1.7.7.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/generex-1.0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/arpack_combined_all-0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-common-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-jaxrs-1.9.13.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-cli-1.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/kubernetes-client-3.0.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/eigenbase-properties-1.1.5.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/pyrolite-4.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-2.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang-2.6.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-jobclient-2.9.0-tests.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-hs-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-kubernetes_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/log4j-1.2.17.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spire_2.11-0.13.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-encoding-1.8.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-common-2.22.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-6.1.26.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-test-2.7.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-net-2.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/avro-1.7.7.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/logging-interceptor-3.8.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-compress-1.4.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-auth-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-net-3.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/metrics-jvm-3.1.5.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-registry-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/machinist_2.11-0.6.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/xz-1.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpclient-4.5.2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jetty-util-6.1.26.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/stax-api-1.0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jsr305-1.3.9.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/netty-all-4.1.17.Final.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/arrow-memory-0.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/antlr-runtime-3.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javax.inject-1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/orc-core-1.4.4-nohive.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-io-2.5.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/metrics-core-3.1.5.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-graphx_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/chill_2.11-0.8.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-launcher_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/paranamer-2.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-compress-1.4.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-client-2.7.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpclient-4.5.2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-hadoop-bundle-1.6.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/avro-1.7.7.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-sharedcachemanager-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/zjsonpatch-0.3.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-framework-2.7.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-guava-2.22.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/gson-2.2.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-catalyst_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/geronimo-
> jcache_1.0_spec-1.0-alpha-1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/snappy-0.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/log4j-1.2.17.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/compress-lzf-1.0.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hive-beeline-1.2.1.spark2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-digester-1.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/metrics-core-3.0.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/super-csv-2.2.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-dataformat-yaml-2.6.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-core-2.6.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/JavaEWAH-0.3.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javax.ws.rs-api-2.0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-mapreduce-client-core-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/apache-log4j-extras-1.2.17.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/asm-3.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/ehcache-3.3.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-json-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/asm-3.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsp-api-2.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javolution-5.5.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-databind-2.6.7.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xmlenc-0.52.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-mesos_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/janino-3.0.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/htrace-core-3.1.0-incubating.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-common-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/breeze_2.11-0.13.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang-2.6.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-format-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/snappy-java-1.0.5.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/mesos-1.4.0-shaded-protobuf.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/woodstox-core-5.0.3.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-nfs-2.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/servlet-api-2.5.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-dbcp-1.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-core-asl-1.9.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-logging-1.1.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/asm-3.2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jcl-over-slf4j-1.7.16.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math-2.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/javax.inject-1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jets3t-0.9.4.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-app-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/libfb303-0.9.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/core-1.1.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-mllib_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-mapreduce-client-shuffle-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jpam-1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-util-6.1.26.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-client-2.7.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-core_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/api-asn1-api-1.0.0-M20.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jtransforms-2.4.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-network-shuffle_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-math3-3.4.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/snappy-java-1.0.5.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-column-1.8.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/mockito-all-1.8.5.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jaxb-api-2.2.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/breeze-macros_2.11-0.13.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-kvstore_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-mapreduce-client-jobclient-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/base64-2.3.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scala-xml_2.11-1.0.5.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/oro-2.0.8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jetty-6.1.26.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/aopalliance-repackaged-2.4.0-b34.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-logging-1.1.3.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-hdfs-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jets3t-0.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/guava-11.0.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-
> kerberos-codec-2.0.0-M15.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-mapreduce-client-common-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/metrics-graphite-3.1.5.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scala-compiler-2.11.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-server-2.22.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/aopalliance-1.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/joda-time-2.9.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-resourcemanager-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/netty-3.9.9.Final.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/activation-1.1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax2-api-3.1.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/gson-2.2.4.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsch-0.1.54.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scalap-2.11.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/py4j-0.10.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/xmlenc-0.52.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-tests-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/curator-client-2.7.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javassist-3.18.1-GA.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-core-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-
> beanutils-core-1.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/kryo-shaded-3.0.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/httpcore-4.4.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/calcite-core-1.2.0-incubating.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-compiler-3.0.8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-yarn-server-common-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/zookeeper-3.4.6.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-
> configuration-1.6.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/stringtemplate-3.2.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/curator-framework-2.7.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/antlr-2.7.7.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsr305-3.0.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-timeline-pluginstorage-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-math3-3.1.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-jackson-1.8.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-mllib-local_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scala-parser-combinators_2.11-1.0.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javax.inject-2.4.0-b34.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-client-2.22.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsch-0.1.54.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/httpclient-4.5.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-httpclient-3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jsr305-3.0.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/lz4-java-1.4.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jta-1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/protobuf-java-2.5.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/snakeyaml-1.15.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-auth-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/datanucleus-core-3.2.10.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/avro-mapred-1.7.7-hadoop2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/aopalliance-1.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javax.servlet-api-3.1.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/etc/hadoop System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-media-jaxb-2.22.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-unsafe_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-xc-1.9.13.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/antlr4-runtime-4.7.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-
> native-client-2.9.0-tests.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-yarn-server-web-proxy-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-sketch_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/objenesis-2.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/guava-14.0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/apacheds-kerberos-codec-2.0.0-M15.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-
> compress-1.4.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/slf4j-api-1.7.16.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/avro-1.7.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/ivy-2.4.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-yarn-client-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_conf__/__hadoop_conf__ System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-api-1.7.25.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-web-proxy-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-pool-1.5.4.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-framework-2.7.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xz-1.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hk2-api-2.4.0-b34.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jettison-1.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/calcite-linq4j-1.2.0-incubating.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hppc-0.7.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-
> annotations-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-module-scala_2.11-2.6.7.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/guice-3.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/kubernetes-model-2.0.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hive-jdbc-1.2.1.spark2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-io-2.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-repl_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-hadoop-1.8.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/osgi-resource-locator-1.0.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-annotations-2.6.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/opencsv-2.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/log4j-1.2.17.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/apacheds-i18n-2.0.0-M15.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-
> collections-3.2.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-common-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-3.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/activation-1.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-io-2.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-codec-1.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/aircompressor-0.8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/nimbus-jose-jwt-3.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-router-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/zstd-jni-1.3.2-2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-yarn_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-
> kerberos-codec-2.0.0-M15.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
> applicationhistoryservice-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/json4s-jackson_2.11-3.2.11.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/validation-api-1.1.0.Final.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-common-1.8.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-lang3-3.5.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okio-1.4.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/metrics-json-3.1.5.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jline-2.12.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/paranamer-2.3.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-codec-1.10.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/fst-2.50.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-configuration-1.6.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-smart-1.1.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-client-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-core-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xmlenc-0.52.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/derby-10.12.1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-hs-plugins-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/__spark_conf__ System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okhttp-2.4.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0-tests.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jcip-annotations-1.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/minlog-1.3.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-io-2.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/flatbuffers-1.2.0-3f79e055.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax2-api-3.1.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/json4s-core_2.11-3.2.11.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hive-cli-1.2.1.spark2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpcore-4.4.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/netty-3.6.2.Final.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hk2-utils-2.4.0-b34.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-yarn-api-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jettison-1.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/curator-recipes-2.7.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-container-servlet-core-2.22.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/api-util-1.0.0-M20.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hamcrest-core-1.3.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math3-3.1.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hk2-locator-2.4.0-b34.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-crypto-1.0.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> applications-distributedshell-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/ST4-4.0.4.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-core-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpcore-4.4.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/guice-servlet-3.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-lang-2.6.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0-tests.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-sql_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/json-smart-1.1.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/snappy-java-1.0.5.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-
> native-client-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/xercesImpl-2.9.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scala-library-2.11.8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-6.1.26.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-common-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-digester-1.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-beanutils-1.7.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/htrace-
> core4-4.1.0-incubating.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsp-api-2.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-yarn-common-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/automaton-1.11-8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-net-3.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/htrace-core4-
> 4.1.0-incubating.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/woodstox-core-5.0.3.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-api-2.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jets3t-0.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/univocity-parsers-2.5.9.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-codec-1.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/stream-2.7.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spire-macros_2.11-0.13.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/datanucleus-rdbms-3.2.9.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang3-3.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hadoop-
> annotations-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-
> core-asl-1.9.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-streaming_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang3-3.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-
> mapper-asl-1.9.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-server-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/zookeeper-3.4.6.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/snappy-java-1.1.2.6.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001 System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-io-2.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-tags_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jodd-core-3.5.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/RoaringBitmap-0.5.11.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-container-servlet-2.22.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-jobclient-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-examples-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/slf4j-log4j12-1.7.16.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/activation-1.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/arrow-vector-0.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/paranamer-2.3.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-annotations-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/asm-3.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/arrow-format-0.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/servlet-api-2.5.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/junit-4.11.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/shapeless_2.11-2.3.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-client-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-module-paranamer-2.7.9.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jsp-api-2.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-
> beanutils-core-1.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/macro-compat_2.11-1.1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-client-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-hive-thriftserver_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> applications-unmanaged-am-launcher-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/stax-api-1.0-2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-cli-1.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xz-1.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-
> mapper-asl-1.9.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-json-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/bonecp-0.8.0.RELEASE.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-
> client-2.9.0-tests.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/bcprov-jdk15on-1.58.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-server-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/chill-java-0.8.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-collections-3.2.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-mapper-asl-1.9.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javax.annotation-api-1.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/gson-2.2.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-mapreduce-client-app-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jdo-api-3.0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/avro-ipc-1.7.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/xz-1.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jul-to-slf4j-1.7.16.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsr305-3.0.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/calcite-avatica-1.2.0-incubating.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/htrace-core4-
> 4.1.0-incubating.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/java-xmlbuilder-1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hive-metastore-1.2.1.spark2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scala-reflect-2.11.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/orc-mapreduce-1.4.4-nohive.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hive-exec-1.2.1.spark2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/xbean-asm5-shaded-4.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-digester-1.8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jcip-annotations-1.0.jar
>
>
> On Sun, Jul 1, 2018 at 10:09 AM, Sean Owen <srowen@gmail.com> wrote:
>
>> Somewhere, you have mismatched versions of Spark on your classpath.
>>
>>
>> On Sun, Jul 1, 2018, 9:01 AM Peter Liu <peter.pliu@gmail.com> wrote:
>>
>>> Hello there,
>>>
>>> I didn't get any response/help from the user list for the following
>>> question and thought people on the dev list might be able to help?:
>>>
>>> I upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload
>>> and got the error (java.lang.AbstractMethodError) never seen before;
>>> see the error exception stack attached in (a) bellow.
>>>
>>> anyone knows if  spark 2.3.1 works well with kafka
>>> spark-streaming-kafka-0-10?
>>>
>>> this link of spark kafka integration page doesn't say anything about any
>>> limitation:
>>> https://spark.apache.org/docs/2.3.1/streaming-kafka-integration.html
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_2.3.1_streaming-2Dkafka-2Dintegration.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=LORvnW8MyMYoQuiRJpv4qo1519IpQ57V-MxUgro0vP0&e=>
>>>
>>> but this discussion seems to say there is indeed an issue when upgrading
>>> to spark 2.3.1:
>>> https://stackoverflow.com/questions/49180931/abstractmethode
>>> rror-creating-kafka-stream
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_49180931_abstractmethoderror-2Dcreating-2Dkafka-2Dstream&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=xIl7y-rgXq2AUuNg8cKeBIRApmRMLnrJbPsf4kc5Zzg&e=>
>>>
>>> i also rebuilt the workload with some spark 2.3.1 jars (see (b) below).
>>> it doesn't seem to help.
>>>
>>> Would be great if anyone could kindly share any insights here.
>>>
>>> Thanks!
>>>
>>> Peter
>>>
>>> (a) the exception
>>> Exception in thread "stream execution thread for [id =
>>> 5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId =
>>> 70e78d5c-665e-4c6f-a0cc-41a56e488e30]" java.lang.AbstractMethodError
>>>         at org.apache.spark.internal.Logging$class.initializeLogIfNeces
>>> sary(Logging.scala:99)
>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializ
>>> eLogIfNecessary(KafkaSourceProvider.scala:369)
>>>         at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(Kafka
>>> SourceProvider.scala:369)
>>>         at org.apache.spark.internal.Logging$class.logDebug(Logging.
>>> scala:58)
>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(
>>> KafkaSourceProvider.scala:369)
>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpda
>>> ter.set(KafkaSourceProvider.scala:439)
>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaPara
>>> msForDriver(KafkaSourceProvider.scala:394)
>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSour
>>> ce(KafkaSourceProvider.scala:90)
>>>         at org.apache.spark.sql.execution.datasources.DataSource.
>>> createSource(DataSource.scala:277)
>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> $$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:80)
>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> $$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:77)
>>>         at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLi
>>> ke.scala:194)
>>>         at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.
>>> scala:80)
>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> $$anonfun$1.applyOrElse(MicroBatchExecution.scala:77)
>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> $$anonfun$1.applyOrElse(MicroBatchExecution.scala:75)
>>>         at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.appl
>>> y(TreeNode.scala:267)
>>>         at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.appl
>>> y(TreeNode.scala:267)
>>>         at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigi
>>> n(TreeNode.scala:70)
>>>         at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(T
>>> reeNode.scala:266)
>>>         at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeN
>>> ode.scala:256)
>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> .logicalPlan$lzycompute(MicroBatchExecution.scala:75)
>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> .logicalPlan(MicroBatchExecution.scala:61)
>>>         at org.apache.spark.sql.execution.streaming.StreamExecution.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.sql.execution.streaming.StreamExecution.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=lU8N2IsjW321Bu4T4BHLHvj1OSsKdPI_R_ahmOgq9rA&e=>
>>> $apache$spark$sql$execution$streaming$StreamExecution$$
>>> runStream(StreamExecution.scala:265)
>>>         at org.apache.spark.sql.execution.streaming.StreamExecution$$
>>> anon$1.run(StreamExecution.scala:189)
>>>
>>> (b)* the build script update:*
>>>
>>> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff
>>> build.sbt spk211-build.sbt.original
>>> 10,11c10,11
>>> < libraryDependencies += "org.apache.spark" % "spark-sql_2.11" %*
>>> "2.3.1"*
>>> < libraryDependencies += "org.apache.spark" % "spark-core_2.11" %*
>>> "2.3.1"*
>>> ---
>>> > libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1"
>>> > libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.1"
>>> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$
>>>
>>>
>>>
>>>
>

Mime
View raw message