spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Liu <peter.p...@gmail.com>
Subject Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)
Date Tue, 03 Jul 2018 23:32:05 GMT
 Thank you, Ryan!
I'll keep you guys updated.

Cheers,

Peter

On Tue, Jul 3, 2018 at 4:22 PM, Shixiong(Ryan) Zhu <shixiong@databricks.com>
wrote:

> -dev +user
>
> Hey Peter,
>
> You should upgrade "org.apache.spark_spark-sql-kafka-0-10_2.11-2.2.0.jar"
> to 2.3.1. The doc link you posted is for DStream. You should take a look at
> https://spark.apache.org/docs/2.3.1/structured-
> streaming-kafka-integration.html. You don't need
> "spark-streaming-kafka-0-10" at all if you are using Structured Streaming.
> This jar is for DStream.
>
>
>
> Best Regards,
> Ryan
>
> On Tue, Jul 3, 2018 at 8:55 AM, Peter Liu <peter.pliu@gmail.com> wrote:
>
>> Hi Sean,
>>
>> thanks for your reply (sorry for the late response - it was a long
>> weekend here).
>>
>> do you happen to have any suggestion for any specific area or jars I
>> should take a close look? (i have the spark environment page attached
>> below).
>>
>> This is very much appreciated!
>>
>> Peter
>>
>> Environment
>> Runtime Information
>> NameValue
>> Java Home /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
>> Java Version 1.8.0-internal (Oracle Corporation)
>> Scala Version version 2.11.8 Spark Properties
>> NameValue
>> spark.serializer org.apache.spark.serializer.KryoSerializer
>> spark.executor.extraJavaOptions -XX:ParallelGCThreads=4
>> spark.driver.host isvhdp11lab.spark.ibm.com
>> spark.history.fs.logDirectory hdfs://isvhdp7lab/sparkLogs
>> spark.eventLog.enabled true
>> spark.executor.heartbeatInterval 60s
>> spark.ui.port 0
>> spark.driver.port 39641
>> spark.yarn.app.id application_1530116533665_0008
>> spark.app.name Consumer Cfg: Stream_Dur300000_RecPerSec6000
>> 0_camp100_recGenNumP10
>> spark.network.timeout 1200
>> spark.scheduler.mode FIFO
>> spark.driver.memory 4g
>> spark.executor.instances 60
>> spark.yarn.secondary.jars org.apache.kafka_kafka-clients
>> -0.10.0.1.jar,org.apache.spark_spark-sql-kafka-0-10_2.
>> 11-2.2.0.jar,org.spark-project.spark_unused-1.0.0.jar,net.
>> jpountz.lz4_lz4-1.3.0.jar,org.xerial.snappy_snappy-java-1.1.
>> 2.6.jar,org.slf4j_slf4j-api-1.7.16.jar
>> spark.executor.id driver
>> spark.streaming.stopGracefullyOnShutdown true
>> spark.yarn.app.container.log.dir *********(redacted)
>> spark.submit.deployMode cluster
>> spark.master yarn
>> spark.ui.filters org.apache.hadoop.yarn.server.
>> webproxy.amfilter.AmIpFilter
>> spark.executor.memory 3g
>> spark.eventLog.dir hdfs://isvhdp7lab/sparkLogs
>> spark.executor.cores 1
>> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFi
>> lter.param.PROXY_HOSTS isvhdp7.aus.stglabs.ibm.com
>> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFi
>> lter.param.PROXY_URI_BASES http://isvhdp7.aus.stglabs.ibm
>> .com:8088/proxy/application_1530116533665_0008
>> spark.yarn.dist.jars file:///home/pgl/SparkStreamin
>> gBenchmark-RemoteConsumer/org.apache.kafka_kafka-clients-0.
>> 10.0.1.jar,file:///home/pgl/.ivy2/jars/org.apache.spark_
>> spark-sql-kafka-0-10_2.11-2.2.0.jar,file:///home/pgl/.ivy2/
>> jars/org.apache.kafka_kafka-clients-0.10.0.1.jar,file:///hom
>> e/pgl/.ivy2/jars/org.spark-project.spark_unused-1.0.0.jar,
>> file:///home/pgl/.ivy2/jars/net.jpountz.lz4_lz4-1.3.0.jar,
>> file:///home/pgl/.ivy2/jars/org.xerial.snappy_snappy-java-
>> 1.1.2.6.jar,file:///home/pgl/.ivy2/jars/org.slf4j_slf4j-api-1.7.16.jar
>> spark.app.id application_1530116533665_0008
>> spark.sql.shuffle.partitions 40 System Properties
>> NameValue
>> java.io.tmpdir /spark2/pgl/yarn/usercache/pgl
>> /appcache/application_1530116533665_0008/container_153011653
>> 3665_0008_02_000001/tmp
>> line.separator
>> path.separator :
>> sun.management.compiler HotSpot 64-Bit Tiered Compilers
>> sun.cpu.endian little
>> java.specification.version 1.8
>> java.vm.specification.name Java Virtual Machine Specification
>> java.vendor Oracle Corporation
>> java.vm.specification.version 1.8
>> user.home /home/pgl
>> file.encoding.pkg sun.io
>> sun.nio.ch.bugLevel
>> sun.arch.data.model 64
>> sun.boot.library.path /home/pgl/root-dir/jdk8_trlopt
>> _55cf01d/jre/lib/ppc64le
>> user.dir /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001
>> java.library.path /home/maha/hadoop-2.9.0/lib/na
>> tive/:/home/maha/hadoop-2.9.0/lib/native/:/home/pgl/hadoop-
>> 2.9.0/lib/native::/usr/java/packages/lib/ppc64le:/usr/
>> lib64:/lib64:/lib:/usr/lib
>> sun.cpu.isalist
>> os.arch
>> ppc64le
>> java.vm.version 25.71-b00
>> java.endorsed.dirs /home/pgl/root-dir/jdk8_trlopt
>> _55cf01d/jre/lib/endorsed
>> java.runtime.version 1.8.0-internal-root_2018_04_19_13_32-b00
>> java.vm.info mixed mode
>> java.ext.dirs /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/ext:/usr/
>> java/packages/lib/ext
>> java.runtime.name OpenJDK Runtime Environment
>> file.separator /
>> java.class.version 52.0
>> java.specification.name Java Platform API Specification
>> sun.boot.class.path /home/pgl/root-dir/jdk8_trlopt
>> _55cf01d/jre/lib/resources.jar:/home/pgl/root-dir/jdk8_
>> trlopt_55cf01d/jre/lib/rt.jar:/home/pgl/root-dir/jdk8_
>> trlopt_55cf01d/jre/lib/sunrsasign.jar:/home/pgl/root-dir/
>> jdk8_trlopt_55cf01d/jre/lib/jsse.jar:/home/pgl/root-dir/
>> jdk8_trlopt_55cf01d/jre/lib/jce.jar:/home/pgl/root-dir/
>> jdk8_trlopt_55cf01d/jre/lib/charsets.jar:/home/pgl/root-
>> dir/jdk8_trlopt_55cf01d/jre/lib/jfr.jar:/home/pgl/root-
>> dir/jdk8_trlopt_55cf01d/jre/classes
>> file.encoding UTF-8
>> user.timezone America/Chicago
>> java.specification.vendor Oracle Corporation
>> sun.java.launcher SUN_STANDARD
>> java.vm.compressedOopsMode Zero based
>> os.version 4.14.0-81.el7a.ppc64le
>> sun.os.patch.level unknown
>> java.vm.specification.vendor Oracle Corporation
>> user.country US
>> sun.jnu.encoding UTF-8
>> user.language en
>> java.vendor.url http://java.oracle.com/
>> java.awt.printerjob sun.print.PSPrinterJob
>> java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
>> awt.toolkit sun.awt.X11.XToolkit
>> os.name Linux
>> java.vm.vendor Oracle Corporation
>> java.vendor.url.bug http://bugreport.sun.com/bugreport/
>> user.name pgl
>> java.vm.name OpenJDK 64-Bit Server VM
>> sun.java.command org.apache.spark.deploy.yarn.ApplicationMaster --class
>> SparkStreamingTest --jar file:/home/pgl/SparkStreamingB
>> enchmark-RemoteConsumer/target/scala-2.11/sparkstreamingbenchmark-remoteconsumer_2.11-0.1-SNAPSHOT.jar
>> --arg 300000 --arg 60000 --arg 20 --arg 10 --arg 100 --arg 80 --arg 80
>> --arg 80 --arg Stream_Dur300000_RecPerSec60000_camp100_recGenNumP10
>> --properties-file /spark2/pgl/yarn/usercache/pgl
>> /appcache/application_1530116533665_0008/container_153011653
>> 3665_0008_02_000001/__spark_conf__/__spark_conf__.properties
>> java.home /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
>> java.version 1.8.0-internal
>> sun.io.unicode.encoding UnicodeLittle Classpath Entries
>> ResourceSource
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax-api-1.0-2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-beanutils-core-1.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guava-11.0.2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/datanucleus-api-jdo-3.2.6.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/guava-11.0.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-util-1.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-hive_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javax.inject-1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-recipes-2.7.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/libthrift-0.9.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/okio-1.13.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-cli-1.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-shuffle-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-3.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/leveldbjni-all-1.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-network-common_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> nodemanager-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/okhttp-3.8.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/junit-4.11.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/json4s-ast_2.11-3.2.11.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-configuration-1.6.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/avro-1.7.7.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/generex-1.0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/arpack_combined_all-0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-common-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-jaxrs-1.9.13.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-cli-1.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/kubernetes-client-3.0.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/eigenbase-properties-1.1.5.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/pyrolite-4.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-2.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang-2.6.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-jobclient-2.9.0-tests.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-hs-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-kubernetes_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/log4j-1.2.17.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spire_2.11-0.13.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-encoding-1.8.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-common-2.22.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-6.1.26.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-test-2.7.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-core-
>> asl-1.9.13.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-net-2.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/avro-1.7.7.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/logging-interceptor-3.8.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-compress-1.4.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-auth-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-net-3.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/metrics-jvm-3.1.5.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-registry-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/machinist_2.11-0.6.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/xz-1.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpclient-4.5.2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jetty-util-6.1.26.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/stax-api-1.0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jsr305-1.3.9.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/netty-all-4.1.17.Final.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/arrow-memory-0.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/antlr-runtime-3.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javax.inject-1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/orc-core-1.4.4-nohive.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-io-2.5.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/metrics-core-3.1.5.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-graphx_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-beanu
>> tils-1.7.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/chill_2.11-0.8.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-launcher_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/paranamer-2.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-compress-1.4.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-client-2.7.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpclient-4.5.2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-hadoop-bundle-1.6.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/avro-1.7.7.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> sharedcachemanager-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/zjsonpatch-0.3.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-framework-2.7.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-guava-2.22.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/gson-2.2.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-catalyst_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/geronimo-jcache
>> _1.0_spec-1.0-alpha-1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/snappy-0.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/log4j-1.2.17.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/compress-lzf-1.0.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hive-beeline-1.2.1.spark2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-digester-1.8.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/metrics-core-3.0.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/super-csv-2.2.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-dataformat-yaml-2.6.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-core-2.6.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/JavaEWAH-0.3.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javax.ws.rs-api-2.0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-mapreduce-client-core-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/apache-log4j-extras-1.2.17.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/asm-3.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/ehcache-3.3.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-json-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/asm-3.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsp-api-2.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javolution-5.5.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-databind-2.6.7.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xmlenc-0.52.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-mesos_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-mapper-
>> asl-1.9.13.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/janino-3.0.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/htrace-core-3.1.0-incubating.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-common-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/breeze_2.11-0.13.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang-2.6.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-format-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/snappy-java-1.0.5.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/mesos-1.4.0-shaded-protobuf.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/woodstox-core-5.0.3.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-nfs-2.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/servlet-api-2.5.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-dbcp-1.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-core-asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-logging-1.1.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/asm-3.2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jcl-over-slf4j-1.7.16.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math-2.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/javax.inject-1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jets3t-0.9.4.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-app-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/libfb303-0.9.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/core-1.1.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-mllib_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-mapreduce-client-shuffle-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jpam-1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-util-6.1.26.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-client-2.7.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-core_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/api-asn1-api-1.0.0-M20.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-i18n
>> -2.0.0-M15.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jtransforms-2.4.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-network-shuffle_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-math3-3.4.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/snappy-java-1.0.5.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-column-1.8.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/mockito-all-1.8.5.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jaxb-api-2.2.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/breeze-macros_2.11-0.13.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-kvstore_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-mapreduce-client-jobclient-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/base64-2.3.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scala-xml_2.11-1.0.5.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/oro-2.0.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jetty-6.1.26.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/aopalliance-repackaged-2.4.0-b34.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-logging-1.1.3.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-hdfs-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jets3t-0.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/guava-11.0.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-kerb
>> eros-codec-2.0.0-M15.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-mapreduce-client-common-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/metrics-graphite-3.1.5.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scala-compiler-2.11.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-server-2.22.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/aopalliance-1.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/joda-time-2.9.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> resourcemanager-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/netty-3.9.9.Final.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/activation-1.1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax2-api-3.1.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/gson-2.2.4.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsch-0.1.54.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-module-jaxb-annotations-2.6.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scalap-2.11.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/py4j-0.10.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/xmlenc-0.52.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> tests-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/curator-client-2.7.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javassist-3.18.1-GA.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-core-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-beanu
>> tils-core-1.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/kryo-shaded-3.0.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/httpcore-4.4.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/calcite-core-1.2.0-incubating.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-compiler-3.0.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-mapper-
>> asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-yarn-server-common-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/zookeeper-3.4.6.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-confi
>> guration-1.6.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/stringtemplate-3.2.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/curator-framework-2.7.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/antlr-2.7.7.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsr305-3.0.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> timeline-pluginstorage-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-math3-3.1.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-jackson-1.8.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-mllib-local_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scala-parser-combinators_2.11-1.0.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javax.inject-2.4.0-b34.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-client-2.22.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsch-0.1.54.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/httpclient-4.5.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-httpclient-3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jsr305-3.0.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/lz4-java-1.4.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jta-1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/protobuf-java-2.5.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/snakeyaml-1.15.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-auth-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/datanucleus-core-3.2.10.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/avro-mapred-1.7.7-hadoop2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/aopalliance-1.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javax.servlet-api-3.1.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/etc/hadoop System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-media-jaxb-2.22.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-unsafe_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-xc-1.9.13.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/antlr4-runtime-4.7.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-native-
>> client-2.9.0-tests.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-yarn-server-web-proxy-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-sketch_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/objenesis-2.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/guava-14.0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/apacheds-kerberos-codec-2.0.0-M15.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-co
>> mpress-1.4.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/slf4j-api-1.7.16.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/avro-1.7.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/ivy-2.4.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-yarn-client-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> conf__/__hadoop_conf__ System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-api-1.7.25.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> web-proxy-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-pool-1.5.4.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-frame
>> work-2.7.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xz-1.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hk2-api-2.4.0-b34.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jettison-1.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/calcite-linq4j-1.2.0-incubating.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hppc-0.7.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-annota
>> tions-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-collect
>> ions-3.2.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-module-scala_2.11-2.6.7.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/guice-3.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/kubernetes-model-2.0.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hive-jdbc-1.2.1.spark2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-io-2.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-repl_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-hadoop-1.8.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/osgi-resource-locator-1.0.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-annotations-2.6.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/opencsv-2.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/log4j-1.2.17.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/apacheds-i18n-2.0.0-M15.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-colle
>> ctions-3.2.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-common-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-3.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/activation-1.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-io-2.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-codec-1.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/aircompressor-0.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-annotat
>> ions-2.7.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/nimbus-jose-jwt-3.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> router-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/zstd-jni-1.3.2-2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-yarn_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-kerber
>> os-codec-2.0.0-M15.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> applicationhistoryservice-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/json4s-jackson_2.11-3.2.11.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/validation-api-1.1.0.Final.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-common-1.8.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-lang3-3.5.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okio-1.4.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/metrics-json-3.1.5.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jline-2.12.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/paranamer-2.3.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-codec-1.10.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/fst-2.50.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-configu
>> ration-1.6.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-smart-1.1.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-client-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-core-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xmlenc-0.52.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/derby-10.12.1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-hs-plugins-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_conf__ System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okhttp-2.4.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0-tests.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jcip-annotations-1.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/minlog-1.3.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-io-2.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/flatbuffers-1.2.0-3f79e055.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax2-api-3.1.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/json4s-core_2.11-3.2.11.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hive-cli-1.2.1.spark2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpcore-4.4.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/netty-3.6.2.Final.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hk2-utils-2.4.0-b34.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-yarn-api-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jettison-1.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/curator-recipes-2.7.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-container-servlet-core-2.22.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/api-util-1.0.0-M20.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hamcrest-core-1.3.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math3-3.1.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hk2-locator-2.4.0-b34.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-crypto-1.0.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-applica
>> tions-distributedshell-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/ST4-4.0.4.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-core-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpcore-4.4.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/guice-servlet-3.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-lang-2.6.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0-tests.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-sql_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/json-smart-1.1.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/snappy-java-1.0.5.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-native-
>> client-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/xercesImpl-2.9.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scala-library-2.11.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-6.1.26.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> common-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-digester-1.8.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-beanutils-1.7.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/htrace-core4-
>> 4.1.0-incubating.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsp-api-2.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-yarn-common-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/automaton-1.11-8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-net-3.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/htrace-core4-4.
>> 1.0-incubating.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/woodstox-core-5.0.3.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-api-2.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jets3t-0.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/univocity-parsers-2.5.9.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-codec-1.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/stream-2.7.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spire-macros_2.11-0.13.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/datanucleus-rdbms-3.2.9.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang3-3.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hadoop-ann
>> otations-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-co
>> re-asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-streaming_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang3-3.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-ma
>> pper-asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-server-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/zookeeper-3.4.6.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/snappy-java-1.1.2.6.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001 System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-io-2.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-tags_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jodd-core-3.5.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/RoaringBitmap-0.5.11.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-container-servlet-2.22.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-jobclient-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-examples-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/slf4j-log4j12-1.7.16.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/activation-1.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/arrow-vector-0.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/paranamer-2.3.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-annotations-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/asm-3.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/arrow-format-0.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/servlet-api-2.5.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/junit-4.11.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/shapeless_2.11-2.3.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-client-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-module-paranamer-2.7.9.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jsp-api-2.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanuti
>> ls-core-1.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/macro-compat_2.11-1.1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-client-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-hive-thriftserver_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-applica
>> tions-unmanaged-am-launcher-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/stax-api-1.0-2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-cli-1.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xz-1.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-mappe
>> r-asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-json-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/bonecp-0.8.0.RELEASE.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-
>> 2.9.0-tests.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/bcprov-jdk15on-1.58.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-server-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/chill-java-0.8.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-collections-3.2.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-mapper-asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javax.annotation-api-1.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/gson-2.2.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-mapreduce-client-app-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jdo-api-3.0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/avro-ipc-1.7.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/xz-1.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jul-to-slf4j-1.7.16.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsr305-3.0.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/calcite-avatica-1.2.0-incubating.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/htrace-core4-4.
>> 1.0-incubating.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/java-xmlbuilder-1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hive-metastore-1.2.1.spark2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scala-reflect-2.11.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/orc-mapreduce-1.4.4-nohive.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hive-exec-1.2.1.spark2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/xbean-asm5-shaded-4.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-digester-1.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jcip-annotations-1.0.jar
>>
>>
>> On Sun, Jul 1, 2018 at 10:09 AM, Sean Owen <srowen@gmail.com> wrote:
>>
>>> Somewhere, you have mismatched versions of Spark on your classpath.
>>>
>>>
>>> On Sun, Jul 1, 2018, 9:01 AM Peter Liu <peter.pliu@gmail.com> wrote:
>>>
>>>> Hello there,
>>>>
>>>> I didn't get any response/help from the user list for the following
>>>> question and thought people on the dev list might be able to help?:
>>>>
>>>> I upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload
>>>> and got the error (java.lang.AbstractMethodError) never seen before;
>>>> see the error exception stack attached in (a) bellow.
>>>>
>>>> anyone knows if  spark 2.3.1 works well with kafka
>>>> spark-streaming-kafka-0-10?
>>>>
>>>> this link of spark kafka integration page doesn't say anything about
>>>> any limitation:
>>>> https://spark.apache.org/docs/2.3.1/streaming-kafka-integration.html
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_2.3.1_streaming-2Dkafka-2Dintegration.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=LORvnW8MyMYoQuiRJpv4qo1519IpQ57V-MxUgro0vP0&e=>
>>>>
>>>> but this discussion seems to say there is indeed an issue when
>>>> upgrading to spark 2.3.1:
>>>> https://stackoverflow.com/questions/49180931/abstractmethode
>>>> rror-creating-kafka-stream
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_49180931_abstractmethoderror-2Dcreating-2Dkafka-2Dstream&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=xIl7y-rgXq2AUuNg8cKeBIRApmRMLnrJbPsf4kc5Zzg&e=>
>>>>
>>>> i also rebuilt the workload with some spark 2.3.1 jars (see (b) below).
>>>> it doesn't seem to help.
>>>>
>>>> Would be great if anyone could kindly share any insights here.
>>>>
>>>> Thanks!
>>>>
>>>> Peter
>>>>
>>>> (a) the exception
>>>> Exception in thread "stream execution thread for [id =
>>>> 5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId =
>>>> 70e78d5c-665e-4c6f-a0cc-41a56e488e30]" java.lang.AbstractMethodError
>>>>         at org.apache.spark.internal.Logging$class.initializeLogIfNeces
>>>> sary(Logging.scala:99)
>>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializ
>>>> eLogIfNecessary(KafkaSourceProvider.scala:369)
>>>>         at org.apache.spark.internal.Logging$class.log(Logging.scala:46
>>>> )
>>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(Kafka
>>>> SourceProvider.scala:369)
>>>>         at org.apache.spark.internal.Logging$class.logDebug(Logging.sca
>>>> la:58)
>>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(
>>>> KafkaSourceProvider.scala:369)
>>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpda
>>>> ter.set(KafkaSourceProvider.scala:439)
>>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaPara
>>>> msForDriver(KafkaSourceProvider.scala:394)
>>>>         at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSour
>>>> ce(KafkaSourceProvider.scala:90)
>>>>         at org.apache.spark.sql.execution.datasources.DataSource.create
>>>> Source(DataSource.scala:277)
>>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> $$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:80)
>>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> $$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:77)
>>>>         at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLi
>>>> ke.scala:194)
>>>>         at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.sca
>>>> la:80)
>>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> $$anonfun$1.applyOrElse(MicroBatchExecution.scala:77)
>>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> $$anonfun$1.applyOrElse(MicroBatchExecution.scala:75)
>>>>         at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.appl
>>>> y(TreeNode.scala:267)
>>>>         at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.appl
>>>> y(TreeNode.scala:267)
>>>>         at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigi
>>>> n(TreeNode.scala:70)
>>>>         at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(T
>>>> reeNode.scala:266)
>>>>         at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeN
>>>> ode.scala:256)
>>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> .logicalPlan$lzycompute(MicroBatchExecution.scala:75)
>>>>         at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> .logicalPlan(MicroBatchExecution.scala:61)
>>>>         at org.apache.spark.sql.execution.streaming.StreamExecution.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.sql.execution.streaming.StreamExecution.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=lU8N2IsjW321Bu4T4BHLHvj1OSsKdPI_R_ahmOgq9rA&e=>
>>>> $apache$spark$sql$execution$streaming$StreamExecution$$runSt
>>>> ream(StreamExecution.scala:265)
>>>>         at org.apache.spark.sql.execution.streaming.StreamExecution$$an
>>>> on$1.run(StreamExecution.scala:189)
>>>>
>>>> (b)* the build script update:*
>>>>
>>>> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff
>>>> build.sbt spk211-build.sbt.original
>>>> 10,11c10,11
>>>> < libraryDependencies += "org.apache.spark" % "spark-sql_2.11" %*
>>>> "2.3.1"*
>>>> < libraryDependencies += "org.apache.spark" % "spark-core_2.11" %*
>>>> "2.3.1"*
>>>> ---
>>>> > libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1"
>>>> > libraryDependencies += "org.apache.spark" % "spark-core_2.11" %
>>>> "2.2.1"
>>>> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$
>>>>
>>>>
>>>>
>>>>
>>
>

Mime
View raw message