kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From big data <bigdatab...@outlook.com>
Subject Re: maven dependency problems
Date Tue, 18 Dec 2018 02:54:24 GMT
When I remove/comment parent dependency like <!--<parent>xxxx</parent> -->
  in Module A pom.xml, it seems ok, and streaming kafka's depends on kafka_2.11:0.8.2.1.

The Modula A's tree:

[INFO] +- org.apache.spark:spark-core_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.7.7:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:provided
[INFO] |  |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- com.twitter:chill_2.11:jar:0.8.0:provided
[INFO] |  |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:provided
[INFO] |  |     +- com.esotericsoftware:minlog:jar:1.3.0:provided
[INFO] |  |     \- org.objenesis:objenesis:jar:2.1:provided
[INFO] |  +- com.twitter:chill-java:jar:0.8.0:provided
[INFO] |  +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:provided
[INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:provided
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:provided
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:provided
[INFO] |  |  |  |  |  \- com.google.inject:guice:jar:3.0:provided
[INFO] |  |  |  |  |     +- javax.inject:javax.inject:jar:1:provided
[INFO] |  |  |  |  |     \- aopalliance:aopalliance:jar:1.0:provided
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:provided
[INFO] |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:provided
[INFO] |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:provided
[INFO] |  |  \- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:provided
[INFO] |  +- org.apache.spark:spark-launcher_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-network-common_2.11:jar:2.1.0:provided
[INFO] |  |  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:provided
[INFO] |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-unsafe_2.11:jar:2.1.0:provided
[INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO] |  +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] |  |  \- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:provided
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.5:provided
[INFO] |  +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.16:compile
[INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.16:provided
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.16:provided
[INFO] |  +- log4j:log4j:jar:1.2.17:compile
[INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.7.16:compile
[INFO] |  +- com.ning:compress-lzf:jar:1.0.3:provided
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO] |  +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:provided
[INFO] |  +- commons-net:commons-net:jar:2.2:compile
[INFO] |  +- org.json4s:json4s-jackson_2.11:jar:3.2.11:provided
[INFO] |  |  \- org.json4s:json4s-core_2.11:jar:3.2.11:provided
[INFO] |  |     +- org.json4s:json4s-ast_2.11:jar:3.2.11:provided
[INFO] |  |     \- org.scala-lang:scalap:jar:2.11.0:provided
[INFO] |  |        \- org.scala-lang:scala-compiler:jar:2.11.0:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:provided
[INFO] |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
[INFO] |  |  +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:provided
[INFO] |  |  |  +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:provided
[INFO] |  |  |  \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:provided
[INFO] |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:provided
[INFO] |  |  \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:provided
[INFO] |  |     \- org.javassist:javassist:jar:3.18.1-GA:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:provided
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.2:provided
[INFO] |  |  +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:provided
[INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:provided
[INFO] |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:provided
[INFO] |  |  \- javax.validation:validation-api:jar:1.1.0.Final:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:provided
[INFO] |  +- io.netty:netty-all:jar:4.0.42.Final:compile
[INFO] |  +- io.netty:netty:jar:3.8.0.Final:provided
[INFO] |  +- com.clearspring.analytics:stream:jar:2.7.0:provided
[INFO] |  +- io.dropwizard.metrics:metrics-core:jar:3.1.2:provided
[INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:provided
[INFO] |  +- io.dropwizard.metrics:metrics-json:jar:3.1.2:provided
[INFO] |  +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:provided
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5:provided
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.6.5:provided
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.5:provided
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.6.5:provided
[INFO] |  +- org.apache.ivy:ivy:jar:2.4.0:provided
[INFO] |  +- oro:oro:jar:2.0.8:provided
[INFO] |  +- net.razorvine:pyrolite:jar:4.13:provided
[INFO] |  +- net.sf.py4j:py4j:jar:0.10.4:provided
[INFO] |  +- org.apache.spark:spark-tags_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.commons:commons-crypto:jar:1.0.0:provided
[INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.apache.spark:spark-streaming_2.11:jar:2.1.0:provided
[INFO] +- org.apache.spark:spark-sql_2.11:jar:2.1.0:provided
[INFO] |  +- com.univocity:univocity-parsers:jar:2.2.1:provided
[INFO] |  +- org.apache.spark:spark-sketch_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-catalyst_2.11:jar:2.1.0:provided
[INFO] |  |  +- org.codehaus.janino:janino:jar:3.0.0:provided
[INFO] |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.0:provided
[INFO] |  |  \- org.antlr:antlr4-runtime:jar:4.5.3:provided
[INFO] |  +- org.apache.parquet:parquet-column:jar:1.8.1:provided
[INFO] |  |  +- org.apache.parquet:parquet-common:jar:1.8.1:provided
[INFO] |  |  \- org.apache.parquet:parquet-encoding:jar:1.8.1:provided
[INFO] |  \- org.apache.parquet:parquet-hadoop:jar:1.8.1:provided
[INFO] |     +- org.apache.parquet:parquet-format:jar:2.3.0-incubating:provided
[INFO] |     \- org.apache.parquet:parquet-jackson:jar:1.8.1:provided
[INFO] +- org.apache.spark:spark-hive_2.11:jar:2.1.0:provided
[INFO] |  +- com.twitter:parquet-hadoop-bundle:jar:1.6.0:provided
[INFO] |  +- org.spark-project.hive:hive-exec:jar:1.2.1.spark2:provided
[INFO] |  |  +- javolution:javolution:jar:5.5.1:provided
[INFO] |  |  +- log4j:apache-log4j-extras:jar:1.2.17:provided
[INFO] |  |  +- org.antlr:antlr-runtime:jar:3.4:provided
[INFO] |  |  |  +- org.antlr:stringtemplate:jar:3.2.1:provided
[INFO] |  |  |  \- antlr:antlr:jar:2.7.7:provided
[INFO] |  |  +- org.antlr:ST4:jar:4.0.4:provided
[INFO] |  |  +- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] |  |  |  \- org.tukaani:xz:jar:1.0:compile
[INFO] |  |  +- com.googlecode.javaewah:JavaEWAH:jar:0.3.2:provided
[INFO] |  |  +- org.iq80.snappy:snappy:jar:0.2:provided
[INFO] |  |  +- stax:stax-api:jar:1.0.1:provided
[INFO] |  |  \- net.sf.opencsv:opencsv:jar:2.3:provided
[INFO] |  +- org.spark-project.hive:hive-metastore:jar:1.2.1.spark2:provided
[INFO] |  |  +- com.jolbox:bonecp:jar:0.8.0.RELEASE:provided
[INFO] |  |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |  |  +- org.apache.derby:derby:jar:10.10.2.0:provided
[INFO] |  |  +- org.datanucleus:datanucleus-api-jdo:jar:3.2.6:provided
[INFO] |  |  +- org.datanucleus:datanucleus-rdbms:jar:3.2.9:provided
[INFO] |  |  +- commons-pool:commons-pool:jar:1.5.4:provided
[INFO] |  |  +- commons-dbcp:commons-dbcp:jar:1.4:provided
[INFO] |  |  \- javax.jdo:jdo-api:jar:3.0.1:provided
[INFO] |  |     \- javax.transaction:jta:jar:1.1:provided
[INFO] |  +- org.apache.avro:avro:jar:1.7.7:compile
[INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] |  +- org.apache.calcite:calcite-avatica:jar:1.2.0-incubating:provided
[INFO] |  +- org.apache.calcite:calcite-core:jar:1.2.0-incubating:provided
[INFO] |  |  +- org.apache.calcite:calcite-linq4j:jar:1.2.0-incubating:provided
[INFO] |  |  \- net.hydromatic:eigenbase-properties:jar:1.1.5:provided
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.10:compile
[INFO] |  +- joda-time:joda-time:jar:2.9.3:provided
[INFO] |  +- org.jodd:jodd-core:jar:3.5.2:provided
[INFO] |  +- org.datanucleus:datanucleus-core:jar:3.2.10:provided
[INFO] |  +- org.apache.thrift:libthrift:jar:0.9.2:provided
[INFO] |  \- org.apache.thrift:libfb303:jar:0.9.2:provided
[INFO] +- org.apache.spark:spark-streaming-kafka_2.11:jar:1.6.3:compile
[INFO] |  \- org.apache.kafka:kafka_2.11:jar:0.8.2.1:compile
[INFO] |     +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.2:compile
[INFO] |     +- com.101tec:zkclient:jar:0.3:compile
[INFO] |     \- org.apache.kafka:kafka-clients:jar:0.8.2.1:compile
[INFO] +- org.scala-lang:scala-library:jar:2.11.8:provided
[INFO] +- org.scala-lang:scala-actors:jar:2.11.8:provided
[INFO] +- org.scalatest:scalatest_2.11:jar:2.2.1:test
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.2:provided
[INFO] |  \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:compile
[INFO] +- org.apache.hbase:hbase-client:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hbase:hbase-annotations:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hbase:hbase-common:jar:1.2.0-cdh5.15.0:compile
[INFO] |  |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile
[INFO] |  |  \- org.apache.hadoop:hadoop-core:jar:2.6.0-mr1-cdh5.15.0:compile
[INFO] |  |     +- tomcat:jasper-runtime:jar:5.5.23:compile
[INFO] |  |     +- tomcat:jasper-compiler:jar:5.5.23:compile
[INFO] |  |     +- javax.servlet.jsp:jsp-api:jar:2.1:compile
[INFO] |  |     +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  |     +- hsqldb:hsqldb:jar:1.8.0.10:compile
[INFO] |  |     \- org.eclipse.jdt:core:jar:3.1.1:compile
[INFO] |  +- org.apache.hbase:hbase-protocol:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  +- com.google.guava:guava:jar:12.0.1:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh5.15.0:compile
[INFO] |  +- org.apache.htrace:htrace-core:jar:3.2.0-incubating:compile
[INFO] |  +- org.jruby.jcodings:jcodings:jar:1.0.8:compile
[INFO] |  +- org.jruby.joni:joni:jar:2.1.2:compile
[INFO] |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.15.0:compile
[INFO] |  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |  |  +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.4:compile
[INFO] |  |  +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] |  |  +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |  |  |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] |  |  |  |     +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] |  |  |  |     \- javax.activation:activation:jar:1.1:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] |  |  +- commons-el:commons-el:jar:1.0:compile
[INFO] |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |  |  +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] |  |  +- com.jcraft:jsch:jar:0.1.42:compile
[INFO] |  |  +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] |  |  \- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.6.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-auth:jar:2.6.0-cdh5.15.0:compile
[INFO] |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] |  +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] |  \- junit:junit:jar:4.12:compile
[INFO] |     \- org.hamcrest:hamcrest-core:jar:1.3:compile
[INFO] \- com.oracle.jdbc:ojdbc6:jar:11.1.0.6.0:compile


But If I open <parent>xxx</parent>, and Module A's tree

[INFO] +- org.apache.spark:spark-core_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.7.7:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:provided
[INFO] |  |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- com.twitter:chill_2.11:jar:0.8.0:provided
[INFO] |  |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:provided
[INFO] |  |     \- com.esotericsoftware:minlog:jar:1.3.0:provided
[INFO] |  +- com.twitter:chill-java:jar:0.8.0:provided
[INFO] |  +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:provided
[INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:provided
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:provided
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:provided
[INFO] |  |  |  |  |  \- com.google.inject:guice:jar:3.0:provided
[INFO] |  |  |  |  |     +- javax.inject:javax.inject:jar:1:provided
[INFO] |  |  |  |  |     \- aopalliance:aopalliance:jar:1.0:provided
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:provided
[INFO] |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:provided
[INFO] |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:provided
[INFO] |  |  \- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:provided
[INFO] |  +- org.apache.spark:spark-launcher_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-network-common_2.11:jar:2.1.0:provided
[INFO] |  |  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-unsafe_2.11:jar:2.1.0:provided
[INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO] |  +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] |  |  \- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:provided
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.7:provided
[INFO] |  +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.25:provided
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.25:provided
[INFO] |  +- log4j:log4j:jar:1.2.17:compile
[INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.7.25:compile
[INFO] |  +- com.ning:compress-lzf:jar:1.0.3:provided
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:provided
[INFO] |  +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:provided
[INFO] |  +- commons-net:commons-net:jar:2.2:compile
[INFO] |  +- org.json4s:json4s-jackson_2.11:jar:3.2.11:provided
[INFO] |  |  \- org.json4s:json4s-core_2.11:jar:3.2.11:provided
[INFO] |  |     +- org.json4s:json4s-ast_2.11:jar:3.2.11:provided
[INFO] |  |     \- org.scala-lang:scalap:jar:2.11.0:provided
[INFO] |  |        \- org.scala-lang:scala-compiler:jar:2.11.0:provided
[INFO] |  |           \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.1:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-client:jar:2.26:provided
[INFO] |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.1:provided
[INFO] |  |  \- org.glassfish.hk2.external:javax.inject:jar:2.5.0-b42:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.26:provided
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:provided
[INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-server:jar:2.26:provided
[INFO] |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.26:provided
[INFO] |  |  \- javax.validation:validation-api:jar:2.0.1.Final:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.26:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.26:provided
[INFO] |  +- io.netty:netty-all:jar:4.1.27.Final:compile
[INFO] |  +- io.netty:netty:jar:3.8.0.Final:provided
[INFO] |  +- com.clearspring.analytics:stream:jar:2.7.0:provided
[INFO] |  +- io.dropwizard.metrics:metrics-core:jar:3.2.6:provided
[INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:3.2.6:provided
[INFO] |  +- io.dropwizard.metrics:metrics-json:jar:3.2.6:provided
[INFO] |  +- io.dropwizard.metrics:metrics-graphite:jar:3.2.6:provided
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.6:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.6:compile
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.9.6:provided
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.9.6:provided
[INFO] |  +- org.apache.ivy:ivy:jar:2.4.0:provided
[INFO] |  +- oro:oro:jar:2.0.8:provided
[INFO] |  +- net.razorvine:pyrolite:jar:4.13:provided
[INFO] |  +- net.sf.py4j:py4j:jar:0.10.4:provided
[INFO] |  +- org.apache.spark:spark-tags_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.commons:commons-crypto:jar:1.0.0:provided
[INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.apache.spark:spark-streaming_2.11:jar:2.1.0:provided
[INFO] +- org.apache.spark:spark-sql_2.11:jar:2.1.0:provided
[INFO] |  +- com.univocity:univocity-parsers:jar:2.2.1:provided
[INFO] |  +- org.apache.spark:spark-sketch_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-catalyst_2.11:jar:2.1.0:provided
[INFO] |  |  +- org.codehaus.janino:janino:jar:3.0.8:provided
[INFO] |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.0:provided
[INFO] |  |  \- org.antlr:antlr4-runtime:jar:4.5.3:provided
[INFO] |  +- org.apache.parquet:parquet-column:jar:1.8.1:provided
[INFO] |  |  +- org.apache.parquet:parquet-common:jar:1.8.1:provided
[INFO] |  |  \- org.apache.parquet:parquet-encoding:jar:1.8.1:provided
[INFO] |  \- org.apache.parquet:parquet-hadoop:jar:1.8.1:provided
[INFO] |     +- org.apache.parquet:parquet-format:jar:2.3.0-incubating:provided
[INFO] |     \- org.apache.parquet:parquet-jackson:jar:1.8.1:provided
[INFO] +- org.apache.spark:spark-hive_2.11:jar:2.1.0:provided
[INFO] |  +- com.twitter:parquet-hadoop-bundle:jar:1.6.0:provided
[INFO] |  +- org.spark-project.hive:hive-exec:jar:1.2.1.spark2:provided
[INFO] |  |  +- javolution:javolution:jar:5.5.1:provided
[INFO] |  |  +- log4j:apache-log4j-extras:jar:1.2.17:provided
[INFO] |  |  +- org.antlr:antlr-runtime:jar:3.4:provided
[INFO] |  |  |  +- org.antlr:stringtemplate:jar:3.2.1:provided
[INFO] |  |  |  \- antlr:antlr:jar:2.7.7:provided
[INFO] |  |  +- org.antlr:ST4:jar:4.0.4:provided
[INFO] |  |  +- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] |  |  |  \- org.tukaani:xz:jar:1.0:compile
[INFO] |  |  +- com.googlecode.javaewah:JavaEWAH:jar:0.3.2:provided
[INFO] |  |  +- org.iq80.snappy:snappy:jar:0.2:provided
[INFO] |  |  +- stax:stax-api:jar:1.0.1:provided
[INFO] |  |  \- net.sf.opencsv:opencsv:jar:2.3:provided
[INFO] |  +- org.spark-project.hive:hive-metastore:jar:1.2.1.spark2:provided
[INFO] |  |  +- com.jolbox:bonecp:jar:0.8.0.RELEASE:provided
[INFO] |  |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |  |  +- org.apache.derby:derby:jar:10.14.1.0:provided
[INFO] |  |  +- org.datanucleus:datanucleus-api-jdo:jar:3.2.6:provided
[INFO] |  |  +- org.datanucleus:datanucleus-rdbms:jar:3.2.9:provided
[INFO] |  |  +- commons-pool:commons-pool:jar:1.6:provided
[INFO] |  |  +- commons-dbcp:commons-dbcp:jar:1.4:provided
[INFO] |  |  \- javax.jdo:jdo-api:jar:3.0.1:provided
[INFO] |  |     \- javax.transaction:jta:jar:1.1:provided
[INFO] |  +- org.apache.avro:avro:jar:1.7.7:compile
[INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] |  +- org.apache.calcite:calcite-avatica:jar:1.2.0-incubating:provided
[INFO] |  +- org.apache.calcite:calcite-core:jar:1.2.0-incubating:provided
[INFO] |  |  +- org.apache.calcite:calcite-linq4j:jar:1.2.0-incubating:provided
[INFO] |  |  \- net.hydromatic:eigenbase-properties:jar:1.1.5:provided
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.6:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.10:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  +- joda-time:joda-time:jar:2.9.9:provided
[INFO] |  +- org.jodd:jodd-core:jar:3.5.2:provided
[INFO] |  +- org.datanucleus:datanucleus-core:jar:3.2.10:provided
[INFO] |  +- org.apache.thrift:libthrift:jar:0.9.2:provided
[INFO] |  \- org.apache.thrift:libfb303:jar:0.9.2:provided
[INFO] +- org.apache.spark:spark-streaming-kafka_2.11:jar:1.6.3:compile
[INFO] |  \- org.apache.kafka:kafka_2.11:jar:1.0.2:compile
[INFO] |     +- org.apache.kafka:kafka-clients:jar:1.0.2:compile
[INFO] |     |  \- org.lz4:lz4-java:jar:1.4:compile
[INFO] |     \- com.101tec:zkclient:jar:0.10:compile
[INFO] +- org.scala-lang:scala-library:jar:2.11.8:provided
[INFO] +- org.scala-lang:scala-actors:jar:2.11.8:provided
[INFO] +- org.scalatest:scalatest_2.11:jar:2.2.1:test
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.2:provided
[INFO] |  \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:provided
[INFO] +- org.apache.hbase:hbase-client:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hbase:hbase-annotations:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hbase:hbase-common:jar:1.2.0-cdh5.15.0:compile
[INFO] |  |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile
[INFO] |  |  \- org.apache.hadoop:hadoop-core:jar:2.6.0-mr1-cdh5.15.0:compile
[INFO] |  |     +- tomcat:jasper-runtime:jar:5.5.23:compile
[INFO] |  |     +- tomcat:jasper-compiler:jar:5.5.23:compile
[INFO] |  |     +- javax.servlet.jsp:jsp-api:jar:2.1:compile
[INFO] |  |     +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  |     +- hsqldb:hsqldb:jar:1.8.0.10:compile
[INFO] |  |     \- org.eclipse.jdt:core:jar:3.1.1:compile
[INFO] |  +- org.apache.hbase:hbase-protocol:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  +- com.google.guava:guava:jar:12.0.1:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh5.15.0:compile
[INFO] |  +- org.apache.htrace:htrace-core:jar:3.2.0-incubating:compile
[INFO] |  +- org.jruby.jcodings:jcodings:jar:1.0.8:compile
[INFO] |  +- org.jruby.joni:joni:jar:2.1.2:compile
[INFO] |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.15.0:compile
[INFO] |  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |  |  +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.4:compile
[INFO] |  |  +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] |  |  +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |  |  |  |  \- javax.xml.bind:jaxb-api:jar:2.3.0:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] |  |  +- commons-el:commons-el:jar:1.0:compile
[INFO] |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |  |  +- com.google.code.gson:gson:jar:2.8.5:compile
[INFO] |  |  +- com.jcraft:jsch:jar:0.1.42:compile
[INFO] |  |  +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] |  |  \- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.6.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-auth:jar:2.6.0-cdh5.15.0:compile
[INFO] |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] |  \- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] +- com.oracle.jdbc:ojdbc6:jar:11.1.0.6.0:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] +- junit:junit:jar:4.12:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.powermock:powermock-module-junit4:jar:1.7.1:test
[INFO] |  \- org.powermock:powermock-module-junit4-common:jar:1.7.1:test
[INFO] |     +- org.powermock:powermock-reflect:jar:1.7.1:test
[INFO] |     \- org.powermock:powermock-core:jar:1.7.1:test
[INFO] |        \- org.javassist:javassist:jar:3.21.0-GA:test
[INFO] \- org.powermock:powermock-api-mockito2:jar:1.7.1:test
[INFO]    +- org.powermock:powermock-api-mockito-common:jar:1.7.1:test
[INFO]    |  \- org.powermock:powermock-api-support:jar:1.7.1:test
[INFO]    \- org.mockito:mockito-core:jar:2.15.0:test
[INFO]       +- net.bytebuddy:byte-buddy:jar:1.7.11:test
[INFO]       +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
[INFO]       \- org.objenesis:objenesis:jar:2.6:provided


The parent pom's dependency tree:(parent pom only has few dependency, same as before)

[INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  +- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- junit:junit:jar:4.12:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.powermock:powermock-module-junit4:jar:1.7.1:test
[INFO] |  \- org.powermock:powermock-module-junit4-common:jar:1.7.1:test
[INFO] |     +- org.powermock:powermock-reflect:jar:1.7.1:test
[INFO] |     \- org.powermock:powermock-core:jar:1.7.1:test
[INFO] |        \- org.javassist:javassist:jar:3.21.0-GA:test
[INFO] \- org.powermock:powermock-api-mockito2:jar:1.7.1:test
[INFO]    +- org.powermock:powermock-api-mockito-common:jar:1.7.1:test
[INFO]    |  \- org.powermock:powermock-api-support:jar:1.7.1:test
[INFO]    \- org.mockito:mockito-core:jar:2.15.0:test
[INFO]       +- net.bytebuddy:byte-buddy:jar:1.7.11:test
[INFO]       +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
[INFO]       \- org.objenesis:objenesis:jar:2.6:test


Other module trees do not have kafka's dependency, so I don't post it.

thanks.

在 2018/12/18 上午10:42, Liam Clarke 写道:

Can you please post the full output of maven dependency:tree from the
parent POM in both scenarios?


Thanks,

Liam Clarke

On Tue, 18 Dec. 2018, 3:26 pm big data <bigdatabase@outlook.com<mailto:bigdatabase@outlook.com>
wrote:



Hi,

No other dependency include kafka's jar.

The project structure is:

pom.xml

|---Module A

   |--pom.xml

|---Module B

|---Module C

When spark-streaming-kafka_2.11:1.6.3 in in Module A's pom.xml, from
dependency tree, we can see in depends on kafka_2.11:0.8.2.1
When we move spark-streaming-kafka_2.11:1.6.3 dependency to the parent
pom.xml, from tree, it depends on kafka_2.11:1.0.2

In whoel project, only one dependency about streaming-kafka, and no
other dependency about kafka_2.11.


在 2018/12/18 上午10:16, Liam Clarke 写道:


Hi,

Is it a transitive dependency of any of your other dependencies?

Cheers,

Liam Clarke

On Tue, 18 Dec. 2018, 2:57 pm big data <bigdatabase@outlook.com<mailto:bigdatabase@outlook.com>
wrote:



Hi,
   our project includes this dependency by:

<dependency>    <groupId>org.apache.spark</groupId>


<artifactId>spark-streaming-kafka_2.11</artifactId>
<version>1.6.3</version></dependency>



 From dependency tree, we can see it dependency kafka_2.11:0.8.2.1


verson.




But when we move this dependency to project's parent pom file, the
dependency "kafka_2.11" version is changed automatically. Why?


You can see Kafka_2.11 version is changed to 1.0.2?  It's so strange, in
out project we DO NOT dependency this jar in other places.










Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message