spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From asma zgolli <zgollia...@gmail.com>
Subject Re: error , saving dataframe , LEGACY_PASS_PARTITION_BY_AS_OPTIONS
Date Wed, 13 Nov 2019 15:17:48 GMT
the line of code that resulted in that error was :

ch.cern.sparkmeasure.StageMetrics.saveData


it also was started while trying to save a dataframe in hbase using hbase
spark connector :

df1.write.options(

      Map(HBaseTableCatalog.tableCatalog -> catalog, HBaseTableCatalog.
newTable -> "4"))

      .format("org.apache.hadoop.hbase.spark")

      .save()


the full stack trace is as follows:



Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry;

at org.apache.spark.sql.DataFrameWriter.saveToV1Source(
DataFrameWriter.scala:277)

at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)

at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)

at ch.cern.sparkmeasure.StageMetrics.saveData(stagemetrics.scala:297)

...........

...........

at scala.Function0$class.apply$mcV$sp(Function0.scala:34)

at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)

at scala.App$$anonfun$main$1.apply(App.scala:76)

at scala.App$$anonfun$main$1.apply(App.scala:76)

at scala.collection.immutable.List.foreach(List.scala:381)

at scala.collection.generic.TraversableForwarder$class.foreach(
TraversableForwarder.scala:35)

at scala.App$class.main(App.scala:76)

Le mer. 13 nov. 2019 à 16:05, Femi Anthony <femibyte@gmail.com> a écrit :

> Can you post the line of code that’s resulting in that error along with
> the stack trace ?
>
> Sent from my iPhone
>
> On Nov 13, 2019, at 9:53 AM, asma zgolli <zgolliasma@gmail.com> wrote:
>
> 
>
> Hello ,
>
>
> I'm using spark 2.4.4 and i keep receiving this error message. Can you
> please help me identify the problem?
>
>
> thank you ,
>
> yours sincerely
> Asma ZGOLLI
>
> PhD student in data engineering - computer science
>
> PJ:
>
>
>
> "main" java.lang.NoSuchMethodError:
> org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry;
>
> at org.apache.spark.sql.DataFrameWriter.saveToV1Source(
> DataFrameWriter.scala:277)
>
>
>
>
>
>

-- 
Asma ZGOLLI

PhD student in data engineering - computer science

Mime
View raw message