spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Lindelöf (Jira) <j...@apache.org>
Subject [jira] [Updated] (SPARK-30305) Unable to overwrite table created by another Spark session
Date Thu, 19 Dec 2019 14:50:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-30305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

David Lindelöf updated SPARK-30305:
-----------------------------------
    Description: 
I'm unable to save a dataframe to a table, even when passing the `mode='overwrite'` argument:

 
{code:java}
def test_pyspark_can_overwrite_table():
    spark = SparkSession.builder.getOrCreate()
    sdf = spark.createDataFrame([('Alice', 1)])
    sdf.write.saveAsTable('alice')
    spark.stop()
    spark = SparkSession.builder.getOrCreate()
    sdf = spark.createDataFrame([('Alice', 1)])
    sdf.write.saveAsTable('alice', mode='overwrite')
{code}
I would expect this to succeed. Instead, I get the output in the attached file. The root exception
reads:
{code:java}
pyspark.sql.utils.AnalysisException: "Can not create the managed table('`foo`'). The associated
location('file:/Users/dlindelof/Work/lodging-brain-mod-lever/scripts/Python/godfather/spark-warehouse/foo')
already exists.;"
{code}
 

 

  was:
I'm unable to save a dataframe to a table, even when passing the `mode='overwrite'` argument:

 
{code:java}
def test_pyspark_can_overwrite_table():
    spark = SparkSession.builder.getOrCreate()
    sdf = spark.createDataFrame([('Alice', 1)])
    sdf.write.saveAsTable('alice')
    spark.stop()
    spark = SparkSession.builder.getOrCreate()
    sdf = spark.createDataFrame([('Alice', 1)])
    sdf.write.saveAsTable('alice', mode='overwrite')
{code}
I would expect this to succeed. Instead, I get the output in the attached file.

 

 


> Unable to overwrite table created by another Spark session
> ----------------------------------------------------------
>
>                 Key: SPARK-30305
>                 URL: https://issues.apache.org/jira/browse/SPARK-30305
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.4.4
>            Reporter: David Lindelöf
>            Priority: Major
>         Attachments: pyspark.out
>
>
> I'm unable to save a dataframe to a table, even when passing the `mode='overwrite'` argument:
>  
> {code:java}
> def test_pyspark_can_overwrite_table():
>     spark = SparkSession.builder.getOrCreate()
>     sdf = spark.createDataFrame([('Alice', 1)])
>     sdf.write.saveAsTable('alice')
>     spark.stop()
>     spark = SparkSession.builder.getOrCreate()
>     sdf = spark.createDataFrame([('Alice', 1)])
>     sdf.write.saveAsTable('alice', mode='overwrite')
> {code}
> I would expect this to succeed. Instead, I get the output in the attached file. The root
exception reads:
> {code:java}
> pyspark.sql.utils.AnalysisException: "Can not create the managed table('`foo`'). The
associated location('file:/Users/dlindelof/Work/lodging-brain-mod-lever/scripts/Python/godfather/spark-warehouse/foo')
already exists.;"
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message