spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marco Gaido (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24170) [Spark SQL] json file format is not dropped after dropping table
Date Thu, 03 May 2018 14:48:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462545#comment-16462545
] 

Marco Gaido commented on SPARK-24170:
-------------------------------------

This is true for every datasource. This is the expected behavior when you set the location,
because by default if the location is set, Spark assumes that the table is external (and not
managed). I am . not sure whether this is the right thing to do, but it is how it works.

cc [~smilegator] [~dongjoon] any further comments on this? Shall we discuss if this is the
right behavior?

> [Spark SQL] json file format is not dropped after dropping table
> ----------------------------------------------------------------
>
>                 Key: SPARK-24170
>                 URL: https://issues.apache.org/jira/browse/SPARK-24170
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0
>         Environment: OS: SUSE 11
> Spark Version: 2.3
>            Reporter: ABHISHEK KUMAR GUPTA
>            Priority: Major
>
> Steps:
>  # Launch spark-sql --master yarn
>  #  create table json(name STRING, age int, gender string, id INT) using org.apache.spark.sql.json
options(path "hdfs:///user/testdemo/");
>  # Execute the below SQL queries 
> INSERT into json
> SELECT 'Shaan',21,'Male',1
> UNION ALL
> SELECT 'Xing',20,'Female',11
> UNION ALL
> SELECT 'Mile',4,'Female',20
> UNION ALL
> SELECT 'Malan',10,'Male',9;
> Below 4 json file format created 
> BLR1000023111:/opt/Antsecure/install/hadoop/namenode/bin # ./hdfs dfs -ls /user/testdemo
> Found 14 items
> -rw-r--r-- 3 spark hadoop 0 2018-04-26 17:44 /user/testdemo/_SUCCESS
> -rw-r--r-- 3 spark hadoop 4802 2018-04-24 18:20 /user/testdemo/customer1.csv
> -rw-r--r-- 3 spark hadoop 92 2018-04-26 17:02 /user/testdemo/json1.txt
> -rw-r--r-- 3 spark hadoop 49 2018-04-26 17:32 /user/testdemo/part-00000-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
> -rw-r--r-- 3 spark hadoop 49 2018-04-26 17:44 /user/testdemo/part-00000-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
> -rw-r--r-- 3 spark hadoop 51 2018-04-26 17:32 /user/testdemo/part-00001-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
> -rw-r--r-- 3 spark hadoop 51 2018-04-26 17:44 /user/testdemo/part-00001-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
> -rw-r--r-- 3 spark hadoop 50 2018-04-26 17:32 /user/testdemo/part-00002-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
> -rw-r--r-- 3 spark hadoop 50 2018-04-26 17:44 /user/testdemo/part-00002-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
> -rw-r--r-- 3 spark hadoop 49 2018-04-26 17:32 /user/testdemo/part-00003-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
> -rw-r--r-- 3 spark hadoop 49 2018-04-26 17:44 /user/testdemo/part-00003-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
>  
> Issue is:
> Now executed below drop command
> spark-sql> drop table json;
>  
> Table dropped successfully but json file still present in the path  /user/testdemo



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message