From issues-return-232770-apmail-spark-issues-archive=spark.apache.org@spark.apache.org Fri Jul 26 12:49:05 2019 Return-Path: X-Original-To: apmail-spark-issues-archive@minotaur.apache.org Delivered-To: apmail-spark-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by minotaur.apache.org (Postfix) with SMTP id 40A721814C for ; Fri, 26 Jul 2019 12:49:05 +0000 (UTC) Received: (qmail 15034 invoked by uid 500); 26 Jul 2019 12:49:04 -0000 Delivered-To: apmail-spark-issues-archive@spark.apache.org Received: (qmail 15021 invoked by uid 500); 26 Jul 2019 12:49:04 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 15012 invoked by uid 99); 26 Jul 2019 12:49:04 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Jul 2019 12:49:04 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 77780E2E86 for ; Fri, 26 Jul 2019 12:49:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 2E83C265F8 for ; Fri, 26 Jul 2019 12:49:00 +0000 (UTC) Date: Fri, 26 Jul 2019 12:49:00 +0000 (UTC) From: "Abhishek Somani (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (SPARK-16996) Hive ACID delta files not seen MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SPARK-16996?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D168= 93807#comment-16893807 ]=20 Abhishek Somani commented on SPARK-16996: ----------------------------------------- We=C2=A0have worked on and open sourced a datasource that will enable users= to work on their Hive ACID Transactional tables=C2=A0using Spark.=C2=A0 =C2=A0 Github:=C2=A0[https://github.com/qubole/spark-acid] =C2=A0 It is available as a Spark package and instructions to use it are on the Gi= thub page. Currently the datasource supports reading from Hive ACID tables = only, and we are working on adding the ability to write into these tables v= ia Spark as well. =C2=A0 Feedback and suggestions are welcome! > Hive ACID delta files not seen > ------------------------------ > > Key: SPARK-16996 > URL: https://issues.apache.org/jira/browse/SPARK-16996 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.2, 1.6.3, 2.1.2, 2.2.0 > Environment: Hive 1.2.1, Spark 1.5.2 > Reporter: Benjamin BONNET > Priority: Major > > spark-sql seems not to see data stored as delta files in an ACID Hive tab= le. > Actually I encountered the same problem as describe here : http://stackov= erflow.com/questions/35955666/spark-sql-is-not-returning-records-for-hive-t= ransactional-tables-on-hdp > For example, create an ACID table with HiveCLI and insert a row : > {code} > set hive.support.concurrency=3Dtrue; > set hive.enforce.bucketing=3Dtrue; > set hive.exec.dynamic.partition.mode=3Dnonstrict; > set hive.txn.manager=3Dorg.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > set hive.compactor.initiator.on=3Dtrue; > set hive.compactor.worker.threads=3D1; > CREATE TABLE deltas(cle string,valeur string) CLUSTERED BY (cle) INTO 1 = BUCKETS > ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' > STORED AS=20 > INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' > OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' > TBLPROPERTIES ('transactional'=3D'true'); > INSERT INTO deltas VALUES("a","a"); > {code} > Then make a query with spark-sql CLI : > {code} > SELECT * FROM deltas; > {code} > That query gets no result and there are no errors in logs. > If you go to HDFS to inspect table files, you find only deltas > {code} > ~>hdfs dfs -ls /apps/hive/warehouse/deltas > Found 1 items > drwxr-x--- - me hdfs 0 2016-08-10 14:03 /apps/hive/warehouse/d= eltas/delta_0020943_0020943 > {code} > Then if you run compaction on that table (in HiveCLI) : > {code} > ALTER TABLE deltas COMPACT 'MAJOR'; > {code} > As a result, the delta will be compute into a base file : > {code} > ~>hdfs dfs -ls /apps/hive/warehouse/deltas > Found 1 items > drwxrwxrwx - me hdfs 0 2016-08-10 15:25 /apps/hive/warehouse/d= eltas/base_0020943 > {code} > Go back to spark-sql and the same query gets a result : > {code} > SELECT * FROM deltas; > a a > Time taken: 0.477 seconds, Fetched 1 row(s) > {code} > But next time you make an insert into Hive table :=20 > {code} > INSERT INTO deltas VALUES("b","b"); > {code} > spark-sql will immediately see changes :=20 > {code} > SELECT * FROM deltas; > a a > b b > Time taken: 0.122 seconds, Fetched 2 row(s) > {code} > Yet there was no other compaction, but spark-sql "sees" the base AND the = delta file : > {code} > ~> hdfs dfs -ls /apps/hive/warehouse/deltas > Found 2 items > drwxrwxrwx - valdata hdfs 0 2016-08-10 15:25 /apps/hive/wareho= use/deltas/base_0020943 > drwxr-x--- - valdata hdfs 0 2016-08-10 15:31 /apps/hive/wareho= use/deltas/delta_0020956_0020956 > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org