From user-return-69268-apmail-spark-user-archive=spark.apache.org@spark.apache.org Sat Apr 22 09:14:34 2017 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 344CE1923E for ; Sat, 22 Apr 2017 09:14:34 +0000 (UTC) Received: (qmail 42993 invoked by uid 500); 22 Apr 2017 09:14:30 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 42823 invoked by uid 500); 22 Apr 2017 09:14:30 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 42813 invoked by uid 99); 22 Apr 2017 09:14:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Apr 2017 09:14:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 911A3C092E for ; Sat, 22 Apr 2017 09:14:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.397 X-Spam-Level: X-Spam-Status: No, score=-0.397 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.796, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id cIlgeun8k26C for ; Sat, 22 Apr 2017 09:14:28 +0000 (UTC) Received: from mail-ua0-f176.google.com (mail-ua0-f176.google.com [209.85.217.176]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id AD7775FDF5 for ; Sat, 22 Apr 2017 09:14:27 +0000 (UTC) Received: by mail-ua0-f176.google.com with SMTP id h2so92829731uaa.1 for ; Sat, 22 Apr 2017 02:14:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=pPBu9OV52Z+/BMYfN1Q7SPY77GvdM3zNd8aahJkF/qc=; b=IJ6bDT3l5rkpKJys4li0o4hPL8jg4yxbCVsm/bKC4XjlDXDDEGNK+NRpBPZ+Aa2jb2 /ysOgIfXqgo9vTyT3iF9TDO39HGu5wH8UkpOIdJpHWgbktF8i2BCrc1VUCVwUPeQnqr5 uU6B4wnfGciIEYVuidrKP95IHlv3lO7Bj1Jj7PXRFPwbBZk2NhYymIK76Wy6RRlYgD4T uDLuIF0DQ3EwFOpLzJNBn6tT5Rbm/Pb/XFDbdaqB44iHIuvviFbCu5/fqjfUBYcvNFvm 8HIjCnZXdGR6eFdU+dyly0IhykM5teieRDHyOYCIYhp2FvdMAFO5fJHgmxrjJuUpSKGa 27HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=pPBu9OV52Z+/BMYfN1Q7SPY77GvdM3zNd8aahJkF/qc=; b=NtfcnQNzuzo1Z5y3F65VIN/YEnbknttIREzUOwQMO1Dh2km4u5XsEH9vB66lNqyOav 4DORaAPggo+R033erS7SiJm8qYMzRpHNSG+WrmluCl/uEuitFW2C0fJmC3OIWgjttCIe ntOWHrQiTg+i56yi4KGa5XTAj1gKuHVvNIG8X68K+V1RH+NuyVfZq5ktNjoL9N/FUlzN 4JxJcaZ+s4WL28W2CiSCoDJWKMBERYmFj7e8EpYdAiajEbjtebUotiT5YkfLrmPXxr8D 06u8Zr30TbBkYy1D/iPAnQhFeiigWl23vzy86GjGXpBRanYNTn5uC1JN2y0voC2oevDZ VZxA== X-Gm-Message-State: AN3rC/6DNeMnWdYugLrvhTuiYixKGqn13T4wL/TVlhEHipEvdDeIltiG orjENhDJDjOQtIBleRvq3m7Xofb6gA== X-Received: by 10.159.41.99 with SMTP id t90mr720382uat.46.1492852461566; Sat, 22 Apr 2017 02:14:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.31.54.80 with HTTP; Sat, 22 Apr 2017 02:14:21 -0700 (PDT) Received: by 10.31.54.80 with HTTP; Sat, 22 Apr 2017 02:14:21 -0700 (PDT) In-Reply-To: References: <7650952C-6886-4DAB-B59F-D92DEFF13695@qvantel.com> From: vincent gromakowski Date: Sat, 22 Apr 2017 11:14:21 +0200 Message-ID: Subject: Re: Spark SQL - Global Temporary View is not behaving as expected To: Hemanth Gudela Cc: "user@spark.apache.org" , Felix Cheung Content-Type: multipart/alternative; boundary=001a114dfb38ac82d9054dbdcc2d --001a114dfb38ac82d9054dbdcc2d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Look at alluxio for sharing across drivers or spark jobserver Le 22 avr. 2017 10:24 AM, "Hemanth Gudela" a =C3=A9crit : > Thanks for your reply. > > > > Creating a table is an option, but such approach slows down reads & write= s > for a real-time analytics streaming use case that I=E2=80=99m currently w= orking on. > > If at all global temporary view could have been accessible across > sessions/spark contexts, that would have simplified my usecase a lot. > > > > But yeah, thanks for explaining the behavior of global temporary view, no= w > it=E2=80=99s clear J > > > > -Hemanth > > > > *From: *Felix Cheung > *Date: *Saturday, 22 April 2017 at 11.05 > *To: *Hemanth Gudela , "user@spark.apache.org= " > > *Subject: *Re: Spark SQL - Global Temporary View is not behaving as > expected > > > > Cross session is this context is multiple spark sessions from the same > spark context. Since you are running two shells, you are having different > spark context. > > > > Do you have to you a temp view? Could you create a table? > > > > _____________________________ > From: Hemanth Gudela > Sent: Saturday, April 22, 2017 12:57 AM > Subject: Spark SQL - Global Temporary View is not behaving as expected > To: > > > > Hi, > > > > According to documentation > , > global temporary views are cross-session accessible. > > > > But when I try to query a global temporary view from another spark shell > like this=C3=A0 > > *Instance 1 of spark-shell* > > ---------------------------------- > > scala> spark.sql("select 1 as col1").createGlobalTempView("gView1") > > > > *Instance 2 of spark-shell *(while Instance 1 of spark-shell is still > alive) > > --------------------------------- > > scala> spark.sql("select * from global_temp.gView1").show() > > org.apache.spark.sql.AnalysisException: Table or view not found: > `global_temp`.`gView1` > > 'Project [*] > > +- 'UnresolvedRelation `global_temp`.`gView1` > > > > I am expecting that global temporary view created in shell 1 should be > accessible in shell 2, but it isn=E2=80=99t! > > Please correct me if I missing something here. > > > > Thanks (in advance), > > Hemanth > > > --001a114dfb38ac82d9054dbdcc2d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Look at alluxio for sharing across drivers or spark jobse= rver

Le=C2= =A022 avr. 2017 10:24 AM, "Hemanth Gudela" <hemanth.gudela@qvantel.com> a =C3=A9crit= =C2=A0:

Thanks = for your reply.

= =C2=A0

Creatin= g a table is an option, but such approach slows down reads & writes for= a real-time analytics streaming use case that I=E2=80=99m currently workin= g on.

If at a= ll global temporary view could have been accessible across sessions/spark c= ontexts, that would have simplified my usecase a lot.<= /p>

= =C2=A0

But yea= h, thanks for explaining the behavior of global temporary view, now it=E2= =80=99s clear J=

= =C2=A0

-Hemant= h

= =C2=A0

From: Felix Cheung <felixcheung_m@hotmail.com>
Date: Saturday, 22 April 2017 at 11.05
To: Hemanth Gudela <hemanth.gudela@qvantel.com>, "user@spark.apache.org"= ; <user@spark= .apache.org>
Subject: Re: Spark SQL - Global Temporary View is not behaving as ex= pected

=C2=A0

Cross session is this context is multiple spark sess= ions from the same spark context. Since you are running two shells, you are= having different spark context.

=C2=A0

Do you have to you a temp view? Could you create a t= able?

=C2=A0

_____________________________
From: Hemanth Gudela <hemanth.gudela@qvantel.com>
Sent: Saturday, April 22, 2017 12:57 AM
Subject: Spark SQL - Global Temporary View is not behaving as expected
To: <user@spa= rk.apache.org>



Hi,

=C2=A0

According to documentation, global temporary views are cross-session accessible.

=C2=A0

But when I try to q= uery a global temporary view from another spark shell like this=C3=A0

Instance 1 of sp= ark-shell

-------------------= ---------------

s= cala> spark.sql("select 1 as col1").createGlobalTempView("= ;gView1")

=C2=A0

Instance 2 of sp= ark-shell (while Instance 1 of spark-shell is st= ill alive)

-------------------= --------------

s= cala> spark.sql("select * from global_temp.gView1").show()

org.apach= e.spark.sql.AnalysisException: Table or view not found: `global_temp`.= `gView1`

'Proj= ect [*]

+- 'U= nresolvedRelation `global_temp`.`gView1`

=C2=A0

I am expecting that= global temporary view created in shell 1 should be accessible in shell 2, = but it isn=E2=80=99t!

Please correct me i= f I missing something here.

=C2=A0

Thanks (in advance)= ,

Hemanth

=C2=A0

--001a114dfb38ac82d9054dbdcc2d--