From user-return-61489-apmail-spark-user-archive=spark.apache.org@spark.apache.org Wed Aug 17 16:23:25 2016 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 17119195C9 for ; Wed, 17 Aug 2016 16:23:25 +0000 (UTC) Received: (qmail 97719 invoked by uid 500); 17 Aug 2016 16:23:19 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 97599 invoked by uid 500); 17 Aug 2016 16:23:19 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 97589 invoked by uid 99); 17 Aug 2016 16:23:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Aug 2016 16:23:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id CBD37180698 for ; Wed, 17 Aug 2016 16:23:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.449 X-Spam-Level: * X-Spam-Status: No, score=1.449 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id BZA8VRyRUucM for ; Wed, 17 Aug 2016 16:23:16 +0000 (UTC) Received: from mail-yb0-f175.google.com (mail-yb0-f175.google.com [209.85.213.175]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id C55015FBBE for ; Wed, 17 Aug 2016 16:23:15 +0000 (UTC) Received: by mail-yb0-f175.google.com with SMTP id e31so25483170ybi.3 for ; Wed, 17 Aug 2016 09:23:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=PLaKnMkOC29cBWivnIYwCl2Yez8KbEYr4AXoAshDR4E=; b=YmWoSno3piUWt3b08nvgZG2U3ZKTVLgQYWZjQDwJNOklSP/Gay+NdV4Z0/8RgBWDrR Zq93O30v9eI2l3xwAIpPbJiaSi2pXkl3Ftko1ROQ0Hc0Gz964inz6F5rnkT+8bT5AuEZ ntjZpqVonFn3XVxjb3KDEj/jBSRbiXT4OxbgUdDclJFdw4LyHIQidURb1qL5+CETrd8n JawmrmdI+JczzM+iw2gHHrwi6qHnDAJYK6ityAghVZrW/7wCP9RlcRn36ZMKjC14mFq1 K037cbJv4gn5qUrq+Ie7gHLEAaQKPC/zFXcyDAxxOEbDgWT+v5Jr2gF1E7ZLvOVMyLxT JKzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=PLaKnMkOC29cBWivnIYwCl2Yez8KbEYr4AXoAshDR4E=; b=J/2J70mAGfHKWtwqRm7ruoEvB4ifuVcT5/mH/GQDYCC6bQWS6O6f66iFp06SO3GukA KO2+zAVhO1ol3WxF0kI4zez9t2+xTe+DNnthHmhkb2fVqIdF6AOAKyXj7AAmIjLY5Rbp EztFfpg0J8KDKQXvqsZTLOAZCPFB5sM+bQb7NckM7lQZc3UbpscgGC5aU53smwrUAtmS GrWWZuXWGSt/gjqhJ9iuUC8Top9YirwmkhQ4MSJE2aq8xFh2dhIgERQF64Ux60j/MGF7 CgkfmgSGhF51zsDboDhuKlVEQjCF6/hLNcSIbaeRoOHvFGv1pAZPqPuq8oDSR4fD9uv4 ZZcg== X-Gm-Message-State: AEkoouuamYtbdvEwdcPComDe4RrBHG6gl7Bnw2UASH1l8UdJLUCTfUvE5D8WAcZDKGnBiQG8leyJjaUDHIyIMg== X-Received: by 10.37.214.194 with SMTP id n185mr29075336ybg.131.1471450988928; Wed, 17 Aug 2016 09:23:08 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.92.194 with HTTP; Wed, 17 Aug 2016 09:23:08 -0700 (PDT) In-Reply-To: References: From: vr spark Date: Wed, 17 Aug 2016 09:23:08 -0700 Message-ID: Subject: Re: Attempting to accept an unknown offer To: Ted Yu Cc: user Content-Type: multipart/alternative; boundary=94eb2c0604ca80642c053a46e198 --94eb2c0604ca80642c053a46e198 Content-Type: text/plain; charset=UTF-8 My code is very simple, if i use other hive tables, my code works fine. This particular table (virtual view) is huge and might have more metadata. It has only two columns. virtual view name is : cluster_table # col_name data_type ln string parti int here is snippet... from pyspark import SparkConf, SparkContext from pyspark.sql import HiveContext import pyspark.sql import json myconf=SparkConf().setAppName("sql") spcont=SparkContext(conf=myconf) sqlcont=HiveContext(spcont) res=sqlcont.sql("select parti FROM h.cluster_table WHERE parti > 408910 and parti <408911 limit 10") print res.printSchema() print 'res' print res df=res.collect() print 'after collect' print df Here is the ouput after i submit the job I0817 09:18:40.606465 31409 sched.cpp:262] New master detected at master@x.y.17.56:6750 I0817 09:18:40.607461 31409 sched.cpp:272] No credentials provided. Attempting to register without authentication I0817 09:18:40.612763 31409 sched.cpp:641] Framework registered with b859f2f3-7484-482d-8c0d-35bd91c1ad0a-6336 16/08/17 09:18:57 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 16/08/17 09:18:57 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException root |-- parti: integer (nullable = true) None res DataFrame[partition_epoch_hourtenth: int] 2016-08-17 09:19:20,648:31315(0x7fafebfb1700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 19ms 2016-08-17 09:19:30,662:31315(0x7fafebfb1700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 13ms W0817 09:20:01.715824 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676564 W0817 09:20:01.716455 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676630 W0817 09:20:01.716645 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676713 W0817 09:20:01.724409 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676554 W0817 09:20:01.724728 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676555 W0817 09:20:01.724936 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676556 W0817 09:20:01.725126 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676557 W0817 09:20:01.725309 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676558. and many more lines like this on the screen with similar message.... On Wed, Aug 17, 2016 at 9:08 AM, Ted Yu wrote: > Please include user@ in your reply. > > Can you reveal the snippet of hive sql ? > > On Wed, Aug 17, 2016 at 9:04 AM, vr spark wrote: > >> spark 1.6.1 >> mesos >> job is running for like 10-15 minutes and giving this message and i >> killed it. >> >> In this job, i am creating data frame from a hive sql. There are other >> similar jobs which work fine >> >> On Wed, Aug 17, 2016 at 8:52 AM, Ted Yu wrote: >> >>> Can you provide more information ? >>> >>> Were you running on YARN ? >>> Which version of Spark are you using ? >>> >>> Was your job failing ? >>> >>> Thanks >>> >>> On Wed, Aug 17, 2016 at 8:46 AM, vr spark wrote: >>> >>>> >>>> W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an >>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492 >>>> >>>> W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an >>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493 >>>> >>>> W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an >>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494 >>>> >>>> W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an >>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495 >>>> >>>> W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an >>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496 >>>> >>>> W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an >>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497 >>>> >>>> W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept an >>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498 >>>> >>>> W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept an >>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499 >>>> >>>> W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an >>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500 >>>> >>>> 2016-08-16 23:18:41,877:16226(0x7f71271b6700):ZOO_WARN@zookeeper_intere >>>> st@1557: Exceeded deadline by 13ms >>>> >>>> 2016-08-16 23:21:12,007:16226(0x7f71271b6700):ZOO_WARN@zookeeper_intere >>>> st@1557: Exceeded deadline by 11ms >>>> >>>> >>>> >>>> >>> >> > --94eb2c0604ca80642c053a46e198 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

My code is very simple, if i use other hive = tables, my code works fine.=C2=A0This =C2=A0particular table (virtua= l view) is huge and might have =C2=A0more metadata.

It has= only two columns.

virtual view name is : cluster_table=C2=A0

# col_name=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = data_type =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0

=C2=A0ln=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0=C2=A0=C2=A0 =C2=A0string =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0

=C2=A0parti=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0=C2=A0int =C2=A0 =C2=A0 =C2=A0 =C2=A0


here is sni= ppet...

from pyspark import SparkConf, SparkCon= text

from pyspark.sql import HiveContext

import pyspark.sql

import json

myconf=3DSparkConf().setAppName("sql")

spcont=3DSparkContext(conf=3Dmyconf)

sqlcont=3DHiveContext(spcont)

res=3Dsqlcont.sql("select=C2=A0 parti FROM h.cluster_tab= le WHERE parti > 408910 and parti <408911=C2=A0=C2=A0limit 10")<= br>

print res.printSchema()

print 'res'

print res

df=3Dres.collect()

print 'after collect'

print df


Here is the ouput after i submit the job

I0817 09:18:40.606465 31409 sched.c= pp:262] New master detected at master@x.y.17.56:6750

I0817 09:18:40.607461 31409 sched.cpp:272] No credential= s provided. Attempting to register without authentication

I0817 09:18:40.612763 31409 sched.cpp:641] Framewor= k registered with b859f2f3-7484-482d-8c0d-35bd91c1ad0a-6336

16/08/17 09:18:57 WARN ObjectStore: Version infor= mation not found in metastore. hive.metastore.schema.verification is not en= abled so recording the schema version 1.2.0

16/08/17 09:18:57 WARN ObjectStore: Failed to get database defaul= t, returning NoSuchObjectException

root

=C2=A0|-- parti: integer (nu= llable =3D true)


None

= res

DataFrame[partition_epoch_hour= tenth: int]

2016-08-17 09:19:20,64= 8:31315(0x7fafebfb1700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline= by 19ms

2016-08-17 09:19:30,662:3= 1315(0x7fafebfb1700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by= 13ms

W0817 09:20:01.715824 31412 = sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c= 0d-35bd91c1ad0a-O168676564

W0817 0= 9:20:01.716455 31412 sched.cpp:1195] Attempting to accept an unknown offer = b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676630

W0817 09:20:01.716645 31412 sched.cpp:1195] Attempting to acc= ept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676713=

W0817 09:20:01.724409 31412 sched.cpp:11= 95] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1a= d0a-O168676554

W0817 09:20:01.7247= 28 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-748= 4-482d-8c0d-35bd91c1ad0a-O168676555

W0817 09:20:01.724936 31412 sched.cpp:1195] Attempting to accept an unkno= wn offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676556

W0817 09:20:01.725126 31412 sched.cpp:1195] Attempti= ng to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O1686765= 57

W0817 09:20:01.725309 31412 sched.cpp:11= 95] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1a= d0a-O168676558.

and many more line= s like this on the screen with similar message....

=


On Wed, Aug 17, 2016 at 9:08 AM, Ted Yu <yuzhihong@= gmail.com> wrote:
Please include user@ in your reply.

Can you rev= eal the snippet of hive sql ?

On Wed, A= ug 17, 2016 at 9:04 AM, vr spark <vrspark123@gmail.com> w= rote:
spark 1.6.1
me= sos
job is running for like 10-15 minutes and giving this message= and i killed it.

In this job, i am creating data = frame from a hive sql. There are other similar jobs which work fine

On Wed, Aug 1= 7, 2016 at 8:52 AM, Ted Yu <yuzhihong@gmail.com> wrote:
Can you provide more infor= mation ?

Were you running on YARN ?
Which vers= ion of Spark are you using ?

Was your job failing = ?

Thanks

On Wed, Aug 17, 2016 at 8:46 AM, vr s= park <vrspark123@gmail.com> wrote:


W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept a= n unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492=

W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept a= n unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493=

W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept a= n unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494=

W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept a= n unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495=

W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept a= n unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496=

W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept a= n unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497=

W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept a= n unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498=

W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept a= n unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499=

W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an unkn= own offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500=C2=A0

= 2016-08-16 23:18:41,877:16226(0x7f71271b6700):ZOO_WARN@zookeeper= _interest@1557: Exceeded deadline by 13ms

2016-08-16 23:21:12,007:16226(0x7f71271b6700):ZOO_WARN@zo= okeeper_interest@1557: Exceeded deadline by 11ms






--94eb2c0604ca80642c053a46e198--