From user-return-61896-apmail-spark-user-archive=spark.apache.org@spark.apache.org Fri Aug 26 20:54:44 2016 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C0F3E19979 for ; Fri, 26 Aug 2016 20:54:44 +0000 (UTC) Received: (qmail 91061 invoked by uid 500); 26 Aug 2016 20:54:40 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 90943 invoked by uid 500); 26 Aug 2016 20:54:40 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 90933 invoked by uid 99); 26 Aug 2016 20:54:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Aug 2016 20:54:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 20ECAC224A for ; Fri, 26 Aug 2016 20:54:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.552 X-Spam-Level: X-Spam-Status: No, score=-0.552 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id b-osAx2zA3ms for ; Fri, 26 Aug 2016 20:54:39 +0000 (UTC) Received: from mail-pa0-f48.google.com (mail-pa0-f48.google.com [209.85.220.48]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 27A595FBB6 for ; Fri, 26 Aug 2016 20:54:39 +0000 (UTC) Received: by mail-pa0-f48.google.com with SMTP id fi15so30372978pac.1 for ; Fri, 26 Aug 2016 13:54:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-transfer-encoding:subject:message-id:date:to :mime-version; bh=Mopu9G00VPkTlKs8738wLWNFH4AO/G6b9mG+Gxn7V4c=; b=I9AsAIYm4gnCNk4pHcUBeKSjBVmF6M2zR+yF2NhYmQMmyNbvWuoWzx7OGWrjmTBN2f qKtBtT0pmlJInr9a7rhi99bkQ/aOIGZmvoUG1rY/VyS0MpIfwC/R+OYRs53U32TPPf9J lEAIH728/Bo5jlaXJFdcxtfi1kIRwmVsyWQUyyFgRRfQmEGgvW51ZQRXCgCuy+ErfQle LYT/OhEKUcRzXYxVBBVvSntLq4vBWSn2j8WTR7gy4WanXEzO0lLdq7vRpRSxTFwhhx89 qFkSWDKEb1F4CakltY3NkZOAeW1Y1V6iqZMbovN3gBWqJNfZwE39HvS82WGdNRV/Qowr MwwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-transfer-encoding:subject :message-id:date:to:mime-version; bh=Mopu9G00VPkTlKs8738wLWNFH4AO/G6b9mG+Gxn7V4c=; b=MvzpVYOnplTtSGlvYXCHbRYkm9meLAbmRU4XJ3UQEATE9HKqprO8N/Rfpw2Bble05d ZtMY4OlgGsYUdypnAM/bZ86Md2Aax2wioyLp/ulXcfnmWoTHsYKS/mMJC3E5TD5FKo7B 0G5/t8WZoJh9zizuHqZUZo25bWOLP7g74vH/jbEQsALME/YJk06xzn1PoNZK/91zP0iw WeZvs700bsXP44UXSl/43i+cIW4iLR3v0H/+vZDeu7wfBf7/xXnaL25W67uEjUNaoiPR ThMo9vlrF4Zq0njTipiQscoDlx1RglIKZBh5c4lcpyb8UhmIV3R5I4PCIGOE5Tr4xcc2 2JgA== X-Gm-Message-State: AE9vXwNILsLm6c+HY0VIB4wd41VdZpGcSXsizLmYIL5FNH1lzu58thVUwkEEIrgOFkJaZA== X-Received: by 10.66.123.105 with SMTP id lz9mr9589702pab.56.1472244877794; Fri, 26 Aug 2016 13:54:37 -0700 (PDT) Received: from [10.111.0.178] ([207.171.30.158]) by smtp.gmail.com with ESMTPSA id 18sm30882612pfn.33.2016.08.26.13.54.36 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Aug 2016 13:54:36 -0700 (PDT) From: Benjamin Kim Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Spark 1.6 Streaming with Checkpointing Message-Id: <964624F7-ACC5-4638-8EE7-6B3E967A05BA@gmail.com> Date: Fri, 26 Aug 2016 13:54:34 -0700 To: user@spark.apache.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) I am trying to implement checkpointing in my streaming application but I = am getting a not serializable error. Has anyone encountered this? I am = deploying this job in YARN clustered mode. Here is a snippet of the main parts of the code. object S3EventIngestion { //create and setup streaming context def createContext( batchInterval: Integer, checkpointDirectory: String, = awsS3BucketName: String, databaseName: String, tableName: String, = partitionByColumnName: String ): StreamingContext =3D { println("Creating new context") val sparkConf =3D new SparkConf().setAppName("S3EventIngestion") val sc =3D new SparkContext(sparkConf) val sqlContext =3D new SQLContext(sc) // Create the streaming context with batch interval val ssc =3D new StreamingContext(sc, Seconds(batchInterval)) =20 // Create a text file stream on an S3 bucket val csv =3D ssc.textFileStream("s3a://" + awsS3BucketName + "/") csv.foreachRDD(rdd =3D> { if (!rdd.partitions.isEmpty) { // process data } }) ssc.checkpoint(checkpointDirectory) ssc } def main(args: Array[String]) { if (args.length !=3D 6) { System.err.println("Usage: S3EventIngestion = = ") System.exit(1) } // Get streaming context from checkpoint data or create a new = one val context =3D StreamingContext.getOrCreate(checkpoint, () =3D> createContext(interval, checkpoint, bucket, = database, table, partitionBy)) //start streaming context context.start() context.awaitTermination() } } Can someone help please? Thanks, Ben= --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscribe@spark.apache.org