From user-return-22934-apmail-spark-user-archive=spark.apache.org@spark.apache.org Mon Dec 22 19:09:21 2014 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3438710B1C for ; Mon, 22 Dec 2014 19:09:21 +0000 (UTC) Received: (qmail 23968 invoked by uid 500); 22 Dec 2014 19:09:18 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 23902 invoked by uid 500); 22 Dec 2014 19:09:18 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 23892 invoked by uid 99); 22 Dec 2014 19:09:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Dec 2014 19:09:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.213.48 as permitted sender) Received: from [209.85.213.48] (HELO mail-yh0-f48.google.com) (209.85.213.48) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Dec 2014 19:08:51 +0000 Received: by mail-yh0-f48.google.com with SMTP id i57so2525371yha.7 for ; Mon, 22 Dec 2014 11:08:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=OuufwJFfETtoPps67FfGwKsJjLvAfYF8jGJ8NmmyDRQ=; b=r5QGAE2jdyVxSeVOSevN947lL5hgDB+R3Yd12bHlZ2OtrZGK2xy3MzwdDr37qnGvdB 3Hzili1bql4IeOZnfzVyyqPXNYuf5ErAmR9nv35EynX0tEobq1B67MM0pT2hmKc4prEy Ltx0PITFer9mqHbA+WZFDWnK/FSdEKlSyIWvZ/OwuXxl+MmWyAy/On+YFNx8L7GfASAS o2xXg0/5wTAoGjhwYhIrPVVv/4tFVeQg0MUfb926Z7CZSlRDEvvyBsyBboQuvcLcHEy6 EyJfe8C5O2vNovITxtfoeLg/Uo6kBdAlrx1PReEVA6dIvLXf3u7AjmE6MobAm3OuXdQJ 8tLQ== MIME-Version: 1.0 X-Received: by 10.170.54.76 with SMTP id 73mr22047624ykw.0.1419275330291; Mon, 22 Dec 2014 11:08:50 -0800 (PST) Received: by 10.170.139.4 with HTTP; Mon, 22 Dec 2014 11:08:50 -0800 (PST) In-Reply-To: <870855689.364663.1419274959022.JavaMail.yahoo@jws11156.mail.ir2.yahoo.com> References: <870855689.364663.1419274959022.JavaMail.yahoo@jws11156.mail.ir2.yahoo.com> Date: Mon, 22 Dec 2014 11:08:50 -0800 Message-ID: Subject: Re: custom python converter from HBase Result to tuple From: Ted Yu To: Antony Mayi Cc: "user@spark.apache.org" Content-Type: multipart/alternative; boundary=001a113949a2e68d1a050ad2c90b X-Virus-Checked: Checked by ClamAV on apache.org --001a113949a2e68d1a050ad2c90b Content-Type: text/plain; charset=UTF-8 Which HBase version are you using ? Can you show the full stack trace ? Cheers On Mon, Dec 22, 2014 at 11:02 AM, Antony Mayi wrote: > Hi, > > can anyone please give me some help how to write custom converter of hbase > data to (for example) tuples of ((family, qualifier, value), ) for pyspark: > > I was trying something like (here trying to tuples of > ("family:qualifier:value", )): > > > class HBaseResultToTupleConverter extends Converter[Any, List[String]] { > override def convert(obj: Any): List[String] = { > val result = obj.asInstanceOf[Result] > result.rawCells().map(cell => > List(Bytes.toString(CellUtil.cloneFamily(cell)), > Bytes.toString(CellUtil.cloneQualifier(cell)), > Bytes.toString(CellUtil.cloneValue(cell))).mkString(":") > ).toList > } > } > > > but then I get a error: > > 14/12/22 16:27:40 WARN python.SerDeUtil: > Failed to pickle Java object as value: $colon$colon, falling back > to 'toString'. Error: couldn't introspect javabean: > java.lang.IllegalArgumentException: wrong number of arguments > > > does anyone have a hint? > > Thanks, > Antony. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org > For additional commands, e-mail: user-help@spark.apache.org > > --001a113949a2e68d1a050ad2c90b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Which HBase version are you using ?

Can= you show the full stack trace ?

Cheers

On Mon, Dec 22, = 2014 at 11:02 AM, Antony Mayi <antonymayi@yahoo.com.invalid= > wrote:
Hi,

can anyone please give me some help how to write custom converter of hbase = data to (for example) tuples of ((family, qualifier, value), ) for pyspark:=

I was trying something like (here trying to tuples of ("family:qualifi= er:value", )):


class HBaseResultToTupleConverter extends Converter[Any, List[String]] { =C2=A0 override def convert(obj: Any): List[String] =3D {
=C2=A0 =C2=A0 val result =3D obj.asInstanceOf[Result]
=C2=A0 =C2=A0 result.rawCells().map(cell =3D> List(Bytes.toString(CellUt= il.cloneFamily(cell)),
=C2=A0 =C2=A0 =C2=A0 Bytes.toString(CellUtil.cloneQualifier(cell)),
=C2=A0 =C2=A0 =C2=A0 Bytes.toString(CellUtil.cloneValue(cell))).mkString(&q= uot;:")
=C2=A0 =C2=A0 ).toList
=C2=A0 }
}


but then I get a error:

14/12/22 16:27:40 WARN python.SerDeUtil:
Failed to pickle Java object as value: $colon$colon, falling back
to 'toString'. Error: couldn't introspect javabean: java.lang.I= llegalArgumentException: wrong number of arguments


does anyone have a hint?

Thanks,
Antony.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


--001a113949a2e68d1a050ad2c90b--