spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rishi Yadav <ri...@infoobjects.com>
Subject Re: Spark SQL : how to find element where a field is in a given set
Date Sun, 02 Nov 2014 11:27:08 GMT
did you create SQLContext?

On Sat, Nov 1, 2014 at 7:51 PM, abhinav chowdary <abhinav.chowdary@gmail.com
> wrote:

> I have same requirement of passing list of values to in clause, when i am
> trying to do
>
> i am getting below error
>
> scala> val longList = Seq[Expression]("a", "b")
> <console>:11: error: type mismatch;
>  found   : String("a")
>  required: org.apache.spark.sql.catalyst.expressions.Expression
>        val longList = Seq[Expression]("a", "b")
>
> Thanks
>
>
> On Fri, Aug 29, 2014 at 3:52 PM, Michael Armbrust <michael@databricks.com>
> wrote:
>
>> This feature was not part of that version.  It will be in 1.1.
>>
>>
>> On Fri, Aug 29, 2014 at 12:33 PM, Jaonary Rabarisoa <jaonary@gmail.com>
>> wrote:
>>
>>>
>>> 1.0.2
>>>
>>>
>>> On Friday, August 29, 2014, Michael Armbrust <michael@databricks.com>
>>> wrote:
>>>
>>>> What version are you using?
>>>>
>>>>
>>>>
>>>> On Fri, Aug 29, 2014 at 2:22 AM, Jaonary Rabarisoa <jaonary@gmail.com>
>>>> wrote:
>>>>
>>>>> Still not working for me. I got a compilation error : *value in is
>>>>> not a member of Symbol.* Any ideas ?
>>>>>
>>>>>
>>>>> On Fri, Aug 29, 2014 at 9:46 AM, Michael Armbrust <
>>>>> michael@databricks.com> wrote:
>>>>>
>>>>>> To pass a list to a variadic function you can use the type ascription
>>>>>> :_*
>>>>>>
>>>>>> For example:
>>>>>>
>>>>>> val longList = Seq[Expression]("a", "b", ...)
>>>>>> table("src").where('key in (longList: _*))
>>>>>>
>>>>>> Also, note that I had to explicitly specify Expression as the type
>>>>>> parameter of Seq to ensure that the compiler converts "a" and "b"
into
>>>>>> Spark SQL expressions.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 28, 2014 at 11:52 PM, Jaonary Rabarisoa <
>>>>>> jaonary@gmail.com> wrote:
>>>>>>
>>>>>>> ok, but what if I have a long list do I need to hard code like
this
>>>>>>> every element of my list of is there a function that translate
a list into
>>>>>>> a tuple ?
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Aug 29, 2014 at 3:24 AM, Michael Armbrust <
>>>>>>> michael@databricks.com> wrote:
>>>>>>>
>>>>>>>> You don't need the Seq, as in is a variadic function.
>>>>>>>>
>>>>>>>> personTable.where('name in ("foo", "bar"))
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa <
>>>>>>>> jaonary@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> What is the expression that I should use with spark sql
DSL if I
>>>>>>>>> need to retreive
>>>>>>>>> data with a field in a given set.
>>>>>>>>> For example :
>>>>>>>>>
>>>>>>>>> I have the following schema
>>>>>>>>>
>>>>>>>>> case class Person(name: String, age: Int)
>>>>>>>>>
>>>>>>>>> And I need to do something like :
>>>>>>>>>
>>>>>>>>> personTable.where('name in Seq("foo", "bar")) ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Jaonary
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
> --
> Warm Regards
> Abhinav Chowdary
>

Mime
View raw message