spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abhinav chowdary <abhinav.chowd...@gmail.com>
Subject Re: Spark SQL : how to find element where a field is in a given set
Date Sun, 02 Nov 2014 02:51:41 GMT
I have same requirement of passing list of values to in clause, when i am
trying to do

i am getting below error

scala> val longList = Seq[Expression]("a", "b")
<console>:11: error: type mismatch;
 found   : String("a")
 required: org.apache.spark.sql.catalyst.expressions.Expression
       val longList = Seq[Expression]("a", "b")

Thanks


On Fri, Aug 29, 2014 at 3:52 PM, Michael Armbrust <michael@databricks.com>
wrote:

> This feature was not part of that version.  It will be in 1.1.
>
>
> On Fri, Aug 29, 2014 at 12:33 PM, Jaonary Rabarisoa <jaonary@gmail.com>
> wrote:
>
>>
>> 1.0.2
>>
>>
>> On Friday, August 29, 2014, Michael Armbrust <michael@databricks.com>
>> wrote:
>>
>>> What version are you using?
>>>
>>>
>>>
>>> On Fri, Aug 29, 2014 at 2:22 AM, Jaonary Rabarisoa <jaonary@gmail.com>
>>> wrote:
>>>
>>>> Still not working for me. I got a compilation error : *value in is not
>>>> a member of Symbol.* Any ideas ?
>>>>
>>>>
>>>> On Fri, Aug 29, 2014 at 9:46 AM, Michael Armbrust <
>>>> michael@databricks.com> wrote:
>>>>
>>>>> To pass a list to a variadic function you can use the type ascription
>>>>> :_*
>>>>>
>>>>> For example:
>>>>>
>>>>> val longList = Seq[Expression]("a", "b", ...)
>>>>> table("src").where('key in (longList: _*))
>>>>>
>>>>> Also, note that I had to explicitly specify Expression as the type
>>>>> parameter of Seq to ensure that the compiler converts "a" and "b" into
>>>>> Spark SQL expressions.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 28, 2014 at 11:52 PM, Jaonary Rabarisoa <jaonary@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> ok, but what if I have a long list do I need to hard code like this
>>>>>> every element of my list of is there a function that translate a
list into
>>>>>> a tuple ?
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 29, 2014 at 3:24 AM, Michael Armbrust <
>>>>>> michael@databricks.com> wrote:
>>>>>>
>>>>>>> You don't need the Seq, as in is a variadic function.
>>>>>>>
>>>>>>> personTable.where('name in ("foo", "bar"))
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa <
>>>>>>> jaonary@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> What is the expression that I should use with spark sql DSL
if I
>>>>>>>> need to retreive
>>>>>>>> data with a field in a given set.
>>>>>>>> For example :
>>>>>>>>
>>>>>>>> I have the following schema
>>>>>>>>
>>>>>>>> case class Person(name: String, age: Int)
>>>>>>>>
>>>>>>>> And I need to do something like :
>>>>>>>>
>>>>>>>> personTable.where('name in Seq("foo", "bar")) ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers.
>>>>>>>>
>>>>>>>>
>>>>>>>> Jaonary
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>


-- 
Warm Regards
Abhinav Chowdary

Mime
View raw message