spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: GenericMutableRow and Row mismatch on Spark 1.5?
Date Wed, 07 Oct 2015 14:06:43 GMT
Hemant:
Can you post the code snippet to the mailing list - other people would be
interested.

On Wed, Oct 7, 2015 at 5:50 AM, Hemant Bhanawat <hemant9379@gmail.com>
wrote:

> Will send you the code on your email id.
>
> On Wed, Oct 7, 2015 at 4:37 PM, Ophir Cohen <ophchu@gmail.com> wrote:
>
>> Thanks!
>> Can you check if you can provide example of the conversion?
>>
>>
>> On Wed, Oct 7, 2015 at 2:05 PM, Hemant Bhanawat <hemant9379@gmail.com>
>> wrote:
>>
>>> Oh, this is an internal class of our project and I had used it without
>>> realizing the source.
>>>
>>> Anyway, the idea is to  wrap the InternalRow in a class that derives
>>> from Row. When you implement the functions of the trait 'Row ', the type
>>> conversions from Row types to InternalRow types has to be done for each of
>>> the types. But, as I can see, the primitive types (apart from String) don't
>>> need conversions. Map and Array would need some handling.
>>>
>>> I will check with the author of this code, I think this code can be
>>> contributed to Spark.
>>>
>>> Hemant
>>> www.snappydata.io
>>> linkedin.com/company/snappydata
>>>
>>> On Wed, Oct 7, 2015 at 3:30 PM, Ophir Cohen <ophchu@gmail.com> wrote:
>>>
>>>> From which jar WrappedInternalRow comes from?
>>>> It seems that I can't find it.
>>>>
>>>> BTW
>>>> What I'm trying to do now is to create scala array from the fields and
>>>> than create Row out of that array.
>>>> The problem is that I get types mismatches...
>>>>
>>>> On Wed, Oct 7, 2015 at 8:03 AM, Hemant Bhanawat <hemant9379@gmail.com>
>>>> wrote:
>>>>
>>>>> An approach can be to wrap your MutableRow in WrappedInternalRow which
>>>>> is a child class of Row.
>>>>>
>>>>> Hemant
>>>>> www.snappydata.io
>>>>> linkedin.com/company/snappydata
>>>>>
>>>>>
>>>>> On Tue, Oct 6, 2015 at 3:21 PM, Ophir Cohen <ophchu@gmail.com>
wrote:
>>>>>
>>>>>> Hi Guys,
>>>>>> I'm upgrading to Spark 1.5.
>>>>>>
>>>>>> In our previous version (Spark 1.3 but it was OK on 1.4 as well)
we
>>>>>> created GenericMutableRow
>>>>>> (org.apache.spark.sql.catalyst.expressions.GenericMutableRow) and
return it
>>>>>> as org.apache.spark.sql.Row
>>>>>>
>>>>>> Starting from Spark 1.5 GenericMutableRow isn't extends Row.
>>>>>>
>>>>>> What do you suggest to do?
>>>>>> How can I convert GenericMutableRow to Row?
>>>>>>
>>>>>> Prompt answer will be highly appreciated!
>>>>>> Thanks,
>>>>>> Ophir
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message