spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: StorageLevel: OFF_HEAP
Date Wed, 18 Mar 2015 21:27:36 GMT
Ranga:
Please apply the patch from:
https://github.com/apache/spark/pull/4867

And rebuild Spark - the build would use Tachyon-0.6.1

Cheers

On Wed, Mar 18, 2015 at 2:23 PM, Ranga <sranga@gmail.com> wrote:

> Hi Haoyuan
>
> No. I assumed that Spark-1.3.0 was already built with Tachyon-0.6.0. If
> not, I can rebuild and try. Could you let me know how to rebuild with 0.6.0?
> Thanks for your help.
>
>
> - Ranga
>
> On Wed, Mar 18, 2015 at 12:59 PM, Haoyuan Li <haoyuan.li@gmail.com> wrote:
>
>> Did you recompile it with Tachyon 0.6.0?
>>
>> Also, Tachyon 0.6.1 has been released this morning:
>> http://tachyon-project.org/ ; https://github.com/amplab/tachyon/releases
>>
>> Best regards,
>>
>> Haoyuan
>>
>> On Wed, Mar 18, 2015 at 11:48 AM, Ranga <sranga@gmail.com> wrote:
>>
>>> I just tested with Spark-1.3.0 + Tachyon-0.6.0 and still see the same
>>> issue. Here are the logs:
>>> 15/03/18 11:44:11 ERROR : Invalid method name: 'getDataFolder'
>>> 15/03/18 11:44:11 ERROR : Invalid method name: 'user_getFileId'
>>> 15/03/18 11:44:11 ERROR storage.TachyonBlockManager: Failed 10 attempts
>>> to create tachyon dir in
>>> /tmp_spark_tachyon/spark-e3538a20-5e42-48a4-ad67-4b97aded90e4/<driver>
>>>
>>> Thanks for any other pointers.
>>>
>>>
>>> - Ranga
>>>
>>>
>>>
>>> On Wed, Mar 18, 2015 at 9:53 AM, Ranga <sranga@gmail.com> wrote:
>>>
>>>> Thanks for the information. Will rebuild with 0.6.0 till the patch is
>>>> merged.
>>>>
>>>> On Tue, Mar 17, 2015 at 7:24 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>>
>>>>> Ranga:
>>>>> Take a look at https://github.com/apache/spark/pull/4867
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Tue, Mar 17, 2015 at 6:08 PM, fightfate@163.com <fightfate@163.com>
>>>>> wrote:
>>>>>
>>>>>> Hi, Ranga
>>>>>>
>>>>>> That's true. Typically a version mis-match issue. Note that spark
>>>>>> 1.2.1 has tachyon built in with version 0.5.0 , I think you may need
to
>>>>>> rebuild spark
>>>>>> with your current tachyon release.
>>>>>> We had used tachyon for several of our spark projects in a production
>>>>>> environment.
>>>>>>
>>>>>> Thanks,
>>>>>> Sun.
>>>>>>
>>>>>> ------------------------------
>>>>>> fightfate@163.com
>>>>>>
>>>>>>
>>>>>> *From:* Ranga <sranga@gmail.com>
>>>>>> *Date:* 2015-03-18 06:45
>>>>>> *To:* user@spark.apache.org
>>>>>> *Subject:* StorageLevel: OFF_HEAP
>>>>>> Hi
>>>>>>
>>>>>> I am trying to use the OFF_HEAP storage level in my Spark (1.2.1)
>>>>>> cluster. The Tachyon (0.6.0-SNAPSHOT) nodes seem to be up and running.
>>>>>> However, when I try to persist the RDD, I get the following error:
>>>>>>
>>>>>> ERROR [2015-03-16 22:22:54,017] ({Executor task launch worker-0}
>>>>>> TachyonFS.java[connect]:364)  - Invalid method name:
>>>>>> 'getUserUnderfsTempFolder'
>>>>>> ERROR [2015-03-16 22:22:54,050] ({Executor task launch worker-0}
>>>>>> TachyonFS.java[getFileId]:1020)  - Invalid method name: 'user_getFileId'
>>>>>>
>>>>>> Is this because of a version mis-match?
>>>>>>
>>>>>> On a different note, I was wondering if Tachyon has been used in
a
>>>>>> production environment by anybody in this group?
>>>>>>
>>>>>> Appreciate your help with this.
>>>>>>
>>>>>>
>>>>>> - Ranga
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Haoyuan Li
>> AMPLab, EECS, UC Berkeley
>> http://www.cs.berkeley.edu/~haoyuan/
>>
>
>

Mime
View raw message