spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 张万新 <kevinzwx1...@gmail.com>
Subject Re: [SS]How to add a column with custom system time?
Date Tue, 12 Sep 2017 07:05:46 GMT
The spark version is 2.2.0

Michael Armbrust <michael@databricks.com>于2017年9月12日周二 下午12:32写道:

> Which version of spark?
>
> On Mon, Sep 11, 2017 at 8:27 PM, 张万新 <kevinzwx1992@gmail.com> wrote:
>
>> Thanks for reply, but using this method I got an exception:
>>
>> "Exception in thread "main"
>> org.apache.spark.sql.streaming.StreamingQueryException: nondeterministic
>> expressions are only allowed in
>>
>> Project, Filter, Aggregate or Window"
>>
>> Can you give more advice?
>>
>> Michael Armbrust <michael@databricks.com>于2017年9月12日周二 上午4:48写道:
>>
>>> import org.apache.spark.sql.functions._
>>>
>>> df.withColumn("window", window(current_timestamp(), "15 minutes"))
>>>
>>> On Mon, Sep 11, 2017 at 3:03 AM, 张万新 <kevinzwx1992@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> In structured streaming how can I add a column to a dataset with
>>>> current system time aligned with 15 minutes?
>>>>
>>>> Thanks.
>>>>
>>>
>>>
>

Mime
View raw message