spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subhajit Purkayastha" <spurk...@p3si.net>
Subject DataFrame Data Manipulation - Based on a timestamp column Not Working
Date Tue, 23 Aug 2016 22:46:29 GMT
Using spark 2.0  & scala 2.11.8, I have a DataFrame with a timestamp column

 

root

|-- ORG_ID: integer (nullable = true)

|-- HEADER_ID: integer (nullable = true)

|-- ORDER_NUMBER: integer (nullable = true)

|-- LINE_ID: integer (nullable = true)

|-- LINE_NUMBER: integer (nullable = true)

|-- ITEM_TYPE_CODE: string (nullable = true)

|-- ORGANIZATION_ID: integer (nullable = true)

|-- INVENTORY_ITEM_ID: integer (nullable = true)

|-- SCHEDULE_SHIP_DATE: timestamp (nullable = true)

|-- ORDER_QUANTITY_UOM: string (nullable = true)

|-- UNIT_SELLING_PRICE: double (nullable = true)

|-- OPEN_QUANTITY: double (nullable = true)

 

[204,94468,56721,197328,1,STANDARD,207,149,2004-01-08
23:59:59.0,Ea,1599.0,28.0]

[204,94468,56721,197331,2,STANDARD,207,151,2004-01-08
23:59:59.0,Ea,1899.05,40.0]

[204,94468,56721,197332,3,STANDARD,207,436,2004-01-08
23:59:59.0,Ea,300.0,24.0]

[204,94468,56721,197335,4,STANDARD,207,3751,2004-01-08
23:59:59.0,Ea,380.0,24.0]

 

I want to manipulate the dataframe data based on a parameter =
demand_time_fence_date

 

var demand_timefence_end_date_instance = new
MutableDateTime(planning_start_date)    

var demand_timefence_days =
demand_timefence_end_date_instance.addDays(demand_time_fence)    

val demand_timefence_end_date =
ISODateTimeFormat.yearMonthDay().print(demand_timefence_end_date_instance)

 

var filter_stmt = "from_unixtime(SCHEDULE_SHIP_DATE,'yyyy-MM-dd') >= "+
demand_timefence_end_date  

 

val sales_order_dataFrame =
sales_order_base_dataFrame.filter(filter_stmt).limit(10)

 

What is the correct syntax to pass the parameter value? 

 

The above filter statement is not working to restrict the dataset

 

Thanks,

 

Subhajit

 

 


Mime
View raw message