spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-28567) pyspark-sql returns error "Paths and fileStatuses are expected to be 1. . Got paths: 1 fileStatuses: 26"
Date Thu, 01 Aug 2019 07:16:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-28567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hyukjin Kwon resolved SPARK-28567.
----------------------------------
    Resolution: Incomplete

> pyspark-sql returns error "Paths and fileStatuses are expected to be 1. . Got paths:
1 fileStatuses: 26"
> --------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-28567
>                 URL: https://issues.apache.org/jira/browse/SPARK-28567
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, SQL
>    Affects Versions: 2.3.2
>         Environment: Sql below..
> query = "WITH hdr as ( \
> select vin_nbr, element_cd, max(element_timstm) as element_timstmp from dev_edm.gk_veh_element
\
> where vin_nbr like '2C%' and element_timstm >= cast('{0}' as timestamp) - interval
'1' day \
> and element_timstm < cast('{1}' as timestamp) \
> and ((yr_nbr=substring('{2}', 1, 4) and mth_nbr=substring('{3}', 6, 2) and day_nbr=substring('{4}',
9, 2)) \
>   or (yr_nbr=substring('{5}', 1, 4) and mth_nbr=substring('{6}', 6, 2) and day_nbr=substring('{7}',
9, 2))) \
> and ( \
> (ELEMENT_CD = 'LEFT_FR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) = 'true'
) or \
> (ELEMENT_CD = 'RIGHT_FR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) = 'true')
or \
> (ELEMENT_CD = 'LEFT_RR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) = 'true'
) or \
> (ELEMENT_CD = 'RIGHT_RR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) = 'true')
or \
> (ELEMENT_CD in ('RIGHT_FRONT_TIRE_PRESSURE', 'RIGHT_REAR_TIRE_PRESSURE', 'LEFT_FRONT_TIRE_PRESSURE',
'LEFT_REAR_TIRE_PRESSURE') ) or \
> (ELEMENT_CD = 'DIESEL_EXHAUST_FLUID'  and (lower(ELEMENT_VAL) = 'on' ) ) or \
> (ELEMENT_CD = 'BRAKE_FLUID_LOW'  and (lower(ELEMENT_VAL) = 'true'  ) )  or \
> (ELEMENT_CD = 'BATTERY_CRITICALLY_LOW'  and (lower(ELEMENT_VAL) = 'true' ) ) or \
> (ELEMENT_CD = 'ODO_READ' ) or \
> (ELEMENT_CD = 'EOL_READ' and ELEMENT_VAL < 30 ) ) \
> group by vin_nbr, element_cd \
> order by vin_nbr, element_cd) \
> SELECT distinct hdr.vin_nbr, hdr.element_cd, \
> case when dtl.element_cd like '%PRESSURE' then cast(cast(round(dtl.element_val * 1000
/ 6894.76) as int) as string) else dtl.element_val end element_val, \
> hdr.element_timstmp, 'BDC LOAD' load_process_nm, 'CFL_SVC' created_user_id, current_timestamp
created_timstm, 'CFL_SVC' updated_user_id, current_timestamp updated_timstm \
> from dev_edm.gk_veh_element_np dtl, hdr \
> where dtl.vin_nbr = hdr.vin_nbr \
> and dtl.element_cd = hdr.element_cd \
> and dtl.element_timstm = hdr.element_timstmp \
> and ((dtl.yr_nbr=substring('{8}', 1, 4) and dtl.mth_nbr=substring('{9}', 6, 2) and dtl.day_nbr=substring('{10}',
9, 2)) \
>   or (dtl.yr_nbr=substring('{11}', 1, 4) and dtl.mth_nbr=substring('{12}', 6, 2) and
dtl.day_nbr=substring('{11}', 9, 2))) \
> and case \
> when dtl.element_cd like '%PRESSURE_STATUS_LOW_FLAG' then  lower(dtl.element_val) = 'true'
\
> when dtl.element_cd = 'DIESEL_EXHAUST_FLUID' then  lower(dtl.element_val) = 'on' \
> when dtl.element_cd = 'BRAKE_FLUID_LOW' then  lower(dtl.element_val) = 'true' \
> when dtl.element_cd = 'BATTERY_CRITICALLY_LOW' then lower(dtl.element_val) = 'true' \
> else 1=1 end".format(crntdate, crntdate, prevdate, prevdate, prevdate, crntdate, crntdate,
crntdate, prevdate, prevdate, prevdate, crntdate, crntdate, crntdate );
> df = hive.executeQuery(query)
> df.show(10)
>            Reporter: Ganesh Kumar
>            Priority: Major
>
> df.show(10) errors out.
> pyspark-sql returns an error " Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: java.lang.IllegalStateException: Requested to generate single split.
Paths and fileStatuses are expected to be 1. Got paths: 1 fileStatuses: 26"
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message