spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sudhir Babu Pothineni <>
Subject ORC file stripe statistics in Spark
Date Tue, 27 Sep 2016 19:43:48 GMT
I am trying to get number of rows each stripe of ORC file?

hivecontext.orcFile doesn't exist anymore? I am using Spark 1.6.0

scala> val hiveSqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
hiveSqlContext: org.apache.spark.sql.hive.HiveContext =

scala> hiveSqlContext.
analyze                   applySchema
asInstanceOf              baseRelationToDataFrame   cacheTable
clearCache                createDataFrame
createDataset             createExternalTable       dropTempTable
emptyDataFrame            experimental
getAllConfs               getConf                   implicits
isCached                  isInstanceOf
isRootContext             jdbc                      jsonFile
jsonRDD                   listenerManager
load                      newSession                parquetFile
range                     read
refreshTable              setConf                   sparkContext
sql                       table
tableNames                tables                    toString
udf                       uncacheTable


View raw message