spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Kumar <kumarami...@gmail.com>
Subject Re: type issue: found RDD[T] expected RDD[A]
Date Wed, 20 Aug 2014 05:06:27 GMT
Hi Evan, Patrick and Tobias,

So, It worked for what I needed it to do. I followed Yana's suggestion of
using parameterized type of  [T <: Product:ClassTag:TypeTag]

more concretely, I was trying to make the query process a bit more fluent
 -some pseudocode but with correct types

val table:SparkTable[POJO] = new
SparkTable[POJO](sqlContext,extractor:String=>POJO)
val data=
       table.atLocation("hdfs://....")
       .withName("tableName")
       .makeRDD("SELECT * FROM tableName")


class SparkTable[T <: Product : ClassTag :TypeTag](val
sqlContext:SQLContext, val extractor: (String) => (T) ) {
  private[this] var location:Option[String] =None
  private[this] var name:Option[String]=None
  private[this] val sc = sqlContext.sparkContext

  def withName(name:String):SparkTable[T]={..}
  def atLocation(path:String):SparkTable[T]={.. }
  def makeRDD(sqlQuery:String):SchemaRDD={
    ...
    import sqlContext._
    val rdd:RDD[String] = sc.textFile(this.location.get)
    val rddT:RDD[T] = rdd.map(extractor)
    val schemaRDD= createSchemaRDD(rddT)
    schemaRDD.registerAsTable(name.get)
    val all = sqlContext.sql(sqlQuery)
    all
  }

}

Best,
Amit

On Tue, Aug 19, 2014 at 9:13 PM, Evan Chan <velvia.github@gmail.com> wrote:

> That might not be enough.  Reflection is used to determine what the
> fields are, thus your class might actually need to have members
> corresponding to the fields in the table.
>
> I heard that a more generic method of inputting stuff is coming.
>
> On Tue, Aug 19, 2014 at 6:43 PM, Tobias Pfeiffer <tgp@preferred.jp> wrote:
> > Hi,
> >
> > On Tue, Aug 19, 2014 at 7:01 PM, Patrick McGloin <
> mcgloin.patrick@gmail.com>
> > wrote:
> >>
> >> I think the type of the data contained in your RDD needs to be a known
> >> case class and not abstract for createSchemaRDD.  This makes sense when
> you
> >> think it needs to know about the fields in the object to create the
> schema.
> >
> >
> > Exactly this. The actual message pointing to that is:
> >
> >     "inferred type arguments [T] do not conform to method
> createSchemaRDD's
> > type parameter bounds [A <: Product]"
> >
> > All case classes are automatically subclasses of Product, but otherwise
> you
> > will have to extend Product and add the required methods yourself.
> >
> > Tobias
> >
>

Mime
View raw message