spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "1427357147@qq.com" <1427357...@qq.com>
Subject Re: Re: the issue about the + in column,can we support the string please?
Date Sun, 01 Apr 2018 12:28:12 GMT
Hi , 
I checked the code.
It seems it is hard to change the code.
Current code, string + int is translated to double + double.
If I change the the string + int to string + sting, it will incompatible whit old version.

Does anyone have better idea about this issue please?



1427357147@qq.com
 
From: Shmuel Blitz
Date: 2018-03-26 17:17
To: 1427357147@qq.com
CC: spark users; dev
Subject: Re: Re: the issue about the + in column,can we support the string please?
I agree.

Just pointed out the option, in case you missed it.

Cheers,
Shmuel

On Mon, Mar 26, 2018 at 10:57 AM, 1427357147@qq.com <1427357147@qq.com> wrote:
Hi,

Using concat is one of the way.
But the + is more intuitive and easy to understand.



1427357147@qq.com
 
From: Shmuel Blitz
Date: 2018-03-26 15:31
To: 1427357147@qq.com
CC: spark?users; dev
Subject: Re: the issue about the + in column,can we support the string please?
Hi,

you can get the same with:

import org.apache.spark.sql.functions._
import sqlContext.implicits._
import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType} 

val schema = StructType(Array(StructField("name", StringType),
                                StructField("age", IntegerType) ))

val lst = List(Row("Shmuel", 13), Row("Blitz", 23))
val rdd = sc.parallelize(lst)

val df = sqlContext.createDataFrame(rdd,schema)

df.withColumn("newName", concat($"name" ,  lit("abc"))  ).show()

On Mon, Mar 26, 2018 at 6:36 AM, 1427357147@qq.com <1427357147@qq.com> wrote:
Hi  all,

I have a table like below:

+---+---------+-----------+
| id|     name|sharding_id|
+---+---------+-----------+
|  1|leader us|      10000|
|  3|    mycat|      10000|
+---+---------+-----------+

My schema is :
root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- sharding_id: integer (nullable = false)

I want add a new column named newName. The new column is based on "name" and append "abc"
after it. My code looks like:

stud_scoreDF.withColumn("newName", stud_scoreDF.col("name") +  "abc"  ).show()
When I run the code, I got the reslult:
+---+---------+-----------+-------+
| id|     name|sharding_id|newName|
+---+---------+-----------+-------+
|  1|leader us|      10000|   null|
|  3|    mycat|      10000|   null|
+---+---------+-----------+-------+


I checked the code, the key code is  in arithmetic.scala. line 165.
It looks like:

override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = dataType match {
  case dt: DecimalType =>
    defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1.$$plus($eval2)")
  case ByteType | ShortType =>
    defineCodeGen(ctx, ev,
      (eval1, eval2) => s"(${ctx.javaType(dataType)})($eval1 $symbol $eval2)")
  case CalendarIntervalType =>
    defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1.add($eval2)")
  case _ =>
    defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1 $symbol $eval2")
}

My issue is:
Can we add case StringType in this class to support string append please?





1427357147@qq.com



-- 
Shmuel Blitz 
Big Data Developer 
Email: shmuel.blitz@similarweb.com 
www.similarweb.com 



-- 
Shmuel Blitz 
Big Data Developer 
Email: shmuel.blitz@similarweb.com 
www.similarweb.com 
Mime
View raw message