spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From RK Aduri <rkad...@collectivei.com>
Subject Re: Breaking down text String into Array elements
Date Tue, 23 Aug 2016 20:20:36 GMT
That’s because of this:

scala> val text = Array((1,"hNjLJEgjxn"),(2,"lgryHkVlCN"),(3,"ukswqcanVC"),(4,"ZFULVxzAsv"),(5,"LNzOozHZPF"),(6,"KZPYXTqMkY"),(7,"DVjpOvVJTw"),(8,"LKRYrrLrLh"),(9,"acheneIPDM"),(10,"iGZTrKfXNr"))
text: Array[(Int, String)] = Array((1,hNjLJEgjxn), (2,lgryHkVlCN), (3,ukswqcanVC), (4,ZFULVxzAsv),
(5,LNzOozHZPF), (6,KZPYXTqMkY), (7,DVjpOvVJTw), (8,LKRYrrLrLh), (9,acheneIPDM), (10,iGZTrKfXNr))

scala> Array(text).getClass()
res1: Class[_ <: Array[Array[(Int, String)]]] = class [[Lscala.Tuple2;

scala> Array(text).length
res2: Int = 1

You see that Array(text) is basically a single element.


> On Aug 23, 2016, at 12:26 PM, Nick Pentreath <nick.pentreath@gmail.com> wrote:
> 
> 
> How about something like 
> 
> scala> val text = (1 to 10).map(i => (i.toString, random_string(chars.mkString(""),
10))).toArray
> 
> text: Array[(String, String)] = Array((1,FBECDoOoAC), (2,wvAyZsMZnt), (3,KgnwObOFEG),
(4,tAZPRodrgP), (5,uSgrqyZGuc), (6,ztrTmbkOhO), (7,qUbQsKtZWq), (8,JDokbiFzWy), (9,vNHgiHSuUM),
(10,CmnFjlHnHx))
> 
> scala> sc.parallelize(text).count
> res0: Long = 10
> 
> By the way not sure exactly why you need the udf registration here?
> 
> 
> On Tue, 23 Aug 2016 at 20:12 Mich Talebzadeh <mich.talebzadeh@gmail.com <mailto:mich.talebzadeh@gmail.com>>
wrote:
> Hi gents,
> 
> Well I was trying to see whether I can create an array of elements. From RDD to DF, register
as TempTable and store it  as a Hive table
> 
> import scala.util.Random
> //
> // UDF to create a random string of charlength characters
> //
> def random_string(chars: String, charlength: Int) : String = {
>   val newKey = (1 to charlength).map(
>     x =>
>     {
>       val index = Random.nextInt(chars.length)
>       chars(index)
>     }
>    ).mkString("")
>    return newKey
> }
> spark.udf.register("random_string", random_string(_:String, _:Int))
> case class columns (col1: Int, col2: String)
> val chars = ('a' to 'z') ++ ('A' to 'Z')
> var text = ""
> val comma = ","
> val terminator = "))"
> var random_char = ""
> for (i  <- 1 to 10) {
>     random_char = random_string(chars.mkString(""), 10)
> if (i < 10) {text = text + """(""" + i.toString + """,""""+random_char+"""")"""+comma}
>    else {text = text + """(""" + i.toString + """,""""+random_char+"""")"""}
> }
> println(text)
> val df = sc.parallelize((Array(text)))
> 
> 
> Unfortunately that only sees it as the text and interprets it as text.
> 
> I can write is easily as a shell script with ${text} passed to Array and it will work.
I was wondering if I could do this in Spark/Scala with my limited knowledge
> 
> Cheers
> 
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>  
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage
or destruction of data or any other property which may arise from relying on this email's
technical content is explicitly disclaimed. The author will in no case be liable for any monetary
damages arising from such loss, damage or destruction.
>  
> 
> On 23 August 2016 at 19:00, Nick Pentreath <nick.pentreath@gmail.com <mailto:nick.pentreath@gmail.com>>
wrote:
> what is "text"? i.e. what is the "val text = ..." definition?
> 
> If text is a String itself then indeed sc.parallelize(Array(text)) is doing the correct
thing in this case.
> 
> 
> On Tue, 23 Aug 2016 at 19:42 Mich Talebzadeh <mich.talebzadeh@gmail.com <mailto:mich.talebzadeh@gmail.com>>
wrote:
> I am sure someone know this :)
> 
> Created a dynamic text string which has format
> 
> scala> println(text)
> (1,"hNjLJEgjxn"),(2,"lgryHkVlCN"),(3,"ukswqcanVC"),(4,"ZFULVxzAsv"),(5,"LNzOozHZPF"),(6,"KZPYXTqMkY"),(7,"DVjpOvVJTw"),(8,"LKRYrrLrLh"),(9,"acheneIPDM"),(10,"iGZTrKfXNr")
> 
> now if I do
> 
> scala> val df = sc.parallelize((Array((1,"hNjLJEgjxn"),(2,"lgryHkVlCN"),(3,"ukswqcanVC"),(4,"ZFULVxzAsv"),(5,"LNzOozHZPF"),(6,"KZPYXTqMkY"),(7,"DVjpOvVJTw"),(8,"LKRYrrLrLh"),(9,"acheneIPDM"),(10,"iGZTrKfXNr"))))
> df: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[230] at parallelize
at <console>:39
> scala> df.count
> res157: Long = 10
> It shows ten Array elements, which is correct.
> 
> Now if I pass that text into Array it only sees one row
> 
> scala> val df = sc.parallelize((Array(text)))
> df: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[228] at parallelize at <console>:41
> scala> df.count
> res158: Long = 1
> 
> Basically it sees it as one element of array
> 
> scala> df.first
> res165: String = (1,"hNjLJEgjxn"),(2,"lgryHkVlCN"),(3,"ukswqcanVC"),(4,"ZFULVxzAsv"),(5,"LNzOozHZPF"),(6,"KZPYXTqMkY"),(7,"DVjpOvVJTw"),(8,"LKRYrrLrLh"),(9,"acheneIPDM"),(10,"iGZTrKfXNr")
> Which is not what I want.
> 
> Any ideas?
> 
> Thanks
> 
> 
> 
> 
> 
> 
> 
> 
> This works fine
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>  
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage
or destruction of data or any other property which may arise from relying on this email's
technical content is explicitly disclaimed. The author will in no case be liable for any monetary
damages arising from such loss, damage or destruction.
>  
> 


-- 
Collective[i] dramatically improves sales and marketing performance using 
technology, applications and a revolutionary network designed to provide 
next generation analytics and decision-support directly to business users. 
Our goal is to maximize human potential and minimize mistakes. In most 
cases, the results are astounding. We cannot, however, stop emails from 
sometimes being sent to the wrong person. If you are not the intended 
recipient, please notify us by replying to this email's sender and deleting 
it (and any attachments) permanently from your system. If you are, please 
respect the confidentiality of this communication's contents.

Mime
View raw message