spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "十六夜涙" <>
Subject spark broadcast unavailable
Date Tue, 09 Dec 2014 06:42:20 GMT
Hi all    In my spark application,I load a csv file and map the datas to a Map vairable for
later uses on driver node ,then broadcast it,every thing works fine untill the exception
occurs.the console log information shows me the broadcast unavailable,I googled this problem,says
spark will  clean up the broadcast,while these's an solution,the author mentioned about re-broadcast,I
followed this way,written some exception handle code with `try` ,`catch`.after compliling
and submitting the jar,I faced anthoner problem,It shows " task not serializable‍".‍‍‍
so here I have  there options:
1,get the right way persisting broadcast
2,solve the "task not serializable" problem re-broadcast variable
3,save the data to some kind of database,although I prefer save data in memory.

here is come code snippets:
  val esRdd = kafkaDStreams.flatMap(_.split("\\n"))
      case esregex(datetime, time_request) =>

        var ipInfo:Array[String]=Array.empty
            ipInfo = Utils.getIpInfo(client_ip,b.value)
          case =>{
            val b = Utils.load(sc,ip_lib_path)
            ipInfo = Utils.getIpInfo(client_ip,b.value)
View raw message