spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lonely Feb <lonely8...@gmail.com>
Subject Re: broadcast hang out
Date Mon, 16 Mar 2015 03:45:07 GMT
yes

2015-03-16 11:43 GMT+08:00 Mridul Muralidharan <mridul@gmail.com>:

> Cross region as in different data centers ?
>
> - Mridul
>
> On Sun, Mar 15, 2015 at 8:08 PM, lonely Feb <lonely8658@gmail.com> wrote:
> > Hi all, i meet up with a problem that torrent broadcast hang out in my
> > spark cluster (1.2, standalone) , particularly serious when driver and
> > executors are cross-region. when i read the code of broadcast i found
> that
> > a sync block read here:
> >
> >   def fetchBlockSync(host: String, port: Int, execId: String, blockId:
> > String): ManagedBuffer = {
> >     // A monitor for the thread to wait on.
> >     val result = Promise[ManagedBuffer]()
> >     fetchBlocks(host, port, execId, Array(blockId),
> >       new BlockFetchingListener {
> >         override def onBlockFetchFailure(blockId: String, exception:
> > Throwable): Unit = {
> >           result.failure(exception)
> >         }
> >         override def onBlockFetchSuccess(blockId: String, data:
> > ManagedBuffer): Unit = {
> >           val ret = ByteBuffer.allocate(data.size.toInt)
> >           ret.put(data.nioByteBuffer())
> >           ret.flip()
> >           result.success(new NioManagedBuffer(ret))
> >         }
> >       })
> >
> >     Await.result(result.future, Duration.Inf)
> >   }
> >
> > it seems that fetchBlockSync method does not have a timeout limit but
> wait
> > forever ? Anybody can show me how to control the timeout here?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message