flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: Exception while running Flink jobs (1.0.0)
Date Mon, 03 Oct 2016 20:26:04 GMT
I think you're running into the same exception I face sometimes..I've
opened a jira for it [1]. Could you please try to apply that patch and see
if things get better?

https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-4719

Best,
Flavio

On 3 Oct 2016 22:09, "Tarandeep Singh" <tarandeep@gmail.com> wrote:

> Now, when I ran it again (with lower task slots per machine) I got a
> different error-
>
> org.apache.flink.client.program.ProgramInvocationException: The program
> execution failed: Job execution failed.
>     at org.apache.flink.client.program.Client.runBlocking(Client.java:381)
>     at org.apache.flink.client.program.Client.runBlocking(Client.java:355)
>     at org.apache.flink.client.program.Client.runBlocking(Client.java:315)
>     at org.apache.flink.client.program.ContextEnvironment.
> execute(ContextEnvironment.java:60)
>     at org.apache.flink.api.java.ExecutionEnvironment.execute(
> ExecutionEnvironment.java:855)
>     ....
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.flink.client.program.PackagedProgram.callMainMethod(
> PackagedProgram.java:505)
>     at org.apache.flink.client.program.PackagedProgram.
> invokeInteractiveModeForExecution(PackagedProgram.java:403)
>     at org.apache.flink.client.program.Client.runBlocking(Client.java:248)
>     at org.apache.flink.client.CliFrontend.executeProgramBlocking(
> CliFrontend.java:866)
>     at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333)
>     at org.apache.flink.client.CliFrontend.parseParameters(
> CliFrontend.java:1189)
>     at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
>     at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$
> mcV$sp(JobManager.scala:714)
>     at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:660)
>     at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:660)
>     at scala.concurrent.impl.Future$PromiseCompletingRunnable.
> liftedTree1$1(Future.scala:24)
>     at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(
> Future.scala:24)
>     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(
> AbstractDispatcher.scala:401)
>     at scala.concurrent.forkjoin.ForkJoinTask.doExec(
> ForkJoinTask.java:260)
>     at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> pollAndExecAll(ForkJoinPool.java:1253)
>     at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> runTask(ForkJoinPool.java:1346)
>     at scala.concurrent.forkjoin.ForkJoinPool.runWorker(
> ForkJoinPool.java:1979)
>     at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:107)
> Caused by: com.esotericsoftware.kryo.KryoException: Unable to find class:
> javaec40-d994-yteBuffer
>     at com.esotericsoftware.kryo.util.DefaultClassResolver.
> readName(DefaultClassResolver.java:138)
>     at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(
> DefaultClassResolver.java:115)
>     at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>     at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>     at org.apache.flink.api.java.typeutils.runtime.kryo.
> KryoSerializer.deserialize(KryoSerializer.java:228)
>     at org.apache.flink.api.java.typeutils.runtime.
> PojoSerializer.deserialize(PojoSerializer.java:431)
>     at org.apache.flink.runtime.plugable.NonReusingDeserializationDeleg
> ate.read(NonReusingDeserializationDelegate.java:55)
>     at org.apache.flink.runtime.io.network.api.serialization.
> SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(
> SpillingAdaptiveSpanningRecordDeserializer.java:124)
>     at org.apache.flink.runtime.io.network.api.reader.
> AbstractRecordReader.getNextRecord(AbstractRecordReader.java:65)
>     at org.apache.flink.runtime.io.network.api.reader.
> MutableRecordReader.next(MutableRecordReader.java:34)
>     at org.apache.flink.runtime.operators.util.ReaderIterator.
> next(ReaderIterator.java:73)
>     at org.apache.flink.runtime.operators.FlatMapDriver.run(
> FlatMapDriver.java:101)
>     at org.apache.flink.runtime.operators.BatchTask.run(
> BatchTask.java:480)
>     at org.apache.flink.runtime.operators.BatchTask.invoke(
> BatchTask.java:345)
>     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
>     at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: javaec40-d994-yteBuffer
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:348)
>     at com.esotericsoftware.kryo.util.DefaultClassResolver.
> readName(DefaultClassResolver.java:136)
>     ... 15 more
>
>
> -Tarandeep
>
> On Mon, Oct 3, 2016 at 12:49 PM, Tarandeep Singh <tarandeep@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am using flink-1.0.0 and running ETL (batch) jobs on it for quite some
>> time (few months) without any problem. Starting this morning, I have been
>> getting errors like these-
>>
>> "Received an event in channel 3 while still having data from a record.
>> This indicates broken serialization logic. If you are using custom
>> serialization code (Writable or Value types), check their serialization
>> routines. In the case of Kryo, check the respective Kryo serializer."
>>
>> My datasets are in Avro format. The only thing that changed today is - I
>> moved to smaller cluster. When I first ran the ETL jobs, they failed with
>> this error-
>>
>> "Insufficient number of network buffers: required 20, but only 10
>> available. The total number of network buffers is currently set to 20000.
>> You can increase this number by setting the configuration key
>> 'taskmanager.network.numberOfBuffers'"
>>
>> I increased number of buffers to 30k. Also decreased number of slots per
>> machine as I noticed load per machine was too high. After that, when I
>> restart the jobs, I am getting the above error.
>>
>> Can someone please help me debug it?
>>
>> Thank you,
>> Tarandeep
>>
>
>

Mime
View raw message