incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: GC overhead limit exceeded
Date Thu, 18 Oct 2012 07:10:23 GMT
On Thu, Oct 18, 2012 at 8:16 AM, 杨定裕 <yangdingyu@gmail.com> wrote:

> Hi, all
> When My adapter send large data to APP, adapter app occurs a error like
> this:
> Maybe the memory is limited and  the following processing is slow?
>

you´d have to be more specific about your app. What is the approximate size
of your messages? What is the available memory in the JVM? How many
messages are you creating per second and per adapter node?
Note that the culprit may not be the serialization but could also be
intermediate objects created by Netty, the comm layer library. It will be
useful to get more feedback on your issue.

>
> How can configure file in a real cluster.
> I found that it is complex in a 20 nodes to create the topology.
>

I´m sure sure what complexity you are referring to. To use 20 nodes, you
just need to define a logical cluster with 20 partitions, and start 20 S4
nodes that point to that cluster configuration (Zookeeper ensemble +
cluster name).


>
> And How can i remove a application in a cluster?
>

Currently you simply clean up zookeeper and kill the S4 nodes. (We plan to
add a more convenient way, like a command you issue to Zookeeper).

>
> I saw s4-0.3 has a configure file "clusters.xml", but s-0.5 has not.
>

This is not needed. In S4 0.5, you define a minimal number of parameter for
the cluster (number of partitions, name) and you start S4 nodes
independently.


Matthieu


> -----error-------------
> Oct 18, 2012 1:50:04 PM
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink
> WARNING: Failed to accept a connection.
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>         at java.util.HashMap.newKeyIterator(HashMap.java:840)
>         at java.util.HashMap$KeySet.iterator(HashMap.java:874)
>         at java.util.HashSet.iterator(HashSet.java:153)
>         at
> sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>         at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:240)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Exception in thread "Thread-7" java.lang.OutOfMemoryError: GC overhead
> limit exceeded
>         at java.lang.reflect.Array.get(Native Method)
>         at
> com.esotericsoftware.kryo.serialize.ArraySerializer.writeArray(ArraySerializer.java:110)
>         at
> com.esotericsoftware.kryo.serialize.ArraySerializer.writeObjectData(ArraySerializer.java:88)
>         at
> com.esotericsoftware.kryo.Serializer.writeObject(Serializer.java:43)
>         at
> com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:182)
>         at
> com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:489)
>         at
> com.esotericsoftware.kryo.ObjectBuffer.writeClassAndObject(ObjectBuffer.java:230)
>         at
> org.apache.s4.comm.serialize.KryoSerDeser.serialize(KryoSerDeser.java:90)
>         at org.apache.s4.comm.tcp.TCPEmitter.send(TCPEmitter.java:178)
>         at org.apache.s4.core.RemoteSender.send(RemoteSender.java:44)
>         at org.apache.s4.core.RemoteSenders.send(RemoteSenders.java:81)
>         at org.apache.s4.core.RemoteStream.put(RemoteStream.java:74)
>         at OLAAdapter.Adapter$Dequeuer.run(Adapter.java:64)
>         at java.lang.Thread.run(Thread.java:619)
>
>

Mime
View raw message