storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Serega Sheypak <serega.shey...@gmail.com>
Subject Re: JsonElement implements Serialization
Date Thu, 07 Jan 2016 16:42:28 GMT
My assumption is that parsing in the most expensive op in described case.
You can modify spout to make it parse raw json to java-bean and serde
java-bean to kryo.
Or you can introduce a bolt that accepts json as string and emits
kryo-serialized bean.

I don't understand why do you work with raw data inside your topology. What
is the goal? I always try to go away from row format of event to
domain-model, I use a set of java-beans representing my domain. It's easier
to maintain and develop.

2016-01-07 17:16 GMT+01:00 Kashyap Mhaisekar <kashyap.m@gmail.com>:

> The JsonElement is from Gson library  (com.google.gson.JsonElement.). The
> implementations of JsonElement abstract class are all final and
> non-serializable. Hence the only way of saving the contents is as a String
> (Json) and parsing it will give the JsonElement object. For us, the parsing
> the string to converting it to JsonElement was pretty slow at 30ms. We
> wanted to serialize/deserialize instead of parsing. The data representing
> Json will be a string again and even if I create a bean, i will need to
> parse the string.
>
> Thanks
> Kashyap
>
> On Thu, Jan 7, 2016 at 3:27 AM, Serega Sheypak <serega.sheypak@gmail.com>
> wrote:
>
>> Why do you serialize jsonelement? Can you parse it and create a java
>> -bean representing data stored as json?
>> What is inside your jsonelement?
>>
>> 2016-01-06 22:59 GMT+01:00 Kashyap Mhaisekar <kashyap.m@gmail.com>:
>>
>>> Here it is -
>>>
>>> This is a class that encapsulates JsonElement (Doesnt implement
>>> Serializable) -
>>> class MyPrimitive {
>>> *private JsonElement element;*
>>>
>>> public MyPrimitive(JsonElement element) {
>>> this.element = element;
>>> }
>>>
>>> public JsonElement getElement() {
>>> return element;
>>> }
>>>
>>> }
>>>
>>> Kryo getting created and called -
>>> Kryo kryo = new Kryo();
>>> kryo.setInstantiatorStrategy(new StdInstantiatorStrategy()); //had to do
>>> this as MyPrimitive does not implement //Serializable and has no default arg
>>>
>>> FieldSerializer<?> serializer = new FieldSerializer<MyFinal>(kryo,
>>> MyPrimitive.class);
>>> kryo.register(MyPrimitive.class, serializer);
>>>
>>> ByteArrayOutputStream stream = new ByteArrayOutputStream();
>>> Output output = new Output(stream);
>>> kryo.writeObject(output, myPrimitive);
>>>                          buffer = stream.toByteArray();
>>> output.close(); // Also calls output.flush()
>>>
>>> I now pass the buffer to another function as an argument and try to
>>> deserialize there.
>>>  Kryo kryo = new Kryo();
>>> kryo.setInstantiatorStrategy(new StdInstantiatorStrategy()); //had to do
>>> this as MyPrimitive does not implement //Serializable and has no default arg
>>> FieldSerializer<?> serializer = new
>>> FieldSerializer<MyFinal>(kryo,
>>> MyPrimitive.class);
>>> kryo.register(MyPrimitive.class, serializer);
>>> //
>>> * Input input = new Input(buffer);*
>>> MyPrimitive myFinal2 = kryo.readObject(input,
>>> MyPrimitive.class);
>>>
>>>
>>> The time taken for kryo.readObject takes anywhere from 30 to 150 ms with
>>> around 100ms being average.
>>>
>>> Thanks Serega.
>>>
>>> Kashyap
>>>
>>> On Wed, Jan 6, 2016 at 3:06 PM, Serega Sheypak <serega.sheypak@gmail.com
>>> > wrote:
>>>
>>>> What are you trying to serialize/deserialize?
>>>> You get out of json to java/groovy bean and serialize them using kryo.
>>>> Can you provide code snippet?
>>>>
>>>> 2016-01-06 21:09 GMT+01:00 Kashyap Mhaisekar <kashyap.m@gmail.com>:
>>>>
>>>>> It takes 100 ms to deserialize. So basically, serializing and
>>>>> deserializing JsonElement (Which has no no-args constructor and does
not
>>>>> implement Serializable) takes 100 ms approx for deserialization.
>>>>>
>>>>> Thanks
>>>>> Kashyap
>>>>>
>>>>> On Wed, Jan 6, 2016 at 1:41 PM, Serega Sheypak <
>>>>> serega.sheypak@gmail.com> wrote:
>>>>>
>>>>>> Sounds weird, Kryo widely used. What do you mean by "slow"?
>>>>>>
>>>>>> 2016-01-05 22:22 GMT+01:00 Kashyap Mhaisekar <kashyap.m@gmail.com>:
>>>>>>
>>>>>>> Serega,
>>>>>>> Kryo has helped, but it is slow. thanks for the pointer. Will
need
>>>>>>> to look into this more.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Kashyap
>>>>>>> On Jan 4, 2016 4:14 AM, "Serega Sheypak" <serega.sheypak@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Why not to switch to Kryo serialization or any other advanced
>>>>>>>> serializers? Express googling give nice example:
>>>>>>>> http://henning.kropponline.de/2015/09/27/storm-serialization-with-avro-using-kryo-serializer/
>>>>>>>>
>>>>>>>> 2016-01-04 5:29 GMT+01:00 Kashyap Mhaisekar <kashyap.m@gmail.com>:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> I was trying out gson in Storm and find that parsing
is slow for
>>>>>>>>> my use case. This is not related to Storm per se, but
I guess many of you
>>>>>>>>> may have experienced the use case below. If so, an interested
in finding
>>>>>>>>> out your approach.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Kashyap
>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>> From: "Kashyap Mhaisekar" <kashyap.m@gmail.com>
>>>>>>>>> Date: Jan 3, 2016 3:55 AM
>>>>>>>>> Subject: [google-gson:2550]: JsonElement implements Serialization
>>>>>>>>> To: "google-gson" <google-gson@googlegroups.com>
>>>>>>>>> Cc:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> Is there any way the JsonElement (Specifically JsonPrimitive)
can
>>>>>>>>> be made serializable? I plan to persist the JsonPrimitive
itself as an
>>>>>>>>> object in cache so that my further process is easier.
Am presently seeing
>>>>>>>>> that parsing strings each time is becoming expensive
in my use case.
>>>>>>>>>
>>>>>>>>> Did anyone try and succeed to Serialize a JsonElement
>>>>>>>>> (JsonPrimitive)?
>>>>>>>>>
>>>>>>>>> Please respond.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Kashyap
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed
to the Google
>>>>>>>>> Groups "google-gson" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails
from it,
>>>>>>>>> send an email to google-gson+unsubscribe@googlegroups.com.
>>>>>>>>> To post to this group, send email to google-gson@googlegroups.com.
>>>>>>>>> Visit this group at https://groups.google.com/group/google-gson.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message