flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: feature request: broadcast POJO objects as part of runtime context
Date Mon, 10 Nov 2014 18:32:46 GMT
I think Stefano was asking for a different way to pass a generic
Configuration object, not just a subclass of it.
For example, in our use case, it would be helpful to broadcast around
generic objects/pojos.
Is that possible?
On Nov 10, 2014 6:46 PM, "Stephan Ewen" <sewen@apache.org> wrote:

> Hi!
>
> I hope I understand correctly what you are trying to do: To have a config
> file available in the functions, you can simply do wither of the following:
>
> -----------------------
> Closure
> -----------------------
>
> Configuration conf = ...
>
> data.map(new RichMapFunction<String, Integer>() {
>
>   public void open (Conficuration c) {
>      // access the conf object here
>      conf.getString(...);
>   }
>
>   public Integer map(String value) {
>     // whatever
>   }
> });
> -----------------------
>
>
> -----------------------
> Config Parameters
> -----------------------
>
> Configuration conf = ...
>
> data.map(new RichMapFunction<String, Integer>() {
>
>   public void open (Conficuration c) {
>      // access the c - it will will have all elements of the conf - see
> withParameters() below
>      c.getString(...);
>   }
>
>   public Integer map(String value) {
>     // whatever
>   }
> })
> .withParameters(conf);
> -----------------------
>
>
> Stephan
>
>
> On Mon, Nov 10, 2014 at 5:36 PM, Stefano Bortoli <s.bortoli@gmail.com>
> wrote:
>
>> Hi,
>>
>> trying to run some legacy code as part of Flink Job, I had to replicate
>> configurations files across my cluster. Not a big deal with a small
>> cluster, but it would be nice to have these configuration objects
>> broadcast-able. Namely, it would be nice to reuse the old "read from conf
>> file" logic to build objects that then could be serialized and used along
>> the processing through the broadcast mechanism.
>>
>> Do you think it will be possible? with the new Kryo serialization it
>> should not be extremely complicated.
>>
>> saluti,
>> Stefano
>>
>>
>>
>

Mime
View raw message