flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Wordindex conversation.
Date Mon, 10 Oct 2016 22:11:54 GMT
Hi,

you can do it like this:

1) you have to split each label record of the main dataset into separate
records:

(0,List(a, b, c, d, e, f, g)) -> (0, a), (0, b), (0, c), ..., (0, g)
(1,List(b, c, f, a, g)) -> (1, b), (1, c), ..., (1, g)

2) join word index dataset with splitted main dataset:

DataSet<Tuple2<Integer, String>> splittedMain = ...
DataSet<Tuple2<Long, String>> wordIdx = ...

DataSet<Integer, Long> joined =
splittedMain.join(wordIdx).where(1).equalTo(1).with(...)

3) Group by Label:

DataSet<Tuple2<Integer, Long[]>> labelsWithIdx =
joined.groupBy(0).reduceGroup(...) // collect all indexes in list / array

Best, Fabian



2016-10-10 23:49 GMT+02:00 Kürşat Kurt <kursat@kursatkurt.com>:

> Hi;
>
>
>
> I have MainDataset (Label,WordList) :
>
>
>
> (0,List(a, b, c, d, e, f, g))
>
> (1,List(b, c, f, a, g))
>
>
>
> ..and, wordIndex dataset(created with .zipWithIndex) :
>
>
>
> wordIndex> (0,a)
>
> wordIndex> (1,b)
>
> wordIndex> (2,c)
>
> wordIndex> (3,d)
>
> wordIndex> (4,e)
>
> wordIndex> (5,f)
>
> wordIndex> (6,g)
>
>
>
> How can i convert mainDataset to indexed wordList dataset like this:
>
> (0,List(1,2,3,4,5,6))
>
> (1,List(2,3,5,0,6)
>
>
>

Mime
View raw message