flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JP de Vooght ...@vooght.de>
Subject Re: WordCount with Flink Docker
Date Sun, 22 Apr 2018 13:14:36 GMT
Just to update that the messages of type "The operator name DataSource
() exceeded the 80 characters length limit and was truncated" occur for
other operations than print statements.

I am still new with Flink and am specifically interested in
print/logging statements. Using Wordcounter in batch with input and
output works well. But cannot see print entries anywhere...

So I took a different approach and submitted the following simpler job
using logging instead of printing to console:

|public class SimpleExample { private static final Logger LOG =
LoggerFactory.getLogger(ParallelMLExample.class); public static void
main(String[] args) throws Exception { final ExecutionEnvironment env;
env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<Integer>
amounts = env.fromElements(1, 29, 40, 50); LOG.info("########## BEFORE
UPDATEMODEL ##########"); List<Integer> collect = amounts.filter(a -> a
> 30).reduce((integer, t1) -> integer + t1).collect();
LOG.info("########## AFTER UPDATEMODEL ##########");
LOG.info(collect.get(0).toString()); } } |

The logs only show the first statement (BEFORE) - why?! I am failing to
understand how to use either print or logging reliably...

|jobmanager_1 | Starting Job Manager jobmanager_1 | config file:
jobmanager_1 | jobmanager.rpc.address: jobmanager jobmanager_1 |
jobmanager.rpc.port: 6123 jobmanager_1 | jobmanager.heap.mb: 1024
jobmanager_1 | taskmanager.heap.mb: 1024 jobmanager_1 |
taskmanager.numberOfTaskSlots: 1 jobmanager_1 |
taskmanager.memory.preallocate: false jobmanager_1 |
parallelism.default: 1 jobmanager_1 | web.port: 8081 jobmanager_1 |
blob.server.port: 6124 jobmanager_1 | query.server.port: 6125
jobmanager_1 | Starting jobmanager as a console application on host
b66ca9ea48e2. taskmanager_1 | Starting Task Manager taskmanager_1 |
config file: taskmanager_1 | jobmanager.rpc.address: jobmanager
taskmanager_1 | jobmanager.rpc.port: 6123 taskmanager_1 |
jobmanager.heap.mb: 1024 taskmanager_1 | taskmanager.heap.mb: 1024
taskmanager_1 | taskmanager.numberOfTaskSlots: 4 taskmanager_1 |
taskmanager.memory.preallocate: false taskmanager_1 |
parallelism.default: 1 taskmanager_1 | web.port: 8081 taskmanager_1 |
blob.server.port: 6124 taskmanager_1 | query.server.port: 6125
taskmanager_1 | Starting taskmanager as a console application on host
5f6223cd3e71. jobmanager_1 | Slf4jLogger started taskmanager_1 |
Slf4jLogger started jobmanager_1 | ########## BEFORE UPDATEMODEL
########## taskmanager_1 | The operator name DataSource (at
main(SimpleExample.java:30)
(org.apache.flink.api.java.io.CollectionInputFormat)) exceeded the 80
characters length limit and was truncated. |


Any advice?


On 04/22/2018 10:33 AM, JP de Vooght wrote:
>
> Thank you Ted!
>
> I still get the message below...
>
> taskmanager_1  | The operator name DataSource (at
> getDefaultTextLineDataSet(WordCountData.java:70
> (org.apache.flink.api.java.io
> <http://org.apache.flink.api.java.io>.CollectionInputFormat)) exceeded
> the 80 characters length limit and was truncated.
>
> The corresponding line 70 in Wordcount.java is
>
> System.out.println("Executing WordCount example with default input
> data set.");
>
> I got the docker image from
> https://github.com/docker-flink/docker-flink and still get this
> message even with all defaults.
>
>
> On 04/22/2018 12:39 AM, Ted Yu wrote:
>> bq. To enable Queryable State, please move the
>> flink-queryable-state-runtime jar from the opt to the lib folder.
>>
>> Have you noticed the above and taken corresponding action ?
>>
>> Thanks
>>
>> On Sat, Apr 21, 2018 at 3:15 PM, JP de Vooght <jp@vooght.de
>> <mailto:jp@vooght.de>> wrote:
>>
>>     Hello,
>>
>>     When I submit a WordCount job from the examples, without any
>>     parameter,
>>     to a standalone cluster running on Docker, I cannot see the
>>     System.out.println() message. Instead I notice the
>>     org.apache.flink.metrics.MetricGroup entry below (last)
>>
>>     jobmanager_1   | Starting Job Manager
>>     jobmanager_1   | config file:
>>     jobmanager_1   | jobmanager.rpc.address: jobmanager
>>     jobmanager_1   | jobmanager.rpc.port: 6123
>>     jobmanager_1   | jobmanager.heap.mb: 1024
>>     jobmanager_1   | taskmanager.heap.mb: 1024
>>     jobmanager_1   | taskmanager.numberOfTaskSlots: 1
>>     jobmanager_1   | taskmanager.memory.preallocate: false
>>     jobmanager_1   | parallelism.default: 1
>>     jobmanager_1   | web.port: 8081
>>     jobmanager_1   | blob.server.port: 6124
>>     jobmanager_1   | query.server.port: 6125
>>     jobmanager_1   | Starting jobmanager as a console application on host
>>     400baa8ff371.
>>     taskmanager_1  | Starting Task Manager
>>     taskmanager_1  | config file:
>>     taskmanager_1  | jobmanager.rpc.address: jobmanager
>>     taskmanager_1  | jobmanager.rpc.port: 6123
>>     taskmanager_1  | jobmanager.heap.mb: 1024
>>     taskmanager_1  | taskmanager.heap.mb: 1024
>>     taskmanager_1  | taskmanager.numberOfTaskSlots: 4
>>     taskmanager_1  | taskmanager.memory.preallocate: false
>>     taskmanager_1  | parallelism.default: 1
>>     taskmanager_1  | web.port: 8081
>>     taskmanager_1  | blob.server.port: 6124
>>     taskmanager_1  | query.server.port: 6125
>>     taskmanager_1  | Starting taskmanager as a console application on
>>     host
>>     531ef6c27264.
>>     jobmanager_1   | Slf4jLogger started
>>     taskmanager_1  | Slf4jLogger started
>>     taskmanager_1  | Could not load Queryable State Client Proxy.
>>     Probable
>>     reason: flink-queryable-state-runtime is not in the classpath. To
>>     enable
>>     Queryable State, please move the flink-queryable-state-runtime
>>     jar from
>>     the opt to the lib folder.
>>     taskmanager_1  | Could not load Queryable State Server. Probable
>>     reason:
>>     flink-queryable-state-runtime is not in the classpath. To enable
>>     Queryable State, please move the flink-queryable-state-runtime
>>     jar from
>>     the opt to the lib folder.
>>     taskmanager_1  | The operator name DataSource (at
>>     getDefaultTextLineDataSet(WordCountData.java:70)
>>     (org.apache.flink.api.java.io
>>     <http://org.apache.flink.api.java.io>.CollectionInputFormat))
>>     exceeded the 80
>>     characters length limit and was truncated.
>>
>>     I am not sure why this happens. I am using the latest Docker
>>     image with
>>     1.4.2 and the corresponding JAR from the 1.4.2 release of the
>>     examples.
>>
>>     TIA
>>
>>     JP
>>
>>
>


Mime
View raw message