flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kedar mhaswade <kedar.mhasw...@gmail.com>
Subject Re: Setting the parallelism in a cluster of machines properly
Date Thu, 26 Apr 2018 17:51:56 GMT
On Thu, Apr 26, 2018 at 10:47 AM, Makis Pap <makisntpap@gmail.com> wrote:

> OK Michael!
>
> I will look into it and will come back at you! Thanks for the help. I
> agree that it is quite suspicious the par = 8
>
> Jps? Meaning?
>
jps is a tool that comes with JDK (see $JAVA_HOME/bin). This is modeled
after the POSIX command ps (process status); jps = java ps => it shows you
the JVM's running on the given computer.


>
> Oh I should mention that the JobManager node is also a TaskManager.
>
> Best,
> Max
>
> On 27 Apr 2018, at 01:39, TechnoMage <mlatta@technomage.com> wrote:
>
> Check that you have slaves and masters set correctly on all machines, and
> in particular the one submitting jobs.  Make sure that from the machine
> submitting the job that it is talking to the correct job manager
> (jobmanager.rpc.address).  It really sounds like you are some how
> submitting jobs to only one taskmanager.
>
> You should also use jps to verify that you only have one jobmanager
> running and the worker machines only have taskmanager running.
>
> Michael
>
> On Apr 26, 2018, at 11:34 AM, Makis Pap <makisntpap@gmail.com> wrote:
>
> So what should the correct configs be then?
>
> I have set numOfSlotsPerTaskManager = 8. Which is reasonable as each has 8
> cpus.
>
> Best,
> Makis
>
> On Fri, 27 Apr 2018, 01:26 TechnoMage, <mlatta@technomage.com> wrote:
>
>> You need to verify your configs are correct.  Check that the local
>> machine sees all the task managers, that is the most likely reason it will
>> reject a higher parallelism.  I use a java program to submit to a 3 node 18
>> slot cluster without issue on a job with 18 parallelism.  I have not used
>> the command line to do this however.
>>
>> Michael
>>
>> > On Apr 26, 2018, at 11:16 AM, m@xi <makisntpap@gmail.com> wrote:
>> >
>> > No man. I have 17 TaskManagers and each has a number of 8 slots.
>> >
>> > Do you think it is better to have 8 TaskManager (1 slot each) ?
>> >
>> > Best,
>> > Max
>> >
>> >
>> >
>> > --
>> > Sent from: http://apache-flink-user-mailing-list-archive.2336050.
>> n4.nabble.com/
>>
>>
>
>

Mime
View raw message