samza-dev mailing list archives

Hey Jae,

I don't believe Samza supports this right now. We've always run in the default queue. This
is definitely a feature we should add. Would you be willing to try to patch your Samza library
to fix it?

The place to fix it is in ClientHelper.scala class. The submitApplicaiton() method has an
appCtx class. I believe you'll need to set the queue there using ApplicationSubmissionContext.setQueue.

The steps are:

1) Add a `yarn.queue` field to YarnConfig.
2) Update YarnJob.scala's submit() method to get the yarn queue, and use it in the client.submitApplication()
call.
3) Update ClientHelper.scala's submitApplication() method to take an optional queue, and if
it's set, add it to the ApplicationSubmissionContext, as I described above.

Cheers,
Chris

From: <Bae>, Jae Hyeon <metacret@gmail.com<mailto:metacret@gmail.com>>
Reply-To: "dev@samza.incubator.apache.org<mailto:dev@samza.incubator.apache.org>" <dev@samza.incubator.apache.org<mailto:dev@samza.incubator.apache.org>>
Date: Monday, December 8, 2014 10:15 AM
To: "dev@samza.incubator.apache.org<mailto:dev@samza.incubator.apache.org>" <dev@samza.incubator.apache.org<mailto:dev@samza.incubator.apache.org>>
Subject: How to set up yarn queue name

Hello

I tried submit Samza job in our EMR hadoop cluster but I got the following error:

[Inline image 1]

So, I asked our company hadoop admin and I got advised to set up the queue name. I added the
following line in Samza job property but it didn't work:

mapreduce.job.queuename=root.adhoc
yarn.scheduler.fair.user-as-default-queue=false

Do you know how to set up yarn queue name?

FYI, I copied our hadoop cluster fair scheduler policy. We disabled default queue.

<?xml version="1.0"?>
<allocations>
    <queue name="adhoc" type="parent">
        <weight>10</weight>
        <maxRunningApps>300</maxRunningApps>
        <queue name="nsharma">
           <weight>5</weight>
        </queue>
    </queue>

    <queue name="default">
        <weight>0.0</weight>
    </queue>

    <queue name="emergencyuseonly">
        <weight>0.0</weight>
        <minResources>0 mb, 0 vcores</minResources>
        <minSharePreemptionTimeout>10000</minSharePreemptionTimeout>
    </queue>

    <queuePlacementPolicy>
        <rule name="nestedUserQueue">
            <rule name="specified" create="false" />
        </rule>
        <rule name="specified" create="false" />
        <rule name="reject" />
    </queuePlacementPolicy>

    <userMaxAppsDefault>20</userMaxAppsDefault>
    <fairSharePreemptionTimeout>60</fairSharePreemptionTimeout>
</allocations>

Thank you
Best, Jae