mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrei Budnik <abud...@mesosphere.com>
Subject Re: Review Request 61530: Enabled retries for `killTasks` in docker executor.
Date Thu, 10 Aug 2017 16:14:55 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61530/
-----------------------------------------------------------

(Updated Aug. 10, 2017, 4:14 p.m.)


Review request for mesos and Alexander Rukletsov.


Bugs: MESOS-6743
    https://issues.apache.org/jira/browse/MESOS-6743


Repository: mesos


Description
-------

Previously, after `docker stop` command failure, docker executor
neither allowed a scheduler to retry `killTask` command, nor retried to
kill a task when task killing was triggered by a failed health check.


Diffs (updated)
-----

  src/docker/executor.cpp 26f12ec002f754fab0d34c01472cf95b499d8007 


Diff: https://reviews.apache.org/r/61530/diff/2/

Changes: https://reviews.apache.org/r/61530/diff/1-2/


Testing
-------

sudo make check

Manual testing:

Emulating `docker stop` errors:
===============================
1. Add `return fmt.Errorf("Emulating error!")` to https://github.com/docker/docker-ce/blob/master/components/engine/daemon/stop.go#L21
2. build docker from sources: http://oyvindsk.com/writing/docker-build-from-source
3. stop docker service and launch dockerd binary, like: `sudo ./bundles/17.06.0-dev/binary-daemon/dockerd`

Emulating docker daemon hang:
=============================
1. `ps aux|grep dockerd` - 2 processes will be found
2. `sudo kill -STOP <PID1> <PID2>` - send SIGSTOP to docker daemon processes just
before sending `docker stop`

Emulating health check failure in docker executor:
==================================================
1. Add
```c++
  static int fake = 0;
  if (++fake > 10) {
    failure();
    return;
  }
```
to `HealthChecker::processCheckResult()` in `src/checks/health_checker.cpp`
2. Add
```c++
       HealthCheck healthCheck;
       healthCheck.set_type(HealthCheck::COMMAND);
       healthCheck.mutable_command()->set_value("exit 0");
       healthCheck.set_delay_seconds(0);
       healthCheck.set_interval_seconds(0);
       healthCheck.set_grace_period_seconds(1);
       _task.mutable_health_check()->CopyFrom(healthCheck);
```
to `CommandScheduler::offers()` in `src/cli/execute.cpp`
3. compile mesos
4. run mesos agent: `sudo GLOG_v=1 ./bin/mesos-agent.sh --resources="cpus:10000;mem:1000000"
--work_dir='/home/some_user/mesos/build/var/agent-1' --containerizers="docker,mesos" --master="127.0.1.1:5050"`
5. launch docker executor: `./src/mesos-execute --master="127.0.1.1:5050" --name="a" --containerizer=docker
--docker_image="ubuntu:xenial" --command="while true; do : ; done"`


Thanks,

Andrei Budnik


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message