mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Qian Zhang <zhq527...@gmail.com>
Subject Re: Review Request 54639: Implemented the 'create()' method of OCI store.
Date Fri, 10 Feb 2017 15:04:18 GMT


> On Feb. 9, 2017, 7:36 a.m., Jie Yu wrote:
> > src/slave/containerizer/mesos/provisioner/oci/paths.hpp, lines 36-39
> > <https://reviews.apache.org/r/54639/diff/3/?file=1621023#file1621023line36>
> >
> >     Can you elaborate a bit more the layout here?
> >     
> >     so `layer_id` is the digest? What is `config.json`? Does it store decompressed
content, or the original content? what about image configs?
> >     
> >     Given this https://github.com/opencontainers/image-spec/blob/master/image-layout.md,
my hunch is that we should probably cache the original blob, as well as decompressed content,
potentially in separate subdirectories.
> >     
> >     ```
> >     <oci_store_dir>
> >        |--- blobs/  # this is origional CAS blobs.
> >                |--- sha256/
> >                        |--- <digest>
> >                        |--- <digest>
> >                        |--- ...
> >        |--- layers/ # this is decompressed filesystem change sets
> >                |--- <id? digest?>
> >                |--- ...
> >     ```
> >     
> >     Looks like we probably do not need `configs` or `manifests` directory for now
because the only media type is json, and can be directly read from `blobs/`.
> >     
> >     Another thing to consider is the conversion to overlayfs opaque file from whiteout
files as you did before. So for `layers` directory, we probably need to have a `backend` in
the path (e.g., `layers/overlayfs/...`).
> 
> Jie Yu wrote:
>     Also, worth taking a look at how Docker (or other OCI system) organize its cache.
> 
> Qian Zhang wrote:
>     Please take a look at https://reviews.apache.org/r/56147/diff/1#4, I have changed
it to:
>     ```
>          |--layers
>             |--<layer_id>
>                     |-- rootfs
>             |--<layer_id>
>                     |-- rootfs
>             |--<config.json>
>     ```
>     Here both `<layer_id>` and `<config.json>` are digest. `<layer_id>/rootfs/`
stores decompressed content(filesystem) extracted by `command::untar()` from the layer tar
ball. `<config.json>` stores the image configuration.
>     
>     Can you please let me know why we need to cache the original blobs? In my current
implementation, the orignal blobs will only be fetched by fetcher into a temp directory under
`staging` directory, `layers` directory will only store the decompressed content, and that
temp directory under `staging` directory will be removed after fetching image is done. I do
not think we need the original blobs once the fetching image is done and we have cached the
image in the store.
>     
>     And actually under `layers/<layer_id>`, there can be two sub-directories: `rootfs`
(for aufs/bind/copy) and `rootfs.overlay` (for overlay), this is same with what I did for
for Docker store. Do you think we need to change it to `layers/non-overlay/<layer_id>`
and `layers/overlay/<layer_id>` for OCI store?

And I checked Docker today, it does not cache blobs as well. When pulling an image, Docker
will download the blobs in a temp directory (`/var/lib/docker/tmp/`), and once the pulling
is done, the blobs in that temp directory will be removed. And Docker will store the decompressed
content of each image layer in `/var/lib/docker/aufs/diff/<layerID>`.


- Qian


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54639/#review164794
-----------------------------------------------------------


On Feb. 1, 2017, 9:17 a.m., Qian Zhang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54639/
> -----------------------------------------------------------
> 
> (Updated Feb. 1, 2017, 9:17 a.m.)
> 
> 
> Review request for mesos, Gilbert Song and Jie Yu.
> 
> 
> Bugs: MESOS-6681
>     https://issues.apache.org/jira/browse/MESOS-6681
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Implemented the 'create()' method of OCI store.
> 
> 
> Diffs
> -----
> 
>   src/CMakeLists.txt 09ef1aee680c6b91476092dbf61d4ca3a8d256bb 
>   src/Makefile.am 6c9be54fe0ce3faaa26e2f090773d1b77bb6e430 
>   src/slave/containerizer/mesos/provisioner/oci/paths.hpp PRE-CREATION 
>   src/slave/containerizer/mesos/provisioner/oci/paths.cpp PRE-CREATION 
>   src/slave/containerizer/mesos/provisioner/oci/store.cpp PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/54639/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Qian Zhang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message