Images versions and pull policy

[pavolloffay]

Purpose of this repository

First, let's define what we expect this repository should provide. I think It should provide the easiest way how to deploy Jaeger on OpenShift with a minimal possible configuration. I think people will anyway modify these templates to fit their workflow and preferred configuration (e.g. pull policy) when using in production.

We have to decide what versions of images we are going to use. Latest vs tagged and what pull policy.

Image versions

Latest

I think this can be risky. We cannot ensure that template will always work.
Possible breaking changes:

• added new service (e.g. for dependency graph)
• service split (e.g. query and UI)
• added new ports (e.g. REST endpoint, new collector)

Tagged version

When using a tagged version there shouldn't be any cases when template won't be compatible with used images. This has also the advantage that images get updates.

SHA

This is like tagged version but without updates for underlying images.

Pull policy

Always

This ensures that images will be updated (e.g. bug or security fixes).

IfNotPresent

Gives the administrator an option to choose the right image (e.g. run some tests before updating images).

Summary

I think the best is to use proper tag version with Always pull policy. This provides the safest out of the box solution. Currently, there is no proper version of Jaeger then I propose to use SHAs as an intermediate solution.

Useful links:

https://kubernetes.io/docs/concepts/configuration/overview/#container-images
https://kubernetes.io/docs/concepts/containers/images/
Email from #8 (comment) http://post-office.corp.redhat.com/archives/obsidian-team-list/2017-May/msg00333.html

[pavolloffay]

cc @jpkrohling @objectiser @yurishkuro

[jpkrohling]

Thanks for the write this down, I think it sums up the options quite well.

To me, the only two real options for versions are latest and a tagged version. SHA is ambiguous (abc123 is the Docker image ID, or the git sha for Jaeger? Or something else?). Besides, SHAs are bad for humans, as we can't tell which one is newer. The only good thing about SHAs is that it uniquely identifies a revision and that's very useful on places where you don't need versions (continuous delivery). In all other cases, you'll always find a "tag" on top of a SHA (git tag, docker image tag, ...).

All in all, I'd leave the version as latest for now, as we don't have a Jaeger version yet. I'm repeating myself, but we don't have a problem to solve yet. Once we have a version, or once we start breaking things, then we should consider solving the problem at hand.

About the pull policy: I'd say we should leave it out of the template and let the default Kubernetes behavior kick in. Let's not deviate from the default behavior without a really good reason here. From the documentation:

The default container image pull policy is IfNotPresent, which causes the Kubelet to not pull an image if it already exists. If you would like to always force a pull, you must specify a pull image policy of Always in your .yaml file (imagePullPolicy: Always) or specify a :latest tag on your image.

Setting a non-latest tag + Always deviates from the default Kubernetes behavior.

My vote is for a version parameter (as we have already) defaulting to latest, and no explicit pull policy.

[yurishkuro]

How do tagged versions help with always pull? The tags will be semver, so if we discover and fix a bug, at minimum it will be released as new patch version. Do these templates understand semver ranges like ~0.5 that automatically picks there latest from 0.5.1, 0.5.2, ...
?

[objectiser]

Doesn't look like kubernetes or openshift understands ranges.

We have been referring to these templates as 'production' versions - and as stated https://kubernetes.io/docs/concepts/configuration/overview/#container-images use of 'latest' in production is not a good idea. So once the server is tagged we should use that version.

In terms of the pull policy - as ranges are not supported, looks like we have two options - either update the version in the templates whenever there is a new release (but that means the templates would need to be re-applied), or use a two level tagging approach: major.minor.micro - to precisely tag a particular version, but also a v{major} tag which moves with each subsequent release of the same major version.

This means that the templates could be setup specifically for a major version, which should be safe from breaking changes, but also means the "Always" pull policy will be required (unfortunately) - which is a pity, but until they provide a range mechanism, looks like the only option.

[pavolloffay]

How do tagged versions help with always pull?

Image version is not the same as underlying bits. Eg. for jaeger release 1.0.0 we create docker image v1.0 and if we release jaeger 1.0.1 we create image 1.0.1 and update v1.0. If pull policy is always pod using that image will pick fixed image on restart (template stays the same). So maintaing v1.0 makes sense if you are are pointing to the one version which gets security and bug fixes and is not breaking change.

[jpkrohling]

As @pavolloffay mentioned, the image version is not the same as the underlying bits. Another reason for that is in case of a security issue on a layer beneath our own, so, we want to have the ability to have v1.0 to point to the same Jaeger version, but different SHA build (which might just be a rebuild).

[yurishkuro]

The backend has been tagged with v0.5.0, but the Docker images built from master don't seem to pick that tag, I only see commit SHAs: https://hub.docker.com/r/jaegertracing/jaeger-collector/tags/

[pavolloffay]

@yurishkuro awesome!! There is probably a bug in https://github.com/uber/jaeger/blob/master/travis/build-docker-images.sh or upload script. Hmm upload script could take argumets to make it cleaner.

Could you please upload it manually for this time?

[yurishkuro]

build-docker-images exists when it's not on the master branch, but tagged release runs with TRAVIS_BRANCH set to the tag number.

# Only push the docker container to Docker Hub for master branch
if [[ "$BRANCH" == "master" && "$TRAVIS_SECURE_ENV_VARS" == "true" ]]; then echo 'upload to Docker Hub'; else echo 'skip docker upload for PR'; exit 0; fi

[yurishkuro]

do we want to keep pushing images for every commit to master, or only for tagged commits?

[yurishkuro]

Made a fix https://github.com/uber/jaeger/pull/246, once it's merged will release 0.5.1 to test.

[yurishkuro]

v0.5.1 released. The images are tagged with 0.5.0, 0.5, and 0

[objectiser]

Great job!

[pavolloffay]

👍

do we want to keep pushing images for every commit to master, or only for tagged commits?

I don't think it's necessary to push an image for every commit. I haven't seen it in other projects, it pollutes the hub. Rather do proper releases more often :)

[yurishkuro]

the all-in-one and crossdock images are still published on every commit to master

[pavolloffay]

ignore