Need to carry out a test in order to get information about the time required for pod startup if image streams from the user namespace are used in deployment configs created in *-che namespace

[ibuziuk]

This test can be carried out on any oso starter cluster and requires:

• creation of image steam in user namespace
• creation of deployment config in *-che namespace which would reference to the image stream from the previous step
• get info about how fast pod startup would be (there is an assumption that switching to image streams / deployment configs for workspace creation on openshift infrastructure might have better performance due to caching)

[amisevsk]

I've tested using image streams to launch Che 6 (centos_jdk8) and Che 7 workspaces:

Setup

ImageStreams can be created via e.g.

oc login <prod-preview account in *-che namespace>
oc import-image che7-che-dev --from eclipse/che-dev:nightly --confirm
oc set image-lookup che7-che-dev

This creates ImageStreams that can be used in place of normal image tags. Notably, they can be used with Kubernetes Deployments without making any changes (i.e. a DeploymentConfig is not necessary).

Che 7

Since many of the images used in Che 7 workspaces come from the plugin registry, to save time I manually updated a Che 7 deployment (obtained via oc export dc <workspace_id> to use ImageStreams. Since it's not started through the Che server, I couldn't get detailed timing data, but from the OpenShift events log, it seems that pulls took roughly 2 seconds each (similar to our current case)

Che 6

I tested 30 workspace starts, using quay.io/openshiftio/che-centos_jdk8 and an ImageStream pointing to the same image. For whatever reason, long pulls are much more common for the centos_jdk8 image than the Che 7 workspace images. There's a negligible difference between the cases at best, which I suspect is probably due to the ImageStream tagging a SHA hash rather than docker image tag (latest, I believe).

Note the area we're interested in is at the far left -- those are the fast pulls, which for Che 6 workspaces are in the 300-400 ms range. For the ImageStream case, I had 8/30 fast pull starts, compared to 15/30 for the quay.io image case. This difference (if caching is the ultimate cause as we suspect) is probably just because I ran the ImageStream case first.

Conclusion

There doens't seem to be any meaningful difference between the cases, so I would lean towards putting off further work here unless we get a more compelling reason to look into it, as maintaining ImageStreams in every user's namespace could be a fairly large burden (and I've had issues with ImageStreams and kubernetes-client in the past).

Same chart as above, but on a log scale:

[ibuziuk]

@amisevsk thanks for detailed document. The only thing that I am wondering is that @l0rd @jfchevrette were suggesting that image streams should be created in user (not che) namespace and used by deployments / deployment-configs created in che namespace. As I understand you created image streams and deployments both in the same *-che namespace, right ?

[jfchevrette]

My initial idea was to create the imagestreams in a single new namespace, such as che-images and then grant read access to the imagestreams from that namespace to all tenants. Creating ImageStreams under each tenant means we will end up with hundreds if not thousands of copies of the same images in the internal registry which could prove problematic with regards to storage and will not add much value.

[amisevsk]

@ibuziuk Yeah, these ImageStreams were created in the same namespace as the workspace (i.e. amisevsk-preview-che).

[ibuziuk]

@jfchevrette @l0rd during the tests that @amisevsk carried out it looks like there is no sufficient differences between usage of images vs imagestreams. Do you think we should continue investigation in this direction ?

@jfchevrette by the way does IMAGE PULLTHROUGH is enabled on starter clusters ?

[1] https://docs.openshift.com/container-platform/3.11/install_config/registry/extended_registry_configuration.html#middleware-repository-pullthrough

[l0rd]

@ibuziuk @amisevsk it doesn't look like there is any interest in going forward with the investigation on imagestreams
cc @gorkem

[ibuziuk]

Closing. Please, reopen if we should consider to continue investigation of potential switch to image streams