Make Coverage report consistent

[dibyom]

Expected Behavior

Coverage job should consistently produce a report for all PRs.

Actual Behavior

When it works, it works. But sometimes, the job passes but it does not actually produce a report that we can view e.g. tektoncd/pipeline#1757

Steps to Reproduce the Problem

This seems to be happening intermittently

Additional Info

• Coverage job might be using a different golang version
• Maybe we can use coveralls.io as an alternative instead?

[ghost]

I'm looking into an incorrect coverage report on tektoncd/pipeline#2233. The complete build log is here: https://storage.googleapis.com/tekton-prow/pr-logs/pull/tektoncd_pipeline/2233/pull-tekton-pipeline-go-coverage/1241017694992470016/build-log.txt

Here's what I've observed so far:

1. go test -coverprofile complains about lots of syntax issues, seemingly related more to the version of Go than anything else. Example snippet:

2020/03/20 15:06:18 Error running 'go test -coverprofile ': error='exit status 2'; combined output='# github.com/tektoncd/pipeline/vendor/knative.dev/pkg/reconciler
vendor/knative.dev/pkg/reconciler/events.go:35:15: undefined: errors.Is
vendor/knative.dev/pkg/reconciler/events.go:51:15: undefined: errors.As
vendor/knative.dev/pkg/reconciler/events.go:86:9: undefined: errors.Is
# github.com/tektoncd/pipeline/vendor/github.com/docker/cli/cli/config
vendor/github.com/docker/cli/cli/config/config.go:109:18: undefined: os.UserHomeDir
# github.com/tektoncd/pipeline/vendor/github.com/google/go-containerregistry/pkg/logs
vendor/github.com/google/go-containerregistry/pkg/logs/logs.go:38:10: l.Writer undefined (type *log.Logger has no field or method Writer)
# github.com/tektoncd/pipeline/cmd/entrypoint
cmd/entrypoint/waiter.go:35: Errorf format %w has unknown verb w
# github.com/tektoncd/pipeline/vendor/github.com/jenkins-x/go-scm/scm
vendor/github.com/jenkins-x/go-scm/scm/issue.go:146:9: undefined: strings.ReplaceAll

2. The set of "concerned" files is confusing. The number of changed files in the PR is 10. The number of files reported by github also looks like 10. But the set of files reported by the tool is as follows:

2020/03/20 15:06:18 GetConcernedFiles(...) started
2020/03/20 15:06:19 github file #0: docs/taskruns.md, concerned=true
2020/03/20 15:06:19 github file #1: pkg/apis/pipeline/v1beta1/task_types.go, concerned=true
2020/03/20 15:06:19 github file #2: pkg/reconciler/taskrun/taskrun.go, concerned=true
2020/03/20 15:06:19 github file #3: pkg/reconciler/taskrun/taskrun.go, concerned=true
2020/03/20 15:06:19 github file #4: pkg/remote/oci.go, concerned=true
2020/03/20 15:06:19 github file #5: pkg/remote/oci.go, concerned=true
2020/03/20 15:06:19 github file #6: pkg/remote/resolver.go, concerned=true
2020/03/20 15:06:19 github file #7: test/builder/task.go, concerned=true
2020/03/20 15:06:19 github file #8: test/builder/task.go, concerned=true
2020/03/20 15:06:19 github file #9: test/remote.go, concerned=true
2020/03/20 15:06:19 GetConcernedFiles(...) completed

There are 10 of them but notice the duplicates, for example test/builder/task.go. Looking back at the PR it's clear to see where these dupes come from - they're files that have an _test.go file associated with them. It looks like this is intentional, but it's confusing none the less.

3. Right up until the very end of the coverage tool's log it appears to understand that there are 7 files needing coverage. But then it mysteriously decides to drop 6 of them. See this snippet of log:

7 files changed, reported by github:
docs/taskruns.md	pkg/apis/pipeline/v1beta1/task_types.go	pkg/reconciler/taskrun/taskrun.go	pkg/remote/oci.go	pkg/remote/resolver.go	test/builder/task.go	test/remote.go	

2020/03/20 15:06:20 
Finding joining set of changed files from profile[count=1] & github
checking if this file is in github change list: test/builder/task.go	Yes!	(Coverage low!)
End of Finding joining set of changed files from profile & github

This happens as it's building the report for the Github message. Appears to be that NewGroupChanges is dropping them based on some difference it infers in a "base list" and a "new list" but it's not clear why. I'm still trying to puzzle out what profile is, why it only has 1 file and why it has an influence on the coverage report.

[pritidesai]

Recently happened in one more Pipeline PR # 2909.

The stats were produced for just 2 out of 13 go files:

File	Old Coverage	New Coverage	Delta
pkg/pod/workingdir_init.go	94.4%	92.9%	-1.6
pkg/reconciler/pipeline/dag/dag.go	98.9%	98.8%	-0.1

tektoncd/pipeline#2909 (comment)

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jerop]

I've seen this happening again - slack post - so will reopen

/remove-lifecycle rotten
/reopen

[tekton-robot]

@jerop: Reopened this issue.

In response to this:

I've seen this happening again - slack post - so will reopen

/remove-lifecycle rotten
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[pritidesai]

/remove-lifecycle rotten

This is still not fixed, noticed in a recent PR tektoncd/pipeline#3700 (comment)

PR #3700 is updating four files but includes coverage report for just one file (pkg/workspace/apply.go):

1. task_types.go
2. workspace_types.go
3. pkg/reconciler/taskrun/resources/apply.go
4. pkg/workspace/apply.go

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.