Improve operator logs by adding the possibility to get the full Ansible logs on it

[camilamacedo86]

Description of the change:

• Add full Ansible result output to the operator logs for Ansible based-operators configurable by EnvVar.

Motivation for the change:

Allow users to have the same full information that can be obtained until the version 0.15.x with the Ansible sidecar container in the operator logs.

Note that we deprecated the inotify-tools and we will no longer scaffold the sidecar container. See #2586. Also, we have been improving the operator logs in order to attend all needs. See: #2580 and #2321.

Local Test
Image: quay.io/camilamacedo86/ansible-operator:logs

See the output with the ENV VAR:

$ kubectl logs deployment.apps/testoperator  -n test
{"level":"info","ts":1582571464.3917449,"logger":"cmd","msg":"Go Version: go1.13.3"}
{"level":"info","ts":1582571464.39191,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1582571464.3919568,"logger":"cmd","msg":"Version of operator-sdk: v0.15.0+git"}
{"level":"info","ts":1582571464.392188,"logger":"cmd","msg":"Watching single namespace.","Namespace":"test"}
{"level":"info","ts":1582571464.753258,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1582571464.753604,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"WORKER_TESTCR_GLOTHRIEL_COM","default":1}
{"level":"info","ts":1582571464.7536469,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_TESTCR_GLOTHRIEL_COM","default":2}
{"level":"info","ts":1582571464.7537,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"glothriel.com","Options.Version":"v1alpha1","Options.Kind":"TestCR"}
{"level":"info","ts":1582571464.7538302,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1582571465.1339865,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1582571465.140328,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1582571465.8697152,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"testoperator-metrics","Service.Namespace":"test"}
{"level":"info","ts":1582571466.225284,"logger":"cmd","msg":"Could not create ServiceMonitor object","Namespace":"test","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1582571466.2253857,"logger":"cmd","msg":"Install prometheus-operator in your cluster to create ServiceMonitor objects","Namespace":"test","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1582571466.2270525,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":1582571466.2276838,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testcr-controller","source":"kind source: glothriel.com/v1alpha1, Kind=TestCR"}
{"level":"info","ts":1582571466.2280617,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"testcr-controller"}
{"level":"info","ts":1582571466.230627,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1582571466.3331325,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"testcr-controller","worker count":1}

--------------------------- Ansible Task StdOut -------------------------------

{"level":"info","ts":1582571469.3538122,"logger":"logging_event_handler","msg":"[playbook task]","name":"example-testcr","namespace":"test","gvk":"glothriel.com/v1alpha1, Kind=TestCR","event_type":"playbook_on_task_start","job":"6129484611666145821","EventData.Name":"Gathering Facts"}
TASK [Gathering Facts] *********************************************************

-------------------------------------------------------------------------------
{"level":"info","ts":1582571470.7043462,"logger":"logging_event_handler","msg":"[playbook debug]","name":"example-testcr","namespace":"test","gvk":"glothriel.com/v1alpha1, Kind=TestCR","event_type":"runner_on_ok","job":"6129484611666145821","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [task A] ******************************** 
ok: [localhost] => {
    "msg": "Watch task!"
}

-------------------------------------------------------------------------------
{"level":"info","ts":1582571470.9242826,"logger":"runner","msg":"Ansible-runner exited successfully","job":"6129484611666145821","name":"example-testcr","namespace":"test"}

--------------------------- Ansible Task Status Event StdOut  -----------------

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


-------------------------------------------------------------------------------

--------------------------- Ansible Debug Result -----------------------------
ansible-playbook 2.9.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file

PLAYBOOK: 21675fce3ecd475c959ce73f9b02b655 *************************************
1 plays in /tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/test/example-testcr/project/21675fce3ecd475c959ce73f9b02b655

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]
META: ran handlers

TASK [testcr : task A] *********************************************************
task path: /opt/ansible/roles/testcr/tasks/main.yml:4
ok: [localhost] => {
    "msg": "Watch task!"
}
META: ran handlers
META: ran handlers
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   



-------------------------------------------------------------------------------

[fabianvf]

I don't think I'd add this argument to the watches.yaml, I'd make it a cli flag/environment variable, since I assume this is something users would want to set after their operator is deployed in the cluster rather than at container build time

[camilamacedo86]

Make sense. I will change it.

[fabianvf]

s/integer/bool/

[camilamacedo86]

Ops. Tks for the good catch. Solved.

[fabianvf]

/lgtm

[jmrodri]

/lgtm