Running jupyter notebooks in public dashboard in automatic mode

[swastis10]

@shankari
I went through the entire project and realized that we are using Crontab.

In docker-compose.dev.yml, CRON_MODE= is empty by default, manually opening the notebooks. We can make changes in the notebook parameters manually.
If I set, CRON_MODE=TRUE it will launch the container in cron mode.
In the crontab file, I set */2 * * * * python bin/generate_plots.py mode_purpose_share.ipynb default &> /var/log/log_file.log which will recalculate the files every 2 mins.

To test if the notebook is recalculated every 2 mins, I did docker-compose -f docker-compose.dev.yml up --build with CRON_MODE=TRUE but nothing happens.

Debugging steps:

1. run docker exec -ti em-public-dashboard-notebook-server-1 bash
2. run source setup/activate.sh && conda env update --name emission --file setup/environment36.notebook.additions.yml to download all pip dependencies.
3. run /usr/src/app/.docker/start_notebook.sh. The logs for the same are:

(emission) root@24ae37d05a34:/usr/src/app# /usr/src/app/.docker/start_notebook.sh
/usr/src/app/e-mission-server /usr/src/app
DB host = db
/usr/src/app
{
    "timeseries": {
        "url": "db",
        "result_limit": 250000
    }
}
Web host = 0.0.0.0
/usr/src/app/e-mission-server /usr/src/app
/usr/src/app/e-mission-server
# conda environments:
#
base                     /root/miniconda-4.12.0
emission              *  /root/miniconda-4.12.0/envs/emission

/usr/src/app
Running crontab without user interaction, setting python path

which means that crontab is enabled.
4. Now, we want to see the crontab logs: The code generates crontab logs at devcron ../crontab >> /var/log/cron.console.stdinout 2>&1 so, I run vim /var/log/cron.console.stdinout and see the following errors:

Traceback (most recent call last):
  File "/root/miniconda-4.12.0/envs/emission/bin/devcron", line 8, in <module>
    sys.exit(main())
  File "/root/miniconda-4.12.0/envs/emission/bin/devcron.py", line 47, in main
    cron.run()
  File "/root/miniconda-4.12.0/envs/emission/bin/devcron.py", line 259, in run
    time.sleep(dt)
KeyboardInterrupt
Traceback (most recent call last):
  File "/root/miniconda-4.12.0/envs/emission/bin/devcron", line 8, in <module>
    sys.exit(main())
  File "/root/miniconda-4.12.0/envs/emission/bin/devcron.py", line 39, in main
    crontab_data = open(args[0]).read()
FileNotFoundError: [Errno 2] No such file or directory: 'crontab'

[shankari]

In the crontab file, I set */2 * * * * python bin/generate_plots.py mode_purpose_share.ipynb default &> /var/log/log_file.log which will recalculate the files every 2 mins.

In which crontab file? the error seems to suggest that there is no crontab file present

FileNotFoundError: [Errno 2] No such file or directory: 'crontab'

[shankari]

Closing this since @swastis10 has "graduated"