Add docker-related files

[atton16]

Hi, I have added the docker-related files here. Please also checkout docker/README.md for how to set it up and running.

[AmplabJenkins]

Can one of the admins verify this patch?

[shankari]

My main question is about mongodb :)
I will test this tomorrow PDT before merging, while waiting for an answer about mongodb...

@brianmickel since you were so excited about the docker container, can you test the script too before I merge?

@atton16 thank you so much for making the onboarding (e-mission/e-mission-docs#28) easier.

This is a partial fix for e-mission/e-mission-docs#26

[shankari]

This is pretty cool. I assume I leave out this and the next two lines if I just want to start the container with defaults?
Because for development, people may want to just start the app and not want to figure out how to set config files.

[shankari]

You probably want to have development versus production/deployment scripts anyway because the port mapping will be different. Development maps 8080:8080 but I would assume production should do 443:443. And production will want to configure SSL certificates as well.

[atton16]

For production, we are using another script that runs as a "web gateway". This helps us separate the problem and handle the SSL modularly. Again, I will be submitting the files in e-mission-docs repo along with the mongodb.

[shankari]

@atton16 I think that the production script should go here as well. Basically, I think that there should be two separate scripts, one for each conceptual module. After the UI component is separated from the server component (#596), we could have three.

We want to have as few script locations as possible - fewer commands to type out - while still supporting flexible deployments. So basically one script per module or tier

[shankari]

So I thought about it some more, and I think that it would be good to have the mongodb as a separate script, but checked in to this repository since it is a dependency. But it is true that in the multi-tier case, the database tier will be configured separately from the appserver tier. So make it easy to copy that script to the host running the database and people can use it for multi-tier (rare) and everything-on-one-host (common).

[shankari]

@sam11b @valin @ecvick85 you can try this out while setting up the new server instead of following the manual steps

[shankari]

please make sure to report back here and tell us how it went!

[shankari]

Is this primarily focused around starting the devapp? What about the analysis pipeline? The pipeline is run periodically, and I currently launch it from a cronjob.
https://github.com/e-mission/e-mission-docs/blob/master/docs/e-mission-server/deploying_your_own_server_to_production.md#the-analysis-pipeline

However, this is clearly a good start, so I approve of merging this once we get confirmation from somebody that it works.

@brianmickel, @sam11b @valin @ecvick85 please let me whether this works for your setup. I'm also donwloading Docker now so that I can test myself.

[atton16]

Oh the daily pipeline task is new for me. Let me try it with our server first then I will update the file once I understand the process more clearly. Also, since the pipeline process can be trigger separately from the server app. I am thinking about building a separate docker image as e-mission-pipeline-runner just to run the pipeline with configurable execution frequency.

[atton16]

Also, you might wanted to consider upload Docker image on Docker Hub to even make it much more easier to others who do not wanted to tangle with the building process (which might yield a different result on different OSs). So others can simple execute a docker run and go!

[shankari]

@sam11b @valin1 @ecvick85 @b-i-l-l-c-a-o @trevor-wu please let me whether this works for your setup. I'm also donwloading Docker now so that I can test myself.

[shankari]

This dockerfile is not working for me. After I launch the e-mission server repo, it looks like port 8080 is redirected.

$ docker container list
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                    NAMES
488a8c84a7ff        e-mission-server:latest   "/usr/bin/tini -- /b…"   3 minutes ago       Up 3 minutes        0.0.0.0:8080->8080/tcp   e-mission-server-1

But I get a "connection refused" for localhost:8080 from my laptop.

The connection was reset

I logged in to the container using

$ docker exec -it 488a8c84a7ff bash
# bash start_script.sh
Connecting to database URL e-mission-mongo-1
...
Running with HTTPS turned OFF - use a reverse proxy on production
Bottle v0.13-dev server starting up (using CherootServer())...
Listening on http://localhost:8080/
Hit Ctrl-C to quit.

Traceback (most recent call last):
  File "emission/net/api/cfc_webapp.py", line 483, in <module>
    run(host=server_host, port=server_port, server='cheroot', debug=True)
  File "/usr/src/app/emission/net/api/bottle.py", line 3713, in run
    server.run(app)
  File "/usr/src/app/emission/net/api/bottle.py", line 3303, in run
    server.start()
  File "/opt/conda/envs/emission/lib/python3.6/site-packages/cheroot/server.py", line 1525, in start
    raise socket.error(msg)
OSError: No socket could be created -- (('127.0.0.1', 8080): [Errno 98] Address already in use) -- (('::1', 8080, 0, 0): [Errno 99] Cannot assign requested address)

So apparently the server is running, but I can't connect to it.

[shankari]

why do you need build-essential?
That installs the entire build toolchain including gcc.
We are not really building any c programs - apt-get install nodejs should install the compiled version of node...

[atton16]

There are a lot of missing dependencies. many of npm packages requires compile from source when you do npm install.

[shankari]

hm, I removed build-essential after container was created and there didn't appear to be any dependencies. So maybe we could remove it at the end after the rest of the installation is complete. I am not sure exactly how much disk space we will save, though...

[shankari]

The server is definitely running but I cannot connect to it.

# apt-get install procps
# ps -aef
....
root        11    10  0 21:34 ?        00:00:00 python emission/net/api/cfc_webapp.py

[shankari]

So apparently the server is running, but I can't connect to it.

Ok so this was because I didn't configure the webserver.conf, which was set to localhost:8080 by default, so the redirection didn't work. When I changed webserver.conf to 0.0.0.0:8080, everything worked.

[shankari]

@atton16 you need to add a step to the README that also states that the webserver.conf should be changed. Also, I think it would be good to make the setup for dev users be essentially configuration-less - see comments below.

Can you provide an ETA for the changes? I'd like to merge them and then create a branch for this version before I merge the GIS-based mode inference into master.

[shankari]

@atton16 this review uses the new suggestions feature so that you can more easily see what I mean

You might also want to add some notes on how to debug; as you saw from e-mission/e-mission-docs#42 it is non-obvious how to debug issues in docker instead of seeing the errors show up directly in the console.

[shankari]

@atton16 is there an ETA on this? I'd like to merge it before I merge #578 so that then I could publish two different images for GIS-based and decision tree-based mode inference.

[shankari]

@atton16 you applied the suggestion, but didn't make the change in the comments.

you can check in the default docker conf as conf/storage/db.conf.docker.sample.
Then you can remove step 4 and change this to source="$(pwd)"/conf/storage/db.conf.docker.sample

[shankari]

@atton16 also, I suggested removing the build-essential package after installing npm.

hm, I removed build-essential after container was created and there didn't appear to be any dependencies. So maybe we could remove it at the end after the rest of the installation is complete. I am not sure exactly how much disk space we will save, though...