Skip to content

Commit 63243bb

Browse files
abessiariabessiari
authored
Solr (#2)
* added support for simplified solr service and and standalone amigo added support for docker push * support for cloning specific branches * aws terraform code * adding google cloud terraform * using correct docker image name for minerva * ignore terraform files and dirs * fixed vars.yaml * adding default for docker_hub_user * Update README.md * Update README.md * adding documentation for aws * Update AWS_README.md * Update AWS_README.md * Update AWS_README.md * deleted old readme file * Update README.md * Update README.md * Update README.md * Update AWS_README.md * Update AWS_README.md * Update README.md * Update README.md * Update AWS_README.md * Update AWS_README.md * Update AWS_README.md * Update AWS_README.md * Update AWS_README.md * Update AWS_README.md * changed location of aws credentials * Update AWS_README.md * Update AWS_README.md * Update AWS_README.md * Update README.md * Update README.md * testing cost estimation * testing cost estimation * testing cost estimation 2 * testing cost estimation 3 * testing cost estimation 4 * testing cost estimation 5 * testing cost estimation 6 * testing cost estimation 7 * Update main.tf * Update main.tf * using golr first ast it builds fast * need to create three docherhub repos * adding conda * securing golr * securing golr * fixing variable host for golr * fixed url for miniconda * adding docker related vars
1 parent dd174ea commit 63243bb

31 files changed

+803
-68
lines changed

‎.gitignore

+4
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
11
minerva
22
noctua
3+
amigo
34
stage_dir
5+
.terraform
6+
.terraform.lock.hcl
7+
terraform.tfstate

‎README.md

+98-29
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,13 @@
11
# Noctua App Stack
22

3-
Install app stack using ansible on a single machine
3+
- Deploy app stack using ansible playbooks on a single machine:
4+
- build_images.yaml
5+
- builds all docker images on local machine.
6+
- push_images.yaml
7+
- pushes images to dockerhub if planning on staging to a remote machine.
8+
- stage.yaml
9+
- Tasks are executed on staging machine which can be local or remote.
10+
411

512
## Requirements
613

@@ -13,71 +20,133 @@ Install app stack using ansible on a single machine
1320
- Notes:
1421
- Docker was given 3 CPUs and 8G RAM. (on mac see Docker Preferences | Resources)
1522
- python 2.7 should work as well.
23+
24+
## Fast Install using miniconda
25+
26+
To install miniconda go to [url](https://docs.conda.io/en/latest/miniconda.html)
27+
28+
This will install ansible, docker ansible module, docker-compose, and terraform.
29+
See dependencies/bare-metal.yml
30+
31+
```
32+
conda env create --file dependencies/bare-metal.yml
33+
conda activate noctua_app_stack
34+
35+
# when done
36+
conda deactivate
37+
38+
# delete a conda environment:
39+
conda env remove -n noctua_app_stack
40+
41+
```
1642

17-
## Installing ansible and ansible docker plugin
43+
## Installing ansible and ansible docker plugin using Pip.
44+
1845

1946
The ansible docker plugin is used to buid docker images.
2047

2148
```sh
2249
pip install ansible
2350
pip install docker
2451
```
25-
## Deploying app stack:
2652

27-
#### Clone this repo.
53+
## Clone this repo.
2854

2955
```sh
3056
git clone https://github.com/abessiari/noctua_app_stack.git
3157
cd noctua_app_stack
3258
```
3359

34-
#### Modify `vars.yaml` as needed. Minimally you need to modify the following variables:
35-
- uri
36-
- username
37-
- password
38-
- barista_lookup_host
39-
- On mac if using wireless, you can use `ipconfig getifaddr en0`
60+
## Building Docker Images:
61+
The playbook <i>build_images.yaml</i> clones minerva, noctua and amigo git repositories
62+
and builds corresponding docker images. The default branch used is <i>master</i>.
63+
To change the branch being cloned, see <i>repo_map</i> in docker-vars.yaml
64+
65+
In order to stage the app stack to a remote machine, Create an account on dockerhub if you do not have one
66+
and a public dockerhub repository named <i>minerva</i>, <i>noctua</i> and <i>golr</i>.
67+
Then set <i>docker_hub_user</i> in docker-vars.yaml or simply
68+
use the -e option when using ansible-playbook command.
69+
70+
71+
#### Build images.
4072

41-
#### Build noctua and minerva docker images.
73+
```sh
74+
ansible-playbook -e docker_hub_user=xxxx build_images.yaml
75+
docker image list | egrep 'minerva|noctua|golr'
76+
```
77+
78+
#### Push images.
79+
- Skip this step if planning on staging locally.
4280

4381
```sh
44-
ansible-playbook build_images.yaml
45-
docker image list | grep minerva
46-
docker image list | grep noctua
82+
ansible-playbook -e docker_hub_user=xxxx push_images.yaml
4783
```
4884

49-
#### Stage artifacts.
50-
- Create and stage blazegraph journal.
51-
- Stage repos
85+
#### Provision machine and stage app stack on the cloud:
86+
- Skip this step if planning on staging locally.
87+
- Refer to [this document](./docs/AWS_README.md) on provisionning an instance on AWS.
88+
89+
## Staging app stack:
90+
91+
#### Staging tasks at a glance:
92+
- Creates blazegraph journal.
93+
- Creates Solr Index
94+
- Clones repos
95+
- noctua-form, noctua-landing-page, noctua-models, go-site
96+
- Creates docker-compose and configuration files from templates.
97+
98+
#### Modify `vars.yaml`.
99+
- These can also be set on command line using the -e flag.
100+
- Barista:
101+
- uri
102+
- username
103+
- password
104+
105+
#### Stage Artifacts.
106+
- Staging tasks at a glance:
107+
- Creates blazegraph journal.
108+
- Creates Solr Index
109+
- Clones repos
52110
- noctua-form, noctua-landing-page, noctua-models, go-site
53-
- Note: Stage the journals below to speed up minerva start up time.
54-
- Create stage_dir if it does not exist
55-
- Copy`blazegraph.jnl` to stage_dir
56-
- Copy`blazegraph-go-lego-reacto-neo.jnl` to stage_dir
111+
- Creates docker-compose and configuration files from templates.
112+
- Staging to a remote machine:
113+
- Refer to [this document](./docs/AWS_README.md) on provisionning an AWS.
57114

58115
```sh
59-
ansible-playbook stage.yaml
116+
# on Mac:
117+
export HOST=`ipconfig getifaddr en0`
118+
ansible-playbook -e "host=$HOST" -i "localhost," stage.yaml
60119
```
61120
#### Bring up stack using docker-compose.
121+
Two docker-compose files are staged:
122+
- docker-compose-golr.yaml
123+
- Uses a lightweight solr image for golr
124+
- docker-compose-amigo.yaml
125+
- Uses the official geneontology/amigo-standalone for golr
62126

63127
```sh
64-
# assuming stage_dir is in current directory
65-
docker-compose -f stage_dir/docker-compose.yaml up -d
128+
# assuming stage_dir is in current directory and docker-compose-golr.yaml is used:
129+
docker-compose -f stage_dir/docker-compose-golr.yaml up -d
66130

67131
# minerva takes a long time to start up the first time
68132
# Tail minerva logs to see its progress
69-
docker-compose -f stage_dir/docker-compose.yaml logs -f minerva
133+
docker-compose -f stage_dir/docker-compose-golr.yaml logs -f minerva
134+
# Or tail all logs
135+
docker-compose -f stage_dir/docker-compose-golr.yaml logs -f
70136

71137
# When minerva is ready all other services should be up
72-
docker-compose -f stage_dir/docker-compose.yaml ps
138+
docker-compose -f stage_dir/docker-compose-golr.yaml ps
73139
```
74140

75141
#### Access noctua from a browser using `http://localhost:{{ noctua_proxy_port }}`
76142
- Use `http://localhost:8080` if default `noctua_proxy_port` was used
77143

78-
#### Bring down stack using docker-compose.
144+
#### Bring down stack using docker-compose.
79145

80146
```sh
81-
# assuming stage_dir is in current directory
82-
docker-compose -f stage_dir/docker-compose.yaml down
147+
docker-compose -f stage_dir/docker-compose-golr.yaml down
148+
# kill works faster ...
149+
docker-compose -f stage_dir/docker-compose-golr.yaml kill
150+
#delete containers:
151+
docker-compose -f stage_dir/docker-compose-golr.yaml rm -f
83152
```

‎allowed-terraform-version.sentinel

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
import "tfplan"
2+
3+
main = rule {
4+
tfplan.terraform_version matches "^0\\.12\\.\\d+$"
5+
}

‎aws/main.tf

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
resource "aws_instance" "noctua_app_stack_server" {
2+
ami = "ami-07dd19a7900a1f049"
3+
instance_type = var.instance_type
4+
vpc_security_group_ids = [aws_security_group.noctua_app_stack_sg.id]
5+
subnet_id = aws_subnet.noctua_app_stack_public_subnet.id
6+
key_name = var.key_name
7+
tags = var.tags
8+
9+
ebs_block_device {
10+
device_name = "/dev/sda1"
11+
delete_on_termination = true
12+
tags = var.tags
13+
volume_size = 100
14+
}
15+
}

‎aws/outputs.tf

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
//output "private_key_path" {
2+
// value = var.private_key_path
3+
//}
4+
5+
output "noctua_port" {
6+
value = var.noctua_port
7+
}
8+
9+
output "barista_port" {
10+
value = var.barista_port
11+
}
12+
13+
output "golr_port" {
14+
value = var.golr_port
15+
}
16+
17+
18+
output "public_ip" {
19+
value = aws_instance.noctua_app_stack_server.public_ip
20+
}

‎aws/provider.tf

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
provider "aws" {
2+
region = "us-west-2"
3+
// shared_credentials_file = "~/.aws/go-credentials"
4+
}

‎aws/securit_groups.tf

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
resource "aws_security_group" "noctua_app_stack_sg" {
2+
name = "nocuta-app-stack-sg"
3+
vpc_id = aws_vpc.noctua_app_stack_vpc.id
4+
tags = var.tags
5+
6+
ingress {
7+
from_port = var.ssh_port
8+
to_port = var.ssh_port
9+
protocol = "tcp"
10+
cidr_blocks = ["0.0.0.0/0"]
11+
}
12+
13+
ingress {
14+
from_port = var.noctua_port
15+
to_port = var.noctua_port
16+
protocol = "tcp"
17+
cidr_blocks = ["0.0.0.0/0"]
18+
}
19+
20+
ingress {
21+
from_port = var.barista_port
22+
to_port = var.barista_port
23+
protocol = "tcp"
24+
cidr_blocks = ["0.0.0.0/0"]
25+
}
26+
27+
ingress {
28+
from_port = var.golr_port
29+
to_port = var.golr_port
30+
protocol = "tcp"
31+
cidr_blocks = ["0.0.0.0/0"]
32+
}
33+
34+
egress {
35+
from_port = 0
36+
to_port = 0
37+
protocol = "-1"
38+
cidr_blocks = ["0.0.0.0/0"]
39+
}
40+
}

‎aws/ssh_key.tf

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
resource "aws_key_pair" "ssh_key" {
2+
key_name = var.key_name
3+
public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1TL5/v2/fy0kyX68/W8YT2bML7XvK6IF84vgSYDCqk2E/UbsLXYbUEk5/nxFOBW+4gEmQP9D80jkAAOEN4zXalctKokib8SMHhCSzv+epCtqsbxKzU71Gfl9elJWNWZcwnCzwkZrO6RDZeu7Jcp3Asx4NiyAKd7PjWuLU8qmJo5rjwbAS4dumvlDLdjVWka4jRT02jVP5+p2J/+JwvZJCGMlEb6d4WKuy8UsO4wZXXzgwrYRPfBu3XaZFA19MoLAO1UyzZwKSsin7yg/+JN8IE7sIwvvL9pExD4DU0krfNyjAO8c88UhzrUbN/vEGxDXEIvvmIQ8k1MtHh8o6WwP3 AEssiari@aessiari-mba"
4+
tags = var.tags
5+
}

‎aws/terraform.tvars

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
variable "public_key" {
2+
default = ""
3+
}

‎aws/vars.tf

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
variable "tags" {
2+
type = map
3+
default = { Name = "testing-noctua-app-stack" }
4+
}
5+
6+
variable "instance_type" {
7+
default = "t2.large"
8+
}
9+
10+
variable "key_name" {
11+
default = "noctua-app-stack-ssh-key"
12+
}
13+
14+
variable "public_key_path" {
15+
default = "~/.ssh/id_rsa.pub"
16+
}
17+
18+
variable "private_key_path" {
19+
default = "~/.ssh/id_rsa"
20+
}
21+
22+
variable "ssh_port" {
23+
type = number
24+
default = 22
25+
description = "ssh server port"
26+
}
27+
28+
variable "noctua_port" {
29+
type = number
30+
default = 8080
31+
description = "noctua server port"
32+
}
33+
34+
variable "barista_port" {
35+
type = number
36+
default = 8090
37+
description = "barista server port"
38+
}
39+
40+
variable "golr_port" {
41+
type = number
42+
default = 8983
43+
description = "golr server port"
44+
}

‎aws/vpc.tf

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
resource "aws_vpc" "noctua_app_stack_vpc" {
2+
cidr_block = "10.0.0.0/16"
3+
enable_dns_support = true
4+
enable_dns_hostnames = true
5+
tags = var.tags
6+
}
7+
8+
resource "aws_subnet" "noctua_app_stack_public_subnet" {
9+
vpc_id = aws_vpc.noctua_app_stack_vpc.id
10+
cidr_block = "10.0.1.0/24"
11+
map_public_ip_on_launch = true
12+
availability_zone = "us-west-2a"
13+
tags = var.tags
14+
}
15+
16+
resource "aws_internet_gateway" "noctua_app_stack_igw" {
17+
vpc_id = aws_vpc.noctua_app_stack_vpc.id
18+
tags = var.tags
19+
}
20+
21+
resource "aws_route_table" "noctua_app_stack_public_routing_table" {
22+
vpc_id = aws_vpc.noctua_app_stack_vpc.id
23+
tags = var.tags
24+
25+
route {
26+
cidr_block = "0.0.0.0/0"
27+
gateway_id = aws_internet_gateway.noctua_app_stack_igw.id
28+
}
29+
}
30+
31+
resource "aws_route_table_association" "noctua_app_stack_public_route_table_association" {
32+
subnet_id = aws_subnet.noctua_app_stack_public_subnet.id
33+
route_table_id = aws_route_table.noctua_app_stack_public_routing_table.id
34+
}

0 commit comments

Comments
 (0)