Skip to content

Commit

Permalink
Added Docker SSL encryption example
Browse files Browse the repository at this point in the history
- Added a common-services.yml file and reworked docker-compose.yml file so that the services are extensions of the services in common-services.yml file
- Added Generate SSL Stores service to docker-compose to generate signed certs and SSL stores
- Added Cassandra SSL service to docker-compose which only accepts encrypted JMX and Native Protocol connections
- Added Initialize Reaper DB SSL service to docker-compose to initialise keyspace in Cassandra SSL container via an encrypted connection
- Added nodetool SSL service to docker-compose to query Cassandra SSL container via an encrypted connection
- Added CQLSH SSL encrypted service to docker-compose
- Replaced intialize-reaper_db and initialize-reaper_db-ssl services to use CQLSH common service
- Added an environment variable to the Reaper Docker container to allow Reaper to use an SSL encrypted connection when connecting with Cassandra storage
- Minor updates to Docker Reaper configuration YAML so that it contains latest Reaper options
- Fixed bug where JVM options were passed to Reaper in the wrong position
- Updated site documentation to include instructions on orchestrating an SSL encrypted environment
  • Loading branch information
ossarga committed Sep 8, 2017
1 parent 080c30c commit d85d438
Show file tree
Hide file tree
Showing 34 changed files with 666 additions and 55 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,6 @@ src/ui/package-lock.json

# Docker Compose persistent storage directory
src/packaging/data

# SSL stores directory
src/packaging/ssl-stores
58 changes: 55 additions & 3 deletions src/docs/content/docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -384,7 +384,11 @@ Note that the above command will build the Reaper JAR and place it in the _src/s

### Start Docker Environment

From the top level directory change to the _src/packaging_ directory
The `docker-compose` services available allow for orchestration of an environment that uses default settings. In addition, services are provided that allow orchestration of an environment in which the connections between the services are SSL encrypted. Services which use SSL encryption contain a `-ssl` suffix in their name.

#### Default Settings Environment

From the top level directory change to the _src/packaging_ directory:

```bash
cd src/packaging
Expand All @@ -407,7 +411,7 @@ Once the Cassandra node is online and accepting CQL connections, create the requ
By default, the `reaper_db` keyspace is created using a replication factor of 1. To change this replication factor, provide the intended replication factor as an optional argument:

```bash
docker-compose run initialize-reaper_db [$REPLICATION_FACTOR]
docker-compose run cqlsh-initialize-reaper_db [$REPLICATION_FACTOR]
```

Wait a few moments for the `reaper_db` schema change to propagate, then start Reaper:
Expand All @@ -416,6 +420,46 @@ Wait a few moments for the `reaper_db` schema change to propagate, then start Re
docker-compose up reaper
```

#### SSL Encrypted Connections Environment

From the top level directory change to the _src/packaging_ directory:

```bash
cd src/packaging
```

Generate the SSL Keystore and Truststore which will be used to encrypt the connections between Reaper and Cassandra.

```bash
docker-compose run generate-ssl-stores
```

Start the Cassandra cluster which encrypts both the JMX and Native Protocol:

```bash
docker-compose up cassandra-ssl
```

The `nodetool-ssl` Docker Compose service can be used to check on the Cassandra node's status:

```bash
docker-compose run nodetool-ssl status
```

Once the Cassandra node is online and accepting encrypted SSL connections via the Native Transport protocol, create the required `reaper_db` Cassandra keyspace to allow Reaper to save its cluster and scheduling data.

By default, the `reaper_db` keyspace is created using a replication factor of 1. To change this replication factor, provide the intended replication factor as an optional argument:

```bash
docker-compose run cqlsh-initialize-reaper_db-ssl [$REPLICATION_FACTOR]
```

Wait a few moments for the `reaper_db` schema change to propagate, then start the Reaper service that will establish encrypted connections to Cassandra:

```bash
docker-compose up reaper-ssl
```


### Access The Environment

Expand All @@ -429,12 +473,20 @@ When adding the Cassandra node to the Reaper UI, use the IP address found via:
docker-compose run nodetool status
```

The helper `cqlsh` Docker Compose service has also been included:
The helper `cqlsh` Docker Compose service has also been included for both the default and SSL encrypted environments:

#### Default Environment

```bash
docker-compose run cqlsh
```

#### SSL Encrypted Environment

```bash
docker-compose run cqlsh-ssl
```

### Destroying the Docker Environment

When terminating the infrastructure, use the following command to stop
Expand Down
37 changes: 37 additions & 0 deletions src/packaging/common-services.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
version: '2.1'

services:
cassandra-common:
image: cassandra:3.11
env_file:
- ./docker-services/cassandra/cassandra.env
mem_limit: 4g
memswap_limit: 4g
mem_swappiness: 0
ports:
- "7000:7000"
- "7001:7001"
- "7199:7199"
- "9042:9042"
volumes:
- ./data/cassandra:/var/lib/cassandra
- ./docker-services/cassandra/jmxremote.access:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/management/jmxremote.access
- ./docker-services/cassandra/jmxremote.password:/etc/cassandra/jmxremote.password

cqlsh-common:
build: ./docker-services/cqlsh
env_file:
- ./docker-services/cqlsh/cql.env

nodetool-common:
build: ./docker-services/nodetool
env_file:
- ./docker-services/nodetool/nodetool.env

reaper-common:
image: cassandra-reaper:latest
env_file:
- ./docker-services/reaper/reaper.env
ports:
- "8080:8080"
- "8081:8081"
112 changes: 87 additions & 25 deletions src/packaging/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,48 +2,110 @@ version: '2.1'

services:
cassandra:
image: cassandra:3.11
extends:
file: ./common-services.yml
service: cassandra-common

cassandra-ssl:
extends:
file: ./common-services.yml
service: cassandra-common
entrypoint: /docker-alt-entrypoint.sh cassandra -f
env_file:
- ./docker-services/cassandra/cassandra.env
mem_limit: 4g
memswap_limit: 4g
mem_swappiness: 0
ports:
- "7000:7000"
- "7001:7001"
- "7199:7199"
- "9042:9042"
- ./docker-services/cassandra-ssl/cassandra-ssl.env
volumes:
- ./data/cassandra:/var/lib/cassandra
- ./docker-services/cassandra/jmxremote.access:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/management/jmxremote.access
- ./docker-services/cassandra/jmxremote.password:/etc/cassandra/jmxremote.password
- ./ssl-stores/cassandra-server-keystore.jks:/etc/ssl/cassandra-server-keystore.jks
- ./ssl-stores/generic-server-truststore.jks:/etc/ssl/generic-server-truststore.jks
- ./docker-services/cassandra-ssl/docker-entrypoint.sh:/docker-alt-entrypoint.sh

cqlsh:
image: cassandra:3.11
command: cqlsh cassandra
extends:
file: ./common-services.yml
service: cqlsh-common
links:
- cassandra

initialize-reaper_db:
build: ./docker-services/initialize-reaper_db
cqlsh-ssl:
extends:
file: ./common-services.yml
service: cqlsh-common
env_file:
- ./docker-services/cqlsh-ssl/cqlsh-ssl.env
links:
- cassandra-ssl
volumes:
- ./ssl-stores/cassandra-server-keystore.jks:/etc/ssl/cassandra-server-keystore.jks
- ./ssl-stores/ca-cert:/etc/ssl/ca-cert

cqlsh-initialize-reaper_db:
extends:
file: ./common-services.yml
service: cqlsh-common
entrypoint: /docker-alt-entrypoint.sh
env_file:
- ./docker-services/cqlsh-initialize-reaper_db/cqlsh-initialize-reaper_db.env
links:
- cassandra
volumes:
- ./docker-services/cqlsh-initialize-reaper_db/docker-entrypoint.sh:/docker-alt-entrypoint.sh

cqlsh-initialize-reaper_db-ssl:
extends:
file: ./common-services.yml
service: cqlsh-common
entrypoint: /docker-alt-entrypoint.sh
env_file:
- ./docker-services/cqlsh-initialize-reaper_db-ssl/cqlsh-initialize-reaper_db-ssl.env
links:
- cassandra-ssl
volumes:
- ./docker-services/cqlsh-initialize-reaper_db-ssl/docker-entrypoint.sh:/docker-alt-entrypoint.sh
- ./ssl-stores/cassandra-server-keystore.jks:/etc/ssl/cassandra-server-keystore.jks
- ./ssl-stores/ca-cert:/etc/ssl/ca-cert

generate-ssl-stores:
build: ./docker-services/generate-ssl-stores
volumes:
- ./resource/ca_cert.conf:/usr/src/app/ca_cert.conf
- ./ssl-stores:/usr/src/app/ssl-stores

nodetool:
image: cassandra:3.11
entrypoint: nodetool --host cassandra --username reaperUser --password reaperPass
extends:
file: ./common-services.yml
service: nodetool-common
links:
- cassandra

reaper:
image: cassandra-reaper:latest
nodetool-ssl:
extends:
file: ./common-services.yml
service: nodetool-common
env_file:
- ./docker-services/reaper/reaper.env
- ./docker-services/nodetool-ssl/nodetool-ssl.env
links:
- cassandra-ssl
volumes:
- ./ssl-stores/cassandra-server-keystore.jks:/etc/ssl/cassandra-server-keystore.jks
- ./ssl-stores/generic-server-truststore.jks:/etc/ssl/generic-server-truststore.jks

reaper:
extends:
file: ./common-services.yml
service: reaper-common
links:
- cassandra
ports:
- "8080:8080"
- "8081:8081"

reaper-ssl:
extends:
file: ./common-services.yml
service: reaper-common
env_file:
- ./docker-services/reaper-ssl/reaper-ssl.env
links:
- cassandra-ssl
volumes:
- ./ssl-stores/reaper-server-keystore.jks:/etc/ssl/reaper-server-keystore.jks
- ./ssl-stores/generic-server-truststore.jks:/etc/ssl/generic-server-truststore.jks

reaper-build-packages:
extends:
Expand Down
4 changes: 4 additions & 0 deletions src/packaging/docker-services/cassandra-ssl/cassandra-ssl.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/usr/bin/env bash

CASSANDRA_KEYSTORE_PASSWORD=keypassword
CASSANDRA_TRUSTSTORE_PASSWORD=trustpassword
113 changes: 113 additions & 0 deletions src/packaging/docker-services/cassandra-ssl/docker-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
#!/bin/bash

# copied from:
# https://github.com/docker-library/cassandra/blob/d83b850cd17bc9198876f8686197c730e29c7448/2.1/docker-entrypoint.sh

set -e

# first arg is `-f` or `--some-option`
if [ "${1:0:1}" = '-' ]; then
set -- cassandra -f "$@"
fi

# allow the container to be started with `--user`
if [ "$1" = 'cassandra' -a "$(id -u)" = '0' ]; then
chown -R cassandra /var/lib/cassandra /var/log/cassandra "$CASSANDRA_CONFIG"
exec gosu cassandra "$BASH_SOURCE" "$@"
fi

if [ "$1" = 'cassandra' ]; then
: ${CASSANDRA_RPC_ADDRESS='0.0.0.0'}

: ${CASSANDRA_LISTEN_ADDRESS='auto'}
if [ "$CASSANDRA_LISTEN_ADDRESS" = 'auto' ]; then
CASSANDRA_LISTEN_ADDRESS="$(hostname --ip-address)"
fi

: ${CASSANDRA_BROADCAST_ADDRESS="$CASSANDRA_LISTEN_ADDRESS"}

if [ "$CASSANDRA_BROADCAST_ADDRESS" = 'auto' ]; then
CASSANDRA_BROADCAST_ADDRESS="$(hostname --ip-address)"
fi
: ${CASSANDRA_BROADCAST_RPC_ADDRESS:=$CASSANDRA_BROADCAST_ADDRESS}

if [ -n "${CASSANDRA_NAME:+1}" ]; then
: ${CASSANDRA_SEEDS:="cassandra"}
fi
: ${CASSANDRA_SEEDS:="$CASSANDRA_BROADCAST_ADDRESS"}

sed -ri 's/(- seeds:).*/\1 "'"$CASSANDRA_SEEDS"'"/' "$CASSANDRA_CONFIG/cassandra.yaml"

for yaml in \
broadcast_address \
broadcast_rpc_address \
cluster_name \
endpoint_snitch \
listen_address \
num_tokens \
rpc_address \
start_rpc \
; do
var="CASSANDRA_${yaml^^}"
val="${!var}"
if [ "$val" ]; then
sed -ri 's/^(# )?('"$yaml"':).*/\2 '"$val"'/' "$CASSANDRA_CONFIG/cassandra.yaml"
fi
done

# Set the Client encryption options in the Cassandra configuration file
#
# grab the line number of the 'client_encryption_options' property then iterate down through the file until the
# first empty line is reached. This will be the end of the block containing all the properties for
# 'client_encryption_options'.
start_line_number=$(grep -n "client_encryption_options:" "$CASSANDRA_CONFIG/cassandra.yaml" | cut -d':' -f1)
count=${start_line_number}
line=$(sed "${count}q;d" "$CASSANDRA_CONFIG/cassandra.yaml")

while [ "${line}" != "" ]
do
((count++))
line=$(sed "${count}q;d" "$CASSANDRA_CONFIG/cassandra.yaml")
done

end_line_number=${count}

for key_val in \
"enabled:true" \
"optional:false" \
"keystore:\/etc\/ssl\/cassandra\-server\-keystore\.jks" \
"keystore_password:${CASSANDRA_KEYSTORE_PASSWORD}" \
"require_client_auth:true" \
"truststore:\/etc\/ssl\/generic\-server\-truststore\.jks" \
"truststore_password:${CASSANDRA_TRUSTSTORE_PASSWORD}" \
"protocol:TLS" \
"algorithm:SunX509" \
"store_type:JKS" \
"cipher_suites:[TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA]"
do
property=$(echo ${key_val} | cut -d':' -f1)
value=$(echo ${key_val} | cut -d':' -f2)
sed -ie "${start_line_number},${end_line_number} s/\(#\ \)\{0,1\}\(${property}:\).*/\2 ${value}/" "$CASSANDRA_CONFIG/cassandra.yaml"
done

# Set the Rack and DC properties
for rackdc in dc rack; do
var="CASSANDRA_${rackdc^^}"
val="${!var}"
if [ "$val" ]; then
sed -ri 's/^('"$rackdc"'=).*/\1 '"$val"'/' "$CASSANDRA_CONFIG/cassandra-rackdc.properties"
fi
done

# Set JVM SSL encryption options for Cassandra
sed -ie 's/#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=true"/JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=true"/g' /etc/cassandra/cassandra-env.sh
sed -ie 's/#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true"/JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true"/g' /etc/cassandra/cassandra-env.sh
sed -ie 's/#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=<enabled-protocols>"/JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1"/g' /etc/cassandra/cassandra-env.sh
sed -ie 's/#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=<enabled-cipher-suites>"/JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=TLS_RSA_WITH_AES_256_CBC_SHA"/g' /etc/cassandra/cassandra-env.sh
sed -ie "s/#JVM_OPTS=\"\$JVM_OPTS -Djavax.net.ssl.keyStore=\/path\/to\/keystore\"/JVM_OPTS=\"\$JVM_OPTS -Djavax.net.ssl.keyStore=\/etc\/ssl\/cassandra-server-keystore\.jks\"/g" /etc/cassandra/cassandra-env.sh
sed -ie "s/#JVM_OPTS=\"\$JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>\"/JVM_OPTS=\"\$JVM_OPTS -Djavax.net.ssl.keyStorePassword=${CASSANDRA_KEYSTORE_PASSWORD}\"/g" /etc/cassandra/cassandra-env.sh
sed -ie "s/#JVM_OPTS=\"\$JVM_OPTS -Djavax.net.ssl.trustStore=\/path\/to\/truststore\"/JVM_OPTS=\"\$JVM_OPTS -Djavax.net.ssl.trustStore=\/etc\/ssl\/generic-server-truststore\.jks\"/g" /etc/cassandra/cassandra-env.sh
sed -ie "s/#JVM_OPTS=\"\$JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>\"/JVM_OPTS=\"\$JVM_OPTS -Djavax.net.ssl.trustStorePassword=${CASSANDRA_TRUSTSTORE_PASSWORD}\"/g" /etc/cassandra/cassandra-env.sh
fi

exec "$@"
Loading

0 comments on commit d85d438

Please sign in to comment.