Skip to content

Latest commit

 

History

History
190 lines (125 loc) · 8.44 KB

localdev.md

File metadata and controls

190 lines (125 loc) · 8.44 KB

Local Development

Prerequisites

Platform notes

  • macOS — Set up your environment with Homebrew as documented, which will include the Xcode CLI Tools.
  • Linux — Install your platform's standard developer tools. This is different for different families of Linux distributions.
  • Windows — Run Linux via Windows Subsystem for Linux v2 (WSL2).

Service flow

Domain names

  • devsec.local — We'll use this domain name to simulate the frontend of the website.

  • api.devsec.local — We'll use this domain name to simulate the API running in AWS Lambda.

  • lambda.devsec.local — We'll never touch this directly. Use api.devsec.local instead.

  • traefik.devsec.local — This is the Traefik dashboard which shows how everything is connected.

Configure aliases

We will use devsec.local and api.devsec.local to simulate the real endpoints which run in production.

This command will use sudo to append these 3 lines to the /etc/hosts file. Since it invokes sudo, you may need to authorize the command with your password.

cat << EOF | sudo tee -a /etc/hosts
127.0.0.1 devsec.local
127.0.0.1 api.devsec.local
127.0.0.1 lambda.devsec.local
127.0.0.1 traefik.devsec.local
EOF

Start backend services

  1. Generate a new Personal Access Token, no particular scopes are required. Save it to your password manager.

  2. Then, set this as an environment variable called GITHUB_TOKEN. This value is passed as a secret to Docker Compose for fetching and building everything.

  3. The local versions of backend services run as containers. From the root of the repository:

    make build-lambda build-serve
cd localdev
docker compose up

    The very first time you run docker compose up, the Docker images will need to build. Subsequent runs will leverage the cached image. Any time the Dockerfile or docker-compose.yml are changed, it is a good idea to explicitly run docker compose up --build.

  4. When you are done, terminate the containers.

    docker compose down

Operating Docker Desktop and Docker Compose is outside the scope of these instructions, but you can read the documentation for yourself.

Traefik (:80)

Traefik is a service which acts as a high-performance reverse proxy in front of our local stack. Traefik runs on port 80, then routes traffic based on where it is sent to.

  • When a request is made to devsec.local on port 80, Traefik will automatically redirect requests to port 1313 where the Hugo web server is running.

  • When a request is made to api.devsec.local on port 80, Traefik will route to the apiproxy container, which will make requests to the lambda containers on your behalf.

Lambda servers (:9000–9010)

In production, we use Amazon API Gateway v2 sitting in front of AWS Lambda.

AWS has open-sourced their Lambda runtimes — namely for Amazon Linux 2023 — so we use that image along with the AWS Lambda Runtime Interface Emulator to create a local AWS Lambda environment that is accurate.

However, passing payloads directly to AWS Lambda is different from going through API Gateway first, so we have a custom reverse proxy server running at api.devsec.local which modifies the original payload to make it look like an API Gateway payload, then forwards that request to lambda.devsec.local which is running AWS Lambda Runtime Interface Emulator in front of our Lambda function.

API Proxy server (:8080)

The only thing this does is receive requests from the Hugo frontend, modify them, pass them to the Lambda servers, receive the response, modify the response, and respond back to the Hugo frontend.

When new endpoints are added to the Lambda function, they must also be added to localdev/api-proxy/main.go. After adding support to the file, compile the changes and restart the localdev Docker environment.

make build-serve
cd localdev
docker compose down
docker compose up --force-recreate

Valkey server (:6379)

Valkey is an open-source fork of Redis, which ceased to be open-source in March 2024. AWS provides ElastiCache Serverless with Valkey support, which devsec.tools uses for caching results.

When the devsec-tools binary is running as a Lambda function it will connect to cache:6379 by default for a caching server. When running in production, you can use the DST_CACHE_HOSTS environment variable to configure production Valkey hosts.

Persistent data

This server uses a persistent volume (localdev_vkdata), so you can stop the Docker container and data will be restored on next restart. If you want to delete the volume data:

  1. Run docker compose down to terminate the local servers.
  2. Run docker volume rm localdev_vkdata to delete the persisted data.

The next time you run docker compose up, the volume will be recreated. Valkey is only used for caching and cache expiration, so it can be deleted without worrying about loss of important data.

Environment variable

When running as a Lambda function, the value of DST_CACHE_HOSTS should be one or more endpoints (delimited by ;) to use for Valkey caching.

# Example: local dev
DST_CACHE_HOSTS="cache:6379"

# Example: production
DST_CACHE_HOSTS="server1.host.com:6379;server2.host.com:6379;server3.host.com:6379"

Traefik Dashboard (:8082)

If you want to see how Traefik routes everything, you can visit http://traefik.devsec.local to view a dashboard.

Endpoints

When launching the local web server, it will tell you which HTTP methods and endpoints are available. It exposes both GET and POST HTTP methods.

GET

For GET endpoints, any parameters are passed as URL-encoded query string parameters. Using the /http endpoint as an example:

GET /http?url=https%3A%2F%2Fapple.com HTTP/1.1
Host: api.devsec.local

POST

For POST endpoints, parameters are passed as a JSON-encoded request body. Using the /http endpoint as an example:

POST /http HTTP/1.1
Host: api.devsec.local
Content-Type: application/json; charset=utf-8

{"url":"https://apple.com"}

Start frontend services

All of this exists in the devsec-ui repository. See that project for further instructions.