Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[LXD-1337] Microcloud demo folder #366

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions demos/microcloud-deploy/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Single Node Microcloud deployment (Nested VMs)

## Prerequisites

- Single Ubuntu node with LXD installed
- A network of type bridged on the host, e.g. from lxc network list
```
+----------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
+----------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| br0 | bridge | NO | | | | 17 | |
+----------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
```

## Get started - init lxd - optional if already initialised

```
sudo lxd init --auto
```

## Init tofu

```
sudo snap install --classic opentofu
tofu init
```

### Variables

| Variable Name | Purpose |
| ------------------------ | ----------------------------------------------------- |
| pro_token | Can be ignored for now |
| lxd_project | LXD Project - will be created |
| bridge_nic | The bridge interface created in the VM |
| lookup_subnet | The CIDR of bridge_nic, e.g. 10.10.32.0/24 |
| ovn_gateway | The IP to use for the OVN Virtual network router |
| ovn_range_start | The start range of OVN to be created |
| ovn_range_end | The end range of OVN to be created |
| ssh_pubkey | The public SSH key to insert into the Microcloud VMs |
| host_bridge_network | The LXD Bridge network |

## Apply the plan

```
tofu apply
```

## Init Microcloud

The terraform will automatically trigger an initialisation, via the non-interactive method - more information can be found here: https://canonical-microcloud.readthedocs-hosted.com/en/latest/how-to/initialise/#non-interactive-configuration


## TODO:

- Additional networks
~~- Microcloud VM bridging~~
- Multinode
- Non-bridged network on the host
124 changes: 124 additions & 0 deletions demos/microcloud-deploy/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
terraform {
required_providers {
lxd = {
source = "terraform-lxd/lxd"
version = "2.0.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There already is a 2.3.0.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

}
ssh = {
source = "loafoe/ssh"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for not using the built-in remote-exec provisioner from Terraform directly? https://developer.hashicorp.com/terraform/language/resources/provisioners/remote-exec

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was a opentofu requirement - I can switch back to the terraform built-in - what do you think?

version = "2.7.0"
}
}
}

provider "lxd" {
}
provider ssh {
# Configuration options
}
locals {
cloud_init = <<EOT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't the MicroCloud preseed file also be included in the cloud init? You can then use runcmd to execute the microcloud init --preseed command.

This way you won't need either SSH or the provisioner (and a key pair) to connect to the machine and to run the command.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could still use the templating engine from Terraform to generate the preseed.yml.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes - it can be included - the issue is that cloud-init is going to run on all three nodes, so you're writing the file out on all three unnecessarily. The issue is without some custom logic in cloud-init, there's no way to decide to pick a 'leader' to run the init phase of this, so I'm not sure what we'd be saving here - do you have some way which we could remove the ssh remote exec and only run init on the leader via cloud-init?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After #383 is being merged, you would need to put the preseed file on all of the nodes anyway.
Also a initiator/leader has to be picked as part of crafting the preseed file (see https://github.com/canonical/microcloud/pull/383/files#diff-14d3c025154c62cbd2816232e7d56a1d80c9389b77d2eebdac004c646af8783d for reference).

With cloud-init you would then run microcloud preseed on all nodes and MicroCloud will pick the selected initiator/leader automatically.

#cloud-config
users:
- name: ubuntu
ssh_authorized_keys:
- ${var.ssh_pubkey}
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /usr/bin/bash
snap:
commands:
1: [snap, refresh, lxd, --channel=5.21/stable, --cohort="+"]
2: [snap, install, microceph, --channel=quincy/stable, --cohort="+"]
3: [snap, install, microovn, --channel=22.03/stable, --cohort="+"]
4: [snap, install, microcloud, --channel=latest/stable, --cohort="+"]
EOT
cloud_init_network = <<EOT
network:
version: 2
ethernets:
enp5s0:
dhcp4: false
bridges:
br0:
dhcp4: true
dhcp6: false
interfaces: [enp5s0]
EOT
}
resource "lxd_project" "microcloud" {
name = "${var.lxd_project}"
description = "Your Microcloud!"
config = {
"features.storage.volumes" = true
"features.images" = false
"features.profiles" = false
}
}

resource "lxd_volume" "mc_sdb_vols" {
count = 3
name = "microcloud-sdb-${count.index}"
content_type = "block"
pool = "default"
project = "${var.lxd_project}"
}
resource "lxd_instance" "microcloud_nodes" {
count = 3
name = "microcloud-${count.index}"
image = "ubuntu:jammy"
type = "virtual-machine"
project = lxd_project.microcloud.name
wait_for_network = true
config = {
"boot.autostart" = true
"cloud-init.user-data" = local.cloud_init
"cloud-init.network-config" =local.cloud_init_network
}

limits = {
cpu = 4
memory = "8GiB"
}
device {
name = "root"
type = "disk"
properties = {
pool = "default"
path = "/"
size = "200GiB"
}
}
device {
name = "sdb"
type = "disk"
properties = {
pool = "default"
source = lxd_volume.mc_sdb_vols[count.index].name
}
}
device {
name = "eth0"
type = "nic"
properties = {
nictype = "bridged"
parent = "${var.host_bridge_network}"
}
}
}

resource "ssh_resource" "microcloud_init" {
host = lxd_instance.microcloud_nodes[0].ipv4_address
user = "ubuntu"
agent = false
private_key = file("~/.ssh/id_rsa")
when = "create" # Default
depends_on = [ lxd_instance.microcloud_nodes ]
file {
content = templatefile("${path.module}/templates/mc-init.tmpl", { instances = [for i in lxd_instance.microcloud_nodes : i.name], lookup_subnet = var.lookup_subnet, bridge_nic = var.bridge_nic, bridge_nic_cidr = var.lookup_subnet, ovn_gateway = var.ovn_gateway, ovn_range_start = var.ovn_range_start, ovn_range_end = var.ovn_range_end, microcloud_one_address = "${lxd_instance.microcloud_nodes[0].ipv4_address}/24"})
destination = "/home/ubuntu/init-mc.yaml"
permissions = "0600"
}
commands = [
"cat /home/ubuntu/init-mc.yaml | sudo microcloud init --preseed",
]
}
47 changes: 47 additions & 0 deletions demos/microcloud-deploy/templates/mc-init.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# `lookup_subnet` limits the subnet when looking up systems with mDNS.
lookup_subnet: ${microcloud_one_address}
# `lookup_interface` limits the interface when looking up systems with mDNS.
lookup_interface: ${bridge_nic}

# `systems` lists the systems we expect to find by their host name.
# `name` represents the host name
# `ovn_uplink_interface` is optional and represents the name of the interface reserved for use with OVN.
# `storage` is optional and represents explicit paths to disks for each system.
systems:
%{ for mc_inst in instances ~}
- name: ${mc_inst}
ovn_uplink_interface: ${bridge_nic}
storage:
ceph:
- path: /dev/sdb
wipe: true
%{ endfor ~}
# `ceph` is optional and represents the Ceph global configuration
ceph:
internal_network: ${bridge_nic_cidr}
public_network: ${bridge_nic_cidr}

# `ovn` is optional and represents the OVN & uplink network configuration for LXD.
ovn:
ipv4_gateway: ${ovn_gateway}
ipv4_range: ${ovn_range_start}-${ovn_range_end}
dns_servers: 8.8.8.8

# `storage` is optional and is used as basic filtering logic for finding disks across all systems.
# Filters are checked in order of appearance.
# The names and values of each key correspond to the YAML field names for the `api.ResouresStorageDisk`
# struct here:
# https://github.com/canonical/lxd/blob/c86603236167a43836c2766647e2fac97d79f899/shared/api/resource.go#L591
# Supported operands: &&, ||, <, >, <=, >=, ==, !=, !
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To cleanup the template, it would be worth removing the comments from the official docs together with any unsused keys.
If you want to learn more about the keys, you can always consult the official docs.

Maybe having a link to the docs at the beginning of the file would be enough.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense

# String values must not be in quotes unless the string contains a space.
# Single quotes are fine, but double quotes must be escaped.
# `find_min` and `find_max` can be used to validate the number of disks each filter finds.
# `cephfs: true` can be used to optionally set up a CephFS file system alongside Ceph distributed storage.
#storage:
# cephfs: true
# ceph:
# - find: size > 10GiB && size < 50GiB
# find_min: 1
# find_max: 2
# wipe: true
#
42 changes: 42 additions & 0 deletions demos/microcloud-deploy/variables.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required newline at the beginning of the file?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

variable "pro_token" {
type = string
}

variable "lxd_project" {
type = string
default = "microcloud2"
}
# TODO Remove this - it can be computed/derived.
variable "bridge_nic" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please can you derive it in this PR if possible?

type = string
default = "br0"
}

variable "host_bridge_network" {
type = string
default = "br0"
}


variable "lookup_subnet" {
type = string
default = "10.10.32.1/24"
}

variable "ovn_gateway" {
type = string
default = "192.168.254.1/24"
}
variable "ovn_range_start" {
type = string
default = "192.168.254.150"
}
variable "ovn_range_end" {
type = string
default = "192.168.254.200"
}
variable "ssh_pubkey" {
type = string
}

Loading