Motivation
Administrators of services need new servers fast. They do not want to wait until the master of the virtualization environment has time to setup their new machines.
The solution is automation. The administrators get access to the inventory tool and can click their new machines themselves. Clever automation in the background does the provisioning of the new machines. At best it takes only a few minutes to setup a new server in such an environment.
In this article I will describe the provisioning of RHEL 8 / CentOS 8 machines using kickstart. Other distributions have other means of automated install processes. It shouldn’t be too difficult to apply the setup described here to other Linux variants.
Setup
In our environment we use Netbox as the central Source of Truth tool. What is documented in Netbox should exist. To enforce Netbox' view of the world in the real environment we use Ansible. It uses Netbox as a dynamic inventory. Ansible’s netbox documentation describes how to use Netbox as a dynamic inventory.
Any administrator with access to the Netbox can add a virtual server that should run on a cluster also defined in Netbox. To add a new virtual server the amount of CPUs, RAM and disk space need to be specified.
Now Ansible can do its work in the background. Since we use RHEL / CentOS the tasks would be
-
Find the right cluster and node to set up the new server
-
Prepare kickstart
-
Prepare storage
-
Create the server
-
Initialize the server
Finding the Correct Cluster
When Ansible wants to create the new server it has to know on what device it should execute its tasks. The inventory data of the virtual machine only contains the name and ID of the cluster it should run on. So lets fetch these data:
- name: Get the cluster for virtual machine
uri:
url: "http://[::1]/api/virtualization/virtual-machines/?name={{ inventory_hostname }}"
method: GET
return_content: yes
headers:
Accept: application/json
Authorization: Token {{netbox_token}}
register: nb_vm
delegate_to: localhost
- name: Get a list of hosts in the cluster
uri:
url: "http://[::1]/api/dcim/devices/?cluster_id={{ nb_vm.json.results[0].cluster.id }}"
method: GET
return_content: yes
headers:
Accept: application/json
Authorization: Token {{netbox_token}}
register: nb_hosts
delegate_to: localhost
- name: Finally select one random host from that list
set_fact:
host: "{{ nb_hosts.json.results[nb_hosts.json.count | random | int].name }}"
Please note that Ansible stored the token to access the Netbox in the
netbox_token
variable. Normally you fetch it from a vault.
Preparing the Storage
We use logical volumes for the storage of the virtual servers. They are located
on a DRBD replicated device for high availability. In order
to create the logical device we need the name of the volume group on that
specific cluster node. The name is stored in a custom field of Netbox called
VGName
. Therefore we retrieve the information about the node we selected from
Netbox again.
- name: Get the VG name on host "{{ host }}"
uri:
url: "http://[::1]/api/dcim/devices/?name={{ host }}"
method: GET
return_content: yes
headers:
Accept: application/json
Authorization: Token {{netbox_token}}
register: nb_host
delegate_to: localhost
- name: Set fact of VG name
set_fact:
vgname: "{{ nb_host.json.results[0].custom_fields.VGName }}"
Creating the Disk with the Kickstart Information
The next task is to create the a disk in the host with the kickstart information. The following entries in the playbook will do:
- name: Create basic disk image
# dd_cmd is a variable defined: dd if=/dev/zero \
# of=/var/lib/libvirt/images/ks-"{{ inventory_hostname }}".img bs=1M count=1
# I found no other way to pass the special chars to the shell module.
shell: "{{ dd_cmd }}"
delegate_to: "{{ host }}"
- name: Create filesystem on ks file
shell: "mkfs -t ext4 /var/lib/libvirt/images/ks-{{ inventory_hostname }}.img"
delegate_to: "{{ host }}"
- name: Give the filesystem the correct label
shell: "e2label /var/lib/libvirt/images/ks-{{ inventory_hostname }}.img OEMDRV"
delegate_to: "{{ host }}"
- name: Mount ks filesytem
shell: "mount -o loop /var/lib/libvirt/images/ks-{{ inventory_hostname }}.img /mnt/ks/"
delegate_to: "{{ host }}"
- name: Copy kickstart file from template
template:
src: ks-template.j2
dest: "/mnt/ks/ks.cfg"
delegate_to: "{{ host }}"
- name: Umount ks file
shell: "umount /mnt/ks"
delegate_to: "{{ host }}"
Please note, that these tasks are delegeted to the node we selected before. The kickstart template, of course, also makes use of the variables defined in Ansible. Since the kickstart file heavily depends on the organisation, it would make no sense to show our complete kickstart file. To show the use of variables, the code snipplet below defines the network configuration of the new host as given in Netbox:
# Part of the ks-template.j2
# Network information
network --bootproto=static --device=ens1 --nameserver=2001:db8::53 --ipv6={{ primary_ip6 }}/64 --activate --ipv6gateway=2001:db8::1
network --hostname={{ inventory_hostname }}.sys4.de
Creating the Storage
Creating the Storage for the new server is no problem at all with the
information we gathered above. Ansible offers the lvol
module for this task.
- name: Create LV for new host
lvol:
lv: "{{ inventory_hostname }}"
size: "{{ disk }}G"
vg: "{{ vgname }}"
delegate_to: "{{ host }}"
Please note that the size also is derived from the definition in Netbox.
Creation of the New Server
The final step when creating the server is to define it in the virsh
environment and to start it. Within Ansible we can use the virt
module.
- name: Create VM
virt:
command: define
xml: "{{ lookup('template', 'c8-template.j2') }}"
autostart: no
delegate_to: "{{ host }}"
- name: Start new VM
virt:
name: "{{ inventory_hostname }}"
command: start
delegate_to: "{{ host }}"
The define
command of the virt
module takes a XML parameter, that can be a
template again. From an existing machine we exported the definition and modified
it with the variables we need. It will only show some interesting parts of the
definition, that show the use of variables.
<domain type='kvm'>
<name>{{ inventory_hostname }}</name>
<uuid>{{ (999999999999999999999 | random | string + (lookup('pipe', 'date +%s%N'))) | to_uuid() }}</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://redhat.com/rhel/8.0"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='MiB'>{{ memory }}</memory>
<currentMemory unit='MiB'>{{ memory }}</currentMemory>
<vcpu placement='static'>{{ vcpus }}</vcpu>
...
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/CentOS-8-x86_64-1905-dvd1.iso'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/{{ vgname }}/{{ inventory_hostname }}'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/ks-{{ inventory_hostname }}.img'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
</disk>
</devices>
...
</domain>
The number of CPUs, RAM and the disc(s) are defined in Netbox. The definition in the virtual machine just uses these variables.
Final Tasks
The final task for the new machine is to modify its definition to boot from the
new installation disc and not from the install CD. The work is done by a shell
command virt-xml
on the cluster node. The problem is the timing. The command
can only be fired after the machine started correctly. After that Ansible has to
wait until the kickstart process has finished its work and then can reboot the
machine.
These Ansible tasks are left to the reader as an exercise.
Feedback, improvments and other comments to ms@sys4.de.