With a current work project, we're after servers with high CPU clock speeds to enable us to do a large amount of alorithmic operations as quickly as possible.
There are not many cloud providers out there that provide high CPU clock speeds (other than Vultr) at reasonable prices.
Instead, we decided to rent our own servers and, by utilising OpenNebula (KVM behind the scenes) to spin up Virtual Machines.
As with anything new, there's quite a steep learning curve, and lots of gotchas on the way, so here's my guide to how to build up a cluster of VMs that are performant, and have external IP addresses.
This guide is more for me to look back on again when I build more of these, but I hope it's useful to some.
Add the OpenNebula repo to your host:
cat << "EOT" > /etc/yum.repos.d/opennebula.repo [opennebula] name=opennebula baseurl=https://downloads.opennebula.org/repo/5.10/CentOS/8/$basearch enabled=1 gpgkey=https://downloads.opennebula.org/repo/repo.key gpgcheck=1 repo_gpgcheck=1 EOT
Add the EPEL Release repo too, and install all the necessary OpenNebula software
yum install epel-release yum install opennebula-server opennebula-sunstone opennebula-ruby opennebula-gate opennebula-flow
Install MySQL Server (which stores all the OpenNebula config), and secure it
yum install mysql-server chkconfig mysqld on service mysqld start mysql_secure_installation
And add a user
CREATE USER 'oneadmin'@'%' IDENTIFIED BY 'PASSWORD'; GRANT ALL PRIVILEGES ON *.* TO 'oneadmin'@'%'; SET GLOBAL TRANSACTION ISOLATION LEVEL READ COMMITTED;
Configure OpenNebula to use this MySQL connection:
vi /etc/one/oned.conf DB = [ backend = "mysql", server = "localhost", port = 0, user = "oneadmin", passwd = "PASSWORD", db_name = "opennebula" ]
We need to edit some default options to make sure that KVM will be performant for both disk IO and network IO
DEFAULT_DEVICE_PREFIX = "hd"
FEATURES = [ PAE = "no", ACPI = "yes", APIC = "no", HYPERV = "no", GUEST_AGENT = "yes", VIRTIO_SCSI_QUEUES = "0" ]
DISK = [ driver = "raw" , cache = "none" , io = "native" , discard = "unmap" ] NIC = [ model="virtio" ]
Let's now start up OpenNebula (the credentials to log in are stored @ /var/lib/one/.one/one_auth)
systemctl start opennebula systemctl start opennebula-sunstone systemctl enable opennebula systemctl enable opennebula-sunstone
Let's check that everything's configured OK
USER 0 INFORMATION ID : 0 NAME : oneadmin GROUP : oneadmin PASSWORD : fd59628069a3e8f5af7720e0ea9358ceb69be192070c27367c032f4e2d4bf1f3 AUTH_DRIVER : core ENABLED : Yes
The front-end is now built, let's build KVM
yum install opennebula-node-kvm yum install centos-release-qemu-ev systemctl restart libvirtd
Now add servers that you wish to use for host nodes to "known_hosts". Here I'm just using a single server:
ssh-keyscan localhost SERVERNAME >> /var/lib/one/.ssh/known_hosts
We need to setup python so OpenNebula can use it to run VNC.
We need to do two things - install Python, and also add a symlink to it:
yum install python36 ln -s /usr/bin/python3.6 /usr/bin/python
By default, OpenNebula uses a virtual bridge (virbr0) and then uses NAT so that your VMs have internet access.
The problem with this is that your VMs are shielded from the Internet, and you cannot connect directly.
If you need direct connections, you will need to amend your Ethernet configuration, and create a network bridge that OpenNebula can use.
Sometimes it takes a bit of trial and error to get everything perfect here - and if it's misconfigured you're likely to lose network connectivity, so make sure you have a Remote Management interface you can use to recover things!
Your network configuration scripts are held @ /etc/sysconfig/network-script
In our example, we have our Ethernet configured as eno1. You need to amend this config to say that it's going to connect to a bridge, and add a new bridge config to take on this config's network details.
Instead of trying to explain it all, I'll just include a before and after view so you can see what we changed. And please be careful, the contents here are case-sensitive!
cat ifcfg-eno1 DEVICE=eno1 BOOTPROTO=none ONBOOT=yes PREFIX=26 IPADDR=23.106.XX.YYY GATEWAY=23.106.XX.YYY DOMAIN=dedi.XXYYY.net DNS2=81.17.XX.YYY DNS3=184.108.40.206 DEFROUTE=yes
cat ifcfg-eno1 DEVICE=eno1 BOOTPROTO=none ONBOOT=yes TYPE=Ethernet BRIDGE=br0 cat ifcfg-br0 DEVICE=br0 TYPE=Bridge BOOTPROTO=none ONBOOT=yes PREFIX=26 IPADDR=23.106.XX.YYY GATEWAY=23.106.XX.YYY DOMAIN=dedi.XXYYY.net DNS2=81.17.XX.YYY DNS3=220.127.116.11 DEFROUTE=yes
With everything working, it's now time to connect to OpenNebula (http://SERVER:9869) and start to create some VMs.
You may need to open up your firewall to enable access to this port and VNC if you wish to use it.
First you need to configure your NIC. Go to Network -> Virtual Networks and create a new Network.
You'll need "BRIDGE" set to "br0" and the DNS and Gateway set as appropriate.
In the Addresses section you'll need to add in the IP range you wish to allocate.
Next you need to add some VM templates. Go to Storage -> MarketPlaces and download any templates you want.
Lots of the templates have SSH keys added in for authentication. If you want to set a root password go to Templates -> VMs and in the Template section add in a context of "PASSWORD" with a value as you wish.
Finally you need to create your VM. Go to Instances -> VMs and add a new VM. Pick the template you wish to use and configure as necessary. Make sure to pick the network interface.
If you're lucky, everything should build and run without issues.
I like to remotely access my VMs, so in the VM I had to amend /etc/ssh/sshd_config to allow remote root login:
PasswordAuthentication yes PermitRootLogin yes
You should now have external Internet connectivity and you should be happy. There are logs available @ /var/log/one/ to help diagnose any issues.
Finally, let's test performance. It's useful to test between the host server and a guest.
Here we'll be testing CPU, Memory, Disk IO and Internet connectivity.
yum install sysbench sysbench --test=cpu run sysbench memory run sysbench fileio prepare --file-num=10 --file-total-size=1G --file-extra-flags=direct wget -O speedtest-cli https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py chmod +x speedtest-cli ./speedtest-cli
If you're looking to add nodes to your cluster, it's particularly easy:
cat << "EOT" > /etc/yum.repos.d/opennebula.repo
yum install epel-release
yum install opennebula-node-kvm
systemctl restart libvirtd
ssh-keyscan HOSTNAME >> /var/lib/one/.ssh/known_hosts
scp -rp /var/lib/one/.ssh HOSTNAME:/var/lib/one/
chown -R oneadmin:oneadmin /var/lib/one/.ssh