A six-part walk through the Neutron data path — from a VM's virtual NIC, across bridges and tunnels, to the wire and back. Read in order; each part assumes the one before it.
6 partsOperational notes from a production Ceph cluster backing Cinder, Glance and Nova — growing it, snapshotting it, unsticking it, and eventually deploying it with cephadm.
6 partsStandalone deep dives into the parts of Neutron that bite you in production: VNI ranges, port security, external networks and SR-IOV passthrough.
5 partsThis coming month marks the last running session of the Openstack User Committee. Over the years, the OpenStack community has grown with many operators being directly involved in the development lifecycle. In efforts to…
As you probably may know, ceph-deploy, the beloved deployment utility for CEPH, is no longer maintained. Cephadm is the new tool/package to deploy CEPH clusters. CERN has a pretty good intro PDF to it. Cephadm includes…
You may run into situations where you need to run pods on your K8s master node. If you’r using RKE, you need to taint two labels on the controller/master node. You can obtain the current labels using kubectl describe…
I’m happy to announce that I will be speaking at the upcoming Stackconf held in Berlin in June 2020. Stay tuned ! Read More
If you’ looking into serverless computing, you probably have bumped into Apache Openwhisk and Knative. Both are the opensource frameworks for serverless computing that allow you to deploy event-driven microservices,…
I’ve written this post as a draft sometime ago, but forgot to post it. The reason I looked into it was to find out how DPDK physically works as the OS/Device level and how it bypasses the network stack. So, when you…
Passthrough has became more and more popular with time. It started initially for simple PCI device assignment to VMs and then grew to be part of high performance network realm in the Cloud such as SR-IOV, Host-level…
Deployment tools for Openstack have become very popular, including the very well known Openstack-Ansible. It makes deploying a Cloud an easy task, at the expense of losing access to the insights of “Behind the Scenes”…
Openstack Neutron provides by default some protections for your VMs’ communications, those protections verify that VMs can not impersonate other VMs. You can easily see how it does that by checking the flow rules in an…
From the title, this is obviously a very common scenario that you may want to do. One thing that we rarely think about though is “backends” for the attached volumes when we create volumes. When you create a volume, the…
Openstack stores quota usage for tenants in the database in quota_usages table. Nova and cinder have by default their own separate databases and in each database you get a new quota_usages table. The structure of the…
Using CEPH as a backend for glance images has slowly become the default deployment methodology in many production deployments. It is usually as easy as creating a new pool in ceph ( glance pool) and creating a user to…
Cold migrations are an integral piece of any QEMU/KVM deployment. It’s cold or “non-live” as you have to power down the VM, move it to the new host and power it back up. Openstack follows the same procedure when it…
A tool that’s less known-about for cinder is cinder-manage. You might have run into it during upgrades. The most common use case is cinder-manage db sync This is normally executed during upgrades to bring the database…
The Openstack User Group for Southwestern Ontario is finally here ! The group will focus on allowing Openstack users in the SWO area to exchange experiences and knowledge of Openstack as well as a way to socialise with…
So, you’ve managed to deploy OpenStack in a production environment, and now you would like to make sure that your precious investment in hardware doesn’t get ruined by poor performance tuning. You might want to consider…
You might find yourself in a position where you need to restrict access by tenants to specific external networks. In Openstack there’s the notion that external networks are accessible by all tenants and anyone can…
If you run into an issue where a Cinder volume you attached to a VM can not be deleted even after detaching it from the VM, and when you look into the logs you find something like ERROR cinder.volume.manager ……. Unable…
You might find yourself in a scenario where you need to backup a CEPH volume attached to an Openstack Instance. CEPH snapshots come automatically to mind as the “state-in-time” solution. Once you take a CEPH snapshot,…
DHCP requests are broadcast requests sent by the VM to its boradcast domain. If a DHCP server exists in this domain, it will respond back providing a DHCP IP lease following the DHCP protocol. In openstack, the same…
If you are expanding your ceph cluster with extra nodes. You will need to prepare the node to have ceph installed and prepare the OSDs to be part of the ceph cluster. In order to do this, you can use ceph-deploy to…
So far we have only spoken about VM communication when they belong to the same network. But what happens when the VM has to communicate with another VM on a different network. The common rule of networking is that…
In the last post, we spoke about VM to VM communication when they belong to the same network and happen to get deployed to the same host. This is a good scenario, but in a big openstack deployment, it’s unlikely that…
In a physical world, machines communicate with each other without routers when they belong to the same network. This is the same case with openstack, VMs communicate over the same network without routers. When two VMs…
As we mentioned in the last post, traffic flows through a set of Linux virtual devices/switches to reach its destination after leaving the VM. Outbound traffic goes downward while inbound traffic moves upwards. The flow…
In order to identify how a VM communicates in Openstack, we need to look into how it is connected logically when it’s created. This will allow us to know the steps that the VM traffic will have to go through before…
