Home KVM Lab: Open vSwitch & VLANs

UPDATE 6/27/14:  Have you ever worked so hard in the lab to find a solution for a problem but could never replicate the solution again?  Well, that’s what happened with this post.  I have completely rewritten this article so that it can be replicated over and over. I promise this time.

Since I’m virtualizing my home networking lab into a few Intel NUCs, I needed way to build my network topologies.  VLANs have been the standard for separating broadcast domains for many, many years.  My home Ethernet switch provided me a way to connect my physical  lab devices and build the topologies I needed for testing.  Each device in my physical lab had a single connection to the Ethernet switch and I used VLANs to provide the interconnection between devices.  I simply trunked all the VLANs to each device and configured only the VLANs I needed on each device to provide the appropriate connectivity.

Removing the physical equipment from my lab didn’t remove the requirement for virtualized network topologies. I needed to bring VLAN support to my KVM environment.  I wanted to preserve many of the same functionality that I had with the physical lab. Open vSwitch is what I decided to use to accomplish that goal.

The Goal

Create multiple VLAN network that KVM guest can use to communicate with each other privately.

  1. Create VLANs on the underlying ethernet network and trunk these VLANs to the Ubuntu host.
  2. Pass the VLANs to the Open vSwitch.
  3. Let Open vSwitch manage the VLAN separation for the KVM guests.
  4. Allow KVM guests on different compute node communicate with each other over the network in separate VLANs.

A picture is worth a thousand words

kvm-ovs-goal

The Ethernet Switch Config

I’m using an Juniper EX2200-C, but you can use any switch that supports VLAN configuration.  I have a single out-of-band network and 5 virtual networks for use by my KVM hosts.

The EX2200-C VLAN configuration:

jack@switch> show configuration vlans 
home-net {
    vlan-id 10;
    l3-interface vlan.10;
}
lab-oob {
    vlan-id 20;
    l3-interface vlan.20;
}
vnet50 {
    vlan-id 50;
}
vnet51 {
    vlan-id 51;
}
vnet52 {
    vlan-id 52;
}
vnet53 {
    vlan-id 53;
}
vnet54 {
    vlan-id 54;
}
vnet55 {
    vlan-id 55;
}

The EX2200-C interface configuration for a single KVM host:

jack@switch> show configuration interfaces ge-0/0/8 
description Idaho;
unit 0 {
    family ethernet-switching {
        port-mode trunk;
        vlan {
            members [ vnet50 vnet51 vnet52 vnet53 vnet54 vnet55 ];
        }
        native-vlan-id 20;
    }
}

My management interface for the compute nodes is on VLAN-20. I didn’t want this traffic to be tagged with a VLAN-id for this VLAN. That is why it is only shown in the configuration as native-vlan-id 20. I’m passing six VLANs (tagged interfaces) to the compute node as well. So I have seven VLANs: one untagged (vlan 20) and six tagged (vlans 50-55).

Linux and Dot1q

The first things I had to do was install VLAN support on my Ubuntu 14.04 Hosts.  Dot1q is the common short name for VLAN tagging – it is short for the IEEE 802.1q standard that defines the virtual LANs on Ethernet Networks.

Install Ubuntu 802.1q support:

sudo apt-get install vlan

Load the 802.1q kernel module:

sudo modprobe 8021q

Allow the 802.1q kernel module to load during startup (for when you reboot):

sudo sh -c 'grep -q 8021q /etc/modules || echo 8021q >> /etc/modules'

That’s it, you can now configure the ethernet interface under Ubuntu.

Open vSwitch Installation

Installation of Open vSwitch is simple:

sudo apt-get install openvswitch-switch openvswitch-common bridge-utils

I’m installing a few extra Open vSwitch packages for future use. Eventually, I will install a open flow controller to make configuring my Open vSwitch more simplified. For now, I’ll just do all the configuration locally.

Open vSwitch Configuration

We need to configure the OpenvSwitch bridge and add the underlying Ubuntu interfaces to the OVS bridge.

Caution: You will need local connectivity to your Ubuntu Host this will break IP connectivity.
sudo ovs-vsctl add-br ovsbr0
sudo ovs-vsctl add-port ovsbr0 em1

Let’s check the configuration:

jack@idaho:~$ sudo ovs-vsctl show
7af354ba-313c-44d6-b997-e601145d76ed
    Bridge "ovsbr0"
        Port "eth0"
            Interface "eth0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
    ovs_version: "2.0.1"

We now have an Open vSwitch bridge named ovsbr0 configured on our Ubuntu host. Let’s get IP connectivity restored so that we can get back to work.

Configure the OVS Interface for IP connectivity

Next, we configure the interfaces on the Ubuntu host.  To keep this information persistent across reboots modify the interfaces file.

sudo vi /etc/network/interfaces

Here is my configuration for the Ubuntu host.  You only need to configure a single interface which will be the primary IP address and out-of-band network for host and guest operating systems.

# # The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual

# The Open vSwitch Bridge Interface
auto ovsbr0
iface ovsbr0 inet static
        address 192.168.1.10
        netmask 255.255.255.0
        gateway 192.168.1.1
        dns-nameservers 8.8.8.8

Note: You may need to reboot the Ubuntu host at this point.

Add the VLAN Networks to the Open vSwitch

Finally we can add the VLANs to the Open vSwitch ovsbr0.

sudo ovs-vsctl add-port ovsbr0 vlan50 tag=50
sudo ovs-vsctl add-port ovsbr0 vlan51 tag=51
sudo ovs-vsctl add-port ovsbr0 vlan52 tag=52
sudo ovs-vsctl add-port ovsbr0 vlan53 tag=53
sudo ovs-vsctl add-port ovsbr0 vlan54 tag=54
sudo ovs-vsctl add-port ovsbr0 vlan55 tag=55

We can verify that the Open vSwitch is now configured with the correct ports.

jack@idaho:~$ sudo ovs-vsctl show
7af354ba-313c-44d6-b997-e601145d76ed
    Bridge "ovsbr0"
        Port "vlan54"
            tag: 54
            Interface "vlan54"
        Port "vlan55"
            tag: 55
            Interface "vlan55"
        Port "vlan53"
            tag: 53
            Interface "vlan53"
        Port "vlan50"
            tag: 50
            Interface "vlan50"
        Port "eth0"
            Interface "eth0"
        Port "vlan51"
            tag: 51
            Interface "vlan51"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "vlan52"
            tag: 52
            Interface "vlan52"
    ovs_version: "2.0.1"

In a future post, I will show how to configure KVM hosts to take advantage of the new lab network that has been created.

 References:

Open vSwitch Frequently Asked Questions
Open vSwitch VLAN Configuration Cookbook

Moving Towards the Virtual

I recently rearranged my office to make better use of my floorspace.  In the process of moving furniture, I had to unplug, uncable and move my physical lab.  I have a bunch of SRX firewalls, switches, Juniper and Cisco routers.

In order to power all that gear, I had to install a dedicated 20A power circuit in my office to handle the power load.  There was also a considerable heat load when the equipment was turned on.  It simply took too much floorspace and power just to have the lab – and that was a cost I was no longer willing to justify.

Before cabling all my gear back up, I decided that I would virtualize my entire network lab.  Almost every vendor on the market is releasing virtual appliances, from firewalls to routers, ssl-vpns, and management stations.  Best of all, most vendors support VMware and KVM for their virtual appliance images.

In order to virtualize my lab I needed a compute node to do all of this on…

My Compute Choice

I initially built a Haswell based computer to be my VM Host.  The idea was to use VMWare ESXi as the hypervisior.  However, a single compute node was not what the rest of the world is using.  A real datacenter has many compute nodes running several guest VMs at a time, all managed from a common pane of glass.  I needed multiple compute nodes.  I was not about to pay for VMWare’s cloud software and building multiple “computers” was too expensive, so I scrapped the large Haswell computer for a better alternative.

My lab is now based on the Intel NUC.  You can pick these up for between $300-$600 dollars fully built out.  I think my NUCs ran about $500 from Amazon.com.  They are small, consume very little power, and are relatively inexpensive.  Here are the specifications of the NUCs that I am using:

Intel NUC D54250WYK Intel NUC D54250WYK, Mini HDMI, Mini DisplayPort, USB 3.0, Intel HD Graphics 5000, 4th Gen Intel Core i5-4250U
Crucial 16GB DDR3-1600 (PC3-12800) Crucial 16GB Kit (8GBx2) DDR3 1600 MT/s (PC3-12800) CL11 SODIMM 204-Pin 1.35V/1.5V Notebook Memory CT2KIT102464BF160B
64GB mSATA SSD MyDigitalSSD 64GB (60GB) 50mm Bullet Proof 4 BP4 50mm mSATA Solid State Drive SSD SATA III 6G – MDMS-BP4-060

I also had an old Late 2009 Mac Mini 2.53 GHz (P8700) Intel Core 2 Duo with 8 GB of RAM laying around.  I repurposed it as a compute node as well.
220px-Mac-mini-1st-gen

 

Since power and floorspace were my biggest concerns, I believe I have found a happy medium between cost, function and scale.  Each Intel NUC uses 15W each and the Mac Mini is between 14W(idle) and 110W. Eventually I would like to replace the Mac Mini with two more Intel NUCs.

My Operating System Choice

I started with VMWare ESXi on my large Haswell box. I couldn’t figure out a good way to manage multiple hosts with vSphere (on a Mac). Then, I started looking at OpenContrail for it’s SDN and multi-compute capabilities. I also wanted to run this natively without installing a guest OS (which would have guest OSs) in ESXi.

OpenContrail uses CentOS, but I’m not a big CentOS fan so I started messing around with several different Linux distro’s. Currently for my lab, I have settled on Ubuntu 14.04 LTS.

My Other Lab Components

To round out my lab network, I have a Juniper EX2200-C and a Synology DS-411 with 8TB of disk.

Synology DS411 EX2200-C

Final Thoughts

This will be the basis for my lab going forward.  The age of network appliance virtualization has come to the networking world.  There will be a need for both custom silicon and VMs to deliver the next-generation networking.  I’m going to use the virtual network appliance in my home lab.

I started working with OpenContrail & OpenStack, but I was having trouble making it work across multi compute nodes.  I realized that I had no clue how the underlying components worked in unison.  In the end I went back to the basics and simply installed KVM with Openvswitch.  I’ll build up my knowledge from there to a full SDN solution.  I just finished my KVM install and will post the details of my trials and tribulations later.

Virtual Lab New Sexy Virtual Lab
Old Lab Old Huge Power Hungry Lab

Junos PyEz Setup for OS X Mavericks

I’ve been working with the Juniper PyEz Framework to help with automation tasks of Juniper Networks gear running Junos.

There was some process that I had to follow for me to make the PyEz framework work on my Mac running OS X Mavericks.  I have created a document and published to Google Docs for your viewing pleasure: Junos PyEz Setup for Mavericks.

osx:~ me$ python
Python 2.7.5 (default, Aug 25 2013, 00:04:04) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin

If you are stuck getting the Junos PyEz Framework working on Mavericks, check out my document and see if that will help.

Notes:

  • I had a clean install of Mavericks, with only the included version of Python for OS X.  If you have changed anything from the defaults you may run into problems.
  • You have to install the Xcode command line tools
  • I understand there are many ways to install the python packages. I choose the way that worked for me.  Do what you must.
  • If you foobar your system – it’s on you.

 

ATT’s Network of the Future and Domain 2.0 White Paper

I recently came across a white paper published by ATT that describes  their Network of the Future, which they are calling a “User-Defined Network Cloud”. The white paper,  ATT Domain 2.0 Vision,  clearly defines some excellent concepts that many of us may have been already thinking about.

I would suggest reading the paper for yourself, but I will highlight some of the parts that really spoke to me.

You have to start with Network Functions Virtualization (NFV)

While the datacenter virtualization trend is in full swing, network virtualization is just beginning.  Routers have had “virtualization” capabilities for many years, but what about load balancers, firewalls, content delivery boxes, session border controllers, and WAN acceleration to name a few?  Network Function Virtualization Infrastructure (NFVI) is the term they are using to differentiate tradition compute and storage components from the network components.  A virtualized network appliance is referred to as a Virtualized Network Functions (VNF).

While routers have virtualization capabilities, they don’t have a good tie-in to the open orchestration systems.  Trending as well, is the need for vRouters on the hypervisor verses the traditional vSwitch. All network functions, from layer 3 (routers) to Layer 4-7 (network services appliances,) will have to have a virtualized component that can be leveraged in new data centers.   ATT summarizes their vision of NFV in their white paper on page 8:

NFV aims to address these problems by evolving standard IT virtualization technology to consolidate many network equipment types onto industry standard high volume servers, switches and storage that can be located in data centers, network PoPs or on customer premises.

Software Defined Networking (SDN)

ATT’s SDN vision builds upon NFV.  One the network functions are virtualized, they can build more open and flexible networks.  The SDN portion of the white paper is pretty standard stuff.
The most relevant SDN quote that applies to 95% of folks looking into SDN is on page 13:

Create a Flexible Fabric for Data Centers and NFV Infrastructure – Use SDN to create virtual network capabilities within an infrastructure fabric, remove middle boxes, and provide customer control of private LAN capabilities.

I wouldn’t skip reading the whole section, though.  ATT’s vision for SDN is comprehensive and extends to most of their business interests including virtual CPE.

Orchestration Control and Policy Management 

The next major of the white paper covers orchestration and policy management topics.   Again, this is more-of-the-same SDN speak, but I believe ATT does a good job identifying the Key Operational Shifts (page 18) with a move to SDN.

Obvious Operational Shifts are:

  • A shift from hardware centric to software centric
  • Highly constrained, independent & disaggregated control planes to Highly integrated & automated control planes driven by customer & operator policies.
  • Separation of service elements & support systems to Integrated orchestration, automation & virtualization.

Not-so-obvious Shifts:

  • Faults as service failures to Faults as capacity reduction events.
  • Optimized provider network & ops process to Optimized customer experience.
  • Quarterly software releases to Continuous software process – “sandbox.”

My Summary

To summarize my thoughts on ATT’s Domain 2.0 Vision, I would have to say the Network of the Future white paper is a must read.  Instead of a nebulous white paper on vanilla SDN concepts, this is a real VISION of a real company and lays out their plan to achieve the next instantiation of the network.  If you are trying to develop your new network vision, read this now.

Troop Guides – Building Future Youth Leaders

This year I took on the role of Assistant Scoutmaster of Troop Guides for my son’s Troop.  My goal was to develop a program that placed more responsibility for the training and development of our new scouts to the more experienced older scouts.  Starting this year, I made the Youth Position of Troop Guide a one year commitment and offered a training class to bring my Youth Leaders up to speed on the new program.   My hopes are that the Troop Guides for 2014 will become the new Troop Guide Trainers for 2015.

I found a great quote from Scouting’s Founder that sums up my goals for this program.

“Scouting is a game for boys
under the leadership of boys
under the direction of a man.”

-Baden-Powell

Objectives of the Program:

  • Help our new scouts achieve the First Class Rank
  • Teach core Scouting Skills
  • Build a cohesive Patrol that will be the foundation for the next generation of leaders in the Troop and in life

I set the following requirements for my Troop Guides:

  • Must have the Star Rank
  • Must be active in the Troop and willing to attend most Troop meeting and outings
  • Agree to a one year commitment as Troop Guide

This is the first edition of the Troop Guide Curriculum, but I have great expectations.  We’re two months into the program and everything is running great.  I already have some changes for next year, but it is more concerned with program flow than content.

You can get my Training PPT deck and  my Curriculum off of Google Docs.

A Journey to Learn Python-foo

I’ve been spending quite a few months trying to round out my skills ahead of the next BIG thing in networking.  It’s all the rage and I bet there is not one vendor, trade rag, or networking blog that doesn’t have something to say about SDN, Automation and Orchestration.

Over the Christmas break I started on a journey to learn Python.  I’ve done some HTML, ASP.net, PHP, VB in the past, but I never really used any of it to stay proficient.  The real key is going must be using Python for more than just networking and automation.  For me, coding is not like riding a bicycle – I can’t jump back on and start riding again.  I have to ramp up and refresh my knowledge.  I think that’s all about to change because I’m in this for the long haul.

So where did I get started learning Python?  I started with Code Academy’s Python Course.  I didn’t buy any books (or a dozen) like I normally do.  I simply started using the awesome resources on the web.  This course took me about a month to complete.  While it started out pretty basic, by the end the content was challenging.  The Code Academy Python Course was perfect to getting my getting my feet wet with Python.

The next thing I did was to try and put some of that new knowledge to the test. I picked up a copy of Violent Python.  I’m not a security Engineer, but this book helped me to understand accessing networked resources and that is what I do want to know.  I had a lot of fun working through the exercises.

Recently, I’ve been playing around with Juniper Networks GitHub repository to stay on my learning journey.  I’ve also been keeping up with a little known resource – the Juniper Techwiki.  Check them out, they are good resources.

Thunderbird Energetica Energy Bars

I got my hands on a sample pack of energy bars from Thunderbird Energetica.  They are fantastic.  So good in fact, it’s taking considerable willpower to prevent me from eating the entire box of 15.  I picked up the Assortment box of the original three flavors: Cherry Walnut Crunch, Cashew Fig Carrot, and Cacao Hemp Walnut.

About the best energy bar I usually ate was a Clif bar, but their first ingredient is SUGAR (brown rice syrup).  Thunderbird Energetica bars are plant powered.  Just straight up natural, raw and pronounceable ingredients.

The Cashew Fig Carrot bar has the following ingredients: Organic Dates, Organic Cashews, Organic Figs, Organic Carrots, Organic Nutmeg, Organic Vanilla, Pink Bolivian Rose Salt.  At 150 calories with 25 coming from fat, this is one of the best bars on the market today. Did I mention how good it tastes?

Two Banana & Oat Recipes

I’m always looking for good treats.  Bananas and oats can be combined to make almost any (semi)healthy treat.  These two recipes can be adapted to avoid most food allergies.

These recipes can be found all over the web, but these are two I have made and adapted for my diet.

Banana Oat Cookies

Ingredients:

  • 3 ripe bananas
  • 1/3 cup apple sauce
  • 2 cups oatmeal (I like Bob’s Red Mill)
  • 1/4 cup almond milk
  • 2 tbsp. Chia Seeds (ground and soaked 6 tbsp. water)
  • 1/2 cup dried fruit (optional)
  • 1 tsp vanilla
  • 1 tsp cinnamon

Instructions:

  1. Combine all ingredients except oats
  2. Add oats and stir
  3. place cookies on parchment paper covered cookie sheet
  4. Bake at 350 for 15-20 minutes

Banana Oatmeal Bars

These are my goto “power” bars

Ingredients:

  • Two ripe banana’s (large)
  • 1 cup brown sugar
  • ¾ cup whole wheat flour
  • 1 tbsp. Vanilla
  • Cinnamon to taste
  • 2 tbsp. Chia Seeds (ground and soaked 6 tbsp. water)
  • 1 tsps. Baking Soda
  • 2 cups Oatmeal (I like Bob’s Red Mill)
  • 2 cups Rice Krispies (whole grain/gluten free variety if you can find them)
  • 1 cup Dried Fruit (like cranberries)

Instructions:

  1. Mix bananas and brown sugar until liquidy
  2. Add whole-wheat flour, soaked Chia seeds, baking soda, vanilla, and cinnamon and mix until well incorporated with banana and sugar mixture.
  3. Add oatmeal and mix
  4. Add Rice Krispies and dried fruit (do not over mix).  You want an even distribution of the fruit in the mixture
  5. Place parchment paper in a 9×13 glass baking dish
  6. Add banana bar mixture to pan and spread evenly
  7. Bake at 350 for 20-25 minutes
  8. Allow to cool and cut into squares

Notes

  • Chia Seeds add essential Omega 3’s
  • Use gluten free oats, flour and Rice Krispies to make gluten free.
  • No added fat, dairy, oil, or animal products
  • A little non-dairy milk can be used thin out the mixture if you have small bananas
  • You can cut back the brown sugar to ¾ cup without problems
  • The Rice Krispies help build volume to the bars

High Hopes and Consolidation

I’ve been writing in a few places, but as of late it has been too complicated to keep track of and update the various places and projects.  I’m consolidating my online presence to a few specific spots.  All my blogging will end up here at JackWParks.com.

The Network Engineer Digest (link will eventually break) will go away – but I may move a few key posts.  I don’t know yet. After moving the Network Engineer Digest multiple time and trying to find reliable IPv6 hosting, the posts are just fubar.  Drawings are missing. Text formatting is messed up. I’m cutting my losses.

I’ve been spending a lot of time developing my son’s Boy Scout Troop website and Wiki page.  If you are a scouter, check out the Troop 279 Wiki page.  I created/compiled a lot of content and there is still much more to do.  It has been my side project for about 6 months now.  I’m always adding more content and trying to make it accessible.

I run the risk of having too much content on too many different subjects in one place, but that’s the way it’ll have to be for awhile.

 

 

 

127.0.0.1 –> ::1