How To: Export a VMWare VM to XenServer

This How To covers exporting a VMWare Virtual Machine to an existing XenServer installation.

A quick one today. I created a VM in VMWare Workstation and after testing some things decided I wanted to add it to my Xen cluster.

Luckily you can export machines into the OVF standard, which can then be imported natively by XenServer.

Select your Virtual Machine in Workstation and power it down. Suspend may also work but I didn’t test this. While still selected, select from the File menu -> Export to OVF and select a location to export the machine to.

export_in_progress

From the File menu in XenCenter click `Import` and select `Import OVF/OVA Package` and follow the steps.

import_xen

How To: Create an LVM

Home Infrastructure Part 3 – Creating an LVM

An LVM is a `Logical Volume Manager`. You can think of it as a system for managing how the space on your disks is used.

It’s sometimes beneficial to use LVM for your storage because you can’t always plan ahead for how large partitions should be sized. LVM means you can have a “virtual” storage pool, and add space when required to increase its capacity by adding new physical drives. You can move your virtual disks around and resize as you need to.

Some basic partitioning

First you want to find the physical device that you want to install LVM on to. A useful command for this is `lsblk`, this lists your block devices (usually things like USB sticks, hard drives but technically a block device is anything that provides buffered access to hardware devices. From the output you can see that there’s a disk at “sda” and a partition at “sda1” already.

Fdisk is a program that will let you partition your drives. The reason I choose fdisk is that it’s present on mostly all systems, although be aware that some distributions (FreeNAS in particular) don’t include versions that allow you to create GPT formatted partition table , in which case run parted.

Run the fdisk command on the disk (in this case /dev/sda).
0-lsblk

What we want to do in this case is create a GPT partition table. This is because the other commonly used type (DOS) has certain restrictions (a 2TB size limit on disks for one, but also a restriction in using the primary/secondary/extended paradigm if you want a large number of partitions).

Configure your disk partitions

Assuming it’s a fresh disk, OR you are happy that you will wipe the disk and it will be irretrievable, follow the steps below. A safe way to try this out would be in a Virtual Machine.


Pick the options :

  • ‘l’ to list your partitions (there should be none if you’ve just created a partition type)
  • ‘n’ to create a new partition. Pick the partition number (default is 1)
  • ‘t’ to set the partition type to Linux LVM (30)
  • ‘w’ to write out the partition table to the disk. You should now have a drive with one unformatted partition at its maximum capacity.

Note: we don’t want to create any more partitions at this point, as we’re just using it as a substrate for the LVM storage. More ‘partitions’ would be added through LVM itself as that’s the technology that manages the partitions.

Configuring LVM

So first we create the physical volume on the Linux LVM partition at /dev/sda1.
Then we want to create the Volume Group. You can think of this as a collection of Logical Volumes which themselves are a bit of virtual storage, like a partition, but that can be moved around and resized at will (well, with some caveats – namely the file system inside)

2-pvcreate

[root@archbang ablive]# pvcreate /dev/sda1                                 
  Physical volume "/dev/sda1" successfully created                         
[root@archbang ablive]# pvdisplay                                          
  "/dev/sda1" is a new physical volume of "40.00 GiB"                      
  --- NEW Physical volume ---                                              
  PV Name               /dev/sda1                                          
  VG Name                                                                  
  PV Size               40.00 GiB                                          
  Allocatable           NO                                                 
  PE Size               0                                                  
  Total PE              0                                                  
  Free PE               0                                                  
  Allocated PE          0                                                  
  PV UUID               sQ1aHi-0gVH-zlAL-MFvp-2vlg-2vDt-CbvtAa             
                                                                           
[root@archbang ablive]# vgcreate vg-root /dev/sda1                         
  Volume group "vg-root" successfully created                              
[root@archbang ablive]# vgdisplay                                          
  --- Volume group ---                                                     
  VG Name               vg-root                                            
  System ID                                                                
  Format                lvm2                                               
  Metadata Areas        1                                                  
  Metadata Sequence No  1                                                  
  VG Access             read/write                                         
  VG Status             resizable                                          
  MAX LV                0                                                  
  Cur LV                0                                                  
  Open LV               0                                                  
  Max PV                0                                                  
  Cur PV                1                                                  
  Act PV                1                                                  
  VG Size               40.00 GiB                                          
  PE Size               4.00 MiB                                           
  Total PE              10239                                              
  Alloc PE / Size       0 / 0                                              
  Free  PE / Size       10239 / 40.00 GiB                                  
  VG UUID               TbuJk8-gGeO-0lfG-MuOQ-7L5u-02Pq-ouAgda             
                                                                           
[root@archbang ablive]#                                                    

Creating your Logical Volumes

Create a Logical Volume, choosing the size (L) and the name (n) and the name of the Volume Group you want to add it to.

lvcreate -L 30G -n root vg-root

You could create a volume that spans the entirity of the available space, or create smaller ones and use them for each partition (/boot, /home, /etc)

Activating your LVM

The Logical Volumes can then be made known to the kernel using the command `lvchange`

lvchange -ay vg-root

This makes them available under `/dev/vg-root` (or equivelent vg name)

mkfs.ext4 /dev/vg-root/root # create a file system (make your own choice here depending on need)

Striping and Mirroring (if you like)

When you write your data to a logical volume, LVM decides how best to store your data on the underlying physical volumes that the volume group resides upon. As you can see above, this means we’re storing out ‘vg-root’ logical volume on the physical volume /dev/sda1 (the LVM partition we made earlier). Don’t worry if you get confused with the terminology, with practice it becomes clear.

Something a bit more advanced would be to create a Striped Logical Volume. This means you can intentionally add multiple physical devices to an LVM and then choose to stripe, which means data can be written and read in parallel with multiple physical drives but it also increases the chances of data loss. This can be thought of as being along the same lines as a Software RAID 1+0.

There’s also a Mirrored Logical Volume, which as the name implies maintains identical copies of the data written onto multiple physical drives. This means that recovery is possible if one of the underlying physical drives fails.

 

So there you have it, the LVM should be up and running.

USB 3.0 problems with Windows 10 on Asus motherboards

I’ve got a relatively old P9-X79 motherboard, and was getting all kinds of weird problems with some high speed USB devices (like the Kinect v2)

kinect
After a bit of research, and trying out different drivers I found this (non-official) Windows 10 drivers page.

Download the USB 3.0 controller drivers for “AsMedia USB 3.0”

Come on Asus, be better.

Latest ASUS drivers for Windows 10

How To: Move a docker container to another host

Saving this here for future reference.

If you wan’t to move a container that you’ve set up (i.e. from development to production) and you don’t yet want to push it to docker hub, here’s a way to move it:

  1. Export the container to a tarball
    docker export <CONTAINER ID> > /home/export.tar
  2. More your tarball to the new machine
  3. Import it back
    cat /home/export.tar | docker import – some-name:latest

Home Environment

Hardware
microserver

HP Proliant Gen8 Microserver
Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
12GB ECC RAM
1x256GB Samsung 840 PRO SSD for operating system volumes
2x1TB HDD for data storage
2x1Gb Ethernet Ports

Operating System

XenLogoBlackGreen
This rig is acting as a virtualisation host running on Arch linux with a Xen 4.5.0 kernel.

The plan was to build a powerful enough home server that I could run a decent amount of virtual machines on top of the hypervisor and have different virtual environments to suit different purposes (media server, home theater, dev environment, network / VPN gateway).

Maintaining them as independent virtual machines gives you the benefits you’d expect – being able to use one physical server to provide you with as many virtual servers as you need. Overprovisioning allows you to allocate more CPU and memory resources than you actually have, and have them dynamically scale up and down to meet the needs of the workload running at that particular time. Transcoding? give the majority of the CPU time and memory resource to the media server for instance.

Storage

Tying in nicely with the choice to go down the hypervisor route is the use of LVM (Logical Volume Manager). This is essentially virtualisation of your physical storage. It allows you to provision your physical disks, which don’t have to be identical in any sense, into a pool of logical volumes. These can be assigned to a single physical volume, spread across many, moved resized etc. at will.

Physically I’ve got 3 drives in this server, a 256GB Samsung 840 Pro SSD, and 2x1TB standard SATA HDDs.

The SSD was designated for the operating systems root partitions to be sure they are as performant as possible. The HDDs are carved up between 1TB of data storage, and 1TB for snapshots to back up the virtual machines and to provide non performance critical generic storage (swapspace, local storage).

Operating System(s)

xentop

Dom0 Hypervisor
Arch linux running Xen 4.5.0
Running as lean as possible with 512mb of allocated RAM and a 40GB root partition.

Guest #1 Media Center (HVM)
Windows 8.1 running as a media center
* PCI passthrough of HD6850 video card for 3d acceleration
* USB passthrough of wireless keyboard

Guest #2 Media Server (HVM)
plex_logo
Ubuntu server 15.04 running as a media server
* Sickrage for triggering downloads of new episodes of TV shows
* Couchpotato for doing the same thing with Movies
* Sabnzbd for downloading from newsgroups
* Transmission for downloading from torrent sites
* Plex media server for indexing media, and making it accessable outside of the LAN (transcoding when needed)

Guest #3 (Still to build) Dev Server (HVM)
docker

Arch server running as a development environment / host for docker containers
* Home automation server will run here (servicing voice commands)

Guest #4 (Still to build) VPN/Network Management Server (HVM)
Arch server running OpenVPN to allow connections in to home LAN.
* Network monitoring (nethogs, iftop, tcpdump etc.)
* Quality of service to dedicate a percentage of traffic to BitTorrent / Newsgroups – so we don’t have to manually configure each client with a limit, and to prioritise things like Skype/SSH/OpenVPN
* Default gateway / Firewall – all traffic from LAN will enter via eth1 and leave via eth0 to router.

Services

SickRage
CouchPotato
Sabnzbd
Transmission

Creating an Intelligent Home Part 1

So my favourite project of 2014 has definitely been a series of API’s to run on my home network.

This has allowed me to do things like issue voice commands through an Android app using the voice recognition APIs and have a system on my network action those commands (like “Download the latest game of thrones”) and have it automagically appear on my home media centre in the living room (running XBMC). This could be drastically extended though and that’s the purpose of this series of blog posts – planning how I’m going to create a smarter home, one that lets me monitor what’s going on inside (webcams, sensors), control the temperature, set up events to happen when I get home etc.

I want to move away from a set of disparate “hacks” and turn it into a standard piece of software that can run on commodity hardware, and has a nice user interface. Ideally it will run on a wide range of hardware, from a Raspberry Pi at the bottom end to a PC, MicroServer or blade.

Technical parts ahead!

Continue reading

How To: Compile Mono for FastCGI & Nginx in Ubuntu 11.10

So if you want to install mono from the comfort of apt-get you can look forward to something like version 2.10 which means that the latest and greatest MVC projects wont work, you’ll run into async issues, and a bunch of other stuff so it’s time to get down and dirty with the mono source code.

Get dependencies

sudo apt-get install build-essential autoconf automake \
bison flex gtk-sharp2-gapi boo gdb valac libfontconfig1-dev \
libcairo2-dev libpango1.0-dev libfreetype6-dev libexif-dev \
libtiff4-dev libgif-dev zlib1g-dev libatk1.0-dev libjpeg8-dev \
libjpeg-turbo8-dev libglib2.0-dev libgtk2.0-dev libglade2-dev \
libart-2.0-dev libgnomevfs2-dev libgnome-desktop-dev \
libgnome2-dev libgnomecanvas2-dev libgnomeui-dev libgnomeprint2.2-dev \
libgnomeprintui2.2-dev libpanel-applet-4-dev \
libnautilus-extension-dev librsvg2-dev libgtkhtml3.14-dev \
libgtksourceview2.0-dev libgtksourceview2.0-dev libvte-dev \
libwnck-dev libnspr4-dev libnss3-dev libwebkit-dev libvala-0.18-dev

Get the source

First we’ll need the following sourcecode:

  • libgdiplus: An Open Source implementation of the GDI+ API. [latest snapshot – see below]
  • mono: Mono open source ECMA CLI, C# and .NET implementation [release 3.2.5]
  • XSP: Monos ASP.NET hosting server. Includes an apache module, fastCGI module and standalone server [release 3.0.11]

note: if we get the latest release version of libgdiplus you may run into an issue during make

/usr/bin/ld: testgdi.o: undefined reference to symbol ‘g_print’
/usr/bin/ld: note: ‘g_print’ is defined in DSO /lib/x86_64-linux-gnu/libglib-2.0.so.0 so try adding it to the linker command line

getting the latest version linked above includes the pull request that fixes this issue.

Compilation

Untar/unzip each of the archives:

unzip master.zip # unzip libgdiplus
tar -zxf 3.2.5 # untar mono
tar -zxf 3.0.11.tar.gz # untar XSP

for each project the process is exactly the same (before you do this though, read the section on uninstalling below)

# change to project directory (mono in this example)
cd mono
# run autogen to configure the build
./autogen.sh --prefix=/usr
# build
make
# install
make install

You may run into missing packages depending on your system configuration, but usually apt-get install the missing dependency is all you really need to do.

Configuring Nginx and FastCGI

coming soon, no really…

Uninstalling

I have in the past stayed away from building software on my production server as it can be a pain to remove or upgrade but I’ve found a great tool called CheckInstall. What this does is keep track of the files being copied during the ‘make install’ step and gives you a handy .deb or .rpm file which you can then use to uninstall (or even use this file on other systems instead of having to repeat the build)

To use it, replace ‘sudo make install’ with ‘sudo checkinstall’

VMWare tools download link (and installing it in a Linux Mint guest)

While trying to get VMWare tools installed inside a Linux Mint guest I thought I’d look around to see if I could find an ISO that I could mount manually inside the guest.

Eventually I figured out that you need to select Ubuntu as your OS inside VMWare Workstation, otherwise if you have selected ‘Linux – Other’ as your OS the mounting process fails (or it possibly VMWare doesn’t even attempt it)

In case anybody else finds they need the Linux version of the tools, here’s the latest TGZ:
download link

(VMWare: please don’t sue…!)

How To: Fix Silex & Composer Autoloading – Class Not Found Error

I started a new project in Silex this weekend and spent a while trying to figure this one out.

Fatal error: Class 'App\Model\MyAmazingClass' not found in /var/www/app/public/index.php on line 18

So, silex depends on Composer for its autoloading functionality. I normally handle my dependencies myself and build stuff from source but this is a quick project, let’s give composer a go. So, off to the composer blog to get my composer.json set up. It tells me to add the following section:

    "autoload": {
        "psr-0": {
            "App" : "src"
        }
    }

Okay, now a “composer update” to register the new dependency with composer and I can see that this line has been added to AutoLoad_Namespaces.php:

    'App' => $baseDir . '/app',

Long story short, I was expecting this line to have mapped my ‘App’ namespace to the ‘/app’ folder and expecting that a model folder under /app with my class in it would be autoloaded (crazy right?). What composer needs you to do is create your ‘App’ namespace folder under the directory that you’ve already defined for it.

Anyway, hope this helps someone else out there.

VMware vSphere client download URL

Each and every time I want to manage my Esxi server from a different machine (laptop, work laptop, home PC) I have to find and download the VMware ESXi windows app. You should be able to navigate to the ESXi machine and download it from there but does it work? Does it hell.

So here (for my own convenience and that of anyone else that’s reading) are the download URLs.

Disclaimer: shamelessly borrowed from Chris Halls blog

vSphere v4.1

– VMware vSphere Client v4.1 : VMware-viclient-all-4.1.0-258902.exe
– VMware vSphere Client v4.1 Update 1 : VMware-viclient-all-4.1.0-345043.exe
– VMware vSphere Client v4.1 Update 2 : VMware-viclient-all-4.1.0-491557.exe
– VMware vSphere Client v4.1 Update 3 : VMware-viclient-all-4.1.0-799345.exe

vSphere v5.0

– VMware vSphere Client v5.0 : VMware-viclient-all-5.0.0-455964.exe
– VMware vSphere Client v5.0 Update 1 : VMware-viclient-all-5.0.0-623373.exe

vSphere v5.1

– VMware vSphere Client v5.1 : VMware-viclient-all-5.1.0-786111.exe
– VMware vSphere Client 5.1.0a : VMware-viclient-all-5.1.0-860230.exe
– VMware vSphere Client 5.1.0b : VMware-viclient-all-5.1.0-941893.exe
– VMware vSphere Client 5.1 Update 1 : VMware-viclient-all-5.1.0-1064113.exe