Devops/Agile/Webscale….too many terms too little time: DevOPs Tools and Terms Cheat Sheet

I’m going to ignore any comments about DevOPs being a vague term like Cloud which means different things to different people.

To me it means trying to use tools and alternative project management approaches (Agile) to deliver repeatable things (applications/networks/virtual servers/physical servers) and continue to manage the configuration of those things after you’ve deployed them (not just running away).

At work I have recently started working more closely with our SDDC team there is plenty mention of a number of these tools and even the project management methodology of iteratively attempting small chunks of a project is new and interesting.  It’s sometimes hard not to slip back into old habits of meetings straying into chats, but breaking projects down into very specific/atomic tasks makes ownership clearer and status.

Either something is done or not done, it can’t be waiting on the next person.  If it is for them to do then it’s a task in their queue. (This is a variation on the Harvard Business idea of Who’s Got the Monkey https://hbr.org/1999/11/management-time-whos-got-the-monkey).

But I digress.  In the first of a series of posts intended to give busy people and/or managers a crib sheet for all the terms here’s a quick view of those that are bandied around in my daily work:

table1

32Gb vSphere Homelab stumbling block: Licensing dilemma

I upgraded my pc to a beefy spec just before Christmas.  i7 4770s giving me 8 threads to throw at VMWorkstation and 16Gb of memory, along with my existing SSD.

A quick reinstall of windows and install of VMWorkstation and I had a great lab to revise for VCAP-DCA510 in a hurry.  However I had promised myself that I’d set aside £120 this month to max out my motherboard with another 16Gb.  There’s another good reason to go that limit.  I didn’t pass 510 and the 550 exam has a different setup including more hosts and VMs/SSO boxes etc.

So I bought the 16Gb and it sat ready to be installed whilst home life happened and stopped me opening up my case.  I did that tonight.  Duly inserted the new memory and booted up.

No errors (time to put the side cover back on now!)….But WAIT….Windows is only showing 16Gb of memory.  However CPU-z dutifully informed me there were 4 banks occcupied and 32gb there.

DOH!!!! I suddenly realised, but had never really considered.  That I was running Win7 Home Premium.  It had sufficed for the last 5 years and like most others I had been in no rush to jump to Windows 8.

A quick visit to MS confirmed my suspicions.

https://msdn.microsoft.com/en-gb/library/windows/desktop/aa366778(v=vs.85).aspx#physical_memory_limits_windows_7

I’ll confess to doing a quick google to see if there was something as cheeky as a reg hack around this “oversight” it quickly became apparent there wasn’t (Kernel hacking isn’t something I really fancy).

So what were my options.

  1. Well the obvious one is Windows 7 Pro.  That is £129 now.  Not a fortune but a lot when you consider I bought all the hardware recently.
  2. Perhaps a spare MSDN license via work?  Well this wouldn’t technically be in the spirit of the license as I do use my PC for other stuff, not just testing
  3. A surprising option was upgrade to Windows 8.1 (Standard/Core/Basic) as when MS released 8 they also increased the memory Limit to 128gb!
    This option comes with a £99 price tag.  Hmm slightly easier to swallow but I don’t really need any of the other features
  4. The final option is a very precarious one.  Windows 10 Tech Preview.
    What!? I hear you cry.  Well as you will all have read this is coming free to Windows 7&8 users soonish anyway (some sites say June, others say months after) so I may be jumping the gun a little bit but it would give me 32Gb in windows…..However…….My licensed version of VMworkstation is 10 which MAY but most likely not run on Windows 10.  So I would need an upgrade to Workstation11.
    Cost? £109

I may test W10 TP on my HP Micrososerver and test if vmworkstation 10 runs on that…but it is frustrating that just to get around a single byte of code in the Win 7 Kernel which says 0x10 not 0x20 is going to set me back another £99 and an OS I’m not itching to jump to…..but ho hum such is progress!

 

I’ll update you on which route I decide

 

 

VCAP-DCA510: Failing to pass doesn’t mean failing to learn

Last Thursday I got the early train to Leeds stationed myself in a coffee shop and proceeded to stuff coffee and pastries into my mouth and last minute vSphere knowledge into my brain.

I had obtained a voucher to book the exam last year.  However I’d been seconded to a “Business Transformation” project.  Whilst this was quite interesting it did cause my technical skills to atrophy.  To cut a long story short I rescheduled the exam a couple of times.  However when a friend retweeted just before Christmas that 510 was going to be retired that was the end of that.  I had 3-4 weeks to revise/read/cram and lab.  Unfortunately Christmas isn’t the season conducive to revision.

So, having gotten my excuses in early, I proceeded to study.  I used the following resources which many others will have referred to and used before me:

  • The Blueprint (no brainer, but I will come back to this one)
  • The Unofficial VCAP Study Guide (as recommended by everyone and anyone)
  • The Official Cert guide (Not free but £25 well spent in my opinion, more again on this later)
  • PluralSight “Scale  & Optimise” online course (I got access to this the week before the exam)
  • Chris Wahl’s VCAP-DCA Study Sheet
  • Josh Andrews’ Test-Track Lab

In real terms I studied every evening for 2 weeks.  I spent a lot of time reading, with occasional visits to my lab PC.  That was sufficient time to absorb and understand all the topics on the blueprint.

There wasn’t anything on the actual exam that surprised, shocked or surprised me.  I’d read about it or practiced it all at least once.

Why didn’t I pass then? Well that’s simple.  Not enough lab time.

It sounds obvious but it really really can’t be overstated.  There were 26 tasks on the exam and 3.5hrs to do them in.  That’s less than 7 minutes per task.  Don’t forget it takes a few minutes to familiarise yourself with the exam environment and hostnames/ips.  So think 6 minutes per task.

Jason Nash in the Optimise and Scale (I’m pretty sure it was) said “Think of it as a customer asking you to do some work but they can only pay for 3.5hrs”

I would put it slightly differently, and this is where the blueprint comes in.

Think of the Blueprint (and definitely Chris Wahl’s Study Sheet/Checklist) as a list of things you will need to do for the customer.  It’s their requirements doc.  It’s not specific.  The IPs or hosts or networks or VMs details will be given to you in the task “on the day”.  But they have an outage window of 3.5 hrs with a hard stop.  You need to have done what they need in that time as users will be let back on the environment afterwards.

Given that premise then you need to be able to do each and every one of the Blueprint/Checklist tasks in 5 minutes.  Not think you can.  Not do it once.  I mean you need to drill yourself on it.
If the task is (and I’ll quote blueprint/checklist items here) change a pathing policy then all you should need it the LUN/Path/Disk detail and which policy.  Bosh.
If the task is migrate all of a host’s, or specific standard, switches to a DVSwitch then 5 minutes later. Bosh.
Set up FT? 5 minutes Bosh!

You get the idea.  If you have to think too long about it, you will waste too much time
(I do think it’s a bit harsh they take your watch off you when you go into the test room!)

A mini resource review is:

  • The many VCAP study blogs out there are great.
  • I would recommend the Official Cert Guide if £25 is in your budget.  It’s got a good structure to it and worth it if only for the practice scenarios it gives you. (Just make sure you get the additional ones from the online website using the code in the book)
  • I got my access to Pluralsight late.  I also got it free as a 2014 vExpert.  but I would really consider spending $29 for 1 month’s access.  Again it won’t break the bank and it’s an easy/convenient way to consume information
  • Chris Wah’s study sheet/checklist should form the basis of your practice tasks that you drill yourself on.  If you can’t do something on it.  Read up on how then practice it in your lab 100 times until you can do it in 3 minutes
  • Finally a massive shout out to Josh Andrews.  His test-track lab which he generously allows remote access to is a great resource that I can’t recommend enough.  It really gives you a feel for the exam and if you’re ready to do the kinds of tasks that you might encounter.

What next?

  1. Request authorisation for VCAP-DCA550 – DONE ✓
  2. Get a voucher/funding from work to book 550
  3. Forget all about AutoDeploy and learn about SSO, vSphere replication, Orchestrator etc*
  4. Revise the bits that are common to 510 and 550 that were highlighted on my score report
  5. Drill in the lab 100s of times until I can perform each task in the time it takes an egg to boil.

*Josh Andrew did a good post about the Blueprint differences here http://sostechblog.com/2014/04/07/vcap5-dca-update-for-vsphere-5-5/

Host Profiles to the Rescue!

I haven’t used host profiles much in anger.  I’ve seen them used as first snapshots of host config, but rarely updated (naughty I know!).

I’m currently in the midst of revising/labbing for my VCAP5-DCA. So I have my nested esxi lab environment running in workstation 10.  All was going well until I tried to test out some VMware Update Manager tasks.  I had installed VUM on a 2k3 VM running on one of my esxi hosts (I may break it out to run in workstation when I have more memory).  I could scan/remediate the host the VM was on, but I couldn’t scan the other host.  I tried vmotioning the VM.  It failed…due to a networking issue.  I cold vmotioned the powered off VM and started it on the 2nd host and VUM could scan it…but not the original host.

A quick run through was that the hosts could ping the default gateway (the VMWorkstation switch) they could even both see my Synology NAS and mount/map NFS/iSCSI LUNs.  But they couldn’t ping each other.  They could ping everything else and everything else could ping them.  They had just plain fallen out with each other!

I ran over the network config in workstation and there is very little to customise on a per-vm setting.  Plus they were just NAT’d exactly the same as the other VMs.
VMworkstation Virtual Network ruled out.

So it was something specific to both hosts.  Now foolishly I’ve been doing lots of lab exercises and not got my esxi hosts set to have non-persistent disks.  So somewhere in the last week or so I have changed the networking and not recorded/documented the random PowerCLI/esxcli/vma command which has stopped them talking.

Then I had an epiphany.  I don’t need to know what’s different.  I’ll just use vSphere/vCenter to set them the same!

Simple Solution

  1. Create host profile of HostA
  2. Apply host profile to HostB

And voila.  Ping was working!

hostprofile

(Note that there is an error but this can be ignored as storage was still visible/accessible on both hosts and is caused because the Host Profile captures storage settings which are unique.  Read here > http://virtualcloudzz.blogspot.co.uk/2013/06/host-profiles-compliance-error-in.html  )

The other message here is:

If you’re using a lab environment set your disks non persistent.

 If you’re in Production….don’t make a change without documenting it and creating a new host profile with similar level of detail/description to explain what changes have been made.

My Real-World VCAP Scenario to test scripting

I’ve been racking my brains thinking of a real-life VCAP-esque scenario that would force me to write a PowerCLI script.

I know Josh Andrews offered up “create 100 powered off VMs with a certain config or clones of an existing template” which is good but not something I’d normally do in my role.  Then one came to me.  So here it is

*****Disclaimer.  I have not sat the VCAP-DCA yet so have no visibility of the questions.  This question is based on a real-life scenario from my work but modified a bit.  It’s main purpose is to test/teach you PowerCLI so you get the hang of the main components of a basic vSphere PowerCLI script.  *****


 

Scenario 1

Your company has reviewed storage use and found that a large amount of storage has been taken up by a group of VMs with names starting VMP.
Upon further inspection an engineer identifies that the pagefiles ARE on a separate drive, which is a the secondary virtual disk, but they are using Tier1/Premium storage.
Your storage team identify sufficient space on Tier2 storage and present an empty LUN which is visible to all your hosts.

You need to ensure that page files on the existing VMs are on Tier2 storage.
A new LUN has been presented with identifier naa.6001405df3d2046d4a02d3fe4db20bd5
Use iSCSI01 as the new datastore name and  use host 192.168.88.211 to create the lun from.
All secondary vmdks were created using the default naming convention.
1) Create a datastore on the new LUN
2) Migrate all VMs second virtual disk for the VMP* VMs to the new LUN


This isn’t rocket science and on a small number of VMs you could easily drag and drop in the GUI. But let’s say it’s 100 VMs or 1000.  Then you really want to be getting home some time today!
Breaking down the first step.  We have a host and a device identifier for the lun.  We have a host to create it from and we have a name for the new datastore.  If some of these weren’t given then they could be found using Get-ScsiLun -vmhost 192.168.88.211

new-datastore -vmhost 192.168.88.211 -name iSCSI01 -path “naa.6001405df3d2046d4a02d3fe4db20bd5” -vmfs -FileSystemVersion 5
lunadd0

Our Datastore is created.  You can watch that happen in vSphere client if you like.
lunadd

Now we need to move all the secondary disks.
We know the VMs are all named with the prefix VMP
We know that their secondary disk follows the default naming convention.  e.g. vmname_1.vmdk
We know they need to be moved to Datastore iSCSI01

This snippet stores the name of the Datastore in one variable and then stores the harddisk details for any VMs hard disk where the VMname matches VMP* and the hard disk filename matches *_1*
It then invokes the move hard disk cmdlet to actually relocate the vmdks.

$myDS = get-datastore -Name “iSCSI01”
$myDisk = get-vm | where-object {$_.name -like “VMP*”} | get-harddisk | where-object {$_.filename -like “*_1*”}
move-harddisk -harddisk $mydisk -datastore $myDS

As seen from the command line, I notice I need to add a -Confirm $false to the New-Datastore command to stop the interactive prompt 🙂
lunadd3

And viewed from VI Client

lunadd2

So there we have it.  To make it a standalone runnable script, simply add 2 lines to the top.  One to add the Powershell Snapin for vSphere automation.  Then connect to a vCenter server.

add-pssnapin vmware.vimautomation.core
connect-viserver vc

$newLun = get-scsilun -vmhost 192.168.88.211 | Where-Object {$_.canonicalName -like “naa*”}
new-datastore -vmhost 192.168.88.211 -name iSCSI01 -path “naa.6001405df3d2046d4a02d3fe4db20bd5” -vmfs -FileSystemVersion 5

$myDS = get-datastore -Name “iSCSI01”
$myDisk = get-vm | where-object {$_.name -like “VMP*”} | get-harddisk | where-object {$_.filename -like “*_1*”}
move-harddisk -harddisk $mydisk -datastore $myDS -Confirm $false

I left out the bit where it took me an hour to see the iSCSI Lun because the networking on my virtual hosts went screwy…!  But otherwise it felt like a productive night!

 

Google doesn’t answer Forums/Communities posts…..YOU do!

Have you ever been here?

It happens, but it’s rare.  You might call it a Googlewhack or something similar but it isn’t.  Google, like a library doesn’t creates content, they merely make it easier to find other people’s work.  What’s really happened is you have found a post that the right person hasn’t looked at and answered.  Which is a shame…like an unloved child every question deserves an answer 🙂

 

Maybe that right person is you?  I’ve been contributing to the VMware communities forums recently https://communities.vmware.com/community/vmtn#src=vmw_so_vex_cneal_850 and I’ve found it interesting and satisfying for a number of reasons.

  1. You get to feel clever 🙂
    Rummage around.  No-one’s expecting you to go and buy a Brocade Fibre Switch and FC HBAs just to help another stranger, but there will almost certainly be a question you can answer.  Being able to explain the solution to another’s problem will help you demonstrate to yourself that you have the understanding of the topic.  Be it SRM handling unique UUIDs, how to run a script inside a VM by calling Powercli’s Invoke-Script.
  2. You can try out a solution on your homelab/work-lab set up.
    It may be that you read all about AutoDeploy for your VCP and passed with flying colours, but you’ve never exported a host profile in your life because your particular environment was inherited or maybe you haven’t got Enterprise Plus licenses in your Prod environment.
  3. You might get corrected by someone cleverer.
    I’m not ashamed to admit I answered someone’s query with a definitive “no that’s not possible in vsphere” only to have a guru type user with thousands of posts and solutions under their belt correct me and inform the person that “it is possible using the following method as of version x.y”.
    So free learning for me!
  4. Free Learning
    If you’re revising for an exam or certification then look in the topics you’re struggling on in the forums and by finding the answer for others, particularly as their problems are real-world, you are getting free exam scenarios/questions that you have to work out the answer to!
  5. You get to feel helpful and be part of the solution 🙂
    As I said at the top.  If the forums were just full of questions…they wouldn’t be much use and it wouldn’t matter how well indexed Google had the site.
    It takes someone to devote a little of their time to helping others.

I’m going to start aiming to do 2hrs per week on the forums minimum, 30 minutes here and there soon adds up.  Give it a try, you just might like it!

So think of it as digital altruism in a subject you are knowledgeable or passionate about.  VMware!

(You could become a forum rockstar like these guys https://communities.vmware.com/people?filterID=all&sortKey=all~statusLevelDesc&sortOrder=0&src=vmw_so_vex_cneal_850)

If you have any queries about the forums then contact VMware’s – Corey Romero https://communities.vmware.com/people/Corey_R?view=profile&src=vmw_so_vex_cneal_850

My vSphere HomeLab

Everyone’s got one, and like petrolheads showing off their v8 or GTI here’s my vSphere homelab.

Originally it was just an HP Microserver accessed via a pc running virtualised esxi hosts.

But I needed to upgrade my main PC and I still had a VMWorkstation 10 license from my VCP5 in June to use up.

So I went for a Core i7 4770s.  It’s quad core with HT so 8 threads for VMs to consume.  Currently only 16gb but it was only built in December and now Christmas is out of the way the first thing after January’s payday will be another 16Gb to take it to 32Gb.  I have a 256Gb Samsung Pro SSD in and run Windows 7 Home Premium +VMWorkstation 10.

I already had the SSD/PSU/Blu-Ray Drive so the upgrade for case/Mobo/CPU/Memory cost me around £500 which is a bargain considering I haven’t upgraded my PC for around 5 years and had been running an E6550 with 4Gb Memory.  Future Proofing doesn’t exist but I think I’ll get another 5 years at least out of this (once I add the next 16Gb) so £100/year for a Micro-ATX sized vSphere Lab+PC is good value in my book!

I will probably at some point recommission the HP N36L Microserver which has a puny CPU but has 16Gb too so I could do real, proper vmotions not virtual vmotions over virtually nested vswitches inside VMWorkstation (serious Inception style confusion).  However it doesn’t support Passthrough/Direct I/O so I’m not rushing on that one 🙂

To help understand iSCSI and NFS storage I use my Synology DS212j NAS with 2x1Tb Western Digital HDDs in, running mirrored (well it’s Synology Hybrid RAID, whatever that is 🙂 but with just two drives it behaves like RAID 1 and eats up 50% of your capacity).  I use the NAS for storing all my other personal stuff so it work quite well as a Lab NAS as well as a home NAS without one impacting on the other.  I’d really recommend it for Lab and home/media/backup/dlna use.  They even support VAAI which makes them a great choice for a vSphere lab.

Here it is as a Google Drawing (I don’t have a visio license!) alongside a photo of the kit.

HomeLabIMG_20150103_000850

I’ve been using it for a week or so now in anger as a lab to learn the VCAP-DCA510 objectives on.  I know I need to build/add a server for VUM (don’t really want or need to install it on the DC) and I need to create some more client VMs on the nested ESXi hosts, but for now it’s running fine and with NAT’d networking as I haven’t had chance to “Design” a network for the vSphere layer and isolate is and manage it properly (I’ll do that for the DCD 🙂

My VCAP-DCA exam is on the 15th Jan.  I got the voucher for VCAP-DCA510 in September but got seconded to a very very very very non-technical piece of work so my hands-on skills started to atrophy quickly.  Like many of us I postponed it as I knew the assignment was temporary.  It finished just before Christmas and when I came to reschedule I got a shock because I couldn’t.  Obviously, now, that is because it is being retired at the end of Jan.  So I am cramming in as much practice and focus as I can leading up to that and am taking some study leave the 3 days leading up to the exam.  I am fortunate that I only have to drive one hour to the test centre.  I have seen tales of people having to fly across the US or drive/fly around Europe.  However relatively speaking it is three to four times further than I normally travel to sit VMware/MS exams.  I would like to see VMware expand the locations in which you can sit the exam by working with testing partners to get their sites up to whatever spec is required.

Administrator Tools and Network Administration Done.
On to my next Chapter/Objective of the Official Cert Guide!

Storage Concepts