Theres logging, then there’s Graylog.

Monitoring systems usually tell you when there’s a problem and what the problem is, but logs can tell you about the problem, what happened before, and what happened after. In other words logs provide a critical source of information when anything happens in your environment, from the seemingly mundane (NTP update) to the more terrifying (all paths down).

It’s always a good idea to collect logs is some form or another, being able to look through historical logs or requests from support people allows you to start looking for the cause, or a pattern. Usually a log entry gets sent from a system to a syslog box and gets added to the log file created for that system. Not only is it a good idea to have external logging for your ESXi servers but you should also log the VCSA/PSC and any supporting infrastructure, eg “first hop” switches, storage, etc… It still surprises me that many companies don’t actively do this.

Capturing all of that creates a huge amount of data which can be very labour intensive to sift through to get what you need. Apart from something to capture the logs, you should also look at a log organiser, something like vRealize Log Insight, which I really like and have marked to blog about at a later date, or Splunk. Unfortunately both solutions come with a price tag.

And this is where Graylog comes in. “Graylog is a fully integrated open source log management platform for collecting, indexing, and analyzing both structured and unstructured data from almost any source.” 

All graylog deployments will have the same basic features: Graylog server, Web interface, Elasticsearch, MongoDB.

../_images/simple_setup.png For a first look I would strongly recommend the Graylog appliance that’s distributed as an OVA. The beauty of the OVA is that is can be deployed as an all-in-on solution for smaller deployments, or configured for a single component via the graylog-ctl script for larger workloads. For those of you asking it does have beta support for Docker.

The interface is similar to other loggers, which is not a bad move in my opinion, as its what works best. scereen_capture

Searches are snappy and respond quickly. The query syntax is simple and doesn’t require you to have a degree in programming. Type esx01 and it will return everything with esx01. Type esx01 esx02 and it with return all entries with esx01 or esx02. But place the two in quotations, “esx01 esx02” and it will look for the exact phrase.

Dashboards are highly customisable and very easy to setup the one below was based on vsan for a rolling 5 minute window and took a couple of minutes to setup.dashboard_vsan_5min

There are a few good free plugin’s and content packs that will help customise the product. API’s are exposed so that you can write your own.

Support is done through the community but can be purchased at three levels, with different SLA’s for response and different ways to contact the company. I’m not sure of the cost but I guess much of that would be around the size of your deployment.

If you don’t want to use the OVA, graylog also has official deb and rpm package repositories for Ubuntu, Centos  and Debian, which make it easy to install with two or three commands. I tested both the OVA, and the package install on CentOS. Both methods were really simple to deploy.

Graylog should be a serious consideration for any company, big or small, and is very good example of an enterprise opensource project.

Nested Home Lab – Part 14 – Enabling VSAN.

We’ve done all the hard work. Now you’ll see how easy it is to actually enable VSAN. Once this is done you’ll have a lab ready to go.

  1. Log in using an account that has permission to configure the environment.Lic-1
  2. Select Home and Hosts and Clusters.AH-1
  3. Expand the Cluster you want to enable VSAN on. Select Manager, Select Settings,  Select (under Virtual SANGeneral, Select Edit.Add_VSAN_01
  4. Check Turn on Virtual SAN. Change the Add disks to storage to Automatic. Click OKAdd_VSAN_02
  5. You should now have a functioning VSAN cluster. Add_VSAN_04

NOTE: You will initially see the error below. Its normal and should clear in a minute or two. If it doesn’t you need to go back and edit your networking.Add_VSAN_03And that’s it. You can now load some VM’s and start testing. In earlier posts I recommended a couple of Linux distro’s, however I have been playing about with Photon and its a really good light weight alternative.

Have fun labbing!

VMworld – Day 4

VMworld has drawn to a close; battle wary attendees slumped in the corridors, faces bathed in the soft light of a million pixels,  or stumbling around looking for that one last elusive caffeine high.

Its_over

I arrived early to take a few minutes to get myself organised, and look at the notes from the previous three days. I had a few questions that I wanted to put to some of the VMware experts.

Session:  How to Manage the Health, Performance and Capacity of Your Data Center, using vSphere with Operations Management

Speaker: Himanshu Singh and Hicham (he-sham) Mourad

Unfortunately, this was rehash of their session from the day before (vRealize Operations Insight: Manage vSphere and Your Entire Data Center).

Session:  Horizon View Troubleshooting – Looking Under the Hood

Speaker: Matt Coppinger, Alex Birch

A good talk from guys in the field. Both of them are Brits so it felt like home. The two had an interesting dynamic, one would start and the other would elaborate. I’m working with View at the moment so this gave me some really good ideas.

Session: A First Look at vSphere Integrated Containers and Photon Platform

Speaker: Dan Wendlandt

I decided to round off the day with a DevOps session. I understand containers and I get why they are important to development but wow did this session stand out as one of the better ones for me. The demo of this technology was really cool. Photon was also talked about. Nicely done presentation.

4

After that Dimos and I went back to the solutions exchange and cornered Tomas a senior systems engineer from VMware and got him to show us everything we could think of about vROI. 60 minutes later Dimos left and 30 minutes after that I let Tomas get back to the other delegates.

My flight was at about 8pm so I got to the airport really early and amused myself by playing spot the VMware bag (36 if anybody’s interested).

VMworld – Day 3

Yes, yes, I know the post is late. I’ll do day 3 and release day 4 immediately after this one. My excuse? Family time.

Anyway back to Day 3.

Today, I was a really busy day. I had resolved to meet with a few people on the solution floor as well as attending as many sessions as I could.

Session: General Session

Speaker: Various

The two General sessions are worth a watch You can see them here

NSX 6.2 got a big mention with a really interesting customer story (look out for the customer with the funky jacket).

2A large part of the focus was talking about where VMware want to push NSX going forward. This is a really interesting space. While NSX can be very complicated, I believe bits of it going to make their way into other products.

Pat Gelsinger also can onto stage. What he had to say was quite interesting about companies needing to change and move forward or face being left behind. However the big elephant in the room wasn’t really talked about EMC/DELL. I was a bit disappointed but in hind site I agree that it wasn’t right to talk about that to the conference attendees before you spoke to the employees.

Session: Solutions Exchange

Dimos and I hit the solutions exchange. There were a few vendors I wanted to see and look at their products, Solidfire being one and HGST being another. Take a look at HGST if you get the change. Their products are very mature and very well priced. What was nice to see a lot more smaller companies participating this year too.

Session: Virtual SAN Customer Panel

Speaker: Rory Choudhuri

Another good session but this time with customers themselves giving updates of their experience with VSAN. This was very enjoyable and I managed to participate quite a lot.

1

Session: Stretched Clusters with Expert

Speaker: Lee Dilworth – Principal Systems Engineer , VMware

A much smaller meeting with seven of us and Lee. This was in the experts lounge and Lee took us through what to look out for in Stretched clusters, SRM, Stretched VSAN. Nice participation with the attendees.

Session: vRealize Operations Insight: Manage vSphere and Your Entire Data Center

Speakers: Himanshu Singh and Hicham (he-sham) Mourad

Good talk. Big focus on Log insight and the benefits of intelligent analysis. Both speakers had a good raport with their audience.

Session: VMware Virtual SAN – Architecture Deep Dive

Speakers: Christian Dickmann, Rawlinson Rivera

I was quite interested this. I wanted to see what had really changed under the hood between 5.5 and 6.1. Well it turns out a lot has changes. Both speakers really know their stuff but i was very impressed with Christian. Looking at how the snapshot process has changed (for the better) and the new Virsto on-disk format was quite cool.

After this I headed back to the sister-in-laws. I decided to give the VMworld party a miss and take the sister-in-law and her boyfriend out for dinner to say thank you for putting up with me for the past few days. We had dinner at a little Japanese restaurant called Yen in Gracia.

 

VMworld – Day 2

So we are now into day 2. My feet hurt.

Its been an interesting day. Firstly the EMC takeover by Dell has overshadowed much of the conversations. I’m not really sure what to make of it and how it will affect VMware.

I took it easy on the drinks at the vExpert party so it wasn’t too bad to get up and into the center. Transport to and from the conference center is easy enough for me so, the journey being only 5 minutes door to door.

Session: Keynote.

Speaker: Various

Nobody really expected Pat Gelsinger to make an appearance as it was very clear he was still working on the EMC deal. There was a video from Micheal Dell which tried to sooth VMware’s customer fears. The big take away for me was a drive to containers, and cloud. Project Photon got a mention as did the new buzz term, “Cloud Native Apps”

A few people complained about the keynote not really having anything revolutionary. Well that is to be expected between major releases. Last year saw the announcement of vsphere 6.0 and a few other projects. This year felt more about encouraging customers to explore those and see if they would be a fit for their business.Keynote

Session: The New vRealize Converged Blueprints.

Speakers: Kal De and Raghavendra Rachamadugu

This was interesting for me. Kal and Rag are very obviously knowledgable about their vRealize and gave a killer demo. unfortunately some speaker prompts were missed in the demo and there were a few silences while the mouse whizzed around the screen without an explanation. Still a successful presentation and, for me, showcased the power of blueprints.Session1

Session: VSAN Pioneer Summit

Speakers: Parag Patel, Christian Dickmann

Very interesting discussions about the future of VSAN and some great comments from other VSAN customers.

Time: Exploring the partner exchange

Lots of different vendors, old and new. EMC, Dell, Fijutsu, Netapp very large players as usual. Also some of the smaller ones too, HGST, SUSE. I got my vExpert hoodie, Thanks Simplivity.

2015-10-13 14.09.43

In the evening I met up with Dimos and headed to the Europe customer party at bestial. Beautiful venue.

2015-10-14 07.16.21

VSAN is fast becoming a passion of mine and its a big focus for me at the conference this time. I thing that this is has the potential to be the most disruptive tech in the last few years, bigger than NSX IMHO. I’ll also be looking into vRealize and View while I’m here.

My feet still hurt.

 

VMworld – the first 24 hours (Day1)

Its been much busier for me this year than previous years. From vRockstar to vExpert.

After arriving yesterday, dropping my bags at my sister-in-laws, I headed out to the vRockstar party at the hardrock Cafe. It was a lot of fun. I ran into Eric sloop of ntpro.nl fame. Really nice friendly guy, actually that’s one thing I can say about the VMware community leaders, all really friendly and easy to talk to. Also caught up with Chris Dearden and the ever charismatic Mike Laverick. I manged to get out of there at about 23:30.

2015-10-11 23.25.35 vRockstar1

 

 

 

 

Waking up the next morning was a bit of a challenge. I did manage to make it to the conference center for 08:30 though to get a couple of labs in before the TAM day started. The Labs were top-notch as always. A slight lag, but it was a good experience over all. Word of advice though, best bring your own laptop if you can as it got filled to capacity very, quickly.2015-10-12 09.56.00

I manged to get two sessions in and one meet the vexperts today:

Session: Ask the Experts Lunch

This was quite a lot of fun. There were about 18 tables, each with a subject matter expert. This allowed you to hop tables and speak to the various experts about almost anything. Managed to talk about storage, cloud aware apps, EVO:RAIL and a couple of others. One thing that stood out was the enthusiasm of the experts for their technology.

Session: Streamlining Data Center Operations, Real World Experience.

Speaker: Colin Fernandes.

This session got off to a bit of a slow start in my opinion but was very useful. The speaker undoubtably knows his stuff. There was a big focus on Log insight. All but one of the real world example were very relevant, the one that stood out was the German health care provider who has a setup on the smaller side with about 250 VM’s but support 1000’s of mobile devices which has enabled them to make their doctors more productive. The session really showed the need for effective monitoring and log analysis.

The question asked was what is Operations really? Colin broke it down to Health/Risk/Consumption/Capacity.

Session: Workplace Transformation Through EUC Transformation.

Speaker: Brian Gammage.

The session was called as a quick talk and dealt mostly with strategy. Look at where you are and where you want to be. When putting together with a strategic vision you need to put in place flexibility. The landscape is constantly shifting and your vision needs to move with that. He gave the example of how long the different generation spend in their jobs. I have been in my current position for 7 years. This is considered a long time for my generation. Will my children even have the concept of a permanent job?

Evening Event: vExpert Reception

And finally the day was finished off with the vExpert reception at the Elephant restaurant and bar. Met some really interesting people: Zlatko Mitev and Thomas Findelkind to name a couple.

I also visited the vGiveback stand with my work college Dimos.

vGiveback

So that’s the first 24 hours. Tuesday looks to be a busy day indeed.

 

Nested Home Lab – Part 13 – Creating VMkernel Ports

For all of this to work we need to get a couple of VMkernel network interfaces created per ESX host. This will give us vMotions and for VSAN connectivity. To delve more into VSAN I would highly recommend that you pick up a copy of Essential Virtual SAN by Duncan Epping and Cormac Hogan.

  1. Log in using an account that has permission to configure the environment.Lic-1
  2. Select Home and Hosts and ClustersAH-1
  3. Click on networking icon, right click  the Distributed Switch, Click Add and Manage Hosts…H_Sw_01
  4. Select Add Hosts, Select  Next.H_Sw_02
  5. Click New Hosts…H_Sw_03
  6. Select the Hosts you want to add to the Distributed Switch, Select OK,  and select Next.H_Sw_04
  7. Make sure Manage physical adapters and Manage VMkernel adapters are selected. Click Next.H_Sw_05
  8. Select the first vmnic you want to add to the dswitch and click Assign uplink.
    H_Sw_06
  9. Select Uplink 1. Click OK. Repeat for all remaining vmnics.H_Sw_07
  10.  Once you have added all your vmnic to their uplinks it should look similar to the below picture. Click Next.H_Sw_08
  11. The next following steps will be about adding in the vmkernel network adapters and will show you how to add in one adapter. You will need to go through and add two adapters to each host; 1 for vMotion and one for VSAN. Click New Adapter.H_Sw_09
  12. Click Browse.H_Sw_10
  13. We’ll be adding a vMotion port group, so select vMotion and click, OK.H_Sw_11
  14. Select Next.H_Sw_12
  15. Select vMotion Traffic. Leave the rest as defaults and click Next.
    H_Sw_13
  16. Select Use Static IPv4 Settings and enter in the network details. Select Next.
  17. check all the details are correct and select finish.H_Sw_15
  18. Go back to step 11 and add vMotion vmknics and VSAN vmknics to all your hosts. If you’ve been following the previous posts/docs, once you’ve finished it should look like the below.H_Sw_16
  19. Select Next.H_Sw_17
  20. Check the details and Select Next.H_Sw_18
    vSphere will go off and add your hosts the the dswitch and create vmknics for VSAN and vMotion.
    So where are we now? We have one more step to go before enabling VSAN.

Nested Home Lab – Part 12 – Distributed Virtual Switch

As we discussed in Part 2, for our basic lab, we wanted to separate out the networking  in to three parts:

Use Example IP Range
VM and Management 192.168.0.0
VSAN (vlan 30) 192.168.100.0
vMotion (vlan 40) 192.168.110.0

We have already set-up our physical host’s network in preparation to create our distributed switch.

So what we are looking to do here is create our distributed switch, create the port groups (tag them), and add our VMkernel interfaces. Easy Right?

Creating a Distributed Switch

  1. Log in using an account that has permission to configure the environment.Lic-1
  2. Select Home and Hosts and Clusters.AH-1
  3. Click on networking icon, right click on your Datacenter, Select Distributed Switch, Click New Distributed Switch.
    DS-2
  4. Enter in a name for your new switch. Click Next.DS-3
  5. Select Distributed Switch: 6.0.0.  Click Next.
    DS-4
  6. Set the number of uplinks to 2 and un-tick create a default port group. Select Next.DS-5
  7. Check that our setting are correct and select FinishDS-6

And thats the switch created.

Adding your port groups

We’re going to be creating two port groups, vMotion with a VLAN tag of 40, and VSAN with a VLAN tag of 30.

  1. Right click on the Distributed Switch. Select Distributed Port Group. Select New Distributed Group…
    DS-7
  2. Give the port group a name. In this case vMotion. Select Next.DS-8
  3. Set VLAN type to VLAN and the VLAN ID to 40. Leave all the other options as defaults.
    DS-9
  4. Double check the settings you specified are correct and click Next.DS-10
  5. Go back and create a Virtual SAN VLAN, but with a VLAN tag of 30.DS-11
  6. You should now have a vSwitch that looks like this:DS-12

For a basic lab, as we are creating, the networking is quite simple. However virtual networking has come a very long way and with the introduction of NSX it is now effecting the whole data centre. I would recommend you read Networking for VMware Administrators by Chris Wahl and Steven Pantol. While the book doesn’t cover NSX (understandable as its a whole discipline in itself) it is very good and I would highly recommend it. 

Nexenta – File Services for VSAN

Where I work we are heavily invested in NetApp and 3PAR. Software vendors that also make the hardware for their software to run on. For a large organisation its attractive, one number to dial when SHTF. However with the rise of software defined everything, we are seeing a shift away from that. VMware, clearly one of the market leaders in the software defined space brought out VSAN, and I really, really, like it. Its evolving very fast to meet customer requirements. Support for all flash, stretched clustering, and ROBO set-ups (which I’m really interested in) to name a few. However missing in this mix of awesomeness is file sharing. If Software defined storage is really to take on the big players it needs to really offer more than a place to put virtual disks. Sure you can run a file server VM but that adds an extra layer of complexity. Where’s the integration?

nexenta_logoIn comes Nexenta with NexentaConnect for VMware Virtual SAN.  Now Nexenta has been on the edges of my radar for a while. I knew they were heavily focused on storage, software defined storage to be exact, but I’d never really looked any deeper. My big focus for the last year has been to push VSAN into the company I work for. I could see the cost savings, the ease of use. In the end, after a successful POC, it was a combination of the two that sold it. During the next two to three years it could save the company a huge amount of money.

Back to Nexenta: NexentaConnect has me very interested. Time to get it in the home Lab.

So having worked with it over the weekend, I have made the following (preliminary) observations:

  1. It does what is says on the tin.
  2. Its very strict with its requirements (I tried to cheat and it failed).
  3. Installing the plugin is a manual task, which some Admins might not like.
  4. The whole process of deploying the first OVA’s to having my first shared volume took just less than an hour.

Once you need the minimum requirements, its quite easy to get going with NexentaConnect and its also easy to pull out if you no longer require it.

There are three main components: The vCenter plugin, the connect manager, and the IO engine. The vCenter plugin is very well integrated and fronts the connect manager, which in turn controls the IO engine. The IO engine is a Virtual Storage Appliance (VSA) that gets deployed into your VSAN cluster..

When you create you first share, whether it be NFS or SMB, an IO engine is deployed to the VSAN datastore. I found deployment of the IO engine to be very fast in my LAB. You can also apply different pre-created VSAN storage policies to each share. Once the IO engine was deployed adding additional share was even faster.

As I don’t currently have a full physical lab, I couldn’t in good conscience do any real performance testing, but it was very quick even in a nested virtual environment.

This product will really shine in smaller shops, which are often overlooked, or ROBO offices. VSAN coupled with a product like this could help a smaller business through the awkward period of transitioning from a small to medium business and at that stage why take it out? That’s not to say that larger organisations shouldn’t take a look and implement if it fits their strategy.

The product as a whole is very interesting and I really do like it. There have been several write ups and youtube videos from several big bloggers detailing various bits of the set-up. Cormac Hogan did a great write-up here and so has Rawlinson Rivera of Punching Clouds.

Its great to see real innovation like this coming from the industry and I highly recommend heading over to Nexenta’s website and take a look.