Do You Know How it Works Under the Hood? Really, Really?

Quick quiz:

Can you create an Active Directory user account that has a blank samAccountName?

Answer: Yes. Oh, not in ADUC, but using almost any other tool, sure. A blank samAccountName is legal so long as it’s unique.

I use this example in classes all the time, because it illustrates one of the difficulties in the Microsoft admin universe: we know our tools pretty well, but not necessarily the underlying technology so well, mainly because the tools have provided a layer of insulation for our entire careers. But without knowing the technology, you’re not as good at planning, troubleshooting, architecture, operations – well, all of it, really.

Here’s another one: do you know how the “dynamic memory” or “memory overcommit” features in VMware and Hyper-V work? If not, and if you’re using that feature, you might be using it in cases where it does more harm than good.

Think about it: in a physical server, you can’t simply yank memory out of a running machine, nor can you just pop in more memory. The guest OS in a VM thinks it’s on a physical machine, so it operates under the same restriction. So how does overcommit work?

The trick with VM memory is that every byte of memory actually being used by a VM must be backed up by physical RAM in the host. Traditionally, the hypervisor had no way of knowing what memory was in use, and what wasn’t, because the guest OS is free to rearrange that stuff constantly. Ergo, every byte assigned to a VM needed to be backed by physical RAM.

Overcommit requires the installation of a special device driver, called the balloon driver, in the guest VM. Device drivers operate in Windows’ kernel mode, which means if they ask for memory, they get it. The assumption by Windows is that device drivers don’t need much RAM, and that denying them RAM will make hardware not work, and so they get what they ask for. So when the hypervisor host asks the balloon driver to release some RAM, the balloon driver asks the guest OS for memory. Whatever the driver gets, it erases, setting the memory contents to 0. That means a known portion of guest memory isn’t in use, so the hypervisor doesn’t have to back that memory with physical RAM. Thus, the VM always thinks it has 4GB or whatever, but not all of that will be in use, because the balloon driver “locks” it.

The operating presumption is that running applications will only release RAM they don’t absolutely, positively need, so the balloon driver won’t impact system operations. It’ll essentially just “gather up” memory that wasn’t really in use.

The problem is that assumptions can sometimes be wrong. For example, some applications maintain large data caches in RAM, basically seeking to use all the RAM they can – the same approach as the balloon driver. When the OS sends a memory panic, because the balloon driver is requesting RAM that isn’t available, these user-mode applications will give up some memory. Their assumption is that the app is better off with less-than-optimal RAM than with the server crashing due to insufficient memory. So app performance suffers – sometimes significantly, depending on the effort involved in rearranging those data caches so that memory can be freed up.

The point is, you can’t make intelligent decisions about these features unless you know how they work, how they interact with other applications, and what the consequences might be. Knowing how things work under the hood is a crucial part of being an effective IT person.

And that “knowing” requires an insatiable curiosity. First-level documentation never discusses these under-the-hood secrets. In most cases, vendor marketing doesn’t either, because they simply want you to believe the feature is a no-brainer to use. So you have to be constantly curious, constantly asking “why” and “how,” and constantly seeking out the answers on your own. Yeah, it’s a lot to keep up with – but it’s what separates the true IT professional from the IT operator who simply pushes buttons and hopes for the best.

One thought on “Do You Know How it Works Under the Hood? Really, Really?

  1. Richard

    And this is a big problem, that a lot of “IT Professionals” don’t care how it really works. And because of this, it’s difficult for a “real” IT Professional to talk and get his message to a “fake” IT Professional. Because they don’t understand what you are talking about. Any advice on this?

Comments are closed