Operating Systems (Hands-On)/Case Study Windows

Objectives

By the end of this lesson, you will be able to:

1. Understand the architecture of Microsoft Windows by identifying its major layers and explaining how objects and message passing are used to structure the operating system.
2. Develop practical familiarity with Win32 programming by navigating Microsoft documentation and implementing basic GUI interactions using messages, controls, and identifiers.
3. Build confidence working within a complex operating system API by synthesizing information from multiple sources and modifying existing code to change behavior, appearance, and interaction.

Required Reading

• Get Started with Win32 and C++
  • Intro to Win32 programming in C++
  • Module 1. Your First Windows Program
• How to Create a Button

Background

Windows operating system versions.

The Windows operating system was created by Microsoft, with its first release in November of 1985. Originally, Windows was referred to as an “operating environment” because it ran on top of Microsoft’s Disk Operating System (DOS). Windows was partly created to allow IBM PCs to remain competitive with the Apple Macintosh, which was released in 1984. (Though, in fact, the computing industry as a whole was moving toward graphical interfaces. Macintosh and Windows were just the two most popular of several options.) DOS, as an operating system, is inherently a single-tasked system where the OS is provided as a series of functions reachable via software interrupts. In contrast, Windows added multitasking features and a more robust set of interactions between programs. As the various releases of the DOS-based Windows system matured, Microsoft added features such as full exposure to advanced features of the Intel 386 processor. This brought process isolation, granular control, and generally faster performance to Windows systems.

Windows 1.0 up through 3.11 were 16-bit operating systems. A second line of Windows development started in the early to mid 1990s which would produce a full 32-bit business-oriented version of Windows called Windows NT. Windows NT was sold in both server and workstation editions, and provided centralized control and desktop standardization. This is where Windows security and management features, such as the concept of a “Windows domain” were born. Windows NT was a full-fledged system in its own right and did not run on top of DOS as did previous versions of Windows.

In 1995, a consumer-facing variant of 32-bit Windows was released as Windows 95. Windows 95 was still based on DOS, but it was the first consumer-facing Windows product which booted into the graphical environment by default. Previous versions of Windows required the user to invoke the GUI using the command win at the DOS prompt. Though, of course, many users opted to configure their systems to automatically run Windows. This was just not the default behavior. Windows 95 introduced the Win32 API which allowed for more robust GUI programming. Windows 95 was followed by Windows 98, then Windows ME. The term “ME” stands for “Millennium Edition”, so chosen because it was released in 2000. It was not called “Windows 2000” because that was the Windows NT line’s release that year. Windows ME was the final version of Windows based on DOS, and was widely regarded as a failure.

Shortly after the release of Windows ME, Microsoft released Windows XP in 2001, a complete merging of the consumer and NT Windows lines. This effectively ended the production of DOS, and all future versions of Windows were built on top of the Windows NT Kernel. Following Windows XP was Windows Vista, 7, 8, 8.1, 10, and the current version Windows 11. There is no longer a separate Windows NT line of operating systems, though there is a server variant of Windows used to create domain controllers, web servers, and the like.

For a full history of Windows, please see Microsoft Windows version history

System Organization

When looking at the organization of the Windows Operating System, we have to keep in mind what it was designed to do. Windows was designed to be a closed-source, proprietary operating system sold for profit which would move in response to market demands. The systems it competes with trace their conceptual lineage to the Xerox Alto which used a pure object-oriented language called SmallTalk as its primary language. This means that these systems provide for complex interactions with built-in graphical systems and complex objects. Windows fits this mold by facilitating message passing among complex objects. Windows, itself, is largely coded in C, with some components written in C++.

Layers of Modern Windows

Simplified layers of the windows operating system.

The organization of the Windows operating system is quite complicated. It is highly modular, and has grown in complexity over 40+ years after its initial release. We will explore the layers of the modern systems beginning at hardware level and building outward the user application layer.

1. Hypervisor - The hypervisor is only present in setups which have features such as Hyper-V or WSL 2 active. When it is present, it is the part of the system that really touches the hardware. This leverages advanced CPU features which allows the processor to run several kernels at once. When activated, the Windows kernel itself runs in a privileged virtualized partition. In the case of Hyper-V, it is possible to run several instances of Windows (and other systems) simultaneously. In the case of WSL 2, the Linux kernel is running as a guest allowing Linux applications to run concurrently with Windows.
2. Hardware Abstraction Layer (HAL) - The HAL is the beginning of the Windows kernel proper forming the first layer of NTOSKRNL.exe , the Windows kernel image. This provides a generalized interface to various buses and interrupts which comprise the PC architecture. The HAL provides services which device drivers use for specific device interactions. This sort of generalized interface allows for greater stability and for features such as plug and play devices.
3. Device Drivers, NTOS Executive, and GUI Driver - On top of the HAL we find the driver layer of the system. This contains the various device drivers, some provided by third party manufacturers, which enable features and services for specific hardware devices. The executive layer is responsible for the management of processes, applications, and threads as well as scheduling. Finally, the GUI driver supplies the actual “windows” that make up “Windows.” Note that significant parts of the GUI system are a proper part of the kernel which makes for fast and responsive graphics. However, should the GUI experience an unrecoverable fault, it will crash the entire system. These GUI crashes were once a leading cause of the dreaded “Blue screen of death.” However, a bit of reshuffling has led to increased GUI stability, making GUI based BSODs rare. (Now the leading cause is faulty device driver code.)
4. NTOS Kernel - The kernel layer provides the entry points which allow the user space to interact with kernel objects. Were we to make true system calls, we would be able to talk directly to this layer. However, we should not do this! The low level system interactions are mostly what Microsoft calls “undocumented” features. They aren’t secret, but when Microsoft calls something “undocumented” what they are really saying is “subject to change without warning.” Targeting the kernel directly will, therefore, make your programs brittle and ephemeral.
5. Native NT API - Here is where the objects and handles really begin. When we have “kernel objects”, what we really have is handles to NT API objects which are, in turn, a collection of resources and calls between the NT API and the NTOS kernel. Subsystems can call this layer to gain low level access, though most of its calls are undocumented and therefore unsafe to call. User applications should most definitely avoid calling the NT API directly as all documented calls are wrapped by higher-level constructs, such as Win32 API.
6.  NT Subsystems - The bulk of the actual calls made by user applications are targeted to NT Subsystems. The term subsystem is used here in a pedagogical sense to group related user-mode services and APIs. These are services which provide the useful operations of the operating system. These subsystems are typically a conglomeration of several subsystems. The diagram shows the major divisions of the ones supplied by Windows:
   1. Modern Windows Applications - this division includes subsystems such as WinRT, COM, Process lifetime Manager, and other services of modern Windows processes. This division also includes runtime environments such as .Net, which is built atop Win32 and WinRT. Most modern Windows programs target this division.
   2. Windows Services - this division includes the more “Traditional” Windows applications environment. Here, you’ll find the NT Services and the application parts of the Win32 subsystem. The Win32 subsystem is divided between the user space subsystem and several kernel objects, which is why this is often thought of as the low level systems programming of Windows because most other subsystems make use of these services. Prior to the advent of .Net in the early 2000s, this was the subsystem which served user applications. Now most developers target the friendlier modern Windows services.
   3. Windows Desktop Apps - This subsystem contains the high-level objects used by GUI applications. This includes the desktop manager, .Net base classes, GUI Shell, Dynamic libraries (such as OLE) and the subsystem API. Most subsystems will utilize these features to provide higher level functionality and quite a few applications target this directly as well.
7. User Application - The user application layer is the true user space of Windows. These are the programs that users directly use. This is somewhat analogous to “processes” in other operating systems with one major caveat. Windows organizes applications into processes and threads. So one user application may represent many subprograms which each run in their own isolation unit, each of which may have one or more threads of execution which share the same address space.

Windows Objects

The primary function of Windows is to facilitate the passing of messages among a series of complex objects. The objects in Windows can be placed throughout the system. They could be complex kernel objects, objects within a subsystem, or objects provided by a user application. Each object in Windows is referred to by an integer called a handle. Everything in Windows, including user application instances, have a handle. Handle identifiers are unique within the context in which they are created and are central to message passing and object creation.

Message Interface

Windows system calls are implemented as messages which are passed between objects. This is accomplished in two ways:

1. LPC - Local Procedure Call dispatches a message within some local context, analogous to a local library call.
2. RPC - Remote Procedure Call dispatches a message to an object located in some other context. This could be a dispatch to a kernel object or to an object provided by some Windows subsystem or application.

It should be noted here that both LPC and RPC invocations look the same to the application programmer. This ambiguity is intentional. Windows presents itself as a singular coherent system, but in fact can be quite fluid. What was an RPC call in one release of Windows could easily become an LPC call in the next without breaking any of the applications which use that call. (Assuming, of course, the applications are using those calls correctly.) This also means that exploring direct system calls in Windows is nearly impossible as we could never be assured that we were talking directly to the kernel, which is dangerous anyway. So we’re going to stick to programming one of its lower-level subsystems.

Win32 API Programming

The best way to get a feel for how the Windows Operating System works is to try your hand at some lower-level Win32 API Programming. (API stands for Application Programming Interface and is the common term applied to application developer facing libraries.) For this purpose, I will hand you over to the authoritative source on the subject, Microsoft. Though I do have a couple of “harder to find things” that I want to point you toward as well as a bit of commentary to offer.

Get Started with Win32 and C++

The first order of business is to wrap our heads around what Win32 programming is like. To do this, I suggest you have a look at the page Get Started with Win32 and C++ Before I recommend specific sections, I want you to pay close attention to the following: * The development environment you need, namely Visual Studio. There are alternatives, but Visual Studio (NOT Visual Studio Code!) will give you the easiest and most authentic experience for learning Win32. * Observe how “Windows” is not just a fancy title. Everything you see on your screen, buttons included, is a “window.” * Take note of the objects at play. Try to get an intuitive feel for how the OS objects relate to each other. Do you see any polymorphism in action? * Pay attention to the nature of strings. Just for fun, when you write your first program, maybe try to get the strings wrong by deleting the leading L from them. What does that do? * Look at the structure of applications and the WinMain entry point. Now with that in mind, read the opening page, linked above along with all of the articles under the subheading of Intro to Win32 programming in C++

Your First Windows Program

Now that you’ve read the preliminaries, it’s time to write some code. We will do this by following along with Module 1. Your First Windows Program. You can copy and paste code, but I suggest you try typing it in as you follow through the sub pages. As you do, consider these thoughts: * Visual Studio code completion is an absolute must with Win32 programming! * Let go of the notion that you’ll remember this verbatim. I can assure you that no one can program Win32 applications without code completion and a browser full of documentation. * How does the message interface work?

The Arcane Art of Microsoft Documentation

One of the key skills of a Win32 developer is to consume Microsoft’s documentation. However, this can lead to some frustration as many things rely on combining information from lots of sources. One of the most common complaints I hear from students is that responding to buttons is hard to figure out. It’s easy enough to create a button, as shown in the article How to Create a Button, and it is easy enough to surmise from Button Control Notificationsthat we will receive the BN_CLICKED notification when the button is clicked. But, to get the full picture of what’s going to happen, we have to keep digging! First, we need to observe that WM_COMMAND is the message that is sent to our WindowProc procedure. Here’s where the documentation leaves us. It has fully specified how this control works, and it leaves us to synthesize the following:

1. In WindowProc, we must look for the WM_COMMAND message to be sent.
2. Upon receipt of the WM_COMMAND message, we need to check the high order word of wParam to check the notification code.
3. If we have a BN_CLICKED notification, we need to handle the button clicked event. That is, we need to perform the action the button commands.

In code, this looks like this:

switch(uMsg) 
{
  // ... other messages

  case WM_COMMAND:
    if(HIWORD(wParam) == BN_CLICKED) {
      // Handle Button Click
    }
    break;

  // ... other messages
}

Now, that begs the question, how do we know which button has been clicked? One temptation is to use the handle. Recall, from the documentation, that the lParam for BN_CLICKED is a handle to the button. We could, indeed, make a global variable to detect a button. Say something like HWND hBtnOk in which we place the handle to the button returned by CreateWindow. This would work! if((HWND)lParam == hBtnOK) would be a valid check for “OK” being pressed. However, this is not usually what most Windows developers do because global variables are dangerous. Were a function to modify this variable, it would make our button become non-responsive. Instead, the discerning Windows programmer would use the control identifier, which is in the low order word of wParam. The control identifier is an integer, and we can therefore simply declare some global constants. This makes the constants reusable in other contexts, and they cannot be changed, so safety has been achieved. Here’s what that looks like.

First, we define some global constants outside of any function:

#define BTN_OK     1
#define BTN_CANCEL 2

It does not matter what values we chose, so long as they are different. They are completely arbitrary. Now, we have to imbue the buttons with their ID when we create them. The creation of our “ok” and “cancel” buttons in our WinMain would look like this:

CreateWindow(
   L"BUTTON",          // Predefined class; Unicode assumed
   L"OK",              // Button text
   WS_TABSTOP | WS_VISIBLE | WS_CHILD | BS_DEFPUSHBUTTON, // Styles
   10,                 // x position
   10,                 // y position
   100,                // Button width
   100,                // Button height
   hwnd,               // Parent window
   (HMENU) BTN_OK,     // Control ID (hMenu)
   hInstance,          // Owning Instance
   NULL);              // Pointer not needed.

CreateWindow(
   L"BUTTON",          // Predefined class; Unicode assumed
   L"CANCEL",          // Button text
   WS_TABSTOP | WS_VISIBLE | WS_CHILD | BS_DEFPUSHBUTTON, // Styles
   120,                // x position
   10,                 // y position
   100,                // Button width
   100,                // Button height
   hwnd,               // Parent window
   (HMENU) BTN_CANCEL, // Control ID (hMenu)
   hInstance,          // Owning Instance
   NULL);              // Pointer not needed.

If you are like most newcomers to Windows, this could be quite frustrating! How are we expected to know that the hMenu parameter specifies the control identifier? Well, the answer lies in yet another documentation page, CreateWindowA Macro. In particular, under the hMenu reference you will find this:

For a child window, hMenu specifies the child-window identifier, an integer value used by a dialog box control to notify its parent about events. The application determines the child-window identifier; it must be unique for all child windows with the same parent window.

So now, putting it all together, we can handle buttons like this from our WindowProc:

switch(uMsg) 
{
  // ... other messages
    
  case WM_COMMAND:
    if(HIWORD(wParam) == BN_CLICKED) {
      if(LOWORD(wParam) == BTN_OK) {
        // handle ok button
      } else if(LOWORD(wParam) == BTN_CANCEL) {
        // handle cancel button
      }
    }
  break;
    
  // ... other messages
}

Of course, you can subdivide this up into subroutines to handle buttons, individual button routines, etc. The point is to see how everything gets packed together. Messages are sent to the instance, the instance dispatches the message, the message handler checks for a notification, and the notification contains identifiers. This pattern repeats for pretty much all the other controls. The documentation spelunking needed to find out how to do this is also common. When in doubt, click and follow all links on the reference pages!

Conclusion

Windows presents a programming model that is object-oriented, message-driven, and heavily layered. While this model can feel opaque and frustrating at first—particularly when navigating documentation—it reflects decades of accumulated design decisions driven by compatibility, performance, and user expectations. The Win32 API exposes this structure more directly than modern frameworks, making it an ideal lens through which to understand how Windows actually operates. In the next lesson, we will step back from Windows itself and compare its design choices with those of UNIX-style systems and the HAWX microkernel, focusing on how different operating systems draw boundaries between mechanisms, policies, and subsystems.

Activities / Assignments

1. Windows System Programming
2. Review Quiz