Processor modes refer to the various ways that the processor creates an operating environment for itself. Specifically, the processor mode controls how the processor sees and manages the system memory and the tasks that use it. There are three different modes of operation, that resulted from the evolution of the PC from its humble beginnings with the Intel 8088 chip.
Real Mode:
The original IBM PC could only address 1 MB of system memory, and the original versions of DOS created to work on it were designed with this in mind. DOS is by its nature a single-tasking operating system, meaning it can only handle one program running at a time. The decisions made in these early days have carried forward until now, and in each new processor, care had to be taken to be able to put the processor in a mode that would be compatible with the original Intel 8088 chip. This is called real mode.
When a processor is running in real mode, it acts like an "8088 on steroids". What this means is that it has the advantage of speed, but it otherwise accesses memory with the same restrictions of the original 8088: a limit of 1 MB of addressable RAM, and slow memory access that doesn't take advantage of the full 32-bit processing of modern CPUs. All processors have this real mode available, and in fact the computer normally starts up in real mode.
Real mode is of course used by DOS and "standard" DOS applications. In fact, today there are relatively few simple DOS programs that just use the standard 640K that DOS makes available. Even within DOS now there are special programs available that will "extend" DOS to allow access to extended memory (over 1 MB) and faster 32-bit access. These are sometimes called DOS extenders. The protocol that describes how to make DOS work in protected mode is called DPMI (DOS protected mode interface). DOS extenders are used by most DOS games (since the standard DOS 640 KB limit has gone from inflexible to downright laughable in recent years as games have become very large).
Note: The first 64 KB of extended memory is actually accessible to the PC in real mode, despite the fact that it is not supposed to be possible. This is the result of a bug in the original IBM AT. This area of memory is called the high memory area (HMA).
Protected Mode:
Starting with the 80286 chip in the IBM AT, a new processor mode was introduced called protected mode. This is a much more powerful mode of operation than real mode, and is used in all modern multitasking operating systems. The advantages of protected mode (compared to real mode) are:
The name of this mode comes from its primary use, which is by multitasking operating systems. Each program that is running has its own assigned memory locations, which are protected from conflict with other programs. If a program tries to use a memory address that it isn't allowed to, a "protection fault" is generated. If you've ever used Windows 3.x, you know exactly what I am talking about. :^)
Although introduced with the 286, the operating system world back in the early 80s was stuck squarely in DOS. The use of protected mode didn't become popular until the rise in dominance of the Microsoft Windows operating system. Protected mode is now currently the way that most people use their PCs. All of the major operating systems today use protected mode, including Windows 3.x, Window 9x, Windows NT, OS/2 and Linux. Even DOS, which normally runs in real mode, can access protected mode using DPMI (DOS protected mode interface), which is often used by DOS games to break the 640 KB DOS conventional memory barrier.
All processors from the 286 on can use protected mode. 386 and later processors can switch on the fly from real to protected mode and vice-versa; the 286 can only switch from real to protected mode once (switching back requires a reboot). Protected mode is also sometimes called 386 Enhanced Mode, since it became mainstream with that family of processors.
Virtual Real Mode:
The third mode of processor operation is actually an additional capability, an enhancement, of protected mode. Protected mode is normally used to run graphical multitasking operating systems such as the various flavors of Windows. There is often a desire to be able to run DOS programs under Windows, but DOS programs need to be run in real mode, not protected mode.
Virtual real mode was created to solve this problem. In essence, it emulates real mode from within protected mode, allowing DOS programs to run. A protected mode operating system such as Windows can in fact create multiple virtual real mode machines, each of which appear to the software running them as if they are the only software running on the machine. Each virtual machine gets its own 1 MB address space, an image of the real hardware BIOS routines, everything.
Virtual real mode is what is used when you use a DOS box or run a DOS game in Windows 95. When you start a DOS application, Windows 95 creates a virtual DOS machine for it to run under. Virtual real mode was introduced starting with the 386 family of processors.