Again another customer is using AWE and /3GB boot.ini switch on a SQL Server box that has 24G of physical memory... So here's a summary:
/PAE is required for a 32bit Windows to manage physical memory that's larger than 4G. PAE is actually a CPU technology. It allows a 32bit CPU to address physical memory that's above 4G. So if you have more than 4G of physical memory, and you're using a 32bit of Windows, you need it in your boot.ini.
AWE on the other hand, is a set of Windows API that a 32bit application can use to access physical memory outside of the 4G virtual memory space. A 32 bit application has an address space of 4G. So it can have 4G of memory, physical plus pagefile, at most. Even if you have more than 4G of RAM installed on the computer and you have /PAE enabled. Because the operating system will have to distribute the RAM to all the processes running.
For some memory intensive application such as database management systems or scientific computing software, 4G of memory may not be enough. Microsoft provided a set of API functions to enable an application to access physical memory that's beyond this 4G of virtual memory limit. So AWE is application specific.
Now, the /3GB switch in boot.ini. We know a 32bit application has a 4G of virtual memory space. Inside this 4G space, 2G is reserved for kernel code. The rest 2G is for the user space code. Again, some application may need more than 2G of user space memory - physical or pagefile.
Why not using AWE - you may ask. Actually you can. And AWE will guarrantee that you'll get physical memory. But to use AWE you may have to re-write your code. For those that don't have a chance to re-write their code, Microsoft offered this /3GB switch in boot.ini. If you enable this switch, Windows will squeeze all the kernel code into 1GB, so that you'll have 3GB of user space memory.
For example, if the notepad in a 32bit Windows were develpped this way, the content of the file being edited was completely stored in memory, then you cannot open a txt file that's larger than 2G. You simply don't have enough virtual memory space.
With /3GB in boot.ini, this same notepad can open a file that's larger than 2G (but still within 3G) because Windows kernel was squeezed into 1G, and now you have a 3G memory space for the user space code.
These all sound perfect - Every option has its unique value, every option has its strength and weakness. So you can pick whichever that fits your requirement. Or even combine them. However, the /3GB switch comes with a price:
Because the kernel is squeezed into a 1GB memory space, its ability of managing RAM is also limited to 16G. So, if you have more than 16G of RAM, you'd better not use this /3GB switch.
In short, if you have more than 4G of RAM, you should use /PAE in your boot.ini. If you have more than 16G of RAM, you should avoid using /3GB switch in your boot.ini. AWE is a feature of an application, you only need it if you have more than 4G of RAM.
Of course, all these only make sense if you're using a 32bit of Windows. With 64bit of Windows, you don't have to worry about any of them.