Tutorial 2: Detecting a Valid PE File

[Iczelion's Win32 Assembly Homepage]

In this tutorial, we will learn how to check if a given file is a valid PE file.
Download the example.

Theory:

How can you verify if a given file is a PE file? That question is difficult to answer. That depends on the length that you want to go to do that. You can verify every data structure defined in the PE file format or you are satisfied with verifying only the crucial ones. Most of the time, it's pretty pointless to verify every single structure in the files. If the crucial structures are valid, we can assume that the file is a valid PE. And we will use that assumption.

The essential structure we will verify is the PE header itself. So we need to know a little about it, programmatically. The PE header is actually a structure called IMAGE_NT_HEADERS. It has the following definition:

IMAGE_NT_HEADERS STRUCT
   Signature dd ?
   FileHeader IMAGE_FILE_HEADER <>
   OptionalHeader IMAGE_OPTIONAL_HEADER32 <>
IMAGE_NT_HEADERS ENDS

Signature is a dword that contains the value 50h, 45h, 00h, 00h. In more human term, it contains the text "PE" followed by two terminating zeroes. This member is the PE signature so we will use it in verifying if a given file is a valid PE one.
FileHeader is a structure that contains information about the physical layout of the PE file such as the number of sections, the machine the file is targeted and so on.
OptionalHeader is a structure that contains information about the logical layout of the PE file. Despite the "Optional" in its name, it's always present.

Our goal is now clear. If value of the signature member of the IMAGE_NT_HEADERS is equal to "PE" followed by two zeroes, then the file is a valid PE. In fact, for comparison purpose, Microsoft has defined a constant named IMAGE_NT_SIGNATURE which we can readily use.

IMAGE_DOS_SIGNATURE equ 5A4Dh
IMAGE_OS2_SIGNATURE equ 454Eh
IMAGE_OS2_SIGNATURE_LE equ 454Ch
IMAGE_VXD_SIGNATURE equ 454Ch
IMAGE_NT_SIGNATURE equ 4550h

The next question: how can we know where the PE header is? The answer is simple: the DOS MZ header contains the file offset of the PE header. The DOS MZ header is defined as IMAGE_DOS_HEADER structure. You can check it out in windows.inc. The e_lfanew member of the IMAGE_DOS_HEADER structure contains the file offset of the PE header.

The steps are now as follows:

1. Verify if the given file has a valid DOS MZ header by comparing the first word of the file with the value IMAGE_DOS_SIGNATURE.
2. If the file has a valid DOS header, use the value in e_lfanew member to find the PE header
3. Comparing the first word of the PE header with the value IMAGE_NT_HEADER. If both values match, then we can assume that the file is a valid PE.

Example:

.386
.model flat,stdcall
option casemap:none
include /masm32/include/windows.inc
include /masm32/include/kernel32.inc
include /masm32/include/comdlg32.inc
include /masm32/include/user32.inc
includelib /masm32/lib/user32.lib
includelib /masm32/lib/kernel32.lib
includelib /masm32/lib/comdlg32.lib

SEH struct
PrevLink dd ?    ; the address of the previous seh structure
CurrentHandler dd ?    ; the address of the exception handler
SafeOffset dd ?    ; The offset where it's safe to continue execution
PrevEsp dd ?      ; the old value in esp
PrevEbp dd ?     ; The old value in ebp
SEH ends

.data
AppName db "PE tutorial no.2",0
ofn OPENFILENAME <>
FilterString db "Executable Files (*.exe, *.dll)",0,"*.exe;*.dll",0
                 db "All Files",0,"*.*",0,0
FileOpenError db "Cannot open the file for reading",0
FileOpenMappingError db "Cannot open the file for memory mapping",0
FileMappingError db "Cannot map the file into memory",0
FileValidPE db "This file is a valid PE",0
FileInValidPE db "This file is not a valid PE",0

.data?
buffer db 512 dup(?)
hFile dd ?
hMapping dd ?
pMapping dd ?
ValidPE dd ?

.code
start proc
LOCAL seh:SEH
mov ofn.lStructSize,SIZEOF ofn
mov ofn.lpstrFilter, OFFSET FilterString
mov ofn.lpstrFile, OFFSET buffer
mov ofn.nMaxFile,512
mov ofn.Flags, OFN_FILEMUSTEXIST or OFN_PATHMUSTEXIST or OFN_LONGNAMES or OFN_EXPLORER or OFN_HIDEREADONLY
invoke GetOpenFileName, ADDR ofn
.if eax==TRUE
    invoke CreateFile, addr buffer, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL
    .if eax!=INVALID_HANDLE_VALUE
       mov hFile, eax
       invoke CreateFileMapping, hFile, NULL, PAGE_READONLY,0,0,0
       .if eax!=NULL
          mov hMapping, eax
          invoke MapViewOfFile,hMapping,FILE_MAP_READ,0,0,0
          .if eax!=NULL
             mov pMapping,eax
             assume fs:nothing
             push fs:[0]
             pop seh.PrevLink
             mov seh.CurrentHandler,offset SEHHandler
             mov seh.SafeOffset,offset FinalExit
             lea eax,seh
             mov fs:[0], eax
             mov seh.PrevEsp,esp
             mov seh.PrevEbp,ebp
             mov edi, pMapping
             assume edi:ptr IMAGE_DOS_HEADER
             .if [edi].e_magic==IMAGE_DOS_SIGNATURE
                add edi, [edi].e_lfanew
                assume edi:ptr IMAGE_NT_HEADERS
                .if [edi].Signature==IMAGE_NT_SIGNATURE
                   mov ValidPE, TRUE
                .else
                   mov ValidPE, FALSE
                .endif
             .else
                 mov ValidPE,FALSE
             .endif
FinalExit:
             .if ValidPE==TRUE
                 invoke MessageBox, 0, addr FileValidPE, addr AppName, MB_OK+MB_ICONINFORMATION
             .else
                invoke MessageBox, 0, addr FileInValidPE, addr AppName, MB_OK+MB_ICONINFORMATION
             .endif
             push seh.PrevLink
             pop fs:[0]
             invoke UnmapViewOfFile, pMapping
          .else
             invoke MessageBox, 0, addr FileMappingError, addr AppName, MB_OK+MB_ICONERROR
          .endif
          invoke CloseHandle,hMapping
       .else
          invoke MessageBox, 0, addr FileOpenMappingError, addr AppName, MB_OK+MB_ICONERROR
       .endif
       invoke CloseHandle, hFile
    .else
       invoke MessageBox, 0, addr FileOpenError, addr AppName, MB_OK+MB_ICONERROR
    .endif
.endif
invoke ExitProcess, 0
start endp

SEHHandler proc C uses edx pExcept:DWORD, pFrame:DWORD, pContext:DWORD, pDispatch:DWORD
    mov edx,pFrame
    assume edx:ptr SEH
    mov eax,pContext
    assume eax:ptr CONTEXT
    push [edx].SafeOffset
    pop [eax].regEip
    push [edx].PrevEsp
    pop [eax].regEsp
    push [edx].PrevEbp
    pop [eax].regEbp
    mov ValidPE, FALSE
    mov eax,ExceptionContinueExecution
    ret
SEHHandler endp
end start

Analysis:

The program opens a file and checks if the DOS header is valid, if it is, it checks the PE header if it's valid. If it is, then it assumes the file is a valid PE. In this example, I use structured exception handling (SEH) so that we don't have to check for every possible error: if a fault occurs, we assume that it's because the file is not a valid PE thus giving our program wrong information. Windows itself uses SEH heavily in its parameter validation routines. If you're interested in SEH, read the article by Jeremy Gordon.

The program displays an open file common dialog to the user and when the user chooses an executable file, it opens the file and maps it into memory. Before it goes on with the verification, it sets up a SEH:

   assume fs:nothing
   push fs:[0]
   pop seh.PrevLink
   mov seh.CurrentHandler,offset SEHHandler
   mov seh.SafeOffset,offset FinalExit
   lea eax,seh
   mov fs:[0], eax
   mov seh.PrevEsp,esp
   mov seh.PrevEbp,ebp

We start by assuming the use of fs register as nothing. This must be done because MASM assumes the use of fs register to ERROR. Next we store the address of the previous SEH handler in our structure for use by Windows. We store the address of our SEH handler, the address where the execution can safely resume if a fault occurs, the current values of esp and ebp so that our SEH handler can get the state of the stack back to normal before it resumes the execution of our program.

   mov edi, pMapping
   assume edi:ptr IMAGE_DOS_HEADER
   .if [edi].e_magic==IMAGE_DOS_SIGNATURE

After we are done with setting up SEH, we continue with the verification. We put the address of the first byte of the target file in edi, which is the first byte of the DOS header. For ease of comparison, we tell the assembler that it can assume edi as pointing to the IMAGE_DOS_HEADER structure (which is the truth). We then compare the first word of the DOS header with the string "MZ" which is defined as a constant in windows.inc named IMAGE_DOS_SIGNATURE. If the comparison is ok, we continue to the PE header. If not, we set the value in ValidPE to FALSE, meaning that the file is not a valid PE.

      add edi, [edi].e_lfanew
      assume edi:ptr IMAGE_NT_HEADERS
      .if [edi].Signature==IMAGE_NT_SIGNATURE
         mov ValidPE, TRUE
      .else
         mov ValidPE, FALSE
      .endif

To get to the PE header, we need the value in e_lfanew of the DOS header. This field contains the file offset of the PE header, relative to the file beginning. Thus we add this value to edi and we get to the first byte of the PE header. It's this place that a fault may occur. If the file is really not a PE file, the value in e_lfanew will be incorrect and thus using it amounts to using a wild pointer. If we don't use SEH, we must check the value of the e_lfanew against the file size which is ugly. If all goes well, we compare the first dword of the PE header with the string "PE". Again there is a handy constant named IMAGE_NT_SIGNATURE which we can use. If the result of comparison is true, we assume the file is a valid PE.
If the value in e_lfanew is incorrect, a fault may occur and our SEH handler will get control. It simply restores the stack pointer, bsae pointer and resumes the execution at the safe offset which is at the FinalExit label.

FinalExit:
   .if ValidPE==TRUE
      invoke MessageBox, 0, addr FileValidPE, addr AppName, MB_OK+MB_ICONINFORMATION
   .else
      invoke MessageBox, 0, addr FileInValidPE, addr AppName, MB_OK+MB_ICONINFORMATION
   .endif

The above code is simplicity itself. It checks the value in ValidPE and displays a message to the user accordingly.

   push seh.PrevLink
   pop fs:[0]

When the SEH is no longer used, we dissociate it from the SEH chain.

---