今天碰到一个 C++ from C#: C++ function (in a DLL) returning false, but C# thinks it's true! 惊呆了！

I'm writing a little C# app that calls a few functions in a C++ API. I have the C++ code building into a DLL, and the C# code calls the API using DllImport. (I am using a .DEF file for the C++ DLL so I don't need extern "C".)

So far the API has one function, which currently does absolutely nothing:

bool Foo()
{
  return false;
}

In C#, I have the following:

public class FooAPI
{
    [DllImport("Foo.dll")]
    public static extern bool Foo();
}

...

bool b = FooAPI.Foo(); 
if (!b) 
{ 
    // Throw an exception 
}

My problem is that, for some reason, b is always evaluating to TRUE. I have a breakpoint on if (!b) and the debugger reports it as 'true', irrelevant of whatever the C++ function is returning.

ANS：

1）Try [return: MarshalAs (UnmanagedType.I1)]. By default, C# interop marshals C# bool as the Win32 BOOL, which is the same as int, while C++ bool is one byte AFAIR. Thus, C#'s default marshaling expects the return value to be a BOOL in the eax register, and picks up some non-zero garbage because C++ bool is returned in al.

2）Your code snippet as posted cannot work. If this was compiled with a C++ compiler then the function name would be ?Foo@@YA_NXZ and your C# code could never find it. The calling convention is important too, it is not the default (CallingConvention.StdCall) unless you tell the compiler. And somewhere you should have told the linker to export the function.

Start by declaring the exported function so it is compatible with default P/Invoke attributes:

extern "C" __declspec(dllexport) 
bool __stdcall Foo() {
    return false;
}

Next problem is that the C++ compiler uses only one byte to store a bool. The machine code generated for your return statement is:

013D138E  xor         al,al

The P/Invoke marshaller will however assume it is a 32-bit integer and check the value of the eax register. Either declare the return type of the function as int or BOOL or use a [return: MarshalAs(UnmanagedType.U1)] attribute in the C# declaration.

C# DllImport with C++ boolean function not returning correctly

Shammah

1# Published in 2011-01-05 20:31:10Z I have the following function in a C++ DLL extern "C" __declspec(dllexport) bool Exist(const char* name) { //if (g_Queues.find(name) != g_Queues.end()) // return true; //else // return false; return false; } Inside my C# class I have the following: [DllImport("Whisper.dll", EntryPoint="Exist", CallingConvention=CallingConvention.Cdecl)] public static extern bool Exist(string name); Yet, whenever I call my function it ALWAYS returns true, even when I commented out my little function and made it return false. I have the feeling there is something wrong with my calling convention or any other issue with P/Invoking my DLL, probably corresponding with the string and const char*, but for now I am completely clueless. What am I doing wrong? Why does it return true instead of false? EDIT: I have figured out this has nothing to do with the const char* or string, because the problem persists with an empty function. I've tried changing the calling convention between Cdecl and StdCall and neither work correctly. I've also managed to debug my DLL and it's being called correctly and does indeed return false, but once back into C# it somehow is true. Changing the CharSet also had no effect. I've made sure I've supplied my C# program with the latest and correct version of my DLL each time, so that shouldn't be an issue aswell. Again, I am completely clueless on why the result is true when I'm in fact returning false. EDIT2: SOReader provided me with a suggestion which fixes another important issue, see my comment. Sadly, it does not fix the return issue. EDIT3: I have concluded that changing the return type of Exist (bool) into (int) suddenly makes it return the correct number (true = 1, false = 0). That would mean that there may be an issue between C++'s bool and C#'s bool. I can continue using an int as a bool, but that would still not explain the original problem. Maybe somebody else can enlighten me on this one? Perhaps it has to do with the fact that I'm using x64 (although both pojects are compiled as x86)

ANS：

1）

I found the solution for your problem. Your declaration should be preceded with this marshaling: [return:MarshalAs(UnmanagedType.I1)] so everything should look like this: [DllImport("Whisper.dll", EntryPoint="Exist", CallingConvention=CallingConvention.Cdecl)] [return:MarshalAs(UnmanagedType.I1)] public static extern bool Exist([MarshalAs(UnmanagedType.LPStr)] string name); I tested it in my very simple example and it worked! EDIT Why this happens? C defines bool as 4 bytes int (as some of you have said) and C++ defines it as 1 byte. C# team decided to use 4 byte bool as default during PInvoke because most of the system API function use 4 bytes values as bool. If you want to change this behavior you have to do it with marshaling specifying that you want to use 1 byte value.

2）

I tested your code and it returns false for me. So there must be something else going on. Are you sure you are recompiling the DLL properly? Try deleting the .DLL and doing a rebuild. Other than that everything seems to be fine assuming . By default the marshalling will handle the .NET string to const char* without having to decorate it with Marshal attributes, whether the DLL is compiled as ANSI or Unicode. See http://msdn.microsoft.com/en-us/library/s9ts558h.aspx#cpcondefaultmarshalingforstringsanchor5

3）

This is actually caused by EAX not being fully cleared out by typical C++ code that returns a bool. It's typical for EAX to contain some bogus value when entering a function, and to return false the compiler would typically emit xor al, al. This clears out only the LSB of EAX, and causes C# code to interpret the resulting non-zero value as true instead of false.

4）

C'sbool is actually int, as there is no boolean type in the original C language. That means that if C#'s DLLImport is designed to interop with C code, then they will expect that C#'s bool to correspond to C's int. While this still doesn't explain why false would become true, fixing it should fix the problem. http://msdn.microsoft.com/en-us/library/system.runtime.interopservices.unmanagedtype.aspx This says that UnmanagedType.Bool is the Win32 BOOL, which is an int.

- See more at: http://www.mzan.com/article/4608876-c-sharp-dllimport-with-c-boolean-function-not-returning-correctly.shtml#sthash.nhsktqYV.dpuf