WCF service performance issue

 

I was recently engaged in a consulting project sponsored by one of the biggest HK Premier customer. The project is kinda SOA and will leverage Microsoft SOA technology Windows Communication Foundation (WCF). The project as a whole may be composed of several different consumers for instance ASP.NET web application, WinForm application etc. and one middle tier WCF service along with backend SQL database. Currently, the problem we are facing is the consumer (client) application sent out the requests and was just pending for WCF service to return.

 

What’s wrong?

 

On the client side, the outstanding requests are “POST /BetSlipIB/services/Login.svc” and the call stacks are similar to:

OS Thread Id: 0xc60 (63)

ESP       EIP    

1debe56c 77005e74 [NDirectMethodFrameStandalone: 1debe56c] System.Net.UnsafeNclNativeMethods+OSSOCK.recv(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags)

1debe584 6be3f864 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)

1debe5bc 6be3f760 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags)

1debe5dc 654417df System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[], Int32, Int32, System.TimeSpan, Boolean)

1debe640 65441757 System.ServiceModel.Channels.SocketConnection.Read(Byte[], Int32, Int32, System.TimeSpan)

…

1debf858 70d41b4c [ContextTransitionFrame: 1debf858]

 

So getting back to the WCF service side, we can dig further from the service hang dump. In fact, for one WCF request, there are two threads running behind one is from CLR ThreadPool (worker thread of ASP.NET ThreadPool) and another is an I/O thread managed by WCF IOThreadScheduler. Wenlong has an awesome post here: http://blogs.msdn.com/wenlong/archive/2008/04/21/wcf-request-throttling-and-server-scalability.aspx. In the below call stack, you can see that the thread is actually not executing anything related to Socket operations and I can’t find any active threads executing something in the WCF service as well. With that, the first thing that came into my mind is it could be the WCF throttling issue. As I remember the default concurrent instances of the WCF service to be running simultaneously is 10 and if it has reached the limit the thread will wait for the outstanding instances to be closed/disposed before continuing serving the next requests. Reference: http://msdn.microsoft.com/en-us/library/ms731379.aspx

OS Thread Id: 0xa5c (63)

ESP       EIP    

1c88ed08 77b55e74 [HelperMethodFrame_1OBJ: 1c88ed08] System.Threading.WaitHandle.WaitOneNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, UInt32, Boolean, Boolean)

1c88edb4 6d2f689f System.Threading.WaitHandle.WaitOne(Int64, Boolean)

1c88edd0 6d2f6855 System.Threading.WaitHandle.WaitOne(Int32, Boolean)

1c88ede4 6d2f681d System.Threading.WaitHandle.WaitOne()

1c88edec 65e4bf70 System.ServiceModel.Activation.HostedHttpRequestAsyncResult.ExecuteSynchronous(System.Web.HttpApplication, Boolean)

1c88ee1c 665945f8 System.ServiceModel.Activation.HttpHandler.ProcessRequest(System.Web.HttpContext)

1c88ee28 6516e2f6 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

1c88ee5c 65141a6c System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

1c88ee9c 65736173 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)

…

1c88f69c 010c239c [ContextTransitionFrame: 1c88f69c]

 

 

Resolution?

 

1. We did a code review and closed any outstanding service instances by adding something like:

            ServiceHost sh = new ServiceHost(typeof(wcf.service.ServiceDuplex));

            sh.Open();

…

            sh.Close();

2. Refer to the KB http://msdn.microsoft.com/en-us/library/ms731379.aspx and tune the WCF throttling parameters.

 

Best Regards,

 

Yawei Wang