I recently had to debug a problem for my current client which was exceptionally weird and I was able to utilize WinDbg to help get to the bottom of the problem relatively quickly. Basically the application in question is an asp.net application that takes a custom object and puts in in an MSMQ message (which currently uses the default binary serializer). The class in question is marked with the Serializable attribute, has a couple of primitive members (e.g., ints, strings) and a couple of NameValueCollections. One of the NameValueCollections is assigned directly from the HttpRequest.Headers NameValueCollection like this:
myObject.Headers = request.Headers;
One day we started getting this error:
System.Runtime.Serialization.SerializationException: Type 'System.Web.HttpHeaderCollection' in Assembly 'System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' is not marked as serializable.
This made absolutely no sense since, like my class, the NameValueCollection class was also marked with the Serializable attribute. Not only that, but where was the System.Web.HttpHeaderCollection coming from?
So since I was already debugging, I took a memory dump of the asp.net process from the command line like this:
cscript.exe adplus.vbs -hang -p 1808 -quiet -o c:temp
The "1808" was simply the PID of my w3wp.exe process and I output the dump to the c:temp directory.
I then launched WinDbg and opened the crash dump that was produced in the previous step. First, you have to run (the sos extensions is what allows WinDbg to understand .NET 2.0):
.loadby sos mscorwks
So the first thing I wanted to see was all the types in memory and their corresponding method tables so I ran:
!dumpheap -stat
A snippet of the results:
00000642802ae6c0 5 320 MyNamespace.Foo
(just using Foo here for the class name. The real output included the fully qualified namespace with correct object name). The 00000642802ae6c0 is the method table for my Foo object. The 5 shows that we currently have 5 of my Foo objects loaded into memory. The 320 is the total size (320/5 tells me each of my objects is taking up 64 bytes).
Now I can narrow in on my issues a little bit. This shows me all 5 of the memory addresses for my Foo objects:
0:000> !dumpheap -mt 00000642802ae6c0
------------------------------
Heap 0
Address MT Size
total 0 objects
------------------------------
Heap 1
Address MT Size
00000000c0012e18 00000642802ae6c0 64
total 1 objects
------------------------------
Heap 2
Address MT Size
000000010053b830 00000642802ae6c0 64
total 1 objects
------------------------------
Heap 3
Address MT Size
0000000140235c60 00000642802ae6c0 64
0000000140430fd8 00000642802ae6c0 64
00000001405275d0 00000642802ae6c0 64
total 3 objects
------------------------------
total 5 objects
Statistics:
MT Count TotalSize Class Name
00000642802ae6c0 5 320 MyNamespace.Foo
Total 5 objects
So now I'm going to pick one of these five objects and drill down further. I'll pick the one highlighted in blue that resides at 00000001405275d0:
0:000> !do 00000001405275d0
Name: MyNamespace.Foo
MethodTable: 00000642802ae6c0
EEClass: 00000642802baf08
Size: 64(0x40) bytes
(C:WINDOWSMicrosoft.NETFramework64v2.0.50727Temporary ASP.NET Filesroote22c255992c7e946assemblydl3fed41db4\000e24c3_21cac701\MyAssemblyName.DLL)
Fields:
MT Field Offset Type VT Attr Value Name
0000000000000000 4000028 8 0 instance 0000000140527610 requests
0000064274e85620 4000029 10 ...meValueCollection 0 instance 0000000140528d60 headers
0000064274e85620 400002a 18 ...meValueCollection 0 instance 0000000140527838 extraRequestData
0000000000000000 400002b 20 0 instance 0000000140527a38 userStateData
000006427881edc8 400002c 28 System.String 0 instance 0000000000000000 endOfLineData
0000064278825318 400002d 30 System.Boolean 0 instance 0 isOptOut
So I know the "headers" member is the one I want for focus on so I inspect that next:
0:000> !do 0000000140528d60
Name: System.Web.HttpHeaderCollection
MethodTable: 00000642bcefad78
EEClass: 00000642bcf3be38
Size: 120(0x78) bytes
(C:\WINDOWS\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
0000064278825318 4001165 44 System.Boolean 0 instance 1 _readOnly
00000642788426f0 4001166 8 ...ections.ArrayList 0 instance 0000000140528f78 _entriesArray
000006427881f650 4001167 10 ...IEqualityComparer 0 instance 00000000ffffc148 _keyComparer
00000642788448c8 4001168 18 ...ections.Hashtable 0 instance 0000000140529000 _entriesTable
0000064274e82360 4001169 20 ...e+NameObjectEntry 0 instance 0000000000000000 _nullKeyEntry
0000064274e9e7e8 400116a 28 ...se+KeysCollection 0 instance 00000001405481c0 _keys
0000064278857210 400116b 30 ...SerializationInfo 0 instance 0000000000000000 _serializationInfo
000006427882b338 400116c 40 System.Int32 0 instance 10 _version
000006427881d280 400116d 38 System.Object 0 instance 000000014054ad48 _syncRoot
000006427881f748 400116e a70 ...em.StringComparer 0 shared static defaultComparer
Domain:Value 0000000000136900:NotInit 0000000000177700:00000000ffffc110
00000642788d8ba8 4001179 48 System.Object[] 0 instance 0000000000000000 _all
00000642788d8ba8 400117a 50 System.Object[] 0 instance 0000000140548160 _allKeys
00000642bceef310 400101b 58 ...m.Web.HttpRequest 0 instance 00000001405265f0 _request
00000642bceef7b8 400101c 60 ....Web.HttpResponse 0 instance 0000000000000000 _response
00000642bcf3ad30 400101d 68 ...IIS7WorkerRequest 0 instance 0000000000000000 _wr
And NOW we see something extremely interesting. The headers member was supposed to be a NameValueCollection so why is my !do showing it to be an HttpHeaderCollection and what is this HttpHeaderCollection anyway and how is this System.Web DLL even compiling? So at this point, it's time for us to crack open trusty old Reflector and find its class definition:
internal class HttpHeaderCollection : HttpValueCollection
So this is an internal class to the System.Web DLL but it's being returned for a NameValueCollection property. Also notice that it is NOT marked with the [Serializable] attribute (which was our problem to begin with). It inherits from the HttpValueCollection (which is yet another internal class) and the HttpValueCollection (while it DOES have the Serializable attribute) inherits from the NameValueCollection object and THAT is how the code is compiling because of course this is a perfectly legal use of polymorphism.
So...in the end, all of these classes are marked with the Serializable attribute except for the HttpHeadersCollection (which I believe to have been an oversight on Microsoft's part) which is causing our serialization problems and we cannot control the internal code of System.Web. The fix ends up being pretty trivial - just iterate the NameValueCollection and adds the string primitives so that all serialization is happy.
But the real moral of the story is that this is a good real world example that shows how to debug an issue that, at first glance, seems impossible. Using WinDbg like this is a very repeatable process that can help you get to the bottom of memory issues (and others) quite fast.
