After failing at breakpointing and debugging request and response headers I almost snapped Wireshark to my connection to determine why I was getting an WebException of “The remote server returned an error: (417) Expectation Failed“. A little searching took me to this site with the solution. The simple solution is to set the static property System.Net.ServicePointManager.Expect100Continue to False before making your call.
Documentation of that property can be found here.
I write lots of desktop tools that help me do my job more efficiently. For instance, I’m normalizing a 98 million row live table and do to the nature of the server, table locks, etc, I’ve decided to pull the data down in batches using a VB.Net app, analyze it and perform a batch update. The high-level code is simple: pull the data, group it and then execute some UPDATE SQL statements. But 98 million rows in a partially non-indexed table takes a looooong time, and maybe its my agency background, but I want some feedback as to what’s going on, and I want more than just a Trace/Console output. At first glance you might just add a progress bar and update it every time you do something, but unfortunately your code loop will block the UI thread from updating until complete (although you can throw some Application.DoEvents in there).
So enter multi-threading. If you think this is overkill then a) you obviously don’t know me and b) I’ve been doing this for years, it only takes a little bit more work to add it. The code below will show you how to create a multi-threaded app with a responsive UI. The code below is all done in Visual Basic 2008 Express Edition.
1) Create a new VB.Net Windows Forms Application.
2) Add a ProgressBar and a Button to the form, leave the names the default names
3) Resize the ProgressBar so you can see more of it
4) Double-click the Button to go into code-behind
5) Add a new class to the project called “Worker” and add the following code to it:
Option Explicit On
Option Strict On
Public Class Worker
Private _curRun As Integer
Private _maxRuns As Integer
Public ReadOnly Property CurRun() As Integer
Get
Return Me._curRun
End Get
End Property
Public ReadOnly Property MaxRuns() As Integer
Get
Return Me._maxRuns
End Get
End Property
Public Sub New(ByVal maxRuns As Integer)
Me._maxRuns = maxRuns
End Sub
Public Sub Start()
For Me._curRun = 1 To Me._maxRuns
System.Threading.Thread.Sleep(250)
Next
End Sub
End Class
This code will be doing our primary work. In this case its not actually doing anything except sleeping, you’ll want to replace the System.Threading.Thread.Sleep(250) with your own logic, obviously. We’ll instantiate this code in our primary UI thread later, telling it how many times we want it to loop. I always prefer to pass my required variables in the constructor and make them read-only properties so that I know that they’ll be set. One other thing that you’ll notice, usually in the For loop you’d do something like For I = 1 to Me._maxRuns, creating a local counter variable. Instead, our code creates a class-level variable and gives read-only permission to it so that other code (on another thread) can determine where we’re at. That’s it from the worker class’s point of view. The only bit of thread-related code in here is the Sleep method and that’s for example purposes only.
The next step is to add some logic to the main form’s code-behind to create our worker, create a thread to execute the worker and create a thread to monitor the worker and report back to the form and changes. Before I slap the code up let me explain how it will work. After clicking the button the form’s primary (and only, initially) thread will create an instance of the Worker class. It will then create a new Thread object and tell it that when we execute it it should call the Worker’s Start() method. This alone would give you a responsive UI but unfortunately the Worker thread can’t safely update the UI because they’re on different threads (and VS 2008 will actually throw errors if you try). You can get around this using events or delegates in the worker object but I prefer to make my worker object 100% completely thread un-aware, meaning that it doesn’t report back to anyone, it just does the work and if someone wants to inspect its progress, fine, but it doesn’t really care. So how do we do this? We introduce a third thread whose only job is to check the status of the worker object and tell the main thread to update the progress bar. The code is so simple that I don’t even bother wrapping it in a separate class, I just include a Monitor() method in the form’s code-behind. The code runs a loop until the worker thread signals that its done, sleeping for a bit between each loop so that we don’t flood the UI with update calls (there’s no sense in sending “set the progress bar to 1″ a million times a second). Threads automatically handle adjusting their ThreadState so we don’t have to worry about that.
One of the things that takes a little bit of getting used to is the concept that one thread cannot modify controls on another thread. So our Monitor() method can’t actually update the UI. Instead what it does is ask the UI thread to update the UI on behalf of the monitor thread. I know, sounds weird. To do this, our UpdateUI method first calls Me.InvokeRequired which basically asks the form to check if the code calling this method is on the same thread. If its on the same thread it just adjust the progress bar. If its not on the same thread it ask the form to call the same method with the same parameters on its behalf using the Invoke() method. Invoke takes two parameters although the second is optional. The first is a delegate which is basically a type-safe function pointer. If you were using Javascript this would be the same as passing the function’s name without parentheses or just using the function(){} syntax. The second is an array of objects that represent the parameters to pass to the method. The syntax may look confusing but that’s just VB’s way of declaring an array. I think they’re fixing that in VB10 but I’m not sure. We also use a similar concept when the main thread is done to re-enable the button. And that’s it.
Option Explicit On
Option Strict On
Imports System.Threading
Public Class Form1
Private MonitorThread As Thread
Private WorkerThread As Thread
Private W As Worker
Private Delegate Sub UpdateUIDelegate(ByVal curIndex As Integer)
Private Delegate Sub WorkerDoneDelegate()
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Me.Button1.Enabled = False 'Disable the button
W = New Worker(50) 'Create our Worker object
Me.ProgressBar1.Maximum = W.MaxRuns 'Set the maximum value of the progress bar
WorkerThread = New Thread(AddressOf W.Start) 'Create our Worker thread and tell it that when we start it it should call our Worker's Start() method
MonitorThread = New Thread(AddressOf Monitor) 'Create our Monitor thread and tell it that when we start it it should call this class's Monitor() method
WorkerThread.Start() 'Start the worker thread
MonitorThread.Start() 'Start the monitor thread
End Sub
Private Sub Monitor()
Do While WorkerThread.ThreadState <> ThreadState.Stopped 'Loop until the Worker thread (and thus the Worker object's Start() method) is done
UpdateUI(W.CurRun) 'Update the progress bar with the current value
Thread.Sleep(250) 'Sleep the monitor thread otherwise we'll be wasting CPU cycles updating the progress bar a million times a second
Loop
WorkerDone() 'If we're here, the worker object is done, call a method to do some cleanup
End Sub
Private Sub UpdateUI(ByVal curIndex As Integer)
If Me.InvokeRequired Then 'See if we need to cross threads
Me.Invoke(New UpdateUIDelegate(AddressOf UpdateUI), New Object() {curIndex}) 'If so, have the UI thread call this method for us
Else
Me.ProgressBar1.Value = curIndex 'Otherwise just update the progress bar
End If
End Sub
Private Sub WorkerDone()
If Me.InvokeRequired Then 'See if we need to cross threads
Me.Invoke(New WorkerDoneDelegate(AddressOf WorkerDone)) 'If so, have the UI thread call this method for us
Else
Me.Button1.Enabled = True 'Otherwise just update the button
End If
End Sub
End Class
One last thing. Debugging a multi-threaded program can be a PITA so one of the things that I do is include a switch to turn it on or off as needed. I prefer to do this with a Compiler Directive by adding #Const MULTI_THREADED = True to the top of my code-behind. Then there’s just a simple change to the button’s click event:
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Me.Button1.Enabled = False 'Disable the button
W = New Worker(50) 'Create our Worker object
#If MULTI_THREADED Then
Me.ProgressBar1.Maximum = W.MaxRuns 'Set the maximum value of the progress bar
WorkerThread = New Thread(AddressOf W.Start) 'Create our Worker thread and tell it that when we start it it should call our Worker's Start() method
MonitorThread = New Thread(AddressOf Monitor) 'Create our Monitor thread and tell it that when we start it it should call this class's Monitor() method
WorkerThread.Start() 'Start the worker thread
MonitorThread.Start() 'Start the monitor thread
#Else
W.Start()
Me.Button1.Enabled = True
#End If
End Sub
When MULTI_THREADED is False we’ll invoke the worker, blocking the UI thread from ever updating, do the work and reset the button. This can save you a lot of time when debugging. This is also why I like to keep my worker object thread un-aware.
I’ve been normalizing a table with 98 million rows for a couple of days now and with a recent change I’ve been able to run my batches in groups of 100,000 instead of just 10,000. The program doing the normalization pulls down 100,000 records at a time, analyzes and groups them and performs an UPDATE for each item in the group, passing the primary keys back to the server in an IN () clause. Everything was fine for a while but with the upgrade to 100,000 and with some groupings sending thousands of keys in the IN clause I started to get the following error:
Internal Query Processor Error: The Query Processor Ran Out Of Stack Space During Query Optimization.Microsoft says to insert the keys into a temp table and join with that but that doesn’t really work for me. Instead I just decided to break up the UPDATE query into ones with more manageable IN clauses.
Originally I had something like:
Dim mySQL = String.Format("UPDATE Tbl SET Col2=xyz WHERE Col1 IN ({0})", Join(TheList.ToArray(), ","))
Using Com As New SqlCommand(mySQL, con)
Com.CommandType = CommandType.Text
Com.ExecuteNonQuery()
End Using
The new code breaks things into more manageable sizes:
Dim AmmountToProcess = 1000
Do While True
Dim Y = TheList.Take(AmmountToProcess)
Dim mySQL = String.Format("UPDATE Tbl SET Col2=xyz WHERE Col1 IN ({0})", Join(Y.ToArray(), ","))
Using Com As New SqlCommand(mySQL, con)
Com.CommandType = CommandType.Text
Com.ExecuteNonQuery()
End Using
TheList.RemoveRange(0, Math.Min(TheList.Count, AmmountToProcess))
If TheList.Count = 0 Then Exit Do
Loop
The new code uses the wonderful Take extension method to grab 1,000 items. Take luckily doesn’t error if you ask for more than available. The smaller UPDATE query runs and then we remove the first 1,000 items (or however many are left) from the list. Repeat until the list is empty.
If you ever make a PNG on the fly using ASP.Net and try to write it directly to the Response.OutputStream you’ll get the exception “a generic error occurred in gdi+“. If you change the output type to JPG or GIF it runs just fine, just PNG breaks.
Now usually .Net exceptions are pretty easy to trace (at least for me) just by name but if you try searching for a generic error guess what you get? Generic results. The problem is that the PNG encoder needs a “seekable stream” when outputing the image and the Response.OutputStream is not seekable. Luckily the fix is easy, just write it to a MemoryStream (which is seekable) and then BinaryWrite the MemoryStream.
Assume that BI is a System.Drawing.Image. Instead of:
BI.Save(Response.OutputStream, System.Drawing.Imaging.ImageFormat.Png)you need to use:
Using MS As New MemoryStream()
BI.Save(MS, System.Drawing.Imaging.ImageFormat.Png)
Response.BinaryWrite(MS.GetBuffer())
End Using
UPDATE: I should found out that in ASP.Net 3.5 the <customErrors> section in web.config now supports the attribute redirectMode which default to ResponseRedirect but can be changed to ResponseRewrite. This will force the proper HTTP status codes to come through for errors.
So I launched a new site porting it from static .html files to dynamic .aspx. I then created an entry for 404’s in web.config and also told IIS to send 404’s to the same page for everything that ASP.Net didn’t handle (images, html files, etc). So simple, right? Unfortunately both ASP.Net and IIS don’t appear to actually send a 404 status code for 404’s if you have a 404 handler specified. IIS actually sends a 200 OK and performs the equivalent of a Server.Transfer. ASP.Net actually sends a 302 Found which is a client-side redirect.
Now, you may not care about this, but if you’re trying to purge search engines of old content you should.
The fix, at first glance, is simple. In Page_Load on the 404 handler page just send the proper status code using Response.StatusCode = 404. This actually fixes the problem with IIS but we’ve still got an ASP.Net problem. ASP.Net will actually still send the 302 redirecting people to the 404 page and then send the 404 code. So if page Blah.aspx doesn’t exist you’ll get a 302 code saying to go to PageNotFound.aspx. When the browser requests PageNotFound.aspx they finally get the 404 response. Now maybe that’s good enough for you, but not for me. When I 404, I want to know right away, not after a redirect.
I found the solution here and have reproduced the code below for easy copy-and-paste. They don’t explicitly mention it in the article but you MUST set the <customErrors> to Off or at least RemoteOnly. I know that sounds weird but the code below still actually processes the <customErrors>, it just does it instead of letting ASP.Net handle it for you.
The article talks about creating a DLL but you can just as easily drop it into App_Code as a file named HttpErrorModule. Then just add <add name="HttpErrorModule" type="HttpErrorModule" /> to your <httpModules>.
One other thing to note, this module actually overrides all errors and sends their proper codes in the response stream. This shouldn’t cause any problems but you should be aware of it when debugging. You could very easily add some logic to check for 404’s only, too.
'Code from: http://www.colincochrane.com/post/2008/01/ASP-NET-Custom-Errors-Preventing-302-Redirects-To-Custom-Error-Pages.aspx
Option Strict On
Option Explicit On
Imports System.Web
Imports System.Web.Configuration
Public Class HttpErrorModule
Implements IHttpModule
Public Sub Dispose() Implements System.Web.IHttpModule.Dispose
'Nothing to dispose.
End Sub
Public Sub Init(ByVal context As System.Web.HttpApplication) Implements System.Web.IHttpModule.Init
AddHandler context.Error, New EventHandler(AddressOf Context_Error)
End Sub
Private Sub Context_Error(ByVal sender As Object, ByVal e As EventArgs)
Dim context As HttpContext = CType(sender, HttpApplication).Context
If (context.Error.GetType Is GetType(HttpException)) Then
' Get the Web application configuration.
Dim configuration As System.Configuration.Configuration = WebConfigurationManager.OpenWebConfiguration("~/web.config")
' Get the section.
Dim customErrorsSection As CustomErrorsSection = CType(configuration.GetSection("system.web/customErrors"), CustomErrorsSection)
' Get the collection
Dim customErrorsCollection As CustomErrorCollection = customErrorsSection.Errors
Dim statusCode As Integer = CType(context.Error, HttpException).GetHttpCode
'Clears existing response headers and sets the desired ones.
context.Response.ClearHeaders()
context.Response.StatusCode = statusCode
If (customErrorsCollection.Item(statusCode.ToString) IsNot Nothing) Then
context.Server.Transfer(customErrorsCollection.Get(statusCode.ToString).Redirect)
Else
context.Response.Flush()
End If
End If
End Sub
End Class
I’m converting a site from static HTML to dynamic ASP.Net but I don’t want to rename every page or worry about dropping in pages in the search engines. So I decided to just map HTML files to the ASP.Net processor. I could have sworn that I had done this before and all I had to do was make the normal IIS setting change (http://msdn.microsoft.com/en-us/library/bb515343.aspx). But that just returned a blank page. The fix is to add a buildProvider and an httpHandler.
<configuration>
<system.web>
<compilation debug="false">
<buildProviders>
<add extension=".html" type="System.Web.Compilation.PageBuildProvider"/>
</buildProviders>
</compilation>
<httpHandlers>
<add verb="*" path="*.html" validate="false" type="System.Web.UI.PageHandlerFactory" />
</httpHandlers>
</system.web>
</configuration>
Ran into this error recently on the system that I moved. My problem was that the program was running under a different user account which didn’t have the needed Adobe InDesign PDF export presets loaded. When my app tried to export a PDF (with no error handling) the system threw an UnhandledException and crashed to desktop with nothing but crap in the event log. To help diagnose the problem I added an UnhandledException handler (sounds weird, I know) that gave me a nice stack trace. I ran it with the debug version so I got line numbers, too. You can add an UnhandledException handler like so:
Sub UnhandledException(ByVal sender As Object, ByVal e As UnhandledExceptionEventArgs)
If e.IsTerminating Then
Dim o As Object = e.ExceptionObject
MessageBox.Show(o.ToString)
End If
End Sub
Public Sub New()
AddHandler AppDomain.CurrentDomain.UnhandledException, AddressOf UnhandledException
End Sub
We moved two servers yesterday that were originally part of a Windows domain. At the new place the domain would no longer exist (we also transffered ownership) so we removed the machines from the domain and set them to be part of a standard workgroup. For the most part this worked accept there was one application that was pulling XML files from one machine to another using UNC paths and standard FileStreams. ASP.Net was throwing an IOException of Logon failure: unknown user name or bad password. Makes sense, the FileStream call wasn’t doing any type of authentication and to the best of my knowledge, there’s no way to specify any credentials (short of using unmanaged code). I started to think of ways around this. In the past I had played with AppDomains which allowed me to reset the credentials but I remember that opened up a whole other can of worms. Then I realized that I was just using the FileStream to give me a String that I was passing to XDocument.Parse(). Well, XDocument also has a Load() method which takes a URL. So, a quick easy solution was to map the UNC path in IIS as a virtual folder, specify the credentials for the remote machine and then use the XDocument.Load() method to load the data. If the document wasn’t an XML document I could have used the same process except use the WebClient instead (Dim S = New System.Net.WebClient().DownloadString("http://..."))
