The importance of the Background Worker component


Serious Problem

I recently reviewed a very large enterprise application for one of the biggest retail companies in the country, and was aghast at the lack of multi-threading in the application. I resolved to ensure I post about it, though this subject has been covered numerous times over the years, and there are a few examples interspersed over the internet. New developers or maybe intermediate developers not working in the data access layer of a project may gloss over why this component is so important. You also don’t want to revisit an application in a few years time and have to make multi-threading modifications.

I came to realise the true importance of this component, because as I spoke with staff, and they talked me through the program working, at key stages during the program I would be told that after clicking a button that makes a database call  “the computer is thinking now, so wait for a while” or “when you click this, you cannot do anything”, or “don’t click that option because it will freeze the screen, and you won’t be able to use the computer for half an hour”.

The list of these warnings goes on and on, about using their system. The administration staff/information workers have tasks they have to run several times every day, and typically, they cannot use their system for half and hour or even more at times, because the queries they are running are executing on the UI thread. This quite frankly is an atrocious situation. Pretty much every part of their application that makes database calls, runs on the UI thread.

What ‘beggars belief’ is that this system is used by one of the largest retail companies in the UK, and that there are doubtlessly more retailers that are in the same boat. These retail organisations realised the power of computing many years ago, and their systems were developed at a time when patterns and practices had not been invented, and some very bad coding practices were cultivated. I estimate that this specific company that has triggered this post, is losing at least £100 000 annually in wages and lost productivity, because all their staff spend a third of their day twiddling their thumbs, while their computer screens are frozen because the UI thread is locked up (and this is a conservative estimate).

Demonstration

Seriously, if you develop any application that makes calls to a database you must learn to love and use this component on every form that makes a database call!

Typically you will create an interface that has a loadDataBackgroundWorker (when the form loads) and a saveDataBackgroundWorker (when the form is closed). There really is no excuse not to use this component, which incidentally, is framework agnostic, and is used in the new Silverlight 2 runtime. I know of some people that have run into issues with the dispatcher component in WPF, and they are using this in their WPF applications. In general, writing multithreaded applications is very difficult, the background worker component makes writing multithreaded applications so much easier.

To demonstrate how this works, create a new windows forms application in C# or Visual Basic (you can use the express editions) and drag a progress bar and two buttons onto the form. Name the appropriate button “startButton” and “MessageButton”.

TheForm

In the code behind, handle the events thus (Visual Basic example is beneath)

C#

using System;

using System.ComponentModel;

using System.Windows.Forms;

 

namespace BackgroundWorkerDemo

{

    public partial class Form1 : Form

    {

        public Form1()

        {

            InitializeComponent();

 

            this.progressBar1.Minimum = 0;

            this.progressBar1.Maximum = 9;

            this.progressBar1.Value = 0;

        }

 

        private void messageButton_Click(object sender, EventArgs e)

        {

            MessageBox.Show("Hello World!");

        }

 

        private void startButton_Click(object sender, EventArgs e)

        {

            for (int i = 1; i < 10; i++)

            {

                this.progressBar1.Value = i;

                System.Threading.Thread.Sleep(1000);

            }

        }

    }

}

Visual Basic

Imports System.ComponentModel

 

Public Class Form1

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

        Me.progressBar1.Minimum = 0

        Me.progressBar1.Maximum = 9

        Me.progressBar1.Value = 0

    End Sub

 

    Private Sub messageButton_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles messageButton.Click

        MessageBox.Show("Hello World!")

    End Sub

 

    Private Sub startButton_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles startButton.Click

        For i As Integer = 1 To 9

            Me.progressBar1.Value = i

            System.Threading.Thread.Sleep(1000)

        Next

    End Sub

End Class

If you run the program, and click on the “show message” button, this shows the message “Hello World!”. If you then click on the start button and try to click the “show message” button you will find that you cannot trigger the message, until the progress bar has completed. Incidentally, System.Threading.Thread.Sleep(1000) just causes the thread the progress bar is running on to pause for a second (1000 milliseconds). If this were not included, the for loop would execute too fast, and you would not see the screen locking up.

This scenario is no different to the problem I have outlined by this major enterprise application. When users of the application click “load orders” or whatever option fetched data from the database, their application is locking the UI thread, and this leads to a very poor user experience, and statements like “the computer is thinking about things”.

Solution

Drag a background worker component onto your form from the toolbox, and set the reports progress property to true

ReportsProgress

Click the events button and handle all 3 events

BackGroundEvents

In the event handlers move the previous code so it looks like this

C#

using System;

using System.Collections.Generic;

using System.ComponentModel;

using System.Data;

using System.Drawing;

using System.Linq;

using System.Text;

using System.Windows.Forms;

 

namespace BackgroundWorkerDemo

{

    public partial class Form1 : Form

    {

        public Form1()

        {

            InitializeComponent();

 

            this.progressBar1.Minimum = 0;

            this.progressBar1.Maximum = 9;

            this.progressBar1.Value = 0;

        }

 

        private void messageButton_Click(object sender, EventArgs e)

        {

            MessageBox.Show("Hello World!");

        }

 

        private void startButton_Click(object sender, EventArgs e)

        {

            this.backgroundWorker1.RunWorkerAsync();

        }

 

 

        private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)

        {

            for (int i = 1; i < 10; i++)

            {

                this.backgroundWorker1.ReportProgress(i);

                System.Threading.Thread.Sleep(1000);

            }

        }

 

        private void backgroundWorker1_ProgressChanged(object sender, ProgressChangedEventArgs e)

        {

            this.progressBar1.Value = (int)e.ProgressPercentage;

        }

 

        private void backgroundWorker1_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)

        {

            this.progressBar1.Value = 0;

        }

    }

}

Visual Basic

Imports System.ComponentModel

 

Public Class Form1

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

        Me.progressBar1.Minimum = 0

        Me.progressBar1.Maximum = 9

        Me.progressBar1.Value = 0

    End Sub

 

    Private Sub messageButton_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) _

    Handles messageButton.Click

        MessageBox.Show("Hello World!")

    End Sub

 

    Private Sub startButton_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) _

    Handles startButton.Click

        Me.BackgroundWorker1.RunWorkerAsync()

    End Sub

 

    Private Sub backgroundWorker1_DoWork(ByVal sender As System.Object, ByVal e As DoWorkEventArgs) _

    Handles BackgroundWorker1.DoWork

        For i As Integer = 1 To 9

            Me.BackgroundWorker1.ReportProgress(i)

            System.Threading.Thread.Sleep(1000)

        Next

    End Sub

 

    Private Sub backgroundWorker1_ProgressChanged(ByVal sender As System.Object, ByVal e As ProgressChangedEventArgs) _

    Handles BackgroundWorker1.ProgressChanged

        Me.progressBar1.Value = CInt(e.ProgressPercentage)

    End Sub

 

    Private Sub backgroundWorker1_RunWorkerCompleted(ByVal sender As System.Object, ByVal e As RunWorkerCompletedEventArgs) _

    Handles BackgroundWorker1.RunWorkerCompleted

        progressBar1.Value = 0

    End Sub

End Class

When you run the application and click the message button after you click start, the long task (i.e. the work done connecting to your database) is executed on another thread asynchronously. this results in a responsive application that does not lock. The users at the aforementioned enterprise can then click into other options in their application and do other work, while the program processes.

Quick Tip : Ensure that you do not add any code that accesses the UI thread in the DoWork event. You may have a label that says connecting…. in your ToolStripLabel for instance. Make sure this code is where the backgroundWorker.RunWorker.Async() is called, and nowhere else. You can add a message so say “completed” or something similar in the RunWorkerCompleted event as that is on the UI thread.

I really cannot reiterate sufficiently clearly just how important it is to wrap every method in your application that makes a database call in a background worker. Not only  does this improve the efficiency and productivity of employees (users), but they come away with a very positive view of the application, which is not what the users at this company have. When training new staff they end up being on tenterhooks just in case they click on the wrong button, and cause the program to freeze. At busy times, or near the end of the day where work needs to be completed, the last thing you need, is an application that hangs for half an hour or even longer before you can use it again.

Please, please, please, use this component as much as you can in all your data access applications.

Choosing the correct database access technology for your application


Over the last year or so, I have been developing a line of business application (LOB). The application itself is quite dull to discuss because unless you know the business problems it tries to specifically solve, then you will probably fall asleep reading about it – so I shan’t compose a disquisition here.

Problem

A core requirement of a typical LOB application is that it is multi-user, and that at any given moment, you usually have a lot of people adding, accessing or modifying data. This presents a number of challenges e.g.

  1. You need to be able to control who has access to certain modules for the application, and who can modify, delete or add data.
  2. You need to ensure that if the same record is modified by 2 users at the same time, a level of concurrency is met.
  3. You need to be able to make certain modules of the application available via the internet or web service.
  4. You need  to shape your data

Linq to SQL/Linq to Entities

This is an inexhaustive list, but will be sufficient to allow the objective of this post to be comprehensible. When I started the aforementioned application development, I elected Linq to SQL for my data access, as that [Linq] made it very easy and super fast, to develop the application.

I did however start to run into some limitations e.g. the DataContext object – central to Linq and the new Entity Framework –  needed to be in the same form for all your Create, Update, Delete or CRUD operations to work correctly. This resulted in all my business logic, presentation logic and data access logic being in the same file. Whilst this worked, the application codebase quickly became unmaintainable and very ugly. It was possible to separate this code, but this involved workarounds, and the object count in the application inflated.

DataSets

The chief problem with Linq to SQL and the Entity Framework is that you must always remain connected to the database, once a connection is broken, then change tracking for your application is lost. After a few months of trying various workarounds, I decided to go back to the tried and tested ADO.NET datasets. I have a tutorial here, which allows you to create a WCF service using datasets. The biggest strength with datasets is that they allow for a disconnected environment. What happens with datasets is that when you request data from a database, you have the tables and columns copied locally (into the dataset) which your users can then modify, and the changes then be persisted back to the underlying database at a later stage. This improves database performance, as the dataset just gets the data and breaks the connection to the database immediately, unlike the always connected Linq to SQL. In multi user applications, this is a serious concern.

Solution – SQL and Stored Procedures

  1. Stored procedures allow the database administrator to give certain users read, write or update permissions on a stored procedure. This allows for an extra layer of security for your data and solves the first problem above.
  2. Stored procedures make it easy to handle concurrency violations.
  3. Once your stored procedures are defined, you can point an ASP.NET, Silverlight or Web Service to them, negating rewriting the same logic.
  4. You can provide joined and computed columns with aggregates. Typically you need to display related tables in the same datagrid, or provide computed columns with totals for example. Stored procedures make this easy, where you would otherwise have to handle datagrid events and perform calculations on the client. This increases code bloat and decreases performance.

At the heart of datasets is SQL, and you have the option to either embed the SQL statements into the dataset, or you can create stored procedures where the statements are saved as functions in SQL Server, and you use these functions in your dataset.

I must admit that at first I found stored procedures extremely daunting, and avoided them as best I could, as they seemed overly complex. Subsequently,I was forced into learning them because my application architecture necessitated them, and have subsequently and might add pleasantly found that they are not as difficult to grasp as I thought. SQL is a pure functional language. Not “pure” in the sense of a functional language like Haskell, but it is composed purely of functions that one uses in their procedures. It is this functional syntax that most C# or Visual Basic.NET developers loathe, as it is completely different to a general purpose programming language.

In truth when you use Linq to SQL or Linq to Entities, your queries are converted to SQL anyway, so in my book, knowing SQL well makes you a better developer because

  1. You understand what the C#/VB abstraction layer is generating, why and how.
  2. You understand functional programming better, compared to the functional constructs added to C# or VB, as SQL is a pure functional language.

Digression

I was in a second hand bookstore (I like collecting books new and old), and picked up a 10 year old book on Transact-SQL. It was one of those “Teach yourself Transact-SQL in 21 days” type books I usually detest, but I am really pleased with it. The book is wonderfully written, but what is most striking is the fact that most SQL has stayed the same over the last decade or so – if not back to the IBM SEQUEL days in the mid–seventies.

Over the last decade Microsoft have introduced many flavours of ADO.NET, and most have always had their drawbacks. The reason I like SQL so much, is that is is not going away anytime soon, so knowing it well, situations you advantageously over the flavour of the month data access solutions, and one can use tried and tested technologies and methodologies, which are essential in multi-user distributed applications.

Domain Driven Design and MVC

Another discovery I have found with using stored procedures, is that you inadvertently design the application domain first, then implement the logic in your application. This is somewhat counter to the style of programming your typical developer likes, but the advantages of this type of design, far outweigh any other considerations. If you look at the MVC Framework Microsoft is pushing at present, you soon discover that stored procedures are your model (In Model View Controller), so all you have left to implement are your View and Controllers, again a default advantage to separating your data access logic from your application.

Application Changes and Updates

A key aspect of enterprise applications is reporting. In general, reporting is a moving target, and companies generally require that their reports are modified frequently to obtain the latest data to give them the competitive edge. With your data access logic tied to Linq to SQL/Entities, one would need to recompile the application, then force every terminal in the enterprise to re-install their application so they can access the latest reports. Using stored procedures, you can simply create a powershell script and run this on the server. The new reports are then available to every terminal, without a re-install, which is very expense if you have hundreds of terminals.

I hope the above has provided you with a basic explanation as to why you need to be judicious about what data access technology you choose, and that the newer technologies are not always the best in some situations.

WPF Composite Application Guidance


Glenn Block has announced that Composite Application Block (CAB) guidance has been released for Windows Presentation Foundation named Prism.

If you’re new to building software or are intermediate, one problem area you are likely to encounter is how to modularise your code. Most applications in general follow similar patterns, whether it be design or code. What CAB offers is a unified way of developing software so if you work on a project written several years before, it is very easy to learn, change, and maintain, because you understand the applications construction. Given the fact that the most expensive part of any software application is in the maintenance of the application, and change requests, there are profound benefits to building modularised applications. It is far easier, consequently cheaper to maintain and change. We all know how astronomicallly high how much software costs to develop.

A lot of the concepts are heavy going to begin with, but if you stick at it you will reap the rewards.

Spec#, a new dawn for programming languages


Spec# pronounced ‘Spec Sharp’ is new attempt at a more cost effective way to develop and maintain high-quality software. This language is being developed by Microsoft Research and has left me completely dumbstruck. My favourite programming language is C# of which I’m self taught, but I was formally trained in C/C++.

Spec# is an extension to C# that introduces the notion of contract based programming. If you have tried Windows Communication Foundation, you will be familiar with the notion of contracts. Spec# takes things to a whole new level, and is very very explicit – this is a complete understatement by the way.

When you are defining your encapsulated fields and methods you have to go into a great more detail about how those fields and methods will operate, all at compile time. In C++, C# or Visual Basic, a developer usually writes a chunk of code and then compiles that code into a .dll or executable. It is only at this stage when problems are identified, and resolutions are effected. Spec# does not allow this because as you write your code it checks everything. One of the key features is that you have to specify values as non-null. This is a big deal.

What non-null stipulations allow for, is very performant code (‘performant’ is a word used by Technical Fellows so it is now a word – as far as I am concerned). This is because code is not being checked for being null which is something that is always done for Common Language Runtime languages. The way the Spec# team achieve this is through their ‘brain-box’ knowledge of Microsoft Intermediate Language.

I will certainly be keeping my eyes on this one.

Moving over to ALT.NET


Kevin Moore has planted the seeds of doubt in me again. In this article he links to Joel Spolsky’s article Why I Hate Frameworks.

Obviously this is a very popular article, and is very humorous and perceptive. Some perceptions on software do indeed last a long time. I have just read through the white-paper for SCSF and that is pretty heavy going. The example in the white-paper is for a National Bank, with thousands of users and appears vastly over engineered for the purposes which I require. Joel’s article really does resonate once you get to about page 10 of the 50 or so pages.

What does a user for a small to medium sized application, that must perform well and adhere to best practices use? It appears as if there are solutions for the mega problems but non for the mini or micros ones. The .NET framework is increasing the ability for individuals for write far fuller applications, with fewer staff than ever before.

Due to my present and abject disgruntlement, I’ve decided to go ALT.NET, and see if that ‘bears any fruits’. This is something I’ve heard of for some time now, in fact, as bar back as this article. I really am getting rather fatigued at the present state of things and hope things like http://www.castleproject.org/ can help bring about the development ascension I yearn.

How to use the Office 2007 Ribbon Control


The lead designer for the ribbon has this video from mix 08.

I really cannot recommend highly enough just how much you should watch this. Especially if you are using a ribbon control in your application. Jensen takes you on a tour of the prototyping the Office team did, consequently you know when to use the ribbon and in what context it is correct to do so. I have come across a lot of developers who have just used the ribbon ‘willy-nilly’ because it was the ‘new kid on the block’, and resulted in creating a really bad application, from a usability point of view. The ribbon solves one key user interface problem, and understanding what this problem is will result in you creating a better application.

There are lessons to be learned here that mean that even if you are not using the ribbon, just the thought process itself is sufficently edifying. It is fascinating just how bad the Office 2003 user interface is, and the scalability that the ribbon creates for your application. One is left beyond doubt just why Office 2007 is far better than the previous version, with an unimpeachable presentation from Jensen.

Thoughts on the Composite Application Block


The problem the Acropolis team had, was wanting to create a general purpose UI framework. The plan was to make it easy for developers to just implement the business logic, when designing applications for the line of business market. This was the driving force for Acropolis. What they soon found out, was that you need designers and very skilled ones at that, to create the compelling interfaces, something their budget did not allow for.

This whole area is ambiguous, because you can implement the Model View Controller framework in Windows Presentation Foundation (excellent example here), and using Windows Communication Foundation/Windows Workflow Foundation or ‘Workflow Services’ as they are collectively known, you pretty much get the design pattern alluded to beneath;

You just have the MVC client instead of the integrated/composite view.

You can manage the scalability of your smart/web client either on the client, or at the service level. If you use databinding in windows forms, you can databind the same logic to Windows Presentation Foundation. It’s still the same datasets/table adapters and so forth. It becomes immediately apparent that it is folly to use Language Integrated Query on the smart/web client because you cannot easily swop or use windows forms/WPF/Web clients, because your data access logic is inadvertently tightly coupled with the UI. Until Linq is developed to easily allow for usage in service based scenarios, and you do not have to jump through technical hurdles and cross layer obstacles to get it to work, steer well clear of it.
If however your business logic resides in the service/workflow, or n-tier datasets/stored procedures, you can service all three with very minimal effort. I have an application that is smart client (windows forms) based, but the WCF service can easily ‘service’ a web or WPF client (once WPF stops running like a dog with three legs).

One can then concentrate solely on the creation of an attractive user interface.

A digression is that, SQL server 2008 now has intellisense and a host of features that pretty much allow for the composition of SQL queries as you would in a rich IDE like Visual Studio using SQL Server Management Studio. With the present rage in functional programming (concurrent programming) and SQL being a ‘functional language’, I actually enjoy creating my data access on the server, for all the ease Linq gives. SQL is a purer functional language than C#, and I prefer it. With concurrency being focused on, just the fact that SQL is functional, means that if your data access logic resides on a many core server, the concurrent data crunching occurs on the server, rather than in parallel Linq queries (ad-hoc ‘for’ loops) on the client. This results in an extremely well performing application. WPF/Silverlight can then consume the limited resources of the client machine