Archive for August, 2010

Integration – Is It Worth It?

Aug 28

One thing I get asked to do a lot is integrate systems with each other or incorporate other systems into one we are currently building. In a lot of scenarios I have been starting to give some pushback on this. Sure integration is cool and all but should it be done?

Expense
The first (and ongoing) issue is the expense of integrating systems. First you have to understand the systems (or at least the APIs being exposed) of each system which can take a lot of time. Secondly we have to write code to integrate the services and adapt concepts from one system to another. This work also needs to be tested and debugged.

Perpetuating The Legacy
The bigger issue I have seen in organizations is that tying a legacy system into your system makes it harder to change or replace the legacy system. If a legacy system has 3-4 systems that depend on it then any changes have to be tested and approved by the consuming systems (expense again).

If we want to replace the legacy system with something better then we have two options. We can either rewrite the system and keep the old API exposed or we can throw it away and force our consumers to change. Rewriting a system and keeping the API may work IF the original API was perfect (or close to perfect). In practice I find that the API is ok but if a person were to rewrite it they could alleviate a lot of the issues and shortcomings discovered over the years. If we do a brand new interface then all our consumers need to change their code which can be…. expensive.

From being in this situation before I have compromised. I created a new system with a better API that was more expressive and simpler to use. I then created a separate project that exposed the old API and translated that to the new API. This does add some performance overhead in translation and lots of things to test (expensive).  It does allow new consumers to use a better API and allowed consumers of the old system to decide if it was worth migrating to the new API.

Integration For Integrations Sake
Many people automatically assume that if the information is in another system that we should integrate the two systems. This does help centralize rules, reduce duplication, and centralize information. This is all true but not always necessary. I would find it acceptable to have some minor duplication if it eliminated a coupling on another system. This is a very careful call to make though as this duplication can lead to duplication of maintainance. Even worse it can lead to their now being two sources of data so when a third system comes along it now consumes both sources for data to try and determine which one is accurate.

One classic example I have is from a client of mine. They have a system that dispatches their vehicles and another system that uses GPS to track their vehicles. They asked for these systems to be integrated as they are always switching between the two. In this case the value of integrating the systems was very low. Instead I recommend that all their operators get two monitors and have both applications open. This is not the 100% perfect solution (unless you are an accountant) but it gets the job done for the right price.

Alternatives
There are ways to integrate but not integrate. One of the simpler ones is to have a button/link/menu item/whatever that fires up a system you would normally integrate with and pass along necessary data. I.e. a button that runs “otherProgram.exe /RunSalesReport 10-10-2009 10-10-2010″ or http://server/SalesReport.aspx?start=10-10-2009&end=10-10-2010. Granted there is a bit of coupling there and some things to maintain but not as much as if we had to call a service, translate the results, and show it on a UI.

Another method is to use manual entry. An example of this would be to use one system to find a product and then type that products manufacture code into your system you are building. Sure it is not as fast and the costs of this slowdown will add up over time but is the cost of integration still higher than the cost it takes for a user to copy/paste a code from one program to another? Not an easy question to answer but one that should be evaluated.

The last method is one typically avoided: double entry. This is where the same (or similar) data is entered into two separate systems. An example of this might be for a sales operation. It may be acceptable for the sales system to have a list of all products, descriptions, and prices of products that they sell. The warehouse system might have its one list of products, dimensions, and inventory levels. As long as when a sale comes in that the order can be filled AND that the sales system does not need to know inventory levels when they make a sale (i.e. if there is none in the warehouse it will just get backordered). This is a very tight line to walk as if an item were entered into one system incorrectly the customer may order 20 Widgets but instead get 20 Gadgets due to the product codes not lining up.

Final Thoughts
I have learned that integration is more expensive than I first thought it would be. I thought that reducing code I had to write would make life easier. It does in some ways but in other ways it adds overhead to development and maintenance. If we just stop and think if it is really necessary to tie two systems together then we might just have a little less coupling and a little more freedom to upgrade some legacy systems. Not integrating is not the only solution…. but neither is integrating.

Filed Under: General, Rants

Learn From My Mistakes?

Aug 21

One of my frustrations with Silverlight has been that the async model is rammed down your throat. I understand the reasoning that it is good to have the UI thread responsive while long running calls are happening. But to me that is something that I code and I control.

The thing that gets me is that there are times where several network operations may have to happen in sequence. Here is a simple contrived example:

var service = new VehicleRentalService();
if (service.AreVehiclesAvailableToRent())
  {
  txtNumberOfCars.Text = service.GetNumberOfCarsForRent();
  if (service.IsTheFlakyOnlineRentalServiceRunning())
      {
       btnBookOnline.Enabled = true;
      }
  }

Now for the above example I would run all this on 1 background thread (leaving the UI nice and responsive) and make sure that I properly delegate the control changes to the UI thread (as controls are not thread safe).

Silverlight forces us to async the calls which leads to a lot of functions that chain together. I find this makes the code a lot harder to read and a lot harder to understand. Here is a hand written rewrite of the above example:

public function Start()
{

var service = new VehicleRentalService();
service.AreVehiclesAvailableToRentCompleted+=VehiclesAvailableToRentResponseRecieved;
service.GetNumberOfCarsForRentCompleted+=NumberOfCarsForRentResponseRecieved;
service.IsTheFlakyOnlineRentalServiceRunningCompleted+=IsTheFlakyOnlineRentalServiceRunningResponseRecieved;

service.AreVehiclesAvailableToRent();

}

public function VehiclesAvailableToRentResponseRecieved(object sender, VehiclesAvailableToRentCompletedEventArgs e)
{
   if (e.Result)
      {
      service.GetNumberOfCarsForRent();
      service.IsTheFlakyOnlineRentalServiceRunning();
     }
}

public function NumberOfCarsForRentResponseRecieved(object sender, NumberOfCarsForRentCompletedEventArgs e)
{
    Dispatcher.BeginInvoke(delegate { txtNumberOfCars.Text = e.Result });
}

public function IsTheFlakyOnlineRentalServiceRunningResponseRecieved(object sender, IsTheFlakyOnlineRentalServiceRunningResponseRecievedCompletedEventArgs e)
{
    Dispatcher.BeginInvoke(delegate { btnBookOnline.Enabled = e.Result });
}

(Some may argue that there should be a GetSystemStatus method on the service that returns all this info to which I would probably agree. For the sake of this post though I am going to run with it).

So the Silverlight code is longer, harder to read, and harder to maintain. It does have the benefit of getting the number of cars for rent and checking if the online booking system is running at the same time which is a nice plus.

For an example a log more complicated than this one I was using a WebRequest to converse with a server in a very back and forth fashion and found that the code quickly became very hard to read so I set about to change things. My idea was to create a class that uses async calls in the background but appear to be synchronous by blocking the caller. i.e.

string customerData = Request.GetResponse(“http://www.dummy.com/getcustomer.aspx”, “CustomerId=1″);
string customerAddress = Request.GetResponse(“http://www.dummy.com/getaddress.aspx”, ParseOutAddressId(customerData));

PLEASE DO NOT USE THIS CODE AS IT DOES NOT WORK (hence the post title):

    public class SyncRequestResponse
    {
        ManualResetEvent _requestCompleted;
        private WebRequest _request;
        private string _postData;
        private Uri _requestUrl;
        private static string _result;

        public string GetResponse(Uri requestUrl, string postData)
        {
            _requestCompleted=new ManualResetEvent(false);
            _requestCompleted.Reset();
            _postData = postData;
            _requestUrl = requestUrl;
            StartRequest();
            _requestCompleted.WaitOne(1000); //block current thread until the response is received or the timeout is reached
            return _result; //the result should be filled now
        }

        private void StartRequest()
        {
            _request = WebRequest.Create(_requestUrl);
            _request.ContentType = “application/x-www-form-urlencoded”;
            _request.Method = “POST”;
            _request.BeginGetRequestStream(GetRequestStreamCompleted, null);
        }

        private void GetRequestStreamCompleted(IAsyncResult result)
        {
                //request stream has returned so fill it
                var content = Encoding.UTF8.GetBytes(_postData);
                var stream = _request.EndGetRequestStream(result );
                stream.Write(content, 0, content.Length);
                stream.Close();
                _request.BeginGetResponse(GetResponseFromRequest, null);
        }

        private void GetResponseFromRequest(IAsyncResult ar)
        {
            var response = _request.EndGetResponse(ar);
            var sr = new StreamReader(response.GetResponseStream());
            _result = sr.ReadToEnd();
            _requestCompleted.Set();
        }
    }

What I quickly found was that the second my WaitOne() line ran was that the background methods would also hang. After a lot of mucking about and some research I found out one very important thing. The WebRequest and WebResponse methods call interacts with the browsers plugin API which runs on…. the UI thread. So by blocking the UI thread it also blocks any calls to the plugin API which blocks my WebRequest methods.

So to solve this I could probably place my sync wrapper onto a background thread to implement the back and forth conversations and then raise an event when it is done but after mucking around it just makes things too complex to read, follow, and maintain. It may still have its uses one day… but it was not as clean as I would like

Solution?
My main issue is when you have async methods that change so for WebRequest calls I decided to inline the callback methods it used. Again it is ugly but I find it simpler to follow a linear action:

public string GetCustomerDataAndBindItToTextBox(Uri requestUrl, string postData)
            {
                var request = WebRequest.Create(requestUrl);
                request.ContentType = “application/x-www-form-urlencoded”;
                request.Method = “POST”;
                request.BeginGetRequestStream(delegate(IAsyncResult result)
                                                   {
                                                       byte[] content = Encoding.UTF8.GetBytes(postData);
                                                       Stream stream = request.EndGetRequestStream(result);
                                                       stream.Write(content, 0, content.Length);
                                                       stream.Close();
                                                       request.BeginGetResponse(delegate (IAsyncResult result2)
                                                                                     {
                                                                                         WebResponse response = request.EndGetResponse(result2);
                                                                                         var sr = new StreamReader(response.GetResponseStream());
                                                                                         this.txtCustomer.Text = sr.ReadToEnd();
                                                                                     }, null);
                                                   }, null);

            }

Filed Under: General

MVVM and Designer Data

Aug 19

Over the last week I have been playing around with Silverlight and the Model View View Model (MVVM) pattern that I have heard such great things about. So far it is a really nice way to separate the UI logic from the UI and make it easily testable.

One of the things I loved in Silverlight was the support for design time data. Design time data allowed you to specify dummy data that would appear in VS/Blend so that you could work with the UI with sample data. This saves so much time as now you don’t have to open the app, (possibly login), navigate to the area you are working on, and then see if you have it the way you want.

The challenge was now to combine MVVM and this dummy design time data. It is a lot simpler than I thought. I simply created a dummy View Model that inherited from my runtime VM and loaded it with sample data.

Runtime ViewModel:
public class SummaryViewModel : BaseViewModel
    {
        public int MoviesOwned { get; set; }
        public int MoviesCurrentlyRentedOut { get; set; }
        public ObservableCollection<Category> Categories { get; set; }

        private readonly MovieServiceClient _service;
        public SummaryViewModel()
        {
            _service = new MovieServiceClient();
            _service.GetAllCategoriesCompleted += GetAllCategoriesCompleted;
            if (!System.ComponentModel.DesignerProperties.IsInDesignTool)            
                _service.GetAllCategoriesAsync();
        }

        private void GetAllCategoriesCompleted(object sender, GetAllCategoriesCompletedEventArgs e)
        {
            Categories = e.Result;
            InvokePropertyChanged(“Categories”);
        }
    }

Design Time View Model:
  public class DesignTimeSummaryViewModelViewModel : SummaryViewModel
    {
        public DesignTimeSummaryViewModelViewModel()
        {
            Categories = new ObservableCollection<Category>();
            Categories.Add(new Category() { Name = “Design Time Test” }); ;
            Categories.Add(new Category() { Name = “Design Time Test 2″ }); ;
            Categories.Add(new Category() { Name = “Design Time Test 3″ }); ;
            MoviesOwned = 10;
            MoviesCurrentlyRentedOut = 5;
        }
    }

Summary.xaml view (designer info in orange, runtime in blue):
<UserControl x:Class=”MovieRental.UI.MainPage”
    xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
    xmlns:d=”http://schemas.microsoft.com/expression/blend/2008″
    xmlns:mc=”http://schemas.openxmlformats.org/markup-compatibility/2006″
    xmlns:local=”clr-namespace:MovieRental.UI.ViewModels”
    mc:Ignorable=”d”
    d:DesignHeight=”300″ d:DesignWidth=”400″
    d:DataContext=”{d:DesignInstance Type=local:DesignTimeSummaryViewModelViewModel,IsDesignTimeCreatable=True}”

    >
    <UserControl.Resources>

        <local:SummaryViewModel x:Key=”vm” />

    </UserControl.Resources>

    <UserControl.DataContext>

        <Binding Source=”{StaticResource vm}” />

    </UserControl.DataContext>

    <Grid x:Name=”LayoutRoot” Background=”White”>
        <StackPanel>
            <StackPanel Orientation=”Horizontal”>
                <TextBlock Text=”Movies Owned” />
                <TextBlock Text=”{Binding MoviesOwned}” Padding=”7,0,0,0″/>
        </StackPanel>
        <StackPanel Orientation=”Horizontal”>
                <TextBlock Text=”Movies Rented”/>
                <TextBlock Text=”{Binding MoviesCurrentlyRentedOut}” Padding=”7,0,0,0″ />
        </StackPanel>
            <ListBox ItemsSource=”{Binding Categories}” DisplayMemberPath=”Name”>
            </ListBox>
        </StackPanel>
    </Grid>
</UserControl>

So far this is working pretty good. The thing that surprised me is that the designer still runs the runtime ViewModels constructor so I had to put a check in around my WCF call to make sure it did not run in the designer via the      System.ComponentModel.DesignerProperties.IsInDesignTool line of code.

Filed Under: General

A Word Macro To Simplify Resume Generation

Aug 8

I have taken the summer off to look after our kids, get caught up on my todo list (yes it is 2 months long), and do some learning. Now that I am mostly caught up I decided to update my resume but find it hard to remember how many years of experience I have in technologies on my skills matrix. I don’t like sending out a resume that shows what year I started with a technology as that does not accurately represent the number of years of experience. I figured that I could compamise on this and have a master document that has what year I started with a technology and have a vba macro that converts it into years of experience. Here is what I did

  1. Created a new word document with a table in it with 3 columns (column #3 contains the year started) e.g.
    Category Skill Experience
    Languages C# 2003
      VB.NET 2003
      VBA 2010
         
    Database MSSQL 2000
      MYSQL 1999
      Oracle 1 year
  2.  Open up the macro editor (Alt+F11) and added a new module to the project
  3. Put in this code:
    Sub UpdateYears()
        Application.ScreenUpdating = False
        currentYear = Year(Date)
        For Each tbl In ActiveDocument.Tables
            For Each rw In tbl.Rows
                Dim experience As Integer
                If IsNumeric(StripJunk(rw.Cells(3))) Then
                    experience = currentYear - StripJunk(rw.Cells(3))
                    If (experience <= 1) Then
                        rw.Cells(3) = experience & " year"
                    Else
                        rw.Cells(3) = experience & " years"
                    End If
                End If
            Next
        Next
    End Sub
    Private Function StripJunk(ByVal s As String)
      StripJunk = Trim(Replace(s, vbCr & Chr(7), ""))
    End Function
  4. Saved my word document as “Word macro-enabled document *.docm”
  5. Ran the macro (in office 2010->click “view” on the ribbon. Then click the macro drop down. Then click “View Macros”. Run the UpdateYears macro
  6. Save a copy of the document as a regular word doc without the macros and send it off.

Hopefully this saves you a few hours of VBA/Macro headaches!

Filed Under: General

Going Digital At Home – Part 4 – Digitization Of Media

Aug 3

The most painful and time consuming part of this project has been digitizing our DVD collection. I thought it would be as simple as downloading a program and rip it is done…. not so much in the end. Most mainstream companies are avoiding the area of DVD digitization due to potential issues of the legality of ripping DVDs. Another issue came down to encoding support in the software I tried (most were doing h.246) which would not play on some of my lower powered devices. The other issue is that there are SO many options and settings to tweak that it takes a long time to get them just right. I started ripping only one chapter of a movie and then watching that on different devices and then tweaking settings.

To start with there are many different ways to digitize your DVDs. The first thing I tried was using VLC media player which (unkown to me) has an option to save a DVD to disk. I found that most outputs it put out was distorted and had lots of green blocks all over. I also tried DVDFab which I could not get the quality I wanted and it was trial software. Handbrake is another popular software but it does not do decryption so you have to install other software (I triewantd AnyDVD which worked awesome) in combination with Handbrake. Handbrake only supported H.246 or MPEG4 and I found the MPEG4 quality was too low (and I could not seem to get it any better).

What I finally got to work is a two step process. The first step was to rip the contents of the DVD to a VOB file using DVDDecryptor which takes about 20 minutes. A VOB file is the raw video/audio/subtitiles from the disk which some programs can read in and convert. The next step was to feed the VOB into a program called MeGUI which is a UI wrapper around AviSynth that works quite well. I found it takes 2-3 hours to encode a video using MeGUI.

Now if you have a lot of movies then you need some automation to this process. What I did was gather up 5 spare DVD roms and combined them into a computer so I could backup 5 DVDs at a time. I then wrote a bit of code (I may publish it later) that starts DVDDecryptor on the command line, rips the disk, ejects the disk, then monitors for a new disk. This allowed me to just walk by and see the ejected trays and then insert new disks which worked well. My big issue was that the raw DVD is about 5 GB so you can fill up storage quick so I moved a lot of the raw files to other computers as the drive on my ripping computer filled up.

Once I had my movies ripped (or my hard drives filled) it was time to compress them down to save some space using MeGUI+Avisynth. When you install the software it asks if you want to download some presets which I did. In the end I used XVid-Balanced to do my encoding. Then I used the one click encoding tool in MeGUI and queued up all my VOB files. As this process takes 2-3 hours (depending on CPU power) I ran it on multiple computers. Many of my computers are multi-core and MeGUI has the option to spin up multiple worker threads but I found that it crashed so I just ran one worker (which still used multiple cores).

Filed Under: General