Archive for General

Dropping my MVP

Feb 2

I have decided to drop out of Microsoft’s Most Valuable Professional Program. I have decided this for many reasons. Mainly I feel the program has no value to me and that MVPs are of no value to MS.

Experts

MS states how they have these “experts” in the community and that are involved in the products. While I have met some really smart MVPs I have also met some that are so out to lunch on things yet MS still considers them experts. To me this completely devalues the MVP program. I am definitely no exception to this. I am a MVP in developer security yet I don’t consider myself to be an expert. Interested yes. Expert no.

The way you get in is by contributing to the community. I had done lots of this and it was really nice to get recognized for it. Once you are in you are supposed to be this expert that provides feedback to the products that MS is building. Now this is where the program breaks down.

Products

Firstly most products you never hear about until it is too late. I hear about more MS technology from the mainstream media than I do from the MVP program. As a MVP who is supposed to be like an insider I thought that they would be looking to us for feedback on these technologies. But they don’t. They will take it after the product is released though but by then there are usually too many issues with the technology to be overcome.

While some products are great there are a big few that still irritate me:

 TFS is garbage if you have used anything else. I should never have to struggle to get things under source control as much as I have with this product. I should never have to struggle for days/weeks to install it. I should never have so much trouble configuring it or applying checkin policies. If any MVP worth their salt was asked about this then MS should know this.

MSBuild is another failed technology to me. It does not come close to the power of Nant so if I am going to pay the XML tax I will pay it with Nant. I will say that it is nice that proj files are msbuild files yet whenever I have had to customize my builds with MSBuild I want to hurt myself. At first I thought it was the learning curve but to me the product just does not meet my needs. Again if an MVP that uses a build technology was asked then MS would have known about this.

MSTest also missed the mark to me when they said it was not geared towards people who do TDD. Really? No one else does TDD? I am the only MVP that does? I think not.

The big one that continues to bug me is Entity Framework. At the ‘08 MVP summit a lot of the MVPs that saw it before it was announced pointed out the serious issues it has. At the ‘09 summit we were excited to see the improvements they had made but instead we were shown the same crap that was the ‘08 entity framework. This spawned off the Entity Framework Vote Of Non Confidence and finally in ’11 we have some of the changes that we said were so needed in ’08. Good job valuing the input of the people that actually have to use the crap you put out.

MS “Listens”/”Cares”

To me it seems that MS having MVPs is a way for MS to feel that they are in touch with the community at large. This is great for a company to do but it seems like the feedback that is given falls on deaf ears for many products. I am not saying that this is for all products though as some do really care about feedback. For the most part though it feels like any feedback given just falls into a void. When I asked for some contacts for TFS/MSBuild/MSTest I was told to send my feedback to my MVP lead and they would forward it on…. Because that is open. You want MVPs to be involved yet filtered (I felt anyways). I have had more interaction with MS staff by tweeting things that I have through the MVP program.

The other big issue I have with MS in general is being treated like we will jump through hoops to help them succeed. There have been a number of “tech x is launching in a few weeks so build something” so we can showcase it emails I have received. Well that is great but how am I going to build a good app on a new technology with little documentation and no access to anyone at MS who can give me some accurate answers to issues? I understand that MS staff are not there to troubleshoot my application but on new/unreleased technology it would be good to have some sort of access if only an internal message board.

There are some internal lists at MS for MVPs as well. I found out about the security one after being a security MVP  for OVER A YEAR. Now that I am on it…. Not much happens. There is little in the way of solicitation for feedback and there is little feedback provided. I assume this is because the security space is pretty quiet but then I am surprised to see things at the summit that I had no idea were being developed.

The most surprising thing to me was when I told my MVP lead that I was not interested in participating in the program. The response was: “No problem.  Thanks for letting me know Dave.  Good luck in the future”. I expected maybe a bit more of a “Why?” response but I am not surprised that MS does not care about what other people think. Maybe, because I have become harder on MS that they are happy to be rid of me. Who knows.

Maybe I am being hard on the program. Our MVP lead is great and tells us that whatever we need he will make happen. Great but how about you let us know some of the things that you can do for us? If you want something like WP7 to succeed get more than 5 dev phones and let us know you have them if we want to build applications. If MS is thinking of a new technology solicit the MVPs for people that use or have an interest in the new tech and get their feedback at the START of development instead of at the end.

Rewards

The MVP award is given to those that contribute to the community. I enjoy doing this and I will continue to do this whenever I can. Unfortunately it costs a lot of time and money to contribute to the community. If I give a talk I have spent 40 to 200 hours preparing (yes 200 hours on a 1 hour talk) plus the costs to travel to another city, food, and accommodation.  MS’s reward is a MSDN subscription and half of a hotel room if I attend the summit (there used to be more benefits but they have been rolled back to save costs). It would probably be more affordable for me to purchase an MSDN subscription on my own and stop speaking entirely. I have gained no additional work from the MVP program, no additional speaking gigs, and have not made relatively few connections to others because of the MVP program.

Fin

The program has become a black mark to me. It does not contain just experts (heck, they let me in). It does not certify that anyone has a clue about anything (and some of the MVP only lists prove this). It has not opened any doors for me. It does not help me stop MS from releasing crap that I have to work with. All it does for me is give me an MSDN subscription and another line on my resume.

Filed Under: General, Rants

Security Trends

Dec 29

I have been around computers for most of my life and for the past ten years I have been involved in corporate IT on both the network side and the programming side. I was reflecting on some of the changes and trends over the past decade and thought I would share them.

  1. We have realized that network security is important. Every organization I visit seems to have a firewall, antivirus, and windows updates turned on. Ten years ago it was every computer was connected straight to the internet with an exposed c$ share it seems like. Finally most corporate (and many home users) seem to have taken some basic security measures.
  2. Users still click on anything that pops up on the screen until they get the application/website to work. I have tried to educate people on this but it has become apparent to many of us in the industry that users just want things to work and it only when they go wrong that they care. For instance, my parents install every plugin they are prompted to as they feel that if it was bad their antivirus would catch it. I don’t fault them for it, it is just human nature. We expect the locks on our house to keep people out when it is simple to smash a window (or cut a hole in the wall). We are coming around the corner on this one and seeing that we can’t educate everyone and things need to just be more secure by default and leak less information.
  3. People care a lot more about their privacy now. People also don’t realize that most places gathering information don’t care about you as a person, they care about you as a demographic. Granted it is not a happy thought to know how much info about you is floating around but if it personalizes my searches and shows me advertisements for things I am interested in so what? It is only an issue (to me) if that information is used for nefarious purposes.
  4. Many more technologies are shipping with a more “secure by default” mentality. The *nix/bsd attitude has been like this since I can remember and Microsoft has really turned a corner and secured their products. I remember when a new client would tell us they were running an IIS server and we would just laugh, wipe the box, and install FreeBSD/Apache for them. Nowdays I only laugh when people have an Adobe product installed (we all do… so ha ha). Adobe has realized that this is starting to hurt them and are making a big security push like MS did years ago.
  5. Schools have not taught about security and they still don’t. I see a bit of growth in this area but I feel this is incredibly lacking. Every time I give a presentation and ask how many people don’t know about SQL injection someone still raises their hands. It is not acceptable to have developers that don’t even know about the most basic/common security holes and how to prevent them. It is like hiring an electrician that does not know how to use a circuit breaker: they might get by for years without an incident but it is bound to happen eventually.
  6. Phishing/scams/chain letters has risen and fallen a lot. I find most people I talk to realize that people are trying to scam them (or will ask others to see if it is a scam) and that they know if they don’t forward this message to ten other people a kitten will die (except for my sister). 
  7. Piracy became common place. Back in the day it was technically difficult to find the software/music/movies and then crack it if necessary. Now my grandma can run a P2P client (or the vast array of other tools) to get her favorite songs. It is common place and it is sad. Not because I love big rich Hollywood types that complain about the money being stolen from them as they drive through Starbucks in a gold plated Bently. It is the fact that for such a huge market that they continually fail to make it simple to get the content we want quickly, conveniently, and be able to use it how we see fit. If I ran an ice cream store and everyone wanted paint flavored ice cream served off a piece of copper pipe you would see me at the hardware store every day. I think they are starting to get it and I hope to see some progress there so that everyone is happy (except me…. until I get a gold plated Starbucks).
  8. The goal of software development was and still is for the most part to deliver working software. Security takes a huge back seat and is almost never a first class citizen. I don’t see this trend changing much. The nice thing is that many languages are shipping in a more secure configuration which helps developers not leave as many gaping holes but I still feel that education is very important for developers.
Filed Under: General, Security

Fluent Build Beta

Dec 19

For those of you that follow me on twitter (@gotwoods is my handle) you may have seen that I have been working on a new open source project called FluentBuild which I am happy to announce is in Beta. This project has kept me quite busy lately but the core of the app is now done.

FluentBuild is a .NET fluent language around builds. It makes doing builds way easier than in Nant, MsBuild, and (in my mind) other build languages. With it you can now write a build file in C# (other language support on the way) within visual studio which allows you to get intellisense, refactoring support, and you are even able to debug your build files.

The project site is http://code.google.com/p/fluent-build/

I just finished putting together a demo video. I recorded my screen at high resolution so full screen viewing is recommended and the volume recorded a bit loud so turn down your speakers to start.

Filed Under: General

Integration – Is It Worth It?

Aug 28

One thing I get asked to do a lot is integrate systems with each other or incorporate other systems into one we are currently building. In a lot of scenarios I have been starting to give some pushback on this. Sure integration is cool and all but should it be done?

Expense
The first (and ongoing) issue is the expense of integrating systems. First you have to understand the systems (or at least the APIs being exposed) of each system which can take a lot of time. Secondly we have to write code to integrate the services and adapt concepts from one system to another. This work also needs to be tested and debugged.

Perpetuating The Legacy
The bigger issue I have seen in organizations is that tying a legacy system into your system makes it harder to change or replace the legacy system. If a legacy system has 3-4 systems that depend on it then any changes have to be tested and approved by the consuming systems (expense again).

If we want to replace the legacy system with something better then we have two options. We can either rewrite the system and keep the old API exposed or we can throw it away and force our consumers to change. Rewriting a system and keeping the API may work IF the original API was perfect (or close to perfect). In practice I find that the API is ok but if a person were to rewrite it they could alleviate a lot of the issues and shortcomings discovered over the years. If we do a brand new interface then all our consumers need to change their code which can be…. expensive.

From being in this situation before I have compromised. I created a new system with a better API that was more expressive and simpler to use. I then created a separate project that exposed the old API and translated that to the new API. This does add some performance overhead in translation and lots of things to test (expensive).  It does allow new consumers to use a better API and allowed consumers of the old system to decide if it was worth migrating to the new API.

Integration For Integrations Sake
Many people automatically assume that if the information is in another system that we should integrate the two systems. This does help centralize rules, reduce duplication, and centralize information. This is all true but not always necessary. I would find it acceptable to have some minor duplication if it eliminated a coupling on another system. This is a very careful call to make though as this duplication can lead to duplication of maintainance. Even worse it can lead to their now being two sources of data so when a third system comes along it now consumes both sources for data to try and determine which one is accurate.

One classic example I have is from a client of mine. They have a system that dispatches their vehicles and another system that uses GPS to track their vehicles. They asked for these systems to be integrated as they are always switching between the two. In this case the value of integrating the systems was very low. Instead I recommend that all their operators get two monitors and have both applications open. This is not the 100% perfect solution (unless you are an accountant) but it gets the job done for the right price.

Alternatives
There are ways to integrate but not integrate. One of the simpler ones is to have a button/link/menu item/whatever that fires up a system you would normally integrate with and pass along necessary data. I.e. a button that runs “otherProgram.exe /RunSalesReport 10-10-2009 10-10-2010″ or http://server/SalesReport.aspx?start=10-10-2009&end=10-10-2010. Granted there is a bit of coupling there and some things to maintain but not as much as if we had to call a service, translate the results, and show it on a UI.

Another method is to use manual entry. An example of this would be to use one system to find a product and then type that products manufacture code into your system you are building. Sure it is not as fast and the costs of this slowdown will add up over time but is the cost of integration still higher than the cost it takes for a user to copy/paste a code from one program to another? Not an easy question to answer but one that should be evaluated.

The last method is one typically avoided: double entry. This is where the same (or similar) data is entered into two separate systems. An example of this might be for a sales operation. It may be acceptable for the sales system to have a list of all products, descriptions, and prices of products that they sell. The warehouse system might have its one list of products, dimensions, and inventory levels. As long as when a sale comes in that the order can be filled AND that the sales system does not need to know inventory levels when they make a sale (i.e. if there is none in the warehouse it will just get backordered). This is a very tight line to walk as if an item were entered into one system incorrectly the customer may order 20 Widgets but instead get 20 Gadgets due to the product codes not lining up.

Final Thoughts
I have learned that integration is more expensive than I first thought it would be. I thought that reducing code I had to write would make life easier. It does in some ways but in other ways it adds overhead to development and maintenance. If we just stop and think if it is really necessary to tie two systems together then we might just have a little less coupling and a little more freedom to upgrade some legacy systems. Not integrating is not the only solution…. but neither is integrating.

Filed Under: General, Rants

Learn From My Mistakes?

Aug 21

One of my frustrations with Silverlight has been that the async model is rammed down your throat. I understand the reasoning that it is good to have the UI thread responsive while long running calls are happening. But to me that is something that I code and I control.

The thing that gets me is that there are times where several network operations may have to happen in sequence. Here is a simple contrived example:

var service = new VehicleRentalService();
if (service.AreVehiclesAvailableToRent())
  {
  txtNumberOfCars.Text = service.GetNumberOfCarsForRent();
  if (service.IsTheFlakyOnlineRentalServiceRunning())
      {
       btnBookOnline.Enabled = true;
      }
  }

Now for the above example I would run all this on 1 background thread (leaving the UI nice and responsive) and make sure that I properly delegate the control changes to the UI thread (as controls are not thread safe).

Silverlight forces us to async the calls which leads to a lot of functions that chain together. I find this makes the code a lot harder to read and a lot harder to understand. Here is a hand written rewrite of the above example:

public function Start()
{

var service = new VehicleRentalService();
service.AreVehiclesAvailableToRentCompleted+=VehiclesAvailableToRentResponseRecieved;
service.GetNumberOfCarsForRentCompleted+=NumberOfCarsForRentResponseRecieved;
service.IsTheFlakyOnlineRentalServiceRunningCompleted+=IsTheFlakyOnlineRentalServiceRunningResponseRecieved;

service.AreVehiclesAvailableToRent();

}

public function VehiclesAvailableToRentResponseRecieved(object sender, VehiclesAvailableToRentCompletedEventArgs e)
{
   if (e.Result)
      {
      service.GetNumberOfCarsForRent();
      service.IsTheFlakyOnlineRentalServiceRunning();
     }
}

public function NumberOfCarsForRentResponseRecieved(object sender, NumberOfCarsForRentCompletedEventArgs e)
{
    Dispatcher.BeginInvoke(delegate { txtNumberOfCars.Text = e.Result });
}

public function IsTheFlakyOnlineRentalServiceRunningResponseRecieved(object sender, IsTheFlakyOnlineRentalServiceRunningResponseRecievedCompletedEventArgs e)
{
    Dispatcher.BeginInvoke(delegate { btnBookOnline.Enabled = e.Result });
}

(Some may argue that there should be a GetSystemStatus method on the service that returns all this info to which I would probably agree. For the sake of this post though I am going to run with it).

So the Silverlight code is longer, harder to read, and harder to maintain. It does have the benefit of getting the number of cars for rent and checking if the online booking system is running at the same time which is a nice plus.

For an example a log more complicated than this one I was using a WebRequest to converse with a server in a very back and forth fashion and found that the code quickly became very hard to read so I set about to change things. My idea was to create a class that uses async calls in the background but appear to be synchronous by blocking the caller. i.e.

string customerData = Request.GetResponse(“http://www.dummy.com/getcustomer.aspx”, “CustomerId=1″);
string customerAddress = Request.GetResponse(“http://www.dummy.com/getaddress.aspx”, ParseOutAddressId(customerData));

PLEASE DO NOT USE THIS CODE AS IT DOES NOT WORK (hence the post title):

    public class SyncRequestResponse
    {
        ManualResetEvent _requestCompleted;
        private WebRequest _request;
        private string _postData;
        private Uri _requestUrl;
        private static string _result;

        public string GetResponse(Uri requestUrl, string postData)
        {
            _requestCompleted=new ManualResetEvent(false);
            _requestCompleted.Reset();
            _postData = postData;
            _requestUrl = requestUrl;
            StartRequest();
            _requestCompleted.WaitOne(1000); //block current thread until the response is received or the timeout is reached
            return _result; //the result should be filled now
        }

        private void StartRequest()
        {
            _request = WebRequest.Create(_requestUrl);
            _request.ContentType = “application/x-www-form-urlencoded”;
            _request.Method = “POST”;
            _request.BeginGetRequestStream(GetRequestStreamCompleted, null);
        }

        private void GetRequestStreamCompleted(IAsyncResult result)
        {
                //request stream has returned so fill it
                var content = Encoding.UTF8.GetBytes(_postData);
                var stream = _request.EndGetRequestStream(result );
                stream.Write(content, 0, content.Length);
                stream.Close();
                _request.BeginGetResponse(GetResponseFromRequest, null);
        }

        private void GetResponseFromRequest(IAsyncResult ar)
        {
            var response = _request.EndGetResponse(ar);
            var sr = new StreamReader(response.GetResponseStream());
            _result = sr.ReadToEnd();
            _requestCompleted.Set();
        }
    }

What I quickly found was that the second my WaitOne() line ran was that the background methods would also hang. After a lot of mucking about and some research I found out one very important thing. The WebRequest and WebResponse methods call interacts with the browsers plugin API which runs on…. the UI thread. So by blocking the UI thread it also blocks any calls to the plugin API which blocks my WebRequest methods.

So to solve this I could probably place my sync wrapper onto a background thread to implement the back and forth conversations and then raise an event when it is done but after mucking around it just makes things too complex to read, follow, and maintain. It may still have its uses one day… but it was not as clean as I would like

Solution?
My main issue is when you have async methods that change so for WebRequest calls I decided to inline the callback methods it used. Again it is ugly but I find it simpler to follow a linear action:

public string GetCustomerDataAndBindItToTextBox(Uri requestUrl, string postData)
            {
                var request = WebRequest.Create(requestUrl);
                request.ContentType = “application/x-www-form-urlencoded”;
                request.Method = “POST”;
                request.BeginGetRequestStream(delegate(IAsyncResult result)
                                                   {
                                                       byte[] content = Encoding.UTF8.GetBytes(postData);
                                                       Stream stream = request.EndGetRequestStream(result);
                                                       stream.Write(content, 0, content.Length);
                                                       stream.Close();
                                                       request.BeginGetResponse(delegate (IAsyncResult result2)
                                                                                     {
                                                                                         WebResponse response = request.EndGetResponse(result2);
                                                                                         var sr = new StreamReader(response.GetResponseStream());
                                                                                         this.txtCustomer.Text = sr.ReadToEnd();
                                                                                     }, null);
                                                   }, null);

            }

Filed Under: General

MVVM and Designer Data

Aug 19

Over the last week I have been playing around with Silverlight and the Model View View Model (MVVM) pattern that I have heard such great things about. So far it is a really nice way to separate the UI logic from the UI and make it easily testable.

One of the things I loved in Silverlight was the support for design time data. Design time data allowed you to specify dummy data that would appear in VS/Blend so that you could work with the UI with sample data. This saves so much time as now you don’t have to open the app, (possibly login), navigate to the area you are working on, and then see if you have it the way you want.

The challenge was now to combine MVVM and this dummy design time data. It is a lot simpler than I thought. I simply created a dummy View Model that inherited from my runtime VM and loaded it with sample data.

Runtime ViewModel:
public class SummaryViewModel : BaseViewModel
    {
        public int MoviesOwned { get; set; }
        public int MoviesCurrentlyRentedOut { get; set; }
        public ObservableCollection<Category> Categories { get; set; }

        private readonly MovieServiceClient _service;
        public SummaryViewModel()
        {
            _service = new MovieServiceClient();
            _service.GetAllCategoriesCompleted += GetAllCategoriesCompleted;
            if (!System.ComponentModel.DesignerProperties.IsInDesignTool)            
                _service.GetAllCategoriesAsync();
        }

        private void GetAllCategoriesCompleted(object sender, GetAllCategoriesCompletedEventArgs e)
        {
            Categories = e.Result;
            InvokePropertyChanged(“Categories”);
        }
    }

Design Time View Model:
  public class DesignTimeSummaryViewModelViewModel : SummaryViewModel
    {
        public DesignTimeSummaryViewModelViewModel()
        {
            Categories = new ObservableCollection<Category>();
            Categories.Add(new Category() { Name = “Design Time Test” }); ;
            Categories.Add(new Category() { Name = “Design Time Test 2″ }); ;
            Categories.Add(new Category() { Name = “Design Time Test 3″ }); ;
            MoviesOwned = 10;
            MoviesCurrentlyRentedOut = 5;
        }
    }

Summary.xaml view (designer info in orange, runtime in blue):
<UserControl x:Class=”MovieRental.UI.MainPage”
    xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
    xmlns:d=”http://schemas.microsoft.com/expression/blend/2008″
    xmlns:mc=”http://schemas.openxmlformats.org/markup-compatibility/2006″
    xmlns:local=”clr-namespace:MovieRental.UI.ViewModels”
    mc:Ignorable=”d”
    d:DesignHeight=”300″ d:DesignWidth=”400″
    d:DataContext=”{d:DesignInstance Type=local:DesignTimeSummaryViewModelViewModel,IsDesignTimeCreatable=True}”

    >
    <UserControl.Resources>

        <local:SummaryViewModel x:Key=”vm” />

    </UserControl.Resources>

    <UserControl.DataContext>

        <Binding Source=”{StaticResource vm}” />

    </UserControl.DataContext>

    <Grid x:Name=”LayoutRoot” Background=”White”>
        <StackPanel>
            <StackPanel Orientation=”Horizontal”>
                <TextBlock Text=”Movies Owned” />
                <TextBlock Text=”{Binding MoviesOwned}” Padding=”7,0,0,0″/>
        </StackPanel>
        <StackPanel Orientation=”Horizontal”>
                <TextBlock Text=”Movies Rented”/>
                <TextBlock Text=”{Binding MoviesCurrentlyRentedOut}” Padding=”7,0,0,0″ />
        </StackPanel>
            <ListBox ItemsSource=”{Binding Categories}” DisplayMemberPath=”Name”>
            </ListBox>
        </StackPanel>
    </Grid>
</UserControl>

So far this is working pretty good. The thing that surprised me is that the designer still runs the runtime ViewModels constructor so I had to put a check in around my WCF call to make sure it did not run in the designer via the      System.ComponentModel.DesignerProperties.IsInDesignTool line of code.

Filed Under: General

A Word Macro To Simplify Resume Generation

Aug 8

I have taken the summer off to look after our kids, get caught up on my todo list (yes it is 2 months long), and do some learning. Now that I am mostly caught up I decided to update my resume but find it hard to remember how many years of experience I have in technologies on my skills matrix. I don’t like sending out a resume that shows what year I started with a technology as that does not accurately represent the number of years of experience. I figured that I could compamise on this and have a master document that has what year I started with a technology and have a vba macro that converts it into years of experience. Here is what I did

  1. Created a new word document with a table in it with 3 columns (column #3 contains the year started) e.g.
    Category Skill Experience
    Languages C# 2003
      VB.NET 2003
      VBA 2010
         
    Database MSSQL 2000
      MYSQL 1999
      Oracle 1 year
  2.  Open up the macro editor (Alt+F11) and added a new module to the project
  3. Put in this code:
    Sub UpdateYears()
        Application.ScreenUpdating = False
        currentYear = Year(Date)
        For Each tbl In ActiveDocument.Tables
            For Each rw In tbl.Rows
                Dim experience As Integer
                If IsNumeric(StripJunk(rw.Cells(3))) Then
                    experience = currentYear - StripJunk(rw.Cells(3))
                    If (experience <= 1) Then
                        rw.Cells(3) = experience & " year"
                    Else
                        rw.Cells(3) = experience & " years"
                    End If
                End If
            Next
        Next
    End Sub
    Private Function StripJunk(ByVal s As String)
      StripJunk = Trim(Replace(s, vbCr & Chr(7), ""))
    End Function
  4. Saved my word document as “Word macro-enabled document *.docm”
  5. Ran the macro (in office 2010->click “view” on the ribbon. Then click the macro drop down. Then click “View Macros”. Run the UpdateYears macro
  6. Save a copy of the document as a regular word doc without the macros and send it off.

Hopefully this saves you a few hours of VBA/Macro headaches!

Filed Under: General

Going Digital At Home – Part 4 – Digitization Of Media

Aug 3

The most painful and time consuming part of this project has been digitizing our DVD collection. I thought it would be as simple as downloading a program and rip it is done…. not so much in the end. Most mainstream companies are avoiding the area of DVD digitization due to potential issues of the legality of ripping DVDs. Another issue came down to encoding support in the software I tried (most were doing h.246) which would not play on some of my lower powered devices. The other issue is that there are SO many options and settings to tweak that it takes a long time to get them just right. I started ripping only one chapter of a movie and then watching that on different devices and then tweaking settings.

To start with there are many different ways to digitize your DVDs. The first thing I tried was using VLC media player which (unkown to me) has an option to save a DVD to disk. I found that most outputs it put out was distorted and had lots of green blocks all over. I also tried DVDFab which I could not get the quality I wanted and it was trial software. Handbrake is another popular software but it does not do decryption so you have to install other software (I triewantd AnyDVD which worked awesome) in combination with Handbrake. Handbrake only supported H.246 or MPEG4 and I found the MPEG4 quality was too low (and I could not seem to get it any better).

What I finally got to work is a two step process. The first step was to rip the contents of the DVD to a VOB file using DVDDecryptor which takes about 20 minutes. A VOB file is the raw video/audio/subtitiles from the disk which some programs can read in and convert. The next step was to feed the VOB into a program called MeGUI which is a UI wrapper around AviSynth that works quite well. I found it takes 2-3 hours to encode a video using MeGUI.

Now if you have a lot of movies then you need some automation to this process. What I did was gather up 5 spare DVD roms and combined them into a computer so I could backup 5 DVDs at a time. I then wrote a bit of code (I may publish it later) that starts DVDDecryptor on the command line, rips the disk, ejects the disk, then monitors for a new disk. This allowed me to just walk by and see the ejected trays and then insert new disks which worked well. My big issue was that the raw DVD is about 5 GB so you can fill up storage quick so I moved a lot of the raw files to other computers as the drive on my ripping computer filled up.

Once I had my movies ripped (or my hard drives filled) it was time to compress them down to save some space using MeGUI+Avisynth. When you install the software it asks if you want to download some presets which I did. In the end I used XVid-Balanced to do my encoding. Then I used the one click encoding tool in MeGUI and queued up all my VOB files. As this process takes 2-3 hours (depending on CPU power) I ran it on multiple computers. Many of my computers are multi-core and MeGUI has the option to spin up multiple worker threads but I found that it crashed so I just ran one worker (which still used multiple cores).

Filed Under: General

Going Digital At Home – Part 3 – The Law

Jul 6

One of the things I never took much time to consider was the legality off all this. My opinion has been (and hopefully always will be) that if I purchased something I should be able to do whatever I want with it. If I own DVD and want to use it as a decoration on my Christmas tree I feel I have that right.

Again, I am not a lawyer and this is just a summation of my research on the topic so do your own research for your own area.

DVD Backups
The first debate is if you can backup or convert your DVD’s to another format. There are two big points in this debate. The first is if you can make a backup of the media. The second is if you can legally circumvent copy protection. In Canada, the current copyright act does not explicitly state if it is legal or not to do either of these. It is legal to make a backup of computer programs, and you are allowed to copy audio/video for personal use. In the current situation it seems quite legal to backup and defeat copy protection. I think this is great as I PURCHASED a product and would like to use it in the manner I see fit. Of course it is illegal for me to share the backup.

From my understanding, in the United States, it is only partially legal to create a backup (yup partially legal). It seems that making a backup copy is totally fine but if you break copy protection it violates the Digital Millenium Copyright Act as it may be considered that you have circumvented copy protection on the DVD. Circumvention though could be in the eye of the beholder as computer software has to open the DVD and decrypt it for playback, capturing the legally decrypted data and placing it on your hard drive may be totally legal.

Canada is working to reform its copyright act so there may be some big changes to modernize it. From reading over Bill C-32, I see this:

29.22 (1) It is not an infringement of copyright for an individual to reproduce a work or other subject-matter or any substantial part of a work or other subject-matter if
(a) the copy of the work or other subject-matter from which the reproduction is made is not an infringing copy;
(b) the individual legally obtained the copy of the work or other subject-matter from which the reproduction is made, other than by borrowing it or renting it, and owns or is authorized to use the medium or device on which it is reproduced;
(c) the individual, in order to make the reproduction, did not circumvent, as defined in section 41, a technological protection measure, as defined in that section, or cause one to be circumvented;
(d) the individual does not give the reproduction away; and
(e) the reproduction is used only for private purposes.

The issue here becomes circumventing technological protection measures. This is where I don’t agree as I purchased it so why should I not be able to do what I want with it in my own home? Frankly, if I can’t do what I want with content I buy then I will cease buying it.

DOWNLOADING MEDIA
In Canada, this is technically legal as long as you don’t distribute it or profit from it. After playing around with converting my own DVDs (that post is coming up), it may be easier to let someone else do the hard work of properly converting a movie and download it from them.

Modifying Your XBOX
I have an old xbox just collecting dust so I installed Xbox Media Center on it. This is not an MS product, it is a replacement OS created by the community (that is awesome) but does require a lot of work to get it installed (there are lots of tutorials out there on the internet about it though). Is it legal to modify your xbox? From what I see, yes. If you mod your xbox it voids your warranty and violates Microsoft’s use policy so it will probably not work online. For me that is fine as I am just using it as a media center hooked up to my TV.

Streaming Media Over Wireless
An interesting potential gotcha is that streaming media over a wireless network may be considered broadcasting. Some media center devices will not stream some content for this very reason (buyer beware for sure). I really think this is a non-issue as most people secure their wireless networks and most neighbors don’t have the technical expertise to watch what you are streaming but it could still be an issue.

Filed Under: General, Internet Law

Going Digital At Home – Part 2 – Sharing

Jun 25

The next step in providing digital content is being able to share your existing media out to the home. There are several ways to do this.

The first thing I want to talk about though is the codec issue. Video and audio can get quite large and there are many different ways to encode media to shrink its size down. In order to play a video you need the proper codec installed (a codec is a bit of code that properly decodes encoded video). The #1 issue with video playback has typically been that you don’t have the right codec installed. Normally when you are missing a codec you go out and download the codec and install it on your computer. When you have a device connected to a T.V. though it may not be possible to add the codec you want (I ran into this with my XBOX 360 for instance).

Simple File Share
Most computers and devices can access file shares quite easily. On your media server just right click and goto sharing and then select the users. There are many tutorials out there on how to share a folder so I wont dig into the basics but there are a few key tricks to it.

Most shares require the user to authenticate upon attempting to access a shared folder so that it can grant or deny access to that share. Now unless you have setup a domain controller for your house, it will not know who you are and want a username and password that has been setup on the computer sharing files. A neat trick is that if you have the same username/password setup on both the client and the server then you will automatically be authenticated. Otherwise it will prompt you for credentials. If you have children then it is adviseable to have a folder for family movies that allows the kids credentials and then a higher rated folder that requires an adults credentials.

The other debate is what type of access the share should have. I typically have shared folders that are read only and one folder that allows write so that new content can be dropped in from any computer and then sorted into the proper readonly folder.

The cons with this is that you need credentials, not all devices work over a file share, not all devices can play all types of media (due to the codec issue). For instance my xbox 360 does not access fileshares so this technique does not work)

Windows Media Sharing
I believe that starting with Windows Media Player 10 is an included service that allows you to share media out that is added to your Media Player library. I think that this solution works pretty good for the scenario where a user has one computer running a client operating system (i.e. XP/Vista/Windows 7) with their media and share it out. I have not tried it as I have a hatred for the bloatware that is Windows Media Player but it may just work for you.

From my quick read it does appear to overcome the codec issue via transcoding. This means that the media player on your computer converts the video on the fly to a format the remote device knows how to play. Again I have not tried this but I leave it up to you to experiment.

Windows Media Center & Media Center Extenders
Media Center is a chunk of software that ships with some editions of Vista and Win7. It allows you to access all your media, photos, and more via a client. The XBOX 360 ships with just such a client so I tried it out. It was horrible. It could not play a lot of the videos I had (due to codec issues), it was slow to navigate, and I found the video was choppy a lot of the time. Granted, I tried this a few years ago and they have probably made improvements but I have moved on to other technologies. I also don’t believe that you can have Media Center on a computer running a server OS (but again, I could be wrong)

Transcoding Server
I have found the best thing for me has been using a transcoding server called TVersity. The app converts video on the fly and streams it out to my devices just fine. The really nice feature is that you can also add internet feeds to the TVersity library. What this means is that I can take a show that published a feed and have my computer download that data, transcode it, and stream it to my xbox 360. I can now watch internet TV… on my TV! There is a lot of neat stuff you can do in this area.

The biggest issue I have found with TVersity is that the service will not respond sometimes and requires me to restart the service. Friends of mine have had no problems but I have had the issue on two separate installs now.

Network Considerations
For all of these solutions you will need to have the appropriate exceptions in your firewall to allow the services that share media to be accessible. Bandwidth can also be an issue with wireless networks. I currently have my server connected  to the router via regular cat 5 cable but the devices I consume media on are wireless. This works fairly well but media sharing is a bandwidth intensive operation so the quality of wireless signal you get may have an effect on streaming. If you can use a hardwire connection then I would recommend it as it is much more reliable than wireless.

Filed Under: General