David Woods

Source Of Truth

david.woods — Thu, 13 Sep 2012 19:11:35 +0000

Sometimes, you get spoiled by how things are done in organizations. I am having one of those moments today so I figured I should write about it. A good development environment needs a source of truth and a clear separation of concerns.

1) Source control is your source of truth. Anything not in source control did not happen. If you make a live edit to a file on the production server it is gone with the next update. This should never be allowed to happen. This is why I do a lot of deployments out of my continuous integration server. The only way to deploy is for the server to get the latest version from source control and install it on the server. If it was not important to put into source control it was not important.

2) Be careful managing your database(s). I usually recommend each developer runs a local database. This way changes to the dev database don’t break other peoples code until you check your source code in. Having a system where change scripts are checked in an run can be troublesome to implement but it is worth it. The benefit being that when you migrate to your different environments you have a suite of scripts already written. Ensure these are in source control

3) Verification is a time consuming process. Having to diff filesystems and databases to find the truth is a time consuming process that is fraught with errors. One project I am inherited has source in four (or more) locations so establishing which is the most current is a frightening process.

4) Establish the different environments very clearly and limit access where appropriate. If you have a mess of servers with some running production and test and development you are going to hit points of confusion or have inadvertent updates to applications. It is simple enough to accidentally update the production database by accident if they are on the same server as your development database. I have always felt that developers should have almost zero access to the production system where possible. Yes, it makes it easier to debug but there are ways to allow developers to debug without making changes (e.g. access to the event log, read only access to the database / filesystem, etc.). I avoid even that due to security implications but if you need to then there are better options than admin rights to the production server.

Security Is Security

david.woods — Fri, 24 Aug 2012 14:31:45 +0000

Something the past few days that has been irking me and seems to be a growing trend in the security space are comments like this:

“This attack is irrelevant because doing Y is cheaper/easier”

You are right. If I were an attacker I would try the simplest method first. If that fails do I go home and watch My Little Pony reruns because I could not telnet to your server with a blank password? No, I try increasingly more complex attacks until I get in.

Security is Security. Just because someone can set the data center on fire does not mean you should not secure the machines within it.

/rant

Local Build Using/Incrementing Team City Version Number

david.woods — Mon, 18 Jun 2012 16:21:12 +0000

On my current project we have a problem in that one of the assemblies we use requires an expensive license that is per computer so our build server would need it’s own license which we are not doing at this point. Now I need to run our publish deploy script locally which means manually increasing the version number on each build. Something that has been missed before causing a new version to overwrite an old version… which is bad… very bad.

To combat this I started digging and found a way to have my build script use and change the TeamCity build counter remotely using their REST API.

First open up http://[Server]/httpAuth/app/rest/projects/ and find your project you want to work with.

Then open up http://[Server]/httpAuth/app/rest/projects/id:project2 and find the configuration you want to work with.

Then open up http://[Server]/httpAuth/app/rest/buildTypes/bt5/settings/buildNumberCounter to get the current build counter

Here is the C# code I use

public class TeamCityVersionIncrementer

{

private readonly string _buildCounterUri;

private readonly NetworkCredential _credentials;

public TeamCityVersionIncrementer()

{

_buildCounterUri = “http://[server]/httpAuth/app/rest/buildTypes/bt5/settings/buildNumberCounter”;

_credentials = new NetworkCredential(“[user]“, “[pass]“);

}

public int Increment()

{

int originalCounter = GetBuildCounterFromTeamCity();

UpdateCounter(originalCounter + 1);

int counterAfterIncrement = GetBuildCounterFromTeamCity();

if (counterAfterIncrement != originalCounter + 1)

{

throw new ApplicationException(“Expected build to be incremented to ” + (originalCounter + 1) + “but was ” + counterAfterIncrement);

}

return counterAfterIncrement;

}

private int GetBuildCounterFromTeamCity()

{

WebRequest webrequest = WebRequest.Create(_buildCounterUri);

webrequest.Credentials = _credentials;

webrequest.Method = “GET”;

WebResponse webResponse = webrequest.GetResponse();

using (var reader = new StreamReader(webResponse.GetResponseStream()))

{

string readToEnd = reader.ReadToEnd();

return int.Parse(readToEnd);

}

private void UpdateCounter(int number)

{

byte[] arr = Encoding.UTF8.GetBytes(number.ToString());

var request = (HttpWebRequest)WebRequest.Create(_buildCounterUri);

request.Method = “PUT”;

request.ContentType = “text/xml”;

request.ContentLength = arr.Length;

request.Credentials = _credentials;

Stream dataStream = request.GetRequestStream();

dataStream.Write(arr, 0, arr.Length);

dataStream.Close();

var response = (HttpWebResponse)request.GetResponse();

if (response.StatusCode != HttpStatusCode.NoContent)

throw new ApplicationException(“Unexepected status code: ” + response.StatusCode);

}

Hope that helps someone in this rare situation!

LinkedIn Hashes Leaked

david.woods — Wed, 06 Jun 2012 16:48:47 +0000

Today it is being reported that LinkedIn has 6.5 million hashes leaked. It is good to see that the passwords were hashed. Unfortunately, it was done in a poor fashion.

The main issue is that salts were not used. Salts add random data per password and then are hashed. This makes things drastically harder to crack. Why? Well in attacking a database of hashes (without salts) the procedure is:

Generate a password
Hash the password
Compare the hash to all items in the database

This means that we generate one hash and check it against every account. If we add salts to the mix we have to work on a password at a time

Load hash and salt from database
Generate a password
Hash the password + salt
Compare the generated hash to the one record

For this scenario we have to crack a user at a time which is a huge time increase. Also, we don’t know how the salt was used internally. It could be:

password + salt
salt + password
salt + password + salt
FirstHalfOfSalt + password + LastHalfOfSalt
etc.

Lack of salting is a big failure here and I don’t see why it was not done. The other issue is the use of SHA-1 as the hashing algorithm which is no longer recommended. SHA-1 is fairly strong but has some collision weaknesses (two words generate the same hash which reduces the amount of time to generate all hashes). It is recommended to use SHA-256 or higher to avoid this issue.

While this may have been an oversight of the developers involved it is more likely a case of hash/crypto phobia. It is scary dealing with hashes and crypto as if you screw up data could be irreversibly lost. As developers we always want recoverability and not to mess with things that are working. Unfortunately for this sort of data we have to face the problem.

Now the big issue people face is migrating hashes from one scheme to another. Most leave it as is because hashes are one way so we can’t reverse the hash to upgrade it to a new format. It is quite easy to do though in a way that allows a seamless migration:

Add in new fields to store the new format into your database
When a user successfully logs in, take the password they typed in (which would be the plain text version) and run it through the new algorithm.
Store the output into the new database fields
Remove the original hash (the one we are replacing) from the database

As users login they get upgraded to the new scheme. It takes very little logic and time to implement this. The more users that login the stronger the database store gets.

Syncing Facebook/Meetup/Google Calendar On My phone

david.woods — Tue, 05 Jun 2012 19:11:29 +0000

One issue I have been running into is not adding my facebook/meetup events onto my phone. I then miss these events or forget until the last minute. I thought there must be a way to automate this and after a bit of tinkering I got it to work.

I am on an iPhone that uses Google Calendar as it’s calendar provider. I assume that the steps will be the same for any device that supports Google Calendar:

Facebook

Click events on the left
click export events
copy the url
login to google calendar
under other calendars click “add by url”
paste in the url for your fb calendar

Meetup

On the main page click the “add to” right above the calendar
Click the Google Calendar icon
It will open up the google calendar website and prompt you to add it

Once this is done the calendars did not appear on my phone. Some people posted to add them by visiting: https://www.google.com/calendar/iphoneselect which did nothing for me.

After a bit more reading I found that if I opened http://m.google.com/sync on my phone I could check off the calendars I wanted to snyc and viola there they were.

Announcing FluentBuild.UI

david.woods — Wed, 23 May 2012 16:01:55 +0000

One of my pet peeves has always been how hard it is to read the output of a build in a dos window. It is hard to navigate and with lots of output it is easy to loose content. To that end I decided to create a windows UI for FluentBuild that I just committed to source.

It still needs some work but it is there for any early adopters (click for a bigger screenshot):

FluentBuild – 1.1 Beta

david.woods — Sun, 20 May 2012 00:47:05 +0000

I have published the FB 1.1 Beta (http://code.google.com/p/fluent-build/downloads/list).

This release contains some big changes.

Task Syntax

Gone are things like Run.[option].[option].Execute() and they have been replaced with Task.Run.Nunit(options=>options.specific.to.runner). This removes the need to remember to use .Execute() all the time (I was constantly forgetting and I wrote the thing). I am fairly happy with it so far.

Exampes:

Task.Build.Csc.Target.Library(compiler => compiler.AddSources(sourceFiles)
.AddRefences(thirdparty_rhino, thirdparty_nunit)
.OutputFileTo(assembly_FluentBuild_Tests));

Task.Run.UnitTestFramework.Nunit(nUnitRunner=>nUnitRunner.FileToTest(assembly_FluentBuild));

Task.Run.Zip(x=>x.Compress

.SourceFolder(directory_compile)
.UsingCompressionLevel.Nine
.To(directory_release.File(“release.zip”)));

FTP Publishing
As I mentioned in a previous post I added some quick and dirty FTP publishing. It only publishes a file at a time currently so its not a full fledged FTP publishing tool.

Errors
For a while I only had the error message publishing but I have switched that to include the full exception as it was problematic to find errors before. There have also been some internal changes to how return codes are handled from programs. If a third party program throws an error then FB will return an error code of 1, not the error code of the third part application.

Logger
The message logger code has been simplified a lot and will probably get another round of cleanup soon. It has been moved to Defaults.Logger for now but will probably get moved right into the BuildFile class you inherit from as that makes more sense to me from a user point of view.

Whats Next?
I am getting fairly happy with the application right now but I would like to do more testing before I release it. I will be tweaking a bit of code and adding some more functional tests to the build but I hope not to have any big changes for the next while.

Rudimentary FTP Support

david.woods — Wed, 16 May 2012 04:00:47 +0000

While it was planned as a feature in the future, necessity bumped it up for me on my current project so I added some rudimentary FTP support.

Task.Publish.Ftp(x => x.Server(“ftp.server.com”)
.UserName(“username”)
.Password(“password”)
.LocalFilePath(@”c:\temp\test.txt”)
.Timeout(new TimeSpan(0,0,15,0))
.RemoteFilePath(“/html/”)
);

It only supports one file at a time and re-authenticates per request for now so uploading a folder structure is not recommended at this point. A more full fledged solution will be coming.

New Run Syntax

david.woods — Tue, 17 Apr 2012 22:04:16 +0000

I came up with a new run syntax that I like:

Task.Run.Zip(x=>x.Compress.SourceFile(“test.dll”).OutputTo(“test.zip”));
Task.Run.Executable(x=>x.PathToExecutable(“test.exe”));

Due to the scope of the change (and the fact that I have not switched to git yet), I have committed this code but there are still a lot more changes to be polished (and some namespace moves still to be done). I hope to have some time for further refinement and a ton of cleanup after this change in the coming week.

New Run Syntax Feedback

david.woods — Thu, 12 Apr 2012 13:58:02 +0000

I have completed the new build sytax of Task.Build(Using.[Compiler].[Options]); which I like way better than the old syntax. It is a bit annoying to have a few extra brackets but already makes it easier to use to me. I liked it so much that I started applying this to the Run syntax but that is where I found things did not work as well. I figured I would throw out the syntax ideas I had and see what people thought:

Option One: Generic run method
+simple to use
+expandable
-need to know what can be run (might be in one namespace)
-how do you handle executable return codes? (or other result objects)

Task.Run(Zip.Compress.SourceFile(“tests.dll”).UsingCompressionLevel.Ten);
Task.Run(Zip.Decompress.SourceFile(“tests.zip”));
Task.Run(Executable.Path(“test.exe”));
Task.Run(Debugger);
Task.Run(Nunit.Assembly(“tests.dll”));
Task.Run(MsTest.Assembly(“tests.dll”));

Option Two: Run narrows down choices then builders are used as arguments
+simple to use
+lets you know options available
+Allows for return codes for items that require it (or other result objects)
-much more language required to learn

Task.Run.Zip(Compress.SourceFile(“tests.dll”).UsingCompressionLevel.Ten);
Task.Run.Zip(Decompress.SourceFile(“tests.zip”));
Task.Run.Executable(Executable.Path(“test.exe”));
Task.Run.Debugger();
Task.Run.UnitTestFramework(Nunit.Assembly(“tests.dll”));
Task.Run.UnitTestFramework(MsTest.Assembly(“tests.dll”));
Option Three: Run narrows down choices to the most narrow point then builders are used as arguments
+simple to use
+lets you know options available
+Allows for return codes for items that require it (or other result objects)
+Intellisense would only give you one option for the arg to pass in
-Bit more verbose with the “Options” builders

Task.Run.Zip.Compress(CompressOptions.SourceFile(“tests.dll”).UsingCompressionLevel.Ten);
Task.Run.Zip.Decompress(DecompressOptions.SourceFile(“tests.zip”));
Task.Run.Executable(Executable.Path(“test.exe”));
Task.Run.Debugger();
Task.Run.UnitTestFramework.Nunit(NunitOptions.Assembly(“tests.dll”));
Task.Run.UnitTestFramework.MsTest(MsTestOptions.Assembly(“tests.dll”));
Option Four: Have a runners factory class that creates the builders
-need to know to do Runners.
+always know where to start (once you know to use Runners.
+simple syntax
-How do you handle executable return codes? (or other result objects)

Task.Run(Runners.Zip.Compress.SourceFile(“tests.dll”).UsingCompressionLevel.Ten);
Task.Run(Runners.Zip.Decompress.SourceFile(“tests.zip”));
Task.Run(Runners.Executable.Path(“test.exe”));
Task.Run(Runners.Debugger);
Task.Run(Runners.Nunit.Assembly(“tests.dll”));
Task.Run(Runners.MsTest.Assembly(“tests.dll”));

Let me know your thoughts or give me other options to make this simple and intuitive!