Sharp Learning Curve Ahead

Not a day after I posted Using Reactive Extensions To Throttle Asynchronous Tasks, Josh Bush was already (kindly) saying “I think your code may have a problem”. The issue with the first example is two-fold: one it doesn’t really work as posted and two, if it did it would behave in a less than ideal way. Basically, it calls wait after each item. Not exactly what I was going for.

It Would Be Cool If…

You have some known/unknown quantity of total work (Y), and you want to limit the number of worker threads in process at any given time (X). What my first try was actually doing was making X asynchronous calls and for every call past that (X+N) it was immediately calling WaitOne on the wait handle.

Josh To The Rescue

I fought with my example for an hour or so and realized that there wasn’t just some really simple thing for me to tack on to the existing code sample. This morning, I read Josh’s post about his approach to limiting the number of asynchronous calls using IEnumerable only. The key to his approach (and little did I know, to mine as well) was the really cool Aggregate extension method.

My Improved Solution

Action<IList<XElement>> saveAction = SaveChunk;
var loader = new BulkPostLoader(@"e:\stackoverflow\062010 so\posts.xml");
var batches = loader.BufferWithCount(5000);
var results = batches.Select(x => saveAction.BeginInvoke(x, null, null));
        
results
    .Aggregate(new HashSet<IAsyncResult>(), (set, result) =>
    {
        set.RemoveWhere(x => x.IsCompleted);
        set.Add(result);
        if(set.Count > 5)
        {
            var inProcess = set.Where(x => !x.IsCompleted).FirstOrDefault();
            if(inProcess != null)
            {
                inProcess.AsyncWaitHandle.WaitOne();
            }
        }
        return set;
    }
    .Subscribe(x => {});

As you can see, the main difference is that I’m using a HashSet to aggregate calls over the stream. Every time I clear out any calls that have completed to prevent completed calls from making my code behave as if the number of calls in process have reached the limit. Every time I add the most recent async handle to the set and then, if the set is above the limit, I take an uncompleted call and wait on it.

Well That Wasn’t So Bad, Was It?

By now you’ve certainly realized I’m not an Rx expert, I’m just sharing what I’m trying to learn as I learn it. Hopefully it’s more helpful than it is distracting.

Tags: Reactive Extensions, Threading

This weekend I discovered two very valuable ways to control asynchronous processes using the fantastic, amazing and immensely useful Reactive Extensions from the Microsoft Could Programmability Team (not the Research team as I had previously thought).

Limiting The Number Of Asynchronous Tasks

I recently blogged about ETL with Couch, Relax and Symbiote here. The source batched asynchronous results and blocked until they were completed causing the work to get queued 5 calls at a time and then block until those 5 finish. Not exactly ideal. (thanks Josh Bush for pointing this out)

EDIT

This first approach has a pretty serious bug in it. It only sort of works as described for a much better solution (thanks again to Josh for providing me with a good idea) please see this new post: An Improved Rx Approach For Limiting Asynchronous Calls

Here’s the code snippet:

Action<IList<XElement>> saveAction = SaveChunk;
var loader = new BulkPostLoader(@"e:\stackoverflow\062010 so\posts.xml");
var batches = loader.BufferWithCount(5000);
var results = batches.Select(x => saveAction.BeginInvoke(x, null, null));
        
results
    .BufferWithCount(5)
    .Subscribe(x => x.ForEach(y => y.AsyncWaitHandle.WaitOne()));

The fix here is probably quite obvious to anyone more familiar with Reactive Extensions than I am. If I want to ensure a steady flow of asynchronous tasks and only limit how many are running at a time, I can use a the .DoWhile extension method and provide a predicate which will only activate the following Subscriber when the condition is met. Here’s the code block from above with the change applied:

Action<IList<XElement>> saveAction = SaveChunk;
var loader = new BulkPostLoader(@"e:\stackoverflow\062010 so\posts.xml");
var batches = loader.BufferWithCount(5000);
var results = batches.Select(x => saveAction.BeginInvoke(x, null, null));
        
results
    .DoWhile(()=> results.Count > 5)
    .Subscribe(x => x.AsyncWaitHandle.WaitOne()));

Clean and simple. Have I mentioned I love the Reactive Extensions?

Throttling Execution Rate

While this next example certainly isn’t the only way one can throttle asynchronous processes, I liked this so much I wanted to share it. The challenge I was facing is that I don’t know how much work is actually available, I can only monitor how quickly work becomes available. I want to maximize throughput and minimize latency and still be kind to the processor.

First step is exposing the work load as an IObservable or IEnumerable and then using the Reactive Extensions to monitor the time intervals between each call. You can then use this interval to calculate a thread sleep time to use when there is no work available.

Here’s some very generic pseudo-code:

// First the code running inside the IObservable which
// is dequeueing messages and then notifying all the
// subscribed observers. This code is running in a
// worker thread (asynchronously) from the main thread.

while(Running)
{
    // this is where we check to see if there's any work
    var message = queue.Dequeue();
    if(message != null)
        NotifyObservers(message)  // calls send on each of the 
                                  // observers
    else
         Thread.Sleep(SleepFor); // SleepFor would be a property
                                 // that we can set externally    
}

// This code runs outside of the IObservable and is
// using the Reactive Extensions to react everytime
// a message is processed. In this case, I'm doing the
// simplest thing and just setting sleep time to the
// last interval between messages but you can do other
// things...

observer
    .TimeInterval()
    .Subscribe(x =>
    {
        observer.SleepFor = x.Interval.Miliseconds;                
    });

While that’s a very simplified version of what I’m actually using, I hope it demonstrates the principle. You can get insight into the time between each message and use it to throttle the loop that’s checking for work so that the code isn’t just cycling as fast as it can when there’s nothing to do and using up valuable processor cycles.

Suggestions? Feedback?

I’d be interested in seeing other patterns similar to these using the Reactive Extensions, it really is a great tool!

Tags: Reactive Extensions, Threading

Recently, one of our newest team members, Josh Bush, encouraged me to make my CouchDB API, originally a part of Symbiote, a free-standing assembly that would allow people to talk to CouchDB without having to use the Symbiote stack. Given that Dan Mohl and Elijah Manor had already convinced me that it was a good idea, I couldn’t justify putting it off any longer.

So Relax isn’t entirely free of it’s Symbiote dependency. It does still reference Symbiote.Core to get certain baseline functionality and also to incorporate interfaces which other Symbiote assemblies can supply to add functionality. The primary example is that you can get Relax to use memcached behind the scenes by using the Symbiote.Eidetic assembly and turning on caching. It’s basically 2 lines of code that can potentially have a nice performance boost.

Get Started?

First off, I plan to do a few screen casts that cover most (if not all) the Relax functionality. Currently, Relax supports the majority of CouchDB’s RESTful API. It will take me a while though to get all the screencasts put together, so in the meantime there are two other ways to get started.

The current wiki page for Relax contains several examples for how to get started and certain tasks. For more detail you can pull down the Symbiote source from GitHub and look at the code for the RelaxDemo console app which is basically the code I use for integration testing.

The source for Relax is here on GitHub. Currently Relax is .Net 4.0 and I do not plan on supporting .Net 3.5 even though the original Relax codebase began in 3.5. A lot of performance/concurrency improvements were introduced because of new classes in the framework and trying to grow the codebase for two significantly different frameworks isn’t something I believe would add a lot of value.

Feedback

I’m very interested in feedback from the community. Even if it’s just “I like this” or “I would rather role my own”, anything is appreciated simply because it lets me know that I’m spending my time on the right things that will help others. You can follow me on twitter using the link to the upper right of the blog or you can e-mail me here: asrobson AT gmail DOT com.

Tags: relax, symbiote, couchdb

Recently, Mike Taylor talked about his concerns about the increase of all the frameworks and libraries in his Pragmatic Bookshelf article, Tangled Up In Tools. If I understand his point correctly, he’s concerned that developers are too quick to adopt a library or framework which may or may not actually be a good fit and could actually cost the team time and functionality that they wouldn’t have lost had they just built what they needed. At the end of his post, he introduces the term ‘radius of comprehension’ which he defines as such:

“Radius of comprehension” is a new term that I am introducing here, because it describes an important concept that I don’t think there is a name for. It is a property of a codebase defined as follows: if you are looking at a given fragment of code, how far away from that bit of the code do you need to have in your mind at that time in order to understand the fragment at hand?

That’s an excellent concept (thanks Mike!). I think it’s one of those things that many programmers will read and think, “Exactly! I just didn’t know what to call it!”. Sure, we’ve had terms like “usability”, “intuitive” or “clean” but those terms are very subjective. Radius of comprehension is objective, you read the definition and you realize that it gives you a better construct for evaluating an API.

Introducing Symbiote

I said recently that I have been working for four months on a new open source set of libraries. Collectively, I think of it as the Symbiote framework. I feel like it’s a fairly unique attempt at taking a lot of different application concerns and technologies and reducing the radius of comprehension required to use these things in your application. It’s not a framework in the sense of, “Just plug in Symbiote and watch it handle everything”, rather it’s a way to centralize the configuration, dependency injection and simplify (or provide) APIs to some of the better open source libraries available.

Symbiote is more about providing useful APIs than enforcing a design methodology or approach. Furthermore, because *everything* in Symbiote uses a DI system and design by contract, you can replace pretty much anything you like with your own implementations. Symbiote isn’t designed to tie you down. It’s meant to provide value via utility.

Uh, Did You Actually Read Mike’s Article?

Yes, I did. The thing is that those of us employed as application/solution developers with business customers (internal or external) know that there’s a significant rift between the ideal circumstances in which to develop software and the realities we face each day which impede the development of software which provides value, meets needs and generally makes everyone happy.

My point here being, you need good tools. How else are you going to avoid technical debt while focusing on the features and aspects of the software which are perceived by your customers as providing value? Either you have a huge team, you take longer to release or you make technical sacrifices which will hurt the longevity/suitability of the solution. It’s that cut and dry.

Simple, Focused APIs For Open Source Libraries

The name is meant to indicate the over-all architecture of the entire framework. The framework consists of several focused libraries which take a dependency on the Symbiote.Core project. The idea is that you only add reference to one of the other Symbiote libraries to gain a specific piece of functionality.

Where It’s At

Symbiote is under heavy development. That means APIs are subject to change. While I’ve managed to isolate a lot of the changes behind the public facing interfaces, that’s not to say you shouldn’t expect some libraries to see more change than others until a library is stable. I’m also working very hard to improve test coverage and provide documentation. You can peek at the bits out on github: git://github.com/arobson/Symbiote.git and you can checkout what’s on the wiki so far at http://sharplearningcurve.com/wiki. The source includes some demo applications which really just help to serve as integration tests as well as a way to introduce the functionality provided.

What’s To Come

I plan on trying to blog fairly frequently about Symbiote libraries and functionality. There’s a lot to Symbiote so it’s going to take a while to cover everything it can do. If you’re impatient or really curious, I recommend pulling the bits down and playing with what’s there. I plan to blog soon about the CouchDB and

As always, I’m interested in feedback. You can e-mail me at asrobson AT gmail DOT com or follow me on Twitter, A_Robson.

Tags: symbiote, open source

In the last post I wrote up the steps required to setup a gitosis server. Now that I have a solution for source control, it’s time to start thinking about the build server. CruiseControl.Net is a nice, easy to use build server and there’s already a lot of support and documentation for it in the community.

One thing that I found a little sub-optimal about using CruiseControl and git together is that I had to build my own build trigger. The downside to the trigger is that it polls the git repository on a timer and when there’s a new commit, it tells CruiseControl, “hey pull the code and build it”. While that works, I’m not really thrilled with a chatty trigger…

A while back a fellow git user, Calvin Bottoms, told me about the hook scripts that come with git and how he was using them. The great thing about the hooks is that they’re basically just shell scripts which git calls in response to changes in the repository. This is exactly what I want; a way to “push” the event to the build server instead of polling. There are several hooks, but the hook I’m particularly interested in is the post-update script. (You can read more about the hooks in the official documentation.)

The only catch is that CruiseControl exposes integration via .Net remoting endpoints. Since my git server runs on Linux, there aren’t a lot of easy ways to call out to a .Net remoting endpoint. The simplest thing I came up with was using curl against a RESTful API. While CruiseControl doesn’t have a RESTful API, it just so happens that ASP.Net MVC comes with a wonderfully simple routing engine that I can use to build a utilitarian one.

So all I need in my MVC “application” is a controller with an action, an empty view (although there are other useful things one could do instead) and the route. The only action I’ll put in my controller is the Build action. It will take the name of the CruiseControl server and the name of the project I want to build. Here’s the code:

public ActionResult Build(string cruiseControlServer, string projectName)
{
    var uri = string.Format(@"tcp://{0}:21234/CruiseServerClient.rem", cruiseControlServer);
    var client = RemotingServices.Connect(typeof(ICruiseServerClient), uri) as ICruiseServerClient;
    var request = new ProjectRequest("build", projectName);
    var response = client.ForceBuild(request);

    ViewData["server"] = cruiseControlServer;
    ViewData["project"] = projectName;

    ViewData.Model = response;

    return new ViewResult();
}

Dead simple. Honestly, it’s too simple. To use this, you really need to recover gracefully from garbage input, or CruiseControl being down or the project name missing, etc. However, this is enough to get started and demonstrate what’s required. If you wanted to, you could actually get away with this being the only code you need to write. The downside is that without a custom route defined, the curl command to call this controller method looks like this:

curl http://localhost/cruisecontrol?cruiseControlServer=localhost&projectName=project

Yuck city. To me, that kind of URL is just begging for typos. Thanks to the Routing engine in MVC, we can remove the existing routes from the Global.asax.cs file and replace it with this:

routes.MapRoute(
   "Build",                                       
   "{cruiseControlServer}/{projectName}",
   new
       {
           controller = "CruiseControl", 
           action = "Build", 
           cruiseControlServer = "", 
           projectName = ""
       });

 

This lets me change my git hook script to:

curl http://localhost/cruisecontrol/localhost/project

Ah, much better! Adding the above call to the post-update hook in a git repository will cause it to call out to our new RESTful call which will trigger the build in CruiseControl. It’s definitely not as robust as it could be but it’s definitely enough to provide continuous integration builds on code pushes to the central/build repository.

Like I said before, I really need to add other things to this and should be able to easily extend functionality from here. What I’ve shown in this post is really just a nice starting point for better integration between CruiseControl and git.

Other posts in this series:

Introduction
ESXi, Debian and Git
Gitosis From Scratch

Tags: git, cruisecontrol, restful

Three months ago, I tried to start a series about my efforts to find a continuous integration solution that I could get on board with. Things have been pretty crazy (more on that someday) since and I haven’t had much time to work on this entry.

What You Should Have At The End Of This Post

By the end of this post, if you’ve followed the steps correctly, you should be able to add new repositories, configure access and add users to a central/shared git server from the comfort of your Windows environment. I’ve tried to provide enough narrative to explain what you’re actually doing, but this is a very long post; I won’t be offended if you just skip to the good parts.

Prerequisites

You need to have msysgit (or alternative) on your machine. You should already have a ssh key pair installed. You need to be comfortable with git and a bash console. If you’re completely new to Linux, stop reading this and look for some good Linux introductions.

Why I Think Gitosis Is Great

Gitosis is like magic. I think that in order to understand why gitosis is magic, you have to understand a little bit about what life would be like using git without gitosis. As I mentioned in my previous post, the trouble with a git server in a Windows domain is that git-daemon won’t run on Windows. Hosting it on Linux isn’t a simple solution because unless you want the headache of joining the Linux server to your domain, none of your developers will have rights to the machine. In order to provide them with access, you have to manage ssh and keys and folder permissions and…  it boils down to a lot of busy work. I hate busywork.

Enter gitosis! Gitosis provides you with a simple way to manage keys, permissions, repositories, web browsing of the repositories and public read-only access. Enough blabbering… it’s time to dive in.

The Environment

For this blog post I’ve created a brand-new VM on version 5.03 of Debian, and version 7.01 of VMWare Workstation. For testing I will use msysgit 1.6.5 on Windows 7. Because I have never been able to get the host OS to communicate with the guest OS by name, I have put an entry in my hosts file on Windows so that I can issue commands to the virtual git server by name rather than IP address. (you can find your hosts file in c:\windows\system32\drivers\etc\hosts)

For the rest of the article, if I say host OS I mean my Windows 7 box running msysgit. If I say guest OS or git server, I’m talking about the Debian VM. Anywhere you see [git-server] just replace it with your git server’s name. Anyplace you see [your@email.com], you should replace it with the email address that you associated with your ssh key.

Installing

Mount the disc to the VM that you installed Debian with. Apt-get will almost always require you to do that.

You need Apache server if you’re going to use gitweb. If it’s not installed this command will handle that:

sudo apt-get install apache2

From the command prompt, type:

sudo apt-get install git-core gitweb git-daemon-run gitosis

This command line is installs 4 packages (and any missing dependencies) for you: git, gitweb, git-daemon and gitosis.

Create A User That “Owns” The Repositories

 

sudo adduser \
    --system \
    --shell /bin/sh \
    --gecos 'git version control' \
    --group \
    --disabled-password \
    --home /home/git \
    git

 

This command line create the user git and sets their home directory to /home/git. This is where gitosis will keep all the managed repositories.

Update Git-Daemon Configuration

The git-daemon needs to be told where we’re going to be keeping the repositories that it will manage because we’re not using the default location it expects (which was /var/cache/git).

sudo vi /etc/service/git-daemon/run

comment out the last line in the file and replace it with:

exec git-daemon --verbose --base-path=/home/git/repositories/ --export-all

Now issue the command:

sudo /usr/bin/git-daemon restart

If this command results in the error:

fatal: unable to allocate any listen sockets on host (null) port 9418

then there’s a good chance git-daemon is already running in the background. My advice is to run ps –A to find the git-daemon process id and issue a kill command. That will cause git-daemon to re-load your changed configuration.

At this point, the git-daemon should provide public, read-only access. Let’s test that on the host OS:

git clone git://[git-server]/gitosis-admin.git

You should see the following response:

Initialized empty Git repository in [path]
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (5/5), done.
remote Total 5 (delta 0), reused 5 (delta 0)
Receiving objects: 100% (5/5), done.

Create The First Gitosis Admin

You need a way to authenticate with gitosis so that you can manage it remotely via it’s gitosis-admin repository. The way you do this is by providing the public key of your ssh key pair to gitosis.

This may sound intimidating but you can do it very simply by following one of two methods:

Using Only The Bash Prompt:

1. On your host OS, locate the .ssh folder contained in your home directory
2. Open the id_rsa.pub file contained in that directory in a text editor
3. Copy all the text
4. On your guest OS, navigate to /tmp by typing cd /tmp at the prompt
5. Type ‘echo’, then in the console menu under Edit choose paste, then finish the command by typing > id_pub.rsa

Using Vi:

1. On the host, locate the .ssh folder contained in your home directory
2. Open the id_rsa.pub file contained in that directory in a text editor
3. Copy all the contents
4. On your guest OS, type vi id_pub.rsa
5. Hit the ‘i’ key and using the console menu, under Edit, choose the paste option
6. Hit escape, hit ‘:’ then type wq and press enter

Initialize Gitosis

Now that we’ve created the public key for the gitosis admin on the git server, we can initialize gitosis with that key.

sudo –H –u git gitosis-init < /tmp/id_rsa.pub

This invokes the command under the git account you created previously and initializes gitosis with your public key so that you’ll be able to authenticate for push and pull operations with the git server.

Testing Gitosis

Believe it or not, we’re actually ready to check and see whether or not gitosis is ready to go. In your git-bash shell on the host OS, navigate to a directory where you keep your git repositories and enter the following:

git clone git@[git-server-name]:gitosis-admin.git

My command looked like this:

git clone git@[git-server]:gitosis-admin.git

This should pull down the gitosis-admin repository which you can use to remotely manage gitosis.

Setting Up GitWeb

For me, this part was the most difficult, it took me a while to figure out all the steps but it’s definitely worth it.

We need to add www-data to the git group so that it will have read permissions to the repositories.

sudo usermod -G www-data,git www-data

Next we need to provide a way to host gitweb in apache. I’ve seen a few different approaches but prefer the way I’m about to show you.

sudo mkdir /var/www/git
sudo chmod 755 /www/git
sudo cp /usr/share/gitweb/* /var/www/git
sudo cp /usr/lib/cgi-bin/gitweb.cgi /var/www/git

Those four lines are creating a virtual directory in apache for gitweb, copying the gitweb.cgi, css, image and icon files over and setting the proper permissions. Next we need to create a configuration block for our gitweb in apache.

vi /etc/apache2/conf.d

<Directory /var/www/git>
    Allow from all
    AllowOverride all
    Order allow,deny
    Options ExecCGI
    <Files gitweb.cgi>
        SetHandler cgi-script
    </Files>
</Directory>
DirectoryIndex gitweb.cgi
SetEnv GITWEB_CONFIG /etc/gitweb.conf

The last file we need to change is the configuration for gitweb itself. Change it to look like the following:

vi /etc/gitweb.conf

# path to git projects (<project>.git)
#projectroot = "/home/git/repositories";

# directory to use for temp files
$git_temp = "/tmp";

#target of the home link on top of all pages
#$home_link = $my_uri || "/";

$ html text to include at home page
$home_text = "indextext.html";

# file with project list; by default, simply scan the projectroot dir.
$projects_list = $projectroot;

# stylesheet to use
$stylesheet = "/git/gitweb.css";

# logo to use
$logo = "/git/git-logo.png";

# the 'favicon'
$favicon = "/git/git-favicon.png";

Now reset apache to reload the new configuration:

sudo /etc/init.d/apache2 restart

Configuring Gitosis

Let’s take a look at the default gitosis configuration file:

[gitosis]

[group gitosis-admin]
writable = gitosis-admin
members = [your@email.com]

Not much there to see, right? Let’s say we want git-daemon to export the repositories so that they can be served to the public as read-only and we also want gitweb to export our repositories for web-browsing. To do that we’d change the config to look like this:

[gitosis]
gitweb = yes
daemon = yes

[group gitosis-admin]
writable = gitosis-admin
members = [your@email.com]

That’s great, but we don’t have any repositories to test this out on (the gitosis-admin repository is special and won’t share out to gitweb). Let’s make a very simple repository to test.

1. On your guest OS make a new directory called test
2. CD to test and type the following commands:

git init
echo This is a test file! > readme.txt
git add .
git commit -m "Adding a readme.txt file"

Let’s add the new repository to gitosis.conf file and create a developer group:

[gitosis]
gitweb = yes
daemon = yes

[repo gitosis-admin]
gitweb = no
daemon = no

[repo test]
gitweb = yes
daemon = yes
owner = [your@email.com]
description = A test repository

[group gitosis-admin]
writable = gitosis-admin
members = [your@email.com]

[group developers]
writable = test
members = [your@email.com]

Now we need to add, commit and push our configuration changes up to the server like so:

git add .
git commit -m "adding configuration for new repository"
git push

If the configuration was successful, you should be able to change directory back to your test repository on the host OS and issue the following command to push it up to the git server:

git push git@[git-server]:test.git master

The easiest way to test this is by navigating to your git server’s url: http://[gitserver]/git

All Done!

At this point you should be able to add repositories, manage access and add users all through the gitosis-admin repository. You should also have gitweb access running as well as public repository accessibility via git-daemon.

Hopefully this has been helpful to others out there who develop in a Windows environment but would like to use a central or shared git repository to coordinate development on their team.

Tags: git, gitosis, debian, linux, gitweb, git-daemon, source control

I’ve seen several tutorials and blog entries that don’t quite get the syntax correct, (especially if the path contains spaces). This is a short blog post, but after an hour of fiddling with the syntax to try and get git to use BC3, I figured having the information available makes it worth it:

The following entry needs to go in your .gitconfig file:

[merge]
    tool = bc3
[mergetool "bc3"]
    cmd = \"[path to beyond compare]/bcomp.com\" \"$PWD/$LOCAL\" \"$PWD/$REMOTE\" \"$PWD/$BASE\" \"$PWD/$MERGED\"
    keepBackup = false
    trustExitCode = false

Just to for clarity’s sake, a concrete example on my machine looks like this: (Beyond Compare 3 installed on a x64 Windows 7 machine):

[merge]
    tool = bc3
[mergetool "bc3"]
    cmd = \"c:/program files (x86)/Beyond Compare 3/bcomp.com\" \"$PWD/$LOCAL\" \"$PWD/$REMOTE\" \"$PWD/$BASE\" \"$PWD/$MERGED\"
    keepBackup = false
    trustExitCode = false

To find where .gitconfig file is located, you can issue the following commands from the git bash prompt:

cd ~
pwd

This will change directory to your home path and then print the full path to the console. Happy merging!

Tags: git, beyond compare

Or maybe I should entitle this blog, “My Secret Shame”. I’ve got a confession to make: until this morning, I’ve listened to maybe a total of 3 podcasts… One with Roy Osherove about testing, one about ASP.Net MVC and Elijah Manor’s first jQuery podcast. Oh, the shame.

I think a lot of it is that it’s difficult for me to do just one thing at a time, and it’s a lot more natural to me to stop mid-read than it is mid-listen. Also, I am a total n00b about podcasts and the best way to get them.

But no more… thanks to the quality of content on Herding Code and jQuery podcast, I could resist no longer. I finally got around to finding a nice podcast application (DoggCatcher) and have been downloading podcasts from both feeds this morning. Right now, I’m listening to episode 71 which talks a lot about things I’m very interested in; NoSQL databases. (more on that later)

Can anyone reading this suggest other good developer/tech podcasts?

Tags: podcasts

Have you ever heard some truism or principle, immediately thought, “Exactly! It’s so simple and obvious!” then looked around at your colleagues and exchanged some knowing laughter. Maybe you even made fun of the poor bastards who didn’t get it? “Huh-huh-huh, yeah, like this guy, Durfin, he’ll NEVER GET IT!”

Well, whenever I used to hear things like “don’t use a hammer to drive screws”, “use the right tool for the job”, etc., I always reacted like that. Like a terd. Never bothering to really reflect upon whether or not my blind application of all things Microsoft was right. I am moving the other direction now and trying to really examine my assumptions about what’s necessary vs. what’s common. Instead of just being all abstract and gushy though, I’m going to be more specific and go over common assumptions that myself and others in Microsoft country have been stumbling over for quite some time now.

If You Need RPC, WCF Is Great!

In my experience, WCF is great if you enjoy spending a lot of time on high-learning-curve, low-return, frustrating, bloat-ware. I won’t revisit all the magical ways WCF can bite you, but I will say that WCF users generally fall into three groups:

1. Geniuses. These folks have no idea why anyone would complain about WCF.
2. The Frustrated. That includes people like me.
3. The clueless. These people think they’re in the first group. They’re using WCF in an over-simplistic, “look, ma, I made codez!”, demo quality applications which create more jobs for people in the first and second group when it all goes to hell.

My point is that WCF is solving a problem that doesn’t really exist outside of the Microsoft ecosystem. With the right tools, you can accomplish the same end-goal in much more simple and elegant ways:

Erlang – has the ability to communicate natively with distributed Erlang nodes built into the language itself. It’s accomplished via a trivial one liner that takes the form: <address> ! <message>. Simple. Elegant. Powerful.

AMQP and XMPP are open protocols which relate specifically to messaging. Lots of stuff use these two. Google’s Wave is built on an extension to XMPP. My favorite AMQP implementation is RabbitMQ (written in Erlang).

Take away here – WCF is overkill. It’s a jackhammer for jeweler’s screws.

If You Need To Store Something, Microsoft SQL Is King!

Microsoft SQL is a good RDMS. Hard to argue that. What are relational stores good for? Analytical processing of data. Most applications I’ve worked on actually need analytics but most of the time is spent figuring out how to make my application model save its state in a schema that’s well suited to analytics.

The NoSQL movement is good and bad. The problem is too many reactionaries (myself included) hear NoSQL and think “YAY, NO MORE RDBMS!!!” That’s actually a terrible idea for many business line applications or anything you need analytics for. If you want reporting, you’re going to find that writing reports against a document store is probably just as torturous as mapping a model to normalized tables.

Use the right tool for the right job. Use the document databases for your transactional store and use relational databases for your analytics. You can do both. (I’m not going to go into that right now). The important thing is realizing that it’s not all or nothing in one approach.

SOA Is The One True Architecture

SOA gets overused so much that it means different things to different people. Many developers, formerly myself, believed that SOA meant lots of atomic services which performed certain tasks and that combining these services was how you built out an application. For those of you who don’t have infinite time and resources to maintain your application, may I suggest that this implementation of SOA is really just 3 tiered architecture which rarely scales out well and is fairly difficult to architect, develop and maintain.

Instead, I would point you to Gregor Hophe, Udi Dahan and others that would suggest SOA is something very different. Messaging based architectures allow you to add services which look more like nodes on a bus. Each node performs a specific role and the beauty here is that the things putting messages on the bus and the things reacting to messages coming off the bus don’t have to know *anything* about each other. They only need to know about the message types and the bus. Ta-dow.

How All Of This Relates

I’m still working on that. In my last post I talked about the fallacy of thinking I was at least aware of everything I needed to know about. These are some of the things I thought I knew which were instrumental in leading me down a path of high frustration, long hours and low return.

I’m not saying I currently have everything figured out, but I am saying that looking into other tools and languages has helped me realize that there are far better ways to solve the problems that applications developers face daily. I’m blogging about this stuff because it’s exciting to me and I feel like it’s information worth passing around.

Tags: .net, erlang, amqp, xmpp, wcf

As I said in the intro, this series of posts is all about me trying to find a solution that I like for continuous integration. It’s about the search. It’s about the learning process. It’s about seeing how many times I run into a wall before getting out of the maze or discovering there’s no end. The title is a dead give-away but in case you missed it, my journey begins using trying out git for source control.

Goal
Last week I decided it was time to actually look into what it would take to get an internal git solution together. If you’re new to this blog, you might not know that I love git. Well, I do. What I didn’t love were my options for running it on Windows. I don’t mind the idea of developers making their root git directories shared amongst the Active Directory Developer’s group. I actually think that’s an acceptable way to provide read-only access so that others can pull/clone/fetch/etc from their peers. But for my primary CI build server, I want a separate, more tightly controlled git repository so I know when and how the source gets there.

Git
Git has 3 ‘bits’ to it: git (core), git-daemon and git-web. Each of these rely on POSIX. That means to run them in Windows you’d have to try something like CYGWIN. Now, I’m not knocking CYGWIN, but it sounded too much to me like I was trying to force something unnatural. Why not just run git the way holy Torvalds intended on an OS of his own grand design?

Git core is what you’ll find the most documentation about because by itself, it’s a complete source control system. Git-daemon is a service that you can use to natively serve up git repositories to the outside world. It’s really not ideal for anyone doing development on Windows boxes though because it expects that anyone interacting with it will have accounts on the box/workgroup/domain that the host OS knows about. Git-web is a terrific web front end for exploring git repositories that runs on Apache. I can’t really say enough nice things about it because I’ve literally never seen anything integrated that easily and perform so well. My favorite feature so far? You can grep a repository’s files from the web interface and it’s lightning fast.

Where To Get It
I’m using Debian, so I apt-get installed my way there. It’s a simple, clean install experience and you don’t even have to recompile the kernel. If you’re curious about using git on windows, I can’t recommend msysgit highly enough.

Debian
Debian was actually not my first choice for OS. I was going to try several different distributions that I had experience with, starting with OpenSUSE but in the years that have passed since I tried OpenSUSE they’ve made some changes to it that made it difficult to get git setup according to some of the useful guides I’ve found.

After several disappointing hours of fiddling with OpenSUSE, I decided to try Debian based on the recommendations of Dave Purdon and Evan Hoff. I have to say, I really like Debian. It’s clean, simple and everything’s where I expect it to be (and where most git-related walk-throughs tell you they’ll be).

Where To Get It
The best way to get your hands on it is using microTorrent and the torrent(s) for Debian. CD disc 1 is sufficient to get your install started although you will end up pulling a few things down from a mirror of your choice to complete the install.

ESXi
I never questioned that I’d be running these servers as virtual machines. And even though I have experience with ESX and how awesome it is, I don’t currently have access to the company ESX server. So to get started I was considering workstation or virtual box or something and Evan Hoff recommended I look into ESXi, VMWare’s free-esque ESX package. If you’re unfamiliar with either, ESX[i] is a Type 1 hypervisor meaning it runs on bare metal. This means no stinky host OS to gobble up your precious resources. ESXi requires you only use it on boxes with 1 processor (up to 6 cores) amongst other things, so I re-provisioned my desktop as my ESXi server. It’s down-right dreamy. ESXi is missing some of the more spectacular and magical things that it’s commercial big brother can do, but heck; it’s FREE.

Where To Get It
ESXi is available here and you do need to register for it if you would like to continue using it after 60 days. The vSphere client software can actually be downloaded from a URL on your ESXi server once it’s up and running.

Important Note:
If you happen to try it out for yourself, beware that you will have to hack about with the Vitural Sphere client to get it to run on Windows 7. (read about how to do that here)

Next Steps
So, shortly after getting ESXi installed, I spun up a VM for Debian, got Debian installed, apt-get installed git, git-daemon and git-web and then promptly ran into a wall. Next post I’ll talk about getting past the road block and go into a lot more step-by-step detail (as well as provide more links and resources).

Tags: continuous integration, git