Sharp Learning Curve Ahead

While my recent work on Symbiote and Relax probably appears to be all over the place, there is a unifying, underlying purpose behind all the work I’m doing. This post is about my short and long term goals. It’s about the technologies and architectures I believe are going to become important in the not-too-distant-yet-not-immediate future.

Who I’m Building For

I’m primarily building tools for our development team at work. We have seven developers (myself included), which work on multiple projects for internal and external customers. We target SMBs, usually via SaaS solutions.

I’m hopeful that more developers on similar teams with similar needs will find that the Symbiote libraries provide a simple, easy way to adopt some of the great open source frameworks available.

Tenets

If I had a technical manifesto, it would ridiculously opinionated, long and in printed form, might be used to even out furniture with wobbly legs. It would also talk a lot about the following tenets:

1. Open standards are the way forward. Proprietary is bad.
2. Distributed architectures will be the best way to take advantage of the new advances in hardware.
3. Open source alternatives to bloated, closed technologies will become vital to small/medium development shops.
4. “Teh Cloud” will become a great place for solutions which aren’t built from proprietary pieces.
5. APIs that are “discoverable”, provide extensibility via dependency injection, and are built around conventions but provide configuration provide the most value. (shameless plug for Symbiote)

Technologies and Architectures I <3

Symbiote isn’t a complete list, but it’s a good start of the architectures and technologies I’m excited about. RabbitMQ, CouchDB and Lucene are three technologies I love. CQRS, messaging, and RESTful are just a few of the architectures that can produce powerful and agile solutions.

Technologies I Avoid

Knowing when a technology is bad for you is a skill that I’ve learned the hard way. It’s been painful. It’s been costly. My poor coworkers are probably developing mental defense mechanisms due to my tendency to wax bitter about certain technologies that infamous for burning projects and small teams to the ground.

Bloated, proprietary, and closed systems are a bad fit for most projects I’ve ever worked on. If you’re going to use technologies built for fortune 500 IT organizations, you need to BE a fortune 500 IT organization. These kinds of technologies are career paths unto their own. If you’re a smaller shop, you generally can’t spare entire people to these things alone.

What I’m Focused On Now

Our team is investing a good bit of time developing technologies around CouchDB. Why? It performs well. Schema-less storage is a huge time-savings for us because it doesn’t require a ton of up-front design and it ‘evolves’ gracefully with our domain model (at least so far). The team is going to need to address certain things that CouchDB doesn’t do out of the box. Those things are:

1. Handling relationships between document types
2. Open search capabilities. Writing views for everything a user may want to search on isn’t practical.
3. Reporting capabilities. Josh Bush and Jim Cowart are doing a brilliant job in this area.

That said, I’m trying to fast track a Relax RC that provides indexing and query services using Lucene. I have a proof of concept for those services and I’m also working on a LINQ provider.

Where To Get More Information

Following me on this blog, or on twitter are good starts. If you’re feeling adventurous and want to play with the code, check out http://github.com/arobson and see the Symbiote and Relax repositories. There is currently a wiki at http://sharplearningcurve.com/wiki and I’m also (slowly) working on a site dedicated to Relax documentation, features and updates.

Tags: architecture, symbiote, relax, couchdb, restful, lucene, nosql

For years now I’ve been trying to create an open source framework that would address many common application development concerns while reducing the amount of time required to get a project started and on it’s way. My professional experience has taught me that development teams are almost never given enough time and are constantly having to cut corners, incur a lot of technical debt, and choose technologies which aren’t a good fit simply because they don’t have time to incorporate anything new.

While I’ve learned a lot throughout the process and written some interesting code, my efforts have largely resulted in generous shipments of fail. The good news? I’ve tried very hard to learn from those failures and make them count.

Nvigorate

The first project I started was called Nvigorate. It centered largely around an ORM. Yep. I’m that kind of crazy. Thanks largely to people like Craig Israel, the ORM actually got to a really decent feature set. The primary issue was that 2 or 3 developers can’t rival the existing code base, time and team around NHibernate. Not to mention that NHibernate has nearly become the standard in shops where open source is an option. I think I gave up right around the time that I learned that NHibernate also had nice fluent mapping API and a LINQ API and could support certain inheritance scenarios that I knew would take a very long time to work into my code base. I learned a lot of valuable lessons trying to make Nvigorate into something viable. Here are just a few:

1. Frameworks that don’t make use of dependency injection are gargbage
2. Beware internal dependencies on other pieces of the framework
3. Create focused, decoupled libraries where possible
4. Lack of unit test coverage can kill a project dead

BourneFramework

This wasn’t really my framework but I contributed a lot to it. Basically, I contributed way too much to it and realized that I was trying to do a lot more than was originally intended. Bourne was initially intended to bring NHibernate integration into the different tiers: mvc, wcf, services, etc. I made the mistake of trying to make it do way to much and as a result the configuration aspect of the API became a big, awkward, fragmented and difficult to explain or use. So from this I learned:

1. Don’t try to make a framework do more than it was initially designed to do, it won’t make “sense”
2. Make the configuration and setup simple as you can
3. One way to configure an API is best.

This leads me to the new open source framework I started. I think it’s got promise. I’ve been working on it for about 4 months now and we’ve been using pieces of it at work. From the feedback I’ve received so far, I think it’s actually easy to learn and easy to use and provides value. I’ll be talking about it more on the blog very soon.

Tags: bourne framework, nvigorate

I’ve been working on an open source project that’s kind of like an aggregator and abstraction framework for incorporating several different open source projects. It has been a lot of fun so far and some of the developers I work with are trying to use it on our newer development efforts.

One of the issues I’ve recently run into is one of the APIs I wrote provides the user with four possible dependencies. Each dependency provides the same functionality but with varying degrees of customization to the object model they’re working with. Initially I thought this was the best way to allow users to completely customize how they choose to work with the underlying system. But recently I’ve discovered that in most cases, developers see the most simple interface and just assume that it will work with their object model regardless of how customized. Syntax errors are the only thing they get. It’s not like the build tells them “Hey, use a different interface if you’ve done this, this and this”. It just barfs on them. It looks like my API has a bug…

I remember Brad Abrams recounting his experience in designing the first version of the .Net Framework; he talks about watching developers new to the framework trying to work with the IO namespace and how developer after developer kept stumbling over the API when trying to do something as simple as open a file and read the contents. He experienced denial, shock, shame, and finally settled into a determination to make the code better and easier to use. (Framework Design Guidelines, 2nd Edition)

It wasn’t that they had bugs in the IO namespace in .Net 1.0. It was that the API wasn’t “discoverable”. How often do we as developers try to learn new APIs through intellisense alone? Especially in open source projects where documentation is scarce, you hope that the API is clearly designed enough and the assemblies structured enough that you can go off one or two examples and discover what you need.

I think that one of the mistakes I made in the design was giving the user too many choices. Not only that, but more importantly my API didn’t even have the decency to make the choices apparent to the end user.

I’m currently working on a fairly extensive re-write that will reduce the API to a single dependency which will inform the user of the constraints put in place on their object model. I’ll provide some classes that help them meet those constraints and go from there. In the end, while it reduces the flexibility of the over-all API, I think it results in a much better design because it removes the likelihood that a developer gets stuck with runtime errors that give them no clue as to what’s going on and leaves them to assume the library is broken.

Tags:

Have you ever heard some truism or principle, immediately thought, “Exactly! It’s so simple and obvious!” then looked around at your colleagues and exchanged some knowing laughter. Maybe you even made fun of the poor bastards who didn’t get it? “Huh-huh-huh, yeah, like this guy, Durfin, he’ll NEVER GET IT!”

Well, whenever I used to hear things like “don’t use a hammer to drive screws”, “use the right tool for the job”, etc., I always reacted like that. Like a terd. Never bothering to really reflect upon whether or not my blind application of all things Microsoft was right. I am moving the other direction now and trying to really examine my assumptions about what’s necessary vs. what’s common. Instead of just being all abstract and gushy though, I’m going to be more specific and go over common assumptions that myself and others in Microsoft country have been stumbling over for quite some time now.

If You Need RPC, WCF Is Great!

In my experience, WCF is great if you enjoy spending a lot of time on high-learning-curve, low-return, frustrating, bloat-ware. I won’t revisit all the magical ways WCF can bite you, but I will say that WCF users generally fall into three groups:

1. Geniuses. These folks have no idea why anyone would complain about WCF.
2. The Frustrated. That includes people like me.
3. The clueless. These people think they’re in the first group. They’re using WCF in an over-simplistic, “look, ma, I made codez!”, demo quality applications which create more jobs for people in the first and second group when it all goes to hell.

My point is that WCF is solving a problem that doesn’t really exist outside of the Microsoft ecosystem. With the right tools, you can accomplish the same end-goal in much more simple and elegant ways:

Erlang – has the ability to communicate natively with distributed Erlang nodes built into the language itself. It’s accomplished via a trivial one liner that takes the form: <address> ! <message>. Simple. Elegant. Powerful.

AMQP and XMPP are open protocols which relate specifically to messaging. Lots of stuff use these two. Google’s Wave is built on an extension to XMPP. My favorite AMQP implementation is RabbitMQ (written in Erlang).

Take away here – WCF is overkill. It’s a jackhammer for jeweler’s screws.

If You Need To Store Something, Microsoft SQL Is King!

Microsoft SQL is a good RDMS. Hard to argue that. What are relational stores good for? Analytical processing of data. Most applications I’ve worked on actually need analytics but most of the time is spent figuring out how to make my application model save its state in a schema that’s well suited to analytics.

The NoSQL movement is good and bad. The problem is too many reactionaries (myself included) hear NoSQL and think “YAY, NO MORE RDBMS!!!” That’s actually a terrible idea for many business line applications or anything you need analytics for. If you want reporting, you’re going to find that writing reports against a document store is probably just as torturous as mapping a model to normalized tables.

Use the right tool for the right job. Use the document databases for your transactional store and use relational databases for your analytics. You can do both. (I’m not going to go into that right now). The important thing is realizing that it’s not all or nothing in one approach.

SOA Is The One True Architecture

SOA gets overused so much that it means different things to different people. Many developers, formerly myself, believed that SOA meant lots of atomic services which performed certain tasks and that combining these services was how you built out an application. For those of you who don’t have infinite time and resources to maintain your application, may I suggest that this implementation of SOA is really just 3 tiered architecture which rarely scales out well and is fairly difficult to architect, develop and maintain.

Instead, I would point you to Gregor Hophe, Udi Dahan and others that would suggest SOA is something very different. Messaging based architectures allow you to add services which look more like nodes on a bus. Each node performs a specific role and the beauty here is that the things putting messages on the bus and the things reacting to messages coming off the bus don’t have to know *anything* about each other. They only need to know about the message types and the bus. Ta-dow.

How All Of This Relates

I’m still working on that. In my last post I talked about the fallacy of thinking I was at least aware of everything I needed to know about. These are some of the things I thought I knew which were instrumental in leading me down a path of high frustration, long hours and low return.

I’m not saying I currently have everything figured out, but I am saying that looking into other tools and languages has helped me realize that there are far better ways to solve the problems that applications developers face daily. I’m blogging about this stuff because it’s exciting to me and I feel like it’s information worth passing around.

Tags: .net, erlang, amqp, xmpp, wcf

So maybe my blog name isn’t self-explanatory. The reason it’s called sharp learning curve ahead is because I think it describes what my ongoing experience in IT has been and continues to be; one learning challenge after another. If you’re like me, you look forward to the challenges and even enjoy the paradigm shifts. Thankfully, I have the extreme good fortune to work with pros like Jim Cowart, Elijah Manor and Evan Hoff; each being exceptional and each of them excellent mentors in their respective areas of expertise. The good thing about working with so many solid people is that you’re constantly learning. Well, if I could summarize some of the things I’ve been learning about lately, I’d say they’ve been along the lines of learning that there are much better tools available than what I’ve been using.

On Friday, Evan was patiently educating me on a rather important fundamental architectural pattern that I somehow managed to know almost nothing about, messaging. Sure I’d heard about service busses and messaging before, but the only place I had heard it applied to something concrete was, I believe, a very weak implementation and so I (foolishly) glossed over the concepts, uninterested at the time, assuming that what those concepts had to offer wasn’t significant. Sadly there hadn’t been any further opportunities for me to expose my ignorance on the topic so someone could correct me. If you’re not so impressed with messaging, I highly recommend this screen cast, it was an excellent overview, and explanation of the how and why of SOA via messaging.

In light of that recent discovery, I now feel like I’ve been trying to hammer nails with a screwdriver for several years now and have just been shown an actual hammer. But the fun doesn’t stop there.

Up until the past few months, I was certain that Nvigorate’s ORM had a reasonable value proposition and even made sense in light of NHibernate (which is practically the de facto ORM for .Net). I started to doubt that with the addition of Fluent NHibernate, which makes the whole adoption process less odious, but still I believed that Nvigorate’s API, extensibility and superior state tracking justified my on-going development. Well, two weeks back, LINQ to NHibernate was released, and having worked briefly on a LINQ adapter in the past, I know what a difficult task that is. Shortly after that, Matt Honeycutt, was telling me about how he had used NHibernate’s schema update functionality to resolve differences between his model and the database schema and update the schema. You see where this is going, right? At this point I could continue to work on the ORM piece of Nvigorate, but I’d spend months trying to match two features. Two features which any developer should be really excited about using and won’t want to wait months for.

I’ve been playing with the Fluent NHibernate and LINQ to NHibernate some this weekend and have acquired Mass Transit so I can start figuring out how all the pieces work together. What about Nvigorate? It’s not going away, there are still a lot of positive things it has to offer outside of the ORM that I’m working on, such as a fluent WCF Proxy class that eliminates the need for generating WCF proxies.

At any rate, I wanted to blog about all of this because I the path I’m about to take is going to really change the direction of this blog in general and explaining why I stopped working on/talking about the Nvigorate ORM piece seemed important.

Tags: nvigorate, nhibernate, patterns