On 02 May, I had the opportunity to return to my alma mater - Rose-Hulman - and speak to a class on Software QA.  It was a great opportunity and discussion spurred by (hopefully funny) war stories and major screwups that I was more than happy to share.

During the discussion, I had the opportunity to speak about our development process here for WhyGoSolo.  The most interesting thing that struck me was a comment I received from the student feedback after:

I thought that Keith had a lot of interesting, informative things to say about his company’s development process.  Of all of the presenters, I think that he is one of the few that has even a semblance of formal development process in a small company setting.  I was intrigued to hear how formal and mature his startup’s development process is – it seems to be a trend that most companies don’t develop a mature process until they have been around for quite a while and wish to receive various industry certifications.

Unfortunately, he's correct.  Most companies don't try to clean up their processes until someone needs to audit them... until there's something else on the line.

What they forget is that there's almost always something on the line. 

Having a formal development process seems like a lot of overhead.  It seems like a lot of extra work...

Until the first time the production database gets dropped.

Or the first time users are locked out of the system.

Or the first time you lose that big opportunity.

Then you and your team are going to stop and consider how you can make it better next time.  It could be as simple as requiring every change to production to be reviewed by a second set of eyes.  it could be as simple as requiring one person to write the database update and someone else to execute it.  It could be as simple as making sure your staging environment gets the review required prior to deployment to production.  The consistency that comes along with a process could be the single most valuable part.

I don't care what your process is: Just get one and get it rolling.

It will never require forms signed in triplicate, filed, lost, buried in peat moss, and filed again... well, almost never.

You've heard us complain about it.

You've seen us struggle with it.

Now we're going to take a step to change things... and you can help.  In fact, we need you to help.

On 26 April 2008, we have the Mid Atlantic Business Plan Competition.  The panel of judges includes a number of VC's, fund manager types, entrepreneurs, and even a more familiar face from the DC Tech scene. ;)

So here's the plan.  We're going to flood the place.  We're going to get the best and brightest people in DC to show up and participate.  When the judges and the press and the professors and the students look around the room, we need to show them that there definitely is a startup community in DC.  That there are entrepreneurs in DC.  That we do have the passion to make our dreams happen and change our environment.  That entrepreneurs don't have to leave DC to make a go at it.

So Keith, what do we do?

First, talk about it.

Pass this post - or any of the other ones - around to other entrepreneurs and smart people you know.  Ask them to spread the word and talk/blog/tweet about it.

Second, show up.

Give up one Saturday afternoon to help us swarm the place!  Jimmy Gardner - a friend and champion of the local DC Tech scene - has created a Facebook Group (DC Technology Business Plan Competition Crashers) and set up an event to coordinate our efforts.

Place: ICC Auditorium, Georgetown University (Building 26 on this map)

Time: 1 - 5pm

My goal is 200+ sharp people from the community. Are you going to be one of them?

As of 11:30pm EST on 07 April, we have 18 people.  About 10%!

Every development team faces it.  There's a major component which although not key to your applicationevery development team has to make the decision... and odds are that you've made the wrong decision... but here's the thing:

Everyone makes the wrong choice.

Wha?

Yes, no matter which way you chose, it was probably wrong...

When we shopped around for a blogging engine for WhyGoSolo, we didn't find a good match.  We briefly reviewed Mephisto and a few others but integrating the look and feel and the User pieces was going to be cumbersome at best and incredibly painful at worst.  It was going to require some hacking at and extending the core functionality and features of whatever system we chose.

The most dangerous part of this one is simple...  although you get a huge amount of functionality quickly, as soon as you start changing and customizing a third-party tool, you've forked it.  Now upgrading isn't as simple as dropping in a new version and testing.  When bug fixes, patches, or security vulnerabilities are disclosed, you can't simply copy and paste.  Each and every patch must be reviewed, tweaked, customized, and decided if it matches with what you've done.  You've created a maintenance nightmare that you can probably never get rid of and you'll have to support forever and ever Amen.  And if the project/company behind it dies...

So what about rolling your own?  Is that just as wrong?  Yes.

We built our own little blogging system for WhyGoSolo.  It started off pretty basic.  Every post had a title, a body, a set of tags, and a single rss feed.  In almost no time, ther requirements grew and we had to support "Publish At", then Archives, then attaching images.  While the functionality has stablized for now, it's created an interesting problem.

So we get all the difficulties fun of supporting a new platform and steadily adding the new functionality that comes standard with a regular blog.  So in this area we get to play catch up... but on the flip side, we get exactly the functionality we want and need and no more.

Generally, I'm pragmatic to the point of telling people "don't build it if it's not your core business".  I believe firmly that the vast majority of us don't have time to support something that isn't part of our business.  We have too many things to do and not enough time to do them.

If you have the opportunity to roll your own... reconsider it.

If you have the opportunity to use an off the shelf component... reconsider it.

Don't worry, more to follow..

All I can say is:  Wow.

As we headed up to the 05 January release, I expected it to be a huge month.  I knew that we'd have some late nights.  I knew we'd have some stressful coding sessions.  And I knew we'd get attention from people and places that we'd never heard of.  We made the release date.  Despite daily updates for the first week or so, we even got a major update out the door.  It was a stressful time, but we had lots of forward progress.  All in all, a good month with some stress but lots of successes.

And then February happened...

February started off slowly.  We released our Facebook Application.  Despite a couple bugs, it went successfully and is broadcasting your listings to all your friends.  After all, if you're interested in something, it's possible your friends might be.  Then we had a couple meetings with friends and allies including Justin Thorp from Clearspring to discuss our widget strategy.  Somewhere along the way, Ann made friends with Jeff Pulver and he wrote about us... and the dam broke.  In the last two weeks of February, we've gotten coverage from ReadWriteWeb, Mashable, Pulver.tv and more than I can publicly thank.

Yet another round of successes to be celebrated... but despite everything else, Ann and I are still perfectionists working with minimal resources.  The team is excellent and the ideas are numerous and somewhere along the way, the release slipped.  As COO/CTO I take responsibility for it and boiled it down to a handful of points:

• We dream big, and we push hard... sometimes too hard.  Prior to this iteration, we've been strict about only applying peer-reviewed fixes to any release branch.  This time we had a mini-release in between iterations.  It was bad form and we should not have done it.  From here until funding, we're doing smaller iterations to make them more managable and easier to deploy.
• We're still shorthanded.  Despite the prowess of our UI Ninja, she's been tied up on other things.  Therefore, we've been shaking the bushes to find another set of skillful hands and eyes to help us.  I went ahead and went public with our request for a User Interface Ninja.  If you think you have the skills and want to work with a great team, check it out.
• We've outgrown our tools.  We've been using Basecamp to manage everything and really gotten past the point where it's useful.  I've said before that Basecamp isn't a real Project Management tool - nothing personal guys, I still love it - but we've passed the point where it breaks down.  We're moving some of our things to Trac to... track and organize them better.
• This is our biggest release since launch.  Our previous couple releases centered around one new piece of functionality and a number of fixes.  This time, we had a pair of new  features (Twitter and Timezones) and an infrastructure upgrade (Rails 2).  In order to do it right, or testing had to be even more extensive than usual.  And our deployment/upgrade was tested repeatedly... just in case.  ;)

So, we have had some minor issues but this time it's not because we're screwing things up.  It's because we're at a new level working with the same resources.  We're getting more and more attention, writeups, mentions, and just general discussion coming more often and faster than we've had.  We have people actively seeking us out, we have investors starting to lurk around, and a number of groups in serious discussion.  When you see the person's eyes light up during a pitch...  wow.

And yes, I know this is the first week of March... 

You've been there.  You're getting close to release.  Your team is working crazy hours, they're patching bugs left and right, your documenation team is swearing at everyone for making changes,  your former Marine boss is spending a lot of time cleaning her weapons... and your PR people have one question for you:

What's in this release?

Oh... crap.  So how can you figure out what was done?  How can you figure out what's in this release?

First, hopefully you have a list of everything that went into this iteration.  We usually use the Categories in Mediawiki to keep the relevant User Stories grouped together.  With a click we can see the relevant User Stories without a problem.  For new functionality, this works well and we're able to keep things well-documented and straight.

Next, ask your team members.  If you've been doing daily standups or any sort of task reporting, they probably have a list of things they've done.  They will remember the particularly annoying/difficult things that they've worked on... and the things they're proud of.  Just ask.

Next, check your bug tracking system.  You have a bug tracking system right?  Or at least a todo list of everything that came up during this release, right?  Although we have Trac functional, the simplicity and speed of Basecamp still has it beat.  As the system and the team grows, we'll probably have to convert...

Finally, you can check your commit logs.  This is the most painful and tedious.  If you require useful commit comments, it will be less painful but will be incredibly time-consuming.  Regardless, it is the most complete way to find everything.

Unfortunately, this most recent time around, we had to do a bit of all four things.  During our day to day usage and via user feedback, we found a number of small things that simply didn't make it to the point of being logged.  Luckily, since each release is centered around 1-2 new things, it was easier than expected...

• Feature: New invites system!
• Fix: Publish Time now editable on blog post edit page
• Fix: Profile image thumbnails are viewable even if the user is non-public so that wall comments and friends lists do not have broken image links
• Improvement: Changed how the front page decides which five new users to view;  It used to be random among all users, and now it is a random five of the 100 most recent members with public profiles
• Fix: Future posts do not show up in the archives
• Improvement: Fixed navigation bar on group community listings page
• Fix: Handshake history improvements
• Fix: Minor miscellaneous HTML fixes

As I write this, we're solidifying the last few things for the next release:  Boone.  And the list is already looking pretty good... ;)

A long, long time ago before I started CaseySoftware, I didn't understand the point of an introduction.  I considered it irrelevant and not all that important to day to day business, goals, or even your personal/professional standing... then I figured out how wrong I was.

An introduction to the right person at the right time under the right circumstances can be the most powerful thing in the world.  It can make or break an opportunity.  It can completely change everything in no time...

A pair of books tipped me off to this... Art of the Start and Love is the Killer App.  It may sound silly, but between these two books, I figured out the point of an introduction has nothing to do with what you can get.  It has everything to do with what you can give...  If you introduce two people with complementary goals, interests, or situations, they will be the ones to benefit.  But before anyone can benefit, you need to make the introduction worthwhile to both sides.

 So what does it take to make a good introduction?

First, you need to know both people involved.  If you don't know both of them, this isn't an introduction, it's a cold call.  Don't waste your time doing a cold call for someone else, they can do it themselves.

Second, you need to vouch for both people.  By writing the introduction, you are lending your name and credibility.. now you need to tell them why.  I make a point of telling how I know each of the people, whether we've worked or collaborated on a project, and something else that might spark their interest.

Third, there needs to be a reason for the introduction and you have to state it quickly.  Look, most of us are busy and don't have the time for a lot of irrelevant email.  Get to the point and get to it quickly.  State what each of them stand to gain from the introduction.

And finally... don't have any expectations.  Even with a personal introduction, quite often one of them won't follow up.  The opportunity might fall through.  Their goals might not line up.  Or the match might not be as good as you hoped.  Don't expect a finder's fee and don't expect the favor to be returned.

So... what's the point?  What do you stand to gain from giving introductions?

In the short term, nothing.   Most of the introductions you pass along will end right there.  A few more will trade a few emails.  And a special few will result in business or a new connection.  While I haven't tracked it, anecdotally, I suspect this is about 10%.

In the long term, everything.  What is the value of helping friends and associates succeed?  What is the value of becoming known as the "connector" in your circles?  What is the value of being contacted first when new opportunities appear?  Honestly, I can't put a value on any of these... they're that big, they're that powerful, and they have the potential to be career and life-changing to you... and the people you help.  There's no way to measure that value.

The other day in "Site Triage and You", I talked about the three types of site issues and how we're addressing them.  But that doesn't cover how we're deploying them..

The core of our deployment strategy boils down to two principles:

Repeatability and The Buddy System

First, all of our code is in Subversion.  Every developer has full access to every line of code.  While there are a few areas which were predominantly written by a single developer, we've had strong coding standards and discussion in place - thanks to Nola - since the beginning.  When you know that at least one other person will look at your code... well, peer pressure can be one of the best things in the world.

Second, deployment happens the same way every time by one person.  We don't copy files up one by one or apply any changes manually.  Too many things can go wrong and all it takes is for someone to forget one command or file just one time.  I stop the server, perform an svn up, run any migrations, and restart.  All scripted, no room for error.  Also, since only one person can do deployment, there's no question to who did what.

Finally, no code goes into production without a peer review.  Every change - no matter who does it or how trivial - is reviewed and blessed by another developer.  Yes, since we don't have anyone on staff fulltime, it can slow things down. Yes, it can be a little tedious to always have to wait.  But I wouldn't change it for the world.  According to our logs, exactly two errors have been deployed since launch... despite deploying new code each and every day.

Despite these rules, we can still improve the process...

First, we can expand our RSpec tests.  In the final days leading to launch, our coverage dropped significantly.  We have quite a few tests that fail.  This needs to change.. and soon.

Second, we can scrub the codebase.  We've found a number of Magic Numbers and duplication that should be eliminated.  Duplicate code means duplicate bugs, duplicate effort, and duplicate stress and requires Refactoring.  Magic Numbers are much nastier as the introduce invisible dependencies.  A Magic Number was the cause of one of our introduced errors.

Finally, we need to expand the team.  Bringing another mid-level and another junior-level or two would be handy.  Not only can we get more done, but the trivial things can be handled while our senior developers move into bigger areas and lead by example.  Now that we're launched and funding is coming into play, we'll have some more options...

And like everything else around here, if you have tips or suggestions, I'm happy to hear them...

Or "Wait a second.. what's that burning smell!?"

Now that things have calmed down a bit, I'd like to give a bit of my perspective...

Whenever you launch a site, there are going to be problems.  Some are going to be tiny and only detectable by the handful of people actively working on it.  Others are going to be so large that you briefly consider giving up technology and living in a tree.  Regardless, they all need to be dealt with... so the questions become:  How and when?

In my book, there are three types of issues and each requires a different response:

First, there are tweaks.  These can be spelling errors, broken images, bad punctuation, or a variety of other things.  These are not functional things at all.  These probably aren't preventing people from using the site.  They're probably not breaking things.  More than anything, they just annoy your users (or your boss) and hurt the credibility of your site.

The best thing about these fixes is that you can apply them just about any time without issue.  You don't have to worry about the database or cached data or even users currently on the site.  You can deploy the changes as often as you want with few problems.

Second, there are errors.  These can be a variety of things... ranging from unexpected behavior to working exactly as expected but in a wrong way.  These tend to discourage people from using the site.  When your users see these, they say "wtf" or "why did it do that?"  For the record, those are both bad.  Odds are that the features are either getting/making bad data and blowing up or it's working exactly as designed and it's just wrong.  The second one is much worse.

These are a bit more problematic...  Most likely you need to both apply the fix and scrub any existing data or user sessions.  The worst part is that users currently in the system can get into an unstable state and you can't being to predict what will happen to them.  These fixes have to be scheduled and handled carefully... preferably while no one is using the site.  Odds are that you only get one shot at this a day or even once a week.

Finally, there are showstoppers.  These are huge.  These are the ones that make your phone ring at 2am and put people's jobs at risk.  It can be something simple like bad css that makes a page unusable or it can be something nasty like a security hole that lets anyone own your site.  Both are bad, but there are magnitudes of difference between the two.

These are huge and problematic and not something to play arond with... These have to be applied as soon as possible because bad things are happening.  Luckily, there's an upside.  Since the site is already broken, you have a perfect opportunity to fix it.  ;)

For WhyGoSolo, it's been an interesting mix of the above...

We've had literally dozens of tweaks and we have quite a few more.  Some we've managed to roll out, some are done but queued for later, others are later on the todo list, and some we've just dropped.

The errors...  we've had quite a few of those.  We've closed a handful each day and tend to deploy and validate before we send out a blast of invites.

Finally, the showstoppers... we've had two.  Both were resolved shortly after discovery, but they still hurt.

If you run into any issues, let us know - support@whygosolo.com - and we'll figure it out.  All feedback is appreciated.

Or how I learned to manage Rails development as a PHP guy.  ;)

When you're building a new system by yourself, you know everything that's there.  More importantly, you know what isn't there.  The organization of the entire system is in your head - or in my case the whiteboard next to me - and documentation and sharing information isn't that important.

Once you add a second person to the mix, things become fractured.  The information has to get shared between two people, documentation becomes more important, and each of you has different skills and weaknesses.  But there are two more important aspects...  you don't always know what they're doing and you steadily lose a complete understanding of the codebase.

As long as the code works, what does it matter?

Well... not quite.

As you lose a complete understanding and a mental inventory of what is and isn't in the code, you begin to run the risk of duplicating efforts.

For example, you create a nifty little function to format dates in a particular way.  And then someone else writes a similar function that is just a bit different.  Each of you use the similar but not identical code for different things in different places and eventually a bug is found.  Even worse, the bug appears in one place but no where else with the same behavior.  And I can guarantee it's just going to get worse...

When you're working on a system, you need to rigorously defend it... but it takes everyone being involved.  When someone makes a check in, someone else needs to take a look at it.  Ask the obvious questions, note concerns, and if something strikes you as odd, ask about it.

The entire point of this is to minimize the paths through the code.  Every variation you have, every line of code, and every different interpretation offers the chance for bugs or unexpected behavior (not always the same) to creep into the code.  Each one requires more work, more effort, and tends to annoy people.  The sooner and faster you can catch/prevent these, the happier everyone will be.

I'm not a Rails guy and you're probably not either...  so although we'd love to be on the front line on this one, we can't always be.  If you don't have all of the relevant skills to perform a code review, you can still facilitate and mediate the discussion whenever it occurs... or better yet, specifically plan on a regular discussion to address these issues and keep people moving in the same direction.

While no company - or worse a startup - wants to consider this aspect, since the point of WhyGoSolo is to learn and connect with other people, I'm sharing this warts and all.

No matter what kind of company you are - funded, bootstrapped, government, Google, mISV, or the dream job that all of your developers have been dreaming about for years... Burnout Happens.

Sometimes it happens quickly.  If the work is boring, uninspired, or conditions/pay are bad, you can guarantee it happening quickly and without warning.  You may see this in horrible retention rates, the inability to recruit new people, or simple dissatisfaction within the team.  For example, a team I was with a few years ago suffered from all of the above.  I left in January of that year.  By mid-November, 7 of the 9 remaining team members had left... and none of the local recruiting companies were willing to work with them.

Sometimes it happens more slowly.  When the work is interesting, new problems are being solved, and/or people believe in the cause, it will take longer and there will be signs.  You'll see this in people being less productive, unable to focus, and making obvious mistakes.  Their heart just isn't in it.  Most often this is because of exhaustion and the desire/need 

So why is this relevant to WhyGoSolo?

For the past month, we've been pushing hard.  We've been pushing amazingly hard from top to bottom.  From Ann giving pitches and demos to the Dev team working on and through lots of code and tests to both Ann and I stradling both worlds.  To be honest, it's tiring... and if there's any time one of us will break and get burned out, this is the time.

Even worse, while a side project is just getting started - yes, WhyGoSolo is a side project for many of us - it's not work.  It's fun, it's new, it's a diversion from the difficult stuff.  But once you get towards launch, all those tedious things still need to be done... so most people want to disengage and look for something "fun" again.  And just to add insult to injury, no less than three of our team members have had major releases in their day jobs in the past few weeks.  Not an ideal environment at all.

So what are we doing to combat this?

1. Prioritize - We've broken the open issues (bugs, tweaks, and features) list down to a simple and straightforward hitlist of things that need to be addressed before the Demo.  Prior to the last couple days, we've had the big list unfiltered and it has caused confusion once or twice.  We don't plan on letting this happen again.
2. Drafting - No, we're not dragging people off the street, instead the team has been following the cycling model of Drafting.  It means that different people take the lead at different times, often to preserve the strength of the other riders.  In our case, as different people's dayjob schedules have been more/less demanding, different people have taken the lead at different times.
3. Milestones - Yes, we've already given one demo last week, but our big one is coming up this week at the Social Times DC Launch Event.  Having a finish line milestone that everyone can sink their teeth into and see something result is fundamental to making it.  Everyone can know that it won't go on forever.  That it can and will end.
4. Credit - While almost every team has a person or two who will push themselves beyond the point of exhaustion just to do it right with or without credit, the other 95% of the team will probably respond positively to encouragement.  It doesn't take much to identify the individuals who have made good contributions or have helped others do the same.  In the past couple weeks, a number of people have simply shined in their involvement.  I took time from our most recent call to identify them and plan to continue the practice indefinitely.
5. A Break - Once we get through this one, we're taking some time off to relax, regroup, and take some deep breaths.  At some companies, they give people a week of vacation time or send everyone out for drinks.  Since we're virtual, we can't do the exact same thing, but we may be able to come up with something.

And don't get me wrong.  We're not perfect and don't pretend to be.  We're working to solve the issues that come up externally or from the top down.  As the guy in charge, those are the ones I'm best situated and able to address but they're only a portion of what happens throughout our days.  Every team has issues that come from elsewhere and these fit into the burnout and exhaustion equation just as much if not more than the other things.

That's one of the joys of a 100% distributed team... we can't go into the next room and talk over coffee or read body language to understand or detect things immediately.  Instead, it takes reaching out to team members individually, regularly, and repeatedly to find out what's on their mind.  You have to build up trust and respect within the team and with the team to begin to address those...