Home » 2013 » March


One of the things I need to keep in mind as a lead is that the list of things I need to know in order to be effective has grown. It’s not just the details of development, design and architecture, along with a baseline knowledge of requirements gathering, project management and QA, but also things like business strategy, budgets and, as proved important today, vendor agreements.

We’ve been looking at changing the web hosting provider for our new site since a little before I came on board. When I joined, it was among my first tasks to compare our current provider with our potential new provider — to assess value of the services for the price.

Part of what I did was review the contracts — in particular, the uptime and support guarantees, and made note of what each was offering. (As the company has tens of millions of dollars a year in sales, every hour of downtime is a potential loss of thousands of dollars — during prime time, tens of thousands.) And by making note of these facts, I was able to recoup a significant chunk of change for the company today.

As it happens, this morning we found the staging site — the one hosted by our current provider — was down. It was down because the load balancer*, which is entirely in the provider’s control, was failing. This halted much of our development. Amazingly, this premium web hosting provider to which we pay several thousand dollars a month was unable to have the site back online by the end of the day, and we since learned it was down much of the weekend before that.

Much frustration ensued, including a push to switch us over to the new provider ASAP. But towards the end of the day, I mentioned to the CEO this little detail of the contract with the provider: For every hour of downtime, we are entitled to be credited 5% of our monthly contract fees. It’s also important to note that the contract does not specify that this credit caps at 100%.

The dollar signs appeared over the CEO’s eyes when he brought up the contract and confirmed that information. Gleefully he asked me to put in a request with that provider for a refund — and forwarded me the pertinent clause of the contract.

I just earned my pay for the month, I think. And I also earned a strong measure of the confidence of my CEO, which is very important to achieving my goals for the company. And I also had an important lesson of being a lead reinforced. I’m going to remember the importance of keeping apprised of the details of many things beyond the development process itself.

*If you don’t know what a load balancer is, it’s a server at the front end of the web server cluster that routes incoming traffic evenly between the web servers so that the servers handle roughly equivalent “load” of that traffic. Basically, if it stops working, your entire site fails.


A story from the past

At a company someone I know worked at, he was part of the dev team. The dev team had a great culture. They were interested in their own success, the success of the team, and the success of other teams within the company. They cared about doing their jobs right.

At this same company, there was an IT team. This IT team was not particularly invested in the success the company, and didn’t show any real signs about being concerned with the quality of their own output. They were often obstacles, rather than partners, in the dev’s team efforts to do good work, and they displayed attitudes of aloofness, indifference, and self-interest.

In a nutshell, IT and development had two distinct and very different cultures.

In many companies, IT and development have some crossover, as some development projects have IT impacts and vice versa. This company was no exception. So the dev team often found themselves grumbling amongst each other at IT’s obduracy.

Dev was hiring, and found a guy who impressed everyone, who had a skillset that Dev really needed, but who didn’t have the skillset for the position for which Dev was hiring. But because of this guy’s skills and his great attitude, the dev team really wanted this guy on board, so they created a position for him. Because this position was one that really had a foot in both IT and development, the company decided to place him in IT.

It took about two weeks for this guy, about whom the entire Dev team was really excited, to be co-opted into IT’s culture. He became self-important, lost his sense of collaboration, quality, or follow-through, and became a highly skilled yet marginally useful employee.

And this is the lesson to be learned from this experience: If the cause of a lack of quality output from a team is due to an institutionalized problem with the culture of the team or the company, trying to improve output by adding people to the team won’t work, because those people will be absorbed into the same culture of failure. It is the culture itself that must be changed to turn the team around.

And that can be one of the hardest things in the world to do because it requires the humility to realize that the problem is due to one’s own poor choices. And it requires the majority, or even all, of the team to accept that and resolve to change together.

And inertia is an awfully hard thing to overcome.

The stake in the ground

Today was the first day where I didn’t leave work frustrated with how difficult it is to learn the things I need to know to do my job. This was a very good thing.

What’s a bad thing is the principal reason why it’s been such a slog for me to date.

The platform I’m currently responsible for getting production-ready was originally built with very few specifications. The specifications were, mostly, “Make it do what the old site did.”

There were not one, but two major problems with this, both of which are fairly obvious.

The first was that the old platform’s processes weren’t documented either. Knowledge of how to use many of them were scattered throughout the company’s employees and management, of course, but with no BA, turning that knowledge into requirements is not an easy task. Other process knowledge left the company with old employees, and needs to be rediscovered.

The second problem is a little less obvious than the first, but still readily apparent: Just because the old site does something doesn’t mean the new site needs to do the same thing. Any new development should be with the goal of meeting business needs, not matching existing processes. Often, old processes are obsolete, or are workarounds for things that didn’t work exactly the right way on the old platform. If your old e-commerce site handles conversion of Canadian to American currency but you’ve since stopped selling to Canada, you don’t need that functionality any more. If your old administration console had a process that exported all orders to an XML file that a user transferred to the fulfillment service’s import function, it’s not necessary to build that export function in the new platform if it can instead export directly to the fulfillment service.

TL’s Fifth Rule of Development: When designing an upgrade to an application or service, the question to ask is not, “What does it do now?” The question to ask is, “What are the business needs to be met?”

So what was built was taking a powerful new platform, and wedging the old platform’s processes onto it, without really examining what the needs are nor understanding the new and possibly better ways the new platform can meet them.

Fortunately a few of the more onerous features haven’t been implemented yet, and I’ve been given the opportunity to put the brakes on what they’ve been doing and ask those questions: What do we need it to do? Can it be done differently? These questions have already yielded results that are going to allow me to meet the company’s needs far more cheaply, and far more simply, than otherwise would have.

And better still, the powers that be at the company let me do just that.

And that is my first major win at my new job.

Tapped out

The week ended with exhaustion and a feeling I was in over my head.

Not because I thought myself incapable, but because the demands placed upon me, Pope and BA/PM to get this project out the door were enormous and I’ve barely scratched the surface of the technology in which I need to become an expert.

The schedule was put together by people who are not developers, based, it seems, solely on the date they decided they need to go live rather than a sound estimate of what the minimum list of tasks is and the time it will reasonably take to complete them.

This is what we call “Schedule-driven development” and it never works. Why?

TL’s Fourth Rule of Development: Time, Cost, Quality. The business owner of the project gets to dictate two at most. Trying to dictate all three will set the stage for failure.

It’s an old rule which I’m sure anyone reading this has previously heard, but it bears repeating because business owners keep breaking the rule, in organization after organization, project after project. Sometimes the same business owner does this repeatedly, wondering why his projects keep crashing and burning.

But that’s the paradigm I find myself facing. There is a set of minimum marketable requirements, a schedule to get them done, and a specified number of resources to accomplish the task.

I knew there would be difficulty in changing the development culture of this company, but I didn’t know the immediate task would be so urgent and with such a broken structure. And I know I can’t begin the process of changing the company’s culture until this immediate task is complete. And my chances of changing the development culture hinge on how well I complete the task.

The good, however, is that both Pope and BA/PM are right there with me, they see exactly the same problems I do, and thus we have a united front in dealing with this challenge, and managing the business leadership when deadlines inevitably slip.

The fact that Pope is part of senior leadership is one reason why we’ve got a chance to make this plan work. Without him, I’d think we were tilting at windmills, at best.

The fire hose

As I get to know my new coworkers I’m starting to assign them nicknames in this blog. The papal conclave that elected a new Pope today inspired me, for a few reasons, to nickname my boss, previously referred to as CTO. He is henceforth “Pope”.

So Pope and I the last two days have been working on bringing me up to speed, and I have surely been “drinking from the fire hose.” I’m mentally exhausted.

Today became a different kind of trial, however, as a critical issue with the current production servers (directly impacting company revenue) consumed much of our time.

I found myself doing something I really never like to do but have many times found myself needed to do: dive into a complicated, poorly-documented set of running processes and try to figure out what the problem is. Pope and I tag-teamed it, with his understanding of processes and my understanding of code, with occasional calls to business users to figure things out. Eventually we figured out that this was a problem with our network, which had recently been reconfigured due to the opening of our new branch office (the one from which I’m working). Essentially, the internal firewall got misconfigured and blocked the processes from communicating with outside processes.

Which tells me the current version of our platform does something very badly: It doesn’t report when things aren’t working right. It “fails silently.”

TL’s Third Rule of Development: Never architect nor write your code under the assumption that everything will work as planned. Always develop with the exception cases in mind.

That rule goes double for any code that talks to something external to itself (a database, an API, a file import or export). Because those always seem to break at the most surprising times.

Fortunately most of this code is going to be retired when the new platform is release. Unfortunately, that’s no guarantee that the new code handles the exception cases appropriately, and a review is indicated.

Other fun things I discovered. A cron job runs every hour, on the hour. This cron job tries to execute two scripts that don’t exist. As far as I can tell, they haven’t existed for over a year because there are web server logs that show the same errors every hour going back that far.

This concerns me greatly because it, obviously, means there has been little to no quality assurance on the processes or the code to date. And I’m obviously worried that the new development has taken place under the same (lack of) constraints.

My solution is to lay the groundwork for a near-term plan to do a full review of the new code base. I know that CEO’s top priority is getting our new platform out the door as quickly as possible, as the E-commerce guy has all kinds of things he wants to do that are dependent on the new platform being live. So that’s first priority. But once that’s done, I want to do that full review and identify potential problem areas. I introduced CEO to the concept of ‘technical debt’ and told him what my post-release goals were in the context of ‘paying down’ the technical debt.

Mostly, I don’t want to be trying, yet again, to figure out how to build stable code on a brittle foundation.

The other item of note today was that CEO was eager to plan to build my team. I told him I had several people in mind, but that I needed a couple weeks to really get to understand the company’s needs before I made any recommendations. Pope was quick to back me up on that, saying he wanted my focus right now to be getting acquainted with the new applications.

But this is a good thing for my desire to build myself and my company a great development team.

Day One

So I walked into the office like I owned the place.

Well, no. I walked in a bit nervously. BA/PM was there (she had a week head start on me) as was CEO. CTO lives out of town and was to arrive in the early afternoon.

The first task is to complete, post-haste, the new E-commerce platform for the company’s primary website. It was, in violation of Rule One, constructed without a clear requirements document — the direction was to ‘make it work like the old one’.

This left a bit of a mess, and coupled with a couple key developers rolling off the project unfortunately quickly (not, to my understanding, for ugly reasons) the mess got larger.

In the interim, outside contractors were brought in to complete work on it, but significant challenges to complete the project remain — most notably aspects of sharing some data with the order fulfillment system. And it’ll be on me to complete development as I’m, for now, the only developer.

I met a few other key players, including the new designer (also starting today) and the E-Commerce guru (who needs this project done to implement some of his plans). We had lunch together.

A note on having lunch together: In an earlier entry, I wrote that “Never have lunch alone” was one of Sparky’s key pieces of advice. The idea is that if you’re lunching with your co-workers, you’re building rapport and bonding. As I have learned from my many years of employment, as a developer and otherwise, whether you’re having lunch with your direct reports, your peers, or your superiors, this has great value in the near and long terms.

Also, when your CEO goes to lunch with you, he usually buys. So, free lunch.

After lunch CTO arrived and we got immediately to work. He gave me and BA/PM the breakdown of the project and the major things to deal with, and then immediately pivoted to a discussion of the deployment architecture, which is currently slated to be very expensive (as in, thousands per month). CTO and I started exploring alternative architectures, and then immediately he started contacting the vendor for bids based on the alternatives, with me riding shotgun, and principally giving him nonverbal cues in reaction to what the vendor was saying.

One of the architectural details pushed by the vendor was a file server. Neither CTO nor I saw a separate file server as necessary for our architecture. The vendor (who, it should be noted, is known for their expertise in our new E-commerce platform) insisted it was a critical and necessary part of the architecture. One glance exchanged between the two of us confirmed that we believed the vendor was full of it. When CTO pushed back, the vendor said he’d talk to someone more technical about it to make sure it would be all right for us.

Tomorrow CTO and I are heading to be on-site with another vendor – the development vendor – to begin the lengthy process of knowledge transfer into my brain.

I expect to be very tired for the next few days. But if I take good notes and keep focused I should be in a much better position to understand the company’s technical needs, and begin to build my team accordingly.

However, it sounds like that will not be my first concern. First, I need to get this platform ready to release.


Very fond farewells

Yesterday was my final day at the old company. Wrapping up and taking care of the details, packing up my stuff, a couple battles with Nerf darts, having both HR and Exidor offically remind me of my non-disclosure/non-compete agreements (really? twice?), turning in my access card, and I departed…

…to a nearby bar where New Kid and I were the erstwhile guests of honor at an informal gathering, where many of our now former coworkers came to wish us well at our respective new positions. Old Company alumni Red and Smiley also showed up, to the warm welcome of all in attendance. Phoenix and Kabob from my team were there, as were Sparky and Cupcake, Cupcake’s team, both the QA guys, and lots and lots of people from other teams. It was fantastic.

From a professional perspective (after all, that’s the focus of this blog) one thing for which I’m very grateful is that I’ve developed these great relationships with Red, Sparky, Smiley and Cupcake, each of whom has warmly offered to be a resource for advice, counsel and ideas moving forward. And as Cupcake is a lead whose soft skills I greatly respect, and the other three all serve in the same kind of management/technical direction role I’m about to enter, I know their support will be invaluable in the days to come.

From a personal perspective, I’m very glad to have formed valuable friendships with everyone in the development group. Parting company with great coworkers who are great people is a very difficult thing for me to do.

But this is also an important point for me to remember: The team that I build going forward can be staffed with the most technically talented people in the state, but if they’re not people that I can form strong relationships with, nor people that can form strong relationships with each other, it’s not going to work. At the old company, the hiring process sussed out whether the candidate was a cultural fit as much as a technical fit. We would pass on people who were technically strong but whose personalities weren’t right for the team, and we would be more willing to bring on someone who was technically weaker but a strong cultural fit. In those instances where we hired despite having concerns about cultural fit, we always came to regret it.

So when I start building my team, I need to understand the culture into which I’m bringing them. And though I believe I know intuitively the kind of person I personally get along with, it wouldn’t hurt at all to spend some time thinking about — even writing down — the character and personality traits I want to see in the people with whom I work.

Eating the elephant

Had a talk with the contractor who’s been responsible for the development of the websites for which I’m going to be responsible starting Monday. Websites that will be going live Any Week Now.

Productive discussion and a brief crash course in the software upon which these sites are built.

But what I also learned is some not wholly unexpected but difficult facts about the environment into which I’m headed. And it’s an environment pretty familiar to me — in many respects, it’s very much like the environment of my current job was the day I arrived.

One of the things I’ve learned through multiple jobs in my career is that most startup companies which have a core web services platform have a point I call Development Adolescence, a period analogous to true adolescence where one transitions from carefree childhood to responsible adulthood. And that transition is a period of difficulty, awkwardness, and often self-created pain and angst.

Development childhood is the period where a new company is initially creating and growing its web services. It needs to grow fast, and it doesn’t usually have a very detailed idea of what it needs its web services to be. So the web services tend to grow organically, rapidly, without a very clear picture of its destination. There are either no clearly defined business owners of the platform, meaning too many people drive the business process of its development and not enough people (outside the developer or developers themselves) have any accountability for that development. The application grows, it becomes harder to maintain because the code has no overall structure for its growth, requirements documentation is sparse to nonexistent and process flow documentation doubly so. “Technical debt”, meaning structural defects in the application that slow new development or require time invested to maintain the existing services, builds up. Developers start fearing the most onerous parts of the codebase and know that the entire system is brittle and can break in unexpected and non-obvious ways by any modification.

The tipping point is reached when either business needs outgrow the capacity of the existing platform to meet them, or the accumulated technical debt renders the platform so unstable or development so painful that the dev team cries out for relief. The business becomes aware that the current path is unsustainable, and that the development process must mature. And thus begins Development Adolescence.

Adolescence is the process of the development team leadership (by which I mean me) establishing a sustainable development framework for both stable growth in the web services platform and for controlling and prioritizing the flow of new change requests coming for the business. This process usually takes place in the context of either a major overhaul or wholesale replacement of the web services platform, transforming it into something that has a much better chance of serving the needs of the business in the medium to long term.

This isn’t awkward in and of itself, but the awkwardness ensues when the business, which has reached the point of understanding that the development process needs to greatly change, doesn’t yet understand that the business process must change too. When adolescence is reached, the business has become accustomed to how they do things — usually, they tell Dev to do something and Dev does it. There is no gatekeeper who prioritizes task, no real requirements gathering, no business owner to agree to the specific changelist, no placing the requirements in the context of the long-term business goals, nothing. And they are resistant to changing how they do things, because what they’ve done so far has worked, for them.

And so it is incumbent upon the development team leadership, along with the senior management responsible for the dev team (usually, the CTO/CIO), to bring the business to understand and buy into the value of changing their processes to work with the new business processes. And this is no easy task. Myself, I favor a top-down approach: Work with the CTO/CIO to get buy in from the CEO and/or the rest of the senior leadership team of the company, who then champion the business process changes with the rest of the business, who in turn adapt to change, slowly, step by step.

The problem with that path is that there’s always the chance you don’t get buy-in from the CEO/SLT. And that makes adolescence far more difficult to get through.

The other viable path is to try to convince the business more holistically by putting processes in place step by step and showing the business the benefits that are reaped by demonstration. This is an even slower, more incremental, far more painful process to adulthood.

Whichever path is taken, the question of how the business will emerge from adolescence is largely a question of how well the business adapts to change (er, okay, that and how well the development team imposes the process upon itself and how closely it adheres to that process, but I’m talking about the business here). And I’ve personally seen this go well, and go poorly, in my own work history.

The company I’m leaving is one where it’s gone poorly. The business adapted to some of the processes, but did a very bad job of bearing the pain of transition and began to slip into the mode of “panic-driven-development”, where the dev group was repeatedly directed to redirect its entire effort to whichever issue was causing the business the most pain at the moment. This results in lots of projects being left half-done, “to be finished later”, as the new greatest pain point becomes the new focus. Over. And Over. And Over Again.

TL’s Second Rule Of Development: Change to development processes is painful. If you’re going to commit to the change, commit to bear the pain, or you will fail.

I don’t know how well my new company will adjust to the change, nor how resistant it will be to changing at all. But I am fairly competent that the CTO, the new BA and I are all of one mind that we need to bring change to the company to usher it through its adolescence. And it will require an enormous, sustained, united effort to do this. It is a monumental task.

But how do you eat an elephant? One bite at a time.


First, a side note: There was massive snowfall yesterday and this morning. The drive to the central office of my new employer is about 40 miles from home. It was orientation, so I felt it important to go anyway. This was a mistake. I came very close to destroying my car and possibly myself via collision with snowplows, and only a well-timed swerve when braking proved impossible kept me from being a snowplow sandwich. It is NOT worth braving dangerous driving weather, even for a very important work meeting.

That said, I did go to the new job, and I did get some orientation. I took notes. Lots of notes.

I got to meet the CEO face to face. He’s a younger guy, five or so years younger than I am, but very ambitious and very likeable. He also doesn’t try to hide it when things aren’t being done as well as they could, and he listens to criticism.

I got to meet a couple other new hires. One’s in IT and seems to be a likely ally, which I will need, as the laptop they assigned me is locked down and I can’t install my own software on it without IT administrator access. This is not an acceptable situation to any self-respecting developer. The new IT person, whom I’ll call 80’s Hair, understood and appreciated this.

The other new hire is the BA-slash-PM-slash-QA expert. She’s going to be wearing three hats, which I suppose is tenable for now, but that role needs to be broken apart. She and I are working on building a good rapport, as we’ll be working closely together in the months to come.

It was also “monthly meeting” day, which meant a group lunch. This was a chili cookoff. And me without my special homemade chili. (I’m quite a foodie, I admit.) But the chili I sampled was plenty tasty and I had time to get to meet more of the people in the main office.

But the big thing today was what we term ‘drinking from the fire hose.’ The concept is being flooded with more information than I can possibly absorb at once, and all I can do is try to take in all I can.

This particular firehose was learning about business processes, specifically order fulfillment and customer service. The former is the basic result of the e-commerce sites which I’ll be responsible for; the latter is the basic conduit of feedback from the customers. The OF guy spent about 90 minutes illustrating his process to us. He showed us how it began with the output from the existing e-commerce site, printing out packing slips, distributing them into bins, filling the bins from inventory, bringing them to the packing area, and packing the orders in bubble envelopes or boxes, labeling them and marking them in the e-commerce site as shipped. I learned about the basic pain points, the basic points of failure, how those points are caught, and the time invested in the process of catching them.

On the other end, the customer service manager and workers had much input from their customers, and that information doesn’t seem to have a distinct process for getting back to development to be incorporated into future development requirements. Both the new BA and I are keenly interested in remedying that situation.

After that, I was exhausted and packed up my new work-assigned laptop for home on the (now much better plowed) long road. I was requested by the CTO to call him to go over some stuff. So, I did. I was tired, but CTO and I talked and planned out the next couple weeks. I also confirmed with him what earlier conversations with the CEO and document review made me suspect: The nearly-complete current development of the core website was done without a clear requirements document — which made its development go badly up until now. CTO’s high priority for when he is in town the next two weeks, and therefore my high priority for the next two weeks, is to define those requirements tightly. Once that’s done, BA can develop a test plan, and I can evaluate the current state of the project against those requirements, and identify the difference between the two. From there I can plan out development and estimate a time to completion — and then release.

Once that’s done then CTO and I can ensure that all future development will be done against a clear set of requirements, and a business owner of those requirements.

TL’s First Rule Of Development: Without clearly defined and accepted requirements, there is no effective way to know when development is complete, nor even to know what exactly needs to be developed.

Sure with vague requirements like “Make it work like that thing worked” you can get a decent idea of what you’re trying to develop, but it’ll never be clear, and whoever the business owner is will be able to push scope creep into your process with nigh-impunity — other than crashing your sprints and breaking your deadline commitments.

A gathering

My former project manager, Smiley, hosted a gathering of mostly current and former employees of my company. Sparky, Red, Phoenix, Kabob and Harley were there — Sparky, Phoenix and Kabob being current along with myself; Red, Smiley and Harley being the ex-employees.

The conversations that were had are not mine to detail — what happens at Smiley’s stays at Smiley’s. But for me, the content of the conversations, including my personal detailing of some of the reasons for my departure and my very candid advice to my team going forward, was part of my process of letting go.

One thing I do when I find a place I like to work is bond with people. And where I work now is no exception. And it’s hard to let go. But this gathering (which was coincidental to my departure) was a good opportunity for me to take a step down that process.

I wonder what the new company will bring me — if I’ll find the people I can bond with, if I find something there worth bonding to.

T minus eight days until I can start finding out.

Well, not counting Tuesday, when I take the day off from Old Job and head to New Job orientation.