Home » Posts tagged "rules of development"

The stake in the ground

Today was the first day where I didn’t leave work frustrated with how difficult it is to learn the things I need to know to do my job. This was a very good thing.

What’s a bad thing is the principal reason why it’s been such a slog for me to date.

The platform I’m currently responsible for getting production-ready was originally built with very few specifications. The specifications were, mostly, “Make it do what the old site did.”

There were not one, but two major problems with this, both of which are fairly obvious.

The first was that the old platform’s processes weren’t documented either. Knowledge of how to use many of them were scattered throughout the company’s employees and management, of course, but with no BA, turning that knowledge into requirements is not an easy task. Other process knowledge left the company with old employees, and needs to be rediscovered.

The second problem is a little less obvious than the first, but still readily apparent: Just because the old site does something doesn’t mean the new site needs to do the same thing. Any new development should be with the goal of meeting business needs, not matching existing processes. Often, old processes are obsolete, or are workarounds for things that didn’t work exactly the right way on the old platform. If your old e-commerce site handles conversion of Canadian to American currency but you’ve since stopped selling to Canada, you don’t need that functionality any more. If your old administration console had a process that exported all orders to an XML file that a user transferred to the fulfillment service’s import function, it’s not necessary to build that export function in the new platform if it can instead export directly to the fulfillment service.

TL’s Fifth Rule of Development: When designing an upgrade to an application or service, the question to ask is not, “What does it do now?” The question to ask is, “What are the business needs to be met?”

So what was built was taking a powerful new platform, and wedging the old platform’s processes onto it, without really examining what the needs are nor understanding the new and possibly better ways the new platform can meet them.

Fortunately a few of the more onerous features haven’t been implemented yet, and I’ve been given the opportunity to put the brakes on what they’ve been doing and ask those questions: What do we need it to do? Can it be done differently? These questions have already yielded results that are going to allow me to meet the company’s needs far more cheaply, and far more simply, than otherwise would have.

And better still, the powers that be at the company let me do just that.

And that is my first major win at my new job.

Tapped out

The week ended with exhaustion and a feeling I was in over my head.

Not because I thought myself incapable, but because the demands placed upon me, Pope and BA/PM to get this project out the door were enormous and I’ve barely scratched the surface of the technology in which I need to become an expert.

The schedule was put together by people who are not developers, based, it seems, solely on the date they decided they need to go live rather than a sound estimate of what the minimum list of tasks is and the time it will reasonably take to complete them.

This is what we call “Schedule-driven development” and it never works. Why?

TL’s Fourth Rule of Development: Time, Cost, Quality. The business owner of the project gets to dictate two at most. Trying to dictate all three will set the stage for failure.

It’s an old rule which I’m sure anyone reading this has previously heard, but it bears repeating because business owners keep breaking the rule, in organization after organization, project after project. Sometimes the same business owner does this repeatedly, wondering why his projects keep crashing and burning.

But that’s the paradigm I find myself facing. There is a set of minimum marketable requirements, a schedule to get them done, and a specified number of resources to accomplish the task.

I knew there would be difficulty in changing the development culture of this company, but I didn’t know the immediate task would be so urgent and with such a broken structure. And I know I can’t begin the process of changing the company’s culture until this immediate task is complete. And my chances of changing the development culture hinge on how well I complete the task.

The good, however, is that both Pope and BA/PM are right there with me, they see exactly the same problems I do, and thus we have a united front in dealing with this challenge, and managing the business leadership when deadlines inevitably slip.

The fact that Pope is part of senior leadership is one reason why we’ve got a chance to make this plan work. Without him, I’d think we were tilting at windmills, at best.

The fire hose

As I get to know my new coworkers I’m starting to assign them nicknames in this blog. The papal conclave that elected a new Pope today inspired me, for a few reasons, to nickname my boss, previously referred to as CTO. He is henceforth “Pope”.

So Pope and I the last two days have been working on bringing me up to speed, and I have surely been “drinking from the fire hose.” I’m mentally exhausted.

Today became a different kind of trial, however, as a critical issue with the current production servers (directly impacting company revenue) consumed much of our time.

I found myself doing something I really never like to do but have many times found myself needed to do: dive into a complicated, poorly-documented set of running processes and try to figure out what the problem is. Pope and I tag-teamed it, with his understanding of processes and my understanding of code, with occasional calls to business users to figure things out. Eventually we figured out that this was a problem with our network, which had recently been reconfigured due to the opening of our new branch office (the one from which I’m working). Essentially, the internal firewall got misconfigured and blocked the processes from communicating with outside processes.

Which tells me the current version of our platform does something very badly: It doesn’t report when things aren’t working right. It “fails silently.”

TL’s Third Rule of Development: Never architect nor write your code under the assumption that everything will work as planned. Always develop with the exception cases in mind.

That rule goes double for any code that talks to something external to itself (a database, an API, a file import or export). Because those always seem to break at the most surprising times.

Fortunately most of this code is going to be retired when the new platform is release. Unfortunately, that’s no guarantee that the new code handles the exception cases appropriately, and a review is indicated.

Other fun things I discovered. A cron job runs every hour, on the hour. This cron job tries to execute two scripts that don’t exist. As far as I can tell, they haven’t existed for over a year because there are web server logs that show the same errors every hour going back that far.

This concerns me greatly because it, obviously, means there has been little to no quality assurance on the processes or the code to date. And I’m obviously worried that the new development has taken place under the same (lack of) constraints.

My solution is to lay the groundwork for a near-term plan to do a full review of the new code base. I know that CEO’s top priority is getting our new platform out the door as quickly as possible, as the E-commerce guy has all kinds of things he wants to do that are dependent on the new platform being live. So that’s first priority. But once that’s done, I want to do that full review and identify potential problem areas. I introduced CEO to the concept of ‘technical debt’ and told him what my post-release goals were in the context of ‘paying down’ the technical debt.

Mostly, I don’t want to be trying, yet again, to figure out how to build stable code on a brittle foundation.

The other item of note today was that CEO was eager to plan to build my team. I told him I had several people in mind, but that I needed a couple weeks to really get to understand the company’s needs before I made any recommendations. Pope was quick to back me up on that, saying he wanted my focus right now to be getting acquainted with the new applications.

But this is a good thing for my desire to build myself and my company a great development team.

Eating the elephant

Had a talk with the contractor who’s been responsible for the development of the websites for which I’m going to be responsible starting Monday. Websites that will be going live Any Week Now.

Productive discussion and a brief crash course in the software upon which these sites are built.

But what I also learned is some not wholly unexpected but difficult facts about the environment into which I’m headed. And it’s an environment pretty familiar to me — in many respects, it’s very much like the environment of my current job was the day I arrived.

One of the things I’ve learned through multiple jobs in my career is that most startup companies which have a core web services platform have a point I call Development Adolescence, a period analogous to true adolescence where one transitions from carefree childhood to responsible adulthood. And that transition is a period of difficulty, awkwardness, and often self-created pain and angst.

Development childhood is the period where a new company is initially creating and growing its web services. It needs to grow fast, and it doesn’t usually have a very detailed idea of what it needs its web services to be. So the web services tend to grow organically, rapidly, without a very clear picture of its destination. There are either no clearly defined business owners of the platform, meaning too many people drive the business process of its development and not enough people (outside the developer or developers themselves) have any accountability for that development. The application grows, it becomes harder to maintain because the code has no overall structure for its growth, requirements documentation is sparse to nonexistent and process flow documentation doubly so. “Technical debt”, meaning structural defects in the application that slow new development or require time invested to maintain the existing services, builds up. Developers start fearing the most onerous parts of the codebase and know that the entire system is brittle and can break in unexpected and non-obvious ways by any modification.

The tipping point is reached when either business needs outgrow the capacity of the existing platform to meet them, or the accumulated technical debt renders the platform so unstable or development so painful that the dev team cries out for relief. The business becomes aware that the current path is unsustainable, and that the development process must mature. And thus begins Development Adolescence.

Adolescence is the process of the development team leadership (by which I mean me) establishing a sustainable development framework for both stable growth in the web services platform and for controlling and prioritizing the flow of new change requests coming for the business. This process usually takes place in the context of either a major overhaul or wholesale replacement of the web services platform, transforming it into something that has a much better chance of serving the needs of the business in the medium to long term.

This isn’t awkward in and of itself, but the awkwardness ensues when the business, which has reached the point of understanding that the development process needs to greatly change, doesn’t yet understand that the business process must change too. When adolescence is reached, the business has become accustomed to how they do things — usually, they tell Dev to do something and Dev does it. There is no gatekeeper who prioritizes task, no real requirements gathering, no business owner to agree to the specific changelist, no placing the requirements in the context of the long-term business goals, nothing. And they are resistant to changing how they do things, because what they’ve done so far has worked, for them.

And so it is incumbent upon the development team leadership, along with the senior management responsible for the dev team (usually, the CTO/CIO), to bring the business to understand and buy into the value of changing their processes to work with the new business processes. And this is no easy task. Myself, I favor a top-down approach: Work with the CTO/CIO to get buy in from the CEO and/or the rest of the senior leadership team of the company, who then champion the business process changes with the rest of the business, who in turn adapt to change, slowly, step by step.

The problem with that path is that there’s always the chance you don’t get buy-in from the CEO/SLT. And that makes adolescence far more difficult to get through.

The other viable path is to try to convince the business more holistically by putting processes in place step by step and showing the business the benefits that are reaped by demonstration. This is an even slower, more incremental, far more painful process to adulthood.

Whichever path is taken, the question of how the business will emerge from adolescence is largely a question of how well the business adapts to change (er, okay, that and how well the development team imposes the process upon itself and how closely it adheres to that process, but I’m talking about the business here). And I’ve personally seen this go well, and go poorly, in my own work history.

The company I’m leaving is one where it’s gone poorly. The business adapted to some of the processes, but did a very bad job of bearing the pain of transition and began to slip into the mode of “panic-driven-development”, where the dev group was repeatedly directed to redirect its entire effort to whichever issue was causing the business the most pain at the moment. This results in lots of projects being left half-done, “to be finished later”, as the new greatest pain point becomes the new focus. Over. And Over. And Over Again.

TL’s Second Rule Of Development: Change to development processes is painful. If you’re going to commit to the change, commit to bear the pain, or you will fail.

I don’t know how well my new company will adjust to the change, nor how resistant it will be to changing at all. But I am fairly competent that the CTO, the new BA and I are all of one mind that we need to bring change to the company to usher it through its adolescence. And it will require an enormous, sustained, united effort to do this. It is a monumental task.

But how do you eat an elephant? One bite at a time.


First, a side note: There was massive snowfall yesterday and this morning. The drive to the central office of my new employer is about 40 miles from home. It was orientation, so I felt it important to go anyway. This was a mistake. I came very close to destroying my car and possibly myself via collision with snowplows, and only a well-timed swerve when braking proved impossible kept me from being a snowplow sandwich. It is NOT worth braving dangerous driving weather, even for a very important work meeting.

That said, I did go to the new job, and I did get some orientation. I took notes. Lots of notes.

I got to meet the CEO face to face. He’s a younger guy, five or so years younger than I am, but very ambitious and very likeable. He also doesn’t try to hide it when things aren’t being done as well as they could, and he listens to criticism.

I got to meet a couple other new hires. One’s in IT and seems to be a likely ally, which I will need, as the laptop they assigned me is locked down and I can’t install my own software on it without IT administrator access. This is not an acceptable situation to any self-respecting developer. The new IT person, whom I’ll call 80’s Hair, understood and appreciated this.

The other new hire is the BA-slash-PM-slash-QA expert. She’s going to be wearing three hats, which I suppose is tenable for now, but that role needs to be broken apart. She and I are working on building a good rapport, as we’ll be working closely together in the months to come.

It was also “monthly meeting” day, which meant a group lunch. This was a chili cookoff. And me without my special homemade chili. (I’m quite a foodie, I admit.) But the chili I sampled was plenty tasty and I had time to get to meet more of the people in the main office.

But the big thing today was what we term ‘drinking from the fire hose.’ The concept is being flooded with more information than I can possibly absorb at once, and all I can do is try to take in all I can.

This particular firehose was learning about business processes, specifically order fulfillment and customer service. The former is the basic result of the e-commerce sites which I’ll be responsible for; the latter is the basic conduit of feedback from the customers. The OF guy spent about 90 minutes illustrating his process to us. He showed us how it began with the output from the existing e-commerce site, printing out packing slips, distributing them into bins, filling the bins from inventory, bringing them to the packing area, and packing the orders in bubble envelopes or boxes, labeling them and marking them in the e-commerce site as shipped. I learned about the basic pain points, the basic points of failure, how those points are caught, and the time invested in the process of catching them.

On the other end, the customer service manager and workers had much input from their customers, and that information doesn’t seem to have a distinct process for getting back to development to be incorporated into future development requirements. Both the new BA and I are keenly interested in remedying that situation.

After that, I was exhausted and packed up my new work-assigned laptop for home on the (now much better plowed) long road. I was requested by the CTO to call him to go over some stuff. So, I did. I was tired, but CTO and I talked and planned out the next couple weeks. I also confirmed with him what earlier conversations with the CEO and document review made me suspect: The nearly-complete current development of the core website was done without a clear requirements document — which made its development go badly up until now. CTO’s high priority for when he is in town the next two weeks, and therefore my high priority for the next two weeks, is to define those requirements tightly. Once that’s done, BA can develop a test plan, and I can evaluate the current state of the project against those requirements, and identify the difference between the two. From there I can plan out development and estimate a time to completion — and then release.

Once that’s done then CTO and I can ensure that all future development will be done against a clear set of requirements, and a business owner of those requirements.

TL’s First Rule Of Development: Without clearly defined and accepted requirements, there is no effective way to know when development is complete, nor even to know what exactly needs to be developed.

Sure with vague requirements like “Make it work like that thing worked” you can get a decent idea of what you’re trying to develop, but it’ll never be clear, and whoever the business owner is will be able to push scope creep into your process with nigh-impunity — other than crashing your sprints and breaking your deadline commitments.