Home » Archive by category "Starting Out"

Investment

One thing I’ve learned in my years as a developer is that I am more eager to go to work, and am more invested in doing the best job I can, when I feel an emotional connection to my team and my employers.

That connection can take the form of friendship, or loyalty, or empathy, but if it’s there, I do better.

Knowing this about myself, I’ve always made an effort to build that connection to my team and to people in other teams throughout my company. I’m naturally a gregarious, goofy guy (sometimes to excess) and I try to use those tendencies to make friends. I also get to understand the business, what it’s doing, how all the pieces fit together and how each team contributes to the goal. And then for me, doing my part becomes more important because I know how what I do impacts everyone else.

I also know that I’ve worked for people at some of my jobs that have encouraged and facilitated this, either deliberately, opportunistically, or instinctively, because they understand that value.

Being in the lead role, it’s important I understand that I, now, also should take the time to foster that connection in my team. I don’t know if it’s in me to inspire loyalty, but even if it’s not, I think I’m not too bad at building a little friendship and empathy amongst my team. But beyond that, it’s important for me to help the people on my team get a grasp of the things that leads like myself have to get to understand implicitly, namely, the various parts of the company and how they work togther, how the work of our team impacts all those other teams, and how those teams impact our own.

By encouraging my own team to build those emotional connections, I encourage them to become more invested in their jobs. And if I succeed, I reap the twin rewards of lower turnover and greater productivity — as well as making some friends along the way.

The counterpoint to all this is an experience I’ve had many times as well — a company where those connections are not encouraged and are not valued. We’ve all been there, whether it takes the form of a boss who’s all stick and no carrot, a team that is every-man-for-himself, or a team so isolated from the rest of the company that they don’t care about anything that doesn’t directly impact them. Think on your experiences in places like this. Did you work harder, or not? Did you have an easier or harder time getting out of bed in the morning? Were you more or less willing to stay late to get the job done?

Yeah, me too.

So it’s really a no-brainer which kind of culture I want to foster in my team. So it’s just a matter of learning how.

Urgency

Some people seem to think that by impressing a sense of sufficient urgency that the impossible can be made possible.

Let’s consider the hypothetical scenario where you’re at home, and you live a 30 minute drive from work when traffic is good. Today, traffic is not good.

Your boss calls you in a panic and says you have to be at work in 15 minutes. You tell him you’ll head out immediately but it’s really unlikely you’ll be there in less than 30.

Your boss tells you that you need to be at work in 15 minutes or the company will lose its most important client. You tell him you’re already on your way, but it’ll still be 30 minutes, even if you speed and weave through traffic.

Your boss tells you that you need to be at work in 15 minutes or the company will go under, and it will be all your fault. You’re upset, and angry, but you tell him there’s no way you can be there in time.

So you break several traffic laws and finally rush into work 25 minutes later, your boss blames you for failing to come to work on time and causing the demise of the company. And you’re silently plotting how to murder your boss without leaving any forensic evidence.

The entire scenario is absurd, of course, yet this is an all too common scenario of software development: The business owner sets a development schedule that’s laughably unrealistic, and when the development lead pushes back, the business owner behaves as if he believes he can make the project complete on schedule by imparting a sufficient sense of urgency to the development team.

It won’t work, of course; the dev team may, of course, work in panic mode for the duration of the project and try in futility to make the deadline, and then when they fail they will be so exhausted and burned out that they will be useless as developers for weeks afterwards, but that means the project fails and the company has a completely demoralized dev team that’s probably been updating their resumes.

Project owners need to listen to their development teams when they say that the project deadline is far too soon. Instead of trying to crack the whip, project owners need to work with the development lead and project manager to figure out the alternative. That alternative may be a reduced list of features, a revised deadline, the addition of resources, or some combination of the three.

But setting an arbitrary deadline and offering only urgency and consequences to the development team for failing to meet it, apart from violating the Fourth Rule of Development, can seriously damage the morale, and by extension the value and effectivenes, of the development team.

It’s about the team

First, a note:

I’ve been rethinking, due to feedback from friends and family, how I present the content of this trifle of a blog. I’ve been writing directly about my experiences while keeping the name of my company, and the names of my coworkers, obfuscated using nicknames and euphemisms. However, that isn’t really obfuscated enough to keep me out of trouble should this blog ever get connected to me by the wrong people. So, henceforth, I shall talk about my learnings without directly relating my experiences at work, so there will be no issue. You can make up your own story about what experiences I had which led me to be taught or reminded of whatever I’m imparting with my blog entries. Experiences outside of work, well, those I may continue to discuss directly as discretion permits.

 

Now then.

One quality of a development team, or indeed any team working together, that is often undervalued is how well the members of the team interact on a personal level. Teams which are comprised of even exceptionally talented people who do not have any affinity for one another will not work exceptionally well together. Teams of much more modest talent who have a strong affinity for one another will become more than the sum of their parts. It’s not hard to imagine circumstances where a less talented team with more mutual affinity will be more productive with better results than a more talented but less cohesive team.

A lot of employers understand that personal affinity is important; they interview for new hires by evaluating technical skill but also how well they believe the candidate’s personality will integrate with the team and/or the company. They call this ‘cultural fit’ usually, but I think they often don’t consider the full implications of how well personalities fit one another, and in particular the magic that can happen where a team comes together comprised of people who develop strong professional and personal bonds with one another.

Here’s a question: How many times have you seen a candidate for a job show strong technical qualifications, but is iffy culturally, and get hired despite misgivings over the cultural fit? Here’s another: How many times have you seen a candidate for a job who presents a superb cultural fit, but is more iffy in technical qualifications, and get hired anyway?

I bet most people in the development world would answer these questions somewhere between “occasionally” and “frequently”, and somewhere between “never” and “almost never” respectively. For me it’s “occasionally” and “never”. But let’s think about this, and consider that this is exactly the opposite of the way things should be.

Skills can be taught. Someone entering a job with some technical deficiencies can be trained to grow into the job. Anyone with development experience has witnessed this — most of us, in fact, have lived it. We learn, adapt, and grow to meet the needs of our jobs even if we don’t start with the necessary skills. If we can’t do that, we’re probably not in the right line of work. We can also learn ‘soft skills’ – leadership skills, communication skills, organization, etc. But we can’t learn to have a different personality.

Oh, sure, we can learn to fake it; learn to pretend that the Nerf gun war that breaks out among the development team isn’t an annoyance and is fun; learn to have detailed conversations about international cuisine like it’s interesting; learn to geek out about the latest episode of Doctor Who or The Walking Dead as if we are interested in either — but if you don’t love shooting your buddies with foam darts, if you don’t relish new dining experiences, or if you don’t giggle inside at the phrase ‘wibbly wobbly, timey wimey’, you aren’t going to change. And fundamentally, you’re never going to build that connection to people who geek out in ways that you don’t.

So why is it that hiring decisions are often made on the principle that trainable qualifications are more valuable than untrainable qualifications?

In the area where I live, positions available for highly skilled developers are much more numerous than developers are available to fill those positions. My friend and colleague Red, whom I’m now calling Mojo (because she prefers Mojo) has said that the best solution is to find a strong cultural fit who’s below the desired skill level, hire that candidate, and invest in that developer through mentoring and training. This course of action makes even greater sense in light of the above.

So with all that in mind, imagine that rare confluence of a great group of developers, designers, project people and leadership who have developed that strong affinity, mutual respect, and friendship with one another. This is a group of people who not only want to be successful as individuals but who want to see each other succeed as well. They look forward to seeing one another and collaborating together.

How much harder and with how much more dedication will this group of people work on a given project when they have that mutual bond motivating them? And what will they be able to accomplish together?

I’d sure as hell like to find out.

Ka-ching

One of the things I need to keep in mind as a lead is that the list of things I need to know in order to be effective has grown. It’s not just the details of development, design and architecture, along with a baseline knowledge of requirements gathering, project management and QA, but also things like business strategy, budgets and, as proved important today, vendor agreements.

We’ve been looking at changing the web hosting provider for our new site since a little before I came on board. When I joined, it was among my first tasks to compare our current provider with our potential new provider — to assess value of the services for the price.

Part of what I did was review the contracts — in particular, the uptime and support guarantees, and made note of what each was offering. (As the company has tens of millions of dollars a year in sales, every hour of downtime is a potential loss of thousands of dollars — during prime time, tens of thousands.) And by making note of these facts, I was able to recoup a significant chunk of change for the company today.

As it happens, this morning we found the staging site — the one hosted by our current provider — was down. It was down because the load balancer*, which is entirely in the provider’s control, was failing. This halted much of our development. Amazingly, this premium web hosting provider to which we pay several thousand dollars a month was unable to have the site back online by the end of the day, and we since learned it was down much of the weekend before that.

Much frustration ensued, including a push to switch us over to the new provider ASAP. But towards the end of the day, I mentioned to the CEO this little detail of the contract with the provider: For every hour of downtime, we are entitled to be credited 5% of our monthly contract fees. It’s also important to note that the contract does not specify that this credit caps at 100%.

The dollar signs appeared over the CEO’s eyes when he brought up the contract and confirmed that information. Gleefully he asked me to put in a request with that provider for a refund — and forwarded me the pertinent clause of the contract.

I just earned my pay for the month, I think. And I also earned a strong measure of the confidence of my CEO, which is very important to achieving my goals for the company. And I also had an important lesson of being a lead reinforced. I’m going to remember the importance of keeping apprised of the details of many things beyond the development process itself.

*If you don’t know what a load balancer is, it’s a server at the front end of the web server cluster that routes incoming traffic evenly between the web servers so that the servers handle roughly equivalent “load” of that traffic. Basically, if it stops working, your entire site fails.

 

The stake in the ground

Today was the first day where I didn’t leave work frustrated with how difficult it is to learn the things I need to know to do my job. This was a very good thing.

What’s a bad thing is the principal reason why it’s been such a slog for me to date.

The platform I’m currently responsible for getting production-ready was originally built with very few specifications. The specifications were, mostly, “Make it do what the old site did.”

There were not one, but two major problems with this, both of which are fairly obvious.

The first was that the old platform’s processes weren’t documented either. Knowledge of how to use many of them were scattered throughout the company’s employees and management, of course, but with no BA, turning that knowledge into requirements is not an easy task. Other process knowledge left the company with old employees, and needs to be rediscovered.

The second problem is a little less obvious than the first, but still readily apparent: Just because the old site does something doesn’t mean the new site needs to do the same thing. Any new development should be with the goal of meeting business needs, not matching existing processes. Often, old processes are obsolete, or are workarounds for things that didn’t work exactly the right way on the old platform. If your old e-commerce site handles conversion of Canadian to American currency but you’ve since stopped selling to Canada, you don’t need that functionality any more. If your old administration console had a process that exported all orders to an XML file that a user transferred to the fulfillment service’s import function, it’s not necessary to build that export function in the new platform if it can instead export directly to the fulfillment service.

TL’s Fifth Rule of Development: When designing an upgrade to an application or service, the question to ask is not, “What does it do now?” The question to ask is, “What are the business needs to be met?”

So what was built was taking a powerful new platform, and wedging the old platform’s processes onto it, without really examining what the needs are nor understanding the new and possibly better ways the new platform can meet them.

Fortunately a few of the more onerous features haven’t been implemented yet, and I’ve been given the opportunity to put the brakes on what they’ve been doing and ask those questions: What do we need it to do? Can it be done differently? These questions have already yielded results that are going to allow me to meet the company’s needs far more cheaply, and far more simply, than otherwise would have.

And better still, the powers that be at the company let me do just that.

And that is my first major win at my new job.

Tapped out

The week ended with exhaustion and a feeling I was in over my head.

Not because I thought myself incapable, but because the demands placed upon me, Pope and BA/PM to get this project out the door were enormous and I’ve barely scratched the surface of the technology in which I need to become an expert.

The schedule was put together by people who are not developers, based, it seems, solely on the date they decided they need to go live rather than a sound estimate of what the minimum list of tasks is and the time it will reasonably take to complete them.

This is what we call “Schedule-driven development” and it never works. Why?

TL’s Fourth Rule of Development: Time, Cost, Quality. The business owner of the project gets to dictate two at most. Trying to dictate all three will set the stage for failure.


It’s an old rule which I’m sure anyone reading this has previously heard, but it bears repeating because business owners keep breaking the rule, in organization after organization, project after project. Sometimes the same business owner does this repeatedly, wondering why his projects keep crashing and burning.

But that’s the paradigm I find myself facing. There is a set of minimum marketable requirements, a schedule to get them done, and a specified number of resources to accomplish the task.

I knew there would be difficulty in changing the development culture of this company, but I didn’t know the immediate task would be so urgent and with such a broken structure. And I know I can’t begin the process of changing the company’s culture until this immediate task is complete. And my chances of changing the development culture hinge on how well I complete the task.

The good, however, is that both Pope and BA/PM are right there with me, they see exactly the same problems I do, and thus we have a united front in dealing with this challenge, and managing the business leadership when deadlines inevitably slip.

The fact that Pope is part of senior leadership is one reason why we’ve got a chance to make this plan work. Without him, I’d think we were tilting at windmills, at best.

The fire hose

As I get to know my new coworkers I’m starting to assign them nicknames in this blog. The papal conclave that elected a new Pope today inspired me, for a few reasons, to nickname my boss, previously referred to as CTO. He is henceforth “Pope”.

So Pope and I the last two days have been working on bringing me up to speed, and I have surely been “drinking from the fire hose.” I’m mentally exhausted.

Today became a different kind of trial, however, as a critical issue with the current production servers (directly impacting company revenue) consumed much of our time.

I found myself doing something I really never like to do but have many times found myself needed to do: dive into a complicated, poorly-documented set of running processes and try to figure out what the problem is. Pope and I tag-teamed it, with his understanding of processes and my understanding of code, with occasional calls to business users to figure things out. Eventually we figured out that this was a problem with our network, which had recently been reconfigured due to the opening of our new branch office (the one from which I’m working). Essentially, the internal firewall got misconfigured and blocked the processes from communicating with outside processes.

Which tells me the current version of our platform does something very badly: It doesn’t report when things aren’t working right. It “fails silently.”

TL’s Third Rule of Development: Never architect nor write your code under the assumption that everything will work as planned. Always develop with the exception cases in mind.

That rule goes double for any code that talks to something external to itself (a database, an API, a file import or export). Because those always seem to break at the most surprising times.

Fortunately most of this code is going to be retired when the new platform is release. Unfortunately, that’s no guarantee that the new code handles the exception cases appropriately, and a review is indicated.

Other fun things I discovered. A cron job runs every hour, on the hour. This cron job tries to execute two scripts that don’t exist. As far as I can tell, they haven’t existed for over a year because there are web server logs that show the same errors every hour going back that far.

This concerns me greatly because it, obviously, means there has been little to no quality assurance on the processes or the code to date. And I’m obviously worried that the new development has taken place under the same (lack of) constraints.

My solution is to lay the groundwork for a near-term plan to do a full review of the new code base. I know that CEO’s top priority is getting our new platform out the door as quickly as possible, as the E-commerce guy has all kinds of things he wants to do that are dependent on the new platform being live. So that’s first priority. But once that’s done, I want to do that full review and identify potential problem areas. I introduced CEO to the concept of ‘technical debt’ and told him what my post-release goals were in the context of ‘paying down’ the technical debt.

Mostly, I don’t want to be trying, yet again, to figure out how to build stable code on a brittle foundation.


The other item of note today was that CEO was eager to plan to build my team. I told him I had several people in mind, but that I needed a couple weeks to really get to understand the company’s needs before I made any recommendations. Pope was quick to back me up on that, saying he wanted my focus right now to be getting acquainted with the new applications.

But this is a good thing for my desire to build myself and my company a great development team.

Day One

So I walked into the office like I owned the place.

Well, no. I walked in a bit nervously. BA/PM was there (she had a week head start on me) as was CEO. CTO lives out of town and was to arrive in the early afternoon.

The first task is to complete, post-haste, the new E-commerce platform for the company’s primary website. It was, in violation of Rule One, constructed without a clear requirements document — the direction was to ‘make it work like the old one’.

This left a bit of a mess, and coupled with a couple key developers rolling off the project unfortunately quickly (not, to my understanding, for ugly reasons) the mess got larger.

In the interim, outside contractors were brought in to complete work on it, but significant challenges to complete the project remain — most notably aspects of sharing some data with the order fulfillment system. And it’ll be on me to complete development as I’m, for now, the only developer.

I met a few other key players, including the new designer (also starting today) and the E-Commerce guru (who needs this project done to implement some of his plans). We had lunch together.

A note on having lunch together: In an earlier entry, I wrote that “Never have lunch alone” was one of Sparky’s key pieces of advice. The idea is that if you’re lunching with your co-workers, you’re building rapport and bonding. As I have learned from my many years of employment, as a developer and otherwise, whether you’re having lunch with your direct reports, your peers, or your superiors, this has great value in the near and long terms.

Also, when your CEO goes to lunch with you, he usually buys. So, free lunch.

After lunch CTO arrived and we got immediately to work. He gave me and BA/PM the breakdown of the project and the major things to deal with, and then immediately pivoted to a discussion of the deployment architecture, which is currently slated to be very expensive (as in, thousands per month). CTO and I started exploring alternative architectures, and then immediately he started contacting the vendor for bids based on the alternatives, with me riding shotgun, and principally giving him nonverbal cues in reaction to what the vendor was saying.

One of the architectural details pushed by the vendor was a file server. Neither CTO nor I saw a separate file server as necessary for our architecture. The vendor (who, it should be noted, is known for their expertise in our new E-commerce platform) insisted it was a critical and necessary part of the architecture. One glance exchanged between the two of us confirmed that we believed the vendor was full of it. When CTO pushed back, the vendor said he’d talk to someone more technical about it to make sure it would be all right for us.

Tomorrow CTO and I are heading to be on-site with another vendor – the development vendor – to begin the lengthy process of knowledge transfer into my brain.

I expect to be very tired for the next few days. But if I take good notes and keep focused I should be in a much better position to understand the company’s technical needs, and begin to build my team accordingly.

However, it sounds like that will not be my first concern. First, I need to get this platform ready to release.

Oy.

Eating the elephant

Had a talk with the contractor who’s been responsible for the development of the websites for which I’m going to be responsible starting Monday. Websites that will be going live Any Week Now.

Productive discussion and a brief crash course in the software upon which these sites are built.

But what I also learned is some not wholly unexpected but difficult facts about the environment into which I’m headed. And it’s an environment pretty familiar to me — in many respects, it’s very much like the environment of my current job was the day I arrived.

One of the things I’ve learned through multiple jobs in my career is that most startup companies which have a core web services platform have a point I call Development Adolescence, a period analogous to true adolescence where one transitions from carefree childhood to responsible adulthood. And that transition is a period of difficulty, awkwardness, and often self-created pain and angst.

Development childhood is the period where a new company is initially creating and growing its web services. It needs to grow fast, and it doesn’t usually have a very detailed idea of what it needs its web services to be. So the web services tend to grow organically, rapidly, without a very clear picture of its destination. There are either no clearly defined business owners of the platform, meaning too many people drive the business process of its development and not enough people (outside the developer or developers themselves) have any accountability for that development. The application grows, it becomes harder to maintain because the code has no overall structure for its growth, requirements documentation is sparse to nonexistent and process flow documentation doubly so. “Technical debt”, meaning structural defects in the application that slow new development or require time invested to maintain the existing services, builds up. Developers start fearing the most onerous parts of the codebase and know that the entire system is brittle and can break in unexpected and non-obvious ways by any modification.

The tipping point is reached when either business needs outgrow the capacity of the existing platform to meet them, or the accumulated technical debt renders the platform so unstable or development so painful that the dev team cries out for relief. The business becomes aware that the current path is unsustainable, and that the development process must mature. And thus begins Development Adolescence.

Adolescence is the process of the development team leadership (by which I mean me) establishing a sustainable development framework for both stable growth in the web services platform and for controlling and prioritizing the flow of new change requests coming for the business. This process usually takes place in the context of either a major overhaul or wholesale replacement of the web services platform, transforming it into something that has a much better chance of serving the needs of the business in the medium to long term.

This isn’t awkward in and of itself, but the awkwardness ensues when the business, which has reached the point of understanding that the development process needs to greatly change, doesn’t yet understand that the business process must change too. When adolescence is reached, the business has become accustomed to how they do things — usually, they tell Dev to do something and Dev does it. There is no gatekeeper who prioritizes task, no real requirements gathering, no business owner to agree to the specific changelist, no placing the requirements in the context of the long-term business goals, nothing. And they are resistant to changing how they do things, because what they’ve done so far has worked, for them.

And so it is incumbent upon the development team leadership, along with the senior management responsible for the dev team (usually, the CTO/CIO), to bring the business to understand and buy into the value of changing their processes to work with the new business processes. And this is no easy task. Myself, I favor a top-down approach: Work with the CTO/CIO to get buy in from the CEO and/or the rest of the senior leadership team of the company, who then champion the business process changes with the rest of the business, who in turn adapt to change, slowly, step by step.

The problem with that path is that there’s always the chance you don’t get buy-in from the CEO/SLT. And that makes adolescence far more difficult to get through.

The other viable path is to try to convince the business more holistically by putting processes in place step by step and showing the business the benefits that are reaped by demonstration. This is an even slower, more incremental, far more painful process to adulthood.

Whichever path is taken, the question of how the business will emerge from adolescence is largely a question of how well the business adapts to change (er, okay, that and how well the development team imposes the process upon itself and how closely it adheres to that process, but I’m talking about the business here). And I’ve personally seen this go well, and go poorly, in my own work history.

The company I’m leaving is one where it’s gone poorly. The business adapted to some of the processes, but did a very bad job of bearing the pain of transition and began to slip into the mode of “panic-driven-development”, where the dev group was repeatedly directed to redirect its entire effort to whichever issue was causing the business the most pain at the moment. This results in lots of projects being left half-done, “to be finished later”, as the new greatest pain point becomes the new focus. Over. And Over. And Over Again.

TL’s Second Rule Of Development: Change to development processes is painful. If you’re going to commit to the change, commit to bear the pain, or you will fail.

I don’t know how well my new company will adjust to the change, nor how resistant it will be to changing at all. But I am fairly competent that the CTO, the new BA and I are all of one mind that we need to bring change to the company to usher it through its adolescence. And it will require an enormous, sustained, united effort to do this. It is a monumental task.

But how do you eat an elephant? One bite at a time.