Technology The New Normal How We Work Customer Stories Testing | All Topics

The New Normal: Failure Domains and Safety

Through this series, we've talked about antifragility, disposable code, high leverage, and team-scale autonomy. Earlier, we looked at the benefits of team-scale autonomy: It breaks dependencies between teams, allowing the average speed of the whole organization to increase. People on the teams will be more fulfilled and productive, too. These are nice benefits you can expect from this style, but it's not all unicorns and rainbows. There is some very real, very hard work that has to be done to get there. It should already be clear that you must challenge assumptions about architecture and development processes. But we also need to talk about critical issues of failure domains and safety.

The New Normal: Data Leverage

Like many developers, I’ve spent a lot of time thinking about objects. I started doing object-oriented programming (OOP) in 1989, and was arguably still doing it up until 2012. The concept of objects arose in GUIs and simulations, where it is natural to think about sending message to things. Over time though, the idea of messages took a back seat to the idea of locking data up in little containers where each class provides a namespace of functions to operate on it.

The New Normal: Team Scale Autonomy

You’re probably familiar with the concept of the two pizza team. This is Amazon founder and CEO Jeff Bezos’ rule that every team should be sized no bigger than you can feed with two large pizzas. At Cognitect, we take this concept one step further: the one chateaubriand team—with sharper tools you can afford to spend the extra money on better food.

The two pizza team is an important concept. The idea is that smaller teams are more self-sufficient and typically have better communication and a greater focus on getting things done because they eliminate dependencies. Every dependency is like one of the Lilliputian ropes that ties Gulliver to the beach. Each one may be simple to deal with on its own, but the collection of 1,000 tiny threads keeps you from breaking free. Eliminate dependencies and your teams move faster. Encourage autonomy and you allow innovation.

The New Normal: Everything Relies on Sharp Tools

You wouldn’t use a toothpick to wear down a mountain. You wouldn’t wear your trainers instead of your racing spikes at a track meet. So why would you use a hand saw to build an infrastructure when you need a scalpel for surgical precision?

Up to this point in the series, we’ve talked about embracing failure, experimenting more and moving faster. We've talked about the "why," now it's time to address the "how." 

The New Normal: The Art of War, Maneuverability, and Microservices

The word "antifragile" may be recent, but some of the concepts are ancient. In "Art of War", the renowned general, strategist and tactician Sun Tzu's states, “…water shapes its course according to the nature of the ground over which it flows...” In an antifragile organization, we want to explore opportunities so resources flow like water into the things that are working, and abandon those that are not. 

Just as water retains no constant shape, there are no constants in an antifragile organization and IT infrastructure. To flow like water, you must be able to shift people and teams easily, create teams and systems easily, be able to tear down systems and remove people from working on projects that aren’t working. This requires an architecture that allows you to act locally but think globally. Some organizations are pursuing microservices to this end. Complex applications are composed of small, independent processes that focus on doing a single responsibility. With microservices, developers decouple software into smaller single-function units that can be replaced individually.

The New Normal: Protected Asset or Disposable Inventory?

The accounting department classifies software development work as a capital expense. That means, at least according to accounting, new software represents new capital investment that should increase productivity. This highlights a big divergence in the way accounting views software versus the way we should look at it.

We used to have a saying, “KLOCs kill.” The more lines of code, the more risk you have. As your system gets bigger it gets more complicated and difficult to work with. Code has a carry cost... you have to keep maintaining it. It has obsolescence risk. Undeployed code is exactly like unfinished automobiles: nobody pays you for it. Maintaining it is a liability not an asset. Excess code is a boat anchor that will weigh you down until you drown.

The New Normal: Minimize Risk by Maximizing Change

I once worked for a startup called "Totality." Our business was outsourced web operations for companies that either didn’t want to invest or lacked the skills to build and staff their own24x7 operations center. We handled all the production management, change management, incident management—essentially the entire ITIL (IT Infrastructure Library) suite of processes.

During my time at Totality, we observed that nearly 50% of all our software outages happened within 24 hours of a software release. Since we were on the hook for uptime, but not new features, our response was obvious: Stop touching things!

Sisyphus on Fire: The Coming Agile Dystopia

Hear me now, for I have seen things. A world of ash and soot. Bodies and brains toiling day after day, pushing boulders up steep mountainsides. When these burdens are overcome, when the peak is crested and victory achieved there is neither reward nor respite. Burning disappointment is the only prize, a hot frustration as the mighty stones are sent careening once again to the depths below in endless repetition. This, my friends is the future of agile development. For the past several years I have consulted with over 20 companies - from tiny startups to giant global enterprises. I have seen the inner workings, corporate cultures, and development practices across a wide range of industries. The trends are disturbing. I have seen the future and it is nasty and brutish. We can and must alter course. Resistance is NOT futile.

Why agile?

In the beginning, before agile, was the waterfall. We were doing multi-year projects, implementing reams of printed specifications, missing ship dates and blowing budgets. Most damning of all, we created products that didn’t meet the customer's needs. As big deadlines loomed, teams fell into death marches, with horrible periods of late nights and heated arguments, broken builds and streams of bugs without end. A game of lose-lose, the epitome of action not equalling progress. 

Then a group of survivors hatched a plan of escape. Let’s break the cycle and put real customer satisfaction first. Let’s welcome changing requirements. Let’s deliver early and often. Let’s  communicate across teams and reflect upon ways to improve. Above all, let’s promote sustainable development - a manifesto for practices that would enable sponsors, developers, and users to maintain a constant pace. Indefinitely. It made so much sense. And a few teams have indeed managed to succeed and thrive. Yet here we are, many of us, every 2 weeks. Surrounded by pizza boxes and despair. The points. The pressure. The last minute check-ins, the skimping on QA. The tortured definitions of “complete.” The pronouncements of SUCCESS (yay!) or FAILURE (boo!) every fortnight. OMG it’s happening again. And again. And again! Every 2 weeks. WTF?

All around me are familiar faces
Worn out places, worn out faces
Bright and early for the daily races
Going nowhere, going nowhere...

— Roland Orzabal, “Mad World”

Many organizations took their few, real deadlines - a Beta version, big conference demo, or the major public release - and rolled the incumbent pressure and "all hands on deck" mentality into a twice monthly cannonball run. Remember those large waterfall endeavors? Even if the product was too little or too late, there was at least a time to recharge at the end. To rest and to dream, a period of calm to plan a new route forward. But in many “agile” organizations, there is no rest and no peace. Just another boulder to push up another hill. Say what you will about the horrors of a waterfall death march, these teams at least got to sleep once they were dead. In stark contrast, many agile teams are rousted from their dirt naps before pushing a single daisy. The “big deadline” is now every sprint’s end. This is endless war. This is not about “keeping commitments.” This is not what agile was meant to be. This is not even madness. This. Is. Sparta

Many so-called “agile” organizations have mistaken the process for the product. The daily “how confident are you this card will be done today?” The inevitable “pretty confident unless things don’t go as planned.” This is a bogus dance. This is project management theater. Even the notion of “delivering working software frequently” to maintain a healthy feedback channel is being corrupted. Rather than a brief reflection on progress toward the finish line, most end-of-sprint reviews fixate on “how well did we do compared to our guesses 2 weeks ago?” 

The truth is that we are running a marathon here, people. Treating it like a series of independent races, a chain of back-to-back “sprints” is not the way to win a marathon, nor is it the way to build software. To be truly agile we iterate, we take stock of progress toward the finish line and we adjust as we go. No particular mile is more important (or detached) from any other. Don’t start, stop, start, stop, start, stop. Focus on the end goal and run, dammit, RUN.

Enough with the burndown charts. Enough with the sprint-end whiplash. Establish clear priorities and guidelines - set a vision of what success looks like. Slow occasionally to take stock and adjust as needed, but keep moving forward at a steady pace. 

Let’s stop rolling boulders up and down meaningless hills, and let’s start moving mountains. 

Walmart Runs Clojure at Scale

Say the words "Black Friday" to a group of people and you'll get back a range of emotions. In 2014 this day of historically epic consumer enthusiasm on the Friday after Thanksgiving generated over $50 billion in sales from over 133 million U.S. shoppers. Although originally named for post-Thanksgiving traffic woes in Philadelphia in the 1960's, Black Friday has long been viewed by retailers (some of whom struggle "in the red" through much of the year) as a chance to get their financial numbers "in the black." But this massive surge in transactions is both an opportunity and a crisis for global retailers. Crowds of shoppers seeking pre-holiday deals can generate huge revenue boosts - along with massive spikes in web traffic and server loads. Last year several major retailers suffered system outages costing them $300,000 an hour, and substantially more during peak shopping times. While CFOs (and shoppers) may look to Black Friday with excited anticipation, many CTOs and IT Departments view their November calendars with increasing anxiety.

In preparation for the 2014 holiday season, project architect Anthony Marcar's team at WalmartLabs rolled out a new system designed from the ground up to handle the projected shopping frenzy. The result? After the staggering wave ebbed, his tweet following the surge said it all: 

Our Clojure system just handled its first Walmart black Friday and came out without a scratch.
— Anthony Marcar, Senior Architect - WalmartLabs

How did Walmart's eReceipts team of 8 developers build a system to process and integrate every purchase at Walmart's 5000+ stores, including online and mobile data? Simple - they used Clojure "all the way down" to build a powerful data system enabling the popular Savings Catcher, Vudu Instawatch, Black Friday's 1-Hour Guarantee, and other programs to improve the customer experience and streamline operations. 

Clojure uses anywhere from 5 to 10 times less code than other programming languages, speeding development time, reducing the instances of bugs, and lowering maintenance costs.

Take a behind-the-scenes look at how and why Walmart uses Clojure to reliably power one of the most demanding retail environments in the world as Anthony presents Clojure at Scale.

Distinguished Magic

"The future is already here — it's just not very evenly distributed." - William Gibson

Decision, decisions. Users are screaming to pay money for your service, but your current infrastructure is going to pop like a balloon if you let any more sign up. Or your cumbersome deployment process has slowed enterprise sales to a crawl as your small support team runs themselves ragged. Is your competition widening the feature gap due to your lengthy test and release cycle? How to process all that customer data into useful trends? Your decisions today will mean soaring profit or smoldering ruin. How will you solve these problems? Traditional answers to these questions lead many organizations down roads from which they never return. But in the future? Roads? Where we’re going, we don’t need roads...  

You may have tried incremental approaches that were so incremental that nothing changed. You may have attempted full system overhauls, only to have the budget and schedule blow up in your face while the problems remained - or got worse. Or you may not even know where to start.

Usually the problem isn’t a lack of skill. It’s not a lack of commitment. Or an aversion to hard work. No, what’s holding you back is a failure of imagination. Most people self-censor (whether they are aware of it or not), limiting themselves to what they think is possible. We tend to stick to what we know how to do or have seen others do. Because we cannot see a path to the future we would like to see, we fail to see that future at all.  Ironically, when we come across someone who did manage to transcend these mental limits, we say that they “saw what others could not see”. Actually these visionaries see less than others - they don’t  see the walls that block others' view of a better world. They don’t impose restrictions on the future. The difference is that they believe in magic.

So forget what you and the people around you already know how to do. Forget what you have time to do. Forget what you have the budget to do. Forget what you think others will let you do. Instead, ask yourself:

If I could wave a magic wand and make the solution appear, what would it look like? 

Ignore how you are going to get there and just think about what things will look like once you arrive. You need to believe.

Working backwards from a “magical” solution will result in a better real world solution than you can get to by crawling forward on your belly around obstacles you find along the road.

"Any sufficiently advanced technology is indistinguishable from magic." — Arthur C. Clarke

With the future of your business at stake, you need a new approach for solving your tough problems.  At Cognitect, we help teams break through preconceptions that hold them back. To envision systems unrestrained by the limits of traditional development tools, to design the right solution to grow your business. The results sometimes seem magical - our modern tech stack of Clojure and Datomic enables development of flexible cloud-based systems at significantly less cost than traditional approaches. Technologies now exist that enable small teams to build enterprise-grade solutions on schedules and budgets that not long ago really would have required a pointy hat and cloak.

As noted by former Chief Architect of Netflix’s Cloud Services Adrian Cockcroft, Clojure allows developers to produce "ridiculously sophisticated things in a very short time.” And immutable data - a hallmark of the Datomic database - grants complete auditability of values and transactions from any previous time, plus the ability to model potential scenarios without committing changes. Relive the past? Peer into the future? Datomic may not actually let you travel through time, but the results are just about indistinguishable.

You can attempt to solve your problem with the traditional approach. But are you planning the right solution? Or just what you can put together with the tools and ideas inside the barriers that have accreted across your organizational field of vision? Or will you forget what is and imagine what could be, without preconceived limitations and constraints? Don’t settle for a faster horse - demand a rocket ship. This may be the path less traveled, but it will make all the difference.

Story Points Must Die: A Spacetime Adventure

Long ago beyond the stars a civilization rose from dust and puddles to spread the gospel of “agile software development” across many worlds. Over time these worlds diverged into two distinct tribes, known as “Team A” and “Team B” (for they were truly unimaginative beings in the ways of names). Both tribes prospered, delivering software throughout the galaxy, but over many generations Team A grew to believe the source of their success was a mysterious force known as “Story Points”. Team A evolved an elaborate ritual that they performed during their traditional bi-weekly gatherings known as “Sprint Planning”.  Team A believed these rituals allowed them to accurately predict the future. Abandoning the process of deep thought and discussion, Team A eventually focused their entire development process on these “points”, which for some reason existed only in quantities of 1, 2, 3, 5, and (very rarely) 8.

While both tribes continued to ship software, fealty to this numeric master began to visibly distort Team A’s demeanor and appearance until travelers from other galaxies could easily tell which of the two tribes they had encountered. Those from Team A came to be known as victims of "Fibonacci’s Curse". 

Let us go now, across space and time in the blink of an eye to gaze upon two such parallel development efforts...

Extremely Honest Programming

The true cost of new features is one of the toughest things for Product Managers to understand. When you've got a full-time team of salaried developers at your side, it's easy to forget the real monetary costs of writing code. The result? Blown schedules, burned out development teams, and lackluster customer response to the "big release". This is where Extremely Honest Programming comes in. EHP is all about net results.