How We Work

Technology The New Normal How We Work Customer Stories Testing | All Topics

How We Work: Agile is an Attitude

One of the wonderful things about working in software is that every company, every system and every project is a little experiment in group dynamics. From the tiniest of startups to the most massive of enterprises, behind every information system are teams of people who spend a lot of time collaborating. And when people work together a lot they tend to develop their own idiosyncratic vocabulary. Certainly we at Cognitect have our own jargon. At Cognitect everybody knows that a Cognation is our periodic company meeting, that the Zoom Party (think group video call) happens on Fridays and that it is better to elide than to complect.  

Everyone at Cognitect also knows the difference between agile and Agile. Since this is print, I’m using some font tricks to represent what is primarily a difference in pronunciation: Think of Agile as being projected from the diaphragm of a Shakespearean actor during a critical bit of the script, perhaps To Agile or not to Agile? or maybe You feelin’ Agile punk?  In contrast, agile is said in the familiar tone you would use for a particularly handy tool: Hey could you hand me the three-quarters-inch agile?

The Two Agiles

The real question, aside from pronunciation, is what’s the difference? To us, Agile (the Kenneth Brannagh version) is the mainstream vision of agile, a family of software development methodologies built around some key practices:

  • Standups
  • Iterations
  • Development stories - with or without points
  • Some version of self organization
  • Retrospectives

Let me start by saying that I’m a fan of each of these practices. In my experience software projects that do some or all of these things tend to succeed while projects that do none of them tend to end up face down in the gutter. But as valuable as they are, I don’t think that these practices are at the core of the monkey wrench version of agile.

To get to that core you need to dig into why these practices became associated with agile (either flavor) in the first place. In fact, you need to go all the way back to the manifesto that started it all, which (in part) says:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

Despite the name -- manifesto sounds like something you work on from the comfort of your Federal prison cell -- there’s a vital idea lurking inside those four sentences. And that idea is:

To get the job done you need to focus on the goal and the people who will get you there.

Processes and tools are great, but in the end it’s the people and not the rule book that get things done. So if the goal is working software, then working software is more important than the tools, processes and documentation that you use or generate along the way. And if working with your customer is a key to success then work with them. And if getting to working software is the point then be prepared to tear up the plan the minute the plan gets in the way. That’s what we at Cognitect mean when we talk about the monkey wrench version of agile: Focus on the goals and the people who will get you there.

Goals and People...

Once you have the goals and people idea in your head, a lot of the standard agile practices do make sense:

  • Frequent short, informal meetings help to keep everyone pulling in the same direction, so sure, let’s have stand ups.
  • Breaking up larger goals into smaller chunks helps get things done, so let’s have iterations.
  • Knowing what you need to do next is always helpful so let’s have stories to describe what we're doing.
  • And if your people have the goal firmly in mind, you can leave it to them to get much of the project organized.

But while the classic agile practices make sense, they are not an end in themselves. At Cognitect we do projects large and small, from enormous e-commerce and financial applications for giant enterprises to much more modest systems for tiny but innovative start ups. For some customers we have built the whole thing, all the way from an idea to launch. At other times we worked side by side with the clients, helping them turn their vision into bytes. Still other clients have pulled us in for training, where our job is to help them get better at their job. And on other projects we have been more like advisors, looking over the plans or the code and offering expert advice.

If there is one thing that is clear from our experience doing projects long and short for customers large and small, it is that one size does not fit all:

  • Stand ups are great, but you can’t go round the room quickly if there are fifty people on the project. At the other end of the spectrum stand ups are also silly if it’s just you and your partner working together day after day.
  • Incremental progress is great, but can you iterate your way to an overall architecture for a giant fault tolerant distributed information system?
  • Stories are wonderful, but what if your project is more research than development? Or what if you are still trying to figure out what it is you need to build? What’s the point value of vision? What if your job is to train another developer? Don’t you need something that looks less like a story and more like a lesson plan?
  • And how much self organization can you have if the vision is fuzzy or the domain is beyond most of the team.

What we’ve learned is that there is no magic recipe, no set of practices that will guarantee success.  We’ve found that if you try to build information systems with a recipe, if you put your faith in a standard playbook of processes and practices, you are setting yourself up to make a bitter discovery about reality: Reality is a jerk.  Sooner or later reality will find something that your rules can’t handle and lob that thing right into the middle of your project.

At its heart, the monkey wrench version of agile says that in order to be agile you need to actually be agile in the sense that civilians use the word: You need to be ready to throw away the rule book and bob and weave and flex with whatever comes next. You need to improvise, adapt and overcome. Above all, agile means keeping your eye on the fundamentals: What are the goals? And how are we going to get there?

… and Results

At the risk of repeating myself, let me say it again. Here at Cognitect we do the standard agile processes and I think we do them well. We’ve iterated our way through hundreds of software projects. We have run more retros than most people have had hot lunches. And we’ve written more stories than Stephen King. But we also know that iterations, retros and stories are just a means to an end. When they make sense, we use them. When they don’t we let them go without a moment’s regret. Agile isn’t a rule book, it is the absence of a rule book. Here at Cognitect our focus is on the goals and the people who will get you there.

As I say, at Cognitect we don’t do Agile. We do agile. In other words, we focus on people and results.

 

REPL Debugging: No Stacktrace Required

Eli Bendersky recently blogged about debugging Clojure code. His post is worth reading, and shows how to read stacktraces and add instrumentation for tracing. In this blog post I am going to debug his example program without using such tooling, relying only on the rapid feedback loop provided by the REPL.

Here's the problem program:

(defn foo
      [n]
      (cond (> n 40) (+ n 20)
            (> n 20) (- (first n) 20)
            :else 0))

Before I even show you this program failing, I am going to deliberately obfuscate Clojure's error reporting. I am going to throw away all the potentially useful information: the exception classes, the stacktraces, and the error messages. To do this I will start a subrepl that snidely elides all information about exceptions caught by the REPL:

(require '[clojure.main :as main])
    (main/repl :caught (fn [_] (println "Broken! HaHa!")))

Now we are ready to see the problem:

(foo 24)
    => Broken! HaHa!

So foo is broken for the value 24. We will look at three ways to track this down:

  • blindly subdivide the problem
  • execute the code in your head
  • mix and match

Blindly Subdivide the Problem

To blindly subdivide the problem, start with the your problem case and explicitly eval all the subpieces. This will often identify which subpiece is the problem, and you can work your way down until you have a problem that you can understand at a glance. Since foo uses only one name, n, I will temporarily def that name at the top level:

(def n 24)

Now I can evaluate every subform in foo, just by putting my cursor at the end of each form and sending that form to my REPL (all Clojure editors support this). The definition of foo has only one form in its body, the cond, so I will eval that first:

(defn foo
      [n]
      (cond (> n 40) (+ n 20)
            (> n 20) (- (first n) 20)
            :else 0)◼︎)
    => Broken! HaHa!

The ◼︎ character indicates the position of the cursor when I ask my REPL to evaluate the previous form. Since the cond also appears broken, I will now evaluate all six of the cond's subforms in quick succession:

(defn foo
      [n]
      (cond (> n 40)◼︎ (+ n 20)◼︎
            (> n 20)◼︎ (- (first n) 20)◼︎
            :else◼︎ 0◼︎))

    ;; results shown in order the ◼︎ appears:
    (1) => false
    (2) => 44
    (3) => true
    (4) => Broken! HaHa!
    (5) => :else
    (6) => 0

Now we are getting somewhere! Of the six subforms of foo, only the fourth one, (- (first n) 20), causes the problem. Subdividing one last time:

(defn foo
      [n]
      (cond (> n 40) (+ n 20)
            (> n 20) (-◼︎ (first n)◼︎ 20◼︎)
            :else 0))


    ;; results shown in order the ◼︎ appears:
    (1) => #object[...]
    (2) => Broken! HaHa!
    (3) => 20

At this point it is pretty clear that calling first on a number is not valid. A quick doc check will show that first is for collections:

(doc first)
    -------------------------
    clojure.core/first
    ([coll])
      Returns the first item in the collection. Calls seq on its
        argument. If coll is nil, returns nil.

If this approach seems dumb to you, good! It is dumb by design, and can mindlessly find things that a mind might space out and miss.
Now let's solve the problem again, smarter:

Execute the Code In Your Head

To the extent that you are familiar with a language, you may be able to execute code in your head even faster than you can navigate forms in an editor. I know Clojure pretty well, so my internal monologue looking at this code would be something like:

  • I see that foo is being passed one arg, the integer 24
  • foo has only one branching form, the cond
  • the value 24 will take me down the second branch
  • hey, you can't call first on a number!

For me and for this problem, there is a good chance that this approach would be faster than blindly subdividing. But this is situational, and we are all fallible, so how about mixing these two approaches?

Mix and Match

The Goldilocks rule tells us that if blindly subdividing the problem is too dumb, and executing the code in your head is too clever, then you should aim for a happy medium of the two. This is in fact my normal operating mode when solving function-level problems in Clojure. How do you know if you are at a happy medium? If you execute the code in your head and don't get the same answer the REPL is giving you, dumb it down a little and subdivide the problem.

It is worth noting that this entire discussion can also be applied to writing code, not just to debugging. When you write code, you are imagining what needs to happen (executing not-yet-written code in your head), and breaking that into a set of steps (subdividing the problem). If, while you are developing, you try each subform at the REPL, you are far less likely to make the kind of mistake we are examining here. Also, this style catches a certain category of errors more quickly than you can catch them with unit testing or TDD.

Conclusion

Clojure's tangible environment (the REPL, namespaces, vars, etc.) make it possible to inspect and modify your program directly, quickly adjusting your focus as you are solving problems. Of course there are many other tools you might use for debugging:

  • stacktraces
  • temporary print statements
  • logging
  • symbolic debuggers

All of these tools can and should be made better for Clojure. But these other tools are all generic, and are likely available in any language. The next time you are debugging a problem, and before you opt for a generic experience, think about what makes Clojure special and what you could do from the REPL.

Developing the Language of the Domain

2U, a company that provides the technology, services and data architecture to transform its clients into digital universities, asked for Cognitect’s help making their operations more efficient. They had a deep client-onboarding pipeline, constrained by the speed of content-creation, and they wanted to eliminate unnecessary bottlenecks in their software systems.

Iteration Zero

During our Iteration Zero, we worked with 2U engineers to sketch out a high-level view of their software architecture, with a particular eye toward information flows. As this diagram grew to cover most of a conference-room wall, it became clear that the architecture had a lot of moving parts and information silos, not surprising for a successful startup after several years of rapid growth.

The key insight of the architecture review was that 2U relied on people to move data between software systems — mostly content, but also configuration, access credentials, and design artifacts. Streamlining these processes meant not only building software systems that could communicate directly with one another, but also incorporating human work-products into an automated workflow.

The Language of the Domain

Working with a mixed team of developers and operations specialists at 2U, we set out to develop a model of the business domain. It started with a simple question: What types of entities — what kinds of stuff — do you work with? Programmers may recognize this as a classic object-oriented design exercise, but we weren’t limited to object-oriented nomenclature or a complex modeling language such as UML. Instead, we took advantage of our primary tools, Clojure and Datomic, to let the customer lead us toward a model that was meaningful to their business.

As we collected answers to these questions, we could quickly encode them in Clojure’s Lisp-like syntax:

(entity University
  "An institution of higher learning"
  (attr :university/shortname string identity #"[a-z]{2,4}"
    "Unique, human-readable abbreviation for this university")
  (attr :university/full-name string non-blank unique
    "Fully-expanded name of the university for public display"))

(entity Program
  "Something offered by a school or college culminating in
  a degree or certificate."
  (attr :program/full-name string non-blank
    "Fully-expanded name of the program for public display"))

(relationship offers
  "Every Program is owned by a single University"
  [Program :program/university University]
  [University :university/programs Program many])

Although this looks like Clojure code, it’s just data. Clojure’s edn reader and clojure.spec made it easy to parse and load this data into a Datomic database. Then it was a simple matter of programming to transform it into a variety of representations.

The very first artifact we deployed at 2U was an internal web app to explore the domain model, including a graph visualization of entities and relationships:

To gather feedback, we printed a poster-sized version of the same diagram, hung it in a corridor next to a stack of post-its, and asked everyone in the company to suggest improvements.

As we collected feedback, we learned that almost every team had a unique perspective on the business. Some of the most commonly-used terms had widely-divergent definitions. It was not exactly a blind-men-and-elephant scenario: None of the definitions was actually incorrect, but neither were any of them complete.

The fable of the blind men and the elephant: Each perceives only one part, on which he bases his (incorrect) conception of the whole animal. By contrast, one team's understanding of a domain concept may be entirely correct for that team's work, yet still inadequate to represent that concept for the business as a whole.

The fable of the blind men and the elephant: Each perceives only one part, on which he bases his (incorrect) conception of the whole animal. By contrast, one team's understanding of a domain concept may be entirely correct for that team's work, yet still inadequate to represent that concept for the business as a whole.

Many of our collaborators told us this was the most valuable part of the process for them, as it expanded their understanding of the business and their role in it. We were applying agile software development practices to a fundamentally cognitive task. The goal was not to produce software but to better understand the domain, helping 2U to learn about itself. The software behind the model and visualizations was merely a means to that end.

A Foundation in Data

Even while the model was still evolving, Datomic’s flexible approach to schema enabled us to start work on a database of domain knowledge. Since Datomic schemas are themselves expressed as data, we could automate the process of keeping the database in sync with the domain model.

;; Sample Datomic schema elements generated from domain model
[{:db/ident       :university/shortname
  :db/doc         "Unique abbreviation for this university"
  :db/valueType   :db.type/string
  :db/cardinality :db.cardinality/one
  :db/unique      :db.unique/identity
  :db/index       true}

 {:db/ident       :program/university
  :db/valueType   :db.type/ref
  :db/cardinality :db.cardinality/one}]

When data is the foundation, repetitive programming tasks can often be replaced by higher-leverage meta-programming. With Datomic, Clojure, and clojure.spec, we took an abstract model and generated a database. Adding Pedestal into the mix, we generated web-service interfaces for managing information about entities in the database. Using ClojureScript, Om, and React, we deployed a series of “mini apps,” rich interactive forms tailored to particular business roles, all sharing the common back-end.

Since 2U was already using Slack for inter-team communication, we integrated a Slack “bot” into various team channels, posting notifications and reminders with direct links into the associated web apps. All of this information flowed back into a dashboard view to help project managers track work-in-progress across the organization.

Lessons Learned

When we started, our stakeholders at 2U had hoped to arrive at a “canonical” data model for a new system which would be the “source of truth” for the rest of the organization. We set out to iterate towards a canonical model on which everyone could agree.

Of course, building consensus takes time. We were fortunate to be paired with an energetic 2U project manager willing to spend long hours asking questions and collecting feedback on our evolving data model. Even so, it took months to arrive at something we felt confident in. Fortunately, the flexibility of Clojure and Datomic allowed us to keep our models up-to-date as we learned about the business domain, even when that required revisiting some of our earliest decisions.

In the end, we realized there wasn’t any one “canonical” model. The various definitions of core business concepts were all equally valid and equally necessary. This led us to our final data model, in which the core entities were represented not as single objects but as faceted collections of views. Each team, each software system could shine a spotlight on one face of the entity, and the sum of those views made up the complete picture. The new services we developed with 2U allowed each team to continue working with the model that best served them while also bridging the communications gap between formerly-siloed systems.

Ultimately, we learned that “canonical” and “truth” are asymptotic targets. In any organization, teams tend to view the rest of the business through the lens of the system they interact with. A diversity of models is inevitable as each team adapts their understanding of the domain to fit the task at hand. For human beings, it’s a necessary cognitive shortcut. For software (and the people who write it) it’s a call to action: to adapt to reality rather than trying to reshape it.

How We Work: Iteration Zero

As a Project Coach at Cognitect, I get to work with awesome people. My colleagues are talented, passionate, and -- occasionally -- opinionated people. Along with software and other sorts of engineers we have people with backgrounds in music, physics, radio announcing and mathematics. But we have all found a common purpose here at Cognitect: Our customers. I also get to work with the even more diverse crew we are lucky enough to call our customers. So when I start a project the only thing I can be sure of is that everyone, everyone wants it to succeed.

What goes into a successful software project? We’ve all seen teams of great people tackle hard technical problems and triumph. Sadly, if you’ve been in the software business any length of time, you have also seen teams of great people stumble. At Cognitect we believe that software projects are mainly about people. Look around at the industry today and you can find hard technical problems, problems that can defeat even the brightest. But much more often project failures are people failures. Way more systems had been brought down by mismatched expectations and personality conflicts than by off-by-one errors or dueling library versions.

If pulling off a successful software project is all about people, then you need to pay attention to the people issues all through the project. Sadly that’s not how it usually is: Projects usually get much more attention in the middle and at the end than they do in the beginning. It’s only human nature: At the beginning of a project, the deadline is as far away as it ever will be. At the start people are generally relaxed, looking forward to a new challenge. It’s usually in the middle when commitments and decisions have been made and the pages are flying off the calendar that we tend to sit up and take notice. But after 23 years of running or coaching projects, I can tell you that the beginning of a project is when you have your biggest opportunity to put your project on the right path.

Before the Project Kick-Off

Have you ever been to a project kick-off meeting that was just a formality to introduce the team, and announce that the project started? Maybe we will vote on a cool code name and if we’re lucky there will be pizza. I’ve been to a lot of those project kick-off meetings and THEY DO NOT WORK.

As I say, the secret to software project success is the people. It’s getting a room full of individuals to work together as a team. And the secret to getting a team to work is to start before the project kick-off meeting. Step one is to figure out who should participate in the project kick-off meeting.

Who should participate in the project kick-off? Obviously, the people who are going to do the work, the developers, designers, architects and anyone else who is going to pitch in should all be there. But you have to cast your net wider than just the people doing the work. You need to ask the key question: Cui bono? Who benefits? Who are you doing this project for? And you have to follow the money. Who is paying? A project kick-off meeting cannot be productive without all the stakeholders and the project sponsor’s participation. Getting all the key players in a room together can be a pain, but if you can’t pull everyone together you may as well vote on the code name while you wait for the pizza.

Notice that I keep saying participate, not attend. Have you ever been in a meeting where a key stakeholder is “attending” the meeting but isn’t really fully participating or even paying attention? I have, and it’s frustrating and disrespectful. Worst of all it’s distracting, distracting to the folks who are really involved.

Eye Zero

At Cognitect, we start our projects with Iteration Zero or I-0 for short. We call it iteration zero because it’s the iteration before the first “real” iteration. The first day of i-0 is the project kick-off meeting. After that, the i-0 activity focuses on learning about the project and business needs, technical details, and the beginnings of a project plan. A typical i-0 lasts two to four days, though we have done them in as little as one day for a small, straightforward project and had one stretch over two weeks for a massive effort.

The critical thing on that first day is to make sure the project goals, values, and motivations are clear to everyone. The whole point is to facilitate good conversations and get everyone participating. I use the classic talking stick technique to make sure that everyone has a say and that no one person (and I’m looking at you, Senior Executive Manager and Chief Architect) dominates the conversation. To get to this shared understanding, I use the three project framework exercises. I borrowed these exercises from Doug DeCarlo many years ago. I was lucky enough to work with Doug -- the author of eXtreme Project Management -- some years ago. Doug was a mentor to me and in the years since I’ve tweaked the exercises to fit my needs.

 

 

Exercise 1: Who is doing what for whom?

The first exercise seems very simple: It starts with this incomplete sentence:

<Who> will <Do> <What> for <Whom>.

The goal is to fill in the blanks and come up with a one-line description of the project. Kick things off by asking everyone to take a few minutes to come up with their own replacements for the <Who>, <Do>, <What> and the <Whom>.

For example, one version of the finished sentence might be:

The Genesis Application Team (Who) will design & build (Do) a new sales lead management software application with existing clients migrated (What) for the products sales team at the Orange company (Whom)

Or it might be:

The Genesis Application Team (Who) will code (Do) a new sales lead management module (What) for the Orange company operations team to deploy (Whom)

Or:

The Genesis Application Team (Who) will fix (Do) the issues around sales lead management (What) for the Orange company senior management (Whom)


Don’t be surprised at how many versions of the sentence your team produces -- it’s just a starting point. The next step is to get the team to agree on a single version of the sentence. Teams often have the most trouble filling in the <Whom>, but I have seen people struggle over every single word of this simple sentence. But if the project is to succeed we need to know who is doing what for whom. Again, don’t let any one person dictate: This should be a team discussion.

 


Exercise 2. The project will be complete when….

The goal of the second exercise is to complete the following sentence:

The project will be complete when…

The idea here is not to list all of the features, tasks, and requirements. Instead, focus on the top-level goals, summed up in two to five bullet points. Something like this:

  • The project will be complete when each account executive can log into the application and review the latest invoice status for their clients.
  • The project will be complete when most User Acceptance Testing feedback is implemented and is ready to be deployed to Production.
  • The project will be complete when there are no known severity-1 or severity-2 defects.

Again, work on sentences as a team, and drive towards team consensus. Emphasize that these criteria are important: When there is a check mark next to all the items, you are done. Or to put it another way, the project isn’t done until that last item is checked off.

 

Exercise 3. Win Conditions

We’ve done fill-in-the-blank and complete-the-sentence so it must be time for a multiple choice question.: Here’s a list of things we would like to get from our project:

  • Schedule
  • Scope
  • Quality
  • Budget
  • Customer Satisfaction
  • Teamwork / Learning

Ask everyone to take a few minutes to think about what are the top three win conditions for them. Then ask them to share their picks along with what they think each condition means. Typically the top three win conditions will vary from person to person.

For example, one answer might be:

  1. Budget is the most important thing since we only have $50,000 to spend.
  2. Customer Satisfaction is the next most important thing -- we need to make our users happy!
  3. And then quality: No critical defects.

The point of the exercise is to focus everyone’s mind on what will make the project a success.

The universal first reaction to this exercise is I choose all of the above. Sadly, there are times when we need to choose, say budget over scope or scope over schedule. Nobody wants to build a system that is behind schedule or lacks important features. As we work on the project we will do our absolute best to ensure it is as good as it can be in every possible way. But in every project I’ve ever been involved in, there has been at least one moment where we have had to choose. And at those moments you need to know what is important. By getting the What’s More Important? discussion out on the table right at the beginning, we won’t find ourselves improvising an answer two days before the deadline.

An equally important goal is to make sure that everyone understands what each of the win conditions means. Sue’s definition of budget might mean within 10% our estimate while Carol’s might mean absolutely no more than $50K. Bob might think that hitting the scope goal means getting the minimum viable product up and running in the lab, while his boss might think scope means all of the features deployed on the corporate cloud. Before you walk out of the room on that first day, make sure the team agrees on what three most important win conditions are and what they mean.

At one of my previous jobs, I joined a team that was already three months into their project. One of the first things I did was run the win condition exercise. The team chose the following:

  1. Schedule
  2. Scope
  3. Quality.

That afternoon I asked the project sponsor, who hadn’t attended the first meeting, to pick her top three win conditions. Her list was this:

  1. Scope
  2. Quality
  3. Teamwork.

Notice that while the rest of the team thought that schedule was the most important thing, it didn’t even make the sponsor’s top three. Good thing you worked that weekend…

But the differences didn’t stop there: To the team, Scope meant this:

We must stick to the feature & functional requirements as defined.

To the project sponsor, Scope -- her top priority-- meant this:

The requirements are changing all the time, and so we need to keep track of them accurately and adjust accordingly.

Once this all came to light, the team realized for the first time that the scope can and is expected to change throughout the project. Something you might want to know.

This example also shows how important it is to check in on the results of the exercises while the project is going on. If the landscape has changed, it’s important to note that, adjust the project framework, and communicate the changes to ensure everyone involved remains on the same page.

A Framework for Success

As you can see the focus of the project framework exercises is on the people first, and the other aspects of the project second. The exercises trigger important conversations, realizations, and lead to a shared understanding. They get people asking questions and talking through issues that might otherwise not come up until it is too late, if at all.

Time and time again I’ve found these set of exercises to be the key to a successful project. Time and time again my clients and colleagues have found them to be very useful. Project kick-off is - emphatically - not just a formality. A great kick-off is the first step to making your project a success.

Works on My Machine: How We Work: Distributed

Working for a distributed company -- Cognitect is scattered across much of the United States and Europe -- does have its ups and downs. I love not having to commute. But I miss hanging out with my coworkers, live and in person. I love that my office is just upstairs, in that spare bedroom. But sometimes I wish I could put more distance between my job and the rest of my life.  I love that the Internet lets me talk to just about anyone, anywhere. And sometimes I wish I could throw my computer, complete with its bogged-down network connection out the window.

Working for a distributed company also means that I get asked "the question" a fair bit. Actually the question is really a family of questions: "What's it like?" is a common variation. So is "Isn't it hard to get things done?" Then there is "What skills do I need to work remotely?" and, of course "How do I talk my boss -- or potential boss -- into this?"

Cognitect opposes discrimination and HB2. #WeAreNotThis

Cognitect strongly values the rights of all citizens to equal treatment under the law.  We find it disheartening that the North Carolina General Assembly would pass, and our Governor would sign, a law that singles out LGBTQ persons for unequal treatment.  The law is heartless on its face; discriminatory and mean-spirited.  We share with other companies, organizations and individuals our dismay that North Carolina would be actively going backwards on issues of fairness and inclusion.  At Cognitect, we believe that no one should be discriminated against based on gender identity or sexual orientation whatsoever, and we will continue to act on those values regardless of what the NCGA does. #WeAreNotThis

 

Sisyphus on Fire: The Coming Agile Dystopia

Hear me now, for I have seen things. A world of ash and soot. Bodies and brains toiling day after day, pushing boulders up steep mountainsides. When these burdens are overcome, when the peak is crested and victory achieved there is neither reward nor respite. Burning disappointment is the only prize, a hot frustration as the mighty stones are sent careening once again to the depths below in endless repetition. This, my friends is the future of agile development. For the past several years I have consulted with over 20 companies - from tiny startups to giant global enterprises. I have seen the inner workings, corporate cultures, and development practices across a wide range of industries. The trends are disturbing. I have seen the future and it is nasty and brutish. We can and must alter course. Resistance is NOT futile.

Why agile?

In the beginning, before agile, was the waterfall. We were doing multi-year projects, implementing reams of printed specifications, missing ship dates and blowing budgets. Most damning of all, we created products that didn’t meet the customer's needs. As big deadlines loomed, teams fell into death marches, with horrible periods of late nights and heated arguments, broken builds and streams of bugs without end. A game of lose-lose, the epitome of action not equalling progress. 

Then a group of survivors hatched a plan of escape. Let’s break the cycle and put real customer satisfaction first. Let’s welcome changing requirements. Let’s deliver early and often. Let’s  communicate across teams and reflect upon ways to improve. Above all, let’s promote sustainable development - a manifesto for practices that would enable sponsors, developers, and users to maintain a constant pace. Indefinitely. It made so much sense. And a few teams have indeed managed to succeed and thrive. Yet here we are, many of us, every 2 weeks. Surrounded by pizza boxes and despair. The points. The pressure. The last minute check-ins, the skimping on QA. The tortured definitions of “complete.” The pronouncements of SUCCESS (yay!) or FAILURE (boo!) every fortnight. OMG it’s happening again. And again. And again! Every 2 weeks. WTF?

All around me are familiar faces
Worn out places, worn out faces
Bright and early for the daily races
Going nowhere, going nowhere...

— Roland Orzabal, “Mad World”

Many organizations took their few, real deadlines - a Beta version, big conference demo, or the major public release - and rolled the incumbent pressure and "all hands on deck" mentality into a twice monthly cannonball run. Remember those large waterfall endeavors? Even if the product was too little or too late, there was at least a time to recharge at the end. To rest and to dream, a period of calm to plan a new route forward. But in many “agile” organizations, there is no rest and no peace. Just another boulder to push up another hill. Say what you will about the horrors of a waterfall death march, these teams at least got to sleep once they were dead. In stark contrast, many agile teams are rousted from their dirt naps before pushing a single daisy. The “big deadline” is now every sprint’s end. This is endless war. This is not about “keeping commitments.” This is not what agile was meant to be. This is not even madness. This. Is. Sparta

Many so-called “agile” organizations have mistaken the process for the product. The daily “how confident are you this card will be done today?” The inevitable “pretty confident unless things don’t go as planned.” This is a bogus dance. This is project management theater. Even the notion of “delivering working software frequently” to maintain a healthy feedback channel is being corrupted. Rather than a brief reflection on progress toward the finish line, most end-of-sprint reviews fixate on “how well did we do compared to our guesses 2 weeks ago?” 

The truth is that we are running a marathon here, people. Treating it like a series of independent races, a chain of back-to-back “sprints” is not the way to win a marathon, nor is it the way to build software. To be truly agile we iterate, we take stock of progress toward the finish line and we adjust as we go. No particular mile is more important (or detached) from any other. Don’t start, stop, start, stop, start, stop. Focus on the end goal and run, dammit, RUN.

Enough with the burndown charts. Enough with the sprint-end whiplash. Establish clear priorities and guidelines - set a vision of what success looks like. Slow occasionally to take stock and adjust as needed, but keep moving forward at a steady pace. 

Let’s stop rolling boulders up and down meaningless hills, and let’s start moving mountains. 

Distinguished Magic

"The future is already here — it's just not very evenly distributed." - William Gibson

Decision, decisions. Users are screaming to pay money for your service, but your current infrastructure is going to pop like a balloon if you let any more sign up. Or your cumbersome deployment process has slowed enterprise sales to a crawl as your small support team runs themselves ragged. Is your competition widening the feature gap due to your lengthy test and release cycle? How to process all that customer data into useful trends? Your decisions today will mean soaring profit or smoldering ruin. How will you solve these problems? Traditional answers to these questions lead many organizations down roads from which they never return. But in the future? Roads? Where we’re going, we don’t need roads...  

You may have tried incremental approaches that were so incremental that nothing changed. You may have attempted full system overhauls, only to have the budget and schedule blow up in your face while the problems remained - or got worse. Or you may not even know where to start.

Usually the problem isn’t a lack of skill. It’s not a lack of commitment. Or an aversion to hard work. No, what’s holding you back is a failure of imagination. Most people self-censor (whether they are aware of it or not), limiting themselves to what they think is possible. We tend to stick to what we know how to do or have seen others do. Because we cannot see a path to the future we would like to see, we fail to see that future at all.  Ironically, when we come across someone who did manage to transcend these mental limits, we say that they “saw what others could not see”. Actually these visionaries see less than others - they don’t  see the walls that block others' view of a better world. They don’t impose restrictions on the future. The difference is that they believe in magic.

So forget what you and the people around you already know how to do. Forget what you have time to do. Forget what you have the budget to do. Forget what you think others will let you do. Instead, ask yourself:

If I could wave a magic wand and make the solution appear, what would it look like? 

Ignore how you are going to get there and just think about what things will look like once you arrive. You need to believe.

Working backwards from a “magical” solution will result in a better real world solution than you can get to by crawling forward on your belly around obstacles you find along the road.

"Any sufficiently advanced technology is indistinguishable from magic." — Arthur C. Clarke

With the future of your business at stake, you need a new approach for solving your tough problems.  At Cognitect, we help teams break through preconceptions that hold them back. To envision systems unrestrained by the limits of traditional development tools, to design the right solution to grow your business. The results sometimes seem magical - our modern tech stack of Clojure and Datomic enables development of flexible cloud-based systems at significantly less cost than traditional approaches. Technologies now exist that enable small teams to build enterprise-grade solutions on schedules and budgets that not long ago really would have required a pointy hat and cloak.

As noted by former Chief Architect of Netflix’s Cloud Services Adrian Cockcroft, Clojure allows developers to produce "ridiculously sophisticated things in a very short time.” And immutable data - a hallmark of the Datomic database - grants complete auditability of values and transactions from any previous time, plus the ability to model potential scenarios without committing changes. Relive the past? Peer into the future? Datomic may not actually let you travel through time, but the results are just about indistinguishable.

You can attempt to solve your problem with the traditional approach. But are you planning the right solution? Or just what you can put together with the tools and ideas inside the barriers that have accreted across your organizational field of vision? Or will you forget what is and imagine what could be, without preconceived limitations and constraints? Don’t settle for a faster horse - demand a rocket ship. This may be the path less traveled, but it will make all the difference.

Story Points Must Die: A Spacetime Adventure

Long ago beyond the stars a civilization rose from dust and puddles to spread the gospel of “agile software development” across many worlds. Over time these worlds diverged into two distinct tribes, known as “Team A” and “Team B” (for they were truly unimaginative beings in the ways of names). Both tribes prospered, delivering software throughout the galaxy, but over many generations Team A grew to believe the source of their success was a mysterious force known as “Story Points”. Team A evolved an elaborate ritual that they performed during their traditional bi-weekly gatherings known as “Sprint Planning”.  Team A believed these rituals allowed them to accurately predict the future. Abandoning the process of deep thought and discussion, Team A eventually focused their entire development process on these “points”, which for some reason existed only in quantities of 1, 2, 3, 5, and (very rarely) 8.

While both tribes continued to ship software, fealty to this numeric master began to visibly distort Team A’s demeanor and appearance until travelers from other galaxies could easily tell which of the two tribes they had encountered. Those from Team A came to be known as victims of "Fibonacci’s Curse". 

Let us go now, across space and time in the blink of an eye to gaze upon two such parallel development efforts...

Extremely Honest Programming

The true cost of new features is one of the toughest things for Product Managers to understand. When you've got a full-time team of salaried developers at your side, it's easy to forget the real monetary costs of writing code. The result? Blown schedules, burned out development teams, and lackluster customer response to the "big release". This is where Extremely Honest Programming comes in. EHP is all about net results.

s/DevOps/Etsily/

Dave Thomas urges us to kill the word "agile". Many of the original leaders in agile software development have now disavowed the term. Why? In short, it's been devalued. Literally, stripped of its values. Today, the word is used as a capitalize noun, as in the hideous phrasing, "Do Agile," rather than as the lowercase adjective it was meant to be.