Through this series, we've talked about antifragility, disposable code, high leverage, and team-scale autonomy. Earlier, we looked at the benefits of team-scale autonomy: It breaks dependencies between teams, allowing the average speed of the whole organization to increase. People on the teams will be more fulfilled and productive, too. These are nice benefits you can expect from this style, but it's not all unicorns and rainbows. There is some very real, very hard work that has to be done to get there. It should already be clear that you must challenge assumptions about architecture and development processes. But we also need to talk about critical issues of failure domains and safety.
Like many developers, I’ve spent a lot of time thinking about objects. I started doing object-oriented programming (OOP) in 1989, and was arguably still doing it up until 2012. The concept of objects arose in GUIs and simulations, where it is natural to think about sending message to things. Over time though, the idea of messages took a back seat to the idea of locking data up in little containers where each class provides a namespace of functions to operate on it.
You’re probably familiar with the concept of the two pizza team. This is Amazon founder and CEO Jeff Bezos’ rule that every team should be sized no bigger than you can feed with two large pizzas. At Cognitect, we take this concept one step further: the one chateaubriand team—with sharper tools you can afford to spend the extra money on better food.
The two pizza team is an important concept. The idea is that smaller teams are more self-sufficient and typically have better communication and a greater focus on getting things done because they eliminate dependencies. Every dependency is like one of the Lilliputian ropes that ties Gulliver to the beach. Each one may be simple to deal with on its own, but the collection of 1,000 tiny threads keeps you from breaking free. Eliminate dependencies and your teams move faster. Encourage autonomy and you allow innovation.
I'm happy to introduce today clojure.spec, a new core library and support for data and function specifications in Clojure.
Cognitect hit a pretty impressive milestone this week that deserves some celebration - we released our 100th Cognicast (the Cognitect podcast for those of you not yet in the know). As our esteemed host Craig Andera likes to say, he "just decided to do it one day, and we have so far failed to stop." But I think that shortchanges the quality of what's come out of it.
You wouldn’t use a toothpick to wear down a mountain. You wouldn’t wear your trainers instead of your racing spikes at a track meet. So why would you use a hand saw to build an infrastructure when you need a scalpel for surgical precision?
Up to this point in the series, we’ve talked about embracing failure, experimenting more and moving faster. We've talked about the "why," now it's time to address the "how."
The word "antifragile" may be recent, but some of the concepts are ancient. In "Art of War", the renowned general, strategist and tactician Sun Tzu's states, “…water shapes its course according to the nature of the ground over which it flows...” In an antifragile organization, we want to explore opportunities so resources flow like water into the things that are working, and abandon those that are not.
Just as water retains no constant shape, there are no constants in an antifragile organization and IT infrastructure. To flow like water, you must be able to shift people and teams easily, create teams and systems easily, be able to tear down systems and remove people from working on projects that aren’t working. This requires an architecture that allows you to act locally but think globally. Some organizations are pursuing microservices to this end. Complex applications are composed of small, independent processes that focus on doing a single responsibility. With microservices, developers decouple software into smaller single-function units that can be replaced individually.
Cognitect strongly values the rights of all citizens to equal treatment under the law. We find it disheartening that the North Carolina General Assembly would pass, and our Governor would sign, a law that singles out LGBTQ persons for unequal treatment. The law is heartless on its face; discriminatory and mean-spirited. We share with other companies, organizations and individuals our dismay that North Carolina would be actively going backwards on issues of fairness and inclusion. At Cognitect, we believe that no one should be discriminated against based on gender identity or sexual orientation whatsoever, and we will continue to act on those values regardless of what the NCGA does. #WeAreNotThis
"The master has failed more times than the beginner has even tried."
- Stephen McCranie
In my earlier post Minimize Risk by Maximizing Change, I talked about embracing change and touched on three paths towards antifragility. Here’s a bit more on game day and chaos, and how they relate not only to software development but to the organization as a whole.
The accounting department classifies software development work as a capital expense. That means, at least according to accounting, new software represents new capital investment that should increase productivity. This highlights a big divergence in the way accounting views software versus the way we should look at it.
We used to have a saying, “KLOCs kill.” The more lines of code, the more risk you have. As your system gets bigger it gets more complicated and difficult to work with. Code has a carry cost... you have to keep maintaining it. It has obsolescence risk. Undeployed code is exactly like unfinished automobiles: nobody pays you for it. Maintaining it is a liability not an asset. Excess code is a boat anchor that will weigh you down until you drown.
I once worked for a startup called "Totality." Our business was outsourced web operations for companies that either didn’t want to invest or lacked the skills to build and staff their own24x7 operations center. We handled all the production management, change management, incident management—essentially the entire ITIL (IT Infrastructure Library) suite of processes.
During my time at Totality, we observed that nearly 50% of all our software outages happened within 24 hours of a software release. Since we were on the hook for uptime, but not new features, our response was obvious: Stop touching things!
No Silver Bullet
In his paper "No Silver Bullet — Essence and Accidents of Software Engineering,” Fred Brooks argues that "there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity." Brooks, a Turing Award winner and the author who also brought us “The Mythical Man-Month: Essays on Software Engineering,” also states that "we cannot expect ever to see two-fold gains every two years" in software development, like there is in hardware development (Moore's law). In other words, there is no silver bullet.
Some things benefit from shocks; they thrive and grow when exposed to volatility, randomness, disorder, and stressors...
We all recognize the traditional approach to risk management: prevent problems from ever occurring. If a process fails one time, institute a review step to make sure it never fails that way again. The next level is resilient systems that can cope with change and survive failures. I propose that the new normal is the exact opposite of preventing problems: survive problems, and if you’re not getting enough problems from the outside, make problems for yourself! This new approach is still counterintuitive in most organizations.
It seems that everyone is talking about microservices as the road to salvation. Why? Why now? The usual explanation is just that it's an architecture style that encourages flexibility and makes a company more competitive. However, like agile development, patterns, and object-oriented development before it, the microservices architecture will not deliver its promises to everyone. If you understand what makes it work, you will know when and how to apply it successfully. Let us look deeper to see what forces really underlie this evolution.
Clojure is a "robust, practical and fast programming language" whose original design goals were aimed at giving developers "succinctness, flexibility and productivity". It turns out that most organizations today are interested in achieving those very goals - as the landscape evolves, they need technologies that help them move faster, respond more effectively, and to take advantage of new architectures and infrastructure. In short, they need simple tools that help them be more agile.
In this, the sixth consecutive year of the State of Clojure survey, we are seeing the evidence that Clojure is moving across the chasm from a niche tool used by explorers and hobbyists to becoming a critical part of the commercial development landscape, and that its adoption is spreading both broadly among companies, but also more deeply within them.
Before we get to the results, we'd like to first thank everyone who took the time to respond to the survey this year. Response was up more than 75% this year, with 2,445 total responses to analyze. This kind of data is invaluable to the people and organizations using (or considering) Clojure today. As always, we'd also like to thank Chas Emerick for starting the survey back in 2010.
We are pleased to announce the release of Clojure 1.8.
Some of the new features for 1.8 are:
- More string functions in clojure.string (portable to ClojureScript): index-of, last-index-of, starts-with?, ends-with?, includes?
- Compiler direct linking - improves performance and startup time
- Socket server and socket server REPL - adds the ability to allow remote Clojure REPL connections
For more information, see the complete list of all changes since Clojure 1.7 for more details.
Thanks to all of those who contributed to Clojure 1.8 (first time contributors in bold):
- Alexander Yakushev
- Alex Miller
- Alex Redington
- Alf Kristian Stoyle
- Ambrose Bonnaire-Sergeant
- Andrew Rosa
- Andy Fingerhut
- Andy Sheldon
- Aspasia Beneti
- Blake West
- Bozhidar Batsov
- Daniel Compton
- Erik Assum
- Gary Fredericks
- Ghadi Shayban
- Gordon Syme
- Howard Lewis Ship
- Jean Niklas L'orange
- Jeremy Heiler
- Jonas Enlund
- Jozef Wagner
- Karsten Schmidt
- Kevin Downey
- Mark Simpson
- Michael Blume
- Nahuel Greco
- Nicola Mometto
- Nikita Prokopov
- Nola Stowe
- Ragnar Dahlén
- Ralf Schmitt
- Rich Hickey
- Russ Olsen
- Shogo Ohta
- Steve Miner
- Stuart Halloway
- Timothy Baldridge
- Tsutomu Yano
- Yanxiang Lou
We're happy to announce that Clojure/west 2016 will be April 15-16th in Seattle, Washington. The event will take place at the Seattle Marriott Waterfront, close to Pike Place Market, the Space Needle, and other downtown Seattle attractions.
- free admission to the conference
- US travel reimbursement (or $550 stipend if international).
We are also now seeking sponsors for the 2016 event! Clojure/west is a great opportunity to reach 500 Clojure developers with hiring, products, or other developer-oriented outreach. If you're interested in being part of the event please contact us to see how we can collaborate on the best option for your team.
Early bird registration for Clojure/west will open later in January - stay tuned!
It's time for the annual State of Clojure Community survey!
If you are a user of Clojure, ClojureScript, or ClojureCLR, we are greatly interested in your responses to the following survey:
The survey contains four pages:
- General questions applicable to any user of Clojure, ClojureScript, or ClojureCLR
- Questions specific to the JVM Clojure (skip if not applicable)
- Questions specific to ClojureScript (skip if not applicable)
- Final comments
The survey will close December 18th. Afterwards we will release all of the data and some analysis.
Many thanks to Chas Emerick for starting and keeping this survey going for so many years!
Hear me now, for I have seen things. A world of ash and soot. Bodies and brains toiling day after day, pushing boulders up steep mountainsides. When these burdens are overcome, when the peak is crested and victory achieved there is neither reward nor respite. Burning disappointment is the only prize, a hot frustration as the mighty stones are sent careening once again to the depths below in endless repetition. This, my friends is the future of agile development. For the past several years I have consulted with over 20 companies - from tiny startups to giant global enterprises. I have seen the inner workings, corporate cultures, and development practices across a wide range of industries. The trends are disturbing. I have seen the future and it is nasty and brutish. We can and must alter course. Resistance is NOT futile.
In the beginning, before agile, was the waterfall. We were doing multi-year projects, implementing reams of printed specifications, missing ship dates and blowing budgets. Most damning of all, we created products that didn’t meet the customer's needs. As big deadlines loomed, teams fell into death marches, with horrible periods of late nights and heated arguments, broken builds and streams of bugs without end. A game of lose-lose, the epitome of action not equalling progress.
Then a group of survivors hatched a plan of escape. Let’s break the cycle and put real customer satisfaction first. Let’s welcome changing requirements. Let’s deliver early and often. Let’s communicate across teams and reflect upon ways to improve. Above all, let’s promote sustainable development - a manifesto for practices that would enable sponsors, developers, and users to maintain a constant pace. Indefinitely. It made so much sense. And a few teams have indeed managed to succeed and thrive. Yet here we are, many of us, every 2 weeks. Surrounded by pizza boxes and despair. The points. The pressure. The last minute check-ins, the skimping on QA. The tortured definitions of “complete.” The pronouncements of SUCCESS (yay!) or FAILURE (boo!) every fortnight. OMG it’s happening again. And again. And again! Every 2 weeks. WTF?
Many organizations took their few, real deadlines - a Beta version, big conference demo, or the major public release - and rolled the incumbent pressure and "all hands on deck" mentality into a twice monthly cannonball run. Remember those large waterfall endeavors? Even if the product was too little or too late, there was at least a time to recharge at the end. To rest and to dream, a period of calm to plan a new route forward. But in many “agile” organizations, there is no rest and no peace. Just another boulder to push up another hill. Say what you will about the horrors of a waterfall death march, these teams at least got to sleep once they were dead. In stark contrast, many agile teams are rousted from their dirt naps before pushing a single daisy. The “big deadline” is now every sprint’s end. This is endless war. This is not about “keeping commitments.” This is not what agile was meant to be. This is not even madness. This. Is. Sparta.
Many so-called “agile” organizations have mistaken the process for the product. The daily “how confident are you this card will be done today?” The inevitable “pretty confident unless things don’t go as planned.” This is a bogus dance. This is project management theater. Even the notion of “delivering working software frequently” to maintain a healthy feedback channel is being corrupted. Rather than a brief reflection on progress toward the finish line, most end-of-sprint reviews fixate on “how well did we do compared to our guesses 2 weeks ago?”
The truth is that we are running a marathon here, people. Treating it like a series of independent races, a chain of back-to-back “sprints” is not the way to win a marathon, nor is it the way to build software. To be truly agile we iterate, we take stock of progress toward the finish line and we adjust as we go. No particular mile is more important (or detached) from any other. Don’t start, stop, start, stop, start, stop. Focus on the end goal and run, dammit, RUN.
Enough with the burndown charts. Enough with the sprint-end whiplash. Establish clear priorities and guidelines - set a vision of what success looks like. Slow occasionally to take stock and adjust as needed, but keep moving forward at a steady pace.
Let’s stop rolling boulders up and down meaningless hills, and let’s start moving mountains.
The Clojure community is full of talented writers and valuable experience, and together we can create great documentation for the language and the ecosystem. With that in mind, we are happy to announce a new initiative to replace the existing http://clojure.org site. The new site will contain most of the existing reference and information content but will also provide an opportunity for additional guides or tutorials about Clojure and its ecosystem.
The new site content is hosted in a GitHub repository and is open for contributions. All contributions require a signed Clojure Contributor Agreement. This repository will accept contributions via pull request and issues with GitHub issues. The contribution and review process is described in more detail on the site contribution page.
We are currently working on the design elements for the site but if you would like to suggest a new guide, tutorial, or other content to be included on the site, please file an issue for discussion or create a thread on the Clojure mailing list with [DOCS] in the subject. There will be an unsession at the Clojure/conj conference next week for additional discussion. This is the beginning of a process, and things will likely evolve in the future. In the meantime, we look forward to seeing your contributions!