Technology The New Normal How We Work Customer Stories Testing | All Topics

Cognitect Support for Clojure Events


Cognitect (formerly Relevance) has run many Clojure events since 2010. The first Conj was put together for the rapidly growing Clojure community in the early days of Clojure and it became an annual event run primarily on the east coast of the US. In 2012 and 2013, Alex Miller started and ran Clojure/west as an independent event, later becoming a Cognitect event after Alex joined Cognitect. EuroClojure was created by the European Clojure community and run by Marco Abis, then transitioned into a Cognitect event in 2015.

Throughout, the goal has to been to provide places for Clojurists to gather and talk about their favorite language, provide opportunities to learn and share tools, techniques, and successful experiences. In the last few years, we've seen the rise of many great new Clojure community events in London, Berlin, Amsterdam, Helsinki, Bangalore, Leuven, Toronto, New Orleans, and some online as well.

Moving forward, Cognitect will continue to produce the Clojure/conj event on an annual basis every fall. We will discontinue the EuroClojure and Clojure/west events and increase our sponsorship of community conferences, provide speakers as desired, and offer other support as needed for worldwide Clojure community events. If you are a Clojure conference organizer, please contact with any questions or requests.

We look forward to seeing you at Clojure events wherever they happen!

State of Clojure 2018 Results

Welcome to the annual State of Clojure 2018 survey results! Thanks so much for taking the time to check in and provide your feedback. We are very fortunate to have data for some of these questions going all the way back to 2010, giving us a long view on how the data is trending. This year, we had 2325 respondents, about the same as last year.

Rapid Uptake of Clojure 1.9

With the release of Clojure 1.9 in December, we expected to see a shift in version usage, and we did. 72% of developers are already using it with about 60% still using Clojure 1.8 as well. Only a small (6%) number of developers are still using versions 1.7 or older.


We also keep an eye on JDK usage. Uptake of Java 1.9, released last year, has been a bit slower with only 29% adopting Java 1.9 so far and 88% of developers using Java 1.8. Only 6% of developers are using Java 1.7 and less than 1% are still using Java 1.6.

In the editor/IDE world we saw some consolidation this year with both Emacs (50%) and IntelliJ/Cursive (29%) making gains. All other editors saw decreases, although there is still a lot of interesting innovation happening around Atom and VS Code, which was not included but saw a lot of mentions in the comments (~5% of total respondents) - will definitely add next year!


In the ClojureScript world, Figwheel continues to dominate as a critical part of most ClojureScript developer's REPL workflow (76%). Clojurists Together is a new community effort to support open source projects in the community and they have been funding work on Figwheel among other projects. Lumo was a new REPL option this year and made a strong showing of 12%.


In CLJS target environments, we saw an increase of +6% targeting Node (to 29%) and +4% targeting Lambda (to 13%) - both things to watch.

In the build tooling world, the entry of the clj tool is driving a lot of reevaluation and change right now. With so many things in flux, this area is sure to evolve significantly in 2018 and it will be interesting to see where we are in 2019. One important omission in the choices this year was shadow-cljs. There were a lot of mentions in the comments and it's clearly an important tool for many to build and deploy - we'll be sure to add it next year.

Interest surging from JavaScript programmers

When we look at which language communities people are coming from, those answers have been remarkably stable for years, but there was significant movement this year for JavaScript (which vaulted over both Python and Ruby). Clearly people are finding ClojureScript (and its strong resonance with React) as an interesting and viable alternative to JavaScript.


As to where Clojurists hang out, we saw significant increases in use of Reddit (+5%) and Slack (+4%) and some decreases in use of the Clojure mailing lists, IRC, and attendance at both in-person and on-line conferences. One new choice added this year was the ClojureVerse Discourse server - it seems to be a useful midpoint between Slack (high volume live chat) and mailing lists (low volume asynchronous discussion). This was a new option yet 17% of respondents reported using it.


Clojure and ClojureScript used in many domains and industries

One of the things we are always watching is the trend of people using Clojure for their day-to-day work. This year, we continued to see about 2/3 of respondents using Clojure for work (compare that to the very first survey back in 2010 when less than 1/3 were doing so). Web development has always been the most dominant domain - in 2010, 53% were doing web dev and these days fully 82% of Clojure devs are involved in some kind of web development (not surprising given how many Clojure devs are using both Clojure and ClojureScript together).


When looking at the industries using Clojure, we added a few choices this year based on prominent results in last year's "Other" category - entertainment (3%), energy/utility (2%), automotive/manufacturing (2%). We also saw a noticeable increase (+3%) in Financial services. Perhaps due to the new choices, we saw small decreases in the largest and most generic categories, enterprise software and consumer software. 


Interest in hiring stays strong

There are several questions about how Clojure and ClojureScript should change or be prioritized for improvement. The results are largely similar to prior years, although the question format changed a little making it hard to directly compare every detail. The top result is clearly error messages though - while spec has started us down a road, that is still a work in progress which will continue this year. Many people have been using the Expound library for taking spec error output and making the data easier to read.

Hiring and staffing is always an interesting one to watch and that increased this year. We often see the seemingly contradictory dual complaints of companies that need more people and developers that have a hard time finding positions. To a large degree this is either a mismatch in the geographic distribution of jobs and people and/or a mismatch in needs and skill levels. It has been very encouraging to see so many large teams growing and hiring of late though.

The need for more docs and tutorials is also one that has gone up and down over the years and seems to be up again this year. While there are a wealth of resources for new Clojure developers now in every format, it is also sometimes difficult for people to find just the right resource for their experience level and need. There have been many good discussions lately about this and lots of active work in the community.

In general, there have been so many new tools, learning resources, companies, etc of late that it's hard to keep up - 2018 is going to be a great year for Clojure!

Check out the data

If you'd like to dig into the full results, you can find the complete set of data from this and former years here:

Note that we are doing the survey about every 14 months so the last survey occurred in late 2016 rather than 2017.

Thanks again for being part of the community!

2018 State of Clojure Community Survey Now Open


It's time for the annual State of Clojure Community survey!

If you are a user of Clojure or ClojureScript, we are greatly interested in your responses to the following survey:

State of Clojure 2018

The survey contains four pages:

  1. General questions applicable to any user of Clojure or ClojureScript
  2. Questions specific to JVM Clojure (skip if not applicable)
  3. Questions specific to ClojureScript (skip if not applicable)
  4. Final comments

The survey will close February 9th. All of the data will be released in February. We are greatly appreciative of your input!


Clojure 1.9


Clojure 1.9 is now available! 

Clojure 1.9 introduces two major new features: integration with spec and command line tools.

spec (rationaleguide) is a library for describing the structure of data and functions with support for:

  • Validation
  • Error reporting
  • Destructuring
  • Instrumentation
  • Test-data generation
  • Generative test generation
  • Documentation

Clojure integrates spec via two new libraries (still in alpha):

This modularization facilitates refinement of spec separate from the Clojure release cycle.

The command line tools (getting startedguidereference) provide:

  • Quick and easy install
  • Clojure REPL and runner
  • Use of Maven and local dependencies
  • A functional API for classpath management (tools.deps.alpha)

The installer is available for Mac developers in brew, for Linux users in a script, and for more platforms in the future.

For more information, see the complete list of all changes in Clojure 1.9 for more details.


Thanks to all of the community members who contributed to Clojure 1.9 (first time contributors in bold):

Adam Clements
Andy Fingerhut
Brandon Bloom
Cameron Desautels
Chad Taylor
Chris Houser
David Bürgin
Eli Lindsey
Gerrit Jansen Van Vuuren
Ghadi Shayban
Greg Leppert
Jason Whitlark
Johan Mena
Jozef Wagner
Lee Yen-Chin
Matthew Boston
Michael Blume
Michał Marczyk
Nicola Mometto
Ruslan Al-Fakikh
Steffen Dienst
Steve Miner
Yegor Timoshenko
Zhuang XiaoDan

State of Clojure 2016 Results and Analysis

Welcome back to the annual State of Clojure survey results. This year we held steady in our response rate as 2,420 of you took the time and effort to weigh in on your experience with Clojure - as always, we appreciate that time and effort very much. And, as always, thanks to Chas Emerick for starting this survey 7 years ago.

Clojure (and ClojureScript) were envisioned as tools that could make programming simple, productive, and fun. They were always aimed squarely at the working developer - someone being paid to solve complicated problems who needed to focus more on the solution and less on the unnecessary complexity surrounding it. While we love the academics, open source developers, and hobbyists who have flocked to Clojure, we are always happy to see signs of commercial adoption.

Last year, we had an outright majority of users (57%) using Clojure at work. This year, that number accelerates up to 67%.

Commercial Clojure use is for products, not just internal tools

A whopping 60% of respondents who use Clojure at work are building applications for people "outside my organization". We changed the wording of the answers to this question from the 2015 survey, so a direct head-to-head comparison isn't possible. However, in 2015, fully 70% of respondents said their use was for "personal" projects, while 42% said "company-wide/enterprise". This year, only 5% answered "just me". Even without the direct results comparison, the data shows a dramatic shift towards building products.

This year we also introduced a new question, asking what industry or industries people develop for. For commercial users, "Enterprise Software" was the leader (at 22%), followed by "Financial services/fintech", "Retail/ecommerce", "Consumer software", "Media/advertising", and "Healthcare". Everything else was at under 5% reporting. When we dig deeper and look at each of those industries in turn, we find that within each one, "outside my organization" is still the most common answer. In fact, only in "Financial services/fintech" do internal tools come within 15% of "outside my organization".

Clojure users are adopting the public cloud

Last year, 51% of respondents said they were deploying into the public cloud. This year, that number is up to 57%, coming almost entirely at the expense of "traditional infrastructure" (private/hybrid cloud was essentially unmoved). Recently, rescale released a report estimating that "we are in fact only at about 6% enterprise cloud penetration today" ( Clojurists in the workforce are considerably ahead of this curve, if true.

There is, unsurprisingly, a heavy correlation between use of the public cloud and developing applications for use "outside my organization". The use of the public cloud also skews heavily towards smaller organizations (companies of fewer than 100 people make up 70% of the public cloud group, while only 55% of the "traditional infrastructure" fell into that category).

There were only two industries where traditional infrastructure dramatically beat public cloud: Government/Military (which seems obvious) and Academia (which seems sad, although it could be a reflection of universities' sunk investment in infrastructure).  And only Telecom had a majority of respondents indicating "private/hybrid", which is almost certainly a reflection of the fact that hybrid cloud offerings are, by and large, products from the Telecom sector.

Clojure has penetrated all kinds of companies, not just startups

If you look at the spread of response for size of organization, while there is a clear winner (11-100), the split is fairly even otherwise. A full 17% of responses were from companies of 1000+ people.

Web development and open source development are the dominant two domains regardless of company size, but coming in at a strong #3 is "building and delivering commercial services", except when you look at responses from 1000+ companies, in which case "enterprise apps" unsurprisingly moves ahead.

"Enterprise software" is the #1 industry regardless of company size. However, #2 is quite distinctly different across sizes -- in smaller companies (< 100 employees), "consumer software" is the strong #2, whereas for companies > 100 employees, financial services is the dominant #2.

(An interesting aside: most industries show a normal bell curve, with most respondents coming from the middle two categories, 11-100 and 101-1000. For example: 

Only two industries show the inverted bell curve, with the most respondents at the edges -- Academia, and Government/Military.

You will note that these are the two industries where "traditional infrastructure" also dominates, so the distribution of respondents either being from the largest [most conservative] and smallest [most disruptive] paints an interesting picture of how industries change.)

One of the biggest barriers to adoption is corporate aversion to new technologies

As was true the last two years, error messages and "hiring and staffing" are the top 2 reasons given for "What has been most frustrating or has prevented you from using Clojure more than you do now?" though both have fallen several percent since then. Interestingly, "Need docs/tutorials" has jumped from #5 in 2015 to #3 now, which corresponds well with a continuing growth of new entrants into the community.

When you break down respondents by size, each category is relatively uniform with one glaring exception: for some reason, companies of 100-1000+ people have a problem with the lack of static typing (it is a strong #3 in that cohort). Everyone else has a carbon copy distribution of the overall answers. When you look by industry, the "enterprise software" crowd would clearly benefit from more tools and a better IDE experience.

What we found fascinating was drilling through the free answer portion of the responses to this question. Next year, we'll be adding a new possible answer: "corporate aversion to new technologies". If it was captured as one of the main responses, it would come in #2 or #3 overall. We clearly have work to do as a community to arm the technologists who wish to adopt Clojure with the materials and support they need to overcome internal inertia or resistance. That's an area we'd love to both see more people contributing, but also letting us at Cognitect know what else we could provide that would be useful.


When you dig into these numbers, you see a technology that has been accepted as a viable tool for crafting solutions across industries, company types and sizes, and target domains. As you might expect, adoption of Clojure seems closely correlated with the adoption of other new technologies, like the public cloud, and Clojure is beset with some of the same headwinds, like corporate aversion to new things. We are encouraged by the maturation of the community and of the ability of the technology and its adherents to tackle the hard problems of commercial software development.

Detailed Results

In addition to the big themes above, this section highlights a few of the more interesting results for specific questions in the survey. For details on all questions, see the full results.

Which dialects of Clojure do you use?

The interesting detail here was that the percentage of respondents using ClojureScript rose yet again, such that 2/3 of users are now using both Clojure and ClojureScript together (this has continually risen from about 1/2 3 years ago):

Clojure increasingly delivers on the promise of a single unified language stack that can be used to cover an entire application.

Prior to using Clojure, ClojureScript, or ClojureCLR, what was your primary development language?

We've changed the way this question is asked and the options provided several times so it's difficult to assess trends. However, it's clear that developers come to Clojure either from imperative/OO languages (Java, C#, C/C++) or from dynamic languages (Ruby, Python, JavaScript, etc) with only small numbers coming from functional programming languages like Scala, Common Lisp, Haskell, Erlang, etc.

What is your *primary* Clojure, ClojureScript, or ClojureCLR development environment?

Due to the general volatility of tools, it's interesting to see how this changes year to year. However, this year things were mostly pretty static with the three most common choices again Emacs/CIDER, Cursive/IntelliJ, and Vim with no major changes in percent use. Sublime, Light Table, and Eclipse/Counterclockwise all became a bit less common. The most interesting development was the rise in the use of Atom which was a new choice and selected by 6% of respondents.

What Clojure, ClojureScript, or ClojureCLR community forums have you used or attended in the last year?

This was a new question this year, trying to get a sense of how people are interacting with other members of the community. The Clojurians slack channel was the most frequently used - this is a great place to connect with others and has taken the place of IRC for many. About half of respondents are using the original language mailing lists, and almost that many have looked at the Clojure subreddit.

Interestingly, most respondents have not attended either local Clojure meetups or Clojure conferences either in-person or remotely. There are many active Clojure meetups and conferences in the world - if you'd like to talk to other Clojurists, take a look and see if one is near you!

Which versions of Clojure do you currently use in development or production?

Library maintainers are often interested in how quickly users are migrating to newer versions of Clojure as they decide whether they can use new features. We can see in this year's survey that most users are on the latest stable version (1.8.0) - 83%, with a third of respondents already using the 1.9 prereleases prior to final release. Less than 5% are using a Clojure version older than Clojure 1.7, which is good news for those that wish to rely on 1.7 features like cljc files or transducers.

What versions of the JDK do you target?

Similar to the prior question, it's useful to track what versions of the JDK are in use in the community. We saw significant consolidation to Java 1.8 over the past year (with Java 1.9 on the horizon) - 95% of users are using it with only about 2% using a version older than Java 1.7. For the moment, Clojure is still supported on Java 1.6 but eventually that support will be dropped.

What tools do you use to compile/package/deploy/release your Clojure projects?

While Leiningen continues to be ubiquitous, boot made significant advances this year, moving from 13% usage to 22% usage.

What has been most frustrating or has prevented you from using Clojure more than you do now?

Error messages continued to be the top frustration for people and we will continue to improve those with the integration of spec in Clojure 1.9. Interestingly, the majority of the other frustrations went down this year compared to last year:

  • Hiring/staffing - from 33% to 30%
  • Scripting - from 33% to 18% (maybe due to the rise of Planck and Lumo)
  • Docs - from 25% to 22% (hopefully the new Clojure and ClojureScript web sites have helped)
  • Static typing - from 23% to 16% (maybe due to the release of spec)
  • Long-term viability - from 20% to 10%
  • Finding libraries - from 16% to 11%
  • Portability - from 10% to 5% (continued uptake of cljc / reader conditionals)

Which JavaЅcript environments do you target?

The most interesting story here is the rise in three areas:

  • React Native - 18% (new choice this year)
  • Electron - 11% (new choice this year)
  • AWS Lambda - 9% (vs 5% last year)

As JavaScript continues to seep into every area of computing, ClojureScript is following along with it and seeing new and interesting uses. 

Which tools do you use to compile/package/deploy/release your ClojureScript projects?

We saw a small increase in Figwheel this year (after a huge jump after its release) with about 2/3 of ClojureScript users now using it. And as we saw in the prior tools question, there is a big jump in the number of ClojureScript developers using boot (from 15 to 23%).

Which ClojureScript REPL do you use most often?

Again, even more usage of Figwheel here (76%, up from 71% last year). We added Planck this year and it registered at 9%. The Lumo repl was not listed as a choice but did make a showing in the comments.

How are you running your ClojureScript tests?

We added this question to gather some information on what seems like an underserved area of the ecosystem. Of those who responded, we saw:

However, there was a lot of information in the "Other" responses as well. At least 60 people (more than replied for the Nashorn choice above) responded that they were either not testing at all or were relying on testing their ClojureScript via cljc tests that ran in Clojure. This is a great area for future improvements with no real consensus and a lot of developers not even doing it at all. Some other choices seen in the comments were Devcards, Karma, Phantom, and doo.

What has been most frustrating or has prevented you from using ClojureScript more than you do now?

The top answer here was "Using JavaЅcript libs with ClojureScript / Google Closure", which was a new choice we added this year. David Nolen and the ClojureScript community have been working hard on some of the biggest pain points in this area, which culminated in the recent release of a new ClojureScript version with better support for externs and modules.

Some of the other choices fell in importance this year (similar to Clojure):

  • "Using ClojureScript REPLs" went from 45% to 34% (rise of Figwheel, Planck, Lumo)
  • "Availability of docs" went from 39% to 31% (new ClojureScript web site)
  • "Long-term viability" went from 15% to 10%

Here you can add any final comments or opinions...

The majority of responses (~62%) here either expressed sentiments of happiness or gratitude (always good to see). Other categories centered around expected themes (many are areas of current or future work): docs/tutorials, error messages, tooling, startup time, etc. One relatively stronger theme this year was the need for better marketing for the purposes of expanding or introducing Clojure within organizations, which is a great area for contribution from the entire community.

The data

If you'd like to dig into the results more deeply, you can find the complete set of data from this and former years here:

Thanks again for providing your responses to help form this picture of our growing community!

Creating a spec for destructuring


A while back David Nolen had a thoughtful post about using spec as a tool for thought, which included an exploration of creating a spec for clojure.core/let.

The latest Clojure alpha actually includes a spec for let that covers destructuring and I thought it might be interesting to walk through the details of how it is implemented.

I'll pick up approximately where David left off. A typical let looks like this:

(let [a 1
      b 2]
  (+ a b))

We can define an initial spec for clojure.core/let by splitting it into bindings and body:

(require '[clojure.spec :as s]
         '[clojure.spec.gen :as gen])

(s/fdef let
  :args (s/cat :bindings ::bindings
               :body (s/* any?)))

We then need to more fully define bindings as a vector of individual bindings. Each binding is made of a binding-form and an init-expr that computes the value of the local binding:

(s/def ::bindings (s/and vector? (s/* ::binding)))
(s/def ::binding (s/cat :binding ::binding-form 
                        :init-expr any?))

The expressions can be anything so we leave those as any?. The binding-form is where things get interesting. Let's first allow for binding-form to be just simple (no namespace) symbols. That's enough to create something to work with.

(s/def ::binding-form simple-symbol?)

Now that we have a full spec, we can actually try a few things. Let's try an example of conforming our bindings.

(s/conform ::bindings '[a 1, b 2])
;;=> [{:binding a, :init-expr 1} {:binding b, :init-expr 2}]

Looks good! We get back a vector of binding maps broken into the binding and the initial expression.

Now we need to expand our spec to include sequential destructuring and map destructuring.

Sequential destructuring

Sequential destructuring binds a series of symbols to the corresponding elements in the bound value. Optionally, the symbols may be followed by a variadic argument (using &) and/or an alias for the overall sequence (using :as).

Some examples:

;; Sequential destructuring examples:
[a b]
[a b :as s]
[a b & r]

To describe a sequential spec we use the spec regex operators:

(s/def ::seq-binding-form
  (s/cat :elems (s/* simple-symbol?)
         :rest  (s/? (s/cat :amp #{'&} :form simple-symbol?))
         :as    (s/? (s/cat :as #{:as} :sym simple-symbol?))))

Let's try it out:

(s/conform ::seq-binding-form '[a b])
;;=> {:elems [a b]}
(s/conform ::seq-binding-form '[a b :as s])
;;=> {:elems [a b], :as {:as :as, :sym s}}
(s/conform ::seq-binding-form '[a b & r])
;;=> {:elems [a b & r]}

Hang on a sec, what happened in the last example? The elems snagged & r as well because & is a symbol. We need to redefine our notion of what a binding symbol is to exclude the symbol &, which is special in the language of destructuring:

(s/def ::local-name (s/and simple-symbol? #(not= '& %)))
(s/def ::seq-binding-form
  (s/cat :elems (s/* ::local-name)
         :rest  (s/? (s/cat :amp #{'&} :form ::local-name))
         :as    (s/? (s/cat :as #{:as} :sym ::local-name))))

(s/conform ::seq-binding-form '[a b & r :as s])
;;=> {:elems [a b], :rest {:amp &, :form r}, :as {:as :as, :sym s}}

That's better. But it turns out I've not really been spec'ing the full truth of sequential destructuring. Each of the ::elems can itself be sequentially destructured, and even the rest arg can be destructured.

We need to back up to the beginning and reconsider the definition of ::binding-form to add the possibility of either a ::local-name (our improved simple symbol) or a sequential destructuring form. (We'll add map later.)

(s/def ::local-name (s/and simple-symbol? #(not= '& %)))

;; WORK IN PROGRESS (still missing ::map-binding-form)
(s/def ::binding-form
  (s/or :sym ::local-name
        :seq ::seq-binding-form))

(s/def ::seq-binding-form
  (s/cat :elems (s/* ::binding-form)
         :rest  (s/? (s/cat :amp #{'&} :form ::binding-form))
         :as    (s/? (s/cat :as #{:as} :sym ::local-name))))

Now ::binding-form is a recursive specification. Binding-forms are either symbols or sequential forms, which may themselves contain binding-forms. The registry provides naming indirection which makes this possible.

Let's try our prior example again and see things have changed.

(s/conform ::seq-binding-form '[a b & r :as s])
;;=> {:elems [[:sym a] [:sym b]], :rest {:amp &, :form [:sym r]}, :as {:as :as, :sym s}}

Our conformed result is a bit more verbose as it now indicates for each binding form what kind of binding it is. While this is more verbose to read, it's also easier to process. Here's how a recursive binding form example looks:

(s/conform ::seq-binding-form '[a [b & c] [d :as e]])
;;=> {:elems [[:sym a]
;;            [:seq {:elems [[:sym b]], :rest {:amp &, :form [:sym c]}}]
;;            [:seq {:elems [[:sym d]], :as {:as :as, :sym e}}]]}

Finally we are ready to look at map destructuring.

Map destructuring

Map destructuring has a number of entry forms that can be used interchangeably:

  • <binding-form> key - for binding either a local name with (get m key) or recursively destructuring
  • :keys [key ...] - for binding locals with the same name as each key to the value retrieved from the map using the key as a keyword. In addition the specified keys can be either symbols or keywords and simple or qualified. In all cases, the local that gets bound is a short symbol and the value is looked up as a keyword.
  • :<ns>/keys [key ...] - same as :keys, but where ns is used as the namespace for every key
  • :syms [sym ...] - for binding locals with the same name as each sym to the value retrieved from the map using sym, which may be either simple or qualified.
  • :<ns>/syms [sym ...] - same as :syms, but where ns is used as the namespace for every symbol.
  • :strs [str ...] - for binding locals with the same name as each sym to the value retrieved from the map using str as a sym, which must be simple.
  • :or {sym expr} - for providing default values for any missing local that would have been bound based on other entries. The keys should always be simple symbols (the same as the bound locals) and the exprs are any expression.
  • :as sym - binds the entire map to a local named sym.

There is a lot of functionality packed into map binding forms and in fact there are really three different map specs combined into this single map. We call this a "hybrid" map spec.

The first part describes just the fixed well-known attributes in a typical s/keys spec:

(s/def ::keys (s/coll-of ident? :kind vector?))
(s/def ::syms (s/coll-of symbol? :kind vector?))
(s/def ::strs (s/coll-of simple-symbol? :kind vector?))
(s/def ::or (s/map-of simple-symbol? any?))
(s/def ::as ::local-name)

(s/def ::map-special-binding
  (s/keys :opt-un [::as ::or ::keys ::syms ::strs]))

The second part describes the basic binding form specs (examples like {n :name} ), although the left hand side here can further destructure.

(s/def ::map-binding (s/tuple ::binding-form any?))

And finally we need to handle the new functionality for namespaced key or symbol sets (like :<ns>/keys or :<ns>/syms) which we'll describe here as a map entry tuple:

(s/def ::ns-keys
    (s/and qualified-keyword? #(-> % name #{"keys" "syms"}))
    (s/coll-of simple-symbol? :kind vector?)))

Then we can put all of these together into the ::map-binding-form by combining them as an s/merge or the well-known attributes and a description of the possible tuple forms:

;; collection of tuple forms
(s/def ::map-bindings
  (s/every (s/or :mb ::map-binding
                 :nsk ::ns-keys
                 :msb (s/tuple #{:as :or :keys :syms :strs} any?)) :into {}))

(s/def ::map-binding-form (s/merge ::map-bindings ::map-special-binding))

And finally we need to go back and define our parent spec to include map bindings:

(s/def ::binding-form
  (s/or :sym ::local-name
        :seq ::seq-binding-form
        :map ::map-binding-form))

And that's it! Here's an example binding form that shows several features of destructuring:

(s/conform ::binding-form
  '[[x1 y1 :as p1]
    [x2 y2 :as p2]
    {:keys [color weight]
     :or {color :black weight :bold}
     :as opts}])
;;=> [:seq {:elems [[:seq {:elems [[:sym x1] [:sym y1]], :as {:as :as, :sym p1}}]
;;                  [:seq {:elems [[:sym x2] [:sym y2]], :as {:as :as, :sym p2}}]
;;                  [:map {:keys [color weight]
;;                         :or {color :black, weight :bold}
;;                         :as opts}]]}]

Now that we have a spec for destructuring, we can reuse it anywhere destructuring is allowed - in fn, defn, for, etc. We could even leverage it to implement destructuring itself. Rather than recursively parsing the binding form, we could simply conform it to receive a more regular structure described in terms of the parts we've defined in the spec.

Works on My Machine: Self Healing Code with clojure.spec

Works On My Machine is the place where Cognitects reflect on tech, culture, and the work we do. The views expressed on Works On My Machine are those of the author and don’t necessarily reflect the official position of Cognitect.

How can we can make code smarter? One of the ways is to be more resilient to errors. Wouldn't it be great if a program could recover from an error and heal itself? This code would be able to rise above the mistakes of its humble programmer and make itself better.

The prospect of self-healing code has been heavily researched and long sought after. In this post, we will take a look at some of the key ingredients from research papers. Then, drawing inspiration for one of them, attempt an experiment in Clojure using clojure.spec.

Self Healing Code Ingredients

The paper Towards Design for Self-healing outlines a few main ingredients that we will need.

  • Failure Detection - This one is pretty straight forward. We need to detect the problem in order to fix it.
  • Fault Diagnosis - Once the failure has been detected, we need to be able to figure out exactly what the problem was so that we can find a solution.
  • Fault Healing - This involves the act of finding a solution and fixing the problem.
  • Validation - Some sort of testing that the solution does indeed solve the problem.

With our general requirements in hand, let's turn to another paper for some inspiration for a process to actually achieve our self healing goal.

Self Healing with Horizontal Donor Code Transfer

MIT developed a system called CodePhage which is a system inspired from the biological world with horizontal gene transfer of genetic material between different organisms. In it they use a "horizontal code transfer system" that fixes software errors by transferring correct code from a set of donor applications.

This is super cool. Could we do something like this in Clojure?

Clojure itself has the fundamental ability with macros to let the code modify itself. The programs can make programs! That is a key building block but clojure.spec is something new and has many other advantages that we can use.

  • clojure.spec gives code the ability to describe itself. With it we can describe the data the functions take as input and output in a concise and composable way.
  • clojure.spec gives us the ability to share these specifications with other code in the global registry.
  • clojure.spec gives us the ability to generate data from the specifications, so we can make example data that fits the function's description.

With the help of clojure.spec, we have all that we need to design and implement a self-healing code experiment.

Self Healing Clojure Experiment

We'll start with a simple problem.

Imagine a programmer has to write a small report program. It will be a function called report that is made up of three helper functions. It takes in a list of earnings and outputs a string summary of the average.

(defn report [earnings]
  (-> earnings

The problem is that our programmer has made an error in the calc-average function. A divide by zero error will be triggered on a specific input.

Our goal will be to use clojure.spec to find a matching replacement function from a set of donor candidates.


Then replace the bad calc-average function with a better one, and heal the report function for future calls.


The Setup

Let's start with the report code. Throughout the code examples I will be using clojure.spec to describe the function and its data. If you haven't yet looked at it, I encourage you to check out the spec Guide.

The first helper function is called clean-bad-data. It takes in a vector of anything and filters out only those elements that are numbers.

(defn clean-bad-data [earnings]
  (filter number? earnings))

Let's create a couple of specs to help us describe it. The first, earnings will be a vector, (for the params) with another vector of anything.

(s/def ::earnings (s/cat :elements (s/coll-of any?)))

The next spec for the output of the function we will call cleaned-earnings. It is going to have a custom generator for the purposes of this experiment, which will constrain the generator to just returning the value [[1 2 3 4 5]] as its example data[^1].

(s/def ::cleaned-earnings (s/with-gen
                            (s/cat :clean-elements (s/coll-of number?))
                            #(gen/return [[1 2 3 4 5]]))

An example of running the function is:

(clean-bad-data [1 2 "cat" 3])
;=>(1 2 3)

If we call spec's exercise on it, it will return the custom sample data from the generator.

(s/exercise ::cleaned-earnings 1)
;=> ([[[1 2 3 4 5]] {:clean-elements [1 2 3 4 5]}])

Now we can spec the function itself with s/fdef. It takes the earnings spec for the args and the cleaned-earnings spec for the return value.

(s/fdef clean-bad-data
        :args ::earnings
        :ret ::cleaned-earnings)

We will do the same for the calc-average function, which has the flaw vital to our experiment - if we pass it an empty vector for the earnings, the count will be zero and result in a run time divide by zero error.

(defn calc-average [earnings]
  (/ (apply + earnings) (count earnings)))

(s/def ::average number?)

(s/fdef calc-average
    :args ::cleaned-earnings
    :ret ::average)

Finally, we will create the rest of the display-report function and finish specing the function for report.

(s/def ::report-format string?)

(defn display-report [avg]
  (str "The average is " avg))

(s/fdef display-report
        :args (s/cat :elements ::average)
        :ret ::report-format)

(defn report [earnings]
  (-> earnings

(s/fdef report
        :args ::earnings
        :ret ::report-format)

Giving a test drive:

(report [1 2 3 4 5])
;=> "The average is 3"

And the fatal flaw:

(report [])
;=>  EXCEPTION! Divide by zero

Now we have our problem setup. We need to have our donor candidates.

The Donor Candidates

We are going to have a separate namespace with them. They will be a number of them, all function speced out. Some of them will not be a match for our spec at all. Those bad ones include:

  • bad-calc-average It returns the first number in the list and doesn't calc the average at all.
  • bad-calc-average2 It returns a good average function but the result is a string. It won't match the spec of our calc-average function.
  • adder It takes a number and adds 5 to it. It also won't match the spec of calc-average.

There is a matching function called better-calc-average that matches the spec of our calc-average function and has the additional check for divide by zero.

(s/def ::numbers (s/cat :elements (s/coll-of number?)))
(s/def ::result number?)

(defn better-calc-average [earnings]
  (if (empty? earnings)
    (/ (apply + earnings) (count earnings))))

This is the one that we will want to use to replace our broken one.

We have the problem. We have the donor candidates. All we need is the self-healing code to detect the problem, select and validate the right replacement function, and replace it.

The Self Healing Process

Our process is going to go like this:

  • Try the report function and catch any exceptions.
  • If we get an exception, look through the stack trace and find the failing function name.
  • Retrieve the failing function's spec from the spec registry
  • Look for potential replacement matches in the donor candidates
    • Check the orig function's and the donor's :args spec and make sure that they are both valid for the failing input
    • Check the orig function's and the donor's :ret spec and make sure that they are both valid for the failing input
    • Call spec exercise for the original function and get a seed value. Check that the candidate function's result when called with the seed value is the same result when called with the original function.
  • If a donor match is found, then redefine the failing function as new function. Then call the top level report form again, this time using the healed good function.
  • Return the result!
(ns self-healing.healing
  (:require [clojure.spec :as s]
            [clojure.string :as string]))

(defn get-spec-data [spec-symb]
  (let [[_ _ args _ ret _ fn] (s/form spec-symb)]
       {:args args
        :ret ret
        :fn fn}))

(defn failing-function-name [e]
  (as-> (.getStackTrace e) ?
    (map #(.getClassName %) ?)
    (filter #(string/starts-with? % "self_healing.core") ?)
    (first ?)
    (string/split ? #"\$")
    (last ?)
    (string/replace ? #"_" "-")
    (str *ns* "/" ?)))

(defn spec-inputs-match? [args1 args2 input]
  (println "****Comparing args" args1 args2 "with input" input)
  (and (s/valid? args1 input)
       (s/valid? args2 input)))

(defn- try-fn [f input]
  (try (apply f input) (catch Exception e :failed)))

(defn spec-return-match? [fname c-fspec orig-fspec failing-input candidate]
  (let [rcandidate (resolve candidate)
        orig-fn (resolve (symbol fname))
        result-new (try-fn rcandidate failing-input)
        [[seed]] (s/exercise (:args orig-fspec) 1)
        result-old-seed (try-fn rcandidate seed)
        result-new-seed (try-fn orig-fn seed)]
    (println "****Comparing seed " seed "with new function")
    (println "****Result: old" result-old-seed "new" result-new-seed)
    (and (not= :failed result-new)
         (s/valid? (:ret c-fspec) result-new)
         (s/valid? (:ret orig-fspec) result-new)
         (= result-old-seed result-new-seed))))

(defn spec-matching? [fname orig-fspec failing-input candidate]
  (println "----------")
  (println "**Looking at candidate " candidate)
  (let [c-fspec (get-spec-data candidate)]
    (and (spec-inputs-match? (:args c-fspec) (:args orig-fspec) failing-input)
         (spec-return-match? fname c-fspec orig-fspec  failing-input candidate))))

(defn find-spec-candidate-match [fname fspec-data failing-input]
  (let [candidates (->> (s/registry)
                        (filter #(string/starts-with? (namespace %) "self-healing.candidates"))
                        (filter symbol?))]
    (println "Checking candidates " candidates)
    (some #(if (spec-matching? fname fspec-data failing-input %) %) (shuffle candidates))))

(defn self-heal [e input orig-form]
  (let [fname (failing-function-name e)
        _ (println "ERROR in function" fname (.getMessage e) "-- looking for replacement")
        fspec-data (get-spec-data (symbol fname))
        _ (println "Retriving spec information for function " fspec-data)
        match (find-spec-candidate-match fname fspec-data [input])]
    (if match
        (println "Found a matching candidate replacement for failing function" fname " for input" input)
        (println "Replacing with candidate match" match)
        (println "----------")
        (eval `(def ~(symbol fname) ~match))
        (println "Calling function again")
        (let [new-result (eval orig-form)]
          (println "Healed function result is:" new-result)
      (println "No suitable replacment for failing function "  fname " with input " input ":("))))

(defmacro with-healing [body]
  (let [params (second body)]
    `(try ~body
          (catch Exception e# (self-heal e# ~params '~body)))))

What are we waiting for? Let's try it out.

Running the Experiment

First we call the report function with a non-empty vector.

(healing/with-healing (report [1 2 3 4 5 "a" "b"]))
;=>"The average is 3"

Now, the big test.

(healing/with-healing (report []))
; ERROR in function self-healing.core/calc-average Divide by zero -- looking for replacement
; Retrieving spec information for function  {:args :self-healing.core/cleaned-earnings, :ret :self-healing.core/average, :fn nil}
; Checking candidates  (self-healing.candidates/better-calc-average self-healing.candidates/adder self-healing.candidates/bad-calc-average self-healing.candidates/bad-calc-average2)
; ----------
; **Looking at candidate  self-healing.candidates/better-calc-average
; ****Comparing args :self-healing.candidates/numbers :self-healing.core/cleaned-earnings with input [[]]
; ****Comparing seed  [[1 2 3 4 5]] with new function
; ****Result: old 3 new 3
; Found a matching candidate replacement for failing function self-healing.core/calc-average  for input []
; Replacing with candidate match self-healing.candidates/better-calc-average
; ----------
; Calling function again
; Healed function result is: The average is 0
;=>"The average is 0"

Since the function is now healed we can call it again and it won't have the same issue.

(healing/with-healing (report []))
;=>"The average is 0"

It worked!

Taking a step back, let's a take a look at the bigger picture.


The self healing experiment we did was intentionally very simple. We didn't include any validation on the :fn component of the spec, which gives us yet another extra layer of compatibility checking. We also only checked one seed value from the spec's exercise generator. If we wanted to, we could have checked 10 or 100 values to ensure the replacement function's compatibility. Finally, (as mentioned in the footnote), we neglected to use any of spec's built in testing check functionality, which would have identified the divide by zero error before it happened.

Despite being just being a simple experiment, I think that it proves that clojure.spec adds another dimension to how we can solve problems in self-healing and other AI areas. In fact, I think we have just scratched the surface on all sorts of new and exciting ways of looking at the world.

For further exploration, there is a talk from EuroClojure about this as well as using clojure.spec with Genetic Programming

[^1]: The reason for this is that if the programmer in our made up example didn't have the custom generator and ran spec's check function, it would have reported the divide by zero function and we would have found the problem. Just like in the movies, where if the protagonist had just done x there would be no crisis that would require them to do something heroic.

2016 State of Clojure Community Survey Now Open


It's time for the annual State of Clojure Community survey!

If you are a user of Clojure, ClojureScript, or ClojureCLR, we are greatly interested in your responses to the following survey:

State of Clojure 2016

The survey contains four pages:

  1. General questions applicable to any user of Clojure, ClojureScript, or ClojureCLR
  2. Questions specific to the JVM Clojure (skip if not applicable)
  3. Questions specific to ClojureScript (skip if not applicable)
  4. Final comments

The survey will close December 23rd. We will release all of the data and our analysis in January. We are greatly appreciative of your input!


Interactive Development with Clojure.spec

clojure.spec provides seamless integration with clojure.test.check's generators. Write a spec, get a functioning generator, and you can use that generator in a REPL as you're developing, or in a generative test.

To explore this, we'll use clojure.spec to specify a scoring function for Codebreaker, a game based on an old game called Bulls and Cows, a predecessor to the board game, Mastermind. You might recognize this exercise if you've read The RSpec Book, however this will be a bit different.

Agility & Robustness: Clojure spec

You can program with high agility and end up with a robust, maintainable program. This talk will show you how to use Clojure and the new spec library to write programs that behave as expected, meet operational requirements, and have the flexibility to accommodate change.

Focus on spec: Combining specs with s/or

In our last post, we looked at `s/and`, a way to combine multiple specs into a compound spec. It should come as no surprise that spec also provides `s/or` to represent a spec made of two or more alternatives. 

Focus on spec: Combining Specs with `and`

Clojure's new spec library provides the means to specify the structure of data and functions that take and return data. In this series, we'll take one Clojure spec feature at a time and examine it in more detail than you can find in the spec guide.

In our last post, we explored the simplest specs: predicate functions and sets. In this post we'll look at how you can start to combine specs using the and spec.

Clojure spec Screencast: Customizing Generators

One benefit of Clojure specs is that they automatically provide data generators that produce values conforming to the spec which can be used for testing. In addition, you can compose your own generator to more precisely match your data model.

Focus on spec: Predicates

Clojure's new spec library provides the means to specify the structure of data and functions that take and return data. All specs definitions are ultimately based on predicates, which are nothing more than Clojure functions that take a value and return a value that is treated as logically true or false.

Clojure spec Screencast: Testing

Clojure spec defines specifications for both data and functions. In addition to validity checking, specs can generate random samples of the data they specify. This capability enables an alternative to unit testing known as generative, or property-based, testing.

Clojure spec Screencast: Leverage

The new Clojure spec library provides support for data and function specification. In this first in a series of screencasts, Stuart Halloway discusses how spec provides leverage to achieve many returns for a small investment in describing your functions with spec.

The New Normal: Data Leverage

Like many developers, I’ve spent a lot of time thinking about objects. I started doing object-oriented programming (OOP) in 1989, and was arguably still doing it up until 2012. The concept of objects arose in GUIs and simulations, where it is natural to think about sending message to things. Over time though, the idea of messages took a back seat to the idea of locking data up in little containers where each class provides a namespace of functions to operate on it.

The New Normal: Everything Relies on Sharp Tools

You wouldn’t use a toothpick to wear down a mountain. You wouldn’t wear your trainers instead of your racing spikes at a track meet. So why would you use a hand saw to build an infrastructure when you need a scalpel for surgical precision?

Up to this point in the series, we’ve talked about embracing failure, experimenting more and moving faster. We've talked about the "why," now it's time to address the "how." 

The New Normal: The Art of War, Maneuverability, and Microservices

The word "antifragile" may be recent, but some of the concepts are ancient. In "Art of War", the renowned general, strategist and tactician Sun Tzu's states, “…water shapes its course according to the nature of the ground over which it flows...” In an antifragile organization, we want to explore opportunities so resources flow like water into the things that are working, and abandon those that are not. 

Just as water retains no constant shape, there are no constants in an antifragile organization and IT infrastructure. To flow like water, you must be able to shift people and teams easily, create teams and systems easily, be able to tear down systems and remove people from working on projects that aren’t working. This requires an architecture that allows you to act locally but think globally. Some organizations are pursuing microservices to this end. Complex applications are composed of small, independent processes that focus on doing a single responsibility. With microservices, developers decouple software into smaller single-function units that can be replaced individually.