Articles

Technology The New Normal How We Work Customer Stories Testing | All Topics

Extending Transit

The transit spec defines several ground types that are supported by
JSON and MessagePack, on which transit piggy-backs to convey data from
one process to another. These include scalar types like strings and
numbers, and composite types like maps and arrays.

The spec also defines several extension types, which are built from
ground types, each represented by a tag and a value. The tag is a
string, which is a scalar ground type. The value can be a scalar or
composite type. When the reader encounters a composite, e.g. an array,
it unpacks its values and reads each one, recursing until it reaches
scalar ground types.

We use the same extension system to support custom types that are
not part of the transit spec. These can be domain-specific types like
Person or Account, or they can be generic datatypes like trees, sorted
sets, tuples, etc. They don't need to be supported by all of the
languages in our system. In fact, they don't even need to be recognized
by the transit libs running in all of the processes in our system!

Transit

We are pleased to announce today the initial release of Transit.

Transit is a format and set of libraries for conveying values between applications written in different programming languages. The key objectives of Transit are to support:

  • Sending values between applications
  • written in different programming languages
  • without requiring schemas/context
    • i.e., to be self-describing at the bottom
  • with extensibility
  • and good performance
  • with reach to the browser

JSON currently dominates similar use cases, but it has a limited set of types, no extensibility, and is verbose. Actual applications of JSON are rife with ad hoc and context-dependent workarounds for these limitations, yielding coupled and fragile programs.

On the other hand, the reach of JSON is undeniable. High performance parsers are widely available. Thus Transit is specified as an encoding to and from both JSON and MessagePack, a binary JSON-like format with widely available parsers. In particular, both formats have parsers written in C for languages like Ruby and Python that reach to C for performance.

Transit supports a minimal but rich set of core types:

  • strings
  • booleans
  • integers (to 64 bits w/o truncation)
  • floats
  • nil/null
  • arrays
  • maps (with arbitrary scalar keys, not just strings)

Transit also includes a wider set of extension types:

  • timestamps
  • UUIDs
  • URIs
  • arbitrary precision integers and decimals
  • symbols, keywords, characters
  • bytes
  • sets
  • lists
  • hypermedia links
  • maps with composite keys

Transit is extensible - users can define extension types in exactly the same way as the included extension types.

The emphasis of Transit is on communication between programs, thus it prioritizes programmatic types and data structures over human readability and document orientation. That said, it does have a readable verbose mode for JSON.

Transit is self describing using tags, and encourages transmitting information using maps, which have named keys/fields which will repeat often in data. These overheads, typical of self-describing formats, are mitigated in Transit by an integrated cache code system, which replaces repetitive data with small codes. This yields not only a reduction in size, but also an increase in performance and memory utilization. Contrast this with gzipping, which, while it may reduce size on the wire, takes time and doesn't reduce the amount of text to be parsed or the number of objects generated in memory after parsing.

We are shipping an 0.8 version of the Transit spec, which has extensive documentation for implementors, as well as interoperable implementations for:

We welcome feedback and suggestions on the transit-format list.

I'd like to thank the team at Cognitect that built Transit:

  • Tim Ewald - Team Lead
  • Brenton Ashworth
  • Timothy Baldridge
  • Bobby Calderwood
  • David Chelimsky
  • Paul deGrandis
  • Benoit Fleury
  • Michael Fogus
  • Yoko Harada
  • Ben Kamphaus
  • Alex Miller
  • David Nolen
  • Russ Olsen

I hope you find Transit useful, and look forward to your feedback.

Rich