THE COGNICAST TRANSCRIPTS
In this episode, we talk to Paul Stadig about Polymorphism and other interesting things.
The complete transcript of this episode is below.
The audio of this episode of The Cognicast is available here.
CRAIG: Hello, and welcome to Episode 113 of The Cognicast, a podcast by Cognitect, Inc. about software and the people who create it. I'm your host, Craig Andera.
All right, announcements as always. Let’s start with the Austin Clojure Meetup. This will of course be in Austin, Texas. It’s happening Wednesday, November 30th. That’s the night before the Conj. If you happen to be in town on Wednesday, November 30th--this is 2016, of course--because you’re going to the Conj, you will be around for the Austin Clojure Meetup where Stu Halloway will be talking about clojure.spec. You can find more about that by searching for Austin Clojure Meetup.
Of course, the Conj itself is happening. That is happening December 1st through the 3rd. I will be there. In fact, I will be there from the evening of the 28th on since we are doing Clojure and Datomic training immediately before the conference. I’ll be doing the Clojure class. Still places left, I believe, in both classes. Certainly check the website, Clojure-conj.org, for more information about that.
I want to mention ClojureBridge. Got a couple coming up, one in London and one in Berlin. I believe those are both on the 25th of November. Information about ClojureBridge is always available at ClojureBridge.org. I’d like to remind you, too, that you can go there to donate. A great way to support that organization.
That’s about all I think I have for announcements. We’ll go ahead. We will go on. This will be Episode 113 of The Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
CRAIG: I believe that I am ready to kick this off. How about you?
CRAIG: Cool. All right, let’s go then. Well, hello, everyone. Welcome. Today is Monday, October 31st, 2016. That would be Halloween, so welcome to a very spooky episode of The Cognicast. Welcome especially to our guest today, Paul Stadig. Welcome to the show, Paul.
CRAIG: We actually have quite a few things to talk about today. But of course, we always start the show with one particular question, a question about art, and specifically about our guest, some experience that our guest has had of art, whatever that might mean to them. Paul, have you picked something out that you’d like to share with us?
PAUL: Yeah. Yeah, yeah. I think the experience I had with art that I wanted to relate was actually a Clojure/conj talk by Chris Ford in 2012. Actually, I think he gave it at Lambdajam too. I think there are a couple of recordings of it out on the Internet there, but it was called Functional Composition, and he was talking about representing music using code, essentially.
I remember it really affected me. It’s an excellent talk. People should go listen to it. The idea of representing music with code was just really interesting to me.
I mean I’ve played instruments before. As a Christian, for me the experience of art always kind of connects with my worldview, and especially this idea that it was very interesting to me that there is like this idea of music. There’s like this music that can be represented either in sheet music or in code, and then it can be manipulated as code with function composition and all these things. It was a really interesting concept that kind of set me off thinking about lots of related concepts in how you can represent music or visual arts with code.
I was actually moved to tears at the end of his talk. I mean it was just very beautiful. He kind of builds up music from basic concepts. It was really great.
CRAIG: Yeah. I was there too, and it was. It is a really, really good talk. He’s an excellent speaker, period. He picked an interesting topic, and that unification of code and art was very, very cool. I totally agree. He gave a talk at Strange Loop just recently.
PAUL: Yeah. Right.
CRAIG: I don’t know if you saw that one.
PAUL: I was actually going to say, probably my second experience with art was the talk he gave actually in 2016 at Strange Loop on African polyphony and polyrhythm, which was also interesting for me because it kind of challenged the first experience I had, which I came away from the first talk thinking, like, wow, this is great. We can just represent all kinds of music as code and all this.
But in his Strange Loop talk he talks about trying to represent some different kinds of African music using code. It was based off of a book he’d read, and it was just really interesting because there are definitely -- I mean my experience with music is mostly like the European tradition. Trying to represent this African music using European sheet music was really quite a challenge. Also, there’s a human element to it. There’s like cultural significance to music that you can’t represent with code, and so that was also kind of an interesting experience as well.
CRAIG: Yeah, I was there in the audience for it, and I have to say he did a really amazing job of blending the cultural aspects in a--as you said, challenging is the right word--way, but in a good sense of that, but also the technical and just the musical portion. It was a very, very cool talk, too. He’s a great guy.
PAUL: Yeah, I think both of those talks are well worth listening to. I’d recommend them both highly.
CRAIG: Yeah. Awesome. Well, cool. As much as it would be awesome just for us both to gush being Chris Ford fan boys the whole hour or whatever, I think we should talk about you because you’ve done some interesting things. You’ve been in the Clojure community for quite a while now. I mean it’s increasingly rare that I run into people that have been in as long or longer than me, which is a good sign because it’s a sign of a healthy community that we’re bringing in new people and it’s growing, but you certainly have been around for quite a while.
PAUL: Yeah. How long have you been around in the Clojure community?
CRAIG: Well, it depends on how you count, but I think I started playing with it in, I want to say, 2008, but I’d have to go look. But I started at what was then Relevance in 2010, so that would definitely be my official “I am now a professional Clojure programmer” start.
PAUL: Yeah. I was taking some graduate classes pursuing a Master’s Degree in Computer Science and one of my classmates kind of pointed. We were doing some projects in Common Lisp. We wrote a Naïve Bayes spam filter in Common Lisp.
One of my classmates was like, hey, this thing is kind of cool. There’s like this lisp for the JVM. I remember seeing it. I was like, oh, yeah, it’s kind of interesting - whatever.
I don’t know if that was even 2007 or 2008. I think it might have been 2007 because I think I started kind of actually playing with Clojure in 2008. Then I joined Sonian in 2009 working full time doing Clojure development, which was cool. I’ve been involved since then, at least, I guess.
CRAIG: Mm-hmm. Yeah, and you and I met while you were still at Sonian. We’ve become friends. We see each other a lot. The joke, of course, is that many people actually have said that you and I look a fair amount alike to the extent where I think we’ve both been at conferences and had people come up to us--
CRAIG: --expecting we were the other person. There was one in particular. I won’t say who it was, but one of your colleagues like ran up to me and said, “Oh, hey, I’ve been looking for you,” blah, blah, blah. Went on for about five seconds and then just stopped and said, “Oh, never mind,” and walked away. I knew exactly what had happened by that point, so.
PAUL: I think it’s more common for people to confuse me for you than you for me because I’m not as prominent in the community, but I do kind of relish. I heard the story that when you started at Relevance someone actually thought you’d hired me or something.
CRAIG: Oh, yeah. Yeah.
CRAIG: Yep, absolutely true.
PAUL: That’s funny.
CRAIG: I won’t out that person either, but yeah. I always thought that you and I -- we kicked this idea around and we haven’t made it happen yet. Maybe I’m ruining the surprise, but I always thought we should do a conference talk where one of us starts on stage and then, at some point, walks behind something and the other person comes out and picks up as if nothing had happened and continues the talk to see if anybody would notice.
PAUL: Yeah. Be like one of those magic shows with two doors and someone walks in one and comes out the other.
CRAIG: Exactly. A very, very, very obscure joke that you and I would enjoy greatly. Yeah, but anyway, you have been around for a long time. You’ve done quite a few interesting and cool things. I guess maybe we’ll go reverse chronologically though because you’ve done one pretty recently that was the proximate cause.
Certainly being who you are, it would have made sense for us to have had you on any number of times before now, but I saw that you’d put out a book. I was like, oh, right, yeah, that might be a great excuse--not that we really need one--to have Paul on the show to talk about that. Maybe I’ll just throw it to you, and you can explain to us what this book is about, how you came to write it, who it’s for, that type of thing.
PAUL: Yeah. It wasn’t really well thought out, actually. It really came about because I was submitting a talk to the Conj for this year, 2016, which was not accepted, which is fine. I started kind of fleshing out the outline for that talk, and I thought, I don’t know. This could make a decent blog post or a little short e-book - whatever. I kind of fleshed it out into an e-book. It was my first experience working with Leanpub and it’s actually available on Amazon as well through the Kindle direct publishing system, and so that was an interesting experience.
The person I’ve written it for, I feel like there’s kind of a gap in the market for Clojure. I use the term “market” loosely. There’s plenty of introductory tutorials how to get started, how to set up Emacs, Cursive, or something, and get connected to REPL and integers and floating points and how to write functions. Then there’s certainly a lot of material where if you want to go deep on things like core.async or core.logic or something, you can go down into the guts of those things.
There’s nothing really, or it seemed to me there wasn’t a lot in the middle there where maybe someone who has actually used Clojure a little bit, is a little familiar with the language, has some familiarity with the different tools it provides you, but they want to build a real system. There are obvious tradeoffs to different types of designs and using the different tools in Clojure, so what I was trying to target was that person who has some moderate experience in Clojure, hasn’t written a lot of large systems or projects, and wants to know, like, hey, I’m trying to write a service abstraction. What’s the best tools from Clojure to, like, do that or whatever? And so that was kind of where I was going.
The book is called Clojure Polymorphism, and I just kind of explored the theme of polymorphism in Clojure, what the different tools are. I did it through some real examples, real code, working through kind of iteratively changing things, trying different approaches, solving the same problems several different ways so you can kind of see it from different perspectives and how the different tools affect the solutions and what the tradeoffs are.
CRAIG: Mm-hmm. Yeah, and so for once I’m actually somewhat prepared for an interview. I actually read your book, and it’s quite good. I encourage people to go and check it out.
I have to say, I think my opinion of the available materials lines up very well with yours. I think that there is an excellent, rich selection of intro material. There are smaller and understandably smaller selection of good advanced stuff. I’m thinking of, for example, Joy of Clojure, I think is maybe the prototypical advanced Clojure book.
CRAIG: Great book. But I agree. I think, although I do believe that there are some things in the middle, and your book is an excellent example of this, it’s not proportional. Right? There should be a lot more of it because we know that there are a lot of people that are coming to Clojure. Look at the attendance numbers at the conferences where every year we have 200, 300 new people who have actually flown somewhere to be at this conference, so we know that there are people coming in. Those people very clearly are going to need help with things like polymorphism.
I think, in your book, you focus a lot on -- and you can expand on this. I feel like you focus a lot on tradeoffs, and I think that’s key.
CRAIG: Is that--?
CRAIG: Was that an effort on your part…?
PAUL: Yeah. Yeah, I mean my idea was kind of that it’s kind of like a comparative architecture course. You can see the same problems solved several different ways, and you can see the tradeoffs. Yeah.
CRAIG: Yeah, and I think that’s key because there’s an old, old joke that says -- who was it that said Lisp programmers know the value of everything and the cost of nothing, or something along those lines? I think that’s really a comment on human nature, which is that when you’re presented with a tool, it’s very easy to see what the tool does for you, but not easy to see the ways in which the tool constrains you. And so I think tradeoffs and comparisons like that are super important.
PAUL: Yeah, and obviously performance is a big tradeoff, and a lot of times that’s kind of more prominent for people, but there are other tradeoffs too. The ease of understanding the system or extending the system. There’s definitely other types of tradeoffs that you can consider and you want to consider.
Now I will say I think you kind of echoed this that there doesn’t seem to be as many resources in that kind of middle area as there are on the beginning and kind of advanced ends. But I did find a book called Clojure Applied, which is an excellent book. Having looked at it, kind of doing my market research to see what kind of resources are available out there, I ran across that book, which I had not seen before. That’s a great book, and it’s kind of along the same vein of, so you have Clojure. You have these sharp tools, like how to use and what are the tradeoffs for different ways to design systems.
CRAIG: Yeah, so this is Ben Vandgrift’s and Alex Miller’s book.
PAUL: Yep. Yep.
CRAIG: Yeah. Yeah, yeah, no, I agree. To be honest, I haven’t read the whole thing, but there’s just too many Clojure books any more for me to have read them all.
CRAIG: But, yeah, I think those two definitely understand the things we’re talking about. You know the other one I would maybe plus one is The Cookbook. It’s not quite the same thing, but I feel like it’s in the same general space. It’s not going to give somebody the same sort of grounded understanding that I feel like you were going for, and we’ll come back to what I mean by that in a minute.
CRAIG: But I do feel like it’s about, okay, you know the language. Let’s lift you up to the point where you can start building things and not have to work out from first principles how to do that.
PAUL: Mm-hmm. Yeah, yeah, and I do. I think I agree with what you’re saying. I saw Cookbooks as a useful tool, and I don’t mean to disparage Cookbooks at all. I think they’re a useful tool to give you patterns or help you get started with things, or if you want to look up, like, how do I talk to a website using clj-http or something. You could look it up.
What I was aiming for was not so much a Cookbook approach as, like, again, kind of looking at the same problem solved with different tools and then drawing some principles from that. You can take those principles and apply them in different ways, so it’s not so much a Cookbook where it’s like here is a chunk of code you can drop in, but more kind of a comparative, you know, look at the tools, approaches, and tradeoffs.
CRAIG: Yeah, I think they’re very much complementary approaches. They’re quite different at what they achieve.
Let’s actually walk through a little bit. Kind of take us through the story of the book here, I mean, because you really do have a pretty clear path where you build up and bring people along and say, okay, we’re going to start with this and then move to this. I wonder if you could maybe give us a bit of a tour of the book.
PAUL: Yeah. There’s really kind of like, I guess you could say, three main sections. There’s a chapter on service abstraction. I use that as an example. That’s my example that I approach from different tools in Clojure. We look at how to implement a service abstraction with, like, plain functions, which a hash map that has functions in it that you can pull out as kind of a way of dispatching with protocols and what the tradeoffs are for those. Then there’s another section, another chapter where I look at data transformation because that’s kind of another common use of polymorphism that I’ve seen.
My experience is you have this data. You want to convert it to JSON or from JSON or whatever, so a lot of times what you’ll want to do is have some transformation function, like to JSON that you want to extend to floats, ints, strings, and various different data structures. That function is polymorphic and you want to define it for all those different types. That’s another chapter kind of looking at that problem and how to approach it.
Then there’s another chapter is kind of JVM specific and talks about some gotchas that I’ve run across in my experience using Clojure in the vein of polymorphism on the JVM because there are a couple of performance considerations and things to think about in addition. That’s kind of the main body of the book. There is also kind of an introductory chapter that just talks about what is polymorphism and a brief overview of what the tools are in Clojure for polymorphism. I don’t go into a lot of detail about the implementation of things like protocols or things. Again, I kind of assume people kind of are aware of the tools and generally how to use them and have even used them a little bit, but want to see these problems, different ways of approaching them, and kind of learn some design principles from them.
CRAIG: Yeah, so this is not a long book.
CRAIG: Like I said, I actually read the whole thing, but it took me maybe 25 minutes. There’s a good amount of meat there, for sure, but I don’t even know how many pages.
PAUL: It’s like 30. I think it’s like 31 pages. It’s an e-book. But, yeah, my goal was not to write something long. It’s hopefully short, but, like you said, packed with information. It’s not a long read, and it’s relatively cheap. In fact, I have a coupon code for people. Maybe I can mention that later. I don’t know, but.
CRAIG: Sure. Mention it now. It’ll wind up in the transcript too, so either way.
PAUL: Okay. Yeah, so if anybody is listening to this and is interested in the book, as I said it’s available on Amazon. You can search for it, Clojure Polymorphism. It’s also available on LeanPub. I have a coupon for Leanpub. I created a really cool URL for it using the bit.ly URL shortener. It’s bit.ly/clojurepoly, all one word, lower case, and that’ll send you straight to the book page with that coupon code applied. It saves you 30% if you’re interested in checking it out.
CRAIG: Very cool. Yeah, people should pick it up. I mean, like you say -- I don’t know. I’m just trying to give an example of something that, when I read it, I was like, “Oh, yeah. I totally forgot about that. And to be honest, I never had completely thought about it even though I was aware of it.” So this was the thing where you’re talking about the fact that there’s a difference when you extend a protocol to a record after the fact versus declaring that the record supports the protocol in the body of the def record and that there is a potentially significant. I mean that it’s all relative performance difference between those two approaches.
I was like, “Oh, that’s right. I had forgotten about that,” so I think--and I don’t know if you want to expand on that at all, but--there’s definitely things in there where even somebody who has been doing Clojure for quite a while can read the book and come away with a better understanding or at least a refreshed understanding of the things that we deal with all the time.
PAUL: Yeah. Yeah, that’s in the JVM kind of gotchas chapter. I mean that has to do with the way protocols are extended to classes. When you extend outside of a def record form or if you extend to, like, a core class, like say java.lang.integer or something, that, the dispatch actually happens through a dispatch table in the protocol function. That’s above the level of the JVM’s actual dispatch on interfaces if you’re calling Java code, so it’s slower.
If you extend a protocol in a def record form, then the class, the def record class that’s generated actually implements an interface that’s part of the protocol. And so when you invoke that thing, it goes through the Java interface dispatch at a much lower level in the JVM, so it’s faster. And so it does make a difference.
There’s definitely a tradeoff there between performance and, I guess, expressivity and how you write your code. Sometimes you might want to extend something in the core library or outside of a def record form, but there’s definitely a tradeoff to be aware of there. That’s only on the JVM though. On ClojureScript there’s not a similar penalty.
CRAIG: Yeah. No, that was a good one - definitely. There’s other things like that in there as well.
PAUL: Yeah. My thought was, again, just a short book. I’d like to write more in this same vein exploring different themes other than polymorphism.
CRAIG: Yeah. I wanted to ask you about that because I think you make that pretty clear in the material and surrounding the book that this is potentially Paul on Clojure volume one, right?
PAUL: Yeah, right.
CRAIG: Maybe a less ambitious title. I’m not sure, but I would certainly read that series. That would be great. What was your, I guess kind of -- what was your experience of writing the book and then carry that through to what you think about continuing?
PAUL: Yeah, I mean, again the way it came about, I was kind of already thinking of this talk for the Clojure/conj, and I don’t know. It happened pretty quickly. I mean I just kind of -- it just flowed, you know, I wrote the book, so I don’t know that the next one would necessarily go so easily. You know I’ve been kind of thinking through different topics.
I actually had this idea for doing one on optimizing Clojure code. The other day at work someone was complaining about something they were trying to do and couldn’t write, find the right function for it, or something. I said, “You know what? It sounds like what you actually need is not a function. You need a data structure. You need a heap data structure because you want to basically have all these items that have, say, a cost to them. You want to pull the lowest cost item out first, and then the next lowest cost and so forth,” and so a heap would be ideal for that.
But I was like, “But I don’t really know of any heap implementations for Clojure offhand,” so I thought, well, it would be interesting to take that as an example and write a heap data structure that can integrate into Clojure and then optimize it, tune it after it’s been written because obviously a first pass is probably not that performant. I don’t know if that would work as a book.
I thought about maybe some YouTube videos. I don’t know, but yeah, I’ve kind of noodled around on some other ideas. If anybody has any that they would particularly like to see covered, certainly let me know.
CRAIG: Awesome. Prepare for a tweet storm because I think people would love to hear more. I mean I could think of a few off the top of my head, things like just kind of in this vein of what is somebody who has come to the language that can write programs in the sense of taking the ideas that they’ve had in other languages and expressing them in Clojure, but not necessarily idiomatically or maybe going beyond that in a way that’s congruent with the philosophy of Clojure. I could imagine, for instance, one of the things that people struggle with when they start out is state. You know.
CRAIG: I’ve got this program, and I write it in Clojure. It looks like Java to some degree where there’s like 20 atoms and everything I do I do by mutating those atoms. It’s like, well, okay. It’s not usually how we do things. There’s another way to structure. That would be one area where I could see.
Another one that I think is a great intermediate topic even though arguably it’s a beginner topic is name spacing, code organization.
CRAIG: I feel like that’s one that people get hung up on a bit.
PAUL: Yeah, name spacing, state and/or concurrency are something I’ve considered. Another one is error handling. I tried as much as I could in Clojure Polymorphism to write in a way that the principles that we’re learning apply to both Clojure and ClojureScript. Error handling is a little more complicated because it tends to be more platform specific. But certainly, I mean there have been times where I’ve interacted with other people at work who are like, “Well, how do I signal errors, catch errors, or whatever?” There are different approaches to that, too, that have their own tradeoffs.
CRAIG: Mm-hmm. Well, cool. Well, we will definitely be interested in tracking your further writings on this or any other topic.
You mentioned at work, and I am embarrassed to admit that I have lost track of where you’re working now. We have known each other since you were at Sonian.
CRAIG: You’re not at Sonian any more. Where are you working these days?
PAUL: I’m at a company called Looking Glass that’s based in Baltimore. We’re in the network security space. Before that I was at a company called Outpace, which was a great experience because I got to work with a lot -- I mean Outpace ended up doing a lot of hiring and hiring some really great people. I got to work with Carin Meier. I think it’s at -- is she at Cognitect now?
CRAIG: Yeah, she’s at Cognitect now. Yep. Yep.
PAUL: As well as lots of other people at Outpace as well, and then before that Sonian, which also is a great team. Looking Glass has a great team too. They’re based in Baltimore. I work from home. I’ve been working remotely for companies, well, since Sonian, since about 2009.
CRAIG: Mm-hmm. Yeah, that’s actually something we should talk about in a minute. I just want to mention that it’s funny. I mean it is a big world now compared to what it used to be, but it’s still a small world. You know. You’ve worked with Carin Meier, who is at Cognitect. Devin Walters was also at Outpace.
PAUL: Mm-hmm. Yep.
CRAIG: And he’s been a guest on the show. He and I have worked together on a bunch of different things and Looking Glass.
PAUL: Yeah, Devin is great.
CRAIG: We’ve done a bunch of work with Looking Glass as well, and I actually taught a Clojure class at a company that Looking Glass had acquired recently.
PAUL: Yeah. Yeah, I was working with Tim Baldridge for a little bit at first at Looking Glass, and then also someone else who I think was working with Cognitect, Creighton Kirkendall who actually was at Outpace. I didn’t know I was going to work with them at Looking Glass until I started at Looking Glass and I saw his name in the chat list. I was like, hey, that’s funny.
CRAIG: Yep, it is funny.
PAUL: Yeah. Yeah, it is. It’s a small world, although it seems to be getting bigger. I’m surprised, having been involved and working as a professional Clojure programmer essentially since 2009, I’m surprised sometimes to hear of some companies. I’m like, oh, wow, they’re doing Clojure too. That’s great.
CRAIG: Yeah, absolutely. It’s the same here, especially at the conferences. Right, so -- oh, yeah. I wanted to talk to you about working remotely. This is something that we find people are very interested in.
CRAIG: We’ve had people request shows on it before. We’ve done shows on it, but I think there’s always something more to say because, as more and more jobs in the world become information jobs and not like you’re in some place actually physically processing bits or responding to events in the real world, I think this is something people are interested in.
It’s obviously transformative, both for an individual’s life to a greater or lesser degree. I think maybe a greater for you. We’ll get you to talk about that in a minute. But also for the world, I mean things like where people live, population density. These things are all impacted to some degree by the ability of companies and their propensity for hiring remote people. I don’t know if you have any general advice or particulars of your experience you’d like to share, but I would love to hear what your take on the remote life is.
PAUL: Yeah. Yeah. No, I have opinions about that I mean having worked remotely for a while. Well, I mean one thing is I draw a distinction between the idea of working remotely versus working on a distributed team. I think those are qualitatively different.
When you’re a remote person working for a company that has an office somewhere and everybody else goes into the office, that’s a different experience than it is if you have a distributed team where essentially, say, you don’t have an office and everybody works not necessarily from home, but wherever they want to work from. I mean I’ve worked with people who go into a coffee shop every day. You don’t have to necessarily work from home.
But, yeah, the flexibility to kind of decide where you want to work is interesting. Like you said, it has implications personally. For one, about three years ago my family actually moved. We were in northern Virginia near the D.C. area and we moved down into central Virginia because we kind of liked that area better.
Not to go into too much detail, but we saw in the future our families kind of moving down in that area too, so we wanted to be closer to them. Being able to move to another place because you like that place and not have to change jobs is kind of cool. It’s kind of a nice thing to be able to do.
I don’t think working remotely works for everybody or certainly not for every profession. Like you said, it’s probably more information related professions, programming, writing, things like that. But even someone who is a programmer or a writer may not necessarily enjoy working remotely. I think that’s fine, but if you embrace it as kind of a new paradigm of working and not just as, like, a remote person with an office and kind of go the distributed team route, there are lots of great tools to help that work well. I think it can work really well.
It has a great impact, I think, on a company who can hire people from essentially anywhere. I’ll put some limitations on that. If you’re working, especially if you’re doing any kind of pair programming, time zones matter. That’s kind of a limitation on the hiring, but essentially you don’t have to limit yourself to one particular city where you’re hiring. Obviously, like you mentioned, it has benefits for employees in kind of deciding where you want to live. There are definitely different ways of doing distributed teams or remote work.
For me, I think one essential component is having meet-ups every once in awhile. I think there’s definitely benefit to face-to-face contact for--I don’t know--just camaraderie, morale, or whatever you want to call it. Sometimes there are tasks that are easier to accomplish face-to-face too, so typically in companies I’ve worked at we’ll have a meet-up maybe two, three, four times a year, or once a quarter, say. Part of that will be like brainstorming on a whiteboard or whatever, some kinds of things that might be a little bit easier to do in person. But I think that the meet-up is kind of an essential part of it.
I think, to me, if I were to say like what are the kind of essential components of distributed teams, I think having regular meet-ups. I think pair programming is definitely very helpful. I think having an open communication system, some kind of chat where there’s transparency and people can see what people are talking about because you miss out on hallway conversations and things not being in an office. But I think you can kind of recreate that to some extent by just having open chat channels where people can come and go, or if they want to go back and reread some back chat in a channel or something. There’s more access to information that way. I think those are important components of a distributed team.
CRAIG: Yeah. That lines up a lot with our experience and with the experience of other people we’ve talked to. I wouldn’t mind getting your take, because you’ve done this a few times now, on what particular tools and practices. I mean you mentioned a few, generally: chat and whatnot. But I’m interested in things, for instance, really detailed things like what do you use for white boarding when you’re distributed? That one is one that I don’t feel like I’ve cracked yet. Then screen sharing is another one where there’s actually a bunch of solutions that work pretty well now, but I’m curious to hear what you’ve found that works the best, at least for you.
PAUL: Yeah. The tools have changed over the years. I mean when I was at Sonian, it’s mostly an audio-based culture there, so we did a lot of Skype and a lot of audio calls. We would have stand-ups, which again I guess I missed that. I think that’s another important component is having stand-ups every day. It helps keep people on track. But we’d do stand-ups with Skype.
Yeah, and there we tended to mostly use chats or, if we wanted to do some kind of design, we might pair on it and write, actually write up the design document, so there wasn’t a lot of white boarding. But I mean you can get creative. I mean at Outpace there were people who had -- Outpace was a more video-based culture there, so there was a lot of video calls and video pairing. People would get kind of creative and have a couple of video cameras, and one of which could be pointed straight down at their desk, and they could just draw things. It’s not as easy to collaborate that way. At least you can see if someone is trying to explain something and wants to draw some pictures. That’s helpful.
We experimented with some white boarding. I think at Outpace I forget exactly what tools, but we started using Hangouts a lot there and then eventually moved to Zoom, which is a pretty solid video conferencing and also screen sharing tool.
As far as chat at Sonian, we did a lot of IRC, which I kind of like. I think it’s a standard and it’s been around forever, and you can get tools and clients and things to work with IRC, but not everybody is comfortable with that. More recently, Slack has been kind of becoming a more de facto tool for chat purposes. IRC is a little better in the sense that it’s more open, I guess, than Slack, necessarily, but Slack does keep the history and IRC doesn’t unless you set something up to capture back chats so you can go back and replay things. You can’t really see what people had been talking about in the past.
CRAIG: Yeah, that’s actually something I really like about chat. As you say, it’s not incompatible with IRC, but the Slack experiences that out of the box. Interestingly, when I was at Room Key a couple of years ago, they switched literally overnight from IRC to Slack and didn’t go back. They quite liked that, and I like Slack. I think it’s a good program. At least in terms of the user experience, it’s a good program.
PAUL: Slack has some options. I mean it does have an IRC bridge. It has the native client, the Web client. There’s also Hipchat. We’ve -- I’ve used that, and we use Hipchat a lot more at Looking Glass these days, so I mean that’s an option too. You know there are tradeoffs between these tools.
PAUL: Yeah, those are kind of some more concrete examples of things that I’ve seen used.
CRAIG: I’m bummed that you haven’t cracked the white boarding problem because that one feels like one that is a missing tool, for me at least. Although, the idea of using two cameras, that’s brilliant. I have to start doing that. That’s fantastic.
PAUL: Even before that, like if I was trying to explain something to somebody, from the day that I started working from home, the first thing I did was I went out and I bought one of these print centers. It’s like a laser printer/scanner/copier, which is also useful just around the house because we homeschool. And so a lot of times we want to copy something or whatever. I would draw little diagrams, scan it in, and then email it or whatever, which is a slower way of doing it, but, yeah, having an extra camera.
I feel like there was some white boarding tool we experimented with like Whiteboard.io. I have no idea. Something like that, but I don’t know. I mean honestly I don’t feel like we’ve had a lot of need for white boarding. Maybe it just depends on the company or the person. Maybe you like to doodle more on white boards. We tend to do white boarding in person at meet-ups, usually.
CRAIG: Yeah, so that’s interesting because I wonder to what extent we let our thought processes be shaped by the available tools. This is something I’ve been thinking about in a bunch of contexts. You just mentioned one. It’s like, well, if we’re in the same room together and there’s a whiteboard, I’m very likely to pick up a marker, go to it, and start drawing. But if I’m on a call, even if it’s a video call with you and we’re trying to kick an idea around, I’m less likely to do that. And so I feel like that must be because -- I don’t think the conversation is different. I think it’s that there is a barrier that I’m instinctively or unconsciously avoiding climbing to get to the end result.
I think, in particular, that that type of barrier is manifest in Clojure tooling itself, to be careful not to say the language. What do I mean by that? Specifically that we oftentimes work with these data structures. Everybody that’s done any Clojure for more than a few months has had a data structure that is a deeply nested map of vectors of maps of vectors with keyword keys and this really complicated thing, or at least nested thing even if it’s not complicated.
Like the default experience in the tooling is to print it, which is awesome, right? Don’t get me wrong. Having that is so much better than, well, I’ve got a pile of objects and no way to look at. You know most debuggers are going to give you a way to expand that out hierarchically, so you know credit there. But I feel like what I want is something richer, something where I can really visualize, make use of that huge--what is it--the prefrontal cortex, whatever the visual center of your brain is, this thing that is really good at processing visual information. Make use of that to throw things up on the screen and sort through them in a way that I at least find more difficult when it is ten screens full of text, if that makes any sense.
PAUL: Yeah. You know I could be wrong about this. I seem to remember. I’ve never used it, but from very early on, like even the 1.0 days in Clojure, wasn’t there some kind of inspector--
PAUL: --that would pop up a swing window, and you could, like, actually expand the parts of your data structure or something?
CRAIG: Yep. Yep, and it still exists.
CRAIG: I mean just like almost everything in Clojure--
CRAIG: --you know it doesn’t go away. I do remember. I’m speaking from mostly ignorance here. I remember trying it and it not being the thing that I’m hoping for. How could I be more vague than that, right?
CRAIG: Yeah, but you’re right. That’s something that I should go back to, look at it, and see where it differs from what it is I think I want or maybe it’s just a matter of familiarity. But I do feel like my process would be improved by incorporating a better visualization experience. Just to be fair, I think people are doing this.
I don’t use Cursive just because I’m a hardened Emacs user, not because I don’t think it’s a good tool. I actually think Colin is doing great work. But I don’t use it, so I don’t know what the tools are there. Then I think if you look at some of the stuff like Kovas was doing with Session and some of the other interactive REPL stuff, that stuff is also super promising.
PAUL: Yeah. Yeah. No, those are definitely cool. I’m the same way. I’m an Emacs user, and I’ve been doing Clojure for a while where we didn’t necessarily have all these tools. Now I’ll say that, but if we want to go back to like the first thing that I ever did with Clojure, the first thing I ever did with Clojure was to try to get Clojure running on Terracotta, which is this shared memory system, distributed memory system for the JVM where you can have several JVMs running on different network nodes that share the same heap, essentially.
I remember one of the things I did was hook up a Java debugger to the JVM because, since it’s on the JVM and Clojure compiles it by code, you can do that, which I think was brilliant. One of the first things I ever did with Clojure was step through the code with a Java debugger where I would step from a line of Clojure code to a line of Clojure code to a line of Java code back to a line of Clojure code. Certainly the JVM, that kind of debugger exists now. It’s maybe not perfect because it’s possibly more oriented towards Java, but certainly you can step through code. I tend to go the println route for debugging. I just add a bunch of printlns everywhere, which is definitely a suboptimal way to debug.
CRAIG: There’s your next book: debugging.
CRAIG: Right? I think that would actually be real useful.
PAUL: I’d have to actually learn to do it right, though.
CRAIG: That’s a great excuse, though, right? I mean I think that’s actually a good question for you. When you were writing the book, they say the best way to learn is to teach, and that’s been my experience that I always learn something when I go to teach it. Did you learn anything that you didn’t know or that you got a new perspective on in the course of writing your book?
PAUL: Yeah, certainly. Having come to Clojure early, I am much more comfortable with the JVM side of Clojure than the ClojureScript side. And so I definitely learned a few things about ClojureScript. For one thing, I thought the penalty for extending a protocol outside of the def record might actually be a penalty in ClojureScript too, but I found that it wasn’t.
CRAIG: The things that you can do with that and, oh, man, having ClojureScript for that is so interesting. Very briefly, I have a tool. It’s a weather simulator. It generates weather, but it doesn’t really matter. It’s this algorithm.
PAUL: It generates weather?
CRAIG: Well, not in the real world. It generates weather data, I should say, or weather-like data would probably be a better way to put it.
PAUL: Okay. That makes sense.
CRAIG: Right, so you run it, and you can see, oh, look, there’s a storm here and it’s moving this way. Over time, it dissipates, that type of thing. It doesn’t matter. It’s some algorithm.
I’ve got that algorithm, and it’s all in cljc files, and so I can run a REPL in the JVM and play with it all I want in that environment. Now I’m going to write a visualization layer in the browser. Of course, it renders as HTML. But now I’d like to have a command line tool for spitting out a bunch of data files to feed into some other tool. Well, the code in the model, the thing that actually generates the weather data has to change not at all.
CRAIG: Right. I mean I really feel like this is crazy awesome. Then on top of that, even the code for doing the UI I have not yet played with, but I am very interested to begin playing with Electron. I don’t know if you’re familiar with that at all.
I’ve done some preliminary experimenting. Many, many other people have done far more than I have that it looks really promising. Startup times are very, very low. I wrote a "Hello, world!" ClojureScript. This is in electron. It’s Node, but took that and compiled it into a Node native executable. The startup time on my laptop, for whatever that’s worth, was, I want to say, 70 milliseconds. It was decent, right?
CRAIG: It was actually something along the lines of what you would expect from a command line. Anyway, sorry I took over the conversation there. I didn’t mean to do that, but I’m kind of excited about the fact that we have these two first class runtimes that we can leverage in different ways and in different contexts from the same language.
PAUL: Yeah. You know what I did? I experimented when ClojureScript first came out because it was kind of the first instance of a Clojure compiler written in Clojure, or at least something from Rich. One of the things I did was I experimented with trying to target ClojureScript to ELisp for Emacs.
PAUL: You could write ClojureScript code and compile it down to Emacs, ELisp files and actually script Emacs with ClojureScript, which I thought would have been cool. I got as far as -- the annoying thing about Clojure, at least at the time that ClojureScript compiler was -- it very quickly gets into using def protocols, def records, and def types.
CRAIG: Yeah, it is pretty awesome. A total random thing you just reminded me of. Today I “discovered” that Emacs has a full on terminal emulator. I’m not talking about EShell or the rather weak shell, but like a full on terminal emulator. It’s powerful enough to run VIM.
CRAIG: You can fire up an Emacs window and have VIM running inside of it. It’s pretty, pretty surreal. Anyway, sorry. That was completely random and nothing to do with the very, very interesting things that you’ve been working on.
I’ve got a bunch of things I want to ask you about yet, but I think I want to steer it back to the book. I mean it’s a short book, which I really like. You’ve been around long enough. You probably remember the days when if a computer book wasn’t heavy enough to cause near fatal injury if it fell off a shelf and hit you on the head then nobody would buy it, right?
CRAIG: I think one of the great things about a lot of the Clojure books out there is that they’re not 500 pages. And so your book at--what did you say it was--38 pages?
PAUL: Like 31, yeah.
CRAIG: Yeah, it’s awesome. I think that’s fantastic. I think that also helps in a conversation like this where we have an hour and we can actually touch on all the major points. But I wanted to loop back and see if there’s anything else that you think we should mention to people about the book.
PAUL: That’s a good question. I should probably have something to say. No, I mean, like you said, it’s a short book. We kind of talked about the meat of it. There are a couple of different examples I go through. There’s a little bit of introduction about the polymorphic tools in Clojure. I’m not sure there’s anything else that’s jumping out to me.
PAUL: Unless there was something that you wanted to….
CRAIG: No, not at all. I really think it’s a very nice, self-contained piece of work and definitely recommend to people. Cool.
Well, then there’s a few other things we could talk about, but I think we may have to save those for another day. We’ll see because I always like to leave space in the episode for anything. We generally have a primary topic, and obviously today was the book, but I always like to leave space in the episode for anything else that the guest would like to talk about, so interesting things you’ve been working on, stuff you’re excited about, or just things that you think are important to talk about. Do you have anything along those lines that you would like to discuss today?
PAUL: I mean something I’ve been thinking about lately, and this is probably a larger topic than maybe we could really do justice to at this point, and it’s not necessarily Clojure related, but I kind of got kicked off on this exploration in the past few years because I was just kind of like looking at the state of software engineering in general and thinking why is it so difficult. Why is it that many companies you go and you work at, it seems like everybody has a different process for developing software? Oftentimes they’re not always effective. You know projects still end up being late. They end up being over budget.
I just kind of started doing this soul searching about, like, what is software engineering, and what other kinds of engineering out there, like what kind of processes do they use? Is there something that we could learn from a different practice of engineering? I kind of started off on this journey looking at a few different resources and ended up a lot of times at Richard Gabriel’s website. He gave a talk at Clojure/west. Actually, I think it was the first Clojure/west conference. Then there were some other things he has written that are really interesting.
I saw a talk by Glenn Vanderburg. This is also online you can find about software engineering. Then also, like, I think it was actually Donald Knuth’s Turing Award lecture. He talked about the relationship between science, engineering, and art, which is really interesting.
But, yeah, I mean I kind of got set off on this path about, like -- I mean I had been told in college, “Science creates a body of knowledge. Engineering applies it to solve problems within constraints.” That was the definition of engineering. But, in particular, Richard Gabriel’s talk, even Glenn Vanderburg’s talk, they both kind of hit on this same topic that actually many times engineering precedes science and discovers something that then later science systematizes or something like that.
Richard Gabriel has a really interesting talk/paper that he’s given called Better Science through Art, which is really fascinating. I found it really fascinating to read and just in it he kind of develops this idea that science, engineering, and art really all use the same process, and it’s really about disciplined noticing.
PAUL: An artist develops a skill of noticing things in a disciplined way and representing them. Engineers as well and scientists all have kind of this at their core, the same idea of disciplined noticing. It’s a really interesting talk.
CRAIG: Yeah, that is interesting and, you know, brings together three of the things I’m interested in. Obviously we talk about art on the show.
CRAIG: We describe ourselves oftentimes as computer scientists, but I am, by education, an engineer and actually think of myself more of a software engineer than a computer scientist, certainly. What’s your take on it all then? I don’t -- not to put words in your mouth, but my guess would be that you would say that it doesn’t really make sense to talk about the question of what you and I do for our jobs is a science, is engineering, or is an art. It just doesn’t.
PAUL: I mean, well, actually it might. I mean it depends on how you view those three disciplines. I don’t think Richard Gabriel’s point is necessarily that they’re the same thing, but they’re kind of different approaches that have kind of a similar thing at the core. But no, I mean I think it does matter. One of the reasons I got kind of kicked off on this exploration and then one of the places I kind of got to is I feel like my experience of writing software is much more creative, much more exploratory. I don’t necessarily know where I’m going, or I may go a certain place and then come back and then go somewhere else, and so there’s kind of a circuitous path that I take to actually find where I’m going.
There’s echoes of that in agile methodology where the idea is that requirements will change. Sometimes you don’t actually know what you want to do or what’s possible until you actually start, and so you kind of get there iteratively. Thinking through all that, I’m thinking, well, what engineering methodologies do we have for writing software? And do they match with that?
If writing software is essentially a creative act that has more in common with, like, actually writing a book where you’re writing and rewriting, and there’s an iterative thing. You may throw sections out, and you write new sections. Does it make sense to think of it as like a science or engineering kind of a thing? I don’t know that anybody today necessarily thinks we can start at the beginning of a project and create a Gantt chart and expect that that’s going to work until the end of the project and everything is going to work perfectly that way. But I think there was still maybe a mismatch between how we view and manage software production versus what the reality of it is, at least in my experience as a more creative kind of thing.
CRAIG: Oh, yeah. I totally agree with that it being creative. I think in and of itself the following is proof of that. The number of times that I have solved a significant work problem while asleep or in the shower or running is way higher than ten. And it might even go as high as 50% of the projects I’ve ever worked on have had -- my contribution would not have been what it was without a moment like that. To me, that just points out that there is a creative aspect to it, an important creative aspect, certainly.
PAUL: Yeah. No, I mean it could be different person-to-person how they approach it, and it may be different problem-to-problem. I mean I don’t know. If my job was essentially to recreate the same system over and over and over, but imagine I was constructing houses and I had a design. I’m stamping out that same design over and over and over. I would have a much better ability to estimate how long it’s going to take. I’d have a much better understanding of what are the kind of things I’m going to run into, how to actually plan that out, and manage that. You’re more interested in the efficiency of the process and getting these things stamped out.
For me, I feel like my job more often is, like, doing new things every day that I’ve never done before, like we have … and ElasticSearch. We’re going to conduct them in this way, which we’ve never done before. What are the implications?
I feel like we’re creating such complex systems these days that you really almost have to take a more empirical approach than an analytic approach, I guess, where from the outset you look at this thing. You say this is a system. These are going to be the performance characteristics. If we make this change to the thing, it’s going to cause us this effect.
I don’t know that we can necessarily do that so easily with the complex systems that I work on. I feel like it’s a more empirical thing. You have to make a change, measure it, and see how it affects things, and then maybe tune it a little more. It’s a more exploratory process, and so I wonder whether at times we’re kind of viewing the software development process in the wrong light.
But like I said, it may be different. I mean maybe for some people it’s like their job is much more creating the same things over and over, and they have a better understanding versus doing new things every day. I think, at the very least, our approach to understanding the process and managing the process should be flexible enough to see that those are two different things, and we have to, like, expect that something that’s more exploratory is going to have to be more of a -- I don’t know. You’re not going to know everything upfront.
PAUL: And you need to expect that. Whereas, something that maybe you have a better handle on how to do you would manage differently.
CRAIG: Yeah. Did you see the talk by -- I forget his name. There was a guy from JPL. He gave a talk at this most recent Strange Loop 2016. Did you happen to catch that?
PAUL: I don’t remember. Do you remember the title?
CRAIG: I don’t, but it was something along the lines of lessons from waterfall or something related. Anyway, I’ll just summarize briefly and then we’ll post a link to the talk. Maybe along the way you’ll remember if you’ve seen it or not.
His observation was that obviously there’s agile, whatever that means to whomever, and that’s sort of a modern, considered to be a modern software practice. There’s waterfall, which is considered to be an antiquated software practice.
CRAIG: But really his observation was that there’s a risk tolerance spectrum. That’s one of the elements that goes into project planning. He’s at JPL. They’re working on projects from NASA. They’re writing software that has to work exactly one time over the course of 30 seconds in space.
CRAIG: And if you get it wrong, a 1/2billion artifact is destroyed, right? Mars Lander smashed into the surface or whatever. Versus the soft launch of your website that sells earmuffs for kittens. You can get that wrong, and you’re not going to go out of business. You’re not out half a billion dollars. It’s not like you can never do it again. And so there is this element of risk tolerance.
He said, look, you know, it’s important. And this is, I think, what you were saying is it’s important to understand where you are on that spectrum so that you can pick methodologies that are appropriate. Some of the aspects of what some people would say is a heavy weight or over-engineered process are actually about appropriate risk mitigation for high risk projects, right?
CRAIG: Like having a big plan and having a testing plan that’s done before you go into production. These things are important for certain types of projects. I think that’s very much congruent with what you were saying.
PAUL: Yeah. There was actually. There was an article by Tom DeMarco called Software Engineering: An Idea Whose Time has Come and Gone?, with a question mark at the end. That was in IEEE Software in 2009. He said kind of a similar thing. He said that a lot of times control over a software project is something that people are concerned about, but he gives an example of, like, if you have a project that’s going to cost $1 million and produce $1.1 million of value, you definitely want to control that very tightly. But if you have a project that’s going to cost $1 million and it’s going to produce $50 million of value, you’re not so much concerned about control and efficiency there. And so you maybe manage it differently. Yeah, that’s an interesting article as well. I think he hits on some of the same points.
CRAIG: Cool. Well, as you rightfully said, this might be a topic that we could spend more than what might be an appropriate amount of time on this particular episode of the show, so I will invite you back. We should talk about this and other things and ceiling wax and however that goes some other time.
CRAIG: But assuming that you think we’ve done that topic justice in this space anyway, maybe we should move on to the final question. What do you say?
CRAIG: Okay. Well, our final question is the same as always. It is a question to our guest. The question is, what advice would you like to share with our audience? This could be any kind of advice that maybe you’ve received or maybe you like to give, but something in the form of advice. What do you have for us?
PAUL: I think the advice I would give, something that I’ve found useful for me is to pursue interests for the love of it. I have found several times in my career that something that I just found interesting and I was, like, reading about or playing with in my spare time ended up becoming important actually in my day-to-day job. I think, especially if you look at software as kind of a creative pursuit, reading lots of other people’s code, writing lots of code, being exposed, even interdisciplinary, ideas outside of software can be useful and help you improve your craft as a software developer. I think being open to ideas from other fields and just pursuing things just for the love of it. If you find something interesting, pursue it, and I think you’ll find it’ll become valuable later on.
CRAIG: I totally agree. That’s excellent advice, in my opinion. Paul, thank you so much for taking the time to come on the show. Always a pleasure to talk to you. I don’t remember the last time we saw each other. I am so bad. I’ve been to Strange Loop, and they all kind of blur together in my mind.
CRAIG: You know what I mean? I can’t tell whether -- in fact, I was talking to Jason Gilman, and I’m like, “Oh, man, I’m sorry I missed you at Strange Loop.”
He’s like, “What are you talking about?”
PAUL: It’s like, we talked.
CRAIG: “We sat next to each other and talked for like 15 minutes.”
CRAIG: I was like, oh, I thought that was in 2014. Anyway, so I’m probably about to do the same thing with you. Were you at Strange Loop this year?
PAUL: I was not at Strange Loop this year, no.
PAUL: I’ve been at every Conj though. I don’t know if you were at the last Conj.
CRAIG: I was.
CRAIG: I was super busy, but I was there. Yeah.
PAUL: Yeah. Yeah.
CRAIG: Are you going this year?
PAUL: I am, yeah. I am on a perfect attendance streak. In fact, I think they should give out perfect attendance awards to those of us who have come to every Conj. I have every shirt. In fact, it’s kind of embarrassing because, I mean, when you look back at family pictures of vacations and stuff from like four or five years ago, I’m wearing a Clojure/conj shirt when we’re at Disney World or whatever. They’re some of my favorite shirts, but no.
Yeah, we might have run into each other at the last Conj, I think that would be the last time.
CRAIG: Yeah. Well, anyway, if you’re going to be at this one, I will too, and so it’ll be great to see you there, but it certainly was great to see you or to talk to you today. If you do go to the Conj, by the way, bring that Conj number one shirt. I’ll bring mine, and we can confuse people even further by wearing them--
PAUL: Right. Nice.
CRAIG: --and standing next to each other. But, yeah, thanks so much for taking the time. Super interesting stuff. The book is excellent. Highly recommend people buy it. You won’t regret it. It’s a good read. Like I said, not much money, not much time. How could you go wrong? Awesome job on that.
But we will go ahead and close it down. Thanks again for coming. This has been The Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
CRAIG: You have been listening to The Cognicast. The Cognicast is a production of Cognitect, Inc. Cognitect are the makers of Datomic, and we provide consulting services around it, Clojure, and a host of other technologies to businesses ranging from the smallest startups to the Fortune 50. You can find us on the Web at cognitect.com and on Twitter, @Cognitect. You can subscribe to The Cognicast, listen to past episodes, and view cover art, show notes, and episode transcripts at our home on the Web, cognitect.com/podcast. You can contact the show by tweeting @Cognicast or by emailing us at email@example.com.
Our guest today was Paul Stadig, on Twitter @PJStadig. Episode cover art is by Michael Parenteau, audio production by Russ Olsen and Daemian Mack. The Cognicast is produced by Kim Foster. Our theme music is Thumbs Up (for Rock N' Roll) by Kill the Noise with Feed Me. I'm your host, Craig Andera. Thanks for listening.