In this episode, we talk to Joseph Wilk about Clojure, music and art.
Headphone concert, one example
Overtone
Visualization with music - Joseph's live coding demo
Sonic Pi
Sam Aaron
Scheme Bricks - Dave Griffiths
If Hemingway Wrote in Javascript by Angus Croll
Quil
Openframeworks
Prolog
Fluxus
DrRacket
thi.ng
EPISODE COVER ART
AUDIO PRODUCTION
PRODUCER
Our theme music for this episode is Thumbs Up (for Rock N' Roll) by killthenoise with Feed Me which was used under a Creative Commons License.
In this episode, we talk to Joseph Wilk about Clojure, music and art.
CRAIG: Hello, and welcome to Episode 97 of The Cognicast, a podcast by Cognitect, Inc. about software and the people who create it. I'm your host, Craig Andera.
Well, as always, we have a few things we want to make you aware of. First up is Clojure/west, a big one for us, of course. Our conference is, April 15th and 16th, 2016, out in Seattle this year.
Coming up before the conference, there will be a couple bits of training that we'll be putting on. There is the Intro to Clojure course. That's the 13th and the 14th, so the two days before the conference. I've taught that class. I think it's pretty good. You should check it out if you're interested in getting into Clojure. We've actually had people who'd been doing Clojure professionally come to the class and find it valuable, so you should check it out. There's information about all the conference and the workshops on the website at clojurewest.org.
In addition to the Intro to Clojure course, there's also a Datomic Workshop taking place on April 14th, the day before the conference. Again, details about that are at clojurewest.org. Of course, you can get your tickets there as well.
All right, well, there are also a couple ClojureBridge events coming up. There are still a few seats left for ClojureBridge London happening February 19th and 20th, and there's a ClojureBridge Boston coming up March 18th and 19th. Again, this is all in 2016. Information about ClojureBridge, including how you can donate, is available as always at clojurebridge.org.
I also want to mention Dutch Clojure Days. Now, this is a free event. You can search for Dutch Clojure Days, and you'll find their website. It's at ticketbase.com/events/dutch-clojure-days-2016. Searching for it is probably your best bet there, but that's happening March 19th in Amsterdam. Again, it's free. It looks like a pretty interesting event. Carlo Sciolla, a former guest on the podcast, let us know about that. He's involved, so I think that'll be a fun events for you to check out.
I think that's everything I want to mention today, so we will go ahead and get on to Episode 97 of the Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
CRAIG: Ready when you are.
JOSEPH: Yep. Sounds good.
CRAIG: Cool. Well then, let's go ahead.
All right, welcome, everybody. Today is Friday, January 22nd in 2016, which, just like it does every year, sneaks up on me and seems unreal that I'm saying that number: 2016. But, here we are, our first interview of the year. We are thrilled to make that interview a conversation with Joseph Wilk, a developer, an artist, and a well known Clojurist, and many other things besides. Welcome to the show, Joseph.
JOSEPH: Thank you.
CRAIG: It's really great to have you here. Glad you were able to make it today. I know it's a little bit later where you are over there in the U.K., but I'm glad you were able to take time.
And so, we're going to start off, as we always do, by asking you a question, one, which I think is especially appropriate. It's a question about art and, as we said, you are an artist - one, in fact, who blends some of our favorite topics with art, as we'll get into. We ask people to relate some experience of art, and I wonder what you'd like to share with us.
JOSEPH: I think a music concert I went to quite a while ago in a very old, abandoned warehouse in central London, which was a headphone concert, and so kind of walking into the performance area was absolute silence. You kind of had your headphone jacks dotted around the room, and you kind of just got to observe everyone enjoying the music, not actually hearing the music.
Something that really, I guess, struck me about that was you kind of had various, I guess, DJs, different performers kind of coming on. Obviously since the sound was like a choice, you actually plugged in, people would unplug their headphones and were just relaxing. The artist, when they came on, had to really grab everyone's attention.
There was one guy who had constructed a costume out of kind of polystyrene. It was kind of like almost this alien shape and, as he came onto the room without any music, everybody just immediately plugged in their headphones and started listening, waiting to hear what this guy was going to sound like. I really liked that aspect of not just the music, but the theater around the performance that is part of that. CRAIG: That's really interesting. I'd never heard of that before, so that's a very cool concept, and it's interesting to me that you point out the connection between the visual aspects of the performance and the audible one, especially given that, now that I think about it, they're always there, right? There's often a lightshow involved in concerts and whatnot, especially for maybe more energetic music. You might not expect that too much with something like a chamber piece, but I guess at that point they're almost separated to the point where you have to pay attention to one or the other.
JOSEPH: Yeah. It's an experience, and you can put on a CD, and you can listen to the music probably better fidelity than you will ever have in a live venue with a big sound system. But, it's about that experience and about being part of it with everyone and experiencing the artist performing.
CRAIG: Well, this whole blend between performance and the visual aspects of performance and the audible aspects of performance, I can't think of a better person to talk to about this, especially on this show, than you because of the work. Some of the things that you've been doing touch directly on this.
I won't try to put words in your mouth. I think most people have probably heard of the things you've done, but I'll let you tell the story. You've done a lot of work with both music and visual aspects and specifically in Clojure. Isn't that right?
JOSEPH: Yeah. I guess I'll start where I started, which was getting kind of into AI generative music and messing around with visuals with that using Overtone, actually through slight sleep deprivation of having a newborn baby. I started coding at crazy hours, and I had the daydream of creating AI generated nursery rhymes for her, which was a lot of fun, but kind of got to the point where I'm not a trained musician, so I'm kind of making it up and learning through code as I go along.
I kind of got to the point where I understood the AI and kind of understand logic, but didn't really understand, I guess, the details around music theory, and so I kind of started diving into that and really kind of, I guess, started moving towards performance from having to learn music theory. Code is a fantastic way to learn music theory, and I've always been a very visual person. I've always done and am obsessed with various movies and lots of cinematic effects, so I kind of like those performances to merge some of the, I guess, cinema aspects into them.
Also, something to distract in that, as I said, I had no music theory, so the first time I ever went and performed kind of coding, live coding and music and visuals, the visuals is also an amazing way to gain you confidence because, with visuals, it's much easier to know if they're good or not or if they're acceptable or not. With music, it's much, much harder because it's such a broad range and so many different audiences, and someone is always going to hate and like it. The visuals almost give you a confidence to go out there and perform both, music as well.
CRAIG: Yeah, so I took a look at some of the performances you've done. They're really interesting. They're sort of, oftentimes I guess I might use the word, swirling visualizations as a backdrop to the code that you're writing, which in turn is producing music, which in turn, I believe, is producing the visuals. And so, this is something you've done live, as you said. Is that something you do frequently?
Also, I wanted to ask you. When we were talking before the show, I said, "Well, how would you like to be introduced?"
You said, "I'd like to be introduced as an artist."
I go, "Okay, that clearly applies in your case."
When you say that you think of yourself as an artist, is that the thing that makes you think of yourself as an artist, or are there other areas where you express yourself artistically as well.
JOSEPH: I guess I kind of like that idea that other people started to call me an artist rather than myself calling me an artist, so that felt like some affirmation that this might be art. I think it was kind of the moments of realizing that artists have lots of different medium and code is a material for art, and that kind of, I guess, a little switch in my brain of realizing that the code is an expressive way to capture thoughts, but is also a way of self-expression and starting to write code that people watch and then realize the effect of that on the world.
I guess, kind of performance art, really, is one of the things that started to connect with. And, in terms of the community and the people around the kind of whole live coding of music and visuals is a lot of artists mainly, so it's kind of going into those circles. You gain a sense of contributing something new, and that people who are already artists enjoy your work. That gives you a bit of self-confidence maybe that you could also be an artist.
I also think I've touched more people through code art than I ever have in products. I think of the things I'm most proud of are the live coded performances I've done and more so that they've been shown around in schools and the kids have kind of been blown away that this is what programming could actually be like. That, to me, be it art or not, is one of the most rewarding bits of code I've ever written, I think.
CRAIG: It's interesting to me that you say that you feel you've touched more people through your performances than through product because I feel like something like a performance, you're interacting with people who are in essence in the room, although obviously there's recorded media. Maybe that's what you mean. Whereas, with product, I feel like there's leverage, right? You can get it out into the world more. I suppose it depends on what you're making, and especially because–correct me if I'm wrong–I believe you're one of the contributors to a number of libraries, but one of them is Sonic Pi.
JOSEPH: Yes. Yeah, that's right.
CRAIG: I guess I'm just a little surprised to hear that you say that, in terms of the number of people at least, the biggest impact you feel you've had has been through the performances.
JOSEPH: Yeah. I guess I should expand on that thought that, in all the products I've built, the code is not part of the experience.
CRAIG: Aah.
JOSEPH: So no one ever sees what I write, and that's pretty much true for everything I've done in a product. But here, the code is a fundamental part of the piece, and people who have no coding background can start to see what code could be and could even read meaning into it and read their own meaning into it as well. It's kind of the thing that I can go home and show my friends and family, and they will get what I'm doing and they will get that that's code. Well, everything else, it's kind of an invisible layer that's hidden away.
CRAIG: Okay. That makes perfect sense to me. I'm actually fascinated by your comment when you say you share it with friends and family. I assume you mean non-programming friends and family.
JOSEPH: People who have never touched code ever.
CRAIG: Now, when you do that, that's interesting. Now, when you do that and you say that they understand it, do you feel that they're making an actual direct connection between the symbols that you're typing and what's happening, or is it a little fuzzier than that where they're like, "Oh, I get it. You're actually making a computer do things by typing," or do you think you're actually saying, "Oh, wow. The letter 'A' there means this." Is it that type of connection that they're getting or something more abstract?
JOSEPH: Yeah, so I purposefully do a lot of play with variable names and trying to create things, which express the sounds or the visuals that are going to happen, so there's already a lot of meaning in the words, so I think, even as like a piece of literature, it's readable and I think it's often the fact that they can see a variable name, something like dockseahorn or something like that, and they already start to form in their head an idea of what that variable means, and then they hear it, and now they've kind of got that connection with the meaning and the sounds. Then they can see me manipulating and changing it.
It's kind of, on many levels, of that semantic meaning in terms of the variable names they kind of get and then the logic. Often with music, it's a lot of patterns and patterns can be very intuitive to understand to everyone, I think. Transposing notes, doing very simple things, people start to either learn through the music or learn the music through the code.
CRAIG: That's really cool. Of course, that applies to all types of coding, even though I suspect some of your audience, at least, doesn't realize that a symbol has a meaning. You manipulate the symbol, and it has this expression in the real or virtual world. That's a powerful concept, I think, that is a lot of programming, so it's cool that you're giving people that. Huh, neat.
JOSEPH: Yeah.
CRAIG: I've actually started doing Sonic Pi with my 11-year-old daughter fairly recently. We had Sam on the show, and he talked a bunch about it when it first came out. One of the great things about it is that there's a curriculum, right? We walked up to it, my daughter and I, and I was like, "Okay. I don't have to think about, 'Okay, well, should we start with this or start with that?'" It's like it goes right from the top.
I hear you talk about giving people this experience of understanding the connection between symbols and things that happen in the real world, like symbolic manipulation, essentially, which is, as we both know, a really important part of what it means to be a software developer. Of course, that's present in a first class way in Sonic Pi. I was wondering whether that was something, that that connection, that experience of creating that connection in other people's heads in that way that's so important was something that you brought to Sonic Pi. When you were coming to it, you're like, "Oh, yeah. We really need to do more of this," or something that you got from Sonic Pi, or that evolved independently - just whether there's any connection there.
JOSEPH: I think one of the things I got from doing a lot of Clojure is the feedback loop. It's like, irrelevant of everything else, the feedback loop has to be fast. If you speak to a lot of people about some very early experiences they have of coding, things like LOGO and moving the turtle around, the feedback loop is super fast.
I think, in fact, a guiding principle of Sonic Pi was more of that feedback loop being so fast that it's very easy to associate meaning to the code, and it's very easy to play with the code and then hear the results. Then it's kind of that rewards cycle as well of, you play a couple of notes. That's great. Then you start shuffling them, loop them around, just very little, simple programming concepts through a very fast feedback loop take you on a very kind of, I guess, very quick journey through programming and through music. I think, actually, it's all about the feedback, and I think developers are most happy when that feedback loop is tight. No one wants a slow feedback loop.
CRAIG: Yeah, yeah, yeah. No, it is key. Actually, you're a great guy to talk to about something that I've been thinking about a lot lately because, like a lot of people who come to Clojure from a language like, say, C+, in my case. The tight feedback that the REPL gives is super important to my experience. It's like one of the big benefits of Clojure is that feedback loop.
But, I find it fascinating that, in general, the Clojure experience of other types of feedback is in fact not what we would like it to be. Specifically, I'm talking about visualization. I had some code recently where I was writing, and it's a control loop. It's actually driving a motor and then the motor is trying to seek some position, and I was seeing oscillations or other types of problems.
Of course, I have the ability to dump it out to a text file, but I'm looking at a pile of numbers. Then I could maybe bring it into Excel, Google sheets, or whatever, and do all that. But I'm like, "Man. There's this type of problem, and then there's the other type of problem that we have all the time in Clojure, which is I've got a giant nested data structure. I can print it out, and that's one type of visualization, but it's honestly not a very good one."
I guess what I'm trying to say is, have you observed the same thing? Have you observed that there is, to some degree, either a hesitancy to use existing visualization techniques or a lack of visualization techniques at all that's kind of the exact polar opposite of this feedback loop we get around the program itself manipulating functions and vars and all that type of thing? JOSEPH: Yeah, yeah. I think it applies to all programming languages and that visual feedback. It is in fact something that I'm doing a lot of exploring about now trying to – I guess in a live coding kind of context, you're always continuously running code and loops are running. Then, by actually presenting visualizations that are behind a block of code that's running, you kind of gain something. I'm trying to think, actually. I guess – I'm not sure I've seen it in many editors where you have some visual representation of the code that's running, and you see both the code and the visual representation. I guess we've kind of got very into debugging sort of style or breakpoints and that sort of stuff, which is a very powerful way to have introspection of what's happening at execution time.
I kind of wonder if it's just that it has a high cost in terms of tooling. If you think of all the most powerful editors–I mean I say powerful–more GUI oriented, powerful editors, they tend to be slow or tend to use a lot of memory or always have some clunkiness, hence why I think a lot of people turn to the very simplicity model of things like Vim and Emacs and the various other editors. With that, you're throwing away that complexity, which I think might be associated with that extra visual level of feedback. Although, Emacs, in itself, I have augmented to try and provide more feedback, but it's hard with just text to try and really convey a lot of meaning without it being noise around the coding experience.
CRAIG: Yeah, so I feel like I agree. When you have an editing environment that becomes bloated because of a variety of things that gets in your way, but I guess it seems to me, perhaps naïvely, that the types of visualizations that we could take advantage of. I'm thinking of something at the simplest, perhaps, as something like the Clojure inspector, which was worked on for a while, for those that remember it. I think it was a swing interface for walking through nested data structures. It doesn't seem to really have gone anywhere. I'm not even sure it totally works anymore. But, that feels like a sort of optional thing that you could rely on when you wanted to. It would be orthogonal. I'm just wondering whether, as you've thought about visualizations, you've come across anything you feel that would be universal or at least broadly applicable that we're just not doing for whatever reason.
JOSEPH: I think there's probably a lot we're not doing, but there may be many reasons why we're not doing them. I guess, in terms of kind of visualizing the data and stuff, it is all a balance of noise and how much the visuals augment the experience and how much value they add.
One of the really simple things, which isn't necessarily about the data, but is more about the closures that I've seen at quite a couple of places, is a tool called Scheme Bricks. It's a project by Dave Griffiths that creates kind of his own editor. His simple idea is that closures are bricks, and every enclosed closure is another brick of a different color.
You start to see a series of bricks, basically, forming the programming language. Then, to kind of get feedback on the data and the data structures, those bricks have various connections with variables to each other and those are conveyed through kind of flashing lines, slightly. Not quite flashing, but these kinds of little lines that indicate almost electrical signal flowing between the various closures.
I thought that was a really interesting experiment in kind of, I guess, capturing that execution aspect of the coding part and also seeing the data and how things are interconnected. It's not necessarily as auxiliary as kind of opening up an inspector and seeing a bunch of stuff. It's more seeing a system live, which I guess is where most of my experience and research lies.
CRAIG: Interesting. I'm curious then when you say you've been doing research. Can you talk a little bit about that? I'm fascinated to hear what you've been working on.
JOSEPH: I have far too many projects. I do a lot. I'm really interested in how visuals can be used to convey a live system, especially in a music context. Talking about Sonic Pi, one of the projects I'm working on is when you have a thread and it's looping infinitely. You're prodding at bits of that code and changing it. How you can convey using visual graphics like an OpenGL window behind that bit of code, what's happening to it, and how fast is it running? Is it blocked? Is there an error? That's a lot of, I guess, UX. And, again, that kind of constraint of nine-year-old kids must understand this as well that it throws away a lot of complexity, which is fantastic and terrible at the same time.
I'm working on a project at the moment involving the British Arts Council, looking at how to celebrate Shakespeare. I'm looking at how I can combine live coding in a performance using Shakespeare's plays. I'm doing a lot of kind of messing around with semantic analysis, more generative-based at the moment. I'm still just playing around with how we combine code and Shakespeare. I'm really inspired a lot by – I forget the name of the book. There's a fantastic book of what code would look like if Shakespeare wrote it or various famous authors as well, so that's something I'm really digging into.
What else am I looking at? And I'm building my own kind of OpenGL graphics state machine engine at the moment, just exploring how people performing through code can very rapidly manipulate graphics with very simple task commands. It's kind of a mix between Quil processing or based using openFrameworks, which is a fantastic visual tool, and what we can do with that as well. It's kind of all in the path of trying to work out what programming is as a performance and how you go along to a place where nobody knows what code is and share it with them and have them gain something from the experience.
CRAIG: I was struck earlier when you were talking about code as a medium for art. The fact that when I look at a piece of code, I can, with some reliability if it was written by somebody I know, say, "Oh, Stuart Sierra wrote that," or, "Oh, that looks like a bit of Luke VanderHart's code," or whatever. It's kind of like the brush strokes on a canvas that experts can distinguish one artist from another.
JOSEPH: And everyone can say, like I had this question a while back where people were like, "What's the most beautiful piece of code you've written?" It was fascinating all the answers of some saying, "The line of code that I deleted. That was my most beautiful line of code." I think it was Fortran or Cobol, like the "go to." There's a one liner that has an entire book written on it all about how it's created generative arts. It's kind of, beauty in code, I think, is definitely a thing we all understand. Even if we don't necessarily always acknowledge it on the wall as a piece of art, we definitely have the concept of beautiful code.
CRAIG: What was your answer to that question, if you had one?
JOSEPH: I did have an answer. Let me think. What did I say my most beautiful bit of code was? It was a single exclamation mark I added on my thesis. It was a massive Prolog program, and I spent probably about three months analyzing this AI planning system. My sole contribution to those three months was a single exclamation mark, which was just, in Prolog, cutting the back tracing. I was so satisfied that I had a big comment next to it for anyone who was reviewing the coding like, "This took three months. This is where I started. This was my contribution." The smallest and, I think, perhaps the most beautiful bit of code I've ever written.
CRAIG: Yeah. I have to assume it made a difference. That was a performance thing, I assume, right?
JOSEPH: Yeah. It was also exhaustively searching, and it's kind of pruning the tree at the right points made it computationally feasible.
CRAIG: Cool.
JOSEPH: Yeah, it was a lot of fun messing with Prolog. I think Prolog is by far where my most beautiful code lies, I think.
CRAIG: Interesting. I have yet to look into Prolog, other than sort of indirectly through things like core.logic and, of course, Datalog.
JOSEPH: Mm-hmm.
CRAIG: I heard your child a moment ago, and you reminded me that you said something earlier that I wouldn't let go, which was that you had written a nursery rhyme generator.
JOSEPH: Yes.
CRAIG: Did that ever start producing interesting nursery rhymes?
JOSEPH: Yes, it did. It appealed to – actually, it's appealed throughout most of my daughter's life. She's now three, and I play her a lot of, like, AI generated stuff, and she likes listening and messing around with that as well. I think what you learn is actually the appreciation levels of a very young baby, as long as it doesn't sound terrible, they're pretty happy.
I also did a whole feedback loop stuff of looking at EEG machines and things like that around that, so she could generate the nursery rhymes or the music herself as well, so it was kind of taking brain scans and feeding that in as data, which was controlling the music, which meant if she was excited, the music would change. If she were bored, it would change, sort of thing. Again, it sounds very fun, but I think ultimately resulted in the fact that a small child's attention span is extremely small, so it's kind of just like continuously drift away and then comes back, drift away and then comes back, and then goes.
CRAIG: That is fascinating. Now, did she have any understanding or awareness of the fact that she was influencing what was going on, or she was just kind of around and it was reacting to what she was feeling?
JOSEPH: Yeah. No, I don't think, at a young age, she had any concept other than it changed, and she noticed the change sort of thing. I don't think she had much concept of what exactly papa was doing.
CRAIG: She's three now?
JOSEPH: Yes.
CRAIG: So she's sort of beginning to understand the world around her in a more concrete way. Do you have plans to revisit that experiment? I think that'd be fascinating.
JOSEPH: Yes. Yeah, I'm always trying to build devices so she can kind of code or make music as well, so I've programmed various devices like monomes, little boxes of LED buttons and the laptop keyboard as well, so she can start to play music and have little kind of co-routines where she presses them and it runs off a little bit of music, so I'm definitely really interested in joining her for the journey of, like, discovering programming, discovery code, and seeing what we can both do together.
CRAIG: That's very cool. Three is very young, but have you had any experience yet where, even in a small way, you feel like she has started to make some kind of connection, like there's a thing that goes on in our brains that I find it hard to explain where the idea of this abstract world that we're manipulating that we were talking a little bit about earlier. Have you seen any? Three is very young, so I wouldn't be surprised if the answer is no, but have you seen any evidence that that type of thing is happening with her?
JOSEPH: I guess if I kind of concrete it down to what I do with computers and messing around with music and things, I think understanding of the fact that she can do something on the keyboard and hit run, and it sounds different, and then she changes it, and it sounds different, kind of that understanding of the world in a more concrete way that she can do something, run it, and then hear its response is definitely developing. It's so hard to tell with anything else. More development, more concept of time, I guess, is the main thing that happens.
CRAIG: Sure. Sure. In your work with older children through Sonic Pi, one of the things I am interested in is thinking about–and obviously you and Sam and the rest of team have done a lot more work with this–is this idea of computer literacy. I think you could have a definition of programming, a useful definition of programming that's very broad to include getting a computer to do what you want, even to carrying out sets of instructions where you don't have to be explicit at every moment that the computer is carrying them out, if that makes sense. I think there's a sense in which those efforts are successful when people don't think of that as programming. They use it as an appliance.
JOSEPH: Yeah.
CRAIG: Am I making any sense?
JOSEPH: Yeah.
CRAIG: Does that line up with what you've experienced?
JOSEPH: Completely. No kid goes, "I'm going to learn a for loop." The kids with Sonic Pi are like, "Okay. I've got this note, and it just goes beep. Now I want it to go beep, beep, beep, beep, beep," sort of thing. It's like, "Well, I need a loop."
"I need to have a baseline, and I want a drum." Well, you need to have threads. You need some concept of concurrency. No one approaches programming in that way of thinking, like, "I need threads." They think, "I want two things sounding at the same time."
It's totally, like, music driven. I think, even at a young age, we have a very well developed sense of what we want from music, and we get bored very easily, which means we're totally driven by the music rather than the code. It's to facilitate us creating the music that I think drives the kids.
CRAIG: Yeah, for sure. It's funny. I sat down with, like I said, my 11-year-old, and we busted out Sonic Pi. I'm like, "Oh, this is great. Let's try it." She loved it. I remember Sam talking about the typical experience where the kids will sit down and type out – I forget the syntax exactly. It's right there in front of me, you know, when you have the thing open, so it's easy to get through.
JOSEPH: Mm-hmm.
CRAIG: Let's say I play 64, you know, play as middle C or whatever it is. He said that kids will often play 64, sleep 1, play 65, sleep 1, play 66, sleep 1, and that's exactly what she did, and she loved it so much. It's going to be really interesting to see if, as you say, right, that she gets motivated to learn something like a for loop, which she's had some exposure to through Scratch, but in this new environment, as a way to do that more easily. I suspect that you've seen that, exactly that transition on any number of occasions.
JOSEPH: Yeah. It's true kids never use the loop because I guess it's more explicit when you write it all down. I still do that myself as well. I'm not sure necessarily there is that much of a driver always for those for loop things. I think it kind of comes down to how easily is the change, right? Experimentation is fun, but then you want to start mutating it and changing it in just the same way we don't like duplication in code. It's kind of the same. It's because it makes some ideas easy to express and change easier to express.
As soon as they're kind of like, "Actually, I now want to change every single note and add a fifth to them."
It's like, "Ah, okay. That's a lot of work. You'd have to go and type every single one of these things here."
It's like, "Oh, is there a faster way to do it?"
"Yeah, we can do this thing called a loop." Again, it's kind of like always driven by some need rather than it being forcing injecting, "Let's do the loop," sort of thing. It's simply an easy way to experiment.
CRAIG: Maybe you can explain a little bit to people some of the time model of Sonic Pi. People should definitely check it out. It's very cool. But, it's a little bit different than a lot of languages, I think, in the sense of when you look at what's on the page, what's happening simultaneously versus what isn't. It's not quite the same way as what people might have encountered before.
JOSEPH: It's the idea that when you play a note and then you have another line after it that plays another note, they occur concurrently because that kind of makes sense with music. If you want to play a chord, you play three notes. You have a play, and then another play, and another play, imperative style, but they all play at once. Then you use a sleep, which we can think of as a beat, so you tell how many beats you want it to sleep, and then again you hit another phase where you can do things concurrently, which works very well for music, in fact, actually.
It's this thing I kind of mentioned a little bit earlier about temporal recursion or the different ways of approaching the timing model. Kind of temporal recursion where you have a little bit of code that kind of keeps calling back on itself is an extremely hard concept to teach to nine-year-olds or even adults, anyone, to be honest with you, who hasn't had a program experience, saying, "This is the fundamental component for you making music is time, and you need to understand temporal recursion in order to do it," is very hard. But saying, "Well, you hit sleep, and it waits, and then you can have as many plays as you want, so you can play five, six, seven notes if you want and, hence, understand what a chord is and all the different things you can do to a chord as well." It fits very nicely with a timing model….
CRAIG: Yeah, and so that makes perfect sense. The thing that's a little weird to me is that, well, I guess maybe not that it's weird to me. It's that I wonder what your experience with newcomers is versus mine. I look at a Sonic Pi program and, if I'm not thinking about it, it says, "Play, play, play, sleep." In my mind those things are sequential. The first play comes before the second play comes before the third play comes before the sleep.
JOSEPH: It's interesting. You say that is sequential, but I guess ultimately, in terms of actual execution of what's going on, they're happening so fast that they might as well be concurrent, unless they're blocking for some reason. If you have three very simple lines of code, they're pretty much happening concurrently, even if they occur one after the other. With the notes, okay, it is actually playing them concurrently. But, if they happen fast enough after each other such that there's no perceivable difference in time, then I think we think of some imperative bit of code of having that flow, but it's so fast that I'm not sure a human would necessarily perceive that if there's no IO blocking.
CRAIG: Yeah, and that makes sense. Certainly when I look at Sonic Pi, I don't find that confusing when I actually spend the minimal amount of effort it takes to remember that's what's going on. I guess my question is more around do you, as you're teaching, say, a nine-year-old this stuff, ever run into any kind of a mismatch between their expectations around when things are happening and the way they're written on the page? Do you know what I'm saying?
Yes, it's happening very, very quickly. But, in my head, in my programmer head, without thinking about it, or when I first walked up to Sonic Pi, I'm like, "Oh, well those are sequential. Yeah, the delta is very small, I guess, depending on how the engine is implemented," blah, blah, blah - all these technical details.
But, if I was absent all that, and I'm a nine-year-old kid–I've never programmed a computer before–do you find that they completely accept that right off the bat? They're like, "Oh, yeah. Those three things are happening at the same time."
JOSEPH: Yeah. Yeah, so I find that there's usually that first bump of like, "Oh, that's how it works." And, in fact, the kids move straight on. The programmers that I show this to stop for a second and find the concept way more kind of, "Oh, wait a minute. I need to think about this for a bit." The kids, as I said, they're more excited by the music. Hence, it's a very small, little step in their first journey on Sonic Pi, and it's a very simple concept. They jump straight on because they want to do more stuff.
CRAIG: That's fascinating. And so I wanted to follow on from there to come back to the idea of visualizations that you were talking about because Sonic Pi has this idea of loops. Am I correct on that?
JOSEPH: Yes.
CRAIG: Remembering the term?
JOSEPH: Yeah, kind of threads that run a loop. Yeah.
CRAIG: Right, and so I think it's great that you didn't call them threads, first of all.
JOSEPH: Yes.
CRAIG: Second of all, it's a neat idea and well executed. I really like the idea of having kind of some visual aspect that emphasizes their independence. I kind of wonder whether there might not be some space in the Clojure world with something like maybe futures or, maybe even more importantly, with core.async, these communicating sequential processes, when I look at my program and can go, "Okay. That thing has an independent," although it's a little bit weird with core.async with "go" because it's notionally independent. But, in practice, it's actually interleaved. But, I wonder whether you think there might be any leverage. I don't know if you've looked at core.async, but just that whole idea of being able to see where things are happening in parallel.
JOSEPH: Yeah. I definitely think, at worst, the output will be some art that's quite interesting. At best, it may actually augment and be a helpful tool. But, I kind of am very open-minded in just trying these things and seeing what it looks like. I think with kind of data flow and stuff like core.async, it could be a very interesting way of running your test and then seeing what's going on - not just the results of the test, but actually seeing the test run and seeing the effect and the blocking of the various aspects of it.
CRAIG: Hmm.
JOSEPH: Yeah, it's definitely something I'm interested in doing more work on.
CRAIG: Interesting. I have talked to Racket people from time to time. They always have really amazing ideas. I vaguely recall, and I haven't done it as seriously as I would like to given that I think I'd get a lot of benefit from it. I haven't played much with DrRacket. I don't know whether there's anything in there that you could mine or if you've looked at it at all. Are you familiar at all with any of the work that's been in that space on the Racket side of the world?
JOSEPH: The Racket, the Lisp language, right?
CRAIG: Yeah, that's right.
JOSEPH: Yeah, so the main thing I know with Racket is a tool called Fluxus that's a live coding visual environment that's more about prototyping live visuals. I don't think I've had a DrRacket.
CRAIG: Okay. It's kind of the GUI that you get out of the box when you install Racket.
JOSEPH: Ah, okay.
CRAIG: There's been some cool stuff in there. I don't know whether any of it applies. I was just curious if you'd seen it just because the Racket community is such a wealth of good ideas. I was wondering whether this was an area that they had something.
JOSEPH: Something I've seen done with Fluxus in the Racket space is you move the code to the graphics library, so you're writing the code in OpenGL, which suddenly means that the code is the graphics, and the graphics are the code, which is very interesting. I don't quite know yet. I'm still trying to experiment with that to see if that's the right way to go because you kind of then have whole fun issues with aliasing and scrolling and lots of various aspects that could be quite jarring, I guess, for someone who is used to coding in Emacs.
Imagine that when you first start typing, your font is massive. And, as you type, the font continues to get smaller and smaller and smaller as it scales itself to the graphics window. It's interesting and the Racket people have done some really cool stuff about that that works really well. It's a lovely idea that the code becomes the graphics. This is something I'm really interested in because even the performances that I've done, the code lays on top of the graphics.
I kind of talked to you that I'd like to augment the graphics to actually connect with the code. Ultimately, my goal is the code should be a fundamental part of the graphics. I want the code to do things along with the visuals and along with the sound. I think maybe the Racket kind of guise of that path of making the code actually part of the OpenGL window and being part of the graphics has quite an interesting scope.
CRAIG: Just to make sure I understand what you mean when you say the code is the graphics, you mean it's still textual, like I'm still typing open parentheses and the same things that I'm still seeing that. It's just that it's being rendered in a different way, or is there something beyond that?
JOSEPH: Yeah. Yeah, exactly. It's still all like everything is textual. You're typing. It's just that what is being rendered, rather than being a character in your editor, is actually a three-dimensional letter C, so you type the letter C. It exists in the graphics space because the problem you obviously have when you have kind of text overlaying a visual is it's very hard to create that connection. It's a lot of work. You have to use coordinates of where the code is. The graphics doesn't really know without you providing a lot of information of what the code is.
But, as soon you say, "Well, the code exists in 3-D space, and all my graphics exist in a 3-D space," then suddenly they can do a lot more with each other like the code could get bigger the louder the volume, or the code will start to wobble slightly because the base is so low. You can drop a beat or all these various things that could have very interesting effects on the code. I don't think you could do that without moving the code into the graphics world.
CRAIG: When you say "the code," of course our code is made up of different pieces. One obvious way to split it is functions, but there are name spaces. There are lexical blocks. There are all sorts of stuff. I guess I'm trying to figure out would it be the case that different functions would appear in different places? Lexical order still matters, even at the name space level, so how do you resolve those types of situations?
JOSEPH: I've seen quite a few kind of going towards, if you're familiar with kind of Max/MSP or some of the kind of patching libraries that a lot of musicians use where you can kind of think of everything of lots of blocks of modules connected with wires, and you're feeding a sound signal through the various blocks in the wires of how the signal moves between the blocks. That's pretty much the same analogy for the code, and I've seen a couple of implementations where people have tried to do that for the functions in a visual kind of music performance, but they just have a bunch of functions, and then they want to show what functions are calling what functions and create like a graph almost, again with some sort of like live flow so you can kind of see the values go through the functions and come out the functions.
CRAIG: Obviously, at a mechanical level in Clojure, we require an order among the functions in order for the compiler to work correctly. I guess you would infer it at that point, or it would be maintained for you.
JOSEPH: Yeah. Well, the function has to have been evaluated before you can call it, so it does exist in some scope, I guess, if you're going to do it ad hoc as the flow goes through the code. I think it would be easy enough, even with the order dependence. With all my live coding experiences, I've very rarely ever found that a problem because I'm evaluating everything and evaluating the functions. As long as they're all defined, then it's more about the execution of them than necessarily the definition.
CRAIG: That makes sense. Yeah, interesting. Interesting stuff. Wow, you're a busy guy. How do you do all this and raise a three-year-old? This is actually a serious question. I should ask this of more of my guests because I could use some advice.
Do you have a way that you approach work that allows you to still do the things that are not work? What's your workflow, I suppose might be a way to put it? How are you fitting in the things that you want to fit in into the 24 hours that are all we all get in each day?
JOSEPH: Yeah. I guess making my work art kind of has helped in that one of the big moves I've made in my career is to give myself more time to do the things that I want to do around art that necessarily don't make me any money. If I can contract and do more freelancing and have that flexibility, then I can kind of balance, and I find that very important having a child that I'm getting that satisfaction of doing the art work that I want to do in my work day such that I can put the computer away and spend time with my daughter and not feel like I have to kind of squeeze in that extra time in the evenings and, hence, miss my time with her.
CRAIG: Yeah.
JOSEPH: It's been a progression. I've kind of had places where I've worked where I used to do four-fifths, so I had one day where I was just doing my art stuff, and that was fantastic. But, then I wanted to do more and more art, and it kind of seemed a natural progression to move towards a more, I guess, balanced work model with the arts and paid work.
CRAIG: As you know, we do advice at the end of the show. I find that our guests are always so – maybe "wise" is the right word. But, by the end of the show, they've already given two or three really great pieces of advice. We're certainly not quite to the point where we close down because I definitely want to make sure that we leave time for anything. I've been asking all the questions, and I want to make sure I leave time for you to talk about any topics that you think are especially interesting to our listeners or that you just think is cool.
JOSEPH: Which I think is cool. There's so much that's cool. I think there's a lot of interesting stuff going on with thi.ng library that's going on at the moment, which is a Clojure and kind of ClojureScript library that I've been playing with a lot, a lot around 3-D generation of interesting shapes dealing with torn fabric, dealing with cells growing, dealing with lots of interesting visual effects. These tend to be exported to a big rendering app. Run 24 hours and you get a wonderful picture, but there's a lot of really interesting, simple systems, kind of very simple rule systems generating very complex, 3-D images.
Some people are now going and getting those 3-D printed as well, so you can, using the ClojureScript library, create and define a very simplistic yet able to generate very complex space, a 3-D image, which you could then go and get 3-D printed and end up being in an art exhibition. I think that's a beautiful balance of what Clojure is absolutely fantastic for and having something very visual to show people and explain the whole interesting of seeing the design and then hearing what the Clojure algorithm is doing, very simplistic stuff, Fibonaccian stuff, and kind of seeing the visual effect of the item first and then taking that back to, "Oh, I see. That's how the Clojure algorithm is working," is pretty amazing, I think. It's definitely something I'm very interested in and am doing at the moment as well.
CRAIG: I haven't heard of this.
JOSEPH: Thi.ng. Yes, it's thi.ng.
CRAIG: I have a 3-D printer sitting next to me right now and, actually, I'm interested in this because one of the things I have found as I have done a bit more – I'm still very much a beginner. Actually, this is really similar to the space that you've been living in for so long, which is I have some experience with 3-D modeling and kind of an interactive way where you go into a 3-D CAD program. I use Google SketchUp because it's easily accessible, free, whatever, but it's not parametric at all. You draw everything. You draw a rectangle, and then you pull up into a prism, and then you punch out holes or whatever.
But, if you want to change something, you have to go in and tweak every line, point, or whatever. And so I've started to move towards using Open SCAD, which is a scripting language that you write Python, Ruby, whatever JavaScript style language, and you produce your models that way. There's actually a Clojure library, which you can use that compiles to the SCAD script language, which therefore you have a REPL. You're making changes, and you're seeing them in real time in a 3-D environment. Then I can print those.
In fact, I've done exactly that to print various pulleys and a few other things that I've been interested in making. Of course, the huge benefit is that since it's information, it's easy to change. Oh, I need this hole to be 5% bigger. There are 20 of them, and I need to go around to 20 different holes and make them 5% bigger and get it all right and all that good stuff. It's very much like the 3-D version of where you're like, "Well, I'm going to do music, but I'm going to do it procedurally or algorithmically or generatively," do the same thing with actual, physical objects. Now that's fascinating.
Thi.ng is a Clojure Script library, or it's a Clojure library?
JOSEPH: Yeah. Clojure and ClojureScript. I guess it's just a collection of libraries around various 3-D aspects like texture, digital fabrication, rendering data. It's kind of everything bundled together to make it very easy. You can kind of render to Web, kind of just the canvas, so you can have these on Web pages and just generate your results there. There's also a facilitation for exporting them out into things like Flux renderer or any kind of 3-D library creating scenes and doing that, again, like you said, kind of interact with the REPL, getting that all set up and then generating your pictures.
CRAIG: Wow.
JOSEPH: It's been used in a lot of art exhibitions, I think mostly in the U.K. I'm not sure if it's been used much outside. But, lots of art exhibitions have been using this library. It's definitely a really interesting thing if you want to explore 3-D kind of generation.
CRAIG: Very, very cool. I'll definitely have to check that out. Thanks for the pointer. Well, Joseph, like so many of our guests, you are busy producing really cool things at such a rapid rate that we could easily, I think, talk for several hours. We do want to respect your time and, of course, keep the episodes down into the sub-two-hour range. I think it probably might be time for us to start wrapping up. Again, I always want to make sure that we give our guests time to share anything that they want to share. If there's anything else you'd like to talk about today, we could certainly do that, but I think we'll have to have you back on at some point, so we can always save more for later too.
JOSEPH: Yeah, that sounds good. I'm happy. We've covered a lot of stuff, so it's cool.
CRAIG: Cool. Yeah, it's been great. Of course, I'm not done ringing information out of you yet. We always have a question that we end with. You're aware of that. Like I said, it's about advice. You've already given us a bunch of good advice, but I wonder if you have any additional piece of advice you'd like to share with us here to close the show.
JOSEPH: I always feel slightly odd giving people advice since I never know what I'm doing ever really, and I'm just making it up as I go along. But, I guess the thing I would say, advice, suggestion, maybe, is to just take your code and go and do something completely fun, pointless, but self-expressive and share that with people. Maybe that'll be a small thing that'll be fun. Maybe it will inspire other people to look at code, to learn code, or try ad play around with it.
It's kind of seeing code in a new light, I think, that it could be creative, and it could be self-expressive. Even if that's just going to a hack thing and programming a robot to run around the room, it's a lot of fun at worst. And, at best, you may discover something to inspire people with.
CRAIG: Very cool. Very, very cool. Well, thanks so much for taking the time to come on the show today. I know we talked about how you make an effort to balance your work with your home life, and I know that we, because of the time difference, are eating into your home life, so I really appreciate you taking the time out to talk to us today. It's been really great to talk to you. You had a lot of really interesting and fun things to say, so thanks a ton.
JOSEPH: Yeah, thanks. It's been a pleasure.
CRAIG: I should also mention, I believe that you are available for consulting work right now. Is that the case?
JOSEPH: Yes, that's right. Yep, commissions, all project, Clojure consultancy, all that stuff.
CRAIG: Well, if anybody out there heard what we were talking about today and thought that it sounded like the sort of thing they could use your help with, I would certainly encourage them to reach out to you. Of course, we always have contact information for our guests on the show notes page, so I just wanted to make sure we give a shout out to you because it's obvious that you are a deep thinker and a skilled developer, so people should be able to take advantage of that.
Anyway, I will thank you again. It has been really great to talk to you. I am definitely looking forward to getting a chance to do so again, whether that's in person, perhaps it's in a conference, or the next time we have you on the show, if nothing else. Thanks a ton. It's been great to have you.
JOSEPH: Definitely. Thanks very much. It's been fun.
CRAIG: All right, the same for us. We will call it there. This has been the Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
CRAIG: You have been listening to the Cognicast. The Cognicast is a production of Cognitect, Inc., whom you can find on the Web at cognitect.com and on Twitter, @Cognitect. You can subscribe to the Cognicast, listen to past episodes, and view cover art and show notes at our home on the Web, cognitect.com/cognicast. You can contact us by tweeting @Cognicast or by emailing us at podcast@cognitect.com.
Our guest today was Joseph Wilk, on Twitter @josephwilk. Episode cover art is by Michael Parenteau. Audio production is by Russ Olsen and by our new editor, whom we welcome, Damian Mack. The Cognicast is produced by Kim Foster. Our theme music is Thumbs Up (for Rock N' Roll) by Kill the Noise with Feed Me. I'm your host, Craig Andera. Thanks for listening.