I've found there are 4 types, or rather 4
intentions, of improvisation in the arts (mostly music and theater, of course, but there are aspects to which it influences other creative efforts like software, but I'll get there in a blog post on my professional/coding page later).
The first and earliest, of course, is to
re-learn how to improvise. And I'll stress
RE-learn. We come into this world (no, I'm not going to quote Bill Hicks) totally free to PLAY, to just come up with anything...like setting up all your stuffies and reading to them like they were a classroom, or telling the #9 that it was in the wrong place (the #6 place, though they are the same shape) and going "sit down, time out". Yeah, both things my kid did before she was 3.
We once knew how to just make crap up and hope it was good or right, or rather at least not care if anybody else thought it was...and then we got conditioned out of it. Some didn't, all of us shouldn't. But here we are. The criticism, the "oh, grow up", the...yeah, whatevers all squeezed the PLAY out of our playtime. And squeezed creativity out with it for far too many. Many a psychologist or educator has spoken out about this curse of our culture (or culture in general), including Fred Rogers and Leo Buscaglia.
This was always the ultimate in free improv: there was never a care for process nor product. Just living in the moment in its purest form. It is very hard to get back to that...but it isn't impossible.
The second intent is probably best described as the quest for 'chemistry'. A lot of improv is allusion: you generally start with what you know, be it movie quotes, song lyrics, or musically, particular genres, favorite bands, or just a particular practice you have. While the better improvisers will, of course, expand this allusion with clever variation, the team as a whole, be it a performance group like the cast of a renfaire, or a band getting together or auditioning a potential new member, has to determine if the ideas generated are compatible
just enough that nobody is going to just halt to go "whoa, dude, what the hell was that". In this, well, there's sometimes generation gap issues. If in a 'sci-fi' mood there's lots of star wars quotes going around and suddenly the new guy throws out a 2001-era battlestar galactica line? well, ok there needs to be some reconciliation here. Now that can be the old crew adapting and studying up the new stuff, because they want to stay relevant...and the best performers never stop learning. The goal is not the product, but the process.
The third intent is improvisation as a means of creating a final product. As Martin Mull describes Robert Klein, someone who "rehearsed and rehearsed that spontaneity". Improvisation as a method of composition. It is tedious at times, esp if someone one in the group is in a bad mood (or is feeling their own ideas are getting buried). But there's still a shared goal: out of these crazy ideas, we can assemble a scene, a song, a show. Standup comics tend to work this way - they just throw a line out or let a stream of consciousness flow on paper, maybe record into a digital recorder (in the old days, a tape recorder) a one-shot joke that came to mind while listening to the radio news driving home, then put the best bits together. Music groups sometimes jam, but a few bands had made it a point of jamming as a means of collecting musical ideas that are definitely 'the group' (as opposed to composed by one member or another) - particularly Genesis since 1980 and Marillion.
A variation of this is the "ok, here's a written idea, what can you do with it". Sometimes the improv out of that doesn't change the words or doesn't change the notes...but it changes the tone: a
visual gag, an ironic accent or emphasis that inverses the text's raw meaning, an alternate chord structure, an odd turn of instrument choices or a key/chord change seemingly out of nowhere but fits the melody. Yes often works this way, where there are ideas separately contributed, then one member (usually Jon, but more recently Steve) picks the good ones and stitches them together then goes back to the band with it and says "hey, lets link these up".
But the goal is a polished product that started life as just an idea seemingly out of nowhere, in a shared environment where everybody had the chance to contribute and give their reaction to what was created. The group, in working with the material, filters out what isn't working, embellishes to fill in the gaps of the written material, and otherwise gives it a vocabulary of identity.
One risk in this process is repetition. Even if the goal is to come up with creative works that aren't repetitive of the past...it can still happen because that's what's in your memory (see improv type 2). So when selecting ideas from the improvs to assemble into a product, many strive to be careful and look for the repetitions - a musical motif, a reference to a show already referenced, a rhyme, a common cliche. Or heck, realizing that this great melody you created turned out to be the cello part of measures 24-33 of Beethoven's 2nd piano concerto, 2nd movement. I hate when that happens. yeah, that's a Billy Crystal riff, thus supporting my point - it was the first thing that came to mind, a repetitive allusion/cliche, but it came AFTER the creative moment - the incongruity of someone accidentally repeating a part from that obscure a classical reference.
That often happens in my head when it starts 'inventing' (not) a Celtic tune. I realize I'm mixing tunes I already know together. And when improvising for
creation, one needs to be able to filter the results, both for repeating themselves, but also the risk of plagiarism.
There's lots of talk about right-brain-left-brain, where one is the creative side and the other the logical filtering side (I forget which is which, never mind my suspicion that maybe it, too, is backwards in left-handed people

). Thus the best creatives are ones where both sides are working equally, as the creative ideas are selectively filtered for originality and effective response, without filtering EVERYTHING out in the name of anxiety or fear. In group settings sometimes one member is not as 'creative' as the others...but if the goal is a final product, they can be the best filter of all of the creative ideas that come out in assembling a final product. For Marillion, that has been their co-producer Michael Hunter for the last 20 years (or Dave Meegan for a few albums before), who sifts through the jams and comes back with ideas he thinks works together, to let the band agree, bridge, and then add lyrics to.
In rock music criticism, there's an odd dislike of allusion. Either you're covering a song, or you're making something new, but none of this "throw in 2 bars of X" (outside of raw sampling). My dad was a stickler on that. One The Flower Kings piece quoted 2 measures of an old Neil Young track, and he instantly was turned off by it - turned off to the whole band and anything else they did...and no matter of my trying to explain that it was an intentional allusion, a callback to that rather than an accidental copy (you know, like
Stravinsky made his entire career on), he just couldn't take it. It was 'stealing'. Thus a curse of modern copyright (particularly the Harrison, Carmen, and Bolton lawsuits), perhaps, and how the public domain has been so limited of late in how our allusions to current culture have to be used within very tight limitations...and even within that, it may not be accepted as an original adaptation. There often needs to be a filter of experience: did you 'copy' someone else (even if unintentionally)?
The fourth improvisation is where the improvisation IS the product. The big popular versions of this are, of course shows like Whose Line and Comedy Sportz, or jazz improvs (when they're improvs), or Grateful Dead solos. Sometimes the improv is trading solos (in King Crimson, that would be 21st Century Schizoid Man) but other times the improv really is
free improv where everybody is listening to each other and trying to fit whatever idea came to mind into that (Easy Money and Thraak). In theater improv the difference is between "Scenes from a Hat" (everybody gets their turn) contrasted with "Song Titles" where the interaction is more important than the particular schtick. "Party Quirks" is a good game that can go either way. Some times the host will just let the guest do their bit and guess it. Other times they'll figure it out early but then let the guest continue to interact with the other guests for the sake of comedy in the moment.
So why did all this come to mind?
There's a scene at the MD Renaissance Festival, The Village Court. It is an all-improv show, a 'intent 4', where the improv is the goal. The structure is a typical court scene (justice, bailiff / prosecutor, defendant, defending solicitor, witnesses, and the audience as the jury...and then throw in some hecklers from the cast not involved for full measure). It works within those as several of the cases involve street bits that the bailiff performed in giving citations to the patrons of the fest, and some of them dutifully come to the court and let it play out, thus giving a wonderful callback to their own day at the fest which certainly lasts longer than the day. If not enough, then the cast will just play out a resolution to their many street improvs during the day. It is an intent 4 improv, where the goal is the improv itself.
But there was once a predecessor that didn't work as well...but the proposed fix to it just made it worse. It was called "Prickett's Court", named for the justice, a fire and brimstone type. The improvs were mostly among the cast rather than directly involving patrons, but the cast available was often limited as any involved in the main stage production couldn't participate. The proposed solution was to turn this intent 4 (free improv) into intent 3 - take the best ideas and just try to repeat them...but those ideas weren't as good because they were created for the moment, as intent 4 should be. There was filtering in the moment to make the best scene one could make
at the time, but not with the goal of making the best scene ever.
Intent 3 is wilder - you take the time (usually without an audience, but the King Crimson ProjeKcts worked with one to get another 'filter'), to generate a LOT of material to then filter the best out of. Improv of intent 4 is not going to generate that as some ideas better than what is on record never actually came out, so trying to produce repeatable scenes from only intent 4 ideas just didn't have enough to work with, and the show fell flat and ended on only the one season.
The improvs may have been hit or miss (*all* improv is, and always will be), but trying to convert it into a written/composed show just killed it. It really is not an easy thing to turn improv of intent 4 (in the moment) into improv of intent 3 (improve with the goal of a composed product).
It sometimes happens (a concert improv from King Crimson in late '72 on one of the first concerts of that line-up eventually became an instrumental break on a track, Fallen Angel, 3 albums later). but those are very rare.
So all that came to mind and finally I got around to putting some words to those impressions and memories.