Over the course of an hour at Thoughtworks last week Ken Perlin described a vision for the future of immersive human interaction.

It was rich and varied subject matter, and it drew a line from Ken's early inspirations right through current research and beyond.

The Internet Society's Joly MacFie was on hand to film it (above), but I'll summarize the thrust of the argument here so you can get a sense. I also want to sprinkle in a few of my own comments and reactions.

What this talk is really about
Over the course of the hour, Ken weaves through a range of subjects including narrative, immersion, imagination, creativity, shorthand pop culture references to the 'future', and human nature - already a lot for one hour.

Add to that a range of technology subjects - wearables, implants, depth & holography, virtual & augmented reality, machine learning, kinematics and software programming - and you find yourself with plenty of rabbit holes to go down.

However, the real vision is all here in this 1-minute clip. In the video below, Professor Whoopee helpfully explains the functioning of a CRT using his 3DBB - his 3-dimensional blackboard (ofcourse). Check it out, and particularly watch how Professor Whoopee uses the 3DBB to communicate and interact with the other characters around him:

The 3DBB is like an immersive environment, in which Professor Whoopee can create and operate a virtual, functioning CRT, which he and the other characters around him can all have a shared, volumetric experience of.

Bearing in mind that context, have a look through a transcript of Ken's closing statements to the audience last Tuesday:

"Eventually, when you and I are face-to-face in an augmented version of reality, and we have nothing but our bodies and our eyes and our hands and each other... then we'll be able to use these very very simple techniques, because we've understood the semantics of how I create something for you in realtime. And what I can offer is this library [of intuitively instantiated, intelligent and directable, interactive yet autonomous virtual objects]..."

"Then the virtual world we have between us becomes something that is not just a replication of our physical reality, but actually the kind of reality we'd really like to be in"

In this vision, we are able to spontaneously create and manipulate logically-consistent virtual objects, or we might say 'directable actors' since they are semi-autonomous. These object/actors surrounding us are not impositions on physical reality. As a consequence of immersion, they are indistinguishable, an effectively inseparable component of reality itself.

The actor/objects are viewable and interoperable in the space between multiple people, similar to how we currently imagine future holographic interfaces.

But we don't interact with these objects by using a portion of space which we would identifiably define as being an interface. The net effect of immersion combined with shared experience is that the virtual-physical reality we inhabit precludes the need for an interface.

All of it, or rather none of it is the interface.

This versus other visions
I think a lot of people will find that hard to imagine. One way to try and imagine it is to contrast it with other visions of the future.

In his Brief Rant on the Future of Interaction Design, Bret Victor takes a decent swipe at the vision presented in this video:

Bret's criticism is to note that the characters in such visions are immersed in an experience which can take on any shape or form (so long as that form blends the possible characteristics of virtual and physical worlds).

And yet to interact with this immersive reality, they turn to their hands and manipulate a little virtual 'phone'.

It doesn't make sense.

A real, modern-day phone is like a glass window which you can swipe and prod, and there isn't a great deal of immersive haptic feedback.

Compare that to your intuitive sense of your place in a book by the relative density of pages in each hand. Or the amount of water by the shift in weight distribution as you tilt a glass.

Natural human interactions are richly physical, and both Ken and Victor point out that we are currently going through a very odd transitional stage - walking around with our heads facing down, glued by our eyes and fingers to screens. Visions for future human interaction should allow that as soon as our dependency on physical devices for virtual interaction goes, so too goes that framework of interaction.

If no interface, then what?
Ken's answer is that the spontaneous creation of shared virtual actors will become a new staple of human communication, in much the same way that the ability to instantly communicate by video with people the other side of the world became a staple of human communication.

These actors will be scriptable and responsive to their environment, much like real actors on a stage. But hang on - we've already seen this type of thing recently haven't we?

The big difference between this and Ken's vision is that in the PS4 you enter into a contrived interaction with a specific subset of actors and scenarios. Because the hardware is not a seamless, integral element of your everyday experience, the user-experience case for being able to instantiate your own actors at will is much weaker.

This begs another question. If the interactions in Ken's vision are not to be contrived, doesn't this rely on each individual user to craft and nurture their own individual libraries of symbolic 'actors'? Or do most people subscribe to commercially available 'packs' of actors, and combine them to curate their own unique library?

Questions start flooding in. In what way does a library contribute to, or become reflective of, a person's personality? And tangentially, if these actors can't provide haptic feedback, are they really so different from the Minority Report interface?

Perhaps a further modification. What if it is the ability to transfer virtual characteristics to physical objects, and have those physical objects respond in a haptically-meaningful way that will provide the most engaging experience?

Yes, great conjecture - but is any of this even possible?
Ken spends most of his talk explaining how far he and his students have come, and how he expects technology to develop in the near future. I'd recommend watching if any of this interests you.

We will keep an eye on Ken's work over at Volumetric and catch up with him again to explore these questions in the future.