Had a really great lunch discussion with Ehud today, and we went deep on what the overall dissertation research plan could be, and how the HRI course can serve those goals.
Coles Notes version – I’m going all-in on the “record a learning experience in 3D+ and explore playback in various modalities”. I think this is incredibly important. There is a lot of work going on, looking at motion capture and performance. There is a lot of work going on, looking at playback (especially in games and movies), but there is a huge opportunity to explore recording a learning experience (active learning classroom scenario) and allowing participants and observers to revisit in various modalities.
Imagine recording a session in such a way that everything is capture – 3D, audio, content of collaboration, position of participants, environmental cues. Everything. Then, if you can revisit that recorded session in VR or augmented reality, able to move around as a ghost as the recording plays back, able to see the interactions from any perspective. To pause, slow down, speed up, or alter time.
So. For the HRI course, I’m planning to record a monologue performance using HD video, and then manually convert that into a robotic performance using Baxter. Then, I will convert it to a VR performance using Unity Blender or something similar. This will give 4 modalities to compare (live performance, video playback, robotic playback, VR playback). These will help me to learn about what is lost in translation, what is gained, and how this might inform the broader project of full omnimedia recording and revisiting of a session.