AR Puppet - Animated Agents in Mixed Reality Environments - PhD topic

For publications see my publication list and the projects below.

Current projects

• AR Lego
  
  This project is an important pilot application for the use of AR agents. It implements a machine maintenance scenario employing two AR agents to educate an untrained user to assemble, test and maintain machines composed of active (engines and sensors) and passive (cogwheels, gears, frames) parts. The two agents are a real, augmented LEGO Mindstorms robot and a virtual repairman. The animated virtual repairman serves as an autonomous telepointer and explains about the robot with expressive, natural gestures in the vicinity of significant physical locations during the construction steps. By monitoring the attributes of the LEGO robot and the current status of assembly application, character behavior is automatically generated for the repairman, which is synchronized to the physical actions triggered on the physical robot (turning engines on/off, setting direction/power, querying various sensors, etc.)
  
  
  
  The above images are screenshots from the AR Lego application that is covered in detail in the paper below and can be seen in the video further down on the page.

  
  Selected publication: I. Barakonyi, T. Psik, D. Schmalstieg, "Agents That Talk And Hit Back: Animated Agents in Augmented Reality", Proc. IEEE and ACM International Symposium on Mixed and Augmented Reality 2004 (ISMAR'04), Nov. 2-5, 2004, Arlington, VA, USA, pp. 141-150. (download here - 1.4Mb)
  
  
• Monkeybridge (joint project with Markus Weilguny and Thomas Psik)
  
  Lifelike animated characters have received significant attention in recent years. Augmented Reality as a new a media demands broader acceptance, which can be achieved by widening its application base to include entertainment and education. In this project we show how animated agents make autonomous decisions based on their observation of an AR environment in which they are embedded. Monkeybridge is a multiplayer game, where users place real and virtual objects onto a physical surface, thus influencing the behavior of the characters.The characters autonomously choose: the path the walk on; decide how to get from one platform to the other, e.g. climb or jump when there is a slight difference in height between platform edges; automatically choose the straightest path from several available tiles; and fall into the water if there is no suitable piece of landing stage to walk on. The game includes many spectacular virtual and physical visual elements such as animated 3D ocean waves, a flock of virtual seagull boids, a physical volcano with real smoke and a lighthouse with rotating lights. Sound effects enhance the game experience.
  
  
  
  The project was presented as a demo at the ISMAR'04 conference in Arlington, VA, USA and at the VR'05 conference in Bonn, Germany. See pictures above and the video further down on the page.
  
  Project description: (download here - 54Kb)
  Project poster: (download here - view - 385Kb)
  
  Selected publication: I. Barakonyi, M. Weilguny, T. Psik, D. Schmalstieg, "MonkeyBridge: Autonomous Agents in Augmented Reality Games", Proc. of the ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE'05), Valencia, Spain, June 15-17, 2005. (download here - 1.3Mb)
  
  
  
• Animated agents in a UbiComp environment (joint project with Joseph Newman)
  
  
  
  Project description: coming soon
  
  
• An AR calibration assistant for an ultra-wideband tracking system (joint project with Joseph Newman and UbiSense Ltd.)
  
  
  
  Project description: coming soon
  
  
• AR Piano Tutor
  
  In this project virtual objects are superimposed on top of a real MIDI keyboard. The input and output channels of the keyboard are connected to a computer via a MIDI interface. The application serves as a piano teacher that teaches about basic chords, scales, etc. The keys pressed by the "student" user, the order and the timing of the piano key presses are all captured by the computer with the help of the MIDI interface, therefore the teacher can immediately "check" whether the student has correctly followed the instructions and may give instant visual feedback over the real keyboard (e.g. marking wrongly pressed or missed keys with different colors). The application is based on a "fishtank" AR setup (PC+monitor+webcam), where the physical MIDI keyboard is tracked with the help of an optical marker.
  
  
  The above images are screenshots from the AR Piano Tutor application that is covered in detail in the paper below and can be seen in the video further down on the page.
  
  Project description : (download here - 370Kb)
  
  Selected publication: I. Barakonyi, D. Schmalstieg, "Augmented Reality Agents in the Development Pipeline of Computer Entertainment", Proc. of the 4th International Conference on Entertainment Computer (ICEC'05), Sanda, Japan, Sept. 19-21, 2005. (Download here - 2.6Mb)
  
  
• Rotoscoping techniques exploiting AR
  
  Modellers and animators often rely on real-life references to build and animate 3D characters for games or film production. Observing the real world by photographing or videotaping the subject, or asking someone to pose helps create more believable, precise and expressive character animation. Professional artists use motion capture techniques or other expensive means (such as the Monkey Kinematic Tracker) of aquiring motion data to create an essential initial data set for the final, refined animation.Since physical and virtual objects appear to coexist in an Augmented Reality (AR) environment, new possibilities in character animation open up. The animated virtual model and the real-world reference can be merged into one, interactive modeling tool. The main goal of this project is to develop such a tool using pose-tracked physical objects as input devices to animate skeleton-based virtual 3D models. An important part of the project is to map the real-time pose information to rotation information for the joints of the character skeleton using Inverse Kinematics and motion mapping techniques. AR enables not only a close interaction with the virtual model by using tangible objects but also the creation of complex motions like walking up stairs or lifting a ball, since the animator can use the actual physical models of the stairs or the ball together with the character to create the motion.
  
  
  The above images are concept images for the AR Rotoscopy application that is covered in detail in the paper below. The working application can be seen in the video further down on the page.
  
  Project description: (download here - 186Kb)
  
  Selected publication: I. Barakonyi, D. Schmalstieg, "Augmented Reality Agents in the Development Pipeline of Computer Entertainment", Proc. of the 4th International Conference on Entertainment Computer (ICEC'05), Sanda, Japan, Sept. 19-21, 2005. (Download here - 2.6Mb)
  
  


• AR videoconferencing
  
  The Augmented Reality Videoconferencing System is a novel remote collaboration tool combining a desktop-based AR system and a videoconference module. The novelty of this system is the combination of these tools with AR applications superimposed on live video background displaying the conference parties’ real environment, merging the advantages of the natural face-to-face communication of videoconferencing and AR’s interaction capabilities with distributed virtual objects using tangible physical artifacts. The simplicity of the system makes it affordable for everyday use. The system is based on concurrent video streaming, optical tracking and 3D application sharing. The technology used provides yields superior quality compared to pure video streaming with successive optical tracking from the compressed streams. The system's collaborative features are demonstrated with a volume rendering application that allows users to display and examine volumetric data simultaneously and to highlight or explore slices of the volume by manipulating an optical marker as a cutting plane interaction device.
  
  The above images are screenshots (with some extra explanation added afterwards to the left image) from the demo application described in the paper below.
  
  Selected publication: I. Barakonyi, T. Fahmy, D. Schmalstieg, "Remote Collaboration Using Augmented Reality Videoconferencing", Proc. of Graphics Interface 2004 (GI'04), May 17-19, London, ON, Canada, pp. 89-96. (download here - 700Kb)
  

Past projects

• Animated talking head for multimodal presentations (done at the Ishizuka Lab at the University of Tokyo)
  
  
  
  Selected publication: N.P. Chandrasiri, I. Barakonyi, T. Naemura, M. Ishizuka, H. Harashima, "Internet Communication Using Real-time Facial Expression Analysis and Synthesis", IEEE Multimedia, Vol.11, No. 3, July-Sept 2004, pp. 20-29. (download here - 1.9Mb)
  
  
• AR conference table
  
  
  
  This was a student project developed by Thomas Baldauf and Harald Entner under my supervision.

Publications

• Istvan Barakonyi, Dieter Schmalstieg, "Ubiquitous Animated Agents for Augmented Reality", In Proc. of IEEE and ACM International Symposium on Mixed and Augmented Reality 2006 (ISMAR'06), Santa Barbara, CA, USA
• Istvan Barakonyi, Dieter Schmalstieg, "Augmented Reality in the Character Animation Pipeline", SIGGRAPH sketch, SIGGRAPH 2006, Boston, MA, USA
• Joseph Newman, Gerhard Schall, Istvan Barakonyi, Andreas Schuerzinger, Dieter Schmalstieg, "Sentient Environments for Augmented Reality", demo submission paper, In Advances in Pervasive Computing, Pervasive 2006, Dublin, Ireland
• Florian Ledermann, Istvan Barakonyi, Dieter Schmalstieg, "Abstraction and Implementation Strategies for Augmented Reality Authoring", book chapter in Emerging Technologies of Augmented Reality: Interfaces and Design, (M. Haller, B. Thomas, M. Billinghurst eds.), Idea Group Publishing, to be published in 2006
• Istvan Barakonyi, Dieter Schmalstieg, "Augmented Reality Agents in the Development Pipeline of Computer Entertainment", Proc. of the 4th International Conference on Entertainment Computer (ICEC'05), Sanda, Japan, Sept. 19-21, 2005. Download (2.6Mb)
• Istvan Barakonyi, Markus Weilguny, Thomas Psik, Dieter Schmalstieg, "MonkeyBridge: Autonomous Agents in Augmented Reality Games", Proc. of the ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE'05), Valencia, Spain, June 15-17, 2005. Download (1.3Mb)
• Istvan Barakonyi, Dieter Schmalstieg, "Exploiting the Physical World as User Interface in Augmented Reality Applications", Proc. of the IEEE Virtual Reality 2005 Workshop on New Directions in 3D User Interfaces, Bonn, Germany, March 12, 2005. Download (1.6Mb)
• Istvan Barakonyi, Thomas Psik, Dieter Schmalstieg, "Agents That Talk And Hit Back: Animated Agents in Augmented Reality", Proc. of the IEEE and ACM International Symposium on Mixed and Augmented Reality 2004 (ISMAR'04), Arlington, VA, USA, Nov. 2-5, 2004, pp. 141-150. Download (1.4Mb)
• Istvan Barakonyi, Tamer Fahmy, Dieter Schmalstieg, "Remote Collaboration Using Augmented Reality Videoconferencing", Proc. of Graphics Interface 2004 (GI'04), London, ON, Canada, May 17-19, 2004, pp. 89-96. Download (701Kb)
• Naiwala P. Chandrasiri, Istvan Barakonyi, Takeshi Naemura, Mitsuru Ishizuka, Hiroshi Harashima, "Internet Communication Using Real-time Facial Expression Analysis and Synthesis", IEEE Multimedia, Vol.11, No. 3, July-Sept 2004, pp. 20-29. Download (1.98Mb)
• Istvan Barakonyi, Dieter Schmalstieg, "AR Puppet: Animated Agents in Augmented Reality ", Proc. of First Central European International Multimedia and Virtual Reality Conference, pp. 35-42, Veszprém, Hungary, May 6-8, 2004. Download (295Kb)
• Istvan Barakonyi, Tamer Fahmy, Dieter Schmalstieg, Karin Kosina, "Collaborative Work with Volumetric Data Using Augmented Reality Videoconferencing", Demo Paper, Proc. of the IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'03), Tokyo, Japan, October 2003. Download (212Kb)
• Istvan Barakonyi, Werner Frieb, Dieter Schmalstieg, "Augmented Reality Videoconferencing for Collaborative Work", Proc. of the 2nd Hungarian Conference on Computer Graphics and Geometry, Budapest, Hungary, June 30 - July 1, 2003. Download (651Kb)
• Naiwala P. Chandrasiri, Istvan Barakonyi, Takeshi Naemura, Mitsuru Ishizuka and Hiroshi Harashima, "Communication over the Internet using a 3D Agent with Real-time Facial Expression Analysis, Synthesis and Text to Speech Capabilities", Proc. of the IEEE International Conference on Communication Systems (ICCS 2002), Singapore, November 2002. Download (399Kb)
• Istvan Barakonyi, Naiwala P. Chandrasiri, Sylvain Descamps, Mitsuru Ishizuka, "Communicating Multimodal Information on the WWW Using a Lifelike, Animated 3D Agent", Proc. of the PRICAI'02 workshop, Tokyo, Japan, August 2002 and the 1st Hungarian Conference on Computer Graphics and Geometry, Budapest, Hungary, May 2002. Download (145Kb)
• Sylvain Descamps, Istvan Barakonyi, Mitsuru Ishizuka, "Making the Web Emotional: Authoring Multimodal Presentations Using a Synthetic 3D Agent", Proc. of the OZCHI 2001 Conference, Perth, Australia, November 2001, pp.25-30. Download (246Kb)
• Istvan Barakonyi, Mitsuru Ishizuka, "A 3D Agent with Synthetic Face and Semiautonomous Behavior for Multimodal Presentations", Proc. of the Multimedia Technology and Applications Conference (MTAC2001, IEEE Computer Society), Irvine, California, USA, November 2001, pp.21-25. Download (247Kb)

Videos

• Balancing
  
  The video shows a simple application demonstrating how a virtual character can react to changes in the physical environment's attributes. A monster-like character stands on an optical marker, which the user moves and tilts around. After initializing the direction of the gravity, the character starts changing its behavior based on the tilt angle of the platform so that it appears as it was trying to keep its balance. If the platform gets too steep, the monster falls down.
  
  
  
  Download video (2.8Mb)
  
  
• AR Lego
  
  This video demonstrates the LEGO robot assembly and maintenance application described in my ISMAR'04 paper. A virtual repairman helps assemble a LEGO Mindstorms(R) robot and explains about its operation by superimposing virtual objects on top of the real robot. The virtual repairman agent's gestures and its explanation about various engines and sensors are synchronized to infrared commands that are automatically generated and sent over to the real robot, which trigger physical actions like switching engines on /off, querying push or light sensors etc. The user holds a tracked PDA, which acts as a hand novel interaction device. Firstly, it renders a dynamically generated GUI using the Personal Universal Controller software to change the controllable attributes of the LEGO robot (engine status, engine power, sensor type etc.) and the virtual character (rendering mode, character type, scale etc.). Moreover, the PDA acts as a tangible interface to position the agent in the user's mixed reality environment and switch between the LEGO robot's and the user's reference frame.
  
  
  
  Download video (49Mb)
  
  
• Controlling characters using the Personal Universal Controller (PUC)
  
  
  
  Download video (14Mb)
  
  
• Monkeybridge
  
  
  
  Download video (111Mb)
  
  
• AR Piano Tutor
  
  
  
  Download video (74Mb)
  
  
• AR Rotoscopy
  
  
  
  (coming soon)
  
  
• AR Videoconferencing
  (coming soon)
  
  
• 3D Chat (video made by N.P. Chandrasiri at the University of Tokyo)
  
  
  
  Download video (27Mb)