The Toyota Research Institute (TRI) specializes in developing assistive home robots, and currently they’re attempting to teach a general purpose robot how to perform useful human-level tasks in actual homes. To achieve this, the team is teaching the robot using an immersive telepresence system. In other words, there’s a model of the robot that is mirroring what the robot is doing.
This virtual reality system allows the human teacher sees what the robot is seeing live, in 3D, straight from its sensors. The human can then choose different behaviors to instruct and then annotate the 3D scene, like how to grasp a handle or drawing the line that defines the axis of rotation of a cabinet door. A person can try various approaches when teaching a task and make use of their creativity to use the robot’s hands as well as tools to perform the task. All of this teaching and testing happens in real homes, which is critical to achieving sufficient capability and reliability.
