Method for an augmented reality character to maintain and exhibit awareness of an observer

• US 9,901,828 B2
• Filed: 12/08/2010
• Issued: 02/27/2018
• Est. Priority Date: 03/30/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Miller, Thomas, Weising, George
• Primary Examiner(s): WANG, JIN CHENG
• Application Number:
  US12/963,585
• Publication Number:
  US 20110242134A1
• Time in Patent Office: 2,638 Days
• Field of Search: 345633-634, 715758-764
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2340/14 : Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2354/00 : Aspects of interface with display user

Abstract

Methods and systems for enabling an augmented reality character to maintain and exhibit awareness of an observer are provided. A portable device held by a user is utilized to capture an image stream of a real environment, and generate an augmented reality image stream which includes a virtual character. The augmented reality image stream is displayed on the portable device to the user. As the user maneuvers the portable device, its position and movement are continuously tracked. The virtual character is configured to demonstrate awareness of the user by, for example, adjusting its gaze so as to look in the direction of the portable device.

Claims

1. A method for causing a virtual character to demonstrate awareness of a user'"'"'s position or activity, the method comprising,

detecting a position of a portable device, the portable device being substantially colocational with the user, the portable device including a camera and a display, the camera capturing a first image stream of a real-world scene, wherein the detection of the position of the portable device comprises identification of an anchor within the first image stream, the anchor being tracked during movement of the portable device;

generating a second image stream based on the first image stream, the second image stream comprising an augmented reality scene formed by inserting a virtual character into the real-world scene as captured in the first image stream, the virtual character being positioned in the augmented reality scene in relation to the identified anchor, the virtual character configured to perform an action indicating awareness of the position or activity of the user;

displaying the second image stream on the display of the portable device;

wherein the action indicating awareness comprises adjusting a view direction, in the augmented reality scene, of the virtual character, the view direction of the virtual character being defined relative to the anchor that is tracked during the movement of the portable device;

wherein the adjustment of the view direction of the virtual character is in the direction of the position of the portable device and is defined by turning of the virtual character from a current view direction to an adjusted view direction facing the portable device, and wherein the adjustment of the view direction of the virtual character changes a relationship of the view direction to the anchor, wherein the adjustment of the view direction of the virtual character includes adjustment of a head or eyes of the virtual character so as to look in the direction of the portable device.

2. The method of claim 1, wherein the insertion of the virtual character into the real-world scene comprises placement of the virtual character on a surface identified within the real-world scene.

3. The method of claim 1, wherein the anchor is selected from the group consisting of one or more augmented reality tags and one or more features within the real-world scene.

4. The method of claim 1, wherein the performance of an action indicating awareness occurs automatically, without requiring a command from the user.

5. The method of claim 1, further comprising,

continuously updating the adjustment of the view direction of the virtual character in the direction of the position of the portable device, as the portable device moves in three-dimensional space.

6. A method for causing a virtual character to demonstrate awareness of a user'"'"'s position or activity, the method comprising,

capturing a first image stream of a real-world scene at a camera of a portable device, the portable device being substantially colocational with the user;

identifying and tracking one or more markers in the first image stream;

tracking the position and orientation of the portable device based on the tracking of the one or more markers;

generating a second image stream based on the first image stream, the second image stream depicting a virtual reality scene positioned with reference to the one or more markers in the first image stream, the virtual reality scene including a virtual character positioned with reference to the one or more markers in the first image stream and configured to perform an action indicating awareness of the position or activity of the user;

displaying the second image stream on a display of the portable device;

wherein the action indicating awareness comprises adjusting a view direction, in the virtual reality scene, of the virtual character, the view direction of the virtual character being defined relative to the one or more markers;

wherein the adjustment of the view direction is in the direction of the position of the portable device, and wherein the adjustment of the view direction changes a relationship of the view direction to the one or more markers; and

continuously updating the adjustment of the view direction as the portable device moves in three-dimensional space.

7. The method of claim 6, wherein the markers are selected from the group consisting of one or more augmented reality tags and one or more features within the real-world scene.

8. The method of claim 6, wherein the performance of an action indicating awareness occurs automatically, without requiring a command from the user.

9. The method of claim 6, wherein the adjustment of the view direction of the virtual character includes adjustment of a head or eyes of the virtual character so as to look in the direction of the portable device.

10. A method for providing user interactivity with a virtual character, the method comprising,

initiating user interactivity with a computer program,

determining user interaction with a virtual character of the computer program,

detecting sensor data from the user during the user'"'"'s interaction with the virtual character, wherein the sensor data includes an image stream captured by a portable device controlled by the user;

tracking a position of the user using the sensor data, wherein the tracking of the position of the user includes identification and tracking of one or more markers within the image stream to enable determination of a position of the portable device;

detecting a position of the virtual character during the user'"'"'s interaction with the virtual character;

identifying a relative positioning between the user'"'"'s position and the position of the virtual character during the user'"'"'s interaction with the virtual character;

identifying a context of the user'"'"'s interaction with the virtual character;

identifying a signaling attribute for the context of the user'"'"'s interaction with the virtual character;

applying the signaling attribute to the virtual character to convey an awareness of the user by the virtual character;

wherein the computer program provides a virtual reality environment positioned with reference to the one or more markers, the virtual reality environment including the virtual character positioned with reference to the one or more markers;

wherein the user interactivity with the computer program includes interactivity with the virtual reality environment;

wherein the signaling attribute is selected from the group consisting of adjustment of the virtual character'"'"'s view direction in the virtual reality environment towards the position of the user so that the adjustment of the view direction changes a relationship of the view direction to the one or more markers, movement of the virtual character in the virtual reality environment towards the position of the user, gesturing by the virtual character in the virtual reality environment towards the position of the user.

11. The method of claim 10, wherein the computer program provides an augmented reality environment, the augmented reality environment including the virtual character;

and

wherein the user interactivity with the computer program includes interactivity with the augmented reality environment.

12. The method of claim 10, wherein the context of the user'"'"'s interaction with the virtual character is selected from the group consisting of a location within a real-world environment, a location within a virtual environment, a stage within the computer program.

13. A method for providing interactivity with a virtual character, comprising:

capturing, by a camera of a portable device, video of a real-world scene having one or more anchors;

generating an augmented reality scene based on the captured image stream, the augmented reality scene including a virtual character positioned in the augmented reality scene in relation to the one or more anchors;

rendering the augmented reality scene on a display of the portable device;

tracking a direction of the camera of the portable device;

adjusting a view direction of the virtual character, defined relative to the one or more anchors, in response to the tracked direction of the camera;

wherein when the direction of the camera is towards the virtual character, then the view direction of the virtual character is adjusted towards a location of the portable device, and wherein the adjustment of the view direction of the virtual character changes a relationship of the view direction to the one or more anchors.

14. The method of claim 13, wherein the adjustment of the view direction of the virtual character towards the location of the portable device is defined by turning of the virtual character from a current view direction to an adjusted view direction facing the portable device.

15. The method of claim 14, wherein when the direction of the camera is towards an object in the augmented reality scene other than the virtual character, then the view direction of the virtual character is adjusted towards the object in the augmented reality scene.