METHOD FOR AN AUGMENTED REALITY CHARACTER TO MAINTAIN AND EXHIBIT AWARENESS OF AN OBSERVER

• US 20110242134A1
• Filed: 12/08/2010
• Published: 10/06/2011
• Est. Priority Date: 03/30/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Miller, Thomas, Weising, George
• Granted Patent: US 9,901,828 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2340/14 : Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2354/00 : Aspects of interface with display user

Abstract

Methods and systems for enabling an augmented reality character to maintain and exhibit awareness of an observer are provided. A portable device held by a user is utilized to capture an image stream of a real environment, and generate an augmented reality image stream which includes a virtual character. The augmented reality image stream is displayed on the portable device to the user. As the user maneuvers the portable device, its position and movement are continuously tracked. The virtual character is configured to demonstrate awareness of the user by, for example, adjusting its gaze so as to look in the direction of the portable device.

Claims

1. A method for causing a virtual character to demonstrate awareness of a user'"'"'s position or activity, the method comprising,

detecting the user'"'"'s position by detecting the position of a portable device, the portable device being substantially colocational with the user, the position and movement of the portable device being controlled by the user, the portable device including a camera and a display;

capturing a first image stream of a real-world scene at the camera;

generating a second image stream based on the first image stream, the second image stream comprising an augmented reality scene formed by inserting a virtual character into the real-world scene as captured in the first image stream, the virtual character configured to perform an action indicating awareness of the position or activity of the user;

displaying the second image stream on the display of the portable device.

2. The method of claim 1, wherein the insertion of the virtual character into the real-world scene comprises placement of the virtual character on a surface identified within the real-world scene.

3. The method of claim 1, wherein the detection of the position of the portable device comprises identification of anchors within the first image stream, the anchors being tracked during movement of the portable device.

4. The method of claim 3, wherein the anchors are selected from the group consisting of one or more augmented reality tags and one or more features within the real-world scene.

5. The method of claim 1, wherein the performance of an action indicating awareness occurs automatically, without requiring a command from the user.

6. The method of claim 1, wherein the action indicating awareness comprises adjusting a view direction of the virtual character.

7. The method of claim 6, wherein the adjustment of the view direction is in the direction of the position of the portable device;

and further comprising,

continuously updating the adjustment of the view direction as the portable device moves in three-dimensional space.

8. The method of claim 7, wherein the adjustment of the view direction of the virtual character includes adjustment of a head or eyes of the virtual character so as to look in the direction of the portable device.

9. The method of claim 5, wherein the adjustment of the view direction is in the direction of a position towards which the camera of the portable device is directed.

10. A method for causing a virtual character to demonstrate awareness of a user'"'"'s position or activity, the method comprising,

capturing a first image stream of a real-world scene at a camera of a portable device, the portable device being substantially colocational with the user, the position and movement of the portable device being controlled by the user;

identifying and tracking one or more markers in the first image stream;

tracking the position and orientation of the portable device based on the tracking of the one or more markers;

generating a second image stream based on the first image stream, the second image stream depicting a virtual reality scene positioned with reference to the markers in the first image stream, the virtual reality scene including a virtual character configured to perform an action indicating awareness of the position or activity of the user;

displaying the second image stream on a display of the portable device.

11. The method of claim 10, wherein the markers are selected from the group consisting of one or more augmented reality tags and one or more features within the real-world scene.

12. The method of claim 10, wherein the performance of an action indicating awareness occurs automatically, without requiring a command from the user.

13. The method of claim 10, wherein the action indicating awareness comprises adjusting a view direction of the virtual character.

14. The method of claim 13, wherein the adjustment of the view direction is in the direction of the position of the portable device;

and further comprising,

continuously updating the adjustment of the view direction as the portable device moves in three-dimensional space.

15. The method of claim 14, wherein the adjustment of the view direction of the virtual character includes adjustment of a head or eyes of the virtual character so as to look in the direction of the portable device.

16. A method for providing user interactivity with a virtual character, the method comprising,

initiating user interactivity with a computer program,

determining user interaction with a virtual character of the computer program,

detecting sensor data from the user during the user'"'"'s interaction with the virtual character;

tracking a position of the user using the sensor data;

detecting a position of the virtual character during the user'"'"'s interaction with the virtual character;

identifying a relative positioning between the user'"'"'s position and the position of the virtual character during the user'"'"'s interaction with the virtual character;

identifying a context of the user'"'"'s interaction with the virtual character;

identifying a signaling attribute for the context of the user'"'"'s interaction with the virtual character;

applying the signaling attribute to the virtual character to convey an awareness of the user by the virtual character.

17. The method of claim 16, wherein the computer program provides a virtual reality environment, the virtual reality environment including the virtual character;

and

wherein the user interactivity with the computer program includes interactivity with the virtual environment.

18. The method of claim 16, wherein the computer program provides an augmented reality environment, the augmented reality environment including the virtual character;

and

wherein the user interactivity with the computer program includes interactivity with the augmented reality environment.

19. The method of claim 16, wherein the sensor data includes an image stream captured by a portable device controlled by the user;

and

wherein the tracking of the position of the user includes identification and tracking of markers within the image stream to enable determination of a position of the portable device.

20. The method of claim 16, wherein the context of the user'"'"'s interaction with the virtual character is selected from the group consisting of a location within a real-world environment, a location within a virtual environment, a stage within the computer program.

21. The method of claim 16, wherein the signaling attribute is selected from the group consisting of adjustment of the virtual character'"'"'s view direction, movement of the virtual character, gesturing by the virtual character.