POSITION-DEPENDENT GAMING, 3-D CONTROLLER, AND HANDHELD AS A REMOTE

• US 20120014558A1
• Filed: 07/13/2010
• Published: 01/19/2012
• Est. Priority Date: 07/13/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Stafford, Jeffrey R., Zhu, Yunpeng, Osman, Steven
• Granted Patent: US 9,159,165 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/5258 by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/5573 player location
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof

Abstract

Methods and systems for using a position of a mobile device with an integrated display as an input to a video game or other presentation are presented. Embodiments include rendering an avatar on a mobile device such that it appears to overlay a competing user in the real world. Using the mobile device'"'"'s position, view direction, and the other user'"'"'s mobile device position, an avatar (or vehicle, etc.) is depicted at an apparently inertially stabilized location of the other user'"'"'s mobile device or body. Some embodiments may estimate the other user'"'"'s head and body positions and angles and reflect them in the avatar'"'"'s gestures.

Claims

1. A system for augmented video, the system comprising:

a video source configured to provide video content to a video display; and

a mobile device with an integrated video camera and display, the mobile device configured to;

track a relative distance and direction of the video display using the video camera;

determine a position coordinate of the mobile device using the tracked relative distance and direction; and

render, on the integrated display, an object in a position based on the determined position coordinate of the mobile device.

2. The method of claim 1 wherein the video source and mobile device are configured to execute a video game, the position coordinate input to the video game.

3. The method of claim 1 wherein the mobile device is configured to control the video display.

4. A method for augmenting video, comprising:

receiving a first position coordinate corresponding to a first user, the first position coordinate relative to a first video display;

receiving a first view direction corresponding to the first user, the first view direction relative to the first video display;

receiving a second position coordinate corresponding to a second user, the second position coordinate relative to a second video display;

determining a direction and range from the first position coordinate to the second position coordinate; and

rendering, on an integrated display of a first mobile device, an object based on the determined direction and range from the first position coordinate to the second position coordinate and based on the received first view direction.

5. The method of claim 4 wherein the object is displayed as if it were co-located with the second user from a vantage point of the first user.

6. The method of claim 4 further comprising:

receiving a second view direction corresponding to the second user, the second view direction relative to the second video display,

wherein the rendering is further based on the received second view direction.

7. The method of claim 6 wherein the object is selected from the group consisting of an avatar and a vehicle.

8. The method of claim 7 wherein the avatar or vehicle is rendered to face in the second view direction.

9. The method of claim 4 further comprising:

estimating a position of the second user'"'"'s head,

wherein the rendering of the object is further based on the estimate of the position of the second user'"'"'s head.

10. The method of claim 4 wherein the position coordinates corresponding to the users are estimates of where the users'"'"' heads or hands are.

11. The method of claim 4 wherein the position coordinates corresponding to the users are position coordinates of mobile devices worn by the users.

12. The method of claim 4 wherein determining the direction and range from the first position coordinate to the second position coordinate comprises co-locating and aligning the first and second frames of references.

13. The method of claim 4 wherein the position coordinates are of a type selected from the group consisting of Cartesian, polar or cylindrical, and spherical.

14. The method of claim 4 wherein the first view direction comprises an azimuth and elevation.

15. The method of claim 4 wherein the first and second video displays are the same display.

16. The method of claim 4 wherein the receivings, determining, and rendering are performed in the order recited.

17. A system for augmenting video, comprising:

a first mobile device having a display;

a second mobile device;

means for determining a relative direction and range from the first mobile device to the second mobile device; and

means for determining a view direction of the first mobile device;

wherein the first mobile device is configured to render on its display an avatar or vehicle from a perspective based on the relative direction and range to the second mobile device and view direction of the first mobile device.

18. The system of claim 17 further comprising:

means for determining a pointing direction of the second mobile device,

wherein the first mobile device is further configured to render on its display the avatar or vehicle facing a direction based on the pointing direction of the second mobile device.

19. The system of claim 17 wherein the first mobile device comprises glasses having an integrated display in at least one lens of the glasses.

20. The system of claim 17 wherein the means for determining the relative direction and range from the first mobile device to the second mobile device comprise means off-board the first mobile device.