Position-dependent gaming, 3-D controller, and handheld as a remote

• US 9,159,165 B2
• Filed: 07/13/2010
• Issued: 10/13/2015
• Est. Priority Date: 07/13/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Stafford, Jeffrey R., Zhu, Yunpeng, Osman, Steven
• Primary Examiner(s): FUJITA, KATRINA R
• Application Number:
  US12/835,671
• Publication Number:
  US 20120014558A1
• Time in Patent Office: 1,918 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/5258 by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/5573 player location
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof

Abstract

Methods and systems for using a position of a mobile device with an integrated display as an input to a video game or other presentation are presented. Embodiments include rendering an avatar on a mobile device such that it appears to overlay a competing user in the real world. Using the mobile device'"'"'s position, view direction, and the other user'"'"'s mobile device position, an avatar (or vehicle, etc.) is depicted at an apparently inertially stabilized location of the other user'"'"'s mobile device or body. Some embodiments may estimate the other user'"'"'s head and body positions and angles and reflect them in the avatar'"'"'s gestures.

Claims

1. A method for augmenting video, comprising:

receiving a first position coordinate corresponding to a first user, the first position coordinate relative to a first video display;

receiving a first view direction corresponding to the first user, the first view direction relative to the first video display;

receiving a second position coordinate corresponding to a second user, the second position coordinate relative to a second video display;

estimating a position of the second user'"'"'s head relative to a first mobile device;

determining a direction and range from the first position coordinate to the second position coordinate; and

rendering, on an integrated display of the first mobile device, an object based on the determined direction and range from the first position coordinate to the second position coordinate and based on the received first view direction and based on the estimated position of the second user'"'"'s head, wherein the first video display and the second video display are different from each other and from the integrated display of the first mobile device.

2. The method of claim 1 wherein the object is displayed as if it were co-located with the second user from a vantage point of the first user.

3. The method of claim 1 further comprising:

receiving a second view direction corresponding to the second user, the second view direction relative to the second video display,

wherein the rendering is further based on the received second view direction.

4. The method of claim 3 wherein the object is selected from the group consisting of an avatar and a vehicle.

5. The method of claim 1 wherein the position coordinates corresponding to the users are estimates of where the users'"'"' heads or hands are.

6. The method of claim 5 wherein the first position coordinate corresponding to the first user includes an estimate of position of the first user'"'"'s hands relative to the first mobile device.

7. The method of claim 1 wherein the position coordinates corresponding to the users are position coordinates of mobile devices worn by the users.

8. The method of claim 4 wherein the first view direction comprises an azimuth and elevation.

9. The method of claim 1 wherein slewing the first mobile device to a first direction causes the rendered object to disappear off to an opposite direction of the first direction.

10. The method of claim 1 wherein the first view direction comprises a bearing direction in relation to a fixed point.

11. The method of claim 1 further comprising:

rendering, on the integrated display of the first mobile device, a background object.

12. The method of claim 1 wherein the position of the second user'"'"'s head relative to a first mobile device is estimated using predefined models.

13. A system for augmented video, the system comprising:

a processor; and

a memory device including instructions that, when executed by the processor, cause the processor to;

receive a first position coordinate corresponding to a first user, the first position coordinate relative to a first video display;

receive a first view direction corresponding to the first user, the first view direction relative to the first video display;

receive a second position coordinate corresponding to a second user, the second position coordinate relative to a second video display;

estimate a position of the second user'"'"'s head relative to a first mobile device;

determine a direction and range from the first position coordinate to the second position coordinate; and

render, on an integrated display of the first mobile device, an object based on the determined direction and range from the first position coordinate to the second position coordinate and based on the received first view direction and based on the estimated position of the second user'"'"'s head, wherein the first video display and the second video display are different from each other and from the integrated display of the first mobile device.

14. The system of claim 13 wherein the object is displayed as if it were co-located with the second user from a vantage point of the first user.

15. The system of claim 13 wherein the memory further includes instructions that, when executed by the processor, cause the processor to:

receive a second view direction corresponding to the second user, the second view direction relative to the second video display,

wherein the rendering is further based on the received second view direction.

16. The system of claim 13 wherein the memory further includes instructions that, when executed by the processor, cause the processor to:

send a remote control command to the first video display.

17. A machine-readable non-transitory storage medium embodying information indicative of instructions for causing one or more machines to perform operations, the operations comprising:

receiving a first position coordinate corresponding to a first user, the first position coordinate relative to a first video display;

receiving a first view direction corresponding to the first user, the first view direction relative to the first video display;

receiving a second position coordinate corresponding to a second user, the second position coordinate relative to a second video display;

estimating a position of the second user'"'"'s head relative to a first mobile device;

determining a direction and range from the first position coordinate to the second position coordinate; and

rendering, on an integrated display of the first mobile device, an object based on the determined direction and range from the first position coordinate to the second position coordinate and based on the received first view direction and based on the estimated position of the second user'"'"'s head, wherein the first video display and the second video display are different from each other and from the integrated display of the first mobile device.

18. The machine-readable non-transitory storage medium of claim 17 wherein the position coordinates corresponding to the users are position coordinates of mobile devices worn by the users.

19. The machine-readable non-transitory storage medium of claim 17 wherein the first view direction comprises an azimuth and elevation.

20. The machine-readable non-transitory storage medium of claim 17 wherein the first mobile device tracks the first video display using markers on the first video display.