Maintaining multiple views on a shared stable virtual space

• US 9,310,883 B2
• Filed: 04/23/2014
• Issued: 04/12/2016
• Est. Priority Date: 03/05/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Weising, George, Miller, Thomas
• Primary Examiner(s): McClellan, James S
• Assistant Examiner(s): CARTER, KEVIN M
• Application Number:
  US14/260,216
• Publication Number:
  US 20140235311A1
• Time in Patent Office: 720 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2009/2457 Display screens, e.g. monitors, video displays
• CPC Class Codes: A63F 3/02 Chess; Similar board games
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2320/0261 : in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 5/08 : Cursor circuits
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors

Abstract

Methods, apparatus, and computer programs for controlling a view of a virtual scene with a handheld device are presented. In one method, images of a real world scene are captured using a device. The method further includes operations for creating an augmented view for presentation on a display of the device by augmenting the images with virtual reality objects, and for detecting a hand in the images as extending into the real world scene. In addition, the method includes operations for showing the hand in the screen as detected in the images, and for generating interaction data, based on an interaction of the hand with a virtual reality object, when the hand makes virtual contact in the augmented view with the virtual reality object. The augmented view is updated based on the interaction data, which simulates on the screen that the hand is interacting with the virtual reality object.

Claims

1. A method comprising:

receiving, by a handheld device, remote images of a remote real world scene sent by a remote device, the remote device being operated by a remote player communicating with the handheld device as operated by a user;

capturing local images of a local real world scene using the handheld device;

creating an augmented view for presentation on a display screen of the handheld device by combining the remote images with the captured local images and one or more virtual reality objects, wherein the augmented view provides a proximity effect to simulate that the remote player and the user share a space that includes the one or more virtual reality objects;

detecting a user hand in the captured local images as extending into the local real world scene;

adding the user hand to the augmented view as detected in the captured local images;

generating interaction data based on an interaction of the user hand with a first virtual reality object from the one or more virtual reality objects when the user hand makes virtual contact in the augmented view with the first virtual reality object;

updating the augmented view based on the interaction data, wherein the interaction data simulates on the display screen that the user hand is interacting with the first virtual reality object; and

transmitting, the interaction data and one or more of the captured local images of the local real world, from the handheld device to the remote device so that the remote device presents a view that includes a change to the first virtual reality object based on the interaction data.

2. The method as recited in claim 1, wherein the interaction of the user hand is an action on the first virtual reality object selected from a group consisting of holding, pushing, pulling, moving, hitting, throwing, opening, closing, turning on or off, pushing a button, firing a weapon, or eating the first virtual reality object.

3. The method as recited in claim 1, wherein creating the augmented view further includes:

determining a location of the handheld device utilizing one or more of image analysis of the captured local images of the local real world scene, or data obtained from inertial systems within the handheld device, or GPS, or ultrasonic triangulation, or WiFi communications, or dead reckoning.

4. The method as recited in claim 1, wherein creating the augmented view further includes:

detecting a table in the remote real world scene using image recognition of the remote images; and

placing in the augmented view a virtual game board on top of the table as if the virtual game board was resting on the table.

5. The method as recited in claim 1, wherein the interaction data is associated with the user hand grabbing the first virtual reality object.

6. The method as recited in claim 5, wherein the interaction includes moving the first virtual reality object after the user hand grabs the first virtual reality object.

7. The method as recited in claim 1, wherein the augmented view simulates that a camera in a back of the handheld device moves within the local real world scene.

8. The method as recited in claim 1, wherein the first virtual reality object is a chess piece, wherein the interaction of the user hand includes one or more of grabbing the chess piece, or moving the chess piece, or releasing the chess piece, the method further including:

transmitting the interaction of the user hand to the remote device.

18. The method as recited in claim 1, wherein the remote images are associated with a remote player playing a game with the user, wherein the one or more virtual reality objects are game objects, the method further including:

receiving remote interaction data from the remote device; and

updating the one or more virtual objects in the augmented view based on the received remote interaction data.

19. The method as recited in claim 18, further including:

capturing rear-facing images with a rear-facing camera of the handheld device; and

transmitting the rear-facing images to the remote device, wherein the remote device is operable to present the rear-facing images to the remote player while playing the game.

20. The method as recited in claim 18, further including:

receiving, by the handheld device, an update of a face of the remote player; and

updating the face of the remote player in the augmented view while leaving a background unchanged in the augmented view.

9. A method comprising:

capturing images of a real world scene using a plurality of handheld devices associated with a plurality of users, the images being captured while the plurality of handheld devices are proximate to each other and located within the real world scene or next to the real world scene;

creating augmented views for presentation on respective display screens of the plurality of handheld devices by augmenting the captured images with one or more virtual reality objects, wherein the augmented view provides a proximity effect to simulate that the plurality of users interact with the one or more virtual reality objects added to the real world scene;

detecting a user hand in first captured images of a first handheld device, the user hand extending into the real world scene;

showing the user hand in a first display screen of the first handheld device as detected in the first captured images;

generating interaction data based on an interaction of the user hand with a first virtual reality object from the one or more virtual reality objects when the user hand makes virtual contact with the first virtual reality object in a first augmented view of the first handheld device;

updating the first augmented view based on the interaction data, wherein the interaction data simulates on the first display screen that the user hand is interacting with the first virtual reality object; and

transmitting the interaction data to other of the plurality of handheld devices so that the other of the plurality of handheld devices present a view that includes a change to the first virtual reality object as the user hand interacts with the first virtual reality object based on the interaction data, wherein the augmented views of the other of the plurality of handheld devices show the changes to the first virtual object as the user hand makes virtual contact with the first virtual object.

10. The method as recited in claim 9, wherein each handheld device from the plurality of handheld devices is operable to detect a user hand of a respective user and generate interaction data based on the interaction of the respective user with any of the one or more virtual reality objects.

11. The method as recited in claim 9, wherein the interaction of the user hand is an action on the first virtual reality object selected from a group consisting of holding, or opening, or smashing.

12. The method as recited in claim 9, further including:

exchanging among the plurality of handheld devices image information, relative motion, and ping data to maintain synchronization of the plurality of handheld devices with respect to a reference point in the real world scene.

13. The method as recited in claim 9, exchanging among the plurality of handheld devices image information, relative motion, and positional information to synchronize calibrated positions for the plurality of handheld devices in the augmented views.

14. The method as recited in claim 9, wherein the augmented views in the respective displays from the plurality of handheld devices are independent from each other and respective augmented views are based on a position of a respective handheld device.

15. The method as recited in claim 9, wherein the first augmented view includes a virtual board game, wherein players holding the respective handheld devices play a virtual board game, wherein a location of the virtual board game with reference to the real world scene is a same location for all of the plurality of handheld devices.

16. A non-transitory computer-readable storage medium storing a computer program, the computer-readable storage medium comprising:

program instructions for receiving, by a handheld device, remote images of a remote real world scene sent by a remote device, the remote device being operated by a remote player communicating with a user operating the handheld device;

program instructions for capturing local images of a local real world scene using the handheld device;

program instructions for creating an augmented view for presentation on a display screen of the handheld device by combining the remote images with the captured local images and one or more virtual reality objects, wherein the augmented view provides a proximity effect to simulate that the remote player and the user share a space that includes the one or more virtual reality objects;

program instructions for capturing local images of a local real world scene using the handheld device;

program instructions for detecting a user hand in the captured local images as extending into the local real world scene;

program instructions for adding the user hand to the augmented view as detected in the captured local images;

program instructions for generating interaction data based on an interaction of the user hand with a first virtual reality object from the one or more virtual reality objects when the user hand makes virtual contact in the augmented view with the first virtual reality object;

program instructions for updating the augmented view based on the interaction data, wherein the interaction data simulates on the display screen that the user hand is interacting with the first virtual reality object; and

program instructions for transmitting, the interaction data and one or more of the captured local images of the local real world, from the handheld device to the remote device so that the remote device presents a view that includes a change to the first virtual reality object based on the interaction data.

17. The storage medium as recited in claim 16, wherein creating augmented views further includes:

adding a shadow of the user hand over the one or more virtual reality objects according to lighting conditions in the real world scene.