Maintaining multiple views on a shared stable virtual space

• US 8,730,156 B2
• Filed: 11/16/2010
• Issued: 05/20/2014
• Est. Priority Date: 03/05/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Weising, George, Miller, Thomas
• Primary Examiner(s): Thompson, James A
• Application Number:
  US12/947,290
• Publication Number:
  US 20110216060A1
• Time in Patent Office: 1,281 Days
• US Class Current: 345/156
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2009/2457 Display screens, e.g. monitors, video displays
• CPC Class Codes: A63F 3/02 Chess; Similar board games
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2320/0261 : in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 5/08 : Cursor circuits
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors

Abstract

Ways for controlling a virtual-scene view in a portable device are presented. In one method, a signal is received and the device is synchronized to make the location of the device a reference point in a three-dimensional (3D) space. A virtual scene with virtual reality elements is generated around the reference point. The current position of the device in the 3D space, with respect to the reference point, is determined and a view of the virtual scene created. The view represents the virtual scene as seen from the current position of the device with a viewing angle based on the position of the device. The created view is displayed in the device, and the view of the virtual scene is changed as the device is moved within the 3D space. In another method, multiple players shared the virtual reality and interact with each other in the virtual reality.

Claims

1. A method for controlling a view of a virtual scene with a portable device, the method comprising:

synchronizing the portable device to a reference point in a physical three-dimensional (3D) space, the reference point being a point in space occupied by the portable device when a signal to synchronize is received by the portable device;

capturing images of the physical 3D space with a camera of the portable device in response to the synchronizing;

tracking a current position of the portable device in the physical 3D space with respect to the reference point, the tracking utilizing image recognition of the captured images and inertial information obtained by inertial sensors in the portable device;

generating a virtual scene defined around the reference point, the virtual scene including virtual reality elements; and

displaying a view of the virtual scene in a display of the portable device based on the current position.

2. The method as recited in claim 1, wherein creating the view of the virtual scene further includes:

blending real elements in the physical 3D space with the virtual reality elements, wherein the virtual elements appear in the displayed view as if the virtual elements were part of the physical 3D space, wherein a view of the virtual elements changes, from a geometrical perspective, in a same way as a view of the real elements changes when the portable device moves within the physical 3D space.

3. The method as recited in claim 2, wherein the real elements include a table, wherein blending further includes:

placing virtual elements on top of the table as if the placed virtual elements were resting on the table.

4. The method as recited in claim 2, wherein the real elements include a wall in a room, wherein blending further includes:

adding an illustration to the wall, the illustration being one of the virtual elements.

5. The method as recited in claim 1, wherein the current position of the portable device includes geometric coordinates of the portable device and geometric coordinates of a surface of the display in the portable device.

6. The method as recited in claim 5, wherein the geometric coordinates of the portable device are equal to geometric coordinates of the camera of the portable device.

7. The method as recited in claim 6, wherein the viewing direction is defined in reference to a vector with origin in a center of the viewing surface of the display and a direction perpendicular to the surface of the display.

8. The method as recited in claim 1, wherein creating the view of the virtual scene further includes:

zooming in a second virtual reality element when the portable device moves closer to the second virtual reality element; and

zooming out the second virtual reality element when the portable device moves away from the second virtual reality element.

9. The method as recited in claim 1, wherein changing the displayed view further includes:

adding image stabilization to the displayed view as the portable device moves.

10. The method as recited in claim 1, further including:

receiving an input to change views; and

changing the creation of the view of the virtual scene such that the view of the virtual scene is calculated from another point different from the current position of the portable device.

11. The method as recited in claim 10, wherein the view of the virtual scene is rotated 180 degrees with respect to a vertical line intersecting the reference point.

12. The method as recited in claim 1, wherein the received signal is generated by pushing a button on the portable device or by touching a touch-sensitive display of the portable device.

13. The method as recited in claim 1, wherein boundaries of the virtual scene are defined by walls of a room where the portable device is located when the signal was received.

14. The method as recited in claim 1, wherein synchronizing the portable device further includes:

resetting position tracking modules in the portable device, the position tracking modules being selected from a group consisting of an accelerometer, a magnetometer, a camera, a depth camera, a compass, or a gyroscope,

wherein the tracking of the current position is performed with information from the position tracking modules.

15. A method comprising:

synchronizing a portable device to a reference point in a physical three-dimensional (3D) space, the reference point being a point in space occupied by the portable device when a signal to synchronize is received by the portable device;

capturing images of the physical 3D space with a camera of the portable device in response to the synchronizing;

generating a virtual scene defined around the reference point, the virtual scene including virtual reality elements;

determining a current position in the physical 3D space of the portable device with respect to the reference point;

displaying a view of the virtual scene in the display of the portable device based on the current position;

tracking a position of a hand of a player in the physical 3D space based on image recognition of the hand in the captured images;

detecting when the position of the hand is where a first virtual element is located; and

enabling, after the detecting, an interaction of the hand with the first virtual element to simulate that the hand is touching the virtual element, wherein the hand is able to manipulate the first virtual element to change a position or a property of the first virtual element as if the first virtual element were a real object.

16. The method as recited in claim 15, wherein the interaction of the hand is an action on the first virtual element selected from a group consisting of interfacing, holding, pushing, pulling, grabbing, moving, smashing, squeezing, hitting, throwing, fighting, opening, closing, turning on or off, pushing a button, firing, or eating.

17. The method as recited in claim 15, wherein creating a view further includes:

adding a shadow of the hand over the virtual elements, according to lighting conditions in the physical 3D space and the virtual scene.

18. A method for sharing a virtual scene among devices, the method comprising:

calculating, by a first device, a first location of the first device relative to a reference point in a physical three-dimensional (3D) space;

calculating, by the first device, a second location in the physical 3D space of a second device relative to the first location of the first device, wherein the first device and the second device are handheld devices;

sending information from the first device to the second device identifying the reference point in the physical 3D space, the information including the reference point, the first location, and the second location;

generating a virtual scene, augmenting the physical 3D space, situated around the reference point, the virtual scene being common to both devices and presented on respective displays of the first device and the second device, the virtual scene changing simultaneously in both devices in response to interaction from the first device or from the second device;

creating a first view of the virtual scene as seen from a current position of the first device;

displaying the first view in the display of the first device; and

changing the displayed first view of the virtual scene as the first device moves within the physical 3D space.

19. The method as recited in claim 18, wherein calculating the location of the second device further includes:

gathering, by the first device, a first relative location information between the first device and the second device, the gathering including one or more of WiFi positional tracking, audio triangulation, or image analysis obtained from a camera of the first device;

determining, by the first device, the second location of the second device with respect to the first location based on the first relative location information; and

sending from the first device the second location and coordinates of the reference point to the second device.

20. The method as recited in claim 18, further including:

gathering, by the first device, a first relative location information between the first device and the second device, the gathering including one or more of WiFi positional tracking, audio triangulation, or image analysis obtained from a camera of the first device or the second device;

receiving by the first device a second relative location information from the second device;

determining, by the first device, the second location of the second device with respect to the first device based on the first and second relative information; and

sending from the first device the second location and coordinates of the reference point to the second device.

21. The method as recited in claim 18, wherein the virtual scene includes a virtual board game, wherein players holding the first device and the second device, respectively, play the virtual board game.

22. The method as recited in claim 18, wherein a first player holding the first device controls movement of a first virtual element in the virtual scene by moving the first device in synchronism with the first virtual element.

23. A method for controlling a view of a virtual scene with a first device, the method comprising:

calculating a first location of a first device relative to a first reference point in a first physical three-dimensional (3D) space;

establishing a communications link between the first device and a second device, the second device being in a second physical 3D space outside the first physical 3D space, the second device having a second location relative to a second reference point in the second physical 3D space, wherein the first device and the second device are handheld devices;

sending from the first device a first image of a first user associated with the first device and receiving by the first device a second image of a second user associated with the second device, the first user and the second user being in different locations;

generating a common virtual scene that includes virtual reality elements, the common virtual scene being presented on a first display of the first device and a second display of the second device, the first device building the common virtual scene around the first reference point, the second device building the common virtual scene around the second reference point, both devices being able to interact with the virtual reality elements;

determining a current position in the first physical 3D space of the first device with respect to the first reference point;

creating a view of the common virtual scene, wherein the view represents the common virtual scene as seen from the current position of the first device;

blending the second image of the second user into the view of the common virtual scene to create a proximity effect that simulates that the second user is near the first user;

displaying the view of the common virtual scene in the display of the first device; and

changing the displayed view of the common virtual scene as the first device moves within the first physical 3D space.

24. The method as recited in claim 23, wherein the communications link includes a direct network connection between the first device and the second device.

25. The method as recited in claim 23, further including:

assigning a virtual location in the first physical 3D space to the second device;

receiving, from the second device, second device interaction information corresponding to interactions between the second device and the common virtual scene; and

changing the view of the common virtual scene according to the received second device interaction information and the virtual location, wherein the second device appears to be in the first physical 3D space and interacting with the first device as if the second device were physically located in the first physical 3D space.

26. The method as recited in claim 23, further including:

updating the second image of the second player periodically.

27. The method as recited in claim 23, further including:

updating the second image of the second player when the second player moves.

28. The method as recited in claim 23, wherein the virtual elements include a chess board and chess pieces, wherein the first device and the second device are used to play a game of chess by manipulating the chess pieces, wherein the second player appears in the view of the virtual scene to be sitting in front of the first player.

29. The method as recited in claim 23, wherein the first and second devices are represented in the view of the common virtual scene as a first object and second object, respectively, wherein movement of the first object matches movement of the first device in the first physical 3D space and movement of the second object matches movement of the second device in the second physical 3D space.

30. A method for controlling a view of a virtual scene with a portable device, the method comprising:

synchronizing the portable device to a reference point in a physical three-dimensional (3D) space, the reference point being a point in space occupied by the portable device when a signal to synchronize is received by the portable device, the portable device including a front camera facing the front of the portable device and a rear camera facing the rear of the portable device, the portable device being a handheld device;

capturing images of the physical 3D space with a camera of the portable device in response to the synchronizing;

tracking a current position of the portable device in the physical 3D space with respect to the reference point, the tracking utilizing image recognition of the captured images and inertial information obtained by inertial sensors in the portable device;

generating a virtual scene defined around the reference point, the virtual scene including virtual reality elements;

creating a view of the virtual scene based on a current position of the portable device, the view capturing a representation of the virtual scene as seen from a current eye position in the physical 3D space of a player holding the portable device, the capturing corresponding to what the player would see through a window into the virtual scene, a window'"'"'s position in the physical 3D space being equal to a position in the physical 3D space of a display in the portable device;

displaying the created view in the display; and

changing the displayed view of the virtual scene as the portable device or the player move within the physical 3D space, wherein a change in a position of an eye of the player holding the portable device while keeping the portable device stationary causes a change of the displayed view.

31. The method as recited in claim 30, wherein images from the front camera are used to determine the current eye position and images from the rear camera are used to get a view of the physical 3D space.

32. The method as recited in claim 30, wherein pulling the display away from the player'"'"'s eyes causes the view to zoom in the virtual scene and pulling the display closer to the player'"'"'s eyes causes the view to zoom out of the virtual scene.

33. A method for controlling a view of a scene with a portable device, the method comprising:

synchronizing the portable device to make a location where the portable device is located a reference point in a physical three-dimensional (3D) space, the reference point being a point in space occupied by the portable device when a signal to synchronize is received by the portable device;

generating a virtual scene in a display of the portable device when viewing the physical 3D space via the portable device, the virtual scene defined around the reference point, the virtual scene including virtual reality elements;

creating a view of the virtual scene as the portable device is moved away from the reference point, wherein the view represents the virtual scene as seen from a current position of the portable device, wherein the view created is independent of a position of an eye of a user holding the portable device;

displaying the created view in the portable device; and

changing the displayed view of the virtual scene as the portable device moves within the physical 3D space.

34. The method as recited in claim 33, further including:

determining a virtual element that generates sound; and

emitting a sound from the portable device corresponding to the sound generated by the virtual element, wherein the sound emitted becomes louder as the portable device is moved closer to a position of the virtual element that generates sound.

35. The method as recited in claim 34, wherein the portable device has stereo speakers, wherein the sound emitted is adjusted to provide a stereo effect according to a relative location between the portable device and the virtual element that generates sound.

36. A computer program embedded in a non-transitory computer-readable storage medium, when executed by one or more processors, for sharing a virtual scene among devices, the computer program comprising:

program instructions for calculating, by a first device, a first location of a first device relative to a reference point in a physical three-dimensional (3D) space, the first device being a handheld device;

program instructions for calculating, by the first device, a second location in the physical 3D space of a second device relative to the first location of the first device;

program instructions for sending information from the first device to the second device identifying the reference point in the physical 3D space, the information including the reference point, the first location, and the second location;

program instructions for generating a virtual scene, augmenting the physical 3D space, situated around the reference point, the virtual scene being common to both devices and presented on respective displays of the first device and the second device, the virtual scene changing simultaneously in both devices in response to interaction from the first device or from the second device;

program instructions for creating a first view of the virtual scene as seen from a current position of the first device;

program instructions for displaying the first view in the display of the first device; and

program instructions for changing the displayed first view of the virtual scene as the first device moves within the physical 3D space.