System and method of virtual interaction

• US 9,122,391 B2
• Filed: 12/28/2010
• Issued: 09/01/2015
• Est. Priority Date: 12/31/2009
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Lee, Richard, Coates, Sam, Grenfell, Paul Kevan
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): TRAN, KIM THANH THI
• Application Number:
  US12/979,636
• Publication Number:
  US 20110298824A1
• Time in Patent Office: 1,708 Days
• Field of Search: 345632-633, 455/404.2, 455/456.1, 455/459
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/31 Communication aspects specific to video games, e.g. between several handheld game devices at close range
• CPC Class Codes: A63F 13/327 using wireless networks, e.g. Wi-Fi® or piconet
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/405 being a wireless ad hoc network, e.g. Bluetooth, Wi-Fi, Pico net
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A system for virtual interaction, comprising two or more portable electronic devices, is provided. Each device comprises, in turn, coordinate referencing means operable to define a coordinate system common to the portable electronic devices with respect to a physically defined reference position, position estimation means operable to detect a the physical position of its respective portable electronic device with respect to the reference position, virtual environment generation means operable to generate a virtual environment, and communication means operable to transmit positional data using the common coordinate system from that portable electronic device to another portable electronic device. The virtual environment is shared in common between the portable electronic devices. The virtual environment uses the common co-ordinate system within which each portable electronic device defines a position for itself responsive to its physical position with respect to the reference position.

Claims

1. A system for virtual interaction, comprising:

two or more portable electronic devices, each device comprising in turn;

a coordinate referencer configured to define a new coordinate system common to said portable electronic devices with respect to a real, physically defined common reference position in a real-world space currently shared by said portable electronic devices, where said common reference position is a point where said portable electronic devices are placed physically adjacent to one another;

a position estimator configured to detect a physical position of its respective portable electronic device with respect to said reference position;

a virtual environment generator configured to generate a virtual environment; and

a communicator configured to transmit positional data using said common coordinate system from that portable electronic device to another portable electronic device; and

in which

said virtual environment is shared in common between said portable electronic devices, said virtual environment using said common co-ordinate system within which each portable electronic device defines a position for itself responsive to its physical position with respect to said common reference position.

2. A system for virtual interaction according to claim 1, in which the position estimator of at least one of said portable electronic devices comprises:

one or more accelerometers; and

a motion tracker configured to estimate a physical position with respect to said reference position from cumulative changes in physical position detected using said one or more accelerometers.

3. A system for virtual interaction according to claim 1, in which said position estimator of at least one of said portable electronic devices comprises:

a video camera configured to capture a sequence of video images; and

an optical flow estimator configured to estimate a change in physical position of a respective portable electronic device as a motion that would account for apparent motion in said sequence of video images captured by the video camera, and to estimate a physical position with respect to said reference position from cumulative estimated changes in physical position.

4. A system for virtual interaction according to claim 1, in which the position estimator of at least one of said portable electronic devices comprises:

a video camera configured to capture a sequence of video images; and

an image processor configured to estimate a physical position of the portable electronic device based upon distortion characteristics of a captured video image of a reference object at said reference position.

5. A system for virtual interaction according to claim 1, in which the position estimator of at least one of the portable electronic devices comprises:

a wireless identification signal transmitter; and

a wireless signal triangulator; and

in which

the at least one of said portable electronic devices triangulates its relative physical position with respect to another one of said portable electronic devices.

6. A system for virtual interaction according to claim 1, in which at least one of said portable electronic devices comprises two or more position estimators;

and in which the at least one of said portable electronic devices is configured to use a second position estimator to adjust the operation of a first position estimator.

7. A system for virtual interaction according to claim 1, in which at least one of said portable electronic device comprises:

a video camera configured to capture a video image; and

in which

the at least one said portable electronic device is configured to detect an error in the correspondence between the real relative position of another portable electronic device and the relative position of the other portable electronic device within said common coordinate system, by analysing a captured video image to detect whether said other portable electronic device lies within a region of said captured video image predicted to encompass to the position of the other portable electronic device based upon its position within said common coordinate system.

8. A system for virtual interaction according to claim 1, in which at least one of said portable electronic devices comprises:

a video camera configured to capture a video image; and

a display screen; and

in which

the at least one said portable electronic device is configured to display a captured video image of the physical environment on said display screen in conjunction with one or more virtual objects included in said shared virtual environment.

9. A system for virtual interaction according to claim 1, in which control of at least one virtual object within said shared virtual environment is shared between said two or more portable electronic devices.

10. A system for virtual interaction according to claim 1, in which boundaries of said shared virtual environment are defined by moving one or more of said portable electronic devices to one or more boundaries of a physical environment and noting the corresponding position of the or each portable electronic device within said shared virtual environment.

11. A method of virtual interaction between two or more portable electronic devices comprises the steps of:

defining a new coordinate system common to said portable electronic devices with respect to a real, physically defined common reference position in a real-world space currently shared by said portable electronic devices, where said common reference position is a point where said portable electronic devices are placed physically adjacent to one another;

estimating a physical position of at least one portable electronic device with respect to said reference position;

generating a virtual environment;

communicating positional data between said portable electronic devices using said common coordinate system; and

in which

said virtual environment is shared in common between said portable electronic devices, the virtual environment using the common co-ordinate system;

and for each portable electronic device,

defining its position in said shared virtual environment responsive to its physical position with respect to said common reference position.

12. A method of virtual interaction between two or more portable electronic devices according to claim 11, in which the step of estimating the physical position of the at least one respective portable electronic device comprises one or more selected from the list consisting of:

i. estimating the physical position with respect to said reference position from cumulative changes in physical position calculated from detected acceleration of said respective portable electronic device;

ii. estimating a physical motion that accounts for apparent motion in a sequence of video images captured by a video camera of said respective portable electronic device, and estimating the physical position with respect to said reference position from cumulative estimated motion;

iii. estimating the physical position with respect to said reference position based upon distortion characteristics of a captured video image of a reference object at said reference position, captured by a video camera of said respective portable electronic device; and

iv. estimating a relative physical position with respect to another portable electronic device based upon triangulated wireless identification signals.

13. A method of virtual interaction between two or more portable electronic devices according to claim 11, comprising the step of:

passing control of at least one virtual object within said shared virtual environment between said two or more portable electronic devices, so as to allocate control of said virtual object to a portable electronic device which most recently interacted with said virtual object.

14. A method of virtual interaction between two or more portable electronic devices according to claim 11, comprising the step of:

defining boundaries of said shared virtual environment by moving one or more of said portable electronic devices to one or more boundaries of a physical environment and noting the corresponding position of the or each portable electronic device within said shared virtual environment.

15. A non-transitory computer-readable storage medium on which computer readable instructions of a computer program are stored, the instructions, when executed by a computer, cause the computer to perform a method of virtual interaction between two or more portable electronic devices comprising:

defining a new coordinate system common to said portable electronic devices with respect to a real, physically defined common reference position in a real-world space currently shared by said portable electronic devices, where said common reference position is a point where said portable electronic devices are placed physically adjacent to one another;

estimating a physical position of at least one portable electronic device with respect to said reference position;

generating a virtual environment;

communicating positional data between said portable electronic devices using said common coordinate system; and

in which

said virtual environment is shared in common between said portable electronic devices, the virtual environment using the common co-ordinate system;

and for each portable electronic device,

defining its position in said shared virtual environment responsive to its physical position with respect to said common reference position.