Method and system for making a selection in 3D virtual environment

• US 8,384,665 B1
• Filed: 07/14/2010
• Issued: 02/26/2013
• Est. Priority Date: 07/14/2006
• Status: Active Grant

• Current Assignee: Wei Yen
• Original Assignee: AiLive, Inc.
• Inventors: Powers, III, William Robert, Musick, JR., Charles, Wilkinson, Dana
• Primary Examiner(s): Hall, Arthur O.
• Assistant Examiner(s): HOANG, BACH V
• Application Number:
  US12/835,755
• Time in Patent Office: 958 Days
• Field of Search: 463/36, 463/37, 463/47, 345156-158, 345162-163, 345/169, 702/141, 702/145, 702150-154
• US Class Current: 345/156
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Techniques for selecting a portion of a 3D virtual environment are disclosed, where the portion of the 3D virtual environment may be one or more objects in the 3D virtual environment or one of scenes in the 3D virtual environment. A motion sensitive device (controller) is used. The controller generates sensor data sufficient to derive position and orientation of the controller in six degrees of freedom. In one embodiment, when selecting the portion of the 3D virtual environment, the controller generates a ray to an interaction of a display screen provided to display the 3D virtual environment, wherein the ray is further projected into the 3D virtual environment by a ray tracing technique as if the user had a real laser pointer that crosses from the physical world into the 3D virtual environment being displayed. Further a user is optionally to use a secondary input device to select one or more points or objects in a 3D space to control a distance along the ray being used.

Claims

1. A system for a user to select a portion of a 3D virtual environment being displayed, the system comprising:

a controller including a plurality of inertia sensors providing sensor signals sufficient to derive changes in motion of the controller in six degrees of freedom;

a processing unit, receiving the sensor signals, configured to derive position and orientation of the controller from the sensor signals, and generate a ray originating from the controller to an interaction of a screen provided to display the 3D virtual environment, wherein the ray is further projected into the 3D virtual environment by a ray tracing technique; and

a video capturing system including at least one camera, disposed in front or in vicinity of the user to capture the motion of the controller being held by the user, wherein the processing unit receives images from the video capturing system, the controller is used to calibrate a relationship with the screen used to display the 3D virtual environment to estimate size and location of the screen with respect to the camera so as to find absolute position and orientation of the controller in a physical world.

2. The system as recited in claim 1, wherein the portion of the 3D virtual environment is an object, and the ray is combined with an application-specific geometry to determine an intersection of the ray with the screen.

3. The system as recited in claim 2, wherein the processing unit is further configured to integrate the motion of the controller with an application being executed, the 3D virtual environment is a display resulted from the application.

4. The system as recited in claim 1, wherein the changes in the motion of the controller are determined in the processing unit, with or without image signals from the camera.

5. The system as recited in claim 4, wherein the controller is subsequently used to control motion of a selected object in the 3D virtual environment.

6. The system as recited in claim 4, wherein the controller is caused to point at four or more corners of the screen to allow determination of an angle between a known location of the controller and each of the corners.

7. The system as recited in claim 4, wherein the processing unit receives images from the video capturing system, the images are processed to determine an equivalent trajectory of the controller in a 2D image plane, the trajectory is used together with the sensor signals to determine the absolute position and orientation of the controller in a physical world.

8. The system as recited in claim 7, wherein the ray is originated from the absolute position and orientation of the controller.

9. The system as recited in claim 1, wherein an additional input is provided by the user to select the portion of the virtual environment along the ray.

10. The system as recited in claim 9, wherein the additional input is from another controller.

11. The system as recited in claim 1, further comprising two or more controllers to select the portion of the 3D virtual environment, wherein respective rays are projected from the two or more controllers and distances between the rays are calculated.

12. The system as recited in claim 1, wherein the portion of the 3D virtual environment is an object capable of moving relatively in the 3D virtual environment or one of scenes in the 3D virtual environment.

13. The system as recited in claim 1, wherein the screen is capable of displaying the 3D virtual environment in 3D, one or more rays extend before or into the screen in order to allow the user to point at the portion that appears to be at a location in a physical world.

14. A method for a user to select a portion of a 3D virtual environment being displayed, the method comprising:

receiving sensor signals from a controller sufficient to derive motion of the controller in six degrees of freedom, wherein the controller includes a plurality of inertia sensors that generate the sensor signals when being manipulated by the user;

deriving in a processing unit the motion of the controller from the sensor signals; and

determining an orientation of the controller by ray originating from the controller to a display screen provided to display the 3D virtual environment, wherein the ray is further projected into the 3D virtual environment by a ray tracing technique;

generating images from a video capturing system disposed in front or in vicinity of the user to capture movements of the controller by the user, wherein said determining an orientation of the controller comprises calibrating a relationship of the controller with the display screen to find an actual size and location of the display screen with respect to the video capturing system so as to find absolute position and orientation of the controller.

15. The method as recited in claim 14, wherein the portion of the 3D virtual environment is an object, and the ray is combined with application-specific geometry to determine an intersection of the ray with the display screen.

16. The method as recited in claim 15, further comprising integrating the motion of the controller with an application being executed in the processing unit, the 3D virtual environment is a display resulted from the application.

17. The method as recited in claim 14, wherein the controller is caused to point at four or more corners of the display screen to allow determination of an angle between a known location of the controller and each of the corners.

18. The method as recited in claim 14, wherein the images are processed in the processing unit to determine an equivalent trajectory of the controller in a 2D image plane, the trajectory is used together with the sensor signals to determine absolute position and orientation of the controller in a physical world.

19. The method as recited in claim 18, wherein the ray is originated from the absolute position and orientation of the controller.

20. The method as recited in claim 18, further comprising receiving an additional input from another device to select the portion of the virtual environment along the ray.

21. The method as recited in claim 20, wherein the another device is another controller, and a distance between the controller and the another controller is determined to infer a depth of the portion along the ray being selected by the controller.

22. The method as recited in claim 14, further comprising projecting rays from two or more controllers to select the portion of the 3D virtual environment, and distances between the rays are calculated.

23. The method as recited in claim 15, wherein the portion of the 3D virtual environment is an object capable of moving relatively to the 3D virtual environment or one of scenes in the 3D virtual environment.