Multi-axis motion-based remote control

• US 8,237,656 B2
• Filed: 07/06/2007
• Issued: 08/07/2012
• Est. Priority Date: 07/06/2007
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hsu, Feng-Hsiung, Gao, Rui
• Primary Examiner(s): Patel, Nitin
• Assistant Examiner(s): PARKER, JEFFREY ALAN
• Application Number:
  US11/774,515
• Publication Number:
  US 20090009469A1
• Time in Patent Office: 1,859 Days
• Field of Search: 345/163, 345/161, 345/158, 345/157, 345/156
• US Class Current: 345/158
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image

Abstract

Motion-based control of an electronic device uses an array of at least three reference elements forming a triangle. An image sensor (e.g., a video camera), which may be located on a user-manipulated device, captures an image of the array. The array image has a pattern formed by a nonparallel projection of the reference triangle onto the image sensor. The pattern carries information of the relative position between the image sensor and the reference element array, and changes as the relative position changes. The pattern is identified and used for generating position information, which may express a multidimensional position of the user-manipulated device with respect to three axes describing a translational position, and three rotational axes describing pitch, roll and yaw motions. The control system and method are particularly suitable for videogames.

Claims

1. A system for motion-based controlling of an electronic device, the system comprising:

a user-manipulated device;

an image sensor;

a reference element array including a first reference element, a second reference element and a third reference element each having a different size from one another and defining vertices of a reference triangle, the reference triangle defining a plane substantially nonparallel to a front surface of a display, one of the image sensor or the reference element array being attached to the user-manipulated device, the reference element array forming an array image on the image sensor through a nonparallel projection of the reference triangle onto the image sensor, the array image having a pattern changing shape as a relative position between the image sensor and the reference element array changes, at least one of the first reference element, the second reference element, or the third reference element elevated off a plane with respect to non-elevated reference elements to facilitate discrimination of the at least one elevated reference element from the non-elevated reference elements, to increase visibility of the elevated reference element, and to increase a degree of shape differentiation of the array image; and

a position determination module configured to;

receive image information from the image sensor, the image information including a reference element image of each reference element, each reference element image comprising multiple pixels;

identify a pattern of reference elements from the image information by clustering the multiple pixels of each reference element image to a separate group and calculating a weighted barycenter of each group based on brightness as a weight; and

generate position information based on the pattern of the reference elements identified in the image information, the position information expressing the relative position between the image sensor and the reference element array.

2. The system as recited in claim 1, wherein the image sensor is a part of a video camera attached to the user-manipulated device, the relative position between the image sensor and the reference element array representing a position of the user-manipulated device relative to the reference element array.

3. The system as recited in claim 2, wherein the video camera is an integral part of the user-manipulated device.

4. The system as recited in claim 1, wherein the electronic device comprises a game console.

5. The system as recited in claim 1, wherein the reference element array is adapted for being placed near the display when the display is connected to the electronic device such that at least one of the reference elements is located in a position outward from the front surface of the display.

6. The system as recited in claim 1, wherein the plane defined by the reference triangle is substantially perpendicular to the front surface of the display.

7. The system as recited in claim 1, wherein the reference element array is adapted for being placed on a top or near a bottom portion of the display.

8. The system as recited in claim 1, wherein the position determination module is configured to generate position information using geometric relations based on the nonparallel projection of the reference triangle through a three-dimensional space onto the image sensor to form the array image.

9. The system as recited in claim 1, wherein the position determination module is configured to generate position information capable of expressing a multi-axis position of the user-manipulated device with respect to three axes (x-axis, y-axis and z-axis) describing a translational position, a first rotational axis describing a pitch motion, a second rotational axis describing a roll motion, and a third rotational axis describing a yaw motion.

10. The system as recited in claim 1, wherein the position determination module receives from the image sensor a series of sequential frames during a time period, each frame containing a shot of the array image, and wherein the position determination module is configured to track the position of the user-manipulated device in real-time.

11. The system as recited in claim 1, wherein the position determination module is located in the electronic device.

12. The system as recited in claim 1, wherein the position determination module is located in the user-manipulated device.

13. The system as recited in claim 1, wherein the user-manipulated device further comprises a supplemental motion detection sensor.

14. The system as recited in claim 1, wherein the position determination module receives information of the array image from the image sensor wirelessly.

18. The system as recited in claim 1, wherein the first reference element, the second reference element and the third reference element each have a monotonic primary color characteristic.

19. The system as recited in claim 18, wherein each reference element is a light emission device that emits a different monotonic primary color.

15. A system for motion-based controlling of an electronic device, the system comprising:

a user-manipulated device including a video camera associated therewith;

a reference element array including a first reference element, a second reference element and a third reference element, each having a different size from one another and defining vertices of a reference triangle, the reference element array located near a display connected to the electronic device such that the reference triangle defines a plane substantially nonparallel to a front surface of the display for positioning at least one of the reference elements in a location forward of the front surface of the display, the reference element array forming a reference element image through a nonparallel projection of the reference triangle onto the video camera when the video camera is at least partially facing the reference element array, the reference element image forming a pattern which varies in shape as the user-manipulated device changes position in relation to the reference element array, at least one of the first reference element, the second reference element, or the third reference element elevated off a plane with respect to non-elevated reference elements to facilitate discrimination of the at least one elevated reference element from the non-elevated reference elements, to increase visibility of the elevated reference element, and to increase a degree of shape differentiation of the reference element image; and

a position determination module configured to;

receive information of the reference element images from the video camera;

identify a pattern of the reference elements from the information of the reference element images based, at least in part, on the different sizes of the first reference element, the second reference element and the third reference element distinguishing the reference elements from one another; and

generate position information based on the identified pattern of the reference elements, the position information expressing a position of the video camera relative to the reference element array.

16. The system as recited in claim 15, wherein the reference element array is configured to be placed on top of the display connected to the electronic device.

17. A method for motion-based control of an electronic device having a display with a front display surface, the method comprising:

generating on an image sensor a first reference element image of a first reference element, a second reference element image of a second reference element, and a third reference element image of a third reference element, the reference elements forming a reference triangle that defines a plane substantially nonparallel to a front surface of the display, the reference element images having a pattern formed by a nonparallel projection of the reference triangle through a three-dimensional space onto the image sensor, the first reference element, the second reference element and the third reference element each having a distinctive size, the first reference element elevated off a plane with respect to the second reference element and the third reference element to facilitate discrimination of the first reference element from the second reference element and the third reference element, to increase visibility of the first reference element, and to increase a degree of shape differentiation among the first reference element image, the second reference element image, and the third reference element image;

receiving information of reference element images from the image sensor;

differentiating the first reference element image, the second reference element image and the third reference element image based, at least in part, on the distinctive sizes of the first reference element, the second reference element and the third reference element;

identifying the pattern formed by the first reference element image, the second reference element image and the third reference element image;

generating multi-axis position information based on the identified pattern of the reference element images, the multi-axis position information expressing a relative position between the image sensor and the reference element array; and

controlling the electronic device using the multi-axis position information.