Apparatus for controlling contents of a computer-generated image using three dimensional measurements

• US 9,785,255 B2
• Filed: 04/13/2016
• Issued: 10/10/2017
• Est. Priority Date: 05/28/2004
• Status: Active Grant

• Current Assignee: UltimatePointer, LLC
• Original Assignee: UltimatePointer, LLC
• Inventors: Banning, Erik Jan
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US15/098,099
• Publication Number:
  US 20160306441A1
• Time in Patent Office: 545 Days
• CPC Class Codes: G03B 21/00 Projectors or projection-type viewers; Accessories therefor devices for changing pictures G03B23/00
• CPC Class Codes: G03B 21/006 using LCD's
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0338 with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03549 Trackballs G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

A method for controlling movement of a computer display cursor based on a point-of-aim of a pointing device within an interaction region includes projecting an image of a computer display to create the interaction region. At least one calibration point having a predetermined relation to said interaction region is established. A pointing line is directed to substantially pass through the calibration point while measuring a position of and an orientation of the pointing device. The pointing line has a predetermined relationship to said pointing device. Movement of the cursor is controlled within the interaction region using measurements of the position of and the orientation of the pointing device.

Claims

1. A non-transitory computer-readable medium or media storing computer-executable instructions for causing a computer to perform a method for controlling contents of a computer-generated image, the method comprising the steps of:

receiving first data from a first sensing device that comprises a first accelerometer and a first gyro, said first data being dependent on a first output of said first accelerometer and a second output of said first gyro and sensitive to three independent positional and three independent orientational coordinates of a first device that is configured to be worn as a headset and contains said first accelerometer and said first gyro;

receiving second data from a second device that is different from the first device and is configured to be wielded by a user in mid-air and contains a user input device configured to allow the user to manipulate two positional coordinates of a feature on the image, said second data being dependent on a third output of said user input device; and

controlling the contents of the computer-generated image based at least in part on the first data and the second data.

2. The non-transitory computer-readable medium or media according to claim 1, wherein the method furthermore comprises the steps of

receiving third data from a second sensing device, the third data being sensitive to two independent positional coordinates of the second device;

receiving fourth data from a base station that is different from the first device and from the second device and contains at least one element of the first sensing device and at least one element of the second sensing device, the fourth data being sensitive to the three independent positional coordinates of the first device and the two independent positional coordinates of the second device;

controlling the contents of the computer-generated image based on the first data and the second data and at least one of the third data and the fourth data;

wherein at least one element of the second sensing device is contained in the second device;

wherein the two independent positional coordinates of the second device and the three independent positional coordinates of the first device are relative to the base station, and

wherein the second device is configured to be handheld.

3. The non-transitory computer-readable medium or media according to claim 1, where said controlling contents of a computer-generated image is controlling position of a feature on said image.

4. A non-transitory computer-readable medium or media storing computer-executable instructions for causing a computer to perform a method for controlling contents of a computer-generated image, the method comprising the steps of:

communicating with and receiving first data from a first communication device that is coupled to a first sensing device that comprises a first accelerometer and a first gyro, the first data being dependent on a first output generated by the first sensing device, the first output being sensitive to three independent orientational coordinates and three independent positional coordinates of a first device that contains said first accelerometer and said first gyro and is configured to be worn as a headset, said first output being dependent on a second output of said first accelerometer and on a third output of said first gyro; and

using said first data to control the contents of the computer-generated image based on the orientational and positional coordinates of the first device.

5. The non-transitory computer-readable medium or media according to claim 4, further comprising the steps of:

communicating with and receiving second data from a second communication device that is coupled to a second sensing device that comprises a second accelerometer and a second gyro, the second data being dependent on a fourth output generated by the second sensing device, the fourth output being sensitive to three independent orientational coordinates and three independent positional coordinates of a second device that is different from the first device and contains said second accelerometer and said second gyro and is configured to be wielded in mid-air, said fourth output being dependent on a fifth output of said second accelerometer and on a sixth output of said second gyro; and

using said second data to control the contents of the computer-generated image based on the orientational and positional coordinates of the second device.

6. The non-transitory computer-readable medium or media according to claim 5, further comprising the steps of:

communicating with and receiving third data from a third communication device that is coupled to the first sensing device and to the second sensing device and is contained in a first base station that is different from the first device and from the second device and that furthermore contains at least one element of the first sensing device and at least one element of the second sensing device; and

using the third data to control the contents of the computer-generated image based on the orientational and positional coordinates of the first device and of the second device,

wherein the third data depends on the first output and on the fourth output;

wherein the three independent positional coordinates of the first device are relative to the base station,

wherein the three independent positional coordinates of the second device are relative to the base station,

wherein the base station comprises at least one of a first power source and a connection to a second power source for providing power to the base station,

wherein the first sensing device further comprises a first lightsource for use in generating said first output, and

wherein the second sensing device further comprises a second lightsource for use in generating said fourth output.

7. The non-transitory computer-readable medium or media according to claim 5, further comprising the steps of:

communicating with and receiving third data from a third communication device that is coupled to the first sensing device and to the second sensing device and is contained in a first base station that is different from the first device and from the second device and that furthermore contains at least one element of the first sensing device and at least one element of the second sensing device;

communicating with and receiving fourth data from a fourth communication device that is coupled to the first sensing device and to the second sensing device and is contained in a second base station that is different from the first device and from the second device and from the first base station and that furthermore contains at least one element of the first sensing device and at least one element of the second sensing device; and

using the third data and the fourth data to control the contents of the computer-generated image based on the orientational and positional coordinates of the first device and of the second device,

wherein the first base station comprises at least one of a first power source and a connection to a second power source for providing power to the first base station,

wherein the second base station comprises at least one of a third power source and a connection to a fourth power source for providing power to the second base station, and

wherein the second device is configured to be handheld.

8. The non-transitory computer-readable medium or media according to claim 7, wherein the first sensing device comprises a first infrared lightsource,

wherein the second sensing device comprises a second infrared lightsource,

wherein the first data depends on output from the first infrared lightsource, and

wherein the second data depends on output from the second infrared lightsource.

9. The non-transitory computer-readable medium or media according to claim 4, wherein the first sensing device comprises an image capturing device and a lightsource, said image capturing device for generating fourth output that is dependent on position of the first device and on light emanating from said lightsource, and

wherein the first data is dependent on the fourth output.

10. The non-transitory computer-readable medium or media according to claim 9, wherein the image capturing device is outside of the first device,

wherein the lightsource is an LED,

wherein a pointing line is provided with a fixed relation to the first device;

wherein the image is rectangular,

wherein a rectangular interaction structure is provided that has the same center and is twice as large as the image, and

wherein the controlling contents of the image is controlling position of a feature on the image and operates in such a way that the feature appears at substantially the upper right corner of the image if the pointing line is directed at substantially the upper right corner of the interaction structure, and at substantially the lower left corner of the image if the pointing line is directed at substantially the lower left corner of the interaction structure.

11. The non-transitory computer-readable medium or media according to claim 4, where said controlling contents of a computer-generated image is controlling position of a feature on said image.

12. An apparatus for controlling contents of an image generated by a computer, the apparatus comprising:

a first device configured to be worn as a headset, said first device comprising a first accelerometer and a first gyro and at least one component of a first sensing device, the first accelerometer and the first gyro being part of a second sensing device, the first sensing device for generating first data for controlling the contents of the image, the second sensing device for generating second data that is dependent on the output of the first accelerometer and the first gyro and is for controlling the contents of the image, the first data being sensitive to three independent positional coordinates of the first device relative to a first base station that comprises at least one component of the first sensing device, the second data being sensitive to three independent orientational coordinates of the first device.

13. The apparatus according to claim 12, further comprising a non-transitory computer-readable medium or media storing computer-executable instructions for causing the computer to perform a method for the controlling of the contents of the image generated by the computer, wherein the controlling of the contents of the image is controlling the position of a feature on the image,

wherein the image is rectangular,

wherein there is provided a pointing line with a fixed relation to the first device,

wherein there is provided a rectangular interaction structure that has the same center and is twice as large as the image, and

wherein the controlling position of the feature operates such that the feature appears substantially in the lower left corner of the image if the pointing line is substantially directed at the lower left corner of the interaction structure, and substantially in the upper right corner of the image if the pointing line is substantially directed at the upper right corner of the interaction structure.

14. The apparatus according to claim 12, wherein the first sensing device comprises a lightsource and an image capturing device for detecting light emanating from said lightsource,

wherein the first data is dependent on output of the image capturing device, and

wherein the lightsource is contained in the first device.

15. The apparatus according to claim 12, further comprising

a second device that is different from the first device and from the first base station and is configured to be wielded in mid-air, said second device comprising a second accelerometer and a second gyro and at least one component of a third sensing device, the second accelerometer and the second gyro being part of a fourth sensing device, the third sensing device for generating third data for controlling the contents of the image, the fourth sensing device for generating fourth data that is dependent on the output of the second accelerometer and the second gyro and is for controlling the contents of the image, the third data being sensitive to three independent positional coordinates of the second device relative to the first base station, the fourth data being sensitive to three independent orientational coordinates of the second device,

wherein the first base station further comprises at least one component of the third sensing device.

16. The apparatus according to claim 15, furthermore comprising a second base station that is different from the first base station, from the first device and from the second device and comprises at least one component of the first sensing device and at least one component of the third sensing device,

wherein the first sensing device comprises a first lightsource,

wherein the third sensing device comprises a second lightsource, and

wherein the first and the third data are dependent on the output of the first lightsource and the second lightsource.

17. The apparatus according to claim 12, wherein the apparatus further comprises:

a second device that is different from the first device and configured to be handheld and wielded in mid-air by a user, the second device comprising a user input device for generating third data for controlling the contents of the image generated by the computer, said user input device configured to allow the user to manipulate two positional coordinates of a feature on the image.

18. The apparatus according to claim 17, wherein the second device further comprises at least one component of a third sensing device for generating fourth data for controlling the contents of the image, the third sensing device comprising an image capturing device and a lightsource, the fourth data being sensitive to two independent positional coordinates of the second device relative to the first base station, said first base station comprising at least one component of the third sensing device, and

wherein the fourth data is dependent on the output of the image capturing device.

19. The apparatus according to claim 18, further comprising said first base station,

wherein the user input device is at least one of a thumbstick and a touchpad, and

wherein the third sensing device is sensitive to three independent positional coordinates of the second device relative to the first base station.

20. The apparatus according to claim 12, where the controlling of the contents of the image is controlling position of a feature on said image.