ALGORITHMS, SOFTWARE AND AN INTERACTION SYSTEM THAT SUPPORT THE OPERATION OF AN ON THE FLY MOUSE

• US 20150022449A1
• Filed: 11/26/2013
• Published: 01/22/2015
• Est. Priority Date: 07/16/2013
• Status: Active Grant

• Current Assignee: Amchaelvisual Technology LLC
• Original Assignee: Amchael Visual Technology Corporation
• Inventors: Cheng, Kai Michael, Tsai, Yushiuan, Lu, Hsien-Chang
• Granted Patent: US 9,507,437 B2
• Time in Patent Office: Days
• US Class Current: 345/163
• CPC Class Codes: G06F 3/03 Arrangements for converting the position or the displacement of a member into a coded form
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence

Abstract

Methods and systems for surface-free pointing and/or command input include a computing device operably linked to an imaging device. The imaging device can be any suitable video recording device including a conventional webcam. At least one pointing/input device is provided including a visible point light source emitting light in a wavelength defining a predetermined color. The at least one pointing/input device may be a conventional optoelectronic computer mouse. The imaging device captures one or more sequential image frames each including a view of a scene including the visible point light source. One or more software programs calculate a position and/or a motion of the at least one point light source in the captured image frames by identifying colored areas corresponding to the point light source and mapping those colored areas to a position and/or motion of a visual marker such as a cursor in a graphical user interface.

Claims

1. A computing system for tracking a pointing/input device, comprising:

a computing device having at least one processor, at least one memory, and at least one graphical user interface;

at least one pointing/input device including at least one visible point light source;

an imaging device operably linked to the computing device and configured for capturing a plurality of sequential image frames each including a view of the at least one point light source held and/or moved in a three-dimensional space and within a field of view of the imaging device; and

at least one computer program product operable on the computing device processor;

wherein the at least one computer program product includes executable instructions for calculating from the plurality of sequential image frames at least a position and/or a motion of the at least one point light source, and for rendering on the graphical user interface a visual marker corresponding to the position and/or the motion of the at least one pointing/input device.

2. The system of claim 1, wherein the at least one point light source emits a light beam having a wavelength defining a predetermined color.

3. The system of claim 2, wherein the at least one computer program product includes executable instructions for mapping and displaying in the graphical user interface the position and/or the motion of the at least one pointing/input device within the field of view of the imaging device, comprising:

determining a position of the at least one point light source in each of said plurality of sequential image frames;

converting the determined position of the at least one point light source into a position of the visual marker in the graphical user interface; and

displaying the visual marker in the graphical user interface.

4. The system of claim 3, wherein the at least one computer program product includes executable instructions for determining a position of the at least one point light source by:

analyzing each of said plurality of sequential image frames to identify one or more regions having an increased color intensity;

applying a first filtering action to one or more regions of said plurality of sequential image frames to remove areas having a decreased color intensity;

applying a second filtering action to one or more areas of said regions of said plurality of sequential image frames regions having the increased color intensity to remove areas having a color that is other than said predetermined color and to isolate areas having said predetermined color; and

identifying a position of said at least one point light source according to said areas having said predetermined color.

5. The system of claim 4, wherein the at least one computer program product includes executable instructions for mapping said position and/or motion of the at least one point light source to a position and/or motion of the visual marker on the graphical user interface by:

determining a first global position of the at least one point light source in a first of the plurality of sequential image frames;

determining a next global position of the at least one point light source in a next of the plurality of sequential image frames;

calculating a difference between the first global position and the next global position of the at least one point light source;

scaling the calculated difference in a non-linear fashion;

calculating one or more intermediate positions between the first global position and the next global position of the at least one point light source; and

moving the visual marker displayed in the graphical user interface along a determined displacement path passing through the first scaled global position, the intermediate global positions, and the next scaled global position.

6. In a computing system environment, a method for tracking a pointing/input device, comprising:

holding and/or moving at least one pointing/input device including at least one visible point light source in a three-dimensional space disposed within a field of view of an imaging device operably connected to a computing device having at least one processor, at least one memory, and at least one graphical user interface;

by the imaging device, capturing a plurality of sequential image frames each including a view of a position of the at least one point light source within the imaging device field of view;

from the captured plurality of sequential image frames, calculating at least a position and a motion of the at least one point light source; and

rendering a visual marker on the graphical user interface corresponding to the position and/or the motion of the at least one point light source.

7. The method of claim 6, further including mapping the motion of the at least one point light source to a predetermined pointing and/or input command.

8. The method of claim 6, including providing at least one point light source emitting a light beam having a fixed wavelength defining a predetermined color.

9. The method of claim 8, including the steps of:

determining a position of the at least one point light source in each of said plurality of sequential image frames;

converting the determined position of the at least one point light source into a position of the visual marker in the graphical user interface; and

displaying the visual marker in the graphical user interface.

10. The method of claim 9, including determining a position of the at least one point light source by the steps of:

analyzing each of said plurality of sequential image frames to identify one or more regions having an increased color intensity;

applying a first filtering action to one or more regions of said plurality of sequential image frames to remove areas having a decreased color intensity;

applying a second filtering action to one or more areas of said regions of said plurality of sequential image frames regions having the increased color intensity to remove areas having a color that is other than said predetermined color and to isolate areas having said predetermined color; and

identifying a position of said at least one point light source according to said areas having said predetermined color.

11. The method of claim 10, including mapping said position of the at least one point light source to a position of the at least one pointing/input device on the graphical user interface by the steps of:

determining a first global position of the at least one point light source in a first of the plurality of sequential image frames;

determining a next global position of the at least one point light source in a next of the plurality of sequential image frames;

calculating a difference between the first global position and the next global position of the at least one point light source;

scaling the calculated difference in a non-linear fashion;

calculating one or more intermediate positions between the first global position and the next global position of the at least one point light source; and

moving the visual marker displayed in the graphical user interface along a determined displacement path passing through the first scaled global position, the intermediate global positions, and the next scaled global position.

12. A non-transitory computer program product for loading on a computing device to render a position and/or motion of a visual marker on a graphical user interface, the computer program product having executable instructions for:

analyzing digital data converted by an imaging device from a plurality of captured sequential image frames providing a view of a pointing/input device including a visible point light source having a wavelength defining a predetermined color, the pointing/input device being held and/or moved in three-dimensional space within a field of view of the imaging device;

determining a position of the at least one point light source in each of said plurality of sequential image frames;

converting the determined position of the at least one point light source into a position of the visual marker in the graphical user interface; and

displaying the visual marker in the graphical user interface.

13. The computer program product of claim 12, further including executable instructions that map and display in the graphical user interface the position and/or the motion of the at least one pointing/input device by:

analyzing each of said plurality of sequential image frames to identify one or more regions having an increased color intensity;

applying a first filtering action to one or more regions of said plurality of sequential image frames to remove areas having a decreased color intensity;

applying a second filtering action to one or more areas of said regions of said plurality of sequential image frames regions having the increased color intensity to remove areas having a color that is other than said predetermined color and to isolate areas having said predetermined color; and

identifying a position of said at least one point light source according to said areas having said predetermined color.

14. The computer program product of claim 13, further including executable instructions that map said position and/or motion of the at least one point light source to a position and/or motion of the visual marker on the graphical user interface by:

determining a first global position of the at least one point light source in a first of the plurality of sequential image frames;

determining a next global position of the at least one point light source in a next of the plurality of sequential image frames;

calculating a difference between the first global position and the next global position of the at least one point light source;

scaling the calculated difference in a non-linear fashion;

calculating one or more intermediate positions between the first global position and the next global position of the at least one point light source; and

moving the visual marker displayed in the graphical user interface along a determined displacement path passing through the first scaled global position, the intermediate global positions, and the next scaled global position.

15. A non-transitory computer program product for loading on a computing device to render a position and/or motion of a visual marker on a graphical user interface, comprising:

a first component for analyzing digital data converted from a plurality of captured sequential image frames each including view of a visible point light source having a wavelength defining a predetermined color and for determining a position of the at least one point light source in each of said plurality of sequential image frames;

a second component for converting the determined position of the at least one point light source into a position of the visual marker in the graphical user interface; and

;

a third component for displaying the visual marker in the graphical user interface.

16. The computer program product of claim 15, wherein the first component is configured to analyze digital data of each of said plurality of sequential image frames and to identify one or more regions having an increased color intensity corresponding to the visible point light source predetermined color.

17. The computer program product of claim 16, wherein the first component is further configured to apply at least one filtering action to digital data of one or more regions of said plurality of sequential image frames to remove areas having a decreased color intensity, to remove areas having a color that is other than said predetermined color, and to isolate areas having said predetermined color.

18. The computer program product of claim 16, wherein the second component is configured to determine a global position of the at least one point light source in digital data from one of the plurality of sequential image frames and to calculate a difference between the global position of the at least one point light source in the one of the sequential image frames and a succeeding global position of the at least one point light source in a succeeding image frame of the plurality of sequential image frames.

19. The computer program product of claim 18, wherein the second component is further configured to scale the calculated difference in a non-linear fashion, to calculate one or more intermediate positions between the global position and the succeeding global position of the at least one point light source.

20. The computer program product of claim 19, wherein the third component is configured to render the visual marker in the graphical user interface and to move the rendered visual marker along a calculated displacement path passing through the scaled global position, the intermediate global positions, and the succeeding scaled global position.