ENHANCED HANDHELD SCREEN-SENSING POINTER

• US 20100259474A1
• Filed: 04/08/2010
• Published: 10/14/2010
• Est. Priority Date: 04/08/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: GestureTek Incorporated (Winning Brands Corp.)
• Inventors: Hildreth, Evan
• Granted Patent: US 9,189,082 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus

Abstract

Enhanced handheld screen-sensing pointing, in which a handheld device captures a camera image of one or more fiducials rendered by a display device, and a position or an angle of the one or more fiducials in the captured camera image is determined. A position on the display device that the handheld device is aimed towards is determined based at least on the determined position or angle of the one or more fiducials in the camera image, and an application is controlled based on the determined position on the display device.

Claims

1. A computer-implemented method comprising:

capturing, by a handheld device, a camera image of one or more fiducials rendered by a display device;

determining a position or an angle of the one or more fiducials in the captured camera image;

determining a position on the display device that the handheld device is aimed towards based at least on the determined position or angle of the one or more fiducials in the camera image; and

controlling an application based on the determined position on the display device.

2. The method of claim 1, further comprising controlling the display device to render the one or more fiducials within a user interface that represents output of the application being controlled.

3. The method of claim 1, wherein the one or more fiducials comprise first and second fiducials.

4. The method of claim 1, wherein the one or more fiducials comprise first through fourth fiducials.

5. The method of claim 1, further comprising:

determining a type of the display device from among multiple, different display device types;

selecting, from among multiple, different filtering techniques, a filtering technique that is suitable for the determined type of the display device; and

applying the selected filtering technique to the captured camera image.

6. The method of claim 1, further comprising:

analyzing a user interface that represents output of the application being controlled for displayed elements that may be used as the one or more fiducials;

selecting the one or more fiducials from the displayed elements within the user interface that represents output of the application being controlled; and

transmitting, to the handheld device, a description of shape and position of the one or more fiducials selected from the displayed elements within the user interface that represents output of the application being controlled.

7. The method of claim 1, wherein the one or more fiducials further comprise one or more corners of a screen of the display device.

8. The method of claim 7, further comprising detecting the one or more corners of the screen based on a difference in brightness or hue.

9. The method of claim 1, wherein the one or more fiducials comprise a user interface control.

10. The method of claim 1, wherein the one or more fiducials comprise a desktop icon.

11. The method of claim 1, wherein the one or more fiducials comprise a video game reference part.

12. The method of claim 1, further comprising accessing, from a memory of the handheld device, information identifying a shape or position of the one or more fiducials.

13. The method of claim 1, further comprising accessing, from a memory of the handheld device, information identifying a timing of a display of the one or more fiducials rendered by the display device.

14. The method of claim 1, further comprising:

determining a type of the display device from among multiple, different display device types;

selecting, from among multiple, different flashing techniques, a flashing technique that is suitable for the determined type of the display device; and

controlling the display device to flash the one or more fiducials on the display device using the selected flashing technique.

15. The method of claim 1, further comprising determining an orientation of the handheld device based at least on the determined position or angle of the one or more fiducials in the camera image, wherein the application is controlled further based on the determined orientation.

16. The method of claim 15, further comprising determining a distance or position of the handheld device, wherein the distance, position, or orientation of the handheld device are determined with respect to the display device.

17. The method of claim 1, further comprising:

capturing, by a second handheld device, a second camera image of the one or more fiducials rendered by the display device;

determining a position or an angle of the one or more fiducials in the second captured camera image; and

determining a second position on the display device that the second handheld device is aimed towards based at least on the determined position or angle of the one or more fiducials in the second camera image,

wherein the application is controlled based on the determined position and the determined second position.

18. The method of claim 17, further comprising:

determining a first distance, position, or orientation of the handheld device with respect to the display device;

determining a second distance, position, or orientation of the second handheld device with respect to the display device; and

determining an identity of a first user and an identity of a second user based on the determined first distance, position, or orientation of the handheld device relative to the determined second distance, position, or orientation of the second handheld device.

19. The method of claim 1, wherein the application comprises a video game application or a media center application.

20. The method of claim 1, wherein controlling the application further comprises changing a view of an avatar in a virtual universe based on determining that an orientation of the handheld device has changed.

21. The method of claim 1, further comprising:

capturing, at the handheld device, a second camera image of the one or more fiducials rendered by the display device;

determining a second position or a second angle of the one or more fiducials in the second captured camera image;

determining a second position on the display device that the handheld device is aimed towards based at least on the determined second position or second angle of the one or more fiducials in the second camera image; and

determining a motion of the handheld device based on comparing the position with the second position,

wherein the application is controlled based on the determined motion.

22. A system comprising:

one or more computers; and

a computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

capturing, by a handheld device, a camera image of one or more fiducials rendered by a display device,

determining a position or an angle of the one or more fiducials in the captured camera image,

determining a position on the display device that the handheld device is aimed towards based at least on the determined position or angle of the one or more fiducials in the camera image, and

controlling an application based on the determined position on the display device.

23. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

capturing, by a handheld device, a camera image of one or more fiducials rendered by a display device;

determining a position or an angle of the one or more fiducials in the captured camera image;

determining a position on the display device that the handheld device is aimed towards based at least on the determined position or angle of the one or more fiducials in the camera image; and

controlling an application based on the determined position on the display device.