Camera and acceleration based interface for presentations

• US 7,852,315 B2
• Filed: 04/07/2006
• Issued: 12/14/2010
• Est. Priority Date: 04/07/2006
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ofek, Eyal, Hsu, Feng-Hsiung
• Primary Examiner(s): Nguyen; Chanh
• Assistant Examiner(s): Yang; Kwang-Su
• Application Number:
  US11/278,998
• Publication Number:
  US 20070236451A1
• Time in Patent Office: 1,712 Days
• Field of Search: 345156-168, 345/8
• US Class Current: 345/157
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

Systems and methods for a camera and acceleration based interface for presentations are described. In one aspect, a handheld device is utilized capture optical information. The optical information is then used to identify the location of a cursor that is presented on a display surface. While a user moves the handheld device, inertial information corresponding to the movement is accumulated on the handheld device. The optical and accumulated inertial information is used to modify position of the cursor with respect to the display surface.

Claims

1. A method implemented at least in part by a computing device, the method comprising:

without knowledge of a position of any specific point of a display surface;

capturing, by a mobile handheld device, optical information of a cursor presented on a presentation displayed on the display surface by a remote computing device, wherein the optical information comprises two or more images;

evaluating the optical information to minimize differences between the two or more images to estimate a projective transformation between the two or more images to identify a current position of the cursor;

accumulating, by the mobile handheld device, inertial information comprising accumulated motion of the mobile handheld device;

the remote computing device utilizing the optical information, the inertial information, or a combination thereof to modify a position of the cursor from the current position to a different position with respect to the presentation, the utilizing further comprising;

determining whether the optical information is sufficient to reliably determine a new position for the cursor;

responsive to determining that the optical information is sufficient, evaluating only the optical information to determine the new position;

responsive to the determining that the optical information is not sufficient, evaluating the inertial information to estimate the new position by accumulating angular accelerations from a previous image of the two or more images to estimate a rotation angle to a current image of the two or more images; and

determining the new position by a product of the rotation angle and a resolution of the current image divided by an angle of the camera.

2. The method of claim 1, further comprising obtaining an initial position of the cursor, wherein obtaining the initial position of the cursor comprises:

presenting the cursor such that the cursor can be detected from any image in the optical information that captures at least a portion of an area associated with the display surface;

reducing a size of the cursor from an initially presented size to a default size; and

tracking the position of the cursor until the default size of the cursor is achieved to obtain the initial position of the cursor.

3. The method of claim 1, further comprising capturing, by one or more optical sensors, the optical information;

wherein the two or more images comprise images of the cursor presented on the presentation; and

wherein a difference in the two or more images is indicative of a change in position of the mobile handheld device.

4. The method of claim 1, further comprising accumulating, by an accelerometer, the inertial information;

wherein the inertial information contains data indicative of a change in position of the mobile handheld device.

5. The method of claim 1, further comprising capturing the optical information and accumulating the inertial information responsive to an action by the user;

wherein one or more of the optical information or the inertial information is indicative of a change in position of the mobile handheld device, wherein the change in position of the mobile handheld device is used to modify the position of the cursor.

6. The method of claim 1, wherein the operations of determining, evaluating, and considering are performed by one of the computing device or the mobile handheld device.

7. The method of claim 1, further comprising calculating the new position using a projective homography indicating a transformation between optical sensors associated with the mobile handheld device and a coordinate system associated with the presentation.

8. The method of 7, further comprising calculating the transformation based on user motions that pan the mobile handheld device with respect to the presentation.

9. A computer-readable storage medium comprising computer-program instructions that, when executed by a processor, perform acts comprising:

without knowledge of a position of any specific point of a display surface;

presenting a presentation on the display surface;

receiving by a camera optical information from a mobile handheld device of a cursor presented on the presentation displayed on the display surface by a remote computing device, wherein the optical information comprises two or more images;

evaluating the optical information to minimize differences between the two or more images to estimate a projective transformation between the two or more images to identify a current position of the cursor;

receiving accumulated inertial information comprising accumulated motion of the handheld device;

the remote computing device utilizing at least part of the optical information and the inertial information to modify a position of the cursor from the current position to a different position with respect to the presentation, the utilizing further comprising;

determining whether the optical information is sufficient to reliably determine a new position for the cursor;

responsive to determining that the optical information is sufficient, evaluating only the optical information to determine the new position;

responsive to the determining that the optical information is not sufficient, evaluating the inertial information to estimate the new position by accumulating angular accelerations from a previous image of the two or more images to estimate a rotation angle to a current image of the two or more images; and

determining the new position by a product of the rotation angle and a resolution of the current image divided by an angle of the camera,

wherein the mobile handheld device is used as one or more of a virtual laser pointer and an element for a user to control the presentation.

10. The computer-readable storage medium of claim 9, wherein a camera is used to capture the optical information including the two or more images;

and

wherein the two or more images comprise images of the cursor presented on the presentation and are indicative of the change in position of the mobile handheld device.

11. The computer-readable storage medium of claim 9, wherein an accelerometer is used to accumulate the inertial information;

and

wherein the inertial information contains data indicative of the change in position of the mobile handheld device.

12. The computer-readable storage medium of claim 9, wherein the presentation is a slideshow presentation.

13. The computer-readable medium of claim 9, wherein the display surface is a projection surface.

14. The computer-readable storage medium of claim 9, wherein the acts further comprise:

determining whether the optical information is sufficient to reliably determine a new position for the cursor;

responsive to determining that the optical information is sufficient, evaluating only the optical information to determine the new position; and

responsive to determining that the optical information is not sufficient, considering a combination of the inertial information and the optical information to determine the new position.

15. The computer-readable storage medium of claim 14, wherein the acts further comprise calculating the new position using a projective homography indicating a transformation between optical sensors associated with the handheld device and a coordinate system associated with the presentation.

16. A system comprising:

a remote computing device; and

a handheld computing device, wherein the handheld computing device comprises;

one or more optical sensors; and

one or more inertial sensors;

wherein the handheld computing device is configured to acquire, without knowledge of a position of any specific point of a display surface, optical and inertial information using the one or more optical sensors and the one or more inertial sensors respectively, the optical information, comprising two or more images, for use by the remote computing device to minimize differences between the two or more images to estimate a projective transformation between the two or more images to determine an initial position of a cursor on a displayed presentation;

wherein the remote computing device is configured, when the optical information is sufficient to track the cursor with respect to the displayed presentation, to use the optical information to control a position of the cursor from one or more of the initial position and subsequent cursor positions with respect to the displayed presentation; and

wherein the remote computing device is configured, when the optical information is not sufficient to track the cursor with respect to the displayed presentation, to use a combination of the optical information and the inertial information to control the position of the cursor from one or more of the initial position and subsequent cursor positions with respect to the displayed presentation and to evaluate the inertial information to estimate the subsequent cursor positions by accumulating angular accelerations from a previous image of the two or more images to estimate a rotation angle to a current image of the two or more images, the subsequent cursor positions being determined by a product of the rotation angle and a resolution of the current image divided by an angle of the camera.

17. The system of claim 16, wherein the handheld computing device comprises one of small form factor mobile computing devices comprising at least a mobile smart phone and a personal digital assistant.

18. The system of claim 16, wherein the displayed presentation is a slideshow presentation.

19. The system of claim 16, wherein the remote computing device is further configured to control the position of the cursor using a protective homography indicating a transformation between the optical information received from the handheld device and a coordinate system associated with the presentation.