Optical flow based tilt sensor

• US 8,213,686 B2
• Filed: 12/06/2010
• Issued: 07/03/2012
• Est. Priority Date: 01/07/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US12/961,050
• Publication Number:
  US 20110074974A1
• Time in Patent Office: 575 Days
• Field of Search: 382/103, 382/107, 3482081-20816, 715719-722, 715/863
• US Class Current: 382/107
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04M 1/72427 for supporting games or graphical animations
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 1/00352 Input means H04N1/00411 takes precedence; input arrangements for computers G06F3/00
• CPC Class Codes: H04N 1/00381 Input by recognition or interpretation of visible user gestures manual input means, e.g. digitisers, writing tablets H04N1/00392; gesture input to computers G06F3/017; recognition algorithms G06V40/20
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0096 Portable devices

Abstract

A method is described for determining a description of motion of a moving mobile camera to determine a user input to an application. The method may involve capturing a series of images from a moving mobile camera and comparing stationary features present in the series of images. Optical flow analysis may be performed on the series of images to determine a description of motion of the moving mobile camera. Based on the determined motion, a user input to an application may be determined and the application may respond to the user input, for example, by updating a user interface of the application.

Claims

1. A method comprising:

accessing, by at least one processor, images captured from an image sensor positioned on a handheld device;

identifying, by the at least one processor, a first stationary feature that is present in the images captured from the image sensor on the handheld device;

identifying, by the at least one processor, a second stationary feature that is present in the images captured from the image sensor on the handheld device, the second stationary feature being different than and spaced apart from the first stationary feature;

determining, by the at least one processor, a description of motion of the handheld device in a depth direction based on changes in distance between the first stationary feature and the second stationary feature within the captured images, the depth direction being a direction of movement toward and away from the first stationary feature and the second stationary feature; and

determining, by the at least one processor, user input for an application based on the determined description of motion of the handheld device in the depth direction.

2. The method of claim 1, wherein determining user input for the application based on the determined description of motion of the handheld device in the depth direction comprises controlling movement of an object rendered in a user interface in a manner corresponding to the determined motion of the handheld device in the depth direction.

3. The method of claim 2, wherein:

determining the description of motion comprises determining a magnitude of motion of the handheld device in the depth direction; and

controlling movement of the object rendered in the user interface in a manner corresponding to the determined motion of the handheld device in the depth direction further comprises controlling the object rendered in the user interface to steer, roll or navigate a distance within the user interface corresponding to the determined magnitude of motion of the handheld device in the depth direction.

4. The method of claim 2, wherein:

determining the description of motion further comprises calculating translation of the handheld device; and

controlling movement of the object rendered in the user interface further comprises controlling the object rendered in the user interface to steer, roll or navigate in the user interface in a manner corresponding to the calculated translation of the handheld device.

5. The method of claim 1, wherein the application is a game.

6. A system comprising:

a handheld device;

an image sensor positioned on the handheld device; and

at least one processor configured to;

access images captured from the image sensor positioned on the handheld device;

identify a first stationary feature that is present in the images captured from the image sensor on the handheld device;

identify a second stationary feature that is present in the images captured from the image sensor on the handheld device, the second stationary feature being different than and spaced apart from the first stationary feature;

determine a description of motion of the handheld device in a depth direction based on changes in distance between the first stationary feature and the second stationary feature within the captured images, the depth direction being a direction of movement toward and away from the first stationary feature and the second stationary feature; and

determine user input for an application based on the determined description of motion of the handheld device in the depth direction.

7. The system of claim 6, wherein the at least one processor is configured to determine user input for the application based on the determined description of motion of the handheld device in the depth direction by controlling movement of an object rendered in a user interface in a manner corresponding to the determined motion of the handheld device in the depth direction.

8. The system of claim 7, wherein the at least one processor is configured to:

determine the description of motion by determining a magnitude of motion of the handheld device in the depth direction; and

control movement of the object rendered in the user interface in a manner corresponding to the determined motion of the handheld device in the depth direction by controlling the object rendered in the user interface to steer, roll or navigate a distance within the user interface corresponding to the determined magnitude of motion of the handheld device in the depth direction.

9. The system of claim 7, wherein the at least one processor is configured to:

determine the description of motion by calculating translation of the handheld device; and

control movement of the object rendered in the user interface by controlling the object rendered in the user interface to steer, roll or navigate in the user interface in a manner corresponding to the calculated translation of the handheld device.

10. The system of claim 6, wherein the application is a game.

11. A system comprising:

a handheld device;

an image sensor positioned on the handheld device; and

at least one processor configured to;

access images captured from the image sensor positioned on the handheld device;

identify a first stationary feature that is present in the images captured from the image sensor on the handheld device;

identify a second stationary feature that is present in the images captured from the image sensor on the handheld device, the second stationary feature being different than and spaced apart from the first stationary feature;

determine positions of the first stationary feature and the second stationary feature within the images captured from the image sensor on the handheld device;

provide a user interface that includes a visual display for an application, the user interface being separate from the handheld device on which the image sensor is positioned;

determine user input for the application based on the determined positions of the first stationary feature and the second stationary feature within the images captured from the image sensor on the handheld device; and

control the user interface that is separate from the handheld device on which the image sensor is positioned based on the determined user input for the application.

12. The system of claim 11, wherein the handheld device is configured to use wireless communications to enable control of the user interface that is separate from the handheld device on which the image sensor is positioned.

13. The system of claim 11, wherein the at least one processor is configured to:

determine positions of the first stationary feature and the second stationary feature within the images captured from the image sensor on the handheld device by determining relative positions between the first stationary feature and the second stationary feature within the images captured from the image sensor on the handheld device; and

determine user input for the application by determining user input for the application based on the relative positions between the first stationary feature and the second stationary feature within the images captured from the image sensor on the handheld device.

14. The system of claim 11, wherein the at least one processor is configured to:

determine positions of the first stationary feature and the second stationary feature within the images captured from the image sensor on the handheld device by determining changes in position of the first stationary feature within the images captured from the image sensor on the handheld device and changes in position of the second stationary feature within the images captured from the image sensor on the handheld device; and

determine user input for the application by determining user input for the application based on the changes in position of the first stationary feature within the images captured from the image sensor on the handheld device and the changes in position of the second stationary feature within the images captured from the image sensor on the handheld device.

15. The system of claim 11, wherein the at least one processor is configured to determine user input for the application by determining a description of motion of the handheld device in a depth direction based on changes in distance between the first stationary feature and the second stationary feature within the captured images and determining user input for the application based on the determined description of motion of the handheld device in the depth direction.

16. The system of claim 11, wherein the at least one processor is configured to control the user interface that is separate from the handheld device on which the image sensor is positioned by controlling movement of an object rendered in the user interface based on the determined user input for the application.

17. The system of claim 16, wherein the at least one processor is configured to:

determine user input for the application by determining a magnitude of motion of the handheld device based on the determined positions of the first stationary feature and the second stationary feature within the images captured from the image sensor on the handheld device; and

control movement of the object rendered in the user interface based on the determined user input for the application by controlling the object rendered in the user interface to move a distance within the user interface corresponding to the determined magnitude of motion of the handheld device.

18. The system of claim 16, wherein the at least one processor is configured to:

determine user input for the application by calculating translation of the handheld device based on the determined positions of the first stationary feature and the second stationary feature within the images captured from the image sensor on the handheld device; and

control movement of the object rendered in the user interface based on the determined user input for the application by controlling the object rendered in the user interface to move in the user interface in a manner corresponding to the calculated translation of the handheld device.

19. The system of claim 11, wherein the at least one processor is configured to control the user interface that is separate from the handheld device on which the image sensor is positioned by controlling a menu navigation function based on a menu object rendered in the user interface of the application.

20. The system of claim 11, wherein the application is a game.

21. An apparatus comprising:

means for accessing images captured from an image sensor positioned on a handheld device;

means for identifying a first stationary feature that is present in the images captured from the image sensor on the handheld device;

means for identifying a second stationary feature that is present in the images captured from the image sensor on the handheld device, the second stationary feature being different than and spaced apart from the first stationary feature;

means for determining a description of motion of the handheld device in a depth direction based on changes in distance between the first stationary feature and the second stationary feature within the captured images, the depth direction being a direction of movement toward and away from the first stationary feature and the second stationary feature; and

means for determining user input for an application based on the determined description of motion of the handheld device in the depth direction.

22. At least one non-transitory computer-readable medium having instructions stored thereon that, when executed, cause at least one processor to:

access images captured from an image sensor positioned on a handheld device;

identify a first stationary feature that is present in the images captured from the image sensor on the handheld device;

identify a second stationary feature that is present in the images captured from the image sensor on the handheld device, the second stationary feature being different than and spaced apart from the first stationary feature;

determine a description of motion of the handheld device in a depth direction based on changes in distance between the first stationary feature and the second stationary feature within the captured images, the depth direction being a direction of movement toward and away from the first stationary feature and the second stationary feature; and

determine user input for an application based on the determined description of motion of the handheld device in the depth direction.