Optical flow based tilt sensor

• US 8,983,139 B2
• Filed: 06/25/2012
• Issued: 03/17/2015
• Est. Priority Date: 01/07/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US13/532,627
• Publication Number:
  US 20120268622A1
• Time in Patent Office: 995 Days
• Field of Search: 382/103, 382/107, 3482081-20816, 715719-722, 715/863
• US Class Current: 382/107
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04M 1/72427 for supporting games or graphical animations
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 1/00352 Input means H04N1/00411 takes precedence; input arrangements for computers G06F3/00
• CPC Class Codes: H04N 1/00381 Input by recognition or interpretation of visible user gestures manual input means, e.g. digitisers, writing tablets H04N1/00392; gesture input to computers G06F3/017; recognition algorithms G06V40/20
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0096 Portable devices

Abstract

A method is described for determining a description of motion of a moving mobile camera to determine a user input to an application. The method may involve capturing a series of images from a moving mobile camera and comparing stationary features present in the series of images. Optical flow analysis may be performed on the series of images to determine a description of motion of the moving mobile camera. Based on the determined motion, a user input to an application may be determined and the application may respond to the user input, for example, by updating a user interface of the application.

Claims

1. A method comprising:

accessing images captured from an image sensor positioned on a device;

identifying a first feature that is present in the images captured from the image sensor;

identifying a second feature that is present in the images captured from the image sensor, the second feature being different than and spaced apart from the first feature; and

determining a description of motion of the device in a depth direction based on changes in distance between the first feature and the second feature in the images captured from the image sensor, the depth direction being a direction of movement substantially toward or substantially away from the first feature and the second feature.

2. The method of claim 1, further comprising:

prior to determining the description of motion of the device, determining that the first feature and the second feature are static features.

3. The method of claim 2, wherein determining that the first feature and the second feature are static features includes selecting the first feature and the second feature from a set of features present in the images captured from the image sensor based on similarity in direction and magnitude of flow vectors associated with the first feature and the second feature.

4. The method of claim 3, wherein the flow vectors are substantially parallel and of substantially equal magnitude.

5. The method of claim 1, wherein the second feature has a fixed physical relationship with the first feature.

6. The method of claim 1, further comprising:

determining user input for an application based on the determined description of motion of the device in the depth direction.

7. The method of claim 1, wherein the first feature includes a first object in the images, any part of the first object in the images, or a corner or edge of the first object in the images.

8. The method of claim 1, wherein a physical position of the first feature or the second feature does not change.

9. A device comprising:

at least one processor;

memory storing instructions that, when executed by the at least one processor, cause the device to;

access digital representations of a scene from a sensor coupled to the device;

identify a first feature that is present in the digital representations from the sensor;

identify a second feature that is present in the digital representations from the sensor, the second feature being different than and spaced apart from the first feature; and

determine a description of motion of the device in a depth direction based on changes in distance between the first feature and the second feature in the digital representations from the sensor, the depth direction being a direction of movement substantially toward or substantially away from the first feature and the second feature.

10. The device of claim 9, wherein the memory stores additional instructions that, when executed by the at least one processor, further cause the device to:

prior to determining the description of motion of the device, determine that the first feature and the second feature are static features.

11. The device of claim 10, wherein determining that the first feature and the second feature are static features includes selecting the first feature and the second feature from a set of features present in the digital representations from the sensor based on similarity in direction and magnitude of flow vectors associated with the first feature and the second feature.

12. The device of claim 11, wherein the flow vectors are substantially parallel and of substantially equal magnitude.

13. The device of claim 9, wherein the memory stores additional instructions that, when executed by the at least one processor, further cause the device to:

determine user input for an application based on the determined description of motion of the device in the depth direction.

14. The device of claim 9, wherein the first feature includes a first object in the digital representations, any part of the first object in the digital representations, or a corner or edge of the first object in the digital representations.

15. The device of claim 9, wherein a physical position of the first feature or the second feature does not change.

16. The device of claim 9, further comprising the sensor, wherein the sensor comprises a camera configured to capture an image of the scene.

17. A non-transitory computer readable medium comprising a device having instructions stored thereon to:

access images captured from an image sensor positioned on an apparatus;

identify a first feature that is present in the images captured from the image sensor;

identify a second feature that is present in the images captured from the image sensor, the second feature being different than and spaced apart from the first feature; and

determine a description of motion of the apparatus in a depth direction based on changes in distance between the first feature and the second feature in the images captured from the image sensor, the depth direction being a direction of movement substantially toward or substantially away from the first feature and the second feature.

18. The computer readable medium of claim 17, having additional instructions stored thereon to:

prior to determining the description of motion of the apparatus, determine that the first feature and the second feature are static features.

19. The computer readable medium of claim 17, having additional instructions stored thereon to:

determine user input for an application based on the determined description of motion of the apparatus in the depth direction.

20. The computer readable medium of claim 17, wherein the first feature includes a first object in the images, any part of the first object in the images, or a corner or edge of the first object in the images.

21. The computer readable medium of claim 17, wherein a physical position of the first feature or the second feature does not change.

22. A device comprising:

means for accessing images captured from an image sensor positioned on the device;

means for identifying a first feature that is present in the images captured from the image sensor;

means for identifying a second feature that is present in the images captured from the image sensor, the second feature being different than and spaced apart from the first feature; and

means for determining a description of motion of the device in a depth direction based on changes in distance between the first feature and the second feature in the images captured from the image sensor, the depth direction being a direction of movement substantially toward or substantially away from the first feature and the second feature.

23. The device of claim 22, further comprising:

means for determining that the first feature and the second feature are static features prior to determining the description of motion of the device.

24. The device of claim 22, further comprising:

means for determining user input for an application based on the determined description of motion of the device in the depth direction.

25. The device of claim 22, wherein the first feature includes a first object in the images, any part of the first object in the images, or a corner or edge of the first object in the images.

26. The device of claim 22, wherein a physical position of the first feature or the second feature does not change.