Optical Flow Based Tilt Sensor

• US 20080137913A1
• Filed: 10/31/2007
• Published: 06/12/2008
• Est. Priority Date: 01/07/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: GestureTek Incorporated (Winning Brands Corp.)
• Inventors: Hildreth, Evan
• Granted Patent: US 7,848,542 B2
• Time in Patent Office: Days
• US Class Current: 382/107
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04M 1/72427 for supporting games or graphical animations
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 1/00352 Input means H04N1/00411 takes precedence; input arrangements for computers G06F3/00
• CPC Class Codes: H04N 1/00381 Input by recognition or interpretation of visible user gestures manual input means, e.g. digitisers, writing tablets H04N1/00392; gesture input to computers G06F3/017; recognition algorithms G06V40/20
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0096 Portable devices

Abstract

A method is described for determining a description of motion of a moving mobile camera to determine a user input to an application. The method may involve capturing a series of images from a moving mobile camera and comparing stationary features present in the series of images. Optical flow analysis may be performed on the series of images to determine a description of motion of the moving mobile camera. Based on the determined motion, a user input to an application may be determined and the application may respond to the user input, for example, by updating a user interface of the application.

Claims

1. A computer-implemented method comprising:

capturing images from a moving mobile camera;

determining a description of motion of the moving mobile camera based on the captured images;

rendering a user interface for an application on an interface device, the user interface including an object; and

automatically controlling a steering, rolling or navigating movement of the object rendered in the user interface in a manner corresponding to the determined motion of the moving mobile camera.

2. The method of claim 1 wherein:

determining a description of motion comprises determining a magnitude and all orientation; and

automatically controlling the steering, rolling or navigating movement of the object rendered in the user interface in a manner corresponding to the determined motion of the moving mobile camera further comprises controlling the object rendered in the user interface to steer, roll or navigate a distance within the user interface corresponding, to the determined magnitude in a direction corresponding to the determined orientation.

3. The method of claim 1 wherein determining a description of motion of the moving mobile camera comprises calculating rotation of the moving mobile camera in two degrees of freedom.

4. The method of claim 1 wherein determining a description of motion of the moving mobile camera comprises calculating rotation of the moving mobile camera in three degrees of freedom.

5. The method of claim 1 wherein:

determining a description of motion of the moving mobile camera comprises calculating translation of the moving mobile camera; and

automatically controlling the steering, rolling or navigating movement of the object rendered in the user interface in a manner corresponding to the determined motion of the moving mobile camera further comprises controlling an avatar rendered in the user interface to steer, roll or navigate in the user interface in a manner corresponding to the calculated translation of the moving mobile camera.

6. The method of claim 1 wherein determining a description of motion of the moving mobile camera comprises calculating motion of the moving mobile camera with respect to an initial position of the moving mobile camera.

7. The method of claim 1 wherein determining a description of motion of the moving mobile camera comprises:

determining motion of features within the captured images; and

determining motion of the mobile camera based on the motion of the features.

8. The method of claim 7 wherein determining a description of motion of the moving mobile camera further comprises:

segmenting the features, based on the determined motion of the features, into a set of moving features that are moving with respect to a real-world frame of reference.

9. The method of claim 1 wherein:

determining a description of motion comprises;

extracting features from the captured images;

comparing images to match features present in more than one image;

determining a flow vector for matched features by calculating the displacement in the position of a feature in a current image relative to the position of the feature in a prior image; and

determining the motion of the mobile camera based on the flow vector determined for the matched features, and

automatically controlling the steering, rolling or navigating movement of the object rendered in the user interface in a manner corresponding to the determined motion of the moving mobile camera further comprises controlling the object rendered in the user interface to move in a direction corresponding to the determined flow vector.

10. The method of claim 9 wherein extracting features from the captured images includes detecting at least one of corners and edges.

11. The method of claim 9 wherein comparing images to match features comprises first comparing a block of an image located proximate to the known position of the feature in another image.

12. The method of claim 9 wherein comparing images to match features comprises first comparing images at a low resolution and, conditioned on whether a match is found, comparing images at a higher resolution.

13. The method of claim 9 wherein determining the motion of the mobile camera based on the flow vectors comprises determining the motion of the mobile camera in real-world units by converting the motion determine in image-dependent units.

14. The method of claim 1 wherein:

capturing images comprises capturing a sequence of images, and

determining a description of motion of the moving mobile camera comprises;

determining motion for a first feature visible in a first subset of images within the sequence of images;

determining-motion for a second feature visible in a second subset of images within the sequence of images, overlapping partially with the first subset, and the second feature having a fixed physical relationship with the first feature, wherein motion for the second feature is based on the motion for the first;

feature even though the first feature is not visible in all of the second subset of images; and

determining the description of motion of the moving mobile camera based on the determined motions for the first and second features.

15. The method of claim 1 wherein:

the object is a ball rendered in the user interface of an application, and

wherein automatically controlling the steering, rolling or navigating movement of the object rendered in the user interface in a manner corresponding to the determined motion of the moving mobile camera further comprises controlling a rolling direction of the ball in the user interface in a manner corresponding to the determined motion of the moving mobile camera.

16. The method of claim 1 wherein:

the object is a vehicle rendered in the user interface of an application, and

automatically controlling steering, rolling or navigating movement of the object rendered in the user interface in a manner corresponding to the determined motion of the moving mobile camera further comprises controlling a steering movement of the vehicle in the user interface in a manner corresponding to the determined motion of the moving mobile camera.

17. The method of claim 1 wherein:

the object is a menu object rendered in the user interface of an application, and

automatically controlling the steering, rolling or navigating movement of the object rendered in the user interface in a manner corresponding to the determined motion of the moving mobile camera further comprises controlling a menu navigation function associated with the object in a manner corresponding to the determined motion of the moving mobile camera.

18. The method of claim 1 wherein:

the object is an articulated character in an application, and

automatically controlling the steering, rolling or navigating movement of the object rendered in the user interface in a manner corresponding to the determined motion of the moving mobile camera further comprises controlling movement of the articulated character.

19. The method of claim 1 wherein:

the object is a particle in a particle simulation application, and

automatically controlling movement of the object rendered in the user interface in a manner corresponding to the determined motion of the moving mobile camera further comprises controlling movement of the particle in the particle simulation application.

20. The method of claim 1 wherein the application is at least one of a game, a tracking application, and a security application.

21. The method of claim 1 wherein the moving mobile camera is physically coupled to a cellular phone.

22. A device comprising:

a camera;

a display device; and

a processor configured to;

capture images from the camera;

determine a description of motion of the handheld device based on the captured images;

render a user interface for an application on the display device, the user interface including an object; and

automatically control a steering, rolling or navigating movement of the object rendered in the user interface in a manner corresponding to the determined motion of the camera.

23. A computer readable storage medium, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

capture images from a moving mobile camera;

determine a description of motion of the moving mobile camera based on the captured images;

render a user interface for an application on an interface device, the user interface including an object; and

automatically control a steering, rolling or navigating movement of the object rendered in the user interface in a manner corresponding to the determined motion of the moving mobile camera.

24. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

determine a description of motion of a moving mobile camera based on captured images;

render a user interface for an application on an interface device, the user interface including a virtual environment including an object; and

navigate through the virtual environment in a manner corresponding to the determined motion of the moving mobile camera while maintaining the object in a fixed position on the user interface.

25. A computer program, product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

determine a description of motion of a moving mobile camera based on captured images;

render a user interface for an application including a virtual environment; and

navigate through the virtual environment in a manner corresponding to the determined motion of the moving mobile camera.

26. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

capture images from a moving mobile camera;

determine a description of motion of the moving mobile camera based on the captured images;

determine an absolute position of the mobile camera;

render a user interface for an application on an interface device, the user interface including an object; and

automatically control movement of the object rendered in the user interface, based on the determined motion of the moving mobile camera and the determined absolute position.

27. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause data processing, apparatus to:

determine a description of motion of a moving mobile camera based on captured images;

render a user interface for a particle simulation application on an interface device; and

automatically control movement of particles in the particle simulation application based on the determined motion of the moving mobile camera.

28. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

determine a description of motion of a moving mobile camera based on captured images;

render a user interface for an application on an interface device, the user interface including an animated character having simulated mass; and

translate the motion into a movement of the animated character; and

automatically control the animated character based on the translated movement.

29. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

determine a description of motion of a moving camera phone-based on captured images; and

automatically control a volume of the camera phone based on the determined description of motion.