Viewing images with tilt control on a hand-held device

• US 8,994,644 B2
• Filed: 01/25/2008
• Issued: 03/31/2015
• Est. Priority Date: 01/26/2007
• Status: Active Grant

• Current Assignee: Motionip LLC
• Original Assignee: Apple Inc.
• Inventors: Cook, Ralph, Mlne, Anders L., Carsanaro, Joseph A.
• Primary Examiner(s): PATEL, PREMAL R
• Application Number:
  US12/449,182
• Publication Number:
  US 20100171691A1
• Time in Patent Office: 2,622 Days
• Field of Search: 345/156, 345/157, 345/158, 345/619, 345/649
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1613 for portable computers cooling arrangements therefor G06F1/203; constructional details or arrangements for pocket calculators, electronic agendas or books G06F15/0216; constructional details of portable telephone sets: with several bodies H04M1/0202
• CPC Class Codes: G06F 1/1637 Details related to the display arrangement, including those related to the mounting of the display in the housing
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G09G 2340/04 Changes in size, position or resolution of an image
• CPC Class Codes: G09G 5/34 for rolling or scrolling

Abstract

A user interface suitable for use in cellular phones and personal digital assistants (PDAs), PC Tablets, as well as laptops, PCs, office equipment, medical equipment, or any other hand-held electronic device, that allows control of the image on the device display by tilting the device to either change the view in perspective, change the magnification, or both, concurrently, by moving the device. Thus, the tilt of the device controls the angle of view of the image, and moving the device perpendicular to the screen controls the magnification.

Claims

1. A handheld electronic device comprising:

a processor;

a display panel in data communication with said processor;

a user-input assembly in data communication with said processor;

one or more motion-sensing devices in data communication with said processor for sensing movement of the handheld electronic device and outputting a motion signal indicative thereof, the one or more motion-sensing devices including an accelerometer;

at least one optical sensor in data communication with said processor;

a memory device coupled to said processor, said memory device including instructions to be executed, wherein said instructions, when executed by said processor, cause said processor to;

determine relative orientation changes of the handheld electronic device over time based at least in part on multiple sequential images captured by the at least one optical sensor at one or more time intervals;

correlate the relative orientation changes with force data measured by the accelerometer to generate measurements of orientation with respect to a horizontal plane and inertial acceleration in three dimensions; and

based at least in part on the measurements of orientation, control display of an object on said display panel as a function of motion of said handheld electronic device, said function interpreting motion along a first axis as a first function that affects the display of the object on said display panel, and said function interpreting motion about a second axis distinct from the first axis as a second function that affects the display of the object on said display panel, wherein the first function is different from the second function.

2. The handheld electronic device of claim 1, wherein the relative orientation changes of the handheld electronic device are determined by comparing overlap in said multiple sequential images.

3. The handheld electronic device of claim 1, wherein the accelerometer measures gravitational forces and inertial acceleration along three orthogonal axes to generate the force data and wherein the optical sensor captures the multiple sequential images at known time intervals.

4. The handheld electronic device of claim 1, wherein the instructions further cause the processor to:

decrease a magnification of the object on said display panel in response to moving the handheld electronic device in a first direction along the first axis, and increase the magnification of the object on said display panel in response to moving the handheld electronic device in a second direction, opposite the first direction, along the first axis.

5. The handheld electronic device of claim 1, wherein the instructions further cause the processor to:

define a threshold for a maximum tilt angle; and

in accordance with a determination that a tilt angle of the handheld electronic device has increased beyond the threshold for the maximum tilt angle, scroll a display of the object on said display panel.

6. The handheld electronic device of claim 1, wherein the instructions further cause the processor to:

scroll a display of the object on said display panel with increasing velocity as a tilt angle of said handheld electronic device increases.

7. The handheld electronic device of claim 6, wherein the scrolling of the display of the object on said display panel comprises orbiting around a perspective view of the object.

8. The handheld electronic device of claim 6, wherein the instructions further cause the processor to:

while scrolling the display of the object on said display panel, center a current perspective view of the object on said display panel in response to a user action.

9. The handheld electronic device of claim 8, wherein the user action is a pre-defined motion of the device, a button push, a screen tap, or a voice command.

10. The handheld electronic device of claim 1, wherein the first function comprises a zoom function and wherein the second function comprises a change in perspective function.

11. A method comprising:

determining, by a handheld electronic device, relative orientation changes of the handheld electronic device over time based at least in part on multiple sequential images captured by at least one optical sensor at one or more time intervals;

correlating, by said handheld electronic device, the relative orientation changes with force data measured by an accelerometer to generate measurements of orientation with respect to a horizontal plane and inertial acceleration in three dimensions; and

based at least in part on the measurements of orientation, controlling, by said handheld electronic device, display of an object on a display panel of the handheld electronic device as a function of motion of said handheld electronic device, said function interpreting motion along a first axis as a first function that affects the display of the object on said display panel, and said function interpreting motion about a second axis distinct from the first axis as a second function that affects the display of the object on said display panel, wherein the first function is different from the second function.

12. The method of claim 11 wherein the relative orientation changes of the handheld electronic device are determined by comparing overlap in said multiple sequential images.

13. The method of claim 11, wherein the accelerometer measures gravitational forces and inertial acceleration along three orthogonal axes to generate the force data and wherein the optical sensor captures the multiple sequential images at known time intervals.

14. The method of claim 11, further comprising:

decreasing a magnification of the object on said display panel in response to moving the handheld electronic device in a first direction along the first axis, and increasing the magnification of the object on said display panel in response to moving the handheld electronic device in a second direction, opposite the first direction, along the first axis.

15. The method of claim 11, further comprising:

defining a threshold for a maximum tilt angle; and

in accordance with a determination that a tilt angle of the handheld electronic device has increased beyond the threshold for the maximum tilt angle, scrolling a display of the object on said display panel.

16. The method of claim 11, further comprising:

scrolling a display of the object on said display panel with increasing velocity as a tilt angle of said handheld electronic device increases.

17. The method of claim 16, wherein the scrolling of the display of the object on said display panel comprises orbiting around a perspective view of the object.

18. The method of claim 16, further comprising:

while scrolling the display of the object on said display panel, centering a current perspective view of the object on said display panel in response to a user action.

19. The method of claim 18, wherein the user action is a pre-defined motion of the device, a button push, a screen tap, or a voice command.

20. The method of claim 11, wherein the first function comprises a zoom function and wherein the second function comprises a change in perspective function.