NATURAL USER INTERFACES FOR MOBILE IMAGE VIEWING

• US 20150002393A1
• Filed: 09/16/2014
• Published: 01/01/2015
• Est. Priority Date: 06/13/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cohen, Michael F., Joshi, Neel Suresh
• Granted Patent: US 10,275,020 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

The mobile image viewing technique described herein provides a hands-free interface for viewing large imagery (e.g., 360 degree panoramas, parallax image sequences, and long multi-perspective panoramas) on mobile devices. The technique controls the imagery displayed on a display of a mobile device by movement of the mobile device. The technique uses sensors to track the mobile device'"'"'s orientation and position, and front facing camera to track the user'"'"'s viewing distance and viewing angle. The technique adjusts the view of a rendered imagery on the mobile device'"'"'s display according to the tracked data. In one embodiment the technique can employ a sensor fusion methodology that combines viewer tracking using a front facing camera with gyroscope data from the mobile device to produce a robust signal that defines the viewer'"'"'s 3D position relative to the display.

Claims

1. A computer-implemented process for viewing imagery on a mobile device, comprising:

tracking a mobile device'"'"'s orientation and position;

using a camera and viewer tracker on the mobile device to track a user'"'"'s face looking at a screen on the mobile device;

computing image transformations of imagery displayed on the screen of the mobile device using the tracked mobile device'"'"'s orientation and position and the tracked user'"'"'s face to allow dynamic change of the perspective of the displayed imagery by moving the mobile device relative to the user'"'"'s face.

2. The computer-implemented process of claim 1, wherein the perspective of the imagery displayed can be changed even when the user'"'"'s face is outside of the camera'"'"'s field of view.

3. The computer-implemented process of claim 1, further comprising:

computing the viewing angle and a viewing distance between the user and the screen on the mobile device by using the tracked orientation and position of the mobile device, and the tracked position of the user'"'"'s face relative to screen of the mobile device in order to compute the image transformations.

4. The computer-implemented process of claim 1, further comprising zooming in or out of the imagery displayed on the screen whenever the distance of the mobile device relative to the user'"'"'s face changes.

5. The computer-implemented process of claim 1, further comprising panning around the imagery displayed on the screen whenever the position of the mobile device changes laterally in relation to the user'"'"'s face.

6. The computer-implemented process of claim 1, changing the imagery displayed on the screen when the mobile device is tilted.

7. The computer-implemented process of claim 1, wherein the mobile device'"'"'s orientation and position is determined by a gyroscope on the mobile device.

8. The computer-implemented process of claim 7 wherein the viewer tracker is used to correct for drift of the gyroscope.

9. The computer-implemented process of claim 1 wherein the imagery comprises location imagery that is displayed by a mapping application.

10. The computer-implemented process of claim 9 wherein information is overlaid on the location imagery.

11. The computer-implemented process of claim 1 wherein the imagery comprises a multi-perspective panorama.

12. A computer-implemented process for viewing imagery on a mobile device, comprising:

tracking a mobile device'"'"'s orientation and position using the mobile device;

using a front-facing camera and viewer tracker on the mobile device to track a user'"'"'s face looking at a screen on the mobile device;

using the mobile device'"'"'s orientation and position and the position of the user'"'"'s face to determine a combined position and rate control for viewing imagery in three dimensions on the screen of the mobile device;

using the combined position and rate control to compute image transformations of the imagery rendered on the screen of the mobile device to allow the dynamic display of different viewpoints in three dimensions of the rendered imagery.

13. The computer-implemented process of claim 12, wherein the imagery is a 360 degree panorama and wherein a user can pan to the left and to the right in the rendered imagery by changing the viewing angle between the user and the screen of the mobile device, and can zoom into the imagery by changing the distance between the user and the screen of the mobile device.

14. The computer-implemented process of claim 12, wherein the imagery is a set of parallax images and wherein the combined position and rate control is used to determine a relative offset of a virtual camera.

15. The computer-implemented process of claim 12, wherein the imagery comprises a series of 360 degree panoramas of the same scene taken at fixed intervals, and a set of long perspective strip panoramas created by clipping out and stitching parts of the series of 360 degree panoramas.

16. The computer-implemented process of claim 15, wherein a user can view left and right in a 360 degree panorama of the series by changing the viewing angle between the user'"'"'s face and the screen of the mobile device and can zoom into a different 360 degree panorama of the series by changing the viewing distance between the user'"'"'s face and the screen of the mobile device.

17. A system for viewing imagery, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,

track a mobile device'"'"'s orientation and position;

use a camera and viewer tracker to track a face of a user viewing imagery rendered on the screen of the mobile device,

compute image transformations of the imagery rendered on the screen of the mobile device that allow dynamic display of different viewpoints of the rendered imagery in both depth and position by moving the mobile device relative to the user'"'"'s face.

18. The system of claim 17 wherein the user can change the viewpoint of the imagery rendered on the screen of the mobile device by changing the viewing angle of the mobile device relative to the user'"'"'s face.

19. The system of claim 17, wherein the user'"'"'s face can be outside of the field of view of the camera and wherein a gyroscope on the mobile device can be used to estimate the location of the face.

20. The system of claim 17, wherein the imagery rendered on the screen can be changed by tilting the mobile phone.