Providing a corrected view based on the position of a user with respect to a mobile platform

• US 9,507,416 B2
• Filed: 02/22/2011
• Issued: 11/29/2016
• Est. Priority Date: 02/22/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kimball, Robert Howard
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US13/032,493
• Publication Number:
  US 20120212508A1
• Time in Patent Office: 2,107 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A mobile platform displays a corrected view of an image and/or augmented reality (AR) data based on the position of the user with respect to the mobile platform. The corrected view is produced by determining a position of the user with respect to the mobile platform using an image of the user from a backward facing camera. The display information is provided in the form of an image or video frame of the environment captured with a forward facing camera or AR data. The position of the user with respect to the mobile platform is used to determine the portion of the display information to be displayed that is aligned with the line of sight between the user and the mobile platform so that the displayed information is aligned with the real world environment.

Claims

1. A method comprising:

determining a position of a user with respect to a mobile platform;

providing display information for an environment, the display information comprising at least one of an image of the environment or augmented reality data;

using the position of the user with respect to the mobile platform to determine a portion of the display information to display that is aligned with a line of sight between the user and the mobile platform, wherein the portion of the display information is less than the entirety of the display information; and

displaying only the portion of the display information that is aligned with the line of sight between the user and the mobile platform when the mobile platform is facing a direction that is not along the line of sight.

2. The method of claim 1, wherein displaying the portion of the display information that is aligned with the line of sight between the user and the mobile platform comprises at least one of displaying a portion of the image, displaying augmented reality data, or displaying augmented reality data over the portion of the image.

3. The method of claim 1, wherein determining the position of the user with respect to the mobile platform comprises:

capturing an image of the user with a backward facing camera on the mobile platform; and

analyzing the image of the user to determine the position of the user within the image.

4. The method of claim 1, further comprising:

determining a distance between the mobile platform and the user; and

using the distance between the mobile platform and the user to further determine the portion of the display information to display, wherein an increased portion of the display information is displayed when the distance between the mobile platform and the user is decreased.

5. The method of claim 4, wherein determining the distance between the mobile platform and the user comprises:

capturing an image of the user with a backward facing camera on the mobile platform; and

analyzing the image of the user using biometrics to determine the distance between the mobile platform and the user.

6. The method of claim 1, wherein providing the display information for the environment comprises capturing an image of the environment with a forward facing camera on the mobile platform.

7. The method of claim 6, wherein using the position of the user with respect to the mobile platform to determine a portion of the display information to display comprises cropping the image of the environment to produce the portion of the image that is aligned with the line of sight.

8. The method of claim 1, wherein using the position of the user with respect to the mobile platform to determine a portion of the display information to display comprises determining a field of view based on a position and orientation of the mobile platform and the position of the user with respect to the mobile platform, and retrieving augmented reality data based on the field of view.

9. The method of claim 1, wherein the mobile platform comprises a forward facing camera to capture an image of the environment and a backward facing camera to capture an image of the user in order to determine the position of the user with respect to the mobile platform.

10. A mobile platform comprising:

a forward facing camera;

a backward facing camera;

a processor connected to receive data from the forward facing camera and the backward facing camera;

memory connected to the processor;

a display connected to the memory; and

software held in the memory and run in the processor to cause the processor to determine a position of a user with respect to the mobile platform based on an image of the user captured by the backward facing camera, provide display information for an environment, the display information comprising at least one of an image of the environment captured by the forward facing camera or augmented reality data, use the position of the user with respect to the mobile platform to determine a portion of the display information to display that is aligned with a line of sight between the user and the mobile platform, wherein the portion of the display information is less than the entirety of the display information, and to display only the portion of the display information on the display.

11. The mobile platform of claim 10, wherein the portion of the display information that is displayed comprises at least one of a portion of the image, augmented reality data, or augmented reality data over the portion of the image.

12. The mobile platform of claim 10, wherein the software causes the processor to analyze the image of the user to determine the position of the user with respect to the mobile platform.

13. The mobile platform of claim 12, wherein the software causes the processor to analyze the image of the user using biometrics to determine a distance between the mobile platform and the user and use the distance between the mobile platform and the user to further determine the portion of the display information to display, wherein an increased portion of the display information is displayed when the distance between the mobile platform and the user is decreased.

14. The mobile platform of claim 10, wherein the software causes the processor to use the position of the user with respect to the mobile platform to determine a portion of the display information to display by cropping the image to produce the portion of the image that is aligned with the line of sight.

15. The mobile platform of claim 12, wherein the software that causes the processor to use the position of the user with respect to the mobile platform to determine a portion of the display information to display comprises software to determine a field of view based on a position and orientation of the mobile platform and the position of the user with respect to the mobile platform and wherein the software causes the processor to retrieve augmented reality data based on the field of view.

16. A system comprising:

means for determining a position of a user with respect to a mobile platform;

means for providing display information for an environment, the display information comprising at least one of an image of the environment or augmented reality data;

means for using the position of the user with respect to the mobile platform to determine a portion of the display information to display that is aligned with a line of sight between the user and the mobile platform, wherein the portion of the display information is less than the entirety of the display information; and

means for displaying only the portion of the display information that is aligned with the line of sight between the user and the mobile platform when the mobile platform is facing a direction that is not along the line of sight.

17. The system of claim 16, wherein the means for displaying the portion of the display information that is aligned with the line of sight between the user and the mobile platform displays at least one of a portion of the image, augmented reality data, or augmented reality data over the portion of the image.

18. The system of claim 16, wherein the means for determining the position of the user with respect to the mobile platform comprises:

means for capturing an image of the user with a backward facing camera on the mobile platform; and

means for analyzing the image of the user to determine the position of the user with respect to the mobile platform.

19. The system of claim 16, further comprising:

means for determining a distance between the mobile platform and the user; and

means for using the distance between the mobile platform and the user to determine the portion of the display information to display, wherein an increased portion of the display information is displayed when the distance between the mobile platform and the user is decreased.

20. The system of claim 19, wherein the means for determining the distance between the mobile platform and the user comprises means for biometrically analyzing an image of the user capturing with a backward facing camera on the mobile platform to determine the distance between the mobile platform and the user.

21. A non-transitory computer-readable medium comprising program code, which, when executed by a computer, cause the computer to perform operations, the program code comprising:

program code to determine a position of a user with respect to a mobile platform based on an image of the user captured by a backward facing camera;

program code to provide display information for an environment, the display information comprising at least one of an image of the environment captured by a forward facing camera or augmented reality data;

program code to use the position of the user with respect to the mobile platform to determine a portion of the display information to display that is aligned with a line of sight between the user and the mobile platform, wherein the portion of the display information is less than the entirety of the display information; and

program code to display only the portion of the display information for the environment that is aligned with the line of sight when the mobile platform is facing a direction that is not along the line of sight.

22. The non-transitory computer-readable medium of claim 21, wherein the program code to determine the position of the user with respect to the mobile platform comprises program code to analyze the image of the user captured with the backward facing camera on the mobile platform to determine the position of the user within the image and to use the position of the user within the image to determine the position of the user with respect to the mobile platform.

23. The non-transitory computer-readable medium of claim 21, further comprising program code to determine a distance between the mobile platform and the user wherein the program code to use the position of the user with respect to the mobile platform to determine the portion of the display information to display further comprises program code to use the distance between the mobile platform and the user to determine the portion of the display information to display, wherein an increased portion of the display information is displayed when the distance between the mobile platform and the user is decreased.