Systems and methods for adjusting a display based on the user's position

• US 9,092,053 B2
• Filed: 06/17/2008
• Issued: 07/28/2015
• Est. Priority Date: 06/17/2008
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kerr, Duncan R., King, Nicholas V.
• Primary Examiner(s): Levy, Amy M
• Application Number:
  US12/140,604
• Publication Number:
  US 20090313584A1
• Time in Patent Office: 2,597 Days
• Field of Search: 715/848, 715/852, 715/782, 715/757, 715/836, 715/849
• US Class Current: 1/1
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G09G 2340/04 Changes in size, position or resolution of an image
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

An electronic device for providing a display that changes based on the user'"'"'s perspective is provided. The electronic device may include a sensing mechanism operative to detect the user'"'"'s position relative a display of the electronic device. For example, the electronic device may include a camera operative to detect the position of the user'"'"'s head. Using the detected position, the electronic device may be operative to transform displayed objects such that the displayed perspective reflects the detected position of the user. The electronic device may use any suitable approach for modifying a displayed object, including for example a parallax transform or a perspective transform. In some embodiments, the electronic device may overlay the environment detected by the sensing mechanism (e.g., by a camera) to provide a more realistic experience for the user (e.g., display a reflection of the image detected by the camera on reflective surfaces of a displayed object).

Claims

1. An electronic device for displaying three-dimensional objects, comprising:

a display;

a sensing mechanism; and

control circuitry that is operative to;

direct the sensing mechanism to capture a first image of a user of the electronic device within a field of view of the sensing mechanism;

detect a first position of the user with respect to a component of the electronic device from the captured first image;

in response to the first position being detected, direct the display to concurrently display;

a three-dimensional object from a first perspective, wherein at least a portion of the three-dimensional object from the first perspective comprises a first reflective surface and at least a portion of the captured first image is applied to at least a portion of the first reflective surface when the three-dimensional object from the first perspective is displayed; and

a first depiction comprising a window and an element at a first element position within the window, wherein the first element position of the element with respect to the boundaries of the window is based on the detected first position of the user with respect to the component of the electronic device;

after the display is directed to display the three-dimensional object from the first perspective, direct the sensing mechanism to capture a second image of the user within the field of view of the sensing mechanism;

detect a current second position of the user with respect to the component of the electronic device from the captured second image, wherein the detected current second position is different than the detected first position;

in response to the second position being detected, transform the displayed three-dimensional object by;

changing the displayed three-dimensional object from the first perspective to a second perspective based on a difference between the first position and the detected current second position, wherein at least a portion of the changed three-dimensional object comprises a second reflective surface; and

applying at least a portion of the captured second image to at least a portion of the second reflective surface;

direct the display to concurrently display;

the transformed three-dimensional object; and

a second depiction comprising the window and the element at a second element position within the window, wherein the second element position of the element with respect to the boundaries of the window is based on the detected current second position of the user with respect to the component of the electronic device;

detect that a position of the user within the field of view of the sensing mechanism has not changed more than a defined threshold amount within a specific period of time; and

remove any depiction of the window from the display when it has been detected that the position of the user within the field of view of the sensing mechanism has not changed more than the defined threshold amount within the specific period of time.

2. The electronic device of claim 1, wherein the control circuitry is further operative to:

after the display is directed to display the transformed three-dimensional object, direct the sensing mechanism to capture a third image of the user;

detect a third position of the user with respect to the electronic device from the captured third image;

in response to the third position being detected, re-transform the transformed three-dimensional object by;

re-changing the transformed three-dimensional object from the second perspective to a third perspective based on a difference between the second position and the third position, wherein at least a portion of the re-changed three-dimensional object comprises a third reflective surface; and

applying at least a portion of the captured third image to at least a portion of the third reflective surface; and

direct the display to display the re-transformed three-dimensional object.

3. The electronic device of claim 1, wherein the control circuitry is operative to change the displayed three-dimensional object by performing a parallax transform on the displayed three-dimensional object.

4. The electronic device of claim 1, wherein the control circuitry is operative to change the displayed three-dimensional object by performing a perspective transform on the displayed three-dimensional object.

5. The electronic device of claim 1, wherein the control circuitry is further operative to:

direct the display to display another three-dimensional object;

transform the displayed another three-dimensional object based on the detected second position; and

direct the display to display the transformed another three-dimensional object.

6. The electronic device of claim 5, wherein the control circuitry is operative to:

transform the displayed three-dimensional object by performing a first type of transform; and

transform the displayed another three-dimensional object by performing a second type of transform.

7. The electronic device of claim 6, wherein the first type of transform is different from the second type of transform.

8. The electronic device of claim 6, wherein the control circuitry is operative to transform at least one of the displayed three-dimensional object and the displayed another three-dimensional object based on a display position of the displayed three-dimensional object relative to a display position of the displayed another three-dimensional object.

9. The electronic device of claim 6, wherein:

the first type of transform comprises a parallax transform; and

the second type of transform comprises a perspective transform.

10. The electronic device of claim 1, wherein the sensing mechanism comprises a lens.

11. The electronic device of claim 1, wherein:

the second reflective surface comprises a curve; and

the applying comprises applying the at least a portion of the captured second image based on the curve.

12. The electronic device of claim 1, wherein the boundaries of the window represent the boundaries of the component of the electronic device.

13. The electronic device of claim 1, wherein the component of the electronic device is the display.

14. The electronic device of claim 1, wherein the component of the electronic device is the sensing mechanism.

15. The electronic device of claim 14, wherein the boundaries of the window represent the field of view of the sensing mechanism.

16. The electronic device of claim 1, wherein the first depiction further comprises at least a portion of the captured first image within the window.

17. The electronic device of claim 16, wherein the second depiction further comprises at least a portion of the captured second image within the window.

18. The electronic device of claim 1, wherein:

the first depiction further comprises a first representation of the user within the window; and

the first element position of the element is placed on a particular feature of the user provided by the first representation of the user.

19. The electronic device of claim 18, wherein:

the second depiction further comprises a second representation of the user within the window; and

the second element position of the element is placed on the particular feature of the user provided by the second representation of the user.

20. The electronic device of claim 19, wherein the particular feature is the face of the user.

21. The electronic device of claim 20, wherein the element comprises an intersection of crosshairs.

22. The electronic device of claim 1, wherein the element comprises an intersection of crosshairs.

23. A method comprising:

capturing a first image of an environment of an electronic device with a sensing mechanism of the electronic device;

detecting with the electronic device a first position of a user within a field of view of the sensing mechanism with respect to a component of the electronic device from the captured first image;

in response to detecting the first position, concurrently displaying with a display of the electronic device;

a three-dimensional object from a first perspective, wherein at least a portion of the three-dimensional object from the first perspective comprises a first reflective surface and at least a portion of the captured first image is applied to at least a portion of the first reflective surface when the three-dimensional object from the first perspective is displayed; and

a first depiction comprising a window and an element at a first element position within the window, wherein the first element position of the element with respect to the boundaries of the window is based on the detected first position of the user with respect to the component of the electronic device;

once the three-dimensional object is displayed from the first perspective, capturing a second image of the environment with the sensing mechanism;

detecting with the electronic device a second position of the user within the field of view of the sensing mechanism with respect to the component of the electronic device from the captured second image, wherein the detected second position is different than the detected first position;

in response to detecting the second position, transforming with the electronic device the displayed three-dimensional object by;

changing the displayed three-dimensional object from the first perspective to a second perspective based on a difference between the first position and the second position, wherein at least a portion of the changed three-dimensional object comprises a second reflective surface; and

mapping with the electronic device at least a portion of the captured second image on at least a portion of the second reflective surface;

concurrently displaying with the display;

the transformed three-dimensional object; and

a second depiction comprising the window and the element at a second element position within the window, wherein the second element position of the element with respect to the boundaries of the window is based on the detected second position of the user with respect to the component of the electronic device;

detecting that a position of the user within the field of view of the sensing mechanism has not changed more than a defined threshold amount within a specific period of time; and

removing any depiction of the window from the display when it has been detected that the position of the user within the field of view of the sensing mechanism has not changed more than the defined threshold amount within the specific period of time.

24. The method of claim 23, further comprising:

once the transformed three-dimensional object is displayed, capturing a third image of the environment with the sensing mechanism;

detecting with the electronic device a third position of the user with respect to the electronic device from the captured third image;

in response to detecting the third position, re-transforming with the electronic device the transformed three-dimensional object by;

re-changing the transformed three-dimensional object from the second perspective to a third perspective based on a difference between the second position and the third position, wherein at least a portion of the re-changed three-dimensional object comprises a third reflective surface; and

mapping with the electronic device at least a portion of the captured third image on at least a portion of the third reflective surface; and

displaying with the display the re-transformed three-dimensional object.

25. The method of claim 23, further comprising:

defining with the electronic device at least one optical property for each surface of a plurality of surfaces of the displayed three-dimensional object, wherein the mapping further comprises applying the at least a portion of the captured second image to at least one surface of the plurality of surfaces based on the defined optical properties.

26. The method of claim 25, wherein:

the at least one optical property comprises a reflection property; and

the mapping is based on the at least one reflection property.

27. The method of claim 25, wherein:

the at least one optical property comprises a reflection property; and

the mapping further comprises not applying the at least a portion of the captured second image to at least one other surface of the plurality of surfaces based on the reflection property for the at least one other surface.

28. The method of claim 23, further comprising updating with the electronic device the display of the three-dimensional object based on the mapped at least a portion of the captured second image.

29. The method of claim 23, wherein:

the captured second image comprises the user; and

the mapped at least a portion of the captured second image comprises at least a portion of a face of the user.

30. A method comprising:

capturing a first image of a user of an electronic device with a sensing mechanism of the electronic device;

detecting with the electronic device a first position of the user relative to the electronic device from the captured first image;

in response to detecting the detected first position, simultaneously displaying using a display of the electronic device;

at least one two-dimensional object from a first perspective, wherein the first perspective is based on the detected first position, and wherein the at least one two-dimensional object comprises at least one application window comprising a depth in the display that is different than another depth of another application window in the display;

at least one three-dimensional object from the first perspective, wherein the at least one three-dimensional object comprises an icon in a dock; and

a first depiction comprising a window and an element at a first element position within the window, wherein the first element position of the element with respect to the boundaries of the window is based on the detected first position of the user relative to the electronic device;

capturing a second image of the user with the sensing mechanism of the electronic device;

detecting a second position of the user relative to the electronic device using the sensing mechanism of the electronic device, wherein the detected second position is different than the detected first position; and

in response to detecting the detected second position, simultaneously displacing;

the displayed at least one two-dimensional object from the first perspective to a second perspective using a first transform, wherein the second perspective is based on the detected second position;

the displayed at least one three-dimensional object from the first perspective to the second perspective using a second transform; and

a second depiction comprising the window and the element at a second element position within the window, wherein the second element position of the element with respect to the boundaries of the window is based on the detected second position of the user relative to the electronic device, and wherein the element comprises an intersection of crosshairs.

31. The method of claim 30, wherein:

the displayed at least one two-dimensional object comprises first metadata that associates the displayed at least one two-dimensional object with the first transform; and

the displayed at least one three-dimensional object comprises second metadata that associates the displayed at least one three-dimensional object with the second transform.

32. The method of claim 30, wherein:

the first transform comprises a parallax transform; and

the second transform comprises a perspective transform.

33. The method of claim 32, wherein:

the display comprises a front plane and a rear plane;

using the parallax transform comprises adjusting only the rear plane based on the detected second position; and

using the perspective transform comprises adjusting both of the rear plane and the front plane based on the detected second position.

34. The method of claim 30, wherein one of the displayed at least one three-dimensional object is displayed within one of the at least one application window.

35. The method of claim 30, wherein:

the displayed at least one two-dimensional object comprises first metadata that associates the displayed at least one two-dimensional object with the first transform;

the displayed at least one three-dimensional object comprises second metadata that associates the displayed at least one three-dimensional object with the second transform;

the first metadata is independent of the position of the displayed at least one two-dimensional object on the display; and

the second metadata is independent of the position of the displayed at least one three-dimensional object on the display.

36. A non-transitory machine-readable medium comprising machine program logic recorded thereon for:

directing with an electronic device a sensing mechanism of the electronic device to capture a first image of a user of the electronic device within a field of view of the sensing mechanism;

detecting with the electronic device a first position of the user with respect to a component of the electronic device from the captured first image;

in response to detecting the first position, concurrently displaying with the electronic device;

a three-dimensional object from a first perspective, wherein at least a portion of the three-dimensional object from the first perspective comprises a first reflective surface and at least a portion of the captured first image is applied to at least a portion of the first reflective surface when the three-dimensional object from the first perspective is displayed; and

a first depiction comprising a window and an element at a first element position within the window, wherein the first element position of the element with respect to the boundaries of the window is based on the detected first position of the user with respect to the component of the electronic device;

once the three-dimensional object is displayed from the first perspective, directing with the electronic device the sensing mechanism to capture a second image of the user within the field of view of the sensing mechanism;

detecting with the electronic device a second position of the user with respect to the component of the electronic device from the captured second image, wherein the detected second position is different than the detected first position;

in response to detecting the second position, transforming with the electronic device the displayed three-dimensional object by;

changing the displayed three-dimensional object from the first perspective to a second perspective based on a difference between the first position and the second position, wherein at least a portion of the changed three-dimensional object comprises a second reflective surface; and

applying at least a portion of the captured second image to at least a portion of the second reflective surface;

concurrently displaying with the electronic device;

the transformed three-dimensional object; and

a second depiction comprising the window and the element at a second element position within the window, wherein the second element position of the element with respect to the boundaries of the window is based on the detected second position of the user with respect to the component of the electronic device;

detecting that a position of the user within the field of view of the sensing mechanism has not changed more than a defined threshold amount within a specific period of time; and

removing any depiction of the window from the display when it has been detected that the position of the user within the field of view of the sensing mechanism has not changed more than the defined threshold amount within the specific period of time.

37. The non-transitory machine-readable medium of claim 36, comprising additional machine program logic recorded thereon for:

after the displaying the transformed three-dimensional object, directing with the electronic device the sensing mechanism to capture a third image of the user;

detecting with the electronic device a third position of the user with respect to the electronic device from the captured third image;

in response to detecting the third position, re-transforming with the electronic device the transformed three-dimensional object by;

re-changing the transformed three-dimensional object from the second perspective to a third perspective based on a difference between the second position and the third position, wherein at least a portion of the re-changed three-dimensional object comprises a third reflective surface; and

applying at least a portion of the captured third image to at least a portion of the third reflective surface; and

displaying with the electronic device the re-transformed three-dimensional object.

38. The non-transitory machine-readable medium of claim 36, comprising additional machine program logic recorded thereon for:

displaying with the electronic device another three-dimensional object;

transforming with the electronic device the displayed another three-dimensional object based on the detected second position; and

displaying with the electronic device the transformed another three-dimensional object.

39. The non-transitory machine-readable medium of claim 38, comprising additional machine program logic recorded thereon for:

transforming with the electronic device the displayed three-dimensional object by performing a first type of transform; and

transforming with the electronic device the displayed another three-dimensional object by performing a second type of transform.