Three Dimensional User Interface Effects on a Display by Using Properties of Motion

• US 20120036433A1
• Filed: 08/04/2010
• Published: 02/09/2012
• Est. Priority Date: 08/04/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Zimmer, Mark, Stahl, Geoff, Hayward, David, Doepke, Frank
• Granted Patent: US 8,913,056 B2
• Time in Patent Office: Days
• US Class Current: 715/702
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 15/20 Perspective computation

Abstract

The techniques disclosed herein use a compass, MEMS accelerometer, GPS module, and MEMS gyrometer to infer a frame of reference for a hand-held device. This can provide a true Frenet frame, i.e., X- and Y-vectors for the display, and also a Z-vector that points perpendicularly to the display. In fact, with various inertial clues from accelerometer, gyrometer, and other instruments that report their states in real time, it is possible to track the Frenet frame of the device in real time to provide a continuous 3D frame-of-reference. Once this continuous frame of reference is known, the position of a user'"'"'s eyes may either be inferred or calculated directly by using a device'"'"'s front-facing camera. With the position of the user'"'"'s eyes and a continuous 3D frame-of-reference for the display, more realistic virtual 3D depictions of the objects on the device'"'"'s display may be created and interacted with by the user.

Claims

1. A graphical user interface method, comprising:

receiving positional data from one or more position sensors disposed within a device;

determining a 3D frame of reference for the device based at least in part on the received positional data;

receiving optical data from one or more optical sensors disposed within the device;

determining a position of a user'"'"'s eyes based at least in part on the received optical data; and

generating a virtual 3D depiction of at least one graphical user interface object on a display of the device,

wherein the at least one graphical user interface object is represented in a virtual 3D operating system environment, and

wherein the act of generating is based at least in part on the determined 3D frame of reference and the position of the user'"'"'s eyes.

2. The graphical user interface method of claim 1, wherein the act of generating further comprises ray tracing from the user'"'"'s eyes through the display of the device and into the virtual 3D operating system environment.

3. The graphical user interface method of claim 1, wherein the act of generating further comprises applying an appropriate perspective transformation to the virtual 3D depiction of the at least one graphical user interface object on the display of the device, wherein the transformation is based at least in part on the determined 3D frame of reference and the determined position of the user'"'"'s eyes.

4. The graphical user interface method of claim 1, further comprising the act of receiving data from a touchscreen interface of the device, wherein the data received from the touchscreen interface is indicative of one or more locations in the virtual 3D operating system environment with which the user desires to interact.

5. The graphical user interface method of claim 4, further comprising the act of displaying a visual indicator at a location on the display of the device, wherein the location of the visual indicator is indicative of a location in the virtual 3D operating system environment with which the user desires to interact.

6. The graphical user interface method of claim 5, further comprising the act of correcting the location on the display of the device that the visual indicator is displayed for parallax problems, wherein the corrected location is based at least in part on the determined 3D frame of reference for the device.

7. The graphical user interface method of claim 1, further comprising the act of adjusting the generated virtual 3D depiction of the at least one graphical user interface object on the display of the device the display in response to movement of the device with respect to the position of the user.

8. The graphical user interface method of claim 7, wherein the act of adjusting further comprises applying an appropriate perspective transformation to the virtual 3D depiction of the at least one graphical user interface object on the display of the device, wherein the transformation is based at least in part on a current determined 3D frame of reference for the device and a current determined position of the user'"'"'s eyes.

9. The graphical user interface method of claim 1, further comprising the act of adjusting the generated virtual 3D depiction of the at least one graphical user interface object on the display of the device in response to movement of the user of the device with respect to the position of the device.

10. The graphical user interface method of claim 9, wherein the act of adjusting further comprises applying an appropriate perspective transformation to the virtual 3D depiction of the at least one graphical user interface object on the display of the device, wherein the transformation is based at least in part on a current determined 3D frame of reference for the device and a current determined position of the user'"'"'s eyes.

11. The graphical user interface method of claim 1, wherein the act of generating further comprises generating a virtual light source for the virtual 3D operating system environment.

12. The graphical user interface method of claim 11, wherein the generated virtual light source for the virtual 3D operating system environment comprises a representation of an ambient light source located in the user'"'"'s environment.

13. The graphical user interface method of claim 11, wherein the generated virtual light source for the virtual 3D operating system environment comprises a representation of a current location of the Sun.

14. A graphical user interface, comprising:

a viewing surface;

a virtual 3D operating system environment; and

one or more graphical user interface objects,

wherein the one or more graphical user interface objects are represented in the virtual 3D operating system environment and depicted on the viewing surface, and

wherein the depiction of the one or more graphical user interface objects on the viewing surface is determined at least in part by a determined 3D frame of reference of the viewing surface with respect to a user of the viewing surface and a determined location of eyes of the user with respect to the viewing surface.

15. The graphical user interface of claim 14, wherein the viewing surface comprises a touchscreen interface.

16. The graphical user interface of claim 15, wherein data received from the touchscreen interface is indicative of one or more locations in the virtual 3D operating system environment with which the user of the viewing surface desires to interact.

17. The graphical user interface of claim 16, further comprising one or more visual indicators on the viewing surface, wherein the one or more visual indicators are indicative of the one or more locations in the virtual 3D operating system environment with which the user of the viewing surface desires to interact.

18. The graphical user interface of claim 17, wherein the locations of the one or more visual indicators on the viewing surface are corrected for parallax problems, wherein a determination of the corrected locations is based at least in part on the determined 3D frame of reference of the viewing surface.

19. The graphical user interface of claim 14, wherein the depiction of the one or more graphical user interface objects on the viewing surface comprises the application of one or more of the following visual effects to the one or more graphical user interface objects:

shading, shine, blur, and alpha masking.

20. A graphical user interface method, comprising:

receiving positional data from one or more position sensors disposed within a device;

determining a 3D frame of reference for the device based at least in part on the received positional data;

receiving optical data from one or more optical sensors disposed within the device;

generating a depiction of a graphical user interface on a display of the device, wherein the graphical user interface comprises at least one graphical user interface object; and

applying one or more visual effects to at least one of the at least one graphical user interface objects,

wherein the one or more applied visual effects are based at least in part on the determined 3D frame of reference and the received optical data.

21. The graphical user interface method of claim 20, wherein the received optical data comprises data indicative of a position of a user'"'"'s eyes.

22. The graphical user interface method of claim 20, wherein the received optical data comprises data indicative of a position of an ambient light source located in the user'"'"'s environment.

23. The graphical user interface method of claim 21, wherein the graphical user interface comprises a “

bento box”

form factor.

24. The graphical user interface method of claim 21, wherein the one or more applied visual effects comprise a spotlight effect applied to at least one of the graphical user interface objects.

25. The graphical user interface method of claim 24, wherein the at least one of the graphical user interface objects to which a spotlight effect is applied is determined by the position of the user'"'"'s eyes.

26. The graphical user interface method of claim 20, wherein the one or more applied visual effects comprise one or more of the following:

shading, shine, blur, and alpha masking.

27. The graphical user interface method of claim 20, wherein the act of generating further comprises the act of simulating a light source shining upon the at least one graphical user interface object.

28. The graphical user interface method of claim 22, wherein the act of generating further comprises the act of simulating a light source shining upon the at least one graphical user interface object, wherein the simulated light source comprises a representation of the ambient light source located in the user'"'"'s environment.

29. The graphical user interface method of claim 27, wherein the act of simulating a light source further comprises repositioning a graphical layer with respect to the at least one graphical user interface object.

30. The graphical user interface method of claim 29, wherein the graphical layer comprises at least one of the following:

a shine map, and a shadow map.

31. A method, comprising:

generating a 2D depiction of a graphical user interface on a display of a device, wherein the graphical user interface comprises at least one graphical user interface object;

receiving positional data from one or more position sensors disposed within the device;

determining a 3D frame of reference for the device based at least in part on the received positional data;

receiving optical data from one or more optical sensors disposed within the device;

determining a position of a user of the device'"'"'s eyes based at least in part on the received optical data;

detecting an activating gesture based at least in part on the received positional data; and

generating a virtual 3D depiction of the graphical user interface on the display of the device in response to the detection of the activating gesture,

wherein the act of generating the virtual 3D depiction is based on the determined 3D frame of reference and the determined position of the user'"'"'s eyes.

32. The method of claim 31, further comprising the acts of:

detecting a deactivating gesture based at least in part on the received positional data; and

generating a 2D depiction of the graphical user interface in response to the detection of the deactivating gesture.

33. The method of claim 31, wherein the activating gesture comprises a one-axis sine wave modulation of the device of a sufficient number of degrees and within a predetermined threshold amount of time.

34. The method of claim 33, wherein the deactivating gesture comprises positional quiescence of the device for at least a predetermined threshold amount of time.

35. The method of claim 31, wherein the act of generating a virtual 3D depiction of the graphical user interface comprises transforming a 2D depiction of at least one graphical user interface object into a 3D depiction of the graphical user interface object.

36. The method of claim 35, wherein the 3D depiction of the graphical user interface object comprises a cube.

37. The method of claim 36, wherein the cube has at least one rounded surface.

38. An apparatus, comprising:

a display;

one or more optical sensors;

one or more positional sensors;

a memory; and

one or more programmable control devices communicatively coupled to the display, the optical sensors, the positional sensors, and the memory,

wherein the memory includes instructions for causing the one or more programmable control devices to;

receive positional data from the one or more position sensors;

determine a 3D frame of reference for the apparatus based at least in part on the received positional data;

receive optical data from the one or more optical sensors;

determine a position of a user of the apparatus'"'"'s eyes based at least in part on the received optical data; and

render a virtual 3D depiction of at least one graphical user interface object on the display,

wherein the instructions to render are based on the determined 3D frame of reference and the determined position of the user'"'"'s eyes.

39. The apparatus of claim 38, wherein the one or more position sensors comprise one or more of the following:

a compass, an accelerometer, a GPS module, and a gyrometer.

40. The apparatus of claim 38, wherein the one or more optical sensors comprise one or more of the following:

a proximity sensor, an image sensor, and a video camera.

41. The apparatus of claim 38, wherein the one or more optical sensors comprises a front-facing camera.

42. The apparatus of claim 41, wherein the instructions to determine a position of a user of the apparatus'"'"'s eyes comprise instructions to perform one or more of the following acts:

calculating the distance from the user'"'"'s eyes to the apparatus'"'"'s display, detecting the face of the user, calculating a distance between the user'"'"'s pupils, and locating an ambient light source in the user'"'"'s environment.

43. The apparatus of claim 38, wherein the apparatus comprises one or more of the following:

a mobile phone, a personal data assistant (PDA), a portable music player, a television, a gaming console, a portable gaming device, a laptop computer, a desktop computer, and a tablet computer.