Three dimensional user interface effects on a display

• US 9,411,413 B2
• Filed: 07/11/2014
• Issued: 08/09/2016
• Est. Priority Date: 08/04/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Motta, Ricardo, Zimmer, Mark, Stahl, Geoff, Hayward, David, Doepke, Frank
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US14/329,777
• Publication Number:
  US 20150009130A1
• Time in Patent Office: 760 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]

Abstract

The techniques disclosed herein may use various sensors to infer a frame of reference for a hand-held device. In fact, with various inertial clues from accelerometer, gyrometer, and other instruments that report their states in real time, it is possible to track a Frenet frame of the device in real time to provide an instantaneous (or continuous) 3D frame-of-reference. In addition to—or in place of—calculating this instantaneous (or continuous) frame of reference, the position of a user'"'"'s head may either be inferred or calculated directly by using one or more of a device'"'"'s optical sensors, e.g., an optical camera, infrared camera, laser, etc. With knowledge of the 3D frame-of-reference for the display and/or knowledge of the position of the user'"'"'s head, more realistic virtual 3D depictions of the graphical objects on the device'"'"'s display may be created—and interacted with—by the user.

Claims

1. A graphical user interface method, comprising:

receiving optical data from one or more optical sensors disposed within a device, wherein the optical data comprises one or more of;

two-dimensional image data, stereoscopic image data, structured light data, depth map data, and Lidar data;

receiving non-optical data from one or more non-optical sensors;

determining a position of a user of the device'"'"'s head based, at least in part, on the received optical data and the received non-optical data;

generating a virtual 3D depiction of at least part of a graphical user interface on a display of the device; and

applying an appropriate perspective transformation to the virtual 3D depiction of the at least part of the graphical user interface on the display of the device,

wherein the acts of generating and applying are based, at least in part, on the determined position of the user of the device'"'"'s head, the received optical data, and the received non-optical data, and

wherein the at least part of the graphical user interface is represented in a virtual 3D operating system environment.

2. The graphical user interface method of claim 1, wherein at least one of the one or more optical sensors comprises:

a front-facing camera, an image sensor, a two-dimensional camera, a stereoscopic camera, an infrared camera, proximity sensor, video camera, or a laser.

3. The graphical user interface method of claim 1, wherein the one or more non-optical sensors do not sense visible light.

4. The graphical user interface method of claim 1, wherein the non-optical data comprises one or more of:

data related to a position of the user of the device'"'"'s head;

data transmitted by electromagnetic or electrostatic fields;

data transmitted by electromagnetic radiation; and

data transmitted by radio frequency waves.

5. The graphical user interface method of claim 1, wherein at least one of the one or more non-optical sensors comprises:

eyewear or headgear worn by a user of the device, or positional sensors capable of providing position information related to the user.

6. The graphical user interface method of claim 1, further comprising the act of:

receiving data from one or more sensors external to the device,

wherein the act of determining a position of a user of the device'"'"'s head is further based, at least in part, on the received data from the one or more sensors external to the device, and

wherein the act of generating is further based, at least in part, on the received data from the one or more sensors external to the device.

7. The graphical user interface method of claim 1, wherein the device comprises at least one of:

a computer monitor, a watch, a music player, a television screen, or a dashboard display in a car or other vehicle.

8. A device, comprising:

a display;

one or more optical sensors;

one or more positional sensors;

a memory; and

one or more programmable control devices communicatively coupled to the display, the optical sensors, the positional sensors, and the memory,

wherein the memory includes instructions for causing the one or more programmable control devices to;

receive optical data from the one or more optical sensors, wherein the optical data comprises one or more of;

two-dimensional image data, stereoscopic image data, structured light data, depth map data, and Lidar data;

receive non-optical data from one or more non-optical sensors;

determine a position of a user of the device'"'"'s head based, at least in part, on the received optical data and the received non-optical data;

generate a virtual 3D depiction of at least part of a graphical user interface on the display; and

apply an appropriate perspective transformation to the virtual 3D depiction of the at least part of the graphical user interface on the display of the device,

wherein the instructions to generate and apply are based, at least in part, on the determined position of the user of the device'"'"'s head, the received optical data, and the received non-optical data, and

wherein the at least part of the graphical user interface is represented in a virtual 3D operating system environment.

9. The device of claim 8, wherein at least one of the one or more optical sensors comprises:

a front-facing camera, an image sensor, a two-dimensional camera, a stereoscopic camera, an infrared camera, proximity sensor, video camera, or a laser.

10. The device of claim 8, wherein the one or more non-optical sensors do not sense visible light.

11. The device of claim 8, wherein the non-optical data comprises one or more of:

data related to a position of the user of the device'"'"'s head;

data transmitted by electromagnetic or electrostatic fields;

data transmitted by electromagnetic radiation; and

data transmitted by radio frequency waves.

12. The device of claim 8, wherein at least one of the one or more non-optical sensors comprises:

eyewear or headgear worn by a user of the device, or positional sensors capable of providing position information related to the user.

13. The device of claim 8, wherein the memory further includes instructions for causing the one or more programmable control devices to:

receive data from one or more sensors external to the device,

wherein the instructions to determine a position of a user of the device'"'"'s head further comprise instructions to determine the position of the user of the device'"'"'s head based, at least in part, on the received data from the one or more sensors external to the device, and

wherein the instructions to generate a virtual 3D depiction of at least part of a graphical user interface on the display further comprise instructions to generate the virtual 3D depiction of at least part of the graphical user interface on the display based, at least in part, on the received data from the one or more sensors external to the device.

14. The device of claim 8, wherein the device comprises at least one of:

a computer monitor, a watch, a music player, a television screen, or a dashboard display in a car or other vehicle.

15. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

receive optical data from one or more optical sensors in a device, wherein the optical data comprises one or more of;

two-dimensional image data, stereoscopic image data, structured light data, depth map data, and Lidar data;

receive non-optical data from one or more non-optical sensors;

determine a position of a user of the device'"'"'s head based, at least in part, on the received optical data and the received non-optical data; and

generate a virtual 3D depiction of at least part of a graphical user interface on a display of the device; and

apply an appropriate perspective transformation to the virtual 3D depiction of the at least part of the graphical user interface on the display of the device,

wherein the instructions to generate and apply are based, at least in part, on the determined position of the user of the device'"'"'s head, the received optical data, and the received non-optical data, and

wherein the at least part of the graphical user interface is represented in a virtual 3D operating system environment.

16. The non-transitory program storage device of claim 15, wherein at least one of the one or more optical sensors comprises:

a front-facing camera, an image sensor, a two-dimensional camera, a stereoscopic camera, an infrared camera, proximity sensor, video camera, or a laser.

17. The non-transitory program storage device of claim 15, wherein the one or more non-optical sensors do not sense visible light.

18. The non-transitory program storage device of claim 15, wherein the non-optical data comprises one or more of:

data related to a position of the user of the device'"'"'s head;

data transmitted by electromagnetic or electrostatic fields;

data transmitted by electromagnetic radiation; and

data transmitted by radio frequency waves.

19. The non-transitory program storage device of claim 15, wherein at least one of the one or more non-optical sensors comprises:

eyewear or headgear worn by a user of the device, or positional sensors capable of providing position information related to the user.

20. The non-transitory program storage device of claim 15, wherein the instructions further comprise instructions for causing the one or more processors to:

receive data from one or more sensors external to the device,

wherein the instructions to determine a position of a user of the device'"'"'s head further comprise instructions to determine the position of the user of the device'"'"'s head based, at least in part, on the received data from the one or more sensors external to the device, and

wherein the instructions to generate a virtual 3D depiction of at least part of a graphical user interface on the display further comprise instructions to generate the virtual 3D depiction of at least part of the graphical user interface on the display based, at least in part, on the received data from the one or more sensors external to the device.