FULL 3D INTERACTION ON MOBILE DEVICES

• US 20140245230A1
• Filed: 12/27/2011
• Published: 08/28/2014
• Est. Priority Date: 12/27/2011
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Durham, Lenitra M., Durham, David, Sharma, Sangita
• Granted Patent: US 9,335,888 B2
• Time in Patent Office: Days
• US Class Current: 715/848
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0486 Drag-and-drop

Abstract

Systems and methods may provide for displaying a three-dimensional (3D) environment on a screen of a mobile device, and identifying a user interaction with an area behind the mobile device. In addition, the 3D environment can be modified based at least in part on the first user interaction. Moreover, the 3D environment may be modified based on movements of the mobile device as well as user interactions with the mobile device, allowing the user to navigate through the virtual 3D environment by moving the mobile/handheld device.

Claims

1. A mobile device comprising:

a screen; and

logic to,

display a three-dimensional (3D) environment via the screen,

identify a first user interaction with an area behind the mobile device, and

modify the 3D environment based at least in part on the first user interaction.

2. The mobile device of claim 1, wherein the first user interaction is to include a hand movement relative to at least one of an in-air location and an external surface associated with a virtual object in the 3D environment.

3. The mobile device of claim 1, further including an air nozzle, wherein the logic is to use the air nozzle to provide tactile feedback if the first user interaction is a hand movement relative to an in-air location.

4. The mobile device of claim 1, further including a rear image sensor, wherein the logic is to use one or more signals from the rear image sensor to identify the first user interaction.

5. The mobile device of claim 1, further including a microphone, wherein the logic is to, use one or more signals from the microphone to identify the first user interaction.

6. The mobile device of claim 1, wherein the logic is to perform one or more of a cursor movement operation, a click operation, a drag and drop operation, a pinch operation, a selection operation and an object rotation operation to modify the 3D environment based at least in part on the first user interaction.

7. The mobile device of claim 1, further including a motion sensor, wherein the logic is to use one or more signals from the motion sensor to identify a movement of the mobile device.

8. The mobile device of claim 1, further including a motion sensor, wherein the logic is to use one or more signals from the front image sensor to identify a movement of the mobile device.

9. The mobile device of claim 1, wherein the 3D environment is to include a virtual desktop computer environment that is substantially larger than the screen of the mobile device.

10. The mobile device of claim 1, wherein the logic is to,

use one or more signals from a user interface of the mobile device to identify a second user interaction with the mobile device, and

modify the 3D environment with at least one of a click operation, a drag and drop operation and a pinch operation based at least in part on the second user interaction.

11. An apparatus comprising:

logic to,

display a three-dimensional (3D) environment via a screen of a mobile device,

identify a first user interaction with an area behind the mobile device, and

modify the 3D environment based at least in part on the first user interaction.

12. The apparatus of claim 11, wherein the first user interaction is to include a hand movement relative to at least one of an in-air location and an external surface associated with a virtual object in the 3D environment.

13. The apparatus of claim 11, wherein the logic is to use one or more signals from at least one of a rear image sensor and a microphone of the mobile device to identify the first user interaction.

14. The apparatus of claim 11, wherein the logic is to use an air nozzle of the mobile device to provide tactile feedback if the first user interaction is a hand movement relative to an in-air location.

15. The apparatus of claim 11, wherein the logic is to perform one or more of a cursor movement operation, a click operation, a drag and drop operation, a pinch operation, a selection operation and an object rotation operation to modify the 3D environment based at least in part on the first user interaction.

16. The apparatus of claim 11, wherein the logic is to,

use one or more signals from at least one of a motion sensor and a front image sensor of the mobile device to identify a movement of the mobile device, and

modify the 3D environment with at least one of a pan operation and a zoom operation based at least in part on the movement of the mobile device.

17. The apparatus of claim 11, wherein the 3D environment is to include a virtual desktop computer environment that is substantially larger than the screen of the mobile device.

18. The apparatus of claim 11, wherein the logic is to,

use one or more signals from a user interface of the mobile device to identify a second user interaction with the mobile device, and

modify the 3D environment with at least one of a click operation, a drag and drop operation and a pinch operation based at least in part on the second user interaction.

19. The apparatus of claim 18, wherein the click operation is to include a computer mouse click operation.

20. A non-transitory computer readable storage medium comprising a set of instructions which, if executed by a processor, cause a mobile device to:

display a three-dimensional (3D) environment via a screen of the mobile device;

identify a first user interaction with an area behind the mobile device; and

modify the 3D environment based at least in part on the first user interaction.

21. The medium of claim 20, wherein the first user interaction is to include a hand movement relative to at least one of an in-air location and an external surface associated with a virtual object in the 3D environment.

22. The medium of claim 20, wherein the instructions, if executed, cause the mobile device to use one or more signals from at least one of a rear image sensor and a microphone of the mobile device to identify the first user interaction.

23. The medium of claim 20, wherein the instructions, if executed, cause the mobile device to use an air nozzle of the mobile device to provide tactile feedback if the first user interaction is a hand movement relative to an in-air location.

24. The medium of claim 20, wherein the instructions, if executed, cause the mobile device to perform one or more of a cursor movement operation, a click operation, a drag and drop operation, a pinch operation, a selection operation and an object rotation operation to modify the 3D environment based on at least in part the first user interaction.

25. The medium of claim 20, wherein the instructions, if executed, cause the mobile device to:

use one or more signals from at least one of a motion sensor and a front image sensor of the mobile device to identify a movement of the mobile device; and

modify the 3D environment with at least one of a pan operation and a zoom operation based at least in part on the movement of the mobile device.

26. The medium of claim 20, wherein the 3D environment is to include a virtual desktop computer environment that is substantially larger than the screen of the mobile device.

27. The medium of claim 20, wherein the instructions, if executed, cause the mobile device to:

use one or more signals from a user interface of the mobile device to identify a second user interaction with the mobile device; and

modify the 3D environment with at least one of a click operation, a drag and drop operation and a pinch operation based at least in part on the second user interaction.

28. A computer implemented method comprising:

displaying a three-dimensional (3D) environment via a screen of a mobile device;

identifying a first user interaction with an area behind the mobile device; and

modifying the 3D environment based at least in part on the first user interaction.

29. The method of claim 28, wherein the first user interaction includes a hand movement relative to at least one of an in-air location and an external surface associated with a virtual object in the 3D environment.

30. The method of claim 28, further including:

using one or more signals from at least one of a rear image sensor and a microphone of the mobile device to identify the first user interaction; and

using an air nozzle of the mobile device to provide tactile feedback if the first user interaction is a hand movement relative to the in-air location.

31. The method of claim 28, further including:

using one or more signals from at least one of a rear image sensor and a microphone of the mobile device to identify the first user interaction; and

using an air nozzle of the mobile device to provide tactile feedback if the first user to interaction is a hand movement relative to the in-air location.

32. The method of claim 28, wherein modifying the 3D environment includes performing one or more of a cursor movement operation, a click operation, a drag and drop operation, a pinch operation, a selection operation and an object rotation operation.

33. The method of claim 28, further including;

use one or more signals from at least one of a motion sensor and a front image sensor of the mobile device identifying a movement of the mobile device; and

modifying the 3D environment with at least one of a pan operation and a zoom operation based at least in part on the movement of the mobile device.

34. The method of claim 28, wherein the 3D environment includes a virtual desktop computer environment that is substantially larger than the screen of the mobile device.

35. The method of claim 28, further including:

use one or more signals from a user interface of the mobile device to identify a second user interaction with the mobile device; and

modifying the 3D environment with at least one of a click operation, a drag and drop operation and a pinch operation based at least in part on the second user interaction.