Full 3D interaction on mobile devices

• US 9,335,888 B2
• Filed: 12/27/2011
• Issued: 05/10/2016
• Est. Priority Date: 12/27/2011
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Durham, Lenitra M., Durham, David, Sharma, Sangita
• Primary Examiner(s): Alvesteffer, Stephen
• Application Number:
  US13/996,088
• Publication Number:
  US 20140245230A1
• Time in Patent Office: 1,596 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0486 Drag-and-drop

Abstract

Systems and methods may provide for displaying a three-dimensional (3D) environment on a screen of a mobile device, and identifying a user interaction with an area behind the mobile device. In addition, the 3D environment can be modified based at least in part on the first user interaction. Moreover, the 3D environment may be modified based on movements of the mobile device as well as user interactions with the mobile device, allowing the user to navigate through the virtual 3D environment by moving the mobile/handheld device.

Claims

1. A mobile device comprising:

a screen;

a sensor, wherein the sensor is to include one or more of a motion sensor and a front image sensor; and

logic to,

display a three-dimensional (3D) environment via the screen,

identify a first user interaction with an area behind the mobile device that is to include a selection of one or more of a region of interest and a virtual object in the 3D environment,

use one or more signals from at least one of the motion sensor and the front image sensor to identify a movement of the mobile device in response to the selection, and

modify the 3D environment based at least in part on the first user interaction that is to include the selection and the movement of the mobile device in response to the selection.

2. The mobile device of claim 1, wherein the first user interaction is to include a hand movement that is to touch at least one of an in-air location associated with one or more of the region of interest and the virtual object when the 3D environment is an in-air environment and an external surface associated with one or more of the region of interest and the virtual object when the 3D environment is pinned to the external surface.

3. The mobile device of claim 1, further including an air nozzle, wherein the logic is to use the air nozzle to provide tactile feedback if the first user interaction is a hand movement relative to an in-air location.

4. The mobile device of claim 1, further including a rear image sensor, wherein the logic is to use one or more signals from the rear image sensor to identify the first user interaction.

5. The mobile device of claim 1, further including a microphone, wherein the logic is to use one or more signals from the microphone to identify the first user interaction.

6. The mobile device of claim 1, wherein the logic is to perform one or more of a cursor movement operation, a click operation, a drag and drop operation, a pinch operation, a selection operation and an object rotation operation to modify the 3D environment based at least in part on the first user interaction.

7. The mobile device of claim 1, wherein the logic is to,

use one or more signals from the motion sensor to identify a movement of the mobile device including a tilt of the mobile device with respect to one or more of the region of interest and the virtual object to pan and a change in distance with respect to one or more of the region of interest and the virtual object to zoom, and

use one or more signals from the front image sensor to identify a movement of the mobile device including a tilt of the mobile device with respect to a user that is to view the 3D environment via the screen to pan and a change in distance with respect to the user that is to view the 3D environment via the screen to zoom.

8. The mobile device of claim 1, wherein the 3D environment is to include a virtual desktop computer environment that is substantially larger than the screen of the mobile device.

9. The mobile device of claim 1, wherein the sensor is to include a user interface of the mobile device, and wherein the logic is to,

use one or more signals from the user interface of the mobile device to identify a movement of the user interface of the mobile device in response to the selection, and

modify the 3D environment based at least in part on the first user interaction that is to include the selection and the movement of the user interface of the mobile device in response to the selection.

10. An apparatus comprising:

logic to,

display a three-dimensional (3D) environment via a screen of a mobile device,

identify a first user interaction with an area behind the mobile device that is to include a selection of one or more of a region of interest and a virtual object in the 3D environment,

use one or more signals from at least one of a motion sensor and a front image sensor of the mobile device to identify a movement of the mobile device in response to the selection, and

modify the 3D environment based at least in part on the first user interaction that is to include the selection and the movement of the mobile device in response to the selection.

11. The apparatus of claim 10, wherein the first user interaction is to include a hand movement that is to touch at least one of an in-air location associated with one or more of the region of interest and the virtual object when the 3D environment is an in-air environment and an external surface associated with one or more of the region of interest and the virtual object when the 3D environment is pinned to the external surface.

12. The apparatus of claim 10, wherein the logic is to use one or more signals from at least one of a rear image sensor and a microphone of the mobile device to identify the first user interaction.

13. The apparatus of claim 10, wherein the logic is to use an air nozzle of the mobile device to provide tactile feedback if the first user interaction is a hand movement relative to an in-air location.

14. The apparatus of claim 10, wherein the logic is to perform one or more of a cursor movement operation, a click operation, a drag and drop operation, a pinch operation, a selection operation and an object rotation operation to modify the 3D environment based at least in part on the first user interaction.

15. The apparatus of claim 10, wherein the 3D environment is to include a virtual desktop computer environment that is substantially larger than the screen of the mobile device.

16. The apparatus of claim 10, wherein the logic is to,

use one or more signals from a user interface of the mobile device to identify a movement of the user interface of the mobile device in response to the selection, and

modify the 3D environment based at least in part on the first user interaction that is to include the selection and the movement of the user interface of the mobile device in response to the selection.

17. The apparatus of claim 16, wherein the click operation is to include a computer mouse click operation.

18. A non-transitory computer readable storage medium comprising a set of instructions which, if executed by a processor, cause a mobile device to:

display a three-dimensional (3D) environment via a screen of the mobile device;

identify a first user interaction with an area behind the mobile device that is to include a selection of one or more of a region of interest and a virtual object in the 3D environment;

use one or more signals from at least one of a motion sensor and a front image sensor of the mobile device to identify a movement of the mobile device in response to the selection; and

modify the 3D environment based at least in part on the first user interaction that is to include the selection and the movement of the mobile device in response to the selection.

19. The medium of claim 18, wherein the first user interaction is to include a hand movement that is to touch at least one of an in-air location associated with one or more of the region of interest and the virtual object when the 3D environment is an in-air environment and an external surface associated with one or more of the region of interest and the virtual object when the 3D environment is pinned to the external surface.

20. The medium of claim 18, wherein the instructions, if executed, cause the mobile device to use one or more signals from at least one of a rear image sensor and a microphone of the mobile device to identify the first user interaction.

21. The medium of claim 18, wherein the instructions, if executed, cause the mobile device to use an air nozzle of the mobile device to provide tactile feedback if the first user interaction is a hand movement relative to an in-air location.

22. The medium of claim 18, wherein the instructions, if executed, cause the mobile device to perform one or more of a cursor movement operation, a click operation, a drag and drop operation, a pinch operation, a selection operation and an object rotation operation to modify the 3D environment based on at least in part the first user interaction.

23. The medium of claim 18, wherein the 3D environment is to include a virtual desktop computer environment that is substantially larger than the screen of the mobile device.

24. The medium of claim 18, wherein the instructions, if executed, cause the mobile device to:

use one or more signals from a user interface of the mobile device to identify a movement of the user interface of the mobile device in response to the selection; and

modify the 3D environment based at least in part on the first user interaction that is to include the selection and the movement of the user interface of the mobile device in response to the selection.

25. A computer implemented method comprising:

displaying a three-dimensional (3D) environment via a screen of a mobile device;

identifying a first user interaction with an area behind the mobile device including a selection of one or more of a region of interest and a virtual object in the 3D environment;

using one or more signals from at least one of a motion sensor and a front image sensor of the mobile device to identify a movement of the mobile device in response to the selection; and

modifying the 3D environment based at least in part on the first user interaction including the selection and the movement of the mobile device in response to the selection.

26. The method of claim 25, wherein the first user interaction includes a hand movement touching at least one of an in-air location associated with one or more of the region of interest and the virtual object when the 3D environment is an in-air environment and an external surface associated with one or more of the region of interest and the virtual object when the 3D environment is pinned to the external surface.

27. The method of claim 25, further including:

using one or more signals from at least one of a rear image sensor and a microphone of the mobile device to identify the first user interaction; and

using an air nozzle of the mobile device to provide tactile feedback if the first user interaction is a hand movement relative to the in-air location.

28. The method of claim 25, wherein modifying the 3D environment includes performing one or more of a cursor movement operation, a click operation, a drag and drop operation, a pinch operation, a selection operation and an object rotation operation.

29. The method of claim 25, wherein the 3D environment includes a virtual desktop computer environment that is substantially larger than the screen of the mobile device.

30. The method of claim 25, further including:

using one or more signals from a user interface of the mobile device to identify movement of the user interface of the mobile device in response to the selection; and

modifying the 3D environment based at least in part on the first user interaction including the selection and the movement of the user interface of the mobile device in response to the selection.

31. A mobile device comprising:

a screen;

a sensor, wherein the sensor is to include a user interface of the mobile device; and

logic to,

display a three-dimensional (3D) environment via the screen,

identify a first user interaction with an area behind the mobile device that is to include a selection of one or more of a region of interest and a virtual object in the 3D environment,

use one or more signals from the user interface of the mobile device to identify a movement of the user interface of the mobile device in response to the selection, and

modify the 3D environment based at least in part on the first user interaction that is to include the selection and the movement of the user interface of the mobile device in response to the selection.

32. An apparatus comprising:

logic to,

display a three-dimensional (3D) environment via a screen of a mobile device,

identify a first user interaction with an area behind the mobile device that is to include a selection of one or more of a region of interest and a virtual object in the 3D environment,

use one or more signals from a user interface of the mobile device to identify a movement of the user interface of the mobile device in response to the selection, and

modify the 3D environment based at least in part on the first user interaction that is to include the selection and the movement of the user interface of the mobile device in response to the selection.

33. A non-transitory computer readable storage medium comprising a set of instructions which, if executed by a processor, cause a mobile device to:

display a three-dimensional (3D) environment via a screen of the mobile device;

identify a first user interaction with an area behind the mobile device that is to include a selection of one or more of a region of interest and a virtual object in the 3D environment;

use one or more signals from a user interface of the mobile device to identify a movement of the user interface of the mobile device in response to the selection; and

modify the 3D environment based at least in part on the first user interaction that is to include the selection and the movement of the user interface of the mobile device in response to the selection.

34. A computer implemented method comprising:

displaying a three-dimensional (3D) environment via a screen of a mobile device;

identifying a first user interaction with an area behind the mobile device including a selection of one or more of a region of interest and a virtual object in the 3D environment;

using one or more signals from a user interface of the mobile device to identify movement of the user interface of the mobile device in response to the selection; and

modifying the 3D environment based at least in part on the first user interaction including the selection and the movement of the user interface of the mobile device in response to the selection.