IN-LINE MEDIATION FOR MANIPULATING THREE-DIMENSIONAL CONTENT ON A DISPLAY DEVICE

• US 20100053151A1
• Filed: 04/09/2009
• Published: 03/04/2010
• Est. Priority Date: 09/02/2008
• Status: Abandoned Application

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Marti, Stefan, Kim, Seung Wook, Imai, Francisco
• Application Number:
  US12/421,363
• Publication Number:
  US 20100053151A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G09G 2356/00 Detection of the display position w.r.t. other display screens
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: H04M 2203/359 Augmented reality
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking

Abstract

A user holds the mobile device upright or sits in front of a nomadic or stationary device, views the monitor from a suitable distance, and physically reaches behind the device with her hand to manipulate a 3D object displayed on the monitor. The device functions as a 3D in-line mediator that provides visual coherency to the user when she reaches behind the device to use hand gestures and movements to manipulate a perceived object behind the device and sees that the 3D object on the display is being manipulated. The perceived object that the user manipulates behind the device with bare hands corresponds to the 3D object displayed on the device. The visual coherency arises from the alignment of the user'"'"'s head or eyes, the device, and the 3D object. The user'"'"'s hand may be represented as an image of the actual hand or as a virtualized representation of the hand, such as part of an avatar.

Claims

1. A method of detecting manipulation of a digital 3D object displayed on a device having a front side with a display and a back side, the method comprising:

detecting a hand within a specific area of the back side of the device, the back side having an sensor;

displaying the hand on the display;

tracking movement of the hand within the specific area of the back side, wherein said movement is caused by a user intending to manipulate the displayed 3D object;

detecting a collision between the displayed hand and the displayed 3D object; and

modifying an image of the 3D object displayed on the device,

wherein the device is a 3D in-line mediator between the user and the 3D object.

2. A method as recited in claim 1 further comprising:

detecting a hand gesture within the specific area.

3. A method as recited in claim 1 wherein modifying an image of the 3D object further comprises:

deforming the image of the 3D object.

4. A method as recited in claim 1 wherein modifying an image of the 3D object further comprises:

moving the image of the 3D object.

5. A method as recited in claim 1 further comprising:

displaying the modified image on the device.

6. A method as recited in claim 1 wherein the user reaches behind the device to manipulate a perceived object corresponding to the 3D object, such that the hand is within the specific area of the back side of the device.

7. A method as recited in claim 6 further comprising:

providing the user with visual coherency when the user reaches behind the device.

8. A method as recited in claim 1 wherein tracking movement of the hand further comprises:

processing depth data of said hand movement.

9. A method as recited in claim 1 further comprising:

executing tracking software.

10. A method as recited in claim 1 wherein the sensor is a tracking component that faces outward from the back side of the device and wherein the sensor is a camera.

11. A method as recited in claim 1 wherein displaying the hand further comprising:

displaying a composited image of the hand on the display.

12. A method as recited in claim 1 wherein displaying the hand further comprising:

displaying a virtualized image of the hand on the display.

13. A method as recited in claim 1 further comprising:

providing haptic feedback to the hand when a collision is detected between the displayed hand and the 3D object.

14. A method as recited in claim 1 wherein there is no contact between the hand and the back side of the device or with the display.

15. A device having a display, the device comprising:

a processor;

a memory storing digital 3D content data;

a tracking sensor component for tracking movement of an object in proximity of the device, wherein the tracking sensor component is on a back side of the device facing away from a user;

a hand tracking module for processing movement data related to a user hand; and

a hand-3D object collision module for detecting a collision between the user hand and a 3D object.

16. A device as recited in claim 15 further comprising:

a face tracking sensor component for tracking face movement in proximity of a front side of a device; and

a face tracking module for processing face movement data related to user face movement in front of the device.

17. A device as recited in claim 15 further comprising:

a hand gesture detection module for detecting user hand gestures made within range of the tracking sensor component.

18. A device as recited in claim 15 further comprising:

a tactile feedback controller for providing tactile feedback to the user hand.

19. A device as recited in claim 15 wherein a tracking sensor component is a camera-based component.

20. A device as recited in claim 15 wherein a tracking sensor component is one of an image differentiator, infrared detector, optic flow component, and spectral processor.

21. A device as recited in claim 15 wherein the tracking sensor component tracks movements of the hand when the user moves the hand behind the device with the range of the tracking sensor component.

22. A device as recited in claim 15 wherein the device is one of a mobile device, a nomadic device, and a stationary device.

23. A device as recited in claim 15 further comprising a network interface for connecting to a network to receive digital 3D content data.

24. An apparatus for manipulating digital 3D content, the apparatus having a front side with a display and a back side, the apparatus comprising:

means for detecting a hand within a specific area of the back side of the apparatus, the back side having an sensor;

means for displaying the hand on the apparatus;

means for tracking movement of the hand within the specific area of the back side, wherein said movement is caused by a user intending to manipulate a displayed 3D object;

means for detecting a collision between the displayed hand and the displayed 3D object; and

means for modifying an image of the 3D object displayed on the apparatus,

wherein the apparatus is a 3D in-line mediator between the user and the 3D object.

25. An apparatus as recited in claim 24 further comprising:

means for detecting a hand gesture within the specific area.

26. An apparatus as recited in claim 24 wherein means for modifying an image of the 3D object further comprises:

means for moving the image of the 3D object.

27. An apparatus as recited in claim 24 further comprising:

means for displaying the modified image on the apparatus.

28. An apparatus as recited in claim 24 wherein the user reaches behind the apparatus to manipulate a perceived object corresponding to the 3D object, such that the hand is within the specific area of the back side of the apparatus.

29. An apparatus as recited in claim 24 wherein means for tracking movement of the hand further comprises:

means for processing depth data of said hand movement.

30. An apparatus as recited in claim 24 wherein the sensor is a tracking component that faces outward from the back side of the apparatus and wherein the sensor is a camera.

31. An apparatus as recited in claim 24 wherein means for displaying the hand further comprises:

means for displaying a composited image of the hand on the apparatus.

32. An apparatus as recited in claim 24 further comprising:

means for providing haptic feedback to the hand when a collision is detected between the displayed hand and the 3D object.