INTERCOMMUNICATION BETWEEN A HEAD MOUNTED DISPLAY AND A REAL WORLD OBJECT

• US 20170061700A1
• Filed: 02/13/2015
• Published: 03/02/2017
• Est. Priority Date: 02/13/2015
• Status: Abandoned Application

• Current Assignee: OTOY, Inc.
• Original Assignee: OTOY, Inc.
• Inventors: Urbach, Julian Michael, Lazareff, Nicolas
• Application Number:
  US14/621,621
• Publication Number:
  US 20170061700A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06Q 20/123 Shopping for digital content
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

User interaction with virtual objects generated in virtual space on a first display device is enabled. Using sensor and camera data of the first display device, a real-world object with a marker on its surface is identified. Virtual objects are generated and displayed in the virtual 3D space relative to the marker on the real-world object. Manipulation of the real-world object in real 3D space results in changes to attributes of the virtual objects in the virtual 3D space. The marker comprises information regarding particular the renders to be generated. Different virtual objects can be generated and displayed based on information comprised in the markers. When the real world object has sensors, sensor data from the real-world object is transmitted to the first display device to enhance the display of the virtual object, or the virtual scene, based on sensor input. Local or remote storage can further define, enhance, or modify characteristics of the real world object.

Claims

1. ) A method comprising:

detecting, by a processor in communication with a first display device, presence of a real-world object comprising a marker on a surface thereof;

identifying, by the processor, position and orientation of the real-world object in real 3D space relative to a user'"'"'s eyes;

rendering, by the processor, a virtual object positioned and oriented in a virtual 3D space relative to the marker, and the virtual object configured for control in the virtual 3D space via manipulations of the real-world object in the real 3D space; and

transmitting render data, by the processor to the first display device, to visually present the virtual object in the virtual 3D space.

2. ) The method of claim 1, wherein the virtual object configured for control via manipulations of the real-world object further comprises:

detecting, by the processor, a change in one of the position and orientation of the real-world object.

3. ) The method of claim 2, further comprising:

altering, by the processor, one or more of a position and an orientation of the virtual object in the virtual space based on the detected change in the real-world object; and

transmitting, by the processor to the first display device, render data to visually display the virtual object at one or more of the altered positions and orientations based on the detected change.

4. ) The method of claim 1, the real world object is a second display device comprising a touchscreen, the first display device is communicatively coupled to a second display device, the coupling enabling exchange of data between the first display device and the second display device.

5. ) The method of claim 4, wherein the marker is detected on the touchscreen of the second display device.

6. ) The method of claim 4, further comprising:

receiving, by the processor, data regarding the user'"'"'s touch input from the second display device; and

manipulating, by the processor, the virtual object or a virtual scene in the virtual space in response to the data regarding the user'"'"'s touch input.

7. ) The method of claim 6, the data regarding the user'"'"'s touch input comprising position information of the user'"'"'s body part on the touchscreen relative to the marker.

8. ) The method of claim 7, the manipulation of the virtual object further comprising:

changing, by the processor, a position of the virtual object in the virtual space to track the position information.

9. ) The method of claim 6, the manipulation of the virtual object further comprising:

changing, by the processor, one or more of a size, shape, lighting and rendering properties of the virtual object in response to the user'"'"'s touch input.

10. ) The method of claim 9, wherein the user'"'"'s touch input corresponds to a gesture selected from a group of gestures consisting of a single or multi-tap, tap-and-hold, rotate, swipe, or pinch-zoom gesture.

11. ) The method of claim 4, further comprising:

receiving, by the processor, data regarding input from at least one of a plurality of sensors comprised in the second device;

manipulating, by the processor, the virtual object or a virtual scene in response to the sensor input data from the second device.

12. ) The method of claim 1, wherein the detecting of real world object comprises detection of a 3D printed model of another object.

13. ) The method of claim 12, wherein the virtual object comprises a virtual outer surface of the other object, the virtual outer surface encodes optical properties of a real-world surface material of the other object.

14. ) The method of claim 13, wherein one or more of geometric and rendering properties of the virtual object are substantially similar to corresponding properties of the 3D printed model.

15. ) The method of claim 14, further comprising:

receiving, by the processor, user input for purchase of render data of the virtual object; and

transmitting, by the processor, to a vendor server information regarding the user'"'"'s purchase of the render data.

16. ) The method of claim 12, wherein one or more of other geometric or rendering properties of the virtual object are different from corresponding properties of the 3D printed model.

17. ) The method of claim 16, further comprising:

receiving, by the processor, user input for purchase of render data of the virtual object; and

transmitting, by the processor to a vendor server, information regarding the user'"'"'s purchase of the render data.

18. ) The method of claim 16, further comprising:

detecting, by the processor, that the user has purchased render data of the virtual object for use with the 3D printed model;

rendering, by the processor, the virtual object in accordance with the purchased render data.

19. ) The method of claim 1, further comprising:

displaying, by the processor, the virtual object on a display of the first display device.

20. ) An apparatus comprising:

a processor;

a non-transitory storage medium having stored thereon processor-executable programming logic, the programming logic comprises;

presence detecting logic that detects in communication with a first display device, presence of a real-world object comprising a marker on a surface thereof;

identifying logic that identifies position and orientation of the real-world object in real 3D space relative to a user'"'"'s eyes;

rendering logic that renders a virtual object positioned and oriented in a virtual 3D space relative to the marker;

manipulation logic that manipulates the virtual object responsive to a manipulation of the real-world object in the real 3D space; and

transmitting logic that transmits render data by the processor to visually display, the virtual object on in the virtual 3D space.

21. ) The apparatus of claim 20 the manipulation logic further comprises:

identifying logic that detects a change in the position or orientation of the real-world object.

22. ) The apparatus of claim 21, the manipulation logic further comprising:

altering logic that alters one or more attributes of the virtual object in the virtual space based on the detected change in the real-world object; and

displaying logic that displays to the user, the virtual object with the altered attributes.

23. ) The apparatus of claim 20, the first display device is communicatively coupled to a second display device, the coupling enabling exchange of data generated by the second display device.

24. ) The apparatus of claim 23, the marker is displayed on the touchscreen of the second display device.

25. ) The apparatus of claim 24, the manipulation logic further comprising:

receiving logic that receives data regarding the user'"'"'s touch input from the second display device; and

logic for manipulating the virtual object in the virtual space in response to the data regarding the user'"'"'s touch input.

26. ) The apparatus of claim 25, the data regarding the user'"'"'s touch input comprising position information of the user'"'"'s body part on the touchscreen relative to the marker.

27. ) The apparatus of claim 26, the manipulation logic further comprising:

altering logic that changes at least one of a position, orientation, size, and rendering properties of the virtual object in the virtual space

28. ) The apparatus of claim 26, the manipulation logic further comprising:

altering logic that changes at least one of a position, orientation, size, geometric and rendering properties of the virtual object in response to the user'"'"'s touch input.

29. ) The apparatus of claim 20, the real world object is a 3D printed model of another object.

30. ) The apparatus of claim 29, the virtual object comprises a virtual outer surface of the other object, the virtual outer surface encodes real-world surface properties of the other object.

31. ) The apparatus of claim 30, the properties of the virtual object are substantially similar to the properties of the 3D printed model.

32. ) The apparatus of claim 30, a size of the virtual object is different from a size of the 3D printed model.

33. ) The apparatus of claim 20, the processor is comprised in the first display device.

34. ) The apparatus of claim 33, further comprising:

display logic that displays the virtual object on a display of the first display device.

35. ) A non-transitory processor-readable storage medium comprising processor-executable instructions for:

detecting, by the processor in communication with a first display device, presence of a real-world object comprising a marker on a surface thereof;

identifying, by the processor, position and orientation of the real-world object in real 3D space relative to a user'"'"'s eyes;

rendering, by the processor, a virtual object positioned and oriented in a virtual 3D space relative to the marker, the virtual object configured for control via a manipulations of the real-world object in the real 3D space; and

transmitting render data, by the processor, to visually display, the virtual object in the virtual 3D space.

36. ) The non-transitory medium of claim 35, instructions for manipulation of the virtual object via manipulation of the real-world object further comprises instructions for:

detecting, by the processor, a change in one of the position and orientation of the real-world object.

37. ) The non-transitory medium of claim 35, the further comprising instructions for:

altering, by the processor, one or more attributes of the virtual object in the virtual space based on the detected change in the real-world object; and

displaying, by the processor to the user, the virtual object with the altered attributes.

38. ) The non-transitory medium of claim 35 the first display device is communicatively coupled to a second display device, the coupling enabling exchange of data generated by the second display device.

39. ) The non-transitory medium of claim 38, the marker is displayed on the touchscreen of the second display device.

40. ) The non-transitory medium of claim 39, further comprising instructions for:

receiving, by the processor, data regarding the user'"'"'s touch input from the second display device; and

manipulating, by the processor, the virtual object in the virtual space in response to the data regarding the user'"'"'s touch input.

41. ) The non-transitory medium of claim 35, the real world object is a 3D printed model of another object, the virtual object comprises a virtual outer surface of the other object, the virtual outer surface encodes real-world surface reflectance properties of the other object, and a size of the virtual object is substantially similar to a size of the 3D printed model.

42. ) The non-transitory medium of claim 41, further comprising instructions for:

rendering, by the processor, the virtual outer surface in response to further input indicating a purchase of the rendering.

43. ) The non-transitory medium of claim 35, the render data for the visual display data comprising display data for an image of the real-world object.

44. ) The non-transitory medium of claim 43, the render data comprises data that causes the virtual object to modify the image of the real-world object in the virtual 3D space