METHOD AND APPARTUS FOR SELECTIVELY INTEGRATING SENSORY CONTENT

• US 20160125656A1
• Filed: 10/31/2015
• Published: 05/05/2016
• Est. Priority Date: 11/04/2014
• Status: Active Grant

• Current Assignee: West Texas Technology Partners LLC
• Original Assignee: Atheer, Inc.
• Inventors: James, Greg, Yang, Allen Yang, Itani, Sleiman
• Granted Patent: US 10,068,369 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

To integrate a sensory property such as occlusion, shadowing, reflection, etc. among physical and notional (e.g. virtual/augment) visual or other sensory content, providing an appearance of similar occlusion, shadowing, etc. in both models. A reference position, a physical data model representing physical entities, and a notional data model are created or accessed. A first sensory property from either data model is selected. A second sensory property is determined corresponding with the first sensory property, and notional sensory content is generated from the notional data model with the second sensory property applied thereto. The notional sensory content is outputted to the reference position with a see-through display. Consequently, notional entities may appear occluded by physical entities, physical entities may appear to cast shadows from notional light sources, etc.

Claims

1. A machine-implemented method, comprising in a processor:

establishing a reference position;

establishing a physical data model at least substantially representing at least one physical entity, said physical data model being spatially dynamic in time;

establishing a notional data model at least substantially representing at least one notional entity, said notional data model being dynamic in time and non-exclusive of spatial coincidence with said physical data model;

determining an occlusion of said notional data model by said physical data model relative to said reference position, wherein a first distance along a first direction from said reference position to said physical data model is less than a second distance along a second direction from said reference position to said notional data model, said second direction being at least substantially similar to said first direction;

generating notional visual content representing at least a portion of said notional data model with at least a portion of said occlusion applied thereto;

outputting to said reference position with a see-through display said notional visual content, at least substantially registered with said physical data model.

2. The method of claim 1, wherein:

further determining said occlusion wherein said first distance along said second direction from said reference position to said physical data model is substantially equal to said second distance along said second direction from said reference position to said notional data model.

3. The method of claim 1, wherein:

said at least one physical entity comprises at least a portion of a viewer.

4. The method of claim 1, wherein:

said at least one notional entity comprises at least one of a virtual reality entity and an augmented reality entity.

5. The method of claim 1, wherein:

said reference point substantially corresponds with an eye of a viewer.

6. The method of claim 1, wherein:

said see-through display is a head mounted display.

7. The method of claim 1, wherein:

said see-through display is at least one of an optical see-through display such that optical physical content at least substantially passes therethrough and a virtual see-through display adapted to output at least a portion of said physical data model therefrom.

8. A machine-implemented method, comprising in a processor:

establishing a reference position;

establishing a physical data model at least substantially representing at least one physical entity, said physical data model being spatially dynamic in time;

establishing a notional data model at least substantially representing at least one notional entity, said notional data model being dynamic in time and non-exclusive of spatial coincidence with said physical data model;

establishing a first sensory property from one of said physical and notional data models;

determining a second sensory property, said second sensory property at least substantially corresponding with said first sensory property for another of said physical and notional data models;

generating notional sensory content representing at least a portion of said notional data model with at least a portion of said second sensory property applied thereto;

outputting to said reference position with a perceive-through display said notional sensory content, at least substantially registered with said physical data model.

9. The method of claim 8, wherein:

said first sensory property comprises at least one of a visual property and an auditory property.

10. The method of claim 8, wherein:

said second sensory property comprises at least one of a visual property and an auditory property.

11. The method of claim 8, wherein:

said at least one physical entity comprises at least a portion of a viewer.

12. The method of claim 8, wherein:

said at least one notional entity comprises at least one of a virtual reality entity and an augmented reality entity.

13. The method of claim 8, wherein:

said reference point substantially corresponds with an eye of a viewer.

14. The method of claim 8, wherein:

said perceive-through display is a head mounted display.

15. The method of claim 8, wherein:

said perceive-through display is at least one of an optical see-through display such that optical physical content at least substantially passes therethrough and a virtual see-through display adapted to output at least a portion of said physical data model therefrom.

16. The method of claim 8, comprising:

establishing said first sensory property from said physical data model;

said second sensory property at least substantially corresponding with said first sensory property for said notional data model.

17. The method of claim 16, wherein:

said first sensory property at least substantially represents a physical environmental phenomenon.

18. The method of claim 17, wherein:

said physical environmental phenomenon comprises at least one of physical illumination, physical shadowing, a physical volumetric effect, and an optical phenomenon.

19. The method of claim 17, wherein:

said physical volumetric effect comprises at least one of a group consisting of at least one falling element, at least one flying element, and at least one suspended element.

20. The method of claim 17, wherein:

said physical volumetric effect comprises at least one of a group consisting of ash, debris, dust, fog, gas, hail, heat distortion, insects, leaves, mist, rain, sleet, smoke, snow, spray, and steam.

21. The method of claim 17, wherein:

said optical phenomenon comprises at least one of a group consisting of diffraction, diffusion, focus, glory, haloing, lens flare, and reflection.

22. The method of claim 17, wherein:

said second sensory property effect at least substantially represents a physical environmental phenomenon.

23. The method of claim 22, wherein:

said physical environmental phenomenon comprises at least one of physical illumination, physical shadowing, a physical volumetric effect, and an optical phenomenon.

24. The method of claim 23, wherein:

said physical volumetric effect comprises at least one of a group consisting of at least one falling element, at least one flying element, and at least one suspended element.

25. The method of claim 23, wherein:

said physical volumetric effect comprises at least one of a group consisting of ash, debris, dust, fog, gas, hail, heat distortion, insects, leaves, mist, rain, sleet, smoke, snow, spray, and steam.

26. The method of claim 23, wherein:

said physical environmental phenomenon comprises at least one of a group consisting of diffraction, diffusion, glory, haloing, lens flare, and reflection.

27. The method of claim 8, comprising:

establishing said first sensory property from said notional data model;

said second sensory property at least substantially corresponding with said first sensory property for said physical data model.

28. The method of claim 27, wherein:

said first sensory phenomenon at least substantially represents a physical environmental phenomenon.

29. The method of claim 28, wherein:

said physical environmental phenomenon comprises at least one of physical illumination, physical shadowing, a physical volumetric effect, and an optical phenomenon.

30. The method of claim 29, wherein:

said physical volumetric effect is a function of said first distance between said reference point and said physical data model.

31. The method of claim 29, wherein:

said physical volumetric effect comprises at least one of a group consisting of at least one falling element, at least one flying element, and at least one suspended element.

32. The method of claim 29, wherein:

said physical volumetric effect comprises at least one of a group consisting of ash, debris, dust, fog, gas, hail, heat distortion, insects, leaves, mist, rain, sleet, smoke, snow, spray, and steam.

33. The method of claim 29, wherein:

said physical environmental phenomenon comprises at least one of a group consisting of diffraction, diffusion, glory, haloing, lens flare, and reflection.

34. The method of claim 34, wherein:

said second sensory property at least substantially represents said physical environmental phenomenon.

35. The method of claim 34, wherein:

said physical environmental phenomenon comprises at least one of physical illumination, physical shadowing, a physical volumetric effect, and an optical phenomenon.

36. The method of claim 35, wherein:

said physical volumetric effect is a function of said first distance between said reference point and said physical data model.

37. The method of claim 35, wherein:

said physical volumetric effect comprises at least one of a group consisting of at least one falling element, at least one flying element, and at least one suspended element.

38. The method of claim 35, wherein:

said physical volumetric effect comprises at least one of a group consisting of ash, debris, dust, fog, gas, hail, heat distortion, insects, leaves, mist, rain, sleet, smoke, snow, spray, and steam.

39. The method of claim 35, wherein:

said physical environmental phenomenon comprises at least one of a group consisting of diffraction, diffusion, glory, haloing, lens flare, and reflection.

40. An apparatus, comprising:

a processor;

a perceive-through display in communication with said processor;

a reference position establisher comprising executable instructions instantiated on said processor, said reference position establisher being adapted to establish a reference position;

a physical data model establisher comprising executable instructions instantiated on said processor, said physical data model establisher being adapted to establish a physical data model at least substantially representing at least one physical entity, said physical data model being spatially dynamic in time;

a notional data model establisher comprising executable instructions instantiated on said processor, said notional data model establisher being adapted to establish a notional data model at least substantially representing at least one notional entity, said notional data model being dynamic in time and non-exclusive of spatial coincidence with said physical data model;

an occlusion determiner comprising executable instructions instantiated on said processor, said occlusion determiner being adapted to determine an occlusion of said notional data model by said physical data model relative to said reference position, wherein a first distance along a first direction from said reference position to said physical data model is less than a second distance along a second direction from said reference position to said notional data model, said second direction being at least substantially similar to said first direction;

an occlusion applier comprising executable instructions instantiated on said processor, said occlusion applier being adapted to apply said occlusion to said notional data model; and

an outputter comprising executable instructions instantiated on said processor, said outputter being adapted to output to said reference position with said perceive-through display at least a portion of said notional data model with said occlusion applied thereto at least substantially registered with said physical data model.

41. An apparatus, comprising:

a processor;

a see-through display in communication with said processor;

a reference position establisher comprising executable instructions instantiated on said processor, said reference position establisher being adapted to establish a reference position;

a physical data model establisher comprising executable instructions instantiated on said processor, said physical data model establisher being adapted to establish a physical data model at least substantially representing at least one physical entity, said physical data model being spatially dynamic in time;

a notional data model establisher comprising executable instructions instantiated on said processor, said notional data model establisher being adapted to establish a notional data model at least substantially representing at least one notional entity, said notional data model being dynamic in time and non-exclusive of spatial coincidence with said physical data model;

a first sensory property establisher comprising executable instructions instantiated on said processor, said first sensory property establisher being adapted to establish a first sensory property from one of said physical and notional data models;

a second sensory property determiner comprising executable instructions instantiated on said processor, said second sensory property determiner being adapted to determine a second sensory property at least substantially corresponding with said first sensory property for another of said physical and notional data models;

a second sensory property applier comprising executable instructions instantiated on said processor, said second sensory property applier being adapted to apply said second sensory property to said notional data model so as to generate notional sensory content therefrom;

an outputter comprising executable instructions instantiated on said processor, said outputter being adapted to output to said reference position said notional sensory content with a perceive-through display at least substantially registered with said physical data model.

42. A machine-implemented method, comprising in a processor:

means for establishing a reference position;

means for establishing a physical data model at least substantially representing at least one physical entity, said physical data model being spatially dynamic in time;

means for establishing a notional data model at least substantially representing at least one notional entity, said notional data model being dynamic in time and non-exclusive of spatial coincidence with said physical data model;

means for determining an occlusion of said notional data model by said physical data model relative to said reference position, wherein a first distance along a first direction from said reference position to said physical data model is less than a second distance along a second direction from said reference position to said notional data model, said second direction being at least substantially similar to said first direction;

means for generating notional visual content representing at least a portion of said notional data model with at least a portion of said occlusion applied thereto;

means for outputting to said reference position with a see-through display said notional visual content at least substantially registered with said physical data model.

43. A machine-implemented method, comprising in a processor:

means for establishing a reference position;

means for establishing a physical data model at least substantially representing at least one physical entity, said physical data model being spatially dynamic in time;

means for establishing a notional data model at least substantially representing at least one notional entity, said notional data model being spatially dynamic in time and non-exclusive of spatial coincidence with said physical data model;

means for establishing a first sensory property from one of said physical and notional data models;

means for determining a second sensory property, said second sensory property at least substantially corresponding with said first sensory property for another of said physical and notional data models;

means for generating notional sensory content representing at least a portion of said notional data model with at least a portion of said second sensory property applied thereto;

means for outputting to said reference position with a perceive-through display said notional sensory content at least substantially registered with said physical data model.