Method and apparatus for selectively integrating sensory content

• US 10,068,369 B2
• Filed: 10/31/2015
• Issued: 09/04/2018
• Est. Priority Date: 11/04/2014
• Status: Active Grant

• Current Assignee: West Texas Technology Partners LLC
• Original Assignee: Atheer, Inc.
• Inventors: James, Greg, Yang, Allen Yang, Itani, Sleiman
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US14/929,322
• Publication Number:
  US 20160125656A1
• Time in Patent Office: 1,039 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

To integrate a sensory property such as occlusion, shadowing, reflection, etc. among physical and notional (e.g. virtual/augment) visual or other sensory content, providing an appearance of similar occlusion, shadowing, etc. in both models. A reference position, a physical data model representing physical entities, and a notional data model are created or accessed. A first sensory property from either data model is selected. A second sensory property is determined corresponding with the first sensory property, and notional sensory content is generated from the notional data model with the second sensory property applied thereto. The notional sensory content is outputted to the reference position with a see-through display. Consequently, notional entities may appear occluded by physical entities, physical entities may appear to cast shadows from notional light sources, etc.

Claims

1. A machine-implemented method, comprising in a processor:

establishing a reference position;

establishing a physical data model representing at least one physical entity, said physical data model being spatially dynamic in time;

establishing a notional data model representing at least one notional entity, said notional data model being dynamic in time and non-exclusive of spatial coincidence with said physical data model;

establishing a first sensory property from said notional data model;

wherein said first sensory phenomenon represents a physical environmental phenomenon;

wherein said physical environmental phenomenon comprises at least one of physical illumination, physical shadowing, a physical volumetric effect, and an optical phenomenon; and

wherein said physical volumetric effect comprises at least one of a group consisting of at least one falling element, at least one flying element, and at least one suspended element;

determining a second sensory property, said second sensory property corresponding with said first sensory property for another of said physical and notional data models;

generating notional sensory content representing at least a portion of said notional data model with at least a portion of said second sensory property applied thereto; and

outputting to said reference position with a perceive-through display said notional sensory content, registered with said physical data model.

2. The method of claim 1, wherein:

said first sensory property comprises at least one of a visual property and an auditory property.

3. The method of claim 1, wherein:

said second sensory property comprises at least one of a visual property and an auditory property.

4. The method of claim 1, wherein:

said at least one physical entity comprises at least a portion of a viewer.

5. The method of claim 1, wherein:

said at least one notional entity comprises at least one of a virtual reality entity and an augmented reality entity.

6. The method of claim 1, wherein:

said reference point corresponds with an eye of a viewer.

7. The method of claim 1, wherein:

said perceive-through display is a head mounted display.

8. The method of claim 1, wherein:

said perceive-through display is at least one of an optical see-through display such that optical physical content passes therethrough and a virtual see-through display adapted to output at least a portion of said physical data model therefrom.

9. The method of claim 1, wherein:

said physical volumetric effect is a function of a distance between said reference point and said physical data model.

10. A machine-implemented method, comprising in a processor:

establishing a reference position;

establishing a physical data model representing at least one physical entity, said physical data model being spatially dynamic in time;

establishing a notional data model representing at least one notional entity, said notional data model being dynamic in time and non-exclusive of spatial coincidence with said physical data model;

establishing a first sensory property from said notional data model;

wherein said first sensory phenomenon represents a physical environmental phenomenon;

wherein said physical environmental phenomenon comprises at least one of physical illumination, physical shadowing, a physical volumetric effect, and an optical phenomenon; and

wherein said physical volumetric effect comprises at least one of a group consisting of ash, debris, dust, fog, gas, hail, heat distortion, insects, leaves, mist, rain, sleet, smoke, snow, spray, and steam;

determining a second sensory property, said second sensory property corresponding with said first sensory property for another of said physical and notional data models;

generating notional sensory content representing at least a portion of said notional data model with at least a portion of said second sensory property applied thereto; and

outputting to said reference position with a perceive-through display said notional sensory content, registered with said physical data model.

11. The method of claim 10, wherein:

said physical volumetric effect is a function of a distance between said reference point and said physical data model.

12. The method of claim 10, wherein:

said first sensory property comprises at least one of a visual property and an auditory property.

13. The method of claim 10, wherein:

said second sensory property comprises at least one of a visual property and an auditory property.

14. The method of claim 10, wherein:

said at least one physical entity comprises at least a portion of a viewer.

15. The method of claim 10, wherein:

said at least one notional entity comprises at least one of a virtual reality entity and an augmented reality entity.

16. The method of claim 10, wherein:

said reference point corresponds with an eye of a viewer.

17. The method of claim 10, wherein:

said perceive-through display is a head mounted display.

18. The method of claim 10, wherein:

said perceive-through display is at least one of an optical see-through display such that optical physical content passes therethrough and a virtual see-through display adapted to output at least a portion of said physical data model therefrom.

19. A machine-implemented method, comprising in a processor:

establishing a reference position;

establishing a physical data model representing at least one physical entity, said physical data model being spatially dynamic in time;

establishing a notional data model representing at least one notional entity, said notional data model being dynamic in time and non-exclusive of spatial coincidence with said physical data model;

establishing a first sensory property from said notional data model;

wherein said first sensory phenomenon represents a physical environmental phenomenon;

wherein said physical environmental phenomenon comprises at least one of physical illumination, physical shadowing, a physical volumetric effect, and an optical phenomenon; and

wherein said physical environmental phenomenon comprises at least one of a group consisting of diffraction, diffusion, glory, haloing, lens flare, and reflection;

determining a second sensory property, said second sensory property corresponding with said first sensory property for another of said physical and notional data models;

generating notional sensory content representing at least a portion of said notional data model with at least a portion of said second sensory property applied thereto; and

outputting to said reference position with a perceive-through display said notional sensory content, registered with said physical data model.

20. The method of claim 19, wherein:

said physical volumetric effect is a function of a distance between said reference point and said physical data model.

21. The method of claim 19, wherein:

said first sensory property comprises at least one of a visual property and an auditory property.

22. The method of claim 19, wherein:

said second sensory property comprises at least one of a visual property and an auditory property.

23. The method of claim 19, wherein:

said at least one physical entity comprises at least a portion of a viewer.

24. The method of claim 19, wherein:

said at least one notional entity comprises at least one of a virtual reality entity and an augmented reality entity.

25. The method of claim 19, wherein:

said reference point corresponds with an eye of a viewer.

26. The method of claim 19, wherein:

said perceive-through display is a head mounted display.

27. The method of claim 19, wherein:

said perceive-through display is at least one of an optical see-through display such that optical physical content passes therethrough and a virtual see-through display adapted to output at least a portion of said physical data model therefrom.