Method and system for obtaining texture data of a space

• US 10,352,693 B2
• Filed: 05/08/2015
• Issued: 07/16/2019
• Est. Priority Date: 07/12/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Abovitz, Rony, Schowengerdt, Brian T., Watson, Mathew D.
• Primary Examiner(s): Martello, Edward
• Application Number:
  US14/707,745
• Publication Number:
  US 20150247723A1
• Time in Patent Office: 1,530 Days
• CPC Class Codes: A61B 2034/101 Computer-aided simulation of surgical operations
• CPC Class Codes: A61B 2562/0204 Acoustic sensors
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 2562/0247 Pressure sensors
• CPC Class Codes: A61B 3/0008 provided with illuminating means A61B3/117 takes precedence; goniolenses used for laser treatment A61F9/009; illuminating means for optical instruments G02B27/00
• CPC Class Codes: A61B 3/022 for testing contrast sensitivity
• CPC Class Codes: A61B 3/024 for determining the visual field, e.g. perimeter types
• CPC Class Codes: A61B 3/028 for testing visual acuity; for determination of refraction, e.g. phoropters
• CPC Class Codes: A61B 3/063 for testing light sensitivity, i.e. adaptation
• CPC Class Codes: A61B 3/066 for testing colour vision
• CPC Class Codes: A61B 3/08 for testing binocular or stereoscopic vision, e.g. strabismus
• CPC Class Codes: A61B 3/085 for testing strabismus
• CPC Class Codes: A61B 3/10 Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
• CPC Class Codes: A61B 3/1015 for wavefront analysis
• CPC Class Codes: A61B 3/102 for optical coherence tomography [OCT]
• CPC Class Codes: A61B 3/1035 for measuring astigmatism testing astigmatism A61B3/036
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 3/12 for looking at the eye fundus, e.g. ophthalmoscopes A61B3/13 takes precedence
• CPC Class Codes: A61B 3/1216 for diagnostics of the iris
• CPC Class Codes: A61B 3/13 Ophthalmic microscopes
• CPC Class Codes: A61B 3/14 : Arrangements specially adapted for eye photography apparatus or arrangements for taking photographs per se G03B
• CPC Class Codes: A61B 3/165 : Non-contacting tonometers
• CPC Class Codes: A61B 34/10 : Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 5/0059 : using light, e.g. diagnosis by transillumination, diascopy, fluorescence A61B5/0093, A61B5/1455, A61B5/02416 and A61B5/0261 take precedence
• CPC Class Codes: A61B 5/0066 : Optical coherence imaging
• CPC Class Codes: A61B 5/0077 : Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/01 : Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/14532 : for measuring glucose, e.g. by tissue impedance measurement
• CPC Class Codes: A61B 5/1455 : using optical sensors, e.g. spectral photometrical oximeters
• CPC Class Codes: A61B 5/14555 : specially adapted for the eye fundus
• CPC Class Codes: A61B 5/369 : Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: A61B 5/398 : Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG]
• CPC Class Codes: A61B 5/6803 : Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A63F 13/00 : Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 : comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 : involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G01B 11/303 : using photoelectric detection means
• CPC Class Codes: G02B 2027/0105 : Holograms with particular structures
• CPC Class Codes: G02B 2027/0127 : comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 : comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 : comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 : Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0185 : Displaying image at variable distance
• CPC Class Codes: G02B 2027/0187 : slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0101 : characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 : Head mounted
• CPC Class Codes: G02B 27/0172 : characterised by optical features
• CPC Class Codes: G02B 27/0179 : Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G02B 27/42 : Diffraction optics , i.e. systems including a diffractive element being designed for providing a diffractive effectG02B27/60 takes precedence
• CPC Class Codes: G02B 27/4205 : having a diffractive optical element [DOE] contributing to image formation, e.g. whereby modulation transfer function MTF or optical aberrations are relevant
• CPC Class Codes: G02B 27/4227 : in image scanning systems
• CPC Class Codes: G02B 6/10 : of the optical waveguide type G02B6/02, G02B6/24 take precedence; devices or arrangements for the control of light by electric, magnetic, electro-magnetic or acoustic means G02F1/00; transferring the modulation of modulated light G02F2/00; optical logic elements G02F3/00; optical analogue/digital converters G02F7/00
• CPC Class Codes: G02B 6/34 : utilising prism or grating G02B6/293 takes precedence
• CPC Class Codes: G06F 16/7837 : using objects detected or recognised in the video content
• CPC Class Codes: G06F 18/22 : Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/005 : Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 : for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 : Scrolling or panning
• CPC Class Codes: G06F 3/0487 : using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/04883 : for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06Q 30/0643 : Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2200/24 : involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10004 : Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/10148 : Varying focus
• CPC Class Codes: G06T 2207/10152 : Varying illumination
• CPC Class Codes: G06T 2207/30041 : Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2210/41 : Medical
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06V 10/20 : Image preprocessing
• CPC Class Codes: G06V 10/40 : Extraction of image or video features
• CPC Class Codes: G06V 10/758 : Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/113 : Recognition of static hand signs
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G16H 40/00 : ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
• CPC Class Codes: G16H 40/20 : for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
• CPC Class Codes: G16H 40/67 : for remote operation
• CPC Class Codes: H04B 10/25891 : Transmission components H04B10/40 takes precedence

Abstract

A method and system determining texture data at least by performing fiducial point registration with a triggerable texture source and generating virtual objects and texture data based at least in part upon a fiducial point. Fiducial point registration textures a desired point with one or more projectors in response to the identification of a desired point in space, generates data with one or more projected patterns, determines a final location or depth of a point using the generated data, and registers the point as a fiducial point by groom a data structure based on a registration requirement pertaining to a distribution of fiduciary points and a requirement pertaining to machine recognition of physical objects. Virtual objects are generated relative to a real-world scene and the generated texture data-based at least in part upon the final location or depth of a fiducial point.

Claims

1. A method for obtaining texture data of a space, comprising:

performing fiducial point registration using at least a triggerable texture source at least by;

identifying a desired point in the space based at least in part on one or more positions of one or more image capturing devices;

in response to identification of the desired point, texturing the desired point at least by controlling one or more projector devices that project a series of patterns with the triggerable texture source to generate one or more projected patterns at the desired point, in absence of distinct feature points and recognizable patterns in at least one pattern of the series of patterns;

generating first and second data with at least the one or more projected patterns, the first and second data corresponding to at least one point that pertains to the desired point in the space;

determining, by a microprocessor of an augmented or virtual reality system or a remote processing module comprising at least one microprocessor, a final location or depth of the at least one point using at least the first and second data; and

registering the at least one point as a fiducial point in a data structure at least by grooming the data structure based in part or in whole upon a registration requirement pertaining to a distribution of fiducial points for the space and further upon determination results of whether the at least one point improves machine recognition of physical objects than existing fiducial points stored in the data structure; and

generating, by the augmented or virtual reality system, one or more virtual objects relative to a real-world scene and texture data for the desired point in a virtual content presented to a user based in whole or in part on the final location or depth of the fiducial point.

2. The method of claim 1, further comprising repeating at least the texturing the physical location, the generating first and second data, and the determining the final location or the depth with a plurality of portions of the one or more projected patterns to acquire additional texture data of the space.

3. The method of claim 1, wherein grooming the data structure comprises adding one or more new points to the data structure when it is determined that the one or more new points improve the world model or provide refinement for at least the existing fiducial points than at least one existing point already stored in the data structure.

4. The method of claim 1, further comprising projecting light onto the space at least by projecting the light onto the space using a fiber-based projector;

and modifying the light with an optical element to form bundles of rays before the light is projected onto the space.

5. The method of claim 1, further comprising adding one or more new points to the data structure when it is determined that the one or more new points improve evenness of the distribution of the fiducial points in the space than the existing fiducial points.

6. The method of claim 1, wherein the series of patterns comprises structured light, the series of patterns is a dynamic series of patterns, or the light comprises patterned light, and the series of patterns includes a plurality of points.

7. The method of claim 1, wherein the light comprises textured light, and wherein the series of patterns comprises a first pattern that is irregular.

8. The method of claim 1, further comprising identifying first and second portions of the respective images, wherein the first and second portions of the patterns correspond to respective pixels of the series of patterns.

9. The method of claim 1, further comprising:

identifying successive light patterns that are projected by one or more fiber-based projectors as structured light having a plurality of dynamic, known patterns;

identifying one or more individual pixels that are individually addressable and textured in the successive light patterns; and

determining the final location or depth of the at least one point based in part or in whole upon the one or more individual pixels.

10. The method of claim 1, wherein the one or more image capturing devices are the one or more projector devices that are configured to provide image capturing functionality.

11. The method of claim 1, wherein the desired point already includes texturing before the one or more projector devices project the series of patterns onto the space.

12. The method of claim 1, further comprising:

identifying data comprising at least one of visual data, audio data, or sensory data;

converting the data into map data; and

constructing a virtual world or a map of the virtual world based at least in part upon the map data.

13. The method of claim 12, further comprising:

determining a location, orientation, placement, or configuration of a physical object perceived by the user in an image of a real-world environment including the space through the augmented or virtual reality system;

rendering the one or more virtual objects in the virtual content in relation to the location, orientation, placement, or configuration of a physical object; and

presenting an image of the real-world enviorment and the virtual content to at least one eye of the user.

14. The method of claim 1, further comprising:

identifying a plurality of data sets of a real-world scene from a database, the plurality of data sets corresponding to multiple levels of resolution for the real-world scene; and

rendering and presenting, by the augmented or virtual reality system, the real-world scene together with the virtual content with fiducial refinement to the user.

15. A method for obtaining texture data of a space, comprising:

identifying a desired point in the space based at least in part on one or more positions of first and second spatially separated image capturing devices; and

in response to identification of the desired point, texturing the desired point at least by controlling one or more projector devices that project a series of patterns with a triggerable texture source to generate one or more projected patterns at the desired point;

identifying a portion of the one or more projected patterns with the first and second spatially separated image capturing devices to generate first and second data corresponding to at least the desired point in images captured by the first and second image capturing devices;

determining, at an augmented or virtual reality system or a remote processing module comprising at least one microprocessor, a final location or depth of at least one point that pertains to the desired point in the space using at least the first and second data;

registering the at least one point as a fiducial point that is specific to at least one physical object in a data structure at least by grooming the data structure based in part or in whole upon a registration requirement pertaining to a distribution of fiducial points in the space and further upon determination results of whether the at least one point improves machine recognition of physical objects than existing fiducial points stored in the data structure; and

generating, at the augmented or virtual reality system, one or more virtual objects relative to a real-world scene for the desired point in a virtual content presented to a user based in whole or in part on the final location or depth of the fiducial point.

16. The method of claim 15, wherein grooming the data structure is based in part or in whole upon a relative evenness of the distribution of the fiducial points in the space.

17. The method of claim 15, wherein the first and second spatially separated image capturing devices comprise fiber-based cameras, and the fiber-based cameras includes fiber-based projectors configured to project light onto the space to form the pattern.

18. A system for generating virtual contents with texture data of a space, the system comprising:

a triggerable texture source comprising one or more projector devices, wherein

the triggerable texture source is configured to texture a desired point in a space at least by controlling the one or more projector devices that project a series of patterns onto to generate one or more projected patterns at the desired point;

one or more image capturing devices configured to generate first and second data with at least the one or more projected patterns, the first and second data corresponding to at least one point that pertains to the desired point in the space;

a microprocessor or a remote processing module comprising at least one microprocessor configured to determine a final location or depth of the at least one point using at least the first and second data;

the microprocessor or the remote processing module further configured to register the at least one point as a fiducial point in a data structure at least by grooming the data structure based in part or in whole upon a registration requirement pertain to a distribution of fiducial points for the space and further upon determination results of whether the at least one point improves machine recognition of physical objects than existing fiducial points stored in the data structure; and

the augmented or virtual reality system further configured to generate one or more virtual objects relative to a real-world scene and texture data for the desired point in a virtual content presented to a user based in whole or in part on the final location or depth of the fiducial point.

19. A computer program product comprising a non-transitory computer-usable storage medium storing thereupon executable code which, when executed by at least one processor, causes the at least one processor to execute a set of acts for generating virtual contents with texture data of a space, the set of acts comprising:

individually addressing a pixel corresponding to a point in a space with a triggerable texture source in absence of distinct feature points in the space at least by;

identifying a desired point in the space based at least in part on one or more positions of one or more image capturing devices; and

in response to identification of the desired point, texturing the desired point at least by controlling one or more projector devices that project a series of patterns with the triggerable texture source to generate one or more projected patterns at the desired point;

generating first and second data with at least the one or more projected patterns, the first and second data corresponding to at least one point that pertains to the desired point in the space;

determining, by a microprocessor of an augmented or virtual reality system or a remote processing module comprising at least one microprocessor, a final location or depth of the at least one point using at least the first and second data; and

registering the at least one point as a fiducial point in a data structure at least by grooming the data structure based in part or in whole upon a registration requirement pertaining to a distribution of fiducial points for the space and further upon determination results of whether the at least one point improves machine recognition of physical objects than existing fiducial points stored in the data structure; and

generating, by the augmented or virtual reality system, one or more virtual objects relative to a real-world scene and texture data for the desired point in a virtual content presented to a user based in whole or in part on the final location or depth of the fiducial point.