System for volume dimensioning via holographic sensor fusion

• US 10,559,086 B1
• Filed: 04/22/2019
• Issued: 02/11/2020
• Est. Priority Date: 05/15/2015
• Status: Active Grant

• Current Assignee: 4DMobile LLC
• Original Assignee: 4DMobile LLC
• Inventors: Miller, Matthew D., Kunert, Steven R., McDermott, Craig, Berry, Josh, Boyer, Mark
• Primary Examiner(s): Perez-Fuentes, Luis
• Application Number:
  US16/390,562
• Time in Patent Office: 295 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06K 7/1408 the method being specifically adapted for the type of code
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 5/44504 Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits

Abstract

A system for volume dimensioning via two-dimensional (2D)/three-dimensional (3D) sensor fusion, based in a tablet, phablet, or like mobile device, is disclosed. The mobile device includes a 2D imager for capturing an image stream of its field of view (FOV), the FOV including target objects. The mobile device includes a 3D imager for collecting 3D imaging data of the FOV including point clouds of each target object within the FOV. Processors of the mobile device positively identify a particular target object by correlating the 2D and 3D image streams and generating a holographic model of the target object overlaid on the video stream, with adjustable surface, edge, and vertex guides. The processors determine the precise dimensions of the target object by measuring the holographic model, and detect and decode object identifiers (2D or 3D) on the surface of the target object to acquire and supplement object data.

Claims

1. An apparatus for volume dimensioning via sensor fusion, comprising:

a housing capable of being carried by an operator;

at least one two-dimensional (2D) image sensor disposed within the housing, the 2D image sensor configured to capture at least one image stream corresponding to a field of view (FOV), the FOV including at least one target object;

at least one three-dimensional (3D) imager disposed within the housing, the 3D imager configured to generate 3D image data associated with the FOV, the 3D image data including at least one plurality of points associated with the target object, each point corresponding to a coordinate set and a distance from the apparatus;

at least one processor disposed within the housing and operatively coupled to the 2D image sensor and the 3D imager, the processor configured to;

a) distinguish the target object within the FOV by analyzing at least one of the captured image stream and the 3D image data;

b) generate at least one holographic model corresponding to the target object by correlating the 3D image data and the captured image stream, the holographic model including at least one of a surface of the target object, a vertex of the target object, and an edge of the target object;

c) determine at least one dimension of the target object by measuring the holographic model;

d) detect at least one object identifier corresponding to the target object by analyzing the holographic model;

and

e) acquire object data corresponding to the target object by decoding the object identifier;

a touch-sensitive display surface disposed within the housing and coupled to the processor, the display surface configured to;

a) display the captured image stream;

b) superimpose the holographic model over the captured image stream;

c) receive control input from the operator;

and

d) adjust the holographic model based on the received control input;

and

at least one wireless transceiver disposed within the housing and configured to establish a wireless link to at least one remote source.

2. The apparatus of claim 1, further comprising:

at least one inertial sensor disposed within the housing and configured to determine at least one of a) an orientation of the apparatus and b) a rate of change of the orientation over time.

3. The apparatus of claim 2, wherein:

the display surface is configured to display at least one touch-sensitive graphical element based on at least one of the determined orientation and the determined rate of change.

4. The apparatus of claim 1, further comprising:

at least one microphone disposed within the housing, the microphone coupled to the processor and configured to receive aural control input from the operator.

5. The apparatus of claim 1, wherein:

the processor is configured to generate the at least one holographic model when the processor has positively identified the target object;

and

the display surface is configured to superimpose the holographic model over the captured image stream by displaying at least one of;

a) an adjustable surface guide corresponding to the surface;

b) an adjustable edge guide corresponding to the edge;

and

c) an adjustable vertex guide corresponding to the vertex.

6. The apparatus of claim 5, wherein adjusting the holographic model includes at least one of:

moving at least one of the adjustable vertex guide, the adjustable surface guide, and the adjustable edge guide;

resizing at least one of the adjustable vertex guide, the adjustable surface guide, and the adjustable edge guide;

and

adjusting the holographic model relative to the captured image stream.

7. The apparatus of claim 1, wherein the processor is configured to determine the at least one dimension to a confidence level, the confidence level based on a plurality of measurements of the holographic model.

8. The apparatus of claim 1, further comprising:

at least one memory coupled to the processor, the memory configured to store reference data associated with the target object.

9. The apparatus of claim 8, wherein the processor is configured to:

generate supplemental object data corresponding to the target object;

and

add the supplemental object data to the acquired object data.

10. The apparatus of claim 9, wherein the processor is configured to upload the supplemental object data to the remote source via the wireless link.

11. The apparatus of claim 9, wherein the processor is configured to generate the supplemental object data by comparing at least one of the determined dimension and the object data to the reference data.

12. The apparatus of claim 1, wherein the processor is configured to acquire the object data by:

identifying at least one three-dimensional (3D) code based on the 3D image data;

and

decoding the 3D code.

13. A system for remote volume dimensioning via sensor fusion, comprising:

a mobile computing device capable of being carried by an operator, the mobile computing device comprising;

at least one two-dimensional (2D) image sensor configured to capture at least one image stream corresponding to a field of view (FOV), the FOV including at least one target object;

at least one three-dimensional (3D) imager configured to generate 3D image data associated with the FOV, the 3D image data including at least one plurality of points associated with the target object, each corresponding to a coordinate set and a distance from the apparatus;

at least one processor disposed within the housing and operatively coupled to the 2D image sensor and the 3D imager, the processor configured to;

a) distinguish the target object within the FOV by analyzing at least one of the captured image stream and the 3D image data;

b) generate at least one holographic model corresponding to the target object by correlating the 3D image data and the captured image stream, the holographic model including at least one of a surface of the target object, a vertex of the target object, and an edge of the target object;

c) determine at least one dimension of the target object by measuring the holographic model;

d) detect at least one object identifier corresponding to the target object by analyzing the holographic model;

and

e) acquire object data corresponding to the target object by decoding the object identifier;

and

at least one wireless transceiver disposed within the housing and configured to establish a wireless link;

and

at least one augmented reality (AR) viewing device communicatively coupled to the mobile computing device via wireless link and wearable by a viewer, the AR viewing device configured to;

a) display the captured image stream to the viewer via a display surface proximate to one or more eyes of the viewer;

b) superimpose the holographic model over the captured image stream;

c) detect control input provided by the viewer;

and

d) adjust the holographic model based on the detected control input.

14. The volume dimensioning system of claim 13, wherein the viewer is located remotely from the mobile computing device.

15. The volume dimensioning system of claim 13, wherein the AR viewing device includes at least one of:

a camera configured to detect a gesture of the viewer;

and

a microphone configured to detect aural control input provided by the viewer.

16. The volume dimensioning system of claim 13, wherein the AR viewing device includes at least one of a virtual reality (VR) viewing device and a mixed reality (MR) viewing device.

17. The volume dimensioning system of claim 13, wherein:

the wireless transceiver is configured to receive the detected control input via the wireless link;

and

the processor is configured to adjust the holographic model based on the received control input.

18. The volume dimensioning system of claim 13, wherein:

the processor is configured to generate the at least one holographic model when the processor has positively identified the target object;

and

the display surface is configured to superimpose the holographic model over the captured image stream by displaying at least one of;

a) an adjustable surface guide corresponding to the surface;

b) an adjustable edge guide corresponding to the edge;

and

c) an adjustable vertex guide corresponding to the vertex.

19. The volume dimensioning system of claim 18, wherein adjusting the holographic model includes at least one of:

moving at least one of the adjustable surface guide, the adjustable vertex guide, and the adjustable edge guide;

resizing at least one of the adjustable surface guide, the adjustable vertex guide, and the adjustable edge guide;

and

adjusting the holographic model relative to the captured image stream.

20. A method for volume dimensioning via sensor fusion, comprising:

capturing, via a two-dimensional (2D) camera attached to a mobile device, a 2D image stream corresponding to a field of view (FOV) and including at least one target object within the FOV;

capturing, via a three-dimensional (3D) imager attached to the mobile device, 3D image data corresponding to the FOV, the 3D image data including a plurality of points corresponding to the target object, each point comprising a coordinate set and a distance from the mobile device;

distinguishing, via at least one processor of the mobile device, the target object from the FOV by analyzing at least one of the captured image stream and the 3D image data;

generating, via at least one processor of the mobile device, a holographic model corresponding to the target object by correlating the 2D image stream and the plurality of points, the holographic model including at least one of a surface of the target object, an edge of the target object, and a vertex of the target object;

determining, via the processor, at least one dimension of the target object by measuring the holographic model;

detecting, via the processor, at least one object identifier corresponding to the target object by analyzing the holographic model;

and

acquiring object data corresponding to the target object by decoding the object identifier.