System for 2D/3D Spatial Feature Processing

• US 20140241614A1
• Filed: 02/28/2013
• Published: 08/28/2014
• Est. Priority Date: 02/28/2013
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Inventors: Lee, Johnny
• Granted Patent: US 9,142,019 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 25/705 Pixels for depth measurement, e.g. RGBZ

Abstract

An electronic device (100) includes a first processor (802) and a second processor (804) coupled to the first processor. The first processor (802) is to receive image data from a first imaging camera (114, 116) and to determine two-dimensional (2D) spatial feature data representing one or more 2D spatial features identified from the image data. The second processor (804) is to determine three-dimensional (3D) spatial feature data representing one or more 3D spatial features identified based on the 2D spatial feature data. Further, the first processor (802) can initiate detection of one or more 2D spatial features from a portion of an image frame prior to receiving the entire image frame.

Claims

1. An electronic device comprising:

a first processor to receive image data from a first imaging camera and to determine two-dimensional (2D) spatial feature data representing one or more 2D spatial features identified from the image data; and

a second processor, coupled to the first processor, to determine three-dimensional (3D) spatial feature data representing one or more 3D spatial features identified based on the 2D spatial feature data.

2. The electronic device of claim 1, wherein the first processor is to initiate detection of one or more 2D spatial features from a portion of an image frame prior to receiving the entire image frame.

3. The electronic device of claim 1, further comprising:

the first imaging camera, disposed at a first surface of the electronic device, and having a first field of view; and

a second imaging camera, disposed at the first surface of the electronic device, and having a second field of view narrower than the first field of view.

4. The electronic device of claim 3, further comprising:

a third imaging camera, disposed at a second surface of the electronic device, and having a third field of view greater than the second field of view; and

wherein the first processor is to determine the 2D spatial feature data further based on one or more 2D spatial features identified from image data captured by the third imaging camera.

5. The electronic device of claim 1, further comprising:

a depth sensor to capture depth data; and

wherein the second processor is to determine the 3D spatial feature data further based on the depth data.

6. The electronic device of claim 5, wherein:

the depth sensor includes;

a modulated light projector; and

the depth data includes;

image data captured by the first imaging camera and representing a reflection of a modulated light pattern projected by the modulated light projector.

7. The electronic device of claim 1, further comprising:

a sensor, coupled to the second processor, to provide non-image sensor data; and

wherein the second processor is to determine the 3D spatial feature data further based on the non-image sensor data.

8. The electronic device of claim 7, wherein:

for each image frame;

the first processor is to capture at least one sensor state of the sensor; and

the first processor is to determine a 2D spatial feature list of 2D spatial features identified in the image frame and to send the 2D spatial feature list and a representation of the at least one sensor state to the second processor.

9. The electronic device of claim 7, wherein the sensor comprises at least one selected from a group consisting of:

an accelerometer;

a gyroscope;

an ambient light sensor;

a magnetometer;

a gravity gradiometer;

a wireless cellular interface;

a wireless local area network interface;

a wired network interface;

a near field communications interface;

a global positioning system interface;

a microphone; and

a keypad.

10. A method comprising:

receiving, at a first processor of an electronic device, first image data captured by a first imaging camera of the electronic device, the first image data representing a first image frame;

determining, at the first processor, a first set of one or more two-dimensional (2D) spatial features from the first image data; and

determining, at a second processor of the electronic device, a set of one or more three-dimensional (3D) spatial features using the first set of one or more 2D spatial features.

11. The method of claim 10, further comprising:

receiving, at the first processor, second image data captured by a second imaging camera of the electronic device, the second image data representing a second image frame;

determining, at the first processor, a second set of one or more 2D spatial features from the second image data; and

wherein determining the set of one or more 3D spatial features includes;

determining the set of one or more 3D spatial features based on correlations between the first set of one or more 2D spatial features and the second set of one or more 2D spatial features.

12. The method of claim 11, further comprising:

aligning image data captured by the first imaging camera and image data captured by the second imaging camera to generate a combined image frame; and

displaying the combined image frame at the electronic device.

13. The method of claim 10, further comprising:

receiving, at the first processor, depth data captured by a depth sensor of the electronic device; and

wherein determining the set of one or more 3D spatial features includes;

determining the set of one or more 3D spatial features further based on the depth data.

14. The method of claim 10, further comprising:

determining, at the first processor, sensor data representative of a sensor state of at least one non-imaging sensor of the electronic device concurrent with the capture of the first image data; and

wherein determining the set of one or more 3D spatial features includes;

determining the set of one or more 3D spatial features further based on the sensor data.

15. A method comprising:

receiving, at a first processor of an electronic device, a first stream of image data captured by a first imaging camera of the electronic device, the first stream of image data representing a first image frame; and

determining, at the first processor, a first set of one or more two-dimensional (2D) spatial features for a portion of the first image frame, and sending first 2D spatial feature data representative of the first set of one or more 2D spatial features to a second processor of the electronic device while continuing to receive at the first processor a portion of the first stream of image data that represents a next portion of the first image frame.

16. The method of claim 15, further comprising:

determining, at the second processor, a first partial set of one or more three-dimensional (3D) spatial features based on the first 2D spatial feature data.

17. The method of claim 16, further comprising:

receiving, at the first processor, depth data captured by a depth sensor of the electronic device; and

wherein determining the first set of one or more 3D spatial features includes;

determining the first set of one or more 3D spatial features further based on the depth data.

18. The method of claim 16, further comprising:

receiving sensor data representative of a sensor state of at least one non-imaging sensor of the electronic device concurrent with receiving the first stream of image data; and

wherein determining the first set of one or more 3D spatial features includes;

determining the first set of one or more 3D spatial features further based on the sensor data.

19. The method of claim 18, wherein:

the non-imaging sensor is a gyroscope; and

wherein determining the first set of one or more 3D spatial features includes;

determining the first set of one or more 3D spatial features further based on an orientation reading from the gyroscope.

20. The method of claim 15, wherein:

the first imaging camera includes a rolling shutter imaging camera having a plurality of rows of pixel sensors; and

receiving the first stream of image data includes;

receiving a row-by-row stream of image data captured by the rolling shutter imaging camera;

the portion of the first image frame includes image data of a first set of one or more rows of the rolling shutter imaging camera; and

the next portion of the image frame includes image data of a second set of one or more rows of the rolling shutter imaging camera.

21. The method of claim 15, further comprising:

receiving, at the first processor, a second stream of image data captured by a second imaging camera of the electronic device, the second stream of image data representing a second image frame; and

determining, at the first processor, a second set of one or more 2D spatial features for the second image frame and streaming second 2D spatial feature data representative of the second set of one or more 2D spatial features to the second processor.

22. The method of claim 21, further comprising:

determining, at the second processor, a second set of one or more three-dimensional (3D) spatial features based on the second 2D spatial feature data;

receiving, at the first processor, depth data captured by a depth sensor of the electronic device; and

wherein determining the second set of one or more 3D spatial features includes;

determining the second set of one or more 3D spatial features further based on the depth data.