SYSTEMS AND METHODS FOR GENERATING A 3D WORLD MODEL USING VELOCITY DATA OF A VEHICLE

• US 20170178352A1
• Filed: 12/19/2016
• Published: 06/22/2017
• Est. Priority Date: 12/18/2015
• Status: Active Grant

• Current Assignee: Iris Automation, Inc.
• Original Assignee: Iris Automation, Inc.
• Inventors: Harmsen, Alexander, Howard, James
• Granted Patent: US 10,242,455 B2
• Time in Patent Office: Days
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/106 Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08G 1/16 Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: G08G 3/02 Anti-collision systems
• CPC Class Codes: G08G 5/04 Anti-collision systems
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A self-contained, low-cost, low-weight guidance system for vehicles is provided. The guidance system can include an optical camera, a case, a processor, a connection between the processor and an on-board control system, and computer algorithms running on the processor. The guidance system can be integrated with a vehicle control system through “plug and play” functionality or a more open Software Development Kit. The computer algorithms re-create 3D structures as the vehicle travels and continuously updates a 3D model of the environment. The guidance system continuously identifies and tracks terrain, static objects, and dynamic objects through real-time camera images. The guidance system can receive inputs from the camera and the onboard control system. The guidance system can be used to assist vehicle navigation and to avoid possible collisions. The guidance system can communicate with the control system and provide navigational direction to the control system.

Claims

1. A method for generating a 3D world model, comprising:

receiving, by a computing device including one or more processors, from an image capture device mounted on a movable entity, a sequence of images including a first image captured at a first time and a plurality of second images captured after the first time, the first image of the sequence of images including a first plurality of pixels having respective parameter values, the plurality of second images each including a respective second plurality of pixels having respective parameter values;

identifying, by the computing device, motion data of the movable entity;

tracking, by the computing device, movement of at least a subset of the first plurality of pixels of the first image across the sequence of images;

estimating, by the computing device responsive to tracking the subset of the first plurality of pixels and using the received motion data, a depth value for each pixel of the subset of the first plurality of pixels; and

generating, by the computing device, using the estimated depth value for each pixel of the subset of the first plurality of pixels, a keyframe to populate a point cloud including a plurality of points, each point corresponding to a respective pixel of the subset of the first plurality of pixels.

2. The method of claim 1, further comprising generating a 3D world model using the generated keyframe.

3. The method of claim 1, wherein the identifying, by the computing device, motion data of the movable entity includes receiving at least one of velocity data or acceleration data of the movable entity.

4. The method of claim 1, further comprising associating, by the computing device, to each image of the sequence of images, a respective velocity of the movable entity corresponding to a time the image was captured by the image capture device.

5. The method of claim 1, wherein tracking, by the computing device, movement of at least a subset of the first plurality of pixels of the first image across the sequence of images includes tracking, by the computing device, movement of at least a subset of the first plurality of pixels of the first image across the sequence of images using the respective parameter value of the pixel.

6. The method of claim 1, wherein estimating, by the computing device, a depth value for each pixel of the subset of the first plurality of pixels includes estimating the depth value for each pixel of the subset of the first plurality of pixels using at least one of velocity data or acceleration data of the movable entity;

and further comprising;

publishing, by the computing device, a depth map including the estimated depth values for each pixel of the subset of the first plurality of pixels of the first image being tracked across the sequence of images, the depth map including, for each tracked pixel, absolute distance information calculated based on a distance the tracked pixel moved across the sequence of images and the velocity data of the movable entity.

7. The method of claim 1, wherein the sequence of images includes a first sequence of images and further comprising:

receiving, by the computing device, a second sequence of images after the first sequence of images; and

updating the keyframe responsive to receiving each image of the second sequence of images.

8. The method of claim 7, wherein the keyframe includes a first keyframe and further comprising updating, by the computing device, the point cloud with a second keyframe, the second keyframe generated using the first keyframe and a subset of the second sequence of images captured after the first keyframe is generated.

9. A system for generating a 3D world model, comprising:

a hardware processor;

a memory coupled to the at least one processor;

an image receiver configured to receive, from an image capture device mounted on a movable entity, a sequence of images including a first image captured at a first time and a plurality of second images captured after the first time, the first image of the sequence of images including a first plurality of pixels having respective parameter values, the plurality of second images each including a respective second plurality of pixels having respective parameter values;

a motion data receiver configured to receive motion data of the movable entity;

a pixel movement tracker configured to track movement of at least a subset of the first plurality of pixels of the first image across the sequence of images;

a depth value estimator configured to estimate, responsive to the tracked subset of the first plurality of pixels and using the received motion data, a depth value for each pixel of the subset of the first plurality of pixels; and

a keyframe generator configured to generate, using the estimated depth value for each pixel of the subset of the first plurality of pixels, a keyframe to populate a point cloud including a plurality of points, each point corresponding to a respective pixel of the subset of the first plurality of pixels.

10. The system of claim 9, further comprising a 3D world model generator configured to generate a 3D world model using the generated keyframe.

11. The system of claim 9, wherein the motion data includes at least one of velocity data or acceleration data of the movable entity.

12. The system of claim 9, wherein the image receiver is configured to associate, to each image of the sequence of images, a respective velocity or acceleration of the movable entity corresponding to a time the image was captured by the image capture device.

13. The system of claim 9, wherein the pixel movement tracker is configured to track the movement of at least the subset of the first plurality of pixels of the first image across the sequence of images using the respective parameter values of the subset of the first plurality of pixels.

14. The system of claim 9, wherein the depth value estimator is configured to estimate the depth value for each pixel of the subset of the first plurality of pixels using velocity data of the movable entity;

and wherein the keyframe generator is further configured to publish a depth map including the estimated depth values for each pixel of the subset of the first plurality of pixels of the first image being tracked across the sequence of images, the depth map including, for each tracked pixel, absolute distance information calculated based on a distance the tracked pixel moved across the sequence of images and the velocity data of the movable entity, the absolute distance information between the movable entity and the dynamic object.

15. The system of claim 9, wherein the sequence of images includes a first sequence of images and wherein:

the image receiver if further configured to receive a second sequence of images after the first sequence of images; and

the keyframe generator is further configured to update the keyframe responsive to receiving each image of the second sequence of images.

16. The system of claim 15, wherein the keyframe includes a first keyframe and wherein the keyframe generator is further configured to update the point cloud with a second keyframe, the second keyframe generated using the first keyframe and a subset of the second sequence of images captured after the first keyframe is generated.

17. A computer readable storage medium including computer-executable instructions stored thereon, which when executed by a computer, causes the computer to:

receive, from an image capture device mounted on a movable entity, a sequence of images including a first image captured at a first time and a plurality of second images captured after the first time, the first image of the sequence of images including a first plurality of pixels having respective parameter values, the plurality of second images each including a respective second plurality of pixels having respective parameter values;

receive motion data of the movable entity;

track movement of at least a subset of the first plurality of pixels of the first image across the sequence of images;

estimate, responsive to tracking the subset of the first plurality of pixels and using the received motion data, a depth value for each pixel of the subset of the first plurality of pixels; and

generate, using the estimated depth value for each pixel of the subset of the first plurality of pixels, a keyframe to populate a point cloud including a plurality of points, each point corresponding to a respective pixel of the subset of the first plurality of pixels.

18. The computer readable storage medium of claim 17, wherein the motion data of the movable entity includes velocity data of the movable entity and having further computer-executable instructions stored thereon, which when executed by the computer, causes the computer to associate, to each image of the sequence of images, a respective velocity of the movable entity corresponding to a time the image was captured by the image capture device.

19. The computer readable storage medium of claim 17, wherein estimating a depth value for each pixel of the subset of the first plurality of pixels includes estimating the depth value for each pixel of the subset of the first plurality of pixels using velocity data of the movable entity;

and having further computer-executable instructions stored thereon, which when executed by the computer, causes the computer to publish a depth map including the estimated depth values for each pixel of the subset of the first plurality of pixels of the first image being tracked across the sequence of images, the depth map including, for each tracked pixel, absolute distance information calculated based on a distance the tracked pixel moved across the sequence of images and the velocity data of the movable entity.

20. The computer readable storage medium of claim 17, wherein the sequence of images includes a first sequence of images and further computer-executable instructions stored thereon, which when executed by the computer, causes the computer to:

receive a second sequence of images after the first sequence of images; and

update the keyframe responsive to receiving each image of the second sequence of images.