Method for estimating free space using a camera system

• US 9,305,219 B2
• Filed: 01/23/2014
• Issued: 04/05/2016
• Est. Priority Date: 01/23/2014
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Ramalingam, Srikumar, Yao, Jian, Taguchi, Yuichi
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): CONNER, SEAN M
• Application Number:
  US14/162,370
• Publication Number:
  US 20150206015A1
• Time in Patent Office: 803 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation
• CPC Class Codes: G06V 10/85 Markov-related models; Markov random fields
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/586 of parking space

Abstract

A method estimates free space near a moving object from a sequence of images in a video acquired of a scene by a camera system arranged on the moving object by first constructing a one-dimensional graph, wherein each node corresponds to a column of pixels in the image. Features are determined in the image, and an energy function is constructed on the graph based on the features. Using dynamic programming, the energy function is maximized to obtain the free space.

Claims

1. A method for estimating free space near a moving object, comprising:

acquiring a sequence of images of a scene by a monocular camera system arranged on the moving object, and for each image in the sequence of images;

constructing a Markov random field as a one-dimensional graph, wherein each node in the graph corresponds to a discrete variable for a column of pixels in the image;

determining features in the image;

constructing an energy function on the one-dimensional graph based on the determined features; and

using dynamic programming to maximize the energy function to obtain a curve, wherein an area under the curve defines the free space near the object, wherein the free space is used for autonomous navigation of the moving object moving from one location to another, and wherein the steps are performed in a processor connected to the monocular camera system,

wherein the energy function comprises a sum of unary and pairwise potential functions, and wherein each potential function is determined using a corresponding feature among the determined features and a corresponding weight parameter learned from a sequence of training images.

2. The method of claim 1, wherein the moving object is a vehicle and the scene includes a road.

3. The method of claim 1, wherein the moving object is a boat and the scene includes water.

4. The method of claim 1, further comprising:

estimating, for each column of pixels in the image, vertical coordinates of an obstacle.

5. The method of claim 1, wherein the free space is in front of or behind the moving object.

6. The method of claim 4, further comprising:

using an estimate of a horizon to restrict the vertical coordinates of the obstacle.

7. The method of claim 6, wherein the estimate of the horizon is determined based on a configuration of the camera system.

8. The method of claim 6, wherein the estimate of the horizon is determined using a set of training images.

9. The method of claim 1, wherein the weight parameter is set manually.

10. The method of claim 1, wherein the weight parameter is determined automatically from a set of training images using a structured support vector machine.

11. The method of claim 1, wherein the features are selected from a group consisting of edges, appearance, homography, geometric context, smoothness, optical flow, a depth map and combinations thereof.

12. The method of claim 11, wherein the appearance is modeled by Gaussian mixture models.

13. The method of claim 1, further comprising:

determining a location prior probability for each pixel in the image to smooth the curve.

14. The method of claim 11, wherein the homography imposes smoothness across the images, wherien the homography is based on a ground plane in the image.

15. The method of claim 1, wherein a truncated quadratic penalty is used to smooth the curve.

16. The method of claim 1, wherein the moving object is a vehicle, and wherein the free space is used for autonomous navigation of the vehicle moving from one location to another.

17. The method of claim 1, wherein the moving object is a boat, and wherein the free space is used for manouvering the boat during berthing or mooring .

18. The method of claim 1, wherein the moving object is a vehicle, and wherein the free space is used for parking assistance for the vehicle.

19. The method of claim 1, wherein the moving object is a an indoor mobile robot, and wherein the free space is used by the indoor mobile robot for moving inside a building.

20. The method of claim 1, wherein the moving object is a vacuum cleaning robot, and wherein the free space is used by the vacuum cleaning robot.

21. A system for estimating free space near a moving object comprising:

a monocular camera system arranged on the moving objects for acquiring a sequence of images of a scene; and

a processor connected to the monocular camera system being operable to;

construct a Markov random field as a one-dimensional graph, wherein each node in the graph corresponds to a discrete variable for a column of pixels in the image,

determine features in the image,

construct an energy function on the one-dimensional graph based on the determined features; and

use dynamic programming to maximize the energy function to obtain a curve, wherein an area under the curve defines the free space near the object, wherein the free space is used for autonomous navigation of the moving object moving from one location to another,

wherein the energy function comprises a sum of unary and pairwise potential functions, and wherein each potential function is determined using a corresponding feature among the determined features and a corresponding weight parameter learned from a sequence of training images.