Method for Estimating Free Space using a Camera System

• US 20150206015A1
• Filed: 01/23/2014
• Published: 07/23/2015
• Est. Priority Date: 01/23/2014
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Ramalingam, Srikumar, Taguchi, Yuichi, Yao, Jian
• Granted Patent: US 9,305,219 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation
• CPC Class Codes: G06V 10/85 Markov-related models; Markov random fields
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/586 of parking space

Abstract

A method estimates free space near a moving object from a sequence of images in a video acquired of a scene by a camera system arranged on the moving object by First constructing a one-dimensional graph, wherein each node corresponds to a column of pixels in the image. Features are determined in the image, and an energy function is constructed on the graph based on the features. Using dynamic programming, the energy function is maximized to obtain the free space.

Claims

1. A method for estimating free space near a moving object from a sequence of images in a video acquired of a scene by a camera system arranged on the moving object, comprising for each image in the video the steps of:

constructing a one-dimensional graph, wherein each node in the graph corresponds to a column of pixels in the image;

determining features in the image;

constructing an energy function on the graph based on the features; and

using dynamic programming to maximize the energy function to obtain a curve, wherein an area under the curve defines the free space, and wherein the steps are performed in a processor.

2. The method of claim 1, wherein the camera system is a monocular video system.

3. The method of claim 1, wherein the camera system is a stereo or multi-camera system.

4. The method of claim 1, wherein the moving object is a vehicle and the scene includes a road.

5. The method of claim 1, wherein the moving object is a boat and the scene includes water.

6. The method of claim 1, further comprising:

estimating, for each column of pixels in the image, vertical coordinates of an obstacle.

7. The method of claim 1, wherein the free space is in front of or behind the moving object.

8. The method of claim 6, further comprising:

using an estimate of a horizon to restrict the vertical coordinates of the obstacle.

9. The method of claim 8, wherein the estimate of the horizon is determined based on a configuration of the camera system.

10. The method of claim 8, wherein the estimate of the horizon is determined using a set of training images.

11. The method of claim 1, wherein the energy function is defined as a sum of potential functions, wherein each potential function is determined using each feature and a corresponding weight parameter.

12. The method of claim 11, wherein the weight parameter is set manually.

13. The method of claim 11, wherein the weight parameter is determined automatically from a set of training images using a structured support vector machine.

14. The method of claim 1, wherein the features are selected from a group consisting of edges, appearance, homography, geometric context, smoothness, optical flow, a depth map and combinations thereof.

15. The method of claim 14, wherein the appearance is modeled by Gaussian mixture models.

16. The method of claim 1, further comprising:

determining a location prior probability for each pixel in the image to smooth the curve.

17. The method of claim 14, wherein the homography imposes smoothness across the images, wherein the homography is based on a ground plane in the image.

18. The method of claim 1, wherein a truncated quadratic penalty is used to smooth the curve.

19. The method of claim 1, wherein the free space is used for autonomous navigation of vehicles moving from one location to another.

20. The method of claim 1, wherein the free space is used for manouvering a boat during berthing or mooring.

21. The method of claim 1, wherein the free space is used for parking assistance for vehicles.

22. The method of claim 1, wherein the free space is used by an indoor mobile robot for moving inside a building.

25. The method of claim 1, wherein the free space is used by a vacuum cleaning robot.