Room layout estimation methods and techniques

• US 10,657,376 B2
• Filed: 03/16/2018
• Issued: 05/19/2020
• Est. Priority Date: 03/17/2017
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Lee, Chen-Yu, Badrinarayanan, Vijay, Malisiewicz, Tomasz, Rabinovich, Andrew
• Primary Examiner(s): Park, Chan S
• Assistant Examiner(s): Rice, Elisa M
• Application Number:
  US15/923,511
• Publication Number:
  US 20180268220A1
• Time in Patent Office: 795 Days
• Field of Search: None
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/36 Indoor scenes

Abstract

Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.

Claims

1. A system comprising:

non-transitory memory configured to store;

a room image for room layout estimation; and

a neural network for estimating a layout of a room, the neural network comprising;

an encoder-decoder sub-network configured to receive the digital room image as an input, wherein the encoder-decoder sub-network comprises an encoder and a decoder; and

a classifier sub-network in communication with the encoder-decoder sub-network configured to classify a room type associated with the room image;

a hardware processor in communication with the non-transitory memory, the hardware processor programmed to;

access the room image;

determine, using the encoder, decoder, and the room image, a plurality of predicted two-dimensional (2D) keypoint maps corresponding to a plurality of room types;

determine, using the encoder, the classifier sub-network and the room image, a predicted room type from the plurality of room types;

determine, using the plurality of predicted 2D keypoint maps and the predicted room type, a plurality of ordered keypoints associated with the predicted room type; and

determine, using the plurality of ordered keypoints, a predicted layout of the room in the room image.

2. The system of claim 1, wherein each room type in the plurality of room types comprises an ordered set of room type keypoints.

3. The system of claim 2, wherein each room type in the plurality of room types comprises a semantic segmentation for regions in the room type, the semantic segmentation comprising an identification as a floor, a ceiling, or a wall.

4. The system of claim 2, wherein a first keypoint order is associated with a first room type of the plurality of room types and a second keypoint order is associated with a second room type of the plurality of room types, wherein the first keypoint order and the second keypoint order are different.

5. The system of claim 1, wherein the room image comprises a monocular image.

6. The system of claim 1, wherein the room image comprises a Red-Green-Blue (RGB) image.

7. The system of claim 1, wherein a dimensionality of the room image is larger than a dimensionality of the predicted 2D keypoint maps.

8. The system of claim 1, wherein the encoder-decoder sub-network comprises an encoder sub-network comprising a plurality of convolutional layers and a plurality of pooling layers.

9. The system of claim 1, wherein the encoder-decoder sub-network comprises a decoder sub-network comprising a plurality of convolutional layers and a plurality of upsampling layers.

10. The system of claim 1, wherein the encoder-decoder sub-network comprises a memory augmented recurrent encoder-decoder (MRED) network.

11. The system of claim 1, wherein the encoder-decoder sub-network comprises a plurality of recurrent layers.

12. The system of claim 11, wherein a number of recurrent iterations of the plurality of recurrent layers is two.

13. The system of claim 11, wherein a number of recurrent iterations of the plurality of recurrent layers is at least three.

14. The system of claim 11, wherein each of the plurality of recurrent layers has a weight matrix, and the weight matrix is the same for all of the plurality of recurrent layers.

15. The system of claim 1, wherein the predicted two-dimensional (2D) keypoint maps comprise heat maps.

16. The system of claim 15, wherein the hardware processor is programmed to extract keypoint locations from the heat maps as maxima of the heat map.

17. The system of claim 1, wherein the hardware processor is programmed to:

access object information from an object recognizer that analyzes the room image; and

combine the object information with the predicted layout of the room.

18. The system of claim 17, wherein the object recognizer is configured to detect cuboids in the room image.

19. A wearable display device comprising:

an outward-facing imaging system configured to capture the room image for room layout estimation; and

the system of claim 1.

20. A system comprising:

non-transitory memory configured to store parameters for the neural network; and

a hardware processor in communication with the non-transitory memory, the hardware processor programmed to;

receive a training room image, wherein the training room image is associated with;

a reference room type from a plurality of room types, and

reference keypoints associated with a reference room layout;

generate a neural network for room layout estimation, wherein the neural network comprises;

an encoder-decoder sub-network configured to receive the training room image as an input and output predicted two-dimensional (2D) keypoints associated with a predicted room layout associated with each of the plurality of room types, wherein the encoder-decoder sub-network comprises an encoder and a decoder, and

a side sub-network in communication with the encoder-decoder network configured to output a predicted room type from the plurality of room types, wherein the predicted room type is determined at least in part by the encoder and side sub-network; and

optimize a loss function based on a first loss for the predicted 2D keypoints and a second loss for the predicted room type; and

update parameters of the neural network based on the optimized loss function.

21. The system of claim 20, wherein a number of the plurality of room types is greater than 5.

22. The system of claim 20, wherein the reference keypoints and the predicted 2D keypoints are associated with a keypoint order.

23. The system of claim 20, wherein a first keypoint order is associated with a first room type of the plurality of room types and a second keypoint order is associated with a second room type of the plurality of room types, wherein the first keypoint order and the second keypoint order are different.

24. The system of claim 20, wherein the training room image comprises a monocular image.

25. The system of claim 20, wherein the training room image comprises a Red-Green-Blue (RGB) image.

26. The system of claim 20, wherein a dimensionality of the training room image is larger than a dimensionality of a map associated with the predicted 2D keypoints.

27. The system of claim 20, wherein the encoder sub-network and the decoder sub-network comprises a plurality of recurrent layers.

28. The system of claim 27, wherein a number of recurrent iterations of the recurrent layers is two or three.

29. The system of claim 27, wherein deep supervision is applied to the recurrent layers.

30. The system of claim 27, wherein weights associated with a first recurrent iteration of the iterations of the recurrent layers are identical to weights associated with a second recurrent iteration of the current layers.

31. The system of claim 27, wherein the plurality of recurrent layers are configured as a memory-augmented recurrent encoder-decoder (MRED) network.

32. The system of claim 20, wherein the side sub-network comprises a room type classifier.

33. The system of claim 20, wherein the first loss for the predicted 2D keypoints comprises a Euclidean loss between the plurality of reference keypoint locations and the predicted 2D keypoints.

34. The system of claim 20, wherein the second loss for the predicted room type comprises a cross entropy loss.

35. The system of claim 20, wherein the predicted 2D keypoints are extracted from a predicted heat map.

36. The system of claim 35, wherein hardware processor is programmed to:

calculate a reference heat map associated with the reference keypoints of the training image; and

calculate the first loss for the predicted 2D keypoints based on a difference between the predicted heat map and the reference heat map.

37. The system of claim 36, wherein the reference heat map comprises a two-dimensional distribution centered at a location for each reference keypoint.

38. The system of claim 36, wherein the reference heat map comprises a background away from the reference keypoints and a foreground associated with the reference keypoints, and the hardware processor is programmed to weight gradients in the reference heat map based on a ratio between the foreground and the background.

39. The system claim 38, wherein to weight the gradients in the reference heat map, the hardware processor is programmed to degrade values of pixels in the background by a degrading factor less than one.