Indoor scene capture system

• US 9,888,215 B2
• Filed: 04/25/2014
• Issued: 02/06/2018
• Est. Priority Date: 04/26/2013
• Status: Expired due to Fees

• Current Assignee: University of Washington
• Original Assignee: University of Washington
• Inventors: Sankar, Aditya, Seitz, Steven Maxwell
• Primary Examiner(s): Bailey, Frederick
• Application Number:
  US14/262,439
• Publication Number:
  US 20140320661A1
• Time in Patent Office: 1,383 Days
• Field of Search: None
• CPC Class Codes: G01C 21/188 for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An indoor scene capture system is provided that, with a handheld device with a camera, collects videos of rooms, spatially indexes the frames of the videos, marks doorways between rooms, and collects videos of transitions from room to room via doorways. The indoor scene capture system may assign a direction to at least some of the frames based on the angle of rotation as determined by an inertial sensor (e.g., gyroscope) of the handheld device. The indoor scene capture system marks doorways within the frames of the videos. For each doorway between rooms, the indoor scene capture system collects a video of transitioning through the doorway as the camera moves from the point within a room through the doorway to a point within the adjoining room.

Claims

1. A method in a handheld device with a camera for collecting a video of rooms, the method comprising:

for each room,

collecting a video of the room as a user holding the handheld device rotates around a point within the room and assigning directions to frames of the video as the frames are collected; and

receiving from the user an indication of locations of corners of the room and doorways within the frames of the video as the frames are collected;

for each doorway between a first room and a second room,

collecting a video of the doorway as the user moves from the point within the first room through the doorway to a point within the second room; and

generating a layout of the rooms based on the collected video of the rooms and the locations of the corners of the rooms and the doorways with in the frames of the video, wherein the generating of the layout is based on walls of a room that meet at a corner being at a right angle.

2. The method of claim 1 wherein the handheld device is a smartphone.

3. The method of claim 1 wherein the directions are assigned based on angle of rotation as determined by an inertial sensor of the handheld device.

4. The method of claim 3 wherein the directions are adjusted to counter drift using a magnetometer of the handheld device.

5. The method of claim 4 including triangulating a distance from the point within the room to a corner of the room based on the parallax of the frames with indicated locations of corners.

6. A method in a handheld device with a camera for collecting a video of rooms, the method comprising:

for each room,

collecting a video of the room as a user holding the handheld device rotates around a point within the room and assigning directions to frames of the video as the frames are collected; and

marking doorways within the frames of the videos;

for each doorway between a first room and a second room,

collecting a video of the doorway as the user moves from the point within the first room through the doorway to a point within the second room; and

generating a layout of a room by assuming different directions of a wall from an initial position of an initial corner to an initial adjacent corner, calculating positions of corners with the corners being right angles, calculating a difference from the initial position to a final position of the initial corner, and selecting the calculated position associated with the smallest calculated difference as representing the layout of the room.

7. The method of claim 6 including aligning layouts of adjoining rooms by translating a layout of a room so that an edge of a doorway is coincident with the corresponding edge of the doorway of an adjoining room and rotating the layout of a room so that a common wall is coincident with the common wall of an adjoining room.

8. A computer-readable storage medium storing computer-executable instructions for controlling a handheld device with a camera to collect videos of rooms, the computer-executable instructions comprising instructions that:

collect a video of a room as a user holding the handheld device rotates around a point within the room;

assign directions to frames of the video as the frames are collected based on readings from an inertial sensor of the handheld device and as later adjusted based on readings of a magnetometer of the handheld device;

mark corners of the room as the frames are collected based on user input; and

generate a layout of the room based on the marked corners and directions of frames containing the marked corners based on the walls that meet at a marked corner being at a right angle.

9. The computer-readable storage medium of claim 8 further comprising instructions that collect a video as the user moves from the point within a room through a doorway indicated by the user to a point within an adjacent room.

10. The computer-readable storage medium of claim 8 further comprising instructions that align adjoining rooms based on the doorway between the adjoining rooms.

11. The computer-readable storage medium of claim 8 further comprising instructions that triangulate distances from the point within the room to each corner based on parallax resulting from frames taken from different directions.

12. A computer-readable storage medium storing computer-executable instructions for controlling a handheld device with a camera to collect videos of rooms, the computer-executable instructions comprising instructions that:

collect a video of a room as a user holding the handheld device rotates around a point within the room;

assign directions to frames of the video as the frames are collected based on readings from an inertial sensor of the handheld device and as later adjusted based on readings of a magnetometer of the handheld device;

mark corners of the room as the frames are collected based on user input;

generate a layout of the room based on the marked corners and directions of frames containing the marked corners;

triangulate distances from the point within the room to each corner based on parallax resulting from frames taken from different directions; and

scale the layout of the room to fit the triangulated distances and calculate the dimension of each wall of the room based on the scaled layout.

13. A method performed by a handheld device for determining a layout of a room with walls and corners from mappings of corners of the room to directions of the corners from a point within the room, each corner having a ray from the point in the direction of the corner, the method comprising:

selecting a starting position along the ray of an initial corner;

selecting an initial adjacent corner that is adjacent to the initial corner;

for each of a plurality of directions,

determining a position along the ray of the initial adjacent corner, assuming the wall between the initial corner and the initial adjacent corner is in that direction;

for each of the remaining corners in order of adjacency until a calculated position is calculated along the ray of the initial corner, calculating a position along the ray of the next adjacent corner, assuming the walls are at right angles; and

calculating a difference between the starting position and the calculated position along the ray of the initial corner; and

selecting the positions of the corners associated with the direction with the smallest difference as the layout of the room.

14. The method of claim 13 wherein the mappings of corners to directions of the corners are generated by:

collecting video of frames as a user holding a handheld device with a camera rotates around the point within the room;

assigning directions to the frames;

receiving an indication of the corners within the frames; and

determining the directions of the corners based on the directions of the frames with the indicated corners.

15. A method performed by a handheld device with a camera for determining dimensions of a layout of a room having walls, the method comprising:

providing frames of a video as a user holding the camera rotates around a point in the room, the frames having directions;

for each corner of the room,

providing mappings of locations of that corner within multiple frames; and

determining a triangulated distance from the point in the room to that corner using triangulation based on the mappings and the directions of the frames;

selecting a scale for the room by minimizing the differences between the triangulated distances and calculated distances assuming that scale; and

calculating the dimensions of the walls of the room assuming the selected scale.