INDOOR SCENE CAPTURE SYSTEM

• US 20140320661A1
• Filed: 04/25/2014
• Published: 10/30/2014
• Est. Priority Date: 04/26/2013
• Status: Active Grant

• Current Assignee: University of Washington
• Original Assignee: University of Washington
• Inventors: Sankar, Aditya, Seitz, Steven Maxwell
• Granted Patent: US 9,888,215 B2
• Time in Patent Office: Days
• US Class Current: 348/158
• CPC Class Codes: G01C 21/188 for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An indoor scene capture system is provided that, with a handheld device with a camera, collects videos of rooms, spatially indexes the frames of the videos, marks doorways between rooms, and collects videos of transitions from room to room via doorways. The indoor scene capture system may assign a direction to at least some of the frames based on the angle of rotation as determined by an inertial sensor (e.g., gyroscope) of the handheld device. The indoor scene capture system marks doorways within the frames of the videos. For each doorway between rooms, the indoor scene capture system collects a video of transitioning through the doorway as the camera moves from the point within a room through the doorway to a point within the adjoining room.

Claims

1. A method in a handheld device with a camera for collecting a video of rooms, the method comprising:

for each room,

collecting a video of the room as a user holding the handheld device rotates around a point within the room and assigning directions to frames of the video as the frames are collected; and

marking doorways within the frames of the videos; and

for each doorway between a first room and a second room,

collecting a video of the doorway as the user moves from the point within the first room through the doorway to a point within the second room.

2. The method of claim 1 wherein the handheld device is a smartphone.

3. The method of claim 1 wherein the directions are assigned based on angle of rotation as determined by an inertial sensor of the handheld device.

4. The method of claim 3 wherein the directions are adjusted to counter drift using a magnetometer of the handheld device.

5. The method of claim 1 including, during the collecting of the video of a room, receiving from the user an indication of locations of corners of the room within the frames.

6. The method of claim 5 including triangulating a distance from the point within the room to a corner of the room based on the parallax of the frames with indicated locations of corners.

7. The method of claim 1 including generating a layout of a room by assuming different directions of a wall from an initial position of an initial corner to an initial adjacent corner, calculating positions of corners with the corners being right angles, calculating a difference from the initial position to a final position of the initial corner, and selecting the calculated position associated with the smallest calculated difference as representing the layout of the room.

8. The method of claim 7 including aligning layouts of adjoining rooms by translating a layout of a room so that an edge of a doorway is coincident with the corresponding edge of the doorway of an adjoining room and rotating the layout of a room so that a common wall is coincident with the common wall of an adjoining room.

9. A computer-readable storage medium storing computer-executable instructions for controlling a handheld device with a camera to collect videos of rooms, the computer-executable instructions comprising instructions that:

collect a video of a room as a user holding the handheld device rotates around a point within the room;

assign directions to frames of the video as the frames are collected based on readings from an inertial sensor of the handheld device and as later adjusted based on readings of a magnetometer of the handheld device;

mark corners of the room as the frames are collected based on user input; and

generate a layout of the room based on the marked corners and directions of frames containing the marked corners.

10. The computer-readable storage medium of claim 9 further comprising instructions that collect a video as the user moves from the point within a room through a doorway indicated by the user to a point within an adjacent room.

11. The computer-readable storage medium of claim 9 further comprising instructions that align adjoining rooms based on the doorway between the adjoining rooms.

12. The computer-readable storage medium of claim 9 further comprising instructions that triangulate distances from the point within the room to each corner based on parallax resulting from frames taken from different directions.

13. The computer-readable storage medium of claim 12 further comprising instructions that scale the layout of the room to fit the triangulated distances and calculate the dimension of each wall of the room based on the scaled layout.

14. A method in a handheld device with a camera and an inertial sensor for collecting video of a room, the method comprising:

collecting frames of the video as a user holding the handheld device rotates around a point within the room;

for at least some of the frames of the video, determining an angle of rotation of the handheld device at the time of the collecting of the frames based on a reading from the inertial sensor;

receiving an indication of an ending frame that corresponds to a full rotation;

determining a difference between the angle of rotation for the ending frame and a full rotation; and

adjusting the angles of rotation for the frames to compensate for the difference.

15. The method of claim 14 wherein the handheld device includes a magnetometer and a full rotation is detected based on readings of the magnetometer.

16. The method of claim 14 wherein the full rotation is detected based on user input.

17. A method performed by a handheld device for determining a layout of a room with walls and corners from mappings of corners of the room to directions of the corners from a point within the room, each corner having a ray from the point in the direction of the corner, the method comprising:

selecting a starting position along the ray of an initial corner;

selecting an initial adjacent corner that is adjacent to the initial corner;

for each of a plurality of directions,

determining a position along the ray of the initial adjacent corner, assuming the wall between the initial corner and the initial adjacent corner is in that direction;

for each of the remaining corners in order of adjacency until a calculated position is calculated along the ray of the initial corner, calculating a position along the ray of the next adjacent corner, assuming the walls are at right angles; and

calculating a difference between the starting position and the calculated position along the ray of the initial corner; and

selecting the positions of the corners associated with the direction with the smallest difference as the layout of the room.

18. The method of claim 17 wherein the mappings of corners to directions of the corners are generated by:

collecting video of frames as a user holding a handheld device with a camera rotates around the point within the room;

assigning directions to the frames;

receiving an indication of the corners within the frames; and

determining the directions of the corners based on the directions of the frames with the indicated corners.

19. A method performed by a handheld device with a camera for determining dimensions of a layout of a room having walls, the method comprising:

providing frames of a video as a user holding the camera rotates around a point in the room, the frames having directions;

for each corner of the room,

providing mappings of locations of that corner within multiple frames; and

determining a triangulated distance from the point in the room to that corner using triangulation based on the mappings and the directions of the frames;

selecting a scale for the room by minimizing the differences between the triangulated distances and calculated distances assuming that scale; and

calculating the dimensions of the walls of the room assuming the selected scale.

20. A method performed by a handheld device for presenting a video of rooms and transitions between rooms, the video of a room collected as a user holding a camera rotates around a point in the room, each frame of a video of a room having a direction, the video of a transition being collected by the user holding the camera moving between rooms, the method comprising:

presenting the video of a room by displaying the frame associated with an initial direction and displaying subsequent frames associated with subsequent directions; and

when a user selects a doorway that is currently being displayed, the doorway connecting an adjoining room,

presenting the video of the transition through that doorway; and

presenting the video of an adjoining room by displaying a frame associated with an initial direction and displaying subsequent frames associated with subsequent directions.