Automated control of image acquisition via use of acquisition device sensors

• US 10,708,507 B1
• Filed: 12/28/2018
• Issued: 07/07/2020
• Est. Priority Date: 10/11/2018
• Status: Active Grant

• Current Assignee: MFTB Holdco Incorporated
• Original Assignee: Zillow Group, Inc.
• Inventors: Dawson, Mitchell David, Guan, Li, Otwell, Andrew H., Hsiao, Dun-Yu
• Primary Examiner(s): Aggarwal, Yogesh K
• Application Number:
  US16/236,187
• Time in Patent Office: 557 Days
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Techniques are described for using computing devices to perform automated operations to control acquisition of images in a defined area, including obtaining and using data from one or more hardware sensors on a mobile device that is acquiring the images, analyzing the sensor data (e.g., in a real-time manner) to determine the geometric orientation of the mobile device in three-dimensional (3D) space, and using that determined orientation to control the acquisition of further images by the mobile device. In some situations, the determined orientation information may be used in part to automatically generate and display a corresponding GUI (graphical user interface) that is overlaid on and augments displayed images of the environment surrounding the mobile device during the image acquisition process, so as to control the mobile device'"'"'s geometric orientation in 3D space.

Claims

1. A computer-implemented method comprising:

determining, by a mobile computing device at a viewing location within a room, a series of target directions from the viewing location to use in acquiring multiple images for use in a composite panorama image of an interior of the room, wherein each of the target directions includes a target geometric orientation in three-dimensional space that includes values for rotation for each of tilt, swivel and pivot of the mobile computing device along three degrees of freedom;

acquiring, from a camera on the mobile computing device, the multiple images, including, for each successive target direction in the series;

determining a current direction of the camera from the viewing location based at least in part on analyzing data from one or more inertial measurement unit (IMU) sensors in the mobile computing device, wherein the current direction includes a current geometric orientation in three-dimensional space that includes values for rotation for each of tilt, swivel and pivot of the mobile computing device along three degrees of freedom;

displaying information on the mobile computing device, including a view currently visible from the camera in the current direction, and a current direction visual indicator overlaid in a center of the view to represent the mobile computing device, and a target direction visual indicator overlaid at a position in the view to identify the successive target direction in the series, wherein the position of the target direction visual indicator in the view is offset from the center of the view in an amount corresponding to a difference between the successive target direction and the determined current direction;

repeatedly updating the displayed information as camera direction changes until the current direction visual indicator matches the target direction visual indicator, including updating the view being displayed to reflect a changing view that is visible from the camera, and updating relative positions of the current direction and target direction visual indicators to reduce the difference based on the changes in the camera direction;

recording, by the camera, video during the displaying of the information and the updating of the displayed information; and

acquiring, from the camera and while the current direction visual indicator matches the target direction visual indicator, one of the multiple images that is in the successive target direction from the viewing location, including storing, by the mobile computing device and without corresponding instructions from a user moving the mobile computing device, information about one or more current frames of the video at a time of the current direction visual indicator matching the target direction visual indicator;

generating, by the mobile computing device and from the acquired multiple images, the composite panorama image to represent the interior of the room; and

displaying the composite panorama image to one or more users.

2. The computer-implemented method of claim 1 wherein the acquiring of the multiple images occurs while a user turns the mobile computing device in a 360 degree circle at the viewing location, wherein the one or more users are using client computing systems remote from the mobile computing device and are different from the user, wherein displaying of the current direction visual indicator includes displaying a rectangular visual element that represents the mobile computing device, wherein displaying of the target direction visual indicator for a successive target direction includes displaying an additional rectangular visual element, and wherein the method further comprises determining, by the mobile computing device, that the current direction visual indicator matches the target direction visual indicator when one of the displayed rectangular visual element and the displayed additional rectangular visual element fully overlaps the other of the displayed rectangular visual element and the displayed additional rectangular visual element.

3. A computer-implemented method comprising:

determining, for a mobile device at a viewing location within a room, a series of target directions from the viewing location to use in acquiring multiple images that capture at least some of an interior of the room;

determining, by the mobile device, a current direction of a camera of the mobile device based at least in part on data from one or more inertial measurement unit (IMU) sensors in the mobile device;

identifying, by the mobile device, one or more differences between the current direction of the camera and one or more of the target directions;

displaying, on the mobile device, information including a view from the camera in the current direction, and a current direction visual indicator in a center of the displayed view that represents the current direction of the camera, and one or more target direction visual indicators to represent the one or more target directions and that are at locations in the displayed view differing from the center of the displayed view by amounts corresponding to the determined one or more differences;

updating, by the mobile device, and until the current direction visual indicator matches one of the target direction visual indicators, the displayed information as a direction of the camera changes, including displaying a changing view from the camera, and maintaining the current direction visual indicator in a center of the displayed changing view, and updating the positions in the displayed changing view of the one or more target direction visual indicators, to enable a visual determination of changing differences from the one or more target directions to a current direction;

recording, by the camera, video during the displaying of the information and the updating of the displayed information; and

acquiring, by the camera and while the current direction visual indicator matches the one target direction visual indicator, one of the multiple images in one of the one or more target directions that is represented by the one target direction visual indicator, including storing, by the mobile device and without corresponding instructions from a user moving the mobile device, information about one or more current frames of the video at a time of the current direction visual indicator matching the one target direction visual indicator.

4. The computer-implemented method of claim 3 further comprising, after the acquiring of the one image in the one target direction, further acquiring images by the mobile device for each of the target directions of the series other than the one target direction, including repeatedly performing the determining of the current direction and the identifying and updating and the acquiring until the images are acquired for each of the target directions of the series other than the one target direction.

5. The computer-implemented method of claim 4 further comprising, after the acquiring of the image for the one target direction and the further acquiring of the images for each of the target directions of the series other than the one target direction, using at least some acquired images for target directions to generate a panorama image that provides a 360-degree view around a vertical axis from the viewing location.

9. The computer-implemented method of claim 3 wherein the determining of the series of target directions includes determining, by at least one computing device and for each of the target directions of the series, a target geometric orientation in three-dimensional space that includes values for rotation for each of tilt, swivel and pivot to correspond to three degrees of freedom, wherein the determining of the current direction of the camera includes determining a current geometric orientation of the camera in the three-dimensional space that includes values for rotation for each of tilt, swivel and pivot to correspond to the three degrees of freedom, and wherein the displaying of the one or more target direction visual indicators includes visually representing the determined target geometric orientation of each of the one or more target direction visual indicators relative to the determined current geometric orientation of the camera.

10. The computer-implemented method of claim 9 wherein the determining of the current direction of the camera further includes using the view from the camera in the current direction to determine a current location of the camera in the three-dimensional space that includes values of translation for each of three perpendicular axes of direction to correspond to three additional degrees of freedom, wherein the determining of the series of target directions includes determining a target location of each of the target directions in the three-dimensional space that includes values of translation for each of the three perpendicular axes of direction, and wherein the displaying of the one or more target direction visual indicators includes visually representing the determined target location of each of the one or more target direction visual indicators relative to the determined current location of the camera.

11. The computer-implemented method of claim 3 wherein the displaying of the current direction visual indicator includes displaying a rectangular visual element that represents the mobile device, wherein the displaying of the one or more target direction visual indicators includes displaying an additional rectangular visual element for each of the one or more target direction visual indicators that represents a different direction of the mobile device, and wherein the method further comprises determining, by the mobile device, that the current direction visual indicator matches the one target direction visual indicator when the displayed rectangular visual element and the displayed additional rectangular visual element of the one target direction visual indicator overlap each other at least in part.

12. The computer-implemented method of claim 3 wherein the displaying of the current direction visual indicator includes displaying a rectangular visual element that represents the mobile device, wherein the displaying of the one or more target direction visual indicators includes displaying an additional rectangular visual element for each of the one or more target direction visual indicators that represents a different direction of the mobile device, and wherein the method further comprises determining, by the mobile device, that the current direction visual indicator matches the one target direction visual indicator when one of the displayed rectangular visual element and the displayed additional rectangular visual element of the one target direction visual indicator fully overlaps the other of the displayed rectangular visual element and the displayed additional rectangular visual element of the one target direction visual indicator.

13. The computer-implemented method of claim 3 wherein the displaying of the current direction visual indicator includes determining a line in the view from the camera in the current direction that corresponds to a level horizontal portion of the room, and displaying at least a horizon line to represent the current direction of the mobile device relative to the level horizontal portion of the room.

14. The computer-implemented method of claim 3 wherein the displaying of the one or more target direction visual indicators includes displaying upper and lower boundaries for each of the one or more target direction visual indicators, and wherein the method further comprises determining, by the mobile device, that the current direction visual indicator matches the one target direction visual indicator when the displayed current direction visual element indicator is between the upper and lower boundaries for the one target direction visual indicator.

15. The computer-implemented method of claim 3 further comprising, after the acquiring of the one image in the one target direction, further updating, by the mobile device, the displayed information to change a visual appearance of the one target direction visual indicator to illustrate that an image has been acquired in the one target direction visual indicator.

16. The computer-implemented method of claim 15 further comprising determining, by the mobile device, one of multiple degrees to which the acquired one image satisfies one or more defined criteria, and wherein the further updating of the displayed information includes adding information to the changed visual appearance of the one target direction visual indicator to visually represent the determined one degree.

17. The computer-implemented method of claim 15 further comprising determining, by the mobile device, a characteristic of the acquired one image from multiple possible characteristics, and wherein the further updating of the displayed information includes adding information to the changed visual appearance of the one target direction visual indicator to visually represent the determined characteristic.

19. The computer-implemented method of claim 3 wherein the displaying of the information and the updating of the displayed information includes overlaying the current direction visual indicator and the one or more target direction visual indicators on the view and the changing view from the camera, and further includes displaying information to guide a user who is moving the mobile device in matching the current direction visual indicator and the one target direction visual indicator.

6. A computer-implemented method comprising:

determining, for a mobile device at a viewing location within a room, a series of target directions from the viewing location to use in acquiring multiple images that capture at least some of an interior of the room;

determining, by the mobile device, a current direction of a camera of the mobile device based at least in part on data from one or more inertial measurement unit (IMU) sensors in the mobile device;

identifying, by the mobile device, one or more differences between the current direction of the camera and one or more of the target directions;

displaying, on the mobile device, information including a view from the camera in the current direction, and a current direction visual indicator in a center of the displayed view that represents the current direction of the camera, and one or more target direction visual indicators to represent the one or more target directions and that are at locations in the displayed view differing from the center of the displayed view by amounts corresponding to the determined one or more differences;

updating, by the mobile device, and until the current direction visual indicator matches one of the target direction visual indicators, the displayed information as a direction of the camera changes, including displaying a changing view from the camera, and maintaining the current direction visual indicator in a center of the displayed changing view, and updating the positions in the displayed changing view of the one or more target direction visual indicators, to enable a visual determination of changing differences from the one or more target directions to a current direction;

acquiring, by the camera and while the current direction visual indicator matches the one target direction visual indicator, one of the multiple images in one of the one or more target directions that is represented by the one target direction visual indicator;

after the acquiring of the one image in the one target direction, further acquiring images by the mobile device for each of the target directions of the series other than the one target direction, including repeatedly performing the determining of the current direction and the identifying and updating and the acquiring until the images are acquired for each of the target directions of the series other than the one target direction;

after the acquiring of the image for the one target direction and the further acquiring of the images for each of the target directions of the series other than the one target direction, using at least some acquired images for target directions to generate a panorama image that provides a 360-degree view around a vertical axis from the viewing location;

for each of multiple additional viewing locations in additional rooms of a building that includes the room, acquiring images from multiple target directions from the additional viewing location, including repeatedly performing, for the additional viewing location, the determining of the current direction and the identifying and updating and the acquiring until the images are acquired for each of the multiple target directions;

generating, for each of the multiple additional viewing locations, a panorama image for the additional viewing location that is based at least in part on the acquired images for the multiple target directions from the additional viewing location;

generating, by at least one computing device, linking information between at least some pairs of generated panorama images that includes determining a direction between the generated panorama images of the pair; and

displaying, by the at least one computing device and to a user, multiple of the generated panorama images, including changing displayed information from one generated panorama image to another panorama image based on user instructions using displayed visual representations of the generated linking information.

7. The computer-implemented method of claim 6 further comprising:

generating, by the at least one computing device, a floor map of the building by analyzing the generated linking information and the generated panorama images for the viewing location and the multiple additional viewing locations; and

displaying, by the at least one computing device and to the user, the generated floor map of the building in response to instructions from the user.

8. The computer-implemented method of claim 7 wherein the mobile device is a handheld smartphone device, and wherein the at least one computing device is the smartphone device.

18. The computer-implemented method of claim 6 wherein the acquiring by the camera of the one image includes saving, by the mobile device and without corresponding instructions from a user moving the mobile device, a current view available from the camera as the one image.

20. A non-transitory computer-readable medium having stored contents that cause one or more computing devices to perform automated operations including at least:

determining, by the one or more computing devices, multiple target directions from a viewing location to use in acquiring multiple images for generating a panorama image from the viewing location;

determining, by the one or more computing devices, a current direction of a camera at the viewing location based at least in part on data from one or more sensors associated with the camera, and one or more differences between the current direction and one or more target directions of the multiple target directions;

displaying a view visible from the camera, and an overlaid visual indicator of a center of the displayed view, and one or more overlaid target direction visual indicators that represent the one or more target directions and that are at positions in the displayed view corresponding to the determined one or more differences;

updating, as camera direction changes, and until one of the target direction visual indicators is at the center of the displayed view, the displayed view visible from the camera and the positions in the displayed view of the one or more target direction visual indicators;

recording, by the camera, video during the displaying of the view and the updating of the displayed view; and

acquiring, by the camera and while the one target direction visual indicator is at the center of the displayed view, one of the multiple images in the target direction that is represented by the one target direction visual indicator, including storing, by the one or more computing devices and without corresponding instructions from a user, information about one or more current frames of the video at a time of the one target direction visual indicator being at the center of the displayed view.

21. The non-transitory computer-readable medium of claim 20 wherein the one or more computing devices include a mobile computing device that contains the camera, wherein the one or more sensors associated with the camera include multiple hardware sensors on the mobile computing device that are part of one or more inertial measurement units, and wherein the displaying of the overlaid visual indicator of the center of the displayed view includes displaying a current direction visual indicator at the center that represents the mobile computing device.

22. The non-transitory computer-readable medium of claim 20 wherein the displaying of the overlaid visual indicator of the center of the displayed view includes displaying horizontal and vertical lines that intersect at the center of the displayed view, and the displaying of the one or more overlaid target direction visual indicators that represent the one or more target directions further includes displaying the one or more overlaid target direction visual indicators to each have a rotation for one or more of tilt, swivel and pivot that represents current tilt, swivel and pivot of a device containing the camera, and wherein the automated operations further include determining to perform the acquiring of the one image based at least in part on the rotation of the one target direction visual indicator for the one or more of tilt, swivel and pivot matching the horizontal and vertical lines.

23. The non-transitory computer-readable medium of claim 20 wherein the displaying of the one or more overlaid target direction visual indicators that represent the one or more target directions further includes displaying upper and lower boundaries, and wherein the automated operations further include determining to perform the acquiring of the one image based at least in part on the one target direction visual indicator being within the upper and lower boundaries.

24. A system comprising:

one or more hardware processors of one or more computing devices; and

one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations including at least;

determining multiple target directions from a viewing location to use in acquiring multiple images from the viewing location;

determining a current direction of a camera at the viewing location based at least in part on data from one or more inertial measurement unit (IMU) sensors associated with the camera;

displaying a view visible from the camera and repeatedly updating the displayed view as the current direction of the camera changes, including overlaying one or more target direction visual indicators on the displayed view that represent one or more target directions of the multiple target directions and that are at changing positions in the displayed view from the updating corresponding to changing differences between the current direction and one or more target directions;

recording, by the camera, video during the displaying of the view and the updating of the displayed view; and

acquiring, by the camera and while one of the one or more target direction visual indicators is at a center of the displayed view, one of the multiple images in the target direction that is represented by the one target direction visual indicator, including storing, without corresponding instructions from a user, information about one or more current frames of the video at a time of the one target direction visual indicator being at the center of the displayed view.

25. The system of claim 24 wherein the one or more computing devices include a mobile computing device that contains the camera, wherein the displaying of the view and the repeated updating of the displayed view includes overlaying on the displayed view at least one current direction visual indicator representing the current direction of the camera, and wherein the automated operations further include determining to perform the acquiring of the one image based at least in part on the at least one current direction visual indicator matching the one target direction visual indicator.