Generating a three-dimensional model using a portable electronic device recording

• US 8,294,766 B2
• Filed: 01/28/2009
• Issued: 10/23/2012
• Est. Priority Date: 01/28/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Tsai, Richard, Just, Andrew, Harris, Brandon
• Primary Examiner(s): Chang, Jungwon
• Application Number:
  US12/361,323
• Publication Number:
  US 20100188503A1
• Time in Patent Office: 1,364 Days
• Field of Search: 348/113, 348/231.5, 348/333.02, 348/148, 348/135, 348/208.1, 701/207, 701/209, 455/404.1, 707/731, 707/758, 345/419
• US Class Current: 348/148
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G06F 1/1613 for portable computers cooling arrangements therefor G06F1/203; constructional details or arrangements for pocket calculators, electronic agendas or books G06F15/0216; constructional details of portable telephone sets: with several bodies H04M1/0202
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 2200/1614 Image rotation following screen orientation, e.g. switching from landscape to portrait mode
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/422 Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

Systems and methods are providing for navigating a three-dimensional model using deterministic movement of an electronic device. An electronic device can load and provide an initial display of a three dimensional model (e.g., of an environment or of an object). As the user moves the electronic device, motion sensing components, positioning circuitry, and other components can detect the device movement and adjust the displayed portion of the three-dimensional model to reflect the movement of the device. By walking with the device in the user'"'"'s real environment, a user can virtually navigate a representation of a three-dimensional environment. In some embodiments, a user can record an object or environment using an electronic device, and tag the recorded images or video with movement information describing the movement of the device during the recording. The recorded information can then be processed with the movement information to generate a three-dimensional model of the recorded environment or object.

Claims

1. A method for generating a three-dimensional model, comprising:

recording a first image of a first portion of an object using an electronic device;

at least one of prior to the recording and during the recording, determining a first distance from the first portion of the object to the electronic device;

associating the determined first distance with the recorded first image using the electronic device;

after the recording the first image;

recording a second image of a second portion of the object using the electronic device; and

detecting with the electronic device a first movement of the electronic device;

associating using the electronic device the recorded second image with the detected first movement;

processing with the electronic device the recorded first image, the determined first distance, the recorded second image, and the detected first movement; and

generating with the electronic device at least a portion of the three-dimensional model in response to the processing.

2. The method of claim 1,

wherein the first portion of the object is different from the second portion of the object.

3. The method of claim 2, wherein the processing comprises:

determining a position of the electronic device based on the determined first distance and the detected first movement; and

converting the determined first distance and the determined position from a first coordinate system that is associated with the electronic device to a second coordinate system that is independent of the electronic device.

4. The method of claim 3, wherein the second coordinate system is static relative to an environment of the object.

5. The method of claim 1, wherein the detecting comprises:

receiving an output of a movement detection component of the electronic device that indicates the first movement.

6. The method of claim 5, wherein the associating comprises:

determining that the received output of the movement detection component is received during the recording the second image.

7. The method of claim 6, wherein the associating further comprises:

processing the received output to determine a change in position of the electronic device relative to the determined first distance.

8. The method of claim 1, wherein the determining comprises receiving with the electronic device a user input of the first distance.

9. The method of claim 1, wherein the determining further comprises determining a second distance from the electronic device to a ground.

10. The method of claim 1, wherein the determining comprises determining using at least one of a radar sensor and a sonar sensor of the electronic device.

11. The method of claim 1, wherein the determining further comprises sensing an initial orientation of the electronic device.

12. The method of claim 11, wherein the sensing comprises sensing with a magnometer of the electronic device.

13. The method of claim 1, further comprising defining at least one recommended path through which to navigate the generated three-dimensional model.

14. An electronic device operative to capture information of an object for a three-dimensional model of the object, comprising:

a lens operative to capture a recording of a plurality of portions of the object;

a motion-sensing component operative to provide at least one output that reflects movement of the lens of the electronic device; and

a processor operative to;

direct the lens to capture the recording;

receive a plurality of outputs provided by the motion-sensing component during the capturing;

associate each output of the received plurality of outputs with a corresponding portion of the recording that is captured at the same time as that output is received;

provide the three-dimensional model of the object based on the associations; and

define at least one recommended path for navigating the provided three-dimensional model based on the received plurality of outputs.

15. The electronic device of claim 14, wherein:

the captured recording comprises at least a first portion and a second portion;

the processor is operative to associate each output of the received plurality of outputs by associating;

a first output of the received plurality of outputs with the first portion of the captured recording; and

a second output of the received plurality of outputs with the second portion of the captured recording; and

the processor is further operative to define at least one mathematical quantity based on the first output and the second output.

16. The electronic device of claim 14, wherein the processor is further operative to:

at least one of prior to the capturing and at a beginning of the capturing, determine an initial position of the electronic device and an initial orientation of the electronic device;

associate the determined initial position and the determined initial orientation with a beginning portion of the captured recording; and

define at least one mathematical quantity based on the determined initial position, the determined initial orientation, and a respective output of the received plurality of outputs that is associated with a portion of the captured recording that is captured after the beginning portion is captured.

17. The electronic device of claim 14, wherein the processor is operative to:

provide the three-dimensional model by generating the three-dimensional model based on the associating.

18. The electronic device of claim 14, wherein the processor is operative to provide the three-dimensional model by:

sending the captured recording and the received plurality of outputs to a host device for generating the three-dimensional model; and

receiving the generated three-dimensional model from the host device.

19. The electronic device of claim 14, wherein the processor is operative to:

provide the three-dimensional model by displaying at least a portion of the three-dimensional model; and

define the at least one recommended path by overlaying the at least one recommended path on at least a portion of the displayed at least a portion of the three-dimensional model.

20. The electronic device of claim 14, wherein the processor is operative to:

provide the three-dimensional model by displaying at least a portion of the three-dimensional model; and

define the at least one recommended path by displaying at least one element that points to the at least one recommended path.

21. The electronic device of claim 20, wherein the at least one element comprises at least one arrow.

22. A host device operative to generate a three-dimensional model of an object, comprising a processor operative to:

receive from an electronic device;

a recording of a plurality of portions of the object, wherein the received recording comprises at least;

a first image of a first portion of the plurality of portions of the object; and

a second image of a second portion of the plurality of portions of the object;

a first distance from the first portion of the plurality of portions of the object to the electronic device, wherein the received first distance is associated with the first image; and

first movement information of the electronic device, wherein the received first movement information is associated with the second image; and

create at least a portion of the three-dimensional model of the object based at least on the received first image, the received second image, the received first distance, and the received first movement information.

23. The host device of claim 22, wherein the first portion of the object is different from the second portion of the object.

24. The host device of claim 22, wherein the received first distance comprises a user input of the first distance to the electronic device.

25. The host device of claim 22, wherein the processor is further operative to receive from the electronic device a second distance from the electronic device to a ground.

26. The host device of claim 25, wherein the processor is further operative to create the at least a portion of the three-dimensional model based at least on the received second distance.

27. The host device of claim 22, wherein the processor is operative to create the at least a portion of the three-dimensional model by determining a position of the electronic device based on the received first distance and the received first movement.

28. The host device of claim 22, wherein the received recording further comprises a third image of a third portion of the object, and wherein the processor is further operative to receive from the electronic device:

second movement information of the electronic device, wherein the received second movement information is associated with the third image.

29. The host device of claim 28, wherein the processor is further operative to:

create at least another portion of the three-dimensional model based at least on the third image and the received second movement information.

30. A method for generating a three-dimensional model, comprising:

capturing with an electronic device a plurality of images of an object;

during the capturing, receiving with the electronic device a plurality of outputs from a motion sensing component of the electronic device;

associating each output of the received plurality of outputs with a corresponding image of the captured plurality of images that is captured at a time when that output is received using the electronic device;

providing with the electronic device the three-dimensional model based on the associating; and

defining with the electronic device at least one recommended path for navigating the provided three-dimensional model based on the received plurality of outputs.

31. The method of claim 30, wherein the associating comprises associating each output of the received plurality of outputs as metadata for the corresponding image.

32. The method of claim 30, wherein the associating comprises linking each output of the received plurality of outputs to the corresponding image in a distinct file.

33. The method of claim 30, wherein:

the plurality of images is captured in a first sequence;

the provided three-dimensional model comprises the captured plurality of images; and

the method further comprises navigating with the electronic device at least a portion of the captured plurality of images of the provided three-dimensional model in an order that is different from the first sequence.

34. The method of claim 30, wherein the captured plurality of images comprises a video.

35. Non-transitory computer readable media for generating a three-dimensional model, comprising computer readable code recorded thereon for:

recording a first image of a first portion of an object using an electronic device;

at least one of prior to the recording and during the recording, determining a first distance from the first portion of the object to the electronic device;

associating using the electronic device the determined first distance with the recorded first image;

after the recording the first image;

recording with the electronic device a second image of a second portion of the object; and

detecting with the electronic device a first movement of the electronic device;

associating using the electronic device the recorded second image with the detected first movement;

processing with the electronic device the recorded first image, the determined first distance, the recorded second image, and the detected first movement; and

generating with the electronic device at least a portion of the three-dimensional model in response to the processing.