Generating a 3D interactive immersive experience from a 2D static image

• US 9,990,760 B2
• Filed: 08/27/2014
• Issued: 06/05/2018
• Est. Priority Date: 09/03/2013
• Status: Active Grant

• Current Assignee: 3DITIZE SL
• Original Assignee: 3DITIZE SL
• Inventors: Aguilera Perez, Jaime, Alonso Blazquez, Fernando, Gmez Fernandez, Juan Bautista
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US14/469,814
• Publication Number:
  US 20150062125A1
• Time in Patent Office: 1,378 Days
• Field of Search: 345419, 345427, 345156
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/543 from line drawings
• CPC Class Codes: G06T 7/75 involving models

Abstract

A two-dimensional (2D) static image may be used to generate a three-dimensional (3D) interactive immersive experience. An image type of the 2D image may first be identified. The image type may be selected from a set of types such as interior, exterior, people, corridor, landscape, and other. Each image type may have an associated main feature type. The main feature of the 2D image may be identified using the corresponding main feature type. Then, unless the 2D image is of the “other” image type, a 3D object with two or more planes may be generated. The planes may intersect on the identified main feature. A virtual camera may be positioned proximate the 3D object, and a 3D view of the 3D object may be generated and displayed for the user. The user may optionally move the virtual camera, within limits, to view the 3D object from other locations.

Claims

1. A computer-implemented method for generating a three-dimensional view from a first two-dimensional image, the method comprising:

receiving a first two-dimensional image;

identifying, from a plurality of image types, a first image type for the entirety of the two-dimensional image, wherein the image type indicates a classification of the nature of a scene represented by the two-dimensional image;

identifying, within the first two-dimensional image, a first main feature having a first main feature type associated with the first image type, wherein the first main feature comprises an element of the scene represented by the two-dimensional image;

at a processor, generating a first plane within a virtual three-dimensional image space;

at the processor, generating a second plane within the virtual three-dimensional image space, wherein the second plane intersects the first plane at a line on the first main feature;

using the processor to automatically project the first two-dimensional image onto the first plane and the second plane to define a first three-dimensional object within the virtual three-dimensional image space;

using the processor to automatically position a virtual camera at a first camera position within the virtual three-dimensional image space proximate the first three-dimensional object;

using three-dimensional rendering capability of the processor to automatically generate a first three-dimensional view of at least a portion of the first three-dimensional object from the virtual camera; and

at a display screen, displaying the first three-dimensional view.

2. The computer-implemented method of claim 1, wherein identifying the first image type comprises, at an input device, receiving first user input indicating the first image type.

3. The computer-implemented method of claim 2, further comprising, prior to identifying the first main feature, selecting the first main feature type from a plurality of main feature types based on the first image type.

4. The computer-implemented method of claim 3, wherein identifying the first main feature comprises, at the input device, receiving second user input identifying the first main feature.

5. The computer-implemented method of claim 1, wherein the plurality of image types comprises an interior image type, an exterior image type, a people image type, a corridor image type, and a landscape image type.

6. The computer-implemented method of claim 1, wherein the first image type comprises an interior image type indicative of representation by the first two-dimensional image of an interior of a structure;

wherein the first main feature comprises a concave vertical corner of the structure;

and wherein identifying the first main feature comprises positioning a vertical line proximate the concave vertical corner.

7. The computer-implemented method of claim 1, wherein the first image type comprises an exterior image type indicative of representation by the first two-dimensional image of an exterior of a structure;

wherein the first main feature comprises a convex vertical corner of the structure;

and wherein identifying the first main feature comprises positioning a vertical line proximate the convex vertical corner.

8. The computer-implemented method of claim 1, wherein the first image type comprises a people image type indicative of representation by the first two-dimensional image of a person;

wherein the first main feature comprises a face of the person;

and wherein identifying the first main feature comprises positioning a rectangle to frame the face.

9. The computer-implemented method of claim 1, wherein the first image type comprises a corridor image type indicative of representation by the first two-dimensional image of a corridor within a structure;

wherein the first main feature comprises the corridor;

and wherein identifying the first main feature comprises positioning a rectangle to define boundaries of the corridor.

10. The computer-implemented method of claim 1, wherein the first image type comprises a landscape image type indicative of representation by the first two-dimensional image of a landscape comprising a background and a floor;

wherein the first main feature comprises a concave horizontal corner defining an intersection between the background and the floor;

and wherein identifying the first main feature comprises positioning a horizontal line proximate the concave horizontal corner.

11. The computer-implemented method of claim 1, wherein the first two-dimensional image comprises an image plane that bounds a field of view;

wherein the first image type comprises a selection from the group consisting of an interior image type and an exterior image type;

wherein the first main feature comprises a vertical corner of a structure;

wherein identifying the first main feature comprises positioning a vertical line proximate the vertical corner;

wherein generating the first plane comprises;

defining an ellipse comprising a major axis within the image plane, wherein, at a first major end of the ellipse, the major axis intersects a first edge of the field of view to define a first intersection and, at a second major end of the ellipse, the major axis intersects a second edge of the field of view to define a second intersection; and

defining the first plane such that the first plane passes through the first intersection and the vertical line;

and wherein generating the second plane comprises generating the second plane such that the second plane passes through the second intersection and the vertical line.

12. The computer-implemented method of claim 11, wherein defining the ellipse further comprises setting a length of a minor axis of the ellipse such that the minor axis has a minor length ranging from one quarter to one half of a major length of the major axis.

13. The computer-implemented method of claim 1, wherein the first two-dimensional image comprises an image plane that bounds a field of view;

wherein the first image type comprises a selection from the group consisting of a people image type and a corridor image type;

wherein the first main feature is selected from the group consisting of a face of a person and a corridor;

wherein identifying the first main feature comprises positioning a rectangle proximate the first main feature, the rectangle comprising a first vertical edge and a second vertical edge;

wherein generating the first plane comprises;

defining an ellipse comprising a major axis within the image plane, wherein, at a first major end of the ellipse, the major axis intersects a first edge of the field of view to define a first intersection and, at a second major end of the ellipse, the major axis intersects a second edge of the field of view to define a second intersection; and

defining the first plane such that the first plane passes through the first intersection and the first vertical edge;

wherein generating the second plane comprises generating the second plane such that the second plane is tangent to the ellipse and passes through the first intersection and the second intersection;

wherein the method further comprises generating a third plane such that the third plane passes through the second vertical edge and the second intersection;

and wherein applying the two-dimensional image to the first plane and the second plane further comprises applying the first two-dimensional image to the third plane to define the first three-dimensional object.

14. The computer-implemented method of claim 13, wherein defining the ellipse further comprises setting a length of a minor axis of the ellipse such that the minor axis has a minor length of about five sixteenths of a major length of the major axis.

15. The computer-implemented method of claim 1, wherein the first two-dimensional image comprises an image plane that bounds a field of view;

wherein the first image type is a landscape image type;

wherein the first main feature comprises a concave horizontal corner defining an intersection between a background and a floor;

wherein identifying the first main feature comprises positioning a horizontal line proximate the concave horizontal corner;

wherein generating the first plane comprises defining the first plane such that the first plane extends horizontally and passes through the horizontal line;

and wherein generating the second plane comprises generating the second plane such that the second plane extends vertically and passes through the horizontal line.

16. The computer-implemented method of claim 1, further comprising:

receiving a second two-dimensional image;

identifying, from the plurality of image types, a second image type for the entirety of the second two-dimensional image, wherein the second image type indicates a classification of the nature of a scene represented by the second two-dimensional image, wherein the second image type is an “

other”

image type;

at a processor, generating a third plane within the virtual three-dimensional image space;

using the processor to automatically project the second two-dimensional image onto the third plane to define a second three-dimensional object within the virtual three-dimensional image space;

using the processor to automatically position the virtual camera at a second camera position within the virtual three-dimensional image space proximate the second three-dimensional object;

using three-dimensional rendering capability of the processor to automatically generate a second three-dimensional view of at least a portion of the second three-dimensional object; and

at the display screen, displaying the second three-dimensional view.

17. The computer-implemented method of claim 1, further comprising:

at an input device, receiving a first user input;

in response to receipt of the first user input, using the processor to automatically position the virtual camera at a second camera position offset from the first camera position within the virtual three-dimensional image space;

using three-dimensional rendering capability of the processor to automatically generate a second three-dimensional view of at least a portion of the first three-dimensional object from the virtual camera; and

at the display screen, displaying the second three-dimensional view.

18. The computer-implemented method of claim 17, further comprising:

at the input device, receiving a second user input indicating a desire to move the virtual camera to a third virtual camera position;

in response to receipt of the second user input, determining that, at the third virtual camera position, a field of view of the virtual camera would not be entirely occluded by the first three-dimensional object; and

in response to determining that, at the third virtual camera position, the field of view would not be entirely occluded by the first three-dimensional object, determining that the virtual camera will not be moved to the third virtual camera position.

19. The computer-implemented method of claim 1, wherein identifying the first image type comprises, at the processor, automatically determining the first image type based on identification of an image feature of the first two-dimensional image.

20. A computer program product for generating a three-dimensional view from a first two-dimensional image, the computer program product comprising:

a non-transitory storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

receiving a first two-dimensional image;

identifying, from a plurality of image types, a first image type for the entirety of the two-dimensional image, wherein the image type indicates a classification of the nature of a scene represented by the two-dimensional image;

identifying, within the first two-dimensional image, a first main feature having a first main feature type associated with the first image type, wherein the first main feature comprises an element of the scene represented by the two-dimensional image;

generating a first plane within a virtual three-dimensional image space;

generating a second plane within the virtual three-dimensional image space, wherein the second plane intersects the first plane at a line on the first main feature;

automatically projecting the first two-dimensional image onto the first plane and the second plane to define a first three-dimensional object within the virtual three-dimensional image space;

automatically positioning a virtual camera at a first camera position within the virtual three-dimensional image space proximate the first three-dimensional object;

using three-dimensional rendering capability of the processor to automatically generate a first three-dimensional view of at least a portion of the first three-dimensional object from the virtual camera; and

causing a display screen to display the first three-dimensional view.

21. The computer program product of claim 20, wherein the computer program code is further configured to cause the at least one processor to identify the first image type by causing an input device to receive first user input indicating the first image type.

22. The computer program product of claim 21, wherein the computer program code is further configured to cause the at least one processor, prior to identifying the first main feature, to select the first main feature type from a plurality of main feature types based on the first image type;

and wherein the computer program code is further configured to cause the at least one processor to identify the first main feature by causing the input device to receive second user input identifying the first main feature.

23. The computer program product of claim 20, wherein the first image type comprises an interior image type indicative of representation by the first two-dimensional image of an interior of a structure;

wherein the first main feature comprises a concave vertical corner of the structure;

and wherein the computer program code is further configured to cause the at least one processor to identify the first main feature by positioning a vertical line proximate the concave vertical corner.

24. The computer program product of claim 20, wherein the first image type comprises an exterior image type indicative of representation by the first two-dimensional image of an exterior of a structure;

wherein the first main feature comprises a convex vertical corner of the structure;

and wherein the computer program code is further configured to cause the at least one processor to identify the first main feature by positioning a vertical line proximate the convex vertical corner.

25. The computer program product of claim 20, wherein the first image type comprises a people image type indicative of representation by the first two-dimensional image of a person;

wherein the first main feature comprises a face of the person;

and wherein the computer program code is further configured to cause the at least one processor to identify the first main feature by positioning a rectangle to frame the face.

26. The computer program product of claim 20, wherein the first image type comprises a corridor image type indicative of representation by the first two-dimensional image of a corridor within a structure;

wherein the first main feature comprises the corridor;

and wherein the computer program code is further configured to cause the at least one processor to identify the first main feature by positioning a rectangle to define boundaries of the corridor.

27. The computer program product of claim 20, wherein the first image type comprises a landscape image type indicative of representation by the first two-dimensional image of a landscape comprising a background and a floor;

wherein the first main feature comprises a concave horizontal corner defining an intersection between the background and the floor;

and wherein the computer program code is further configured to cause the at least one processor to identify the first main feature by positioning a horizontal line proximate the concave horizontal corner.

28. The computer program product of claim 20, wherein the first two-dimensional image comprises an image plane that bounds a field of view;

wherein the first image type comprises a selection from the group consisting of an interior image type and an exterior image type;

wherein the first main feature comprises a vertical corner of a structure;

wherein the computer program code is further configured to cause the at least one processor to identify the first main feature by positioning a vertical line proximate the vertical corner;

wherein the computer program code is further configured to cause the at least one processor to generate the first plane by;

defining an ellipse comprising a major axis within the image plane, wherein, at a first major end of the ellipse, the major axis intersects a first edge of the field of view to define a first intersection and, at a second major end of the ellipse, the major axis intersects a second edge of the field of view to define a second intersection; and

defining the first plane such that the first plane passes through the first intersection and the vertical line;

and wherein the computer program code is further configured to cause the at least one processor to generate the second plane by generating the second plane such that the second plane passes through the second intersection and the vertical line.

29. The computer program product of claim 20, wherein the first two-dimensional image comprises an image plane that bounds a field of view;

wherein the first image type comprises a selection from the group consisting of a people image type and a corridor image type;

wherein the first main feature is selected from the group consisting of a face of a person and a corridor;

wherein the computer program code is further configured to cause the at least one processor to identify the first main feature by positioning a rectangle proximate the first main feature, the rectangle comprising a first vertical edge and a second vertical edge;

wherein the computer program code is further configured to cause the at least one processor to generate the first plane by;

defining an ellipse comprising a major axis within the image plane, wherein, at a first major end of the ellipse, the major axis intersects a first edge of the field of view to define a first intersection and, at a second major end of the ellipse, the major axis intersects a second edge of the field of view to define a second intersection; and

defining the first plane such that the first plane passes through the first intersection and the first vertical edge;

wherein the computer program code is further configured to cause the at least one processor to generate the second plane by generating the second plane such that the second plane is tangent to the ellipse and passes through the first intersection and the second intersection;

wherein the computer program code is further configured to cause the at least one processor to generate a third plane such that the third plane passes through the second vertical edge and the second intersection;

and wherein the computer program code is further configured to cause the at least one processor to apply the two-dimensional image to the first plane and the second plane by applying the first two-dimensional image to the third plane to define the first three-dimensional object.

30. The computer program product of claim 20, wherein the first two-dimensional image comprises an image plane that bounds a field of view;

wherein the first image type is a landscape image type;

wherein the first main feature comprises a concave horizontal corner defining an intersection between a background and a floor;

wherein the computer program code is further configured to cause the at least one processor to identify the first main feature by positioning a horizontal line proximate the concave horizontal corner;

wherein the computer program code is further configured to cause the at least one processor to generate the first plane by defining the first plane such that the first plane extends horizontally and passes through the horizontal line;

and wherein the computer program code is further configured to cause the at least one processor to generate the second plane by generating the second plane such that the second plane extends vertically and passes through the horizontal line.

31. The computer program product of claim 20, wherein the computer program code is further configured to cause the at least one processor to perform the steps of:

causing an input device to receive a first user input;

in response to receipt of the first user input, automatically positioning the virtual camera at a second camera position offset from the first camera position within the virtual three-dimensional image space;

using three-dimensional rendering capability of the processor to automatically generate a second three-dimensional view of at least a portion of the first three-dimensional object from the virtual camera; and

causing the display screen to display the second three-dimensional view.

32. The computer program product of claim 31, wherein the computer program code is further configured to cause the at least one processor to perform the steps of:

causing the input device to receive a second user input indicating a desire to move the virtual camera to a third virtual camera position;

in response to receipt of the second user input, determining that, at the third virtual camera position, a field of view of the virtual camera would not be entirely occluded by the first three-dimensional object; and

in response to determining that, at the third virtual camera position, the field of view would not be entirely occluded by the first three-dimensional object, determining that the virtual camera will not be moved to the third virtual camera position.

33. A system for generating a three-dimensional view from a first two-dimensional image, the system comprising:

a processor, configured to;

receive a first two-dimensional image;

identify, from a plurality of image types, a first image type for the entirety of the two-dimensional image, wherein the image type indicates a classification of the nature of a scene represented by the two-dimensional image;

identify, within the first two-dimensional image, a first main feature having a first main feature type associated with the first image type, wherein the first main feature comprises an element of the scene represented by the two-dimensional image;

generate a first plane within a virtual three-dimensional image space;

generate a second plane within the virtual three-dimensional image space, wherein the second plane intersects the first plane at a line on the first main feature;

automatically project the first two-dimensional image onto the first plane and the second plane to define a first three-dimensional object within the virtual three-dimensional image space;

automatically position a virtual camera at a first camera position within the virtual three-dimensional image space proximate the first three-dimensional object; and

using three-dimensional rendering capability of the processor to automatically generate a first three-dimensional view of at least a portion of the first three-dimensional object from the virtual camera; and

a display screen, communicatively coupled to the processor, configured to display the first three-dimensional view.

34. The system of claim 33, further comprising,

an input device, communicatively coupled to the processor, configured to receive first user input indicating the first image type;

wherein the processor is further configured to identify the first image type based on the first user input.

35. The system of claim 34, wherein the processor is further configured, prior to identification of the first main feature, to select the first main feature type from a plurality of main feature types based on the first image type;

wherein the input device is further configured to receive second user input identifying the first main feature;

and wherein the processor is further configured to identify the first main feature based on the second user input.

36. The system of claim 33, wherein the first image type comprises an interior image type indicative of representation by the first two-dimensional image of an interior of a structure;

wherein the first main feature comprises a concave vertical corner of the structure;

and wherein the processor is further configured to identify the first main feature by positioning a vertical line proximate the concave vertical corner.

37. The system of claim 33, wherein the first image type comprises an exterior image type indicative of representation by the first two-dimensional image of an exterior of a structure;

wherein the first main feature comprises a convex vertical corner of the structure;

and wherein the processor is further configured to identify the first main feature by positioning a vertical line proximate the convex vertical corner.

38. The system of claim 33, wherein the first image type comprises a people image type indicative of representation by the first two-dimensional image of a person;

wherein the first main feature comprises a face of the person;

and wherein the processor is further configured to identify the first main feature by positioning a rectangle to frame the face.

39. The system of claim 33, wherein the first image type comprises a corridor image type indicative of representation by the first two-dimensional image of a corridor within a structure;

wherein the first main feature comprises the corridor;

and wherein the processor is further configured to identify the first main feature by positioning a rectangle to define boundaries of the corridor.

40. The system of claim 33, wherein the first image type comprises a landscape image type indicative of representation by the first two-dimensional image of a landscape comprising a background and a floor;

wherein the first main feature comprises a concave horizontal corner defining an intersection between the background and the floor;

and wherein the processor is further configured to identify the first main feature by positioning a horizontal line proximate the concave horizontal corner.

41. The system of claim 33, wherein the first two-dimensional image comprises an image plane that bounds a field of view;

wherein the first image type comprises a selection from the group consisting of an interior image type and an exterior image type;

wherein the first main feature comprises a vertical corner of a structure;

wherein the processor is further configured to identify the first main feature by positioning a vertical line proximate the vertical corner;

wherein the processor is further configured to generate the first plane by;

defining an ellipse comprising a major axis within the image plane, wherein, at a first major end of the ellipse, the major axis intersects a first edge of the field of view to define a first intersection and, at a second major end of the ellipse, the major axis intersects a second edge of the field of view to define a second intersection; and

defining the first plane such that the first plane passes through the first intersection and the vertical line;

and wherein the processor is further configured to generate the second plane by generating the second plane such that the second plane passes through the second intersection and the vertical line.

42. The system of claim 33, wherein the first two-dimensional image comprises an image plane that bounds a field of view;

wherein the first image type comprises a selection from the group consisting of a people image type and a corridor image type;

wherein the first main feature is selected from the group consisting of a face of a person and a corridor;

wherein the processor is further configured to identify the first main feature by positioning a rectangle proximate the first main feature, the rectangle comprising a first vertical edge and a second vertical edge;

wherein the processor is further configured to generate the first plane by;

defining an ellipse comprising a major axis within the image plane, wherein, at a first major end of the ellipse, the major axis intersects a first edge of the field of view to define a first intersection and, at a second major end of the ellipse, the major axis intersects a second edge of the field of view to define a second intersection; and

defining the first plane such that the first plane passes through the first intersection and the first vertical edge;

wherein the processor is further configured to generate the second plane by generating the second plane such that the second plane is tangent to the ellipse and passes through the first intersection and the second intersection;

wherein the processor is further configured to generate a third plane such that the third plane passes through the second vertical edge and the second intersection;

and wherein the processor is further configured to apply the two-dimensional image to the first plane and the second plane by applying the first two-dimensional image to the third plane to define the first three-dimensional object.

43. The system of claim 33, wherein the first two-dimensional image comprises an image plane that bounds a field of view;

wherein the first image type is a landscape image type;

wherein the first main feature comprises a concave horizontal corner defining an intersection between a background and a floor;

wherein the processor is further configured to identify the first main feature by positioning a horizontal line proximate the concave horizontal corner;

wherein the processor is further configured to generate the first plane by defining the first plane such that the first plane extends horizontally and passes through the horizontal line;

and wherein the processor is further configured to generate the second plane by generating the second plane such that the second plane extends vertically and passes through the horizontal line.

44. The system of claim 33, further comprising:

an input device configured to receive a first user input;

wherein the processor is further configured to;

in response to receipt of the first user input, position the virtual camera at a second camera position offset from the first camera position within the virtual three-dimensional image space; and

using three-dimensional rendering capability of the processor to automatically generate a second three-dimensional view of at least a portion of the first three-dimensional object from the virtual camera; and

and wherein the display screen is further configured to display the second three-dimensional view.

45. The system of claim 44, wherein the input device is further configured to receive a second user input indicating a desire to move the virtual camera to a third virtual camera position;

wherein the processor is further configured to;

in response to receipt of the second user input, determine that, at the third virtual camera position, a field of view of the virtual camera would not be entirely occluded by the first three-dimensional object; and

in response to determining that, at the third virtual camera position, the field of view would not be entirely occluded by the first three-dimensional object, determine that the virtual camera will not be moved to the third virtual camera position.