Device, system, and process for searching image data based on a three-dimensional arrangement

• US 9,886,789 B2
• Filed: 05/31/2011
• Issued: 02/06/2018
• Est. Priority Date: 05/31/2011
• Status: Active Grant

• Current Assignee: Rakuten Group, Inc.
• Original Assignee: Rakuten, Inc. (Rakuten Group, Inc.)
• Inventors: Takami, Shinya
• Primary Examiner(s): Hajnik, Daniel
• Application Number:
  US14/119,233
• Publication Number:
  US 20140118343A1
• Time in Patent Office: 2,443 Days
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/20 Perspective computation

Abstract

An information providing device can efficiently search for image data which includes an image of an object matching a user'"'"'s preference, or can efficiently recommend information related to an image-capturing spot matching the user'"'"'s preference. The information providing device receives three-dimensional arrangement information which indicates an arrangement of an object in three-dimensional space, and line-of-sight information for specifying a point-of-view position and a line-of-sight direction in the three-dimensional space, from a terminal device, generates two-dimensional arrangement information which indicates, in a two-dimensional plane, an arrangement of the object projected on the two-dimensional plane based on the received three-dimensional arrangement information and line-of-sight information, and compares arrangement information stored in a memory means that stores image data and the arrangement information which indicates, in an image, an arrangement of the object included in the image indicated by the image data, and the generated two-dimensional arrangement information, and searches for the image data based on a result of the comparison.

Claims

1. An information providing device that a terminal device can access through a network, the information providing device comprising:

at least one memory configured to store computer program code, respective image data of at least one image, and arrangement information corresponding to respective image data of the at least one image;

a receiver configured to receive three-dimensional arrangement information which indicates an arrangement of an object in three-dimensional space, and line-of-sight information for specifying a point-of-view position and a line-of-sight direction in the three-dimensional space, from the terminal device; and

at least one processor configured to access said memory, read said computer program code, and execute according to said computer program code,

said computer program code comprising;

generating code configured to cause at least one of said at least one processor to generate two-dimensional arrangement information which indicates, in a two-dimensional plane, an arrangement of the object projected on the two dimensional plane based on the received three-dimensional arrangement information and line-of sight information, the line-of-sight information being based on an overlap of objects in the two-dimensional plane; and

searching code configured to cause at least one of said at least one processor to

compare the arrangement information stored in the at least one memory and the generated two-dimensional arrangement information, and

search the respective image data based on a result of the comparison,

wherein arrangement information indicates, in an image, relative locations of areas occupied by an image of the object included in the image indicated by the image data,

wherein the searching code is further configured to cause at least one of said at least one processor to search the respective image data based on a similarity between the relative locations of areas occupied by the image of the object in the arrangement information stored in the at least one memory and the generated two-dimensional arrangement information,

wherein the receiver is further configured to receive the line-of-sight information for specifying a plurality of pairs of point-of-view positions and line-of-sight directions,

wherein the generating code is further configured to cause at least one of said at least one processor to generate a plurality of items of the two-dimensional arrangement information based on the plurality of pairs specified based on the received line-of sight information, and

wherein the searching code is further configured to cause at least one of said at least one processor to compare the arrangement information stored in the at least one memory and the plurality of items of the generated two-dimensional arrangement information.

2. The information providing device according to claim 1, wherein

said computer program code further comprises selecting code configured to cause at least one of said at least one processor to select two-dimensional arrangement information which indicates an arrangement which minimizes a degree of the overlap of objects in the two-dimensional plane, from the plurality of the generated two-dimensional arrangement information, and

the searching code is further configured to cause at least one of said at least one processor to compare the arrangement information stored in the at least one memory and the selected two-dimensional arrangement information.

3. The information providing device according to claim 2, wherein

the at least one memory is further configured to store position information of the corresponding to respective image data of the at least one image;

the receiver is further configured to receive condition information which indicates a condition related to position information; and

the searching code is further configured to cause at least one of said at least one processor to search for image data of the respective image data associated with the stored position information which satisfies the condition indicated by the received condition information.

4. The information providing device according to claim 3, further comprising a transmitter,

wherein the at least one memory is further configured to store position information corresponding to respective image data of the at least one image, and

wherein said computer program code further comprises transmitting code configured to cause at least one of said at least one processor to control the transmitter to transmit the position information of image data of the respective image data searched according to the searching code, to the terminal device.

5. The information providing device according to claim 1, wherein

the receiver is further configured to receive selection information which indicates an object selected by a user from objects arranged in the three-dimensional space;

and

the searching code is further configured to cause at least one of said at least one processor to compare an arrangement of the object indicated by the received selection information among arrangements indicated by the generated two-dimensional arrangement information, and an arrangement of an object corresponding to the object indicated by the received selection information among arrangements indicated by the arrangement information stored in the at least one memory.

6. The information providing device according to claim 1, wherein

the receiver is further configured to receive selection information which indicates an object selected by a user from objects arranged in the three-dimensional space;

and

the generating code is further configured to cause at least one of said at least one processor to generate the two-dimensional arrangement information by excluding objects which are other than object information indicated by the received selection information among objects arranged in the three-dimensional space, from projection objects on the two-dimensional plane.

7. The information providing device according to claim 1, wherein

said computer program code further comprises display control code configured to cause at least one of said at least one processor to display object candidates which represent objects arranged in the three-dimensional space, on a display screen of the terminal device such that a user can select an object candidate, and

the receiver is further configured to receive three-dimensional arrangement information which indicates an arrangement of the object candidate which is selected by the user and which is arranged in virtual three-dimensional space, as three-dimensional arrangement information which indicates the arrangement of the object in the three-dimensional space.

8. The information providing device according to claim 1, wherein

the receiver is further configured to receive color information which indicates a color of the object in the three-dimensional space;

and

the searching code is further configured to cause at least one of said at least one processor to compare the color of the object indicated by the received color information and a color of an object included in the image indicated by the image data.

9. The information providing device according to claim 1, wherein

the at least one memory is further configured to store position information corresponding to respective image data of the at least one image;

the receiver is further configured to receive condition information which indicates a condition related to position information; and

the searching code is further configured to cause at least one of said at least one processor to search for image data of the respective image data associated with the stored position information which satisfies the condition indicated by the received condition information.

10. The information providing device according to claim 1, further comprising a transmitter,

wherein said computer program code further comprises transmitting code configured to cause at least one of said at least one processor to control the transmitter to transmit image data of the respective image data searched according to the searching code, to the terminal device.

11. The information providing device according to claim 1, further comprising a transmitter,

wherein the at least one memory is further configured to store position information corresponding to respective image data of the at least one image, and

wherein said computer program code further comprises transmitting code configured to cause at least one of said at least one processor to control the transmitter to transmit the position information of image data of the respective image data searched according to the searching code, to the terminal device.

12. An information providing method in an information providing device that a terminal device can access through a network, the information providing method comprising:

receiving three-dimensional arrangement information which indicates an arrangement of an object in three-dimensional space, and line-of-sight information for specifying a point-of-view position and a line-of-sight direction in the three-dimensional space, from the terminal device;

generating two-dimensional arrangement information which indicates, in a two-dimensional plane, an arrangement of the object projected on the two dimensional plane based on the received three-dimensional arrangement information and line-of sight information, the line-of-sight information being based on an overlap of objects in the two-dimensional plane; and

comparing the generated two-dimensional arrangement information with arrangement information stored in a memory corresponding to respective image data of at least one image, the memory storing the respective image data and the arrangement information corresponding to the respective image data which indicates, in the at least one image, relative locations of areas occupied by an image of the object included in the image indicated by the image data, and

searching for the image data based on a result of the comparison,

wherein the searching comprises searching the respective image data based on a similarity between the relative locations of areas occupied by the image of the object in the arrangement information stored in the at least one memory and the generated two-dimensional arrangement information,

wherein the receiving three-dimensional arrangement information comprises receiving the line-of-sight information for specifying a plurality of pairs of point-of-view positions and line-of-sight directions,

wherein the generating two-dimensional arrangement information comprises generating a plurality of items of the two-dimensional arrangement information based on the plurality of pairs specified based on the received line-of sight information, and

wherein the searching further comprises comparing the arrangement information stored in the at least one memory and the plurality of items of the generated two-dimensional arrangement information.

13. An information providing system comprising:

a terminal device; and

an information providing device that the terminal device can access through a network,

wherein the terminal device comprises;

a transmitter;

at least one first memory configured to store first computer program code;

at least one first processor configured to access said first memory, and execute according to said first computer program code,

wherein said first computer program code comprises transmitting code configured to cause at least one of said at least one first processor to control the transmitter to transmit three-dimensional arrangement information which indicates an arrangement of an object in three-dimensional space and line-of-sight information for specifying a point-of-view position and a line-of-sight direction in the three-dimensional space, to the information providing device;

wherein the information providing device comprises;

at least one second memory configured to store second computer program code, image data of at least one image, and the arrangement information corresponding to respective image data of the at least one image;

a receiver configured to receiver the three-dimensional arrangement information and the line-of-sight information transmitted from the terminal device; and

at least one second processor configured to access said second memory, and execute according to said second computer program code,

wherein said second computer program code comprises;

generating code configured to cause at least one of said at least one second processor to generate two-dimensional arrangement information which indicates, in a two-dimensional plane, relative locations of areas occupied by an image of the object projected on the two dimensional plane based on the received three-dimensional arrangement information and line-of sight information, the line-of-sight information being based on an overlap of objects in the two-dimensional plane; and

searching code configured to cause at least one of said at least one second processor to

compare the arrangement information stored in the at least one first memory and the generated two-dimensional arrangement information, and

search the respective image data based on a result of the comparison,

wherein arrangement information indicates, in an image, an arrangement of the object included in the image indicated by the image data,

wherein the searching code is further configured to cause at least one of said at least one processor to search the respective image data based on a similarity between the relative locations of areas occupied by the image of the object in the arrangement information stored in the at least one memory and the generated two-dimensional arrangement information,

wherein at least one of the first computer program code and the second computer program code further comprises;

specifying code configured to cause at least one of said at least one first processor or at least one second processor to specify a reference position to which a line of sight is directed in virtual three-dimensional space;

the first computer program code further comprises;

acquiring code configured to cause at least one of said at least one first processor to acquire three-dimensional arrangement information which indicates, in virtual three-dimensional space, an arrangement of a virtual object arranged in the virtual three-dimensional space by a user'"'"'s operation; and

setting code configured to cause at least one of said at least one first processor to set the line-of-sight information by moving the point-of view position while a direction to the reference position specified based on the point-of-view position is regarded as the line-of-sight direction, according to the user'"'"'s operation, and

the transmitting code is further configured to cause at least one of said at least one first processor to control the transmitter to transmit the acquired three-dimensional arrangement information and the set line-of-sight information.

14. The information providing system according to claim 13,

wherein the specifying code is further configured to cause at least one of said at least one first processor or at least one second processor to specify, as the reference position, a position of a virtual object of a maximum size among virtual objects arranged in the virtual three-dimensional space.

15. The information providing system according to claim 13, wherein, when a plurality of virtual objects is arranged in the virtual three-dimensional space, the specifying code is further configured to cause at least one of said at least one first processor or at least one second processor to specify, as the reference position, a position which is a gravity center of positions of the plurality of arranged virtual objects.

16. The information providing system according to claim 13, wherein

at least one of first computer program code and the second computer program code further comprises:

second generating code configured to cause at least one of said at least one first processor or at least one second processor to generate a plurality of pieces of two-dimensional arrangement information which indicates, in the two-dimensional plane, the arrangement of the virtual object projected on the two-dimensional plane based on the acquired three-dimensional arrangement information, and for which at least one of point-of-view positions and line-of-sight directions is different;

the specifying code is further configured to cause at least one of said at least one first processor or at least one second processor to specify the reference position on a line of a sight indicated by a point-of-view position and a line-of-sight direction used to generate two-dimensional arrangement information which minimizes a degree of the overlap of virtual objects arranged in the two-dimensional plane among the plurality of pieces of the generated two-dimensional arrangement information; and

the setting code is further configured to cause at least one of said at least one first processor to set, as an initial state, the point-of-view position and the line-of sight direction used to generate the two-dimensional arrangement information which minimizes the degree of the overlap of the virtual objects arranged on the two-dimensional plane, and changes the point-of-view position and the line-of-sight direction from the initial state according to a user'"'"'s second operation.