Reality augmenting method, client device and server

• US 9,922,461 B2
• Filed: 12/15/2014
• Issued: 03/20/2018
• Est. Priority Date: 04/14/2014
• Status: Active Grant

• Current Assignee: Baidu Online Network Technology (Beijing) Co., Ltd (Baidu Incorporated)
• Original Assignee: Baidu Online Network Technology (Beijing) Co., Ltd (Baidu Incorporated)
• Inventors: Yang, Chunyong, He, Zhouzhou, Fu, Zhengchao, Sui, Xin
• Primary Examiner(s): Harrison, Chante
• Application Number:
  US14/570,173
• Publication Number:
  US 20150294503A1
• Time in Patent Office: 1,191 Days
• Field of Search: 345629, 345632, 345633, 345635
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A reality augmenting method, a client device and a server are provided. The reality augmenting method includes: obtaining information related to an object to be identified, in which the information includes image information of the object; sending the information to a server, receiving augmented information of the object and display position information of the augmented information returned from the server according to the information; and displaying the augmented information and the image information simultaneously according to the display position information.

Claims

1. A reality augmenting method, executed by an instruction execution system of a terminal, and comprising:

obtaining, by the instruction execution system via an input device of the terminal, information related to a first object to be identified, wherein the information comprises image information of the first object, sound information of the first object, and position information of the first object;

sending, by the instruction execution system, the information to a server,

receiving, by the instruction execution system, augmented information of the first object and display position information of the augmented information returned from the server according to the information, wherein

the server is configured to identify the information and to obtain an identified result, and to generate augmented information of the first object and display position information of the augmented information according to the identified result, and

the server is further configured to identify the information by acts of obtaining type information corresponding to the first object,

determining a sequence of identifying models to be used according to the type information,

identifying the information using the identifying models according to the sequence and obtaining the identified result, and/or

identifying the information using the identifying models in parallel and obtaining the identified result, in which the identifying model comprises at least one of an image information identifying model, a sound information identifying model and a position information identifying model, and

the server is further configured to identify the information using the identifying models and to obtain an identified result by acts of;

identifying image information of the first object and obtaining a plurality of second objects to be identified in the image information, a title of each second object, an identifying probability of each second object, and location information of each second object, and obtaining the identified result according to the title and the position information of a current second object when determining that an identifying probability of the current second object is larger than a preset threshold, and re-identifying a current second object when determining that an identifying probability of the current second object is smaller than the preset threshold, until the identifying probability of the current second object is larger than the preset threshold, wherein the identifying probability is a weighted sum of a first probability corresponding to the image information identifying model, a second probability corresponding to the sound information identifying model and a third probability corresponding to the position information identifying model; and

displaying, by the instruction execution system, the augmented information and the image information simultaneously according to the display position information.

2. The method according to claim 1, wherein sending, by the instruction execution system, the information to a server comprises:

preprocessing, by the instruction execution system, the information to obtain processed information and sending, by the instruction execution system, the processed information to the server.

3. The method according to claim 1, wherein displaying, by the instruction execution system, the augmented information and the image information simultaneously according to the display position information comprises:

displaying, by the instruction execution system, the augmented information and the image information simultaneously in a predetermined mode comprising a front-projected holographic display mode.

4. A reality augmenting method executed by an instruction execution system of the server, and comprising:

receiving, by the instruction execution system, information related to a first object to be identified sent by a client device, wherein the information comprises image information of the first object, sound information of the first object and location information of the first object;

identifying, by the instruction execution system, the information;

obtaining, by the instruction execution system, an identified result;

generating, by the instruction execution system, augmented information of the first object and display position information of the augmented information according to the identified result; and

returning, by the instruction execution system, the augmented information and the display position information to the client device such that the client device displays the augmented information and the image information simultaneously according to the display position information,

wherein

identifying, by the instruction execution system, the information comprises;

obtaining, by the instruction execution system, type information corresponding to the first object,

determining, by the instruction execution system, a sequence of identifying models to be used according to the type information, identifying, by the instruction execution system, the information using the identifying models according to the sequence, and obtaining, by the instruction execution system, the identified result; and

/or

identifying, by the instruction execution system, the information using the identifying models in parallel, and obtaining, by the instruction execution system, the identified result, in which the identifying model comprise at least one of an image information identifying model, a sound information identifying model and a position information identifying model,

wherein identifying, by the instruction execution system, the information using the identifying models, and obtaining, by the instruction execution system, an identified result comprises;

identifying, by the instruction execution system, image information of the first object, and obtaining, by the instruction execution system, a plurality of second objects to be identified in the image information, a title of each second object, an identifying probability of each second object, and location information of each second object; and

obtaining, by the instruction execution system, the identified result according to the title and the position information of a current second object when determining that an identifying probability of the current second object is larger than a preset threshold;

re-identifying, by the instruction execution system, a current second object when determining that an identifying probability of the current second object is smaller than the preset threshold, until the identifying probability of the current second object is larger than the preset threshold, wherein the identifying probability is a weighted sum of a first probability corresponding to the image information identifying model, a second probability corresponding to the sound information identifying model and a third probability corresponding to the position information identifying model.

5. The method according to claim 4, wherein generating, by the instruction execution system, augmented information of the first object according to the identified result comprises:

obtaining, by the instruction execution system, associated information of the first object from a local information source according to the identified result, integrating, by the instruction execution system, the associated information and the identified result, and generating, by the instruction execution system, the augmented information.

6. The method according to claim 4, wherein

when identifying, by the instruction execution system, the information using the identifying models according to the sequence, and obtaining, by the instruction execution system, the identified result comprises:

identifying, by the instruction execution system, the information using the identifying models according to the sequence and in combination with information obtained from the local information source, and obtaining, by the instruction execution system, the identified result; and

when identifying, by the instruction execution system, the information using the identifying models in parallel and obtaining, by the instruction execution system, the identified result comprises;

identifying, by the instruction execution system, the information using the identifying models in parallel and in combination with information obtained from the local information source, and obtaining, by the instruction execution system, a search result.

7. A client device, comprising:

a processor;

a memory configured to store an instruction executable by the processor;

wherein the processor is configured to;

obtain, via an obtaining module of the client device, information related to a first object to be identified, wherein the information comprises image information of the first object, sound information of the first object and location information of the first object;

send the information to a server, receive augmented information of the first object and display position information of the augmented information returned from the server according to the information, wherein

the server is configured to identify the information and to obtain an identified result, and to generate augmented information of the first object and display position information of the augmented information according to the identified result, and

the server is further configured to identify the information by acts of obtaining type information corresponding to the first object, determining a sequence of identifying models to be used according to the type information, and identifying the information using the identifying models according to the sequence and obtaining the identified result, and/or identifying the information using the identifying models in parallel and obtaining the identified result, in which the identifying model comprise at least one of an image information identifying model, a sound information identifying model and a position information identifying model, and

the server is further configured to identify the information using the identifying models and to obtain an identified result by acts of;

identifying image information of the first object and obtaining a plurality of second objects to be identified in the image information, a title of each second object, an identifying probability of each second object, and location information of each second object, and obtaining the identified result according to the title and the position information of a current second object when determining that an identifying probability of the current second object is larger than a preset threshold, and re-identifying a current second object when determining that an identifying probability of the current second object is smaller than the preset threshold, until the identifying probability of the current second object is larger than the preset threshold, wherein the identifying probability is a weighted sum of a first probability corresponding to the image information identifying model, a second probability corresponding to the sound information identifying model and a third probability corresponding to the position information identifying model; and

display the augmented information and the image information simultaneously according to the display position information.

8. The client device according to claim 7, wherein the processor is configured to preprocess the information to obtain processed information, and send the processed information to the server.

9. The client device according to claim 7, wherein the processor is configured to display the augmented information and the image information simultaneously in a predetermined mode including a front-projected holographic display mode.

10. A server, comprising:

a processor;

a memory configured to store an instruction executable by the processor;

wherein the processor is configured to;

receive information related to a first object to be identified sent by a client device, wherein the information comprises image information of the first object, sound information of the first object and location information of the first object;

identify the information to obtain an identified result, generate augmented information of the first object and display position information of the augmented information according to the identified result; and

return the augmented information and the display position information to the client device such that the client device displays the augmented information and the image information according to the display position information, wherein

the processor is configured to identify the information by acts of;

obtaining type information corresponding to the first object, determining a sequence of identifying models to be used according to the type information, identifying the information using the identifying models according to the sequence and obtaining the identified result; and

/or

identifying the information using the identifying models in parallel and obtaining the identified result, in which the identifying model comprise at least one of an image information identifying model, a sound information identifying model and a position information identifying model,

wherein the processor is configured to identify the information using the identifying models and obtain an identified result by acts of;

identifying image information of the first object, and obtaining a plurality of second objects to be identified in the image information, a title of each second object, an identifying probability of each second object, and location information of each second object; and

obtaining the identified result according to the title and the position information of a current second object when determining that an identifying probability of the current second object is larger than a preset threshold;

re-identifying a current second object when determining that an identifying probability of the current second object is smaller than the preset threshold, until the identifying probability of the current second object is larger than the preset threshold, wherein the identifying probability is a weighted sum of a first probability corresponding to the image information identifying model, a second probability corresponding to the sound information identifying model and a third probability corresponding to the position information identifying model.

11. The server according to claim 10, wherein the processor is configured to:

obtain associated information of the first object from a local information source according to the identified result, integrate the associated information and the identified result, and generate the augmented information.

12. The server according to claim 11, wherein the processor is configured to identify the information using the identifying models according to the sequence and in combination with information obtained from the local information source, and obtain the identified result;

and/or

identify the information using the identifying models in parallel and in combination with information obtained from the local information source, and obtain a search result.