Method and device for presenting multimedia information

• US 10,325,408 B2
• Filed: 01/20/2017
• Issued: 06/18/2019
• Est. Priority Date: 01/22/2016
• Status: Active Grant

• Current Assignee: NextVPU (Shanghai) Co., Ltd.
• Original Assignee: NextVPU (Shanghai) Co., Ltd.
• Inventors: Feng, Xinpeng, Zhou, Ji
• Primary Examiner(s): Lett, Thomas J
• Application Number:
  US15/410,891
• Publication Number:
  US 20170213391A1
• Time in Patent Office: 879 Days
• Field of Search: 345423
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2219/004 Annotating, labelling
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G09G 2310/08 Details of timing specific for flat panels, other than clock recovery
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

A method and a device for presenting multimedia information are disclosed. The method includes: acquiring representation information, the representation information including electromagnetic-field spectral information for representing an object, the electromagnetic-field spectral information being observable for a naked eye and/or acquirable for a device; establishing a four-dimensional time-space model for characterizing the representation information according to the acquired representation information, the four-dimensional time-space model having an attribute for characterizing in a digital form variation of the representation information over time; and presenting the representation information characterized by the four-dimensional time-space model. In the solution, the four-dimensional time-space model has an attribute for characterizing in a digital form variation of the representation information over time. Thereby, the problem of delay in presenting representation information may be solved to some degree. Therefore, the solution may solve the defects of delay in the related art to some degree.

Claims

1. A method for presenting multimedia information, which is performed by a computing device, the method comprising:

by one or more processors, acquiring representation information, the representation information including electromagnetic-field spectral information for representing an object, the electromagnetic-field spectral information being observable for a naked eye and/or acquirable for a device;

by the one or more processors, establishing a four-dimensional time-space model for characterizing the representation information according to the acquired representation information, the four-dimensional time-space model having an attribute for characterizing in a digital form variation of the representation information over time; and

by the one or more processors, presenting the representation information characterized by the four-dimensional time-space model on a presenting unit,

wherein the step of establishing a four-dimensional time-space model for characterizing the representation information according to the acquired representation information, comprises;

processing the representation information to obtain first annotation information;

obtaining first point-cloud information containing geometric information and second point-cloud information containing texture information according to the first annotation information and the representation information;

merging the first point-cloud information and the second point-cloud information to obtain target point-cloud information;

obtaining visual information according to the target point-cloud information;

obtaining a space model according to the visual information, merging space models of a plurality of moments to obtain a space module; and

obtaining the four-dimensional time-space model according to the obtained space module, the first annotation information and second annotation information.

2. The method according to claim 1, wherein the representation information further comprises acoustic-field information which can be sensed by ear and/or acquirable for a device;

before the step of establishing a four-dimensional time-space model according to the acquired representation information, the method further comprises;

calculating acoustic-field information of an object corresponding to the representation information according to the representation information; and

the step of establishing a four-dimensional time-space model for characterizing the representation information according to the representation information comprises;

establishing a four-dimensional time-space model for characterizing the representation information and the acoustic-field information according to the representation information and the acoustic-field information.

3. The method according to claim 1, wherein the representation information further comprises acoustic-field information which can be sensed by ear and/or acquirable for a device;

before the step of obtaining a space model according to the visual information, the method further comprises;

calculating acoustic-field information of an object corresponding to the representation information according to the representation information; and

the step of obtaining a space model according to the visual information comprises;

merging the visual information and the acoustic-field information to obtain the space model.

4. The method according to claim 1, wherein after the step of merging the first point-cloud information and the second point-cloud information to obtain target point-cloud information, and before the step of obtaining visual information according to the target point-cloud information, the method further comprises:

processing the target point-cloud information to obtain second annotation information; and

the step of obtaining visual information according to the target point-cloud information comprises;

obtaining the visual information according to the second annotation information and the target point-cloud information.

5. The method according to claim 4, wherein the step of obtaining the visual information according to the annotation information and the target point-cloud information comprises:

optimizing a geometric vertex position of the target point-cloud information and calculating a normal of the target point-cloud information, to obtain a first result;

performing a surface fitting process and a triangular meshing process on the first result to obtain a second result; and

obtaining the visual information according to the second result.

6. The method according to claim 4, wherein the step of obtaining the visual information according to the second annotation information and the target point-cloud information comprises:

calculating surface normal information of the object according to the second annotation information and the target point-cloud information; and

obtaining the visual information according to the surface normal information.

7. The method according to claim 1, wherein the step of processing the representation information to obtain first annotation information comprises:

performing digital image process and analysis on the representation information to obtain the first annotation information.

8. The method according to claim 1, wherein the step of obtaining first point-cloud information containing geometric information according to the first annotation information and the representation information comprises:

processing the representation information according to the first annotation information to obtain coordinate information of an object corresponding to the representation information; and

generating first point-cloud information containing the geometric information according to the coordinate information.

9. The method according to claim 1, wherein the step of obtaining second point-cloud information containing texture information according to the first annotation information and the representation information comprises:

extracting information from the representation information according to the first annotation information in a point-by-point manner and/or by image synthesis, to obtain the second point-cloud information containing texture information.

10. The method according to claim 1, wherein before the step of presenting the representation information characterized by the four-dimensional time-space model, the method further comprises:

merging the four-dimensional time-space model and a first time-space model, to obtain a target four-dimensional time-space model, the first time-space model is for characterizing representation information of an object at a place where the multimedia information is presented; and

the step of presenting the representation information characterized by the four-dimensional time-space model comprises;

presenting the representation information characterized by the four-dimensional time-space model and the representation information characterized by the first time-space model according to the target four-dimensional time-space model.

11. The method according to claim 1, wherein before the step of presenting the representation information characterized by the four-dimensional time-space model, the method further comprises:

merging the four-dimensional time-space model and a first time-space model and a second time-space model which are located locally at the device, to obtain a target four-dimensional time-space model, the first time-space model is for characterizing representation information of an object at a place where the multimedia information is presented, and the second time-space model is for characterizing representation information of a virtual object; and

the step of presenting the representation information characterized by the four-dimensional time-space model comprises;

presenting the representation information characterized by the four-dimensional time-space model, the representation information characterized by the first time-space model and the representation information characterized by the second time-space model according to the target four-dimensional time-space model.

12. The method according to claim 1, wherein the representation information further comprises acoustic-field information which can be sensed by ear and/or acquirable for a device;

and the four-dimensional time-space model further characterizes acoustic-field information of an object corresponding to the representation information; and

the method further comprises;

playing the acoustic-field information characterized by the four-dimensional time-space model.

13. The method according to claim 1, wherein before the step of presenting the representation information characterized by the four-dimensional time-space model, the method further comprises:

determining a front orientation of a device for presenting the multimedia information; and

the step of presenting the representation information characterized by the four-dimensional time-space model comprises;

presenting the representation information characterized by the four-dimensional time-space model according to the front orientation.

14. The method according to claim 1, wherein the method further comprises:

determining a front orientation of a device for presenting the multimedia information and target multimedia information; and

feeding back the front orientation and the target multimedia information to a device for sending the representation information.

15. The method according to claim 1, wherein the step of presenting the representation information characterized by the four-dimensional time-space model comprises:

decoding the four-dimensional time-space model to obtain a decoded four-dimensional time-space model; and

presenting the representation information characterized by the four-dimensional time-space model according to the decoded four-dimensional time-space model.

16. A computing device for presenting multimedia information, comprising:

a processor; and

a memory for storing instructions executable by the processor;

wherein when executing the instructions, the processor is configured to perform;

acquiring representation information, the representation information including electromagnetic-field spectral information for representing an object, the electromagnetic-field spectral information being observable for a naked eye and/or acquirable for a computing device;

establishing a four-dimensional time-space model for characterizing the representation information according to the acquired representation information, the four-dimensional time-space model having an attribute for characterizing in a digital form variation of the representation information over time; and

presenting the representation information characterized by the four-dimensional time-space model on a presenting unit,

wherein for the step of establishing a four-dimensional time-space model for characterizing the representation information according to the acquired representation information, the processor is configured to perform;

processing the representation information to obtain first annotation information;

obtaining first point-cloud information containing geometric information, second point-cloud information containing texture information according to the first annotation information and the representation information;

merging the first point-cloud information and the second point-cloud information to obtain target point-cloud information;

obtaining visual information according to the target point-cloud information;

obtaining a space model according to the visual information, merging space models of a plurality of moments to obtain a space module; and

obtaining the four-dimensional time-space model according to the obtained space module, the first annotation information and second annotation information.

17. The computing device according to claim 16, wherein the representation information further comprises acoustic-field information which can be sensed by ear and/or acquirable for a computing device;

the processor is configured to perform;

before the step of establishing a four-dimensional time-space model according to the acquired representation information, calculating acoustic-field information of an object corresponding to the representation information according to the representation information; and

for the step of establishing a four-dimensional time-space model for characterizing the representation information according to the representation information, the processor is configured to perform;

establishing a four-dimensional time-space model for characterizing the representation information and the acoustic-field information according to the representation information and the acoustic-field information.

18. The computing device according to claim 16, wherein the representation information further comprises acoustic-field information which can be sensed by ear and/or acquirable for a computing device;

the processor is configured to perform;

before the step of obtaining a space model according to the visual information, calculating acoustic-field information of an object corresponding to the representation information according to the representation information; and

for the step of obtaining a space model according to the visual information, the processor is configured to perform;

merging the visual information and the acoustic-field information to obtain the space model.