Augmented reality viewing of printer image processing stages

• US 10,593,120 B1
• Filed: 08/28/2018
• Issued: 03/17/2020
• Est. Priority Date: 08/28/2018
• Status: Active Grant

• Current Assignee: KYOCERA Document Solutions, Inc. (Kyocera Corporation)
• Original Assignee: KYOCERA Document Solutions, Inc. (Kyocera Corporation)
• Inventors: Oliver, Don
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US16/115,212
• Publication Number:
  US 20200074733A1
• Time in Patent Office: 567 Days
• Field of Search: None
• CPC Class Codes: G06F 3/1205 resulting in increased flexibility in print job configuration, e.g. job settings, print requirements, job tickets
• CPC Class Codes: G06F 3/1207 resulting in the user being informed about print result after a job submission
• CPC Class Codes: G06F 3/1256 User feedback, e.g. print preview, test print, proofing, pre-flight checks
• CPC Class Codes: G06F 3/1259 Print job monitoring, e.g. job status
• CPC Class Codes: G06F 3/1292 Mobile client, e.g. wireless printing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method of generating an Augmented Reality (AR) display environment includes establishing a data connection at a mobile device to a printer image processing pipeline and generating an augmentation object of a virtual image based on the received printer image. Live video data of the physical environment is received from an imaging sensor on the mobile device, and a local 3D model of the physical environment utilizing the live video data is generated. Device tracking data from the mobile device is used to adapt the local 3D model. The augmentation object is combined with the adapted local 3D model to create an AR display environment, followed by configuring the mobile device to display the AR display environment. A mobile device includes a processor and memory with instructions to configure the device to generate an AR display environment.

Claims

1. A method of generating an Augmented Reality (AR) display environment, comprising:

establishing a data connection at a mobile device, wherein the data connection connects to a printer image processing pipeline comprising printer image data, and the mobile device comprises an imaging sensor and a display;

receiving the printer image data on the mobile device from the printer image processing pipeline, wherein the printer image data comprises pixels;

generating an augmentation object of a virtual image based on the received printer image data, comprising;

calculating two-dimensional (2D) coordinates for the four corners of the virtual image, wherein the virtual image is rectangular in shape and includes four corners, comprising;

determining the distance between each corner of the virtual image to an adjacent corner of the virtual image, wherein the distance is the visible dimension in pixels of the display of the mobile device used to display the virtual image;

calculating apparent 2D coordinates of each corner of the virtual image, wherein the apparent 2D coordinates are percentages relative to the longest visible dimension of the virtual image in the AR display environment on the display; and

calculating a pixel resolution from the apparent 2D coordinates, wherein the pixel resolution includes the apparent 2D coordinates of each corner of the virtual image;

receiving live video data of the physical environment from the imaging sensor on the mobile device, wherein the physical environment includes physical objects with detectable features, and the content of the live video data collected by the imaging sensor is at least partially directed by a user input;

generating a local three-dimensional (3D) model of the physical environment utilizing the live video data;

receiving mobile device tracking data from the mobile device;

adapting the local 3D model based on the mobile device tracking data, thereby creating an adapted local 3D model;

combining the augmentation object with the adapted local 3D model to create the AR display environment; and

configuring the mobile device to display the AR display environment.

2. The method of claim 1, wherein the augmentation object comprises at least one of location data defining a desired location in 3D space of the virtual image, structural data defining the 3D surface structure of the virtual image, image data defining the 2D surface texture of the virtual image to be applied to the surfaces defined by the local 3D model, and combinations thereof.

3. The method of claim 1, wherein the tracking data comprises data from at least one of a camera, an accelerometer, a compass, a proximity sensor, a motion sensor, RF receivers, a light sensor, a positioning system, and combinations thereof.

4. The method of claim 1, wherein the mobile device is at least one of a mobile phone, a tablet, an AR headset, and combinations thereof.

5. The method of claim 1, wherein the user input includes at least one of directing the imaging sensor to a portion of the physical environment, moving the imaging sensor closer to or further from the physical objects in the physical environment, selecting a portion of the virtual image data in the AR display environment to display, and combinations thereof.

6. The method of claim 1, wherein the generating of the local 3D model of the physical environment utilizing the live video data comprises assigning tracking anchor points to the live video data based on the detection of features of the physical objects in the physical environment and establishing an initial reference frame based on the user input.

7. The method of claim 6, further comprising:

receiving additional live video data of the physical environment from the imaging sensor on the mobile device; and

updating the local 3D model of the physical environment utilizing the additional live video data, including updating the assigned tracking anchor points based on the additional live video data.

8. The method of claim 1, wherein the printer image processing pipeline comprises a cloud server, wherein the cloud server performs at least one of the following:

receiving image data from the printer image processing pipeline, transmitting image data from the printer image processing pipeline, generating the local 3D model of the physical environment, receiving the mobile device tracking data from the mobile device, adapting the local 3D model based on the mobile device tracking data, thereby creating the adapted local 3D model, combining the augmentation object with the adapted local 3D model to create the AR display environment, configuring the mobile device to display the AR display environment, and combinations thereof.

9. The method of claim 1, further comprising:

receiving additional live video data of the physical environment from the imaging sensor on the mobile device;

updating the local 3D model of the physical environment utilizing the additional live video data;

receiving updated mobile device tracking data from the mobile device;

adapting the updated local 3D model based on the updated mobile device tracking data, thereby creating an updated adapted local 3D model;

combining the augmentation object with the updated adapted local 3D model to create an updated AR display environment; and

configuring the mobile device to display the updated AR display environment.

10. The method of claim 1, further comprising:

selecting a rectangular part of the virtual image, wherein the rectangular part is determined by the user input; and

calculating 2D coordinates for the four corners of the rectangular part of the virtual image, comprising;

determining the distance between each corner of the rectangular part of the virtual image to an adjacent corner of the full virtual image, wherein the distance is the visible dimension in pixels of the display of the mobile device used to display the virtual image;

calculating apparent 2D coordinates of each corner of the rectangular part of the virtual image, wherein the apparent 2D coordinates of the rectangular part are percentages relative to the longest visible dimension of the full virtual image in the AR display environment on the display; and

calculating a pixel resolution for the rectangular part of the virtual image from the apparent 2D coordinates for the rectangular part, wherein the pixel resolution for the rectangular part includes the apparent 2D coordinates of each corner of the rectangular part of the virtual image.

11. The method of claim 1, further comprising:

transmitting the pixel resolution to the printer image processing pipeline;

receiving pixel image data for the pixel resolution from the printer image processing pipeline;

generating a modified augmentation object of the virtual image based on the received pixel image data from the printer image processing pipeline;

combining the modified augmentation object with the adapted local 3D model to create an updated AR display environment; and

configuring the mobile device to display the updated AR display environment.

12. A system, comprising:

a processor;

an imaging sensor;

a display; and

a memory storing instructions that, when executed by the processor, configure the system to;

establish a data connection at a mobile device, wherein the data connection connects to a printer image processing pipeline comprising printer image data;

receive printer image data on the mobile device from the printer image processing pipeline, wherein the printer image data comprises pixels;

generate an augmentation object of a virtual image based on the received printer image data, comprising;

calculating two-dimensional (2D) coordinates for the four corners of the virtual image, wherein the virtual image is rectangular in shape and includes four corners, comprising;

determining the distance between each corner of the virtual image to an adjacent corner of the virtual image, wherein the distance is the visible dimension in pixels of the display of the mobile device used to display the virtual image;

calculating apparent 2D coordinates of each corner of the virtual image, wherein the apparent 2D coordinates are percentages relative to the longest visible dimension of the virtual image in the AR display environment on the display; and

calculating a pixel resolution from the apparent 2D coordinates, wherein the pixel resolution includes the apparent 2D coordinates of each corner of the virtual image;

receive live video data of the physical environment from the imaging sensor on the mobile device, wherein the physical environment includes physical objects with detectable features, and the content of the live video data collected by the imaging sensor is at least partially directed by a user input;

generate a local three-dimensional (3D) model of the physical environment utilizing the live video data;

receive mobile device tracking data from the mobile device;

adapt the local 3D model based on the mobile device tracking data, thereby creating an adapted local 3D model;

combine the augmentation object with the adapted local 3D model to create an AR environment; and

configure the mobile device to display the AR environment.

13. The system of claim 12, wherein local modeling logic is used to generate the local 3D model.

14. The system of claim 12, wherein local augmentation logic is used to generate the augmentation object.

15. The system of claim 12, wherein device tracking logic is used to adapt the local 3D model based on the mobile device tracking data.

16. The system of claim 12, wherein a rendering engine is used to combine the augmentation object with the adapted local 3D model to create the AR environment.

17. The system of claim 12, wherein the printer image processing pipeline comprises a cloud server, wherein the cloud server performs at least one of the following:

receiving image data from the printer image processing pipeline, transmitting image data from the printer image processing pipeline, generating the local 3D model of the physical environment, receiving the mobile device tracking data from the mobile device, adapting the local 3D model based on the mobile device tracking data, thereby creating the adapted local 3D model, combining the augmentation object with the adapted local 3D model to create the AR environment, configuring the mobile device to display the AR environment, and combinations thereof.

18. The system of claim 12, further comprising:

configuring the mobile device to;

receive additional live video data of the physical environment from an imaging sensor on the mobile device;

update the local 3D model of the physical environment utilizing the additional live video data;

receive updated mobile device tracking data from the mobile device;

adapt the updated local 3D model based on the updated mobile device tracking data, thereby creating an updated adapted local 3D model;

combine the augmentation object with the updated adapted local 3D model to create an updated AR environment; and

configure the mobile device to display the updated AR environment.