WO2015051622A1

WO2015051622A1 - Ultrasound fusion imaging method and ultrasound fusion imaging navigation system

Info

Publication number: WO2015051622A1
Application number: PCT/CN2014/074451
Authority: WO
Inventors: 康锦刚; 王广志; 朱磊; 张倩; 丁辉; 杨明雷; 丛龙飞
Original assignee: 深圳迈瑞生物医疗电子股份有限公司; 清华大学
Priority date: 2013-10-09
Filing date: 2014-03-31
Publication date: 2015-04-16
Also published as: CN104574329A; EP3056151A4; US20170020489A1; EP3056151B1; CN104574329B; US10751030B2; EP3056151A1

Abstract

The present application relates to an ultrasound fusion imaging method and an ultrasound fusion imaging navigation system. The ultrasound fusion imaging method comprises: a selection step, comprising selecting at least one ultrasound image from at least one previously stored piece of ultrasound video data according to an input instruction, wherein the ultrasound video data comprises an ultrasound image obtained by acquiring a target object from at least one plane and location-pointing information corresponding to the ultrasound image; a registration step, comprising registering the selected at least one ultrasound image with a modality image, wherein the registration process uses the location-pointing information of the at least one ultrasound image; and a fusion step, comprising image fusion of the registered ultrasound image with the modality image. The present invention differs from the existing registration-fusion methods based on real-time ultrasound in using a recorded registration video by scanning a target object prior to registration, and then selecting one or more ultrasound images to perform registration.

Description

Ultrasound fusion imaging method, ultrasound fusion imaging navigation system

Technical field

The present application relates to ultrasound imaging techniques, and more particularly to a method for fusing ultrasound images to pre-acquired modal images, and an ultrasound fusion imaging navigation system. Background technique

Imaging of the target subject in the clinic can use more than one imaging system, allowing medical personnel to obtain multiple modal medical images, such as computed tomography (CT) images, magnetic resonance (Magnetic Resonance, MR) ) images, ultrasound images, etc. The principle of ultrasound fusion imaging navigation is to establish the spatial correspondence between real-time ultrasound images and other modal data acquired in advance (such as CT or MR images) through a spatial positioning device (usually a magnetic positioning sensor attached to the probe). And superimposing the ultrasound and the corresponding modal data facet to achieve the fusion of the two images, realize the common guidance of the two images to the diagnosis and treatment process, give full play to the advantages of CT or MR high resolution and the real-time characteristics of ultrasound. Clinicians provide more diagnostic information to improve treatment outcomes.

In ultrasound fusion imaging navigation, an important step is to register the ultrasound image with the modal data, which is essentially the position of the point (or plane) in the ultrasound image in the world coordinate system and the point in the modal data ( Or the planes in the world coordinate system one-to-one correspondence, accurately obtaining the position of the target point in the world coordinate system has a great influence on improving the registration accuracy.

The current registration technology is based on real-time ultrasound. The medical staff obtains the ultrasound image used to provide the registration information by freezing the current frame. This method of processing the real-time ultrasound image frame by frame has become a habit. . In addition, in order to obtain an ultrasound image of a specific section of a certain depth of breath, the doctor often needs the patient to control the breathing well, especially for some patients with abdominal breathing to perform abdominal body fusion. Displacement, rotation and deformation of the organ can cause large errors and therefore require the patient to remain in a particular breathing state for a longer period of time, for example. This puts high demands on doctors and patients who use ultrasound fusion imaging navigation. If the patient's breathing control is poor or the doctor's operating experience is insufficient, it often fails to achieve satisfactory results, such as resulting in low registration accuracy, fusion. Imaging success rate Drop. The current method for eliminating the effects of respiration is mainly based on the doctor manually determining the respiratory phase or adding a sensor for simple respiratory gating, but the effect is not good. Summary of the invention

Based on this, it is necessary to provide an ultrasound fusion imaging method and a ultrasound fusion imaging navigation system that have a good effect of eliminating respiratory effects.

A method for fusing an ultrasound image with a pre-acquired modal image, comprising: a selecting step of selecting at least one frame of ultrasound image from at least one piece of pre-stored ultrasound video data according to the input instruction, the ultrasound video The data includes an ultrasonic image obtained by acquiring the target object from at least one plane, and position pointing information corresponding to each frame of the ultrasonic image, the position pointing information being determined by a position sensor fixed to the ultrasonic probe during the ultrasonic image acquiring process. Generating a condition of the inductive ultrasonic probe; a registration step of registering the selected at least one frame of the ultrasound image with the modal image, wherein the at least one frame of the ultrasound image is used in the registration process Information; fusion step, image fusion of the registered ultrasound image and modal image.

In one embodiment, the multi-frame ultrasound image is selected in the selecting step; the method further includes: establishing a breathing model step, establishing a breathing model according to the ultrasound video data; and a breathing correction step, selecting the selected multi-frame ultrasound The multi-frame ultrasound image is corrected to the same breathing depth using a breathing model before and/or during fusion of the image and the modal image.

A method for fusing an ultrasound image with a pre-acquired modal image, comprising: a selecting step of selecting a multi-frame ultrasound image from at least one piece of ultrasound video data, the ultrasound video data comprising from at least one plane to a target object Performing the acquired ultrasonic image and position pointing information corresponding to each frame of the ultrasonic image, the position pointing information being obtained by the position sensor fixed on the ultrasonic probe during the ultrasonic image acquiring process;

Establishing a breathing model step of establishing a breathing model based on the ultrasound video data;

a registration step of registering the multi-frame ultrasound image to be registered with the modal image;

a fusion step of performing image fusion on the registered ultrasound image and modal image;

Wherein, before the registration of the multi-frame ultrasound image and the modal image to be registered and/or the fusion process, the multi-frame ultrasound image to be registered is corrected to the same breathing depth by using the breathing model . An ultrasonic fusion imaging navigation system includes: a probe and a position sensor fixed on the probe; an acquisition module, configured to collect the target object from at least one plane, and obtain at least one piece of ultrasound video including information for registration Data, for each frame of ultrasound image in each piece of ultrasound video data, recording position pointing information generated by the position sensor during the ultrasound image acquisition process according to the sensed movement condition of the ultrasound probe; And playing back the pre-stored ultrasound video data according to the input instruction; the selecting module, configured to select at least one frame of the ultrasound image from the played back ultrasound video data according to the input selection instruction; the registration module, configured to select Aligning at least one frame of the ultrasound image with the modal image, wherein the registration uses the positional orientation information of the at least one frame of the ultrasound image; the fusion module is configured to register the ultrasound image and the modality The image is image blended.

The above-mentioned ultrasonic fusion imaging method and the ultrasonic fusion imaging navigation system adopt a registration fusion method different from the existing real-time ultrasound, but record a registration video of the target object before registration, and then select one or more frames. The frame ultrasound image is registered to eliminate the effect of breathing.

1 is a schematic structural view of an ultrasonic fusion imaging navigation system according to an embodiment;

2 is a schematic flow chart of a method for fusing an ultrasonic image and a pre-acquired modal image according to an embodiment;

3 is a schematic diagram of spatial transformation of an embodiment;

4 is a schematic diagram showing a plurality of registration plane display modes in an embodiment, wherein each window displays an ultrasound image;

5 is a schematic flow chart of establishing a breathing model according to an embodiment;

6 is a schematic diagram showing changes in breathing depth with time according to an embodiment;

7 is a schematic diagram showing the displacement of a target organ in a three-direction of the world coordinate system with respect to a reference position as a function of respiratory depth by linear fitting according to an embodiment;

FIG. 8 is a schematic flow chart of performing registration fusion after establishing a breathing model according to an embodiment. detailed description

The following instructions are provided for full understanding Specific details. However, it will be understood by those skilled in the art that the present invention may be practiced without such detail. In some instances, well known structures and functions are not shown or described in detail in order to avoid obscuring the description of the embodiments.

The terms "including", "comprising", etc., are intended to be interpreted in an inclusive meaning rather than an exclusive or exhaustive meaning. not limited to".

The present invention will be further described in detail below with reference to the accompanying drawings.

A block diagram of an ultrasound fusion imaging navigation system is shown in Figure 1 (excluding the dashed box): The ultrasound probe 101 transmits ultrasound to the body examination site, and the received echo signal is processed by the ultrasound imaging module 102 to obtain an ultrasound image of the target organ. The modal image data acquired in advance, such as a CT or MR image, is introduced into the registration fusion module 105 before registration; the position sensor 103 fixed on the probe 101 continuously provides position information as the probe moves, Information on the spatial orientation of the probe's six degrees of freedom (including vertical, lateral, longitudinal, pitch, roll, and sway) is obtained by the position controller 104; the registration fusion module 105 uses the image information and position information to image the ultrasound image and the modal data Registration and fusion are performed; display module 106 is used to display the fusion results.

In conjunction with the ultrasound fusion imaging navigation system shown in FIG. 1, the fusion method of the ultrasound image and the pre-acquired modal image provided in this embodiment is as shown in FIG. 2, and includes the following steps S1: S13:

The selecting step S 1 1 selects at least one frame of the ultrasonic image from the pre-stored at least one piece of the ultrasound video data according to the input instruction.

The pre-stored ultrasound video data includes: acquiring a target object (a target organ such as a liver) in advance, and obtaining an ultrasound video containing registration information, that is, a registration video, for each frame of the ultrasound image, the recording is fixed at the same time The positional orientation information of the position sensor on the ultrasonic probe is R _prote ( ), R _probe {t). The position sensor is generated according to the movement condition of the induced ultrasonic probe during the ultrasonic image acquisition process, that is, the content of the registration video is in addition to the ultrasound. In addition to the image data, positional orientation information of the position sensor is also included. The position sensor may be an electromagnetic induction based position sensor, an optical principle based position sensor, or other types of positions based on an acoustic principle or the like. sensor. In the embodiments of the present application, a position sensor based on electromagnetic induction is taken as an example for description.

In the selecting step, the input instruction may be an instruction input from an external user, or may be An instruction that is automatically triggered when the registration is fused inside the system. In an implementation, when selecting, the pre-stored ultrasound video may be played back, and then selected during playback, and the input instruction and selection instruction received by the system may be determined by the user's needs, and the commonly used correlation is combined. The technology is implemented as long as it can satisfy the playback of the ultrasound video and the selection of the frame image. In the playback ultrasound video of this step, the pre-stored ultrasound video may be played frame by frame from beginning to end, or the ultrasound video may be dragged to the frame position corresponding to the slice of interest by, for example, a progress bar or a knob rotation. In some implementations, selection may also be made by some pre-set conditions, such as presetting a certain frame in the selected ultrasound video, such as the first 20 frames.

In the registration step S12, the selected at least one frame of the ultrasound image is registered with the modal image, and the system simultaneously uses the positional pointing information corresponding to the ultrasound images to participate in the registration.

In order to improve the accuracy of the entire field of registration, it is also possible to select two or more frames of ultrasound images for registration in the registration video. Of course, the prerequisite for selecting a multi-frame ultrasound image for registration is that the ultrasound images of these frames have the same or similar respiratory states, or that these ultrasound images are acquired in the same or similar respiratory state. In general, if the ultrasound video is acquired while the patient is holding the breath, the ultrasound images of all frames in the video have similar breathing states, and the multi-frame ultrasound images can be directly registered simultaneously.

In the image registration, it is necessary to find a spatial transformation to map an image to the image, so that the points corresponding to the same position in the two images are correspondingly matched, so as to achieve the purpose of correctly merging the information. Based on this, in the ultrasound fusion imaging navigation system, the registration of the ultrasound image and the modal image can be realized by the spatial transformation relationship of FIG. 3, that is, the point in the ultrasound image is first transformed from the coordinate system of the ultrasound image to the position sensor. The coordinate system is transformed from the coordinate system of the position sensor to the world coordinate system, and finally the world coordinate system is transformed into the coordinate system of the modal image. The world coordinate system in this embodiment and other embodiments is a coordinate system used as a reference, and may be arbitrarily designated, for example, a magnetic field generator coordinate system. Of course, other coordinate systems may be employed as the reference coordinate system. The spatial transformation relationship shown in Figure 3 can be expressed in the form of a formula:

Xsec― P ' Rprobe - ' ^-us ( 1 ) where is the coordinates of a point in the ultrasound image, _ec is the coordinates of the point in the modal image, ^ is the transformation matrix of the ultrasound image space to the position sensor space, ^ ₆ It is the transformation matrix of the position sensor space to the reference coordinate system such as the magnetic field generator space, and P is the transformation matrix of the reference coordinate system to the modal image space. For the transformation matrix A, since the position sensor is fixed on the probe and the depth of the ultrasonic probe is constant,

A is fixed, so the transformation matrix A can be obtained by the calibration method in combination with the positional pointing information R _pra o before registration. For details, refer to the related art of spatial transformation of the existing ultrasound image to the position sensor space, which will not be described in detail.

For the transformation matrix R _PRATE , it can be directly read by the positioning controller connected to the position sensor. As the probe moves, the R _PRATE changes continuously, and can also be implemented by referring to commonly used related technologies.

For the transformation matrix P, which can also be called a registration matrix, it can be calculated according to formula (1) by finding corresponding points or faces in the ultrasound image space and the modal image space. In one example, certain points or areas of the body examination site are marked, and the position of the marked points or regions is obtained after imaging in the ultrasound image space, and these points are also obtained by imaging in the modal image space. Or the position of the region _et ., so that P can be obtained by the formula (1); in another example, after transforming some points or regions of the ultrasound image space to the world coordinate system, the modal image is Some points or regions are also transformed into the world coordinate system. The image matching method is used to obtain the position of the point or region in which the ultrasound image is spatially imaged. Corresponding to the position _ec of the point or region in the modal image space, it is also possible to calculate P. . One of ordinary skill in the art will readily appreciate that an inverse transform can be applied to obtain the inverse coordinate transformation from one image space to another.

The fusion step S13 performs image fusion on the registered ultrasound image and the modal image.

The techniques used for image fusion can be specifically referred to the existing image fusion processing methods, such as image fusion methods based on spatial domain, such as image pixel gray value maximal (small) fusion method, image pixel gray value weighted fusion method, etc., or A transform domain-based image fusion method, such as a multi-resolution pyramid fusion method, a Fourier transform-based image fusion method, etc., will not be described in detail herein.

Thus, the registration and fusion of the ultrasound image and the modal image are achieved by the above steps, and the registration method employed here is different from the existing registration method. The existing method is based on real-time ultrasound, which is performed by freezing one frame at a time. The registration method of this embodiment is to record a registration video of the target object before registration, and select one or more frames of ultrasound images for registration by video playback.

In addition to the above steps, the fusion method of the ultrasound image and the modal image of an embodiment further includes: a multi-frame display step of selecting a multi-frame ultrasound image for registration in the registration step, after displaying the registration or fusion While the ultrasound image is being framed, the intersection and angle between the images of these frames are also displayed, as shown in Figure 4, The two boxes show the intersection line position 305 between the different frame ultrasound images and the angle 307 between the two frames. If the multi-frame registration or fused image is to be displayed, one of the frames may be selected as the reference frame, and the displayed intersection line position and the angle between the two are the remaining frames and the reference frame. Intersection lines and angles. An advantage of this embodiment is that the relative positional relationship of the frames can be more intuitively reflected, facilitating subsequent fine-tuning of the registration.

In the application of ultrasound fusion imaging navigation system, the respiratory control ability of many patients is usually poor. The registration and fusion of the registration video is performed under the free breathing state of the patient. However, since the person is free to breathe, the organ is also Along with the more complicated movements, the motion forms include rigid motions such as translation and rotation, as well as non-rigid motions such as integral or local deformation caused by inter-organ compression. This causes the organ movement caused by breathing to have a large effect on the fusion result, so it is necessary to reduce or eliminate the effect of breathing on registration and fusion. Based on this, the present embodiment proposes a method for registering and fusing an ultrasound image and a pre-acquired modal image, which is first based on the positional information of the ultrasound video and the position sensor, and is used to describe the organ with the breathing. The regular breathing model of the movement then uses the breathing model to obtain a time-varying corrected spatial map and apply it to registration and fusion to achieve the goal of attenuating or eliminating respiratory effects.

Here, the effect of breathing movement is eliminated or attenuated by means of a correction, which can be expressed in the form of a formula:

Xsec = P · T Rp · A · X _U s) ( 2 ) where Γ is some spatial mapping method for correction, which can be linear mapping, affine mapping and other forms of nonlinear mapping, ie r can be defined Any continuous mapping in the middle of three dimensions.

In general, the breathing movement during free breathing is relatively regular and can be approximated as a periodic movement. When breathing, the patient's abdominal epidermis moves mainly along the front and rear directions of the human body, which can be approximated as reciprocating motion. For a target organ whose movement is mainly caused by breathing, similar to the movement of the abdomen epidermis, its motion can also be approximated as periodic. It is assumed that the movement of such target organs, such as the liver, with the breathing is a rigid motion, and a linear model can be used to describe its regularity with the movement of the breathing. In this embodiment, a linear model is taken as an example. For other embodiments having non-rigid components, the present embodiment can be simply deduced in combination with an algorithm formula commonly used in non-rigid motion. For a linear model, all points in space have the same mapping relationship, and equation (2) can be expressed as

Xsec- PT Rprobe ^ Xus ( 3 ) At this time, the spatial map T is degenerated into one matrix.

In order to establish a breathing model, it is necessary to add a respiration sensor 109, such as the dotted line portion shown in Fig. 1, to the patient's abdominal epidermis for tracking the abdominal epidermis of the patient, based on the ultrasonic fusion imaging navigation system of Embodiment 1. How the sensor is positioned to move with the breath. The respiration sensor can be a position sensor based on electromagnetic induction or other type of position sensor.

For the convenience of the following description, the related terms are first explained as follows:

(1) Reference position: may be any point on the path of the breathing sensor with the abdomen movement, for example, it may be an intermediate position of the sensor movement path;

(2) Respiratory depth: The displacement of the respiration sensor relative to the reference position at a time corresponding to the ultrasound image of a frame, called the respiration depth, is used to approximate the state of the respiration, which can be represented by d(t), which can be passed through the respiration position. The information R _rep (0 is obtained, which can be obtained by using the conversion method of the position information of the commonly used respiratory sensor, and will not be described in detail;

(3) Reference breathing depth: The breathing depth corresponding to the reference position is called the reference breathing depth and can be expressed as d ₀ .

(4) Reference frame relative motion: to be at the reference breathing depth ί. The position of the lower target organ is the reference position, and in the world coordinate system, the amount of movement of the target organ relative to the reference position at different breathing depths is called the reference system relative motion amount. This embodiment uses rigid motion as an example for illustration, and ignores the rotational motion to consider only the relative displacement of the reference frame, which is used here to describe.

As shown in FIG. 5, the method for establishing a breathing model of this embodiment includes the following steps S21 to S25:

Step S21, setting a reference breathing depth and a reference position for the pre-stored ultrasonic video.

The pre-stored ultrasound video is acquired from a plurality of planes that are at an angle to each other (eg, "for a positive integer"), and a plane acquires an ultrasound video to obtain an n-segment ultrasound video US V, · , ζ · = 1 , the frame number of the video is indicated, and USV^) can be used to represent the first frame of the ultrasound image in the first ultrasound video. At the same time, for each frame of the acquired ultrasound image, the positional pointing information R _pro of the position sensor fixed to the ultrasonic probe, and the respiratory position information R _resp (t) of the respiratory sensor fixed on the patient's abdomen epidermis are recorded. During each acquisition of ultrasound video, the acquisition time includes one or more breathing cycles.

As described in the respiratory motion characteristics described above, respiratory motion exhibits periodic characteristics, and the phases of each respiratory motion vary during the cycle, and this motion is a repetitive process during the cycle, similar to Figure 6. The waveform of the sinusoid is shown, where the horizontal axis represents the breathing cycle and the vertical axis represents the breathing depth 40, and each repeated curve represents one breathing cycle. In Fig. 6, the horizontal dotted line is the given reference position d ₀ , and the breathing depth varies with the video frame number /, where the reference position is set to the middle position of the sensor motion path. Of course, the reference position may also be other positions, such as the lowest position or the highest position of the sensor motion path. Since the same breathing depth may correspond to an inhalation state or an exhalation state, it is also possible to further distinguish the respiratory state into an expiratory phase and an inhalation phase, and similarly, a sinusoidal waveform can also be obtained.

Step S22: For each video segment, select a reference frame to obtain a motion displacement V d(t) of the target object corresponding to the reference frame corresponding to the ultrasound image of the other frame.

For each segment of the ultrasound video USV, if the ultrasound image corresponding to a certain depth of breath ί/„ is used as the reference frame, the target organ corresponding to the ultrasound image of the other frame can be obtained by motion tracking, such as template matching. The motion shift of the reference frame V d(t ).

Step S23, converting to the same reference coordinate system to eliminate the influence of probe jitter.

Since it is difficult to ensure that the probe is absolutely stationary when the breath correction video is acquired, it is necessary to limit the motion tracking to the plane of the reference frame to eliminate the influence of probe jitter. In this step, assume χ. Is the position of a point in the reference frame, .ψ) is the position of the point obtained by tracking in the ultrasound image of the t-th frame, using R;,,,,/,,« and respectively representing the reference frame and the frame/frame The position of the probe corresponding to the ultrasound image, x. And x (0 points corresponding to the world coordinate system are represented as /" ₀ and /;? (/), respectively:

m ₀ = Rprobe_0 · A - XQ ( 4 ) m{t) = Rprob j) - A - x(t) ( 5 ) Assume that the relative displacement of the reference frame of the target organ corresponding to other frames (ie non-reference frames) is in the reference frame The projected component on the corresponding plane is expressed as proji WXdit))) ^ The motion displacement of the target organ corresponding to the other frame relative to the reference frame is the observed value of the projected component. As assumed before, at this time. The projection component of the plane corresponding to the reference frame is proj mit mo) , which is a general expression of the observation value V ) of Λ·(^(ί ))), namely:

Vi{d{t)) = proji(m(i)-m ₀ ) ( 6 )

Here, by shifting the reference frame and the non-reference frame into the same world coordinate system for projection, the displacement calculation error caused by the probe jitter is eliminated. When collecting each piece of ultrasound video, you can use a probe clip or the like Set the position of the probe to keep the position of the probe as far as possible, and keep the position of the patient unchanged. At this time, the position of the probe can be considered to remain unchanged during the acquisition, R _prob = R _{probe 0} , then ν «ή) = χ (ή - x ₀ step S24, according to the motion displacement (ί) obtained in step S22), calculate the relative displacement of the reference system of the target organ corresponding to the ultrasound image of the other frame at different breathing depths in the same reference coordinate system. .

Set the same reference breathing depth / „ for all segment videos, at a certain breathing depth i/(/)::=D, target

The relative displacement of the reference system relative to the reference position can be obtained by the following formula (7) by the optimization method, that is, J (D) is the corresponding value when the outermost layer is the minimum value:

(Z)) = argmin(X( ∑ || P^(0) || ² )) where argmin() is a function that minimizes the function in its parentheses.

Equation (7) is solved at multiple breathing depths to obtain the displacement W(d) at the depth of the breath. Step S25, obtaining a breathing model by fitting different respiratory depths and corresponding reference frame relative displacements.

In the embodiment of the present invention, the "breathing model" refers to a law in which the displacement of the target organ varies with the depth of the breathing. The term "establishing a breathing model" refers to the calculation or determination of the target organ displacement as a function of respiratory depth based on existing ultrasound video data, that is, the mathematical expression of the law of the target organ displacement as a function of respiratory depth.

In this step, with ί as the independent variable, (i, the point pair is fitted in a certain way, the deviation of the target organ displacement with the breathing depth can be obtained, and the establishment of the breathing model is completed.

In the present embodiment, in equation (7), the target organ motion and the reference frame relative displacement JV (d) observed at a certain breathing depth D in plane i (ie, video I) are measured by the square of the difference modulus of the two vectors. (t)) The error between the projection / "•/ iWD) on the plane, ie II, · (,) is called) II ² , other methods can also describe the magnitude of the error in other ways, Such as Ι ^ ^ 'Ί )) ( )) 11 and so on.

The three straight lines shown in FIG. 7 are schematic diagrams of the displacement of the target organ in the three directions of the world coordinate system with respect to the reference position as a function of the breathing depth d by linear fitting. Other embodiments may be used with other fitting methods, such as by other types of curves such as a quadratic curve or a cubic curve. In this embodiment, after obtaining the breathing model, before the ultrasound image and the modal image are registered by the method of Embodiment 1, the breathing model is used for correction, that is, the ultrasound image of different frames is used by the breathing model before registration. Correct to the same breathing depth (ie, the same breathing state) to achieve the purpose of eliminating or reducing the respiratory effects. The specific calibration process is described below.

Suppose t is a certain frame in the frame to be registered selected by the doctor, and the corresponding breathing depth of the frame is 0. According to the breathing model, the relative displacement of the reference system relative to the reference position at the depth of respiration is

W{d{t)) ₀ hypothesis. is a point in the frame whose position is R _/W in the world coordinate system, ( ) · A · Λ-, then according to formula (3), it is corrected to The position of this point after the reference breathing depth 4 becomes:

T{W{d{t)))■ Rpro t)■ A■ X (8) where Γ( /(/))) is the ultrasound image frame/corresponding respiratory depth as an independent variable, corrected according to the breathing model Breath correction matrix. In this embodiment, r(i(/(/))) may be obtained by linearly compensating for the respiratory effect in accordance with the law of respiratory motion in three dimensions using the relative displacement of the reference system of the target organ, assuming W{d{ i))={W _x {d{t)\ W _y {d{i)\ W _z {d{t))), then the breathing correction matrix in homogeneous coordinate system is:

1 0 0

0 1 0

T((W(d(t)))

0 0 1 -W _z (d(t))

0 0 0 1

(9) where [- -w _y {d{t)) - ^(4))] is the translation vector.

In other embodiments, T(W(ci(†))) may also be obtained by compensating for the motion law in one or more dimensions after some processing, such as nonlinear transformation or some directions. Give different weights on them.

This embodiment uses a breathing model for correction before registration. In other embodiments, it is also possible to perform a respiration correction during the fusion process after the registration is completed. Assuming that the reference breathing depth at registration is do, the established breathing model can be used to correct the effects of breathing during the fusion process in real time. The principle of correction is the same as correcting different frames to the same breathing depth during the aforementioned registration process. Then, the correspondence between _M and _ec in the fusion after correction can be embodied by the formula (3) into the formula (10):

X _sec = P- T( W(d(t)))-R _P ro t)-AX _us ( 10)

Figure 8 shows the positional orientation based on ultrasound video and position sensor in the case of establishing a breathing model. Schematic diagram of the process of registration and fusion of information and position information of the respiratory sensor. Specifically include:

S31, collecting ultrasound video. Step S32, playing back the video, and selecting one or more frames of images for registration. In step S33, it is judged whether or not the breathing correction is performed on the registration frame. If so, step S34 is performed to correct the registration frame to the same breathing depth according to the breathing pattern, and the flow proceeds to step S35. If no, in step S35, the selected image is registered. In step S36, it is determined whether the breathing result is corrected for the fusion result. If so, step S37 is performed to correct the fusion result to the same breathing depth according to the breathing model, and the flow proceeds to step S38. If no, step S38 is performed to display the result of the fusion. During registration, the doctor can perform video playback, select one or more frames of ultrasound images for registration, and extract the position sensor of the position sensor to register for registration. If the selected multi-frame ultrasound image is not at the same depth of breath, the breath model can be used to correct to the same depth and then register. After the registration is completed, if you want to observe the fusion results at the same breathing depth, you can also use the breathing model to correct to that depth.

The above embodiment describes how to correct a multi-frame ultrasound image for registration to a certain breathing depth using a breathing model, and how to correct the fusion result to a certain breathing depth, assuming that the corrected target depth is the time when the breathing model is established. The reference depth d ₀ , of course, the corrected target depth can be understood for any meaningful breathing depth, when the breathing correction matrix is changed from r(ff(i ( ))) to ( (^»)) - T( W(d(t))) ₀ The fusion method of the ultrasound image and the pre-acquired modal image provided by the embodiment includes the following steps:

a playback step of playing or playing back pre-stored at least one piece of ultrasound video data according to the input play command, the ultrasound video data comprising an ultrasound image obtained by acquiring the target object from at least one plane, and corresponding to each frame of the ultrasound image Position pointing information and respiratory position information, wherein the position pointing information is sensed by a position sensor fixed to the ultrasonic probe during the ultrasonic image acquisition process, and the respiratory position information is a respiratory sensor fixed to the target object during the ultrasonic image acquisition process Inductive breathing of the target object is obtained;

In the registration step, the multi-frame ultrasound image is selected from the ultrasound video data to be registered with the modal image, and the registration method may adopt a common image registration algorithm instead of the registration method provided by the foregoing embodiment of the present application;

The fusion step performs image fusion on the multi-frame ultrasound image and the modal image after registration, and in the fusion process, the multi-frame ultrasound image after registration is corrected to the same breathing depth by using the breathing model, thereby being The fusion results were observed at a depth of breath.

In the fusion step, the establishment of the breathing model and the method of correcting to the same breathing depth can be referred to the corresponding portions in the foregoing embodiment 2, and will not be repeated here.

The method for fusing an ultrasound image and a pre-acquired modal image provided by this embodiment includes the following steps:

a selecting step of selecting a multi-frame ultrasound image from at least one piece of ultrasound video data, wherein the ultrasound video data comprises an ultrasound image obtained by acquiring the target object from at least one plane, and position pointing information corresponding to each frame of the ultrasound image, the positional pointing The information is obtained by a position sensor fixed on the ultrasonic probe during the ultrasonic image acquisition process. Meanwhile, the ultrasonic video data further includes respiratory position information corresponding to each frame of the ultrasonic image, and the respiratory position information is in the ultrasonic image acquisition process. Inducing the breathing of the target object by a breathing sensor fixed to the target object;

Wherein, before the multi-frame ultrasound image to be registered and the modal image are registered and/or during the fusion process, the multi-frame ultrasound image to be registered is corrected to the same breathing depth by the breathing model.

In one implementation, the steps of establishing a breathing model include:

The relative motion calculation sub-step, for each piece of ultrasound video data, selecting one frame of the ultrasound image corresponding to the reference breathing depth as the reference frame, and acquiring the motion amount of the target object corresponding to the reference frame corresponding to the other frame ultrasound image, according to the amount of motion, in the same Calculating the relative motion amount of the reference frame at different breathing depths of the target object corresponding to the ultrasound image of the other frame under the reference coordinate system;

The fitting modeling substeps are performed to fit different respiratory depths and their corresponding reference frame relative motion amounts to obtain a breathing model.

In still another embodiment, correcting the multi-frame ultrasound image to be registered to the same respiratory depth comprises: a calculation sub-step, combining the corresponding respiratory position with any frame ultrasound image in the multi-frame ultrasound image to be registered Information, obtaining a breathing depth corresponding to the ultrasound image of the frame, and obtaining a relative motion amount of the reference frame of the target object relative to the reference position in the frame ultrasound image according to the breathing model;

a calibration substep, according to the breathing model, the relative motion amount of the reference frame corresponding to the multi-frame ultrasound image Going to the same scheduled breathing depth.

For a detailed description of the above steps and sub-steps, reference may be made to the corresponding parts in the foregoing embodiment 2, and will not be repeated here. The multi-frame ultrasound image to be registered in the registration step of this embodiment may be a multi-frame image selected when the breathing model is established, or may be a multi-frame ultrasound image obtained based on the real-time ultrasound image.

In summary, in order to solve the two problems that the existing registration technology is based on real-time ultrasound, the registration accuracy and the fusion success rate are not high, and the method of eliminating the respiratory effect is not effective, the embodiments of the present application propose a new review based on the review. Sexual ultrasound video combined with position sensor information for registration fusion, while also establishing a breathing model for respiratory correction. In order to perform respiratory correction, the patient is required to breathe normally. Ultrasound video of more than one respiratory cycle is collected at several angles of each other angle, one video is collected at one position, and the sensor position and pointing information on the ultrasonic probe corresponding to each frame of ultrasonic data are recorded. And the position of the respiratory sensor fixed to the patient's abdomen, and then the patient's breathing model is established based on the video and sensor information. Prior to registration, in the acquired ultrasound video containing the information required for registration, each frame of ultrasound data simultaneously records the sensor position and pointing information fixed to the ultrasound probe, and the position of the respiratory sensor fixed to the patient's abdomen. When registering, the doctor is allowed to perform frame-by-frame or continuous video playback search, one or more frames of data are selected for registration, and the system simultaneously extracts corresponding ultrasonic probe position sensor information for registration. When multi-frame registration is selected, the physician can use the breathing model and respiratory sensor information to correct ultrasound data from different frames to the same breathing state. In the fusion process, the breathing model can also be used to correct the fusion result according to the position information of the respiratory sensor in real time, so as to eliminate or attenuate the respiratory effect.

The method for registering and merging with other modal data and respirating the registration data and the fusion result based on the ultrasound video proposed in the embodiments of the present application is applicable not only to the liver but also to other abdominal organs such as the kidney and the prostate.

A person skilled in the art can understand that all or part of the process of implementing the above embodiment method can be completed by a computer program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, the flow of an embodiment of the methods as described above may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM). However, it is not to be construed as limiting the scope of the invention. It should be noted that for the field A person skilled in the art can make several modifications and improvements without departing from the spirit and scope of the invention. Therefore, the scope of the invention should be determined by the appended claims.

Claims

claims

1. A method for fusing ultrasound images with pre-acquired modal images, which is characterized by including:

The selection step is to select at least one frame of ultrasound image from at least one piece of pre-stored ultrasound video data according to the input instruction. The ultrasound video data includes an ultrasound image obtained by collecting the target object from at least one plane, and each frame. Position pointing information corresponding to the ultrasound image, the position pointing information is sensed by a position sensor fixed on the ultrasound probe during the acquisition process of the ultrasound image;

The registration step is to register the selected at least one frame of ultrasound image with the modal image, using the position and pointing information of the at least one frame of ultrasound image in the registration process; and

The fusion step is to perform image fusion on the registered ultrasound image and the modal image.

2. The method according to claim 1, characterized in that,

Selecting multiple frames of ultrasound images in the selection step;

The method also includes:

The step of establishing a respiratory model is to establish a respiratory model based on the ultrasound video data;

In the breathing correction step, before registering the selected multiple frames of ultrasound images and modal images and/or during the fusion process, the breathing model is used to correct the multiple frames of ultrasound images to the same breathing depth.

3. The method of claim 2, characterized in that,

The ultrasonic video data also includes respiratory position information corresponding to each frame of the ultrasonic image. The respiratory position information is obtained by sensing the breathing of the target object by a respiratory sensor fixed on the target object during the acquisition of the ultrasonic image;

The steps of establishing a breathing model include:

Relative motion calculation sub-step: for each piece of ultrasound video data, select an ultrasound image corresponding to the reference breathing depth as the reference frame, and obtain the amount of movement of the target object corresponding to the other frames of ultrasound images relative to the reference frame. According to the amount of movement, In the same reference coordinate system, calculate the relative motion of the target object corresponding to the other frames of ultrasound images at different breathing depths in the reference system;

The fitting modeling sub-step is to fit the different breathing depths and their corresponding relative motion amounts of the reference system to obtain a breathing model.

4. The method according to claim 3, characterized in that: the objects corresponding to the other frames of ultrasound images The relative motion amount of the target object relative to the reference frame at different breathing depths satisfies: Minimize the error between the relative motion amount of the target object relative to the reference frame and the relative motion amount of the reference frame at the same predetermined breathing depth.

5. The method of claim 4, wherein the amount of motion is a rigid amount of motion, the amount of relative motion of the reference system is a displacement amount, and the reference systems of the target objects corresponding to the other frames of ultrasound images at different breathing depths The relative amount of motion is calculated by the following formula:

)))

Among them, is the breathing depth, J¾i ( )) is the displacement of the target object relative to the base frame, W (D) is the displacement of the target object at the breathing depth D, ρη).„ is the projection of the plane corresponding to the base frame Component, " is the segment number of all segments of ultrasound video data, i=i,.. .,n, f - is the error function of proj W{D))^.

6. The method of claim 3, wherein correcting multiple frames of ultrasound images to the same breathing depth includes:

Calculation sub-step: for any one of the multiple frames of ultrasound images, combined with its corresponding breathing position information, obtain the breathing depth corresponding to the frame of ultrasound image, and obtain the target object in the frame of the ultrasound image according to the breathing model The amount of relative movement of the reference frame relative to the reference position;

The correction sub-step is to correct the relative motion of the reference system corresponding to the multiple frames of ultrasound images to the same predetermined breathing depth according to the breathing model.

7. The method of claim 6, wherein the amount of motion is a rigid amount of motion, the amount of relative motion of the reference system is a displacement amount, and the correction sub-step includes:

According to the displacement amount, the translation vector of the target object in each dimension of the reference coordinate system is obtained; According to the breathing model, the rotation factor is determined;

According to the translation vector and rotation factor, the breathing correction matrix is obtained;

According to the respiration correction matrix, the relative motion amount of the reference system corresponding to the multiple frames of ultrasound images is corrected to the same predetermined respiration depth.

8. The method of claim 6, wherein correcting the relative motion of the reference system corresponding to the multiple frames of ultrasound images to the same predetermined breathing depth specifically includes:

For any point on any one of the multiple frames of ultrasound images, calculate its reference coordinates The position in the system is calculated using the formula ζ ³ · r(R _prote . to obtain the position of the point at the predetermined breathing depth, where P is the transformation matrix from the ultrasound image space to the modal image space, and Γ is used for correction. Spatial mapping method, R _prate is the transformation matrix from the position sensor space to the world coordinate system, is the transformation matrix from the ultrasound image space to the position sensor space, x _us is the coordinate of the point in the ultrasound image space.

9. The method according to claim 7, characterized in that, after the breathing correction step, the fusion step is obtained by the following formula:

Xsec = PT(W(d(t)))-R _probe (t)-AX _us

Among them, represents the coordinates of a pixel in the ultrasound image, ^^ represents the coordinates of the pixel in the modal image, P represents the transformation matrix from the ultrasound image space to the modal image space, r(wv/(/))) represents respiration Correction matrix, R^. o is the transformation matrix from the position sensor space to the world coordinate system, ^ represents the transformation matrix from the ultrasound image space to the position sensor space.

10. The method of claim 1, further comprising:

The multi-frame display step displays the registered or fused multiple frames of ultrasound images, and simultaneously displays the intersection lines and angles between the registered or fused multiple frames of ultrasound images.

1 1. A method for fusing ultrasound images with images of pre-acquired modalities, characterized by including:

The selection step is to select multiple frames of ultrasound images from at least one piece of ultrasound video data. The ultrasound video data includes ultrasound images obtained by collecting the target object from at least one plane, and position pointing information corresponding to each frame of ultrasound image, so The position and pointing information is sensed by a position sensor fixed on the ultrasound probe during the ultrasound image acquisition process;

The registration step is to register the multi-frame ultrasound image to be registered with the modal image;

The fusion step is to perform image fusion on the registered ultrasound image and modal image;

Wherein, before registering and/or during the fusion process of the multiple frames of ultrasound images to be registered and the modal images, the breathing model is used to correct the multiple frames of ultrasound images to be registered to the same respiration. depth.

12. The method of claim 11, wherein the ultrasound video data further includes respiratory position information corresponding to each frame of ultrasound image, and the respiratory position information is obtained by fixing the ultrasonic image during the acquisition process. The respiration sensor on the target object senses the respiration of the target object; The steps of establishing a breathing model include:

13. The method of claim 12, wherein correcting the multiple frames of ultrasound images to be registered to the same breathing depth includes:

Calculation sub-step: for any one of the multiple frames of ultrasound images, combined with its corresponding respiratory position information, obtain the breathing depth corresponding to the frame of ultrasound image, and obtain the target object in the frame of the ultrasound image according to the breathing model The amount of relative movement of the reference frame relative to the reference position;

14. An ultrasound fusion imaging navigation system, characterized by including:

A probe and a position sensor fixed on the probe;

An acquisition module, configured to collect the target object from at least one plane to obtain at least one segment of ultrasound video data containing registration information, and record position and pointing information for each frame of ultrasound image in each segment of ultrasound video data, so The above position and pointing information is sensed by the position sensor during the ultrasonic image acquisition process;

A selection module, configured to select at least one frame of ultrasound image from pre-stored ultrasound video data according to the input instruction;

A registration module, configured to register the selected at least one frame of ultrasound image with the modal image, using the position and pointing information of the at least one frame of ultrasound image in the registration process;

The fusion module is used to perform image fusion on registered ultrasound images and modal images.

15. The ultrasonic fusion imaging navigation system according to claim 14, further comprising: a respiration sensor fixed on the target object;

Respiration correction module, used to register the selected multi-frame ultrasound images and modal images before and /Or during the fusion process, a breathing model is used to correct the multiple frames of ultrasound images to the same breathing depth; the acquisition module is also used to record the breathing position for each frame of ultrasound image in each piece of ultrasound video data. Information, the respiration sensor is obtained by sensing the target object's respiration during the ultrasonic image acquisition process.