US9792668B2 - Photographic image acquistion device and method - Google Patents
Photographic image acquistion device and method Download PDFInfo
- Publication number
- US9792668B2 US9792668B2 US14/981,518 US201514981518A US9792668B2 US 9792668 B2 US9792668 B2 US 9792668B2 US 201514981518 A US201514981518 A US 201514981518A US 9792668 B2 US9792668 B2 US 9792668B2
- Authority
- US
- United States
- Prior art keywords
- image
- patch
- patches
- super
- panchromatic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G06K9/6215—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4007—Interpolation-based scaling, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4015—Demosaicing, e.g. colour filter array [CFA], Bayer pattern
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
- H04N23/661—Transmitting camera control signals through networks, e.g. control via the Internet
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/133—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing panchromatic light, e.g. filters passing white light
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/48—Increasing resolution by shifting the sensor relative to the scene
-
- H04N5/2258—
-
- H04N5/23232—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2621—Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
-
- H04N5/349—
-
- H04N9/045—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/64—Circuits for processing colour signals
- H04N9/68—Circuits for processing colour signals for controlling the amplitude of colour signals, e.g. automatic chroma control circuits
- H04N9/69—Circuits for processing colour signals for controlling the amplitude of colour signals, e.g. automatic chroma control circuits for modifying the colour signals by gamma correction
Definitions
- the present disclosure relates to photographic image acquisition devices and methods thereof, and in particular to still and video cameras having at least one imaging lens optically coupled to at least one image sensor that includes a focal plane array of photosensitive elements.
- Spatial resolution of photographic image acquisition devices is limited by the spatial resolution of the image sensor that includes a focal plane array of photosensitive elements, and the point spread function (optical blur) of the imaging lens optically coupled to the sensor.
- the first is by raising the spatial density of the focal plane array, and the second is by increasing the optical format of the image sensor and the lens to accommodate a larger number of photosensitive elements.
- the first approach requires size reduction of the photosensitive elements that causes a reduction in the number of photons collected by each photosensitive element per unit of time, and thus worsens the signal-to-noise ratio and the dynamic range of the image.
- the effective image resolution may drop due to the elevated image noise that drowns small image details, and due to an increased motion blur caused by a longer exposure time required to compensate for the elevated image noise.
- the second approach leads to an exponential cost increase of the image sensor due to an exponential dependency between the physical dimensions of the focal plane array and the fabrication cost.
- the same exponential cost increase applies to the corresponding large-format lenses.
- FIG. 1 is a diagram of a photographic image acquisition device in keeping with one embodiment of the present disclosure.
- FIG. 2A is a depiction of an example panchromatic image according to an embodiment of the present disclosure.
- FIG. 2B is a depiction of an example luminosity component of a multi-spectral image according to an embodiment of the present disclosure illustrating a mismatch with the panchromatic image due to parallax.
- FIG. 3A is a depiction of an example panchromatic image according to an embodiment of the present disclosure.
- FIG. 3B is a depiction of a luminosity component of a multi-spectral image within a larger field of view than the panchromatic image of FIG. 3A , according to one embodiment of the present disclosure.
- FIG. 4A is a depiction of another example panchromatic image of a scene according to an embodiment of the present disclosure.
- FIG. 4B is a depiction of another example of a luminosity subcomponent image of a multi-spectral image of the scene (according to effective pixel count) that is four times smaller than the panchromatic image of FIG. 4A .
- FIG. 4C is a depiction of a super-resolved image derived from super-resolution image reconstruction of the panchromatic image of FIG. 4A and the luminosity subcomponent image of FIG. 4B , according to one embodiment of the present disclosure.
- FIG. 5 is a flow chart illustrating one embodiment of an iterative method for super-resolution image reconstruction according to an embodiment of the present disclosure.
- employing post-acquisition computational methods may increase spatial resolution of the images post-capture.
- One of such methods is pan-sharpening, the fusion of at least two images acquired by at least two separate image sensors: a higher-resolution panchromatic image sensor and a lower-resolution multi-spectral sensor.
- the multi-spectral image sensor may include Bayer color filter array deposited on top of its focal plane array, as practiced in the art.
- Pan-sharpening is generally considered as a global substitution of the luminosity component or an intensity component of the multi-spectral image with the higher-resolution panchromatic image. For the substitution to succeed, both images are fully matched in scale and the field of view, and perfectly registered to each other globally and locally, with no parallax present.
- These pan-sharpening techniques have been developed for and applied in aerial and space imaging where the distance from the focal plane to the objects in the field of view is essentially infinite, and where the panchromatic image and the multispectral image are acquired sequentially while flying over the same land surface area, thus eliminating the parallax problem, e.g., so that both fields of view (the panchromatic and the multi-spectral) are fully matched and perfectly registered to each other globally and locally.
- Pan-sharpening methods may also allow a mismatch between the panchromatic and the multi-spectral images due to parallax. These methods may include additional means for resolving such a mismatch, for example, a light-projection-based depth-of-scene-estimation device integrated into the photographic image acquisition device. It may be evident to those skilled in the art that the task of pan-sharpening in the case of mismatched panchromatic and multi-spectral images is substantially complicated due to parallax and occlusions.
- super-resolution image reconstruction As yet another alternative to post-capture image resolution improvement, computational methods collectively known as super-resolution image reconstruction (or super-resolution for short) aim at reverting the effects of blurring in the lens and downsampling in the focal plane array. In contrast to pan-sharpening, super-resolution does not fuse a separately acquired higher-resolution image with a lower-resolution image. Instead, super-resolution exploits the intrinsic properties of the lower-resolution image itself.
- the desired sub-pixel offsets analogous to the sub-pixel offsets caused by the relative motion between the camera and the scene are found to exist in the low-resolution image itself due to a property of nonlocal self-similarity and redundancy at the scale of small image patches.
- multiple similar or substantially similar image patches are typically present at different locations in the same image, where each patch includes a small group of pixels, e.g., a square of 9-by-9 pixels.
- the image patches are a product of sampling similar areas of the scene by a finite-resolution sampling grid of the image sensor, the image patches generally include random, sub-pixel offsets relative to the grid, and thus provide additional sub-pixel resolution information.
- the effective resolution increase factor up to a factor of three in each of the two image dimensions
- Further embodiments include patch-based, super-resolution methods that employ a database of examples of high-resolution images and their blurred and subsampled low-resolution copies (otherwise referred to herein as “example-based” methods).
- Such database of examples is utilized to extract a compact dictionary of pairs of low-resolution and high-resolution image patches using a variety of learning techniques.
- This dictionary of patch pairs may be made substantially compact due to the property of self-similarity and redundancy at the scale of small image patches.
- a low-resolution image may then be super-resolved using said dictionary of pairs.
- the effective resolution increase factor may exceed that of the non-example based super-resolution approach, e.g., the approach based on sub-pixel sampling offsets.
- the super-resolved images may be less reliable in some applications because they are based on example-based predictions of what the high-resolution image might look like, as opposed to relying on the actual data present in the image itself.
- Further embodiments include example-based, super-resolution methods based on previously discussed patch-pair dictionary learning, while the number of entries in the dictionary is substantially reduced.
- Such dictionaries are composed of elementary patch atoms, such that any small image patch is closely approximated by a linear sum of a very small subset of these atoms.
- the ability to approximate any image patch by a linear sum of a few elementary atoms is due to a fundamental property of sparsity of natural images in certain mathematical domains.
- the property of sparsity and the property of nonlocal self-similarity and redundancy at the patch scale are closely related.
- Sparse coding methods are known for having high computational cost due to the combinatorial nature of identifying the best combination of patch atoms that matches a given patch.
- This property of cross-scale nonlocal patch similarity is the basis of another example-based super-resolution method, which in contrast to the other example-based methods does not require an external database of high-resolution images or the dictionary of patch pairs.
- This cross-scale method first locates a pair of similar patches A and B in the actual image and its downscaled copy, and then applies the coordinates of patch B in the downscaled copy as a pointer to a corresponding location in the actual (non-downscaled) image to extract a higher-resolution patch C corresponding to the downscaled patch B, which may not generally coincide with the location of patch A.
- Higher-resolution patch C is further used in formation of a super-resolved image on a pixel-by-pixel basis by applying C at the same location in the super-resolved image as the location of patch A in the actual image.
- the super-resolved image is formed using patches from locations in the actual image that are generally different from the “correct” locations in the unknown high-resolution image.
- this substitution produces visually acceptable results.
- this example-based method is combined with the method based on sub-pixel sampling offsets, the combined result is further improved in resolution.
- the above technique may be applied over multiple iterations with a gradual increase in the cross-scale factor and using the intermediate results as a starting point for the next iteration.
- An additional step of back-projection of the newly formed high-resolution image onto the low-resolution image via blurring and subsampling may be referred to as a means for verification, regularization, and error avoidance.
- the degree of match depends on the choice of the database of examples used in dictionary learning, and the choice of the scale gap between the high-resolution and the low-resolution image pairs in the database.
- increasing the effective pixel count by a factor of four may generally produce a close match between the reconstructed image and an unknown high-resolution image, while increasing the effective pixel count by a factor of sixteen or higher may not produce a meaningful result due to the predictive nature of the example-based methods.
- Super-resolution image reconstruction involves not only the step of upsampling of the low-resolution image, but may also include a step of reversing the optical blur incurred in the process of the photographic acquisition of the low-resolution image.
- Reversing the blur employs an estimation of the point-spread function (PSF) of the imaging lens, which is generally unknown, but is typically assumed to have a Gaussian shape.
- PSF point-spread function
- Methods of blur kernel estimation may include, as an example, an approach based on inferring the blur kernel by maximizing the match between the low-resolution patches and their blurred and down-sampled higher-resolution matches found in the same image.
- post-capture computational super-resolution methods are computationally expensive.
- the amount of computational power that would be required to perform such computations in real-time during image acquisition exceeds by orders of magnitude the computational power typically available in most cameras, including high-end cameras used in certain military applications.
- the additional bandwidth to transmit the super-resolved images and the additional storage space to record them would be impractical in many applications.
- example-based approaches to super-resolution are available, as well as various non-example-based ones that rely on sub-pixel offsets present in multiple similar image patches.
- At least one category of the example-based approaches relies on a database of high-resolution images and their blurred and subsampled low-resolution copies, while another relies on the examples found in the low-resolution image itself, with no external database used.
- the effective resolution increase factor may be lower in the non-example based methods relative to the example-based methods; however, the usage of the latter may be limited in some applications due to their predictive nature.
- the pan-sharpening methods rely on a separate higher-resolution image independently acquired by a panchromatic image sensor, but do not involve an increase of resolution beyond that of the panchromatic sensor.
- FIG. 1 is a block diagram of a photographic image acquisition device 100 A according to one embodiment of the present disclosure.
- the photographic image acquisition device 100 A may, for example, include a primary lens 100 , a secondary lens 101 , a primary image sensor 102 , a secondary image sensor, one or more pre-processors 104 and 105 , one or more encoders 106 and 107 , a computer readable storage medium 108 , and a media access controller (MAC) 109 .
- the photographic image acquisition device 100 A may be a distributed, networked system that communicates over a computer network 200 , and which may further, or alternatively, include a remote recorder (or viewer) 200 A.
- the photographic image acquisition device 100 A may further include a computing device 202 having a post-processor 210 and that is operatively coupled to an encoder 206 and a storage medium 208 located remotely from photographic image acquisition device 100 A, e.g., within a cloud computing system.
- the primary lens 100 is optically coupled to the primary image sensor 102 , a multi-spectral sensor, and the secondary lens 101 is optically coupled to the secondary image sensor 103 , a panchromatic sensor.
- Images acquired by the primary sensor 102 are pre-processed by the pre-processor 104
- the images acquired by the secondary sensor 103 are pre-processed by the pre-processor 105 .
- the pre-processing may include various processing steps, including pixel defect correction, color interpolation, noise filtering, unsharp masking, color correction, white balance control, gamma correction, tone mapping, color space conversion, as well as other image processing steps now known or later practiced in the art.
- the images pre-processed by pre-processors 104 and 105 are encoded by encoder 106 and 107 , respectively.
- the encoding may include image compression and encoding of the compressed images into one or more standard compression formats, such as H.264, H.265, JPEG, and the like.
- Encoded images are further recorded onto the storage medium 108 , which may include an SD memory card, a hard disk drive, a solid state drive, or the like.
- the encoded images are streamed over the computer network 200 or over other means of data transmission to the remote recorder (or viewer) 200 A by the media access controller 109 , and/or an Ethernet controller, a wireless network adapter, or the like means of data streaming over the computer network 200 .
- the recorded images are optionally retrieved from the storage medium 108 into the post-processor 110 for decoding and joint post-processing to produce super-resolved images.
- only a small region of interest is retrieved from storage medium 108 and super-resolved by the post-processor 110 , as opposed to the entire image at full field of view, to reduce processing power and computational requirements.
- the super-resolved images are further re-encoded by encoder 106 and optionally recorded onto storage medium 108 or streamed over the computer network 200 or over other means of data transmission to the remote recorder (or viewer) 200 A by the media access controller 109 .
- the encoded images are retrieved from the remote recorder 200 A and streamed over the computer network 200 back to the disclosed photographic image acquisition device via the media access controller 109 to be decoded and post-processed by the post-processor 110 , to produce a super-resolved image.
- the joint post-processing of images to produce a super-resolved image is performed by the remote post-processor 210 that receives the encoded images from the image acquisition device 100 A over the computer network via the media access controller 109 or the like means of data transmission.
- the present disclosure provides the photographic image acquisition device 100 A and methods thereof that raise the spatial resolution of the acquired images substantially beyond the individual capabilities of either the pan-sharpening approach or the super-resolution approach, and to advantageously produce a multiplicative effect on resolution increase unachievable by employing either of the two approaches separately.
- the multiplicative effect may be gained from the two previously mentioned components of the disclosure (the super-resolution reconstruction component and the pan-sharpening component) being made mutually interdependent through the property of nonlocal patch similarity and redundancy as will be explained in detail.
- the super-resolution of the multi-spectral image by itself cannot achieve a high scaling factor, while the pan-sharpening by itself cannot reach beyond the resolution of the panchromatic sensor.
- the photographic image acquisition device 100 A may also employ a pan-sharpening method that allows for a mismatch between the panchromatic and the multi-spectral images due to parallax without involving any additional hardware or methods for depth of scene estimation. In this way, the present disclosure also seeks to reduce the computational burden, bandwidth and storage space typically required in managing extreme resolution images and video.
- the optical axes of the primary lens 100 and the secondary lens 101 are set parallel to each other, and the primary image sensor 102 and secondary image sensor 103 are set in the same geometric plane, such that their focal plane arrays receive optical projections of the same scene.
- the focal length of the primary lens 100 is shorter than that of the secondary lens 101 , and in another embodiment, the primary lens 100 is a fisheye lens that projects an optically-warped image onto the focal plane of the primary image sensor 102 .
- the image sensors include identical focal plane arrays of identical photosensitive elements, except that the primary sensor 102 is a multi-spectral image sensor, which may include a Bayer color filter array deposited on top of its focal plane array, while the secondary image sensor 103 (or sensors) may be panchromatic.
- the images from the primary multi-spectral image sensor e.g., sensor 102
- the secondary panchromatic image sensors e.g., sensor(s) 103
- the multi-spectral image sensor 102 may generally produce noisier images under limited illumination than the panchromatic image sensor 103 due to the color filter array impeding light propagation from the lens to the focal plane array, and an infrared cut filter that is typically installed in front of the multi-spectral image sensor to prevent the infrared content of the scene from contaminating the spectral subcomponents.
- the resolution of the focal plane array of the secondary (e.g., panchromatic) image sensor 103 is higher than that of the primary (or multi-spectral) image sensor 102 .
- the resolution of the primary image sensor 102 is further reduced due to a larger optical field of view as a consequence of a shorter focal length of the primary lens.
- the pre-processing performed by the pre-processors 104 and 105 may include pixel defect correction, color interpolation, noise filtering, unsharp masking, color correction, white balance control, gamma correction, tone mapping, color space conversion, as well as other steps now known or latter practiced in the art.
- the encoding by the encoders 106 , 107 and/or 206 may include image compression and encoding of compressed images into one or more standard compression formats, such as H.264, H.265, JPEG, and the like.
- the storage medium 108 or 208 may include an SD memory card, a hard disk drive, a solid state drive, or the like.
- the means for streaming the encoded images over the computer network 200 may include an Ethernet transceiver, a Wi-Fi transceiver (e.g., based on the Wi-FiTM standard of the Wi-Fi Alliance), and the like.
- the secondary image sensor 103 captures images at a different frame rate than the primary image sensor 102 , or with a shorter exposure time, so that the scene content is further mismatched.
- One embodiment also includes computational means (such as will be discussed) for decoding the recorded images acquired by the primary and the secondary image sensors, and a joint post-processing reconstruction of the super-resolved images from the images acquired by the primary and the secondary image sensors, the super-resolved images further re-encoded, and recorded onto the storage medium 108 , or optionally streamed to the remote recorder (or viewer) 200 A over a computer network or via other means of data transmission.
- the decoding of the recorded images and joint post-processing reconstruction of the super-resolved images is performed by the remote post-processor 210 .
- the remote post-processor 210 may be located within the computing device 202 (such as a computer or server) connected to a local area network as the photographic image acquisition device 100 A.
- the remote post-processor 210 may receive the images from the computing device 202 , or a computer connected to a wide area network, or a specialized post-processor connected to the local or wide area network, or a post-processor connected directly to the disclosed image acquisition device, or the like.
- the post-processing reconstruction of the super-resolved images is limited to a small, zoomed region of interest, as opposed to the entire recorded image at full field of view, to advantageously reduce the computational cost, bandwidth and storage space required for the super-resolved image.
- the joint post-processing reconstruction of the super-resolved images includes a method of resolution increase involving fusion of the panchromatic and the multi-spectral images with a concurrent super-resolution reconstruction thereof, both the fusion and the super-resolution reconstruction exploiting nonlocal self-similarity and redundancy at the scale of small image patches.
- the disclosed method of resolution increase may advantageously allow a mismatch between the two images due to parallax.
- the disclosed method may further exploit a larger difference in the fields of view when the focal length of the primary imaging lens 100 is shorter than that of the secondary lens 101 .
- the method may further exploit a larger difference in image content when the frame rates or exposure times of the two image sensors are not identical, or when the panchromatic images are acquired asynchronously to the multi-spectral images.
- FIG. 5 is a flow chart illustrating one embodiment of an iterative method 500 for super-resolution image reconstruction according to an embodiment of the present disclosure.
- the method 500 may be performed by the image acquisition device 100 A, the remote recorder (or viewer) 200 A, the computing device 202 and post-processor 210 , or a combination thereof.
- the computational processing may be performed across the network 200 in a cloud server or the like upon receipt of input images to be processed.
- the method 500 of resolution increase may include a plurality of steps grouped into a first general step, a second general step, and a third general step that are computing steps taken to enhance resolution of an image captured by both the primary image sensor 102 and the secondary image sensor 103 .
- the method 500 may begin by acquiring a multi-spectral image of a scene with a multi-spectral image sensor ( 504 ).
- the method 500 may continue by acquiring a panchromatic image of the scene with a panchromatic sensor ( 508 ).
- the first general step may be to generate an initial crude estimate of an intended super-resolved image computed by the post-processor 110 or 210 or by extracting a luminosity subcomponent image from the multi-spectral image ( 512 ).
- the method 500 may continue by upsampling the luminosity subcomponent image to bring it up to an intended scale of the super-resolved image, to generate a “luminosity image” that is upsampled by one or more interpolation techniques ( 516 ).
- the upsampling is via Lanczos interpolation or a similar interpolation technique such as, for example, the bi-cubic interpolation, to generate the luminosity image.
- luminosity pixel values may be extracted from the multi-spectral image by a weighted mixture of red/green/blue (RGB) subcomponent pixel values at known ratios, for example, at approximately 60% of green, 30% of red, and 10% of blue.
- RGB red/green/blue
- Luminosity pixel values may be substituted in one embodiment with luma values that are extracted similarly except for the gamma correction of the RGB components.
- a series of patch extraction and patch processing may be performed to generate a new super-resolved image of higher-resolution.
- the method 500 may continue with determining whether at least one pixel still remains within the luminosity image to be processed ( 520 ), and as will be seen later, this luminosity image may be a new, higher-resolution version of the luminosity image. If there is not, and all the pixels have been processed, the method may continue to block 548 , to determine whether all iterations on the higher-resolution version of the luminosity image have been performed (discussed later).
- the method 500 may extract a small image patch centered the pixel ( 524 ).
- the method may continue by extracting a plurality of panchromatic image patches that are smaller than the luminosity patch by substantially the same ratio as the ratio between the two image sizes (e.g., the size of the panchromatic image and the size of the luminosity subcomponent image) ( 528 ).
- the method may continue by bringing the luminosity patch and the panchromatic patches to the same scale by either downsampling the luminosity patch or upsampling the panchromatic patch accordingly ( 532 ).
- Upsampling of the plurality panchromatic patches to the scale of the luminosity patch or an alternative downsampling of the luminosity patch to the scale of panchromatic patches can be performed in a variety of ways.
- the upsampling of the plurality of panchromatic patches may be performed via Lanczos interpolation or a similar interpolation technique such as, for example, the bi-cubic interpolation.
- the scale ratio is an integer value, for example two (“2”)
- the downsampling of the luminosity patch may involve calculating each pixel value of the downsampled patch by averaging values of four neighbor pixel values of the non-downsampled image by a 2-by-2 neighborhood, for example.
- downsampling may involve calculating each pixel value of the downsampled luminosity patch by weighing the values of pixels surrounding the pixel in the non-downsampled image in proportion to their geometric proximity to the nodes of the pixel grid of the downsampled image.
- the plurality of panchromatic patches may contain patches extracted from the panchromatic image rotated at a plurality of different angles in the image plane relative to the luminosity image, or may contain patches extracted from the panchromatic image scaled at a plurality of different scale ratios relative to the upsampled luminosity image scale.
- the method 500 may continue by matching the scaled luminosity patch against the plurality of panchromatic patches using a measure of patch similarity, such as the Euclidian distance between the patches, or another similarity metric ( 536 ).
- the luminosity patch may be processed using a pre-trained neural network to improve signal-to-noise ratio and resolution, and to reduce blurring, prior to matching the patch with each of the plurality of the panchromatic patches.
- the luminosity patch may be processed using the pre-trained neural network to improve the similarity between the luminosity patch and the panchromatic patches using pairs of low and high resolution patches.
- the method 500 may then continue by calculating a new pixel value from multiple pixel values at the centers of the panchromatic image patches by weighing relative contributions of the plurality of panchromatic image patches in proportion to their respective similarity to the luminosity patch ( 540 ).
- the method 500 may continue by inserting the new pixel value into the luminosity image at the coordinates corresponding to a center of the interpolated luminosity patch, to replace the original pixel value ( 544 ).
- the measure of patch similarity may involve local gradients of the first, second, and optionally higher orders.
- the method 500 may continue within the general second step by iterating over the pixels of the luminosity image to gradually replace all of its pixels, to form a new higher-resolution image as an improved estimate of the intended super-resolved image in place of the initial crude estimate made prior to the second general step ( 520 ).
- the method 500 may continue within the second general step by performing further iteration(s) over the entire image, e.g., from block 520 through block 544 , but by processing the new higher-resolution image ( 548 ).
- each iteration forms a new higher-resolution image (e.g., enhanced version of the luminosity image) as an improved estimate of the intended super-resolved image in place of the higher-resolution image formed at the previous iteration.
- the number of iterations may be at least two in one embodiment.
- the method 500 may continue (in a third general step) to determine whether a target scale for the super-resolved image has been met ( 552 ). If yes, the target scale has been met, the method may continue with upsampling chromaticity components of the multi-spectral, super-resolved image to the target scale before performing the steps of block 560 ( 556 ). If no, the target scale for the super-resolved image has not been met, the method may loop back to block 516 , to upsample the super-resolved image to the larger (target) scale ( 516 ).
- the method may continue with encoding and storing (or transmitting) the super-resolved image at the target scale ( 560 ).
- each of the iterations may additionally involve a reversal of optical blur applied to each of the newly obtained estimates of the intended super-resolved image.
- the point-spread function (PSF) of the imaging lens also known as the blur kernel, is generally unknown, but may be assumed to have a Gaussian shape.
- the blur kernel is estimated using a calibration procedure employing acquisition of a test image by the photographic image acquisition device 100 A or the remote recorder 200 A at two different distances corresponding to the ratio of the super-resolved image and the low-resolution image.
- An external computing device such as the remote recorder 200 A may then minimize the difference between the images by applying the blur kernel to the higher-resolution test image, downsampling the resultant processed image, and iteratively modifying the kernel weights.
- the minimization in the present embodiment may include the least squares method or the like.
- the plurality of panchromatic patches extracted in the second general step is extracted not only from a given panchromatic image, but also from multiple temporally proximal panchromatic images (e.g., video frames) acquired shortly before and after the currently-processed image.
- the area of the panchromatic image for extracting the plurality of the panchromatic patches is limited to a close geometric proximity of a given luminosity image patch.
- the extraction of the panchromatic patches is done at random, followed by a subsequent additional patch extraction near the most similar patches found initially, where the sequence is iterated multiple times until a sufficient number of similar patches are found.
- the third general step of the disclosed hybrid method is performed after the multiple iterations ( 548 ) are completed, and includes the upsampling of the chromaticity subcomponent images of the multi-spectral image to the scale of the super-resolved luminosity image via Lanczos interpolation or a similar interpolation technique such as, for example, the bi-cubic interpolation ( 556 ).
- the spatial resolution of the chromaticity subcomponents attained via interpolation is not increased, the resolution is not critical for visual perception of small image details of the super-resolved image as long as its luminosity subcomponent is super-resolved.
- the chromaticity subcomponent pixel values of the multi-spectral image may be further adjusted.
- the adjustment of chromaticity values is performed prior to interpolation of the chromaticity subcomponent images by first down-sampling the super-resolved luminosity image to the scale of the chromaticity images and then adjusting the chromaticity at that scale.
- an additional optical blur reversal is performed on the chromaticity components using a default blur kernel as a generic model of the optical blur, for example, a Gaussian-shaped default kernel, or an estimated kernel from the blur kernel estimation procedure as previously discussed.
- a default blur kernel as a generic model of the optical blur, for example, a Gaussian-shaped default kernel, or an estimated kernel from the blur kernel estimation procedure as previously discussed.
- the degree of patch similarity between the two images e.g., the panchromatic and the multi-spectral images
- the degree of patch similarity between the two images is generally higher than the degree of similarity with patches extracted from an external database of example images. This is the case because both image sensors of the disclosed image acquisition device capture a substantially-similar scene content under the identical illumination conditions, and because both images are similarly pre-processed and encoded as disclosed.
- the similarity at the patch level is still high if the fisheye lens produces a conformal stereographic projection, after adjusting for local rotation and scaling factor for each patch.
- the secondary image sensor 103 coupled to the secondary lens is mounted on a motorized pan-and-tilt mechanism to acquire multiple images throughout a larger field of view of the primary image sensor 102 , where the acquisition is performed at a substantially slower frame rate than the frame rate of the primary sensor 102 .
- the secondary lens 101 is a motorized varifocal or a motorized parfocal lens to allow changes of the field of view and the corresponding resolution of the resultant images.
- pan-sharpening methods Unlike pan-sharpening methods, disclosed image fusion is no longer treated as a global substitution of the luminosity or intensity of the multi-spectral image with the higher-resolution panchromatic image. For such substitution to succeed, both images have to be fully matched in scale and the field of view, perfectly registered to each other globally and locally, with no parallax present.
- the present disclosure employs nonlocal self-similarity and redundancy at the patch level, and thus does not depend on a perfect global and local correspondence between the images, and eliminates any no-parallax requirements.
- the disclosed methods include super-resolution image reconstruction performed concurrently with image pan-sharpening to achieve a large multiplicative resolution increase factor.
- This multiplicative effect is gained because the super-resolution reconstruction component and the pan-sharpening component are made mutually interdependent through the property of nonlocal patch similarity and redundancy: without the panchromatic image, the super-resolution of the multi-spectral image by itself cannot achieve a high scaling factor, while the pan-sharpening by itself cannot reach beyond the resolution of the panchromatic sensor.
- the multi-spectral sensor 102 and the panchromatic sensor 103 include identical arrays of photosensitive elements where the multi-spectral sensor 102 includes a Bayer color filter.
- the effective computational, super-resolution factor relative to the panchromatic image resolution is two in each dimension. In this case, the multiplicative factor of the disclosed method is 3.5 in each dimension, or an effective increase in image pixel density of over 12.
- Certain embodiments of the disclosure may be used for video surveillance systems, including land-based, vehicle-based, and airborne surveillance. Such systems may be employed to aid in the imaging and detection of facial features, lettering, manmade versus natural phenomena, and more.
- the systems may be a low-cost, low-weight, small-sized payload camera, while the remote post-processor 210 may be employed to produce the computationally super-resolved images that are similar or superior in image resolution as previously achievable only in high-cost, heavy-weight, larger-size video capture systems.
- FIGS. 2A and 2B are illustrations of two images acquired by the disclosed photographic image acquisition device 100 A.
- FIG. 2A illustrates a panchromatic image
- FIG. 2B illustrates a luminosity component of a multi-spectral image at the same scale as the panchromatic image. Note that the panchromatic image of FIG. 2A does not perfectly match the luminosity image of FIG. 2B due to parallax.
- a small image patch 111 in the luminosity image of FIG. 2B is occluded in the panchromatic image due to parallax; however, there are multiple similar patches 112 and 113 found in the panchromatic image that are used in fusion and super-resolution reconstruction as disclosed with reference to FIG. 5 .
- an image patch 114 in the luminosity image of FIG. 2B is occluded in the panchromatic image of FIG. 2A due to parallax; however, there are multiple similar patches 115 and 116 found in the panchromatic image that are used in fusion and super-resolution reconstruction as disclosed with reference to FIG. 5 .
- the scale relationship between the size of small image patches and the entire field of view depicted in FIGS. 2A and 2B is overstated for illustration purposes, and is not representative of the actual scale relationship.
- FIG. 3A is a depiction of an example panchromatic image according to an embodiment of the present disclosure.
- FIG. 3B is a depiction of a luminosity component of a multi-spectral image within a larger field of view than the panchromatic image of FIG. 3A , according to one embodiment of the present disclosure.
- Patch 117 in the luminosity image has no corresponding region in the panchromatic image due to the difference in the fields of view; however, there are multiple similar patches 118 and 119 found in the panchromatic image that are used in fusion and super-resolution reconstruction as disclosed with reference to FIG. 5 .
- the scale relationship between the size of small image patches and the entire field of view depicted in FIGS. 3A and 3B is overstated for illustration purposes, and is not representative of the actual scale relationship.
- FIG. 4A is a depiction of another example panchromatic image of a scene according to an embodiment of the present disclosure.
- FIG. 4B is a depiction of another example of a luminosity subcomponent image of a multi-spectral image of the scene (according to effective pixel count) that is four times smaller than the panchromatic image of FIG. 4A .
- FIG. 4C is a derived from super-resolution image reconstruction of the panchromatic image of FIG. 4A and the luminosity subcomponent image of FIG. 4B , according to one embodiment of the present disclosure.
- the white arrow between FIG. 4B and FIG. 4C represents the upsampling of the low-resolution luminosity subcomponent image of the multi-spectral image to the intended scale of the super-resolved image by Lanczos interpolation, or by another interpolation technique such as, for example, the bi-cubic interpolation.
- This upsampling may be an initial crude estimate of the intended super-resolved image at the target scale.
- the black arrow between FIGS. 4A and 4C represents the super-resolution reconstruction of the interpolated luminosity image using nonlocal similarity and redundancy of small image patches in the panchromatic image and the interpolated luminosity image.
- the scale relationships between images depicted in FIGS. 4A and 4B represent the difference in effective pixel count relative to the field of view.
- the luminosity subcomponent image shown in FIG. 4B is illustrated approximately four times smaller than the panchromatic image of FIG. 4A to reflect the scenario in one embodiment where both image sensors include an identical array of photosensitive elements while the multi-spectral image sensor includes the Bayer color filter array that reduces its effective pixel count relative to the panchromatic sensor by an approximate factor of at least 1.75 in each dimension due to color interpolation.
- a patch 120 lies within the interpolated luminosity image (shown in FIG. 4C ) and a plurality of similar patches 121 , 122 , 123 , 124 lies within the panchromatic image of FIG. 4A that are approximately four times smaller than patch 120 .
- the scale relationship between the size of small image patches and the entire field of view depicted in FIGS. 4A through 4C is overstated for illustration purposes, and is not representative of the actual scale relationship.
- FIG. 4C Also shown in FIG. 4C is the final super-resolved image with a super-resolution factor of approximately four (two in each dimension) relative to the panchromatic image of FIG. 4A .
- the resolution increase by approximately a factor of four by employing nonlocal patch similarity and redundancy is feasible under a variety of conditions.
- Resolution increase by approximately a factor of four relative to the panchromatic image as shown in FIG. 4A corresponds to an approximate factor of twelve relative to the luminosity image, and the multi-spectral image itself.
- Said resolution increase factor may be achievable by the present disclosure as disclosed heretofore; however, the range of said resolution increase may be limited by the signal-to-noise-ratio of the image, the scene content, and other variables, known to those skilled in the art.
- Another embodiment of the present disclosure includes a similar iterative method directed to parallax compensation in photographic multi-aperture image acquisition systems.
- This embodiment may employ a primary aperture and one or more secondary apertures optically projecting substantially the same scene onto the primary and secondary digital image sensors 102 and 103 .
- These apertures may be identical or non-identical in regards to their F-stop number, the lenses being identical or non-identical in regards to their focal length. (The F-stop number is the ratio of focal length to the diameter of the entrance of the pupil, and a quantitative measure of lens speed.)
- the image sensors 102 and 103 may be identical or non-identical in regards to their resolution, spectral characteristics, or sensitivity.
- the parallax compensation may be needed where, for example, the data from the sensors 102 and 103 are used to produce an enhancement of one or more of image characteristics of the primary image, such as potentially its resolution, dynamic range, signal-to-noise ratio, color, or low-light performance, which is achieved via fusion of the primary aperture image with the secondary aperture image or images.
- image characteristics of the primary image such as potentially its resolution, dynamic range, signal-to-noise ratio, color, or low-light performance
- This further embodiment may then include the steps discussed with reference to FIG. 5 , such as extracting an image patch surrounding each pixel of the primary aperture image and extracting a plurality of image patches from the secondary aperture image or images.
- Each patch of the plurality of secondary aperture image patches may then be matched against the primary aperture image patch using a measure of patch similarity, and a new pixel value may be calculated from multiple pixel values at the centers of the secondary aperture image patches by weighing their relative contributions in proportion to their similarity to the primary aperture image patch.
- the new pixel value may then be inserted into the primary aperture image at the coordinates corresponding to the center of the primary aperture image patch thereby replacing the original pixel value at these coordinates.
- the above-described process may be iterated over all pixels of the primary aperture image to gradually replace all of its pixels to form an enhanced fused image in place of the initial primary aperture image.
- the process may be repeated one or more times over the entire primary image, using the previous iteration resultant image as the new starting point for the next iteration, where each iteration forms an improved fused image in place of the image formed at the previous iteration.
- the iterations may employ a regularization step using Total Variation regularization or alternatively Tikhonov regularization.
- the iterations may additionally employ a reversal of optical blur applied to the newly obtained fused image.
- the plurality of secondary aperture image patches may be extracted not only from a given secondary aperture image, but also from multiple temporally proximal images (video frames) acquired shortly before and/or shortly after the given image.
- the fusion of the primary and secondary aperture image or images may advantageously exploit nonlocal self-similarity and redundancy at the scale of small image patches between the primary and the secondary images, as discussed above with respect to other embodiments, thereby advantageously exploiting mismatches between the images due to parallax, and advantageously eliminating the requirement of global or local registration of the images.
Abstract
Description
Claims (29)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/981,518 US9792668B2 (en) | 2014-08-18 | 2015-12-28 | Photographic image acquistion device and method |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201462038801P | 2014-08-18 | 2014-08-18 | |
US14/828,503 US9225889B1 (en) | 2014-08-18 | 2015-08-17 | Photographic image acquisition device and method |
US14/981,518 US9792668B2 (en) | 2014-08-18 | 2015-12-28 | Photographic image acquistion device and method |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/828,503 Continuation US9225889B1 (en) | 2014-08-18 | 2015-08-17 | Photographic image acquisition device and method |
Publications (2)
Publication Number | Publication Date |
---|---|
US20160117800A1 US20160117800A1 (en) | 2016-04-28 |
US9792668B2 true US9792668B2 (en) | 2017-10-17 |
Family
ID=54932571
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/828,503 Active US9225889B1 (en) | 2014-08-18 | 2015-08-17 | Photographic image acquisition device and method |
US14/981,518 Active 2035-12-01 US9792668B2 (en) | 2014-08-18 | 2015-12-28 | Photographic image acquistion device and method |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/828,503 Active US9225889B1 (en) | 2014-08-18 | 2015-08-17 | Photographic image acquisition device and method |
Country Status (2)
Country | Link |
---|---|
US (2) | US9225889B1 (en) |
WO (1) | WO2016028819A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10482576B2 (en) * | 2018-03-19 | 2019-11-19 | Mitsubishi Electric Research Laboratories, Inc. | Systems and methods for multi-spectral image super-resolution |
Families Citing this family (45)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8724928B2 (en) * | 2009-08-31 | 2014-05-13 | Intellectual Ventures Fund 83 Llc | Using captured high and low resolution images |
US9984440B2 (en) * | 2013-06-18 | 2018-05-29 | Adobe Systems Incorporated | Iterative patch-based image upscaling |
US10931866B2 (en) | 2014-01-05 | 2021-02-23 | Light Labs Inc. | Methods and apparatus for receiving and storing in a camera a user controllable setting that is used to control composite image generation performed after image capture |
US9866764B2 (en) * | 2014-11-21 | 2018-01-09 | Motorola Mobility Llc | Method and apparatus for synchronizing auto exposure between chromatic pixels and panchromatic pixels in a camera system |
US9538126B2 (en) * | 2014-12-03 | 2017-01-03 | King Abdulaziz City For Science And Technology | Super-resolution of dynamic scenes using sampling rate diversity |
US9846919B2 (en) | 2015-02-16 | 2017-12-19 | Samsung Electronics Co., Ltd. | Data processing device for processing multiple sensor data and system including the same |
JP6594101B2 (en) * | 2015-08-19 | 2019-10-23 | キヤノン株式会社 | Image processing apparatus, image processing method, and image processing program |
US10341561B2 (en) * | 2015-09-11 | 2019-07-02 | Facebook, Inc. | Distributed image stabilization |
EP3440625B1 (en) * | 2016-04-05 | 2021-06-09 | Koninklijke Philips N.V. | Medical image processing |
KR102519803B1 (en) * | 2016-04-11 | 2023-04-10 | 삼성전자주식회사 | Photographying apparatus and controlling method thereof |
US10412291B2 (en) | 2016-05-19 | 2019-09-10 | Scenera, Inc. | Intelligent interface for interchangeable sensors |
US10509459B2 (en) | 2016-05-19 | 2019-12-17 | Scenera, Inc. | Scene-based sensor networks |
CN106204489B (en) * | 2016-07-12 | 2019-04-16 | 四川大学 | The single image super resolution ratio reconstruction method converted in conjunction with deep learning and gradient |
CN106251316B (en) * | 2016-08-29 | 2018-09-18 | 珠海格力电器股份有限公司 | The restorative procedure and device of photo |
US10693843B2 (en) | 2016-09-02 | 2020-06-23 | Scenera, Inc. | Security for scene-based sensor networks |
CN110249620B (en) * | 2017-02-17 | 2021-04-23 | 富士胶片株式会社 | Image processing apparatus, image processing method, and camera system |
CN106952228B (en) * | 2017-03-10 | 2020-05-22 | 北京工业大学 | Super-resolution reconstruction method of single image based on image non-local self-similarity |
US10051201B1 (en) | 2017-03-20 | 2018-08-14 | Google Llc | Camera system including lens with magnification gradient |
CA3064779A1 (en) * | 2017-09-15 | 2019-03-21 | Kent Imaging | Hybrid visible and near infrared imaging with an rgb color filter array sensor |
US20190147296A1 (en) * | 2017-11-15 | 2019-05-16 | Nvidia Corporation | Creating an image utilizing a map representing different classes of pixels |
KR102314703B1 (en) * | 2017-12-26 | 2021-10-18 | 에스케이하이닉스 주식회사 | Joint dictionary generation method for image processing, interlace based high dynamic range imaging apparatus using the joint dictionary and image processing method of the same |
CN108156382A (en) * | 2017-12-29 | 2018-06-12 | 上海爱优威软件开发有限公司 | A kind of photo processing method and terminal |
CN108259875B (en) * | 2018-02-05 | 2020-03-24 | 成都天诚慧芯科技有限公司 | Digital image gamma correction hardware implementation method and system |
US11157745B2 (en) | 2018-02-20 | 2021-10-26 | Scenera, Inc. | Automated proximity discovery of networked cameras |
CN109115348B (en) * | 2018-07-24 | 2020-06-30 | 哈尔滨工业大学 | Three-dimensional temperature reconstruction combination method based on flame light field refocusing image |
CN109543544B (en) * | 2018-10-25 | 2021-04-09 | 北京市商汤科技开发有限公司 | Cross-spectrum image matching method and device, electronic equipment and storage medium |
CN109302600B (en) * | 2018-12-06 | 2023-03-21 | 成都工业学院 | Three-dimensional scene shooting device |
US10990840B2 (en) | 2019-03-15 | 2021-04-27 | Scenera, Inc. | Configuring data pipelines with image understanding |
CA3144811A1 (en) * | 2019-07-09 | 2021-01-14 | Pricer Ab | Stitch images |
US11303877B2 (en) | 2019-08-13 | 2022-04-12 | Avigilon Corporation | Method and system for enhancing use of two-dimensional video analytics by using depth data |
US20230058472A1 (en) * | 2020-01-28 | 2023-02-23 | Gopro, Inc. | Sensor prioritization for composite image capture |
EP3885813A1 (en) | 2020-03-27 | 2021-09-29 | Leica Microsystems CMS GmbH | Method and device for estimating a sted resolution |
KR20210121508A (en) * | 2020-03-30 | 2021-10-08 | 삼성전자주식회사 | Image system including image signal processing unit and operation method of image signal processing unit |
US11481876B2 (en) * | 2020-04-08 | 2022-10-25 | Mitsubishi Electric Research Laboratories, Inc. | Systems and methods for blind multi-spectral image fusion |
EP3944184A1 (en) * | 2020-07-20 | 2022-01-26 | Leica Geosystems AG | Dark image enhancement |
CN111861929A (en) * | 2020-07-24 | 2020-10-30 | 深圳开立生物医疗科技股份有限公司 | Ultrasonic image optimization processing method, system and device |
CN112016683B (en) * | 2020-08-04 | 2023-10-31 | 杰创智能科技股份有限公司 | Data reinforcement learning and training method, electronic device and readable storage medium |
CN111932594B (en) * | 2020-09-18 | 2023-12-19 | 西安拙河安见信息科技有限公司 | Billion pixel video alignment method and device based on optical flow and medium |
US20220114700A1 (en) * | 2020-10-08 | 2022-04-14 | Nvidia Corporation | Upsampling an image using one or more neural networks |
US11651474B2 (en) | 2020-11-04 | 2023-05-16 | Bae Systems Information And Electronic Systems Integration Inc. | Real-time super resolution at long standoff ranges |
KR20220135724A (en) * | 2021-03-31 | 2022-10-07 | 서울대학교산학협력단 | Method and device of processing data conforming to statistical distribution |
CN115802183B (en) * | 2021-09-10 | 2023-10-20 | 荣耀终端有限公司 | Image processing method and related device |
WO2023149963A1 (en) * | 2022-02-01 | 2023-08-10 | Landscan Llc | Systems and methods for multispectral landscape mapping |
CN117252927B (en) * | 2023-11-20 | 2024-02-02 | 华中科技大学同济医学院附属协和医院 | Catheter lower intervention target positioning method and system based on small target detection |
CN117437131B (en) * | 2023-12-21 | 2024-03-26 | 珠海视新医用科技有限公司 | Electronic staining method and device for endoscope image, equipment and storage medium |
Citations (35)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5949914A (en) | 1997-03-17 | 1999-09-07 | Space Imaging Lp | Enhancing the resolution of multi-spectral image data with panchromatic image data using super resolution pan-sharpening |
US6011875A (en) | 1998-04-29 | 2000-01-04 | Eastman Kodak Company | Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening |
US6476865B1 (en) | 2001-03-07 | 2002-11-05 | Eastman Kodak Company | Sparsely sampled image sensing device with color and luminance photosites |
US7199348B2 (en) | 2004-08-25 | 2007-04-03 | Newport Imaging Corporation | Apparatus for multiple camera devices and method of operating same |
US20070103595A1 (en) | 2005-10-27 | 2007-05-10 | Yihong Gong | Video super-resolution using personalized dictionary |
US20090110285A1 (en) | 2007-10-26 | 2009-04-30 | Technion Research And Development Foundation Ltd | Apparatus and method for improving image resolution using fuzzy motion estimation |
US7769229B2 (en) | 2006-11-30 | 2010-08-03 | Eastman Kodak Company | Processing images having color and panchromatic pixels |
US7826685B2 (en) | 2006-12-01 | 2010-11-02 | Harris Corporation | Spatial and spectral calibration of a panchromatic, multispectral image pair |
US7830430B2 (en) | 2005-07-28 | 2010-11-09 | Eastman Kodak Company | Interpolation of panchromatic and color pixels |
US7835594B2 (en) | 2006-12-01 | 2010-11-16 | Harris Corporation | Structured smoothing for superresolution of multispectral imagery based on registered panchromatic image |
US7889921B2 (en) | 2007-05-23 | 2011-02-15 | Eastman Kodak Company | Noise reduced color image using panchromatic image |
US8115825B2 (en) | 2008-02-20 | 2012-02-14 | Apple Inc. | Electronic device with two image sensors |
US20120044328A1 (en) | 2010-08-17 | 2012-02-23 | Apple Inc. | Image capture using luminance and chrominance sensors |
US20120086850A1 (en) | 2009-04-20 | 2012-04-12 | Yeda Research And Development Co., Ltd. | Super resolution from a single signal |
US20120268623A1 (en) | 2004-05-05 | 2012-10-25 | Centre National De La Recherche Scientifique-Cnrs | Image data processing method by reducing image noise, and camera integrating means for implementing said method |
US8319822B2 (en) | 2007-12-27 | 2012-11-27 | Google Inc. | High-resolution, variable depth of field image device |
US8320712B2 (en) | 2008-03-07 | 2012-11-27 | Korea Aerospace Research Institute | Satellite image fusion method and system |
US20130129207A1 (en) | 2011-11-18 | 2013-05-23 | Dehong Liu | Method for Pan-Sharpening Panchromatic and Multispectral Images Using Dictionaries |
US8471918B2 (en) | 2006-03-22 | 2013-06-25 | Panasonic Corporation | Imaging device with plural imaging regions and parallax computing portion |
US8487996B2 (en) * | 2011-04-25 | 2013-07-16 | Skybox Imaging, Inc. | Systems and methods for overhead imaging and video |
US8581995B2 (en) | 2011-01-25 | 2013-11-12 | Aptina Imaging Corporation | Method and apparatus for parallax correction in fused array imaging systems |
US8619082B1 (en) | 2012-08-21 | 2013-12-31 | Pelican Imaging Corporation | Systems and methods for parallax detection and correction in images captured using array cameras that contain occlusions using subsets of images to perform depth estimation |
US20140036112A1 (en) | 2009-05-04 | 2014-02-06 | Digitaloptics Corporation | Dual lens digital zoom |
US8711452B2 (en) | 2005-07-28 | 2014-04-29 | Omnivision Technologies, Inc. | Processing color and panchromatic pixels |
US20140118569A1 (en) | 2009-03-19 | 2014-05-01 | Digitaloptics Corporation | Dual sensor camera |
WO2014083489A1 (en) | 2012-11-28 | 2014-06-05 | Corephotonics Ltd. | High-resolution thin multi-aperture imaging systems |
US20140192238A1 (en) | 2010-10-24 | 2014-07-10 | Linx Computational Imaging Ltd. | System and Method for Imaging and Image Processing |
US20140267759A1 (en) | 2013-03-15 | 2014-09-18 | Digitalglobe, Inc. | Estimation of fine structure within pixels in satellite imagery |
US20140368681A1 (en) | 2013-06-14 | 2014-12-18 | Sony Corporation | Methods and devices for parallax elimination |
US20150036014A1 (en) | 2010-12-14 | 2015-02-05 | Pelican Imaging Corporation | Systems and Methods for Synthesizing High Resolution Images Using Images Captured by an Array of Independently Controllable Imagers |
US20150109524A1 (en) | 2013-10-18 | 2015-04-23 | The Lightco Inc. | Methods and apparatus for implementing and/or using a camera device |
US9020256B2 (en) | 2011-04-22 | 2015-04-28 | Exelis Inc. | System and method for combining color information with spatial information in multispectral images |
US20150117332A1 (en) | 2013-10-31 | 2015-04-30 | Qinghua Li | Systems, methods, and devices for efficient device-to-device channel contention |
US20150116547A1 (en) | 2013-10-26 | 2015-04-30 | The Lightco Inc. | Methods and apparatus for use with multiple optical chains |
US20150124083A1 (en) | 2012-07-04 | 2015-05-07 | Linx Computational Imaging Ltd. | Image Processing in a Multi-Channel Camera |
-
2015
- 2015-08-17 US US14/828,503 patent/US9225889B1/en active Active
- 2015-08-18 WO PCT/US2015/045767 patent/WO2016028819A1/en active Application Filing
- 2015-12-28 US US14/981,518 patent/US9792668B2/en active Active
Patent Citations (42)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5949914A (en) | 1997-03-17 | 1999-09-07 | Space Imaging Lp | Enhancing the resolution of multi-spectral image data with panchromatic image data using super resolution pan-sharpening |
US6011875A (en) | 1998-04-29 | 2000-01-04 | Eastman Kodak Company | Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening |
US6476865B1 (en) | 2001-03-07 | 2002-11-05 | Eastman Kodak Company | Sparsely sampled image sensing device with color and luminance photosites |
US20120268623A1 (en) | 2004-05-05 | 2012-10-25 | Centre National De La Recherche Scientifique-Cnrs | Image data processing method by reducing image noise, and camera integrating means for implementing said method |
US7199348B2 (en) | 2004-08-25 | 2007-04-03 | Newport Imaging Corporation | Apparatus for multiple camera devices and method of operating same |
US8711452B2 (en) | 2005-07-28 | 2014-04-29 | Omnivision Technologies, Inc. | Processing color and panchromatic pixels |
US7830430B2 (en) | 2005-07-28 | 2010-11-09 | Eastman Kodak Company | Interpolation of panchromatic and color pixels |
US20070103595A1 (en) | 2005-10-27 | 2007-05-10 | Yihong Gong | Video super-resolution using personalized dictionary |
US8471918B2 (en) | 2006-03-22 | 2013-06-25 | Panasonic Corporation | Imaging device with plural imaging regions and parallax computing portion |
US7769229B2 (en) | 2006-11-30 | 2010-08-03 | Eastman Kodak Company | Processing images having color and panchromatic pixels |
US7826685B2 (en) | 2006-12-01 | 2010-11-02 | Harris Corporation | Spatial and spectral calibration of a panchromatic, multispectral image pair |
US7835594B2 (en) | 2006-12-01 | 2010-11-16 | Harris Corporation | Structured smoothing for superresolution of multispectral imagery based on registered panchromatic image |
US7889921B2 (en) | 2007-05-23 | 2011-02-15 | Eastman Kodak Company | Noise reduced color image using panchromatic image |
US20090110285A1 (en) | 2007-10-26 | 2009-04-30 | Technion Research And Development Foundation Ltd | Apparatus and method for improving image resolution using fuzzy motion estimation |
US8854420B2 (en) | 2007-12-27 | 2014-10-07 | Google Inc. | High-resolution, variable depth of field image device |
US8319822B2 (en) | 2007-12-27 | 2012-11-27 | Google Inc. | High-resolution, variable depth of field image device |
US8115825B2 (en) | 2008-02-20 | 2012-02-14 | Apple Inc. | Electronic device with two image sensors |
US8320712B2 (en) | 2008-03-07 | 2012-11-27 | Korea Aerospace Research Institute | Satellite image fusion method and system |
US8913145B2 (en) | 2009-03-19 | 2014-12-16 | Digitaloptics Corporation | Dual sensor camera |
US20140118569A1 (en) | 2009-03-19 | 2014-05-01 | Digitaloptics Corporation | Dual sensor camera |
US20120086850A1 (en) | 2009-04-20 | 2012-04-12 | Yeda Research And Development Co., Ltd. | Super resolution from a single signal |
US20140036112A1 (en) | 2009-05-04 | 2014-02-06 | Digitaloptics Corporation | Dual lens digital zoom |
US20120044328A1 (en) | 2010-08-17 | 2012-02-23 | Apple Inc. | Image capture using luminance and chrominance sensors |
US20140192238A1 (en) | 2010-10-24 | 2014-07-10 | Linx Computational Imaging Ltd. | System and Method for Imaging and Image Processing |
US20150036015A1 (en) | 2010-12-14 | 2015-02-05 | Pelican Imaging Corporation | Systems and Methods for Dynamic Refocusing of High Resolution Images Generated Using Images Captured by a Plurality of Imagers |
US20150036014A1 (en) | 2010-12-14 | 2015-02-05 | Pelican Imaging Corporation | Systems and Methods for Synthesizing High Resolution Images Using Images Captured by an Array of Independently Controllable Imagers |
US8581995B2 (en) | 2011-01-25 | 2013-11-12 | Aptina Imaging Corporation | Method and apparatus for parallax correction in fused array imaging systems |
US9020256B2 (en) | 2011-04-22 | 2015-04-28 | Exelis Inc. | System and method for combining color information with spatial information in multispectral images |
US8487996B2 (en) * | 2011-04-25 | 2013-07-16 | Skybox Imaging, Inc. | Systems and methods for overhead imaging and video |
US20130129207A1 (en) | 2011-11-18 | 2013-05-23 | Dehong Liu | Method for Pan-Sharpening Panchromatic and Multispectral Images Using Dictionaries |
US20150124083A1 (en) | 2012-07-04 | 2015-05-07 | Linx Computational Imaging Ltd. | Image Processing in a Multi-Channel Camera |
US8619082B1 (en) | 2012-08-21 | 2013-12-31 | Pelican Imaging Corporation | Systems and methods for parallax detection and correction in images captured using array cameras that contain occlusions using subsets of images to perform depth estimation |
WO2014083489A1 (en) | 2012-11-28 | 2014-06-05 | Corephotonics Ltd. | High-resolution thin multi-aperture imaging systems |
US20150085174A1 (en) | 2012-11-28 | 2015-03-26 | Corephotonics Ltd. | High resolution thin multi-aperture imaging systems |
US20140267759A1 (en) | 2013-03-15 | 2014-09-18 | Digitalglobe, Inc. | Estimation of fine structure within pixels in satellite imagery |
US20140368681A1 (en) | 2013-06-14 | 2014-12-18 | Sony Corporation | Methods and devices for parallax elimination |
US20150109468A1 (en) | 2013-10-18 | 2015-04-23 | The Lightco Inc. | Image capture control methods and apparatus |
US20150109484A1 (en) | 2013-10-18 | 2015-04-23 | The Lightco Inc. | Zoom related methods and apparatus |
US20150109482A1 (en) | 2013-10-18 | 2015-04-23 | The Lightco Inc. | Methods and apparatus for capturing images using optical chains and/or for using captured images |
US20150109524A1 (en) | 2013-10-18 | 2015-04-23 | The Lightco Inc. | Methods and apparatus for implementing and/or using a camera device |
US20150116547A1 (en) | 2013-10-26 | 2015-04-30 | The Lightco Inc. | Methods and apparatus for use with multiple optical chains |
US20150117332A1 (en) | 2013-10-31 | 2015-04-30 | Qinghua Li | Systems, methods, and devices for efficient device-to-device channel contention |
Non-Patent Citations (12)
Title |
---|
Buades, "A Non-Local Algorithm for Image Denoising," Computer Vision and Pattern Recognition, 2:60-65, 2005. |
Duran et al., "Implementation of Nonlocal Pansharpening Image Fusion"; Image Processing On Line; Feb. 2014, pp. 1-15. |
Glasner et al., "Super-Resolution from a Single Image," Computer Vision, IEEE 12th International Conference, 349-356, 2009. |
International Search Report and Written Opinion dated Nov. 23, 2015, in co-pending PCT application No. PCT/US15/45767, 12 pages. |
International Search Report and Written Opinion filed November 23, 2015 in co-pending PCT application No. PCT/US15/45767. |
Michaeli et al., "Nonparametric Blind Super Resolution," Weizmann Institute of Science, Israel. |
Michaeli et al., "Nonparametric Blind Super-Resolution," Weizmann Institute of Science, Israel, Dec. 1-8, 2013, pp. 945-952. |
Protter et al., "Generalizing the Nonlocal-Means to Super-Resolution Reconstruction," IEEE Transactions on Image Processing, vol. 18, No. 1, Jan. 2009. |
Xu, et al., "Deep Convolutional Neural Network for Image Deconvolution," Advances in Neural Information Processing Systems, 2014. |
Yang et al., "Image Super-Resolution as Sparse Representation of Raw Image Patches," IEEE Transactions on Image Processing (TIP), vol. 19, Issue 11, Jun. 23, 2008, pp. 1-8. |
Yang, et al., "Fast Direct Super-Resolution by Simple Functions," ICCV, 2013. |
Yang, et al., Image Super-Resolution via Sparse Representation, Image Processing, IEEE Transactions on Image Processing, vol. 19, No. 11, 2861-2873. |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10482576B2 (en) * | 2018-03-19 | 2019-11-19 | Mitsubishi Electric Research Laboratories, Inc. | Systems and methods for multi-spectral image super-resolution |
Also Published As
Publication number | Publication date |
---|---|
US20160117800A1 (en) | 2016-04-28 |
US9225889B1 (en) | 2015-12-29 |
WO2016028819A1 (en) | 2016-02-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9792668B2 (en) | Photographic image acquistion device and method | |
US10891527B2 (en) | Systems and methods for multi-spectral image fusion using unrolled projected gradient descent and convolutinoal neural network | |
US10366472B2 (en) | Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers | |
US8947501B2 (en) | Scene enhancements in off-center peripheral regions for nonlinear lens geometries | |
JP5726057B2 (en) | Camera and method for acquiring a sequence of frames of a scene as video | |
Chierchia et al. | A nonlocal structure tensor-based approach for multicomponent image recovery problems | |
CN113168670A (en) | Bright spot removal using neural networks | |
CN113170030A (en) | Correction of photographic underexposure using neural networks | |
US20140063314A1 (en) | System And Method Of Video Compressive Sensing For Spatial-Multiplexing Cameras | |
Joze et al. | Imagepairs: Realistic super resolution dataset via beam splitter camera rig | |
US11244426B2 (en) | Method for image super resolution imitating optical zoom implemented on a resource-constrained mobile device, and a mobile device implementing the same | |
Chakrabarti et al. | Rethinking color cameras | |
KR20190036442A (en) | Hyperspectral Imaging Reconstruction Method Using Artificial Intelligence and Apparatus Therefor | |
Lv et al. | An integrated enhancement solution for 24-hour colorful imaging | |
KR20190036459A (en) | Hyperspectral Imaging Reconstruction Method Using Artificial Intelligence and Apparatus Therefor | |
Simpkins et al. | An introduction to super-resolution imaging | |
Nguyen et al. | Self-supervised super-resolution for multi-exposure push-frame satellites | |
Ito et al. | Compressive epsilon photography for post-capture control in digital imaging | |
CN112750092A (en) | Training data acquisition method, image quality enhancement model and method and electronic equipment | |
Jerez et al. | Single pixel spectral image fusion with side information from a grayscale sensor | |
EP4167134A1 (en) | System and method for maximizing inference accuracy using recaptured datasets | |
Dewil et al. | Self-supervision versus synthetic datasets: which is the lesser evil in the context of video denoising? | |
JP2017103756A (en) | Image data processing apparatus and method | |
Huang et al. | High-fidelity hyperspectral snapshot of physical world: System architecture, dataset and model | |
Prette et al. | Towards unsupervised multi-temporal satellite image super-resolution |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ENTROPIX, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KORKIN, MIKHAIL;REEL/FRAME:037370/0890 Effective date: 20150817 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: KHALILOV, SAMIR, AZERBAIJAN Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: PAO, CHWEN-LIANG, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: TOPVIEW OPTRONICS CORP., TAIWAN Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: LEE, Y. S., KOREA, REPUBLIC OF Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: TONG, VINCENT, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: HUANG, ERIC, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: MITAINE TRAVEL LTD., VIRGIN ISLANDS, BRITISH Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: S. AND A. ITOH REVOCABLE TRUST (DATED NOVEMBER 17TH, 2009, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: SMITH, TEFFT W., SOUTH CAROLINA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: ZHANG OPTICS, INC., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: FELLOW, INC., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: VENTECH LLC, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: VIDEO TECHNICA CO., LTD., JAPAN Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: CHEGO LLC, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: R3D ALT INC., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: WHITE, DENNIS, AUSTRALIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: SMITH, TEFFT W, II, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: SAMARYANOV, BORIS, ISRAEL Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: WHEELER, NATHAN H., OREGON Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: FOUNDATION CAPITAL VIII, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: SHARFMAN, ALEXANDER, CANADA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 Owner name: MIRAN N. AHMAD AND HUMA J. AHMAD, CO-TRUSTEES OF THE AHMAD FAMILY TRUST DATED 01/08/16, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:ENTROPIX, INC.;REEL/FRAME:053540/0190 Effective date: 20200803 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: MICROENTITY |
|
FEPP | Fee payment procedure |
Free format text: SURCHARGE FOR LATE PAYMENT, MICRO ENTITY (ORIGINAL EVENT CODE: M3554); ENTITY STATUS OF PATENT OWNER: MICROENTITY |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, MICRO ENTITY (ORIGINAL EVENT CODE: M3551); ENTITY STATUS OF PATENT OWNER: MICROENTITY Year of fee payment: 4 |