US20010040632A1 - Multiple sampling via a time-indexed method to achieve wide dynamic ranges - Google Patents
Multiple sampling via a time-indexed method to achieve wide dynamic ranges Download PDFInfo
- Publication number
- US20010040632A1 US20010040632A1 US09/907,099 US90709901A US2001040632A1 US 20010040632 A1 US20010040632 A1 US 20010040632A1 US 90709901 A US90709901 A US 90709901A US 2001040632 A1 US2001040632 A1 US 2001040632A1
- Authority
- US
- United States
- Prior art keywords
- image sensor
- digital
- memory space
- pixel elements
- frames
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
- H04N1/00127—Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
- H04N1/00281—Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture with a telecommunication apparatus, e.g. a switched network of teleprinters for the distribution of text-based information, a selective call terminal
- H04N1/00307—Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture with a telecommunication apparatus, e.g. a switched network of teleprinters for the distribution of text-based information, a selective call terminal with a mobile telephone apparatus
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/70—SSIS architectures; Circuits associated therewith
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
- H04N7/142—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N2201/00—Indexing scheme relating to scanning, transmission or reproduction of documents or the like, and to details thereof
- H04N2201/0008—Connection or combination of a still picture apparatus with another apparatus
- H04N2201/0065—Converting image data to a format usable by the connected apparatus or vice versa
- H04N2201/0068—Converting from still picture data
Abstract
An architecture for a digital pixel sensor is disclosed in which the dynamic range of the sensor is increased by taking samples of a subject to be recorded, where each sample is taken over an interval of a different duration than the other samples. In the preferred embodiment of the invention, an array of pixel elements is fabricated in an integrated circuit that also includes a memory space for storing selectively digital signals of the samples from the digital elements.
Description
- This application is a continuation of co-pending U.S. application Ser. No.: 09/567,786, filed on May 9, 2000.
- The invention relates to image sensor systems. More particularly, the invention relates to an image sensor architecture and associated method for facilitating image multiple sampling using a time-indexed approach to achieve a wide dynamic range.
- Digital photography is one of the most exciting technologies to have emerged during the twentieth century. With the appropriate hardware and software (and a little knowledge), anyone can put the principles of digital photography to work. Digital cameras, for example, are on the cutting edge of digital photography. Recent product introductions, technological advancements, and price cuts, along with the emergence of email and the World Wide Web, have helped make the digital cameras one of the hottest new category of consumer electronics products.
- Digital cameras, however, do not work in the same way as traditional film cameras do. In fact, they are more closely related to computer scanners, copiers, or fax machines. Most digital cameras use an image sensor or photosensitive device, such as charged-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) to sense a scene. The photosensitive device reacts to light reflected from the scene and can translate the strength of that reaction into a numeric equivalent. By passing light through red, green, and blue filters, for example, the reaction can be gauged for each separate color spectrum. When the readings are combined and evaluated via software, the camera can determine the specific color of each element of the picture. Because the image is actually a collection of numeric data, it can easily be downloaded into a computer and manipulated for more artistic effects.
- Nevertheless, there are many cases in which digital cameras simply can not be used because of the limited resolution of the image sensors in today's digital cameras. Film-based photographs have immeasurably higher resolution than digital cameras. While traditional film-based technology typically has a resolution of tens millions of pixels, the image sensors in the digital cameras that could be produced at a price that is acceptable to consumers is slightly more than a million pixels today.
- Dynamic range is another critical figure of merit for image sensors used in digital cameras. The dynamic range of an image sensor is often not wide enough to capture scenes with both highlights and dark shadows. This is especially the case for CMOS sensors which, in general, have lower dynamic range than CCDs.
- Previously suggested solutions for widening the dynamic range of these devices can be divided into three categories:
- Compressing the response curve;
- Multiple sampling; and
- Control over integration time.
- The response curve is compressed by using a sensor that has a logarithmic response. There are two ways of doing this:
- The first approach is to use a CMOS sensor that operates in an instantaneous current read out mode. In this mode, the photocurrent generated by a photodetector is fed into a device that has a logarithmic response, for example a diode connected MOS transistor, to compress the sensor transfer curve. Although this scheme can achieve very wide dynamic range, the resulting image quality is generally poor due to a low signal-to-noise ratio (SNR).
- The second approach to compress the response curve uses a technique referred to as well capacity adjusting. Here, the dynamic range is enhanced by increasing well capacity one or more times during exposure time. During integration well capacity is monotonically increased to its maximum value. The excess photo-generated charge is drained via an overflow gate. This scheme, however, suffers from large fixed pattern noise and degradation in the SNR.
- In multiple sampling, a scene is imaged several times at different exposure times and the data are combined to construct a high dynamic range image. For this approach to work at reasonable capture times, the read out process must be performed at speeds much higher than normal active pixel sensor (APS) speeds. The multiple sampling scheme effectively achieves a wide dynamic range. In reality, much data may need to be read out, which can be particularly burdensome for many types of image sensors.
- Controlling integration time is the third method that has some promising aspects in comparison with others. In essence, the exposure time of each pixel is individually adjusted so that they do not get saturated at the end of each integration period. There are many ways of achieving this.
- One way is to place a set-reset flip-flop and an AND gate at each pixel to control the integration start time to achieve local exposure control. However, this approach suffers the following limitations:
- Each pixel is large due to the inclusion of the flip-flop and the AND gate.
- A large ‘timestamp’ memory is needed to store the exposure time of all pixels. The exposure time of each pixel can be determined by trying out various exposure times. When capturing a moving scene, the exposure times change so the ‘timestamp’ memory must be updated, which not only is burdensome but also causes image lag.
- Moreover, in addition to the column and row decoders used for pixel read out, another column and row decoders are needed to control the flip-flops.
- A second way is known for an individual pixel reset (IPR) to achieve local exposure control, namely a second reset transistor is added to the standard three-transistor APS design so that the integration start time of each pixel can be controlled externally. The second way keeps the pixel size small but requires a large external memory to store the exposure time for all of the pixels, and further requires memory refreshing and additional column and row decoders. Moreover, multiple reset pulses might need to be applied to each pixel throughout the reset period. The time control for resetting pulses could be quite complicated.
- There is therefore a great need for a wide dynamic range image sensor that overcomes some of the above shortcomings and, in particular, outputs image data having a wide dynamic range. Further, the sensor should not require an external timestamp memory and control logic to update the exposure times.
- An architecture for a digital pixel sensor is disclosed in which the dynamic range of the sensor is increased by taking samples of a subject to be recorded, where each sample is taken over an interval of a different duration than the other samples. The use of different recording intervals allows integration of multiple photodetector signals relative to a threshold value and thus expands the dynamic range of the photodetector without saturating the picture elements in the image.
- In the preferred embodiment of the invention, an array of pixel elements is fabricated in an integrated circuit. Each of the pixel elements outputs a digital signal and comprises a photodetector and an analog to digital converter. The photodetector is integrated with the analog to digital converter.
- An array of threshold memory cells, each corresponding to one of the pixel elements, is also provided. The threshold memory assures that a picture element which corresponds to a particular threshold memory cell does not exceed a threshold value and, therefore, does not provide a saturated signal for the picture element. Alternatively, the threshold memory assures that a signal in a picture element which corresponds to a particular threshold memory cell is read out into a data memory cell only when the signal exceeds a value in the particular threshold memory cell. In essence, the threshold memory avoids the readout of unnecessary values to the memory cells (discussed below).
- An array of time memory cells, each corresponding to one of the pixel elements, establishes a different exposure time for each of the pixel elements for each of multiple samples. An array of memory elements, each coupled to one of the pixel elements, is also provided. The memory elements are also fabricated in the integrated circuit. The memory elements only receive a value from a corresponding one of the pixel elements when the content in a corresponding one of the threshold memory cells permits. In this way, multiple samples may be collected for a subject to be recorded to thereby extend the dynamic range of a photodetector.
- Integration of the photodetector and the memory that implements this mechanism into the same integrated circuit avoids the latency that would be experienced if an external memory was used.
- FIG. 1A is a block diagram that shows a CMOS image sensor or photosensitive chip in which the invention may be practiced;
- FIG. 1B is a block diagram which shows a photodiode modeled as a current source and a capacitor;
- FIG. 2 is a block diagram which shows the architecture of a digital pixel sensor, as described in U.S. Pat. No. 5,461,425;
- FIG. 3 is a block diagram which shows an image sensor that includes a threshold memory, a time index memory, and a separate data memory, where each of the memories and the digital pixel sensor are integrated into the same sensor according to the invention;
- FIG. 4 is a graph which shows an example of multiple exposures;
- FIG. 5A is a block diagram which shows a pair of exemplary threshold memory cells, time index memory cells, and corresponding data memory cells according to the invention; and
- FIGS. 5B and 5C are graphs which show, respectively, two corresponding time integration processes according to the invention.
- In the following detailed description of the invention, some specific details are set forth to provide a thorough understanding of the presently preferred embodiment of the invention. However, it should be apparent to those skilled in the art that the invention may be practiced in embodiments that do not use the specific details set forth herein. Well known methods, procedures, components, and circuitry have not been described in detail to avoid unnecessarily obscuring the invention.
- In the following discussion, in references to the drawings like numerals refer to like parts throughout the several views.
- FIG. 1A shows an image sensor or
photosensitive chip 100 in which the invention may be practiced. Theimage sensor 100 may be used in an image capturing device (e.g. a digital camera) for either stationary or video photography, and produces digital image data. Thephotosensitive chip 100, which is typically fabricated on a substrate such as CMOS, comprises a plurality of photodetectors that are arranged in an array. For color applications, a mosaic of selectively transmissive filters is superimposed in registration with each of the photodetectors so that a first, second, and third selective group of photodetectors are made to sense three different color ranges, for example, the red, green, and blue range of the visible spectrum, respectively. The number of the photodetectors in thephotosensitive chip 100 typically determines the resolution of digital images resulting therefrom. The horizontal resolution is a function of the number of photodetectors in arow 102, and the vertical resolution is a function of the number of photodetectors in acolumn 104. - Each of the photodetectors comprises a photosensor that produces an electronic signal when it is exposed to light. Generally, the photosensor is a photodiode or a photogate in a CMOS sensor. FIG. 1B shows a
photodiode 120 that is modeled as acurrent source 122 and acapacitor 124. When a reset signal is applied at aReset terminal 130, thecapacitor 124 is fully charged by and nearly to Vcc through thetransistor 128, at which point thephotodiode 120 is ready for light integration. - As soon as the reset signal is dropped (i.e. the voltage level is changed), light integration starts. As more and more incident photons from light126 strike the surface of the
photodiode 120, the current ofcurrent source 122 increases. Thecapacitor 124 starts to discharge through thecurrent source 122. Typically, the photodiode collects more photons for higher photon intensities and, as a result, the resistance of theresistor 122 decreases. Consequently, a faster discharge signal Vout is produced. In other words, the signal from Vout is proportional to the incident photons which strike thephotodiode 120. This signal is alternatively referred to herein as an electronic signal or pixel charge signal. Optionally, acircuit 130 may be employed to enhance the electronic signal Vout to a desired level so that the output, i.e. the pixel charge signal, is effectively coupled to following circuitry. - Operation of an image sensor comprises two processes: the light integration process, as described above; and the read out process. Each of these two processes is sustained for a controlled time interval. In the light integration process, each photodetector is initiated to accumulate incident photons of the light and the accumulation is reflected as a pixel charge signal. After the light integration process, the photodetectors start the read out process during which the pixel charge signal in each photodetector is read out via read out circuitry to a data bus or video bus. The interval during which the light integration process proceeds is referred to as exposure control or electronic shuttering, and it controls how much charge is accumulated by each of the photodiodes.
- FIG. 2 duplicates FIG. 1 of U.S. Pat. No. 5,461,425 and shows that each photodetector14 includes an A/D converter in addition to a photosensor. Each of the photodetectors is referred to as a sensor pixel or a sensor element or digital pixel. This is done to indicate that the photodetector herein includes an analog-to-digital conversion circuit, as opposed to a photodetector which is commonly seen in a conventional image sensor, and which includes a photosensor and produces an analog signal. Further, the pixel element herein is different from a conventional image sensor because it outputs digital signals that can be read out at a much higher speed than an analog signal can be read out in a conventional image sensor. Hence, the resultant image sensor is considered a digital pixel sensor (DPS). The preferred embodiment of the invention is based on such architecture in which a sensor element includes a photosensor and an analog-to-digital conversion circuit.
- The image sensor of FIG. 2 is formed on a single
integrated circuit chip 10. Theimage sensor core 12 comprises a two-dimensional array of light detecting elements, each connected to a dedicated AND converter which outputs a stream of bits representative of the analog output of the light detecting element. The combination of a light detecting element and A/D converter constitutes a single pixel element 14. Each pixel element 14 includes identical circuitry. Digital filters 16 onchip 10 are connected to receive the digital streams from each pixel element 14 and convert each digital stream to an eight-bit byte representative of one of 256 levels of light intensity detected by the respective pixel element 14. - In operation, an image is focused on the
image sensor core 12 such that a different portion of the focused image impinges on each pixel element 14. Each light detecting element comprises a phototransistor whose conductivity is related to the intensity of light impinging upon the base of the phototransistor. The analog current through the phototransistor thus corresponds to the intensity of light impinging upon the phototransistor. The analog signals from all phototransistors in the core 12 are simultaneously converted into serial bit streams output from dedicated A/D converters clocked using a common clock driver 18. The serial bit streams, over a period of time, i.e. over a frame period, can then be processed by filters 16 (on-chip or off-chip) to derive a signal representative of the intensity of light impinging on the phototransistor. - After each clock cycle, one bit is latched at an output of each A/D converter within each pixel element14. To now transfer each bit generated by the pixel elements 14 to the filters 16 after each clock cycle, each of the rows of pixel elements 14 are addressed in sequence, using row decoder 20, until all rows of pixel elements 14 have been addressed. Upon addressing each row, the one-bit output of each pixel element 14 in the addressed row is coupled to a corresponding bit line 22. The filters 16 process the bit stream from each pixel element 14 to generate an eight-bit value per pixel element 14 corresponding to the average intensity of light impinging on the respective pixel element 14 for that frame period. These eight-bit values may then be output from the
chip 10, using a suitable multiplexer or shift register, and temporarily stored in a bit-mapped memory 24. The memory 24 may then act as a frame buffer, where the light intensity values in memory 24 are sequentially addressed for controlling the light output of corresponding pixels in a monitor. - In a particular embodiment of FIG. 2, assume that sixty-four separate filters16 are used for converting the bit streams output on sixty-four bit lines 22 to eight-bit values. (A multiplexer at the output of the core 12 may reduce the number of required filters to, for example, sixteen.) The preferred interaction of filters 16 with memory 24 is as follows. Immediately after a row of pixel elements 14 has been addressed, a control circuit 26, using the address generated by row decoder 20, fetches a previous (or interim) eight-bit value stored in memory 24 for each pixel element 14 in the addressed row and loads this previous value into the proper one of the 64 filters 16 about to receive a new bit from that pixel element 14. Conventional memory addressing techniques and circuitry may be used for this process. The single bit output of the respective A/D converters in the addressed pixel elements 14 is then applied to a respective one of the sixty-four filters 16 containing the previous eight-bit value for that pixel element 14. Each filter 16 then updates the previous eight-bit value with the new single bit of information to generate a new interim value. The now updated eight-bit value generated by each filter 16 is then transferred back into memory 24, under control of the control circuit 26.
- Referring to FIG. 3, there is shown an
image sensor 300 based on the digital pixel sensor according to one embodiment of the invention. Thedigital pixel sensor 302 may be implemented according to U.S. Pat. No. 5,461,425 or U.S. Pat. No. 5,801,657, and outputs digital signals representing one or more images of a scene. A sense amplifier and latches 304 are coupled to thedigital pixel sensor 302 to facilitate read out of the digital signals from thedigital pixel sensor 302. Unlike the prior art, animage sensor 300 in accordance with the invention also includes a memory 304 (referred to herein as a threshold memory) for storing threshold values, a memory 308 (referred to herein as a time index memory) for storing time index values, and a digital ordata memory 310 that is large enough to accommodate a frame of image data fromsensor 302. - According to one embodiment of the invention, it is assumed that the
sensor 302 is of N by M pixels and has k-bits. Thus, the size of thethreshold memory 306 is of N by M bits, and the size of thetime index memory 308 is of N by M by m bits, where m is the time resolution. The presently preferred pixel resolution ofsensor 302 is 1000 by 1000 in 10 bits. Thus, thethreshold memory 306 is a one-megabit memory, thetime index memory 308 is a two-megabit memory when the time index is set to be T, 2T, 4T and 8T (i.e. two-bit resolution), and thedigital memory 306 preferably has a size of at least 1.2 megabytes. - As a result of the above memory configuration, each of the pixel elements in the
sensor 302 can be stamped by each of the memory cells in thethreshold memory 306, thetime index memory 308, and thedata memory 310. - In operation, at each of the time indexes, for example, T, 2T, 4T and 8T, the
sensor 302 is exposed to a target multiple (e.g. four) times, resulting in four images at four different exposure times. FIG. 4 shows an example of the multiple exposures. As shown in FIG. 4,frame 1 is created after time T,frame 2 is created aftertime 2T,frame 3 is created after time 4T, andframe 4 is created aftertime 8T. One of the advantages of having multiple images of the same target is the ability to expand the dynamic range of the image thus captured. Because of the relative short exposure time,frame 1 typically captures information that is related to high illumination areas in the target. Likewise, because of the relatively long exposure time,frame 4 typically captures information that is related to low illumination areas in the target.Frame 2 andframe 3 thus capture information that is related to gradually increased illumination areas in the target. As a result, the combination of the multiple images provides a very wide dynamic range. - Generally, each of the frames is read out to a memory so that subsequent processing to combine the frames is possible. The architecture illustrated on FIG. 2 shows that the image data are read out to a memory24 through a plurality of lines (e.g. pins of sensor 10). Given the exposure times, the generated image data must be read out fast enough so that it does not affect the following frame. As is well known, the number of lines is limited to a practical packaging solution and often far less than what is needed to accommodate the required speed. Therefore, the limited number of the lines becomes a bottleneck for data transmission from the sensors 14 to the memory 24.
- One of features in the invention is to place an on-chip memory in the image sensor, shown as the
data memory 310 in FIG. 3. Thus, there is no bottleneck for data transmission from thesensors 302 to thememory 310. In operation, after one exposure time, a frame of data can be immediately read out to thememory 310. - According to one aspect of the invention, after the first frame of data is read out into the
memory 310, the second frame of data is selectively read out into thememory 310 to improve, update, or enhance the pixel values contained therein. Selection is controlled by the contents of thecorresponding threshold memory 306. - FIG. 5A shows a pair of exemplary
threshold memory cells 502 and 504, exemplary timeindex memory cells data memory cells resultant signal 514 exceeds a predefined threshold V1. Thus, a flag, such as a binary value “1” which represents that the threshold value V1 has been exceeded, is stored in thecell 502, the exposure time T is stored in thecell 506, and the resultant signal or a representation thereof (e.g. thevalue 240 in eight-bit precision) is stored in thecell 510. The value of threshold V1 is usually so determined that further exposure to the photosensor that producesresultant signal 514 can cause the photosensor to become saturated. Therefore, in view of the flag in thecell 502, there is no need to enhance the value stored in thecell 510 after the first exposure time T. In reality, further update of thecell 510 could cause the loss of the data therein as it is now clear that the next value would be a saturated value. - It is now assumed that a resultant signal516 produced by an adjacent photodetector is below the threshold V1, as shown in FIG. 5C. Therefore, the cell 504 does not store the flag “1,” assuming that the cell 504 was reset to “0” when the operation starts. This permits the corresponding
data cell 512 to be updated or enhanced with new value that results from a next exposure. It should be noted that the exact contents to be stored in thecells - One of the key features of the invention is to provide a stamp on each of the photodetectors in the
sensor 302 or each of the data cells in thememory 310 to prevent any saturated values from overwriting useful information in thememory 310. The contents in the time index memory are used individually so that the final image can be regenerated correctly. This allows the contents in thememory 310 to be updated properly after additional exposure times, or allows the frames of data to be combined properly. - The advantages and benefits provided by the
image sensor 300 are numerous. One of the advantages is the elimination of the data transmission bottleneck presented in the architecture of U.S. Pat. No. 5,461,425. Secondly, the integration of the on-chip memory 310 with thedigital pixel sensor 302 does not affect the performance of thedigital pixel sensor 302 but, rather, improves the overall performance ofimage sensor 300 significantly. Such improvements include that of matched bandwidth when reading out the digital signals from thedigital pixel sensor 302. Thirdly, a threshold memory is used to prevent the read out of unnecessary values to the data memory. In addition, the time index memory is used to provide supporting (weighted) information for properly combining the frames of data to produce a final digital image having expanded dynamic range. - According to one embodiment of the invention, the architecture200 is preferably implemented in a CMOS image sensor. The resultant image sensor may be advantageously employed in digital cameras that can provide superior or comparable image qualities as opposed to film image qualities.
- Although the invention is described herein with reference to the preferred embodiment, one skilled in the art will readily appreciate that other applications may be substituted for those set forth herein without departing from the spirit and scope of the present invention. Accordingly, the invention should only be limited by the claims included below.
Claims (14)
1. An image sensor comprising:
a two dimensional array of digital pixel elements producing frames of digital signals when the image sensor is operated to sense a scene, the frames representing sequentially increased exposures to the scene within a snapshot of the scene; and
a data memory space for storing a time index and a threshold value for each of the digital pixel elements, the data memory space further for storing at least one of the digital signals of one of the frames when the corresponding threshold detects that the one of the digital signals is not saturated.
2. The image sensor of , wherein there are at least two of such frames in the snapshot.
claim 1
3. The image sensor of , wherein both the two dimensional array of the digital pixel elements and the data memory space are integrated in a same integrated circuit.
claim 1
4. The image sensor of , wherein each of the digital pixel elements includes a photodetector producing an analog signal when the image sensor is operated to sense a scene.
claim 1
5. The image sensor of , wherein each of the digital pixel elements is operatively connected to an analog-to-digital conversion circuit that digitizes the analog signal to one of the digital signals.
claim 4
6. An image sensor comprising:
a two dimensional array of digital pixel elements producing a 1st, a 2nd . . . a Nth frame of digital signals when the image sensor is operated to sense a scene, the 1st, 2nd . . . and Nth frames respectively representing an exposure at a 1st sample time, a 2nd sample time, . . . and a Nth sample time, wherein the 1st sample time, the 2nd sample time, . . . and the Nth sample time are within a snapshot of the scene;
a data memory space coupled to the two dimensional array of digital pixel elements;
a readout circuit to readout each of the 1t, 2nd . . . Nth frames respectively to the data memory space; and
wherein the data memory space includes a threshold for each of the digital pixel elements, when one of the digital signals in one of the 1st, 2nd . . . or Nth frames is acceptable with respect to the threshold, the one of the digital signals is read into in the data memory space in association with a time index to indicate which one of the 1st, 2nd . . . or Nth frames.
7. The image sensor of , wherein the threshold is updated when the one of the digital signals is read into in the data memory space.
claim 6
8. The image sensor of , wherein the time index is updated when one of the digital signals from another one of the 1st, 2nd . . . or Nth frames is read into in the data memory space.
claim 7
9. The image sensor of , wherein each of the digital pixel elements includes a photodetector that is sensitive to light and produces an analog signal representing the light.
claim 6
10. The image sensor of , wherein the photodetector is electronically coupled to an analog-to-digital conversion circuit, and the analog signal from the photodetector is locally converted to one of the digital signals via the analog-to-digital conversion circuit.
claim 9
11. The image sensor of , wherein each of the 2nd . . . and Nth frames is accumulated in time from a corresponding preceding frame.
claim 6
12. The image sensor of , wherein the two dimensional array of digital pixel elements and the data memory space are integrated on a semiconductor core.
claim 6
13. The image sensor of , wherein the semiconductor core is based on complementary metal-oxide semiconductor (CMOS).
claim 12
14. The image sensor of , wherein N is an integer greater than 2.
claim 6
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/907,099 US20010040632A1 (en) | 2000-05-09 | 2001-07-16 | Multiple sampling via a time-indexed method to achieve wide dynamic ranges |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/567,786 US6975355B1 (en) | 2000-02-22 | 2000-05-09 | Multiple sampling via a time-indexed method to achieve wide dynamic ranges |
US09/907,099 US20010040632A1 (en) | 2000-05-09 | 2001-07-16 | Multiple sampling via a time-indexed method to achieve wide dynamic ranges |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/567,786 Continuation US6975355B1 (en) | 1999-07-30 | 2000-05-09 | Multiple sampling via a time-indexed method to achieve wide dynamic ranges |
Publications (1)
Publication Number | Publication Date |
---|---|
US20010040632A1 true US20010040632A1 (en) | 2001-11-15 |
Family
ID=24268636
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/907,099 Abandoned US20010040632A1 (en) | 2000-05-09 | 2001-07-16 | Multiple sampling via a time-indexed method to achieve wide dynamic ranges |
US09/909,528 Expired - Lifetime US6970195B1 (en) | 2000-05-09 | 2001-07-20 | Digital image sensor with improved color reproduction |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/909,528 Expired - Lifetime US6970195B1 (en) | 2000-05-09 | 2001-07-20 | Digital image sensor with improved color reproduction |
Country Status (1)
Country | Link |
---|---|
US (2) | US20010040632A1 (en) |
Cited By (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020085109A1 (en) * | 1999-11-08 | 2002-07-04 | Casio Computer Co., Ltd. | Photosensor system and drive control method thereof |
US20020140842A1 (en) * | 2001-03-30 | 2002-10-03 | Olding Benjamin P. | Method and apparatus for storing image information for multiple sampling operations in a digital pixel sensor |
EP1339227A2 (en) * | 2002-02-25 | 2003-08-27 | Kabushiki Kaisha Toshiba | Image pickup apparatus |
US20030206236A1 (en) * | 2002-05-06 | 2003-11-06 | Agfa Corporation | CMOS digital image sensor system and method |
WO2003103279A2 (en) * | 2002-05-29 | 2003-12-11 | The Regents Of The University Of Michigan | Method, processor and computed tomography (ct) machine for generating images utilizing high and low sensitivity data collected from a flat panel detector having an extended dynamic range |
US6765610B1 (en) * | 1999-11-10 | 2004-07-20 | Casio Computer Co., Ltd. | Photosensor system and drive control method thereof |
US6879344B1 (en) * | 1999-11-08 | 2005-04-12 | Casio Computer Co., Ltd. | Photosensor system and drive control method thereof |
US20060108511A1 (en) * | 2004-11-24 | 2006-05-25 | Eastman Kodak Company | Light detection circuit |
US20060170662A1 (en) * | 2004-12-07 | 2006-08-03 | Haruhisa Kurane | Image pickup device |
US20070103569A1 (en) * | 2003-06-02 | 2007-05-10 | National University Corporation Shizuoka University | Wide dynamic range image sensor |
US20070153117A1 (en) * | 2005-12-30 | 2007-07-05 | Yen-Yu Lin | Apparatus and method for adjusting display-related setting of an electronic device |
US20070153098A1 (en) * | 2005-12-30 | 2007-07-05 | Mediatek Incorporation | Luminance compensation apparatus and method |
US20080094481A1 (en) * | 2006-10-19 | 2008-04-24 | Barinder Singh Rai | Intelligent Multiple Exposure |
EP1942660A1 (en) * | 2007-01-02 | 2008-07-09 | STMicroelectronics (Research & Development) Limited | Improvements in image sensor noise reduction |
US20100321525A1 (en) * | 2009-06-17 | 2010-12-23 | Canon Kabushiki Kaisha | Image capturing apparatus, image capturing method, and storage medium |
US20130155282A1 (en) * | 2011-12-15 | 2013-06-20 | STMicroelectronics (R&D) Ltd. | Image sensor readout method and apparatus |
US10218913B2 (en) | 2017-04-18 | 2019-02-26 | Qualcomm Incorporated | HDR/WDR image time stamps for sensor fusion |
US11303835B2 (en) | 2019-11-15 | 2022-04-12 | Samsung Electronics Co., Ltd. | Pixel array and image sensor including the same |
Families Citing this family (34)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2002030109A1 (en) * | 2000-10-03 | 2002-04-11 | Sony Corporation | Imaging device an imaging method |
JP2004048561A (en) * | 2002-07-15 | 2004-02-12 | Fuji Photo Film Co Ltd | Imaging unit and photometric instrument |
US7430011B2 (en) * | 2003-01-22 | 2008-09-30 | Omnivision Technologies, Inc. | Image sensor having dual automatic exposure control |
TWI429066B (en) | 2005-06-02 | 2014-03-01 | Sony Corp | Semiconductor image sensor module and manufacturing method thereof |
US8179296B2 (en) | 2005-09-30 | 2012-05-15 | The Massachusetts Institute Of Technology | Digital readout method and apparatus |
US7496291B2 (en) * | 2006-03-21 | 2009-02-24 | Hewlett-Packard Development Company, L.P. | Method and apparatus for interleaved image captures |
US20070285547A1 (en) * | 2006-05-30 | 2007-12-13 | Milligan Edward S | CMOS image sensor array optimization for both bright and low light conditions |
US20100226495A1 (en) | 2007-10-29 | 2010-09-09 | Michael Kelly | Digital readout method and apparatus |
US20080165257A1 (en) * | 2007-01-05 | 2008-07-10 | Micron Technology, Inc. | Configurable pixel array system and method |
US20080239119A1 (en) * | 2007-03-31 | 2008-10-02 | Olivier Patrick Boireau | Image capturing system and technique |
GB0709026D0 (en) * | 2007-05-10 | 2007-06-20 | Isis Innovation | High speed imaging with slow scan cameras using pixel level dynami shuttering |
US7812869B2 (en) * | 2007-05-11 | 2010-10-12 | Aptina Imaging Corporation | Configurable pixel array system and method |
US8542315B2 (en) * | 2007-11-27 | 2013-09-24 | Broadcom Corporation | Method and apparatus for expanded dynamic range imaging |
KR20100054540A (en) * | 2008-11-14 | 2010-05-25 | 삼성전자주식회사 | Pixel circuit, photo-electricity converter, and image sensing system thereof |
EP2706904B1 (en) | 2011-05-12 | 2021-01-06 | DePuy Synthes Products, Inc. | Improved image sensor for endoscopic use |
US8605853B2 (en) | 2011-07-01 | 2013-12-10 | Massachusetts Institute Of Technology | Methods and apparatus for in-pixel filtering in focal plane arrays |
US9509917B2 (en) | 2012-07-26 | 2016-11-29 | DePuy Synthes Products, Inc. | Wide dynamic range using monochromatic sensor |
CN104486986B (en) | 2012-07-26 | 2018-06-01 | 德普伊辛迪斯制品公司 | Continuous videos in light deficiency environment |
EP2877080B1 (en) | 2012-07-26 | 2020-07-22 | DePuy Synthes Products, Inc. | Ycbcr pulsed illumination scheme in a light deficient environment |
CA2878512A1 (en) | 2012-07-26 | 2014-01-30 | Olive Medical Corporation | Camera system with minimal area monolithic cmos image sensor |
US9777913B2 (en) | 2013-03-15 | 2017-10-03 | DePuy Synthes Products, Inc. | Controlling the integral light energy of a laser pulse |
US10517469B2 (en) | 2013-03-15 | 2019-12-31 | DePuy Synthes Products, Inc. | Image sensor synchronization without input clock and data transmission clock |
US10251530B2 (en) | 2013-03-15 | 2019-04-09 | DePuy Synthes Products, Inc. | Scope sensing in a light controlled environment |
JP2016519591A (en) | 2013-03-15 | 2016-07-07 | オリーブ・メディカル・コーポレイションOlive Medical Corporation | Super-resolution and color motion artifact correction in pulse color imaging system |
EP2967286B1 (en) | 2013-03-15 | 2021-06-23 | DePuy Synthes Products, Inc. | Minimize image sensor i/o and conductor counts in endoscope applications |
KR102105409B1 (en) * | 2013-12-09 | 2020-05-29 | 삼성전자주식회사 | Image sensor including heterogeneous analog to digital converters |
JP6573960B2 (en) | 2014-03-21 | 2019-09-11 | デピュイ・シンセス・プロダクツ・インコーポレイテッド | Card edge connector for imaging sensors |
JP6355527B2 (en) * | 2014-10-31 | 2018-07-11 | 富士フイルム株式会社 | Endoscope system and operating method thereof |
US11463636B2 (en) | 2018-06-27 | 2022-10-04 | Facebook Technologies, Llc | Pixel sensor having multiple photodiodes |
US11956413B2 (en) | 2018-08-27 | 2024-04-09 | Meta Platforms Technologies, Llc | Pixel sensor having multiple photodiodes and shared comparator |
DE102018216199A1 (en) * | 2018-09-24 | 2020-03-26 | Robert Bosch Gmbh | Image sensor element for outputting an image signal and method for producing an image sensor element for outputting an image signal |
US11218660B1 (en) | 2019-03-26 | 2022-01-04 | Facebook Technologies, Llc | Pixel sensor having shared readout structure |
CN110855957B (en) * | 2019-11-01 | 2022-05-10 | RealMe重庆移动通信有限公司 | Image processing method and device, storage medium and electronic equipment |
US11910114B2 (en) | 2020-07-17 | 2024-02-20 | Meta Platforms Technologies, Llc | Multi-mode image sensor |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5309243A (en) * | 1992-06-10 | 1994-05-03 | Eastman Kodak Company | Method and apparatus for extending the dynamic range of an electronic imaging system |
US5461425A (en) * | 1994-02-15 | 1995-10-24 | Stanford University | CMOS image sensor with pixel level A/D conversion |
US5909026A (en) * | 1996-11-12 | 1999-06-01 | California Institute Of Technology | Integrated sensor with frame memory and programmable resolution for light adaptive imaging |
US6229133B1 (en) * | 1997-10-27 | 2001-05-08 | Texas Instruments Incorporated | Image sensing device with delayed phase frequency modulation |
US6330030B1 (en) * | 1998-01-05 | 2001-12-11 | Intel Corporation | Digital image sensor with low device count per pixel analog-to-digital conversion |
US6614560B1 (en) * | 1997-07-15 | 2003-09-02 | Silverbrook Research Pty Ltd | Integrated camera circuit including image sensor, image processing, and printer drive circuits |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5187569A (en) * | 1989-03-03 | 1993-02-16 | Asahi Kogaku Kogyo Kabushiki Kaisha | Solid state imaging device and electronic still camera using same |
JP2830482B2 (en) * | 1991-01-11 | 1998-12-02 | 株式会社ニコン | High dynamic range imaging device |
EP0509820B1 (en) * | 1991-04-19 | 1998-01-07 | Canon Kabushiki Kaisha | Image pickup apparatus |
US5943514A (en) * | 1996-07-19 | 1999-08-24 | Asahi Kogaku Kogyo Kabushiki Kaisha | Focusing system |
US5734426A (en) * | 1996-11-15 | 1998-03-31 | Omnivision Technologies, Inc. | Method of automatic exposure control in a MOS imaging array |
US6665010B1 (en) * | 1998-07-21 | 2003-12-16 | Intel Corporation | Controlling integration times of pixel sensors |
US6229113B1 (en) * | 1999-07-19 | 2001-05-08 | United Technologies Corporation | Method and apparatus for producing a laser drilled hole in a structure |
-
2001
- 2001-07-16 US US09/907,099 patent/US20010040632A1/en not_active Abandoned
- 2001-07-20 US US09/909,528 patent/US6970195B1/en not_active Expired - Lifetime
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5309243A (en) * | 1992-06-10 | 1994-05-03 | Eastman Kodak Company | Method and apparatus for extending the dynamic range of an electronic imaging system |
US5461425A (en) * | 1994-02-15 | 1995-10-24 | Stanford University | CMOS image sensor with pixel level A/D conversion |
US5909026A (en) * | 1996-11-12 | 1999-06-01 | California Institute Of Technology | Integrated sensor with frame memory and programmable resolution for light adaptive imaging |
US6614560B1 (en) * | 1997-07-15 | 2003-09-02 | Silverbrook Research Pty Ltd | Integrated camera circuit including image sensor, image processing, and printer drive circuits |
US6229133B1 (en) * | 1997-10-27 | 2001-05-08 | Texas Instruments Incorporated | Image sensing device with delayed phase frequency modulation |
US6330030B1 (en) * | 1998-01-05 | 2001-12-11 | Intel Corporation | Digital image sensor with low device count per pixel analog-to-digital conversion |
Cited By (32)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020085109A1 (en) * | 1999-11-08 | 2002-07-04 | Casio Computer Co., Ltd. | Photosensor system and drive control method thereof |
US6879344B1 (en) * | 1999-11-08 | 2005-04-12 | Casio Computer Co., Ltd. | Photosensor system and drive control method thereof |
US6867811B2 (en) * | 1999-11-08 | 2005-03-15 | Casio Computer Co., Ltd. | Photosensor system and drive control method thereof |
US6765610B1 (en) * | 1999-11-10 | 2004-07-20 | Casio Computer Co., Ltd. | Photosensor system and drive control method thereof |
US20020140842A1 (en) * | 2001-03-30 | 2002-10-03 | Olding Benjamin P. | Method and apparatus for storing image information for multiple sampling operations in a digital pixel sensor |
US6987536B2 (en) * | 2001-03-30 | 2006-01-17 | Pixim, Inc. | Method and apparatus for storing image information for multiple sampling operations in a digital pixel sensor |
US20040165091A1 (en) * | 2002-02-25 | 2004-08-26 | Yasuo Takemura | Image pickup apparatus |
EP1339227A3 (en) * | 2002-02-25 | 2004-03-10 | Kabushiki Kaisha Toshiba | Image pickup apparatus |
EP1339227A2 (en) * | 2002-02-25 | 2003-08-27 | Kabushiki Kaisha Toshiba | Image pickup apparatus |
US20030206236A1 (en) * | 2002-05-06 | 2003-11-06 | Agfa Corporation | CMOS digital image sensor system and method |
US6868138B2 (en) | 2002-05-29 | 2005-03-15 | The Regents Of The University Of Michigan | Method, processor and computed tomography (CT) machine for generating images utilizing high and low sensitivity data collected from a flat panel detector having an extended dynamic range |
WO2003103279A3 (en) * | 2002-05-29 | 2004-01-08 | Univ Michigan | Method, processor and computed tomography (ct) machine for generating images utilizing high and low sensitivity data collected from a flat panel detector having an extended dynamic range |
WO2003103279A2 (en) * | 2002-05-29 | 2003-12-11 | The Regents Of The University Of Michigan | Method, processor and computed tomography (ct) machine for generating images utilizing high and low sensitivity data collected from a flat panel detector having an extended dynamic range |
US20070103569A1 (en) * | 2003-06-02 | 2007-05-10 | National University Corporation Shizuoka University | Wide dynamic range image sensor |
US7889253B2 (en) * | 2003-06-02 | 2011-02-15 | National University Corporation Shizuoka University | Wide dynamic range image sensor |
US7462811B2 (en) * | 2004-11-24 | 2008-12-09 | Eastman Kodak Company | Light detection circuit |
US20060108511A1 (en) * | 2004-11-24 | 2006-05-25 | Eastman Kodak Company | Light detection circuit |
US20060170662A1 (en) * | 2004-12-07 | 2006-08-03 | Haruhisa Kurane | Image pickup device |
US7643075B2 (en) | 2004-12-07 | 2010-01-05 | Seiko Epson Corporation | Image pickup device |
US20070153117A1 (en) * | 2005-12-30 | 2007-07-05 | Yen-Yu Lin | Apparatus and method for adjusting display-related setting of an electronic device |
DE102006018943B4 (en) * | 2005-12-30 | 2008-03-20 | Mediatek Inc. | Apparatus and method for luminance compensation |
DE102006018943A1 (en) * | 2005-12-30 | 2007-07-12 | Mediatek Inc. | Apparatus and method for luminance compensation |
US20070153098A1 (en) * | 2005-12-30 | 2007-07-05 | Mediatek Incorporation | Luminance compensation apparatus and method |
US8139054B2 (en) | 2005-12-30 | 2012-03-20 | Mediatek Inc. | Luminance compensation apparatus and method |
US20080094481A1 (en) * | 2006-10-19 | 2008-04-24 | Barinder Singh Rai | Intelligent Multiple Exposure |
EP1942660A1 (en) * | 2007-01-02 | 2008-07-09 | STMicroelectronics (Research & Development) Limited | Improvements in image sensor noise reduction |
US20100321525A1 (en) * | 2009-06-17 | 2010-12-23 | Canon Kabushiki Kaisha | Image capturing apparatus, image capturing method, and storage medium |
US8314852B2 (en) * | 2009-06-17 | 2012-11-20 | Canon Kabushiki Kaisha | Image capturing apparatus, image capturing method, and storage medium |
US20130155282A1 (en) * | 2011-12-15 | 2013-06-20 | STMicroelectronics (R&D) Ltd. | Image sensor readout method and apparatus |
US9087756B2 (en) * | 2011-12-15 | 2015-07-21 | Stmicroelectronics (Research & Development) Limited | Image sensor readout method and apparatus |
US10218913B2 (en) | 2017-04-18 | 2019-02-26 | Qualcomm Incorporated | HDR/WDR image time stamps for sensor fusion |
US11303835B2 (en) | 2019-11-15 | 2022-04-12 | Samsung Electronics Co., Ltd. | Pixel array and image sensor including the same |
Also Published As
Publication number | Publication date |
---|---|
US6970195B1 (en) | 2005-11-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6975355B1 (en) | Multiple sampling via a time-indexed method to achieve wide dynamic ranges | |
US20010040632A1 (en) | Multiple sampling via a time-indexed method to achieve wide dynamic ranges | |
US6791611B2 (en) | Dual ported memory for digital image sensor | |
US6922210B2 (en) | Memory updating for digital pixel sensors | |
KR100844317B1 (en) | Cmos sensor array with a memory interface | |
US6680748B1 (en) | Multi-mode camera and method therefor | |
US6721008B2 (en) | Integrated CMOS active pixel digital camera | |
US6380880B1 (en) | Digital pixel sensor with integrated charge transfer amplifier | |
EP1947842B1 (en) | Image sensors with blooming reduction mechanisms | |
US6498576B1 (en) | Selective analog-to-digital conversion for a digital pixel sensor | |
US6778212B1 (en) | Digital image sensor with on -chip programmable logic | |
US7326904B2 (en) | In-pixel kTC noise suppression using circuit techniques | |
US6452152B1 (en) | Sense amplifier having a precision analog reference level for use with image sensors | |
US7026596B2 (en) | High-low sensitivity pixel | |
JP2000209504A (en) | Active pixel sensor | |
JP2001251558A (en) | Interlaced-type alternate pixel design for high sensitivity cmos image sensor | |
JPH05145857A (en) | Wide dynamic range camera | |
Stevanovic et al. | A high speed camera system based on an image sensor in standard cmos technology | |
US7071981B1 (en) | Image sensing system and method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |