US8462141B2 - Unified memory architecture and display controller to prevent data feed under-run - Google Patents

Unified memory architecture and display controller to prevent data feed under-run Download PDF

Info

Publication number
US8462141B2
US8462141B2 US12/596,235 US59623507A US8462141B2 US 8462141 B2 US8462141 B2 US 8462141B2 US 59623507 A US59623507 A US 59623507A US 8462141 B2 US8462141 B2 US 8462141B2
Authority
US
United States
Prior art keywords
pixel data
data
route
display
input
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US12/596,235
Other versions
US20100073388A1 (en
Inventor
Roman Mostinski
Mikhail Bourgart
Edward Vaiberman
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Xinguodu Tech Co Ltd
NXP BV
NXP USA Inc
Original Assignee
Freescale Semiconductor Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Freescale Semiconductor Inc filed Critical Freescale Semiconductor Inc
Assigned to FREESCALE SEMICONDUCTOR INC. reassignment FREESCALE SEMICONDUCTOR INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BOURGART, MIKHAIL, MOSTINSKI, ROMAN, VIABERMAN, EDWARDS
Assigned to CITIBANK, N.A. reassignment CITIBANK, N.A. SECURITY AGREEMENT Assignors: FREESCALE SEMICONDUCTOR, INC.
Assigned to CITIBANK, N.A. reassignment CITIBANK, N.A. SECURITY AGREEMENT Assignors: FREESCALE SEMICONDUCTOR, INC.
Publication of US20100073388A1 publication Critical patent/US20100073388A1/en
Assigned to CITIBANK, N.A., AS COLLATERAL AGENT reassignment CITIBANK, N.A., AS COLLATERAL AGENT SECURITY AGREEMENT Assignors: FREESCALE SEMICONDUCTOR, INC.
Publication of US8462141B2 publication Critical patent/US8462141B2/en
Application granted granted Critical
Assigned to CITIBANK, N.A., AS NOTES COLLATERAL AGENT reassignment CITIBANK, N.A., AS NOTES COLLATERAL AGENT SECURITY AGREEMENT Assignors: FREESCALE SEMICONDUCTOR, INC.
Assigned to CITIBANK, N.A., AS NOTES COLLATERAL AGENT reassignment CITIBANK, N.A., AS NOTES COLLATERAL AGENT SECURITY AGREEMENT Assignors: FREESCALE SEMICONDUCTOR, INC.
Assigned to FREESCALE SEMICONDUCTOR, INC. reassignment FREESCALE SEMICONDUCTOR, INC. PATENT RELEASE Assignors: CITIBANK, N.A., AS COLLATERAL AGENT
Assigned to FREESCALE SEMICONDUCTOR, INC. reassignment FREESCALE SEMICONDUCTOR, INC. PATENT RELEASE Assignors: CITIBANK, N.A., AS COLLATERAL AGENT
Assigned to FREESCALE SEMICONDUCTOR, INC. reassignment FREESCALE SEMICONDUCTOR, INC. PATENT RELEASE Assignors: CITIBANK, N.A., AS COLLATERAL AGENT
Assigned to MORGAN STANLEY SENIOR FUNDING, INC. reassignment MORGAN STANLEY SENIOR FUNDING, INC. ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS Assignors: CITIBANK, N.A.
Assigned to MORGAN STANLEY SENIOR FUNDING, INC. reassignment MORGAN STANLEY SENIOR FUNDING, INC. ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS Assignors: CITIBANK, N.A.
Assigned to MORGAN STANLEY SENIOR FUNDING, INC. reassignment MORGAN STANLEY SENIOR FUNDING, INC. SUPPLEMENT TO THE SECURITY AGREEMENT Assignors: FREESCALE SEMICONDUCTOR, INC.
Assigned to NXP, B.V., F/K/A FREESCALE SEMICONDUCTOR, INC. reassignment NXP, B.V., F/K/A FREESCALE SEMICONDUCTOR, INC. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to NXP B.V. reassignment NXP B.V. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to NXP USA, INC. reassignment NXP USA, INC. CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: FREESCALE SEMICONDUCTOR INC.
Assigned to NXP USA, INC. reassignment NXP USA, INC. CORRECTIVE ASSIGNMENT TO CORRECT THE NATURE OF CONVEYANCE LISTED CHANGE OF NAME SHOULD BE MERGER AND CHANGE PREVIOUSLY RECORDED AT REEL: 040652 FRAME: 0180. ASSIGNOR(S) HEREBY CONFIRMS THE MERGER AND CHANGE OF NAME. Assignors: FREESCALE SEMICONDUCTOR INC.
Assigned to MORGAN STANLEY SENIOR FUNDING, INC. reassignment MORGAN STANLEY SENIOR FUNDING, INC. CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE PATENTS 8108266 AND 8062324 AND REPLACE THEM WITH 6108266 AND 8060324 PREVIOUSLY RECORDED ON REEL 037518 FRAME 0292. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS. Assignors: CITIBANK, N.A.
Assigned to SHENZHEN XINGUODU TECHNOLOGY CO., LTD. reassignment SHENZHEN XINGUODU TECHNOLOGY CO., LTD. CORRECTIVE ASSIGNMENT TO CORRECT THE TO CORRECT THE APPLICATION NO. FROM 13,883,290 TO 13,833,290 PREVIOUSLY RECORDED ON REEL 041703 FRAME 0536. ASSIGNOR(S) HEREBY CONFIRMS THE THE ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS.. Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to NXP B.V. reassignment NXP B.V. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to MORGAN STANLEY SENIOR FUNDING, INC. reassignment MORGAN STANLEY SENIOR FUNDING, INC. CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE APPLICATION 11759915 AND REPLACE IT WITH APPLICATION 11759935 PREVIOUSLY RECORDED ON REEL 037486 FRAME 0517. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS. Assignors: CITIBANK, N.A.
Assigned to NXP B.V. reassignment NXP B.V. CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE APPLICATION 11759915 AND REPLACE IT WITH APPLICATION 11759935 PREVIOUSLY RECORDED ON REEL 040928 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE RELEASE OF SECURITY INTEREST. Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to NXP, B.V. F/K/A FREESCALE SEMICONDUCTOR, INC. reassignment NXP, B.V. F/K/A FREESCALE SEMICONDUCTOR, INC. CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE APPLICATION 11759915 AND REPLACE IT WITH APPLICATION 11759935 PREVIOUSLY RECORDED ON REEL 040925 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE RELEASE OF SECURITY INTEREST. Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/363—Graphics controllers
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/02—Handling of images in compressed format, e.g. JPEG, MPEG
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2360/00—Aspects of the architecture of display systems
    • G09G2360/12—Frame memory handling
    • G09G2360/125—Frame memory handling using unified memory architecture [UMA]
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/39—Control of the bit-mapped memory
    • G09G5/395—Arrangements specially adapted for transferring the contents of the bit-mapped memory to the screen
    • G09G5/397—Arrangements specially adapted for transferring the contents of two or more bit-mapped memories to the screen simultaneously, e.g. for mixing or overlay

Definitions

  • This invention relates to improvements in or relating to display controller architecture, particularly but not exclusively when dealing with variable data feeds.
  • Isochronous displays require data to be displayed in a synchronous manner, with a refresh process. Isochronous displays cannot tolerate delays in the arrival of data. These situations are known as data under-run conditions. Data under-run conditions may cause displays to become corrupt with visibly bad pixels and sometimes even bad rows or fields. Accordingly, there is a requirement to provide a display controller which overcomes this problem. This is particularly the case with resource limited and cost limited handheld devices, but the problem exists with any isochronous display.
  • FIG. 1 shows an example of a display controller which includes a direct memory access (DMA) engine 100 . This passes data to an input FIFO 102 . The output from the input FIFO is passed through a palette random access memory (RAM) 106 and a dithering engine 108 to a formatter 110 . The output data is then passed to the pins via an output FIFO 112 .
  • DMA direct memory access
  • One object of the present invention is to provide an efficient method of feeding data to a display. Another object of the present invention is to reduce visible display corruption. A still further object to the present invention is to provide a display controller architecture which overcomes at least some of the disadvantages of the prior art.
  • the present invention provides a method and system as described in the accompanying claims.
  • FIG. 1 is a block diagram of a typical display controller in accordance with the prior art
  • FIG. 2 is a block diagram of a display controller architecture in accordance with one embodiment of the invention, given by way of example,
  • FIG. 3 is a flow chart of the method steps in accordance with one embodiment of the invention, given by way of example,
  • the display controller includes a direct memory access (DMA) engine 202 which is connected to an input FIFO 204 .
  • the input FIFO is connected to two circuit parts, the left-hand side (or the main pipeline or route) 208 and a right-hand side (or the secondary pipeline or route) 210 .
  • the left-hand side 208 includes two combiners 212 , 214 and a dithering engine 216 .
  • the output from the dithering engine is passed through a third combiner 218 to a formatter 220 .
  • the output from the formatter goes to an output FIFO 222 and then provides the output data to the pins.
  • This part of the processing circuit is substantially similar to the prior art and functions in a similar way there to.
  • the right-hand side 210 includes a compressor 224 and a combiner 226 .
  • the output from the combiner passes into a combined palette RAM 228 and a spare FIFO 230 .
  • the output from the combined palette ram and spare FIFO (details of which will be described in greater detail below) is passed through a divider 232 .
  • the output from the divider may be passed either through the third combiner 218 or into an expander 234 . This is then passed into an extrapolator 236 and the output thereof is passed to the first combiner 212 . The details of how this operates will now be described in more detail.
  • the pixels in a real display image have a high degree of two-dimensional correlation.
  • the visual display corruption will be lower if the two-dimensional extrapolation is used instead of the last pixel repetition as is utilised in the prior art.
  • the last pixel repetition is actually a simple one-dimensional extrapolation.
  • This invention deals with the issue of providing two-dimensional extrapolation. This is achieved by storing a whole line of pixel data for the line before the line it is currently being processed. This storage is effected locally to avoid the need for additional access to an external memory. This is due to the fact that access to the external memories would be complicated to implement, time-consuming, power consuming and may introduced too many delays.
  • the palette RAM includes a lookup table that is generally not used in cases of high colour depth when memory bandwidth limitations become an issue. When there are no memory issues the palette RAM is used to augment the colour of the resultant image.
  • the lookup table in the palette RAM is used to store the previous row of pixel data.
  • the lookup table may store multiple rows of pixel data or other parts of the image as the case may be due to the fact that the compression reduces the size of information constituting a row of pixel data. Compression rates of between 20% and 50% can be achieved by using run length encoding (RLE). It will be appreciated that other types of encoding and compression may be used to reduce the amount of storage space required for a specific amount of information, to be stored for later use if a data latency event occurs.
  • the DMA engine fetches data from the refresh buffer, located in the system memory and passes the data to the input FIFO.
  • the data from the input FIFO is then channelled down both the right-hand side 210 and the left-hand side (the main display pipeline) 208 .
  • the main display pipeline consists of the dithering engine, formatter and output FIFO as previously described.
  • the data is compressed and then stored in the combination palette RAM and spare FIFO. As the data is in compressed form the amount of memory required is optimised.
  • a display controller (not shown per se) determines what data should be output and from which FIFO. When the system is operating normally and a steady flow of synchronous data is being received the display controller passes the data from the input FIFO to the output FIFO for output to the data pins.
  • the display controller switches to data sourcing the data for the output FIFO from the left-hand side 208 to the right-hand side 210 , in particular from the combination palette RAM and spare FIFO ( 228 , 230 ).
  • data is output from the spare FIFO and undergoes decompression in the expander (or decompressor) and passes through an extrapolator which ensures the synchronisation of the data from the spare FIFO with the normal flow of data from the left-hand side. In other words, when there is no longer any data from the left-hand side, data from the spare FIFO is displayed.
  • the data may be read from the spare FIFO and sent to the formatter if the input FIFO experiences an under-run condition. This means that is less delay in switching from the LHS to the RHS, but requires more access to the spare FIFO.
  • the dataflow from the spare FIFO continues until the system memory becomes available again. If this occurs midway through a row of pixel data the display controller can immediately switch to data from the input FIFO (and the left-hand side) and returns interpolated values to the spare FIFO. This ensures that if there is a further under-run condition there is still date in the spare FIFO which can be used.
  • the display controller Whenever the system memory becomes available the display controller immediately displays pixel data from the start position of that available data and skips any pixel data from the spare FIFO which it has not yet read.
  • FIG. 3 shows that any flowcharts of the method steps carried out within the present invention and by the display controller.
  • Pixel data is received by the DMA engine (step 300 ) and is duplicated (step 302 ).
  • One version of the data is sent to the input FIFO (step 304 ) and the second version of the data is sent to the combination parallel RAM spare FIFO (step 306 ).
  • the determination is made as to whether there is an under-run condition (step 308 ). If no under-run condition is detected ( 310 ) the pixel data from the input FIFO is sent to the output FIFO for onward transmission to the data pins (step 312 ). The transmission of data continues in this manner until an under-run condition is detected.
  • the process reverts to step 308 in a predetermined manner (step 314 ).
  • step 318 If an under-run condition is detected (yes, 316 ) and output is activated from the palette RAM spare FIFO combination (step 318 ). This causes the last row of pixel data to be passed to the output FIFO for onward transmission to the data pins (step 320 ). At the same time when the under-run condition is detected a further determination is made to determine whether the under-run condition is finished (step 322 ). If the under-run condition has finished (yes, 326 ) the output of stored last row pixel data to the output FIFO is stopped (step 328 ). At the same time pixel data is sent from the input FIFO to the output FIFO for onward transmission to the data pins (step 312 ).
  • the present invention can be implemented in any situation where a synchronous display of data is required.
  • Different types of compression and decompression could be used and the invention is not limited to RLE.
  • different types of extrapolation could also be used and is not limited to be by linear or two-dimensional extrapolation indicated in the example.
  • the local memory in this example the combined palette RAM and spare FIFO
  • the local memory can take any appropriate form and may be provided by another memory elements in the device as a whole or otherwise.
  • the memory need not be limited to a FIFO, but may be another type of memory which can be accessed to provide the required pixel data in the required sequence.
  • palette RAM spare FIFO combination (or any other appropriate memory) could be used for devices where there are more than one stream of pixel data.
  • a device with a dual display may use only one palette RAM and spare FIFO combination for two main transmission tracks.
  • This invention has the effect of alleviating the visible effect of latency buffer under-run conditions. This is achieved by reusing the FIFO and a colour lookup table, which reside in the display controller to store compressed data from previous scan lines. In addition missing pixels are reconstructed using two-dimensional spatial correlation of pixel data and pixel data available from previous scan lines and current scan lines.
  • the FIFO need not be enlarged for this task, unlike the circumstances with the prior art.
  • the additional hardware required is of the minimum which is particularly important in the cost sensitive handheld device environment.

Abstract

A display controller for controlling data in an isochronous display where fluctuation of data feed latency occurs, the display controller including an input memory which receives pixel data and transmits the pixel data through a main route and a secondary route; wherein pixel data is transmitted through the main route and is processed for delivery to the display in a predetermined manner; characterized in that the secondary route comprises a memory for storing a two-dimensional section of the pixel data that corresponds at least in part to the pixel data being transmitted through the main route at that time; further characterized in that the display controller includes a detector for identifying a data feed latency event and in response there to switching the transmission of the pixel data to the secondary route and processing the pixel data through secondary route for delivery to the display such that when a data feed latency event occurs the stored two-dimensional section of the pixel data from the secondary route is displayed on the display.

Description

FIELD OF THE INVENTION
This invention relates to improvements in or relating to display controller architecture, particularly but not exclusively when dealing with variable data feeds.
BACKGROUND OF THE INVENTION
There are many environments in which isochronous (synchronous) displays exist. For example LCD and CRT displays may be isochronous. Isochronous displays require data to be displayed in a synchronous manner, with a refresh process. Isochronous displays cannot tolerate delays in the arrival of data. These situations are known as data under-run conditions. Data under-run conditions may cause displays to become corrupt with visibly bad pixels and sometimes even bad rows or fields. Accordingly, there is a requirement to provide a display controller which overcomes this problem. This is particularly the case with resource limited and cost limited handheld devices, but the problem exists with any isochronous display.
There are a number of solutions that have been proposed to this problem. One such proposal is disclosed in the US 2003/0142058 (Inventor: Maghielse) and makes use of a first in first out (FIFO) type latency buffer. When the FIFO under-runs the display controller either stops or throttles the display pixel clocks; or repeats the last pixel data keeping the clock running. FIG. 1 shows an example of a display controller which includes a direct memory access (DMA) engine 100. This passes data to an input FIFO 102. The output from the input FIFO is passed through a palette random access memory (RAM) 106 and a dithering engine 108 to a formatter 110. The output data is then passed to the pins via an output FIFO 112.
This has a number of disadvantages, one disadvantage being that not all displays will tolerate the stalling of the pixel clock. In addition, long repetition time of the last pixel can be visible in most cases and thereby does not solve the problem of “bad pixels”.
In order to minimise costs of hand held and mobile devices a number of proposals have been made which adopt a system architecture and unified memory and avoid using separate memory for display refresh buffers. Because of this the display refresh process depends not only on the display controller access to memory but also on other device activities such as the CPU, the DMA and so on. The access by other devices may cause fluctuating memory latency. It is this fluctuating memory latency that can also cause the data under-run in a display controller FIFO.
One object of the present invention is to provide an efficient method of feeding data to a display. Another object of the present invention is to reduce visible display corruption. A still further object to the present invention is to provide a display controller architecture which overcomes at least some of the disadvantages of the prior art.
SUMMARY OF THE INVENTION
The present invention provides a method and system as described in the accompanying claims.
BRIEF DESCRIPTION OF THE DRAWINGS
Reference will now be made, by way of example, to the accompanying drawings, in which:
FIG. 1 is a block diagram of a typical display controller in accordance with the prior art,
FIG. 2 is a block diagram of a display controller architecture in accordance with one embodiment of the invention, given by way of example,
FIG. 3 is a flow chart of the method steps in accordance with one embodiment of the invention, given by way of example,
 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
Referring now to FIG. 2 a display controller architecture is shown generally at 200. The display controller includes a direct memory access (DMA) engine 202 which is connected to an input FIFO 204. The input FIFO is connected to two circuit parts, the left-hand side (or the main pipeline or route) 208 and a right-hand side (or the secondary pipeline or route) 210. The left-hand side 208 includes two combiners 212, 214 and a dithering engine 216. The output from the dithering engine is passed through a third combiner 218 to a formatter 220. The output from the formatter goes to an output FIFO 222 and then provides the output data to the pins. This part of the processing circuit is substantially similar to the prior art and functions in a similar way there to.
The right-hand side 210 includes a compressor 224 and a combiner 226. The output from the combiner passes into a combined palette RAM 228 and a spare FIFO 230. The output from the combined palette ram and spare FIFO (details of which will be described in greater detail below) is passed through a divider 232. The output from the divider may be passed either through the third combiner 218 or into an expander 234. This is then passed into an extrapolator 236 and the output thereof is passed to the first combiner 212. The details of how this operates will now be described in more detail.
The pixels in a real display image have a high degree of two-dimensional correlation. The visual display corruption will be lower if the two-dimensional extrapolation is used instead of the last pixel repetition as is utilised in the prior art. The last pixel repetition is actually a simple one-dimensional extrapolation. This invention deals with the issue of providing two-dimensional extrapolation. This is achieved by storing a whole line of pixel data for the line before the line it is currently being processed. This storage is effected locally to avoid the need for additional access to an external memory. This is due to the fact that access to the external memories would be complicated to implement, time-consuming, power consuming and may introduced too many delays.
In accordance with the present invention of the additional local memory is provided by the combination palette RAM and spare FIFO. The palette RAM includes a lookup table that is generally not used in cases of high colour depth when memory bandwidth limitations become an issue. When there are no memory issues the palette RAM is used to augment the colour of the resultant image. In the present invention the lookup table in the palette RAM is used to store the previous row of pixel data. By use of compression and decompression techniques the lookup table may store multiple rows of pixel data or other parts of the image as the case may be due to the fact that the compression reduces the size of information constituting a row of pixel data. Compression rates of between 20% and 50% can be achieved by using run length encoding (RLE). It will be appreciated that other types of encoding and compression may be used to reduce the amount of storage space required for a specific amount of information, to be stored for later use if a data latency event occurs.
The manner in which the present invention works will now be described. The DMA engine fetches data from the refresh buffer, located in the system memory and passes the data to the input FIFO. The data from the input FIFO is then channelled down both the right-hand side 210 and the left-hand side (the main display pipeline) 208. The main display pipeline consists of the dithering engine, formatter and output FIFO as previously described. With respect to the data channelled down the right-hand side 210 the data is compressed and then stored in the combination palette RAM and spare FIFO. As the data is in compressed form the amount of memory required is optimised. A display controller (not shown per se) determines what data should be output and from which FIFO. When the system is operating normally and a steady flow of synchronous data is being received the display controller passes the data from the input FIFO to the output FIFO for output to the data pins.
If the input FIFO experiences an under-run condition the display controller switches to data sourcing the data for the output FIFO from the left-hand side 208 to the right-hand side 210, in particular from the combination palette RAM and spare FIFO (228, 230). When this occurs data is output from the spare FIFO and undergoes decompression in the expander (or decompressor) and passes through an extrapolator which ensures the synchronisation of the data from the spare FIFO with the normal flow of data from the left-hand side. In other words, when there is no longer any data from the left-hand side, data from the spare FIFO is displayed.
As an alternative, the data may be read from the spare FIFO and sent to the formatter if the input FIFO experiences an under-run condition. This means that is less delay in switching from the LHS to the RHS, but requires more access to the spare FIFO.
The dataflow from the spare FIFO continues until the system memory becomes available again. If this occurs midway through a row of pixel data the display controller can immediately switch to data from the input FIFO (and the left-hand side) and returns interpolated values to the spare FIFO. This ensures that if there is a further under-run condition there is still date in the spare FIFO which can be used.
Whenever the system memory becomes available the display controller immediately displays pixel data from the start position of that available data and skips any pixel data from the spare FIFO which it has not yet read.
FIG. 3 shows that any flowcharts of the method steps carried out within the present invention and by the display controller. Pixel data is received by the DMA engine (step 300) and is duplicated (step 302). One version of the data is sent to the input FIFO (step 304) and the second version of the data is sent to the combination parallel RAM spare FIFO (step 306). At a certain point in time, either as a result of lack of available pixel data or perhaps on a temporary basis the determination is made as to whether there is an under-run condition (step 308). If no under-run condition is detected (310) the pixel data from the input FIFO is sent to the output FIFO for onward transmission to the data pins (step 312). The transmission of data continues in this manner until an under-run condition is detected. The process reverts to step 308 in a predetermined manner (step 314).
If an under-run condition is detected (yes, 316) and output is activated from the palette RAM spare FIFO combination (step 318). This causes the last row of pixel data to be passed to the output FIFO for onward transmission to the data pins (step 320). At the same time when the under-run condition is detected a further determination is made to determine whether the under-run condition is finished (step 322). If the under-run condition has finished (yes, 326) the output of stored last row pixel data to the output FIFO is stopped (step 328). At the same time pixel data is sent from the input FIFO to the output FIFO for onward transmission to the data pins (step 312).
It will be appreciated that the present invention can be implemented in any situation where a synchronous display of data is required. Different types of compression and decompression could be used and the invention is not limited to RLE. In addition different types of extrapolation could also be used and is not limited to be by linear or two-dimensional extrapolation indicated in the example. Also it is not necessary that a line of data is stored in the memory, instead a different two-dimensional section of data maybe stored instead.
It will further be appreciated that the local memory (in this example the combined palette RAM and spare FIFO) can take any appropriate form and may be provided by another memory elements in the device as a whole or otherwise. The memory need not be limited to a FIFO, but may be another type of memory which can be accessed to provide the required pixel data in the required sequence.
It will be still further appreciated that the palette RAM spare FIFO combination (or any other appropriate memory) could be used for devices where there are more than one stream of pixel data. For example a device with a dual display may use only one palette RAM and spare FIFO combination for two main transmission tracks.
This invention has the effect of alleviating the visible effect of latency buffer under-run conditions. This is achieved by reusing the FIFO and a colour lookup table, which reside in the display controller to store compressed data from previous scan lines. In addition missing pixels are reconstructed using two-dimensional spatial correlation of pixel data and pixel data available from previous scan lines and current scan lines. The FIFO need not be enlarged for this task, unlike the circumstances with the prior art. In addition the additional hardware required is of the minimum which is particularly important in the cost sensitive handheld device environment.
Various other alternatives to the example shown will be appreciated to be included within the scope of the present invention.

Claims (13)

The invention claimed is:
1. A display controller for controlling data in an isochronous display where fluctuation of data feed latency occurs, the display controller comprising:
an input memory which receives at an input of the input memory pixel data and transmits at an output of the input memory the pixel data through a main route and a secondary route; wherein, when in operation pixel data is transmitted through the main route and is processed for delivery to the display in a predetermined manner;
the secondary route comprises a secondary memory for storing a two-dimensional section of the pixel data that corresponds at least in part to the pixel data being transmitted through the main route at that time;
the display controller further comprises a detector for identifying a data feed latency event, and, in response thereto switching the transmission of the pixel data to the secondary route and processing the pixel data through the secondary route for delivery to the display such that when a data feed latency event occurs, causing missing pixel data in the main route, the stored two-dimensional section of the pixel data from the secondary route is displayed on the display to provide a two dimensional extrapolation to take the place of the missing pixel data;
wherein the secondary route comprises an encoder for encoding on a line by line basis the pixel data and a decoder which can decode the stored encoded pixel data when required, and wherein the secondary memory stores encoded data: and
wherein the decoder comprises an expander and an extrapolator, the extrapolator to receive at a first input of the extrapolator pixel data output from the expander and to receive at a second input of the extrapolator pixel data output from the input memory, wherein the extrapolation is based on the pixel data received at the first input of the extrapolator and the second input of the extrapolator.
2. A display controller as claimed in claim 1, wherein the secondary memory stores at least one line of pixel data that corresponds to the pixel data being transmitted through the main route at that time.
3. A display controller as claimed in claim 2, wherein the secondary memory is provided by an already existing memory within the display controller.
4. A display controller as claimed in claim 1, wherein the encoder comprises a compressor.
5. A display controller as claimed in claim 1, wherein the secondary memory comprises a FIFO.
6. A display controller as claimed in claim 1, wherein the secondary memory is provided by an already existing memory within the display controller.
7. A display controller as claimed in claim 6, wherein the secondary memory is provided by a palette RAM.
8. A display controller as claimed in claim 7, wherein the palette RAM data to be selectively routed from an output of the palette RAM through a portion of the main route and processed for delivery to the display.
9. A display controller as claimed in claim 1, wherein the extrapolated pixel data to be selectively routed from an output of the extrapolator through the main route and processed for delivery to the display.
10. A method for controlling data in an isochronous display where fluctuation of data feed latency occurs, the method comprising:
receiving pixel data from an input memory;
transmitting the received pixel data through a main route and a secondary route,
processing pixel data transmitted through the main route for delivery to the display in a predetermined manner;
storing in a memory in the second route a two-dimensional section of the pixel data that corresponds at least in part to the pixel data being transmitted through the main route at that time;
identifying a data feed latency event;
switching the transmission of the pixel data to the secondary route; and
processing the pixel data through the secondary route for delivery to the display such that when a data feed latency event occurs, causing missing pixel data in the main route, the stored two-dimensional section of the pixel data from the secondary route is displayed on the display to provide a two-dimensional extrapolation to take the place of the missing pixel data;
wherein the secondary route comprises an encoder for encoding on a line by line basis the pixel data and a decoder which can decode encoded pixel data when required, and wherein a secondary memory stores encoded data; and
wherein the decoder comprises an expander and an extrapolator, the extrapolator to receive at a first input of the extrapolator pixel data output from the expander and to receive at a second input of the extrapolator pixel data output from the input memory, wherein the extrapolation is based on the pixel data received at the first input of the extrapolator and the second input of the extrapolator.
11. The method of claim 10, wherein the step of storing comprises storing a line of pixel data corresponding to the pixel data being transmitted through the main route at that time.
12. A non-transitory computer readable medium having a computer program stored thereon, the computer program comprising instructions that when said computer program is executed on a computer system carries out a method for controlling data in an isochronous display where fluctuation of data feed latency occurs, the method comprising:
receiving pixel data from an input memory;
transmitting the received pixel data through a main route and a secondary route;
processing pixel data transmitted through the main route for delivery to the display in a predetermined manner;
storing in a memory in the second route a two-dimensional section of the pixel data that corresponds at least in part to the pixel data being transmitted through the main route at that time;—identifying a data feed latency event;
switching the transmission of the pixel data to the secondary route; and
processing the pixel data through the secondary route for delivery to the display such that when a data feed latency event occurs, causing missing pixel data in the main route, the stored two-dimensional section of the pixel data from the secondary route is displayed on the display to provide a two-dimensional extrapolation to take the place of the missing pixel data;
wherein the secondary route comprises an encoder for encoding on a line by line basis the pixel data and a decoder which can decode encoded pixel data when required, and wherein a secondary memory stores encoded data: and
wherein the decoder comprises an expander and an extrapolator, the extrapolator to receive at a first input of the extrapolator pixel data output from the expander and to receive at a second input of the extrapolator pixel data output from the input memory, wherein the extrapolation is based on the pixel data received at the first input of the extrapolator and the second input of the extrapolator.
13. The non-transitory computer readable medium of claim 12, wherein the secondary memory is provided by an already existing memory within the display controller.
US12/596,235 2007-04-26 2007-04-26 Unified memory architecture and display controller to prevent data feed under-run Active 2029-07-23 US8462141B2 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/IB2007/051554 WO2008132556A1 (en) 2007-04-26 2007-04-26 Unified memory achitecture and display controller to prevent data feed under-run

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US12/779,618 Continuation-In-Part US8574179B2 (en) 2004-09-17 2010-05-13 Enhanced biomechanical stimulation device

Publications (2)

Publication Number Publication Date
US20100073388A1 US20100073388A1 (en) 2010-03-25
US8462141B2 true US8462141B2 (en) 2013-06-11

Family

ID=38566019

Family Applications (1)

Application Number Title Priority Date Filing Date
US12/596,235 Active 2029-07-23 US8462141B2 (en) 2007-04-26 2007-04-26 Unified memory architecture and display controller to prevent data feed under-run

Country Status (2)

Country Link
US (1) US8462141B2 (en)
WO (1) WO2008132556A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150179234A1 (en) * 2013-12-25 2015-06-25 Kabushiki Kaisha Toshiba Semiconductor system and power source chip

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8675004B2 (en) * 2010-01-11 2014-03-18 Apple Inc. Buffer underrun handling
WO2013150342A1 (en) 2012-04-05 2013-10-10 Freescale Semiconductor, Inc. Diagnostic data generation apparatus, integrated circuit and method of generating diagnostic data
JP2015005835A (en) * 2013-06-19 2015-01-08 シャープ株式会社 Image processing device, image forming device, and recording medium
GB2549311B (en) * 2016-04-13 2019-09-11 Advanced Risc Mach Ltd Data processing systems
US10387993B2 (en) * 2017-09-29 2019-08-20 Intel Corporation Fault-tolerant graphics display engine
CN111128089B (en) * 2020-03-27 2020-06-19 南京芯驰半导体科技有限公司 Display controller with data underrun self-recovery function and method

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5488385A (en) 1994-03-03 1996-01-30 Trident Microsystems, Inc. Multiple concurrent display system
GB2319878A (en) 1996-11-27 1998-06-03 Motorola Inc Display system
US5764201A (en) 1996-01-16 1998-06-09 Neomagic Corp. Multiplexed yuv-movie pixel path for driving dual displays
US5963192A (en) * 1996-10-11 1999-10-05 Silicon Motion, Inc. Apparatus and method for flicker reduction and over/underscan
US6490534B1 (en) * 2000-04-25 2002-12-03 Henry Pfister Camera measurement system
US20030142058A1 (en) 2002-01-31 2003-07-31 Maghielse William T. LCD controller architecture for handling fluctuating bandwidth conditions
US20040207644A1 (en) 1998-11-09 2004-10-21 Broadcom Corporation Graphics display system with anti-flutter filtering and vertical scaling feature
US20050184993A1 (en) 2004-02-24 2005-08-25 Ludwin Albert S. Display processor for a wireless device
US20080174606A1 (en) * 2007-01-23 2008-07-24 Srikanth Rengarajan Method and apparatus for low power refresh of a display device

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5488385A (en) 1994-03-03 1996-01-30 Trident Microsystems, Inc. Multiple concurrent display system
US5764201A (en) 1996-01-16 1998-06-09 Neomagic Corp. Multiplexed yuv-movie pixel path for driving dual displays
US5963192A (en) * 1996-10-11 1999-10-05 Silicon Motion, Inc. Apparatus and method for flicker reduction and over/underscan
GB2319878A (en) 1996-11-27 1998-06-03 Motorola Inc Display system
US5867140A (en) * 1996-11-27 1999-02-02 Motorola, Inc. Display system and circuit therefor
US20040207644A1 (en) 1998-11-09 2004-10-21 Broadcom Corporation Graphics display system with anti-flutter filtering and vertical scaling feature
US6490534B1 (en) * 2000-04-25 2002-12-03 Henry Pfister Camera measurement system
US20030142058A1 (en) 2002-01-31 2003-07-31 Maghielse William T. LCD controller architecture for handling fluctuating bandwidth conditions
US20050184993A1 (en) 2004-02-24 2005-08-25 Ludwin Albert S. Display processor for a wireless device
US20080174606A1 (en) * 2007-01-23 2008-07-24 Srikanth Rengarajan Method and apparatus for low power refresh of a display device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
International Search Report and Written Opinion correlating to PCT/IB2007/051554 dated Nov. 12, 2007.

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150179234A1 (en) * 2013-12-25 2015-06-25 Kabushiki Kaisha Toshiba Semiconductor system and power source chip

Also Published As

Publication number Publication date
WO2008132556A1 (en) 2008-11-06
US20100073388A1 (en) 2010-03-25

Similar Documents

Publication Publication Date Title
US8462141B2 (en) Unified memory architecture and display controller to prevent data feed under-run
US7944451B2 (en) Providing pixels from an update buffer
US9619866B2 (en) Image transformation
US7366816B2 (en) Method and apparatus for adaptively adjusting the bandwidth of a data transmission channel having multiple buffered paths
US5907330A (en) Reducing power consumption and bus bandwidth requirements in cellular phones and PDAS by using a compressed display cache
US9319677B2 (en) Video decoding method and system thereof
US8451280B2 (en) Display control device having a frame buffer for temporarily storing image data to be displayed on either one of a first display device or a second display device
KR101796603B1 (en) Mechanism for memory reduction in picture-in-picture video generation
US20060053233A1 (en) Method and system for implementing a remote overlay cursor
US9947298B2 (en) Variable compression management of memory for storing display data
US20130148740A1 (en) Method and apparatus for processing partial video frame data
JPH07501164A (en) Video/graphics memory system
US10733954B2 (en) Method of processing display data
EP3005285A1 (en) Management of memory for storing display data
US20130002521A1 (en) Screen relay device, screen relay system, and computer -readable storage medium
US20160005379A1 (en) Image Generation
US6717989B1 (en) Video decoding apparatus and method for a shared display memory system
US20210099671A1 (en) System and Method for Redirecting Audio and Video Data Streams in a Display-Server Computing System
US7903119B2 (en) Compression of image regions according to graphics command type
CN113728622A (en) Method and device for wirelessly transmitting image, storage medium and electronic equipment
US20060071922A1 (en) Device and method for up/down converting data output
CN117278693A (en) Image data processing circuit, method, electronic device, and medium
JP2001350458A (en) Device and method for plotting image
KR20050117894A (en) Method and apparatus for controling of liquid crystal display

Legal Events

Date Code Title Description
AS Assignment

Owner name: FREESCALE SEMICONDUCTOR INC.,TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MOSTINSKI, ROMAN;BOURGART, MIKHAIL;VIABERMAN, EDWARDS;REEL/FRAME:023383/0279

Effective date: 20070527

Owner name: FREESCALE SEMICONDUCTOR INC., TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MOSTINSKI, ROMAN;BOURGART, MIKHAIL;VIABERMAN, EDWARDS;REEL/FRAME:023383/0279

Effective date: 20070527

AS Assignment

Owner name: CITIBANK, N.A.,NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNOR:FREESCALE SEMICONDUCTOR, INC.;REEL/FRAME:024079/0082

Effective date: 20100212

Owner name: CITIBANK, N.A.,NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNOR:FREESCALE SEMICONDUCTOR, INC.;REEL/FRAME:024085/0001

Effective date: 20100219

Owner name: CITIBANK, N.A., NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNOR:FREESCALE SEMICONDUCTOR, INC.;REEL/FRAME:024079/0082

Effective date: 20100212

Owner name: CITIBANK, N.A., NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNOR:FREESCALE SEMICONDUCTOR, INC.;REEL/FRAME:024085/0001

Effective date: 20100219

AS Assignment

Owner name: CITIBANK, N.A., AS COLLATERAL AGENT,NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNOR:FREESCALE SEMICONDUCTOR, INC.;REEL/FRAME:024397/0001

Effective date: 20100413

Owner name: CITIBANK, N.A., AS COLLATERAL AGENT, NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNOR:FREESCALE SEMICONDUCTOR, INC.;REEL/FRAME:024397/0001

Effective date: 20100413

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: CITIBANK, N.A., AS NOTES COLLATERAL AGENT, NEW YOR

Free format text: SECURITY AGREEMENT;ASSIGNOR:FREESCALE SEMICONDUCTOR, INC.;REEL/FRAME:030633/0424

Effective date: 20130521

AS Assignment

Owner name: CITIBANK, N.A., AS NOTES COLLATERAL AGENT, NEW YOR

Free format text: SECURITY AGREEMENT;ASSIGNOR:FREESCALE SEMICONDUCTOR, INC.;REEL/FRAME:031591/0266

Effective date: 20131101

AS Assignment

Owner name: FREESCALE SEMICONDUCTOR, INC., TEXAS

Free format text: PATENT RELEASE;ASSIGNOR:CITIBANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:037355/0723

Effective date: 20151207

Owner name: FREESCALE SEMICONDUCTOR, INC., TEXAS

Free format text: PATENT RELEASE;ASSIGNOR:CITIBANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:037356/0553

Effective date: 20151207

Owner name: FREESCALE SEMICONDUCTOR, INC., TEXAS

Free format text: PATENT RELEASE;ASSIGNOR:CITIBANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:037356/0143

Effective date: 20151207

AS Assignment

Owner name: MORGAN STANLEY SENIOR FUNDING, INC., MARYLAND

Free format text: ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS;ASSIGNOR:CITIBANK, N.A.;REEL/FRAME:037486/0517

Effective date: 20151207

AS Assignment

Owner name: MORGAN STANLEY SENIOR FUNDING, INC., MARYLAND

Free format text: ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS;ASSIGNOR:CITIBANK, N.A.;REEL/FRAME:037518/0292

Effective date: 20151207

AS Assignment

Owner name: MORGAN STANLEY SENIOR FUNDING, INC., MARYLAND

Free format text: SUPPLEMENT TO THE SECURITY AGREEMENT;ASSIGNOR:FREESCALE SEMICONDUCTOR, INC.;REEL/FRAME:039138/0001

Effective date: 20160525

FPAY Fee payment

Year of fee payment: 4

AS Assignment

Owner name: NXP, B.V., F/K/A FREESCALE SEMICONDUCTOR, INC., NETHERLANDS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040925/0001

Effective date: 20160912

Owner name: NXP, B.V., F/K/A FREESCALE SEMICONDUCTOR, INC., NE

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040925/0001

Effective date: 20160912

AS Assignment

Owner name: NXP B.V., NETHERLANDS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040928/0001

Effective date: 20160622

AS Assignment

Owner name: NXP USA, INC., TEXAS

Free format text: CHANGE OF NAME;ASSIGNOR:FREESCALE SEMICONDUCTOR INC.;REEL/FRAME:040652/0180

Effective date: 20161107

AS Assignment

Owner name: NXP USA, INC., TEXAS

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE NATURE OF CONVEYANCE LISTED CHANGE OF NAME SHOULD BE MERGER AND CHANGE PREVIOUSLY RECORDED AT REEL: 040652 FRAME: 0180. ASSIGNOR(S) HEREBY CONFIRMS THE MERGER AND CHANGE OF NAME;ASSIGNOR:FREESCALE SEMICONDUCTOR INC.;REEL/FRAME:041354/0148

Effective date: 20161107

AS Assignment

Owner name: MORGAN STANLEY SENIOR FUNDING, INC., MARYLAND

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE PATENTS 8108266 AND 8062324 AND REPLACE THEM WITH 6108266 AND 8060324 PREVIOUSLY RECORDED ON REEL 037518 FRAME 0292. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS;ASSIGNOR:CITIBANK, N.A.;REEL/FRAME:041703/0536

Effective date: 20151207

AS Assignment

Owner name: SHENZHEN XINGUODU TECHNOLOGY CO., LTD., CHINA

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE TO CORRECT THE APPLICATION NO. FROM 13,883,290 TO 13,833,290 PREVIOUSLY RECORDED ON REEL 041703 FRAME 0536. ASSIGNOR(S) HEREBY CONFIRMS THE THE ASSIGNMENT AND ASSUMPTION OF SECURITYINTEREST IN PATENTS.;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:048734/0001

Effective date: 20190217

AS Assignment

Owner name: NXP B.V., NETHERLANDS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:050744/0097

Effective date: 20190903

AS Assignment

Owner name: MORGAN STANLEY SENIOR FUNDING, INC., MARYLAND

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE APPLICATION11759915 AND REPLACE IT WITH APPLICATION 11759935 PREVIOUSLY RECORDED ON REEL 037486 FRAME 0517. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT AND ASSUMPTION OF SECURITYINTEREST IN PATENTS;ASSIGNOR:CITIBANK, N.A.;REEL/FRAME:053547/0421

Effective date: 20151207

AS Assignment

Owner name: NXP B.V., NETHERLANDS

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVEAPPLICATION 11759915 AND REPLACE IT WITH APPLICATION11759935 PREVIOUSLY RECORDED ON REEL 040928 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE RELEASE OF SECURITYINTEREST;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:052915/0001

Effective date: 20160622

AS Assignment

Owner name: NXP, B.V. F/K/A FREESCALE SEMICONDUCTOR, INC., NETHERLANDS

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVEAPPLICATION 11759915 AND REPLACE IT WITH APPLICATION11759935 PREVIOUSLY RECORDED ON REEL 040925 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE RELEASE OF SECURITYINTEREST;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:052917/0001

Effective date: 20160912

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8