US8825720B1 - Scaling asynchronous reclamation of free space in de-duplicated multi-controller storage systems - Google Patents

Scaling asynchronous reclamation of free space in de-duplicated multi-controller storage systems Download PDF

Info

Publication number
US8825720B1
US8825720B1 US13/085,377 US201113085377A US8825720B1 US 8825720 B1 US8825720 B1 US 8825720B1 US 201113085377 A US201113085377 A US 201113085377A US 8825720 B1 US8825720 B1 US 8825720B1
Authority
US
United States
Prior art keywords
data
controller
segment
duplicated
controllers
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US13/085,377
Inventor
Ying Xie
Srikant Varadan
Fabiano Botelho
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
EMC Corp
Original Assignee
EMC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by EMC Corp filed Critical EMC Corp
Priority to US13/085,377 priority Critical patent/US8825720B1/en
Assigned to EMC CORPORATION reassignment EMC CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BOTELHO, FABIANO, VARADAN, SRIKANT, XIE, YING
Application granted granted Critical
Publication of US8825720B1 publication Critical patent/US8825720B1/en
Assigned to CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT reassignment CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT SECURITY AGREEMENT Assignors: ASAP SOFTWARE EXPRESS, INC., AVENTAIL LLC, CREDANT TECHNOLOGIES, INC., DELL INTERNATIONAL L.L.C., DELL MARKETING L.P., DELL PRODUCTS L.P., DELL SOFTWARE INC., DELL SYSTEMS CORPORATION, DELL USA L.P., EMC CORPORATION, EMC IP Holding Company LLC, FORCE10 NETWORKS, INC., MAGINATICS LLC, MOZY, INC., SCALEIO LLC, SPANNING CLOUD APPS LLC, WYSE TECHNOLOGY L.L.C.
Assigned to THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT reassignment THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT SECURITY AGREEMENT Assignors: ASAP SOFTWARE EXPRESS, INC., AVENTAIL LLC, CREDANT TECHNOLOGIES, INC., DELL INTERNATIONAL L.L.C., DELL MARKETING L.P., DELL PRODUCTS L.P., DELL SOFTWARE INC., DELL SYSTEMS CORPORATION, DELL USA L.P., EMC CORPORATION, EMC IP Holding Company LLC, FORCE10 NETWORKS, INC., MAGINATICS LLC, MOZY, INC., SCALEIO LLC, SPANNING CLOUD APPS LLC, WYSE TECHNOLOGY L.L.C.
Assigned to EMC IP Holding Company LLC reassignment EMC IP Holding Company LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: EMC CORPORATION
Assigned to THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A. reassignment THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A. SECURITY AGREEMENT Assignors: CREDANT TECHNOLOGIES, INC., DELL INTERNATIONAL L.L.C., DELL MARKETING L.P., DELL PRODUCTS L.P., DELL USA L.P., EMC CORPORATION, EMC IP Holding Company LLC, FORCE10 NETWORKS, INC., WYSE TECHNOLOGY L.L.C.
Assigned to THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A. reassignment THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A. SECURITY AGREEMENT Assignors: CREDANT TECHNOLOGIES INC., DELL INTERNATIONAL L.L.C., DELL MARKETING L.P., DELL PRODUCTS L.P., DELL USA L.P., EMC CORPORATION, EMC IP Holding Company LLC, FORCE10 NETWORKS, INC., WYSE TECHNOLOGY L.L.C.
Assigned to FORCE10 NETWORKS, INC., DELL SOFTWARE INC., DELL USA L.P., AVENTAIL LLC, EMC IP Holding Company LLC, MOZY, INC., CREDANT TECHNOLOGIES, INC., ASAP SOFTWARE EXPRESS, INC., DELL INTERNATIONAL, L.L.C., SCALEIO LLC, DELL PRODUCTS L.P., EMC CORPORATION, MAGINATICS LLC, WYSE TECHNOLOGY L.L.C., DELL SYSTEMS CORPORATION, DELL MARKETING L.P. reassignment FORCE10 NETWORKS, INC. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
Assigned to DELL INTERNATIONAL L.L.C., DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.), EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.), EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC), DELL USA L.P., DELL PRODUCTS L.P., DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.), SCALEIO LLC, DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.) reassignment DELL INTERNATIONAL L.L.C. RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001) Assignors: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Assigned to DELL USA L.P., DELL PRODUCTS L.P., DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.), SCALEIO LLC, EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC), DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.), DELL INTERNATIONAL L.L.C., DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.), EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.) reassignment DELL USA L.P. RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001) Assignors: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G06F17/30156—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/1727—Details of free space management performed by the file system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
    • G06F12/023—Free address space management
    • G06F12/0253—Garbage collection, i.e. reclamation of unreferenced memory

Definitions

  • FIG. 1 is block diagram illustrating a de-duplicated multi-controller file system, embodiment 100 according to the present invention.
  • FIGS. 2 and 3 illustrate embodiments of a process for processing data in a de-duplicated multi-controller file system.
  • FIGS. 4 and 5 present block diagrams illustrating a de-duplicated multi-controller file system in accordance to an embodiment.
  • FIG. 6 is a block diagram illustrating an embodiment of a process for generating a set of contiguous data segments from a data stream.
  • FIG. 7 is a block diagram illustrating hierarchical data structure in accordance with an embodiment.
  • FIG. 8 is a block diagram illustrating a marking process flow in a mark-and-sweep operation.
  • the invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor.
  • these implementations, or any other form that the invention may take, may be referred to as techniques.
  • the order of the steps of disclosed processes may be altered within the scope of the invention.
  • a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task.
  • the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
  • FIG. 1 is a block diagram illustrating a de-duplicated multi-controller file system, embodiment 100 , according to the present invention.
  • the file system 100 includes a plurality of worker controllers 102 each managing one or more storage collection partitions (only one collection partition per controller is shown for simplicity) 104 of the storage space 106 of the file system 100 .
  • the file system 100 also includes at least a master controller 108 .
  • the master controller 108 also functions as a worker controller 102 .
  • the master controller 108 can be a separate controller separate from the worker controllers.
  • the functional divisions of the master and worker controllers in accordance with various embodiments will be discussed below in reference to other Figures.
  • the file system 100 receives data stream 110 transmitted by one or more data sources (e.g., backup servers) 112 and stores the data stream as de-duplicated data segments organized in hierarchical data structures.
  • the de-duplicated data segments are proportioned or allocated across the plurality of worker controllers 102 for storage in the associated storage collection partitions 104 .
  • the plurality of worker controllers 102 each manages one or more collection partitions assigned to the controller and do not share storage space with other worker controllers. One controller does not have access to the storage space of another controller.
  • the master controller is in communication with each of the plurality of worker controllers.
  • FIG. 2 illustrates an embodiment of a process for processing data in a de-duplicated multi-controller file system.
  • a set of de-duplicated data segments organized in a hierarchical data structure is received.
  • the hierarchical data structure represents one or more data files and can be used to re-construct the data files.
  • the hierarchical data structure includes de-duplicated data segments of the data files and one or more successive levels of references leading to the data files.
  • the hierarchical data structure includes a segment tree structure such as a hash tree structure such as a Merkel tree structure.
  • the set of de-duplicated data segments are allocated across a plurality of controllers, where each of the plurality of controllers manages one or more storage collection partitions of the file system, and where each of the de-duplicated data segments and reference(s) referencing the de-duplicated data segment are restricted to be allocated to the same controller.
  • the hierarchical data structure includes a hash tree structure such as a Merkel tree structure.
  • the set of de-duplicated data segments form the bottom level (denoted as L0 level) of the hierarchical data structure.
  • the set of de-duplicated L0 segments are allocated across the plurality of controllers while restricting each of the de-duplicated data segments to be co-allocated to the same controller with reference(s) referencing the de-duplicated L0 segment in the hierarchical data structure.
  • the reference(s) referencing the L0 data segment are stored in metadata segments denoted as L1s.
  • FIG. 3 illustrates an embodiment of a process for processing data in a de-duplicated multi-controller file system.
  • the multi-controller file system shares a single namespace.
  • all files of the file system are traversed, file path are executed.
  • a local controller updates a local live vector to indicate that the data segment is alive.
  • the local controller manages one or more collection partitions (local storage space) associated with the local controller.
  • the local live vector is maintained by the local controller and can only be updated by the local controller.
  • the local controller updates the local live vector whenever a data segment/data chunks under its management is referenced by a file of the file system.
  • a first storage space allocated to a first data segment located in a first collection partition and managed by a first controller is reclaimed by the first controller if a local live vector associated with the first data segment and local to the first controller and the first collection partition indicates that the first data segment is not alive
  • a second storage space allocated to second data segment located in a second collection partition managed by a second controller is reclaimed by the second controller if a local live vector associated with the second data segment and local to the second controller and the second collection partition indicates the second data segment is not alive, and so on so forth.
  • FIG. 4 is a block diagram illustrating a de-duplicated multi-controller file system in accordance to an embodiment. Although a single master is shown, multiple master controllers are possible. Although three worker controllers are shown, any other numbers of worker controllers are possible. The example illustrates the functional division of master and worker controller. Although the master controller is shown as functionally separate from worker controllers, in various embodiments, the master controller and a worker controller can be a single controller as shown in FIG. 1 .
  • one or more data sources 402 such as one or more backup servers transmit data stream (e.g., one or more data files) 404 to a de-duplicated multi-controller file system 406 .
  • the data stream 404 is buffered in a data cache 408 .
  • the data is sent to a data segment generator 410 which breaks the data stream into contiguous data segments.
  • the generated data segments are passed to a reference generator 412 which generates references to each of the contiguous data segments.
  • references to the references to the data segments are generated.
  • cascading levels of references leading to the data segments are generated.
  • the data segments and the references can be organized in a hierarchical data structure such as a segment tree such as a hash tree.
  • the hierarchical data structure includes a Merkel tree.
  • the data segments and the references are then passed to a load balancer/distributor 414 which distributes the data segments and references across a plurality of worker controllers 416 of the file system for storage in associated one or more collection partitions 422 .
  • the distribution may be carried out according to a policy.
  • the policy may be a load balancing policy based on various parameters such as data access pattern, storage capacity, service agreement, quality of service guarantee, data source, and/or data type, etc.
  • the data segments and references are evenly distributed across the plurality of controllers.
  • each de-duplicated data segment and reference(s) referencing the data segment/block are restricted to be allocated in the same worker controller.
  • Each of the worker controllers 416 includes a local segment redundancy filter 418 which filters out duplicate copies of identical data segments in the received data segments to generate a set of de-duplicate data segments.
  • the de-duplicated data segments and references are passed to one or more receiving containers 420 of the local worker controller 416 . Once one or more receiving containers are filled, they are passed to be stored in the associated storage collection partitions 422 . In the example shown, the worker controllers 416 append the content of a receiving container to the end of an append-only log 424 .
  • the de-duplicated data segments and the references are packed in different receiving containers. In various embodiments, the de-duplicated data segments and each reference level of the hierarchical data structure are packed in different receiving containers. In various embodiments the de-duplicated data segments and the corresponding reference(s) referencing the data segments are restricted to be packed in the same receiving container.
  • FIG. 5 is a block diagram illustrating a de-duplicated multi-controller file system in accordance to an embodiment. Although a single master is shown, multiple master controllers are possible. Although three worker controllers are shown, any other numbers of worker controllers are possible. The example illustrates the functional division of master and worker controller. Although the master controller is shown as functionally separate from worker controllers, in various embodiments, the master controller and a worker controller can be a single controller as shown in FIG. 1 .
  • the master controller 502 is shown to include a namespace manager 504 and a segment tree manager 506 .
  • Each worker controller 508 manages one or more corresponding storage collection partitions (only one collection partition per worker controller is shown for simplicity) 510 .
  • Each of the worker controllers includes a local live vector manager 512 that maintains a local live vector 514 and a local garbage collector 516 .
  • the entire file system includes a single unified namespace that contains a logical grouping of unique identifiers for one or more data files.
  • the namespace includes a directory of the file system that contains references to inodes of data files managed under the namespace.
  • An inode of a data file stores the top level reference (root) of hierarchical data structure (e.g., segment tree structure) representing the data file. The top level reference allows the hierarchical data structure to be accessed.
  • the namespace manager, segment tree manager, and local controllers work cooperatively to traverse the entire file system and execute all the file paths of the file system.
  • Local worker controllers each updates a locally maintained live vector to indicate a locally stored data segment or data chunk 518 is alive when the data segment or data chunk is referenced by one or more files managed in the namespace. For example, if data segment A stored in collection partition A managed by controller A is referenced by a data file of the namespace, controller A updates a local live vector A to indicate that segment A is alive.
  • the local live vector A can only be updated by controller A, other controllers are not allowed to update the local live vector A of controller A.
  • Maintaining local live vectors adds to the scalability of the de-duplicated multi-controller file system, since adding on a new controller will localize the computation of updating live vectors for the added controller to the new controller.
  • Existing master and worker controllers will not see their work load increase significantly.
  • a local controller reclaims memory storage spaces (e.g., data segments or chunks) that are not marked alive in a local live vector maintained by the local controller. For example, if data segment A stored in collection partition A managed by controller A is marked as alive by controller A, the garbage collector reclaims the memory storage space and put it back into the common storage system.
  • reclaiming memory storage spaces include copying over only memory storage spaces marked as alive to a new location (e.g., new data container), such as the end of an append-only log.
  • reclaiming memory storage space using a local controller increases the scalability of the de-duplicated multi-controller file system.
  • a new controller and associated storage space is added, existing master and worker controllers do not need to perform additional duties.
  • the added work load in reclaiming operation is localized to the added controller.
  • FIG. 6 is a block diagram illustrating an embodiment of a process for generating a set of contiguous data segments from a data stream.
  • a data stream is received, for example from a backup server.
  • the data stream is broken down into a set of contiguous data segments.
  • the data segments are variable in size.
  • the data segments have fixed size.
  • the data segments are generated based on one or more of the following factors: contents, access pattern, and data type.
  • references are generated for the data segments.
  • the data segments are de-duplicated by eliminating redundant copies of identical data segment.
  • FIG. 7 is a block diagram illustrating hierarchical data structure 702 representing a data stream 704 in accordance with an embodiment.
  • the example shows a Merkel tree having 4 levels of data.
  • the Merkel tree can have more or less than 4 levels of data. In some embodiments it can go up to 6 levels.
  • the example shows a binary Merkel tree having a span of 2 where each parent node includes two child nodes, other Merkel tree configuration where span is greater than 2 is possible.
  • the span for one level may vary from span of another level.
  • the span of one node in a level can be different from the span of another node in the same level.
  • Each node or leave of the Merkel tree shown includes a data segment/block that includes a data portion and metadata portion.
  • the metadata portion includes a hash of the data portion.
  • the hash is also a reference that can serve as a pointer to the location storing the data portion.
  • the metadata portion may also include other information such as the type and size of the data segment/block (node).
  • the data portion of a parent node includes the hash value of its child nodes.
  • node L0(1) of the Merkel tree includes data segment/block S 1 of the data stream as its data portion and a hash of data segment/block S 1 as its reference (L0(1) reference) and as at least a part of its metadata portion. The same happens for L0(2) and other nodes, so on and so forth.
  • Node L1(1) is a parent node of L0(1) and L0(2).
  • the data portion of node L1(1) includes the references for L0(1) and L0(2) (or L0(1) reference and L0(2) reference), where L0(1) reference is a hash of L0(1) data segment/bock and L0(2) reference is a hash of L0(2) data block.
  • the metadata portion of node L1(1) includes reference to L1(1) (also denoted as L1(1) reference).
  • the reference for L1(1) is computed based on the data portion of L1(1) which is L0(1) reference plus L0(2) reference.
  • Node L2(1) is a parent node of L1(1) and L1(2).
  • the data portion of the node L2(1) includes the reference for L1(1) plus reference to L1(2).
  • the metadata portion of L2(1) includes the reference for L2(1) which is a hash of L2(1) data portion.
  • L3 node is the root node or super reference of the Merkel tree.
  • the data portion of node L3 includes the references for L2(1) and L2(2).
  • the metadata portion of L3 includes the reference to L3 which is a hash of content L3 data portion.
  • FIG. 8 is a block diagram illustrating a marking process flow in a mark-and-sweep operation.
  • a master namespace manager identifies the inode of a file to be traversed and from the inode identifies the top level root of the tree structure for the file.
  • the namespace manager passes the top level root to a segment tree manager.
  • the segment tree manager reads the next level references and passes the references to appropriate worker controllers.
  • the appropriate worker controllers receive the references.
  • the worker controllers return the contents back to the master controller.
  • the local worker controllers each update a locally maintained live vector independently and asynchronously to mark all local data segments/data blocks that are referenced by the file as alive based on the references received at the local controller.
  • the local live vector is maintained by the local worker controller and is not accessible by other worker controllers.
  • the local live vector maintains state information regarding whether data segment/blocks local to the local controller and associated storage space are alive.
  • a decision is made as to whether the last segment(s) read are L1 segment(s), where the L1 segments include references to the corresponding child L0 segments? If the last read references are L1 references, the process proceeds to 816 , otherwise it loops back to step 804 .

Abstract

Scaling Asynchronous reclamation of free space in de-duplicate multi-controller storage systems is disclosed. In one example, a set of de-duplicated data segments/blocks organized in a hierarchical data structure is received, the set of de-duplicated data segments/blocks are allocated across a plurality of controllers of the file system, where each de-duplicated data segment is restricted to be co-allocated to the same controller as the reference(s) referencing the de-duplicated data segment/block in the hierarchical data structure. In various embodiments, the file system is traversed, each controller of the file system updates its own local live vector and reclaim storage space that are not marked as alive.

Description

BACKGROUND OF THE INVENTION
Reclaiming unused storage space in de-duplicated file systems is challenging because files share data segments or data chunks. Once a file is deleted, storage space assigned to its data segments or data chunks may not be reclaimed unless they are not shared by other files. Mark-and-sweep garbage collection approach has been used to solve this problem. This approach typically includes the steps of traversing the entire file system, marking all data segments used in the file system as alive, and reclaiming storage space taken up by all unused data segments. However, such an approach is currently not scalable in de-duplicated multi-controller file systems since it is largely a synchronous process and conflicts often arise and must be resolved when multiple controllers are attempting to access the same data segment or data block. As such, an improved method for reclaiming unused storage space in a de-duplicated multi-controller file system is needed.
BRIEF DESCRIPTION OF THE DRAWINGS
Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
FIG. 1 is block diagram illustrating a de-duplicated multi-controller file system, embodiment 100 according to the present invention.
FIGS. 2 and 3 illustrate embodiments of a process for processing data in a de-duplicated multi-controller file system.
FIGS. 4 and 5 present block diagrams illustrating a de-duplicated multi-controller file system in accordance to an embodiment.
FIG. 6 is a block diagram illustrating an embodiment of a process for generating a set of contiguous data segments from a data stream.
FIG. 7 is a block diagram illustrating hierarchical data structure in accordance with an embodiment.
FIG. 8 is a block diagram illustrating a marking process flow in a mark-and-sweep operation.
 DETAILED DESCRIPTION
The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
FIG. 1 is a block diagram illustrating a de-duplicated multi-controller file system, embodiment 100, according to the present invention. The file system 100 includes a plurality of worker controllers 102 each managing one or more storage collection partitions (only one collection partition per controller is shown for simplicity) 104 of the storage space 106 of the file system 100. The file system 100 also includes at least a master controller 108. In the example shown, the master controller 108 also functions as a worker controller 102. In various other embodiments, the master controller 108 can be a separate controller separate from the worker controllers. The functional divisions of the master and worker controllers in accordance with various embodiments will be discussed below in reference to other Figures. The file system 100 receives data stream 110 transmitted by one or more data sources (e.g., backup servers) 112 and stores the data stream as de-duplicated data segments organized in hierarchical data structures. The de-duplicated data segments are proportioned or allocated across the plurality of worker controllers 102 for storage in the associated storage collection partitions 104. The plurality of worker controllers 102 each manages one or more collection partitions assigned to the controller and do not share storage space with other worker controllers. One controller does not have access to the storage space of another controller. The master controller is in communication with each of the plurality of worker controllers.
FIG. 2 illustrates an embodiment of a process for processing data in a de-duplicated multi-controller file system.
At 202, a set of de-duplicated data segments organized in a hierarchical data structure is received. In various embodiments, the hierarchical data structure represents one or more data files and can be used to re-construct the data files. In various embodiments, the hierarchical data structure includes de-duplicated data segments of the data files and one or more successive levels of references leading to the data files. In various embodiments, the hierarchical data structure includes a segment tree structure such as a hash tree structure such as a Merkel tree structure.
At 204, the set of de-duplicated data segments are allocated across a plurality of controllers, where each of the plurality of controllers manages one or more storage collection partitions of the file system, and where each of the de-duplicated data segments and reference(s) referencing the de-duplicated data segment are restricted to be allocated to the same controller.
In various embodiments, the hierarchical data structure includes a hash tree structure such as a Merkel tree structure. In various embodiments, the set of de-duplicated data segments form the bottom level (denoted as L0 level) of the hierarchical data structure. The set of de-duplicated L0 segments are allocated across the plurality of controllers while restricting each of the de-duplicated data segments to be co-allocated to the same controller with reference(s) referencing the de-duplicated L0 segment in the hierarchical data structure. In various embodiments, the reference(s) referencing the L0 data segment are stored in metadata segments denoted as L1s.
Since the majority of computation in traversing data files is carried out at the L0 and L1 level (since there are a lot more of L0 and L1 data segments/blocks than other types of data segments/blocks), limiting L0 and L1 data segments/blocks to the same controller and associated storage space minimizes chattering between controllers and localizes the computation needed to traverse data files to the local controller. This increase the scalability of the file system since adding an additional controller will localize the added computation to the added controller and will minimize burden to the existing master and worker controllers.
FIG. 3 illustrates an embodiment of a process for processing data in a de-duplicated multi-controller file system. The multi-controller file system shares a single namespace. At 302, all files of the file system are traversed, file path are executed. At 304, if a data segment or data chunk is referenced or otherwise used by a data file of the file system, a local controller updates a local live vector to indicate that the data segment is alive. The local controller manages one or more collection partitions (local storage space) associated with the local controller. The local live vector is maintained by the local controller and can only be updated by the local controller. The local controller updates the local live vector whenever a data segment/data chunks under its management is referenced by a file of the file system. At 306, once the entire file system has been traversed and all relevant local live vectors of the file system updated, storage spaces that are not marked alive are reclaimed by local controllers. For example, a first storage space allocated to a first data segment located in a first collection partition and managed by a first controller is reclaimed by the first controller if a local live vector associated with the first data segment and local to the first controller and the first collection partition indicates that the first data segment is not alive, a second storage space allocated to second data segment located in a second collection partition managed by a second controller is reclaimed by the second controller if a local live vector associated with the second data segment and local to the second controller and the second collection partition indicates the second data segment is not alive, and so on so forth.
FIG. 4 is a block diagram illustrating a de-duplicated multi-controller file system in accordance to an embodiment. Although a single master is shown, multiple master controllers are possible. Although three worker controllers are shown, any other numbers of worker controllers are possible. The example illustrates the functional division of master and worker controller. Although the master controller is shown as functionally separate from worker controllers, in various embodiments, the master controller and a worker controller can be a single controller as shown in FIG. 1.
In the example shown, one or more data sources 402 such as one or more backup servers transmit data stream (e.g., one or more data files) 404 to a de-duplicated multi-controller file system 406. The data stream 404 is buffered in a data cache 408. Once the data cache reaches a certain level, the data is sent to a data segment generator 410 which breaks the data stream into contiguous data segments. The generated data segments are passed to a reference generator 412 which generates references to each of the contiguous data segments. In various embodiments, references to the references to the data segments are generated. In various embodiments, cascading levels of references leading to the data segments are generated. The data segments and the references can be organized in a hierarchical data structure such as a segment tree such as a hash tree. In various embodiments the hierarchical data structure includes a Merkel tree. The data segments and the references are then passed to a load balancer/distributor 414 which distributes the data segments and references across a plurality of worker controllers 416 of the file system for storage in associated one or more collection partitions 422. The distribution may be carried out according to a policy. The policy may be a load balancing policy based on various parameters such as data access pattern, storage capacity, service agreement, quality of service guarantee, data source, and/or data type, etc. In various embodiments, the data segments and references are evenly distributed across the plurality of controllers. In various embodiments, each de-duplicated data segment and reference(s) referencing the data segment/block are restricted to be allocated in the same worker controller. Each of the worker controllers 416 includes a local segment redundancy filter 418 which filters out duplicate copies of identical data segments in the received data segments to generate a set of de-duplicate data segments. The de-duplicated data segments and references are passed to one or more receiving containers 420 of the local worker controller 416. Once one or more receiving containers are filled, they are passed to be stored in the associated storage collection partitions 422. In the example shown, the worker controllers 416 append the content of a receiving container to the end of an append-only log 424. In various embodiments, the de-duplicated data segments and the references are packed in different receiving containers. In various embodiments, the de-duplicated data segments and each reference level of the hierarchical data structure are packed in different receiving containers. In various embodiments the de-duplicated data segments and the corresponding reference(s) referencing the data segments are restricted to be packed in the same receiving container.
FIG. 5 is a block diagram illustrating a de-duplicated multi-controller file system in accordance to an embodiment. Although a single master is shown, multiple master controllers are possible. Although three worker controllers are shown, any other numbers of worker controllers are possible. The example illustrates the functional division of master and worker controller. Although the master controller is shown as functionally separate from worker controllers, in various embodiments, the master controller and a worker controller can be a single controller as shown in FIG. 1.
The master controller 502 is shown to include a namespace manager 504 and a segment tree manager 506. Each worker controller 508 manages one or more corresponding storage collection partitions (only one collection partition per worker controller is shown for simplicity) 510. Each of the worker controllers includes a local live vector manager 512 that maintains a local live vector 514 and a local garbage collector 516. The entire file system includes a single unified namespace that contains a logical grouping of unique identifiers for one or more data files. In the example shown, the namespace includes a directory of the file system that contains references to inodes of data files managed under the namespace. An inode of a data file stores the top level reference (root) of hierarchical data structure (e.g., segment tree structure) representing the data file. The top level reference allows the hierarchical data structure to be accessed.
During the mark phase of a mark-and sweep garbage collection operation, the namespace manager, segment tree manager, and local controllers work cooperatively to traverse the entire file system and execute all the file paths of the file system. Local worker controllers each updates a locally maintained live vector to indicate a locally stored data segment or data chunk 518 is alive when the data segment or data chunk is referenced by one or more files managed in the namespace. For example, if data segment A stored in collection partition A managed by controller A is referenced by a data file of the namespace, controller A updates a local live vector A to indicate that segment A is alive. The local live vector A can only be updated by controller A, other controllers are not allowed to update the local live vector A of controller A. Maintaining local live vectors adds to the scalability of the de-duplicated multi-controller file system, since adding on a new controller will localize the computation of updating live vectors for the added controller to the new controller. Existing master and worker controllers will not see their work load increase significantly.
During the memory reclaiming phase of a mark-and-sweep garbage collection operation, a local controller reclaims memory storage spaces (e.g., data segments or chunks) that are not marked alive in a local live vector maintained by the local controller. For example, if data segment A stored in collection partition A managed by controller A is marked as alive by controller A, the garbage collector reclaims the memory storage space and put it back into the common storage system. In various embodiments, reclaiming memory storage spaces (sweeping) include copying over only memory storage spaces marked as alive to a new location (e.g., new data container), such as the end of an append-only log. In various embodiments, reclaiming memory storage space using a local controller increases the scalability of the de-duplicated multi-controller file system. When a new controller and associated storage space is added, existing master and worker controllers do not need to perform additional duties. The added work load in reclaiming operation is localized to the added controller.
FIG. 6 is a block diagram illustrating an embodiment of a process for generating a set of contiguous data segments from a data stream. At 602, a data stream is received, for example from a backup server. At 604, the data stream is broken down into a set of contiguous data segments. In various embodiments, the data segments are variable in size. In various embodiments, the data segments have fixed size. In various embodiments, the data segments are generated based on one or more of the following factors: contents, access pattern, and data type. At 606, references are generated for the data segments. At 608, the data segments are de-duplicated by eliminating redundant copies of identical data segment.
FIG. 7 is a block diagram illustrating hierarchical data structure 702 representing a data stream 704 in accordance with an embodiment. The example shows a Merkel tree having 4 levels of data. In various embodiments, the Merkel tree can have more or less than 4 levels of data. In some embodiments it can go up to 6 levels. Although the example shows a binary Merkel tree having a span of 2 where each parent node includes two child nodes, other Merkel tree configuration where span is greater than 2 is possible. In various embodiments, the span for one level may vary from span of another level. In various embodiments, the span of one node in a level can be different from the span of another node in the same level. Each node or leave of the Merkel tree shown includes a data segment/block that includes a data portion and metadata portion. The metadata portion includes a hash of the data portion. The hash is also a reference that can serve as a pointer to the location storing the data portion. The metadata portion may also include other information such as the type and size of the data segment/block (node). The data portion of a parent node includes the hash value of its child nodes. For example, node L0(1) of the Merkel tree includes data segment/block S1 of the data stream as its data portion and a hash of data segment/block S1 as its reference (L0(1) reference) and as at least a part of its metadata portion. The same happens for L0(2) and other nodes, so on and so forth. Node L1(1) is a parent node of L0(1) and L0(2). The data portion of node L1(1) includes the references for L0(1) and L0(2) (or L0(1) reference and L0(2) reference), where L0(1) reference is a hash of L0(1) data segment/bock and L0(2) reference is a hash of L0(2) data block. The metadata portion of node L1(1) includes reference to L1(1) (also denoted as L1(1) reference). The reference for L1(1) is computed based on the data portion of L1(1) which is L0(1) reference plus L0(2) reference. Node L2(1) is a parent node of L1(1) and L1(2). The data portion of the node L2(1) includes the reference for L1(1) plus reference to L1(2). The metadata portion of L2(1) includes the reference for L2(1) which is a hash of L2(1) data portion. L3 node is the root node or super reference of the Merkel tree. The data portion of node L3 includes the references for L2(1) and L2(2). The metadata portion of L3 includes the reference to L3 which is a hash of content L3 data portion.
FIG. 8 is a block diagram illustrating a marking process flow in a mark-and-sweep operation. At 802, a master namespace manager identifies the inode of a file to be traversed and from the inode identifies the top level root of the tree structure for the file. At 804, the namespace manager passes the top level root to a segment tree manager. At 806, the segment tree manager reads the next level references and passes the references to appropriate worker controllers. At 808, the appropriate worker controllers receive the references. At 810, the worker controllers return the contents back to the master controller. At 812, the local worker controllers each update a locally maintained live vector independently and asynchronously to mark all local data segments/data blocks that are referenced by the file as alive based on the references received at the local controller. The local live vector is maintained by the local worker controller and is not accessible by other worker controllers. The local live vector maintains state information regarding whether data segment/blocks local to the local controller and associated storage space are alive. At 814, a decision is made as to whether the last segment(s) read are L1 segment(s), where the L1 segments include references to the corresponding child L0 segments? If the last read references are L1 references, the process proceeds to 816, otherwise it loops back to step 804. Since this is a marking operation, we only need to know whether a data segment/block is referenced, we do not need to know the actual content of the data segment/block. As such only the references to the actual data segment need to be read. As such the file traversing can stop at reading the L1 segments. At 816, a decision is made as to whether all files of the file system have been executed? If yes, the process ends. If no, the process loops back to step 802.
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.

Claims (21)

What is claimed is:
1. A method of processing data in a de-duplicated multi-controller file system, comprising:
receiving a data stream;
breaking the data stream into a set of continguous data segments;
generating references for each of the set of continguous data segments;
identifying a set of de-duplicated data segments from the set of contiguous data segments;
organizing a hierarchical data structure representing the de-duplicated multi-controller file system in its entirety by a segment tree, wherein:
1) successive levels of references lead to the set of de-duplicated data segments;
2) the set of de-duplicated data segments form a bottom level of the hierarchical data structure; and
3) references referencing a bottom level data L0 segment are stored in a metadata L1 segment on a second-to-bottom level of the hierarchical data structure, such that L1 segments are parents to L0 segments;
proportioning the set of de-duplicated data segments across a plurality of controllers based at least in part by limiting co-allocation of related L0 and L1 segments to a same controller;
reclaiming memory using a mark-and-sweep garbage collection operation such that the single controller only marks local L0 segments previously co-allocated to a specified controller;
wherein each node of the hierarchical data structure includes a portion for storing data and a portion for storing metadata of the data;
and
wherein each of the plurality of controllers do not share storage space with any of the other controllers in the plurality of controllers.
2. The method of claim 1, wherein the hierarchical data structure is a single piece of data.
3. The method of claim 1, wherein the segment tree comprises a Merkle tree.
4. The method of claim 1, wherein each of the plurality of controllers is configured to manage one or more storage collection partitions of the file system.
5. The method of claim 1, wherein proportioning is also based on one or more of the following factors: data access pattern, quality of service guarantee, data size, data type, and backup date.
6. The method of claim 1, wherein each parent node of the hierarchical data structure includes reference(s) to its child node(s).
7. The method of claim 1, wherein the metadata includes a hash of the data.
8. The method of claim 1, wherein the set of de-duplicated data segments are variable in size.
9. The method of claim 1, wherein the file system includes a single namespace that represents all the data files of the file system.
10. The method of claim 1, wherein reclaiming memory comprises:
traversing the entire file system;
for each of a plurality of controllers of the file system updating a local live vector associated with the controller to mark all local live data segments;
wherein a data segment is marked alive if the data segment is referenced
wherein the plurality of controllers do not share storage space; and
wherein the local live vector can only be updated by a local controller.
11. The method of claim 10, wherein a local live vector is local to a controller and can only be updated by the local controller for data segments local to the controller.
12. The method of claim 10, wherein updating the local live vectors are carried out asynchronously by the plurality of controllers for their respective local storage collection partition(s).
13. The method of claim 10, further comprising:
reclaiming the storage space assigned to the data segment if the local live vector associated with the data segment does not mark the data segment as alive.
14. The method of claim 10, wherein reclaiming the storage space includes not copying forward the data segment to another storage location if the data segment is not marked alive.
15. The method of claim 10, wherein reclaiming the storage space includes putting the storage space back to a memory pool for use.
16. The method of claim 10, wherein reclaiming storage space is carried out by a local controller asynchronously from other controllers.
17. A method of processing data in a de-duplicated multi-controller file system, comprising:
receiving a data stream;
breaking the data stream into a set of contiguous data segments;
generating references for the data segments;
organizing the data segments and the references in a hierarchical data structure representing the de-duplicated multi-controller file system in its entirety by a segment tree;
deduplicating the data segments by eliminating duplicate copies of identical copies of the data segments;
wherein a set of de-duplicated data segments form a bottom level of the hierarchical data structure and references referencing a bottom level data L0 segment are stored in a metadata L1 segment on a second-to-bottom level of the hierarchical data structure;
proportioning the de-duplicated data segments across a plurality of controllers based at least in part by a load-balancing policy;
wherein the L0 and L1 segments that are related are restricted to be co-located in a same controller;
wherein each node of the hierarchical data structure includes a portion for storing data and a portion for storing metadata of the data;
and
wherein each of the plurality of controllers do not share storage space with any of the other controllers in the plurality of controllers.
18. A method of processing data in a de-duplicated multi-controller file system, comprising:
traversing the file system;
for each data segment managed by a controller of the file system, updating a local live vector associated with the data segment to indicate that the data segment is alive if the data segment is referenced by a data file of the de-duplicated multi-controller file system;
wherein each data segment is organized in a hierarchical data structure representing the de-duplicated multi-controller file system in its entirety by a segment tree;
wherein a set of de-duplicated data segments form a bottom level of the hierarchical data structure and references referencing a bottom level data L0 segment are stored in a metadata L1 segment on a second-to-bottom level of the hierarchical data structure;
wherein each data segment is proportioned to be managed by the controller based at least in part by a load-balancing policy and limiting co-allocation of related L0 and L1 segments to a same controller;
wherein the controller does not share storage space with any other controllers associated with the de-duplicated multi-controller file system;
wherein each node of the hierarchical data structure includes a portion for storing data and a portion for storing metadata of the data; and
wherein the local live vector is local to the controller and can only be updated by the controller.
19. A method of reclaiming storage space in a de-duplicated multi-controller file system, comprising:
representing the de-duplicated multi-controller file system in its entirety by a segment tree;
wherein a set of de-duplicated data segments form a bottom level of the segment tree and references referencing a bottom level data L0 segment are stored in a metadata L1 segment on a second-to-bottom level of the segment tree;
proportioning de-duplication across a plurality of controllers based at least in part by a load-balancing policy and limiting co-allocation of related L0 and L1 segments to a same controller;
for each of the plurality of controllers of the file system:
assigning one or more storage collection partitions to be managed by the controller;
configuring the controller to reclaim only local storage space that are not marked alive by a local live vector;
wherein the local live vector is local to the controller and can only be updated by the controller for storage spaces local to the controller;
wherein the plurality of controllers are each configured to asynchronously reclaim storage space in one or more collection partitions managed by the controller; and
wherein each of the plurality of controllers do not share storage space with any of the other controllers in the plurality of controllers.
20. A de-duplicated multi-controller file system for processing data, comprising:
a process; and
a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to:
receive a set of de-duplicated data segments organized in a hierarchical data structure representing the de-duplicated multi-controller file system in its entirety by a segment tree and having successive levels of references leading to the set of de-duplicated data segments;
wherein the set of de-duplicated data segments form a bottom level of the hierarchical data structure and references referencing a bottom level data L0 segment are stored in a metadata L1 segment on a second-to-bottom level of the hierarchical data structure;
proportion the set of de-duplicated data segments across a plurality of controllers based at least in part by a load-balancing policy;
reclaim memory using a mark-and-sweep garbage collection operation such that the single controller only marks local L0 segments previously co-allocated to a specified controller;
wherein each node of the hierarchical data structure includes a portion for storing data and a portion for storing metadata of the data;
wherein each of the L0 and L1 segments are restricted to be co-allocated to a same controller; and
wherein each of the plurality of controllers do not share storage space with any of the other controllers in the plurality of controllers.
21. A computer program product for processing data in a de-duplicated multi-controller file system, the computer program product being embodied in a computer readable storage medium and comprising computer instructions for:
receiving a set of de-duplicated data segments organized in a hierarchical data structure representing the de-duplicated multi-controller file system in its entirety by a segment tree and having successive levels of references leading to the set of de-duplicated data segments;
wherein the set of de-duplicated data segments form a bottom level of the hierarchical data structure and references referencing a bottom level data L0 segment are stored in a metadata L1 segment on a second-to-bottom level of the hierarchical data structure;
proportioning the set of de-duplicated data segments across a plurality of controllers based at least in part by a load-balancing policy and limiting co-allocation of related L0 and L1 segments to a same controller;
reclaiming memory using a mark-and-sweep garbage collection operation such that the single controller only marks local L0 segments previously co-allocated to a specified controller;
wherein each node of the hierarchical data structure includes a portion for storing data and a portion for storing metadata of the data;
wherein each of the de-duplicated data segments and all reference(s) referencing the de-duplicated data segment are restricted to be co-allocated to a single controller; and
wherein each of the plurality of controllers do not share storage space with any of the other controllers in the plurality of controllers.
US13/085,377 2011-04-12 2011-04-12 Scaling asynchronous reclamation of free space in de-duplicated multi-controller storage systems Active US8825720B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US13/085,377 US8825720B1 (en) 2011-04-12 2011-04-12 Scaling asynchronous reclamation of free space in de-duplicated multi-controller storage systems

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US13/085,377 US8825720B1 (en) 2011-04-12 2011-04-12 Scaling asynchronous reclamation of free space in de-duplicated multi-controller storage systems

Publications (1)

Publication Number Publication Date
US8825720B1 true US8825720B1 (en) 2014-09-02

Family

ID=51400133

Family Applications (1)

Application Number Title Priority Date Filing Date
US13/085,377 Active US8825720B1 (en) 2011-04-12 2011-04-12 Scaling asynchronous reclamation of free space in de-duplicated multi-controller storage systems

Country Status (1)

Country Link
US (1) US8825720B1 (en)

Cited By (55)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130282676A1 (en) * 2012-03-28 2013-10-24 Quantum Corporation Garbage collection-driven block thinning
CN104361068A (en) * 2014-11-06 2015-02-18 华中科技大学 Method and system for concurrent blocking for data deduplication process
WO2016043757A1 (en) * 2014-09-18 2016-03-24 Hewlett Packard Enterprise Development Lp Data to be backed up in a backup system
US9348531B1 (en) * 2013-09-06 2016-05-24 Western Digital Technologies, Inc. Negative pool management for deduplication
US9361302B1 (en) * 2013-03-04 2016-06-07 Emc Corporation Uniform logic replication for DDFS
US20160210061A1 (en) * 2015-01-21 2016-07-21 Tektronix, Inc. Architecture for a transparently-scalable, ultra-high-throughput storage network
US20160371295A1 (en) * 2015-06-18 2016-12-22 International Business Machines Corporation Removal of reference information for storage blocks in a deduplication system
US20160371294A1 (en) * 2015-06-18 2016-12-22 International Business Machines Corporation Detection of data affected by inaccessible storage blocks in a deduplication system
US9594674B1 (en) * 2014-09-30 2017-03-14 EMC IP Holding Company LLC Method and system for garbage collection of data storage systems using live segment records
US9785643B1 (en) * 2013-09-06 2017-10-10 Veritas Technologies Llc Systems and methods for reclaiming storage space in deduplicating data systems
US9965487B2 (en) 2015-06-18 2018-05-08 International Business Machines Corporation Conversion of forms of user data segment IDs in a deduplication system
US9965488B2 (en) 2015-06-18 2018-05-08 International Business Machines Corporation Back referencing of deduplicated data
US10095759B1 (en) * 2014-01-27 2018-10-09 Microstrategy Incorporated Data engine integration and data refinement
US10095616B2 (en) 2012-03-28 2018-10-09 Quantum Corporation Garbage collection for virtual environments
US10235371B1 (en) * 2016-06-28 2019-03-19 EMC IP Holding Company LLC Incremental physical locality repair for live data
US10310888B2 (en) 2017-10-31 2019-06-04 Rubrik, Inc. Virtual machine linking
US20190272221A1 (en) * 2015-12-30 2019-09-05 Commvault Systems, Inc. System for redirecting requests after a secondary storage computing device failure
US10496313B2 (en) 2014-09-22 2019-12-03 Hewlett Packard Enterprise Development Lp Identification of content-defined chunk boundaries
WO2020159585A1 (en) * 2019-02-01 2020-08-06 EMC IP Holding Company LLC Scalable garbage collection for deduplicated storage
US10789002B1 (en) * 2017-10-23 2020-09-29 EMC IP Holding Company LLC Hybrid data deduplication for elastic cloud storage devices
US10872037B2 (en) 2019-04-19 2020-12-22 EMC IP Holding Company LLC Estimating worker nodes needed for performing garbage collection operations
DE112017000167B4 (en) * 2016-01-12 2020-12-31 International Business Machines Corporation Distributed data deduplication in a processor grid
US10956275B2 (en) 2012-06-13 2021-03-23 Commvault Systems, Inc. Collaborative restore in a networked storage system
US10990518B1 (en) * 2016-03-31 2021-04-27 EMC IP Holding Company LLC Method and system for I/O parallel distributed garbage collection of a deduplicated datasets
US11016696B2 (en) 2018-09-14 2021-05-25 Commvault Systems, Inc. Redundant distributed data storage system
US11016859B2 (en) 2008-06-24 2021-05-25 Commvault Systems, Inc. De-duplication systems and methods for application-specific data
US11113246B2 (en) 2014-10-29 2021-09-07 Commvault Systems, Inc. Accessing a file system using tiered deduplication
US11119911B2 (en) * 2016-03-17 2021-09-14 Alibaba Group Holding Limited Garbage collection method and device
US11119984B2 (en) 2014-03-17 2021-09-14 Commvault Systems, Inc. Managing deletions from a deduplication database
US11157450B2 (en) 2013-01-11 2021-10-26 Commvault Systems, Inc. High availability distributed deduplicated storage system
US11169888B2 (en) 2010-12-14 2021-11-09 Commvault Systems, Inc. Client-side repository in a networked deduplicated storage system
US11226865B2 (en) * 2019-01-18 2022-01-18 EMC IP Holding Company LLC Mostly unique file selection method for deduplication backup systems
WO2022048201A1 (en) * 2020-09-04 2022-03-10 北京沃东天骏信息技术有限公司 Data processing method and apparatus, and electronic device and storage medium
US11288235B2 (en) 2009-07-08 2022-03-29 Commvault Systems, Inc. Synchronized data deduplication
US11301420B2 (en) 2015-04-09 2022-04-12 Commvault Systems, Inc. Highly reusable deduplication database after disaster recovery
US11321189B2 (en) 2014-04-02 2022-05-03 Commvault Systems, Inc. Information management by a media agent in the absence of communications with a storage manager
US11386085B2 (en) 2014-01-27 2022-07-12 Microstrategy Incorporated Deriving metrics from queries
US11392490B2 (en) * 2019-03-29 2022-07-19 EMC IP Holding Company LLC Marking impacted similarity groups in garbage collection operations in deduplicated storage systems
US11422976B2 (en) 2010-12-14 2022-08-23 Commvault Systems, Inc. Distributed deduplicated storage system
US11429499B2 (en) 2016-09-30 2022-08-30 Commvault Systems, Inc. Heartbeat monitoring of virtual machines for initiating failover operations in a data storage management system, including operations by a master monitor node
US11442896B2 (en) 2019-12-04 2022-09-13 Commvault Systems, Inc. Systems and methods for optimizing restoration of deduplicated data stored in cloud-based storage resources
US11449394B2 (en) 2010-06-04 2022-09-20 Commvault Systems, Inc. Failover systems and methods for performing backup operations, including heterogeneous indexing and load balancing of backup and indexing resources
US20220308925A1 (en) * 2021-03-29 2022-09-29 EMC IP Holding Company LLC Scale out deduplicated file system as microservices
US11463264B2 (en) 2019-05-08 2022-10-04 Commvault Systems, Inc. Use of data block signatures for monitoring in an information management system
US11507305B2 (en) 2019-03-29 2022-11-22 EMC IP Holding Company LLC Concurrently performing normal system operations and garbage collection
US11550680B2 (en) 2018-12-06 2023-01-10 Commvault Systems, Inc. Assigning backup resources in a data storage management system based on failover of partnered data storage resources
US11567965B2 (en) 2020-01-23 2023-01-31 Microstrategy Incorporated Enhanced preparation and integration of data sets
US11614970B2 (en) 2019-12-06 2023-03-28 Microstrategy Incorporated High-throughput parallel data transmission
US11645175B2 (en) 2021-02-12 2023-05-09 Commvault Systems, Inc. Automatic failover of a storage manager
US11663099B2 (en) 2020-03-26 2023-05-30 Commvault Systems, Inc. Snapshot-based disaster recovery orchestration of virtual machine failover and failback operations
US11687424B2 (en) 2020-05-28 2023-06-27 Commvault Systems, Inc. Automated media agent state management
US11698727B2 (en) 2018-12-14 2023-07-11 Commvault Systems, Inc. Performing secondary copy operations based on deduplication performance
US11822545B2 (en) 2014-01-27 2023-11-21 Microstrategy Incorporated Search integration
US11829251B2 (en) 2019-04-10 2023-11-28 Commvault Systems, Inc. Restore using deduplicated secondary copy data
US11921715B2 (en) 2014-01-27 2024-03-05 Microstrategy Incorporated Search integration

Citations (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6931487B2 (en) * 2001-10-22 2005-08-16 Hewlett-Packard Development Company L.P. High performance multi-controller processing
US7299325B1 (en) * 2004-06-30 2007-11-20 Sun Microsystems, Inc. Method of garbage collection on a data storage system
US20090037491A1 (en) * 2007-07-30 2009-02-05 International Business Machines Corporation Storage system and method for updating a hash tree
US7519635B1 (en) * 2008-03-31 2009-04-14 International Business Machines Corporation Method of and system for adaptive selection of a deduplication chunking technique
US20090182789A1 (en) * 2003-08-05 2009-07-16 Sepaton, Inc. Scalable de-duplication mechanism
US20090234892A1 (en) * 2008-03-14 2009-09-17 International Business Machines Corporation Method and system for assuring integrity of deduplicated data
US20090259701A1 (en) * 2008-04-14 2009-10-15 Wideman Roderick B Methods and systems for space management in data de-duplication
US20100064166A1 (en) * 2008-09-11 2010-03-11 Nec Laboratories America, Inc. Scalable secondary storage systems and methods
US20100082558A1 (en) * 2008-10-01 2010-04-01 International Business Machines Corporation Policy-based sharing of redundant data across storage pools in a deduplicating system
US20100088296A1 (en) * 2008-10-03 2010-04-08 Netapp, Inc. System and method for organizing data to facilitate data deduplication
US20100094817A1 (en) * 2008-10-14 2010-04-15 Israel Zvi Ben-Shaul Storage-network de-duplication
US20100106933A1 (en) * 2008-10-27 2010-04-29 Netapp, Inc. Method and system for managing storage capacity in a storage network
US20100235333A1 (en) * 2009-03-16 2010-09-16 International Business Machines Corporation Apparatus and method to sequentially deduplicate data
US20100250896A1 (en) * 2009-03-30 2010-09-30 Hi/Fn, Inc. System and method for data deduplication
US7904423B2 (en) * 2005-12-29 2011-03-08 Amazon Technologies, Inc. Distributed storage system with web services client interface
US20110099154A1 (en) * 2009-10-22 2011-04-28 Sun Microsystems, Inc. Data Deduplication Method Using File System Constructs
US7949662B2 (en) * 1995-04-11 2011-05-24 Kinetech, Inc. De-duplication of data in a data processing system
US20110283085A1 (en) * 2010-05-17 2011-11-17 Oracle International Corporation System and method for end-to-end data integrity in a network file system
US8086799B2 (en) * 2008-08-12 2011-12-27 Netapp, Inc. Scalable deduplication of stored data
US8108447B2 (en) * 2010-03-11 2012-01-31 Symantec Corporation Systems and methods for garbage collection in deduplicated data systems
US8108446B1 (en) * 2008-06-27 2012-01-31 Symantec Corporation Methods and systems for managing deduplicated data using unilateral referencing
US8108638B2 (en) * 2009-02-06 2012-01-31 International Business Machines Corporation Backup of deduplicated data
US8131687B2 (en) * 2008-11-13 2012-03-06 International Business Machines Corporation File system with internal deduplication and management of data blocks
US8321648B2 (en) * 2009-10-26 2012-11-27 Netapp, Inc Use of similarity hash to route data for improved deduplication in a storage server cluster
US20120323859A1 (en) * 2011-06-14 2012-12-20 Netapp, Inc. Hierarchical identification and mapping of duplicate data in a storage system
US8380688B2 (en) * 2009-11-06 2013-02-19 International Business Machines Corporation Method and apparatus for data compression

Patent Citations (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7949662B2 (en) * 1995-04-11 2011-05-24 Kinetech, Inc. De-duplication of data in a data processing system
US6931487B2 (en) * 2001-10-22 2005-08-16 Hewlett-Packard Development Company L.P. High performance multi-controller processing
US20090182789A1 (en) * 2003-08-05 2009-07-16 Sepaton, Inc. Scalable de-duplication mechanism
US7299325B1 (en) * 2004-06-30 2007-11-20 Sun Microsystems, Inc. Method of garbage collection on a data storage system
US7904423B2 (en) * 2005-12-29 2011-03-08 Amazon Technologies, Inc. Distributed storage system with web services client interface
US20090037491A1 (en) * 2007-07-30 2009-02-05 International Business Machines Corporation Storage system and method for updating a hash tree
US20090234892A1 (en) * 2008-03-14 2009-09-17 International Business Machines Corporation Method and system for assuring integrity of deduplicated data
US7519635B1 (en) * 2008-03-31 2009-04-14 International Business Machines Corporation Method of and system for adaptive selection of a deduplication chunking technique
US20090259701A1 (en) * 2008-04-14 2009-10-15 Wideman Roderick B Methods and systems for space management in data de-duplication
US8108446B1 (en) * 2008-06-27 2012-01-31 Symantec Corporation Methods and systems for managing deduplicated data using unilateral referencing
US8086799B2 (en) * 2008-08-12 2011-12-27 Netapp, Inc. Scalable deduplication of stored data
US20100064166A1 (en) * 2008-09-11 2010-03-11 Nec Laboratories America, Inc. Scalable secondary storage systems and methods
US20100082558A1 (en) * 2008-10-01 2010-04-01 International Business Machines Corporation Policy-based sharing of redundant data across storage pools in a deduplicating system
US20100088296A1 (en) * 2008-10-03 2010-04-08 Netapp, Inc. System and method for organizing data to facilitate data deduplication
US20100094817A1 (en) * 2008-10-14 2010-04-15 Israel Zvi Ben-Shaul Storage-network de-duplication
US20100106933A1 (en) * 2008-10-27 2010-04-29 Netapp, Inc. Method and system for managing storage capacity in a storage network
US8131687B2 (en) * 2008-11-13 2012-03-06 International Business Machines Corporation File system with internal deduplication and management of data blocks
US8108638B2 (en) * 2009-02-06 2012-01-31 International Business Machines Corporation Backup of deduplicated data
US20100235333A1 (en) * 2009-03-16 2010-09-16 International Business Machines Corporation Apparatus and method to sequentially deduplicate data
US20100250896A1 (en) * 2009-03-30 2010-09-30 Hi/Fn, Inc. System and method for data deduplication
US20110099154A1 (en) * 2009-10-22 2011-04-28 Sun Microsystems, Inc. Data Deduplication Method Using File System Constructs
US8321648B2 (en) * 2009-10-26 2012-11-27 Netapp, Inc Use of similarity hash to route data for improved deduplication in a storage server cluster
US8380688B2 (en) * 2009-11-06 2013-02-19 International Business Machines Corporation Method and apparatus for data compression
US8108447B2 (en) * 2010-03-11 2012-01-31 Symantec Corporation Systems and methods for garbage collection in deduplicated data systems
US20110283085A1 (en) * 2010-05-17 2011-11-17 Oracle International Corporation System and method for end-to-end data integrity in a network file system
US20120323859A1 (en) * 2011-06-14 2012-12-20 Netapp, Inc. Hierarchical identification and mapping of duplicate data in a storage system

Cited By (75)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11016859B2 (en) 2008-06-24 2021-05-25 Commvault Systems, Inc. De-duplication systems and methods for application-specific data
US11288235B2 (en) 2009-07-08 2022-03-29 Commvault Systems, Inc. Synchronized data deduplication
US11449394B2 (en) 2010-06-04 2022-09-20 Commvault Systems, Inc. Failover systems and methods for performing backup operations, including heterogeneous indexing and load balancing of backup and indexing resources
US11169888B2 (en) 2010-12-14 2021-11-09 Commvault Systems, Inc. Client-side repository in a networked deduplicated storage system
US11422976B2 (en) 2010-12-14 2022-08-23 Commvault Systems, Inc. Distributed deduplicated storage system
US10095616B2 (en) 2012-03-28 2018-10-09 Quantum Corporation Garbage collection for virtual environments
US20130282676A1 (en) * 2012-03-28 2013-10-24 Quantum Corporation Garbage collection-driven block thinning
US10956275B2 (en) 2012-06-13 2021-03-23 Commvault Systems, Inc. Collaborative restore in a networked storage system
US11157450B2 (en) 2013-01-11 2021-10-26 Commvault Systems, Inc. High availability distributed deduplicated storage system
US9361302B1 (en) * 2013-03-04 2016-06-07 Emc Corporation Uniform logic replication for DDFS
US9785643B1 (en) * 2013-09-06 2017-10-10 Veritas Technologies Llc Systems and methods for reclaiming storage space in deduplicating data systems
US9348531B1 (en) * 2013-09-06 2016-05-24 Western Digital Technologies, Inc. Negative pool management for deduplication
US11386085B2 (en) 2014-01-27 2022-07-12 Microstrategy Incorporated Deriving metrics from queries
US11921715B2 (en) 2014-01-27 2024-03-05 Microstrategy Incorporated Search integration
US11822545B2 (en) 2014-01-27 2023-11-21 Microstrategy Incorporated Search integration
US11625415B2 (en) 2014-01-27 2023-04-11 Microstrategy Incorporated Data engine integration and data refinement
US10095759B1 (en) * 2014-01-27 2018-10-09 Microstrategy Incorporated Data engine integration and data refinement
US11119984B2 (en) 2014-03-17 2021-09-14 Commvault Systems, Inc. Managing deletions from a deduplication database
US11188504B2 (en) 2014-03-17 2021-11-30 Commvault Systems, Inc. Managing deletions from a deduplication database
US11321189B2 (en) 2014-04-02 2022-05-03 Commvault Systems, Inc. Information management by a media agent in the absence of communications with a storage manager
WO2016043757A1 (en) * 2014-09-18 2016-03-24 Hewlett Packard Enterprise Development Lp Data to be backed up in a backup system
US10496313B2 (en) 2014-09-22 2019-12-03 Hewlett Packard Enterprise Development Lp Identification of content-defined chunk boundaries
US9594674B1 (en) * 2014-09-30 2017-03-14 EMC IP Holding Company LLC Method and system for garbage collection of data storage systems using live segment records
US11921675B2 (en) 2014-10-29 2024-03-05 Commvault Systems, Inc. Accessing a file system using tiered deduplication
US11113246B2 (en) 2014-10-29 2021-09-07 Commvault Systems, Inc. Accessing a file system using tiered deduplication
CN104361068B (en) * 2014-11-06 2017-06-16 华中科技大学 Parallel method of partition and system during a kind of data deduplication
CN104361068A (en) * 2014-11-06 2015-02-18 华中科技大学 Method and system for concurrent blocking for data deduplication process
US20160210061A1 (en) * 2015-01-21 2016-07-21 Tektronix, Inc. Architecture for a transparently-scalable, ultra-high-throughput storage network
US11301420B2 (en) 2015-04-09 2022-04-12 Commvault Systems, Inc. Highly reusable deduplication database after disaster recovery
US20160371294A1 (en) * 2015-06-18 2016-12-22 International Business Machines Corporation Detection of data affected by inaccessible storage blocks in a deduplication system
US20160371295A1 (en) * 2015-06-18 2016-12-22 International Business Machines Corporation Removal of reference information for storage blocks in a deduplication system
US10042854B2 (en) * 2015-06-18 2018-08-07 International Business Machines Corporation Detection of data affected by inaccessible storage blocks in a deduplication system
US9965488B2 (en) 2015-06-18 2018-05-08 International Business Machines Corporation Back referencing of deduplicated data
US10248656B2 (en) * 2015-06-18 2019-04-02 International Business Machines Corporation Removal of reference information for storage blocks in a deduplication system
US9965487B2 (en) 2015-06-18 2018-05-08 International Business Machines Corporation Conversion of forms of user data segment IDs in a deduplication system
US20190272221A1 (en) * 2015-12-30 2019-09-05 Commvault Systems, Inc. System for redirecting requests after a secondary storage computing device failure
US10956286B2 (en) 2015-12-30 2021-03-23 Commvault Systems, Inc. Deduplication replication in a distributed deduplication data storage system
US10877856B2 (en) * 2015-12-30 2020-12-29 Commvault Systems, Inc. System for redirecting requests after a secondary storage computing device failure
DE112017000167B4 (en) * 2016-01-12 2020-12-31 International Business Machines Corporation Distributed data deduplication in a processor grid
US11119911B2 (en) * 2016-03-17 2021-09-14 Alibaba Group Holding Limited Garbage collection method and device
US10990518B1 (en) * 2016-03-31 2021-04-27 EMC IP Holding Company LLC Method and system for I/O parallel distributed garbage collection of a deduplicated datasets
US11003624B2 (en) * 2016-06-28 2021-05-11 EMC IP Holding Company LLC Incremental physical locality repair for live data
US10235371B1 (en) * 2016-06-28 2019-03-19 EMC IP Holding Company LLC Incremental physical locality repair for live data
US11429499B2 (en) 2016-09-30 2022-08-30 Commvault Systems, Inc. Heartbeat monitoring of virtual machines for initiating failover operations in a data storage management system, including operations by a master monitor node
US10789002B1 (en) * 2017-10-23 2020-09-29 EMC IP Holding Company LLC Hybrid data deduplication for elastic cloud storage devices
US10884783B2 (en) 2017-10-31 2021-01-05 Rubrik, Inc. Virtual machine linking
US10528383B2 (en) 2017-10-31 2020-01-07 Rubrik, Inc. Virtual machine linking
US10310888B2 (en) 2017-10-31 2019-06-04 Rubrik, Inc. Virtual machine linking
US10866827B2 (en) 2017-10-31 2020-12-15 Rubrik, Inc. Virtual machine linking
US11016696B2 (en) 2018-09-14 2021-05-25 Commvault Systems, Inc. Redundant distributed data storage system
US11550680B2 (en) 2018-12-06 2023-01-10 Commvault Systems, Inc. Assigning backup resources in a data storage management system based on failover of partnered data storage resources
US11698727B2 (en) 2018-12-14 2023-07-11 Commvault Systems, Inc. Performing secondary copy operations based on deduplication performance
US11226865B2 (en) * 2019-01-18 2022-01-18 EMC IP Holding Company LLC Mostly unique file selection method for deduplication backup systems
GB2594888B (en) * 2019-02-01 2023-07-12 Emc Ip Holding Co Llc Scalable garbage collection for deduplicated storage
IE20190232A1 (en) * 2019-02-01 2020-08-06 Emc Ip Holding Co Llc Scalable garbage collection for deduplicated storage
WO2020159585A1 (en) * 2019-02-01 2020-08-06 EMC IP Holding Company LLC Scalable garbage collection for deduplicated storage
US11068390B2 (en) 2019-02-01 2021-07-20 EMC IP Holding Company LLC Scalable garbage collection for deduplicated storage
CN113366424A (en) * 2019-02-01 2021-09-07 Emc Ip控股有限公司 Scalable garbage collection for deduplication storage
CN113366424B (en) * 2019-02-01 2024-03-08 Emc Ip控股有限公司 Scalable garbage collection for deduplication storage
GB2594888A (en) * 2019-02-01 2021-11-10 Emc Ip Holding Co Llc Scalable garbage collection for deduplicated storage
US11507305B2 (en) 2019-03-29 2022-11-22 EMC IP Holding Company LLC Concurrently performing normal system operations and garbage collection
US11392490B2 (en) * 2019-03-29 2022-07-19 EMC IP Holding Company LLC Marking impacted similarity groups in garbage collection operations in deduplicated storage systems
US11829251B2 (en) 2019-04-10 2023-11-28 Commvault Systems, Inc. Restore using deduplicated secondary copy data
US10872037B2 (en) 2019-04-19 2020-12-22 EMC IP Holding Company LLC Estimating worker nodes needed for performing garbage collection operations
US11409652B2 (en) 2019-04-19 2022-08-09 EMC IP Holding Company LLC Estimating worker nodes needed for performing garbage collection operations
US11463264B2 (en) 2019-05-08 2022-10-04 Commvault Systems, Inc. Use of data block signatures for monitoring in an information management system
US11442896B2 (en) 2019-12-04 2022-09-13 Commvault Systems, Inc. Systems and methods for optimizing restoration of deduplicated data stored in cloud-based storage resources
US11614970B2 (en) 2019-12-06 2023-03-28 Microstrategy Incorporated High-throughput parallel data transmission
US11567965B2 (en) 2020-01-23 2023-01-31 Microstrategy Incorporated Enhanced preparation and integration of data sets
US11663099B2 (en) 2020-03-26 2023-05-30 Commvault Systems, Inc. Snapshot-based disaster recovery orchestration of virtual machine failover and failback operations
US11687424B2 (en) 2020-05-28 2023-06-27 Commvault Systems, Inc. Automated media agent state management
WO2022048201A1 (en) * 2020-09-04 2022-03-10 北京沃东天骏信息技术有限公司 Data processing method and apparatus, and electronic device and storage medium
US11645175B2 (en) 2021-02-12 2023-05-09 Commvault Systems, Inc. Automatic failover of a storage manager
US11893422B2 (en) * 2021-03-29 2024-02-06 EMC IP Holding Company LLC Scale out deduplicated file system as microservices
US20220308925A1 (en) * 2021-03-29 2022-09-29 EMC IP Holding Company LLC Scale out deduplicated file system as microservices

Similar Documents

Publication Publication Date Title
US8825720B1 (en) Scaling asynchronous reclamation of free space in de-duplicated multi-controller storage systems
US11775428B2 (en) Deletion immunity for unreferenced data
US10248336B1 (en) Efficient deletion of shared snapshots
EP3818450B1 (en) Using a storage system to optimize and maintain the metadata associated with a plurality of small files
US10678751B2 (en) Data protection and long term retention
US11301147B2 (en) Adaptive concurrency for write persistence
US10031672B2 (en) Snapshots and clones in a block-based data deduplication storage system
CN109725851B (en) Intelligent snapshot tiering
US9817750B2 (en) Profile-dependent write placement of data into a non-volatile solid-state storage
US11449485B1 (en) Sequence invalidation consolidation in a storage system
EP3814931A1 (en) Fast migration of metadata
US10860738B2 (en) Augmented metadata and signatures for objects in object stores
WO2015034827A1 (en) Replication of snapshots and clones
US10496295B2 (en) Representing a storage array as two or more logical arrays with respective virtual local area networks (VLANS)
US11500817B2 (en) Asynchronous deletion of large directories
US20190325048A1 (en) Transaction encoding and transaction persistence according to type of persistent storages
US20230394010A1 (en) File system metadata deduplication
EP3292473A1 (en) Logical arrays running on physical hardware where reserve physical capacity is available
CN105279166A (en) File management method and system
US20220309046A1 (en) Deduplicating metadata based on a common sequence of chunk identifiers

Legal Events

Date Code Title Description
AS Assignment

Owner name: EMC CORPORATION, MASSACHUSETTS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:XIE, YING;VARADAN, SRIKANT;BOTELHO, FABIANO;REEL/FRAME:026429/0191

Effective date: 20110607

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT, TEXAS

Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040136/0001

Effective date: 20160907

Owner name: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT, NORTH CAROLINA

Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040134/0001

Effective date: 20160907

Owner name: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLAT

Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040134/0001

Effective date: 20160907

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., A

Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040136/0001

Effective date: 20160907

AS Assignment

Owner name: EMC IP HOLDING COMPANY LLC, MASSACHUSETTS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:EMC CORPORATION;REEL/FRAME:040203/0001

Effective date: 20160906

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551)

Year of fee payment: 4

AS Assignment

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., T

Free format text: SECURITY AGREEMENT;ASSIGNORS:CREDANT TECHNOLOGIES, INC.;DELL INTERNATIONAL L.L.C.;DELL MARKETING L.P.;AND OTHERS;REEL/FRAME:049452/0223

Effective date: 20190320

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., TEXAS

Free format text: SECURITY AGREEMENT;ASSIGNORS:CREDANT TECHNOLOGIES, INC.;DELL INTERNATIONAL L.L.C.;DELL MARKETING L.P.;AND OTHERS;REEL/FRAME:049452/0223

Effective date: 20190320

AS Assignment

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., TEXAS

Free format text: SECURITY AGREEMENT;ASSIGNORS:CREDANT TECHNOLOGIES INC.;DELL INTERNATIONAL L.L.C.;DELL MARKETING L.P.;AND OTHERS;REEL/FRAME:053546/0001

Effective date: 20200409

AS Assignment

Owner name: WYSE TECHNOLOGY L.L.C., CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: SCALEIO LLC, MASSACHUSETTS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: MOZY, INC., WASHINGTON

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: MAGINATICS LLC, CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: FORCE10 NETWORKS, INC., CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: EMC IP HOLDING COMPANY LLC, TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: EMC CORPORATION, MASSACHUSETTS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL SYSTEMS CORPORATION, TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL SOFTWARE INC., CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL PRODUCTS L.P., TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL MARKETING L.P., TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL INTERNATIONAL, L.L.C., TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL USA L.P., TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: CREDANT TECHNOLOGIES, INC., TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: AVENTAIL LLC, CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: ASAP SOFTWARE EXPRESS, INC., ILLINOIS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8

AS Assignment

Owner name: SCALEIO LLC, MASSACHUSETTS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC), MASSACHUSETTS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL PRODUCTS L.P., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL INTERNATIONAL L.L.C., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL USA L.P., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

AS Assignment

Owner name: SCALEIO LLC, MASSACHUSETTS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC), MASSACHUSETTS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL PRODUCTS L.P., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL INTERNATIONAL L.L.C., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL USA L.P., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329