US8447740B1 - Stream locality delta compression - Google Patents
Stream locality delta compression Download PDFInfo
- Publication number
- US8447740B1 US8447740B1 US12/291,989 US29198908A US8447740B1 US 8447740 B1 US8447740 B1 US 8447740B1 US 29198908 A US29198908 A US 29198908A US 8447740 B1 US8447740 B1 US 8447740B1
- Authority
- US
- United States
- Prior art keywords
- data segment
- data
- segment
- sketch
- similar
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
- G06F16/1756—De-duplication implemented within the file system, e.g. based on file segments based on delta files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
Abstract
Description
Claims (71)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/291,989 US8447740B1 (en) | 2008-11-14 | 2008-11-14 | Stream locality delta compression |
US13/764,358 US9069785B2 (en) | 2008-11-14 | 2013-02-11 | Stream locality delta compression |
US14/723,196 US9690802B2 (en) | 2008-11-14 | 2015-05-27 | Stream locality delta compression |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/291,989 US8447740B1 (en) | 2008-11-14 | 2008-11-14 | Stream locality delta compression |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/764,358 Continuation US9069785B2 (en) | 2008-11-14 | 2013-02-11 | Stream locality delta compression |
Publications (1)
Publication Number | Publication Date |
---|---|
US8447740B1 true US8447740B1 (en) | 2013-05-21 |
Family
ID=48365422
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/291,989 Active 2029-08-07 US8447740B1 (en) | 2008-11-14 | 2008-11-14 | Stream locality delta compression |
US13/764,358 Active 2029-03-03 US9069785B2 (en) | 2008-11-14 | 2013-02-11 | Stream locality delta compression |
US14/723,196 Active US9690802B2 (en) | 2008-11-14 | 2015-05-27 | Stream locality delta compression |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/764,358 Active 2029-03-03 US9069785B2 (en) | 2008-11-14 | 2013-02-11 | Stream locality delta compression |
US14/723,196 Active US9690802B2 (en) | 2008-11-14 | 2015-05-27 | Stream locality delta compression |
Country Status (1)
Country | Link |
---|---|
US (3) | US8447740B1 (en) |
Cited By (63)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110126026A1 (en) * | 2009-11-25 | 2011-05-26 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US20120059804A1 (en) * | 2010-09-03 | 2012-03-08 | Arm Limited | Data compression and decompression using relative and absolute delta values |
US20120151177A1 (en) * | 2010-12-14 | 2012-06-14 | Microsoft Corporation | Data Deduplication in a Virtualization Environment |
US20130066868A1 (en) * | 2011-09-13 | 2013-03-14 | Exagrid Systems, Inc. | Systems and Methods for Version Chain Clustering |
US20130304746A1 (en) * | 2012-01-31 | 2013-11-14 | Cleversafe, Inc. | Retrieving indexed data from a dispersed storage network |
US20140258248A1 (en) * | 2013-03-06 | 2014-09-11 | Dell Products, Lp | Delta Compression of Probabilistically Clustered Chunks of Data |
US20140289208A1 (en) * | 2013-03-21 | 2014-09-25 | Fujitsu Limited | Data compression apparatus, data compression method, data decompression apparatus, and data decompression method |
US20150019797A1 (en) * | 2013-07-14 | 2015-01-15 | CNEXLABS, Inc. | Method and Apparatus for Providing Improved Garbage Collection Process In Solid State Drive |
US20150074291A1 (en) * | 2005-09-29 | 2015-03-12 | Silver Peak Systems, Inc. | Systems and methods for compressing packet data by predicting subsequent data |
US8990171B2 (en) | 2011-09-01 | 2015-03-24 | Microsoft Corporation | Optimization of a partially deduplicated file |
US20150120749A1 (en) * | 2013-10-30 | 2015-04-30 | Microsoft Corporation | Data management for connected devices |
US20150134623A1 (en) * | 2011-02-17 | 2015-05-14 | Jitcomm Networks Pte Ltd | Parallel data partitioning |
US20150234616A1 (en) * | 2009-06-25 | 2015-08-20 | Emc Corporation | System and method for providing long-term storage for data |
US9253277B2 (en) | 2007-07-05 | 2016-02-02 | Silver Peak Systems, Inc. | Pre-fetching stored data from a memory |
US9280550B1 (en) * | 2010-12-31 | 2016-03-08 | Emc Corporation | Efficient storage tiering |
US9363248B1 (en) | 2005-08-12 | 2016-06-07 | Silver Peak Systems, Inc. | Data encryption in a network memory architecture for providing data based on local accessibility |
US9367557B1 (en) | 2013-09-26 | 2016-06-14 | Emc Corporation | System and method for improving data compression |
US9397951B1 (en) | 2008-07-03 | 2016-07-19 | Silver Peak Systems, Inc. | Quality of service using multiple flows |
US9411815B1 (en) | 2013-09-26 | 2016-08-09 | Emc Corporation | System and method for improving data compression in a deduplicated storage system |
EP3062227A1 (en) * | 2015-02-27 | 2016-08-31 | Exagrid Systems, Inc. | Scalable grid deduplication |
US9438538B2 (en) | 2006-08-02 | 2016-09-06 | Silver Peak Systems, Inc. | Data matching using flow based packet data storage |
US9483494B1 (en) * | 2013-03-14 | 2016-11-01 | Emc Corporation | Opportunistic fragmentation repair |
US9514146B1 (en) | 2013-09-26 | 2016-12-06 | Emc Corporation | System and method for improving data compression of a storage system in an online manner |
US20160364295A1 (en) * | 2009-11-25 | 2016-12-15 | International Business Machines Corporation | Adaptive dispersed storage network (dsn) and system |
US9549048B1 (en) * | 2005-09-29 | 2017-01-17 | Silver Peak Systems, Inc. | Transferring compressed packet data over a network |
US9571698B1 (en) * | 2012-03-30 | 2017-02-14 | EMC IP Holding Company LLC | Method and system for dynamic compression module selection |
US9584403B2 (en) | 2006-08-02 | 2017-02-28 | Silver Peak Systems, Inc. | Communications scheduler |
US9594753B1 (en) * | 2013-03-14 | 2017-03-14 | EMC IP Holding Company LLC | Fragmentation repair of synthetic backups |
US9613071B1 (en) | 2007-11-30 | 2017-04-04 | Silver Peak Systems, Inc. | Deferred data storage |
US9626224B2 (en) | 2011-11-03 | 2017-04-18 | Silver Peak Systems, Inc. | Optimizing available computing resources within a virtual environment |
US20170123908A1 (en) * | 2015-10-30 | 2017-05-04 | International Business Machines Corporation | Optimizing secondary storage in a dispersed storage network |
US9712463B1 (en) | 2005-09-29 | 2017-07-18 | Silver Peak Systems, Inc. | Workload optimization in a wide area network utilizing virtual switches |
US9717021B2 (en) | 2008-07-03 | 2017-07-25 | Silver Peak Systems, Inc. | Virtual network overlay |
US9843802B1 (en) * | 2012-03-30 | 2017-12-12 | EMC IP Holding Company LLC | Method and system for dynamic compression module selection |
US9843702B1 (en) * | 2012-03-30 | 2017-12-12 | EMC IP Holding Company LLC | Method and system for dynamic compression module selection |
US9875344B1 (en) | 2014-09-05 | 2018-01-23 | Silver Peak Systems, Inc. | Dynamic monitoring and authorization of an optimization device |
US9906630B2 (en) | 2011-10-14 | 2018-02-27 | Silver Peak Systems, Inc. | Processing data packets in performance enhancing proxy (PEP) environment |
US9948496B1 (en) | 2014-07-30 | 2018-04-17 | Silver Peak Systems, Inc. | Determining a transit appliance for data traffic to a software service |
US9967056B1 (en) | 2016-08-19 | 2018-05-08 | Silver Peak Systems, Inc. | Forward packet recovery with constrained overhead |
US20180232387A1 (en) * | 2017-02-15 | 2018-08-16 | Paypal, Inc. | Data transfer size reduction |
US10073855B2 (en) | 2015-05-21 | 2018-09-11 | Exagrid Systems, Inc. | Dynamic and optimized management of grid system resources |
US10114831B2 (en) | 2012-08-16 | 2018-10-30 | Exagrid Systems, Inc. | Delta version clustering and re-anchoring |
US10164861B2 (en) | 2015-12-28 | 2018-12-25 | Silver Peak Systems, Inc. | Dynamic monitoring and visualization for network health characteristics |
US10216434B2 (en) * | 2015-12-30 | 2019-02-26 | International Business Machines Corporation | Detailed memory device statistics with drive write location determination |
US10216754B1 (en) * | 2013-09-26 | 2019-02-26 | EMC IP Holding Company LLC | System and method for balancing compression and read performance in a storage system |
US10257082B2 (en) | 2017-02-06 | 2019-04-09 | Silver Peak Systems, Inc. | Multi-level learning for classifying traffic flows |
US10303656B2 (en) | 2015-08-13 | 2019-05-28 | Exagrid Systems, Inc. | Parallelizing and deduplicating backup data |
US10318159B1 (en) * | 2017-06-14 | 2019-06-11 | EMC IP Holding Company LLC | Method and system for physical locality repair in a storage system |
US10394757B2 (en) | 2010-11-18 | 2019-08-27 | Microsoft Technology Licensing, Llc | Scalable chunk store for data deduplication |
US10432484B2 (en) | 2016-06-13 | 2019-10-01 | Silver Peak Systems, Inc. | Aggregating select network traffic statistics |
US10637721B2 (en) | 2018-03-12 | 2020-04-28 | Silver Peak Systems, Inc. | Detecting path break conditions while minimizing network overhead |
US10771394B2 (en) | 2017-02-06 | 2020-09-08 | Silver Peak Systems, Inc. | Multi-level learning for classifying traffic flows on a first packet from DNS data |
US10805840B2 (en) | 2008-07-03 | 2020-10-13 | Silver Peak Systems, Inc. | Data transmission via a virtual wide area network overlay |
US20200326869A1 (en) * | 2019-04-12 | 2020-10-15 | Samsung Electronics Co., Ltd. | Half-match deduplication |
US10838990B1 (en) | 2013-09-26 | 2020-11-17 | EMC IP Holding Company LLC | System and method for improving data compression of a storage system using coarse and fine grained similarity |
US10892978B2 (en) | 2017-02-06 | 2021-01-12 | Silver Peak Systems, Inc. | Multi-level learning for classifying traffic flows from first packet data |
US10938961B1 (en) | 2019-12-18 | 2021-03-02 | Ndata, Inc. | Systems and methods for data deduplication by generating similarity metrics using sketch computation |
US11044202B2 (en) | 2017-02-06 | 2021-06-22 | Silver Peak Systems, Inc. | Multi-level learning for predicting and classifying traffic flows from first packet data |
US11119995B2 (en) * | 2019-12-18 | 2021-09-14 | Ndata, Inc. | Systems and methods for sketch computation |
US11150997B2 (en) | 2015-08-19 | 2021-10-19 | Exagrid Systems, Inc. | Adaptive bandwidth management of a replication process |
US11212210B2 (en) | 2017-09-21 | 2021-12-28 | Silver Peak Systems, Inc. | Selective route exporting using source type |
US11336295B2 (en) | 2011-09-13 | 2022-05-17 | Exagrid Systems, Inc. | Systems and methods for version chain clustering |
US20230267041A1 (en) * | 2014-09-08 | 2023-08-24 | Pure Storage, Inc. | Selecting Storage Units Based on Storage Pool Traits |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8447740B1 (en) * | 2008-11-14 | 2013-05-21 | Emc Corporation | Stream locality delta compression |
US8719234B2 (en) * | 2012-01-25 | 2014-05-06 | International Business Machines Corporation | Handling rewrites in deduplication systems using data parsers |
US10146787B2 (en) * | 2013-07-26 | 2018-12-04 | Quest Software Inc. | Transferring differences between chunks during replication |
IN2013MU04013A (en) * | 2013-12-23 | 2015-08-07 | Tata Consultancy Services Ltd | |
US20170060769A1 (en) * | 2014-05-01 | 2017-03-02 | Coho Data, Inc. | Systems, devices and methods for generating locality-indicative data representations of data streams, and compressions thereof |
EP3659043B1 (en) * | 2017-08-25 | 2023-02-22 | Huawei Technologies Co., Ltd. | Apparatus and method for storing received data blocks as deduplicated data blocks |
US10419022B2 (en) | 2017-11-30 | 2019-09-17 | International Business Machines Corporation | Run-length base-delta encoding for high-speed compression |
Citations (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5574906A (en) * | 1994-10-24 | 1996-11-12 | International Business Machines Corporation | System and method for reducing storage requirement in backup subsystems utilizing segmented compression and differencing |
US5990810A (en) * | 1995-02-17 | 1999-11-23 | Williams; Ross Neil | Method for partitioning a block of data into subblocks and for storing and communcating such subblocks |
US6349296B1 (en) * | 1998-03-26 | 2002-02-19 | Altavista Company | Method for clustering closely resembling data objects |
US6374250B2 (en) * | 1997-02-03 | 2002-04-16 | International Business Machines Corporation | System and method for differential compression of data from a plurality of binary sources |
US20030097359A1 (en) * | 2001-11-02 | 2003-05-22 | Thomas Ruediger | Deduplicaiton system |
US6658423B1 (en) * | 2001-01-24 | 2003-12-02 | Google, Inc. | Detecting duplicate and near-duplicate files |
WO2004012338A2 (en) | 2002-07-31 | 2004-02-05 | Btg International Limited | Lossless data compression |
US20050044294A1 (en) | 2003-07-17 | 2005-02-24 | Vo Binh Dao | Method and apparatus for window matching in delta compressors |
US20050086246A1 (en) * | 2003-09-04 | 2005-04-21 | Oracle International Corporation | Database performance baselines |
US20050131939A1 (en) * | 2003-12-16 | 2005-06-16 | International Business Machines Corporation | Method and apparatus for data redundancy elimination at the block level |
US20050182780A1 (en) * | 2004-02-17 | 2005-08-18 | Forman George H. | Data de-duplication |
US20050219075A1 (en) * | 2004-03-18 | 2005-10-06 | Storer James A | In-place differential compression |
EP1641219A2 (en) * | 2004-09-24 | 2006-03-29 | Microsoft Corporation | Efficient algorithm for finding candidate objects for remote differential compression |
US20060101060A1 (en) | 2004-11-08 | 2006-05-11 | Kai Li | Similarity search system with compact data structures |
US20070239946A1 (en) | 2006-04-11 | 2007-10-11 | Data Domain, Inc. | Efficient data storage using two level delta resemblance |
US7287019B2 (en) * | 2003-06-04 | 2007-10-23 | Microsoft Corporation | Duplicate data elimination system |
US20080013830A1 (en) * | 2006-07-11 | 2008-01-17 | Data Domain, Inc. | Locality-based stream segmentation for data deduplication |
US20080109428A1 (en) | 2006-11-07 | 2008-05-08 | University Of Washington | Efficient top-k query evaluation on probabilistic data |
US20080133561A1 (en) | 2006-12-01 | 2008-06-05 | Nec Laboratories America, Inc. | Methods and systems for quick and efficient data management and/or processing |
US20080256143A1 (en) * | 2007-04-11 | 2008-10-16 | Data Domain, Inc. | Cluster storage using subsegmenting |
US7567188B1 (en) * | 2008-04-10 | 2009-07-28 | International Business Machines Corporation | Policy based tiered data deduplication strategy |
US20090193223A1 (en) * | 2008-01-24 | 2009-07-30 | George Saliba | Methods and systems for vectored data de-duplication |
US20090234892A1 (en) * | 2008-03-14 | 2009-09-17 | International Business Machines Corporation | Method and system for assuring integrity of deduplicated data |
US7669023B2 (en) * | 2007-07-10 | 2010-02-23 | Hitachi, Ltd. | Power efficient storage with data de-duplication |
US7747584B1 (en) * | 2006-08-22 | 2010-06-29 | Netapp, Inc. | System and method for enabling de-duplication in a storage system architecture |
US7921077B2 (en) * | 2006-06-29 | 2011-04-05 | Netapp, Inc. | System and method for managing data deduplication of storage systems utilizing persistent consistency point images |
US7949662B2 (en) * | 1995-04-11 | 2011-05-24 | Kinetech, Inc. | De-duplication of data in a data processing system |
US8190835B1 (en) * | 2007-12-31 | 2012-05-29 | Emc Corporation | Global de-duplication in shared architectures |
Family Cites Families (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6667700B1 (en) | 2002-10-30 | 2003-12-23 | Nbt Technology, Inc. | Content-based segmentation scheme for data compression in storage and transmission including hierarchical segment representation |
US20080104146A1 (en) * | 2006-10-31 | 2008-05-01 | Rebit, Inc. | System for automatically shadowing encrypted data and file directory structures for a plurality of network-connected computers using a network-attached memory with single instance storage |
EP2078269A4 (en) * | 2006-10-31 | 2011-02-23 | Rebit Inc | System for automatically shadowing data and file directory structures that are recorded on a computer memory |
US20080195597A1 (en) * | 2007-02-08 | 2008-08-14 | Samsung Electronics Co., Ltd. | Searching in peer-to-peer networks |
US8046509B2 (en) * | 2007-07-06 | 2011-10-25 | Prostor Systems, Inc. | Commonality factoring for removable media |
US20090204636A1 (en) * | 2008-02-11 | 2009-08-13 | Microsoft Corporation | Multimodal object de-duplication |
US8352540B2 (en) * | 2008-03-06 | 2013-01-08 | International Business Machines Corporation | Distinguishing data streams to enhance data storage efficiency |
US8825617B2 (en) | 2008-03-14 | 2014-09-02 | International Business Machines Corporation | Limiting deduplication based on predetermined criteria |
US7519635B1 (en) | 2008-03-31 | 2009-04-14 | International Business Machines Corporation | Method of and system for adaptive selection of a deduplication chunking technique |
US8645333B2 (en) * | 2008-05-29 | 2014-02-04 | International Business Machines Corporation | Method and apparatus to minimize metadata in de-duplication |
US8108353B2 (en) * | 2008-06-11 | 2012-01-31 | International Business Machines Corporation | Method and apparatus for block size optimization in de-duplication |
US8832034B1 (en) * | 2008-07-03 | 2014-09-09 | Riverbed Technology, Inc. | Space-efficient, revision-tolerant data de-duplication |
US7913114B2 (en) * | 2008-07-31 | 2011-03-22 | Quantum Corporation | Repair of a corrupt data segment used by a de-duplication engine |
US7814149B1 (en) * | 2008-09-29 | 2010-10-12 | Symantec Operating Corporation | Client side data deduplication |
US8082228B2 (en) | 2008-10-31 | 2011-12-20 | Netapp, Inc. | Remote office duplication |
US8447740B1 (en) * | 2008-11-14 | 2013-05-21 | Emc Corporation | Stream locality delta compression |
-
2008
- 2008-11-14 US US12/291,989 patent/US8447740B1/en active Active
-
2013
- 2013-02-11 US US13/764,358 patent/US9069785B2/en active Active
-
2015
- 2015-05-27 US US14/723,196 patent/US9690802B2/en active Active
Patent Citations (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5574906A (en) * | 1994-10-24 | 1996-11-12 | International Business Machines Corporation | System and method for reducing storage requirement in backup subsystems utilizing segmented compression and differencing |
US5990810A (en) * | 1995-02-17 | 1999-11-23 | Williams; Ross Neil | Method for partitioning a block of data into subblocks and for storing and communcating such subblocks |
US7949662B2 (en) * | 1995-04-11 | 2011-05-24 | Kinetech, Inc. | De-duplication of data in a data processing system |
US6374250B2 (en) * | 1997-02-03 | 2002-04-16 | International Business Machines Corporation | System and method for differential compression of data from a plurality of binary sources |
US6349296B1 (en) * | 1998-03-26 | 2002-02-19 | Altavista Company | Method for clustering closely resembling data objects |
US6658423B1 (en) * | 2001-01-24 | 2003-12-02 | Google, Inc. | Detecting duplicate and near-duplicate files |
US20030097359A1 (en) * | 2001-11-02 | 2003-05-22 | Thomas Ruediger | Deduplicaiton system |
WO2004012338A2 (en) | 2002-07-31 | 2004-02-05 | Btg International Limited | Lossless data compression |
US7287019B2 (en) * | 2003-06-04 | 2007-10-23 | Microsoft Corporation | Duplicate data elimination system |
US20050044294A1 (en) | 2003-07-17 | 2005-02-24 | Vo Binh Dao | Method and apparatus for window matching in delta compressors |
US20050086246A1 (en) * | 2003-09-04 | 2005-04-21 | Oracle International Corporation | Database performance baselines |
US20050131939A1 (en) * | 2003-12-16 | 2005-06-16 | International Business Machines Corporation | Method and apparatus for data redundancy elimination at the block level |
US20050182780A1 (en) * | 2004-02-17 | 2005-08-18 | Forman George H. | Data de-duplication |
US20050219075A1 (en) * | 2004-03-18 | 2005-10-06 | Storer James A | In-place differential compression |
EP1641219A2 (en) * | 2004-09-24 | 2006-03-29 | Microsoft Corporation | Efficient algorithm for finding candidate objects for remote differential compression |
US20060101060A1 (en) | 2004-11-08 | 2006-05-11 | Kai Li | Similarity search system with compact data structures |
US20070239946A1 (en) | 2006-04-11 | 2007-10-11 | Data Domain, Inc. | Efficient data storage using two level delta resemblance |
US7921077B2 (en) * | 2006-06-29 | 2011-04-05 | Netapp, Inc. | System and method for managing data deduplication of storage systems utilizing persistent consistency point images |
US20080013830A1 (en) * | 2006-07-11 | 2008-01-17 | Data Domain, Inc. | Locality-based stream segmentation for data deduplication |
US7747584B1 (en) * | 2006-08-22 | 2010-06-29 | Netapp, Inc. | System and method for enabling de-duplication in a storage system architecture |
US20080109428A1 (en) | 2006-11-07 | 2008-05-08 | University Of Washington | Efficient top-k query evaluation on probabilistic data |
US20080133561A1 (en) | 2006-12-01 | 2008-06-05 | Nec Laboratories America, Inc. | Methods and systems for quick and efficient data management and/or processing |
US20080256143A1 (en) * | 2007-04-11 | 2008-10-16 | Data Domain, Inc. | Cluster storage using subsegmenting |
US7669023B2 (en) * | 2007-07-10 | 2010-02-23 | Hitachi, Ltd. | Power efficient storage with data de-duplication |
US8190835B1 (en) * | 2007-12-31 | 2012-05-29 | Emc Corporation | Global de-duplication in shared architectures |
US20090193223A1 (en) * | 2008-01-24 | 2009-07-30 | George Saliba | Methods and systems for vectored data de-duplication |
US20090234892A1 (en) * | 2008-03-14 | 2009-09-17 | International Business Machines Corporation | Method and system for assuring integrity of deduplicated data |
US7567188B1 (en) * | 2008-04-10 | 2009-07-28 | International Business Machines Corporation | Policy based tiered data deduplication strategy |
Non-Patent Citations (27)
Title |
---|
Ajtai et al. "Compactly Encoding Unstructured Inputs with Differential Compression" J. ACM 49, 3. pp. 318-367. May 2002. |
Andrei Z. Broder "Identifying and Filtering Near-Duplicate Documents", COM '00: Proceedings of the 11th Annual Symposium on Combinatorial Pattern Matching. pp. 1-10, Springer-Verlag, 2000. |
Andrei Z. Broder, "Algorithms for Duplicate Documents", Lecture Notes, Feb. 18, 2005. |
Andrei Z. Broder, "On the Resemblance and Containment of Documents", In Proceedings of Compression and Complexity of Sequences 1997. |
Andrew Tridgell "Efficient Algorithms for Sorting and Synchronization". Doctor of Philosophy Thesis, The Australian National University, Feb. 1999. |
Bobbarjung et al. "Improving Duplicate Elimination in Storage Systems", ACM Transactions on Storage, vol. 2, No. 4, Nov. 2006, pp. 424-448. |
Broder et al. "Min-Wise Independent Permutations", Journal of Computer Systems and Sciences, vol. 60(3), pp. 630-659 (2000) (special issue for STOC '98), preliminary version in Proceedings of the 30th Annual ACM Symposium on Theory of Computing (1998). |
Broder et al. "On the Computation of the Minima of Random Functions ", paper submitted for the conference: Algorithm Engineering, 3rd International Workshop, WAE '99, London, UK, Jul. 19-21, 1999. |
Broder et al. "Syntactic Clustering of the Web", Computer Networks. 29(8-13): 1157-1166. 1997. http://www.std.org/~msm/common/clustering.html. |
Broder et al. "Syntactic Clustering of the Web", Computer Networks. 29(8-13): 1157-1166. 1997. http://www.std.org/˜msm/common/clustering.html. |
Chan et al. "Cache-Based Compaction: A New Technique for Optimizing Web Transfer", Mobile and Wireless Internet: Protocols, Algorithms and Systems, edited by Kia Makki et. al. Kluwer Academic Publishers, Jul. 2003. |
Douglis et al. "Application-Specific Delta-Encoding via Resemblance Detection", In Proc. of the USENIX Annual Technical Conference, Jun. 2003. |
Fred Douglis et al, "Application-specific Delta-encoding via Resemblance Detection", IBM T.J. Watson Research Center, 2002. * |
Jain et al. "TAPER: Tiered Approach for Eliminating Redundancy in Replica Synchronization ", 4th USENIX Conference on File and Storage Technologies (FAST '05), Dec. 14-16, 2005, San Francisco, CA. |
Joshua P. MacDonald, "File System Support for Delta Compression", Masters thesis. Department of Electrical Engineering and Computer Science, University of California at Berkeley, 2000. |
Karp et al. "Efficient randomized pattern-matching algorithms", IBM Journal of Research and Development 31 (2), 249-260. Mar. 1987. |
Kulkarni et al. "Redundancy Elimination Within Large Collections of Files" https://iweb.datadomain.com/princeton/files/bib/douglis04.pdf, USENIX Annual Technical Conference, Boston, MA, Jun. 2004. |
Monika Henzinger, "Finding Near-Duplicate Web Pages: A Large-Scale Evaluation of Algorithms", In 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, vol. 2006, pp. 284-291, 2006. |
Moses S. Charikar, "Similarity Estimation Techniques from Rounding Algorithms", Proceedings of the 34th Annual ACM Symposium on Theory of Computing, pp. 380-388. 2002. |
Muthitacharoen et al. "A Low-bandwidth Network File System ", In Proceedings of the ACM 18th Symposium on Operating Systems Principles. Banff, Canada. Oct. 2001. |
Nevin Heintze, Scalable Document Fingerprinting (Extended Abstract), Bell Laboratories, 1996. * |
Promod Mandagere et al, "Data De-duplication using Object Based Storage", University of Mennesota Digital Technology Center Intelligent Storage Consortium, May 18, 2007. * |
Randal C. Burns, "Differential Compression: A Generalized Solution for Binary Files" MS Thesis, University of California at Santa Cruz, 1997. |
Spring et al. "A Protocol-Independent Technique for Eliminating Redundant Network Traffic", In Proc. of ACM SIGCOMM, pp. 87-95, Aug. 2000. |
Suel et al. "Algorithms for Delta Compression and Remote File Synchronization", In Khalid Sayood, editor, Lossless Compression Handbook. Academic Press, 2002. |
Trendafilov et al. "Zdelta: An Efficient Delta Compression Tool", Technical Report TR-CIS-2002-02, Polytechnic University, Jun. 26, 2002. |
Udi Manber, "Finding Similar Files in a Large File System", In Proceedings of the USENIX Winter 1994 Technical Conference, San Francisco, CA, USA, 1994. |
Cited By (130)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9363248B1 (en) | 2005-08-12 | 2016-06-07 | Silver Peak Systems, Inc. | Data encryption in a network memory architecture for providing data based on local accessibility |
US10091172B1 (en) | 2005-08-12 | 2018-10-02 | Silver Peak Systems, Inc. | Data encryption in a network memory architecture for providing data based on local accessibility |
US9712463B1 (en) | 2005-09-29 | 2017-07-18 | Silver Peak Systems, Inc. | Workload optimization in a wide area network utilizing virtual switches |
US20150074291A1 (en) * | 2005-09-29 | 2015-03-12 | Silver Peak Systems, Inc. | Systems and methods for compressing packet data by predicting subsequent data |
US9549048B1 (en) * | 2005-09-29 | 2017-01-17 | Silver Peak Systems, Inc. | Transferring compressed packet data over a network |
US9363309B2 (en) * | 2005-09-29 | 2016-06-07 | Silver Peak Systems, Inc. | Systems and methods for compressing packet data by predicting subsequent data |
US9438538B2 (en) | 2006-08-02 | 2016-09-06 | Silver Peak Systems, Inc. | Data matching using flow based packet data storage |
US9584403B2 (en) | 2006-08-02 | 2017-02-28 | Silver Peak Systems, Inc. | Communications scheduler |
US9961010B2 (en) | 2006-08-02 | 2018-05-01 | Silver Peak Systems, Inc. | Communications scheduler |
US9253277B2 (en) | 2007-07-05 | 2016-02-02 | Silver Peak Systems, Inc. | Pre-fetching stored data from a memory |
US9613071B1 (en) | 2007-11-30 | 2017-04-04 | Silver Peak Systems, Inc. | Deferred data storage |
US10805840B2 (en) | 2008-07-03 | 2020-10-13 | Silver Peak Systems, Inc. | Data transmission via a virtual wide area network overlay |
US11412416B2 (en) | 2008-07-03 | 2022-08-09 | Hewlett Packard Enterprise Development Lp | Data transmission via bonded tunnels of a virtual wide area network overlay |
US10313930B2 (en) | 2008-07-03 | 2019-06-04 | Silver Peak Systems, Inc. | Virtual wide area network overlays |
US9397951B1 (en) | 2008-07-03 | 2016-07-19 | Silver Peak Systems, Inc. | Quality of service using multiple flows |
US11419011B2 (en) | 2008-07-03 | 2022-08-16 | Hewlett Packard Enterprise Development Lp | Data transmission via bonded tunnels of a virtual wide area network overlay with error correction |
US9717021B2 (en) | 2008-07-03 | 2017-07-25 | Silver Peak Systems, Inc. | Virtual network overlay |
US10108353B2 (en) * | 2009-06-25 | 2018-10-23 | EMC IP Holding Company LLC | System and method for providing long-term storage for data |
US20150234616A1 (en) * | 2009-06-25 | 2015-08-20 | Emc Corporation | System and method for providing long-term storage for data |
US9043616B2 (en) * | 2009-11-25 | 2015-05-26 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US20140331104A1 (en) * | 2009-11-25 | 2014-11-06 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US20150213279A1 (en) * | 2009-11-25 | 2015-07-30 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US20150235032A1 (en) * | 2009-11-25 | 2015-08-20 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US9021273B2 (en) * | 2009-11-25 | 2015-04-28 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US9208331B2 (en) * | 2009-11-25 | 2015-12-08 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US20130262854A1 (en) * | 2009-11-25 | 2013-10-03 | Cleversafe, Inc. | Data de-duplication in a dispersed storage network utilizing data characterization |
US20110126026A1 (en) * | 2009-11-25 | 2011-05-26 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US20160364295A1 (en) * | 2009-11-25 | 2016-12-15 | International Business Machines Corporation | Adaptive dispersed storage network (dsn) and system |
US20170177230A1 (en) * | 2009-11-25 | 2017-06-22 | International Business Machines Corporation | Adaptive dispersed storage network (dsn) and system |
US20140310572A1 (en) * | 2009-11-25 | 2014-10-16 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US9672109B2 (en) * | 2009-11-25 | 2017-06-06 | International Business Machines Corporation | Adaptive dispersed storage network (DSN) and system |
US9823845B2 (en) * | 2009-11-25 | 2017-11-21 | International Business Machines Corporation | Adaptive dispersed storage network (DSN) and system |
US8819452B2 (en) * | 2009-11-25 | 2014-08-26 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US9760286B2 (en) * | 2009-11-25 | 2017-09-12 | International Business Machines Corporation | Adaptive dispersed storage network (DSN) and system |
US9747457B2 (en) * | 2009-11-25 | 2017-08-29 | International Business Machines Corporation | Efficient storage of encrypted data in a dispersed storage network |
US8762346B2 (en) * | 2009-11-25 | 2014-06-24 | Cleversafe, Inc. | Data de-duplication in a dispersed storage network utilizing data characterization |
US20120059804A1 (en) * | 2010-09-03 | 2012-03-08 | Arm Limited | Data compression and decompression using relative and absolute delta values |
US8548962B2 (en) * | 2010-09-03 | 2013-10-01 | Arm Limited | Data compression and decompression using relative and absolute delta values |
US10394757B2 (en) | 2010-11-18 | 2019-08-27 | Microsoft Technology Licensing, Llc | Scalable chunk store for data deduplication |
US10073854B2 (en) | 2010-12-14 | 2018-09-11 | Microsoft Technology Licensing, Llc | Data deduplication in a virtualization environment |
US9342244B2 (en) | 2010-12-14 | 2016-05-17 | Microsoft Technology Licensing, Llc | Data deduplication in a virtualization environment |
US20120151177A1 (en) * | 2010-12-14 | 2012-06-14 | Microsoft Corporation | Data Deduplication in a Virtualization Environment |
US8959293B2 (en) * | 2010-12-14 | 2015-02-17 | Microsoft Corporation | Data deduplication in a virtualization environment |
US10042855B2 (en) * | 2010-12-31 | 2018-08-07 | EMC IP Holding Company LLC | Efficient storage tiering |
US20160202911A1 (en) * | 2010-12-31 | 2016-07-14 | Emc Corporation | Efficient storage tiering |
US9280550B1 (en) * | 2010-12-31 | 2016-03-08 | Emc Corporation | Efficient storage tiering |
US20150134623A1 (en) * | 2011-02-17 | 2015-05-14 | Jitcomm Networks Pte Ltd | Parallel data partitioning |
US8990171B2 (en) | 2011-09-01 | 2015-03-24 | Microsoft Corporation | Optimization of a partially deduplicated file |
US20130066868A1 (en) * | 2011-09-13 | 2013-03-14 | Exagrid Systems, Inc. | Systems and Methods for Version Chain Clustering |
US10498356B2 (en) * | 2011-09-13 | 2019-12-03 | Exagrid Systems, Inc. | Systems and methods for version chain clustering |
US11336295B2 (en) | 2011-09-13 | 2022-05-17 | Exagrid Systems, Inc. | Systems and methods for version chain clustering |
US9906630B2 (en) | 2011-10-14 | 2018-02-27 | Silver Peak Systems, Inc. | Processing data packets in performance enhancing proxy (PEP) environment |
US9626224B2 (en) | 2011-11-03 | 2017-04-18 | Silver Peak Systems, Inc. | Optimizing available computing resources within a virtual environment |
US20130304746A1 (en) * | 2012-01-31 | 2013-11-14 | Cleversafe, Inc. | Retrieving indexed data from a dispersed storage network |
US9465861B2 (en) * | 2012-01-31 | 2016-10-11 | International Business Machines Corporation | Retrieving indexed data from a dispersed storage network |
US9843702B1 (en) * | 2012-03-30 | 2017-12-12 | EMC IP Holding Company LLC | Method and system for dynamic compression module selection |
US9843802B1 (en) * | 2012-03-30 | 2017-12-12 | EMC IP Holding Company LLC | Method and system for dynamic compression module selection |
US9571698B1 (en) * | 2012-03-30 | 2017-02-14 | EMC IP Holding Company LLC | Method and system for dynamic compression module selection |
US10114831B2 (en) | 2012-08-16 | 2018-10-30 | Exagrid Systems, Inc. | Delta version clustering and re-anchoring |
US9798731B2 (en) * | 2013-03-06 | 2017-10-24 | Dell Products, Lp | Delta compression of probabilistically clustered chunks of data |
US20140258248A1 (en) * | 2013-03-06 | 2014-09-11 | Dell Products, Lp | Delta Compression of Probabilistically Clustered Chunks of Data |
US9483494B1 (en) * | 2013-03-14 | 2016-11-01 | Emc Corporation | Opportunistic fragmentation repair |
US9594753B1 (en) * | 2013-03-14 | 2017-03-14 | EMC IP Holding Company LLC | Fragmentation repair of synthetic backups |
US20140289208A1 (en) * | 2013-03-21 | 2014-09-25 | Fujitsu Limited | Data compression apparatus, data compression method, data decompression apparatus, and data decompression method |
US9898404B2 (en) * | 2013-07-14 | 2018-02-20 | Cnex Labs | Method and apparatus for providing improved garbage collection process in solid state drive |
US20150019797A1 (en) * | 2013-07-14 | 2015-01-15 | CNEXLABS, Inc. | Method and Apparatus for Providing Improved Garbage Collection Process In Solid State Drive |
US10838990B1 (en) | 2013-09-26 | 2020-11-17 | EMC IP Holding Company LLC | System and method for improving data compression of a storage system using coarse and fine grained similarity |
US9367557B1 (en) | 2013-09-26 | 2016-06-14 | Emc Corporation | System and method for improving data compression |
US9411815B1 (en) | 2013-09-26 | 2016-08-09 | Emc Corporation | System and method for improving data compression in a deduplicated storage system |
US9514146B1 (en) | 2013-09-26 | 2016-12-06 | Emc Corporation | System and method for improving data compression of a storage system in an online manner |
US11144507B2 (en) | 2013-09-26 | 2021-10-12 | EMC IP Holding Company LLC | System and method for balancing compression and read performance in a storage system |
US10216754B1 (en) * | 2013-09-26 | 2019-02-26 | EMC IP Holding Company LLC | System and method for balancing compression and read performance in a storage system |
US9767154B1 (en) | 2013-09-26 | 2017-09-19 | EMC IP Holding Company LLC | System and method for improving data compression of a storage system in an online manner |
US20150120749A1 (en) * | 2013-10-30 | 2015-04-30 | Microsoft Corporation | Data management for connected devices |
AU2014342430B2 (en) * | 2013-10-30 | 2019-08-15 | Microsoft Technology Licensing, Llc | Data management for connected devices |
US10061791B2 (en) * | 2013-10-30 | 2018-08-28 | Microsoft Technology Licensing, Llc | Data management for connected devices |
US11381493B2 (en) | 2014-07-30 | 2022-07-05 | Hewlett Packard Enterprise Development Lp | Determining a transit appliance for data traffic to a software service |
US9948496B1 (en) | 2014-07-30 | 2018-04-17 | Silver Peak Systems, Inc. | Determining a transit appliance for data traffic to a software service |
US11374845B2 (en) | 2014-07-30 | 2022-06-28 | Hewlett Packard Enterprise Development Lp | Determining a transit appliance for data traffic to a software service |
US10812361B2 (en) | 2014-07-30 | 2020-10-20 | Silver Peak Systems, Inc. | Determining a transit appliance for data traffic to a software service |
US11921827B2 (en) * | 2014-09-05 | 2024-03-05 | Hewlett Packard Enterprise Development Lp | Dynamic monitoring and authorization of an optimization device |
US9875344B1 (en) | 2014-09-05 | 2018-01-23 | Silver Peak Systems, Inc. | Dynamic monitoring and authorization of an optimization device |
US11954184B2 (en) | 2014-09-05 | 2024-04-09 | Hewlett Packard Enterprise Development Lp | Dynamic monitoring and authorization of an optimization device |
US10719588B2 (en) | 2014-09-05 | 2020-07-21 | Silver Peak Systems, Inc. | Dynamic monitoring and authorization of an optimization device |
US11868449B2 (en) | 2014-09-05 | 2024-01-09 | Hewlett Packard Enterprise Development Lp | Dynamic monitoring and authorization of an optimization device |
US10885156B2 (en) | 2014-09-05 | 2021-01-05 | Silver Peak Systems, Inc. | Dynamic monitoring and authorization of an optimization device |
US20210192015A1 (en) * | 2014-09-05 | 2021-06-24 | Silver Peak Systems, Inc. | Dynamic monitoring and authorization of an optimization device |
US11940876B2 (en) * | 2014-09-08 | 2024-03-26 | Pure Storage, Inc. | Selecting storage units based on storage pool traits |
US20230267041A1 (en) * | 2014-09-08 | 2023-08-24 | Pure Storage, Inc. | Selecting Storage Units Based on Storage Pool Traits |
US10387374B2 (en) * | 2015-02-27 | 2019-08-20 | Exagrid Systems, Inc. | Scalable grid deduplication |
US11163727B2 (en) | 2015-02-27 | 2021-11-02 | Exagrid Systems, Inc. | Scalable grid deduplication |
EP3062227A1 (en) * | 2015-02-27 | 2016-08-31 | Exagrid Systems, Inc. | Scalable grid deduplication |
US20160253351A1 (en) * | 2015-02-27 | 2016-09-01 | Thomas G. Hansen | Scalable Grid Deduplication |
US10073855B2 (en) | 2015-05-21 | 2018-09-11 | Exagrid Systems, Inc. | Dynamic and optimized management of grid system resources |
US11182345B2 (en) | 2015-08-13 | 2021-11-23 | Exagrid Systems, Inc. | Parallelizing and deduplicating backup data |
US10303656B2 (en) | 2015-08-13 | 2019-05-28 | Exagrid Systems, Inc. | Parallelizing and deduplicating backup data |
US11150997B2 (en) | 2015-08-19 | 2021-10-19 | Exagrid Systems, Inc. | Adaptive bandwidth management of a replication process |
US10042706B2 (en) * | 2015-10-30 | 2018-08-07 | International Business Machines Corporation | Optimizing secondary storage in a dispersed storage network |
US20170123908A1 (en) * | 2015-10-30 | 2017-05-04 | International Business Machines Corporation | Optimizing secondary storage in a dispersed storage network |
US10771370B2 (en) | 2015-12-28 | 2020-09-08 | Silver Peak Systems, Inc. | Dynamic monitoring and visualization for network health characteristics |
US10164861B2 (en) | 2015-12-28 | 2018-12-25 | Silver Peak Systems, Inc. | Dynamic monitoring and visualization for network health characteristics |
US11336553B2 (en) | 2015-12-28 | 2022-05-17 | Hewlett Packard Enterprise Development Lp | Dynamic monitoring and visualization for network health characteristics of network device pairs |
US10216434B2 (en) * | 2015-12-30 | 2019-02-26 | International Business Machines Corporation | Detailed memory device statistics with drive write location determination |
US10432484B2 (en) | 2016-06-13 | 2019-10-01 | Silver Peak Systems, Inc. | Aggregating select network traffic statistics |
US11601351B2 (en) | 2016-06-13 | 2023-03-07 | Hewlett Packard Enterprise Development Lp | Aggregation of select network traffic statistics |
US11757740B2 (en) | 2016-06-13 | 2023-09-12 | Hewlett Packard Enterprise Development Lp | Aggregation of select network traffic statistics |
US11757739B2 (en) | 2016-06-13 | 2023-09-12 | Hewlett Packard Enterprise Development Lp | Aggregation of select network traffic statistics |
US10848268B2 (en) | 2016-08-19 | 2020-11-24 | Silver Peak Systems, Inc. | Forward packet recovery with constrained network overhead |
US11424857B2 (en) | 2016-08-19 | 2022-08-23 | Hewlett Packard Enterprise Development Lp | Forward packet recovery with constrained network overhead |
US9967056B1 (en) | 2016-08-19 | 2018-05-08 | Silver Peak Systems, Inc. | Forward packet recovery with constrained overhead |
US10326551B2 (en) | 2016-08-19 | 2019-06-18 | Silver Peak Systems, Inc. | Forward packet recovery with constrained network overhead |
US11729090B2 (en) | 2017-02-06 | 2023-08-15 | Hewlett Packard Enterprise Development Lp | Multi-level learning for classifying network traffic flows from first packet data |
US11044202B2 (en) | 2017-02-06 | 2021-06-22 | Silver Peak Systems, Inc. | Multi-level learning for predicting and classifying traffic flows from first packet data |
US10771394B2 (en) | 2017-02-06 | 2020-09-08 | Silver Peak Systems, Inc. | Multi-level learning for classifying traffic flows on a first packet from DNS data |
US10257082B2 (en) | 2017-02-06 | 2019-04-09 | Silver Peak Systems, Inc. | Multi-level learning for classifying traffic flows |
US10892978B2 (en) | 2017-02-06 | 2021-01-12 | Silver Peak Systems, Inc. | Multi-level learning for classifying traffic flows from first packet data |
US11582157B2 (en) | 2017-02-06 | 2023-02-14 | Hewlett Packard Enterprise Development Lp | Multi-level learning for classifying traffic flows on a first packet from DNS response data |
US20180232387A1 (en) * | 2017-02-15 | 2018-08-16 | Paypal, Inc. | Data transfer size reduction |
US11182340B2 (en) * | 2017-02-15 | 2021-11-23 | Paypal, Inc. | Data transfer size reduction |
US10318159B1 (en) * | 2017-06-14 | 2019-06-11 | EMC IP Holding Company LLC | Method and system for physical locality repair in a storage system |
US11805045B2 (en) | 2017-09-21 | 2023-10-31 | Hewlett Packard Enterprise Development Lp | Selective routing |
US11212210B2 (en) | 2017-09-21 | 2021-12-28 | Silver Peak Systems, Inc. | Selective route exporting using source type |
US10887159B2 (en) | 2018-03-12 | 2021-01-05 | Silver Peak Systems, Inc. | Methods and systems for detecting path break conditions while minimizing network overhead |
US11405265B2 (en) | 2018-03-12 | 2022-08-02 | Hewlett Packard Enterprise Development Lp | Methods and systems for detecting path break conditions while minimizing network overhead |
US10637721B2 (en) | 2018-03-12 | 2020-04-28 | Silver Peak Systems, Inc. | Detecting path break conditions while minimizing network overhead |
US10860233B2 (en) * | 2019-04-12 | 2020-12-08 | Samsung Electronics Co., Ltd. | Half-match deduplication |
US20200326869A1 (en) * | 2019-04-12 | 2020-10-15 | Samsung Electronics Co., Ltd. | Half-match deduplication |
US11627207B2 (en) | 2019-12-18 | 2023-04-11 | Ndata, Inc. | Systems and methods for data deduplication by generating similarity metrics using sketch computation |
US11119995B2 (en) * | 2019-12-18 | 2021-09-14 | Ndata, Inc. | Systems and methods for sketch computation |
US10938961B1 (en) | 2019-12-18 | 2021-03-02 | Ndata, Inc. | Systems and methods for data deduplication by generating similarity metrics using sketch computation |
Also Published As
Publication number | Publication date |
---|---|
US9069785B2 (en) | 2015-06-30 |
US20150261779A1 (en) | 2015-09-17 |
US20130232125A1 (en) | 2013-09-05 |
US9690802B2 (en) | 2017-06-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9690802B2 (en) | Stream locality delta compression | |
US8751462B2 (en) | Delta compression after identity deduplication | |
US9418133B2 (en) | Data replication with delta compression | |
You et al. | Deep Store: An archival storage system architecture | |
US9934237B1 (en) | Metadata optimization for network replication using representative of metadata batch | |
US8756238B2 (en) | Index searching using a bloom filter | |
US9110964B1 (en) | Metadata optimization for network replication using differential encoding | |
Shilane et al. | Wan-optimized replication of backup datasets using stream-informed delta compression | |
US7478113B1 (en) | Boundaries | |
US8639669B1 (en) | Method and apparatus for determining optimal chunk sizes of a deduplicated storage system | |
US8712963B1 (en) | Method and apparatus for content-aware resizing of data chunks for replication | |
US8631052B1 (en) | Efficient content meta-data collection and trace generation from deduplicated storage | |
US8166012B2 (en) | Cluster storage using subsegmenting | |
US9514146B1 (en) | System and method for improving data compression of a storage system in an online manner | |
EP2256934B1 (en) | Method and apparatus for content-aware and adaptive deduplication | |
US9678688B2 (en) | System and method for data deduplication for disk storage subsystems | |
US8667032B1 (en) | Efficient content meta-data collection and trace generation from deduplicated storage | |
US8825626B1 (en) | Method and system for detecting unwanted content of files | |
US11157188B2 (en) | Detecting data deduplication opportunities using entropy-based distance | |
US8756249B1 (en) | Method and apparatus for efficiently searching data in a storage system | |
US9063664B1 (en) | Method and system for processing checksum of a data stream to optimize deduplication | |
WO2009131585A1 (en) | Data processing apparatus and method of processing data | |
JP6807395B2 (en) | Distributed data deduplication in the processor grid | |
US9087086B1 (en) | Method and system for handling object boundaries of a data stream to optimize deduplication | |
Vikraman et al. | A study on various data de-duplication systems |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: DATA DOMAIN, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HUANG, MARK;SHILANE, PHILIP;WALLACE, GRANT;AND OTHERS;SIGNING DATES FROM 20090210 TO 20090211;REEL/FRAME:022480/0944 |
|
AS | Assignment |
Owner name: DATA DOMAIN, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HUANG, MARK;SHILANE, PHILIP;WALLACE, GRANT;AND OTHERS;SIGNING DATES FROM 20090210 TO 20090222;REEL/FRAME:022936/0008 |
|
AS | Assignment |
Owner name: DATA DOMAIN LLC, DELAWARE Free format text: CONVERSION;ASSIGNOR:DATA DOMAIN, INC.;REEL/FRAME:023994/0223 Effective date: 20091218 |
|
AS | Assignment |
Owner name: DATA DOMAIN HOLDING, INC., MASSACHUSETTS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DATA DOMAIN LLC;REEL/FRAME:024029/0539 Effective date: 20091222 |
|
AS | Assignment |
Owner name: EMC CORPORATION, MASSACHUSETTS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DATA DOMAIN HOLDING, INC.;REEL/FRAME:024072/0833 Effective date: 20091231 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT, NORTH CAROLINA Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040134/0001 Effective date: 20160907 Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT, TEXAS Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040136/0001 Effective date: 20160907 Owner name: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLAT Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040134/0001 Effective date: 20160907 Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., A Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040136/0001 Effective date: 20160907 |
|
AS | Assignment |
Owner name: EMC IP HOLDING COMPANY LLC, MASSACHUSETTS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:EMC CORPORATION;REEL/FRAME:040203/0001 Effective date: 20160906 |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
AS | Assignment |
Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., T Free format text: SECURITY AGREEMENT;ASSIGNORS:CREDANT TECHNOLOGIES, INC.;DELL INTERNATIONAL L.L.C.;DELL MARKETING L.P.;AND OTHERS;REEL/FRAME:049452/0223 Effective date: 20190320 Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., TEXAS Free format text: SECURITY AGREEMENT;ASSIGNORS:CREDANT TECHNOLOGIES, INC.;DELL INTERNATIONAL L.L.C.;DELL MARKETING L.P.;AND OTHERS;REEL/FRAME:049452/0223 Effective date: 20190320 |
|
AS | Assignment |
Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., TEXAS Free format text: SECURITY AGREEMENT;ASSIGNORS:CREDANT TECHNOLOGIES INC.;DELL INTERNATIONAL L.L.C.;DELL MARKETING L.P.;AND OTHERS;REEL/FRAME:053546/0001 Effective date: 20200409 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
AS | Assignment |
Owner name: WYSE TECHNOLOGY L.L.C., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: SCALEIO LLC, MASSACHUSETTS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: MOZY, INC., WASHINGTON Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: MAGINATICS LLC, CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: FORCE10 NETWORKS, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: EMC IP HOLDING COMPANY LLC, TEXAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: EMC CORPORATION, MASSACHUSETTS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: DELL SYSTEMS CORPORATION, TEXAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: DELL SOFTWARE INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: DELL PRODUCTS L.P., TEXAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: DELL MARKETING L.P., TEXAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: DELL INTERNATIONAL, L.L.C., TEXAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: DELL USA L.P., TEXAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: CREDANT TECHNOLOGIES, INC., TEXAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: AVENTAIL LLC, CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 Owner name: ASAP SOFTWARE EXPRESS, INC., ILLINOIS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001 Effective date: 20211101 |
|
AS | Assignment |
Owner name: SCALEIO LLC, MASSACHUSETTS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001 Effective date: 20220329 Owner name: EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.), TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001 Effective date: 20220329 Owner name: EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC), MASSACHUSETTS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001 Effective date: 20220329 Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.), TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001 Effective date: 20220329 Owner name: DELL PRODUCTS L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001 Effective date: 20220329 Owner name: DELL INTERNATIONAL L.L.C., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001 Effective date: 20220329 Owner name: DELL USA L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001 Effective date: 20220329 Owner name: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.), TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001 Effective date: 20220329 Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.), TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001 Effective date: 20220329 |
|
AS | Assignment |
Owner name: SCALEIO LLC, MASSACHUSETTS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001 Effective date: 20220329 Owner name: EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.), TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001 Effective date: 20220329 Owner name: EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC), MASSACHUSETTS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001 Effective date: 20220329 Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.), TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001 Effective date: 20220329 Owner name: DELL PRODUCTS L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001 Effective date: 20220329 Owner name: DELL INTERNATIONAL L.L.C., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001 Effective date: 20220329 Owner name: DELL USA L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001 Effective date: 20220329 Owner name: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.), TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001 Effective date: 20220329 Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.), TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001 Effective date: 20220329 |