Data deduplication engine, supporting optional compression and public key encryption.

🚢 Data Toolkit for Sailor Language Models

Self-contained C# library for data deduplication using Sqlite

Fast and efficient content-defined chunking for data deduplication. Java implementation of FastCDC as library.

Optimal distributed data deduplication and supervised learning pipeline using Apache Spark

General deduping engine for JDBC sources with output to JDBC/csv targets

A JAVA project that splits data using hashing techniques and removes duplicate blocks to save cloud storage. This project also uses the CloudSim framework for cloud storage simulation.

PolyDeDupe: Multi-Lingual Data Deduplication

Fellow is a package for creating people that can be unified by their shared values via a singleton list on the class

Practical backups. The Unix toolkit way.

This repository contains SQL scripts and documentation for cleaning and standardizing data in the NashvilleHousing table within the sqlproject2 database. The project aims to prepare the dataset for analysis by addressing inconsistencies, filling missing values, standardizing formats, and removing duplicates.

A calculator for storage and transmission of deduplicated data presentation in charts and tables

This is a server client architecture based data deduplication implementation

A Python-based tool for preprocessing, cleaning, and analyzing text datasets, designed to filter, deduplicate, sort data, and generate statistical insights.

The HR Roster Change Detection Pipeline is an automated solution for processing HR roster data. Leveraging Apache Airflow and PostgreSQL, it enables seamless data ingestion, deduplication, and change detection, streamlining HR operations.