iscc-sct¶

A cross-lingual, similarity-preserving binary code for text. Semantically similar texts, including translations, produce codes with low hamming distance.

Introduction¶

The ISCC framework already includes a Text-Code based on lexical similarity for near-duplicate matching. The Semantic Text-Code (SCT) is a planned additional ISCC-UNIT that captures a broader, more abstract similarity. It is engineered to be robust against rephrasing and, most notably, translations that lexical matching cannot detect.

iscc-sct turns any text into a compact binary code built from a binarized, multilingual document embedding. The same content expressed in different languages maps to (near-)identical codes, opening up cross-lingual content identification and similarity detection.

Key features:

• Semantic similarity - deep multilingual embeddings capture the meaning of the text
• Translation matching - near-identical codes for the same content across languages
• Bit-length flexibility - codes from 32 up to 256 bits for adjustable granularity
• ISCC compatible - codes integrate directly with existing ISCC-based systems
• 60+ languages - one model covers a broad range of scripts and language pairs

Quick start¶

pip install "iscc-sct[cpu]"

uv add "iscc-sct[cpu]"

uvx "iscc-sct[cpu]" "path/to/textfile.txt"

Generate a Semantic Text-Code from Python:

import iscc_sct as sct

text = "This is some sample text. It can be a longer document or even an entire book."
print(sct.create(text, bits=256).iscc)
# ISCC:CADV3GG6JH3XEVRNSVYGCLJ7AAV3BOT5J7EHEZKPFXEGRJ2CTWACGZI

Or from the command line:

iscc-sct "path/to/textfile.txt"

How it works¶

Text -> split -> embed -> mean-pool -> binarize -> ISCC header + base32

1. Split the text into overlapping chunks at syntactically sensible boundaries.
2. Embed each chunk with a multilingual sentence-transformer running on ONNX.
3. Aggregate the chunk embeddings into one mean-pooled, normalized document vector.
4. Binarize the vector (positive components become 1-bits), truncate to the requested bit-length, prefix the ISCC header, and base32-encode it.

This process is robust to variation and translation, enabling cross-lingual matching from a short Simprint.

Documentation¶

Source code on GitHub