gunz_cm.compressions#
Submodules#
Compression codecs for GZCM contact matrix tiles.
Benchmarks (GM12878 chr1 @ 50kb, tile_size=512, window=1Mb):
(*) cmc_zstd offers the best balance of compression ratio and convert speed.
bsc_cmc: CMC transforms (binarization, diagonal transform) + BSC entropy coding. Same compression as CMC with faster access. Best overall codec for storage-constrained use.
GZCM v4 adds the lz4 codec (block format, lazy lz4.block import). See
gunz_cm.compressions.lz4 and specs/gzcm-v4-design.md §4.7.
GZCM v5.1 adds a codec registry with an explicit wire-format contract
(WireFormat in gunz_cm.compressions._protocol). Each codec’s
wire_format attribute (set automatically by the registry) tells the
container writer whether to prepend the 8-byte (rows, cols) shape header
or not. This is the fix for Bug 0.3 (writer prepended the header over LZ4’s
own 12-byte header, causing decode to fail). Adding a new codec is now a
single register_codec(...) call.
Examples
>>> from gunz_cm.compressions import CmcZstdEncoder, CmcZstdDecoder
>>> encoder = CmcZstdEncoder(tile_size=512)
>>> encoded = encoder.encode_tile(tile_data)
>>> decoder = CmcZstdDecoder(tile_size=512)
>>> decoded = decoder.decode_tile(encoded)
>>> from gunz_cm.compressions import get_codec, list_codecs
>>> list_codecs()
['cmc', 'cmc_zstd', 'bsc', 'bsc_cmc', 'zstd', 'lz4']
>>> enc_cls, dec_cls, wire_format = get_codec("lz4")
- class gunz_cm.compressions.BscCmcDecoder(tile_size: int = 512, resolution: int = 50000, dtype: ~numpy.dtype = <class 'numpy.uint32'>, diag_mode: int = 0, bsc_bin: str | pathlib.Path | None = None, ld_library_path: str | pathlib.Path | None = None)[source]#
Bases:
objectBSC + CMC Transforms decoder for contact matrix tiles.
Decodes BSC-compressed data that was encoded with CMC transforms. Reverses BSC entropy coding then CMC’s domain-specific transforms.
- Parameters:
tile_size (int, default=512) – Tile size for block processing.
resolution (int, default=50000) – Hi-C resolution in bp.
dtype (np.dtype, default=np.uint32) – Data type for decoded tiles.
bsc_bin (str or pathlib.Path, optional) – Explicit path to the bsc binary. If None, resolved via GUNZ_CM_BSC_BIN env var or system PATH.
ld_library_path (str or pathlib.Path, optional) – Explicit LD_LIBRARY_PATH for the bsc subprocess. If None, resolved via GUNZ_CM_BSC_LD_LIBRARY_PATH env var.
Examples
- class gunz_cm.compressions.BscCmcEncoder(tile_size: int = 512, resolution: int = 50000, level: int = 3, diag_mode: int = 0, bsc_bin: str | pathlib.Path | None = None, ld_library_path: str | pathlib.Path | None = None)[source]#
Bases:
objectBSC + CMC Transforms encoder for contact matrix tiles.
Applies CMC’s domain-specific transforms (diagonal transform, binarization) before BSC entropy coding. Combines BSC’s speed with CMC’s structured transforms.
- Parameters:
tile_size (int, default=512) – Tile size for block processing.
resolution (int, default=50000) – Hi-C resolution in bp.
level (int, default=3) – BSC compression level (0-9, higher = better compression).
bsc_bin (str or pathlib.Path, optional) – Explicit path to the bsc binary. If None, resolved via GUNZ_CM_BSC_BIN env var or system PATH.
ld_library_path (str or pathlib.Path, optional) – Explicit LD_LIBRARY_PATH for the bsc subprocess. If None, resolved via GUNZ_CM_BSC_LD_LIBRARY_PATH env var.
Examples
- encode_tile(mat: ndarray) bytes[source]#
Encode a single contact matrix tile.
- Parameters:
mat (np.ndarray) – 2D contact matrix tile (upper triangular).
- Returns:
Compressed bitstream (shape info + encoded data).
- Return type:
Examples
- class gunz_cm.compressions.BscDecoder(tile_size: int = 512, resolution: int = 50000, dtype: ~numpy.dtype = <class 'numpy.uint32'>, bsc_bin: str | pathlib.Path | None = None, ld_library_path: str | pathlib.Path | None = None)[source]#
Bases:
objectBSC decoder for contact matrix tiles.
Uses bsc CLI subprocess for true BSC (Block Sorting Compression) decompression.
- Parameters:
tile_size (int, default=512) – Tile size for block processing.
resolution (int, default=50000) – Hi-C resolution in bp.
dtype (np.dtype, default=np.uint32) – Data type for decoded tiles.
bsc_bin (str or pathlib.Path, optional) – Explicit path to the bsc binary. If None, resolved via GUNZ_CM_BSC_BIN env var or system PATH.
ld_library_path (str or pathlib.Path, optional) – Explicit LD_LIBRARY_PATH for the bsc subprocess. If None, resolved via GUNZ_CM_BSC_LD_LIBRARY_PATH env var.
Examples
- decode_tile(payload: bytes, shape: tuple[int, int] | None = None) ndarray[source]#
Decode a single BSC-compressed tile.
- Parameters:
payload (bytes) – BSC-compressed bitstream. When
shapeis provided (v3 GZCM path), the first 8 bytes encode(rows, cols)asnp.int32and are stripped before decompression. WhenshapeisNonethe payload is treated as raw compressed bytes (legacy / codec tests).shape (tuple of (int, int), optional) – Actual tile shape
(rows, cols). Required for edge tiles whererows != self.tile_sizeorcols != self.tile_size. IfNone, falls back to(self.tile_size, self.tile_size)and assumes no 8-byte header is present.
- Returns:
Decoded contact matrix tile with the requested shape.
- Return type:
np.ndarray
Examples
- class gunz_cm.compressions.BscEncoder(tile_size: int = 512, resolution: int = 50000, level: int = 3, bsc_bin: str | pathlib.Path | None = None, ld_library_path: str | pathlib.Path | None = None)[source]#
Bases:
objectBSC encoder for contact matrix tiles.
Uses bsc CLI subprocess for true BSC (Block Sorting Compression).
- Parameters:
tile_size (int, default=512) – Tile size for block processing.
resolution (int, default=50000) – Hi-C resolution in bp.
level (int, default=3) – Compression level (0-9, higher = better compression).
bsc_bin (str or pathlib.Path, optional) – Explicit path to the bsc binary. If None, resolved via GUNZ_CM_BSC_BIN env var or system PATH.
ld_library_path (str or pathlib.Path, optional) – Explicit LD_LIBRARY_PATH for the bsc subprocess. If None, resolved via GUNZ_CM_BSC_LD_LIBRARY_PATH env var.
Examples
- encode_tile(mat: ndarray) bytes[source]#
Encode a single contact matrix tile.
- Parameters:
mat (np.ndarray) – 2D contact matrix tile.
- Returns:
BSC-compressed bitstream.
- Return type:
Examples
- class gunz_cm.compressions.CmcDecoder(tile_size: int = 256, resolution: int = 50000, diag_transform: bool = True)[source]#
Bases:
objectCMC decoder for contact matrix tiles.
- Parameters:
Examples
- class gunz_cm.compressions.CmcEncoder(tile_size: int = 256, resolution: int = 50000, diag_transform: bool = True)[source]#
Bases:
objectCMC encoder for contact matrix tiles.
- Parameters:
Examples
- encode_tile(mat: ndarray) bytes[source]#
Encode a single contact matrix tile.
- Parameters:
mat (np.ndarray) – 2D contact matrix tile (upper triangular).
- Returns:
CMC-encoded bitstream.
- Return type:
Examples
- class gunz_cm.compressions.CmcZstdDecoder(tile_size: int = 256, resolution: int = 50000, dtype: ~numpy.dtype = <class 'numpy.uint32'>)[source]#
Bases:
objectCMC Transforms + Zstd decoder for contact matrix tiles.
Uses Zstd decompression then reverses CMC’s domain-specific transforms.
- Parameters:
Examples
- class gunz_cm.compressions.CmcZstdEncoder(tile_size: int = 256, resolution: int = 50000, level: int = 3)[source]#
Bases:
objectCMC Transforms + Zstd encoder for contact matrix tiles.
Uses CMC’s domain-specific transforms (diagonal transform, binarization) with Zstd entropy coding for better compression and faster decode.
- Parameters:
Examples
- encode_tile(mat: ndarray) bytes[source]#
Encode a single contact matrix tile.
- Parameters:
mat (np.ndarray) – 2D contact matrix tile (upper triangular).
- Returns:
Compressed bitstream (shape info + encoded data).
- Return type:
Examples
- class gunz_cm.compressions.Codec(*args, **kwargs)[source]#
Bases:
ProtocolMinimal interface every GZCM tile codec must satisfy.
Implementations:
CmcEncoder,ZstdEncoder,CmcZstdEncoder,BscEncoder,BscCmcEncoder,Lz4Encoder. Decoders are paired via the registry.The
runtime_checkable()decorator lets the registry assertisinstance(cls, Codec)at registration time without requiring explicit subclassing.- decode_tile(payload: bytes, shape) numpy.ndarray[source]#
Decode bytes back to a 2-D uint tile of the given shape.
- Parameters:
payload (bytes) – For
OPAQUE_PAYLOAD: the raw encoded bytes (writer-prepended 8-byte shape header has already been stripped by the reader). ForSELF_DESCRIBING: the full payload including the encoder’s own header.shape (tuple[int, int]) –
(rows, cols). ForOPAQUE_PAYLOADthis comes from the writer’s 8-byte shape header. ForSELF_DESCRIBINGthis should match what the encoder embedded in its own header.
- Returns:
2-D array of shape
shapewith integer dtype.- Return type:
- encode_tile(tile) bytes[source]#
Encode a 2-D uint tile to bytes.
- Parameters:
tile (numpy.ndarray) – 2-D array of shape
(rows, cols)with integer dtype.- Returns:
The encoded payload. The wire-format contract (
OPAQUE_PAYLOADvsSELF_DESCRIBING) determines whether the writer prepends an 8-byte shape header before this payload.- Return type:
- property wire_format: WireFormat#
What the writer must do (or not do) around this codec’s payload.
Bases:
RuntimeErrorRaised when a codec is registered but its native binary is missing.
Codec name (e.g.,
"cmc").- Type:
Human-readable reason (e.g., the FileNotFoundError message).
- Type:
- class gunz_cm.compressions.Lz4Decoder(tile_size: int = 256, resolution: int = 50000, dtype=<class 'numpy.uint32'>)[source]#
Bases:
objectLZ4 block decoder for contact matrix tiles.
- Parameters:
tile_size (int, default=256) – Default tile side length used when
shapeis not provided todecode_tile.resolution (int, default=50000) – Hi-C resolution in bp. Stored for metadata symmetry with
Lz4Encoder; the decoder itself does not use it.dtype (numpy data type, default=numpy.uint32) – Output dtype for decoded tiles. Accepts anything
numpy.dtype()understands.
Examples
- decode_tile(payload: bytes, shape: tuple[int, int] | None = None) ndarray[source]#
Decode a single compressed tile.
Two payload conventions are supported:
v4 GZCM path (
shapeprovided):payloadbegins with the 12-byte(rows, cols, uncompressed_size)header followed by the lz4 block body.rows/colsare taken fromshape(so the caller can override if the tile is non-rectangular after a future change); theuncompressed_sizefield is the authoritative hint forlz4.block.decompress.Legacy / direct codec path (
shapeisNone):payloadis treated as a raw lz4 block body and the tile is reshaped to(tile_size, tile_size).
- Parameters:
- Returns:
Decoded contact matrix tile.
- Return type:
np.ndarray
Examples
- class gunz_cm.compressions.Lz4Encoder(tile_size: int = 256, resolution: int = 50000, mode: str = 'default', acceleration: int = 1, hc_level: int = 9)[source]#
Bases:
objectLZ4 block encoder for contact matrix tiles.
- Parameters:
tile_size (int, default=256) – Tile side length in bins. Stored for
get_compression_info; the encoder itself operates on whatever shape is passed toencode_tile.resolution (int, default=50000) – Hi-C resolution in bp. Stored for metadata.
mode (str, default="default") – LZ4 block mode passed to
lz4.block.compress. One of"default","fast","high_compression".acceleration (int, default=1) – LZ4 acceleration factor (>=1). Higher values trade compression ratio for speed;
1is the slowest but best ratio.hc_level (int, default=9) – High-compression level (4-16) used when
mode='high_compression'. Ignored otherwise.9matches the scheme-picker candidatelz4-hc-9defined inspecs/gzcm-v4-design.md§4.7.
Examples
- encode_tile(mat: ndarray) bytes[source]#
Encode a single contact matrix tile.
The 12-byte header (rows, cols, uncompressed_size) is prepended so the decoder can reshape edge tiles and call
lz4.block.decompresswith the correctuncompressed_size.We call
lz4.block.compress(..., store_size=False)so the uncompressed size is NOT also written into the body of the compressed block. The block decoder receives the size exclusively through our 12-byte header — otherwise thedecompresscall below would see a size prefix that disagrees withuncompressed_size.- Parameters:
mat (np.ndarray) – 2D contact matrix tile (any rectangular shape).
- Returns:
12-byte header + lz4 block-compressed payload.
- Return type:
Examples
- exception gunz_cm.compressions.UnknownCodecError(name: str, available: list[str])[source]#
Bases:
KeyErrorRaised by
get_codecwhen the requested codec name is not registered.
- class gunz_cm.compressions.WireFormat(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]#
Bases:
EnumHow a codec’s encoded payload is delimited on disk.
- OPAQUE_PAYLOAD#
The encoder produces an opaque byte stream with no shape information. The writer prepends an 8-byte
(rows: int32, cols: int32)shape header before the encoded payload so the decoder can reshape on read. Used by: zstd, cmc, bsc, bsc_cmc, cmc_zstd.
- SELF_DESCRIBING#
The encoder adds its own header (size, shape, or both) inside the encoded payload. The writer MUST NOT prepend any header; doing so would shift the encoder’s header and corrupt the payload. Used by: lz4 (12-byte
(rows, cols, uncompressed_size)header).
- OPAQUE_PAYLOAD = 'opaque'#
- SELF_DESCRIBING = 'self'#
- class gunz_cm.compressions.ZstdDecoder(tile_size: int = 256, resolution: int = 50000, dtype: ~numpy.dtype = <class 'numpy.uint32'>, use_zstd: bool = True)[source]#
Bases:
objectZstd decoder for contact matrix tiles.
- Parameters:
Examples
- decode_tile(payload: bytes, shape: tuple[int, int] | None = None) ndarray[source]#
Decode a single compressed tile.
- Parameters:
payload (bytes) – Compressed bitstream. When
shapeis provided (v3 GZCM path), the first 8 bytes encode(rows, cols)asnp.int32and are stripped before decompression. WhenshapeisNonethe payload is treated as raw compressed bytes (legacy / codec tests).shape (tuple of (int, int), optional) – Actual tile shape
(rows, cols). Required for edge tiles whererows != self.tile_sizeorcols != self.tile_size. IfNone, falls back to(self.tile_size, self.tile_size)and assumes no 8-byte header is present.
- Returns:
Decoded contact matrix tile with the requested shape.
- Return type:
np.ndarray
Examples
- class gunz_cm.compressions.ZstdEncoder(tile_size: int = 256, resolution: int = 50000, level: int = 3, use_zstd: bool = True)[source]#
Bases:
objectZstd encoder for contact matrix tiles.
- Parameters:
Examples
- encode_tile(mat: ndarray) bytes[source]#
Encode a single contact matrix tile.
- Parameters:
mat (np.ndarray) – 2D contact matrix tile.
- Returns:
Compressed bitstream.
- Return type:
Examples
- gunz_cm.compressions.codec_available(name: str) bool[source]#
Return True if the named codec’s runtime dependencies are satisfied.
A codec that is not registered at all returns False (it is
unknown, notunavailable; the distinction lets the caller decide whether to register it first or just skip it).
- gunz_cm.compressions.get_codec(name: str) tuple[type, type, gunz_cm.compressions._protocol.WireFormat][source]#
Look up a registered codec by name.
- Returns:
Callers instantiate the encoder/decoder with
tile_size=...and usewire_formatto decide whether to prepend a shape header.- Return type:
(Encoder class, Decoder class, WireFormat)
- Raises:
UnknownCodecError – If
nameis not in the registry at all.CodecUnavailableError – If
nameis registered but the codec requires a native binary (CMC, BSC, …) that is not installed on this machine. Usecodec_available()to probe before requesting.
- gunz_cm.compressions.list_available_codecs() list[str][source]#
Return the names of registered codecs whose binaries are present.
Excludes codecs that depend on a missing native binary (cmc / cmc_zstd / bsc_cmc when GUNZ_CM_CMC_DIR is unset; bsc when neither GUNZ_CM_BSC_BIN nor bsc on PATH resolves).
- gunz_cm.compressions.list_codecs() list[str][source]#
Return the names of all registered codecs (sorted).
Includes codecs that depend on native binaries (e.g.,
cmc,bsc) even when those binaries are missing. Uselist_available_codecs()to filter by availability, andcodec_available()to probe a single codec.
- gunz_cm.compressions.register_codec(name: str, encoder_cls: type, decoder_cls: type, wire_format: WireFormat) None[source]#
Register a codec in the global registry.
- Parameters:
name (str) – Codec identifier used in
convert_to_gzcm(..., compression=name).wire_format (WireFormat) – Whether the writer should prepend the 8-byte shape header. For codecs whose encoder adds its own header (e.g. LZ4), use
WireFormat.SELF_DESCRIBING. Otherwise useWireFormat.OPAQUE_PAYLOAD.
Human-readable reason a codec is unavailable, for error messages.