GZCM Format

GZCM (GunZ Contact Matrix) is a binary container format for Hi-C contact matrices, designed for efficient storage, memory-mapped access, and optional compression.

Version Overview

Version	Storage Type	Compression	Use Case
v1	Dense row-major	None	Small matrices, simple use
v2	Tiled 4D blocks	None	Large matrices, streaming writes
v3	Tiled + CMC	BSC, ZSTD, CMC	Maximum compression for large Hi-C data
v4	Regions + per-tile codecs	zstd, lz4, bsc, cmc, cmc_zstd, bsc_cmc (mixed within file)	Opt-in (v2.28.0+): smallest files, fastest selective reads

File Layout

All versions share the same header structure:

┌─────────────────────────────────────┐
│ Magic: "GZCM" (4 bytes)            │
├─────────────────────────────────────┤
│ Header Length (4 bytes, uint32)     │
├─────────────────────────────────────┤
│ Header JSON (4096-byte aligned)      │
├─────────────────────────────────────┤
│ Array Data (version-dependent)       │
└─────────────────────────────────────┘

Header Schema

The JSON header contains version, metadata, and array descriptions:

{
  "version": 3,
  "metadata": {
    "resolution": 10000,
    "region": "chr1",
    "source_file": "/path/to/input.hic"
  },
  "arrays": {
    "matrix": {
      "dtype": "float32",
      "shape": [1000, 1000],
      "offset": 4096,
      "order": "C"
    }
  }
}

---

Version Details

v1 — Dense Storage

The simplest format. Matrix is stored as a flat row-major array.

Characteristics:

• No chunking — entire matrix must fit in memory

• No compression — uses full float32 storage

• Direct memory-mapping available

When to use: Small matrices (< 100k bins) or when simplicity is preferred.

v2 — Tiled Storage

Matrix is divided into fixed-size blocks (tiles) for cache-efficient access.

Characteristics:

• Streaming writes without full matrix in memory

• Cache-efficient block processing

• Random block access

• Block sizes: 256, 512, or 1024

When to use: Large matrices where you want block-wise processing without loading the full matrix.

v3 — Compressed Tiles

Tiles are independently compressed using CMC (Contact Matrix Codec).

Characteristics:

• ~10-25x compression for Hi-C data

• Random tile access (decompress single tile)

• Supported codecs: cmc, cmc_zstd (recommended), bsc, bsc_cmc, zstd

• Lossless only

When to use: Large matrices where storage size is a concern. Recommended for most production use.

v4 — Regions + Per-Tile Codecs (v2.28.0+)

GZCM v4 splits a matrix into a tree of regions (typically a 2-3 level hierarchy ending in tile leaves), and lets each tile pick its own codec from the registry below. This trades a small per-tile metadata cost (8-byte tile indexes + per-region descriptor) for two big wins:

1. Smaller files: the v4 codec picker (gunz_cm.compressions.scheme_picker) scores each candidate on a 5% sample of the tile and picks the best; on chr1 of 4DNFI1UEG1D this lands ~30–50% smaller than v3 zstd at comparable read speed.

2. Faster selective reads: GzcmV4Reader.get_tile_payload(region_id, tile_index) returns the raw tile bytes and codec name without decoding the entire matrix, so callers can stream just the tiles they need.

v4 reader dispatch is handled by GZCMDataset._open_reader(): the same GZCMDataset("foo.gzcm") constructor transparently loads v1, v2, v3, or v4 files based on the header’s version field. There is no need to specify a version on read.

When to use: when you have a v4 pipeline (workflow that opts into v4 via convert_to_gzcm(..., version=4)), or when reading a v4-produced file. v4 is opt-in for writing — v3 remains the default and is still the right choice when you don’t need the per-tile codec freedom.

For migration details and CLI examples, see the GZCM v3 → v4 migration guide.

---

CLI Usage

Convert to GZCM

# Convert .hic to GZCM v3 (recommended default)
gunz-cm converters to-gzcm input.hic output.gzcm chr1 10000 --version 3 --compression bsc_cmc

# Convert .hic to GZCM v4 (opt-in; per-region codec picker)
gunz-cm converters to-gzcm input.hic output_v4.gzcm chr1 10000 --version 4

# Convert to GZCM v2 (tiled, no compression)
gunz-cm converters to-gzcm input.hic output.gzcm chr1 10000 --version 2

# Convert to GZCM v1 (dense)
gunz-cm converters to-gzcm input.hic output.gzcm chr1 10000 --version 1

Normalize GZCM

# Knight-Ruiz normalization
gunz-cm converters normalize input.gzcm normalized.gzcm --method kr

# ICE normalization
gunz-cm converters normalize input.gzcm normalized.gzcm --method ice

---

Python API

Writing

from gunz_cm.io.gnz import GZCMWriter
import numpy as np

# Write GZCM v3 (compressed)
writer = GZCMWriter("matrix.gzcm", overwrite=True, version=3)
writer.set_metadata({"resolution": 10000, "region": "chr1"})
writer.init_streaming_array("matrix", (1000, 1000), dtype=np.float32)
writer.write()

mm = writer.get_array_writable("matrix")
mm[:] = contact_matrix
mm.flush()

Reading

from gunz_cm.io.gnz import GZCMReader

reader = GZCMReader("matrix.gzcm")
matrix = reader.get_array("matrix")
print(f"Shape: {matrix.shape}, Metadata: {reader.metadata}")

Reading a v4 file (per-tile dispatch)

from gunz_cm.io.gnz import GZCMReader

# GZCMReader transparently dispatches to GzcmV4Reader when the
# header reports version=4. Same API for v1/v2/v3/v4.
reader = GZCMReader("matrix_v4.gzcm")

# Region-level access
region = reader.get_region_by_name("chr1_region_0")
print(f"Region {region.region_id}: codec_per_tile={region.codec_per_tile[:3]}...")

# Per-tile payload fetch (returns raw bytes + codec name; caller
# decodes via gunz_cm.compressions.decode_v4_tile_payload or any
# registered decoder).
payload, codec = reader.get_tile_payload(region.region_id, 0)
print(f"Tile 0: codec={codec}, payload={len(payload)} bytes")

Chunked Access

from gunz_cm.io.gnz import GZCMChunkedReader

chunked = GZCMChunkedReader("matrix.gzcm", chunk_size=1024)
for chunk, r, c in chunked.iter_chunks():
    process(chunk)

Streaming Normalization

from gunz_cm.io.gnz import kr_normalize_gzcm, ice_normalize_gzcm

# Knight-Ruiz normalization
weights = kr_normalize_gzcm("matrix.gzcm", "normalized.gzcm")

# ICE normalization
weights = ice_normalize_gzcm("matrix.gzcm", "normalized.gzcm")

---

Supported Codecs

Codecs are registered once via gunz_cm.compressions.register_codec(...) and shared by v3 and v4. v3 files pick ONE codec per file; v4 files can mix codecs per-tile (the per-tile picker chooses).

Codec	Description	Recommended	Native binary
bsc_cmc	BSC + CMC (v3 default)	Yes — best compression	requires bsc + CMC
cmc_zstd	CMC + ZSTD	Yes — good cross-platform	requires CMC
cmc	CMC only	For compatibility	requires CMC
bsc	BSC only	Good but less portable	requires bsc CLI on PATH (or GUNZ_CM_BSC_BIN)
zstd	ZSTD only (pure Python)	Fastest convert, least compression	none (pure Python)
lz4	LZ4 HC (pure Python, self-describing payload)	New in v2.28.0; great for v4	none (pure Python)

Checking which codecs are installed

Native codecs (CMC family, BSC) need an external library on the box. Use the probe before requesting a codec:

from gunz_cm.compressions import (
    list_available_codecs,
    codec_available,
    CodecUnavailableError,
    get_codec,
)

# Show only codecs that actually work on this machine.
list_available_codecs()
# e.g. ['bsc', 'cmc_zstd', 'lz4', 'zstd']  on a machine without CMC

# Check one codec:
codec_available("cmc")  # False if GUNZ_CM_CMC_DIR is unset

# Lookup; raises UnknownCodecError if unregistered, CodecUnavailableError
# if registered but binary missing. The error tells you which env var to
# set to make the codec available.
try:
    enc_cls, dec_cls, wire_format = get_codec("cmc")
except CodecUnavailableError as exc:
    print(f"CMC missing: {exc.reason}")

The picker (gunz_cm.compressions.scheme_picker) silently filters out unavailable codecs before scoring — you get a valid result without it ever crashing on the missing binary. Errors only surface when you explicitly opt into a missing codec, which is the v5.2 hardening rule.

---

Compression Ratios

Typical compression ratios for Hi-C data at10kb resolution:

Tile Size	Compression Ratio
64	~10-20x
128	~15-25x

---

See Also

• GZCM v1 Specification — Dense storage format

• GZCM v2 Specification — Tiled storage format

• GZCM v3 Specification — CMC compressed format

• GZCM v4 Specification — Regions + per-tile codec picker

• Codec Guide — Detailed codec selection

• GZCM v3 → v4 migration guide