Skip to content

codec

zarr.abc.codec

CodecJSON module-attribute 

CodecJSON = str | Mapping[str, object]

The widest type of JSON-like input that could specify a codec.

CodecJSON_V3 module-attribute 

CodecJSON_V3 = str | NamedConfig[str, Mapping[str, object]]

The JSON representation of a codec for Zarr V3.

ArrayArrayCodec dataclass

Bases: BaseCodec[NDBuffer, NDBuffer]

Base class for array-to-array codecs.

Source code in zarr/abc/codec.py 
185
186
class ArrayArrayCodec(BaseCodec[NDBuffer, NDBuffer]):
    """Base class for array-to-array codecs."""

compute_encoded_size abstractmethod 

compute_encoded_size(
    input_byte_length: int, chunk_spec: ArraySpec
) -> int

Given an input byte length, this method returns the output byte length. Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

Parameters:

  • input_byte_length (int) –
  • chunk_spec (ArraySpec) –

Returns:

Source code in zarr/abc/codec.py 
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
@abstractmethod
def compute_encoded_size(self, input_byte_length: int, chunk_spec: ArraySpec) -> int:
    """Given an input byte length, this method returns the output byte length.
    Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

    Parameters
    ----------
    input_byte_length : int
    chunk_spec : ArraySpec

    Returns
    -------
    int
    """
    ...

decode async 

decode(
    chunks_and_specs: Iterable[
        tuple[CodecOutput | None, ArraySpec]
    ],
) -> Iterable[CodecInput | None]

Decodes a batch of chunks. Chunks can be None in which case they are ignored by the codec.

Parameters:

  • chunks_and_specs (Iterable[tuple[CodecOutput | None, ArraySpec]]) –

    Ordered set of encoded chunks with their accompanying chunk spec.

Returns:

Source code in zarr/abc/codec.py 
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
async def decode(
    self,
    chunks_and_specs: Iterable[tuple[CodecOutput | None, ArraySpec]],
) -> Iterable[CodecInput | None]:
    """Decodes a batch of chunks.
    Chunks can be None in which case they are ignored by the codec.

    Parameters
    ----------
    chunks_and_specs : Iterable[tuple[CodecOutput | None, ArraySpec]]
        Ordered set of encoded chunks with their accompanying chunk spec.

    Returns
    -------
    Iterable[CodecInput | None]
    """
    return await _batching_helper(self._decode_single, chunks_and_specs)

encode async 

encode(
    chunks_and_specs: Iterable[
        tuple[CodecInput | None, ArraySpec]
    ],
) -> Iterable[CodecOutput | None]

Encodes a batch of chunks. Chunks can be None in which case they are ignored by the codec.

Parameters:

  • chunks_and_specs (Iterable[tuple[CodecInput | None, ArraySpec]]) –

    Ordered set of to-be-encoded chunks with their accompanying chunk spec.

Returns:

Source code in zarr/abc/codec.py 
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
async def encode(
    self,
    chunks_and_specs: Iterable[tuple[CodecInput | None, ArraySpec]],
) -> Iterable[CodecOutput | None]:
    """Encodes a batch of chunks.
    Chunks can be None in which case they are ignored by the codec.

    Parameters
    ----------
    chunks_and_specs : Iterable[tuple[CodecInput | None, ArraySpec]]
        Ordered set of to-be-encoded chunks with their accompanying chunk spec.

    Returns
    -------
    Iterable[CodecOutput | None]
    """
    return await _batching_helper(self._encode_single, chunks_and_specs)

evolve_from_array_spec 

evolve_from_array_spec(array_spec: ArraySpec) -> Self

Fills in codec configuration parameters that can be automatically inferred from the array metadata.

Parameters:

  • array_spec (ArraySpec) –

Returns:

Source code in zarr/abc/codec.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
def evolve_from_array_spec(self, array_spec: ArraySpec) -> Self:
    """Fills in codec configuration parameters that can be automatically
    inferred from the array metadata.

    Parameters
    ----------
    array_spec : ArraySpec

    Returns
    -------
    Self
    """
    return self

from_dict classmethod 

from_dict(data: dict[str, JSON]) -> Self

Create an instance of the model from a dictionary

Source code in zarr/abc/metadata.py 
40
41
42
43
44
45
46
@classmethod
def from_dict(cls, data: dict[str, JSON]) -> Self:
    """
    Create an instance of the model from a dictionary
    """

    return cls(**data)

resolve_metadata 

resolve_metadata(chunk_spec: ArraySpec) -> ArraySpec

Computed the spec of the chunk after it has been encoded by the codec. This is important for codecs that change the shape, data type or fill value of a chunk. The spec will then be used for subsequent codecs in the pipeline.

Parameters:

  • chunk_spec (ArraySpec) –

Returns:

  • ArraySpec –
Source code in zarr/abc/codec.py 
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
def resolve_metadata(self, chunk_spec: ArraySpec) -> ArraySpec:
    """Computed the spec of the chunk after it has been encoded by the codec.
    This is important for codecs that change the shape, data type or fill value of a chunk.
    The spec will then be used for subsequent codecs in the pipeline.

    Parameters
    ----------
    chunk_spec : ArraySpec

    Returns
    -------
    ArraySpec
    """
    return chunk_spec

to_dict 

to_dict() -> dict[str, JSON]

Recursively serialize this model to a dictionary. This method inspects the fields of self and calls x.to_dict() for any fields that are instances of Metadata. Sequences of Metadata are similarly recursed into, and the output of that recursion is collected in a list.

Source code in zarr/abc/metadata.py 
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
def to_dict(self) -> dict[str, JSON]:
    """
    Recursively serialize this model to a dictionary.
    This method inspects the fields of self and calls `x.to_dict()` for any fields that
    are instances of `Metadata`. Sequences of `Metadata` are similarly recursed into, and
    the output of that recursion is collected in a list.
    """
    out_dict = {}
    for field in fields(self):
        key = field.name
        value = getattr(self, key)
        if isinstance(value, Metadata):
            out_dict[field.name] = getattr(self, field.name).to_dict()
        elif isinstance(value, str):
            out_dict[key] = value
        elif isinstance(value, Sequence):
            out_dict[key] = tuple(v.to_dict() if isinstance(v, Metadata) else v for v in value)
        else:
            out_dict[key] = value

    return out_dict

validate 

validate(
    *,
    shape: tuple[int, ...],
    dtype: ZDType[TBaseDType, TBaseScalar],
    chunk_grid: ChunkGrid,
) -> None

Validates that the codec configuration is compatible with the array metadata. Raises errors when the codec configuration is not compatible.

Parameters:

  • shape (tuple[int, ...]) –

    The array shape

  • dtype (dtype[Any]) –

    The array data type

  • chunk_grid (ChunkGrid) –

    The array chunk grid

Source code in zarr/abc/codec.py 
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
def validate(
    self,
    *,
    shape: tuple[int, ...],
    dtype: ZDType[TBaseDType, TBaseScalar],
    chunk_grid: ChunkGrid,
) -> None:
    """Validates that the codec configuration is compatible with the array metadata.
    Raises errors when the codec configuration is not compatible.

    Parameters
    ----------
    shape : tuple[int, ...]
        The array shape
    dtype : np.dtype[Any]
        The array data type
    chunk_grid : ChunkGrid
        The array chunk grid
    """

ArrayBytesCodec dataclass

Bases: BaseCodec[NDBuffer, Buffer]

Base class for array-to-bytes codecs.

Source code in zarr/abc/codec.py 
189
190
class ArrayBytesCodec(BaseCodec[NDBuffer, Buffer]):
    """Base class for array-to-bytes codecs."""

compute_encoded_size abstractmethod 

compute_encoded_size(
    input_byte_length: int, chunk_spec: ArraySpec
) -> int

Given an input byte length, this method returns the output byte length. Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

Parameters:

  • input_byte_length (int) –
  • chunk_spec (ArraySpec) –

Returns:

Source code in zarr/abc/codec.py 
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
@abstractmethod
def compute_encoded_size(self, input_byte_length: int, chunk_spec: ArraySpec) -> int:
    """Given an input byte length, this method returns the output byte length.
    Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

    Parameters
    ----------
    input_byte_length : int
    chunk_spec : ArraySpec

    Returns
    -------
    int
    """
    ...

decode async 

decode(
    chunks_and_specs: Iterable[
        tuple[CodecOutput | None, ArraySpec]
    ],
) -> Iterable[CodecInput | None]

Decodes a batch of chunks. Chunks can be None in which case they are ignored by the codec.

Parameters:

  • chunks_and_specs (Iterable[tuple[CodecOutput | None, ArraySpec]]) –

    Ordered set of encoded chunks with their accompanying chunk spec.

Returns:

Source code in zarr/abc/codec.py 
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
async def decode(
    self,
    chunks_and_specs: Iterable[tuple[CodecOutput | None, ArraySpec]],
) -> Iterable[CodecInput | None]:
    """Decodes a batch of chunks.
    Chunks can be None in which case they are ignored by the codec.

    Parameters
    ----------
    chunks_and_specs : Iterable[tuple[CodecOutput | None, ArraySpec]]
        Ordered set of encoded chunks with their accompanying chunk spec.

    Returns
    -------
    Iterable[CodecInput | None]
    """
    return await _batching_helper(self._decode_single, chunks_and_specs)

encode async 

encode(
    chunks_and_specs: Iterable[
        tuple[CodecInput | None, ArraySpec]
    ],
) -> Iterable[CodecOutput | None]

Encodes a batch of chunks. Chunks can be None in which case they are ignored by the codec.

Parameters:

  • chunks_and_specs (Iterable[tuple[CodecInput | None, ArraySpec]]) –

    Ordered set of to-be-encoded chunks with their accompanying chunk spec.

Returns:

Source code in zarr/abc/codec.py 
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
async def encode(
    self,
    chunks_and_specs: Iterable[tuple[CodecInput | None, ArraySpec]],
) -> Iterable[CodecOutput | None]:
    """Encodes a batch of chunks.
    Chunks can be None in which case they are ignored by the codec.

    Parameters
    ----------
    chunks_and_specs : Iterable[tuple[CodecInput | None, ArraySpec]]
        Ordered set of to-be-encoded chunks with their accompanying chunk spec.

    Returns
    -------
    Iterable[CodecOutput | None]
    """
    return await _batching_helper(self._encode_single, chunks_and_specs)

evolve_from_array_spec 

evolve_from_array_spec(array_spec: ArraySpec) -> Self

Fills in codec configuration parameters that can be automatically inferred from the array metadata.

Parameters:

  • array_spec (ArraySpec) –

Returns:

Source code in zarr/abc/codec.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
def evolve_from_array_spec(self, array_spec: ArraySpec) -> Self:
    """Fills in codec configuration parameters that can be automatically
    inferred from the array metadata.

    Parameters
    ----------
    array_spec : ArraySpec

    Returns
    -------
    Self
    """
    return self

from_dict classmethod 

from_dict(data: dict[str, JSON]) -> Self

Create an instance of the model from a dictionary

Source code in zarr/abc/metadata.py 
40
41
42
43
44
45
46
@classmethod
def from_dict(cls, data: dict[str, JSON]) -> Self:
    """
    Create an instance of the model from a dictionary
    """

    return cls(**data)

resolve_metadata 

resolve_metadata(chunk_spec: ArraySpec) -> ArraySpec

Computed the spec of the chunk after it has been encoded by the codec. This is important for codecs that change the shape, data type or fill value of a chunk. The spec will then be used for subsequent codecs in the pipeline.

Parameters:

  • chunk_spec (ArraySpec) –

Returns:

  • ArraySpec –
Source code in zarr/abc/codec.py 
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
def resolve_metadata(self, chunk_spec: ArraySpec) -> ArraySpec:
    """Computed the spec of the chunk after it has been encoded by the codec.
    This is important for codecs that change the shape, data type or fill value of a chunk.
    The spec will then be used for subsequent codecs in the pipeline.

    Parameters
    ----------
    chunk_spec : ArraySpec

    Returns
    -------
    ArraySpec
    """
    return chunk_spec

to_dict 

to_dict() -> dict[str, JSON]

Recursively serialize this model to a dictionary. This method inspects the fields of self and calls x.to_dict() for any fields that are instances of Metadata. Sequences of Metadata are similarly recursed into, and the output of that recursion is collected in a list.

Source code in zarr/abc/metadata.py 
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
def to_dict(self) -> dict[str, JSON]:
    """
    Recursively serialize this model to a dictionary.
    This method inspects the fields of self and calls `x.to_dict()` for any fields that
    are instances of `Metadata`. Sequences of `Metadata` are similarly recursed into, and
    the output of that recursion is collected in a list.
    """
    out_dict = {}
    for field in fields(self):
        key = field.name
        value = getattr(self, key)
        if isinstance(value, Metadata):
            out_dict[field.name] = getattr(self, field.name).to_dict()
        elif isinstance(value, str):
            out_dict[key] = value
        elif isinstance(value, Sequence):
            out_dict[key] = tuple(v.to_dict() if isinstance(v, Metadata) else v for v in value)
        else:
            out_dict[key] = value

    return out_dict

validate 

validate(
    *,
    shape: tuple[int, ...],
    dtype: ZDType[TBaseDType, TBaseScalar],
    chunk_grid: ChunkGrid,
) -> None

Validates that the codec configuration is compatible with the array metadata. Raises errors when the codec configuration is not compatible.

Parameters:

  • shape (tuple[int, ...]) –

    The array shape

  • dtype (dtype[Any]) –

    The array data type

  • chunk_grid (ChunkGrid) –

    The array chunk grid

Source code in zarr/abc/codec.py 
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
def validate(
    self,
    *,
    shape: tuple[int, ...],
    dtype: ZDType[TBaseDType, TBaseScalar],
    chunk_grid: ChunkGrid,
) -> None:
    """Validates that the codec configuration is compatible with the array metadata.
    Raises errors when the codec configuration is not compatible.

    Parameters
    ----------
    shape : tuple[int, ...]
        The array shape
    dtype : np.dtype[Any]
        The array data type
    chunk_grid : ChunkGrid
        The array chunk grid
    """

ArrayBytesCodecPartialDecodeMixin

Mixin for array-to-bytes codecs that implement partial decoding.

Source code in zarr/abc/codec.py 
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
class ArrayBytesCodecPartialDecodeMixin:
    """Mixin for array-to-bytes codecs that implement partial decoding."""

    async def _decode_partial_single(
        self, byte_getter: ByteGetter, selection: SelectorTuple, chunk_spec: ArraySpec
    ) -> NDBuffer | None:
        raise NotImplementedError

    async def decode_partial(
        self,
        batch_info: Iterable[tuple[ByteGetter, SelectorTuple, ArraySpec]],
    ) -> Iterable[NDBuffer | None]:
        """Partially decodes a batch of chunks.
        This method determines parts of a chunk from the slice selection,
        fetches these parts from the store (via ByteGetter) and decodes them.

        Parameters
        ----------
        batch_info : Iterable[tuple[ByteGetter, SelectorTuple, ArraySpec]]
            Ordered set of information about slices of encoded chunks.
            The slice selection determines which parts of the chunk will be fetched.
            The ByteGetter is used to fetch the necessary bytes.
            The chunk spec contains information about the construction of an array from the bytes.

        Returns
        -------
        Iterable[NDBuffer | None]
        """
        return await concurrent_map(
            list(batch_info),
            self._decode_partial_single,
            config.get("async.concurrency"),
        )

decode_partial async 

decode_partial(
    batch_info: Iterable[
        tuple[ByteGetter, SelectorTuple, ArraySpec]
    ],
) -> Iterable[NDBuffer | None]

Partially decodes a batch of chunks. This method determines parts of a chunk from the slice selection, fetches these parts from the store (via ByteGetter) and decodes them.

Parameters:

  • batch_info (Iterable[tuple[ByteGetter, SelectorTuple, ArraySpec]]) –

    Ordered set of information about slices of encoded chunks. The slice selection determines which parts of the chunk will be fetched. The ByteGetter is used to fetch the necessary bytes. The chunk spec contains information about the construction of an array from the bytes.

Returns:

Source code in zarr/abc/codec.py 
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
async def decode_partial(
    self,
    batch_info: Iterable[tuple[ByteGetter, SelectorTuple, ArraySpec]],
) -> Iterable[NDBuffer | None]:
    """Partially decodes a batch of chunks.
    This method determines parts of a chunk from the slice selection,
    fetches these parts from the store (via ByteGetter) and decodes them.

    Parameters
    ----------
    batch_info : Iterable[tuple[ByteGetter, SelectorTuple, ArraySpec]]
        Ordered set of information about slices of encoded chunks.
        The slice selection determines which parts of the chunk will be fetched.
        The ByteGetter is used to fetch the necessary bytes.
        The chunk spec contains information about the construction of an array from the bytes.

    Returns
    -------
    Iterable[NDBuffer | None]
    """
    return await concurrent_map(
        list(batch_info),
        self._decode_partial_single,
        config.get("async.concurrency"),
    )

ArrayBytesCodecPartialEncodeMixin

Mixin for array-to-bytes codecs that implement partial encoding.

Source code in zarr/abc/codec.py 
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
class ArrayBytesCodecPartialEncodeMixin:
    """Mixin for array-to-bytes codecs that implement partial encoding."""

    async def _encode_partial_single(
        self,
        byte_setter: ByteSetter,
        chunk_array: NDBuffer,
        selection: SelectorTuple,
        chunk_spec: ArraySpec,
    ) -> None:
        raise NotImplementedError  # pragma: no cover

    async def encode_partial(
        self,
        batch_info: Iterable[tuple[ByteSetter, NDBuffer, SelectorTuple, ArraySpec]],
    ) -> None:
        """Partially encodes a batch of chunks.
        This method determines parts of a chunk from the slice selection, encodes them and
        writes these parts to the store (via ByteSetter).
        If merging with existing chunk data in the store is necessary, this method will
        read from the store first and perform the merge.

        Parameters
        ----------
        batch_info : Iterable[tuple[ByteSetter, NDBuffer, SelectorTuple, ArraySpec]]
            Ordered set of information about slices of to-be-encoded chunks.
            The slice selection determines which parts of the chunk will be encoded.
            The ByteSetter is used to write the necessary bytes and fetch bytes for existing chunk data.
            The chunk spec contains information about the chunk.
        """
        await concurrent_map(
            list(batch_info),
            self._encode_partial_single,
            config.get("async.concurrency"),
        )

encode_partial async 

encode_partial(
    batch_info: Iterable[
        tuple[
            ByteSetter, NDBuffer, SelectorTuple, ArraySpec
        ]
    ],
) -> None

Partially encodes a batch of chunks. This method determines parts of a chunk from the slice selection, encodes them and writes these parts to the store (via ByteSetter). If merging with existing chunk data in the store is necessary, this method will read from the store first and perform the merge.

Parameters:

  • batch_info (Iterable[tuple[ByteSetter, NDBuffer, SelectorTuple, ArraySpec]]) –

    Ordered set of information about slices of to-be-encoded chunks. The slice selection determines which parts of the chunk will be encoded. The ByteSetter is used to write the necessary bytes and fetch bytes for existing chunk data. The chunk spec contains information about the chunk.

Source code in zarr/abc/codec.py 
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
async def encode_partial(
    self,
    batch_info: Iterable[tuple[ByteSetter, NDBuffer, SelectorTuple, ArraySpec]],
) -> None:
    """Partially encodes a batch of chunks.
    This method determines parts of a chunk from the slice selection, encodes them and
    writes these parts to the store (via ByteSetter).
    If merging with existing chunk data in the store is necessary, this method will
    read from the store first and perform the merge.

    Parameters
    ----------
    batch_info : Iterable[tuple[ByteSetter, NDBuffer, SelectorTuple, ArraySpec]]
        Ordered set of information about slices of to-be-encoded chunks.
        The slice selection determines which parts of the chunk will be encoded.
        The ByteSetter is used to write the necessary bytes and fetch bytes for existing chunk data.
        The chunk spec contains information about the chunk.
    """
    await concurrent_map(
        list(batch_info),
        self._encode_partial_single,
        config.get("async.concurrency"),
    )

BaseCodec dataclass

Bases: Metadata, Generic[CodecInput, CodecOutput]

Generic base class for codecs.

Codecs can be registered via zarr.codecs.registry.

Warnings

This class is not intended to be directly, please use ArrayArrayCodec, ArrayBytesCodec or BytesBytesCodec for subclassing.

Source code in zarr/abc/codec.py 
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
class BaseCodec(Metadata, Generic[CodecInput, CodecOutput]):
    """Generic base class for codecs.

    Codecs can be registered via zarr.codecs.registry.

    Warnings
    --------
    This class is not intended to be directly, please use
    ArrayArrayCodec, ArrayBytesCodec or BytesBytesCodec for subclassing.
    """

    is_fixed_size: bool

    @abstractmethod
    def compute_encoded_size(self, input_byte_length: int, chunk_spec: ArraySpec) -> int:
        """Given an input byte length, this method returns the output byte length.
        Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

        Parameters
        ----------
        input_byte_length : int
        chunk_spec : ArraySpec

        Returns
        -------
        int
        """
        ...

    def resolve_metadata(self, chunk_spec: ArraySpec) -> ArraySpec:
        """Computed the spec of the chunk after it has been encoded by the codec.
        This is important for codecs that change the shape, data type or fill value of a chunk.
        The spec will then be used for subsequent codecs in the pipeline.

        Parameters
        ----------
        chunk_spec : ArraySpec

        Returns
        -------
        ArraySpec
        """
        return chunk_spec

    def evolve_from_array_spec(self, array_spec: ArraySpec) -> Self:
        """Fills in codec configuration parameters that can be automatically
        inferred from the array metadata.

        Parameters
        ----------
        array_spec : ArraySpec

        Returns
        -------
        Self
        """
        return self

    def validate(
        self,
        *,
        shape: tuple[int, ...],
        dtype: ZDType[TBaseDType, TBaseScalar],
        chunk_grid: ChunkGrid,
    ) -> None:
        """Validates that the codec configuration is compatible with the array metadata.
        Raises errors when the codec configuration is not compatible.

        Parameters
        ----------
        shape : tuple[int, ...]
            The array shape
        dtype : np.dtype[Any]
            The array data type
        chunk_grid : ChunkGrid
            The array chunk grid
        """

    async def _decode_single(self, chunk_data: CodecOutput, chunk_spec: ArraySpec) -> CodecInput:
        raise NotImplementedError  # pragma: no cover

    async def decode(
        self,
        chunks_and_specs: Iterable[tuple[CodecOutput | None, ArraySpec]],
    ) -> Iterable[CodecInput | None]:
        """Decodes a batch of chunks.
        Chunks can be None in which case they are ignored by the codec.

        Parameters
        ----------
        chunks_and_specs : Iterable[tuple[CodecOutput | None, ArraySpec]]
            Ordered set of encoded chunks with their accompanying chunk spec.

        Returns
        -------
        Iterable[CodecInput | None]
        """
        return await _batching_helper(self._decode_single, chunks_and_specs)

    async def _encode_single(
        self, chunk_data: CodecInput, chunk_spec: ArraySpec
    ) -> CodecOutput | None:
        raise NotImplementedError  # pragma: no cover

    async def encode(
        self,
        chunks_and_specs: Iterable[tuple[CodecInput | None, ArraySpec]],
    ) -> Iterable[CodecOutput | None]:
        """Encodes a batch of chunks.
        Chunks can be None in which case they are ignored by the codec.

        Parameters
        ----------
        chunks_and_specs : Iterable[tuple[CodecInput | None, ArraySpec]]
            Ordered set of to-be-encoded chunks with their accompanying chunk spec.

        Returns
        -------
        Iterable[CodecOutput | None]
        """
        return await _batching_helper(self._encode_single, chunks_and_specs)

compute_encoded_size abstractmethod 

compute_encoded_size(
    input_byte_length: int, chunk_spec: ArraySpec
) -> int

Given an input byte length, this method returns the output byte length. Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

Parameters:

  • input_byte_length (int) –
  • chunk_spec (ArraySpec) –

Returns:

Source code in zarr/abc/codec.py 
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
@abstractmethod
def compute_encoded_size(self, input_byte_length: int, chunk_spec: ArraySpec) -> int:
    """Given an input byte length, this method returns the output byte length.
    Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

    Parameters
    ----------
    input_byte_length : int
    chunk_spec : ArraySpec

    Returns
    -------
    int
    """
    ...

decode async 

decode(
    chunks_and_specs: Iterable[
        tuple[CodecOutput | None, ArraySpec]
    ],
) -> Iterable[CodecInput | None]

Decodes a batch of chunks. Chunks can be None in which case they are ignored by the codec.

Parameters:

  • chunks_and_specs (Iterable[tuple[CodecOutput | None, ArraySpec]]) –

    Ordered set of encoded chunks with their accompanying chunk spec.

Returns:

Source code in zarr/abc/codec.py 
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
async def decode(
    self,
    chunks_and_specs: Iterable[tuple[CodecOutput | None, ArraySpec]],
) -> Iterable[CodecInput | None]:
    """Decodes a batch of chunks.
    Chunks can be None in which case they are ignored by the codec.

    Parameters
    ----------
    chunks_and_specs : Iterable[tuple[CodecOutput | None, ArraySpec]]
        Ordered set of encoded chunks with their accompanying chunk spec.

    Returns
    -------
    Iterable[CodecInput | None]
    """
    return await _batching_helper(self._decode_single, chunks_and_specs)

encode async 

encode(
    chunks_and_specs: Iterable[
        tuple[CodecInput | None, ArraySpec]
    ],
) -> Iterable[CodecOutput | None]

Encodes a batch of chunks. Chunks can be None in which case they are ignored by the codec.

Parameters:

  • chunks_and_specs (Iterable[tuple[CodecInput | None, ArraySpec]]) –

    Ordered set of to-be-encoded chunks with their accompanying chunk spec.

Returns:

Source code in zarr/abc/codec.py 
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
async def encode(
    self,
    chunks_and_specs: Iterable[tuple[CodecInput | None, ArraySpec]],
) -> Iterable[CodecOutput | None]:
    """Encodes a batch of chunks.
    Chunks can be None in which case they are ignored by the codec.

    Parameters
    ----------
    chunks_and_specs : Iterable[tuple[CodecInput | None, ArraySpec]]
        Ordered set of to-be-encoded chunks with their accompanying chunk spec.

    Returns
    -------
    Iterable[CodecOutput | None]
    """
    return await _batching_helper(self._encode_single, chunks_and_specs)

evolve_from_array_spec 

evolve_from_array_spec(array_spec: ArraySpec) -> Self

Fills in codec configuration parameters that can be automatically inferred from the array metadata.

Parameters:

  • array_spec (ArraySpec) –

Returns:

Source code in zarr/abc/codec.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
def evolve_from_array_spec(self, array_spec: ArraySpec) -> Self:
    """Fills in codec configuration parameters that can be automatically
    inferred from the array metadata.

    Parameters
    ----------
    array_spec : ArraySpec

    Returns
    -------
    Self
    """
    return self

from_dict classmethod 

from_dict(data: dict[str, JSON]) -> Self

Create an instance of the model from a dictionary

Source code in zarr/abc/metadata.py 
40
41
42
43
44
45
46
@classmethod
def from_dict(cls, data: dict[str, JSON]) -> Self:
    """
    Create an instance of the model from a dictionary
    """

    return cls(**data)

resolve_metadata 

resolve_metadata(chunk_spec: ArraySpec) -> ArraySpec

Computed the spec of the chunk after it has been encoded by the codec. This is important for codecs that change the shape, data type or fill value of a chunk. The spec will then be used for subsequent codecs in the pipeline.

Parameters:

  • chunk_spec (ArraySpec) –

Returns:

  • ArraySpec –
Source code in zarr/abc/codec.py 
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
def resolve_metadata(self, chunk_spec: ArraySpec) -> ArraySpec:
    """Computed the spec of the chunk after it has been encoded by the codec.
    This is important for codecs that change the shape, data type or fill value of a chunk.
    The spec will then be used for subsequent codecs in the pipeline.

    Parameters
    ----------
    chunk_spec : ArraySpec

    Returns
    -------
    ArraySpec
    """
    return chunk_spec

to_dict 

to_dict() -> dict[str, JSON]

Recursively serialize this model to a dictionary. This method inspects the fields of self and calls x.to_dict() for any fields that are instances of Metadata. Sequences of Metadata are similarly recursed into, and the output of that recursion is collected in a list.

Source code in zarr/abc/metadata.py 
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
def to_dict(self) -> dict[str, JSON]:
    """
    Recursively serialize this model to a dictionary.
    This method inspects the fields of self and calls `x.to_dict()` for any fields that
    are instances of `Metadata`. Sequences of `Metadata` are similarly recursed into, and
    the output of that recursion is collected in a list.
    """
    out_dict = {}
    for field in fields(self):
        key = field.name
        value = getattr(self, key)
        if isinstance(value, Metadata):
            out_dict[field.name] = getattr(self, field.name).to_dict()
        elif isinstance(value, str):
            out_dict[key] = value
        elif isinstance(value, Sequence):
            out_dict[key] = tuple(v.to_dict() if isinstance(v, Metadata) else v for v in value)
        else:
            out_dict[key] = value

    return out_dict

validate 

validate(
    *,
    shape: tuple[int, ...],
    dtype: ZDType[TBaseDType, TBaseScalar],
    chunk_grid: ChunkGrid,
) -> None

Validates that the codec configuration is compatible with the array metadata. Raises errors when the codec configuration is not compatible.

Parameters:

  • shape (tuple[int, ...]) –

    The array shape

  • dtype (dtype[Any]) –

    The array data type

  • chunk_grid (ChunkGrid) –

    The array chunk grid

Source code in zarr/abc/codec.py 
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
def validate(
    self,
    *,
    shape: tuple[int, ...],
    dtype: ZDType[TBaseDType, TBaseScalar],
    chunk_grid: ChunkGrid,
) -> None:
    """Validates that the codec configuration is compatible with the array metadata.
    Raises errors when the codec configuration is not compatible.

    Parameters
    ----------
    shape : tuple[int, ...]
        The array shape
    dtype : np.dtype[Any]
        The array data type
    chunk_grid : ChunkGrid
        The array chunk grid
    """

BytesBytesCodec dataclass

Bases: BaseCodec[Buffer, Buffer]

Base class for bytes-to-bytes codecs.

Source code in zarr/abc/codec.py 
193
194
class BytesBytesCodec(BaseCodec[Buffer, Buffer]):
    """Base class for bytes-to-bytes codecs."""

compute_encoded_size abstractmethod 

compute_encoded_size(
    input_byte_length: int, chunk_spec: ArraySpec
) -> int

Given an input byte length, this method returns the output byte length. Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

Parameters:

  • input_byte_length (int) –
  • chunk_spec (ArraySpec) –

Returns:

Source code in zarr/abc/codec.py 
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
@abstractmethod
def compute_encoded_size(self, input_byte_length: int, chunk_spec: ArraySpec) -> int:
    """Given an input byte length, this method returns the output byte length.
    Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

    Parameters
    ----------
    input_byte_length : int
    chunk_spec : ArraySpec

    Returns
    -------
    int
    """
    ...

decode async 

decode(
    chunks_and_specs: Iterable[
        tuple[CodecOutput | None, ArraySpec]
    ],
) -> Iterable[CodecInput | None]

Decodes a batch of chunks. Chunks can be None in which case they are ignored by the codec.

Parameters:

  • chunks_and_specs (Iterable[tuple[CodecOutput | None, ArraySpec]]) –

    Ordered set of encoded chunks with their accompanying chunk spec.

Returns:

Source code in zarr/abc/codec.py 
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
async def decode(
    self,
    chunks_and_specs: Iterable[tuple[CodecOutput | None, ArraySpec]],
) -> Iterable[CodecInput | None]:
    """Decodes a batch of chunks.
    Chunks can be None in which case they are ignored by the codec.

    Parameters
    ----------
    chunks_and_specs : Iterable[tuple[CodecOutput | None, ArraySpec]]
        Ordered set of encoded chunks with their accompanying chunk spec.

    Returns
    -------
    Iterable[CodecInput | None]
    """
    return await _batching_helper(self._decode_single, chunks_and_specs)

encode async 

encode(
    chunks_and_specs: Iterable[
        tuple[CodecInput | None, ArraySpec]
    ],
) -> Iterable[CodecOutput | None]

Encodes a batch of chunks. Chunks can be None in which case they are ignored by the codec.

Parameters:

  • chunks_and_specs (Iterable[tuple[CodecInput | None, ArraySpec]]) –

    Ordered set of to-be-encoded chunks with their accompanying chunk spec.

Returns:

Source code in zarr/abc/codec.py 
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
async def encode(
    self,
    chunks_and_specs: Iterable[tuple[CodecInput | None, ArraySpec]],
) -> Iterable[CodecOutput | None]:
    """Encodes a batch of chunks.
    Chunks can be None in which case they are ignored by the codec.

    Parameters
    ----------
    chunks_and_specs : Iterable[tuple[CodecInput | None, ArraySpec]]
        Ordered set of to-be-encoded chunks with their accompanying chunk spec.

    Returns
    -------
    Iterable[CodecOutput | None]
    """
    return await _batching_helper(self._encode_single, chunks_and_specs)

evolve_from_array_spec 

evolve_from_array_spec(array_spec: ArraySpec) -> Self

Fills in codec configuration parameters that can be automatically inferred from the array metadata.

Parameters:

  • array_spec (ArraySpec) –

Returns:

Source code in zarr/abc/codec.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
def evolve_from_array_spec(self, array_spec: ArraySpec) -> Self:
    """Fills in codec configuration parameters that can be automatically
    inferred from the array metadata.

    Parameters
    ----------
    array_spec : ArraySpec

    Returns
    -------
    Self
    """
    return self

from_dict classmethod 

from_dict(data: dict[str, JSON]) -> Self

Create an instance of the model from a dictionary

Source code in zarr/abc/metadata.py 
40
41
42
43
44
45
46
@classmethod
def from_dict(cls, data: dict[str, JSON]) -> Self:
    """
    Create an instance of the model from a dictionary
    """

    return cls(**data)

resolve_metadata 

resolve_metadata(chunk_spec: ArraySpec) -> ArraySpec

Computed the spec of the chunk after it has been encoded by the codec. This is important for codecs that change the shape, data type or fill value of a chunk. The spec will then be used for subsequent codecs in the pipeline.

Parameters:

  • chunk_spec (ArraySpec) –

Returns:

  • ArraySpec –
Source code in zarr/abc/codec.py 
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
def resolve_metadata(self, chunk_spec: ArraySpec) -> ArraySpec:
    """Computed the spec of the chunk after it has been encoded by the codec.
    This is important for codecs that change the shape, data type or fill value of a chunk.
    The spec will then be used for subsequent codecs in the pipeline.

    Parameters
    ----------
    chunk_spec : ArraySpec

    Returns
    -------
    ArraySpec
    """
    return chunk_spec

to_dict 

to_dict() -> dict[str, JSON]

Recursively serialize this model to a dictionary. This method inspects the fields of self and calls x.to_dict() for any fields that are instances of Metadata. Sequences of Metadata are similarly recursed into, and the output of that recursion is collected in a list.

Source code in zarr/abc/metadata.py 
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
def to_dict(self) -> dict[str, JSON]:
    """
    Recursively serialize this model to a dictionary.
    This method inspects the fields of self and calls `x.to_dict()` for any fields that
    are instances of `Metadata`. Sequences of `Metadata` are similarly recursed into, and
    the output of that recursion is collected in a list.
    """
    out_dict = {}
    for field in fields(self):
        key = field.name
        value = getattr(self, key)
        if isinstance(value, Metadata):
            out_dict[field.name] = getattr(self, field.name).to_dict()
        elif isinstance(value, str):
            out_dict[key] = value
        elif isinstance(value, Sequence):
            out_dict[key] = tuple(v.to_dict() if isinstance(v, Metadata) else v for v in value)
        else:
            out_dict[key] = value

    return out_dict

validate 

validate(
    *,
    shape: tuple[int, ...],
    dtype: ZDType[TBaseDType, TBaseScalar],
    chunk_grid: ChunkGrid,
) -> None

Validates that the codec configuration is compatible with the array metadata. Raises errors when the codec configuration is not compatible.

Parameters:

  • shape (tuple[int, ...]) –

    The array shape

  • dtype (dtype[Any]) –

    The array data type

  • chunk_grid (ChunkGrid) –

    The array chunk grid

Source code in zarr/abc/codec.py 
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
def validate(
    self,
    *,
    shape: tuple[int, ...],
    dtype: ZDType[TBaseDType, TBaseScalar],
    chunk_grid: ChunkGrid,
) -> None:
    """Validates that the codec configuration is compatible with the array metadata.
    Raises errors when the codec configuration is not compatible.

    Parameters
    ----------
    shape : tuple[int, ...]
        The array shape
    dtype : np.dtype[Any]
        The array data type
    chunk_grid : ChunkGrid
        The array chunk grid
    """

CodecJSON_V2

Bases: TypedDict, Generic[TName]

The JSON representation of a codec for Zarr V2

Source code in zarr/abc/codec.py 
43
44
45
46
class CodecJSON_V2(TypedDict, Generic[TName]):
    """The JSON representation of a codec for Zarr V2"""

    id: ReadOnly[TName]

CodecPipeline

Base class for implementing CodecPipeline. A CodecPipeline implements the read and write paths for chunk data. On the read path, it is responsible for fetching chunks from a store (via ByteGetter), decoding them and assembling an output array. On the write path, it encodes the chunks and writes them to a store (via ByteSetter).

Source code in zarr/abc/codec.py 
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
class CodecPipeline:
    """Base class for implementing CodecPipeline.
    A CodecPipeline implements the read and write paths for chunk data.
    On the read path, it is responsible for fetching chunks from a store (via ByteGetter),
    decoding them and assembling an output array. On the write path, it encodes the chunks
    and writes them to a store (via ByteSetter)."""

    @abstractmethod
    def evolve_from_array_spec(self, array_spec: ArraySpec) -> Self:
        """Fills in codec configuration parameters that can be automatically
        inferred from the array metadata.

        Parameters
        ----------
        array_spec : ArraySpec

        Returns
        -------
        Self
        """
        ...

    @classmethod
    @abstractmethod
    def from_codecs(cls, codecs: Iterable[Codec]) -> Self:
        """Creates a codec pipeline from an iterable of codecs.

        Parameters
        ----------
        codecs : Iterable[Codec]

        Returns
        -------
        Self
        """
        ...

    @classmethod
    def from_array_metadata_and_store(cls, array_metadata: ArrayMetadata, store: Store) -> Self:
        """Creates a codec pipeline from array metadata and a store path.

        Raises NotImplementedError by default, indicating the CodecPipeline must be created with from_codecs instead.

        Parameters
        ----------
        array_metadata : ArrayMetadata
        store : Store

        Returns
        -------
        Self
        """
        raise NotImplementedError(
            f"'{type(cls).__name__}' does not implement CodecPipeline.from_array_metadata_and_store."
        )

    @property
    @abstractmethod
    def supports_partial_decode(self) -> bool: ...

    @property
    @abstractmethod
    def supports_partial_encode(self) -> bool: ...

    @abstractmethod
    def validate(
        self,
        *,
        shape: tuple[int, ...],
        dtype: ZDType[TBaseDType, TBaseScalar],
        chunk_grid: ChunkGrid,
    ) -> None:
        """Validates that all codec configurations are compatible with the array metadata.
        Raises errors when a codec configuration is not compatible.

        Parameters
        ----------
        shape : tuple[int, ...]
            The array shape
        dtype : np.dtype[Any]
            The array data type
        chunk_grid : ChunkGrid
            The array chunk grid
        """
        ...

    @abstractmethod
    def compute_encoded_size(self, byte_length: int, array_spec: ArraySpec) -> int:
        """Given an input byte length, this method returns the output byte length.
        Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

        Parameters
        ----------
        byte_length : int
        array_spec : ArraySpec

        Returns
        -------
        int
        """
        ...

    @abstractmethod
    async def decode(
        self,
        chunk_bytes_and_specs: Iterable[tuple[Buffer | None, ArraySpec]],
    ) -> Iterable[NDBuffer | None]:
        """Decodes a batch of chunks.
        Chunks can be None in which case they are ignored by the codec.

        Parameters
        ----------
        chunk_bytes_and_specs : Iterable[tuple[Buffer | None, ArraySpec]]
            Ordered set of encoded chunks with their accompanying chunk spec.

        Returns
        -------
        Iterable[NDBuffer | None]
        """
        ...

    @abstractmethod
    async def encode(
        self,
        chunk_arrays_and_specs: Iterable[tuple[NDBuffer | None, ArraySpec]],
    ) -> Iterable[Buffer | None]:
        """Encodes a batch of chunks.
        Chunks can be None in which case they are ignored by the codec.

        Parameters
        ----------
        chunk_arrays_and_specs : Iterable[tuple[NDBuffer | None, ArraySpec]]
            Ordered set of to-be-encoded chunks with their accompanying chunk spec.

        Returns
        -------
        Iterable[Buffer | None]
        """
        ...

    @abstractmethod
    async def read(
        self,
        batch_info: Iterable[tuple[ByteGetter, ArraySpec, SelectorTuple, SelectorTuple, bool]],
        out: NDBuffer,
        drop_axes: tuple[int, ...] = (),
    ) -> None:
        """Reads chunk data from the store, decodes it and writes it into an output array.
        Partial decoding may be utilized if the codecs and stores support it.

        Parameters
        ----------
        batch_info : Iterable[tuple[ByteGetter, ArraySpec, SelectorTuple, SelectorTuple]]
            Ordered set of information about the chunks.
            The first slice selection determines which parts of the chunk will be fetched.
            The second slice selection determines where in the output array the chunk data will be written.
            The ByteGetter is used to fetch the necessary bytes.
            The chunk spec contains information about the construction of an array from the bytes.

            If the Store returns ``None`` for a chunk, then the chunk was not
            written and the implementation must set the values of that chunk (or
            ``out``) to the fill value for the array.

        out : NDBuffer
        """
        ...

    @abstractmethod
    async def write(
        self,
        batch_info: Iterable[tuple[ByteSetter, ArraySpec, SelectorTuple, SelectorTuple, bool]],
        value: NDBuffer,
        drop_axes: tuple[int, ...] = (),
    ) -> None:
        """Encodes chunk data and writes it to the store.
        Merges with existing chunk data by reading first, if necessary.
        Partial encoding may be utilized if the codecs and stores support it.

        Parameters
        ----------
        batch_info : Iterable[tuple[ByteSetter, ArraySpec, SelectorTuple, SelectorTuple]]
            Ordered set of information about the chunks.
            The first slice selection determines which parts of the chunk will be encoded.
            The second slice selection determines where in the value array the chunk data is located.
            The ByteSetter is used to fetch and write the necessary bytes.
            The chunk spec contains information about the chunk.
        value : NDBuffer
        """
        ...

compute_encoded_size abstractmethod 

compute_encoded_size(
    byte_length: int, array_spec: ArraySpec
) -> int

Given an input byte length, this method returns the output byte length. Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

Parameters:

  • byte_length (int) –
  • array_spec (ArraySpec) –

Returns:

Source code in zarr/abc/codec.py 
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
@abstractmethod
def compute_encoded_size(self, byte_length: int, array_spec: ArraySpec) -> int:
    """Given an input byte length, this method returns the output byte length.
    Raises a NotImplementedError for codecs with variable-sized outputs (e.g. compressors).

    Parameters
    ----------
    byte_length : int
    array_spec : ArraySpec

    Returns
    -------
    int
    """
    ...

decode abstractmethod async 

decode(
    chunk_bytes_and_specs: Iterable[
        tuple[Buffer | None, ArraySpec]
    ],
) -> Iterable[NDBuffer | None]

Decodes a batch of chunks. Chunks can be None in which case they are ignored by the codec.

Parameters:

  • chunk_bytes_and_specs (Iterable[tuple[Buffer | None, ArraySpec]]) –

    Ordered set of encoded chunks with their accompanying chunk spec.

Returns:

Source code in zarr/abc/codec.py 
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
@abstractmethod
async def decode(
    self,
    chunk_bytes_and_specs: Iterable[tuple[Buffer | None, ArraySpec]],
) -> Iterable[NDBuffer | None]:
    """Decodes a batch of chunks.
    Chunks can be None in which case they are ignored by the codec.

    Parameters
    ----------
    chunk_bytes_and_specs : Iterable[tuple[Buffer | None, ArraySpec]]
        Ordered set of encoded chunks with their accompanying chunk spec.

    Returns
    -------
    Iterable[NDBuffer | None]
    """
    ...

encode abstractmethod async 

encode(
    chunk_arrays_and_specs: Iterable[
        tuple[NDBuffer | None, ArraySpec]
    ],
) -> Iterable[Buffer | None]

Encodes a batch of chunks. Chunks can be None in which case they are ignored by the codec.

Parameters:

  • chunk_arrays_and_specs (Iterable[tuple[NDBuffer | None, ArraySpec]]) –

    Ordered set of to-be-encoded chunks with their accompanying chunk spec.

Returns:

Source code in zarr/abc/codec.py 
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
@abstractmethod
async def encode(
    self,
    chunk_arrays_and_specs: Iterable[tuple[NDBuffer | None, ArraySpec]],
) -> Iterable[Buffer | None]:
    """Encodes a batch of chunks.
    Chunks can be None in which case they are ignored by the codec.

    Parameters
    ----------
    chunk_arrays_and_specs : Iterable[tuple[NDBuffer | None, ArraySpec]]
        Ordered set of to-be-encoded chunks with their accompanying chunk spec.

    Returns
    -------
    Iterable[Buffer | None]
    """
    ...

evolve_from_array_spec abstractmethod 

evolve_from_array_spec(array_spec: ArraySpec) -> Self

Fills in codec configuration parameters that can be automatically inferred from the array metadata.

Parameters:

  • array_spec (ArraySpec) –

Returns:

Source code in zarr/abc/codec.py 
279
280
281
282
283
284
285
286
287
288
289
290
291
292
@abstractmethod
def evolve_from_array_spec(self, array_spec: ArraySpec) -> Self:
    """Fills in codec configuration parameters that can be automatically
    inferred from the array metadata.

    Parameters
    ----------
    array_spec : ArraySpec

    Returns
    -------
    Self
    """
    ...

from_array_metadata_and_store classmethod 

from_array_metadata_and_store(
    array_metadata: ArrayMetadata, store: Store
) -> Self

Creates a codec pipeline from array metadata and a store path.

Raises NotImplementedError by default, indicating the CodecPipeline must be created with from_codecs instead.

Parameters:

  • array_metadata (ArrayMetadata) –
  • store (Store) –

Returns:

Source code in zarr/abc/codec.py 
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
@classmethod
def from_array_metadata_and_store(cls, array_metadata: ArrayMetadata, store: Store) -> Self:
    """Creates a codec pipeline from array metadata and a store path.

    Raises NotImplementedError by default, indicating the CodecPipeline must be created with from_codecs instead.

    Parameters
    ----------
    array_metadata : ArrayMetadata
    store : Store

    Returns
    -------
    Self
    """
    raise NotImplementedError(
        f"'{type(cls).__name__}' does not implement CodecPipeline.from_array_metadata_and_store."
    )

from_codecs abstractmethod classmethod 

from_codecs(codecs: Iterable[Codec]) -> Self

Creates a codec pipeline from an iterable of codecs.

Parameters:

Returns:

Source code in zarr/abc/codec.py 
294
295
296
297
298
299
300
301
302
303
304
305
306
307
@classmethod
@abstractmethod
def from_codecs(cls, codecs: Iterable[Codec]) -> Self:
    """Creates a codec pipeline from an iterable of codecs.

    Parameters
    ----------
    codecs : Iterable[Codec]

    Returns
    -------
    Self
    """
    ...

read abstractmethod async 

read(
    batch_info: Iterable[
        tuple[
            ByteGetter,
            ArraySpec,
            SelectorTuple,
            SelectorTuple,
            bool,
        ]
    ],
    out: NDBuffer,
    drop_axes: tuple[int, ...] = (),
) -> None

Reads chunk data from the store, decodes it and writes it into an output array. Partial decoding may be utilized if the codecs and stores support it.

Parameters:

  • batch_info (Iterable[tuple[ByteGetter, ArraySpec, SelectorTuple, SelectorTuple]]) –

    Ordered set of information about the chunks. The first slice selection determines which parts of the chunk will be fetched. The second slice selection determines where in the output array the chunk data will be written. The ByteGetter is used to fetch the necessary bytes. The chunk spec contains information about the construction of an array from the bytes.

    If the Store returns None for a chunk, then the chunk was not written and the implementation must set the values of that chunk (or out) to the fill value for the array.

  • out (NDBuffer) –
Source code in zarr/abc/codec.py 
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
@abstractmethod
async def read(
    self,
    batch_info: Iterable[tuple[ByteGetter, ArraySpec, SelectorTuple, SelectorTuple, bool]],
    out: NDBuffer,
    drop_axes: tuple[int, ...] = (),
) -> None:
    """Reads chunk data from the store, decodes it and writes it into an output array.
    Partial decoding may be utilized if the codecs and stores support it.

    Parameters
    ----------
    batch_info : Iterable[tuple[ByteGetter, ArraySpec, SelectorTuple, SelectorTuple]]
        Ordered set of information about the chunks.
        The first slice selection determines which parts of the chunk will be fetched.
        The second slice selection determines where in the output array the chunk data will be written.
        The ByteGetter is used to fetch the necessary bytes.
        The chunk spec contains information about the construction of an array from the bytes.

        If the Store returns ``None`` for a chunk, then the chunk was not
        written and the implementation must set the values of that chunk (or
        ``out``) to the fill value for the array.

    out : NDBuffer
    """
    ...

validate abstractmethod 

validate(
    *,
    shape: tuple[int, ...],
    dtype: ZDType[TBaseDType, TBaseScalar],
    chunk_grid: ChunkGrid,
) -> None

Validates that all codec configurations are compatible with the array metadata. Raises errors when a codec configuration is not compatible.

Parameters:

  • shape (tuple[int, ...]) –

    The array shape

  • dtype (dtype[Any]) –

    The array data type

  • chunk_grid (ChunkGrid) –

    The array chunk grid

Source code in zarr/abc/codec.py 
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
@abstractmethod
def validate(
    self,
    *,
    shape: tuple[int, ...],
    dtype: ZDType[TBaseDType, TBaseScalar],
    chunk_grid: ChunkGrid,
) -> None:
    """Validates that all codec configurations are compatible with the array metadata.
    Raises errors when a codec configuration is not compatible.

    Parameters
    ----------
    shape : tuple[int, ...]
        The array shape
    dtype : np.dtype[Any]
        The array data type
    chunk_grid : ChunkGrid
        The array chunk grid
    """
    ...

write abstractmethod async 

write(
    batch_info: Iterable[
        tuple[
            ByteSetter,
            ArraySpec,
            SelectorTuple,
            SelectorTuple,
            bool,
        ]
    ],
    value: NDBuffer,
    drop_axes: tuple[int, ...] = (),
) -> None

Encodes chunk data and writes it to the store. Merges with existing chunk data by reading first, if necessary. Partial encoding may be utilized if the codecs and stores support it.

Parameters:

  • batch_info (Iterable[tuple[ByteSetter, ArraySpec, SelectorTuple, SelectorTuple]]) –

    Ordered set of information about the chunks. The first slice selection determines which parts of the chunk will be encoded. The second slice selection determines where in the value array the chunk data is located. The ByteSetter is used to fetch and write the necessary bytes. The chunk spec contains information about the chunk.

  • value (NDBuffer) –
Source code in zarr/abc/codec.py 
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
@abstractmethod
async def write(
    self,
    batch_info: Iterable[tuple[ByteSetter, ArraySpec, SelectorTuple, SelectorTuple, bool]],
    value: NDBuffer,
    drop_axes: tuple[int, ...] = (),
) -> None:
    """Encodes chunk data and writes it to the store.
    Merges with existing chunk data by reading first, if necessary.
    Partial encoding may be utilized if the codecs and stores support it.

    Parameters
    ----------
    batch_info : Iterable[tuple[ByteSetter, ArraySpec, SelectorTuple, SelectorTuple]]
        Ordered set of information about the chunks.
        The first slice selection determines which parts of the chunk will be encoded.
        The second slice selection determines where in the value array the chunk data is located.
        The ByteSetter is used to fetch and write the necessary bytes.
        The chunk spec contains information about the chunk.
    value : NDBuffer
    """
    ...