Skip to content

Buffer

zarr.buffer

Implementations of the Zarr Buffer interface.

See Also

zarr.abc.buffer: Abstract base class for the Zarr Buffer interface.

zarr.buffer.cpu

Buffer

Bases: Buffer

A flat contiguous memory block

We use Buffer throughout Zarr to represent a contiguous block of memory.

A Buffer is backed by a underlying array-like instance that represents the memory. The memory type is unspecified; can be regular host memory, CUDA device memory, or something else. The only requirement is that the array-like instance can be copied/converted to a regular Numpy array (host memory).

Notes

This buffer is untyped, so all indexing and sizes are in bytes.

Parameters:

  • array_like (ArrayLike) –

    array-like object that must be 1-dim, contiguous, and byte dtype.

Source code in zarr/core/buffer/cpu.py 
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
class Buffer(core.Buffer):
    """A flat contiguous memory block

    We use Buffer throughout Zarr to represent a contiguous block of memory.

    A Buffer is backed by a underlying array-like instance that represents
    the memory. The memory type is unspecified; can be regular host memory,
    CUDA device memory, or something else. The only requirement is that the
    array-like instance can be copied/converted to a regular Numpy array
    (host memory).

    Notes
    -----
    This buffer is untyped, so all indexing and sizes are in bytes.

    Parameters
    ----------
    array_like
        array-like object that must be 1-dim, contiguous, and byte dtype.
    """

    def __init__(self, array_like: ArrayLike) -> None:
        super().__init__(array_like)

    @classmethod
    def create_zero_length(cls) -> Self:
        return cls(np.array([], dtype="B"))

    @classmethod
    def from_buffer(cls, buffer: core.Buffer) -> Self:
        """Create a new buffer of an existing Buffer

        This is useful if you want to ensure that an existing buffer is
        of the correct subclass of Buffer. E.g., MemoryStore uses this
        to return a buffer instance of the subclass specified by its
        BufferPrototype argument.

        Typically, this only copies data if the data has to be moved between
        memory types, such as from host to device memory.

        Parameters
        ----------
        buffer
            buffer object.

        Returns
        -------
            A new buffer representing the content of the input buffer

        Notes
        -----
        Subclasses of `Buffer` must override this method to implement
        more optimal conversions that avoid copies where possible
        """
        return cls.from_array_like(buffer.as_numpy_array())

    @classmethod
    def from_bytes(cls, bytes_like: BytesLike) -> Self:
        """Create a new buffer of a bytes-like object (host memory)

        Parameters
        ----------
        bytes_like
            bytes-like object

        Returns
        -------
            New buffer representing `bytes_like`
        """
        return cls.from_array_like(np.frombuffer(bytes_like, dtype="B"))

    def as_numpy_array(self) -> npt.NDArray[Any]:
        """Returns the buffer as a NumPy array (host memory).

        Notes
        -----
        Might have to copy data, consider using `.as_array_like()` instead.

        Returns
        -------
            NumPy array of this buffer (might be a data copy)
        """
        return np.asanyarray(self._data)

    def __add__(self, other: core.Buffer) -> Self:
        """Concatenate two buffers"""

        other_array = other.as_array_like()
        assert other_array.dtype == np.dtype("B")
        return self.__class__(
            np.concatenate((np.asanyarray(self._data), np.asanyarray(other_array)))
        )

__add__ 

__add__(other: Buffer) -> Self

Concatenate two buffers

Source code in zarr/core/buffer/cpu.py 
110
111
112
113
114
115
116
117
def __add__(self, other: core.Buffer) -> Self:
    """Concatenate two buffers"""

    other_array = other.as_array_like()
    assert other_array.dtype == np.dtype("B")
    return self.__class__(
        np.concatenate((np.asanyarray(self._data), np.asanyarray(other_array)))
    )

as_array_like 

as_array_like() -> ArrayLike

Returns the underlying array (host or device memory) of this buffer

This will never copy data.

Returns:

  • The underlying 1d array such as a NumPy or CuPy array. –
Source code in zarr/core/buffer/core.py 
233
234
235
236
237
238
239
240
241
242
def as_array_like(self) -> ArrayLike:
    """Returns the underlying array (host or device memory) of this buffer

    This will never copy data.

    Returns
    -------
        The underlying 1d array such as a NumPy or CuPy array.
    """
    return self._data

as_buffer_like 

as_buffer_like() -> BytesLike

Returns the buffer as an object that implements the Python buffer protocol.

Notes

Might have to copy data, since the implementation uses .as_numpy_array().

Returns:

  • An object that implements the Python buffer protocol –
Source code in zarr/core/buffer/core.py 
258
259
260
261
262
263
264
265
266
267
268
269
def as_buffer_like(self) -> BytesLike:
    """Returns the buffer as an object that implements the Python buffer protocol.

    Notes
    -----
    Might have to copy data, since the implementation uses `.as_numpy_array()`.

    Returns
    -------
        An object that implements the Python buffer protocol
    """
    return memoryview(self.as_numpy_array())  # type: ignore[arg-type]

as_numpy_array 

as_numpy_array() -> NDArray[Any]

Returns the buffer as a NumPy array (host memory).

Notes

Might have to copy data, consider using .as_array_like() instead.

Returns:

  • NumPy array of this buffer (might be a data copy) –
Source code in zarr/core/buffer/cpu.py 
 97
 98
 99
100
101
102
103
104
105
106
107
108
def as_numpy_array(self) -> npt.NDArray[Any]:
    """Returns the buffer as a NumPy array (host memory).

    Notes
    -----
    Might have to copy data, consider using `.as_array_like()` instead.

    Returns
    -------
        NumPy array of this buffer (might be a data copy)
    """
    return np.asanyarray(self._data)

create_zero_length classmethod 

create_zero_length() -> Self

Create an empty buffer with length zero

Returns:

  • New empty 0-length buffer –
Source code in zarr/core/buffer/cpu.py 
50
51
52
@classmethod
def create_zero_length(cls) -> Self:
    return cls(np.array([], dtype="B"))

from_array_like classmethod 

from_array_like(array_like: ArrayLike) -> Self

Create a new buffer of an array-like object

Parameters:

  • array_like (ArrayLike) –

    array-like object that must be 1-dim, contiguous, and byte dtype.

Returns:

  • New buffer representing `array_like` –
Source code in zarr/core/buffer/core.py 
165
166
167
168
169
170
171
172
173
174
175
176
177
178
@classmethod
def from_array_like(cls, array_like: ArrayLike) -> Self:
    """Create a new buffer of an array-like object

    Parameters
    ----------
    array_like
        array-like object that must be 1-dim, contiguous, and byte dtype.

    Returns
    -------
        New buffer representing `array_like`
    """
    return cls(array_like)

from_buffer classmethod 

from_buffer(buffer: Buffer) -> Self

Create a new buffer of an existing Buffer

This is useful if you want to ensure that an existing buffer is of the correct subclass of Buffer. E.g., MemoryStore uses this to return a buffer instance of the subclass specified by its BufferPrototype argument.

Typically, this only copies data if the data has to be moved between memory types, such as from host to device memory.

Parameters:

  • buffer (Buffer) –

    buffer object.

Returns:

  • A new buffer representing the content of the input buffer –
Notes

Subclasses of Buffer must override this method to implement more optimal conversions that avoid copies where possible

Source code in zarr/core/buffer/cpu.py 
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
@classmethod
def from_buffer(cls, buffer: core.Buffer) -> Self:
    """Create a new buffer of an existing Buffer

    This is useful if you want to ensure that an existing buffer is
    of the correct subclass of Buffer. E.g., MemoryStore uses this
    to return a buffer instance of the subclass specified by its
    BufferPrototype argument.

    Typically, this only copies data if the data has to be moved between
    memory types, such as from host to device memory.

    Parameters
    ----------
    buffer
        buffer object.

    Returns
    -------
        A new buffer representing the content of the input buffer

    Notes
    -----
    Subclasses of `Buffer` must override this method to implement
    more optimal conversions that avoid copies where possible
    """
    return cls.from_array_like(buffer.as_numpy_array())

from_bytes classmethod 

from_bytes(bytes_like: BytesLike) -> Self

Create a new buffer of a bytes-like object (host memory)

Parameters:

  • bytes_like (BytesLike) –

    bytes-like object

Returns:

  • New buffer representing `bytes_like` –
Source code in zarr/core/buffer/cpu.py 
82
83
84
85
86
87
88
89
90
91
92
93
94
95
@classmethod
def from_bytes(cls, bytes_like: BytesLike) -> Self:
    """Create a new buffer of a bytes-like object (host memory)

    Parameters
    ----------
    bytes_like
        bytes-like object

    Returns
    -------
        New buffer representing `bytes_like`
    """
    return cls.from_array_like(np.frombuffer(bytes_like, dtype="B"))

to_bytes 

to_bytes() -> bytes

Returns the buffer as bytes (host memory).

Warnings

Will always copy data, only use this method for small buffers such as metadata buffers. If possible, use .as_numpy_array() or .as_array_like() instead.

Returns:

  • `bytes` of this buffer (data copy) –
Source code in zarr/core/buffer/core.py 
271
272
273
274
275
276
277
278
279
280
281
282
283
def to_bytes(self) -> bytes:
    """Returns the buffer as `bytes` (host memory).

    Warnings
    --------
    Will always copy data, only use this method for small buffers such as metadata
    buffers. If possible, use `.as_numpy_array()` or `.as_array_like()` instead.

    Returns
    -------
        `bytes` of this buffer (data copy)
    """
    return bytes(self.as_numpy_array())

NDBuffer

Bases: NDBuffer

An n-dimensional memory block

We use NDBuffer throughout Zarr to represent a n-dimensional memory block.

A NDBuffer is backed by a underlying ndarray-like instance that represents the memory. The memory type is unspecified; can be regular host memory, CUDA device memory, or something else. The only requirement is that the ndarray-like instance can be copied/converted to a regular Numpy array (host memory).

Notes

The two buffer classes Buffer and NDBuffer are very similar. In fact, Buffer is a special case of NDBuffer where dim=1, stride=1, and dtype="B". However, in order to use Python's type system to differentiate between the contiguous Buffer and the n-dim (non-contiguous) NDBuffer, we keep the definition of the two classes separate.

Parameters:

  • array (NDArrayLike) –

    ndarray-like object that is convertible to a regular Numpy array.

Source code in zarr/core/buffer/cpu.py 
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
class NDBuffer(core.NDBuffer):
    """An n-dimensional memory block

    We use NDBuffer throughout Zarr to represent a n-dimensional memory block.

    A NDBuffer is backed by a underlying ndarray-like instance that represents
    the memory. The memory type is unspecified; can be regular host memory,
    CUDA device memory, or something else. The only requirement is that the
    ndarray-like instance can be copied/converted to a regular Numpy array
    (host memory).

    Notes
    -----
    The two buffer classes Buffer and NDBuffer are very similar. In fact, Buffer
    is a special case of NDBuffer where dim=1, stride=1, and dtype="B". However,
    in order to use Python's type system to differentiate between the contiguous
    Buffer and the n-dim (non-contiguous) NDBuffer, we keep the definition of the
    two classes separate.

    Parameters
    ----------
    array
        ndarray-like object that is convertible to a regular Numpy array.
    """

    def __init__(self, array: NDArrayLike) -> None:
        super().__init__(array)

    @classmethod
    def create(
        cls,
        *,
        shape: Iterable[int],
        dtype: npt.DTypeLike,
        order: Literal["C", "F"] = "C",
        fill_value: Any | None = None,
    ) -> Self:
        # np.zeros is much faster than np.full, and therefore using it when possible is better.
        if fill_value is None or (isinstance(fill_value, int) and fill_value == 0):
            return cls(np.zeros(shape=tuple(shape), dtype=dtype, order=order))
        else:
            return cls(np.full(shape=tuple(shape), fill_value=fill_value, dtype=dtype, order=order))

    @classmethod
    def empty(
        cls, shape: tuple[int, ...], dtype: npt.DTypeLike, order: Literal["C", "F"] = "C"
    ) -> Self:
        return cls(np.empty(shape=shape, dtype=dtype, order=order))

    @classmethod
    def from_numpy_array(cls, array_like: npt.ArrayLike) -> Self:
        return cls.from_ndarray_like(np.asanyarray(array_like))

    def as_numpy_array(self) -> npt.NDArray[Any]:
        """Returns the buffer as a NumPy array (host memory).

        Warnings
        --------
        Might have to copy data, consider using `.as_ndarray_like()` instead.

        Returns
        -------
            NumPy array of this buffer (might be a data copy)
        """
        return np.asanyarray(self._data)

    def __getitem__(self, key: Any) -> Self:
        return self.__class__(np.asanyarray(self._data.__getitem__(key)))

    def __setitem__(self, key: Any, value: Any) -> None:
        if isinstance(value, NDBuffer):
            value = value._data
        self._data.__setitem__(key, value)

all_equal 

all_equal(other: Any, equal_nan: bool = True) -> bool

Compare to other using np.array_equal.

Source code in zarr/core/buffer/core.py 
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
def all_equal(self, other: Any, equal_nan: bool = True) -> bool:
    """Compare to `other` using np.array_equal."""
    if other is None:
        # Handle None fill_value for Zarr V2
        return False
    # Handle positive and negative zero by comparing bit patterns:
    if (
        np.asarray(other).dtype.kind == "f"
        and other == 0.0
        and self._data.dtype.kind not in ("U", "S", "T", "O", "V")
    ):
        _data, other = np.broadcast_arrays(self._data, np.asarray(other, self._data.dtype))
        void_dtype = "V" + str(_data.dtype.itemsize)
        return np.array_equal(_data.view(void_dtype), other.view(void_dtype))
    # use array_equal to obtain equal_nan=True functionality
    # Since fill-value is a scalar, isn't there a faster path than allocating a new array for fill value
    # every single time we have to write data?
    _data, other = np.broadcast_arrays(self._data, other)
    return np.array_equal(
        self._data,
        other,
        equal_nan=equal_nan
        if self._data.dtype.kind not in ("U", "S", "T", "O", "V")
        else False,
    )

as_ndarray_like 

as_ndarray_like() -> NDArrayLike

Returns the underlying array (host or device memory) of this buffer

This will never copy data.

Returns:

  • The underlying array such as a NumPy or CuPy array. –
Source code in zarr/core/buffer/core.py 
449
450
451
452
453
454
455
456
457
458
def as_ndarray_like(self) -> NDArrayLike:
    """Returns the underlying array (host or device memory) of this buffer

    This will never copy data.

    Returns
    -------
        The underlying array such as a NumPy or CuPy array.
    """
    return self._data

as_numpy_array 

as_numpy_array() -> NDArray[Any]

Returns the buffer as a NumPy array (host memory).

Warnings

Might have to copy data, consider using .as_ndarray_like() instead.

Returns:

  • NumPy array of this buffer (might be a data copy) –
Source code in zarr/core/buffer/cpu.py 
173
174
175
176
177
178
179
180
181
182
183
184
def as_numpy_array(self) -> npt.NDArray[Any]:
    """Returns the buffer as a NumPy array (host memory).

    Warnings
    --------
    Might have to copy data, consider using `.as_ndarray_like()` instead.

    Returns
    -------
        NumPy array of this buffer (might be a data copy)
    """
    return np.asanyarray(self._data)

as_scalar 

as_scalar() -> ScalarType

Returns the buffer as a scalar value

Source code in zarr/core/buffer/core.py 
474
475
476
477
478
def as_scalar(self) -> ScalarType:
    """Returns the buffer as a scalar value"""
    if self._data.size != 1:
        raise ValueError("Buffer does not contain a single scalar value")
    return cast("ScalarType", self.as_numpy_array()[()])

create classmethod 

create(
    *,
    shape: Iterable[int],
    dtype: DTypeLike,
    order: Literal["C", "F"] = "C",
    fill_value: Any | None = None,
) -> Self

Create a new buffer and its underlying ndarray-like object

Parameters:

  • shape (Iterable[int]) –

    The shape of the buffer and its underlying ndarray-like object

  • dtype (DTypeLike) –

    The datatype of the buffer and its underlying ndarray-like object

  • order (Literal['C', 'F'], default: 'C' ) –

    Whether to store multi-dimensional data in row-major (C-style) or column-major (Fortran-style) order in memory.

  • fill_value (Any | None, default: None ) –

    If not None, fill the new buffer with a scalar value.

Returns:

  • New buffer representing a new ndarray_like object –
Notes

A subclass can overwrite this method to create a ndarray-like object other then the default Numpy array.

Source code in zarr/core/buffer/cpu.py 
148
149
150
151
152
153
154
155
156
157
158
159
160
161
@classmethod
def create(
    cls,
    *,
    shape: Iterable[int],
    dtype: npt.DTypeLike,
    order: Literal["C", "F"] = "C",
    fill_value: Any | None = None,
) -> Self:
    # np.zeros is much faster than np.full, and therefore using it when possible is better.
    if fill_value is None or (isinstance(fill_value, int) and fill_value == 0):
        return cls(np.zeros(shape=tuple(shape), dtype=dtype, order=order))
    else:
        return cls(np.full(shape=tuple(shape), fill_value=fill_value, dtype=dtype, order=order))

empty classmethod 

empty(
    shape: tuple[int, ...],
    dtype: DTypeLike,
    order: Literal["C", "F"] = "C",
) -> Self

Create an empty buffer with the given shape, dtype, and order.

This method can be faster than NDBuffer.create because it doesn't have to initialize the memory used by the underlying ndarray-like object.

Parameters:

  • shape (tuple[int, ...]) –

    The shape of the buffer and its underlying ndarray-like object

  • dtype (DTypeLike) –

    The datatype of the buffer and its underlying ndarray-like object

  • order (Literal['C', 'F'], default: 'C' ) –

    Whether to store multi-dimensional data in row-major (C-style) or column-major (Fortran-style) order in memory.

Returns:

  • buffer –

    New buffer representing a new ndarray_like object with empty data.

See Also

NDBuffer.create Create a new buffer with some initial fill value.

Source code in zarr/core/buffer/cpu.py 
163
164
165
166
167
@classmethod
def empty(
    cls, shape: tuple[int, ...], dtype: npt.DTypeLike, order: Literal["C", "F"] = "C"
) -> Self:
    return cls(np.empty(shape=shape, dtype=dtype, order=order))

from_ndarray_like classmethod 

from_ndarray_like(ndarray_like: NDArrayLike) -> Self

Create a new buffer of a ndarray-like object

Parameters:

Returns:

  • New buffer representing `ndarray_like` –
Source code in zarr/core/buffer/core.py 
412
413
414
415
416
417
418
419
420
421
422
423
424
425
@classmethod
def from_ndarray_like(cls, ndarray_like: NDArrayLike) -> Self:
    """Create a new buffer of a ndarray-like object

    Parameters
    ----------
    ndarray_like
        ndarray-like object

    Returns
    -------
        New buffer representing `ndarray_like`
    """
    return cls(ndarray_like)

from_numpy_array classmethod 

from_numpy_array(array_like: ArrayLike) -> Self

Create a new buffer of Numpy array-like object

Parameters:

  • array_like (ArrayLike) –

    Object that can be coerced into a Numpy array

Returns:

  • New buffer representing `array_like` –
Source code in zarr/core/buffer/cpu.py 
169
170
171
@classmethod
def from_numpy_array(cls, array_like: npt.ArrayLike) -> Self:
    return cls.from_ndarray_like(np.asanyarray(array_like))

as_numpy_array_wrapper 

as_numpy_array_wrapper(
    func: Callable[[NDArray[Any]], bytes],
    buf: Buffer,
    prototype: BufferPrototype,
) -> Buffer

Converts the input of func to a numpy array and the output back to Buffer.

This function is useful when calling a func that only support host memory such as GZip.decode and Blosc.decode. In this case, use this wrapper to convert the input buf to a Numpy array and convert the result back into a Buffer.

Parameters:

  • func (Callable[[NDArray[Any]], bytes]) –

    The callable that will be called with the converted buf as input. func must return bytes, which will be converted into a Buffer before returned.

  • buf (Buffer) –

    The buffer that will be converted to a Numpy array before given as input to func.

  • prototype (BufferPrototype) –

    The prototype of the output buffer.

Returns:

  • The result of `func` converted to a `Buffer` –
Source code in zarr/core/buffer/cpu.py 
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
def as_numpy_array_wrapper(
    func: Callable[[npt.NDArray[Any]], bytes], buf: core.Buffer, prototype: core.BufferPrototype
) -> core.Buffer:
    """Converts the input of `func` to a numpy array and the output back to `Buffer`.

    This function is useful when calling a `func` that only support host memory such
    as `GZip.decode` and `Blosc.decode`. In this case, use this wrapper to convert
    the input `buf` to a Numpy array and convert the result back into a `Buffer`.

    Parameters
    ----------
    func
        The callable that will be called with the converted `buf` as input.
        `func` must return bytes, which will be converted into a `Buffer`
        before returned.
    buf
        The buffer that will be converted to a Numpy array before given as
        input to `func`.
    prototype
        The prototype of the output buffer.

    Returns
    -------
        The result of `func` converted to a `Buffer`
    """
    return prototype.buffer.from_bytes(func(buf.as_numpy_array()))

zarr.buffer.gpu

Buffer

Bases: Buffer

A flat contiguous memory block on the GPU

We use Buffer throughout Zarr to represent a contiguous block of memory.

A Buffer is backed by a underlying array-like instance that represents the memory. The memory type is unspecified; can be regular host memory, CUDA device memory, or something else. The only requirement is that the array-like instance can be copied/converted to a regular Numpy array (host memory).

Notes

This buffer is untyped, so all indexing and sizes are in bytes.

Parameters:

  • array_like (ArrayLike) –

    array-like object that must be 1-dim, contiguous, and byte dtype.

Source code in zarr/core/buffer/gpu.py 
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
class Buffer(core.Buffer):
    """A flat contiguous memory block on the GPU

    We use Buffer throughout Zarr to represent a contiguous block of memory.

    A Buffer is backed by a underlying array-like instance that represents
    the memory. The memory type is unspecified; can be regular host memory,
    CUDA device memory, or something else. The only requirement is that the
    array-like instance can be copied/converted to a regular Numpy array
    (host memory).

    Notes
    -----
    This buffer is untyped, so all indexing and sizes are in bytes.

    Parameters
    ----------
    array_like
        array-like object that must be 1-dim, contiguous, and byte dtype.
    """

    def __init__(self, array_like: ArrayLike) -> None:
        if cp is None:
            raise ImportError(
                "Cannot use zarr.buffer.gpu.Buffer without cupy. Please install cupy."
            )

        if array_like.ndim != 1:
            raise ValueError("array_like: only 1-dim allowed")
        if array_like.dtype != np.dtype("B"):
            raise ValueError("array_like: only byte dtype allowed")

        if not hasattr(array_like, "__cuda_array_interface__"):
            # Slow copy based path for arrays that don't support the __cuda_array_interface__
            # TODO: Add a fast zero-copy path for arrays that support the dlpack protocol
            msg = (
                "Creating a zarr.buffer.gpu.Buffer with an array that does not support the "
                "__cuda_array_interface__ for zero-copy transfers, "
                "falling back to slow copy based path"
            )
            warnings.warn(
                msg,
                category=ZarrUserWarning,
                stacklevel=2,
            )
        self._data = cp.asarray(array_like)

    @classmethod
    def create_zero_length(cls) -> Self:
        """Create an empty buffer with length zero

        Returns
        -------
            New empty 0-length buffer
        """
        return cls(cp.array([], dtype="B"))

    @classmethod
    def from_buffer(cls, buffer: core.Buffer) -> Self:
        """Create an GPU Buffer given an arbitrary Buffer
        This will try to be zero-copy if `buffer` is already on the
        GPU and will trigger a copy if not.

        Returns
        -------
            New GPU Buffer constructed from `buffer`
        """
        return cls(buffer.as_array_like())

    @classmethod
    def from_bytes(cls, bytes_like: BytesLike) -> Self:
        return cls.from_array_like(cp.frombuffer(bytes_like, dtype="B"))

    def as_numpy_array(self) -> npt.NDArray[Any]:
        return cast("npt.NDArray[Any]", cp.asnumpy(self._data))

    def __add__(self, other: core.Buffer) -> Self:
        other_array = other.as_array_like()
        assert other_array.dtype == np.dtype("B")
        gpu_other = Buffer(other_array)
        gpu_other_array = gpu_other.as_array_like()
        return self.__class__(
            cp.concatenate((cp.asanyarray(self._data), cp.asanyarray(gpu_other_array)))
        )

__add__ 

__add__(other: Buffer) -> Self

Concatenate two buffers

Source code in zarr/core/buffer/gpu.py 
110
111
112
113
114
115
116
117
def __add__(self, other: core.Buffer) -> Self:
    other_array = other.as_array_like()
    assert other_array.dtype == np.dtype("B")
    gpu_other = Buffer(other_array)
    gpu_other_array = gpu_other.as_array_like()
    return self.__class__(
        cp.concatenate((cp.asanyarray(self._data), cp.asanyarray(gpu_other_array)))
    )

as_array_like 

as_array_like() -> ArrayLike

Returns the underlying array (host or device memory) of this buffer

This will never copy data.

Returns:

  • The underlying 1d array such as a NumPy or CuPy array. –
Source code in zarr/core/buffer/core.py 
233
234
235
236
237
238
239
240
241
242
def as_array_like(self) -> ArrayLike:
    """Returns the underlying array (host or device memory) of this buffer

    This will never copy data.

    Returns
    -------
        The underlying 1d array such as a NumPy or CuPy array.
    """
    return self._data

as_buffer_like 

as_buffer_like() -> BytesLike

Returns the buffer as an object that implements the Python buffer protocol.

Notes

Might have to copy data, since the implementation uses .as_numpy_array().

Returns:

  • An object that implements the Python buffer protocol –
Source code in zarr/core/buffer/core.py 
258
259
260
261
262
263
264
265
266
267
268
269
def as_buffer_like(self) -> BytesLike:
    """Returns the buffer as an object that implements the Python buffer protocol.

    Notes
    -----
    Might have to copy data, since the implementation uses `.as_numpy_array()`.

    Returns
    -------
        An object that implements the Python buffer protocol
    """
    return memoryview(self.as_numpy_array())  # type: ignore[arg-type]

as_numpy_array 

as_numpy_array() -> NDArray[Any]

Returns the buffer as a NumPy array (host memory).

Notes

Might have to copy data, consider using .as_array_like() instead.

Returns:

  • NumPy array of this buffer (might be a data copy) –
Source code in zarr/core/buffer/gpu.py 
107
108
def as_numpy_array(self) -> npt.NDArray[Any]:
    return cast("npt.NDArray[Any]", cp.asnumpy(self._data))

create_zero_length classmethod 

create_zero_length() -> Self

Create an empty buffer with length zero

Returns:

  • New empty 0-length buffer –
Source code in zarr/core/buffer/gpu.py 
81
82
83
84
85
86
87
88
89
@classmethod
def create_zero_length(cls) -> Self:
    """Create an empty buffer with length zero

    Returns
    -------
        New empty 0-length buffer
    """
    return cls(cp.array([], dtype="B"))

from_array_like classmethod 

from_array_like(array_like: ArrayLike) -> Self

Create a new buffer of an array-like object

Parameters:

  • array_like (ArrayLike) –

    array-like object that must be 1-dim, contiguous, and byte dtype.

Returns:

  • New buffer representing `array_like` –
Source code in zarr/core/buffer/core.py 
165
166
167
168
169
170
171
172
173
174
175
176
177
178
@classmethod
def from_array_like(cls, array_like: ArrayLike) -> Self:
    """Create a new buffer of an array-like object

    Parameters
    ----------
    array_like
        array-like object that must be 1-dim, contiguous, and byte dtype.

    Returns
    -------
        New buffer representing `array_like`
    """
    return cls(array_like)

from_buffer classmethod 

from_buffer(buffer: Buffer) -> Self

Create an GPU Buffer given an arbitrary Buffer This will try to be zero-copy if buffer is already on the GPU and will trigger a copy if not.

Returns:

  • New GPU Buffer constructed from `buffer` –
Source code in zarr/core/buffer/gpu.py 
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
@classmethod
def from_buffer(cls, buffer: core.Buffer) -> Self:
    """Create an GPU Buffer given an arbitrary Buffer
    This will try to be zero-copy if `buffer` is already on the
    GPU and will trigger a copy if not.

    Returns
    -------
        New GPU Buffer constructed from `buffer`
    """
    return cls(buffer.as_array_like())

from_bytes classmethod 

from_bytes(bytes_like: BytesLike) -> Self

Create a new buffer of a bytes-like object (host memory)

Parameters:

  • bytes_like (BytesLike) –

    bytes-like object

Returns:

  • New buffer representing `bytes_like` –
Source code in zarr/core/buffer/gpu.py 
103
104
105
@classmethod
def from_bytes(cls, bytes_like: BytesLike) -> Self:
    return cls.from_array_like(cp.frombuffer(bytes_like, dtype="B"))

to_bytes 

to_bytes() -> bytes

Returns the buffer as bytes (host memory).

Warnings

Will always copy data, only use this method for small buffers such as metadata buffers. If possible, use .as_numpy_array() or .as_array_like() instead.

Returns:

  • `bytes` of this buffer (data copy) –
Source code in zarr/core/buffer/core.py 
271
272
273
274
275
276
277
278
279
280
281
282
283
def to_bytes(self) -> bytes:
    """Returns the buffer as `bytes` (host memory).

    Warnings
    --------
    Will always copy data, only use this method for small buffers such as metadata
    buffers. If possible, use `.as_numpy_array()` or `.as_array_like()` instead.

    Returns
    -------
        `bytes` of this buffer (data copy)
    """
    return bytes(self.as_numpy_array())

NDBuffer

Bases: NDBuffer

A n-dimensional memory block on the GPU

We use NDBuffer throughout Zarr to represent a n-dimensional memory block.

A NDBuffer is backed by a underlying ndarray-like instance that represents the memory. The memory type is unspecified; can be regular host memory, CUDA device memory, or something else. The only requirement is that the ndarray-like instance can be copied/converted to a regular Numpy array (host memory).

Notes

The two buffer classes Buffer and NDBuffer are very similar. In fact, Buffer is a special case of NDBuffer where dim=1, stride=1, and dtype="B". However, in order to use Python's type system to differentiate between the contiguous Buffer and the n-dim (non-contiguous) NDBuffer, we keep the definition of the two classes separate.

Parameters:

  • array (NDArrayLike) –

    ndarray-like object that is convertible to a regular Numpy array.

Source code in zarr/core/buffer/gpu.py 
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
class NDBuffer(core.NDBuffer):
    """A n-dimensional memory block on the GPU

    We use NDBuffer throughout Zarr to represent a n-dimensional memory block.

    A NDBuffer is backed by a underlying ndarray-like instance that represents
    the memory. The memory type is unspecified; can be regular host memory,
    CUDA device memory, or something else. The only requirement is that the
    ndarray-like instance can be copied/converted to a regular Numpy array
    (host memory).

    Notes
    -----
    The two buffer classes Buffer and NDBuffer are very similar. In fact, Buffer
    is a special case of NDBuffer where dim=1, stride=1, and dtype="B". However,
    in order to use Python's type system to differentiate between the contiguous
    Buffer and the n-dim (non-contiguous) NDBuffer, we keep the definition of the
    two classes separate.

    Parameters
    ----------
    array
        ndarray-like object that is convertible to a regular Numpy array.
    """

    def __init__(self, array: NDArrayLike) -> None:
        if cp is None:
            raise ImportError(
                "Cannot use zarr.buffer.gpu.NDBuffer without cupy. Please install cupy."
            )

        # assert array.ndim > 0
        assert array.dtype != object
        self._data = array

        if not hasattr(array, "__cuda_array_interface__"):
            # Slow copy based path for arrays that don't support the __cuda_array_interface__
            # TODO: Add a fast zero-copy path for arrays that support the dlpack protocol
            msg = (
                "Creating a zarr.buffer.gpu.NDBuffer with an array that does not support the "
                "__cuda_array_interface__ for zero-copy transfers, "
                "falling back to slow copy based path"
            )
            warnings.warn(
                msg,
                stacklevel=2,
            )
        self._data = cp.asarray(array)

    @classmethod
    def create(
        cls,
        *,
        shape: Iterable[int],
        dtype: npt.DTypeLike,
        order: Literal["C", "F"] = "C",
        fill_value: Any | None = None,
    ) -> Self:
        ret = cls(cp.empty(shape=tuple(shape), dtype=dtype, order=order))
        if fill_value is not None:
            ret.fill(fill_value)
        return ret

    @classmethod
    def empty(
        cls, shape: tuple[int, ...], dtype: npt.DTypeLike, order: Literal["C", "F"] = "C"
    ) -> Self:
        return cls(cp.empty(shape=shape, dtype=dtype, order=order))

    @classmethod
    def from_numpy_array(cls, array_like: npt.ArrayLike) -> Self:
        """Create a new buffer of Numpy array-like object

        Parameters
        ----------
        array_like
            Object that can be coerced into a Numpy array

        Returns
        -------
            New buffer representing `array_like`
        """
        return cls(cp.asarray(array_like))

    def as_numpy_array(self) -> npt.NDArray[Any]:
        """Returns the buffer as a NumPy array (host memory).

        Warnings
        --------
        Might have to copy data, consider using `.as_ndarray_like()` instead.

        Returns
        -------
            NumPy array of this buffer (might be a data copy)
        """
        return cast("npt.NDArray[Any]", cp.asnumpy(self._data))

    def __getitem__(self, key: Any) -> Self:
        return self.__class__(self._data.__getitem__(key))

    def __setitem__(self, key: Any, value: Any) -> None:
        if isinstance(value, NDBuffer):
            value = value._data
        elif isinstance(value, core.NDBuffer):
            gpu_value = NDBuffer(value.as_ndarray_like())
            value = gpu_value._data
        self._data.__setitem__(key, value)

all_equal 

all_equal(other: Any, equal_nan: bool = True) -> bool

Compare to other using np.array_equal.

Source code in zarr/core/buffer/core.py 
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
def all_equal(self, other: Any, equal_nan: bool = True) -> bool:
    """Compare to `other` using np.array_equal."""
    if other is None:
        # Handle None fill_value for Zarr V2
        return False
    # Handle positive and negative zero by comparing bit patterns:
    if (
        np.asarray(other).dtype.kind == "f"
        and other == 0.0
        and self._data.dtype.kind not in ("U", "S", "T", "O", "V")
    ):
        _data, other = np.broadcast_arrays(self._data, np.asarray(other, self._data.dtype))
        void_dtype = "V" + str(_data.dtype.itemsize)
        return np.array_equal(_data.view(void_dtype), other.view(void_dtype))
    # use array_equal to obtain equal_nan=True functionality
    # Since fill-value is a scalar, isn't there a faster path than allocating a new array for fill value
    # every single time we have to write data?
    _data, other = np.broadcast_arrays(self._data, other)
    return np.array_equal(
        self._data,
        other,
        equal_nan=equal_nan
        if self._data.dtype.kind not in ("U", "S", "T", "O", "V")
        else False,
    )

as_ndarray_like 

as_ndarray_like() -> NDArrayLike

Returns the underlying array (host or device memory) of this buffer

This will never copy data.

Returns:

  • The underlying array such as a NumPy or CuPy array. –
Source code in zarr/core/buffer/core.py 
449
450
451
452
453
454
455
456
457
458
def as_ndarray_like(self) -> NDArrayLike:
    """Returns the underlying array (host or device memory) of this buffer

    This will never copy data.

    Returns
    -------
        The underlying array such as a NumPy or CuPy array.
    """
    return self._data

as_numpy_array 

as_numpy_array() -> NDArray[Any]

Returns the buffer as a NumPy array (host memory).

Warnings

Might have to copy data, consider using .as_ndarray_like() instead.

Returns:

  • NumPy array of this buffer (might be a data copy) –
Source code in zarr/core/buffer/gpu.py 
204
205
206
207
208
209
210
211
212
213
214
215
def as_numpy_array(self) -> npt.NDArray[Any]:
    """Returns the buffer as a NumPy array (host memory).

    Warnings
    --------
    Might have to copy data, consider using `.as_ndarray_like()` instead.

    Returns
    -------
        NumPy array of this buffer (might be a data copy)
    """
    return cast("npt.NDArray[Any]", cp.asnumpy(self._data))

as_scalar 

as_scalar() -> ScalarType

Returns the buffer as a scalar value

Source code in zarr/core/buffer/core.py 
474
475
476
477
478
def as_scalar(self) -> ScalarType:
    """Returns the buffer as a scalar value"""
    if self._data.size != 1:
        raise ValueError("Buffer does not contain a single scalar value")
    return cast("ScalarType", self.as_numpy_array()[()])

create classmethod 

create(
    *,
    shape: Iterable[int],
    dtype: DTypeLike,
    order: Literal["C", "F"] = "C",
    fill_value: Any | None = None,
) -> Self

Create a new buffer and its underlying ndarray-like object

Parameters:

  • shape (Iterable[int]) –

    The shape of the buffer and its underlying ndarray-like object

  • dtype (DTypeLike) –

    The datatype of the buffer and its underlying ndarray-like object

  • order (Literal['C', 'F'], default: 'C' ) –

    Whether to store multi-dimensional data in row-major (C-style) or column-major (Fortran-style) order in memory.

  • fill_value (Any | None, default: None ) –

    If not None, fill the new buffer with a scalar value.

Returns:

  • New buffer representing a new ndarray_like object –
Notes

A subclass can overwrite this method to create a ndarray-like object other then the default Numpy array.

Source code in zarr/core/buffer/gpu.py 
169
170
171
172
173
174
175
176
177
178
179
180
181
@classmethod
def create(
    cls,
    *,
    shape: Iterable[int],
    dtype: npt.DTypeLike,
    order: Literal["C", "F"] = "C",
    fill_value: Any | None = None,
) -> Self:
    ret = cls(cp.empty(shape=tuple(shape), dtype=dtype, order=order))
    if fill_value is not None:
        ret.fill(fill_value)
    return ret

empty classmethod 

empty(
    shape: tuple[int, ...],
    dtype: DTypeLike,
    order: Literal["C", "F"] = "C",
) -> Self

Create an empty buffer with the given shape, dtype, and order.

This method can be faster than NDBuffer.create because it doesn't have to initialize the memory used by the underlying ndarray-like object.

Parameters:

  • shape (tuple[int, ...]) –

    The shape of the buffer and its underlying ndarray-like object

  • dtype (DTypeLike) –

    The datatype of the buffer and its underlying ndarray-like object

  • order (Literal['C', 'F'], default: 'C' ) –

    Whether to store multi-dimensional data in row-major (C-style) or column-major (Fortran-style) order in memory.

Returns:

  • buffer –

    New buffer representing a new ndarray_like object with empty data.

See Also

NDBuffer.create Create a new buffer with some initial fill value.

Source code in zarr/core/buffer/gpu.py 
183
184
185
186
187
@classmethod
def empty(
    cls, shape: tuple[int, ...], dtype: npt.DTypeLike, order: Literal["C", "F"] = "C"
) -> Self:
    return cls(cp.empty(shape=shape, dtype=dtype, order=order))

from_ndarray_like classmethod 

from_ndarray_like(ndarray_like: NDArrayLike) -> Self

Create a new buffer of a ndarray-like object

Parameters:

Returns:

  • New buffer representing `ndarray_like` –
Source code in zarr/core/buffer/core.py 
412
413
414
415
416
417
418
419
420
421
422
423
424
425
@classmethod
def from_ndarray_like(cls, ndarray_like: NDArrayLike) -> Self:
    """Create a new buffer of a ndarray-like object

    Parameters
    ----------
    ndarray_like
        ndarray-like object

    Returns
    -------
        New buffer representing `ndarray_like`
    """
    return cls(ndarray_like)

from_numpy_array classmethod 

from_numpy_array(array_like: ArrayLike) -> Self

Create a new buffer of Numpy array-like object

Parameters:

  • array_like (ArrayLike) –

    Object that can be coerced into a Numpy array

Returns:

  • New buffer representing `array_like` –
Source code in zarr/core/buffer/gpu.py 
189
190
191
192
193
194
195
196
197
198
199
200
201
202
@classmethod
def from_numpy_array(cls, array_like: npt.ArrayLike) -> Self:
    """Create a new buffer of Numpy array-like object

    Parameters
    ----------
    array_like
        Object that can be coerced into a Numpy array

    Returns
    -------
        New buffer representing `array_like`
    """
    return cls(cp.asarray(array_like))