Skip to content
Snippets Groups Projects
Commit f8c76c01 authored by Eric Blake's avatar Eric Blake
Browse files

python: Speed up pread


Instead of malloc'ing a C buffer, nbd_pread()ing into it, then copying
it into an immutable Python bytes object, we can instead pre-create a
correctly-sized Python bytearray object (with uninitialized contents),
then nbd_pread() directly into that object's underlying bytes.

While the data copying might not be the bottleneck compared to the
networking costs, it does have noticeable results; on my machine:

$ export script='
m=1024*1024
size=h.get_size()
for i in range(size // m):
  buf = h.pread(m, m*i)
'
$ time ./run nbdkit -U - pattern 10G --run 'nbdsh -u $uri -c "$script"'

sped up from about 7.8s pre-patch to about 7.1s post-patch,
approximately a 10% speedup.

Note that this slightly changes the python API: h.pread[_structured]
now returns a mutable 'bytearray' object, rather than an immutable
'bytes' object.  This makes it possible to modify the just-read string
in place, rather than having to create yet another memory buffer for
any modifications, which offers even more speedups when writing a
read-modify-write paradigm in python.  But the change is
backwards-compatible - python already states that a read-write buffer
may be returned instead of readonly for any client that already
operates on a buffer in only a readonly manner.

The generated code for nbd_internal_py_pread[_structured] changes as:

| --- python/methods.c.bak        2022-05-27 17:08:43.035658709 -0500
| +++ python/methods.c    2022-05-27 17:08:50.678669864 -0500
| @@ -2295,7 +2295,7 @@
|    struct nbd_handle *h;
|    int ret;
|    PyObject *py_ret = NULL;
| -  char *buf = NULL;
| +  PyObject *buf = NULL;
|    Py_ssize_t count;
|    uint64_t offset_u64;
|    unsigned long long offset; /* really uint64_t */
| @@ -2309,19 +2309,20 @@
|    h = get_handle (py_h);
|    if (!h) goto out;
|    flags_u32 = flags;
| -  buf = malloc (count);
| -  if (buf == NULL) { PyErr_NoMemory (); goto out; }
| +  buf = PyByteArray_FromStringAndSize (NULL, count);
| +  if (buf == NULL) goto out;
|    offset_u64 = offset;
|
| -  ret = nbd_pread (h, buf, count, offset_u64, flags_u32);
| +  ret = nbd_pread (h, PyByteArray_AS_STRING (buf), count, offset_u64, flags_u32);
|    if (ret == -1) {
|      raise_exception ();
|      goto out;
|    }
| -  py_ret = PyBytes_FromStringAndSize (buf, count);
| +  py_ret = buf;
| +  buf = NULL;
|
|   out:
| -  free (buf);
| +  Py_XDECREF (buf);
|    return py_ret;
|  }

Reviewed-by: default avatarRichard W.M. Jones <rjones@redhat.com>
parent 56d083b8
No related branches found
No related tags found
No related merge requests found
Pipeline #552113764 failed
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment