Loading README.md +101 −80 Original line number Diff line number Diff line Loading @@ -12,17 +12,18 @@ does everything from a single python interpreter for the sake of clarity, but the real point is to share arrays between python interpreters. ```python import numpy as np import SharedArray as sa # Create an array in shared memory # Create an array in shared memory. a = sa.create("shm://test", 10) # Attach it as a different array. This can be done from another # python interpreter as long as it runs on the same computer. b = sa.attach("shm://test") # See how they are actually sharing the same memory block # See how they are actually sharing the same memory. a[0] = 42 print(b[0]) Loading @@ -37,43 +38,50 @@ interpreters. # Now destroy the array "test" from memory. sa.delete("test") # The array b is not affected, but once you destroy it then the # data are lost. # The array b is still there, but once you destroy it then the # data is gone for real. print(b[0]) ``` ## Functions ### `SharedArray.create(name, shape, dtype=float)` ### SharedArray.create(name, shape, dtype=float) This function creates an array identified by `name`, which can use the `file://` prefix to indicate that the data backend will be a file, or `shm://` to indicate that the data backend shall be a POSIX shared memory object. For backward compatibility `shm://` is assumed when no prefix is given. The `shape` and `dtype` arguments are the same as the numpy function `numpy.zeros()` and the returned array is indeed initialized to zero. This function creates an array in shared memory and returns a numpy array that uses the shared memory as data backend. The contents of the array will not be deleted when this array is destroyed, either implicitly or explicitly by calling `del`, it will simply be detached from the current process. To delete a shared array and therefore reclaim system resources use the `SharedArray.delete()` function. The shared memory is identified by `name`, which can use the `file://` prefix to indicate that the data backend will be a file, or `shm://` to indicate that the data backend shall be a POSIX shared memory object. For backward compatibility `shm://` is assumed when no prefix is given. Most operating systems implement strong file caching so using a file as a data backend won't usually affect performance. ### `SharedArray.attach(name)` The `shape` and `dtype` arguments are the same as the numpy function `numpy.zeros()` and the returned array is indeed initialized to zero. This function attaches the previously created array identified by `name`, which can use the `file://` prefix to indicate that the array is stored as a file, or `shm://` to indicate that the array is stored as a POSIX shared memory object. For backward compatibility `shm://` is assumed when no prefix is given The content of the array lives in shared memory and/or in a file and won't be lost when the numpy array is deleted, nor when the python interpreter exits. To delete a shared array reclaim system resources use the `SharedArray.delete()` function. ### SharedArray.attach(name) The contents of the array will not be deleted when this array is destroyed, either implicitly or explicitly by calling `del`, it will simply be detached from the current process. To delete a shared array and therefore reclaim system resources use the `SharedArray.delete()` function. This function attaches a previously created array in shared memory identified by `name`, which can use the `file://` prefix to indicate that the array is stored as a file, or `shm://` to indicate that the array is stored as a POSIX shared memory object. For backward compatibility `shm://` is assumed when no prefix is given. ### `SharedArray.delete(name)` An array may be simultaneously attached from multiple different processes (i.e. python interpreters). The content of the array lives in shared memory and/or in a file and won't be lost when the numpy array is deleted, nor when the python interpreter exits. To delete a shared array reclaim system resources use the `SharedArray.delete()` function. ### SharedArray.delete(name) This function destroys the previously created array identified by `name`, which can use the `file://` prefix to indicate that the array Loading @@ -83,9 +91,10 @@ is assumed when no prefix is given After calling `delete`, the array will not be attachable anymore, but existing attachments will remain valid until they are themselves destroyed. destroyed. The data is reclaimed by the system when the very last attachment is deleted. ### `SharedArray.list()` ### SharedArray.list() This function returns a list of previously created arrays stored as POSIX SHM objects, along with their name, data type and dimensions. Loading @@ -93,7 +102,7 @@ At the moment this function only works on Linux because it accesses files exposed under `/dev/shm`. There doesn't seem to be a portable method of doing that. ### `SharedArray.msync(array, flags)` ### SharedArray.msync(array, flags) This function is a wrapper around `msync(2)` and is only useful when using file-backed arrays (i.e. not POSIX shared memory). msync(2) Loading @@ -102,29 +111,29 @@ are exported as constants in the module definition (see below) and are a 1:1 map of the `msync(2)` flags, please refer to the manual page of `msync(2)` for details. ### `SharedArray.mlock(array)` ### SharedArray.mlock(array) This function is a wrapper around `mlock(2)`: lock the memory map into RAM, preventing that memory from being paged to the swap area. ### `SharedArray.munlock(array)` ### SharedArray.munlock(array) This function is a wrapper around `munlock(2)`: unlock the memory map, allowing that memory to be paged to the swap area. ## Constants ### `SharedArray.MS_ASYNC` ### SharedArray.MS_ASYNC Flag for `SharedArray.msync()`. Specifies that an update be scheduled, but the call returns immediately. ### `SharedArray.MS_SYNC` ### SharedArray.MS_SYNC Flag for `SharedArray.msync()`. Requests an update and waits for it to complete. ### `SharedArray.MS_INVALIDATE` ### SharedArray.MS_INVALIDATE Flag for `SharedArray.msync()`. Asks to invalidate other mappings of the same file (so that they can be updated with the fresh values just Loading @@ -133,12 +142,32 @@ written). ## Requirements * Python 2.7 or 3+ * Numpy 1.8 * Numpy 1.8+ * Posix shared memory interface SharedArray uses the posix shm interface (`shm_open` and `shm_unlink`) and so should work on most operating systems that follow the posix standards (Linux, *BSD, etc.). and so should work on most POSIX operating systems (Linux, BSD, etc.). ## Installation The extension uses the `distutils` python package that should be familiar to most python users. To test the extension directly from the source tree, without installing, type: ```sh python setup.py build_ext --inplace ``` To build and install the extension system-wide, type: ```sh python setup.py build sudo python setup.py install ``` The package is also available on PyPI and can be installed using the pip tool. ## FAQ Loading @@ -146,13 +175,17 @@ standards (Linux, *BSD, etc.). A few people have reported segfaults with very large arrays using POSIX shared memory. This is not a bug in SharedArray but rather an indication that the system ran out of POSIX shared memory. On Linux a `tmpfs` virtual filesystem is used to provide POSIX shared memory, and by default it is given only about 20% of the total available memory, depending on the distribution. That amount can be changed by re-mounting the `tmpfs` filesystem with the `size=100%` option: indication that the system ran out of POSIX shared memory. On Linux a `tmpfs` virtual filesystem is used to provide POSIX shared memory, and by default it is given only about 20% of the total available memory, depending on the distribution. That amount can be changed by re-mounting the `tmpfs` filesystem with the `size=100%` option: ```sh sudo mount -o remount,size=100% /run/shm ``` Also you can make the change permanent, on next boot, by setting `SHM_SIZE=100%` in `/etc/defaults/tmpfs` on recent Debian Loading @@ -166,27 +199,15 @@ arrays created with a previous version of SharedArray aren't compatible with the new version (the location of the metadata changed). Save your work before upgrading. ## Installation The extension uses the `distutils` python package that should be familiar to most python users. To test the extension directly from the source tree, without installing, type: python setup.py build_ext --inplace To build and install the extension system-wide, type: python setup.py build sudo python setup.py install ## Contact For updates and the browse the code, the canonical repository is: <https://parad0x.org/git/python/shared-array/> This package is hosted on [GitLab](https://gitlab.com) at: <https://gitlab.com/tenzing/shared-array> Packages are also available on PyPi at: <https://pypi.python.org/pypi/SharedArray> For bug reports, feature requests, suggestions, patches and everything else related to SharedArray, please contact the maintainer at: <mat@parad0x.org>. else related to SharedArray, feel free to raise issues on the [project page](https://gitlab.com/tenzing/shared-array). You can also contact the maintainer directly by email at <mat@parad0x.org>. setup.py +1 −1 Original line number Diff line number Diff line Loading @@ -41,7 +41,7 @@ setup(name = 'SharedArray', # Contact author = 'Mathieu Mirmont', author_email = 'mat@parad0x.org', url = 'http://parad0x.org/git/python/shared-array/about', url = 'https://gitlab.com/tenzing/shared-array', # License license = 'https://www.gnu.org/licenses/gpl-2.0.html', Loading Loading
README.md +101 −80 Original line number Diff line number Diff line Loading @@ -12,17 +12,18 @@ does everything from a single python interpreter for the sake of clarity, but the real point is to share arrays between python interpreters. ```python import numpy as np import SharedArray as sa # Create an array in shared memory # Create an array in shared memory. a = sa.create("shm://test", 10) # Attach it as a different array. This can be done from another # python interpreter as long as it runs on the same computer. b = sa.attach("shm://test") # See how they are actually sharing the same memory block # See how they are actually sharing the same memory. a[0] = 42 print(b[0]) Loading @@ -37,43 +38,50 @@ interpreters. # Now destroy the array "test" from memory. sa.delete("test") # The array b is not affected, but once you destroy it then the # data are lost. # The array b is still there, but once you destroy it then the # data is gone for real. print(b[0]) ``` ## Functions ### `SharedArray.create(name, shape, dtype=float)` ### SharedArray.create(name, shape, dtype=float) This function creates an array identified by `name`, which can use the `file://` prefix to indicate that the data backend will be a file, or `shm://` to indicate that the data backend shall be a POSIX shared memory object. For backward compatibility `shm://` is assumed when no prefix is given. The `shape` and `dtype` arguments are the same as the numpy function `numpy.zeros()` and the returned array is indeed initialized to zero. This function creates an array in shared memory and returns a numpy array that uses the shared memory as data backend. The contents of the array will not be deleted when this array is destroyed, either implicitly or explicitly by calling `del`, it will simply be detached from the current process. To delete a shared array and therefore reclaim system resources use the `SharedArray.delete()` function. The shared memory is identified by `name`, which can use the `file://` prefix to indicate that the data backend will be a file, or `shm://` to indicate that the data backend shall be a POSIX shared memory object. For backward compatibility `shm://` is assumed when no prefix is given. Most operating systems implement strong file caching so using a file as a data backend won't usually affect performance. ### `SharedArray.attach(name)` The `shape` and `dtype` arguments are the same as the numpy function `numpy.zeros()` and the returned array is indeed initialized to zero. This function attaches the previously created array identified by `name`, which can use the `file://` prefix to indicate that the array is stored as a file, or `shm://` to indicate that the array is stored as a POSIX shared memory object. For backward compatibility `shm://` is assumed when no prefix is given The content of the array lives in shared memory and/or in a file and won't be lost when the numpy array is deleted, nor when the python interpreter exits. To delete a shared array reclaim system resources use the `SharedArray.delete()` function. ### SharedArray.attach(name) The contents of the array will not be deleted when this array is destroyed, either implicitly or explicitly by calling `del`, it will simply be detached from the current process. To delete a shared array and therefore reclaim system resources use the `SharedArray.delete()` function. This function attaches a previously created array in shared memory identified by `name`, which can use the `file://` prefix to indicate that the array is stored as a file, or `shm://` to indicate that the array is stored as a POSIX shared memory object. For backward compatibility `shm://` is assumed when no prefix is given. ### `SharedArray.delete(name)` An array may be simultaneously attached from multiple different processes (i.e. python interpreters). The content of the array lives in shared memory and/or in a file and won't be lost when the numpy array is deleted, nor when the python interpreter exits. To delete a shared array reclaim system resources use the `SharedArray.delete()` function. ### SharedArray.delete(name) This function destroys the previously created array identified by `name`, which can use the `file://` prefix to indicate that the array Loading @@ -83,9 +91,10 @@ is assumed when no prefix is given After calling `delete`, the array will not be attachable anymore, but existing attachments will remain valid until they are themselves destroyed. destroyed. The data is reclaimed by the system when the very last attachment is deleted. ### `SharedArray.list()` ### SharedArray.list() This function returns a list of previously created arrays stored as POSIX SHM objects, along with their name, data type and dimensions. Loading @@ -93,7 +102,7 @@ At the moment this function only works on Linux because it accesses files exposed under `/dev/shm`. There doesn't seem to be a portable method of doing that. ### `SharedArray.msync(array, flags)` ### SharedArray.msync(array, flags) This function is a wrapper around `msync(2)` and is only useful when using file-backed arrays (i.e. not POSIX shared memory). msync(2) Loading @@ -102,29 +111,29 @@ are exported as constants in the module definition (see below) and are a 1:1 map of the `msync(2)` flags, please refer to the manual page of `msync(2)` for details. ### `SharedArray.mlock(array)` ### SharedArray.mlock(array) This function is a wrapper around `mlock(2)`: lock the memory map into RAM, preventing that memory from being paged to the swap area. ### `SharedArray.munlock(array)` ### SharedArray.munlock(array) This function is a wrapper around `munlock(2)`: unlock the memory map, allowing that memory to be paged to the swap area. ## Constants ### `SharedArray.MS_ASYNC` ### SharedArray.MS_ASYNC Flag for `SharedArray.msync()`. Specifies that an update be scheduled, but the call returns immediately. ### `SharedArray.MS_SYNC` ### SharedArray.MS_SYNC Flag for `SharedArray.msync()`. Requests an update and waits for it to complete. ### `SharedArray.MS_INVALIDATE` ### SharedArray.MS_INVALIDATE Flag for `SharedArray.msync()`. Asks to invalidate other mappings of the same file (so that they can be updated with the fresh values just Loading @@ -133,12 +142,32 @@ written). ## Requirements * Python 2.7 or 3+ * Numpy 1.8 * Numpy 1.8+ * Posix shared memory interface SharedArray uses the posix shm interface (`shm_open` and `shm_unlink`) and so should work on most operating systems that follow the posix standards (Linux, *BSD, etc.). and so should work on most POSIX operating systems (Linux, BSD, etc.). ## Installation The extension uses the `distutils` python package that should be familiar to most python users. To test the extension directly from the source tree, without installing, type: ```sh python setup.py build_ext --inplace ``` To build and install the extension system-wide, type: ```sh python setup.py build sudo python setup.py install ``` The package is also available on PyPI and can be installed using the pip tool. ## FAQ Loading @@ -146,13 +175,17 @@ standards (Linux, *BSD, etc.). A few people have reported segfaults with very large arrays using POSIX shared memory. This is not a bug in SharedArray but rather an indication that the system ran out of POSIX shared memory. On Linux a `tmpfs` virtual filesystem is used to provide POSIX shared memory, and by default it is given only about 20% of the total available memory, depending on the distribution. That amount can be changed by re-mounting the `tmpfs` filesystem with the `size=100%` option: indication that the system ran out of POSIX shared memory. On Linux a `tmpfs` virtual filesystem is used to provide POSIX shared memory, and by default it is given only about 20% of the total available memory, depending on the distribution. That amount can be changed by re-mounting the `tmpfs` filesystem with the `size=100%` option: ```sh sudo mount -o remount,size=100% /run/shm ``` Also you can make the change permanent, on next boot, by setting `SHM_SIZE=100%` in `/etc/defaults/tmpfs` on recent Debian Loading @@ -166,27 +199,15 @@ arrays created with a previous version of SharedArray aren't compatible with the new version (the location of the metadata changed). Save your work before upgrading. ## Installation The extension uses the `distutils` python package that should be familiar to most python users. To test the extension directly from the source tree, without installing, type: python setup.py build_ext --inplace To build and install the extension system-wide, type: python setup.py build sudo python setup.py install ## Contact For updates and the browse the code, the canonical repository is: <https://parad0x.org/git/python/shared-array/> This package is hosted on [GitLab](https://gitlab.com) at: <https://gitlab.com/tenzing/shared-array> Packages are also available on PyPi at: <https://pypi.python.org/pypi/SharedArray> For bug reports, feature requests, suggestions, patches and everything else related to SharedArray, please contact the maintainer at: <mat@parad0x.org>. else related to SharedArray, feel free to raise issues on the [project page](https://gitlab.com/tenzing/shared-array). You can also contact the maintainer directly by email at <mat@parad0x.org>.
setup.py +1 −1 Original line number Diff line number Diff line Loading @@ -41,7 +41,7 @@ setup(name = 'SharedArray', # Contact author = 'Mathieu Mirmont', author_email = 'mat@parad0x.org', url = 'http://parad0x.org/git/python/shared-array/about', url = 'https://gitlab.com/tenzing/shared-array', # License license = 'https://www.gnu.org/licenses/gpl-2.0.html', Loading