Skip to content

GitLab

Commit b0614abc authored by Dan Gass's avatar Dan Gass
Browse files
Options

change representation of view 

show true offset in dumps of view (just in dump requests)
improve view tests for representations
parent 79ec10fd
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 2864 additions and 1034 deletions
.gitignore
View file @ b0614abc
......@@ -2,8 +2,8 @@ cd # Compiled python modules.
*.pyc
# Setuptools distribution folder.
*/dist/
*/build/
dist/
build/
# Python egg metadata, regenerated from source files by setuptools.
**/*.egg-info
......@@ -40,5 +40,3 @@ docs/_build/
.tox/
.coverage
# Regression test baseline updates
*.update.py
.gitlab-ci.yaml
View file @ b0614abc
......@@ -60,7 +60,7 @@ py37_plum-boost:
pylint_plum-py:
stage: lint
image: python:3.6
image: python:3.7
before_script:
- pip install tox
- cd plum-py
......@@ -69,7 +69,7 @@ pylint_plum-py:
pylint_plum-boost:
stage: lint
image: python:3.6
image: python:3.7
before_script:
- pip install tox
- cd plum-boost
......@@ -78,7 +78,7 @@ pylint_plum-boost:
pylint_plum-benchmark:
stage: lint
image: python:3.6
image: python:3.7
before_script:
- pip install tox
- cd plum-benchmark
......@@ -87,14 +87,14 @@ pylint_plum-benchmark:
pylint_plum-tests:
stage: lint
image: python:3.6
image: python:3.7
before_script:
- pip install tox
- cd plum-tests
script:
- tox -vv -e pylint
build_plum-py:
build_plum-py_py36:
stage: build
image: python:3.6
before_script:
......@@ -107,7 +107,7 @@ build_plum-py:
- plum-py/.tox/build/tmp
when: always
build_plum-boost:
build_plum-boost_py36:
stage: build
image: python:3.6
before_script:
......@@ -120,9 +120,35 @@ build_plum-boost:
- plum-boost/.tox/dist
when: always
build_plum-py_py37:
stage: build
image: python:3.7
before_script:
- pip install tox
- cd plum-py
script:
- tox -vv -e build
artifacts:
paths:
- plum-py/.tox/build/tmp
when: always
build_plum-boost_py37:
stage: build
image: python:3.7
before_script:
- pip install tox
- cd plum-boost
script:
- tox -vv -e build
artifacts:
paths:
- plum-boost/.tox/dist
when: always
build_plum-benchmark:
stage: build
image: python:3.6
image: python:3.7
before_script:
- pip install tox
- cd plum-benchmark
......@@ -135,7 +161,7 @@ build_plum-benchmark:
build_plum-tests:
stage: build
image: python:3.6
image: python:3.7
before_script:
- pip install tox
- cd plum-tests
......@@ -148,7 +174,7 @@ build_plum-tests:
make_docs:
stage: build
image: python:3.6
image: python:3.7
before_script:
- pip install tox
- cd plum-py
......
README.rst
View file @ b0614abc
......@@ -15,13 +15,13 @@
:alt: Documentation Status
The |plum| Python package provides classes and utility functions to
efficiently pack and unpack memory bytes and conveniently access the
memory data. The package offers a collection of basic types which may
be further customized and combined in any combination to handle
complex data structures. Most significant features include:
efficiently pack and unpack bytes and conveniently access bytes
within a buffer. The package offers a collection of basic types which may
be further customized and combined to handle
complex data structures. Features include:
- Arbitrary Nesting (e.g. structure of array of structure of bitfields of bitfields)
- Easy to Create Custom Types
- Speed Potential (goal of nearing :mod:`struct` performance)
- Visual Memory Summaries (including partial dumps in exception messages)
- :doc:`Visual Memory Summaries </tutorials/features/dumps>`
- :doc:`Speed Efficient </tutorials/tips_tricks/speed>`
- :doc:`Arbitrary Nesting </tutorials/features/nesting>`
- Extensible API
contribution_waiver.txt
View file @ b0614abc
......@@ -92,7 +92,7 @@ By adding my name to this signature list and dating, I acknowledge:
--------- --------------------------------------------------------
01Mar2019 Jan Novak (job.jan.novak@gmail.com)
16Mar2019 Tyler N. Thieding (python@thieding.com)
29Apr2019 Daniel Mark Gass (dan.gass@gmail.com)
12Dec2019 Daniel Mark Gass (dan.gass@gmail.com)
****************
......
docs/about.rst
View file @ b0614abc
......@@ -37,54 +37,29 @@ What's Next?
- Improve documentation.
- Add simple examples to reference pages.
- Write tutorials.
- Refine regression tests.
- Standardize patterns.
- Develop additional shareable conformance test cases for common features.
- Add feature test cases.
- Add exception (user experience) test cases.
- Add types.
- IpAddr
- MacAddr
- Pointer
- Increase performance.
The long term goal aims to rival performance of the :mod:`struct` standard library
module, a difficult task given that :mod:`struct` is very lean and efficient.
A complimentary :mod:`plum_c` package exists to help achieve this goal. Currently
this proof of concept package offers an alternative version of the :func:`unpack`
function that is implemented in "C" that supports unpacking all |plum| types.
(:mod:`plum_c` source code exists within the |plum| repository but is not yet
available on **PyPi**).
A complimentary ``plum-boost`` package exists to help achieve this goal.
Currently the package offers an alternative versions of the :func:`unpack`,
:func:`unpack_from`, :func:`pack` and :func:`pack_into` functions. These
alternative implementations are implemented in "C" and accelerate
packing and unpacking integers and structures.
Additionally, the :mod:`plum_c` package provides an integer unpack accelerator.
If :mod:`plum_c` is installed, |plum| attaches the integer unpack accelerator
an extra class attribute to the |plum| integer types. Usage of the |plum|
unpack function ignores the extra accelerator attribute and utilizes the normal
Python implementation. Usage of the :mod:`plum_c` unpack function recognizes
the accelarator and utilizes it to boost performance significantly.
Currently the ``plum-boost`` package is not yet available.
Future plans include:
- implement "C" :func:`pack` utility function
- make accelerators for more types (prioritizing the |Structure| type)
- further optimize "C" and Python implementations
- Expand support for "views" of memory.
- Develop additional memory view interfaces such that memory may be local (e.g. a large file)
or remote (e.g. another process or even memory in a networked device).
- Expand support for "views" of buffers.
- For the remaining core |plum| types, develop associated "view" types.
- Use cases:
- interpret portions of large memory image with low overhead
(items aren't parsed until accessed)
- bounce around a memory image via pointer dereferencing
- access memory in networked or remote devices
- access memory in other processes
docs/alias.txt
View file @ b0614abc
......@@ -11,7 +11,6 @@
.. |Float| replace:: :class:`~plum.float.Float`
.. |InsufficientMemoryError| replace:: :class:`~plum.InsufficientMemoryError`
.. |Int| replace:: :class:`~plum.int.Int`
.. |Memory| replace:: :class:`~plum.Memory`
.. |Nil| replace:: :class:`~plum.nil.Nil`
.. |pack()| replace:: :func:`~plum.pack`
.. |PackError| replace:: :class:`~plum.PackError`
......
docs/conf.py
View file @ b0614abc
......@@ -165,6 +165,6 @@ man_pages = [
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'plum', u'plum Documentation',
author, 'plum', 'Pack/Unpack Memory.',
author, 'plum', 'Pack/Unpack Bytes.',
'Miscellaneous'),
]
docs/index.rst
View file @ b0614abc
......@@ -20,64 +20,17 @@ more complete usage examples. The following summarizes the
|plum| types available in this package.
+-------------------------------+----------------------+-----------------------------------------------------+
| Module | Types | Description |
+===============================+======================+=====================================================+
| :mod:`plum.array` | |Array| | list of uniformly typed items |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.bytearray` | |ByteArray| | array of bytes |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.float` | |Float| | float base class |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.float.big` | Float32, Float64 | big endian floating point numbers |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.float.little` | Float32, Float64 | little endian floating point numbers |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.float.native` | Float32, Float64 | native endian floating point numbers |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int` | |Int| | integer base class |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.big` | SInt8, 16, 32, 64 | big endian signed integers |
| +----------------------+-----------------------------------------------------+
| | UInt8, 16, 32, 64 | big endian unsigned integers |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.bitfields` | |BitFields| | integer with bitfield accessors |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.enum` | |Enum| | integer enumerated constants base class |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.enum.big` | Enum8, 16, 32, 64 | big endian integer enumerated constants |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.enum.little` | Enum8, 16, 32, 64 | little endian integer enumerated constants |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.enum.native` | Enum8, 16, 32, 64 | native endian integer enumerated constants |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.flag` | |Flag| | integer with bit flags base class |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.flag.big` | Flag8, 16, 32, 64 | big endian integer with bit flags |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.flag.little` | Flag8, 16, 32, 64 | little endian integer with bit flags |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.flag.native` | Flag8, 16, 32, 64 | native endian integer with bit flags |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.little` | SInt8, 16, 32, 64 | little endian signed integers |
| +----------------------+-----------------------------------------------------+
| | UInt8, 16, 32, 64 | little endian unsigned integers |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.int.native` | SInt8, 16, 32, 64 | native endian signed integers |
| +----------------------+-----------------------------------------------------+
| | UInt8, 16, 32, 64 | native endian unsigned integers |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.nil` | |Nil| | no memory bytes |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.str` | |Str| | string base class |
| +----------------------+-----------------------------------------------------+
| | |AsciiStr| | ASCII encoded string |
| +----------------------+-----------------------------------------------------+
| | |AsciiZeroTermStr| | ASCII encoded zero terminated string |
| +----------------------+-----------------------------------------------------+
| | |Utf8Str| | UTF-8 encoded string |
| +----------------------+-----------------------------------------------------+
| | |Utf8ZeroTermStr| | UTF-8 encoded zero terminated string |
+-------------------------------+----------------------+-----------------------------------------------------+
| :mod:`plum.structure` | |Structure| | structure of uniquely typed items |
+-------------------------------+----------------------+-----------------------------------------------------+
========================================================== ================================================
Plum Type Description
========================================================== ================================================
:doc:`plum.array </tutorials/types/array>` list of uniformly typed
:doc:`plum.bytearray </tutorials/types/bytearray>` array of bytes
:doc:`plum.float </tutorials/types/float>` floating point
:doc:`plum.int </tutorials/types/int>` integers
:doc:`plum.int.bitfields </tutorials/types/int_bitfields>` integer with bitfield accessors
:doc:`plum.int.enum </tutorials/types/int_enum>` integer enumerated constants
:doc:`plum.int.flag </tutorials/types/int_flag>` integer with bit flags
:doc:`plum.nil </tutorials/types/nil>` no bytes
:doc:`plum.str </tutorials/types/str/index>` strings
:doc:`plum.structure </tutorials/types/structure/index>` structure of uniquely typed
========================================================== ================================================
docs/quickstart.rst
View file @ b0614abc
This diff is collapsed.
docs/reference/index.rst
View file @ b0614abc
################
[plum] Reference
################
####################
[plum] API Reference
####################
.. include:: ../alias.txt
.. currentmodule:: plum
**********
Plum Types
**********
============================== ================================================
Module Description
============================== ================================================
:mod:`plum.array` list of uniformly typed
:mod:`plum.bytearray` array of bytes
:mod:`plum.float` floating point
:mod:`plum.int` integer
:mod:`plum.int.bitfields` integer with bitfield accessors
:mod:`plum.int.enum` integer enumerated constants
:mod:`plum.int.flag` integer with bit flags
:mod:`plum.nil` no bytes
:mod:`plum.str` string
:mod:`plum.structure` structure of uniquely typed
============================== ================================================
.. toctree::
:hidden:
array <types/array.rst>
bytearray <types/bytearray.rst>
float <types/float.rst>
float.big <types/float_big.rst>
float.little <types/float_little.rst>
float.native <types/float_native.rst>
int <types/int.rst>
int.big <types/int_big.rst>
int.little <types/int_little.rst>
int.native <types/int_native.rst>
int.bitfields <types/int_bitfields.rst>
int.enum <types/int_enum.rst>
int.flag <types/int_flag.rst>
nil <types/nil.rst>
str <types/str.rst>
structure <types/structure.rst>
*****************
Utility Functions
*****************
=============================== =======================================================================
=============================== ===========================================================================
Function Description
=============================== =======================================================================
:func:`calcsize` Get size of packed item (in bytes).
:func:`dump` Print packed memory summary.
:func:`exhaust` Verify all memory bytes consumed.
:func:`getbytes` Get memory bytes.
:func:`getdump` Get packed memory summary.
:func:`getvalue` Unpack item from memory bytes of a plum view.
:func:`pack` Pack items and return memory bytes.
:func:`pack_and_getdump` Pack items and return memory bytes and summary.
:func:`pack_into` Pack items into memory bytes.
:func:`pack_into` Pack items into memory bytes and return memory summary.
:func:`setvalue` Pack value into memory bytes of a plum view.
:func:`unpack` Unpack item from memory bytes.
:func:`unpack_from` Unpack item from binary file stream.
:func:`unpack_and_getdump` Unpack item from memory bytes and get memory summary.
:func:`unpack_from_and_getdump` Unpack item from binary file stream and get memory summary.
:func:`view` Create plum view of memory bytes.
=============================== =======================================================================
=============================== ===========================================================================
:func:`getbytes` Get bytes from bytes buffer.
:func:`pack` Pack items as bytes.
:func:`pack_and_dump` Pack items as bytes and produce packed bytes summary.
:func:`pack_into` Pack items as bytes into a bytes buffer.
:func:`pack_into_and_dump` Pack items as bytes into a bytes buffer and produce bytes summary.
:func:`unpack` Unpack item(s) from bytes.
:func:`unpack_and_dump` Unpack item(s) from bytes and produce packed bytes summary.
:func:`unpack_from` Unpack item(s) from within a bytes buffer.
:func:`unpack_from_and_dump` Unpack item(s) from within a bytes buffer and produce packed bytes summary.
=============================== ===========================================================================
.. toctree::
:hidden:
calcsize() <utilities/calcsize.rst>
dump() <utilities/dump.rst>
exhaust() <utilities/exhaust.rst>
getbytes() <utilities/getbytes.rst>
getdump() <utilities/getdump.rst>
getvalue() <utilities/getvalue.rst>
pack() <utilities/pack.rst>
pack_and_getdump() <utilities/pack_and_getdump.rst>
pack_and_dump() <utilities/pack_and_dump.rst>
pack_into() <utilities/pack_into.rst>
pack_into_and_getdump() <utilities/pack_into_and_getdump.rst>
setvalue() <utilities/setvalue.rst>
pack_into_and_dump() <utilities/pack_into_and_dump.rst>
unpack() <utilities/unpack.rst>
unpack_from() <utilities/unpack_from.rst>
unpack_and_getdump() <utilities/unpack_and_getdump.rst>
unpack_from_and_getdump() <utilities/unpack_from_and_getdump.rst>
view() <utilities/view.rst>
unpack_and_dump() <utilities/unpack_and_dump.rst>
unpack_from_and_dump() <utilities/unpack_from_and_dump.rst>
**********
......@@ -72,64 +98,3 @@ Exceptions
PackError <exceptions/PackError.rst>
SizeError <exceptions/SizeError.rst>
UnpackError <exceptions/UnpackError.rst>
*******
Modules
*******
============================== ================================================
Module Description
============================== ================================================
:mod:`plum.array` list of uniformly typed items
:mod:`plum.bytearray` array of bytes
:mod:`plum.float` float base class
:mod:`plum.float.big` big endian floats
:mod:`plum.float.little` little endian floats
:mod:`plum.float.native` native endian floats
:mod:`plum.int` integer base class
:mod:`plum.int.big` big endian integers
:mod:`plum.int.bitfields` integer with bitfield accessors
:mod:`plum.int.enum` integer enumerated constants base class
:mod:`plum.int.enum.big` big endian integer enumerated constants
:mod:`plum.int.enum.little` little endian integer enumerated constants
:mod:`plum.int.enum.native` native endian integer enumerated constants
:mod:`plum.int.flag` integer with bit flags base class
:mod:`plum.int.flag.big` big endian integer with bit flags
:mod:`plum.int.flag.little` little endian integer with bit flags
:mod:`plum.int.flag.native` native endian integer with bit flags
:mod:`plum.int.little` little endian integers
:mod:`plum.int.native` native endian integers
:mod:`plum.memory` memory bytes interface
:mod:`plum.nil` no memory bytes
:mod:`plum.str` string
:mod:`plum.structure` structure of uniquely typed items
============================== ================================================
.. toctree::
:hidden:
array <modules/array.rst>
bytearray <modules/bytearray.rst>
float <modules/float.rst>
float.big <modules/float_big.rst>
float.little <modules/float_little.rst>
float.native <modules/float_native.rst>
int <modules/int.rst>
int.big <modules/int_big.rst>
int.bitfields <modules/int_bitfields.rst>
int.enum <modules/int_enum.rst>
int.enum <modules/int_enum_big.rst>
int.enum <modules/int_enum_little.rst>
int.enum <modules/int_enum_native.rst>
int.flag <modules/int_flag.rst>
int.flag <modules/int_flag_big.rst>
int.flag <modules/int_flag_little.rst>
int.flag <modules/int_flag_native.rst>
int.little <modules/int_little.rst>
int.native <modules/int_native.rst>
memory <modules/memory.rst>
nil <modules/nil.rst>
str <modules/str.rst>
structure <modules/structure.rst>
docs/reference/modules/int_enum_big.rst deleted 100644 → 0
View file @ 79ec10fd
####################################
[plum.int.enum.big] Module Reference
####################################
.. include:: ../../alias.txt
.. automodule:: plum.int.enum.big
.. autoclass:: Enum8
.. autoclass:: Enum16
.. autoclass:: Enum32
.. autoclass:: Enum64
docs/reference/modules/int_enum_little.rst deleted 100644 → 0
View file @ 79ec10fd
#######################################
[plum.int.enum.little] Module Reference
#######################################
.. include:: ../../alias.txt
.. automodule:: plum.int.enum.little
.. autoclass:: Enum8
.. autoclass:: Enum16
.. autoclass:: Enum32
.. autoclass:: Enum64
docs/reference/modules/int_enum_native.rst deleted 100644 → 0
View file @ 79ec10fd
#######################################
[plum.int.enum.native] Module Reference
#######################################
.. include:: ../../alias.txt
.. automodule:: plum.int.enum.native
.. autoclass:: Enum8
.. autoclass:: Enum16
.. autoclass:: Enum32
.. autoclass:: Enum64
docs/reference/modules/int_flag_big.rst deleted 100644 → 0
View file @ 79ec10fd
####################################
[plum.int.flag.big] Module Reference
####################################
.. include:: ../../alias.txt
.. automodule:: plum.int.flag.big
.. autoclass:: Flag8
.. autoclass:: Flag16
.. autoclass:: Flag32
.. autoclass:: Flag64
docs/reference/modules/int_flag_little.rst deleted 100644 → 0
View file @ 79ec10fd
#######################################
[plum.int.flag.little] Module Reference
#######################################
.. include:: ../../alias.txt
.. automodule:: plum.int.flag.little
.. autoclass:: Flag8
.. autoclass:: Flag16
.. autoclass:: Flag32
.. autoclass:: Flag64
docs/reference/modules/int_flag_native.rst deleted 100644 → 0
View file @ 79ec10fd
#######################################
[plum.int.flag.native] Module Reference
#######################################
.. include:: ../../alias.txt
.. automodule:: plum.int.flag.native
.. autoclass:: Flag8
.. autoclass:: Flag16
.. autoclass:: Flag32
.. autoclass:: Flag64
docs/reference/modules/memory.rst deleted 100644 → 0
View file @ 79ec10fd
##############################
[plum.memory] Module Reference
##############################
.. include:: ../../alias.txt
.. automodule:: plum.memory
.. autoclass:: MemoryInterface
:members:
.. autoclass:: LocalMemory