Commit ed1b7919 authored by Dan Gass's avatar Dan Gass
Browse files

Start Alpha 13 branch

clean up structure type construction
parent e4ff4954
......@@ -63,21 +63,21 @@ Structures
A |Structure| subclass offers a succinct and intuitive method for
defining a bytes layout for packing and unpacking structured data.
Within the subclass definition, a class variable annotation specifies
the member name, followed by a colon, then the |plum| type for
controlling how the member is packed/unpacked. Optionally, a default
value for the member (to use in the absence of that member when packing
the structure) follows an equal sign. |Structure| types support
:class:`list` behaviors but also provide attribute access to the members.
Within the subclass definition, add member definitions as class
attributes to specify the member type. Type annotations are optional.
Specifying a default value for the member facilitates instantiating
or packing without specifying the member and is optional. |Structure|
types support :class:`list` behaviors but also provide attribute
access to the members.
For example:
>>> from plum.structure import Structure
>>> from plum.structure import Member, Structure
>>> from plum.int.big import UInt8, UInt16
>>>
>>> class MyStruct(Structure):
... byte: UInt8
... word: UInt16 = 1 # has default value
... byte: int = Member(cls=UInt8)
... word: int = Member(cls=UInt16, default=1)
...
>>> # use dictionaries to specify member values,
>>> # you may omit members with defaults
......@@ -124,12 +124,12 @@ supports string conversion to facilitate logging and debugging as
well as supports calls to print the summary to the console.
For example:
>>> from plum.structure import Structure
>>> from plum.structure import Member, Structure
>>> from plum.int.little import UInt8, UInt16
>>>
>>> class MyStruct(Structure):
... byte: UInt8
... word: UInt16
... byte: int = Member(cls=UInt8)
... word: int = Member(cls=UInt16)
...
>>> mystruct = MyStruct(byte=1, word=2)
>>> mystruct.dump()
......@@ -155,15 +155,15 @@ the dump summary table:
>>> from plum.int.enum import Enum
>>> from plum.int.little import UInt8
>>> from plum.structure import Structure
>>> from plum.structure import Member, Structure
>>>
>>> class Pet(Enum, nbytes=1):
... CAT = 0
... DOG = 1
...
>>> class Family(Structure):
... nkids: UInt8
... pet: Pet
... nkids: int = Member(cls=UInt8)
... pet: Pet = Member(cls=Pet)
...
>>> family = Family.unpack(b'\x02\x01')
>>> family.dump()
......@@ -319,37 +319,6 @@ format:
bytearray(b'\x01\x02\x03\x04')
The :func:`pack_freeform` function facilitates packing multiple items without
an up front format. Instead, it accepts |plum| type class and value pairs either
in serialized or grouped form:
>>> from plum import pack_freeform
>>>
>>> # serialized class/value pairs
>>> pack_freeform(UInt8, 1, UInt16, 2)
bytearray(b'\x01\x00\x02')
>>>
>>> # grouped class/value pairs
>>> pack_freeform(
... (UInt8, 1),
... (UInt16, 2))
bytearray(b'\x01\x00\x02')
:func:`pack_freeform` accepts |plum| type instances instead of
or in any combination with class/value pairs:
>>> pack_freeform(
... UInt8, 1, # serialized
... UInt8(2), # instance
... (UInt8, 3) # grouped
... )
bytearray(b'\x01\x02\x03')
.. Tip::
Passing class and value arguments in serialized form avoids an extra
object instantiation and slightly increases packing performance.
Additional pack functions and methods exist for packing directly into
buffers or obtaining byte summary dumps. Visit the
:doc:`Packing Tutorial </tutorials/features/packing>` for more details.
......
......@@ -59,10 +59,6 @@ Utility Functions
:func:`pack_and_dump` Pack values as bytes and produce bytes summary following a format.
:func:`pack_into` Pack values as bytes into a buffer following a format.
:func:`pack_into_and_dump` Pack values as bytes into a buffer following a format and produce bytes summary.
:func:`pack_freeform` Pack values as bytes.
:func:`pack_freeform_and_dump` Pack values as bytes and produce bytes summary.
:func:`pack_freeform_into` Pack values as bytes into a buffer.
:func:`pack_freeform_into_and_dump` Pack values as bytes into a buffer and produce bytes summary.
:func:`unpack` Unpack item(s) from bytes.
:func:`unpack_and_dump` Unpack item(s) from bytes and produce packed bytes summary.
:func:`unpack_from` Unpack item(s) from within a bytes buffer.
......@@ -77,10 +73,6 @@ Utility Functions
pack_and_dump() <utilities/pack_and_dump.rst>
pack_into() <utilities/pack_into.rst>
pack_into_and_dump() <utilities/pack_into_and_dump.rst>
pack_freeform() <utilities/pack_freeform.rst>
pack_freeform_and_dump() <utilities/pack_freeform_and_dump.rst>
pack_freeform_into() <utilities/pack_freeform_into.rst>
pack_freeform_into_and_dump() <utilities/pack_freeform_into_and_dump.rst>
unpack() <utilities/unpack.rst>
unpack_from() <utilities/unpack_from.rst>
unpack_and_dump() <utilities/unpack_and_dump.rst>
......
......@@ -10,19 +10,19 @@
.. autoclass:: Structure(mapping, **kwargs)
.. autofunction:: dims_of
.. autoclass:: DimsMember
.. autofunction:: dims_via
.. autoclass:: Member
.. autofunction:: member
.. autoclass:: SizeMember
.. autofunction:: size_of
.. autoclass:: TypeMember
.. autofunction:: size_via
.. autoclass:: VariableDimsMember
.. autofunction:: type_of
.. autoclass:: VariableSizeMember
.. autofunction:: type_via
.. autoclass:: VariableTypeMember
.. seealso::
......
#########################################
[plum] Utility Reference: pack_freeform()
#########################################
.. include:: ../../alias.txt
.. currentmodule:: plum
.. autofunction:: pack_freeform
##################################################
[plum] Utility Reference: pack_freeform_and_dump()
##################################################
.. include:: ../../alias.txt
.. currentmodule:: plum
.. autofunction:: pack_freeform_and_dump
##############################################
[plum] Utility Reference: pack_freeform_into()
##############################################
.. include:: ../../alias.txt
.. currentmodule:: plum
.. autofunction:: pack_freeform_into
#######################################################
[plum] Utility Reference: pack_freeform_into_and_dump()
#######################################################
.. include:: ../../alias.txt
.. currentmodule:: plum
.. autofunction:: pack_freeform_into_and_dump
......@@ -8,6 +8,20 @@ Versions increment per `semver <http://semver.org/>`_ (except for alpha series).
Alpha (0.1.0aX) Versions (only serial number X increments)
**********************************************************
+ 0.1.0a13 (2020-Feb-9)
- Allow structure subclasses to be further subclassed.
- Require use of :class:`Member()` for all structure member definitions.
Previously type annotations were used exclusively to define members.
This change facilitates allowing other class attributes to have
type annotations and facilitates structure members having a different
type annotation than the plum type.
- Change variable structure member mechanisms:
- ``type_of()``, ``type_via()`` now are :class:`TypeMember` and :class:`VariableTypeMember`
- ``dims_of()``, ``dims_via()`` now are :class:`DimsMember` and :class:`VariableDimsMember`
- ``size_of()``, ``size_via()`` now are :class:`SizeMember` and :class:`SizeTypeMember`
- Eliminate ``member()`` function.
- Eliminate all "freeform" pack functions/methods.
+ 0.1.0a12 (2020-Feb-04)
- Fix memory leaks in plum-boost.
- Fix bitfield representations when embedded within a structure or another
......
......@@ -39,7 +39,8 @@ call the :meth:`__unpack__` method of their subelements.
|Structure| types call the :meth:`__unpack__` method and pass the structure
instance within a list as ``parents`` (or if ``parents`` is already a list,
appends the structure instance to the list). This allows unpacking logic to inspect
previously unpacked structure members. The :func:`size_via` function demonstrated in
previously unpacked structure members. The :class:`SizeMember` and
:class:`VariableSizeMember` member definitions demonstrated in
the following example works this way. It swaps out the ``array`` member type
with a subclass of the |Array| type specified and implements the subclass's
:meth:`__unpack__` method to inspect the previously unpacked ``count`` member
......@@ -47,14 +48,14 @@ and uses it to unpack the array. This prevents the normally greedy array
originally specified from consuming the ``bookend`` member.
>>> from plum import unpack
>>> from plum.structure import Structure, size_via
>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>>
>>> class Struct1(Structure):
... count: UInt8
... array: Array = size_via('count')
... bookend: UInt8
... count: int = SizeMember(cls=UInt8)
... array: list = VariableSizeMember(size_member=count, cls=Array)
... bookend: int = Member(cls=UInt8)
...
>>> unpack(Struct1, b'\x02\x01\x02\x99').dump()
+--------+----------------+-------+-------+---------+
......@@ -77,7 +78,7 @@ the original |Array| as the member type and achieves the same net effect
as the :func:`dims_via` function did:
>>> from plum import getbytes, unpack
>>> from plum.structure import Structure
>>> from plum.structure import Member, Structure
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>>
......@@ -90,9 +91,9 @@ as the :func:`dims_via` function did:
... return item, offset
...
>>> class Struct2(Structure):
... count: UInt8
... array: CustomArray
... bookend: UInt8
... count: int = Member(cls=UInt8)
... array: list = Member(cls=CustomArray)
... bookend: int = Member(cls=UInt8)
...
>>> unpack(Struct2, b'\x02\x01\x02\x99').dump()
+--------+----------------+-------+-------+-------------+
......@@ -125,7 +126,8 @@ type is checked to determine if it implements the following method:
.. code-block:: none
@classmethod
def __touchup__(cls, value, parent):
def __touchup__(cls, parent):
...
return value
......@@ -136,30 +138,29 @@ method if it exists. The member's initial value from the structure instantiation
is passed as ``value`` while the sibling members are present in the structure
passed as ``parent``.
The following example demonstrates the spirit of how the :func:`dims_of` works
in the examples in the last section. :func:`dims_of` swaps out the ``count``
member type with a subclass of the ``UInt8`` type specified and includes a special
:meth:`touchup` subclass method similar to the ``UInt8`` subclass defined below.
In the example structure definition, the subclass replaces the original ``UInt8``
as the member type and achieves the same net effect as the :func:`dims_of` function:
The following example demonstrates the spirit of how the :class:`SizeMember` and
:class:`VariableSizeMember` work in the examples in the last section.
:class:`SizeMember` swaps out the ``count`` member type with a subclass of the ``UInt8``
type specified and includes a special :meth:`__touchup__` subclass method similar to the
``UInt8`` subclass defined below. In the example structure definition, the subclass
replaces the original ``UInt8`` as the member type and achieves the same net effect as
the :class:`SizeMember` member definition:
>>> from plum import pack
>>> from plum.structure import Structure
>>> from plum.structure import Member, Structure
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>>
>>> class Count(UInt8):
...
... @classmethod
... def __touchup__(cls, value, parent):
... if value is None:
... value = len(parent.array)
... return value
... def __touchup__(cls, parent):
... return len(parent.array)
...
>>> class Struct3(Structure):
... count: Count
... array: Array
... bookend: UInt8
... count: int = Member(cls=Count)
... array: list = Member(cls=Array)
... bookend: int = Member(cls=UInt8)
...
>>> Struct3(array=[1, 2], bookend=0x99).dump()
+--------+----------------+-------+-------+---------+
......
......@@ -15,12 +15,12 @@ supports string conversion to facilitate logging and debugging as
well as supports calls to print the summary to the console.
For example:
>>> from plum.structure import Structure
>>> from plum.structure import Member, Structure
>>> from plum.int.little import UInt8, UInt16
>>>
>>> class MyStruct(Structure):
... byte: UInt8
... word: UInt16
... byte: int = Member(cls=UInt8)
... word: int = Member(cls=UInt16)
...
>>> mystruct = MyStruct(byte=1, word=2)
>>> mystruct.dump()
......
......@@ -15,14 +15,14 @@ increases for every level to help visualize the nesting.
>>> from plum import unpack_and_dump
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>> from plum.structure import Structure
>>> from plum.structure import Member, Structure
>>>
>>> class InnerArray(Array, dims=(3,), item_cls=UInt8):
... pass
...
>>> class MyStruct(Structure):
... count: UInt8
... array: InnerArray
... count: int = Member(cls=UInt8)
... array: list = Member(cls=InnerArray)
...
>>> class OuterArray(Array, dims=(2,), item_cls=MyStruct):
... pass
......
......@@ -15,12 +15,12 @@ packing bytes from the convenient forms supported by the various
The tutorial uses the following representative |plum| type, but the
examples apply to all |plum| types or derivatives of them.
>>> from plum.structure import Structure
>>> from plum.structure import Member, Structure
>>> from plum.int.big import UInt8, UInt16
>>>
>>> class Sample(Structure):
... m1: UInt16
... m2: UInt8
... m1: int = Member(cls=UInt16)
... m2: int = Member(cls=UInt8)
...
>>>
......@@ -96,79 +96,18 @@ format. For exammple:
bytearray(b'\x01\x02\x03\x04')
*******************************
Free Form Pack Utility Function
*******************************
The :func:`pack_freeform` function facilitates packing multiple items without
an up front format. Instead, it accepts |plum| type class and value pairs either
in serialized or grouped form:
>>> from plum import pack_freeform
>>>
>>> # serialized class/value pairs
>>> pack_freeform(UInt8, 1, UInt16, 2)
bytearray(b'\x01\x00\x02')
>>>
>>> # grouped class/value pairs
>>> pack_freeform(
... (UInt8, 1),
... (UInt16, 2))
bytearray(b'\x01\x00\x02')
:func:`pack_freeform` accepts |plum| type instances instead of
or in any combination with class/value pairs:
>>> pack_freeform(
... UInt8, 1, # serialized
... UInt8(2), # instance
... (UInt8, 3) # grouped
... )
bytearray(b'\x01\x02\x03')
.. Tip::
Passing class and value arguments in serialized form avoids an extra
object instantiation and slightly increases packing performance.
The :func:`pack_freeform` also accepts |plum| type class/pairs or instances
as keyword arguments.
>>> pack_freeform(a=(UInt8, 0), b=UInt8(1))
bytearray(b'\x00\x01')
The keyword argument name can be used to improve your code readability but
also helps readability of byte summary dumps resulting from a packing
exception or resulting from using the "add dump" alternative form:
>>> from plum import pack_freeform_and_dump
>>>
>>> buffer, dump = pack_freeform_and_dump(a=(UInt8, 0), b=UInt8(1))
>>> buffer
bytearray(b'\x00\x01')
>>> print(dump)
+--------+--------+-------+-------+-------+
| Offset | Access | Value | Bytes | Type |
+--------+--------+-------+-------+-------+
| 0 | a | 0 | 00 | UInt8 |
+--------+--------+-------+-------+-------+
| 1 | b | 1 | 01 | UInt8 |
+--------+--------+-------+-------+-------+
**************************************
Packing Directly Within a Bytes Buffer
**************************************
The :meth:`pack_into` |plum| type method as well as the :func:`pack_into`
and :func:`pack_freeform_into` utility functions have the same packing
behaviors discussed in the previous sections except the functions take
required buffer and offset arguments. The buffer may be any writeable
bytes-like buffer. The offset argument specifies the number of bytes from
the beginning of the buffer to start placing packed item bytes.
utility function have the same packing behaviors discussed in the previous
sections except the functions take required buffer and offset arguments.
The buffer may be any writeable bytes-like buffer. The offset argument
specifies the number of bytes from the beginning of the buffer to start
placing packed item bytes.
>>> from plum import pack_freeform_into, pack_into
>>> from plum import pack_into
>>> from plum.int.big import UInt8
>>>
>>> OFFSET = 1
......@@ -192,11 +131,6 @@ the beginning of the buffer to start placing packed item bytes.
>>> pack_into(fmt, buffer, OFFSET, 0xcc, 0xdd)
>>> buffer
bytearray(b'\x00\xcc\xdd')
>>>
>>> # free form utility function
>>> pack_freeform_into(buffer, OFFSET, UInt8, 0xee, UInt8, 0xff)
>>> buffer
bytearray(b'\x00\xee\xff')
**Caveats**
......@@ -208,7 +142,7 @@ an exception:
>>> buffer
bytearray(b'\x00\x00')
>>>
>>> pack_freeform_into(buffer, 3, UInt8(0x99))
>>> pack_into(UInt8, buffer, 3, 0x99)
Traceback (most recent call last):
...
plum._exceptions.PackError: offset 3 out of range for 2-byte buffer
......@@ -222,7 +156,7 @@ extended:
>>> buffer
bytearray(b'\x00\x00')
>>>
>>> pack_freeform_into(buffer, 1, UInt16(0xaabb))
>>> pack_into(UInt16, buffer, 1, 0xaabb)
>>> buffer
bytearray(b'\x00\xaa\xbb')
......
......@@ -15,12 +15,12 @@ unpacking bytes into the convenient forms supported by the various
The tutorial uses the following representative |plum| type, but the
examples apply to all |plum| types or derivatives of them.
>>> from plum.structure import Structure
>>> from plum.structure import Member, Structure
>>> from plum.int.big import UInt8, UInt16
>>>
>>> class Sample(Structure):
... m1: UInt16
... m2: UInt8
... m1: int = Member(cls=UInt16)
... m2: int = Member(cls=UInt8)
...
>>>
......
......@@ -8,17 +8,18 @@
To improve code readability, avoid creating a custom |Structure| subclass
where it makes sense. Doing so simplifies your code, saves memory (each
Python type consumes a few hundered bytes), and reduces import times.
Instead, use keyword arguments to pass values to the :func:`pack_freeform` utility.
Each value provided as a keyword argument must be an instance of a |plum|
type (e.g. ``UInt8``):
Instead, use a dictionary format and keyword arguments to pass values to
the :func:`pack` utility. For example:
>>> from plum import pack_freeform
>>> from plum import pack
>>> from plum.int.little import UInt8
>>>
>>> # this is cryptic
>>> pack_freeform(UInt8, 12, UInt8, 25)
>>> fmt = (UInt8, UInt8)
>>> pack(fmt, 12, 25)
bytearray(b'\x0c\x19')
>>>
>>> # this self-documents
>>> pack_freeform(month=UInt8(12), day=UInt8(25))
>>> fmt = {'month': UInt8, 'day': UInt8}
>>> pack(fmt, month=12, day=25)
bytearray(b'\x0c\x19')
......@@ -45,14 +45,14 @@ argument:
>>> from plum import pack, unpack
>>> from plum.bytearray import ByteArray
>>> from plum.int.little import UInt8
>>> from plum.structure import Structure
>>> from plum.structure import Member, Structure
>>>
>>> class ByteArray1(ByteArray, nbytes=4):
... """Interpret bytes as a byte array of length 4."""
...
>>> class Struct1(Structure):
... barray: ByteArray1
... bookend: UInt8
... barray: bytearray = Member(cls=ByteArray1)
... bookend: int = Member(cls=UInt8)
...
>>> struct1 = unpack(Struct1, b'\x00\x01\x02\x03\x99')
>>> struct1.dump()
......@@ -76,8 +76,8 @@ with when the initial value contains fewer than ``nbytes`` bytes:
... """Interpret bytes as a byte array of length 4."""
...
>>> class Struct2(Structure):
... barray: ByteArray2
... bookend: UInt8
... barray: bytearray = Member(cls=ByteArray2)
... bookend: int = Member(cls=UInt8)
...
>>> struct2 = Struct2(barray=ByteArray2([1,2]), bookend=0x99)
>>> struct2.dump()
......@@ -95,20 +95,20 @@ with when the initial value contains fewer than ``nbytes`` bytes:
Sized Structure Member
**********************
To couple a size member to a byte array member, use the :func:`size_of`
function to specify the associated member name. The following example
To couple a size member to a byte array member, use the :class:`SizeMember`
and :class:`VariableSizeMember` classes. The following example
demonstrates. See the :doc:`Sized Object Tutorial <structure/sizedobject>`
for details.
>>> from plum import pack, unpack
>>> from plum.bytearray import ByteArray
>>> from plum.int.little import UInt8
>>> from plum.structure import Structure, size_of
>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>>
>>> class Struct3(Structure):
... size: UInt8 = size_of('barray')
... barray: ByteArray
... bookend: UInt8
... size: int = SizeMember(cls=UInt8)
... barray: bytearray = VariableSizeMember(size_member=size, cls=ByteArray)
... bookend: int = Member(cls=UInt8)
...
>>> # size member automatically filled in
>>> struct3 = Struct3(barray=[0] * 8, bookend=0x99)
......@@ -135,41 +135,6 @@ for details.
+--------+----------------+--------------------+-------+-----------+
Alternatively, use the :func:`size_via` function with the byte array member
to specify the associated size member name:
>>> from plum.structure import size_via
>>>
>>> class Struct4(Structure):
... size: UInt8
... barray: ByteArray = size_via('size')
... bookend: UInt8
...
>>> # size member automatically filled in
>>> struct4 = Struct4(barray=[0] * 8, bookend=0x99)
>>> struct4.dump()
+--------+----------------+------------------------------------------------+-------------------------+-----------+
| Offset | Access | Value | Bytes | Type |
+--------+----------------+------------------------------------------------+-------------------------+-----------+