Commit 78973944 authored by Dan Gass's avatar Dan Gass
Browse files

differentiate lists from tuples

start supporting "free" packing with boost
parent 5a2bf048
File mode changed from 100644 to 100755
......@@ -286,28 +286,69 @@ Packing Items
The previous quick start examples utilized the :meth:`pack` method
to produce the byte sequence. Alternatively, the :func:`pack` function
facilitates packing multiple items at once and accepts |plum| type
class and value pairs. This avoids the need for creating |Structure|
types or concatenating binary strings together:
facilitates packing multiple items at once and mirrors the standard library
:func:`struct.pack` function except instead of accepting a format string, it
accepts |plum| types:
>>> from plum import pack
>>> from plum.int.little import UInt16, UInt8
>>> pack(UInt8, 1, UInt16, 2)
bytearray(b'\x01\x02\x00')
>>> from plum.int.big import UInt16, UInt8
>>>
>>> # format as single plum type accepts a value as positional argument
>>> fmt = UInt8
>>> pack(fmt, 1)
bytearray(b'\x01')
>>>
>>> # format as tuple/list of types accepts values as positional arguments
>>> fmt = (UInt8, UInt16)
>>> pack(fmt, 1, 2)
bytearray(b'\x01\x00\x02')
>>>
>>> # format as dictionary of types accepts values as keyword arguments
>>> fmt = {'a': UInt8, 'b': UInt16}
>>> pack(fmt, a=1, b=2)
bytearray(b'\x01\x00\x02')
The :func:`pack` function supports packing arbitrarily nested values within
by specifying the structure using tuples/lists and dictionaries within the
format:
>>> fmt = {'a': (UInt8, UInt8), 'b': {'i': UInt8, 'ii': UInt8}}
>>>
>>> pack(fmt, a=(1, 2), b={'i': 3, 'ii': 4})
bytearray(b'\x01\x02\x03\x04')
The :func:`pack_freeform` function facilitates packing multiple items without
an up front format. Instead, it accepts |plum| type class and value pairs either
in serialized or grouped form:
>>> from plum import pack_freeform
>>>
>>> # serialized class/value pairs
>>> pack_freeform(UInt8, 1, UInt16, 2)
bytearray(b'\x01\x00\x02')
>>>
>>> # grouped class/value pairs
>>> pack_freeform(
... (UInt8, 1),
... (UInt16, 2))
bytearray(b'\x01\x00\x02')
Additionally, :func:`pack` accepts |plum| type instances instead of
or in any combination with class/value arguments:
:func:`pack_freeform` accepts |plum| type instances instead of
or in any combination with class/value pairs:
>>> pack(
... UInt8, 1, # separate
>>> pack_freeform(
... UInt8, 1, # serialized
... UInt8(2), # instance
... (UInt8, 3) # grouped
... )
bytearray(b'\x01\x02')
bytearray(b'\x01\x02\x03')
.. Tip::
Passing separate class and value arguments avoids an extra
object instantiation and increases packing performance.
.. Tip::
Passing class and value arguments in serialized form avoids an extra
object instantiation and slightly increases packing performance.
Additional pack functions and methods exist for packing directly into
buffers or obtaining byte summary dumps. Visit the
......@@ -319,8 +360,9 @@ Unpacking Items
The previous quick start examples utilized the :meth:`unpack` method
to convert a byte sequence into a convenient Python form. Alternatively,
the :func:`unpack` function exists for the same purpose but also to
unpack multiple items at once:
the :func:`unpack` function facilitates unpacking multiple items at once
and mirrors the standard library :func:`struct.pack` function except
instead of accepting a format string, it accepts |plum| types:
>>> from plum import unpack
>>> from plum.int.big import UInt8
......@@ -337,6 +379,18 @@ unpack multiple items at once:
{'a': 3, 'b': 4}
The :func:`unpack` function supports organizing the unpacked values into an
arbitrarily nested structure by specifying the structure using tuples/lists
and dictionaries within the format:
>>> from pprint import pprint
>>>
>>> fmt = {'a': (UInt8, UInt8), 'b': {'i': UInt8, 'ii': UInt8}}
>>>
>>> unpack(fmt, b'\x01\x02\x03\x04')
{'a': (1, 2), 'b': {'i': 3, 'ii': 4}}
In some use cases such as when the number of items or the type of item
to unpack depends on previously unpacked items, items must be unpacked
one at a time. Each unpack operation consumes a piece, but not all
......@@ -364,7 +418,7 @@ completely consumed the bytes. In the following example, the extra unconsumed by
plum._exceptions.ExcessMemoryError: 1 unconsumed bytes
Additional pack functions and methods exist for unpacking directly from a
Additional unpack functions and methods exist for unpacking directly from a
buffer or obtaining byte summary dumps. Visit the
:doc:`Unpacking Tutorial </tutorials/features/unpacking>` for more details.
......
......@@ -51,19 +51,23 @@ Plum Types
Utility Functions
*****************
=============================== ===========================================================================
Function Description
=============================== ===========================================================================
:func:`getbytes` Get bytes from bytes buffer.
:func:`pack` Pack items as bytes.
:func:`pack_and_dump` Pack items as bytes and produce packed bytes summary.
:func:`pack_into` Pack items as bytes into a bytes buffer.
:func:`pack_into_and_dump` Pack items as bytes into a bytes buffer and produce bytes summary.
:func:`unpack` Unpack item(s) from bytes.
:func:`unpack_and_dump` Unpack item(s) from bytes and produce packed bytes summary.
:func:`unpack_from` Unpack item(s) from within a bytes buffer.
:func:`unpack_from_and_dump` Unpack item(s) from within a bytes buffer and produce packed bytes summary.
=============================== ===========================================================================
=================================== ================================================================================
Function Description
=================================== ================================================================================
:func:`getbytes` Get bytes from bytes buffer.
:func:`pack` Pack values as bytes following a format.
:func:`pack_and_dump` Pack values as bytes and produce bytes summary following a format.
:func:`pack_into` Pack values as bytes into a buffer following a format.
:func:`pack_into_and_dump` Pack values as bytes into a buffer following a format and produce bytes summary.
:func:`pack_freeform` Pack values as bytes.
:func:`pack_freeform_and_dump` Pack values as bytes and produce bytes summary.
:func:`pack_freeform_into` Pack values as bytes into a buffer.
:func:`pack_freeform_into_and_dump` Pack values as bytes into a buffer and produce bytes summary.
:func:`unpack` Unpack item(s) from bytes.
:func:`unpack_and_dump` Unpack item(s) from bytes and produce packed bytes summary.
:func:`unpack_from` Unpack item(s) from within a bytes buffer.
:func:`unpack_from_and_dump` Unpack item(s) from within a bytes buffer and produce packed bytes summary.
=================================== ================================================================================
.. toctree::
:hidden:
......@@ -73,6 +77,10 @@ Utility Functions
pack_and_dump() <utilities/pack_and_dump.rst>
pack_into() <utilities/pack_into.rst>
pack_into_and_dump() <utilities/pack_into_and_dump.rst>
pack_freeform() <utilities/pack_freeform.rst>
pack_freeform_and_dump() <utilities/pack_freeform_and_dump.rst>
pack_freeform_into() <utilities/pack_freeform_into.rst>
pack_freeform_into_and_dump() <utilities/pack_freeform_into_and_dump.rst>
unpack() <utilities/unpack.rst>
unpack_from() <utilities/unpack_from.rst>
unpack_and_dump() <utilities/unpack_and_dump.rst>
......
############################################
#########################################
[plum] Utility Reference: pack_and_dump()
############################################
#########################################
.. include:: ../../alias.txt
......
#########################################
[plum] Utility Reference: pack_freeform()
#########################################
.. include:: ../../alias.txt
.. currentmodule:: plum
.. autofunction:: pack_freeform
##################################################
[plum] Utility Reference: pack_freeform_and_dump()
##################################################
.. include:: ../../alias.txt
.. currentmodule:: plum
.. autofunction:: pack_freeform_and_dump
##############################################
[plum] Utility Reference: pack_freeform_into()
##############################################
.. include:: ../../alias.txt
.. currentmodule:: plum
.. autofunction:: pack_freeform_into
#######################################################
[plum] Utility Reference: pack_freeform_into_and_dump()
#######################################################
.. include:: ../../alias.txt
.. currentmodule:: plum
.. autofunction:: pack_freeform_into_and_dump
......@@ -8,6 +8,11 @@ Versions increment per `semver <http://semver.org/>`_ (except for alpha series).
Alpha (0.1.0aX) Versions (only serial number X increments)
**********************************************************
+ 0.1.0a9 (2019-Dec-23)
- change pack functions to mirror unpack with accepting format
- provide "freeform" alternative pack functions (previous pack function API)
- boost working (but doesn't help formatted packing)
+ 0.1.0a8 (2019-Dec-12)
- Add view support.
- Integer
......
......@@ -25,9 +25,9 @@ examples apply to all |plum| types or derivatives of them.
>>>
*********************
Packing a Single Item
*********************
************
Pack Methods
************
**[Plum Type Instance Method]**
......@@ -36,11 +36,11 @@ of |plum| types (e.g. :class:`Sample` from above) offer a :meth:`pack` method
that packs the instance and returns the resulting :class:`bytearray`.
For example:
>>> # off the shelf plum type packed into a bytearray
>>> # pack an off the shelf plum type instance into a bytearray
>>> UInt16(1).pack()
bytearray(b'\x00\x01')
>>>
>>> # derivative plum type packed into a bytearray
>>> # pack a derivative plum type instance into a bytearray
>>> Sample(m1=1, m2=2).pack()
bytearray(b'\x00\x01\x02')
......@@ -59,173 +59,144 @@ the plum type, call the :meth:`pack` method with the value to be packed:
bytearray(b'\x00\x01\x02')
**[Utility Function]**
*********************
Pack Utility Function
*********************
The :func:`pack` utility function also accepts |plum| types and values to be
packed and returns the resulting :class:`bytearray`:
The :func:`pack` utility function mirrors the standard library
:func:`struct.pack` function except instead of accepting a format
string, it accepts |plum| types:
>>> from plum import pack
>>> from plum.int.big import UInt16, UInt8
>>>
>>> # pass plum instance
>>> pack(Sample(m1=1, m2=2))
bytearray(b'\x00\x01\x02')
>>> # format as single plum type accepts a value as positional argument
>>> fmt = UInt8
>>> pack(fmt, 1)
bytearray(b'\x01')
>>>
>>> # pass plum type and value separately
>>> pack(Sample, {'m1': 1, 'm2': 2})
bytearray(b'\x00\x01\x02')
>>> # format as tuple/list of types accepts values as positional arguments
>>> fmt = (UInt8, UInt16)
>>> pack(fmt, 1, 2)
bytearray(b'\x01\x00\x02')
>>>
>>> # format as dictionary of types accepts values as keyword arguments
>>> fmt = {'a': UInt8, 'b': UInt16}
>>> pack(fmt, a=1, b=2)
bytearray(b'\x01\x00\x02')
**********************
Packing Multiple Items
**********************
The :func:`pack` function supports packing arbitrarily nested values within
by specifying the structure using tuples/lists and dictionaries within the
format. For exammple:
**[Unnamed Items]**
>>> fmt = {'a': (UInt8, UInt8), 'b': {'i': UInt8, 'ii': UInt8}}
>>>
>>> pack(fmt, a=(1, 2), b={'i': 3, 'ii': 4})
bytearray(b'\x01\x02\x03\x04')
The :func:`pack` utility function accepts multiple items to pack as
positional arguments:
>>> from plum import pack
>>>
>>> # pass plum instances
>>> pack(UInt8(0xaa), Sample(m1=1, m2=2), UInt8(0xff))
bytearray(b'\xaa\x00\x01\x02\xff')
>>>
>>> # pass plum type and values separately
>>> pack(UInt8, 0xaa, Sample, {'m1': 1, 'm2': 2}, UInt8, 0xff)
bytearray(b'\xaa\x00\x01\x02\xff')
*******************************
Free Form Pack Utility Function
*******************************
The :func:`pack_freeform` function facilitates packing multiple items without
an up front format. Instead, it accepts |plum| type class and value pairs either
in serialized or grouped form:
>>> from plum import pack_freeform
>>>
>>> # pass plum type and value pairs
>>> pack(
... (UInt8, 0xaa),
... (Sample, {'m1': 1, 'm2': 2}),
... (UInt8, 0xff))
bytearray(b'\xaa\x00\x01\x02\xff')
>>> # serialized class/value pairs
>>> pack_freeform(UInt8, 1, UInt16, 2)
bytearray(b'\x01\x00\x02')
>>>
>>> # pass any combination of the three
>>> pack(
... UInt8, 0xaa,
... Sample(m1=1, m2=2),
... (UInt8, 0xff))
bytearray(b'\xaa\x00\x01\x02\xff')
>>> # grouped class/value pairs
>>> pack_freeform(
... (UInt8, 1),
... (UInt16, 2))
bytearray(b'\x01\x00\x02')
**[Named Items]**
:func:`pack_freeform` accepts |plum| type instances instead of
or in any combination with class/value pairs:
Alternatively (or in addition to unnamed items), assign names to the items
to be packed by passing types and values as keyword arguments:
>>> pack_freeform(
... UInt8, 1, # serialized
... UInt8(2), # instance
... (UInt8, 3) # grouped
... )
bytearray(b'\x01\x02\x03')
>>> # pass plum instances
>>> pack(first=UInt8(0xaa), middle=Sample(m1=1, m2=2), last=UInt8(0xff))
bytearray(b'\xaa\x00\x01\x02\xff')
>>>
>>> # pass plum type and value pairs
>>> pack(
... first=(UInt8, 0xaa),
... middle=(Sample, {'m1': 1, 'm2': 2}),
... last=(UInt8, 0xff))
bytearray(b'\xaa\x00\x01\x02\xff')
.. Tip::
Passing class and value arguments in serialized form avoids an extra
object instantiation and slightly increases packing performance.
The :func:`pack_freeform` also accepts |plum| type class/pairs or instances
as keyword arguments.
>>> pack_freeform(a=(UInt8, 0), b=UInt8(1))
bytearray(b'\x00\x01')
The keyword argument name can be used to improve your code readability but
also helps readability of byte summary dumps resulting from a packing
exception or resulting from using the "add dump" alternative form:
>>> from plum import pack_freeform_and_dump
>>>
>>> # pass any combination of the two
>>> pack(
... first=UInt8(0xaa),
... middle=Sample(m1=1, m2=2),
... last=(UInt8, 0xff))
bytearray(b'\xaa\x00\x01\x02\xff')
Naming items to be packed enhances resulting byte summary dumps in the
byte summary dump included in packing exceptions (and enhances byte
summary dumps produced by the :func:`pack_and_dump` alternative covered
in a later section). For example:
>>> pack(
... first=UInt8(0xaa),
... middle=Sample(m1=1, m2=2),
... last=(int, 0xff))
Traceback (most recent call last):
...
plum._exceptions.PackError:
<BLANKLINE>
+--------+-------------+-------+-------+---------------+
| Offset | Access | Value | Bytes | Type |
+--------+-------------+-------+-------+---------------+
| 0 | first | 170 | aa | UInt8 |
+--------+-------------+-------+-------+---------------+
| | middle | | | Sample |
| 1 | [0] (.m1) | 1 | 00 01 | UInt16 |
| 3 | [1] (.m2) | 2 | 02 | UInt8 |
+--------+-------------+-------+-------+---------------+
| | last | 255 | | int (invalid) |
+--------+-------------+-------+-------+---------------+
<BLANKLINE>
TypeError occurred during pack operation:
<BLANKLINE>
invalid plum type
>>> buffer, dump = pack_freeform_and_dump(a=(UInt8, 0), b=UInt8(1))
>>> buffer
bytearray(b'\x00\x01')
>>> print(dump)
+--------+--------+-------+-------+-------+
| Offset | Access | Value | Bytes | Type |
+--------+--------+-------+-------+-------+
| 0 | a | 0 | 00 | UInt8 |
+--------+--------+-------+-------+-------+
| 1 | b | 1 | 01 | UInt8 |
+--------+--------+-------+-------+-------+
**************************************
Packing Directly Within a Bytes Buffer
**************************************
The :func:`pack_into` utility function and the :meth:`pack_into` |plum|
type methods have the same packing behaviors discussed in the previous
sections except the functions take required buffer and offset
arguments. The buffer may be any writeable bytes-like buffer. The
offset argument specifies the number of bytes from the beginning of
the buffer to start placing packed item bytes.
**[Single Item]**
The :meth:`pack_into` |plum| type method as well as the :func:`pack_into`
and :func:`pack_freeform_into` utility functions have the same packing
behaviors discussed in the previous sections except the functions take
required buffer and offset arguments. The buffer may be any writeable
bytes-like buffer. The offset argument specifies the number of bytes from
the beginning of the buffer to start placing packed item bytes.
>>> from plum import pack_into
>>> from plum import pack_freeform_into, pack_into
>>> from plum.int.big import UInt8
>>>
>>> OFFSET = 1
>>>
>>> buffer = bytearray(3)
>>> buffer
bytearray(b'\x00\x00\x00')
>>>
>>> # Plum Type Instance Method
>>> UInt8(0xaa).pack_into(buffer, 1)
>>> UInt8(0xaa).pack_into(buffer, OFFSET)
>>> buffer
bytearray(b'\x00\xaa\x00')
>>>
>>> # Plum Type class Method
>>> UInt8.pack_into(buffer, 1, 0xbb)
>>> UInt8.pack_into(buffer, OFFSET, 0xbb)
>>> buffer
bytearray(b'\x00\xbb\x00')
>>>
>>> # utility function (separate class/value)
>>> pack_into(buffer, 1, UInt8, 0xcc)
>>> buffer
bytearray(b'\x00\xcc\x00')
>>> # utility function (instance)
>>> pack_into(buffer, 1, UInt8(0xdd))
>>> buffer
bytearray(b'\x00\xdd\x00')
**Multiple Items**
>>> buffer = bytearray(4)
>>> buffer
bytearray(b'\x00\x00\x00\x00')
>>>
>>> # utility function (plum instances)
>>> pack_into(buffer, 1, UInt8(0xaa), UInt8(0xbb))
>>> buffer
bytearray(b'\x00\xaa\xbb\x00')
>>>
>>> # utility function (separate class/value)
>>> pack_into(buffer, 1, UInt8, 0xcc, UInt8, 0xdd)
>>> # formatted utility function
>>> fmt = (UInt8, UInt8)
>>> pack_into(fmt, buffer, OFFSET, 0xcc, 0xdd)
>>> buffer
bytearray(b'\x00\xcc\xdd\x00')
bytearray(b'\x00\xcc\xdd')
>>>
>>> # utility function (class/value pairs)
>>> pack_into(
... buffer, 1,
... (UInt8, 0xee),
... (UInt8, 0xff))
>>> # free form utility function
>>> pack_freeform_into(buffer, OFFSET, UInt8, 0xee, UInt8, 0xff)
>>> buffer
bytearray(b'\x00\xee\xff\x00')
bytearray(b'\x00\xee\xff')
**Caveats**
......@@ -237,11 +208,12 @@ an exception:
>>> buffer
bytearray(b'\x00\x00')
>>>
>>> pack_into(buffer, 3, UInt8(0x99))
>>> pack_freeform_into(buffer, 3, UInt8(0x99))
Traceback (most recent call last):
...
plum._exceptions.PackError: offset 3 out of range for 2-byte buffer
Specifying an offset within the limits of the byte buffer but writing an
item that is larger than what fits results in the byte buffer being
extended:
......@@ -250,7 +222,7 @@ extended:
>>> buffer
bytearray(b'\x00\x00')
>>>
>>> pack_into(buffer, 1, UInt16(0xaabb))
>>> pack_freeform_into(buffer, 1, UInt16(0xaabb))
>>> buffer
bytearray(b'\x00\xaa\xbb')
......@@ -261,44 +233,23 @@ Obtaining Bytes Summary Dump
For every pack method/function shown in the previous sections, an "and_dump"
variation exists to provide a bytes summary dump in addition to the packed
item bytes. The following example shows the utility function variations. But
variations exist for all the |plum| type pack methods.
item bytes. The following example shows one variation. But variations exist
for all the |plum| type pack functions/methods.
>>> from plum import pack_and_dump, pack_into_and_dump
>>> from plum.int.big import UInt8
>>> from plum import pack_and_dump
>>>
>>> # positional items (unnamed)
>>> buffer, dump = pack_and_dump(UInt8(0xaa), Sample(m1=1, m2=2))
>>> fmt = {'first': UInt8, 'last': Sample}
>>> buffer, dump = pack_and_dump(fmt, first=0xaa, last=Sample(m1=1, m2=2))
>>> buffer
bytearray(b'\xaa\x00\x01\x02')
>>> print(dump) # notice no names in access column
+--------+-----------+-------+-------+--------+
| Offset | Access | Value | Bytes | Type |
+--------+-----------+-------+-------+--------+
| 0 | | 170 | aa | UInt8 |
+--------+-----------+-------+-------+--------+
| | | | | Sample |
| 1 | [0] (.m1) | 1 | 00 01 | UInt16 |
| 3 | [1] (.m2) | 2 | 02 | UInt8 |
+--------+-----------+-------+-------+--------+
>>>
>>> buffer = bytearray(6)
>>> buffer
bytearray(b'\x00\x00\x00\x00\x00\x00')
>>>
>>> # key-worded items (named)
>>> dump = pack_into_and_dump(buffer, 1, first=UInt8(0xaa), last=Sample(m1=1, m2=2))
>>> buffer
bytearray(b'\x00\xaa\x00\x01\x02\x00')
>>> print(dump) # notice names in access column and offset reflects placement in buffer
+--------+-------------+-------+-------+--------+
| Offset | Access | Value | Bytes | Type |
+--------+-------------+-------+-------+--------+
| 1 | first | 170 | aa | UInt8 |
+--------+-------------+-------+-------+--------+
| | last | | | Sample |
| 2 | [0] (.m1) | 1 | 00 01 | UInt16 |
| 4 | [1] (.m2) | 2 | 02 | UInt8 |
| 0 | ['first'] | 170 | aa | UInt8 |
| | ['last'] | | | Sample |
| 1 | [0] (.m1) | 1 | 00 01 | UInt16 |
| 3 | [1] (.m2) | 2 | 02 | UInt8 |
+--------+-------------+-------+-------+--------+
......