Modules¶
nistitl¶
This module defines classes used to parse and generate NIST-ITL binary files.
The creation of a NIST file is done first with the creation of a message:
>>> msg = Message()
The message automatically contains a type 1 record. It can be accessed using direct accessors. For instance:
>>> msg.TOT = 'TOTFORTEST'
>>> print(msg.TOT)
TOTFORTEST
Note that some fields can be accessed only in read-only mode:
>>> print(msg.CNT.NIST == b'1.003:1\x1f0')
True
>>> msg.CNT = 12 #doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
AttributeError: can't set attribute
Record are created and then added to the message:
>>> r2 = AsciiRecord(2)
>>> r2.IDC = 1
>>> msg += r2
Many fields (as per the NIST-ITL specifications) are declared automatically. Other fields are created individually by specifying the record type, the field number, a value, a type and a format.
>>> f12 = Field(2, 12, 'TEST12', alias='f12')
>>> f12.add_subfields( SubField('TEST12-SF1') )
>>> f12.add_subfields( SubField('TEST12-SF2') )
>>> r2 = r2 + f12
>>> print(f12.NIST == b'2.012:TEST12-SF1\x1eTEST12-SF2')
True
The alias can be used to access the field in the record. This makes the code easier to read.
>>> print(r2.f12)
['TEST12-SF1', 'TEST12-SF2']
Adding sub fields has deleted the value of the field itself:
>>> print(r2['f12']._value)
<BLANKLINE>
Or to change a value
>>> msg[0].DAT = '20090924'
>>> msg[0].TCN = '12345'
And then the full message can be generated
>>> print(msg.NIST == b'1.001:123\x1d1.002:0400\x1d1.003:1\x1f1\x1e2\x1f1\x1d1.004:TOTFORTEST\x1d1.005:20090924\x1d1.007:000\x1d1.008:000\x1d1.009:12345\x1d1.011:00.00\x1d1.012:00.00\x1c2.001:45\x1d2.002:1\x1d2.012:TEST12-SF1\x1eTEST12-SF2\x1c')
True
Two records of the same type can be created in the message. Record are created and then added to the message:
>>> r2 = AsciiRecord(2)
>>> r2.IDC = 2
>>> msg += r2
>>> f12 = Field(2, 12, 'TEST12', alias='f12')
>>> r2 += f12
>>> s =msg.NIST
>>> print(s == b'1.001:127\x1d1.002:0400\x1d1.003:1\x1f2\x1e2\x1f1\x1e2\x1f2\x1d1.004:TOTFORTEST\x1d1.005:20090924\x1d1.007:000\x1d1.008:000\x1d1.009:12345\x1d1.011:00.00\x1d1.012:00.00\x1c2.001:45\x1d2.002:1\x1d2.012:TEST12-SF1\x1eTEST12-SF2\x1c2.001:30\x1d2.002:2\x1d2.012:TEST12\x1c')
True
A NIST buffer can be parsed easily with:
>>> msg = Message()
>>> msg.parse(s)
>>> f12 = msg[(2,'1')][12]
>>> print(f12.NIST == b'2.012:TEST12-SF1\x1eTEST12-SF2')
True
>>> print(f12[0].value)
TEST12-SF1
>>> print(f12[1].value)
TEST12-SF2
>>> f12 = msg[(2,'2')][12]
>>> print(f12.NIST == b'2.012:TEST12')
True
Record can be directy accessed through their position (index) in the full message:
>>> print(msg[0]['CNT'][0].values == ['1', '2'])
True
>>> print(msg[0]['CNT'][1].values == ['2', '1'])
True
-
class
nistitl.
AsciiRecord
(type, autocreate=True, autosort=True)¶ Bases:
object
An ASCII record. ASCII records, or text record, contain text data encoded in latin-1 organized in fields, subfields and items. They can also contain a final field with binary data.
When creating an AsciiRecord you can specify the option
autosort
toTrue
(the default) or toFalse
. When true, the fields will be sorted by numeric order, except forBinaryField
that will remain at the end.-
NIST
¶ Calculation of the NIST binary representation of this record. A bytes object is returned.
>>> r = AsciiRecord(10) >>> r.IDC = 2 >>> r._999 = b'my image data' >>> print(r.NIST) b'10.001:40\x1d10.002:2\x1d10.999:my image data\x1c'
>>> r = AsciiRecord(1, autosort=True) >>> r.TCN = 'TCN' >>> r.TCR = 'TCR' >>> r.DAT = '20200904' >>> print(r.NIST) b'1.001:114\x1d1.002:0400\x1d1.003:\x1d1.004:\x1d1.005:20200904\x1d1.007:000\x1d1.008:000\x1d1.009:TCN\x1d1.010:TCR\x1d1.011:00.00\x1d1.012:00.00\x1c'
>>> r = AsciiRecord(1, autosort=False) >>> r.TCN = 'TCN' >>> r.TCR = 'TCR' >>> r.DAT = '20200904' >>> print(r.NIST) b'1.001:114\x1d1.002:0400\x1d1.003:\x1d1.004:\x1d1.005:20200904\x1d1.007:000\x1d1.008:000\x1d1.009:TCN\x1d1.011:00.00\x1d1.012:00.00\x1d1.010:TCR\x1c'
-
__add__
(f)¶ Add the field f to the record after some consistency checks.
>>> r = AsciiRecord(2) >>> r = r + Field(2, 3, 'OK') >>> print(str(r)) 2.001: LEN : 0 2.002: IDC : 0 2.003: : OK
-
__delitem__
(tag)¶ Remove the field identified by tag from the record. If there is not field for this tag, no exception is raised.
>>> r = AsciiRecord(2) >>> r += Field(2, 3, 'OK') >>> print(str(r)) 2.001: LEN : 0 2.002: IDC : 0 2.003: : OK
>>> del r[3] >>> print(str(r)) 2.001: LEN : 0 2.002: IDC : 0
>>> del r[4]
-
__getattr__
(tag)¶ Fields value can be accessed directly using one the following syntax:
- record._number: get the value for one tag using the number of the tag
- record.alias get the value for one alias. The field must already exist
Build a record with some fields:
>>> r = AsciiRecord(10) >>> r.IDC = 4 # known >>> r.SRC = 'my src' # new one, predeclared >>> r._7 = 'my vll' # by the tag number
Access the values
>>> r.IDC 4 >>> r._4 'my src'
>>> r.UUU # unknown alias Traceback (most recent call last): ... nistitl.NistException: UNKNOWN_ATTRIBUTE: Unkown or bad attribute <UUU> while trying to retrieve a field in record of type 10
>>> r._76a # not an integer Traceback (most recent call last): ... nistitl.NistException: UNKNOWN_ATTRIBUTE: Unkown or bad attribute <_76a> while trying to retrieve a field in record of type 10
-
__getitem__
(tag)¶ Get a field from this record. The field is identified by tag, a number or an alias. If there is no such field,
None
is returned.>>> r = AsciiRecord(2) >>> r += Field(2, 3, 'OK', alias='test') >>> r[3].value 'OK' >>> r['test'].value 'OK' >>> print(r[4]) None
-
__iadd__
(f)¶ Add the field f to the record after some consistency checks.
>>> r = AsciiRecord(2) >>> r += Field(2, 3, 'OK') >>> print(str(r)) 2.001: LEN : 0 2.002: IDC : 0 2.003: : OK
-
__init__
(type, autocreate=True, autosort=True)¶ Initialize self. See help(type(self)) for accurate signature.
-
__setattr__
(tag, v)¶ Fields can be set directly using one the following syntax:
- record._number: set the value for one tag using the number of the tag
- record.alias set the value for one alias. The field must already exist or the alias must be a standard pre-defined alias (not all of them are defined in this library).
>>> r = AsciiRecord(10) >>> r.IDC = 4 # known >>> r._2 = 5 # known, by the tag number >>> r.SRC = 'my src' # new one, predeclared >>> r._7 = 'my vll' # by the tag number >>> r._345 = 'no alias' >>> print(str(r)) 10.001: LEN : 0 10.002: IDC : 5 10.004: SRC : my src 10.007: VLL : my vll 10.345: : no alias
Other usage will generate exceptions
>>> r.UUU = 'unknown alias' Traceback (most recent call last): ... nistitl.NistException: UNKNOWN_ATTRIBUTE: Bad attribute name <UUU> while trying to define a field in record of type 10
>>> r._76a = 'not an integer' Traceback (most recent call last): ... nistitl.NistException: UNKNOWN_ATTRIBUTE: Bad attribute name <_76a> while trying to define a field in record of type 10
It is possible to set the 999 tag with a bytes object:
>>> r.DATA = b'data' >>> print(str(r)) 10.001: LEN : 0 10.002: IDC : 5 10.004: SRC : my src 10.007: VLL : my vll 10.345: : no alias 10.999: DATA : <buffer, size=4>
-
__str__
()¶ Build and return a summary of the record. It can be useful to document the content of a NIST.
>>> r = AsciiRecord(14) >>> r += Field(14, 3, 'OK') >>> r += BinaryField(14, 999, b'data') >>> print(str(r)) 14.001: LEN : 0 14.002: IDC : 0 14.003: IMP : OK 14.999: DATA : <buffer, size=4>
-
-
class
nistitl.
BinaryField
(record, tag, value=b'', format='%d.%03d:', alias='')¶ Bases:
object
A binary field used as the last field in an
AsciiRecord
record.-
NIST
¶ Calculation of the NIST binary representation of this field. A bytes object is returned.
-
__init__
(record, tag, value=b'', format='%d.%03d:', alias='')¶ Creation of a new binary field.
- record indicates the record’s type and must be an integer.
- tag is the tag number and will be converted to an integer.
- value the initial value. Must be a bytes object.
- format is the format used to generate the tag number in the NIST.
- alias is a name used as a shortcut when accessing the value.
-
value
¶ Access to the value of the field. The value is a bytes object.
-
-
class
nistitl.
BinaryRecord
(type)¶ Bases:
object
A binary record. Binary records are fully binary and are not parsed at all. Only the
IDC
is extracted.>>> r = BinaryRecord(4) >>> r.IDC = 1 >>> r.value = b'data' >>> print(r.NIST) b'\x00\x00\x00\t\x01data'
>>> print(str(r)) 4.001: LEN : 9 4.002: IDC : 1 4.---: : <buffer, size=4>
-
IDC
¶ Shortcut to access the IDC of the record.
-
NIST
¶ Calculation of the NIST binary representation of this record. A bytes object is returned.
-
__init__
(type)¶ Initialize self. See help(type(self)) for accurate signature.
-
__str__
()¶ Return str(self).
-
length
¶ Access to the record length (in bytes)
-
pack
(format, *args)¶ A direct wrapper on struct.pack. If successful, the format is kept in the object and used in __str__ to improve the output.
There is always an additional argument corresponding to additional data added to the value (usually an image data).
>>> r = BinaryRecord(4) >>> r.IDC = 1 >>> r.pack("!LHB", 3, 2, 1, b'my image data') >>> print(r.NIST) b'\x00\x00\x00\x19\x01\x00\x00\x00\x03\x00\x02\x01my image data'
>>> print(str(r)) 4.001: LEN : 25 4.002: IDC : 1 4.---: : <buffer, size=20> -> 3 -> 2 -> 1
-
unpack
(format)¶ Unpack the value of the record (without the length and the IDC) and return the value described in the format and the remaining buffer (usually an image data)
If successful, the format is kept in the object and used in __str__ to improve the output.
>>> r = BinaryRecord(4) >>> r.IDC = 1 >>> r.pack("!LHB", 3, 2, 1, b'my image data') >>> r.unpack("!BHL") # format changed for the test! (0, 0, 50332161, b'my image data')
>>> print(str(r)) 4.001: LEN : 25 4.002: IDC : 1 4.---: : <buffer, size=20> -> 0 -> 0 -> 50332161
-
-
class
nistitl.
Field
(record, tag, value='', type='FSI', format='%d.%03d:', alias='')¶ Bases:
object
A class representing a field added to an
AsciiRecord
.A field can be configured to accept only a single value (no subfields, no items)
>>> f = Field(2, 3, type="F") >>> f.value = 'ok' >>> f.value = [1, 2] Traceback (most recent call last): ... nistitl.NistException: BAD_FIELD_VALUE: Field 2.003: cannot have subfields >>> f.value = [[1, 2]] Traceback (most recent call last): ... nistitl.NistException: BAD_FIELD_VALUE: Field 2.003: cannot have subfields
Or only a subfield:
>>> f = Field(2, 3, type="S") >>> f.value = [1, 2] >>> f.value = 'ok' Traceback (most recent call last): ... nistitl.NistException: BAD_FIELD_VALUE: Field 2.003: cannot have a value (only subfields and/or items) >>> f.value = [[1, 2]] Traceback (most recent call last): ... nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield cannot have items
Or only subfields with items:
>>> f = Field(2, 3, type="I") >>> f.value = [[1, 2]] >>> f.value = [1, 2] Traceback (most recent call last): ... nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield cannot have a value, only items >>> f.value = 'ok' Traceback (most recent call last): ... nistitl.NistException: BAD_FIELD_VALUE: Field 2.003: cannot have a value (only subfields and/or items)
Or subfields can be forbidden unless they have items:
>>> f = Field(2, 3, type="FI") >>> f.add_subfields( SubField('a', type="SI") ) Traceback (most recent call last): ... nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield of 2.003: cannot have a value: 'a'
Or a subfield with items can be forbidden:
>>> f = Field(2, 3, type="SF") >>> f.add_subfields( SubField(['a', 'b'], type="SI") ) Traceback (most recent call last): ... nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield of 2.003: cannot have items
-
NIST
¶ Calculation of the NIST binary representation of this field. A bytes object is returned.
If the value of the field is None, it is interpreted as an empty field.
>>> f = Field(2, 2, None) >>> f.NIST b'2.002:'
-
__getitem__
(idx)¶ Access the idx-th sub field in the field.
>>> f = Field(2, 3, ['one', 'two']) >>> print(f[0].value) one >>> print(f[1].value) two
-
__init__
(record, tag, value='', type='FSI', format='%d.%03d:', alias='')¶ Creation of a new field.
record indicates the record’s type and must be an integer.
tag is the tag number and will be converted to an integer.
value the initial value. Must be a string or a list.
type indicates the possible type for the field content. It can be a combination of:
- ‘F’: a single field, i.e. a simple value
- ‘S’: this field can contain sub fields
- ‘I’: this field can contain items
format is the format used to generate the tag number in the NIST.
alias is a name used as a shortcut when accessing the value.
-
__len__
()¶ Return the number of subfields in the field.
>>> f = Field(2, 3, ['one', 'two']) >>> print(len(f)) 2 >>> f = Field(2, 21, "hello") >>> print(len(f)) 0
-
add_subfields
(*sf)¶ Add a set of subfields to this field. Some consistency checks are run.
-
reset
()¶ Reset this field value and the list of subfields.
-
value
¶ Access to the value of the field. The value is a string or a list of subfield values.
-
-
class
nistitl.
Message
¶ Bases:
object
The Message class is the main entry point. It is used to parse an existing NIST file as well as for building a new one.
-
CNT
¶ Calculation of the
CNT
field of the type-1 record. Returns theField
for theCNT
.>>> msg = Message() >>> r = AsciiRecord(2) >>> r.IDC = 3 >>> msg = msg + r >>> msg.TOT = 'TEST' >>> len(msg) 2 >>> msg.CNT.NIST b'1.003:1\x1f1\x1e2\x1f3'
-
NIST
¶ Calculation of NIST binary content. A bytes is returned containing all the records previously added.
>>> msg = Message() >>> msg.TOT = 'TEST' >>> msg[0].DAT = '20190517' >>> msg[0].TCN = 'doctest' >>> msg.NIST b'1.001:115\x1d1.002:0400\x1d1.003:1\x1f0\x1d1.004:TEST\x1d1.005:20190517\x1d1.007:000\x1d1.008:000\x1d1.009:doctest\x1d1.011:00.00\x1d1.012:00.00\x1c'
-
TOT
¶ Shortcut to the TOT of the message (stored in the type 1 record)
-
__add__
(record)¶ Add a record to the list of records of the message. Example:
>>> msg = Message() >>> r = AsciiRecord(2) >>> msg = msg + r >>> len(msg._records) 2
-
__delitem__
(key)¶ Delete a record. key can be:
- a tuple or list containing type record type (a numeric) and the record IDC (a numeric)
- a single numeric value: the index in the list of records.
>>> msg = Message() >>> r = AsciiRecord(2)
Delete the second record, whatever its type:
>>> msg = msg + r >>> del msg[1] >>> len(msg) 1
Access the first type 2 record with IDC = 0:
>>> msg = msg + r >>> del msg[(2, 0)] >>> len(msg) 1
Type 1 record cannot be deleted (create a new message and move or copy the records if you have a need for this):
>>> msg = msg + r >>> del msg[(1, 0)] Traceback (most recent call last): ... nistitl.NistException: CANNOT_DELETE_TYPE1: Cannot delete the type 1 record
>>> del msg[0] Traceback (most recent call last): ... nistitl.NistException: CANNOT_DELETE_TYPE1: Cannot delete the type 1 record
-
__getitem__
(key)¶ Return the record identified by key. key can be:
- a tuple or list with two integers: the record type and the record IDC
- a single numeric value: the index in the list of records.
>>> msg = Message() >>> r = AsciiRecord(2) >>> msg = msg + r
Access the second record, whatever its type:
>>> print(msg[1].type) 2
Access the first type 2 record with IDC = 0:
>>> print(msg[(2, 0)].type) 2
Since type 1 has no IDC, it can only be accessed with an index of 0
>>> print(msg[0].type) 1
If there is no record for the key, an exception is raised:
>>> msg[100] Traceback (most recent call last): ... IndexError: list index out of range
>>> msg[(2, 100)] Traceback (most recent call last): ... IndexError: list index out of range
-
__iadd__
(record)¶ Add a record to the list of records of the message. Example:
>>> msg = Message() >>> r = AsciiRecord(2) >>> msg += r >>> len(msg._records) 2
-
__init__
()¶ Initialize self. See help(type(self)) for accurate signature.
-
__isub__
(record)¶ Remove one specific record from the message.
>>> msg = Message() >>> r = AsciiRecord(2) >>> msg = msg + r >>> len(msg) 2 >>> msg -= r >>> len(msg) 1
>>> msg -= r Traceback (most recent call last): ... nistitl.NistException: RECORD_NOT_FOUND: Cannot remove a record: record not found
-
__iter__
()¶ Iterate over all the records.
>>> msg = Message() >>> r = AsciiRecord(2) >>> msg = msg + r >>> for record in msg: ... print(record.type) 1 2
-
__len__
()¶ Return the number of records in the message.
>>> msg = Message() >>> print(len(msg)) 1
-
__str__
()¶ Build and return a summary of the NIST content. It can be useful to document the content of a NIST.
>>> msg = Message() >>> r = AsciiRecord(2) >>> r.IDC = 3 >>> msg = msg + r >>> msg.TOT = 'TEST' >>> msg[0].DAT = '20190517' >>> msg[0].TCN = 'doctest' >>> msg[0].TCR = 'TCR' >>> print(str(msg)) 1.001: LEN : 0 1.002: VER : 0400 1.003: CNT : [[1, 1], [2, 3]] 1.004: TOT : TEST 1.005: DAT : 20190517 1.007: DAI : 000 1.008: ORI : 000 1.009: TCN : doctest 1.010: TCR : TCR 1.011: NSR : 00.00 1.012: NTR : 00.00 2.001: LEN : 0 2.002: IDC : 3
-
__sub__
(record)¶ Remove one specific record from the message.
>>> msg = Message() >>> r = AsciiRecord(2) >>> msg = msg + r >>> len(msg) 2 >>> msg = msg - r >>> len(msg) 1
>>> msg = msg - r Traceback (most recent call last): ... nistitl.NistException: RECORD_NOT_FOUND: Cannot remove a record: record not found
-
iter
(type)¶ Return an iterator on all the records of type type.
>>> msg = Message() >>> r = AsciiRecord(2) >>> msg = msg + r >>> for record in msg.iter(2): ... print(record.type) 2
-
parse
(buffer)¶ Parse a NIST buffer and initialize the list of records corresponding to the content. buffer must be a bytes object, not a string.
>>> msg = Message() >>> r = AsciiRecord(2) >>> r.IDC = 3 >>> msg = msg + r >>> msg.TOT = 'TEST' >>> msg[0].DAT = '20190517' >>> msg[0].TCN = 'doctest'
>>> new_msg = Message() >>> new_msg.parse( msg.NIST ) >>> len(new_msg) 2
If the buffer is truncated, an exception is raised:
>>> new_msg = Message() >>> new_msg.parse( msg.NIST[:-1] ) Traceback (most recent call last): ... nistitl.NistException: NIST_TOO_SHORT: NIST buffer too short (missing bytes) when parsing record 2
If there are extra bytes to the buffer, this may indicate a binary record is truncated:
>>> new_msg = Message() >>> new_msg.parse( msg.NIST + b'x12345' ) Traceback (most recent call last): ... nistitl.NistException: BAD_CONTENT: Could not recognize binary record, bad content or bad record
This works also with type 14 records that end with a binary buffer:
>>> msg = Message() >>> msg.TOT = 'TEST' >>> msg[0].DAT = '20190517' >>> msg[0].TCN = 'doctest' >>> r = AsciiRecord(14) >>> r += Field(14, 3, 'OK') >>> r += BinaryField(14, 999, b'data') >>> msg = msg + r
>>> new_msg = Message() >>> new_msg.parse( msg.NIST[:-1]) Traceback (most recent call last): ... nistitl.NistException: NIST_TOO_SHORT: NIST buffer too short (missing bytes) when parsing record 14
>>> new_msg = Message() >>> new_msg.parse( msg.NIST + b'x') Traceback (most recent call last): ... nistitl.NistException: NIST_TOO_LONG: NIST buffer too long (extra bytes)
-
reset
(autocreate=True, autosort=True)¶ Reset the record list, leaving only a blank new type-1 record. if autocreate is True, the type-1 record is initialized. If autosort is True, the tags are sorted by numeric order.
-
-
class
nistitl.
NistError
¶ Bases:
enum.Enum
An enumeration.
-
exception
nistitl.
NistException
(message, error)¶ Bases:
Exception
Exception class used to report errors generated from this module.
-
__init__
(message, error)¶ Initialize self. See help(type(self)) for accurate signature.
-
__str__
()¶ Return str(self).
-
__weakref__
¶ list of weak references to the object (if defined)
-
-
class
nistitl.
SubField
(value='', type='SI')¶ Bases:
object
A SubField class, representing a subfield added to a
Field
.A subfield can be typed to accept only items:
>>> sf = SubField(type="I") >>> sf.value = ['a', 'b'] >>> sf.value = 'error' Traceback (most recent call last): ... nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield cannot have a value, only items
Or only a single value:
>>> sf = SubField(type="S") >>> sf.value = 'ok' >>> sf.value = ['a', 'b'] Traceback (most recent call last): ... nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield cannot have items
Or both:
>>> sf = SubField(type="SI") >>> sf.value = 'ok' >>> sf.value = ['a', 'b']
-
NIST
¶ Calculation of the NIST binary representation of this subfield. A bytes object is returned.
If the value of the subfield is None, it is interpreted as an empty subfield.
>>> sf = SubField(None) >>> sf.NIST b''
If the value is not a string, it is converted to a string:
>>> sf = SubField(12) >>> sf.NIST b'12'
-
__getitem__
(idx)¶ Return th idx-th item.
>>> sf = SubField(type="SI") >>> sf.value = ['a', 'b'] >>> sf[0] 'a' >>> sf[1] 'b'
-
__init__
(value='', type='SI')¶ Creation of a new subfield.
value the initial value. Must be a string or a list of strings.
type indicates the possible type for the field content. It can be a combination of:
- ‘S’: a single subfield, i.e. a simple value
- ‘I’: this subfield can contain items
-
__len__
()¶ Return the number of items in the subfield.
>>> f = SubField(type="I", value=['one', 'two']) >>> print(len(f)) 2
-
add_values
(*i)¶ Add multiple values as items of this subfield.
-
value
¶ Access to the value of the subfield. The value is a string or a list of strings.
-
-
nistitl.
parse_record
(buffer, push_record, push_field, push_subfield, push_value)¶ Parse a single “text” record