Modules

nistitl

This module defines classes used to parse and generate NIST-ITL binary files.

The creation of a NIST file is done first with the creation of a message:

>>> msg = Message()

The message automatically contains a type 1 record. It can be accessed using direct accessors. For instance:

>>> msg.TOT = 'TOTFORTEST'
>>> print(msg.TOT)
TOTFORTEST

Note that some fields can be accessed only in read-only mode:

>>> print(msg.CNT.NIST == b'1.003:1\x1f0')
True
>>> msg.CNT = 12
Traceback (most recent call last):
    ...
AttributeError: can't set attribute

Record are created and then added to the message:

>>> r2 = AsciiRecord(2)
>>> r2.IDC = 1
>>> msg += r2

Many fields (as per the NIST-ITL specifications) are declared automatically. Other fields are created individually by specifying the record type, the field number, a value, a type and a format.

>>> f12 = Field(2, 12, 'TEST12', alias='f12')
>>> f12.add_subfields( SubField('TEST12-SF1') )
>>> f12.add_subfields( SubField('TEST12-SF2') )
>>> r2 = r2 + f12
>>> print(f12.NIST == b'2.012:TEST12-SF1\x1eTEST12-SF2')
True

The alias can be used to access the field in the record. This makes the code easier to read.

>>> print(r2.f12)
['TEST12-SF1', 'TEST12-SF2']

Adding sub fields has deleted the value of the field itself:

>>> print(r2['f12']._value)

Or to change a value

>>> msg[0].DAT = '20090924'
>>> msg[0].TCN = '12345'

And then the full message can be generated

>>> print(msg.NIST == b'1.001:123\x1d1.002:0400\x1d1.003:1\x1f1\x1e2\x1f1\x1d1.004:TOTFORTEST\x1d1.005:20090924\x1d1.007:000\x1d1.008:000\x1d1.009:12345\x1d1.011:00.00\x1d1.012:00.00\x1c2.001:45\x1d2.002:1\x1d2.012:TEST12-SF1\x1eTEST12-SF2\x1c')
True

Two records of the same type can be created in the message. Record are created and then added to the message:

>>> r2 = AsciiRecord(2)
>>> r2.IDC = 2
>>> msg += r2
>>> f12 = Field(2, 12, 'TEST12', alias='f12')
>>> r2 += f12
>>> s =msg.NIST
>>> print(s == b'1.001:127\x1d1.002:0400\x1d1.003:1\x1f2\x1e2\x1f1\x1e2\x1f2\x1d1.004:TOTFORTEST\x1d1.005:20090924\x1d1.007:000\x1d1.008:000\x1d1.009:12345\x1d1.011:00.00\x1d1.012:00.00\x1c2.001:45\x1d2.002:1\x1d2.012:TEST12-SF1\x1eTEST12-SF2\x1c2.001:30\x1d2.002:2\x1d2.012:TEST12\x1c')
True

A NIST buffer can be parsed easily with:

>>> msg = Message()
>>> msg.parse(s)
>>> f12 = msg[(2,'1')][12]
>>> print(f12.NIST == b'2.012:TEST12-SF1\x1eTEST12-SF2')
True
>>> print(f12[0].value)
TEST12-SF1
>>> print(f12[1].value)
TEST12-SF2
>>> f12 = msg[(2,'2')][12]
>>> print(f12.NIST == b'2.012:TEST12')
True

Record can be directy accessed through their position (index) in the full message:

>>> print(msg[0]['CNT'][0].values == ['1', '2'])
True
>>> print(msg[0]['CNT'][1].values == ['2', '1'])
True

class nistitl.AsciiRecord(type, autocreate=True, autosort=True)

Bases: object

An ASCII record. ASCII records, or text record, contain text data encoded in latin-1 organized in fields, subfields and items. They can also contain a final field with binary data.

When creating an AsciiRecord you can specify the option autosort to True (the default) or to False. When true, the fields will be sorted by numeric order, except for BinaryField that will remain at the end.

property NIST

Calculation of the NIST binary representation of this record. A bytes object is returned.

>>> r = AsciiRecord(10)
>>> r.IDC = 2
>>> r._999 = b'my image data'
>>> print(r.NIST)
b'10.001:40\x1d10.002:2\x1d10.999:my image data\x1c'

>>> r = AsciiRecord(1, autosort=True)
>>> r.TCN = 'TCN'
>>> r.TCR = 'TCR'
>>> r.DAT = '20200904'
>>> print(r.NIST)
b'1.001:114\x1d1.002:0400\x1d1.003:\x1d1.004:\x1d1.005:20200904\x1d1.007:000\x1d1.008:000\x1d1.009:TCN\x1d1.010:TCR\x1d1.011:00.00\x1d1.012:00.00\x1c'

>>> r = AsciiRecord(1, autosort=False)
>>> r.TCN = 'TCN'
>>> r.TCR = 'TCR'
>>> r.DAT = '20200904'
>>> print(r.NIST)
b'1.001:114\x1d1.002:0400\x1d1.003:\x1d1.004:\x1d1.005:20200904\x1d1.007:000\x1d1.008:000\x1d1.009:TCN\x1d1.011:00.00\x1d1.012:00.00\x1d1.010:TCR\x1c'

__add__(f)

Add the field f to the record after some consistency checks.

>>> r = AsciiRecord(2)
>>> r = r + Field(2, 3, 'OK')
>>> print(str(r))
 2.001: LEN                           : 0
 2.002: IDC                           : 0
 2.003:                               : OK

__delitem__(tag)

Remove the field identified by tag from the record. If there is not field for this tag, no exception is raised.

>>> r = AsciiRecord(2)
>>> r += Field(2, 3, 'OK')
>>> print(str(r))
 2.001: LEN                           : 0
 2.002: IDC                           : 0
 2.003:                               : OK

>>> del r[3]
>>> print(str(r))
 2.001: LEN                           : 0
 2.002: IDC                           : 0

>>> del r[4]

__getattr__(tag)

Fields value can be accessed directly using one the following syntax:

• record._number: get the value for one tag using the number of the tag

• record.alias get the value for one alias. The field must already exist

Build a record with some fields:

>>> r = AsciiRecord(10)
>>> r.IDC = 4           # known
>>> r.SRC = 'my src'    # new one, predeclared
>>> r._7 = 'my vll'     # by the tag number

Access the values

>>> r.IDC
4
>>> r._4
'my src'

>>> r.UUU       # unknown alias
Traceback (most recent call last):
...
nistitl.NistException: UNKNOWN_ATTRIBUTE: Unkown or bad attribute <UUU> while trying to retrieve a field in record of type 10

>>> r._76a      # not an integer
Traceback (most recent call last):
...
nistitl.NistException: UNKNOWN_ATTRIBUTE: Unkown or bad attribute <_76a> while trying to retrieve a field in record of type 10

__getitem__(tag)

Get a field from this record. The field is identified by tag, a number or an alias. If there is no such field, None is returned.

>>> r = AsciiRecord(2)
>>> r += Field(2, 3, 'OK', alias='test')
>>> r[3].value
'OK'
>>> r['test'].value
'OK'
>>> print(r[4])
None

__iadd__(f)

Add the field f to the record after some consistency checks.

>>> r = AsciiRecord(2)
>>> r += Field(2, 3, 'OK')
>>> print(str(r))
 2.001: LEN                           : 0
 2.002: IDC                           : 0
 2.003:                               : OK

__init__(type, autocreate=True, autosort=True)

__setattr__(tag, v)

Fields can be set directly using one the following syntax:

• record._number: set the value for one tag using the number of the tag

• record.alias set the value for one alias. The field must already exist or the alias must be a standard pre-defined alias (not all of them are defined in this library).

>>> r = AsciiRecord(10)
>>> r.IDC = 4           # known
>>> r._2 = 5           # known, by the tag number
>>> r.SRC = 'my src'    # new one, predeclared
>>> r._7 = 'my vll'     # by the tag number
>>> r._345 = 'no alias'
>>> print(str(r))
10.001: LEN                           : 0
10.002: IDC                           : 5
10.004: SRC                           : my src
10.007: VLL                           : my vll
10.345:                               : no alias

Other usage will generate exceptions

>>> r.UUU = 'unknown alias'
Traceback (most recent call last):
...
nistitl.NistException: UNKNOWN_ATTRIBUTE: Bad attribute name <UUU> while trying to define a field in record of type 10

>>> r._76a = 'not an integer'
Traceback (most recent call last):
...
nistitl.NistException: UNKNOWN_ATTRIBUTE: Bad attribute name <_76a> while trying to define a field in record of type 10

It is possible to set the 999 tag with a bytes object:

>>> r.DATA = b'data'
>>> print(str(r))
10.001: LEN                           : 0
10.002: IDC                           : 5
10.004: SRC                           : my src
10.007: VLL                           : my vll
10.345:                               : no alias
10.999: DATA                          : <buffer, size=4>

__str__()

Build and return a summary of the record. It can be useful to document the content of a NIST.

>>> r = AsciiRecord(14)
>>> r += Field(14, 3, 'OK')
>>> r += BinaryField(14, 999, b'data')
>>> print(str(r))
14.001: LEN                           : 0
14.002: IDC                           : 0
14.003: IMP                           : OK
14.999: DATA                          : <buffer, size=4>

class nistitl.BinaryField(record, tag, value=b'', format='%d.%03d:', alias='')

Bases: object

A binary field used as the last field in an AsciiRecord record.

property NIST

Calculation of the NIST binary representation of this field. A bytes object is returned.

__init__(record, tag, value=b'', format='%d.%03d:', alias='')

Creation of a new binary field.

• record indicates the record’s type and must be an integer.

• tag is the tag number and will be converted to an integer.

• value the initial value. Must be a bytes object.

• format is the format used to generate the tag number in the NIST.

• alias is a name used as a shortcut when accessing the value.

property value

Access to the value of the field. The value is a bytes object.

class nistitl.BinaryRecord(type)

Bases: object

A binary record. Binary records are fully binary and are not parsed at all. Only the IDC is extracted.

>>> r = BinaryRecord(4)
>>> r.IDC = 1
>>> r.value = b'data'
>>> print(r.NIST)
b'\x00\x00\x00\t\x01data'

>>> print(str(r))
 4.001: LEN                           : 9
 4.002: IDC                           : 1
 4.---:                               : <buffer, size=4>

property IDC

Shortcut to access the IDC of the record.

property NIST

Calculation of the NIST binary representation of this record. A bytes object is returned.

__init__(type)

__str__()

Return str(self).

property length

Access to the record length (in bytes)

pack(format, *args)

A direct wrapper on struct.pack. If successful, the format is kept in the object and used in __str__ to improve the output.

There is always an additional argument corresponding to additional data added to the value (usually an image data).

>>> r = BinaryRecord(4)
>>> r.IDC = 1
>>> r.pack("!LHB", 3, 2, 1, b'my image data')
>>> print(r.NIST)
b'\x00\x00\x00\x19\x01\x00\x00\x00\x03\x00\x02\x01my image data'

>>> print(str(r))
 4.001: LEN                           : 25
 4.002: IDC                           : 1
 4.---:                               : <buffer, size=20>
                                      -> 3
                                      -> 2
                                      -> 1

unpack(format)

Unpack the value of the record (without the length and the IDC) and return the value described in the format and the remaining buffer (usually an image data)

If successful, the format is kept in the object and used in __str__ to improve the output.

>>> r = BinaryRecord(4)
>>> r.IDC = 1
>>> r.pack("!LHB", 3, 2, 1, b'my image data')
>>> r.unpack("!BHL")    # format changed for the test!
(0, 0, 50332161, b'my image data')

>>> print(str(r))
 4.001: LEN                           : 25
 4.002: IDC                           : 1
 4.---:                               : <buffer, size=20>
                                      -> 0
                                      -> 0
                                      -> 50332161

class nistitl.Field(record, tag, value='', type='FSI', format='%d.%03d:', alias='')

Bases: object

A class representing a field added to an AsciiRecord.

A field can be configured to accept only a single value (no subfields, no items)

>>> f = Field(2, 3, type="F")
>>> f.value = 'ok'
>>> f.value = [1, 2]
Traceback (most recent call last):
...
nistitl.NistException: BAD_FIELD_VALUE: Field 2.003: cannot have subfields
>>> f.value = [[1, 2]]
Traceback (most recent call last):
...
nistitl.NistException: BAD_FIELD_VALUE: Field 2.003: cannot have subfields

Or only a subfield:

>>> f = Field(2, 3, type="S")
>>> f.value = [1, 2]
>>> f.value = 'ok'
Traceback (most recent call last):
...
nistitl.NistException: BAD_FIELD_VALUE: Field 2.003: cannot have a value (only subfields and/or items)
>>> f.value = [[1, 2]]
Traceback (most recent call last):
...
nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield cannot have items

Or only subfields with items:

>>> f = Field(2, 3, type="I")
>>> f.value = [[1, 2]]
>>> f.value = [1, 2]
Traceback (most recent call last):
...
nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield cannot have a value, only items
>>> f.value = 'ok'
Traceback (most recent call last):
...
nistitl.NistException: BAD_FIELD_VALUE: Field 2.003: cannot have a value (only subfields and/or items)

Or subfields can be forbidden unless they have items:

>>> f = Field(2, 3, type="FI")
>>> f.add_subfields( SubField('a', type="SI") )
Traceback (most recent call last):
...
nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield of 2.003: cannot have a value: 'a'

Or a subfield with items can be forbidden:

>>> f = Field(2, 3, type="SF")
>>> f.add_subfields( SubField(['a', 'b'], type="SI") )
Traceback (most recent call last):
...
nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield of 2.003: cannot have items

property NIST

Calculation of the NIST binary representation of this field. A bytes object is returned.

If the value of the field is None, it is interpreted as an empty field.

>>> f = Field(2, 2, None)
>>> f.NIST
b'2.002:'

__getitem__(idx)

Access the idx-th sub field in the field.

>>> f = Field(2, 3, ['one', 'two'])
>>> print(f[0].value)
one
>>> print(f[1].value)
two

__init__(record, tag, value='', type='FSI', format='%d.%03d:', alias='')

Creation of a new field.

• record indicates the record’s type and must be an integer.

• tag is the tag number and will be converted to an integer.

• value the initial value. Must be a string or a list.

• type indicates the possible type for the field content. It can be a combination of:

  • ‘F’: a single field, i.e. a simple value

  • ‘S’: this field can contain sub fields

  • ‘I’: this field can contain items

• format is the format used to generate the tag number in the NIST.

• alias is a name used as a shortcut when accessing the value.

__len__()

Return the number of subfields in the field.

>>> f = Field(2, 3, ['one', 'two'])
>>> print(len(f))
2
>>> f = Field(2, 21, "hello")
>>> print(len(f))
0

add_subfields(*sf)

Add a set of subfields to this field. Some consistency checks are run.

reset()

Reset this field value and the list of subfields.

property value

Access to the value of the field. The value is a string or a list of subfield values.

class nistitl.Message

Bases: object

The Message class is the main entry point. It is used to parse an existing NIST file as well as for building a new one.

property CNT

Calculation of the CNT field of the type-1 record. Returns the Field for the CNT.

>>> msg = Message()
>>> r = AsciiRecord(2)
>>> r.IDC = 3
>>> msg = msg + r
>>> msg.TOT = 'TEST'
>>> len(msg)
2
>>> msg.CNT.NIST
b'1.003:1\x1f1\x1e2\x1f3'

property NIST

Calculation of NIST binary content. A bytes is returned containing all the records previously added.

>>> msg = Message()
>>> msg.TOT = 'TEST'
>>> msg[0].DAT = '20190517'
>>> msg[0].TCN = 'doctest'
>>> msg.NIST
b'1.001:115\x1d1.002:0400\x1d1.003:1\x1f0\x1d1.004:TEST\x1d1.005:20190517\x1d1.007:000\x1d1.008:000\x1d1.009:doctest\x1d1.011:00.00\x1d1.012:00.00\x1c'

property TOT

Shortcut to the TOT of the message (stored in the type 1 record)

__add__(record)

Add a record to the list of records of the message. Example:

>>> msg = Message()
>>> r = AsciiRecord(2)
>>> msg = msg + r
>>> len(msg._records)
2

__delitem__(key)

Delete a record. key can be:

• a tuple or list containing type record type (a numeric) and the record IDC (a numeric)

• a single numeric value: the index in the list of records.

>>> msg = Message()
>>> r = AsciiRecord(2)

Delete the second record, whatever its type:

>>> msg = msg + r
>>> del msg[1]
>>> len(msg)
1

Access the first type 2 record with IDC = 0:

>>> msg = msg + r
>>> del msg[(2, 0)]
>>> len(msg)
1

Type 1 record cannot be deleted (create a new message and move or copy the records if you have a need for this):

>>> msg = msg + r
>>> del msg[(1, 0)]
Traceback (most recent call last):
...
nistitl.NistException: CANNOT_DELETE_TYPE1: Cannot delete the type 1 record

>>> del msg[0]
Traceback (most recent call last):
...
nistitl.NistException: CANNOT_DELETE_TYPE1: Cannot delete the type 1 record

__getitem__(key)

Return the record identified by key. key can be:

• a tuple or list with two integers: the record type and the record IDC

• a single numeric value: the index in the list of records.

>>> msg = Message()
>>> r = AsciiRecord(2)
>>> msg = msg + r

Access the second record, whatever its type:

>>> print(msg[1].type)
2

Access the first type 2 record with IDC = 0:

>>> print(msg[(2, 0)].type)
2

Since type 1 has no IDC, it can only be accessed with an index of 0

>>> print(msg[0].type)
1

If there is no record for the key, an exception is raised:

>>> msg[100]
Traceback (most recent call last):
...
IndexError: list index out of range

>>> msg[(2, 100)]
Traceback (most recent call last):
...
IndexError: list index out of range

__iadd__(record)

Add a record to the list of records of the message. Example:

>>> msg = Message()
>>> r = AsciiRecord(2)
>>> msg += r
>>> len(msg._records)
2

__init__()

__isub__(record)

Remove one specific record from the message.

>>> msg = Message()
>>> r = AsciiRecord(2)
>>> msg = msg + r
>>> len(msg)
2
>>> msg -= r
>>> len(msg)
1

>>> msg -= r
Traceback (most recent call last):
...
nistitl.NistException: RECORD_NOT_FOUND: Cannot remove a record: record not found

__iter__()

Iterate over all the records.

>>> msg = Message()
>>> r = AsciiRecord(2)
>>> msg = msg + r
>>> for record in msg:
...     print(record.type)
1
2

__len__()

Return the number of records in the message.

>>> msg = Message()
>>> print(len(msg))
1

__str__()

Build and return a summary of the NIST content. It can be useful to document the content of a NIST.

>>> msg = Message()
>>> r = AsciiRecord(2)
>>> r.IDC = 3
>>> msg = msg + r
>>> msg.TOT = 'TEST'
>>> msg[0].DAT = '20190517'
>>> msg[0].TCN = 'doctest'
>>> msg[0].TCR = 'TCR'
>>> print(str(msg))
 1.001: LEN                           : 0
 1.002: VER                           : 0400
 1.003: CNT                           : [[1, 1], [2, 3]]
 1.004: TOT                           : TEST
 1.005: DAT                           : 20190517
 1.007: DAI                           : 000
 1.008: ORI                           : 000
 1.009: TCN                           : doctest
 1.010: TCR                           : TCR
 1.011: NSR                           : 00.00
 1.012: NTR                           : 00.00
 2.001: LEN                           : 0
 2.002: IDC                           : 3

__sub__(record)

Remove one specific record from the message.

>>> msg = Message()
>>> r = AsciiRecord(2)
>>> msg = msg + r
>>> len(msg)
2
>>> msg = msg - r
>>> len(msg)
1

>>> msg = msg - r
Traceback (most recent call last):
...
nistitl.NistException: RECORD_NOT_FOUND: Cannot remove a record: record not found

iter(type)

Return an iterator on all the records of type type.

>>> msg = Message()
>>> r = AsciiRecord(2)
>>> msg = msg + r
>>> for record in msg.iter(2):
...     print(record.type)
2

parse(buffer)

Parse a NIST buffer and initialize the list of records corresponding to the content. buffer must be a bytes object, not a string.

>>> msg = Message()
>>> r = AsciiRecord(2)
>>> r.IDC = 3
>>> msg = msg + r
>>> msg.TOT = 'TEST'
>>> msg[0].DAT = '20190517'
>>> msg[0].TCN = 'doctest'

>>> new_msg = Message()
>>> new_msg.parse( msg.NIST )
>>> len(new_msg)
2

If the buffer is truncated, an exception is raised:

>>> new_msg = Message()
>>> new_msg.parse( msg.NIST[:-1] )
Traceback (most recent call last):
...
nistitl.NistException: NIST_TOO_SHORT: NIST buffer too short (missing bytes) when parsing record 2

If there are extra bytes to the buffer, this may indicate a binary record is truncated:

>>> new_msg = Message()
>>> new_msg.parse( msg.NIST + b'x12345' )
Traceback (most recent call last):
...
nistitl.NistException: BAD_CONTENT: Could not recognize binary record, bad content or bad record

This works also with type 14 records that end with a binary buffer:

>>> msg = Message()
>>> msg.TOT = 'TEST'
>>> msg[0].DAT = '20190517'
>>> msg[0].TCN = 'doctest'
>>> r = AsciiRecord(14)
>>> r += Field(14, 3, 'OK')
>>> r += BinaryField(14, 999, b'data')
>>> msg = msg + r

>>> new_msg = Message()
>>> new_msg.parse( msg.NIST[:-1])
Traceback (most recent call last):
...
nistitl.NistException: NIST_TOO_SHORT: NIST buffer too short (missing bytes) when parsing record 14

>>> new_msg = Message()
>>> new_msg.parse( msg.NIST + b'x')
Traceback (most recent call last):
...
nistitl.NistException: NIST_TOO_LONG: NIST buffer too long (extra bytes)

reset(autocreate=True, autosort=True)

Reset the record list, leaving only a blank new type-1 record. if autocreate is True, the type-1 record is initialized. If autosort is True, the tags are sorted by numeric order.

class nistitl.NistError(value)

Bases: Enum

exception nistitl.NistException(message, error)

Bases: Exception

Exception class used to report errors generated from this module.

__init__(message, error)

__str__()

Return str(self).

__weakref__

list of weak references to the object

class nistitl.SubField(value='', type='SI')

Bases: object

A SubField class, representing a subfield added to a Field.

A subfield can be typed to accept only items:

>>> sf = SubField(type="I")
>>> sf.value = ['a', 'b']
>>> sf.value = 'error'
Traceback (most recent call last):
...
nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield cannot have a value, only items

Or only a single value:

>>> sf = SubField(type="S")
>>> sf.value = 'ok'
>>> sf.value = ['a', 'b']
Traceback (most recent call last):
...
nistitl.NistException: BAD_SUBFIELD_VALUE: Subfield cannot have items

Or both:

>>> sf = SubField(type="SI")
>>> sf.value = 'ok'
>>> sf.value = ['a', 'b']

property NIST

Calculation of the NIST binary representation of this subfield. A bytes object is returned.

If the value of the subfield is None, it is interpreted as an empty subfield.

>>> sf = SubField(None)
>>> sf.NIST
b''

If the value is not a string, it is converted to a string:

>>> sf = SubField(12)
>>> sf.NIST
b'12'

__getitem__(idx)

Return th idx-th item.

>>> sf = SubField(type="SI")
>>> sf.value = ['a', 'b']
>>> sf[0]
'a'
>>> sf[1]
'b'

__init__(value='', type='SI')

Creation of a new subfield.

• value the initial value. Must be a string or a list of strings.

• type indicates the possible type for the field content. It can be a combination of:

  • ‘S’: a single subfield, i.e. a simple value

  • ‘I’: this subfield can contain items

__len__()

Return the number of items in the subfield.

>>> f = SubField(type="I", value=['one', 'two'])
>>> print(len(f))
2

add_values(*i)

Add multiple values as items of this subfield.

property value

Access to the value of the subfield. The value is a string or a list of strings.

nistitl.parse_record(buffer, push_record, push_field, push_subfield, push_value)

Parse a single “text” record