dfdl:newVariableInstance statements. They must all be for unique variables, it
is a Schema Definition Error otherwise. The order of execution is specified in
Section 9.5 Evaluation
Order for Statement Annotations.
There is no short form syntax for creating variable instances.
<dfdl:newVariableInstance ref="EDIFACT_DS" defaultValue=","/>
<dfdl:newVariableInstance ref="lengthUnitBits">
{ if (../hdr/fmtCode eq "bits") then 1 else 8 }
</dfdl:newVariableInstance>
Variable instances get their values either by default, by external definition, or by subsequent assignment using the dfdl:setVariable statement annotation.
<dfdl:setVariable
ref = QName
value?
= logical value or dfdl expression >
<!-- Contains: logical value or dfdl expression (value) -->
</dfdl:setVariable>
The dfdl:setVariable annotation can be used on a simpleType, group reference, sequence or choice. It may be used on an element or element reference only if the element is of simple type. It is a Schema Definition Error if dfdl:setVariable appears on an element of complex type, or an element reference to an element of complex type.
The ref property is a QName. That is, it may be qualified with a namespace prefix.
The syntax supports both a value attribute and the 'value' being specified by the element value. Only one or the other may be present (otherwise it is a Schema Definition Error). To set the value to "" (empty string), the value attribute syntax must be used, or the expression { "" } must be used as the element value.
The value must match the type of the variable as specified by dfdl:defineVariable.
A dfdl:setVariable value expression may
refer to the value of this element using a relative path value ".".
Use of relative path expressions is recommended wherever possible as this will
allow the behavior of the parser to be more effectively scoped. However this
practice is not enforced and there may be situations in which use of an
absolute path is in fact necessary.
The expression must not contain forward references to elements which have not yet been processed.
The declaration of a variable must be in scope at the point of the assignment, and at the point of reference.
In normal processing, the value of an instance can only be set once using dfdl:setVariable. Attempting to set the value of the variable instance for a second time is a Schema Definition Error. In addition, if a reference to the variable's value has already occurred and returned a default or an externally supplied value, then no assignment (even a first one) can occur. An exception to this behavior occurs whenever the DFDL processor backtracks because it is processing multiple branches of a choice or as a result of speculative parsing. In this case the variable state is also rewound.
A dfdl:setVariable will override any default value specified on either dfdl:defineVariable or dfdl:newVariableInstance, or externally.
The
resolved set of annotations for an annotation point may contain multiple
dfdl:setVariable statements. They must all be for unique variables and it is a
Schema Definition Error otherwise. The order of execution is specified in
Section 9.5 Evaluation
Order for Statement Annotations.
There is no short form syntax for variable assignment.
<xs:element name="ds" type="xs:string">
<xs:annotation>< xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:setVariable ref="EDI:EDIFACT_DS" value="{.}" />
<dfdl:setVariable ref="delta"> {.} </dfdl:setVariable>
</xs:appinfo></xs:annotation>
</xs:element>
In the above example, the element named "ds" contains the string to be used as the EDI:EDIFACT_DS delimiter at other places in the data, so the above defines the value of the EDI:EDIFACT_DS variable to take on the value of this element.
This section describes the rules that govern the scope over which DFDL representation properties apply
The scope of the representational properties on each of the component format annotations is given in Table 9 DFDL annotation scoping
|
Annotation Point |
Property Scope |
|
Schema declaration |
dfdl:format representation properties apply lexically as default properties over all components in the schema |
|
Element declaration |
dfdl:element properties apply locally |
|
Element reference |
dfdl:element properties apply locally |
|
Simple type definition |
dfdl:simpleType properties apply locally |
|
Sequence |
dfdl:sequence properties apply locally |
|
Choice |
dfdl:choice properties apply locally |
|
Group reference |
dfdl:group properties apply locally |
Table 9 DFDL annotation
scoping
Note: This table lists DFDL annotations on schema components. DFDL annotations can also be placed on other DFDL annotations, such as a dfdl:format within a dfdl:defineFormat, to provide a named reusable format definition. In this case the annotation applies only where the named format is referenced.
DFDL representation properties explicitly defined on annotations, other than a dfdl:format on an xs:schema declaration, apply locally to that component only. The explicitly defined properties are the combination of any defined locally on the annotation and any defined on the dfdl:defineFormat annotation referenced by a local dfdl:ref property. When a property is defined both locally and on the dfdl:defineFormat, the locally defined property takes precedence.
The dfdl:format annotation on the top level xs:schema declaration provides defaults for the DFDL representation properties at every DFDL-annotatable component contained in the schema document. They do not apply to any components in any included or imported schema document (these may have their own defaults).
A dfdl:format annotation on the top level xs:schema declaration may provide defaults for some or all the DFDL representation properties at every annotation point within the schema document. The default properties may be specified in attribute or element form. (Short form is not allowed on the xs:schema element.)
The dfdl:ref property is not a representation property so no default can be set.
The dfdl:escapeSchemeRef property provides a default reference to a dfdl:defineEscapeScheme, the properties of dfdl:escapeScheme are not defaulted individually.
DFDL representation properties defined explicitly on a component apply only to that component and override the default value of that property provided by a default format specified by an xs:schema dfdl:format annotation.
The example below demonstrates the overriding of the encoding property. The value'ASCII' is the default value for the title element, but then it is overridden by the locally defined utf-8 value for the encoding property, which takes precedence.
<xs:schema>
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:format encoding="ASCII" />
</xs:appinfo>
</xs:annotation>
<xs:element name="book">
<xs:complexType>
<xs:sequence>
<xs:element name="title" type="xs:string">
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:element encoding="utf-8" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="pages" type="xs:int"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
The DFDL representation properties contained in a referenced dfdl:defineFormat are combined with any DFDL representation properties defined locally on a construct as if they had been defined locally. If the same property is defined locally in and in the referenced dfdl:defineFormat then the local property takes precedence. The combined set of explicit DFDL properties has precedence over any defaults set by a dfdl:format on the xs:schema.
<xs:schema>
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:defineFormat name='myFormat'>
<dfdl:format encoding="ASCII" />
</dfdl:defineFormat>
</xs:appinfo>
</xs:annotation>
<xs:element name="book">
<xs:complexType>
<xs:sequence>
<xs:element name="title" type="xs:string">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element ref='myFormat' encoding="UTF-8" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="pages" type="xs:int"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
The example above demonstrates the overriding of an encoding property. The 'ASCII' format encoding from the 'myFormat' is overridden by the UTF-8 format encoding, which as a locally defined property takes precedence.
The DFDL properties from the following types of reference are combined using the rules below:
Rules
"Applicable" properties are all the DFDL properties that apply to that schema component. For example all the DFDL properties that apply to a particular xs:simpleType (as defined by section 13).
<xs:simpleType name="newType">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:simpleType alignment="16"/>
</xs:appinfo>
</xs:annotation>
<xs:restriction base="xs:integer">
<xs:maxInclusive value="10"/>
</xs:restriction>
</xs:simpleType>
<xs:element name="testElement1" type="newType">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element representation="binary"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
The locally defined dfdl:alignment property with value '16' from the xs:simpleType 'newType' is combined with the locally defined dfdl:representation property with value 'binary' and applied to element 'testElement1',
<xs:simpleType name="otherNewType">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:simpleType alignment="64"/>
</xs:appinfo>
</xs:annotation>
<xs:restriction base="newType">
<xs:maxInclusive value="5"/>
</xs:restriction>
</xs:simpleType>
<xs:simpleType name="newType">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:simpleType representation='binary'/>
</xs:appinfo>
</xs:annotation>
<xs:restriction base="xs:int">
<xs:maxInclusive value="10"/>
</xs:restriction>
</xs:simpleType>
The
locally defined dfdl:representation property with value 'binary' is combined
with the locally defined dfdl:alignment property with value '64' from the
xs:simpleType restriction 'otherNewType'.
<xs:sequence>
<xs:element ref="testElement1">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element binaryNumberRep ="binary"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
<xs:element name="testElement1" type="newType">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element representation="binary"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:simpleType name="newType">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:simpleType alignment="16"/>
</xs:appinfo>
</xs:annotation>
<xs:restriction base="xs:int">
<xs:maxInclusive value="10"/>
</xs:restriction>
</xs:simpleType>
The locally defined dfdl:alignment property
with value '16' from the xs:simpleType 'newType' is combined with the locally
defined dfdl:representation property with value 'binary' and locally defined
dfdl:binaryNumberRep with a value of 'binary'
<!-- SCHEMA1 -->
<xs:schema targetNamespace="" xmlns:tns1="http://tns1">
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:format encoding="ASCII" byteOrder="littleEndian"
initiator="" terminator=""
sequenceKind="ordered"
/>
</xs:appinfo>
</xs:annotation>
<xsd:import namespace="http://tns2" schemaLocation="SCHEMA2.xsd"/>
<xs:element name="book">
<xs:complexType>
<xs:group ref="tns2:ggrp1"
dfdl:separator=","></xs:group>
</xs:complexType>
</xs:element>
</xs:schema>
<!-- SCHEMA2 -->
<xs:schema targetNamespace=""
xmlns:tns2="http://tns2">
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:format encoding="UTF-8" byteOrder="littleEndian"
initiator=""
sequenceKind="ordered"
/>
</xs:appinfo>
</xs:annotation>
<xs:group
name="ggrp1" >
<xs:sequence
dfdl:separatorPosition="infix" >
<xs:element name="customer" type="xs:string"
dfdl:length="8" dfdl:lengthKind="explicit"
/>
</xs:sequence>
</xs:group>
</xs:schema>
The
DFDL properties applied to the xs:sequence in xs:group "ggrp1" in
SCHEMA2 when referenced from the group reference in SCHEMA1 are
A DFDL Parser is an application or code library that takes as input:
It is able to use the DFDL schema description to interpret the data stream and realize the DFDL Information Model. This information set could then be written out (for example it could be realized as an XML text string) or it could be accessed by an application through an API (for example, a DOM-like tree could be created in memory for access by applications).
Symmetrically, there is a notion of a DFDL Unparser. The unparser works from an instance of the DFDL Information Model, a DFDL annotated schema and writes out to a target data stream in the appropriate representation formats.
Often both parser and unparser would be implemented in the same body of software and so we do not always distinguish them. Collectively they are called a DFDL Processor. The parser and unparser may, of course, be different bodies of software. Conforming DFDL processors may implement only a parser, because the unparser is an optional feature of DFDL.
The DFDL logical parser is a recursive-descent parser[7] having guided, but potentially unbounded look ahead that is used to resolve points of uncertainty. A DFDL parser reads a specification (the DFDL schema) and it recursively walks down and up the schema as it processes the data. This is done in a manner consistent with the scoping of properties and variables described in Section 8 Property Scoping Rules.
The unbounded look ahead means that there are situations where the parser must speculatively attempt to parse data where the occurrence of a processing error causes the parser to suppress the error, back out and make another attempt.
Implementations of DFDL may provide control mechanisms for limiting the speculative search behavior of DFDL parsers. The nature of these mechanisms is beyond the scope of the DFDL specification which defines the behavior of conforming parsers only on correct data. That is, data that can be parsed without any effective processing errors. Any such control mechanisms must be documented by the implementation and are thus implementation-defined.
The logical parser recursively descends the DFDL schema beginning with the distinguished global element declaration (specified for the processor in an implementation-defined manner, see Section 18). Depending on the kind of schema construct that is encountered and the DFDL annotations on it, and the pre-existing context, the parser performs specific parsing operations on the data stream. These parsing operations typically recognize and consume data from the stream and construct values in the logical model. For values of complex types and for arrays, these logical model values may incorporate values created by recursive parsing.
DFDL Implementations are free to use whatever techniques for parsing they wish so long as the semantics are equivalent to that of the speculative recursive-descent logical parser described in this specification. It is required that implementations distinguish the various kinds of errors (Schema Definition Error, processing error, etc.) no matter what time they are detected. Some implementations may not detect certain Schema Definition Errors until data are being parsed; however, they must still distinguish Schema Definition Errors (which indicate that the schema itself is not meaningful), from parsing errors (which indicate that the input data doesn't satisfy the requirements of the schema), or unparsing errors (which indicate that the infoset does not satisfy the requirements of the schema).
Data in a format describable via a DFDL schema obeys the grammar given here. A given DFDL schema is read by the DFDL processor to provide specific meaning to the terminals and decisions in this grammar.
The bits of the data are divided into two broad categories:
The content is the bits of data that are interpreted to compute a logical value.
Framing is the term used to describe the delimiters, length fields, and other parts of the data stream which are present, and may be necessary to determine the length or position of the content of DFDL Infoset items.
Note that sometimes the framing is not strictly necessary for parsing, but adds useful redundancy to the data format, allowing corrupt data to be more robustly detected, and sometimes the framing adds human readability to the data format.
In the grammar tables below, the terminal symbols are shown in bold italic font.
|
Productions |
|
Document
SimpleElement = SimpleLiteralNilElementRep | SimpleEmptyElementRep | SimpleNormalRep SimpleEnclosedElement = SimpleElement | AbsentElementRep ComplexElement = ComplexLiteralNilElementRep | ComplexNormalRep | ComplexEmptyElementRep ComplexEnclosedElement = ComplexElement | AbsentElementRep EnclosedElement = SimpleEnclosedElement | ComplexEnclosedElement |
|
AbsentElementRep = Absent |
|
SimpleEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming ComplexEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming EmptyElementLeftFraming = LeadingAlignment EmptyElementInitiator PrefixLength EmptyElementRightFraming = EmptyElementTerminator TrailingAlignment |
|
SimpleLiteralNilElementRep = NilElementLeftFraming [NilLiteralCharacters | NilElementLiteralContent] NilElementRightFraming ComplexLiteralNilElementRep =
NilElementLeftFraming NilLiteralValue
NilElementLeftFraming = LeadingAlignment NilElementInitiator PrefixLength NilElementRightFraming = NilElementTerminator TrailingAlignment NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill |
|
SimpleNormalRep = LeftFraming PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming PrefixLength ComplexContent
LeftFraming = LeadingAlignment Initiator RightFraming = Terminator TrailingAlignment PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent SimpleContent = LeftPadding [ SimpleLogicalValue
= SimpleNormalValue
| NilLogicalValue ComplexContent
= ComplexValue ElementUnused ComplexValue =
Sequence | Choice
|
|
Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedContent [ Separator EnclosedContent ]* PostfixSeparator ] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedContent ] ChoiceUnused EnclosedContent = [ EnclosedElement | Array | Sequence | Choice ] Array = [ EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] StopValue = SimpleElement |
|
LeadingAlignment = LeadingSkip AlignmentFill TrailingAlignment = TrailingSkip RightPadOrFill
= RightPadding
| RightFill
| RightPadding
RightFill |
Table 10 DFDL Grammar Productions
XML Schema and DFDL properties are used to control constraints on the terminals of the above grammar, as well as repetition (the "*" operator), and alternatives (the "|" operator). For a given set of XML Schema and DFDL properties, and prior data, any terminal may be allowed to be length zero, to contain specific data, or to contain a variety of different admissible data.
Some definitions are needed to cover the range of representations that are possible in the data stream for an element. These definitions are with respect to the grammar above.
An element occurrence has a nil representation if the element has XSDL nillable property 'true' and the occurrence either:
The LeadingAlignment, TrailingAlignment, PrefixLength regions may be present.
An element occurrence has an empty
representation if the occurrence does not have a nil representation
and it conforms to the grammar for SimpleEmptyElementRep or
ComplexEmptyElementRep. Specifically, the EmptyElementInitiator and EmptyElementTerminator
regions must be conformant with dfdl:emptyValueDelimiterPolicy and the
occurrence's SimpleContent or ComplexContent region in the data the
occurrence's content in the data stream is of length zero. (If
non-conformant it is not a processing error and the representation is not
empty). LeadingAlignment, TrailingAlignment, PrefixLength regions may be
present.
The empty representation is special in DFDL
because when parsing it is used to determine when default values are created in
the Infoset. The empty representation can require initiators or terminators be
present so as to enable data formats which
explicitly distinguish occurrences with empty string/hexBinary values from
occurrences that are missing or are absent. The empty
representation is special in DFDL, because when parsing it is this
condition that can trigger the creation of a default value for an element
occurrence. See Section 9.4 Element
Defaults below about default
values.
The empty representation is not possible for fixed-length elements with a non-zero length.
An element occurrence has a normal representation if the occurrence does not have the nil representation or the empty representation and it conforms to the grammar for SimpleNormalRep or ComplexNormalRep.
An element occurrence has an absent representation if the occurrence does not have a nil or empty or normal representation, and it conforms to the grammar for AbsentElementRep. Specifically, the occurrence's representation in the data stream is of length zero. Consequently, the Initiator, Terminator, LeadingAlignment, TrailingAlignment, PrefixLength regions must not be present.
Example of an absent representation: During unparsing, if an optional element does not have an item in the infoset then nothing is output. However if a separator of an enclosing structure is subsequently output as the immediate next thing, then a subsequent parse of the element may return a representation of length zero. If this happens, and this zero-length representation does not conform to any of the nil representation, the empty representation, or the normal representation, then it is the absent representation, and it behaves as if the element occurrence is 'missing'. (The term 'missing' is defined below.)
The point of this term 'absent representation', is that often we know the location where an element or group's representation would be in the data based on the delimiters of an enclosing group. (An example: if there are adjacent delimiters of an enclosing sequence.) When this location in the data, which is of zero length, cannot be a nil, empty, or normal representation, then we say it has absent representation, or "the representation is absent".
We use the term zero-length representation to describe the situations where any of the above representations turn out to be of length zero due to specific combinations of data type and format properties:
The nil representation can be a zero-length
representation if dfdl:nilValue is
‘%ES;’ or
‘%WSP*;’ appearing
on its own as a literal nil value is
"%ES;", and there is no framing or framing is suppressed
by dfdl:nilValueDelimiterPolicy.
The empty representation can be a zero-length representation if there is no framing or framing is suppressed by dfdl:emptyValueDelimiterPolicy.
The normal representation can be a zero-length representation if the type is xs:string or xs:hexBinary and there is no framing.
The absent representation always has a zero-length representation.
If the nil representation may be zero-length, then the absent representation cannot occur because zero-length will be interpreted as nil representation.
If the nil representation may not be zero length, but the empty representation is zero-length, then the absent representation cannot occur because zero-length will be interpreted as the empty representation.
If the nil and empty representations can not be zero-length, but the normal representation may be zero length then the absent representation cannot occur because zero length will be interpreted as a normal representation.
If the nil representation may not be zero-length, the empty representation is not zero-length, and the normal representation may not be zero-length, then a zero-length representation is the absent representation, or "is absent".
When parsing, an element occurrence is missing if it does not have nil, empty, or normal representations, or it has the absent representation.
When parsing, the term missing really covers two situations. Firstly it subsumes absent representation. Secondly it applies when an element does not have a representation at all in the data stream, that is, when we do not even have the constructs in the data stream to determine the location of the representation of the element; hence, none of the concepts above apply. This will be made clearer in the examples below. If an element occurrence is missing when parsing, no item is ever added to the Infoset.
When unparsing, an element occurrence is missing if there is no item in the infoset. For a required element occurrence, it is this condition that can trigger the creation of a default value in the augmented infoset. See Section 9.4 Element Defaults below about default values. For an optional element occurrence, no item is ever added to the augmented Infoset nor any representation ever output in the data stream.
The following examples illustrate missing and empty representation.
<xs:sequence dfdl:separator="," dfdl:terminator="@"
dfdl:separatorSuppressionPolicy="trailingEmpty">
<xs:element name="A" type="xs:string"
dfdl:lengthKind="delimited"/>
<xs:element name="B" type="xs:string" minOccurs="0"
dfdl:lengthKind="delimited"/>
<xs:element name="C" type="xs:string" minOccurs="0"
dfdl:lengthKind="delimited"/>
</xs:sequence>
In data stream aaa,@ element B has the empty representation, and element C does not have a representation so is missing.
<xs:sequence dfdl:separator=","
dfdl:separatorSuppressionPolicy="anyEmpty">
<xs:element name="A" type="xs:string"
dfdl:lengthKind="delimited" dfdl:initiator="A:"
dfdl:emptyValueDelimiterPolicy=initiator"/>
<xs:element name="B" type="xs:string" minOccurs="0"
dfdl:lengthKind="delimited" dfdl:initiator="B:"
dfdl:emptyValueDelimiterPolicy="initiator"/>
<xs:element name="C" type="xs:string" minOccurs="0"
dfdl:lengthKind="delimited" dfdl:initiator="C:"
dfdl:emptyValueDelimiterPolicy=initiator"/>
</xs:sequence>
In data stream A:aaaa,C:cccc element B does not have a representation so is missing.
In data stream A:aaaa,B:,C:cccc element B has the empty representation.
In the data stream A:aaaa,,C:cccc element B has the absent representation so is missing.
The overlapping nature of the possible representations: normal, empty, nil, and absent, creates a number of ambiguities where taking an Infoset, unparsing it, and reparsing it will result in a second Infoset that is not the same as the original. However taking the second Infoset, unparsing it, and reparsing it, will result in a third Infoset which is the same as the second.
When unparsing, if a string Infoset item happens to contain a string that matches either one of the nilValues or the default value, it is not given any special treatment. The string's characters are output, or if the value is the empty string, zero length content is output. (In both cases along with an initiator or terminator if defined.) This creates an ambiguity where one can unparse an Infoset item which has member [nilled] true, but when reparsed will produce an Infoset item which has member [nilled] false.
These
ambiguities are natural and unavoidable. If the nilValue is the 3-character
string "nil", then encountering the characters "nil" in the
data stream will parse to produce an Infoset item with [nilled] true in the Infoset. If you unparsed a string infoset item
with contents of the 3 characters "nil", this will be output as the
letters "nil", which on parse will not produce a string with the
characters "nil", but rather an Infoset item with no data value and
member [nilled] true.
To
avoid this issue, one can use validation, along with a pattern that prevents
the string from matching any of the nil values.
A DFDL parser proceeds by determining the existence of occurrences of schema components. It does this by examining the data and the schema, so as to:
a) Establish representation
b) Resolve points of uncertainty
These two activities are defined below. They are mutually recursive in the expected way as a DFDL schema is a recursive nest of schema components.
Establishing the representation of an occurrence of a schema component and resolving points of uncertainty involve the concepts of known-to-exist and known-not-to-exist.
An occurrence of a schema component is said to be known-to-exist when any of these positive discriminations hold:
If none of those hold because they are not applicable then the occurrence is still known-to-exist if ALL of the following hold, and no processing error occurs during their determination:
Note that validation errors or recoverable errors do not prevent determination that a component is known-to-exist.
Note that it is possible for an occurrence of a schema component to be known-to-exist due to a positive discrimination, but then subsequently a processing error occurs when evaluating a statement annotation such as a dfdl:assert or a dfdl:setVariable, or a processing error occurs when determining the representation, or in the case of normal representation and simpleType, when converting that representation's content into a value of the type. This processing error does not change the fact that the schema component was determined to be known-to-exist. This is important in the discussion of resolving Points of Uncertainty below.
An occurrence of a schema component is known-not-to-exist when any of these negative
discriminations holds:
If none of those hold because they are not
applicable, then a schema component is known-not-to-exist when any of the
following hold:
Note that validation errors or recoverable errors do not cause a component to be known-not-to-exist.
Note: based on the above, when processing a sequence for which a separator is defined, the presence of a match in the data for the separator is not sufficient to cause the parser to determine that an associated component is known-to-exist. See Section 14.2 Sequence Groups with Separators for details.
Unless an element occurrence is known-not-to-exist, it must be established if it has the nil, empty, normal, or absent representation.
The first step is to
see if the SimpleContent or ComplexContent region is of length zero as a first
approximation.The first step is to see if the content is
trivially of length zero. This is dfdl:lengthKind dependent.
If the result is length zero as described above, the representation is then established by checking, in order for:
If the result is not length zero, the representation is then established by checking, in order, for:
If the result is length zero as described above, the representation is then established by checking for:
To establish any other representations requires that the parser descends into the complex type for the element, and returns successfully (that is, no unsuppressed processing error occurs). If the result is zero bits consumed, the representation is then established by checking, in order, for:
Otherwise the element has normal representation.
Note: The DFDL parser shall not recursively parse the schema components inside a complex element when it has already established that the element occurrence is missing[11].
A point of uncertainty occurs in the data stream when there is more than one schema component that might occur at that point. Points of uncertainty can be nested.
Any one of the following constructs is a potential point of uncertainty:
The parser resolves these points of uncertainty by way of a set of construct-specific rules given below along with determining whether schema components are known-to-exist or known-not-to-exist. For some of these constructs, whether there is an actual point of uncertainty depends on the representation of the constructs in the data.
An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch. Sequential choice resolution occurs by parsing each choice branch in schema definition order until one is known-to-exist. It is a processing error if none of the choice branches are known-to-exist. Direct-dispatch choice resolution occurs by matching the value of the dfdl:choiceDispatchKey property to the value of one of the dfdl:choiceBranchKey property values of one of the choice branches. It is a processing error if none of the choice branches have a matching value in their dfdl:choiceBranchKey property.
An element in an unordered xs:sequence is always a point of uncertainty. It is resolved by parsing for the child components of the sequence in schema definition order at each point in the data stream where a component can exist until the required number of occurrences of each child component is known- to-exist or the sequence is terminated by delimiters or specified length.
An element in a sequence with one or more floating elements is always a point of uncertainty. It is resolved by parsing for the expected element at that point in the data stream. If the expected element is known-not-to-exist then an occurrence of each floating element is parsed in schema definition order.
When parsing an array, points of uncertainty only occur for certain values of occursCountKind, as follows:
|
occursCountKind |
Details
of Point of Uncertainty |
|
fixed |
No point of uncertainty (maxOccurs occurrences expected). |
|
implicit |
A point of uncertainty exists after minOccurs occurrences found and until maxOccurs found. |
|
parsed |
A point of uncertainty exists for all occurrences |
|
expression |
No point of uncertainty (occursCount occurrences expected) |
|
stopValue |
No point of uncertainty (stopValue must always be present, even when minOccurs is 0). |
Table 11: Points of Uncertainty and dfdl:occursCountKind
An optional element point of uncertainty is resolved by parsing the element until it is either known-to-exist or known-not-to-exist. Whether an optional element is an actual point of uncertainty depends on property dfdl:occursCountKind as described above. (Property dfdl:occursCountKind is defined in Section 16.1 dfdl:occursCountKind property.)
For an array element, the point of uncertainty is resolved for each occurrence separately by parsing the occurrence until it is either known-to-exist or known-not-to-exist.
A point of uncertainty can be resolved because a schema component has been determined to be known-to-exist due to positive discrimination. In that case, if a subsequent processing error occurs when completing the parsing of that schema component this will cause the next enclosing schema component surrounding this point of uncertainty to be determined to be known-not-to exist.
For example, when parsing an element occurrence for an array with a variable number of occurrences, a positive discrimination tells the parser that the currently-being-parsed occurrence is known-to-exist. If a subsequent processing error occurs while completing the parsing of this occurrence, then the entire array is then known-not-to-exist.
Another example is a choice. If a discriminator resolves the choice point of uncertainty to the first of the choice's alternatives, a subsequent processing error causes the entire choice construct to be determined to be known-not-to-exist.
This will cause the next enclosing point of uncertainty to try the next possible alternative, or if there isn't one, will cause an unsuppressed processing error.
The behavior of a DFDL processor on an unsuppressed processing error is not specified, but it is allowable for implementations to abort further parsing. Any other behavior is implementation-defined.
A DFDL processor can create element defaults in the Infoset for both simple and complex elements. This happens quite differently for parsing and unparsing as will be explained in this section.
A simple element has a default value if any of these are true:
If empty representation is established when parsing, the possibility of applying an element default arises. Essentially, if a required occurrence of an element has empty representation, then an element default will be applied if present, though there are a couple of variations on this rule. Remember that in order to have established empty representation, the occurrence must be compliant with the dfdl:emptyValueDelimiterPolicy for the element, and for a complex element the parser must have descended into the type and returned with no unsuppressed processing error.
The rules for applying element defaults are not dependent on dfdl:occursCountKind. However, if a required occurrence does not produce an item in the Infoset after the rules have been applied, then whether it is a processing error or a validation error (if validation is enabled) does depend on dfdl:occursCountKind (see Section 16.1 dfdl:occursCountKind property).
The sections below
indicate when an item is added to the infoset, and whether it has a default or
other value. If there is no processing error then regardless of whether an item
is added to the infoset or not, any side-effects due to dfdl:discriminator
statements evaluating to true, or dfdl:setVariable statements, are retained.
There are three main cases to consider:
Required occurrence: If the element has a default value then an item is added to the Infoset using the default value, otherwise nothing is added to the Infoset.
Optional occurrence: Nothing is added to the Infoset.
Required occurrence: If the element has a default value then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value.
Optional occurrence:
if dfdl:emptyValueDelimiterPolicy is applicable and is not 'none'Optional
occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none'[12] , then
an item is added to the Infoset using empty string (type xs:string) or empty
hexBinary (type xs:hexBinary) as the value, otherwise nothing is added to the
Infoset.
Note: To prevent unwanted empty strings or empty hexBinary values from being added to the Infoset, use XSD minLength > '0' and a dfdl:assert that uses the dfdl:checkConstraints() function, to raise a processing error.
Required occurrence: An item is added to the Infoset.
Optional occurrence:
if dfdl:emptyValueDelimiterPolicy is applicable and is not 'none'Optional
occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none'[13] , then
an item is added to the Infoset, otherwise nothing is added to the Infoset.
A complex element can
parse, by recursive descent, and construct a complex element in
the infoset containing a single child element. This can occur[1] :For
both required and optional occurrences, the Infoset item may also have a child
item.
As an example, consider a sequence S0 with a separator that contains among other content an optional non-nillable non-initiated element E1 of complex type. The content of the type is a sequence S1 with a different separator and the first child is a required non-initiated element E2 of type xs:string. The dfdl:lengthKind of both E1 and E2 is 'delimited'. The representation of E1 has zero length, that is, the data contains adjacent S0 separators. On processing E1, the parser will establish a point of uncertainty and descend into E1's complex type and process E2. It scans for in-scope delimiters and immediately encounters S0 separator. E2 has the empty representation, so E1 is added to the Infoset along with a value of empty string for E2. All other content of S1 is missing, so the parser returns from the descent. E1 is therefore known-to-exist. Because the position in the data has not changed, E1 therefore has the empty representation. Because E1 is empty and optional it is not added to the Infoset, and the Infoset items for E1 and E2 are discarded.
If an element is missing from the Infoset when unparsing, the possibility of applying an element default arises. Essentially if a required occurrence of an element is missing, then an element default will be applied if present, and the resulting item is added to the augmented Infoset.
The rules for applying element defaults are not dependent on dfdl:occursCountKind. However if a required occurrence does not produce an item in the augmented Infoset after the rules have been applied then whether it is a processing error or a validation error (if enabled) is dependent on dfdl:occursCountKind (see Section 16.1 dfdl:occursCountKind property).
There are two main cases to consider.
Required occurrence: If an element has a default value then an item is added to the augmented Infoset using the default value, otherwise nothing is added.
Optional occurrence: Nothing is added to the augmented Infoset.
Required occurrence: An item is added to the augmented Infoset as specified below.
Optional occurrence: Nothing is added to the augmented Infoset.
For a required occurrence, the unparser descends into the complex type:
For a sequence, each child element is examined in schema order and the rules for simple and complex elements applied (recursively). The lack of a default may give rise to a processing error, as described above.
For a choice, each branch is examined in schema order and the above rules applied recursively to the branch. The lack of a default may give rise to a processing error, as described above, and if so the error is suppressed and the next branch is tried, otherwise that branch is selected. It is a processing error if no choice branch is ultimately selected.
Given a component
of a DFDL schema, there is a resolved set of annotations for it.
Of these, some are statement annotations and the order of their evaluation relative to the actual processing of the schema component itself (parsing or unparsing via its format annotations) is as given in the ordered lists below.
For elements and element references:
1. dfdl:discriminator or dfdl:assert(s) with testKind 'pattern' (parsing only)
1.
dfdl:element
following property scoping rules, which includes establishing representation as
described in Section 9.3.2 and conversion to the element type for simple types dfdl:element
following property scoping rules
2.
3. dfdl:setVariable(s) - in lexical order, innermost schema component first
4. dfdl:discriminator or dfdl:assert(s) with testKind 'expression' (parsing only)
For sequences, choices and group references:
The
dfdl:setVariable annotations at any one annotation point of the schema are
always executed in lexical order. However, dfdl:setVariable annotations can
also be found in different annotation points that are combined into the
resolved set of annotations for one schema component. In this case, the order
of execution of the dfdl:setVariable statements from any one annotation point
remains lexical. The order of execution of the dfdl:setVariable annotations
different annotation points follows the principle of innermost first, meaning
that a schema component that references another schema component has its
dfdl:setVariable statements executed after those of the referenced schema
component. For example, if an element reference and an element declaration both
have dfdl:setVariable statements, then those on the element declaration will
execute before those on the element reference. Similarly dfdl:setVariable
statements on a base simple type execute before those of a simple type derived
from it. The dfdl:setVariable statements on a simple type execute before those
on an element having that simple type (whether by reference, or when the simple
type is lexically nested within the element declaration). The dfdl:setVariable
statements on the sequence or choice within a global group definition execute
before those on a group reference.
The
dfdl:newVariableInstance annotations at any one annotation point of the schema
are always executed in lexical order. However, dfdl:newVariableInstance
annotations can also be found in different annotation points that are combined
into the resolved set of annotations for one schema component. In this case,
the order of execution of the dfdl:newVariableInstance statements from any one
annotation point remains lexical. The order of execution of the
dfdl:newVariableInstance annotations different annotation points follows the
principle of innermost first, meaning that a schema component that contains or
references another schema component has its dfdl:newVariableInstance statements
executed after those of the contained or referenced schema component. For example,
if a group reference and the sequence or choice group of a group definition
both have dfdl:newVariableInstance statements, then those on the global group
definition will execute before those on the group reference.
Implementations are free to optimize by recognizing and executing discriminators or asserts with testKind 'expression' earlier so long as the resulting behavior is consistent with what results from the description above.
When parsing, an
attempt to evaluate a discriminator must be made even if preceding statements
or the parse of the schema component ended in a processing error.
This is because a
discriminator's expression could evaluate to true thereby resolving a point of
uncertainty even if the complete parsing of the construct ultimately caused a
processing error.
Such discriminator
evaluation has access to the DFDL Infoset of the attempted parse as it existed
immediately before detecting the parse failure. Attempts to reference parts of
the DFDL Infoset that do not exist are processing errors.
The resolved set of dfdl:setVariable
statements for an element are executed after the parsing of the element.
This is in contrast to the resolved set of dfdl:setVariable statements for a
group which are executed before the parsing of
the group.
For elements, this
implies that these variables are set after the evaluation of expressions
corresponding to any computed DFDL properties for that element, and so the
variables may not be referenced from expressions that compute these DFDL
properties.
That is, if an
expression is used to provide the value of a property (such as dfdl:terminator
or dfdl:byteOrder), the evaluation of that property expression occurs before
any dfdl:setVariable annotation from the resolved set of annotations for that
element are executed; hence, the expression providing the value of the property
may not reference the variable. Schema authors can insert sequences to provide
more precise control over when variables are set.
The next sections specify the core set of DFDL v1.0 properties that may be used in DFDL annotations in DFDL Schemas to describe data formats.
It is a Schema Definition Error when a DFDL schema does not contain a definition for a representation property that is needed to interpret the data. For example, a DFDL schema containing any textual data must provide a definition of the character set encoding property (dfdl:encoding) for that textual data, and if it is not part of the format properties context for that data, then it is a Schema Definition Error.
Furthermore, no default values are provided for representation properties as built-in definitions by any DFDL processor. This requires DFDL schemas to be explicit about the representation properties of the data they describe, and avoids any possibility of DFDL schemas that are meaningful for some DFDL processors but not others.
The properties are organized as follows:
Where properties are specific to a physical representation, the property name may choose to reflect this. Where properties are related to a specific logical type grouping (defined below), the property name may choose to reflect this.
A
limited number of properties can take a DFDL expression which must return a
value of the proper type for the property. Those properties that take an
expression explicitly state in the description. Other properties do not take an
expression.
The
property description defines which schema component that the property may be
specified on. In addition most DFDL properties may be specified on a
dfdl:format annotation.
|
Property Name |
Description |
|
byteOrder |
Enum or DFDL Expression Valid values 'bigEndian',
'littleEndian'. This property can be computed by way of an
expression which returns the string 'bigEndian' or 'littleEndian'. The
expression must not contain forward references to elements which have not yet
been processed. Note that there is, intentionally, no such
thing as 'native' endian[14]. This property applies to all Number, Calendar, and Boolean types with representation binary. Specifically that is binary integers, binary booleans, all packed decimals, binary floats, binary seconds and binary milliseconds. This
property is never
used to establish
the byte
order for
text /strings with Unicode fixed-width encodings that do
not specify the byte order (UTF-16 and UTF-32). Annotation: dfdl:element, dfdl:simpleType |
|
bitOrder |
Enum Valid values 'mostSignificantBitFirst',
'leastSignificantBitFirst'. The bits of a byte each have a place value or significance of 2n, for n from 0 to 7. Hence, the byte value 255 = 27 + 26 + 25 + 24 + 23 + 22 + 21 + 20. A bit can always be unambiguously identified as the 2n-bit. The bit order is the correspondence of a bit's numeric significance to the bit position (1 to 8) within the byte. Value 'mostSignificantBitFirst' means:
Value 'leastSignificantBitFirst' means:
This property applies to all content and framing since it determines which bits of a byte occupy what bit positions. Content and framing are defined in terms of regions of the data stream, and these regions are defined in terms of the starting bit position and ending bit position; hence, dfdl:bitOrder is relevant to determining the specific bits of any grammar region (see Section 9.2) when the region's starting bit position or ending bit position are not on a byte boundary. The bit order can
only change on byte boundaries, and alignment of up to 7 bits will be skipped
(parsing) or inserted (unparsing) Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
|
encoding |
Enum or DFDL Expression Values are one of: · IANA charset name · CCSID[15] · DFDL standard encoding name · Implementation-specific encoding name This property can be computed by way of an expression which returns an appropriate string value. The expression must not contain forward references to elements which have not yet been processed. Note that there is, deliberately, no concept of 'native' encoding[16]. Conforming DFDL v1.0 processors must accept at least 'UTF-8', 'UTF-16', 'UTF-16BE', 'UTF-16LE', 'ASCII', and 'ISO-8859-1' as encoding names. Encoding names are case-insensitive, so 'utf-8' and 'UTF-8' are equivalent. Unicode character
set encodings that do not specify a byte order (such as UTF-16 or UTF-32) can
have their byte-order controlled by a document-level byte-order-mark (BOM).
See Section 11.1 Unicode
Byte Order Mark (BOM) The encoding name 'UTF-8' is interpreted strictly and does not include variants such as CESU-8. DFDL standard encoding names are defined in Section 34 Appendix D: DFDL Standard Encodings. When supported, a conforming DFDL implementation must implement them in a uniform manner so that they are portable across all DFDL implementations that implement them. Additional implementation-defined encoding names may be provided only for character set encodings for which there is no IANA name standard nor CCSID standard nor DFDL standard encoding. These implementation-defined encodings must have "X-" as a prefix to their name, as they are subject to being superseded by IANA or DFDL standard encoding names. Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
|
utf16Width |
Enum Valid values are 'fixed', 'variable'. Applies only when encoding is 'UTF-16', 'UTF-16BE', UTF16-LE' or their CCSID equivalents. Specifies whether the encoding 'UTF-16' should be treated as a fixed or variable width encoding. 'UTF-16' can contain characters which require two codepoints (called a surrogate pair) to represent. When utf16Width is 'fixed', these surrogate code points are treated as separate characters. When utf16Width is 'variable', then surrogate pairs are converted into a single character on parsing, and such a character is split into two characters on unparsing. When utf16Width is 'variable', then on parsing an un-paired surrogate codepoint causes a decode error, which can be controlled via dfdl:encodingErrorPolicy described below. Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
|
ignoreCase |
Enum Valid values are 'yes', 'no'. Whether mixed case data is accepted when matching delimiters and data values on input. This affects the behavior of matching for these properties: dfdl:initiator, dfdl:terminator, dfdl:separator, dfdl:nilValue, dfdl:textStandardExponentRep, dfdl:textStandardInfinityRep, dfdl:textStandardNaNRep, dfdl:textStandardZeroRep, dfdl:textBooleanTrueRep, and dfdl:textBooleanFalseRep. Property ignoreCase plays no part when comparing
an element value with an XSDL enum facet, matching an element value to an
XSDL pattern facet, or comparing an element value with the XSDL fixed
property. It is therefore not used by validation (when validation is
enabled), nor by the dfdl:checkConstraints function. On unparsing always use the delimiters or value as specified. Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
|
encodingErrorPolicy |
Enum Valid values are 'error' or 'replace'. This property applies whenever
dfdl:encoding is applicable. This property provides control of how decoding and encoding errors are handled when
converting the data to text, or text to data. This includes converting when
scanning for delimiters, matching regular expression length or test patterns,
matching textual data type representation patterns against the data, and of
course isolating the text content that will become the value of an element
(parsing) or constructing the content from the value (unparsing). When parsing,
an error can occur when decoding characters from their encoded form into the
DFDL Infoset character set (ISO10646). This can occur due to invalid byte
sequences, or not enough bytes found to make up the full encoding of a
character. If
'replace', then
the Unicode
replacement character (U+FFFD)
is substituted
for the
offending errors,
one replacement
character for
any incorrect
fragment of
an encoding.
If 'error' then a processing error occurs. When unparsing, the errors that can occur when encoding characters from Unicode/ISO 10646 into the specified encoding include when no mapping is provided by the encoding character set specification
and when there is not enough space to output the entire encoding of the character (e.g., need 2 bytes for a 2-byte character codepoint, but only 1 byte remains in the available length.) If 'replace' then encoding-specific replacement/substitution character is output. It is
a processing error if no such character is defined, and it is a processing
error if there is any error when attempting to output the replacement (such
as not enough room for the representation of the entire encoding of the
replacement character). If ‘error' then a processing error occurs. See Section 11.2 Character Encoding and Decoding Errors for further details. Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
Table 12 Properties Common to both Content and Framing
DFDL does
not provide any special treatment of Unicode Byte-Order Marks. They are treated
as a Unicode ZWNBS character.
When parsing, these are the errors that can
occur when decoding characters into Unicode/ISO 10646.
When unparsing, these are the errors that
can occur when encoding characters from Unicode/ISO 10646 into the specified
encoding.
The subsections below describe how these
errors are handled.
The property dfdl:encodingErrorPolicy has
two possible values: 'error' and 'replace'.
If 'error', then any error when decoding
characters while parsing causes a processing error. For unparsing, any error
when encoding characters causes a processing error.
When parsing, it does not matter if this
happens when scanning for delimiters, matching a regular expression, matching a
literal nil value, or constructing the value of a textual element.
There is one exception. When
dfdl:lengthUnits is 'bytes', the 'not enough data' decoding error is ignored,
and the data making up the fragment character is skipped over. Symmetrically,
when unparsing the 'not enough room' encoding error is ignored and the
left-over bytes are filled with the dfdl:fillByte.
Detection of
character set decoding errors is often implementation-dependent because DFDL
Implementations are free to optimize processing speed by skipping character
decoding or encoding whenever possible. For example: when
character set encodings are fixed-width, it is possible to determine lengths in
bytes or bits from the length in characters by multiplying the length value by
the character width, without having to decode any characters.
When parsing,
character decoding errors MUST be detected when
a)
the decoding results
in a character being placed into the DFDL Infoset
b)
the decoding is
necessary to identify a delimiter
c)
the decoding is
necessary to determine a match or non-match of a regular expression in a
dfdl:assert or dfdl:discriminator with testKind=’pattern’.
When unparsing,
character encoding errors MUST be detected when
a) an
unmapped character appears in the infoset value of an element.
In all other cases, character set decoding and encoding errors MAY NOT be detected.
If 'replace' then any error when decoding
characters results in the insertion of the Unicode Replacement Character
(U+FFFD) as the replacement for that error.
It does not matter if this error and
replacement happens when scanning for delimiters, matching a regular
expression, matching a literal nil value, or constructing the value of a
textual element.
There is one exception. When
dfdl:lengthUnits is 'bytes', the 'not enough data' decoding error is ignored,
no replacement character is created. The data making up the fragment character
is skipped over. (It will be filled with the dfdl:fillByte when unparsing.)
Note that the "." wildcard in
regular expressions will match the Unicode Replacement Character, so
".*" and ".+" regular expressions can potentially cause
very large matches (up to the entire data stream) to occur when data contains
errors and dfdl:encodingErrorPolicy 'replace'. DFDL Schema authors are advised
that bounded length negated regular expressions can help in this case. E.g.,
"[^\uFFFD]{0,50}" says to match any character (excluding the Unicode
Replacement Character), but only up to length 50.
It is also worth noting that the Unicode
Replacement Character can appear in data as an ordinary character, and this
cannot be distinguished from the insertion of the Unicode Replacement Character
due to a decoding error. This is likely to happen for data that is (a)
initially parsed by a DFDL parser with dfdl:encodingErrorPolicy 'replace', and
(b) which contains some decoding errors, but (c) is nevertheless successfully
parsed, (d) is written back out to a file or other data repository, and (e) is
parsed again. The written data will have replaced data errors with the Unicode
Replacement Character, and so if the data is parsed again, it will no longer
have errors, but will have the Unicode Replacement Character as a regular
character in the data.
If dfdl:lengthUnits is 'characters', then a
Unicode Replacement Character counts as contributing a single character to the
length.
If the data contains more than one adjacent
decode error, then the specific number of Unicode Replacement Characters that
are inserted as the replacement of these errors is implementation- dependent.
That is, some implementations may view, for example, three consecutive
erroneous bytes as three separate decode errors, others may view them as a
single or two decode errors. All implementations MUST, however, insert some
number of Unicode Replacement Characters, and then continue to decode
characters following the erroneous data.
The trimming of pad characters always
happens after Unicode Replacement Characters have been inserted into the data.
For unparsing, each encoding has a
replacement/substitution character specified by the ICU. This character is
substituted for the unmapped character or the character that has too large an
encoding to fit in the available space.
There is one exception. When
dfdl:lengthUnits is 'bytes', the 'not enough room' encoding error is ignored.
The left-over bytes are filled with the dfdl:fillByte (they are skipped when
parsing.)
The definitions of these substitution
characters can be conveniently found for many encodings in the ICU Converter
Explorer (http://demo.icu-project.org/icu-bin/convexp).
An encoding error is a processing error if
the encoding does not provide a substitution/replacement character definition.
(This would be rare, but could occur if a DFDL implementation allows many encodings
beyond the minimum set.)
The following specific situations involving
encodings UTF-16, UTF-16LE, and UTF-16BE when dfdl:utf16Width
"fixed", and they do not cause a decoding or encoding error.
In all these cases the code-point(s) becomes
a character code in the DFDL Information Item for the string.
There can be situations where data wants to be preserved exactly even if
it contains errors.
It is suggested that if a DFDL schema author wants to preserve information containing data where the encodings have these kinds of errors, that they model such data as xs:hexBinary, or as xs:string but using an encoding such as iso-8859-1 which preserves all bytes.
Byte order and bit order are separate concepts. However, of the possible combinations, only the following are allowed:
Other combinations must produce Schema Definition Errors.
Consider a structure of 4 logical elements. The total length is 16 bits. Assume dfdl:lengthUnits is 'bits', dfdl:representation is 'binary', dfdl:binaryNumberRep is 'binary':
<element name="A" type="xs:int"
dfdl:length="3"/> <!-- having value 3 -->
<element
name="B" type="xs:int" dfdl:length="7"/>
<!-- having value 9 -->
<element name="C" type="xs:int"
dfdl:length="4"/> <!-- having value 5 -->
<element name="D"
type="xs:int" dfdl:length="2"/> <!-- having value 1
-->
The above are colorized so as to highlight the corresponding bits in the data below.
In a format where dfdl:bitOrder is 'mostSignificantBitFirst':
01100010
01010101
AAABBBBB
BBCCCCDD
Significance M L M L
Bit
Position 12345678 12345678
Byte
Position ----1--- ----2---
As presented here, the bits corresponding to each element appear left to right, and all bits for an individual element are adjacent. Within the bits of an individual element the most significant bit is on the left, least significant on the right, consistent with the way the bytes themselves are presented.
In contrast, in a format where dfdl:bitOrder is 'leastSignificantBitFirst':
01001011 01010100
BBBBBAAA DDCCCCBB
Bit
Position 87654321 87654321
Byte
Position ----1--- ----2---
In the above presentation note how the bits of the element 'B' do not appear adjacent to each other. The most significant bits of byte N are adjacent to the least significant bits of byte N+1.
When working exclusively with data having dfdl:bitOrder 'leastSignificantBitFirst', it is useful to present data with bytes Right to Left. That is, with the bytes starting at byte 1 on the right, and increasing to the left.
01010100 01001011
DDCCCCBB
BBBBBAAA
Significance M L M L
Bit
Position 87654321 87654321
With this reorientation, the bits of the element 'B' are once again displayed adjacently. Within the bits of an individual element the most significant bit is on the left, least significant on the right, consistent with the way the bytes themselves are presented.
Often the specification documents for data formats that with least-significant-bit-first bit order will describe data using this Right-to-Left presentation style.
When any grammar region
appears before (to the left of) or after (to the right of) another grammar
region in the grammar rules of Section 9.2, and the boundary between the two
falls within a byte rather than on a byte boundary, then the dfdl:bitOrder determines
which bits are occupied by the regions.
In general, the notion of before means occupying lower-numbered bit positions, and the bit positions are numbered according to dfdl:bitOrder. Hence, when dfdl:bitOrder is 'mostSignificantBitFirst', grammar regions that are before, will occupy more-significant bits, and when dfdl:bitOrder is 'leastSignificantBitFirst', grammar regions that are before will occupy less-significant bits.
Several properties are common across the various framing styles or are used to distinguish them. Generally these have to do with position and length for text, bit fields, or opaque data.
Alignment properties control the leading
alignment and trailing alignment regions.
When the alignment properties are applied to
an array element, the properties are applied to each occurrence of the element;
that is, not only to the first occurrence.
The following properties are used to define
alignment rules.
|
Property
Name |
Description |
|
alignment |
Non-negative Integer or 'implicit' A non-negative number that gives the alignment required for the beginning of the item. If alignment is needed then the size of the AlignmentFill grammar region will be non-zero if the item must be aligned to a boundary. 'implicit' specifies that the natural alignment for the representation type is used. See the table of implicit alignments Table 14 Implicit Alignment in bits for simple elements. The 'implicit' alignment of a complex element is the alignment of its model group. The 'implicit' alignment of a model group is always 1. If alignment is 'implicit' then dfdl:alignmentUnits is ignored. For textual data, minimum alignment is mandated by the character-set encoding, and this property must be 'implicit' or set to a multiple of the character-set's mandatory alignment. See Section 12.1.2. Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
|
alignmentUnits |
Enum Valid values are 'bits' or 'bytes' Scales the alignment so alignment can be specified in either units of bits or units of bytes. Only used when dfdl:alignment not 'implicit' Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
|
fillByte |
DFDL String Literal A single byte specified as a DFDL byte value entity or a single character. If a character is specified, it must be a single-byte character in the applicable encoding. Used on unparsing to fill empty space such as between two aligned elements. Used to fill these regions specified in the grammar: RightFill, ElementUnused, ChoiceUnused, LeadingSkip, AlignmentFill, and TrailingSkip. Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
|
leadingSkip |
Non-negative Integer A non-negative number of bytes or bits, depending on dfdl:alignmentUnits, to skip before alignment is applied. Gives the size of the grammar region having the same name. Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
|
trailingSkip |
Non-negative Integer A non-negative
number of bytes or bits, depending on dfdl:alignmentUnits, to skip after the
element, but before considering the alignment of the next element.
Gives the size of the grammar region having the same name. If dfdl:trailingSkip is specified when dfdl:lengthKind is 'delimited' then a dfdl:terminator must be specified. Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
Table 13 Aligned Data Properties
There are two properties which control the data alignment by controlling the length of the AlignmentFill region
An element's representation is aligned to N units if P is the first position in the representation and P mod N = 1. When parsing, the position of the first unit of the data stream is 1.
For example, if dfdl:alignment is 4, and dfdl:alignmentUnits is 'bytes', then the element's representation must begin at 1 or 1 plus a multiple of 4 bytes. That is, 1, 5, 9, 13, 17 and so on.
The length of the AlignmentFill
region is measured in bits. If alignmentUnits is 'bytes' then we multiply the
alignment value by 8 to get the bit alignment, If the position in the data stream of
the start of the AlignmentFill region is bit position N, then the length
of the AlignmentFill region is the smallest non-negative integer L such
that (L + N) mod B = 1. The position of the first bit of the aligned
component is P = L + N.
The LeadingSkip and TrailingSkip regions length are controlled by two properties of corresponding names and the dfdl:alignmentUnits property.
When dfdl:alignment is 'implicit' the following alignment values are applied for each logical type.
|
Type |
Alignment |
||
|
text |
binary |
||
|
String |
Encoding Specific (usually 8 bits, with exceptions: See Section 12.1.2) |
Not applicable |
|
|
Float |
32 |
||
|
Double |
64 |
||
|
Decimal, Integer, nonNegativeInteger |
Packed decimals: 8 |
binary: 8 |
|
|
Long, UnsignedLong |
binary: 64 |
||
|
Int, UnsignedInt |
binary: 32 |
||
|
Short, UnsignedShort |
binary: 16 |
||
|
Byte, UnsignedByte |
binary: 8 |
||
|
DateTime |
binarySeconds: 32, binaryMilliseconds:64 |
||
|
Date |
binarySeconds: 32, binaryMilliseconds:64 |
||
|
Time |
binarySeconds: 32, binaryMilliseconds:64 |
||
|
Boolean |
32 |
||
|
HexBinary |
Not applicable |
8 |
Table 14 Implicit Alignment in bits
Note: The above table specifies the implicit alignment in bits, but this does not imply that dfdl:alignmentUnits 'bits' can be specified for all simple types. Rather, dfdl:alignmentUnits and dfdl:lengthUnits are independent and have their own rules for when they are applicable.
The term textual data is used to describe data of type xs:string, data with dfdl:representation "text", as well as data being matched to delimiters (parsing) or output as delimiters (unparsing), and data being matched to regular expressions (parsing only - as in a dfdl:assert with testKind 'pattern', or an element with dfdl:lengthKind 'pattern').
Textual data has mandatory alignment that is character-set-encoding dependent. That is, these mandates come from the character set encoding specified by the dfdl:encoding property.
When processing textual data, it is a Schema Definition Error if the dfdl:alignment and dfdl:alignmentUnits properties are used to specify alignment that is not a multiple of the encoding-specified mandatory alignment.
If the data is not aligned to the proper boundary for the encoding when textual data is processed, then bits are skipped (parsing) or filled from dfdl:fillByte (unparsing) to achieve the mandatory alignment.
All required character set encodings in DFDL have 8-bit/1-byte alignment.
DFDL standard encodings specify their alignment. See Section 34 Appendix D: DFDL Standard Encodings.
Some implementations
may include additional implementation-defined encodings which have other
alignments.
Note the 16-bit and 32-bit Unicode character set encodings UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, UTF-32LE, all have 8-bit/1-byte alignment.
Packed decimal data must have a multiple of 4-bit alignment. It is a Schema Definition Error otherwise.
When dfdl:alignmentUnits
is 'bits', and the dfdl:alignment is not a multiple of 8, then the
dfdl:bitOrder property affects the alignment by controlling which bits are
skipped as part of the grammar AlignmentFill region.
In general, the AlignmentFill region is before the regions it is aligning, and within a byte, the meaning of 'before' is interpreted with respect to the dfdl:bitOrder.
When dfdl:bitOrder is 'mostSignificantBitFirst', then bits with more significance are before bits with less significance, so the AlignmentFill region occupies the most significant bits of the byte.
When dfdl:bitOrder is 'leastSignificantBitFirst', then bits with less significance are before bits with more significance, so the AlignmentFill region occupies the least significant bits of the byte.
Consider a structure of 2 logical elements. Assume dfdl:lengthUnits='bits', dfdl:representation='binary', dfdl:binaryNumberRep='binary' dfdl:alignmentUnits='bits', and assume the data is at the begining of the data stream.
<element name="A" type="xs:int"
dfdl:length="2" dfdl:alignment='8'/>
<!-- having value 1 -->
<element
name="B" type="xs:int" dfdl:length="4"
dfdl:alignment='4'/>
<!--
having value 5 -->
The above are colorized so as to highlight the corresponding bits in the data below. The total length due to the alignment region appearing before element 'B' will be 8 bits.
In a format where dfdl:bitOrder is 'mostSignificantBitFirst' the data can be visualized as:
01000101
AAxxBBBB
Significance M L
Bit
Position 12345678
In the above, the AlignmentFill region is marked with 'x' characters, and contains all 0 bit values.
In a format where dfdl:bitOrder is 'leastSignificantBitFirst' the presentation is different:
01010001
BBBBxxAA
Significance M L
Bit
Position 87654321
In the above the AlignmentFill region still appears before element 'B', and in this case that is in less significant bits of the byte than the bits of content of element 'B', and these bits are displayed to the right of the bits of element 'B'.
The following properties apply to all objects that use text delimiters to delimit, that is, to initiate and/or terminate data. Delimiters can apply to binary data; however they are most often called 'text' delimiters because the concept is much more commonly used for textual data formats.
|
Property
Name |
Description |
|
initiator |
List of DFDL String Literals or DFDL Expression Specifies
a whitespace separated list of alternative DFDL String Literals one of which
marks the beginning of the element or group of elements. This
property can be computed by way of an expression which returns a string
containing a whitespace separated list of DFDL
String Literals. The expression must not contain
forward references to elements which have not yet been processed. Each
string literal in the list, whether apparent in the schema, or returned as
the value of an expression, is restricted to allow only certain kinds of DFDL
String Literal syntax: ·
DFDL
character entities are allowed. ·
DFDL
Byte Value entities ( %#rXX; ) are allowed. ·
DFDL
Character Classes NL, WSP, WSP+, WSP*, and ES are allowed.
·
If
the ES entity or the WSP* entity appear alone as one of the string literals
in the list, then dfdl:initiatedContent must be "no" . If the above rules are not followed it
is a Schema Definition Error. The Initiator region contains one of the initiator strings defined by dfdl:initiator. When parsing, the list of values is processed in a greedy manner, meaning it takes all the initiators, that is, each of the string literals in the whitespace separated list, and matches
them each against the data. The initiator with the longest match is the one
that is selected as having been 'found'. Once a matching
initiator is found, no other matches will be subsequently attempted (ie, there is no backtracking). When an initiator is specified, it is a processing error if the component is required and one of the values is not found. If dfdl:initiator is "" (the empty string), then the Initiator region is of length zero, and no initiator is expected. It is not permitted for an expression to return an empty string. That is a Schema Definition Error. On unparsing the first initiator in the list is automatically inserted into the Initiator region. If
dfdl:ignoreCase is 'yes' then the case of the string is ignored by the
parser. Annotation:
dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
|
terminator |
List of DFDL
String Literals or DFDL Expression Specifies
a whitespace separated list of alternative text strings that one of which
marks the end of an element or group of elements. The strings MUST be
searched for in the longest first order. This
property can be computed by way of an expression which returns a string of
whitespace separated list of values.
The expression must not contain forward references to elements which
have not yet been processed. This property can be used to determine the length of an element as described in Section 12.3.2 dfdl:lengthKind 'delimited'. Each
string literal in the list, whether apparent in the schema, or returned as
the value of an expression, is restricted to allow only certain kinds of DFDL
String Literal syntax: ·
DFDL
character entities are allowed. ·
DFDL
Byte Value entities ( %#rXX; ) are allowed. ·
DFDL
Character Classes NL, WSP, WSP+, WSP*, and ES are allowed.
·
Neither
the ES entity nor the WSP* entity may appear on their own as one of the
string literals in the list when the parser is determining the length of a
component by scanning for delimiters. If the
above rules are not followed it is a Schema Definition Error. The Terminator
region contains the terminator string. If dfdl:terminator is "" (the empty string), then the terminator region is of length zero, and no terminator is expected. It is not permitted for an expression to return an empty string, that is a Schema Definition Error. When parsing, the list of values is processed in a greedy manner, meaning it takes all the terminators, that is, each of the string literals in the whitespace separated list, and matches
them each against the data. The terminator with the longest match is the one
that is selected as having been 'found'. Once a matching
terminator is found, no other matches will be subsequently attempted (ie, there is no backtracking). When a
terminator is expected it is a processing error if no matching terminator is
found. However, if dfdl:documentFinalTerminatorCanBeMissing is specified then
it is not an error if the last terminator in the data stream is not found. On
unparsing the first terminator in the list is automatically inserted in the
Terminator region. If
dfdl:ignoreCase is 'yes' then the case of the string is ignored by the
parser. Annotation:
dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
|
emptyValueDelimiterPolicy |
Enum Valid
values are 'none', 'initiator', 'terminator' or 'both' Indicates
that when an element in the data stream is empty, an initiator (if one is defined),
a terminator (if one is defined), both an initiator and a terminator (if
defined) or neither must be present. Ignored
if both dfdl:initiator and dfdl:terminator are "" (empty string). 'initiator'
indicates that, on parsing, if the content region (which can be either the
SimpleContent region or the ComplexContent region defined in Section
9.2) is empty then the
dfdl:initiator must be present. It also indicates that on unparsing when the
content region is empty that the dfdl:initiator will be output. 'terminator'
indicates that, on parsing, if the content region is empty then the
dfdl:terminator must be present. It also indicates that on unparsing when the
content region is empty the dfdl:terminator will be output. 'both'
indicates that, on parsing, if
the content region is empty both the dfdl:initiator and dfdl:terminator must
be present. On unparsing when the content region is empty the dfdl:initiator
followed by the dfdl:terminator will be output. 'none'
indicates that if the content region is empty neither the dfdl:initiator or
dfdl:terminator must be present. On unparsing when the content region is
empty nothing will be output. It is a
Schema Definition Error if dfdl:emptyValueDelimiterPolicy set to 'none' or
'terminator' when the parent group has dfdl:initiatedContent 'yes'. This property
plays an important role in establishing empty representation. See 9.2.2 Empty Representation for details. This property is ignored if the element is fixed-length
and length is not zero (as no empty representation is possible). The value of dfdl:emptyValueDelimiterPolicy
should only be checked if there is a dfdl:initiator or dfdl:terminator in
scope. If so, and dfdl:emptyValueDelimiterPolicy is not set, it is a Schema
Definition Error. If dfdl:initiator is not "" and
dfdl:terminator is "" and dfdl:emptyValueDelimiterPolicy is
'terminator' it is a Schema Definition Error. If
dfdl:terminator is not "" and dfdl:initiator is "” and
dfdl:emptyValueDelimiterPolicy is 'initiator' it is a Schema
Definition Error. It is not an SDE if dfdl:emptyValueDelimiterPolicy
is 'both' and one or both of dfdl:initiator and dfdl:terminator is
"". This is to accommodate the common use of setting 'both' as a
schema-wide setting. It is a Schema Definition Error
if dfdl:emptyValueDelimiterPolicy is
in effect and is set to 'none' or 'terminator' when the parent
xs:sequence has dfdl:initiatedContent 'yes'. Annotation:
dfdl:element, dfdl:simpleType |
|
documentFinalTerminatorCanBeMissing |
Enum Valid
values are 'yes', 'no' When
the dfdl:documentFinalTerminatorCanBeMissing property is true, then when an
element is the last element in the data stream, then on parsing, it is not an
error if the terminator is not found. For
example, if the data are in a file, and the format specifies lines terminated
by the newline character (typically LF or CRLF), then if the last line is
missing its newline, then this would normally be an error, but if
dfdl:documentFinalTerminatorCanBeMissing is true, then this is not a
processing error. On unparsing
the terminator is always written out regardless of the state of this
property. Annotation:
dfdl:format (but applies to elements only) |
|
outputNewLine |
DFDL String Literal or
DFDL Expression Specifies the
character or characters that will be used to replace the %NL; character class
entity during unparse It is a Schema
Definition Error if any of the characters are not in the set of characters
allowed by the DFDL entity %NL; Only individual characters or the %CR;%LF;
combination are allowed. It is a Schema
Definition Error if the DFDL entity %NL; is specified This
property can be computed by way of an expression which returns a DFDL string
literal. The expression must not contain forward references to elements which
have not yet been processed. Annotation: dfdl:element, dfdl:simpleType, dfdl:sequence, dfdl:choice, dfdl:group |
Table 15 Properties for Specifying Delimiters
These properties are used to determine the content length of an element and apply to elements of all types (simple and complex).
|
Property Name |
Description |
|
lengthKind |
Enum Controls how the content length of the component is determined. Valid values are: 'explicit', 'delimited', 'prefixed', 'implicit', 'pattern', 'endOfParent' A full description of each enumeration is given in the later sections. 'explicit' means
the length of the element is given by the dfdl:length property. 'delimited' means the element length is determined by scanning for a terminator or separator. 'prefixed' means the length of the element is given by an immediately preceding PrefixLength data region the format of which is specified using dfdl:prefixLengthType. 'implicit means the length is to be determined in terms of the type of the element and its schema-specified properties if any. 'pattern' means the length of the element is given by scanning for a regular expression specified using the dfdl:lengthPattern property. 'endOfParent' means that the length extends to the end of the containing (parent) construct. Annotation: dfdl:element, dfdl:simpleType |
|
lengthUnits |
Enum Valid values 'bytes', 'characters', ''bits'. Specifies the units to be used whenever a length is being used to extract or write data. Applicable when dfdl:lengthKind is 'explicit', 'implicit' (for xs:string and xs:hexBinary) or 'prefixed'. Usage is restricted as follows:
Annotation: dfdl:element, dfdl:simpleType |
Table 16 Properties for Specifying Length
When dfdl:lengthKind is 'explicit' the length of the item is given by the dfdl:length property.
When the value of the dfdl:length property is a constant, it is used both when parsing and unparsing.
When unparsing an element with dfdl:lengthKind
'explicit' and where dfdl:length is an expression, then the data in the Infoset
is treated as variable length and not fixed-length and the dfdl:length
property, whether literal constant or expression, is
evaluated to provide the length to use. .
The behaviour is the same as dfdl:lengthKind 'prefixed'.
See Section 12.3.4.
When
parsing and dfdl:lengthKind is 'explicit', delimiter scanning is turned off and in-scope delimiters are not looked for within or between elements.
|
Property
Name |
Description |
|
length |
Non-negative Integer or DFDL Expression. Only used when lengthKind is 'explicit'. Specifies the length of this element in units that are specified by the dfdl:lengthUnits property. This property can be computed by way of an expression which returns a non-negative integer. The expression must not contain forward references to elements which have not yet been processed. Annotation: dfdl:element, dfdl:simpleType |
Table 17 The dfdl:length Property
When dfdl:lengthKind 'explicit', the method of extracting data is described in section: 12.3.7 Elements of Specified Length
On parsing, the length of an element with dfdl:lengthKind 'delimited' is determined by scanning the datastream for the delimiter.
The data stream is scanned for any of
· the element's terminator (if specified)
· an enclosing construct's separator or terminator
· the end of an enclosing element designated by its known length
· the end of the data stream
dfdl:lengthKind 'delimited' may be specified for
· elements of simple type with text representation
· elements of number or calendar simple type with dfdl:representation 'binary' that have a packed decimal representation
· elements of type xs:hexBinary
· elements of complex type.
The rules for resolving ambiguity between delimiters are:
When unparsing a simple element with text representation, the length in the data stream is the length of the content region, padded to dfdl:textOutputMinLength or the XSD minLength facet if dfdl:textPadKind is 'padChar'.
When unparsing a simple element with binary representation, then for hexBinary the length is the number of bytes in the infoset value padded to the XSD minLength facet value using dfdl:fillByte, and for the other types the length is the minimum number of bytes to represent the value and any sign.
When unparsing a complex element, the length is that of the ComplexContent region.
When a simple or complex element has a specified length,dfdl:lengthKind 'pattern', or dfdl:lengthKind 'endOfParent' then delimiter scanning is suspended for the duration of the processing of that element.
This allows formats to be parsed which are delimited, but have nested elements which contain non-character data so long as that nested data can be isolated from the delimited data context surrounding it.
Formats involving binary data, most notably packed decimals, can use delimiter scanning but care must be taken that the delimiters cannot match data represented in these formats. In particular, the delimiters must be chosen with knowledge that BCD data can contain any byte both of whose nibbles are 0 to 9 (that is, excluding A to F). Packed data adds bytes with a sign indicator, that is, a nibble in the range A to F.
General binary data can contain any bit
pattern whatsoever, so delimiter scanning for numbers and calendars with
dfdl:representation 'binary' are disallowed, with the specific exception of
packed decimals. Delimiter scanning is also allowed for type xs:hexBinary.
Implementation Note: Scanning for delimiters when data is binary, or when using byte-value (aka raw byte) entities in delimiters, means that a simple character based delimiter scanner is not sufficient, as the delimiter may not be representable as characters.
When dfdl:lengthKind is 'implicit', the length is determined in terms of the type of the element and its schema-specified properties.
For complex elements,
'implicit' means the length is determined by the combined lengths of the contained
children, that is the ComplexValue region, and the ElementUnused region is of
size 0. However, note that alignment regions inside the contained children
within the ComplexValue region may be of different lengths depending on the
ComplexValue's starting position alignment.
For complex elements,
'implicit' means the length is determined by the combined lengths of the
contained children, that is the ComplexContent
region. However, note that alignment regions inside the
ComplexContent region may be of different lengths depending on the
ComplexContent's starting position alignment.
For simple elements
the length is fixed and is given in Table
18 Length in Bits for SimpleTypes when
dfdl:lengthKind is 'implicit' Table
18 Length in Bits for SimpleTypes when
dfdl:lengthKind is 'implicit'.
|
Type |
Length |
||
|
text |
binary |
||
|
String |
The XSD maxLength facet gives length in
characters, but this is also the length in bytes. (See note below: character
set encoding must be single-byte.) Multply by 8 to get number of bits. |
Not applicable |
|
|
Float |
Not allowed |
32 bits |
|
|
Double |
Not allowed |
64 bits |
|
|
Decimal,
Integer, nonNegativeInteger |
Not allowed |
packed
decimal: Not allowed |
binary: Not allowed |
|
Long,
UnsignedLong |
Not allowed |
binary: 64 bits |
|
|
Int, UnsignedInt |
Not allowed |
binary: 32 bits |
|
|
Short,
UnsignedShort |
Not allowed |
binary: 16 bits |
|
|
Byte,
UnsignedByte |
Not allowed |
binary: 8 bits |
|
|
DateTime |
Not allowed |
binarySeconds: 32 bits,
binaryMilliseconds: 64 bits. |
|
|
Date |
Not allowed |
binarySeconds: Not allowed |
|
|
Time |
Not allowed |
binarySeconds: Not allowed |
|
|
Boolean |
Length of longest of dfdl:textBooleanTrueRep and
dfdl:textBooleanFalseRep values |
32 bits |
|
|
HexBinary |
Not applicable |
The XSD maxLength facet gives the length
in bytes. Multiply by 8 to convert to number of bits. |
Table 18 Length in Bits for SimpleTypes when
dfdl:lengthKind is 'implicit'
When dfdl:lengthKind is 'implicit', the
method of extracting data is described in section: 12.3.7 Elements of Specified Length.
It is a Schema
Definition Error if type is xs:string and dfdl:lengthKind is 'implicit' and
dfdl:lengthUnits is 'bytes' and encoding is not an SBCS (exactly 1 byte per
character code) encoding. This prevents a scenario where validation against the
XSD maxLength facet is in characters but parsing and unparsing using the XSD
maxLength facet is in bytes.
When dfdl:lengthKind is 'prefixed' the
length of the element is given by the integer value of the PrefixLength region specified using
dfdl:prefixLengthType. The property dfdl:prefixIncludesPrefixLength also can be
used to adjust the length appropriately.
When dfdl:lengthKind is 'prefixed' the
method of extracting data is described in section: 12.3.7 Elements of Specified Length
When dfdl:lengthKind is 'prefixed', delimiter scanning is turned off and in-scope delimiters are not looked for within or between elements.
|
Property Name |
Description |
|
prefixIncludesPrefixLength |
Enum Valid values are 'yes', 'no' Whether the length given by a prefix
includes the length of the prefix as well as the length of the content region
(which can be either the SimpleContent region or the ComplexContent region
defined in Section 9.2 DFDL Data Syntax Grammar).) Used only when dfdl:lengthKind 'prefixed'. Annotation: dfdl:element, dfdl:simpleType |
|
prefixLengthType |
QName Name of a simple type derived from
xs:integer or any subtype of it. This type, with its DFDL annotations
specifies the representation of the length prefix, which is in the
PrefixLength region. It is a Schema Definition Error if the xs:simpleType specifies any of:
Annotation: dfdl:element, dfdl:simpleType |
Table 19 Properties for dfdl:lengthKind 'prefixed'
The representation of the element is in two parts.
When parsing, the length of the element's content is obtained by parsing the simple type specified by dfdl:prefixLengthType to obtain an integer value. Note that all required properties must be present on the specified simple type or defaulted because there is no element declaration to supply any missing required properties.
If the dfdl:prefixIncludesPrefixLength
property is 'yes' then the length of the element's content is the value of the
prefix length minus the length of the content of the prefix length.
If the prefix type is dfdl:lengthKind
'implicit' or 'explicit' then the dfdl:lengthUnits properties of both the
prefix type and the element must be the same.
The
DFDL properties that specify the format of the prefix come from annotations
directly on the dfdl:prefixLengthType's type definition, and from the default
format annotation for the schema document containing the definition of that
type. . If the using
element resides in a separate schema, the simple type does not pick up values
from the element's schema's default dfdl:format annotation.
When unparsing, the length of the element's content region must be determined first as described below. Then the value of the prefix length must be adjusted using dfdl:prefixIncludesPrefixLength.
Then the prefix length can be written to the data stream using the properties on the dfdl:prefixLengthType, and finally the element's content can be written to the data stream.
Consider this example:
<xs:element name="myString" type="xs:string"
dfdl:lengthKind="prefixed"
dfdl:prefixIncludesPrefixLength="no"
dfdl:prefixLengthType="packed3"/>
<xs:simpleType name="packed3"
dfdl:representation="binary"
dfdl:binaryNumberRep="packed"
dfdl:lengthKind="explicit"
dfdl:length="2" >
<xs:restriction base="integer" />
</xs:simpleType>
In the above, the string has a prefix length
of type 'packed3' containing 3 packed decimal digits.
The property dfdl:prefixIncludesPrefixLength
is an enumeration which allows the length computation to be varied to include
or exclude the length of the prefix element itself.
The prefix length's value contains the
length measured in units given by dfdl:lengthUnits.
When parsing, if the dfdl:lengthUnits are
bits, then any number of bits can be in the representation.However, the same is
not true when unparsing. The DFDL Infoset does not store the number of bits in
a number, so the number of bits will always be a multiple of 8 bits.
When unparsing, the value of the prefix is
computed automatically by obtaining the length of the element's content.
For a simple element with text
representation, the length is computed as for dfdl:lengthKind 'delimited'.
For a simple element with binary
representation, the length is given in the table below.
For a complex element, the length is that of
the ComplexContent region.
|
Type |
Length
|
|
|
String |
Not applicable |
|
|
Float |
32 |
|
|
Double |
64 |
|
|
Decimal,
Integer, NonNegativeInteger |
Compute the minimum number of bytes to
represent the value (per dfdl:binaryNumberRep) and sign (if applicable).
Multiply by 8 for number of bits. |
|
|
Long,
UnsignedLong |
packed decimal: as Decimal |
binary: 64 |
|
Int, UnsignedInt |
binary: 32 |
|
|
Short,
UnsignedShort |
binary: 16 |
|
|
Byte,
UnsignedByte |
binary: 8 |
|
|
DateTime |
binarySeconds: 32, binaryMilliseconds:64 |
|
|
Date |
binarySeconds: Not allowed |
|
|
Time |
binarySeconds: Not allowed |
|
|
Boolean |
32 |
|
|
HexBinary |
Compute the number of
bytes in the infoset value padded to the value of the XSD minLength facet
(which gives minimum length in bytes) using dfdl:fillByte if necessary. This
gives the unparse length in bytes. Multiply by 8 for the number of bits. |
Table 20 Unparse Lengths (in Bits) for Binary Data with dfdl:lengthKind 'prefixed'
It is possible
for a prefix length, as specified by dfdl:prefixLengthType, to itself have a
prefix length
It is a Schema
Definition Error if this nesting exceeds 1 deep. That is, an element can have a
prefix length, which defines a PrefixLength region (see Section 9.2 DFDL Data Syntax Grammar). The
PrefixLength region can itself have a type which also specifies a prefix
length, thereby defining a PrefixPrefixLength region. It is a Schema Definition
Error unless the type associated with the PrefixPrefixLength is different from
the type associated with the PrefixLength.
The dfdl:lengthKind 'pattern' means the length of the element is given by a regular expression specified using the dfdl:lengthPattern property. The DFDL processor scans the data stream to determine a string value that is the match to a regular expression. The pattern is only used on parsing.
When dfdl:lengthKind is 'pattern', delimiter scanning is turned off and in-scope delimiters are not looked for within or between elements.
|
Property Name |
Description |
|
lengthPattern |
DFDL Regular Expression. Only used when lengthKind is 'pattern'. Specifies a regular expression that, on parsing, is executed against the datastream to determine the length of the element. The data stream beginning at the starting offset of the content region (which can be either the SimpleContent region or the ComplexContent region defined in Section 9.2 DFDL Data Syntax Grammar) of the element is interpreted as a stream of characters in the encoding of the element, and the regular expression contained in the dfdl:lengthPattern property is executed against that stream of characters. When the element is complex the encoding used is the dfdl:encoding of the complex element itself. It is a Schema Definition Error
if there is no value for the dfdl:encoding property in scope. DFDL Escape Schemes (per dfdl:escapeSchemeRef) are not used when executing the regular expression. If the pattern matching of the regular expression reads data that cannot be decoded into characters of the current encoding, then the behavior is controlled by the dfdl:encodingErrorPolicy property. See dfdl:encodingErrorPolicy in Section 11 Properties Common to both Content and Framing. Annotation: dfdl:element, dfdl:simpleType |
Table 21 The dfdl:lengthPattern Property
On unparsing the behavior is the same as for dfdl:lengthKind 'prefixed'.
When the DFDL regular expression is matched against
data:
·
The data is considered to be text in the
character set encoding specified by the dfdl:encoding property, regardless of
the actual representation of the element.
·
The data is decoded from the specified encoding
into Unicode before the actual matching takes place.
·
If there is no match (ie, the length of the data
found to match the pattern is zero) it is not a processing error but instead it
means the length is zero.
The dfdl:lengthKind 'endOfParent' means that
the element is terminated either by the end of the data stream, or the end of
an enclosing complex element with dfdl:lengthKind ‘explicit’,
‘pattern’, ‘prefixed’ or ‘endOfParent’, or
the end of an enclosing choice with dfdl:choiceLengthKind
‘explicit’. The ‘parent’ element or choice does not
have to be the immediate enclosing component of the element, but there must be
no other components defined between the element specifying dfdl:lengthKind
'endOfParent' and the end of the parent.
A convenient way of describing the parent is as a 'box',
being defined as a portion of the data stream that has an established content
length prior to the parsing of its children. If the parent is such a ‘box’ then
the element specifying dfdl:lengthKind ‘endOfParent’ is the last
element in the ‘box’ and its content extends to the end of the
‘box’.
A dfdl:lengthKind
of 'endOfParent' can only be used
on simple and complex elements in the following locations:
It is a Schema
Definition Error if:
The effective length units of the parent are:
· dfdl:lengthUnits if parent is an element with dfdl:lengthKind ‘explicit’ or ‘prefixed’;
· ‘characters’ if parent is an element with dfdl:lengthKind ‘pattern’;
· ‘bytes’ if parent is a choice with dfdl:choiceLengthKind ‘explicit’;
· ‘characters’ if the element is the document root;
· the effective length units of the parent’s parent if parent is an element with dfdl:lengthKind ‘endOfParent’
If the element is in a sequence then it is a Schema Definition Error if:
If the element is in a choice where dfdl:choiceLengthKind is 'implicit' then it is a Schema Definition Error if:
A simple element
must have one of:
A complex element
can have dfdl:lengthKind 'endOfParent'. If so then its last child element can
be any dfdl:lengthKind including 'endOfParent'.
The
dfdl:lengthKind 'endOfParent' can also be used on the document root to allow
the last element to consume the data up to the end of the data stream.
The
use of dfdl:lengthKind ‘endOfParent’ is distinct from the situation
where the length of the last element in the parent is known but is not sufficient
to fill the parent. In the latter case the remaining data are ignored on
parsing and filled with dfdl:fillByte on unparsing.
When parsing an
element with dfdl:lengthKind ‘endOfParent’, delimiter scanning is
turned off and in-scope terminating delimiters are not looked for within the
element.
When unparsing an element with dfdl:lengthKind ‘endOfParent’, if the parent is a complex element with dfdl:lengthKind 'explicit' where dfdl:length is not an expression, or a choice with dfdl:choiceLengthKind 'explicit', then the element with dfdl:lengthKind 'endOfParent' is padded or filled in the usual manner to the required length, by completing the LeftPadding, RightPadOrFill, ElementUnused, or ChoiceUnused regions of the data as appropriate.
An element has a specified length when dfdl:lengthKind is 'explicit', 'implicit' (simple type only) or 'prefixed'. The units that the length represents are specified by the dfdl:lengthUnits property except where noted in Section 12.3.3.
Using specified length, it is possible for
an element to have content length longer than needed to represent just the data
value. For example, a simple text element may be padded in the RightPadding region if the data is not long enough.
When
an element has specified length, but appears inside a complex type element
having delimited length kind, delimiter scanning is turned off and in-scope delimiters are not looked for within or between elements.
An element of specified length with
dfdl:lengthKind 'implicit' or 'explicit' where dfdl:length is not an expression
has a known length when unparsing.
However, anAn
element of specified length with dfdl:lengthKind 'prefixed' or 'explicit' where
dfdl:length is an expression is considered to have a variable
length when unparsing.Specifically, the processor
automatically determines the value to store in the prefix, based on the length
of the infoset element, and the properties which modify the interpretation of
the prefix length value, such as dfdl:prefixIncludesPrefixLength. [MJB2]
For For
dfdl:lengthKind 'explicit' (expression), whether
parsing or unparsing the expression is
evaluated to obtain the length. When unparsing the processor cannot
automatically determine in what way the length information is to be stored as
it comes from an expression that may access one or more elements and perform
any calculation.Hence, n. Normally
the value of the the element
or elements involved in the length calculation would
be computed using dfdl:outputValueCalc, using an expression that measures the
length of the element using by way of functions
such as dfdl:contentLength or dfdl:valueLength.
1.
For dfdl:lengthKind
'prefixed' the processor automatically determines the value to
store in the prefix, based on the length of the infoset element, and the
properties which modify the interpretation of the prefix length value, such as
dfdl:prefixIncludesPrefixLength.
When parsing, if the data stream ends without enough data to parse an element, that is, N bits are needed based on the dfdl:length, but only M < N bits are available, then it is a processing error.
If dfdl:lengthUnits is 'characters' then the length (in bits) of the content region (i.e., SimpleContent or ComplexContent defined in Section 9.2 DFDL Data Syntax Grammar) will depend on the encoding of the characters.
For a simple element, dfdl:lengthUnits 'characters' may only be used for textual elements, it is a Schema Definition Error otherwise.
Some DFDL implementations may support character set encodings where the characters are not a multiple of 8-bits wide. Encodings which are 5, 6, 7, and 9 bits wide are rare, but do exist, so the overall length of the content region may not be a multiple of 8-bits wide.
Textual data is defined to mean either data of type string or data where the dfdl:representation property is 'text'.
For a textual element, the dfdl:lengthUnits property can be either 'bytes' or 'characters'.
If a textual element has dfdl:lengthUnits of 'bytes', and the dfdl:encoding is not SBCS, then it is possible for a partial character encoding to appear after the code units of the characters. In this case, the following rules apply:
It is a Schema Definition Error if type is xs:string and dfdl:textPadKind is not 'none' and dfdl:lengthUnits is 'bytes' and dfdl:encoding is not an SBCS encoding and the XSD minLength facet is not zero. This prevents a scenario where validation against the XSD minLength facet is in characters but padding would be performed in bytes.
This
section discusses the dfdl:lengthKind 'explicit' and 'prefixed' specified
lengths for the different binary representations. When dfdl:lengthKind is
'implicit', see Section 12.3.3 dfdl:lengthKind 'implicit'.
The
dfdl:lengthUnits can be 'bytes' or 'bits' unless otherwise stated. It is Schema
Definition Error if dfdl:lengthUnits is 'characters'.
It is a Schema Definition Error if the specified dfdl:length for an element of dfdl:lengthKind 'explicit' is a string literal integer such that the length of the data exceeds the capacity of the simple type.
It is a processing error if the specified length
for an element of dfdl:lengthKInd 'prefixed' or 'explicit' (with dfdl:length an
expression) is an integer such that the length of the data exceeds the capacity of the simple type.
Non-floating
point numbers with binary representation and dfdl:binaryNumberRep 'binary' are
represented as a bit string which contains a base-2 representation.
The
value of the specified length is constrained per the table below. The lengths
are expressed in bits and are inclusive.
|
Type |
Minimum value of length |
Maximum value of length |
|
xs:byte |
2 |
8 |
|
xs:short |
2 |
16 |
|
xs:int |
2 |
32 |
|
xs:long |
2 |
64 |
|
xs:unsignedByte |
1 |
8 |
|
xs:unsignedShort |
1 |
16 |
|
xs:unsignedInt |
1 |
32 |
|
xs:unsignedLong |
1 |
64 |
|
xs:nonNegativeInteger |
1 |
Implementation-dependent
(but not less than 64) |
|
xs:integer |
2 |
Implementation-dependent
(but not less than 64) |
|
xs:decimal |
|
Implementation-dependent
(but not less than 64) |
Table 22: Allowable Specified Lengths in Bits for
Base-2 Binary Number Elements
See
Section <IP_ADDRESS> Converting Base-2 Binary Numbers for details of the conversion to/from
numeric values.
For
binary elements of types xs:float or xs:double, a specified length must be either
exactly 4 bytes or exactly 8 bytes respectively.
The
dfdl:lengthUnits property must be 'bytes'. It is a Schema Definition Error
otherwise.
See
Section 13.8 Properties Specific to Float/Double with
Binary Representation.
Non-floating
point numbers with binary representation and dfdl:binaryNumberRep 'packed',
'bcd', or 'ibm4690Packed', are represented as a bit string of 4 bit nibbles.
The term packed decimal is used to describe such numbers.
It
is a Schema Definition Error if the specified length is not a multiple of 4
bits.
The
maximum specified length of a packed decimal number is implementation-defined.
See
Section 13.7 Properties Specific
to Number with Binary RepresentationProperties Specific
to Numbers with Binary Representation for details of the conversion of the packed
decimal bit string to/from a numeric value.
The
specified length of a binary element of type xs:boolean is as for type
xs:unsignedInt described in <IP_ADDRESS>.1 Length of Base-2
Binary Number ElementsLength of Base-2
Binary Number Elements.
See
also Section 13.10 Properties Specific to Boolean with Binary
Representation for details of how the data is converted
to/from a Boolean value.
Calendars
with binary representation and dfdl:binaryCalendarRep
‘binarySeconds’ or ‘binaryMilliseconds’ are represented
as a bit string which contains a base-2 representation.The specified length
must be either exactly 4 bytes or exactly 8 bytes respectively.
The
dfdl:lengthUnits property must be 'bytes'. It is a Schema Definition Error
otherwise.
See Section 13.13 Properties Specific to Calendar with Binary Representation for details of how the data is converted to/from the calendar type.
Calendars with binary representation and
dfdl:binaryCalendarRep 'packed', 'bcd', or 'ibm4690Packed', are represented as
a bit string of 4 bit nibbles. The term packed decimal is
used to describe such calendars.
It is a Schema Definition Error if the
specified length is not a multiple of 4 bits.
The maximum specified length of a packed
decimal calendar is implementation-dependent defined
(but not less than 9 bytes, which corresponds to calendar pattern
'yyyyMMddhhmmssSSS')[22].
See Section 13.13 Properties Specific to Calendar with Binary
Representation for details of how the data is converted
to/from the calendar type.
The dfdl:lengthUnits property must be
'bytes'. It is a Schema Definition Error otherwise.
When
unparsing a specified length element of type xs:hexBinary, and the simple
content region is larger than the length of the element in the Infoset, then
the remaining bytes are filled using the dfdl:fillByte property.
The
dfdl:fillByte is not used to trim an
element of type xs:hexBinary when parsing.
A complex element of specified length is
defining a 'box' in which its child elements exist. An example of this would be
a fixed-length record element with a variable number of children elements. The
dfdl:lengthUnits may be 'bytes' or 'characters' and it is a Schema Definition
Error otherwise.
It is possible
that the children may not entirely fill the full length of the complex element.
An example is a complex element with a specified length of 100 characters, which
contains a sequence of child elements that use up less than 100 characters of
data, perhaps because an optional element is not present. In this case the
remaining unused data is called the ElementUnused region in the data syntax
grammar of section 9.2. Another example is a complex element with a specified
length of 100 bytes, which contains a sequence of child elements the last of
which has dfdl:lengthKind 'endOfParent', dfdl:representation 'text' and a
multi-byte dfdl:encoding such that the element does not use up all the bytes of
data. In this case the remaining unused bytes comprise the child element's RightFill
region in the data syntax grammar of section 9.2. In both examples, the unused
area is skipped when parsing, and is filled with the dfdl:fillByte on
unparsing.
Note that a poorly chosen value for
dfdl:fillByte may fill the region with data that cannot be decoded in the
character set encoding, resulting in a decode error when this data is
subsequently parsed again. When dfdl:lengthUnits is 'characters' the value for
dfdl:fillByte should be chosen so as to avoid this error.
The 'representation' property identifies the physical representation of the element. The DFDL logical types are grouped to illustrate which physical representations apply to each logical type.
These properties provide the correct interpretation of the data found in the SimpleContent grammar region.
The allowable physical representations for each logical type grouping are also shown, where the logical type groupings are defined as:
|
Logical Type Group |
Types |
|
Number |
xs:double,
xs:float, xs:decimal, xs:integer and its restrictions (xs:int,
xs:unsignedLong, etc.) |
|
String |
xs:string |
|
Calendar |
xs:dateTime,
xs:date, xs:time |
|
Opaque |
xs:hexBinary |
|
Boolean |
xs:boolean |
Table 23 Logical type groups
|
Property Name |
Description |
|
representation |
Enum Valid values are dependent on logical type. Number: 'text, 'binary' String: representation is assumed to be 'text' and the dfdl:representation property is not examined Calendar: 'text, 'binary' Boolean: 'text, 'binary' Opaque: representation is assumed to be 'binary' and the dfdl:representation property is not examined. Annotation: dfdl:element, dfdl:simpleType |
Table 24 Properties Common to All Simple Types
The permitted representation properties for each logical type are shown in Table 25: Logical Type to Representation properties
|
Logical type |
dfdl:representation |
Additional representation property |
|
String |
Assumed to be text |
|
|
Float, Double |
text |
dfdl:textNumberRep: |
|
binary |
dfdl:binaryFloatRep:
|
|
|
Decimal, Integer, nonNegativeInteger |
text |
dfdl:textNumberRep: |
|
binary |
dfdl:binaryNumberRep: |
|
|
Long, Int, Short, Byte, UnsignedLong, Unsignedint, Unsignedshort, UnsignedByte |
text |
dfdl:textNumberRep: |
|
binary |
dfdl:binaryNumberRep: |
|
|
DateTime, Date, Time |
text |
|
|
|
binary |
dfdl:binaryCalendarRep: |
|
Boolean |
text |
|
|
binary |
|
|
|
HexBinary |
Assumed to be binary |
|
Table 25: Logical Type to Representation properties
|
Property Name |
Description |
|
textPadKind |
Enum Valid values 'none', 'padChar'. Indicates whether to pad
the data value on unparsing. This controls the contents of the LeftPadding
and RightPadding regions of the data syntax grammar in section 9.2. 'none': No padding occurs.
When dfdl:lengthKind is 'implicit' or
'explicit' (and dfdl:length is not an expression) the unparsed data
value must match the expected length otherwise it is a processing error. 'padChar': The data value
is padded using the dfdl:textStringPadCharacter, dfdl:textNumberPadCharacter,
dfdl:textBooleanPadCharacter or dfdl:textCalendarPadCharacter depending on the type of the element.
The padding characters populate the LeftPadding and/or RightPadding
regions depending on dfdl:textStringJustification,
dfdl:textNumberJustification,
dfdl:textBooleanJustification,
or dfdl:textCalendarJustification, depending on the type of the element. When dfdl:lengthKind is
'implicit' the data value is padded to the implicit length for the type. When dfdl:lengthKind is
'explicit' (and dfdl:length is not an expression) the data value is padded to
the length given by the dfdl:length property. When dfdl:lengthKind is
'explicit' (and dfdl:length is an expression), 'delimited', 'prefixed',
'pattern' the data value is padded to the length given by the XSD minLength
facet for type 'xs:string' or dfdl:textOutputMinLength property for other types. When dfdl:lengthKind is
'endOfParent' the data value is padded to the available length. Annotation: dfdl:element,
dfdl:simpleType |
|
textTrimKind |
Enum Valid values 'none',
'padChar' Indicates whether to trim
data on parsing.
This controls the expected contents of the LeftPadding and RightPadding
regions of the data syntax grammar in section 9.2. When 'none' no trimming
takes place. When 'padChar' the element
is trimmed of the dfdl:textStringPadCharacter, dfdl:textNumberPadCharacter,
dfdl:textBooleanPadCharacter or dfdl:textCalendarPadCharacter depending on the type of the element. The padding characters populate the LeftPadding
and/or RightPadding regions depending on
dfdl:textStringJustification, dfdl:textNumberJustification, or
dfdl:textCalendarJustification, depending on the type of the element. Annotation: dfdl:element ,
dfdl:simpleType |
|
textOutputMinLength |
Non-negative
Integer. Only
used when dfdl:textPadKind is 'padChar' and dfdl:lengthKind is 'delimited',
'prefixed', 'pattern', 'explicit' (when dfdl:length is an expression) or
'endOfParent', and type is not xs:string Specifies
the minimum content length during unparsing for simple types that do not
allow the XSD minLength facet to be specified. For
dfdl:lengthKind 'delimited', 'pattern' and 'endOfParent' the length units are
always characters, for other dfdl:lengthKinds the length units are specified
by the dfdl:lengthUnits property. If dfdl:textOutputMinLength is zero or less than the length of the representation text then no padding occurs. Annotation:
dfdl:element, dfdl:simpleType |
|
escapeSchemeRef |
QName or empty String The name of the
dfdl:defineEscapeScheme annotation that provides the additional properties
used to describe the escape scheme. If the value is the empty string then
escaping is explicitly turned off. See:
Section 7.6 The dfdl:escapeScheme Annotation Element, and Section 7.5 The dfdl:defineEscapeScheme Defining
Annotation Element. Annotation:
dfdl:element, dfdl:simpleType |
Table 26 Properties Common to All Simple Types with Text Representation
The dfdl:escapeScheme annotation is used
within a dfdl:defineEscapeScheme annotation to group the properties of an
escape scheme and allows a common set of properties to be defined that can be
reused.
An escape scheme is needed when the content
of a text element contains sequences of characters that are the same as an
in-scope separator or terminator. If the characters are not escaped, a parser
scanning for a separator or terminator would erroneously find the character
sequence in the content.
An escape scheme defines the properties that
describe the text escaping rules. There are two variants on such schemes:
·
The use
of a single escape character to cause the next character to be interpreted
literally. The escape character itself is escaped by the escape escape
character.
·
The use
of a pair of escape strings to cause the enclosed group of characters to be
interpreted literally. The ending escape string is escaped by the escape escape
character.
On
parsing, the escape scheme is applied after pad characters are trimmed and on
unparsing before pad characters are added. A pad character is not escaped by an
escape character. When parsing, pad characters are trimmed without reference to
an escape scheme. When unparsing, pad characters are added without reference to
an escape scheme.
On unparsing, the application of escape scheme processing takes place before the application of the dfdl:emptyValueDelimiterPolicy property.
|
Property Name |
Description |
|
escapeKind |
Enum Valid values 'escapeCharacter',
'escapeBlock' The type of escape mechanism defined in
the escape scheme When 'escapeCharacter': On unparsing a
single character of the data is escaped by adding an dfdl:escapeCharacter
or dfdl:escapeEscapeCharacter immediately
before it. The
characters to escape are determined by property dfdl:escapeCharacterPolicy.
On parsing any in-scope terminating delimiter encountered in the data is not interpreted as such when it is immediately preceded by the dfdl:escapeCharacter (when not itself preceded by the dfdl:escapeEscapeCharacter). Occurrences of the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data as determined by property dfdl:escapeCharacterPolicy, unless the dfdl:escapeCharacter is preceded by the dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter does not precede the dfdl:escapeCharacter, respectively. When 'escapeBlock': On unparsing the
entire data are escaped by adding dfdl:escapeBlockStart to the beginning and
dfdl:escapeBlockEnd to the end of the data. The data is either always escaped
or escaped when needed as specified by dfdl:generateEscapeBlock. If the data
is escaped and contains the dfdl:escapeBlockEnd then first character of each
appearance of the dfdl:escapeBlockEnd is escaped by the
dfdl:escapeEscapeCharacter. On parsing the dfdl:escapeBlockStart string must be the first characters in the (trimmed) data in order to activate the escape scheme. The dfdl:escapeBlockStart string is removed from the beginning of the data. Until a matching dfdl:escapeBlockEnd string (that is, one not preceded by the dfdl:escapeEscapeCharacter) is found in the data, any in-scope terminating delimiter encountered in the data is not interpreted as such, and any dfdl:escapeEscapeCharacters are removed when they precede an dfdl:escapeBlockEnd string. The matching dfdl:escapeBlockEnd string is removed from the data.. The matching dfdl:escapeBlockEnd does not have to be the last character(s) in the (trimmed) data in order to de-activate the escape scheme. A dfdl:escapeBlockStart occurring anywhere in the data other than the first characters has no significance. Annotation: dfdl:escapeScheme |
|
escapeCharacter |
DFDL String Literal or DFDL Expression Specifies one character that escapes the
subsequent character. Used when dfdl:escapeKind is
'escapeCharacter' It is a Schema Definition Error if
dfdl:escapeCharacter is empty when dfdl:escapeKind is 'escapeCharacter' This property can be computed by way of an
expression which returns a character. The expression must not contain forward
references to elements which have not yet been processed. Escape
and Quoting Character Restrictions:
The string literal is restricted to allow only certain kinds of DFDL String
Literal syntax:
It is a Schema Definition Error if the string literal contains any of the disallowed constructs. Escape characters contribute to the Annotation: dfdl:escapeScheme |
|
escapeBlockStart |
DFDL
String Literal The
string of characters that denotes the beginning of a sequence of characters
escaped by a pair of escape strings. Used
when dfdl:escapeKind is 'escapeBlock' It is a Schema Definition Error
if dfdl:escapeBlockStart is empty when dfdl:escapeKind is 'escapeBlock' The string literal value is restricted
in the same way as described in "Escape and Quoting Character
Restrictions" in the description of the dfdl:escapeCharacter property. An
dfdl:escapeBlockStart string
contributes to the simple
value region (SimpleLogicalValue or NilLiteralValue) Annotation: dfdl:escapeScheme |
|
escapeBlockEnd |
DFDL
String Literal The
string of characters that denotes the end of a sequence of characters escaped
by a pair of escape strings. Used
when dfdl:escapeKind is 'escapeBlock' . It is a Schema Definition Error
if dfdl:escapeBlockEnd is empty when dfdl:escapeKind is 'escapeBlock'. When parsing, it is
a processing error if the end of the data for the element is reached and the
escapeBlockEnd is not found in the data. The string literal value is restricted
in the same way as described in "Escape and Quoting Character
Restrictions" in the description of the escapeCharacter property. A
dfdl:escapeBlockEnd
string contributes to the simple
value region (SimpleLogicalValue or NilLiteralValue) Annotation: dfdl:escapeScheme |
|
escapeEscapeCharacter |
DFDL String Literal or DFDL Expression Specifies one character that escapes an immediately following dfdl:escapeCharacter or first character of dfdl:escapeBlockEnd. Used when dfdl:escapeKind is
'escapeCharacter' or 'escapeBlock'. This property can be computed by way of an
expression which returns a character. The expression must not contain forward
references to elements which have not yet been processed. The string literal value is restricted
in the same way as described in "Escape and Quoting Character
Restrictions" in the description of the escapeCharacter property. If the empty string is specified then no
escaping of escape characters occurs. It is explicitly allowed for both the
dfdl:escapeCharacter and the dfdl:escapeEscapeCharacter to be the same
character. In that case processing functions as if the dfdl:escapeCharacter
escapes itself. Escape-escape
characters contribute to the simple value region (SimpleLogicalValue or
NilLiteralValue) of the field. Annotation: dfdl:escapeScheme |
|
extraEscapedCharacters |
List of DFDL String Literals A whitespace separated list of
single characters that must be escaped in addition to the in-scope
delimiters. If there are no extra characters to escape the property should be
set to "". The string literal values are
restricted in the same way as described in "Escape and Quoting Character
Restrictions" in the description of the dfdl:escapeCharacter property. This property only applies on
unparsing. Extra escaped
characters contribute to the simple value region (SimpleLogicalValue or
NilLiteralValue) of the field. Annotation: dfdl:escapeScheme |
|
generateEscapeBlock |
Enum Valid values 'always', 'whenNeeded' Controls when escaping is used on
unparsing when dfdl:escapeKind is 'escapeBlock'. If 'always' then escaping is always occurs as described in dfdl:escapeKind. If 'whenNeeded' then escaping occurs as
described in dfdl:escapeKind when the data contains any of the following:
Annotation:
dfdl:escapeScheme |
|
escapeCharacterPolicy |
Enum Valid values are
‘all’, ‘delimiters’. Controls when
escape characters are removed during parsing, and output during unparsing,
when dfdl:escapeKind is 'escapeCharacter'. When 'all': During
unparsing the following are escaped as described in dfdl:escapeKind when they
are in the data.
During parsing,
occurrences of dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are
interpreted and removed from the data as described in dfdl:escapeKind. When 'delimiters': During
unparsing the following are escaped as described in dfdl:escapeKind when they
are in the data.
During parsing,
occurrences of dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are
interpreted and removed from the data as described in dfdl:escapeKind, except
that dfdl:escapeCharacter is only removed when it immediately precedes an
in-scope terminating delimiter. Annotation:
dfdl:escapeScheme |
Table 27 Escape Scheme Properties
Consider
a dfdl:escapeScheme annotation with the following properties:
·
dfdl:escapeBlockStart="start"
·
dfdl:escapeBlockEnd="end"
·
dfdl:escapeEscapeCharacter="#"
If
this is used to serialize a DFDL Infoset element of type xs:string with value
“A hash is a #”, then the value is wrapped with the
dfdl:escapeBlockStart and dfdl:escapeBlockEnd, giving simple content
"startA hash is a #end". If this data was parsed, the
"#end" will be treated as an escaped escape block end and the parse
will fail, reporting that there is no escape block end in the data.
In this scenario, the data is not compliant with the escape scheme, and the
DFDL serializer must issue a processing error.
Bidirectional
text is a feature expected in a future revision of the
DFDL standard.consists of mainly right-to-left text with some
left-to-right nested segments (such as an Arabic text with some information in
English), or vice versa (such as an English letter with a Hebrew address nested
within it.)
Note:
the bidirectional properties apply to the content of the element and not to the
initiator, terminator or separator if defined.
|
Property name |
Description |
|
textBidi |
Enum Valid
Annotation: dfdl:element, dfdl:simpleType (representation text) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table 28 Properties for Bidirectional support for All Simple Types with Text representation
|
Property Name |
Description |
|
textStringJustification |
Enum Valid values 'left', 'right', 'center' Unparsing: 'left': Justifies to the
left and adds padding chars to the string contents if the string is too
short, to the length determined by the dfdl:textPadKind property. 'right': Justifies to the
right and adds padding chars to the string contents if the string is too
short, to the length determined by the dfdl:textPadKind property. 'center': Adds equal
padding chars left and right of the string contents if the string is too
short, to the length determined by the dfdl:textPadKind property. It adds one
extra padding char on the left if needed. Parsing: 'left': Trims any pad
characters from the right of the string, according to dfdl:textTrimKind
property. 'right': Trims any pad
characters from the left of the string, according to dfdl:textTrimKind
property. 'center' Trims any pad
characters from the left and right of the string, according to
dfdl:textTrimKind property. Annotation: dfdl:element,
dfdl:simpleType |
|
textStringPadCharacter |
DFDL String Literal The value that is used
when padding or trimming string elements. If a character, then it
can be specified using a literal character or using DFDL entities. If a byte, then it must be