when there are zero other occurrences). The parser stops looking for
occurrences once a stop value has been found. If validation is enabled, it is a
validation error if less than XSDL minOccurs occurrences are found or
defaulted, or more than XSDL maxOccurs occurrences are found, not including the
stop value.
When
unparsing, the behavior is the same as for 'parsed', with the addition that a
stop value occurrence is output after the last infoset occurrence. If
dfdl:occursStopValue provides multiple stop values then the first is used.
The
stop value itself is not added to
the infoset when parsing. It is a processing error if a stop value is found in
the infoset when unparsing. (This insures that the array can be reparsed, as
the stop value will be placed automatically and only at the end.)
It
is a Schema Definition Error if dfdl:occursStopValue is not provided or in
scope.
It
is a Schema Definition Error if the type of the element is complex.
It
is a Schema Definition Error if any of the stop values provided by
dfdl:occursStopValue do not conform to the simple type of the element.
When parsing, required occurrences that have empty representation may trigger the application of a default value, as described in Section 9.4.2 Element Defaults When Parsing.
When unparsing, required occurrences that are missing from the infoset may trigger the application of a default value, as described in Section 9.4.3 Element Defaults When Unparsing.
The application of default values is not dependent on dfdl:occursCountKind, only on whether the occurrence is required or optional, whether there is a default value specified, and whether the data contains the empty representation (parsing) or is missing (unparsing). Section 9.4 Element Defaults contains the details.
If the value of a DFDL property of an array element (other than dfdl:occursCount) is given by a DFDL Expression, then the expression must be re-evaluated for each occurrence of the element in case the value changes.
Arrays can have points of uncertainty depending on the value of dfdl:occursCountKind. See Section 9.3.3 Points of Uncertainty for details.
In some situations arrays of elements and sequence groups of elements seem to be similar; however, there is no notion of the array itself independent of its contained elements. Arrays are distinctly different from sequence groups in this way.
A sequence can have its own initiator, and a complex element having that sequence as its content can also have its own initiator, so you could express two different initiators.
Unlike a sequence group, an array does not have its own initiator, terminator, or alignment. Those properties apply to each element occurrence of the array. To give an alignment, initiator, separator or terminator to an entire array you must enclose the element declaration for the array in a sequence group and specify the alignment, separator, initiator and terminator on the sequence group.
To prevent an infinite loop, the parsing of an array that
is potentially unbounded must terminate when the following are true:
·
The occurrence is a point of
uncertainty;
·
The position in the data does not move
during the parsing of the occurrence (including any associated Separator,
PrefixSeparator or PostfixSeparator region);
·
The occurrence is known-to-exist with
empty representation or nil representation.
An array is potentially unbounded if any of the following
are true:
·
dfdl:occursCountKind is 'stopValue'
·
dfdl:occursCountKind is 'parsed'
·
dfdl:occursCountKind is 'implicit' and
XSDL maxOccurs is unbounded
When dfdl:occursCountKind is 'stopValue' this results in a
processing error because the stop value will never be encountered.
Further, to prevent unnecessary consumption of resources
for large bounded values of XSDL maxOccurs, the parsing of an array must
terminate when the following are true:
·
dfdl:occursCountKind is 'implicit';
·
The occurrence is a point of
uncertainty;
·
The position in the data does not move
during the parsing of the occurrence (including any associated Separator,
PrefixSeparator or PostfixSeparator region);
·
The
occurrence is known-to-exist with empty representation.
When parsing a single array, it is possible to extract occurrences that have different representations (nil, empty, normal, absent) although with some values of dfdl:lengthKind certain combinations of representations are not possible.
Occurrences with nil representation are added to the infoset with [nilled] member true.
Occurrences with empty representation may or may not be added to the infoset, as described in Section 9.4. If a required occurrence is not added to the infoset, it may be a processing error, dependent on dfdl:occursCountKind as described in section 16.1.
Occurrences with absent representation are not added to the infoset. For a required occurrence it may be a processing error, dependent on dfdl:occursCountKind as described in section 16.1.
Consider parsing an array where optional occurrences with empty representation are present in the data, but there are also later optional occurrences present with normal representation. Such an array is called a 'sparse array'.
If the indices of the occurrences are
significant and need to be preserved, then the array may be modelled using an
element with XSDL nillable 'true', dfdl:nilKind 'literalValue', dfdl:nilValue
'%ES;' and dfdl:nilValueDelimiterPolicy the same as
dfdl:emptyValueDelimiterPolicy. The occurrences with empty representation now become
occurrences with nil representation, and will produce nil values in the
infoset, so the absolute positions of all occurrences are preserved.
If the indices of the occurrences are not significant, then the array should be modelled using an element with XSDL nillable 'false'. Optional occurrences with empty representation will not create items in the infoset, so the absolute position of any optional occurrences with normal representation is not preserved. Optional occurrences with empty representation are therefore skipped.
This section describes properties which allow the creation of calculated elements. When parsing, the value of a calculated element is derived using a DFDL Expression, and not by processing bytes from the data stream. When unparsing, the value of a calculated element is derived using a DFDL Expression, and is not obtained from the infoset in the usual way.
Calculated elements allow a technique that is commonly called layering. In this technique, some elements are said to be in the physical layer, and some in the logical layer. When parsing, the logical layer values are computed from physical layer values. When unparsing the opposite occurs, that is the physical layer values are computed from the logical layer values.
Calculated elements are commonly used with hidden elements so as to hide the physical layer elements so that they do not become part of the infoset.
When a DFDL Schema is used to both parse and unparse data, then a calculated element on parsing will normally have one or more calculated elements on unparsing.
These properties apply to elements of simple type.
|
Property
Name |
Description |
|
inputValueCalc |
DFDL Expression An expression that calculates the value of
the element when parsing. It is a Schema Definition Error if the
result type of the expression does not conform to the base type of the
element. The element value created using
dfdl:inputValueCalc is validated like any other element value (when
validation is enabled). An element that specifies a
dfdl:inputValueCalc expression has no representation of its own in the data
stream. All other DFDL representation properties are ignored. When an element which carries this
property appears in a sequence that has a separator, no separator is
associated with the element. When parsing, no separator is expected in the
input data. When unparsing, no separator is written to the output data. The element must not be optional nor an
array nor be global. The DFDL Expression must not refer to this
element nor cause a circular reference to this element. The expression must
not contain forward references to elements which have not yet been processed. It is a Schema Definition Error if this
property is specified on an element which has an XSDL fixed or default
property. It is a Schema Definition Error if
dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same
element. It is not possible to place this property
in scope on a dfdl:format annotation.
If this property appears on an element
declaration or element reference schema component, the appearance of any other
DFDL properties on that component is a Schema Definition Error. If this property appears on an element
reference, then DFDL properties expressed on the referenced global element
declaration or its type are ignored. If this property appears on an element
declaration, then DFDL properties expressed on its type are ignored. Annotation: dfdl:element |
|
outputValueCalc |
DFDL Expression An expression that calculates the value of
the current element when unparsing. The element must not be optional nor an array
nor be global. It is a Schema Definition Error if the
result type of the expression does not conform to the base type of the
element. The value created using
dfdl:outputValueCalc is validated like any other element value (when
validation is enabled). The value for the element, if any, in the
infoset is ignored. The DFDL expression must not refer to this
element nor cause a circular reference to this element. The expression may
contain forward references to elements which have not yet been processed. It is a Schema Definition Error if
dfdl:outputValueCalc is specified on an element which has an XSDL fixed or
default property. It is a Schema Definition Error if
dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same
element. It is not possible to place this property
in scope on a dfdl:format annotation. Annotation:
dfdl:element |
Table 54 Calculated Value Properties
Consider this simple example. The data stream contains two elements giving the number of rows and number of columns of an array of numbers. The representation of the array is stored after these two elements.
<xs:complexType name="array">
<xs:sequence dfdl:initiator="" >
<xs:sequence dfdl:hiddenGroupRef="tns:hiddenArrayCounts"/>
<xs:element name="rows" maxOccurs="unbounded"
dfdl:occursCountKind="expression"
dfdl:occursCount="{ ../nrows }">
<xs:complexType>
<xs:sequence>
<xs:element name="cols" type="xs:float" maxOccurs="unbounded"
dfdl:occursCountKind="expression"
dfdl:occursCount=" { ../../ncols } " />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
<xs:group name="hiddenArrayCounts"
>
<xs:sequence>
<xs:element name="nrows" type="xs:unsignedInt"
dfdl:representation="binary"
dfdl:lengthKind="implicit"
dfdl:outputValueCalc="{ count(../rows) }"/>
<xs:element name="ncols" type="xs:unsignedInt"
dfdl:representation="binary"
dfdl:lengthKind="implicit"
dfdl:outputValueCalc=
"{ if ( count(../rows) ge 1 )
then
count(../rows[1]/cols)
else
0
}"/>
</xs:sequence>
</xs:group>
In the example above we see that there are two hidden elements named 'nrows' and 'ncols'. These hidden elements' values are computed when unparsing from the number of occurrences in the 'rows' and 'cols' repeating elements. The 'rows' and 'cols' repeating elements number of occurrences are computed when parsing from the hidden elements 'nrows' and 'ncols'.
Logically, the data is a date.
<xs:element
name="d" type="date"/>
Physically, it is stored as 3 single byte integers.
The format of this data is expressed as this schema:
<xs:sequence
dfdl:representation="binary">
<xs:element name="mm" type="byte" />
<xs:element name="dd" type="byte" />
<xs:element name="yy" type="byte"/>
</xs:sequence>
This physical representation can
be hidden so that it does not become part of the infoset:
<xs:sequence>
<xs:sequence dfdl:hiddenGroupRef="tns:hiddenpDate"/>
<xs:element name="d"
type="date">
…
</xs:element>
</xs:sequence>
<xs:group name="hiddenpDate" >
<xs:sequence>
<xs:element name="pdate">
<xs:complexType>
<xs:sequence dfdl:representation="binary">
<xs:element name="mm" type="byte" />
<xs:element name="dd" type="byte" />
<xs:element name="yy" type="byte"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:group>
A calculation can be used to compute the logical date element 'd' from the physical 'pdate' when parsing:
<xs:sequence>
... hidden pdate here ...
<xs:element name="d" type="date">
<xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element>
<dfdl:property name="inputValueCalc">
{
fn:date(fn:concat(if(../pdate/yy gt 50 )then "19" else "20",
if ( ../pdate/yy gt 9 )
then xs:string(../pdate/yy)
else fn:concat("0",
xs:string(../pdate/yy)),
"-",
xs:string(../pdate/mm),
"-",
xs:string(../pdate/dd)))
}
</dfdl:property>
</dfdl:element>
</xs:appinfo></xs:annotation>
</xs:element>
...
</xs:sequence>
The expression above assembles a string
resembling, for example, "2005-12-17" or "1957-3-9" which
is the string representation of a date that is acceptable to the fn:date
constructor function. The hidden element 'pdate' is
referenced by relative paths. The expression '../pdate/yy'
accesses an element of type 'int', and the xs:string
constructor function turns it into an integer.
Finally, we must handle the unparse case where the physical layer is computed from the logical layer:
<xs:sequence dfdl:representation="binary"
<xs:element name="mm" type="byte"
dfdl:outputValueCalc="{ fn:month-from-date(../d) }" />
<xs:element name="dd" type="byte"
dfdl:outputValueCalc="{ fn:day-from-date(../d) }" />
<xs:element name="yy" type="byte"
dfdl:outputValueCalc="{ fn:year-from-date(../d) idivmod 100 }"/>
</xs:sequence>
The entire example in one place:
<xs:sequence>
<xs:sequence dfdl:hiddenGroupRef="tns:hiddenpDate"/>
<xs:element name="d" type="date">
<xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element>
<dfdl:property name="inputValueCalc">
{
fn:date(fn:concat(if(../pdate/yy gt 50) then "19" else "20",
if ( ../pdate/yy gt 9 )
then xs:string(../pdate/yy)
else fn:concat("0",
xs:string(../pdate/yy)),
"-",
xs:string(../pdate/mm),
"-",
xs:string(../pdate/dd)))
}
</dfdl:property>
</dfdl:element>
</xs:appinfo></xs:annotation>
</xs:element>
...
</xs:sequence>
<xs:group name="hiddenpDate" >
<xs:sequence>
<xs:element name="pdate">
<xs:complexType>
<xs:sequence dfdl:representation="binary">
<xs:element name="mm" type="byte"
dfdl:outputValueCalc="{ fn:month-from-date(../d) }" />
<xs:element name="dd" type="byte"
dfdl:outputValueCalc="{ fn:day-from-date(../d) }" />
<xs:element name="yy" type="byte"
dfdl:outputValueCalc="{ fn:year-from-date(../d) idivmod 100 }" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:group>
The above sequence contains logically only a single date element.
In addition to providing the DFDL schema and data to be parsed or serialized, DFDL Schemas can also be parameterized by external variables.
DFDL processors can provide implementation-defined means to specify:
1. The data to be processed: a data stream when parsing or an infoset when unparsing.
2. The DFDL schema to be used
3. The distinguished global element declaration to be used (specifying both name of element and namespace of that name)
4. Values for external variables
Notice also that a DFDL Schema, like any XML schema, can have multiple top-level element declarations; hence, the distinguished global element declaration is necessary to indicate which of these top-level element declarations is to be the starting point for processing data. The distinguished global element declaration may be omitted if the DFDL schema contains only one top-level element declaration.
The mechanism by which a DFDL processor is controlled is not specified by this standard. For example, command line DFDL processors may use command line options, but DFDL processors embedded in other kinds of software systems may need other mechanisms.
For convenience, a standard set of named DFDL format definitions may be provided with DFDL processors. These built-in format definitions may be imported by DFDL schema authors.
DFDL conformance can be claimed for schema documents and for processors
A schema document conforms to this specification if it conforms to the subset of XML Schema 1.0 defined in section 5.1 DFDL Subset of XML Schema and consists of components which individually and collectively satisfy all the relevant constraints specified in this document.
Conformance may be claimed separately for a DFDL parser, a DFDL unparser or a DFDL processor that parses and unparses.
See Section 21 Optional DFDL Features for the list of optional feature
It is the intention of the DFDL Work Group to provide a conformance test suit to help verify conformance with this specification.
The following table lists the features of the DFDL language that are considered optional for DFDL processor implementations. This list admits very small subsets of the full DFDL specification. For example, a binary-only subset without any expressions or variables is specifically allowed.
|
Feature |
Detection |
|
Validation |
External switch |
|
Named Formats |
dfdl:defineFormat or dfdl:ref |
|
Choices |
xs:choice in xsd |
|
Arrays where size not known in advance |
dfdl:occursCountKind 'implicit', 'parsed', 'stopValue' |
|
Expressions |
Use of a DFDL expression in any property value |
|
End of parent |
dfdl:lengthKind "endOfParent" |
|
Simple type restrictions |
xs:simpleType in xsd |
|
Text representation for types other than String |
dfdl:representation "text" for Number, Calendar or Boolean types |
|
Delimiters |
dfdl:separator <> "" or dfdl:initiator <> "" or dfdl:terminator <> "" or dfdl:lengthKind "delimited" |
|
Nils |
XSDL nillable 'true' in xsd |
|
Defaults |
XSDL default or fixed in xsd |
|
|
|
|
Lengths in Bits |
dfdl:alignmentUnits 'bits' or dfdl:lengthUnits 'bits' |
|
Delimited lengths and representation binary element |
dfdl:representation 'binary' (or implied binary) and dfdl:lengthKind 'delimited' |
|
Regular expressions |
dfdl:lengthKind 'pattern', dfdl:assert with dfdl:testkind 'pattern' , dfdl:discriminator with dfdl:testkind 'pattern' |
|
Zoned numbers |
dfdl:textNumberRep 'zoned' |
|
IBM 390 packed numbers |
dfdl:binaryNumberRep 'packed' |
|
IBM 390 packed calendars |
dfdl:binaryCalendarRep 'packed' |
|
IBM 390 floats |
dfdl:binaryFloatRep 'ibm390Hex' |
|
Unordered sequences |
dfdl:sequenceKind 'unordered' |
|
Floating elements |
dfdl:floating 'yes' |
|
dfdl functions in expression language |
dfdl:functions in expression |
|
Hidden groups |
dfdl:hiddenGroupRef <> '' |
|
Calculated values |
dfdl:inputValueCalc <> '' or dfdl:outputValueCalc <> '' |
|
Escape schemes |
dfd:defineEscapeScheme in xsd |
|
Extended encodings |
Any dfdl:encoding value beyond the core list |
|
UTF-16
Variable Width Characters |
dfdl:utf16Width=”variable” |
|
Asserts |
dfdl:assert in xsd |
|
Discriminators |
dfdl:discriminator in xsd |
|
Prefixed lengths |
dfdl:lengthKind 'prefixed' |
|
Variables |
dfdl:defineVariable, dfdl:newVariableInstances, dfdl:setVariable Variables in DFDL expression language Note that variables as a feature is dependent on the Expressions feature. |
|
BCD calendars |
dfdl:binaryCalendarRep "bcd" |
|
BCD numbers |
dfdl:binaryNumberRep "bcd" |
|
Multiple schemas |
xs:include or xs:import in xsd |
|
IBM 4690 packed numbers |
dfdl:binaryNumberRep "ibm4690Packed" |
|
IBM 4690 packed calendars |
dfdl:binaryCalendarRep "ibm4690Packed" |
|
DFDL Byte Value Entities |
Use of %#r syntax in a DFDL String Literal other than the dfdl:fillByte property |
|
DFDL Standard Character Set Encodings |
dfdl:encoding name begins with "X-DFDL-". |
|
Bit Order - Least Significant Bit First |
dfdl:bitOrder with value 'leastSignificantBitFirst' |
Table 55 Optional DFDL features
In order to provide portability of a DFDL
schema, a minimal or extended conforming processor must issue warnings about
any DFDL properties it does not implement. This warning can simply state that
the property was not recognized.
(This allows the implementation to simply
have no knowledge of properties it does not need for the subset of features it
implements.)
For example if
the hidden groups feature is not implemented, then the implementation will most
likely not recognize the dfdl:hiddenGroupRef property at all. Such an
implementation must issue a warning that the dfdl:hiddenGroupRef property was
not recognized.
It is a Schema Definition Error if a DFDL schema uses an optional feature that is not supported by a minimal or extended conforming processor.
The following list gives the order in which DFDL properties are examined when the DFDL parser is positioned at a particular component in the DFDL schema, and about to parse the bitstream modeled by that component.
The following list gives the order
in which DFDL properties are examined when the DFDL unparser is positioned at a
particular component in the DFDL Infoset, and about to unparsed and thereby
create the bitstream which is the representation of that component.
 It allows properties in the DFDL schema to be dependent on the value of an occurrence of an element or the value of a DFDL variable. For example the length of the content of an element can be made dependent on the value of another element in the document.
The main uses of the expression language are as follows:
The DFDL expression language is a subset of
XPath 2.0 [XPath]. DFDL uses a subset of XML schema and has a
simpler information model, so only a subset of XPath 2.0 expressions is
meaningful in DFDL Schemas. For example there are no attributes in DFDL so the
attribute axis is not needed.
XPath
2.0 specification [XPATH2] allows implementation-dependent evaluation of
expressions thereby allowing either lazy (sequential) evaluation or full
(parallel) evaluation of expressions with OR and AND clauses. This flexibility
is not desirable in DFDL 1.0 implementations, so the specification is changed to prescribe lazy
(sequential) evaluation left-to-right.
In addition, DFDL expressions never return node-sequences having more than one node. DFDL expressions either return a simple value, a node sequence containing exactly one node/value, or an empty node sequence. Node sequences of length greater than one can be used within the expression, just not as the final result. Alternatively, one can state this as there are no constructs in DFDL which can accept a node sequence of more than one node; hence, DFDL expressions can never return a node sequence of more than one node as their final result.
For nilled elements, an attempt to get the value of a nilled element returns an empty node sequence.
DFDL implementations MUST comply with the error
code behaviour in Appendix G of the XPath 2.0 spec and map these to the correct
DFDL failure type. All but one of XPath's errors map to a Schema Definition
Error. The exception is XPTY0004, which is used both for static and dynamic
cases of type mismatch. A static type mismatch maps to a Schema Definition
Error, whereas a dynamic type mismatch maps to a processing error. A DFDL
implementation should distinguish the two kinds of XPTY0004 error if it is able
to do so, but if unable it should map all XPTY0004 errors to a Schema
Definition Error
Implementation Note: DFDL implementations may use
off-the-shelf XPath 2.0 processors, but will need to pre-process DFDL expressions
to ensure that the behaviour matches the DFDL specification:
·
Ensure that what is returned as the result is not a
sequence with length > 1 by appropriate use of fn:exactly-one().
·
Check for the disallowed use of those XPath 2.0
functions that are not in the DFDL subset
XPath 2.0
specification [XPATH2] defines its functions to be in namespace http://www.w3.org/2005/xpath-functions.
The DFDL specification assumes
namespace prefix “fn:” is bound to
this namespace.
The DFDL expression language operates on the DFDL infoset with the addition of the hidden elements. That is, it operates on the augmented infoset.
In
general, a DFDL expression can only reference an element that precedes the
position in the schema where the expression is declared, and it is a schema
definition otherwise, with the following exceptions:
·
An
assert or discriminator on a component may reference an element that is a
descendent of the component.
·
A
dfdl:outputValueCalc property may reference an element that follows the
position in the schema where the property is specified.
Implementations may have implementation-dependent
defined limitations on the use of
forward or backward reference, or may provide controls for bounding the reach
of such references. These mechanisms are beyond the scope of this
specification.
A variable is a binding between a (qualified) name and a (typed) value. Variables are defined using the dfdl:defineVariable annotation (see 7.7); defining a variable causes an initial instance also to be created. Further instances of variables are created using the dfdl:newVariableInstance annotation. Instances of variables are assigned a value using the dfdl:setVariable annotation. Variables are referenced in expressions by preceding the QName with '$'.
This section describes the semantics of variables. Any implementation consistent with the behavior described here is acceptable.
The memory where the information about a variable is stored during DFDL processing is called the variable memory. A variable is a name that is associated with a storage tuple in the variable memory.
Specifically, the variable memory contains:
The variable memory is initialized when a dfdl:defineVariable annotation is encountered.
Each time a dfdl:newVariableInstance annotation is encountered, the parser captures the current value of the counter from the variable memory. It then creates a new variable memory where the location counter's value is one greater, and where the list of locations has been augmented with a new tuple at the location given by the prior value of the location counter. The tuple is initialized based on the specifics of the dfdl:defineVariable annotation.
Upon exit of the scope where the new variable instance was created, the newly created variable memory is discarded and the prior variable memory is restored.
Note that the above algorithm insures that each time a dfdl:newVariableInstance is encountered, a fresh location is initialized for it, and once the scope containing that variable goes out of scope, the instance tuple for the variable can no longer be reached. A different variable instance tuple may now be visible if there is one still in an enclosing scope.
The flags in the variable memory tuples are interpreted and modified as follows:
|
DFDL
annotation |
before annotation processed |
after annotation processed |
||||
|
has-been-set |
has-been-referenced |
has-value |
has-been-set |
has-been-referenced |
has-value |
|
|
defineVariable (without default or external value) |
tuple doesn't exist |
false |
false |
false |
||
|
defineVariable (with default value) |
tuple doesn't exist |
false |
false |
true |
||
|
defineVariable (with external value) |
tuple doesn't exist |
false |
false |
true |
||
|
newVariableInstance (without default value) |
tuple doesn't exist |
false |
false |
false |
||
|
newVariableInstance (with default value) |
tuple doesn't exist |
false |
false |
true |
||
|
setVariable |
tuple doesn't exist |
Schema Definition Error |
||||
|
false |
false |
false |
true |
false |
true |
|
|
false |
false |
true |
true |
false |
true (also value changed to new value) |
|
|
false |
true |
true |
Schema Definition Error – set after reference not allowed. |
|||
|
true |
any |
true |
Schema Definition Error - double set not allowed. |
|||
|
reference variable (from DFDL expression) |
tuple doesn't exist |
Schema Definition Error |
||||
|
false |
false |
false |
Schema Definition Error – undefined variable |
|||
|
any |
any |
true |
false |
true (value is returned) |
true |
Table 56 Memory States for Expression Language Variables
The above table describes a set of rules which might be abbreviated as:
An exception to this behavior occurs whenever the DFDL processor backtracks because it is processing multiple arms of a choice or as a result of speculative parsing. In this case the variable state is also rewound.
It is a Schema Definition Error if a dfdl:setVariable or a variable reference occurs and there is no corresponding variable name defined by a dfdl:defineVariable annotation.
It is a Schema Definition Error if a dfdl:setVariable provides a value of incorrect type which does not correspond to the type specified by the dfdl:defineVariable.
It is a Schema Definition Error if a variable reference in an expression is able to return a value of incorrect type for the evaluation of that expression. That is, DFDL - including the expressions contained in it - is a statically type-checkable language. DFDL implementations may issue these Schema Definition Errors prior to processing time.
Even if the errors are detected at processing time, the errors associated with write-after-read, and double-write are Schema Definition Errors because they indicate the schema is not properly designed to use variables consistent with their single-assignment behavior.
DFDL expressions follow the XPath 2.0 syntax rules but are always enclosed in curly braces "{" and "}".
When a property accepts either a DFDL string literal or a DFDL expression, and the value is a string literal starting with a "{" character, then "{{"must be used to escape the "{" character.
The syntax "{}" is a Schema Definition Error as it results in an empty XPath 2.0 expression which is not legal. It is not the equivalent of setting the property to empty string.
Examples
{ /book/title }
{ $x+2 }
{ if (fn:exists(../field1))
then 1 else 0 }
The result of evaluating the expression must be a single atomic value of the type expected by the context, and it is a Schema Definition Error otherwise. Some XPath expressions naturally return a sequence of values, and in this case it is also Schema Definition Error if an expression returns a sequence containing more than one item.
Additionally:
Refer to XML Path Language (XPath) 2.0 [XPath] for a description of XPath expressions
|
DFDL Expression |
::= |
"{" Expr "}" |
|
Expr |
::= |
ExprSingle |
|
ExprSingle |
::= |
IfExpr |
|
IfExpr |
::= |
"if" "(" Expr ")" "then" ExprSingle "else" ExprSingle |
|
OrExpr |
::= |
AndExpr ( "or" AndExpr )* |
|
AndExpr |
::= |
ComparisonExpr ( "and" ComparisonExpr )* |
|
ComparisonExpr |
::= |
AdditiveExpr ( (ValueComp |
|
AdditiveExpr |
::= |
MultiplicativeExpr ( ("+" | "-") MultiplicativeExpr )* |
|
MultiplicativeExpr |
::= |
IntersectExceptExpr |
|
IntersectExceptExpr
|
::= |
UnaryExpr (
("intersect" | "except") UnaryExpr )* |
|
UnaryExpr |
::= |
("-" | "+")* ValueExpr |
|
ValueExpr |
::= |
PathExpr |
|
ValueComp |
::= |
"eq" | "ne" | "lt" | "le" | "gt" | "ge" |
|
PathExpr |
::= |
("/"
RelativePathExpr?) |
|
RelativePathExpr |
::= |
StepExpr (("/") StepExpr)* |
|
StepExpr |
::= |
AxisStep |
|
AxisStep |
::= |
(ReverseStep | ForwardStep) Predicate? |
|
ForwardStep |
::= |
(ForwardAxis NodeTest) | AbbrevForwardStep |
|
ForwardAxis |
::= |
("child"
"::") |
|
AbbrevForwardStep |
::= |
NodeTest | ContextItemExpr |
|
ReverseStep |
::= |
(ReverseAxis NodeTest) | AbbrevReverseStep |
|
ReverseAxis |
::= |
("parent"
"::") |
|
AbbrevReverseStep |
::= |
".." |
|
NodeTest |
::= |
NameTest |
|
NameTest |
::= |
QName |
|
FilterExpr |
::= |
PrimaryExpr Predicate? |
|
Predicate |
::= |
"[" Expr "]" |
|
PrimaryExpr |
::= |
Literal | VarRef | ParenthesizedExpr | ContextItemExpr | FunctionCall |
|
Literal |
::= |
NumericLiteral | StringLiteral |
|
NumericLiteral |
::= |
IntegerLiteral | DecimalLiteral | DoubleLiteral |
|
VarRef |
::= |
"$" VarName |
|
VarName |
::= |
QName |
|
ParenthesizedExpr |
::= |
"("
Expr ")" |
|
ContextItemExpr |
::= |
"." |
|
FunctionCall |
::= |
QName "(" (ExprSingle ("," ExprSingle)*)? ")" |
Table 57 DFDL Expression Language
Notes:
5.
NameTest - These QNames are path steps that refer
to elements in the DFDL infoset. If such an element is in a namespace, then the
NameTest QName must have a prefix which is bound to the namespace.
Specifically, any default namespace is not used to implicitly qualify these
NameTest QNames. This behavior is consistent with XPath expression usage in XML Schema [Walmsley] [MJB5] such
as in the path property of the xs:selector and xs:field elements within
xs:key and xs:unique constraints, and in related XML standards such as
XSLT. Note however, that this behavior is different from the way QNames are
used in other places in XML and DFDL Schemas such as the ref property of an
element reference, or the dfdl:ref property of a DFDL format annotation. There
a QName with no prefix must always be referring to a global declaration or
definition, and so is augmented with the default namespace when needed.
In the function signatures below a '?' following an argument name, argument type or result type indicates that the argument/result can be a node or value of the expected type or it can have no value.
The arguments to the constructors are all of type xs:anyAtomicType. Since the expression language can be statically type checked, it is a Schema Definition Error if the type of the argument is not one of the DFDL-supported subtypes of xs:anyAtomicType,
However, many statically type-correct values will still not be convertible to the result type. It is a processing error if the supplied argument value is not convertible to the constructed type.
The following constructor
functions for the built-in types are supported:
|
Function |
|
xs:string($arg as xs:anyAtomicType) as xs:string |
|
xs:boolean($arg as xs:anyAtomicType) as xs:boolean |
|
xs:decimal($arg as xs:anyAtomicType) as xs:decimal |
|
xs:float($arg as xs:anyAtomicType) as xs:float |
|
xs:double($arg as xs:anyAtomicType) as xs:double |
|
xs:dateTime($arg as xs:anyAtomicType) as xs:dateTime |
|
xs:time($arg as xs:anyAtomicType) as xs:time |
|
xs:date($arg as xs:anyAtomicType) as xs:date |
|
xs:hexBinary($arg as xs:anyAtomicType) as xs:hexBinary |
|
xs:integer($arg as xs:anyAtomicType) as xs:integer |
|
xs:long($arg as xs:anyAtomicType) as xs:long |
|
xs:int($arg as xs:anyAtomicType) as xs:int |
|
xs:short($arg as xs:anyAtomicType) as xs:short |
|
xs:byte($arg as xs:anyAtomicType) as xs:byte |
|
xs:nonNegativeInteger($arg as xs:anyAtomicType) as xs:nonNegativeInteger |
|
xs:unsignedLong($arg as xs:anyAtomicType) as xs:unsignedLong |
|
xs:unsignedInt($arg as xs:anyAtomicType) as xs:unsignedInt |
|
xs:unsignedShort($arg as xs:anyAtomicType) as xs:unsignedShort |
|
xs:unsignedByte($arg as xs:anyAtomicType) as xs:unsignedByte |
Table 58 Basic Constructors
A special constructor function is provided for constructing a xs:dateTime value from an xs:date value and an xs:time value.
|
Function |
|
fn:dateTime($arg1 as xs:date, $arg2 as xs:time) as xs:dateTime |
Table 59 Special Constructor for xs:dateTime
The following additional
constructor functions are defined on the boolean type.
|
Function |
Meaning |
|
fn:true() |
Constructs the xs:boolean value 'true'. |
|
fn:false() |
Constructs the xs:boolean value 'false'. |
Table 60 Boolean functions
The following functions are
defined on boolean values. The return type of these functions is xs:boolean.:
|
Function |
Meaning |
|
fn:not($arg?) |
If $arg is the empty sequence or a node with [nilled] true, fn:not returns true. If $arg is a sequence containing a node with [nilled] false or [nilled] having no value (that is, a node corresponding to a non-nillable element), fn:not returns false. If $arg is a value of type xs:boolean or a derived from xs:boolean, fn:not returns the boolean inverse of $arg. If $arg is a value of type xs:string or a type derived from xs:string, fn:not returns true if the operand value has zero length; otherwise it returns false. If $arg is a value of any numeric type or a type derived from a numeric type, fn:not returns true if the operand value is NaN or is numerically equal to zero; otherwise it returns false. In all other cases, fn:not raises a processing error. Inverts the xs:boolean value of the argument. |
Table 61 Boolean functions
The following functions are
defined on numeric types. Each function returns a value of the same type as the
type of its argument. The argument must be convertible to a number type.
|
Function |
Meaning |
|
fn:abs($arg as numeric) |
Returns the absolute value of the argument. |
|
fn:ceiling($arg as numeric) |
Returns the smallest number with no fractional part that is greater than or equal to the argument. |
|
fn:floor($arg as numeric) |
Returns the largest number with no fractional part that is less than or equal to the argument. |
|
fn:round($arg as numeric) |
Rounds to the nearest number with no fractional part. When the value is x.5, it rounds toward positive infinity. |
|
fn:round-half-to-even($arg as numeric) fn:round-half-to-even($arg as numeric, $precision as xs:integer) |
Takes a number and a precision and returns a number rounded to the given precision. If the fractional part is exactly half, the result is the number whose least significant digit is even. |
Table 62 Numeric Functions
The following functions are
defined on values of type xs:string and types derived from it. In the functions below which compare strings, DFDL always uses
the default unicode collation algorithm (which is a comparison of codepoint
values).
|
Function |
Meaning |
|
fn:concat( $arg1 as xs:anyAtomicType,
$arg2 as xs:anyAtomicType, ... ) |
Concatenates two or more xs:anyAtomicType arguments
cast to xs:string. |
|
fn:substring($sourceString as xs:string, $startingLoc
as xs:double) fn:substring($sourceString as xs:string, $startingLoc
as xs:double, $length as xs:double) |
Returns the xs:string located at a specified place
within an argument xs:string. |
|
fn:string-length($arg as xs:string) |
Returns the length of the argument as an xs:integer |
|
fn:upper-case($arg as xs:string) |
Returns the upper-cased value of the argument. |
|
fn:lower-case($arg as xs:string) |
Returns the lower-cased value of the argument. |
|
fn:contains($arg1
as xs:string, $arg2 as xs:string) |
Returns
xs:boolean indicating whether one xs:string contains another xs:string. |
|
fn:starts-with($arg1
as xs:string, $arg2 as xs:string) |
Returns
xs:boolean indicating whether the value of one xs:string begins with the characters of another xs:string. |
|
fn:ends-with($arg1
as xs:string, $arg2 as xs:string) |
Returns
xs:boolean indicating whether the value of one xs:string ends with the characters of another xs:string. |
|
fn:substring-before($arg1
as xs:string, $arg2 as xs:string) |
Returns
the characters of one xs:string that precede in that xs:string the characters of another xs:string. |
|
fn:substring-after($arg1
as xs:string, $arg2 as xs:string) |
Returns
the characters of xs:string that follow in that xs:string the characters of another xs:string. |
Table 63 String Functions
|
Function |
Meaning |
|
fn:year-from-dateTime($arg as xs:dateTime) |
Returns
the year from an xs:dateTime value as an xs:integer. |
|
fn:month-from-dateTime($arg as xs:dateTime) |
Returns
the month from an xs:dateTime value as an xs:integer. |
|
fn:day-from-dateTime($arg as xs:dateTime) |
Returns
the day from an xs:dateTime value as an xs:integer. |
|
fn:hours-from-dateTime($arg as xs:dateTime) |
Returns
the hours from an xs:dateTime value as an xs:integer. |
|
fn:minutes-from-dateTime($arg as xs:dateTime) |
Returns
the minutes from an xs:dateTime value as an xs:integer. |
|
fn:seconds-from-dateTime($arg as xs:dateTime) |
Returns
the seconds from an xs:dateTime value as an xs:decimal. |
|
fn:year-from-date($arg as xs:date) |
Returns
the year from an xs:date value as an xs:integer. |
|
fn:month-from-date($arg as xs:date) |
Returns
the month from an xs:date value as an xs:integer. |
|
fn:day-from-date($arg as xs:date) |
Returns
the day from an xs:date value as an xs:integer. |
|
|
Returns
the hours from an xs:time value as an xs:integer. |
|
|
Returns
the minutes from an xs:time value as an xs:integer. |
|
|
Returns
the seconds from an xs:time value as an xs:decimal. |
Table 64 Date and Time Functions
The following functions are
defined on sequences. (Note that DFDL v1.0 does not support sequences of length
> 1.)
|
Function |
Meaning |
|
fn:empty($arg?) |
Indicates whether or not the provided sequence is
empty. |
|
fn:exists($arg?) |
Indicates whether or not the provided sequence is not
empty. |
|
fn:exactly-one($arg?) |
|
|
fn:count($arg) |
Returns the number of items in the value of $arg as an xs:integer. Returns 0 if $arg is the empty sequence. |
Table 65 Node Sequence Test Functions
This section discusses functions
and operators on nodes.
|
Function |
Meaning |
|
fn:local-name() fn:local-name($arg) |
Returns the local name of the context node or the
specified node as an xs:string. |
|
fn:namespace-uri() fn:namespace-uri($arg) |
Returns the namespace URI as an xs:string for the
argument node or the context node if the argument is omitted. Returns empty
string if the argument/context node is in no namespace. |
Table 66 Node functions
This section discusses functions
related to nillable elements.
|
Function |
Meaning |
|
fn:nilled($arg?) |
Returns an xs:boolean true when the argument
node Infoset member [nilled] is
true and false when [nilled] is
false. If the argument is not an element node, returns the empty sequence. If
the argument is the empty sequence, returns the empty sequence. If the argument
is an element node and [nilled]
has no value returns the empty sequence. |
Table 67 Nillable Element Functions
|
Function |
Meaning |
|
dfdl:contentLength($node, $lengthUnits) |
Returns the length of the supplied node's SimpleContent
region for elements of simple type, or ComplexContent region for elements of
complex type. These regions are defined in Section 9.2 DFDL
Data Syntax Grammar. The value is returned
as an xs:unsignedLong. The second argument is of type xs:string and must be
'bytes', 'characters', or 'bits' (Schema Definition Error otherwise) and
determines the units of length. |
|
dfdl:valueLength($node, $lengthUnits) |
Returns the length of the supplied node's For simple types, the valueLength() function returns a
length which excludes any padding or filling. The second argument is is of type xs:string and must be
'bytes', 'characters', or 'bits' (Schema Definition Error otherwise) and
determines the units of length. |
|
dfdl:testBit($data, $bitPos) |
Returns Boolean true if the bit number given by the
xs:nonNegativeInteger $bitPos is set on in the xs:unsignedByte given by
$data, otherwise returns Boolean false. |
|
dfdl:setBits($bit1, $bit2, ... $bit8) |
Returns an unsigned byte being the value of the bit
positions provided by the Boolean arguments, where true is1, false is 0. The
number of arguments must be 8. |
|
dfdl:occursIndex() |
Returns the position of the current item of an array as an xs:nonNegativeInteger. The first element is at position 1. The
function may be used on non-array elements so long as it appears within the
dynamic scope of some array element. In
this case it returns the index of the current item of the innermost enclosing
array element. It
is a Schema Definition Error if this function is called when there is no
enclosing array element. |
|
dfdl:checkConstraints($node) |
Returns boolean true if the specified node value satisfies the XML schema facet constraints that are associated with it. Returns false if the specified node does not meet the constraints or does not exist. The facets that are checked are · minLength, maxLength · pattern · enumeration · maxInclusive, maxExclusive, minExclusive, minInclusive · totalDigits · fractionDigits See Section 5.2 for which facets are checked for each simple type. Additionally the XSD fixed property is checked. It is a Schema Definition Error if the argument is a complex element. |
|
Returns a string containing a DFDL string literal constructed from the $arg string argument. If $arg contains any '%' and/or space characters, then the return value replaces each '%' with '%%' and each space with '%SP;', otherwise $arg is returned unchanged. |
|
|
dfdl:decodeDFDLEntities
($arg) |
Returns a string constructed from the $arg string argument. If $arg contains syntax matching DFDL Character Entities syntax, then the corresponding characters are used in the result. Any characters in $arg not matching the DFDL Character Entities syntax remain unchanged in the result. It is a Schema Definition Error if $arg contains syntax matching DFDL Byte Value Entities syntax. |
|
dfdl:containsDFDLEntities($arg)
|
Returns a Boolean indicating whether the $arg string argument contains one or more DFDL entities. |
|
dfdl:timeZoneFromDateTime($arg) dfdl:timeZoneFromDate($arg) dfdl:timeZoneFromTime ($arg) |
Returns the timezone component, if any, of $arg as an xs:string.
The $arg is of type xs:dateTime, xs:date and xs:time respectively. If $arg has a timezone component, then the result is a string in the format of an ISO Time zone designator. Interpreted as an offset from UTC, its value may range from +14:00 to -14:00 hours, both inclusive. The UTC time zone is represented as "+00:00". If the $arg has no timezone component, then "" (empty string) is returned. |
|
dfdl:checkRangeInclusive($node,
$val1, $val2) |
Returns
boolean true if the specified node value is in the range given by $val1 and
$val2. |
Table 68 DFDL Functions
Notes:
dfdl:valueLength(path, lengthUnits) - returns the value length which excludes any padding or filling which might be added for a specified length
If the element declaration in the DFDL schema corresponding to the infoset item is not potentially represented, then the unpadded length is defined to be 0.
The value length includes the length contributions from introduced escape characters needed to escape contained delimiters (if such are defined, and will appear in the output representation).
The value length is also a function of the dfdl:encoding property. Multi-byte and variable-width character set encodings will commonly contribute more bytes to the value length than a single-byte character set would.
The value length is computed from the DFDL infoset value, ignoring the dfdl:length or dfdl:textOutputMinLength property. Other DFDL properties which affect the length of a text or binary representation are respected, it is only an explicit length which is ignored.
For
a complex type, this means a bottom up totaling of the dfdl:contentLength() of
all the contents and framing of the complex type.
dfdl:contentLength(path, lengthUnits) – returns the length of the content of the infoset data item as identified by the path argument. This includes padding or filling or truncation which might be carried out for a specified length item.
If the element declaration in the DFDL schema corresponding to the infoset item is not potentially represented (e.g., has an dfdl:inputValueCalc property), then the length is defined to be 0.
When unparsing with dfdl:lengthKind "explicit", the calculation of dfdl:contentLength() returns the value of the dfdl:length property.
For both dfdl:contentLength() and dfdl:valueLength(), the content length excludes any alignment filling as well as excluding any leading or trailing skip bytes. That is, the returned length is about the length of the content, and not about the position of that content in the output data stream.
Use dfdl:encodeDFDLEntities() when the value of a DFDL property is obtained from the data stream using an expression, and the type of the property is DFDL String Literal or List of DFDL String Literals, and the values extracted from the data stream could contain '%' or space characters. If the data already contains DFDL entities, this function should not be used.
Use dfdl:decodeDFDLEntities() when you need to create a value which contains characters for which DFDL Character Entities are needed. An example is to create data containing the NUL (character code 0) codepoint. This character code is not allowed in XML documents, including DFDL Schemas; hence, it must be specified using a DFDL Character Entity. Within a DFDL Expression, use this function to obtain a string containing this character.
There is sometimes a need to create a number
type from hex binary, and a hex binary type from a number. Accordingly the
following new DFDL specific functions are provided.
|
Function |
Meaning |
|
dfdl:byte ($arg) dfdl:unsignedByte ($arg) dfdl:short ($arg) dfdl:unsignedShort ($arg) dfdl:int ($arg) dfdl:unsignedInt ($arg) dfdl:long ($arg) dfdl:unsignedLong ($arg) |
These constructor functions behave
identically to the XPath 2.0 constructor functions of the same names, with
one exception. The argument can be a quoted string beginning with the letter
'x', in which case the remainder of the string is hexadecimal digits that
represent a big-endian twos complement representation of a binary number. |
|
dfdl:hexBinary ($arg) |
This constructor function behaves
identically to the XPath 2.0 constructor function of the same name, with one
exception. The argument can also be a long, unsignedLong, or any subtype
thereof, and in that case a xs:hexBinary value containing a number of hex
digits is produced. The ordering and number of the digits correspond to a
binary big-endian twos-complement implementation of the type of the argument.
Digits 0-9, A-F are used. The number of digits produced depends on
the type of $arg, being 2, 4, 8 or 16. If $arg is a literal number then the
type is the smallest signed type (long, int, short, byte) that can contain
the value. |
Table 69: DFDL Constructor Functions
Examples:
|
Function |
|
|
fn:error() fn:error($id as
xs:string) fn:error($id as
xs:string, $desc as xs:string,
$obj?) |
Causes
a processing error. This
function does not return a value. A processing error ends the evaluation of
the expression. The
$id argument is an error code identifier string that distinguishes this error
from others. The string should have the structure of an XSD QName; the namespace
URI conventionally identifies the component, subsystem, or authority
responsible for defining the meaning of the error code, while the local part
identifies the specific error condition. This information is incorporated
into any diagnostic messages created by the DFDL implementation in response
to the processing error in an implementation-dependent manner. If the $id
argument string does not have the form of an XSD QName, or the QName cannot
be interpreted as a meaningful namespace prefix and local identifier, then
the processing error still occurs but the diagnostic message is created in an
implementation-dependent manner. The
$desc is a natural-language description of the error condition. This string
will appear in any diagnostic messages created by the DFDL implementation in
response to the processing error. The
$obj? argument is an arbitrary value used to convey additional information
about the error and it is used to construct the diagnostic message in an implementation-dependent
manner. If
any argument is not supplied the processing error occurs but the diagnostic
message created is implementation-dependent. |
It is possible for
expressions and lengths of elements in a DFDL schema to interact badly,
resulting in circular deadlocks where an [7] expression
is unable to evaluate because it depends in some way on the length of something
that depends on the expression itself.
Expression deadlocks
are always Schema Definition Errors.
One scenario where such a deadlock can arise is due to what is called the interior-alignment problem. If a dfdl:outputValueCalc expression depends on the valueLength of a following complex element which due to interior alignments, has a length that depends on its starting position, then we have a circular deadlock.
A DFDL regular expression may be specified for the
dfdl:lengthPattern format property and the dfdl:testPattern property of the
dfdl:assert and dfdl:discriminator annotations. DFDL regular expressions do not
interpret DFDL entities.
A DFDL regular expression is defined by a set of valid pattern
characters. For portability,
a DFDL regular expression pattern is restricted to the inclusive subset of the
ICU regular expression [ICURegex] and the Java(R) 7 regular expression [JavaRegex] with the Unicode flags UNICODE_CASE and
UNICODE_CHARACTER_CLASS turned on.
DFDL regular expressions thereby conform to Unicode Technical Standard
#18 , Unicode Regular Expressions, level 1 [UnicodeRegex].
The following regular expression constructs
are not common to both ICU and Java(R) 7 and it is a Schema Definition Error if
any are used in a DFDL regular expression:
|
Construct |
Meaning |
Notes |
|
\N{UNICODE
CHARACTER NAME} |
Match the named
character |
ICU only |
|
\X |
Match a
Grapheme Cluster |
ICU only |
|
\Uhhhhhhhh |
Match the
character with the hex value hhhhhhhh |
ICU only |
|
(?# ... ) |
Free-format
comment |
ICU only |
|
(?w-w) |
UREGEX_UWORD -
Controls the behaviour of \b in a pattern |
ICU only |
|
(?d-d) |
UNIX_LINES -
Enables Unix lines mode |
Java 7 only |
|
(?u-u) |
UNICODE_CASE -
Enables Unicode-aware case folding |
Java 7 only (1) |
|
(?U-U) |
UNICODE_CHARACTER_CLASS
- Enables the Unicode version of predefined character classes and POSIX
character classes |
Java 7 only (2) |
Table 70 Disallowed Regular Expression Constructs
Notes:
(1) Implementations using Java 7 must set flag UNICODE_CASE by default to match ICU.
(2) Implementations using Java 7 must set flag UNICODE_CHARACTER_CLASS by default to match ICU.
Additionally, the behaviour of the word
character construct (\w) is not consistent in ICU and Java 7. In Java 7 \w is
[\p{Alpha}\p{gc=Mn}\p{gc=Me}\p{gc=Mc}\p{Digit}\p{gc=Pc}],
which is a larger set than ICU where \w is [\p{Ll}\p{Lu}\p{Lt}\p{Lo}\p{Nd}].
The use of \w is not recommended in DFDL regular expressions in conjunction with Unicode encodings, and an implementation must issue a warning if such usage is detected.
Character properties are detailed by the Unicode Regular Expressions [UnicodeRegex].
All locations must be properly initialized before writing so as to prevent accidental (or purposeful) transmission of data in the unused parts of data formats. Even when a DFDL description does not specify that data should be written to a particular part of the output representation, a defined pattern should always be written.
When unparsing data it is a Schema Definition Error if the representation properties that control filling and padding are not defined by the DFDL schema. The DFDL processor must fail if they are not defined so that it is certain no region of the output data has unspecified contents.
If regions within a DFDL-described data object are encrypted, then when decrypting them proper means must be used to assure secure passage of passwords to the decrypting software. Such means are beyond the scope of the DFDL language specification.
In addition, if encryption passwords/keys are stored in DFDL schema-described data, then proper means must be used to assure that the decrypted form of these passwords is not revealed. Such means are beyond the scope of the DFDL language specification.
Michael J. Beckerle, (corresponding author)
Tresys Technology/Owl Cyber Defense
Columbia, MD
USA
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Stephen M. Hanson, (corresponding author)
IBM Software Group,
Hursley,
Winchester, UK
Email<EMAIL_ADDRESS>Alan W. Powell,
Email<EMAIL_ADDRESS>We
greatly acknowledge the contributions made to this document by the following
and all the other people who provided constructive and valuable input in the
group discussions.
Tim
Kimber, IBM Software Group, Hursley, UK
Suman
Kalia, IBM Software Group, Markham, Ontario, Canada
Stephanie
Fetzer, IBM Software Group, Charlotte, USA
Martin
Westhead, Groupon, Stanford Univ, CA, USA
James
Myers, Univ. of Michigan, MI, USA
Jonathan
W. Cranford, MITRE Corporation, USA
Tom
Sugden, UBS Investment Bank, UK
Tara
Gibson, PNNL, Richland, WA, USA
Robert
McGrath, NCSA, Urbana-Champaign, IL, USA
Geoff
Judd, IBM Software Group, Hursley, UK
Dewey
M. Sasser, MA, USA
David
A. Loose, Opera Solutions, MA, USA
Eric
S. Smith, eBay Enterprise, MA, USA
Kristoffer
H. Rose, Two Sigma Investments, NY, USA
Simon
Parker, Bloomberg Polar Lake, Dublin, IE
Peter
A. Lambros, IBM Software Group, Hursley, UK
Dave
Glick, SIPC, USA
Steve
Marting, Progeny, USA
Alejandro Rodriguez, Amazon, WA, USA
The OGF takes no position regarding the
validity or scope of any intellectual property or other rights that might be
claimed to pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights might or
might not be available; neither does it represent that it has made any effort
to identify any such rights. Copies
of claims of rights made available for publication and any assurances of
licenses to be made available, or the result of an attempt made to obtain a
general license or permission for the use of such proprietary rights by implementers
or users of this specification can be obtained from the OGF Secretariat.
The OGF invites any interested party to
bring to its attention any copyrights, patents or patent applications, or other
proprietary rights which may cover technology that may be required to practice
this recommendation. Please address
the information to the OGF Executive Director.
This document and the information contained herein is provided on an "As Is" basis and the OGF disclaims all warranties, express or implied, including but not limited to any warranty that the use of the information herein will not infringe any rights or any implied warranties of merchantability or fitness for a particular purpose.
Copyright (C) Open Grid Forum (2005-2014). Some Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included as references to the derived portions on all such copies and derivative works. The published OGF document from which such works are derived, however, may not be modified in any way, such as by removing the copyright notice or references to the OGF or other organizations, except as needed for the purpose of developing new or updated OGF documents in conformance with the procedures defined in the OGF Document Process, or as required to translate it into languages other than English. OGF, with the approval of its board, may remove this restriction for inclusion of OGF document content for the purpose of producing standards in cooperation with other international standards bodies.
The limited
permissions granted above are perpetual and will not be revoked by the OGF or
its successors or assignees.
ICU - Copyright (c) 1995-2014 International Business Machines Corporation and others
XPATH - Copyright © 2007 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.
|
"Introduction to ASN.1", |
|
|
X.690 : Information
technology - ASN.1 encoding rules: Specification of Basic Encoding Rules
(BER), Canonical Encoding Rules (CER) and Distinguished Encoding Rules (DER) |
|
|
"ASN.1 Encoding Control Notation," |
|
|
"ASN.1 Packed Encoding Rules (PER)," http://www.itu.int/ITU-T/studygroups/com17/languages/X.691-0207.pdf |
|
|
"Avro," |
|
|
"Binary Format Description (BFD) Language," http://collaboratory.emsl.pnl.gov/sam/bfd/ |
|
|
Disc Interchange Service Company, Inc., Westford, MA, USA., "EBCDIC to ASCII Conversion of Signed Fields," http://www.3480-3590-data-conversion.com/article-signed-fields.html |
|
|
"Coded Character Set Identifiers (CCSID)," http://www.ibm.com/software/globalization/ccsid/ccsid_registered.html |
|
|
W3C, "Efficient XML Interchange (EXI) Format 1.0 (Second Edition)," http://www.w3.org/TR/exi |
|
|
Hierarchical Data
Format (HDF) |
|
|
IANA, "Character Sets," |
|
|
icu-project.org, "Formatting Dates and Times," |
|
|
icu-project.org, "icu::DecimalFormat Class Reference," http://icu.sourceforge.net/apiref/icu4c/classDecimalFormat.html#_details |
|
|
icu-project.org, "Locale," |
|
|
icu-project.org, "Regular Expressions," |
|
|
Universal Coded Character Set http://standards.iso.org/ittf/PubliclyAvailableStandards/c056921_ISO_IEC_10646_2012.zip |
|
|
Date and Time Format - ISO 8601 |
|
|
Oracle, "Class Pattern," http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html |
|
|
Michael J Beckerle, Stephen M Hanson, Alan W Powell. GFD-P-R.174: Data Format Description Language (DFDL) v1.0 Specification. Open Grid Forum. January 2011. |
|
|
IANA - Internet Assigned Numbers Authority, "Time Zone Database," http://www.iana.org/time-zones |
|
|
Introducing JSON |
|
|
Network Common Data Form (NetCDF) |
|
|
OMG "CAM" TD Model: Object Management Group (OMG) "UML Profile and Interchange Models for Enterprise Application Integration (EAI) Specification" formal/04-03-26, March 2004. Section 7.3.2. Available at http://www.omg.org/cgi-bin/doc?formal/2004-03-26Available at http://www.omg.org/cgi-bin/doc?formal/2004-03-26 |
|
|
William H. Burge, "Recursive Programming Techniques," Addison-Wesley Longman, Incorporated, 1975. ISBN-10:<PHONE_NUMBER> |, ISBN-13:<PHONE_NUMBER>505 0 |
|
|
S. Bradner, "RFC 2119: Key words for use in RFCs to Indicate Requirement Levels," IETF (Internet Engineering Task Force). RFC 2119: Key words for use in RFCs to Indicate Requirement Levels. S. Bradner. 1997. |
|
|
W3C, "W3C XML Schema Definition Language (XSD): Component Designators," http://www.w3.org/TR/xmlschema-ref/ |
|
|
M. Slee, A. Agarwal and M. Kwiatkowski, "Thrift: Scalable Cross-Language Services Implementation," http://thrift.apache.org/static/files/thrift-20070401.pdf |
|
|
Unified
Modeling Language http://www.uml.org/ |
|
|
The Unicode Consortium, "Unicode," http://www.unicode.org/. |
|
|
Unicode, Inc., "Unicode Common Locale Data Repository," http://sites.google.com/site/cldr/. |
|
|
Unicode, Inc., "Unicode Regular Expressions," http://www.unicode.org/reports/tr18/ |
|
|
Unicode, Inc., "Unicode Locale Data Markup Language (LDML)," http://www.unicode.org/reports/tr35/. |
|
|
[Walmsley] |
Definitive XML
Schema (Walmsley, ISBN 0-13-065567-8) page 390, Section 17.8, Table 17-6 says
"A child element-type name which must be prefixed if it is in a
namespace". |
RFC 4506: XDR: External Data Representation Standard IETF (Internet Engineering Task Force), 2006. http://tools.ietf.org/html/rfc4506 |
|
|
W3C, "Extensible Markup Language (XML) 1.0 (Fifth Edition)," 26 November 2008. http://www.w3.org/TR/REC-xml |
|
|
W3C, "Extensible Markup Language (XML) 1.1 (Second Edition)," 16 August 2006. http://www.w3.org/TR/xml11/ |
|
|
W3C, "XML Information Set (Second Edition)," 4 February 2004. http://www.w3.org/TR/xml-infoset |
|
|
W3C, "Namespaces in XML 1.0 (Third Edition)," 8 December 2009. http://www.w3.org/TR/REC-xml-names/ |
|
|
W3C, "XML Schema," http://www.w3.org/XML/Schema |
|
|
W3C, "XML Path Language (XPath) 2.0 (Second Edition)," http://www.w3.org/TR/xpath20/ |
|
|
W3C, XML Schema Part 1: Structures http://www.w3.org/TR/xmlschema-1/ XML Schema Part 2: Datatypes http://www.w3.org/TR/xmlschema-2/ , 28
October 2004. |
dfdl:escapeKind 'escapeCharacter', dfdl:escapeCharacter '/', dfdl:escapeEscapeCharacter '/', dfdl:separator ';', dfdl:extraEscapedCharacters '?'
|
Logical Data |
Physical Data / Representation |
|
…………….……………….. |
…………….……………….. |
|
……………/.……………….. |
……………//.……………….. |
|
……………/.…/…………….. |
……………//.…//…………….. |
|
……………//.……………….. |
……………////.……………….. |
|
/…………….……………….. |
//…………….……………….. |
|
…………….………………../ |
…………….………………..// |
|
/……………/.……………….. |
//……………//.……………….. |
|
……………./………………../ |
…………….//………………..// |
|
…………….;……………….. |
……………./;……………….. |
|
……………./;……………….. |
…………….///;……………….. |
|
;…………….……………….. |
/;…………….……………….. |
|
…………….?……………….. |
……………./?……………….. |
Table 71 Examples of Escape Character Same as dfdl:escapeEscapeCharacter
dfdl:escapeKind 'escapeCharacter', dfdl:escapeCharacter '/', dfdl:escapeEscapeCharacter '%%’[31], dfdl:separator ‘;’, dfdl:extraEscapedCharacters ‘?’
|
Logical Data |
Physical Data / Representation |
|
…………….……………….. |
…………….……………….. |
|
……………/.……………….. |
……………%/.……………….. |
|
……………/.…/…………….. |
……………%/.…%/…………….. |
|
……………//.……………….. |
……………%/%/.……………….. |
|
/…………….……………….. |
%/…………….……………….. |
|
…………….………………../ |
…………….………………..%/ |
|
/……………/.……………….. |
%/……………%/.……………….. |
|
……………./………………../ |
…………….%/………………..%/ |
|
…………….;……………….. |
……………./;……………….. |
|
……………./;……………….. |
…………….%//;……………….. |
|
;…………….……………….. |
/;…………….……………….. |
|
…………….?……………….. |
……………./?……………….. |
|
…………….%……………….. |
…………….%……………….. |
|
…………….%/……………….. |
…………….%%/……………….. |
|
……………./%……………….. |
…………….%/%……………….. |
Table 72 Examples (1) of Escape Character Different from dfdl:escapeEscapeCharacter
Dfdl:escapeKind 'escapeCharacter', dfdl:escapeCharacter '/', dfdl:escapeEscapeCharacter '%%', dfdl:separator 'sep', dfdl:extraEscapedCharacters '?'
|
Logical Data |
Physical Data / Representation |
|
…………….sep……………….. |
……………./sep……………….. |
|
……………./sep……………….. |
…………….%//sep……………….. |
|
sep…………….……………….. |
/sep…………….……………….. |
Table 73 Examples (2) of Escape Character Different from dfdl:escapeEscapeCharacter
dfdl:escapeKind 'escapeBlock', dfdl:escapeBlockStart '[', dfdl:escapeBlockEnd ']', dfdl:escapeEscapeCharacter '%%', dfdl:separator ';', dfdl:extraEscapedCharacters '?'
|
Logical Data |
Physical Data / Representation |
|
…………….……………….. |
…………….……………….. |
|
[…………….……………….. |
[[…………….………………..] |
|
]…………….……………….. |
]…………….……………….. |
|
……………[.……………….. |
……………[.……………….. |
|
……………].……………….. |
……………].……………….. |
|
…………….………………..] |
…………….………………..] |
|
[[…………….……………….. |
[[[…………….………………..] |
|
…………….………………..]] |
…………….………………..]] |
|
…………….[[……………….. |
…………….[[……………….. |
|
…………….]]……………….. |
…………….]]……………….. |
|
[…………….………………..] |
[[…………….………………..%]] |
|
[…………….]……………….. |
[[…………….%]………………..] |
|
…………….[………………..] |
…………….[………………..] |
|
[……………[.………………..] |
[[……………[.………………..%]] |
|
[…………….]………………..] |
[[…………….%]………………..%]] |
|
[[…………….………………..] |
[[[…………….………………..%]] |
|
[…………….………………..]] |
[[…………….………………..%]%]] |
|
[[…………….………………..]] |
[[[…………….………………..%]%]] |
|
…………….%……………….. |
…………….%……………….. |
|
…………….%%……………….. |
…………….%%……………….. |
|
…………….%[……………….. |
…………….%[……………….. |
|
…………….%]……………….. |
…………….%]……………….. |
|
%[…………….……………….. |
%[…………….……………….. |
|
…………….………………..%] |
…………….………………..%] |
|
%[…………….………………..%] |
%[…………….………………..%] |
|
[…………%….………………..] |
[[…………%….………………..%]] |
|
[…………%]….………………..] |
[[…………%%]….………………..%]] |
|
…………….;……………….. |
[…………….;………………..] |
|
…………….%;……………….. |
[…………….%;………………..] |
|
[…………….;………………..] |
[[…………….;………………..%]] |
|
…………….?……………….. |
[…………….?………………..] |
Table 74 Examples of Escape Block with Different Start and End Characters
dfdl:escapeKind 'escapeBlock', dfdl:escapeBlockStart ' '', dfdl:escapeBlockEnd ''', dfdl:escapeEscapeCharacter '%', dfdl:separator ';', dfdl:extraEscapedCharacters '?'
|
Logical Data |
Physical Data / Representation |
|
…………….……………….. |
…………….……………….. |
|
'…………….……………….. |
'%'…………….………………..' |
|
……………'.……………….. |
……………'.……………….. |
|
…………….………………..' |
…………….………………..' |
|
''…………….……………….. |
'%'%'…………….………………..' |
|
…………….………………..'' |
…………….………………..'' |
|
…………….''……………….. |
…………….''……………….. |
|
'…………….………………..' |
'%'…………….………………..%'' |
|
'…………….'……………….. |
'%'…………….%'………………..' |
|
…………….'………………..' |
…………….'………………..' |
|
'……………'.………………..' |
'%'……………%'.………………..%'' |
|
''…………….………………..' |
'%'%'…………….………………..%'' |
|
'…………….………………..'' |
'%'…………….………………..%'%'' |
|
''…………….………………..'' |
'%'%'…………….………………..%'%'' |
|
…………….%……………….. |
…………….%……………….. |
|
…………….%%……………….. |
…………….%%……………….. |
|
…………….%'……………….. |
…………….%'……………….. |
|
%'…………….……………….. |
%'…………….……………….. |
|
…………….………………..%' |
…………….………………..%' |
|
'…………….………………..%' |
'%'…………….………………..%%'' |
|
%'…………….………………..%' |
%'…………….………………..%' |
|
'…………%….………………..' |
'%'…………%….………………..%'' |
|
'…………%'….………………..' |
'%'…………%%'….………………..%'' |
|
…………….;……………….. |
'…………….;………………..' |
|
…………….%;……………….. |
'…………….%;………………..' |
|
'…………….;………………..' |
'%'…………….;………………..%'' |
|
…………….?……………….. |
'…………….?………………..' |
Table 75 Examples of Escape Block with Same Start and End Characters
DFDL is intended to be a description language. That is, the capture of a data format should be as descriptive/declarative as possible.
An additional quite critical goal for DFDL is that it allows very high performance implementations, including use of parallel processing wherever possible.
DFDL contains an expression language with variables for use in creating parameterized DFDL schemas.
However, the way variables can be used in DFDL is quite constrained. Specifically, the variables are single-assignment.
Single-assignment variables solve a number of problems.
First, they keep the schema more declarative, because the name of a variable represents a value, not a location. Before assignment, the value is not yet known, after the assignment the value is known, but the consumer of the value need only know the name, and need not be aware of the mechanism by which it gets its value or when.
Second, single-assignment variables avoid over-constraining the implementation, thereby preserving the potential for high-performance and parallel processing.
Some digression is useful here: Any variable creates a data dependency in order of processing. The part of the schema reading/using the variable's value depends upon the data value coming from the part of the schema providing that value. This kind of data dependency is inherent and inescapable. Values must be created before they can be used.
However, if you consider a variable to be a location that can be assigned repeatedly, then things are more complex because you not only have data dependency on the value (one part of the schema writes the location, another reads that location), but you have the dependency in the other direction: you must read the location before it can be used again for the next value. This is usually called anti-dependency. Anti-dependency is the enemy of high-performance and parallel execution. It forces specific and artificial sequential ordering on things that is due to the way variable names are allocated to storage locations.
If variables are single-assignment only, then only data-dependencies exist. Anti-dependencies don't exist, and implementations are free to work in any way consistent with the (inescapable) data dependencies.
A DFDL String Literal describes characters or bytes that can appear in the
data. The syntax of a DFDL String Literal is described in Section 6.3. This
appendix describes how a DFDL String Literal must be processed by a DFDL
processor.
A DFDL String literal is a sequence of one or more string literal parts. Each string literal part describes zero or
more characters or exactly one byte. To process a DFDL string literal correctly
it is necessary to parse it into its string literal parts using the grammar in Table 2 DFDL
Character Entity, Character Class Entity, and Byte Value Entity Syntax of Section 6.3.1 DFDL
String Literals.
When parsing, a DFDL processor must be able to recognize a DFDL String
Literal in the data. The recognition algorithm is used whenever the DFDL
processor needs to process a property of type 'DFDL String Literal'.
The recognition algorithm is as follows:
for each DfdlStringLiteralPart
if this DfdlStringLiteralPart is found in the data at the current offset
advance the offset by the width of the characters matched
else
return false
end if
return true
Each type of string literal part and/or the DfdlESEntity is recognised as
described in the table below:
|
String Literal Part |
Recognition algorithm |
|
LiteralString |
The literal
string is specifying a sequence of Unicode characters. The string part is
recognized if the same Unicode characters are found in the data at the
current offset. Implementation
note: It is not necessary for either the LiteralString or the data to be
encoded using a Unicode encoding. The matching algorithm must operate as if
the LiteralString and the data were both converted to Unicode before
comparison. |
|
DfdlCharEntity |
The entity is specifying a single Unicode character. The recognition algorithm is the same as for a single-character
LiteralString. |
|
ByteValue
|
The entity is specifying a single raw byte value. The string part is recognized if the literal byte in the data at the
current byte offset matches the raw byte value. This entity is specifying a raw byte value and not a character. It
will not always be safe to interpret the byte value as a character in the
component's encoding. |
|
DfdlNLEntity |
The entity is specifying a newline character or character sequence. The string part is recognized if a newline character sequence is found
in the data at the current byte offset: The newline character sequences are defined in Table 4
DFDL Character Class Entities The recognition algorithm must be greedy. If the CR character is
matched then the DFDL processor must check to see whether it is followed by
an LF character and if so, consume that character also. |
|
DfdlWSPEntity |
The entity is specifying a single whitespace character. The string part if recognized if one of the whitespace characters is
found in the data at the current byte offset. The whitespace
characters are defined in Table
4
DFDL Character Class Entities |
|
DfdlWSPStarEntity |
The entity is specifying zero or more whitespace characters. The string part is recognized when the entire sequence of whitespace
characters starting at the current offset has been consumed. The recognition
of this entity cannot fail because it can match zero whitespace characters. The recognition algorithm must be greedy. All possible whitespace
characters must be consumed. |
|
DfdlWSPPlusEntity |
The entity is specifying one or more whitespace characters. The string part is recognized when the entire non-empty sequence of
whitespace characters starting at the current offset has been consumed. The recognition algorithm must be greedy. All possible whitespace
characters must be consumed. |
|
DfdlESEntity |
The entity is specifying an empty string. The string part is recognized if the data available for matching is
zero-length. This entity is only allowed in contexts
where the available data will be constrained by other DFDL properties. |
Table 76 Recognizing DFDL String Literal Part
This appendix provides a specification template for the definition of DFDL standard encodings, and defines two such encodings for use in DFDL schemas.
These encodings are optional - DFDL implementations are not required to support them to be considered to be in conformance with the DFDL specification.
However, if implemented, then a conformant DFDL implementation must follow the specification in this appendix.
Additional encoding names are needed for several reasons:
All indexing of positions are 1-based meaning the first position is
position 1.
Bytes are unsigned integers with values from 0 to 255 (decimal).
Hexadecimal numbers have the prefix "0x".
Decimal numbers have no prefix when the context is clear, and are explicitly labeled with (decimal) to resolve ambiguity.
A DFDL standard encoding specification must include:
This encoding is used by MIL-STD-2045 [MILSTD2045] and a variety of related binary data standards.
X-DFDL-US-ASCII-7-BIT-PACKED
The characters are exactly the 128 US-ASCII characters with the same code unit values.
Fixed width. Each code unit is 7-bits wide.
This encoding differs from US-ASCII in that it specifically uses only 7-bits per character within each byte[33]. A subsequent character or the next data field may begin in the very next bit after a 7-bit character code of this encoding.
Hence, 8 characters will fit exactly into 7 bytes of data as no bits are wasted.
1 bit. That is, a code unit may begin on any bit boundary.
Not applicable - code units are always smaller than 1 byte.
The table below shows how the 8 character string 'UNIT1234' is encoded into 7 bytes, when the bit order is least-signficant-bit first.
|
Logical character |
7-bit code unit |
Bit placement 27
20 |
Byte value |
Byte number |
|
U |
1010101 |
X1010101 |
|
|
|
N |
1001110 |
0XXXXXXX XX100111 |
0x55 |
1 |
|
I |
1001001 |
01XXXXXX XXX10010 |
0x67 |
2 |
|
T |
1010100 |
100XXXXX XXXX1010 |
0x92 |
3 |
|
1 |
0110001 |
0001XXXX XXXXX011 |
0x1A |
4 |
|
2 |
0110010 |
10010XXX XXXXXX01 |
0x93 |
5 |
|
3 |
0110011 |
110011XX XXXXXXX0 |
0xCD |
6 |
|
4 |
0110100 |
0110100X |
0x68 |
7 |
Table 77: Example of DFDL Standard Encoding X-DFDL-US-ASCII-7-BIT-PACKED
Below is the same data shown horizontally with the bytes numbered from right to left. The bits corresponding to the characters are shown in distinct colors. The characters that each bit contributes to are written below each of the bits and in the corresponding color.
01010101 01100111 10010010 00011010 10010011 11001101 01101000
Byte -----1-- -----2-- -----3-- -----4-- -----5-- -----6-- -----7--
Char NUUUUUUU IINNNNNN TTTIIIII 1111TTTT 22222111 33333322 44444443
This kind of data is often better visualized by numbering the bytes in the opposite direction that is right to left, which leads to this presentation:
01101000 11001101 10010011 00011010 10010010 01100111 01010101
Byte -----7-- -----6-- -----5-- -----4-- -----3-- -----2-- -----1--
Char 44444443 33333322 22222111 1111TTTT TTTIIIII IINNNNNN NUUUUUUU
In the above, the bits corresponding to each character code unit are more easily recognized, but the characters appear right-to-left (i.e., backward for English).
The bits below represent a 3-bit unsigned integer containing value 7, followed by the ASCII string 'ABC' followed by the ASCII DEL character (character code 0x7F). This illustrates a string not beginning on a byte boundary. Again the bit ordering is least-signficant-bit first.
The bits are written in increasing position and place value from right-to-left:
1111111
1000011 1000010 1000001 111
DEL C B A 7
In the above example, if we number the bits from the right, starting with position 1, the character code for 'A' occupies bits 4 to 10. If we divide the data above into bytes with vertical bars we must start on the right to get:
char: DEL C B
A
bits: 01111111 | 1000011 1
|000010 10 | 00001 111
byte: 7F
87
1C
0F
This encoding is used by MIL-STD-6016D (See [MILSTD6016]) and a variety of related binary data standards.
X-DFDL-US-ASCII-6-BIT-PACKED
The characters are a subset of the 128 US-ASCII characters. The assignment of numeric code units to the characters is different for some of the characters and the same for others.
The correspondence of ASCII characters to their code point values is given by the table below.
|
Character |
Code Point Value (Decimal) |
Notes |
|
@ |
0 |
These characters' code points are the same as the US-ASCII code point, minus 64 (decimal) |
|
A to Z |
1 through 26 |
|
|
[ |
27 |
|
|
\ |
28 |
|
|
] |
29 |
|
|
^ |
30 |
|
|
_ |
31 |
|
|
SPACE |
32 |
These characters' code points are the same as in US-ASCII. |
|
! |
33 |
|
|
" |
34 |
|
|
# |
35 |
|
|
$ |
36 |
|
|
% |
37 |
|
|
& |
38 |
|
|
' APOSTROPHE |
39 |
|
|
( |
40 |
|
|
) |
41 |
|
|
* |
42 |
|
|
+ |
43 |
|
|
, COMMA |
44 |
|
|
- |
45 |
|
|
. PERIOD |
46 |
|
|
/ |
47 |
|
|
0 to 9 |
48 through 57 |
|
|
: |
58 |
|
|
; |
59 |
|
|
< |
60 |
|
|
= |
61 |
|
|
> |
62 |
|
|
? |
63 |
Table 78: Translation Table for DFDL Standard Encoding X-DFDL-US-ASCII-6-BIT-PACKED
Fixed width. Each code unit is 6-bits wide. A subsequent character or the next data field may begin in the very next bit after a 6-bit character code of this encoding.
Hence, 4 characters will fit into 3 bytes of data as no bits are wasted.
1 bit. That is, a code unit may begin on any bit boundary.
Not applicable. Code units are always less than 1 byte in size.
The table below shows how the 8 character string '1234' is encoded into 3 bytes. The bit order is least-significant-bit first:
|
Logical character |
6-bit code unit |
Bit placement 27
20 |
Byte value |
Byte number |
|
1 |
110001 |
XX110001 |
|
|
|
2 |
110010 |
10XXXXXX XXXX1100 |
0xB1 |
1 |
|
3 |
110011 |
0011XXXX XXXXXX11 |
0x3C |
2 |
|
4 |
110100 |
110100XX |
0xD3 |
3 |
Table 79: Example of DFDL Standard Encoding X-DFDL-US-ASCII-6-BIT-PACKED
|
[CCSID] |
"Coded
Character Set Identifiers (CCSID),"
http://www.ibm.com/software/globalization/ccsid/ccsid_registered.html |
|
Michael J Beckerle, Steven M Hanson, Alan W Powell. Data Format Description Language (DFDL) v1.0 Specification. Open Grid Forum. (http://redmine.ogf.org/dmsf/dfdl-wg) GFD-P-R.207 (2014) |
|
|
[IANA] |
IANA,
"Character Sets," http://www.iana.org/assignments/character-sets |
|
Wikipedia:
Baudot code, http://en.wikipedia.org/wiki/Baudot_code#ITA2 |
|
|
CONNECTIONLESS DATA TRANSFER APPLICATION LAYER STANDARD, MIL-STD-2045-47001D w/CHANGE 1, 23 June 2008 (available publicly from US Dept. of Defense at http://assistdocs.com/) |
|
|
Tactical Data Link Message Standard MIL-STD-6016 This standard requires authorization to access. http://en.wikipedia.org/wiki/Link_16 provides some information about the standard. http://www.assistdocs.com/search/document_details.cfm?ident_number=123964 provides links for those with authorization to access the various versions of the standard. |
[1] A very complex analysis is needed to identify this
sort of grammar ambiguity in general. While we believe this may be decidable
for DFDL v1.0, future versions of DFDL may add features (such as recursive
types) which make this analysis undecidable.
[2] X-DFDL-US-ASCII-7-BIT-PACKED is a DFDL standard encoding, which uses the US-ASCII characters, but each code unit is stored occupying only 7 bits, not a whole 8-bit-byte. DFDL standard encodings are defined in a separate specification. See Section 34 Appendix D: DFDL Standard Encodings.
[3] The purpose of this member is to support construction of a W3C standard Post Schema Validation Infoset (PSVI) from a DFDL Infoset.
[4] Also to support PSVI construction.
[5] The purpose of unions is to allow multiple constraints via facets such as multiple independent range restrictions on numbers. This enhances the ability to do rich validation of data.
[6] By reserved we mean that conforming DFDL v1.0 implementations MAY NOT assign semantics to them.
[7] A "top-down" parser built from a set of mutually-recursive procedures or a non-recursive equivalent where each such procedure usually implements one of the productions of the grammar. Thus the structure of the resulting program closely mirrors that of the grammar it recognises. See [RDP].
[8]
DFDL discriminators are
described in section: 7.4 The
dfdl:discriminator Statement Annotation Element.
[9] This is a corner case that only happens when type is xs:string or xs:hexBinary and the maxLength facet is 0. Such an element can only be of length 0.
[10] It is a Schema Definition Error if a complex element has XSDL nillable ‘true’ and dfdl:lengthKind ‘implicit’.
[11] The rationale for this is that otherwise this could give rise to misleading error messages where the parser reported that required child elements were missing required occurrences. (This is consistent with XML Schema validation, where if a required element is missing, it gets reported as such, and there is nothing reported about its children).
[12] If other than ‘none’, either an initiator, terminator or both must have been found in the data stream.
[13]
If other than ‘none’, either an initiator, terminator or both must
have been found in the data stream.
[14] The concept of native-endian is avoided in DFDL since a DFDL schema containing such a property binding would not contain a complete description of data, but rather an incomplete one which would behave differently based on characteristics of the machine and implementation where the DFDL processor is executed. In DFDL this same behavior is achieved through the use of explicit parameterization using DFDL variables to set dfdl:byteOrder. See Section 7.7.2 Predefined Variables.
[15] CCSID stands for Coded Character Set ID, a decimal number syntax for a coded charater set specifier. [CCSID].
[16] The concept of native character encoding is avoided in DFDL since a DFDL schema containing such a property binding would not contain a complete description of data, but rather an incomplete one which would behave differently based on characteristics of the operating environment where the DFDL processor executes. In DFDL this same behavior is achieved through the use of explicit parameterization using DFDL variables to set dfdl:encoding. See Section 7.7.2 Predefined Variables.
[17]
While UTF-8 has no true notion of byte-order, the 16-bit codepoint for a
byte-order mark is often translated into a 3-byte utf-8 sequence of bytes that
appear at the start of a document. This information is helpful to establish
that the document is encoded in Unicode (specifically UTF-8).
[18] A
way of eliminating a BOM at the beginning of a string so that it does not end
up in the infoset is to model it as a separate hidden element before the
string. This BOM element can be either required or optional depending on
whether a BOM is expected or optional at the beginning of the string.
[19] Used by data format MIL-STD-2045
[20]This feature allows DFDL to describe the needed “one more level” of prefix that is needed for modeling an ASN.1 format, but without the complexities of general recursion.
[21] Type decimal must be a minimum of 8 bits because lengthUnits 'bits' is not allowed for xs:decimal.
[22] This is the smallest pattern
that contains all the digit-only symbols. SSS is the minimum precision that must be supported for fractional seconds, but in can be more, hence why 'not less
than 9 bytes'.
[23] Reference for this CA Realia 0x20 overpunch for negative sign is the article: "EBCDIC to ASCII Conversion of Signed Fields" [CARealia] where it says:
COBOL compilers that run on ASCII platforms have a "signed" data type that operates in a similar manner to the EBCDIC Signed field -- that is, they over punch the sign on the LSD. However, this is not standardized in ASCII, and different compilers use different overpunch codes. For example, Computer Associates' Realia compiler uses a 30 hex for positive values and a 20 hex for negative values, but Micro Focus® and Microsoft® use 30 hex for positive values and 70 hex for negative values.
[24] Implementations which use current versions of the popular ICU library will allow 309 digits as maximum integer digits.
[25] Note that XSDL 1.1 moved to IEEE 754-2008 only because of new decimal support, and not for enhanced float support. That's why in XSDL 1.1 there are still just the xs:float and xs:double built-in types. Any future support for half-precision and quad-precision in XSDL would very likely be implemented by adding new built-in types that derive from xs:anySimpleType. It is likely therefore that future DFDL support for half-precision and quad-precision will build on XSDL.
[26] Note that DFDL does not support a pure month or day or year, as it does not support the XSD simple types xs:gMonth, xs:gDay, and xs:gYear.
[27] Absent representation implies processing error for ‘implicit’ when less than or equal to minOccurs.
[28] Absent representation always implies zero-length. Nil, empty, and normal representations can also be zero-length with the right combinations of properties. See Section 9.2.5 Zero-length Representation.
[29] An Infoset item value of nil means the Infoset item [nilled] member is true, and the [dataValue] member has no value. See Section 4.1.2 Element Information Items.
[30]The NTE segment in the X12 EDI standard is an example of a floating element.
[31] The ‘%’ character needs escaping when used as a literal character in DFDL properties. It is escaped by doubling it, i.e., preceding it by another ‘%’ character.
[32] It is anticipated this appendix will be migrated into
a separate document to facilitate the addition of new encodings without revising
the DFDL specification.
[33] Though US-ASCII contains only 128 characters DFDL assumes the US-ASCII encoding to use one character per 8-bit byte.
[34]
This list is maintained separately from the other
references in the specification to facilitate future migration of this material
into a separate document.
[MJB2]This language was moved from below. There is no change to the behavior for prefixed unparsing.
[MJB3]These two bullets should be indented one more level, but MS-Word will not do so.
[MJB4]These two bullets should be indented one more level but MS-Word will not do so.
[MJB5]Test that hyperlink works to reference.
[MJB6]Verify formatting of table once change is accepted.


This extension adds the "Quicker..." context menu entry which opens a lightweight dialog to select all available context menu entries more easily by typing parts of the name or choosing one stored in the history.
Usage
The Quicker Context Dialog can be completely controlled using the keyboard:


API Quote Machine
API Quote Machine
Generate Quote
Click the button to generate a quote here.
Tweet This Quote
Previously Generated Quotes