This is a OGF DFDL Workgroup draft document in
preparation.
Use with caution. The material herein is
subject to change, and URL links to this version will be broken when newer
revisions and the final version are released.
This draft document is an updated version of
GFD-P-R.207 with the errata and clarifications applied to it that have been
gathered since its first publication.
The updates are
indicated with change bars, text coloring, and text strikethrough in the style
of this sentence.
Data Format Description Language (DFDL) v1.0 Specification
Status of This Document
Grid Final Draft (GFD)
Obsoletes
This document obsoletes GFD-P-R.174 dated January 2011 [OBSOLETE_DFDL].
Copyright Notice
Copyright © Global Grid Forum (2004-2006). Some Rights Reserved. Distribution is unlimited.
Copyright © Open Grid Forum (2006-2019). Some Rights Reserved. Distribution is unlimited
Abstract
This document provides a definition of a standard Data
Format Description Language (DFDL).
This language allows description of text, dense binary, and legacy data
formats in a vendor-neutral declarative manner. DFDL is an extension to the XML
Schema Description Language (XSDL).
Document Change History
|
Revision ID |
Date |
Description |
|
R11 |
2019-12-09 |
Edited in
minor/typo fixes. |
|
R10 |
2019-12-01 |
Erratum 5.38
Updated UML Diagram in Section 4.1 to have missing members on element class. Marked some
comments as resolved. Date changed to
December 2019 |
|
R09 |
2019-11-25 |
Errata Merged 5.39 Section
15.1.1 Clarification for choice branches that are zero-occurrences arrays. 5.40 Section 14.1
Empty Sequence and Hidden Group References 5.41 Section 2.6
Dynamic Type error must cause SDE. 5.42 Section 21
utf16Width ‘variable’ property value is optional. 5.43 Sections
6.3.1,21,4.1.2, <IP_ADDRESS>, 13.3, 22.1.1, 2.2.1 Bidirectional text properties
other than textBidi=”no” removed and reserved for future use. 5.44 Sections,
9.2, 12.3.3, 23.5.3 Data syntax grammar corrections related to
ComplexContent. 5.45 Sections
9.2, 13.2.1, 23.5.3. Data syntax grammar corrections related to SimpleContent
and SimpleValue. 5.46 New Section
23.6 on Circular Deadlocks. 5.47 Section
<IP_ADDRESS> Detection of Encoding/Decoding Errors. 5.48 Section
23.5.5 fn:error() function. 5.49 Section 5.3
Allowing import of other annotation language schemas. 5.50 Sections
4.1.1, 9.2, 11, 11.1 Remove Byte-order-mark (BOM) processing. 5.51 Section 12.2
Entity %ES; allowed alone in terminator when not delimited. 5.52 Section 15
Clarify choiceDispatchKey evaluating to empty string is SDE. 5.53 Section
13.16 Clarifications on nilValueDelimiterPolicy. 5.54 Section 12.2
Clarifications on emptyValueDelimiterPolicy. 5.55 Section
9.3.2 Clarifications on Establishing Representation. 5.56 Section 9.5,
9.4.2, <IP_ADDRESS>, <IP_ADDRESS> Clarifications on Element Defaults. 5.57 Section
9.2.2 Clarifications on Empty Representation. 5.58 Section 14.2
Clarifications on Separators and Suppression. 5.59 Section
14.2.1, 14.2.2, 14.2.3 Clarifications on Potentially Trailing Groups and
Elements. |
|
R07 |
2019-10-11 |
Fixed typos with
errata-related changes per SMH email of 2019-10-04. |
|
R06 |
2019-09-26 |
Errata Merged 5.19. Section 16.1.1, 16.1.2 Correct occursCountKind ‘fixed’ and
‘implicit’ to state that the unparser stops looking for
occurrences in the Infoset once maxOccurs has been reached. 5.20. Section 23.1, 13.11, 15, 7.9 outputValueCalc maximum forward reference is
implementation-defined. No forward reference from calendarLanguage,
choiceDispatchKey, setVariable. 5.21. Section 15. choiceBranchKey is case
sensitive. 5.22. Section 13.2.1. Parsing
block escape scheme processing error if the end of data for the element and
escapeBlockEnd is not found. 5.23 Section 13.16
nilKind, nilValue, and nilValueDelimiterPolicy properties apply to nillable
complex type elements as well as simple types. 5.24 Sections 13.11.1, <IP_ADDRESS>.6.
Fractional-seconds “S” corrections. Packed decimal calendar max
digits is implementation defined. 5.25 Section
17. Allow inputValueCalc on root of a choice branch. 5.26 Section 2. RFC2119 termiinology, except for required and optional. 5.27 Section
11. Bit order change requires byte alignment both parsing and
unparsing. 5.28 Section 13.6. textStandardNaNRep and textStandardInfinityRep: Schema Definition Error if empty string. 5.29 Sections 23.4, 23.5.3. Add XPath 2.0 'intersect' and "except" operators. Add dfdl:checkRangeInclusivem dfdl:checkRangeExclusive functions. 5.30 Section <IP_ADDRESS>. Clarify scanning text can require byte-level scans, character level
is insufficient. 5.31 Section 9.5. Clarify Evaluation Order for Statement Annotations section on property expression evaluation times, and grammar regions. 5.32 Section <IP_ADDRESS>, 9.2.5 ES and WSP* corrections for nil rep 5.33 Section 13.9 textBooleanTrueRep cannot be empty string. 5.34 Section 13.2 Fix textPadKind omission of dfdl:textBooleanJustification case. 5.35 Section
5.2 Remove redundant description from pattern facet in list. 5.36 Section
15, <IP_ADDRESS>, <IP_ADDRESS>, 9.3.3,15.1.2 choiceBranchKey is List of DFDL String
Literals. 5.37 Section 15 Specify SDE for choiceDispatchKey forward reference. 5.38 Section 4.1.2 Add element information item [array]. That is all the errata
up to 2019-09-25. |
|
R05 |
2019-09-17 |
Errata Merged 5.4 Section
<IP_ADDRESS> fn:exactly-one definition should match XPath 5.5 Section 23
binding fn: prefix to XPath function namespace. 5.6 Section 5.1
elementFormDefault and form attributes 5.7 Section 13.6 textNumberPadCharacter
clarifications 5.8 Sections
14.2, 14.2.2, 14.2.3, 16 Clarify the behaviour of separator suppression. 5.9 Section 16.1 Clarify
the behaviour when maxOccurs or occursCount is zero. 5.10 Sections 3, 9.2.2, <IP_ADDRESS>, <IP_ADDRESS>,
12.2, 13.16 Definition for ‘fixed-length element’ and
clarifications related to dfdl:emptyValueDelimiterPolicy (that it is not
applicable). 5.11 Section 23.4
Clarify unqualfied path steps. 5.12 Section
<IP_ADDRESS>.1 Clarify decimal minimum bits. 5.13 Section
<IP_ADDRESS> Clarifications on converting base-2 binary numbers. 5.14 Section 13.7
Clarifications for dfdl:binaryDecimalVirtualPoint 5.15 Section 12.3
Correction to dfdl:lengthUnits ‘bits’ allowed for packed
calendars. 5.16. Section
12.3.3, 12.3.4 Correction that binarySeconds and binaryMilliseconds are not
allowable representations for types xs:date and xs:time. 5.17. Section 13.11 Correct regular expression for
dfdl:calendarTimeZone. 5.18. Section 3.11 Correction for dfdl:lengthKind
‘expression’ – when unparsing the expression is evaluated
to obtain the length. |
|
R04 |
2019-09-16 |
Upgraded from MS-Word “Compatibility Mode” to “normal” so tools we use don’t have to deal with compatibility mode. |
|
R03 |
2019-09-13 |
Errata Merged 5.1 merged. Section <IP_ADDRESS> Escape Scheme Examples 5.2 merged. Section 23. Clarification of expression evaluation order. 5.3 merged. Section 13.2.1. New property escapeCharacterPolicy added to dfdl:escapeScheme. Dfdl:escapeKind language adjusted to take new property into consideration. |
|
R02 |
2019-09-13 |
Changes for Minor typographical fixes accepted. Two Comment bubbles remain where formatting changes could not be applied. |
|
R01 |
2019-09-09 |
Minor typographical fixes. (Remove this entry once changes accepted.) https://redmine.ogf.org/issues/233 through update #28. |
Contents
Data
Format Description Language (DFDL) v1.0 Specification
2. Notational and Definitional
Conventions
2.2 Schema Definition Error (SDE)
2.3.1 Ambiguity of Data Formats
2.6 Specific Errors Classified
2.7 Optional Checks and Warnings
4. The DFDL Information Set
(Infoset)
4.1.1 Document Information Item
4.1.2 Element
Information Items
4.3 DFDL Information Item Order
5. DFDL Schema Component Model
5.2 XSD Facets, min/maxOccurs,
default, and fixed
5.2.3 MaxInclusive, MaxExclusive,
MinExclusive, MinInclusive, TotalDigits, FractionDigits
5.3 Compatibility with Other
Annotation Language Schemas
6.2 The DFDL Annotation Elements
6.3.3 DFDL Regular Expressions
7. Syntax of DFDL Annotation Elements
7.1 Component Format Annotations
7.1.3 Empty String as a
Representation Property Value
7.2 dfdl:defineFormat - Reusable
Data Format Definitions
7.2.1 Inheritance for dfdl:defineFormat
7.2.2 Using/Referencing a Named
Format Definition
7.3 The dfdl:assert Statement
Annotation Element
7.3.1 Properties for dfdl:assert
7.3.2 Controlling
the Timing of Statement Evaluation
7.4 The dfdl:discriminator
Statement Annotation Element
7.4.1 Properties for
dfdl:discriminator
7.5 The dfdl:defineEscapeScheme
Defining Annotation Element
7.5.1 Using/Referencing a Named
escapeScheme Definition
7.6 The dfdl:escapeScheme
Annotation Element
7.7 The dfdl:defineVariable
Annotation Element
7.8 The dfdl:newVariableInstance
Statement Annotation Element
7.9 The dfdl:setVariable
Statement Annotation Element
8.1 Providing Defaults for DFDL
properties
8.2 Combining DFDL Representation
Properties from a dfdl:defineFormat
8.3 Combining DFDL Properties
from References
9. DFDL Processing Introduction
9.2.5 Zero-length Representation
9.2.7 Examples of Missing and Empty
Representation
9.3.1 Known-to-exist and
Known-not-to-exist
9.3.2 Establishing
Representation
9.4.1 Definition 'default value'
9.4.2 Element Defaults When Parsing
9.4.3 Element Defaults When
Unparsing
9.5 Evaluation Order for
Statement Annotations
9.5.1 Asserts
and Discriminators with testKind 'expression'
9.5.2 Discriminators
with testKind 'expression'
9.5.3 Elements
and setVariable
10. Core Representation Properties
and their Format Semantics
11. Properties Common to both
Content and Framing
11.1 Unicode Byte Order Mark (BOM)
11.2 Character Encoding and
Decoding Errors
11.2.1 Property
dfdl:encodingErrorPolicy
11.2.2 Unicode UTF-16
Decoding/Encoding Non-Errors
11.2.3 Preserving Data Containing
Decoding Errors
11.4.1 Example Using Right-to-Left
Display for 'leastSignificantBitFirst'
11.4.2 dfdl:bitOrder and Grammar
Regions
12.1.2 Mandatory Alignment for
Textual Data
12.1.3 Mandatory Alignment for
Packed Decimal Data
12.2 Properties for Specifying
Delimiters
12.3 Properties for Specifying
Lengths
12.3.1 dfdl:lengthKind 'explicit'
12.3.2 dfdl:lengthKind 'delimited'
12.3.3 dfdl:lengthKind 'implicit'
12.3.4 dfdl:lengthKind 'prefixed'
12.3.5 dfdl:lengthKind 'pattern'
12.3.6 dfdl:lengthKind 'endOfParent'
12.3.7 Elements of Specified Length
13.1 Properties Common to All
Simple Types
13.2 Properties Common to All
Simple Types with Text representation
13.2.1 The dfdl:escapeScheme
Properties
13.3 Properties for Bidirectional
support for All Simple Types with Text representation
13.4 Properties Specific to String
13.5 Properties Specific to Number
with Text or Binary Representation
13.6 Properties Specific to Number
with Text Representation
13.6.1 The dfdl:textNumberPattern
Property
13.6.2 Converting logical numbers
to/from text representation
13.7 Properties Specific to Number
with Binary Representation
13.7.1 Converting Logical Numbers
to/from Binary Representation
13.8 Properties Specific to
Float/Double with Binary Representation
13.9 Properties Specific to
Boolean with Text Representation
13.10 Properties Specific to
Boolean with Binary Representation
13.11 Properties specific to
Calendar with Text or Binary Representation
13.11.1 The dfdl:calendarPattern
property
13.11.2 The
dfdl:calendarCheckPolicy Property
13.12 Properties Specific to
Calendar with Text Representation
13.13 Properties Specific to
Calendar with Binary Representation
13.14 Properties Specific to Opaque
Types (xs:hexBinary)
13.16 Properties
for Nillable Elements
14.2 Sequence Groups with
Separators
14.2.1 Separators and Suppression
14.2.2 Parsing Sequence Groups with
Separators
14.2.3 Unparsing Sequence Groups
with Separators
14.3 Unordered Sequence Groups
14.3.1 Restrictions for Unordered
Sequences
14.3.2 Parsing an Unordered Sequence
14.3.3 Unparsing an Unordered
Sequence
15.1.1 Resolving Choices via
Speculation
15.1.2 Resolving Choices via Direct
Dispatch
16. Properties for Array Elements
and Optional Elements
16.1 The dfdl:occursCountKind
property
16.1.1 dfdl:occursCountKind 'fixed'
16.1.2 dfdl:occursCountKind
'implicit'
16.1.3 dfdl:occursCountKind 'parsed'
16.1.4 dfdl:occursCountKind
'expression'
16.1.5 dfdl:occursCountKind
'stopValue'
16.2 Default Values for Arrays
16.3 Arrays with DFDL Expressions
16.6 Forward Progress Requirement
16.7 Parsing Occurrences with
Non-Normal Representation
17. Calculated Value Properties
18. External Control of the DFDL
Processor
22.1.1 dfdl:element (simple) and
dfdl:simpleType
22.1.3 dfdl:sequence and dfdl:group
(when reference is to a sequence)
22.1.4 dfdl:choice and dfdl:group
(when reference is to a choice)
22.2.1 dfdl:element (simple) and
dfdl:simpleType
22.2.3 dfdl:sequence and dfdl:group
(when reference is a sequence)
22.2.4 dfdl:choice and dfdl:group
(when reference is a choice)
23.1 Expression Language Data
Model
23.2.1 Rewinding of Variable Memory
State
23.2.2 Variable Memory State
Transitions
23.5 Constructors, Functions and
Operators
23.5.1 Constructor Functions for XML
Schema Built-in Types
23.5.2 Standard XPath Functions
23.5.4 DFDL
Constructor Functions
23.6 Unparsing and Circular
Expression Deadlock Errors
27. Intellectual Property Statement
31. Appendix A: Escape Scheme Use
Cases
31.1 Escape Character Same as
dfdl:escapeEscapeCharacter
31.2 Escape Character Different
from dfdl:escapeEscapeCharacter
31.3 Escape Block with Different
Start and End Characters
31.4 Escape Block with Same Start
and End Characters
32. Appendix B: Rationale for Single-Assignment Variables
33. Appendix C: Processing of
DFDL String literals
33.1 Interpreting a DFDL String
Literal
33.2 Recognizing a DFDL String
Literal
33.3 Recognizing DFDL String
Literal Part
34. Appendix D: DFDL Standard
Encodings
34.4 Encoding
X-DFDL-US-ASCII-7-BIT-PACKED
34.5 Encoding
X-DFDL-US-ASCII-6-BIT-PACKED
34.6 References for Appendix D
We envisage three dominant kinds of data in the future, as follows:
1. Textual data defined by a format specific schema such as XML[XML] or JSON[JSON].
2. Binary data in standard formats.
3. Data with DFDL descriptors.
Textual XML data is the most successful data interchange standard to date. All such data are by definition new, by which we mean created in the XML era. Because of the large overhead that XML tagging imposes, there is often a need to compress and decompress XML data. However, there is a high-cost for compression and decompression that is unacceptable to some applications. Standardized binary data formats are also relatively new, and are suitable for larger data because of the reduced costs of encoding and more compact size. Examples of standard binary formats are data described by modern versions of ASN.1[ASN1], or the use of XDR [XDR]. These techniques lack the self-describing nature of XML-data. Scientific formats, such as NetCDF[NetCDF] and HDF[HDF] are used by some communities to provide self-describing binary data. There are also standardized binary-encoded XML data formats such as EXI [EXI].
It is an important observation that both XML format and standardized binary formats are prescriptive in that they specify or prescribe a representation of the data. To use them your applications must be written to conform to their encodings and mechanisms of expression.
DFDL suggests an entirely different scheme. The approach is descriptive in that one chooses an appropriate data representation for an application based on its needs and one then describes the format using DFDL so that multiple programs can directly interchange the described data. DFDL descriptions can be provided by the creator of the format, or developed as needed by third parties intending to use the format. That is, DFDL is not a format for data; it is a way of describing any data format. DFDL is intended for data commonly found in scientific and numeric computations, as well as record-oriented representations found in commercial data processing.
DFDL can be used to describe legacy data files, to simplify transfer of data across domains without requiring global standard formats, or to allow third-party tools to easily access multiple formats. DFDL can also be a powerful tool for supporting backward compatibility as formats evolve.
DFDL is designed to provide flexibility and also permit implementations that achieve very high levels of performance. DFDL descriptions are separable and native applications do not need to use DFDL libraries to parse their data formats. DFDL parsers can also be highly efficient. The DFDL language is designed to permit implementations that use lazy evaluation of formats and to support seekable, random access to data. The following goals can be achieved by DFDL implementations:
· Density. Fewest bytes to represent information (without resorting to compression). Fastest possible I/O.
· Optimized I/O. Applications can write data aligned to byte, word, or even page boundaries and to use memory-mapped I/O to insure access to data with the smallest number of machine cycles for common use cases without sacrificing general access.
DFDL can describe the same types of abstract data that other binary or textual data formats can describe and, furthermore, it can describe almost any possible representation scheme for those data. It is the spirit of DFDL to support canonical data descriptions that correspond closely to the original in-memory representation of the data, and also to provide sufficient information to write as well as to read the given format.
The question arises of why DFDL is needed in an era when there are so many standard data formats available. Ultimately, it is because there are a number of social phenomena in the way software is developed that have lead to the situation today where DFDL is needed to standardize descriptions of diverse data formats.
First, programs are very often written speculatively, that is, without any advance understanding of how important they will become. Given this situation, little effort is expended on data formats since it remains easier to program the I/O in the most straightforward way possible with the programming tools in use. Even something as simple as using an XML-based data format is harder than just using the native I/O libraries of a programming language.
In time, however, it is realized that a software program is important because either many people are using it, or it has become important for business or organizational needs to start using it in larger scale deployments. At that point it is often too late to go back and change the data formats. For example, there may be real or perceived business costs to delaying the deployment of a program for a rewrite just to change the data formats, particularly if such rewriting will reduce the performance of the program and increase the costs of deployment. (It takes longer to program, but at least it's slower when you are doneJ)
Additionally, the need for data format standardization for interchange with other software may not be clear at the point where a program first becomes 'important'. Eventually, however, the need for data interchange with the program becomes apparent.
The above phenomena are not something that is going away any time soon. There are, of course, efforts to smoothly integrate standardized data format handling into programming languages. Nevertheless, we see a critical role for DFDL since it allows after-the-fact description of a data format.
DFDL is a language for describing data formats. A DFDL description allows data to be read from its native format and to be presented as an instance of an information set or indeed converted to the corresponding XML document. DFDL also allows data to be taken from an instance of an information set and written out to its native format.
DFDL achieves this by leveraging W3C XML Schema Definition Language (XSDL) 1.0. [XSDL]
An XML schema is written for the logical model of the data. The schema is augmented with special DFDL annotations. These annotations are used to describe the native representation of the data. This is an established approach that is already being used today in commercial systems.
Consider the following XML data:
<w>5</w>
<x>7839372</x>
<y>8.6E-200</y>
<z>-7.1E8</z>
The logical model for this data can be described by the following fragment of an XML schema document that simply provides description of the name and type of each element:
<xs:complexType name="example1">
<xs:sequence>
<xs:element name="w" type="xs:int"/>
<xs:element name="x" type="xs:int"/>
<xs:element name="y" type="xs:double"/>
<xs:element name="z" type="xs:float"/>
</xs:sequence>
</xs:complexType>
Now, suppose we have the same data but represented in a non-XML format. A binary representation of the data could be visualized like this (shown as hexadecimal):
0000 0005 0077 9e8c
169a 54dd 0a1b 4a3f
ce29 46f6
To describe this in DFDL, we take our original XML schema document that described the data model and we annotate the type definition as follows:
<xs:complexType>
<xs:sequence>
<xs:element name="w" type="xs:int">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element representation="binary"
binaryNumberRep="binary"
byteOrder="bigEndian"
lengthKind="implicit"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="x" type="xs:int ">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element representation="binary"
binaryNumberRep="binary"
byteOrder="bigEndian"
lengthKind="implicit"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="y" type="xs:double">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element representation="binary"
binaryFloatRep="ieee"
byteOrder="bigEndian"
lengthKind="implicit"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="z" type="xs:float" >
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element representation="binary"
byteOrder="bigEndian"
lengthKind="implicit"
binaryFloatRep="ieee" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
This simple DFDL annotation expresses that the data are represented in a binary format and that the byte order will be big endian. This is all that a DFDL parser needs to read the data.
Consider if the same data are represented in a text format:
5,7839372,8.6E-200,-7.1E8
Once again, we can annotate the same data model, this time with properties that provide the character encoding, the field separator (comma) and the decimal separator (period):
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:sequence encoding="UTF-8" separator="," />
</xs:appinfo>
</xs:annotation>
<xs:element name="w" type="xs:int">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element representation="text"
encoding="UTF-8"
textNumberRep ="standard"
textNumberPattern="####0"
textStandardDecimalSeparator="."
lengthKind="delimited"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="x" type="xs:int">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element representation="text"
encoding="UTF-8"
textNumberRep ="standard"
textNumberPattern="#######0"
textStandardDecimalSeparator="."
lengthKind="delimited"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="y" type="xs:double">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element representation="text"
encoding="UTF-8"
textNumberRep ="standard"
textNumberPattern="0.0E+000"
textStandardDecimalSeparator="."
lengthKind="delimited"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="z" type="xs:float">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element representation="text"
encoding="UTF-8"
textNumberRep
="standard"
textNumberPattern="0.0E0"
textStandardDecimalSeparator="."
lengthKind="delimited"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
Many properties are repeatedly expressed in the example for the sake of simplicity. Later sections of this specification will define the mechanisms DFDL provides to avoid this repetitiveness.
DFDL maps data from a non-XML representation to an instance of an information set. This can be thought of as a data transformation. However, DFDL is not intended to be a general transformation language and, in particular, DFDL does not intend to provide a mechanism to map data to arbitrary XML models. There are two specific limitations on the data models that DFDL can work to:
This latter point deserves some elaboration. The XML schema used must be suitable for describing the physical data format. There must be a correspondence between the XML schema's constructs and the physical data structures. For example, generally the elements in the XML schema must match the order of the physical data. DFDL does allow for certain physically unordered formats as well.
The key concept here is that when using DFDL, you do not get to design an XML schema to your preference and then populate it from data. That would involve describing the data format, and describing a transformation for mapping it to the XML schema you have designed. DFDL is only about the format part of this problem. There are other languages, such as XSLT, which are for transformation. In DFDL, you describe only the format of the data, and this format constrains the nature of the XML schema you must use in its description.
The goals of version 1.0 are as follows:
The general features of version 1.0 are as follows:
a) Text and binary data parsing and unparsing
b) Validate the data when parsing and unparsing using XSDL validation.
c) Defaulted input and output for missing representations
d) Reference – use of the value of a previously read element in subsequent expressions
e) Choice – capability to select among format variations
f) Hidden sequence of elements – A description of an intermediate representation whose corresponding Infoset is not exposed in the final result.
g) Basic Math – in DFDL expressions
h) Out-of-type value handling (e.g., The string value 'NIL' to indicate nil for an integer)
i) Speculative parsing to resolve uncertainty.
j) Very general parsing capability: Lookahead/Push-back
Version 1.0 of DFDL is a language capable of expressing a wide range of binary and text-based data formats.
DFDL is capable of describing binary data as found in the data structures of COBOL, C, PL1, Fortran, etc. In particular, it is able to describe repeating sub-arrays where the length of an array is stored in another location of the structure.
DFDL is capable of describing a wide variety of textual data formats such as HL7, X12, and SWIFT. Textual data formats often use syntax delimiters, such as initiators, separators and terminators to delimit fields.
DFDL has certain composition properties. I.e., two formats can be nested or concatenated and a working format results.
The following topics have been deferred to future versions of the standard:
- Extensibility: There are real examples of proprietary data format description languages that we use as our base of experience from which to derive standard DFDL. However, there are no examples of extensible format description languages. Therefore, while extensibility is desirable in DFDL, there is not yet a base of experience with extensibility from which to derive a standard.
- Rich Layering: Some formats require data to be described in multiple passes. Combining these into one DFDL schema requires very rich layering functionality. In these layers one element's value becomes the representation of another element. DFDL V1.0 allows description of only a limited kind of layering.
1. Prescriptive systems:
a. Javascript Object Notation (JSON) [JSON]
b. EXI (binary XML) [EXI]
c.
Thrift
[Thrift]
d. Avro [AVRO]
e. ASN.1 with any of the prescribed encoding rules: Basic Encoding Rules (BER), Distinguished Encoding Rules (DER), Canonical Encoding Rules(CER)[ASN1CER] or Packed Encoding Rules (PER) [ASN1PER]
2. Descriptive systems:
a.
ASN1 Encoding Control Notation (also known as ITU-T
X.692) [ASN1ECN]
b. BFD: Binary Format Description (BFD) Language [BFD]
The key words must, must
not, required, shall, shall
not, should, should not, recommended,
may,
and may not and optional
in this document are to be interpreted as described in [RFC2119]. The terms required
and optional in this document are used as described in Section 3
(Glossary). Note that for reasons of clarity these words
are not always capitalized in this document.
Examples are for illustration purposes only and for clarity they will often not include all the necessary DFDL properties.
Where the phrase "must be consistent with" is used, it is assumed that a conforming DFDL implementation must check for the consistency and issue appropriate diagnostic messages when an inconsistency is found.
There are several kinds of failures that can occur when a DFDL processor is handling data and/or a DFDL schema.
When the DFDL schema itself contains an error, it implies that the DFDL processor cannot process data because the DFDL schema is not meaningful. It may be ambiguous, or contain conflicting definitions. Equivalently, we can say that there is no possible data that conforms to the schema; hence, the schema cannot be meaningful. All conforming DFDL processors must detect all Schema Definition Errors, and must issue some kind of appropriate diagnostic message. The behavior of a DFDL processor after a Schema Definition Error is detected is out of scope for this specification.
When a Schema definition error can be detected statically, that is given only the schema, it is desirable, though not required by the DFDL standard, that such errors be detected and diagnostic messages issued before any data are processed. Of course not all Schema Definition Errors can be detected without reference to data as some representation properties may obtain their values from the data (see also section 2.3.1 Ambiguity of Data Formats).
The expression language included within DFDL is strongly, statically type checkable. This means that type checking of expressions can be performed without processing data, and implementations are encouraged to perform this checking statically so that Schema Definition Errors having to do with type inconsistencies can be detected before processing data.
Note that Schema Definition Errors cannot be suppressed by points of uncertainty.
A DFDL processor MUST implement the Schema Component Constraint: Unique Particle Attribution defined in XML Schema Part 1: Structures [XSDLV1] that applies to the DFDL schema subset.
Two elements overlap if
A schema will violate the unique attribution constraint if it contains two particles which overlap and which either
Or
If a DFDL schema contains no Schema Definition Errors, then there is the additional possibility of a processing error when processing data using a DFDL schema. A processing error occurs if the data does not conform to the format described by the schema, that is to say, the data is not well-formed relative to the schema.
Processing errors can be suppressed by a point of uncertainty. See section 9.3.3.
It is expected that DFDL implementations will provide additional implementation-defined mechanisms for dealing with effective processing errors, such as the means of specifying retry points or the means of skipping some data so as to recover from the error in some way.
Exceptions that occur in the evaluation of the DFDL expression language are processing errors.
Non-conformance with the XSDL minOccurs or maxOccurs constraints is either a processing error or only a validation error depending on the settings of certain DFDL properties (see section 16 below).
A data format using delimiters may be ambiguous if
the delimiters are not distinct, and a data format description which has fixed
data requirements (that is, where some elements have fixed values) may be
ambiguous even with fixed-length elements.[1]
If the delimiter string values are stored within
the data, perhaps as elements of a header part of the data, then this ambiguity
certainly cannot be examined until the data is available.
Given an ambiguous grammar, a DFDL implementation
may successfully parse a particular input data stream. That is, the part of the
schema with the ambiguity may not be exercised by a particular data stream, or
the data may parse successfully anyway because the ambiguity may not cause any
kind of failure or processing error.
Hence, to insure compatible behavior, DFDL v1.0 implementations MUST NOT detect grammar ambiguities as errors. Implementations are of course free to issue warnings to help users identify these situations, but ambiguity is neither a Schema Definition Error nor a Processing Error.
Usually, the behavior of the unparser is symmetric to the behavior of the parser; however, there are cases where the DFDL schema will accept several equivalent representations for the same logical data. In this case it would be ambiguous which of these equivalent representations should be produced by the unparser. The DFDL standard contains representation properties which are used to eliminate this ambiguity. It is a Schema Definition Error if a DFDL schema is being used to unparse data and there is any ambiguity about the representation.
Logical validation checks are constraints expressed in XSDL, and they apply to the logical values of the infoset. Hence, parsing must successfully construct the infoset from the representation in order for validation checks to be meaningful. This implies that validation errors cannot affect the ability of a DFDL processor to successfully parse or unparse data; that is, validation errors are independent of whether the data is well-formed with respect to the DFDL schema.
DFDL processors may provide both validating and non-validating behaviors on either or both of parse and unparse. (A DFDL implementation could support validate on parse, but not support it on unparse and still be considered conforming.)
Validation on unparsing takes place on the augmented infoset that is created by the unparser as a side-effect of creating the output data stream.
When resolving points of uncertainty (during parsing), validation errors are ignored.
The way a validation error is presented to the execution context of a DFDL processor is not specified by the DFDL language. The validity of an element is recorded in the DFDL Infoset, see Section 4 The DFDL Information Set (Infoset).
The following DFDL schema constructs are allowed in DFDL and are checked when validating:
Note that validation is distinct from the checking of DFDL assert or discriminator predicates. When a DFDL discriminator or assert is used to discriminate a choice or other point of uncertainty when parsing, then that assert or discriminator is essential to parsing and it is evaluated irrespective of whether validation is enabled or disabled.
There is also a function dfdl:checkConstraints available in the DFDL Expression language. This can be used to explicitly include checking of the XSD facet constraints as part of parsing a specific element. Such checking is part of parsing, and does not create validation errors. See Section 23.5.3 DFDL Functions for details.
This error type is used with the dfdl:assert annotation when parsing to permit the checking of physical format constraints without terminating a parse. For example, some formats will have redundancy by having known lengths, as well as delimiters. A recoverable error can be issued, using an assert to check a physical length constraint when property lengthKind is 'delimited'.
Recoverable errors are independent of validation, and when resolving points of uncertainty, recoverable errors are ignored.
This section
clarifies which errors are Schema Definition Errors and which are processing
errors.
The following are
processing errors:
·
Arithmetic
Errors
o
Division
by zero
o
Integer
Arithmetic Underflow
o
Integer
Arithmetic Overflow
o
Note:
Floating point math can produce NaN (Not a Number) values. This is not an
error, nor are properly typed operations on floating point NaN values.
·
Expression
Errors
o
Dynamic
Type Error – unable to
convert to target type
§
Example:
non-digits found in string argument to xs:int(…) constructor.
§
Note:
if a DFDL Implementation cannot distinguish Dynamic Type Errors from Static
Type Errors, then a Dynamic Type Error should must
cause a Schema Definition Error
o
Index
out of bounds error – index not <= number of occurrences, or is <
1.
§
Note:
same error for dfdl:testBit if bitPos is not 1..8, or for character positions
in a string-value
o
Indexing
of non-array non-optional element
§
Example:
x[1] when x is declared and has both minOccurs="1" and
maxOccurs="1" explicitly, or by not stating either or both of them.
o
Illegal
argument value (correct type, illegal value)
·
Parse
Errors
o
Delimiter
not found
o
Data
not convertible to type
o
Assertion
failed
o
Discriminator
failed
o
Required
occurrence not found
o
No
choice alternative successfully parsed.
o
Character
set decoding failure and dfdl:encodingErrorPolicy is 'error'
·
Unparse
Errors
o
Truncation
scenarios where truncation is being disallowed
o
Rounding
error – rounding needed but not allowed. (Unparsing)
o
No
choice alternative successfully unparsed.
o
Character
set encoding failure and dfdl:encodingErrorPolicy is 'error'
·
Implementation-defined
Limit Errors - Implementations can have fixed or adjustable limits that some
formats and some data may exceed at processing time. This specification does
not further specify what these errors are, but some possible examples are:
o
Data
longer than allowed for representation of a given data type
§
Example:
exceed maximum length of representation of xs:decimal when dfdl:representation
is "text".
o
Expression
references too far back into infoset (parsing)
o
Expression
references too far forward into infoset (unparsing)
o
Number
of array elements exceeds limit.
o
Regular
expression exceeds time limit
The following are Schema Definition Errors, regardless of whether they are detected in advance of processing or once processing begins:
·
Errors
in XML Schema Construction and Structure
o
See XML
Schema Specification Part 1, Section 5.1 [XSDLV1]
·
Use of
XSD constructs outside of DFDL subset
·
Implementation-defined
Limitations
o
Use of
DFDL schema constructs not supported by this implementation.
§
Example:
xs:choice is an optional part of the DFDL specification (see section 21). If
not supported, it must be rejected as a Schema Definition Error.
§
Example:
use of packed-decimal when it is not supported by the implementation.
§
Example:
use of dfdl:assert when it is not supported by the implementation (See Spec
section 21 on DFDL Subsets)
§
Note:
Unrecognized DFDL properties or property values can produce a warning and an
implementation can attempt to process data despite the warning.
o
Exceeding
implementation-dependent limits for schema size/complexity
§
Example:
schema too large – simply a limit on how large the schema can be, how
many files, how many top-level constructs, etc.
·
Schema
Not Valid
o
See XML
Schema Specification Part 1, Section 5.2 [XSDLV1]
·
UPA
violation (Unique Particle Attribution)
·
Reference
to DFDL global definition not found
o
Format
definition (dfdl:defineFormat)
o
Escape
schema definition (dfdl:defineEscapeScheme)
o
Variable
Definition (dfdl:defineVariable)
·
DFDL
Annotations not well-formed or not valid
·
DFDL
Annotations Incompatible
o
E.g.,
dfdl:assert and dfdl:discriminator at same combined annotation point, or more
than one format annotation at an annotation point.
·
DFDL
Properties and their values
o
Property
not applicable to DFDL annotation
o
Property
value not suitable for property
o
Property
conflict
§
Between
Element Reference and Element Declaration
§
Between
Element Declaration and Simple Type Definition
§
Between
Simple Type Definition and Base Simple Type Definition
§
Between
Group Reference and Sequence/Choice of Group Definition
o
Required
property not found
·
Expressions
o
Expression
syntax error
o
Named
child element doesn't exist – E.g., /a/b, and there is no child b in
existence.
§
Note:
no child possible in the schema is a different error, but also a Schema
Definition Error, as /a/b would not have a type in that case.
§
Note:
This is an SDE, as schema authors are advised to use fn:exists(…) to test
for existence of elements when it is possible that they not exist.
o
Variable
read but not defined
o
Variable
assigned after read
o
Variable
assigned more than once
o
Static
Type error – type is incorrect for usage
§
Note:
if an implementation is unable to distinguish Static Type Errors from Dynamic Type
Errors, then both should cause Schema Definition Errors.
o
Path
step definition not found – e.g., /a/n:b but no definition for n:b as
local or global element.
o
Not
enough arguments for function
o
Expression
value is not single node
§
Most
DFDL expression contexts require an expression to identify a single node, not
an array (aka sequence of nodes). There are a few exceptions such as the
fn:count(…) function, where the path expression must be to an array or
optional element.
o
Expression
value is not array element or optional element.
§
Some
DFDL expression contexts require an array or an optional element.
§
Example:
The fn:count(...) function argument must be to an array or optional element. It
is an SDE if the argument expression is otherwise.
·
Regular
Expressions
o
Syntax
error
A DFDL processor:
There are two exceptions to this, which must be checked:
· Global simple types that are referenced by prefixLengthType property
· Global elements that are the document root.
Some situations suggest likely errors, but a DFDL processor cannot be certain. In these situations, a DFDL processor may issue warnings to assist a DFDL schema author in identifying likely errors. An important case of this is when the DFDL processor encounters a schema component and annotation where there are explicitly properties that are not relevant to the component as defined. Depending on the specifics of the component and property the DFDL processor can or must take certain actions. If the:
Adjacent - Two parts of the input/output stream are adjacent if they are at consecutive addresses.
Addressable Unit - This is the unit of storage that makes up the input or output stream holding the representation of the data. The units are bits, bytes, or characters.
Annotation point - A location within a DFDL schema where DFDL annotation elements are allowed to appear.
Applicable properties - All the DFDL properties that apply to that type of schema construct. For example all the DFDL properties that apply to an xs:simpleType.
Array - The set of adjacent elements whose XSDL element declaration specifies the potential for it to have more than one occurrence (XSD property maxOccurs > '1' or 'unbounded'). Of course any given array can have any number of element occurrences, including zero elements or exactly 1 element as long as the occurrence constraints are met. If XSD property maxOccurs is 'unbounded' then there is no constraint to the maximum number of occurrences, though implementations may have implementation-defined maximum capabilities. An optional element (where XSD property maxOccurs is '1', minOccurs is '0') is not considered to be an array as described in this document. Note that a sequence is not to be confused with an array. A sequence is a complex tuple type for an element; the children of a sequence can be of different types. All elements of an array have the same type and have the same information item members except for the value member.
Array Element – An element declaration or reference with XSD property maxOccurs > '1' or 'unbounded'.
Augmented Infoset - When unparsing one begins with the DFDL schema and conceptually with the
logical infoset. As the values of items are filled in by defaulting, and by use
of the DFDL outputValueCalc property (including on hidden items), these new
item values augment the infoset. The resulting infoset is called the augmented
infoset.
Binary - There are two meanings for this word
depending on context.
Binary
Representation - Of type
xs:hexBinary, or of other type with property dfdl:representation 'binary'. Note
that type xs:string can never have binary representation.
Bit Order - .Within a binary integer, if the
most-significant bit is assigned bit position 1, then the bit order is most-significant-bit first. If the
least-significant bit is assigned bit position 1, then the bit order is least-significant-bit first. When the
bit order is most-significant-bit first, then the least-significant bit of byte
N is considered to be adjacent to the most-significant bit of byte N+1. When
the bit order is least-significant-bit first, then the most-significant bit of
byte N is considered to be adjacent to the least-significant bit of byte N+1.
Bit Position - The data stream is assumed to be a collection of consecutively numbered unsigned bytes. Each byte is a numeric value from 0 to 255. The bits of a byte are referred to by their numerical significance as the 2n bit, for n from 0 to 7. Hence, the byte value 255 = 27 + 26 + 25 + 24 + 23 + 22 + 21 + 20. The 27-bit is the most-significant bit, and the 20-bit is the least significant bit. The bits within each byte are assigned numbered bit positions 1 to 8 according to the bit order. Given a bit-order, every bit in the data stream has a unique bit position.
Bit String - The ordered set of bits from a first bit with bit position N, to bit position N+M is a bit string of length M bits.
Byte - The term "byte" refers to an 8-bit octet. Can also refer to an integer with value from 0 to 255 inclusive.
CCSID - see Coded Character Set Identifier.
Character - An ISO10646 [ISO10646] character having a unique character code as its identifier. This concept is independent of font, typeface, size, and style, so 'F', 'F', 'F', are all the same character 'F'.
Character Code - The canonical integer used to identify a character in the ISO10646 [ISO10646] standards. This number identifies the character, but can be independent of any specific character set encoding of the character. Example: The '{' character known in Unicode [Unicode] as LEFT CURLY BRACKET has character code U+007B. However, depending on the character set encoding, the value 0x7B may or may not appear in the representation of that character.
Character Set - An abstract set of characters that are assigned (or mapped to) a representation by a particular character set encoding. For most character set encodings their character set is a subset of the Unicode character set.
Character Set Encoding - Often abbreviated to just 'encoding'. A specific representation of a character set as bytes or bits of data. A character set encoding is usually identified by a standard character set encoding name or a recognized alias name, or by a coded character set identifier or CCSID [CCSID]. These identifiers are standardized. The names and aliases are standardized by the IANA [IANA] (where unfortunately, they are called character set names). CCSIDs are an industry standard. Examples of character set encoding names are UTF-8, USASCII, GB2312, ebcdic-cp-it, ISO-8859-5, UTF-16BE, Shift_JIS. There are also additional DFDL standard character set encodings, see DFDL Standard Encoding. The DFDL standard also allows for implementation-defined character set encodings to be supported..
Character Width - The number of code units or alternatively the number of bytes or bits used to represent a character in a specific character set encoding is called the character width. Encodings are either fixed width (all characters encoded using the same width), or variable-width (different characters are encoded using different widths). For example the UTF-32 character set encoding has 4-byte character width, whereas USASCII has a 1-byte character width. UTF-8 is variable width, and any specific character has width 1, 2, 3, or 4 bytes. See also Fixed-Width Character Encoding and Variable-Width Character Encoding
Code Point - The integer that identifies a character within a character set encoding. A code point is represented by one or more code units. When a character set is fixed width, then there is no distinction between a code unit and a code point. For unicode character set encodings, there is no distinction between a character code and a code point. Examples:
Code Unit - When a character set encoding uses differing variable width representations for characters, the units making up these variable width representations are called code units. For example the UTF-8 encoding uses between 1 and 4 code units to represent characters, and for UTF-8, the individual code units are single bytes. DFDL's interpretation of the UTF-16 encoding is either fixed or variable width. When format property dfdl:utf16Width 'variable' then UTF-16 is variable width and this encoding uses either one or two code units per character, but in this case each individual code unit is a 16-bit value. When a character set is fixed width, then there is no distinction between a code unit and a code point.
Coded Character Set Identifier (CCSID) - An alternate identifier of a character set encoding. Originally created by IBM, CCSIDs are a broadly used industry standard. See [CCSID].]
Component - A construct within a DFDL schema that may contain a DFDL annotation.
Content - The content is the bits of data that are interpreted to compute a logical value.
Content Model - Used in describing the syntactic structure of XSD and DFDL annotations of it. An element of a schema can have empty, simple, or element-only content. An element declaration for an element of complex type containing a xs:sequence element is said to have a sequence in its content model.
Contiguous - An element has a contiguous representation if all parts of its representation are adjacent in the input/output stream. Most simple types have contiguous representations naturally. Groups containing elements that are themselves contiguous are also considered to have contiguous representations irrespective of alignment fill or padding of any kind that exists within the group. Similarly, arrays of elements that are themselves contiguous are also contiguous. An example of a non-contiguous representation would be a nillable element, where a flag is used to determine whether or not the element is nil, and the location of that flag is not adjacent to the value representation.
Count - The number of occurrences of an element.
Data Stream - Data where the format is being described
by a DFDL schema. This use of 'stream' implies only that there is a numbering
scheme that specifies a unique bit position for every bit within the data. This use of
'stream' does not imply anything about whether the data is persistently stored
or not, nor does it imply anything about whether there are sequential or random
access capabilities for access to the data.
DBCS - See Double-Byte Character Set
Decimal - This term is used several different ways
distinguished by context:
Defining Annotations - The annotation elements dfdl:defineFormat, dfdl:defineVariable, and dfdl:defineEscapeScheme
Delimiter - A character or string used to separate, or mark the start and end of, items of data. In DFDL, dfdl:lengthKind 'delimited' scans the data for initiators, separators, and terminators.
Delimiter scanning - When parsing, the process of scanning for a specific item in the input data which either marks the end of an item or the beginning of a subsequent item. Delimiter scanning also takes into account escape schemes so as to allow the delimiters to appear within data if properly escaped.
DFDL – Data Format Description Language
DFDL Processor - A program that uses DFDL schemas in order to process data described by them.
DFDL Schema - An XML schema containing DFDL annotations to describe data format.
DFDL Standard Encoding - A character set for
which there is no IANA name or CCSID but the name and definition of which DFDL
implementations must agree on. See Section 34
Appendix D: DFDL
Standard EncodingsAppendix D: DFDL
Standard Character Set Encodings.
Double-Byte Character Set (DBCS) - A character set encoding where each character code consists of one code unit which uses exactly 2 bytes.
Dynamic extent - This is a characteristic of the data stream. When parsing data corresponding to a schema component, the collection of bits within the data stream that contain any aspect of the representation of that schema component make up the component's dynamic extent.
Dynamic scope - This is a characteristic of parts of the DFDL schema. When a definition or declaration contains or references another declaration or definition, then the contained definition or declaration is said to be in the dynamic scope of the enclosing one. The important characteristic of dynamic scoping is that it traverses references. When parsing, the dynamic scope of an element declaration includes all definitions and declarations used as part of parsing that element.
Element - A part of the data described by an element declaration in the schema and presented as an element information item in the infoset.
Encoding - See Character Set Encoding.
Explicit properties - The explicit properties are the
combination of any defined locally on the annotation and any defined by a
dfdl:defineFormat annotation referenced by a local dfdl:ref property.
Fixed-Length Element - An element of specified length where dfdl:lengthKind is 'explicit' but dfdl:length is not an expression, or dfdl:lengthKind is 'implicit' (of simple type only). Note that choice branches where dfdl:choiceLengthKind is 'explicit' are also referred to as ‘fixed-length’ but are not necessarily elements.
Fixed-Width Character Encoding - A character set encoding where
all characters are encoded using a single code unit for their representation.
Note that a code unit is not necessarily a single byte. It may be more than one
byte, or some number of bits less than a byte. Examples of different fixed
widths are:
Fixed Array Element - An array element where XSDL minOccurs is equal to XSDL maxOccurs.
Format annotations - The annotation elements dfdl:format,
dfdl:element, dfdl:simpleType, dfdl:group, dfdl:sequence, dfdl:choice, and
dfdl:escapeScheme.
Format property – A DFDL property carried on a DFDL format annotation.
Framing - The term used to describe the delimiters, length fields, and other parts of the data stream which are present, and may be necessary to determine the length or position of the content of an element.
Implementation-defined feature - A
feature where the implementation has discretion in how it is performed, and the
implementation must document how it is performed.
Implementation-dependent feature - A
feature where the implementation has discretion in how it is performed, but the
implementation is not required to document how the feature is performed.
Index - The position of an occurrence in a count, starting at 1.
Item - A DFDL information set consists of a number of information items; or just items for short.
Least-Significant Bit - Often abbreviated to LSB. In a binary integer the least significant bit is the bit having the least place value. Within an 8-bit unsigned byte, the bit with place value 20 (or 1) is the least significant bit.
Length - When discussing data items and their representations, the term 'length' is used to refer to the measure of the size of the representation of an item in units of bits, bytes, or characters. The length of an array is the number of bits, bytes, or characters making up its representation, and has nothing to do with the number of occurrences of the array. Any element occurrence has length. Only array elements and optional elements have numbers of occurrences other than 1.
Lexical scope - In a DFDL Schema document, the lexical scope of any element is the collection of schema declarations, definitions, and annotations contained within the element textually.
Local properties – Local properties are the properties defined on an annotation in either short, attribute or element form
Logical layer - A DFDL Schema with all the DFDL annotations ignored is an ordinary XSDL schema. The logical structure described by this XSDL is called the DFDL logical layer.
Most-Significant Bit - Often abbreviated to MSB. In a binary integer the most significant bit is the bit having the greatest place value. Within an 8-bit unsigned byte, the bit with place value 27 is the most significant bit.
Nibble - 4 bits. A single hexadecimal digit (0 to 9, A to F) is often referred to as a nibble as it can be represented in exactly 4 bits.
Node - The term Node is a shorter equivalent to Element Information Item of the DFDL Infoset described in Section 4.1.2 Element Information Items.
Non-representation property – A format property that is not a representation property, specifically dfdl:ref, dfdl:hiddenGroupRef, dfdl:choiceBranchKey, dfdl:choiceDispatchKey, dfdl:inputValueCalc, dfdl:outputValueCalc. See also representation property.
Occurrence - An instance of an element in the data, or an item in the DFDL Infoset.
Optional Element - An element declaration or reference where XSDL minOccurs is equal to zero.
Optional Occurrence - An occurrence with an index greater than XSDL minOccurs.
Packed decimal – A physical representation of a decimal and integer numbers where each digit is packed into one nibble (4 bits) of a byte. There are several variants, some also include a sign nibble and some include a padding nibble. The term covers all the following enums of the dfdl:binaryNumberRep and dfdl:binaryCalendarRep properties – 'packed' (IBM 390 packed), 'bcd' (standard binary coded decimals or BCDs) and 'ibm4690Packed' (IBM 4690 packed).
Potentially represented - An element declaration in the schema describes a potentially represented item if that element declaration does not have a dfdl:inputValueCalc property. Whether the element declaration describes an occurrence that is actually represented or not depends on whether the element declaration is for an optional element, and whether the element has a corresponding value in the augmented infoset.
Physical Layer - A DFDL Schema adds DFDL annotations onto
an XSDL language schema. The annotations describe the physical representation
or physical layer of the data.
Point of Uncertainty - A point of uncertainty occurs in the data stream when there is more than one schema component that might occur at that point.
Representation property - A format property that is used to describe a physical characteristic of a component. Such a property will apply to one or more grammar regions of the component. See also non-representation property.
Required Element - An element declaration or reference where
XSDL minOccurs is greater than zero.
Required Occurrence - An occurrence with an index less than or equal
to XSDL minOccurs.
Required Property – A DFDL property that must have a value. The required properties for each xs:schema component are listed in the Property Precedence tables in section 23.
Resolved set of annotations - When DFDL annotations appear on a group
reference and the sequence or choice of the referenced global group, or appear
among an element reference, an element declaration, and its type definition,
then they are combined together and the resulting set of annotations is
referred to as the resolved set of annotations for the schema component.
SBCS - See Single Byte Character Set
Scan – Examine the input data looking for delimiters such
as separators and terminators, or matches to regular expressions.
Single-Byte Character Set (SBCS) - A character set encoding where each character code consists of one code unit which is exactly a single byte (8 bits).
Schema - The set of all declarations and definitions in the schema, including all included and imported schemas taken together. This includes both the XSDL declarations and definitions, and the DFDL definitions provided in the top-level DFDL annotations.
Schema Component Designator (SCD) - A notation for referring to one of the components of a DFDL Schema. This is being standardized by W3C. See http://www.w3.org/TR/xmlschema-ref.
Schema Definition Order – The order that the schema components are defined in a schema document.
Specified length - An item has specified length when dfdl:lengthKind is "implicit" (simple type only), "explicit", or "prefixed".
Speculative Parsing – When the parser reaches a point of uncertainty it attempts to parse each option in turn until one is known-to-exist or known-not-to-exist.
Statement annotations - The annotation elements dfdl:assert,
dfdl:discriminator, dfdl:setVariable, and dfdl:newVariableInstance. Also called
DFDL Statements.
Statically - A DFDL Implementation can analyze a DFDL
schema and determine the presence of many kinds of errors. This is called
static analysis, compilation of the schema, or determining the presence of the
error statically.
Surrogate Pair - A Unicode character whose character code value is greater than 0xFFFF can be encoded into variable-width UTF-16BE or UTF-16LE (which are variable-width encodings when the DFDL property utf16Width is 'variable'). In this case the representation uses two adjacent code units each of which is called a surrogate, and the pair of which is called a surrogate pair.
Target length - When unparsing, the length (in dfdl:lengthUnits) of an item's representation is the target length. The length of the content corresponding to a logical data value in the infoset may be shorter or longer than the target length, in which case padding or truncation may be necessary to make the logical data content conform to the target length. Rules for when padding and truncation occur, and how they are applied are specific to simple data types, and are controlled by a number of DFDL format properties.
Text - Consisting of characters in some character set encoding. Normally we think of text data as being human readable, but many character set encodings contain special control characters that are not human readable but we call data containing these text anyway. The dfdl:encoding property is required in order to decode/encode the text.
Text Representation - Of type xs:string, or of other types (except xs:hexBinary) with property dfdl:representation 'text'. Note that type xs:hexBinary never has text representation. This term specifically refers to the representation of the SimpleContent region being textual.
Textual - See Text.
Twos-Complement - A very common scheme for representing binary integers within data. A positive integer consisting of N bits is represented as its base-2 absolute value. A negative integer is represented as the complement (all bits inverted) of its absolute value plus 1.
Unicode - A character set defined by the Unicode Consortium,
and standardized at the International Standards Organization (ISO) as ISO10646.
Unit - See Addressable Unit.
Unpadded length - This is the length of the content of an item of the infoset, prior to any filling or padding which might be introduced due to dfdl:lengthKind "prefixed" or dfdl:lengthKind "explicit". It is equal to or smaller than the target length.
Validity - A
DFDL Infoset is said to be valid with respect to a DFDL schema if each Infoset
item is valid with respect to its corresponding DFDL schema component. Validity
is about the Infoset and the values it holds. It is independent of the data
representation when parsing or unparsing. See Section 2.4 Validation Errors, for a list of the specific value checks
that are performed when validating a DFDL Infoset against a DFDL schema.
Variable-Width Character Encoding - A character set encoding where characters are encoded using one or
more code units for their representation depending on which specific character
is being encoded. Examples with their ranges of varying width:
·
1 to 4
bytes: UTF-8
·
1 or 2
16-bit code units: UTF-16 when property dfdl:utf16Width is 'variable'
·
1 or 2
bytes: Shift-JIS
Well-formed - Data is said to be well-formed with respect
to a DFDL schema if a DFDL processor can parse the data into a DFDL Infoset, or
a DFDL processor can unparse to that data from a DFDL Infoset. The validity of
values in the infoset is not necessary for data to be well-formed.
Width - See Character Width.
This section defines an abstract data set called the DFDL Information Set (Infoset). Its purpose is to define the abstract data structure that must be provided:
The DFDL Infoset contains enough information so that a DFDL schema can be defined that will unparse the infoset and reparse the resultant datastream to produce the same infoset.
There is no requirement for DFDL-described data to be valid in order to have a DFDL information set.
DFDL information sets may be created by methods (not described in this specification) other than parsing DFDL-described data.
A DFDL information set consists of a number of information items; or just items for short. The information set for any well-formed DFDL-described data will contain at least a document information item and one element information item. An information item is an abstract description of a part of some DFDL-described data: each information item has a set of associated named members. In this specification, the member names are shown in square brackets, [thus]. The types of information item are listed in Section 4.1 Information Items.
The DFDL Information Set does not require or favor a specific interface or class of interfaces. This specification presents the information set as a modified tree for the sake of clarity and simplicity, but there is no requirement that the DFDL Information Set be made available through a tree structure; other types of interfaces, including (but not limited to) event-based and query-based interfaces, are also capable of providing information conforming to the DFDL Information Set.
The terms "information set" and "information item" are similar in meaning to the generic terms "tree" and "node", as they are used in computing. However, the former terms are used in this specification to reduce possible confusion with other specific data models.
The DFDL Information Set is similar in purpose to the XML Information Set [XMLInfoset], however, it is not identical, nor a perfect subset, as there are important differences.
An information set contains two different types of information items, as explained in the following sections. Every information item has members. For ease of reference, each member is given a name, indicated [thus].
There is exactly one document information item in the information set, and all other information items are accessible through the [root] member of the document information item.
There is no specific DFDL schema component that corresponds to this item. It is a concrete artifact describing the information set.
The document information item has the following members:
[root] The element information item corresponding to the root element declaration of the DFDL Schema.
[dfdlVersion] String. The version of the DFDL specification to which this information set conforms. For DFDL V1.0 this is 'dfdl-1.0'
[schema] String. This member is reserved for future use.
[unicodeByteOrderMark]
Enum. When the encoding of the root element of the document is exactly UTF-8, UTF-16, or UTF-32 (or CCSID equivalent), the member value indicates whether the document starts with a
Byte-order-mark (BOM), and what the
value of the mark was. If there is a BOM
at the start of the data stream, then for UTF-8 encoding the value is 'UTF-8'; for UTF-16 encoding the value is 'UTF-16LE' or 'UTF-16BE'; for UTF-32 the value is 'UTF-32LE' or 'UTF-32BE'. If there is no BOM then the member value is empty. When the encoding of the root element of the document is any other encoding, the member value is empty.
When unparsing, if this member is not empty and the encoding is UTF-8, UTF-16,
or UTF32, then this member's value is used to determine the specific byte-order
mark written, and for UTF-16 and UTF-32, the byte order used when characters
are encoded to the output data stream.
There is an element information item for each value parsed from the non-hidden DFDL-described data. This corresponds to an occurrence of a non-hidden element declaration of simple type in the DFDL Schema and is known as a simple element information item.
There is an element information item for each explicitly declared structure in the DFDL-described data. This corresponds to an occurrence of an element declaration of complex type in the DFDL Schema and is known as a complex element information item.
In this information set, as in an XML document, an array is just a set of adjacent elements with the same name and namespace. (To represent the array explicitly, introduce a new complex type element to contain the array elements only.)
One of the element information items is the [root] member of the document information item, corresponding to the root element declaration of a DFDL Schema, and all other element information items are accessible by recursively following its [children] member.
An element information item has the following members:
[namespace] String. The namespace, if any, of the element. If the element does not belong to a namespace, the value is the empty string.
[name] String. The local part of the element name.
[document] The document information item representing the DFDL information set that contains this element. This element is empty except in the root element of an information set.
[dataType] String. The name of the XML Schema 1.0 built-in simple type to which the value corresponds. DFDL supports a subset of these types listed in section 5.1 DFDL Subset of XML Schema. In a complex element information item this member has no value.
[dataValue] The value in the value space (as defined by XML Schema Part 2: Datatypes [XSDLV1]) of the [datatype] member. In a complex element information item this member has no value. If the [nilled] member is true, then this member has no value.
For
information items
of datatype
xs:string, the
value is
an ordered
collection of
unsigned 16-bit
integer codepoints
each having
any value
from 0x0000
to 0xFFFF.
Where defined,
these are
interpreted as
the ISO646
character codes.
Codepoints disallowed
by ISO
10646, such
as 0xD800
to 0xDFFF
are explicitly
allowed by
the DFDL
infoset. The codepoints of the string are stored in 'implicit' (also known as logical), left-to-right bidirectional ordering and orientation. DFDL's
infoset represents
Unicode characters
with character
codes beyond
0xFFFF by
way of
surrogate pairs
(2 adjacent
codepoints) in
a manner
consistent with
the UTF-16
encoding of
ISO 10646. The value can have length
0, in which case the value may be referred to as an 'empty string'.
For information items of datatype xs:hexBinary, the value is an ordered collection of unsigned 8-bit bytes each having value from 0 to 255. The length of this collection can be 0 in which case the value may be referred to as an 'empty hexBinary'.
[nilled] Boolean. True if the nillable item is nil. False if the nillable item is not nil. If the element is not nillable this member has no value. If this member is true then for a simple element the [dataValue] member has no value, and for a complex element the [children] member has no value. If this member is true then the Infoset item is said to be nil or nilled.
[array]
Boolean. True if the item is an array, meaning that it corresponds to an
element having maxOccurs value greater than 1, or ‘unbounded’.
[children] An ordered set of zero or more element information items. The order they appear in the set is the order implied by the DFDL Schema. 'Ordered set' is not formally defined here, but two operations are assumed: 'count' gives the number of information items, and 'at (index)' gives the element at ordinal position 'index' starting from 1. In a simple element information item this member has no value. In a document information item this member contains exactly one element information item. If the [nilled] member is true then this member has no value.
[parent] The complex element information item which contains this information item in its [children] member. In the root element of an information set this member is empty.
[schema] String. A reference to a schema component associated with this information item, if any. If not empty, the value must be an absolute or relative Schema Component Designator [SCD].
[valid] Boolean[3]. True if the element is valid as determined by a DFDL implementation that performs validation checking. A complex element information item is not valid if any of its [children] are not valid. Empty if validation is not enabled.
[unionMemberSchema][4] String. For simple element information items, this member contains an SCD reference to the member of the union that matched the value of the element. Empty if validation is not enabled. Empty if the element's type is not a union.
On unparsing, any non-empty values for the [valid] or [unionMemberSchema] members are ignored. However, in the augmented infoset which is built during the unparse operation [valid] will have a value, and [unionMemberSchema] may have a value.
Some members may sometimes have the value no value, and it is said that such a member has no value. This value is distinct from all other values. In particular it is distinct from the empty string, the empty set, and the empty list, each of which simply has no members.
On parsing and unparsing information items will be presented in the order they are defined in the DFDL Schema.
By way of illustration, the DFDL information set is presented below as an object model using a Unified Modeling Language (UML) class diagram, augmented using the Object Constraint Language (OCL) [UML].
The structure of the information set follows the Composite design pattern. In case of inconsistency or ambiguity, the preceding discussion takes precedence.
DFDL is able to describe the format of the physical representation for data whose structure conforms to this model. Note that this model allows hierarchically nested data, but does not allow representation of arbitrary connected graphs of data objects.
Figure 1 DFDL Infoset Object Model
When unparsing, one begins with
the DFDL schema and conceptually with the logical infoset. As the values of
items are filled in by defaulting, and by use of the dfdl:outputValueCalc
property (including on hidden
items) (see section 17 Calculated Value Properties), these new item values augment
the infoset. The resulting infoset is called the augmented infoset.
An element declaration in the schema describes a potentially represented item if that element declaration
does not have a dfdl:inputValueCalc property (see section 17 Calculated Value Properties). Whether the element
declaration describes an item that is actually represented or not depends on whether
the element declaration is for an optional element and whether the element has
a corresponding value in the augmented infoset.
In expressions, the function dfdl:contentLength() and dfdl:valueLength()
can be called to determine the length of an item. If an element declaration is
not potentially represented, then these functions are defined to return 0.
When unparsing, an element declaration and the infoset are considered as
follows. An implementation may use any technique consistent with this algorithm:
a) If the element declaration has a
dfdl:outputValueCalc property then the expression which is the
dfdl:outputValueCalc property value is evaluated and the resulting value
becomes the value of the element item in the augmented infoset. Any pre-existing
value for the infoset item is superseded by this new value.
References to other augmented infoset items from within the
dfdl:outputValueCalc expression must obtain their values from the augmented
infoset directly (when the value is already present) or by recursively using
these methods (a) and (b) as needed.
b) If the element declaration has no
corresponding value in the augmented infoset, and the element declaration is
for a required occurrence, and it has
a default value specified, then an element item having the
default value is created in the augmented infoset.
c) If any Infoset item's value is
requested recursively as a part of (a) above and (a) does not apply, and the
corresponding value is not present, and (b) does not apply then it is a
processing error.
Given this augmented infoset, then if the potentially represented element
declaration has a corresponding infoset item then that item is converted to its
representation according to its DFDL properties. If the element declaration is
for a required occurrence, and there is no value in the augmented infoset then
it is a processing error.
Because rule (a) above is used even if the augmented infoset item already
exists and has a value, it is possible for a dfdl:outputValueCalc expression to
be evaluated multiple times. DFDL implementations are free to cache values and
avoid this repeated evaluation for efficiency, as the semantics of DFDL require
that the dfdl:outputValueCalc expression return the same value every time it is
evaluated.
When using DFDL, the format of data is described by means of a DFDL Schema.
The DFDL Schema Component Model is shown in conceptual UML in Figure 2. First we show the model for elements, groups and the top of the type hierarchy.
The shaded boxes have direct corresponding element syntax and therefore appear in DFDL schema
Figure 2 DFDL Schema UML diagram
The simple types are shown in Figure 3. The graph shows all the types defined by XML Schema version 1.0, and the subset of these types supported by DFDL are shown as shaded.
Figure 3 DFDL simple types
These types are defined as they are in XML Schema, with exceptions for:
· String – In DFDL a string can contain any character codes. None are reserved. (Including the character with character code U+0000, which is not permitted in XML documents.)
Each object defined by a class in the above UML is called a DFDL Schema component.
We express the DFDL Schema Model using a subset of the XML Schema Description Language (XSDL). XSDL provides a standardized schema language suitable for expressing the DFDL Schema Model.
A DFDL Schema is an XML schema containing only a restricted subset of the constructs available in full W3C XML Schema Description Language. Within this XML schema, special DFDL annotations are distributed that carry the information about the data's format or representation.
A DFDL Schema is a valid XML schema. However, the converse is not true since the DFDL Schema Model does not include many concepts that appear in XML schema.
The DFDL subset of XSDL is a general model for hierarchically-nested data. It avoids the XSDL features used to describe the peculiarities of XML as a syntactic textual representation of data, and features that are simply not needed by DFDL.
The following lists detail the similarities and differences between general XSDL and this subset.
DFDL Schemas consist of:
Note: xs:nonNegativeInteger is treated as an unsigned xs:integer.
The following constructs from XML Schema are not used as part of the DFDL Schema Model of DFDL v1.0 schemas; however, they are all reserved[6] for future use since the data model may be extended to use them in future versions of DFDL:
XSD element declarations and references can carry several properties that express constraints on the described data. These constraints are mainly used for validation. These properties include:
The facets and the types they are applicable to are:
The facets (but not maxOccurs nor minOccurs) are also checked by the dfdl:checkConstraints DFDL expression language function.
The following sections describe these in more detail.
The XSDL minOccurs property is used:
The XSDL maxOccurs property is used:
For some values of dfdl:occursCountKind such as 'implicit', it is a processing error when an array is found to have a number of occurrences not conforming to XSDL minOccurs in the absence of a default value specification. For other values of dfdl:occursCountKind such as 'parsed', it is only a validation error if an array is found to have fewer than XSDL minOccurs occurrences. See Section 16, Properties for Array Elements and Optional Elements, for more details.
These facets are used:
· Used for validation only
The format of numbers is not derived from these facets. Rather dfdl properties are used to specify the format.
· Allowed only on elements of type xs:string or types derived from it in Figure 3 DFDL simple types.
· Used for validation only
It is important to avoid confusion of the pattern facet with other uses of regular expressions that are needed in DFDL (for example, to determine the length of an element by regular expression matching).
Note: in XSD, pattern is about the lexical representation of the data, and since all is text there, everything has a lexical representation. In DFDL only strings are guaranteed to have a lexical and logical value that is identical.
Enumerations are used to provide a list of valid values in XSD.
Note: in DFDL we do not use XSD enumeration as a means to define symbolic constants. These are captured using dfdl:defineVariable constructs so they can be referenced from expressions.
The 'default' property is used:
Note that the 'fixed' and 'default' properties are mutually exclusive on an element declaration.
The 'fixed' property is used in the same ways as the 'default' property but in addition:
Note that the 'fixed' and 'default' properties are mutually exclusive on an element declaration.
A DFDL Schema only
applies annotations on a subset of the XML Schema constructs. However, a DFDL
schema may be annotated not only by DFDL annotations, but by other XML
annotation languages.
The XML schemas of
those other annotation languages MAY use any constructs of XML Schema, including
those prohibited by DFDL (such as attribute declarations.)
A DFDL implementation MUST ignore any schema file included or imported by a DFDL schema if the top level xs:schema element of that schema does not have an XML namespace binding for the DFDL namespace.
Using DFDL, a data format is described by placing special annotations at various positions within an XML schema. This XML schema conveys the basic structure of the data format, while the annotations fill in the detail. Annotations are used to describe aspects such as the file encoding and byte ordering, as well as declaring variables for reference elsewhere, and specifying properties that govern the capabilities of the DFDL processor. A DFDL processor requires these annotations, along with the structural information of the enclosing XML schema, to make sense of the physical data model.
The xs:appinfo source URI http://www.ogf.org/dfdl/ is used to distinguish DFDL annotations from other annotations.
The element and attribute names in the DFDL syntax are in a namespace defined by the URI http://www.ogf.org/dfdl/dfdl-1.0/. All symbols in this namespace are reserved. DFDL implementations may not provide extensions to the DFDL standard using names in this namespace. Within this specification, the namespace prefix for DFDL is "dfdl" referring to the namespace http://www.ogf.org/dfdl/dfdl-1.0/.
Attributes on DFDL annotations that are not in the DFDL namespace or or in no namespace are ignored.
A DFDL Schema document contains XML schema annotation elements that define and assign names to parts of the format specification. These names are defined using the target namespace of the schema document where they reside, and are referenced using QNames in the usual manner. A DFDL schema document can include or import another schema document, and namespaces work in the usual manner for XML schema documents. The schema is the schema including all additional schemas referenced through import and include. Generally, in this specification, when we refer to the DFDL Schema we mean the schema. When we refer to a specific document we will use the term DFDL Schema document.
DFDL annotations must be positioned specifically where DFDL annotations are allowed within an XML schema document. These positions are known as annotation points. When an annotation is positioned at an annotation point, it binds some additional information to the schema component that encloses it. The description of a data format is achieved by correctly placing annotations on the structural components of the schema.
DFDL specifies a collection of annotations for different purposes. They are organized into three different annotation types: Format Annotations, Statement Annotations, and Defining Annotations
At any single annotation point of the schema there can be only one format annotation, but there can be several statement annotations although there are rules about which of those are allowed to co-exist as well which will be described in sections about those specific annotation types.
|
Annotation Type |
Annotation Element |
Description |
|
Format Annotation |
choice |
Defines the physical data format properties of an xs:choice
group. See section 7.1. |
|
element |
Defines the physical data format properties of an xs:element
and xs:element reference. See section 7.1. |
|
|
format |
Defines the physical data format properties for multiple DFDL
schema constructs. Used on an xs:schema and as a child of a dfdl:defineFormat
annotation. This includes aspects such as the encodings, separators, and many
more. See section 7.1. |
|
|
group |
Defines the physical data format properties of an xs:group
reference. See section 7.1. |
|
|
property |
Used in the syntax of format annotations. See section <IP_ADDRESS>. |
|
|
sequence |
Defines the physical data format properties of an xs:sequence
group. See section 7.1. |
|
|
simpleType |
Defines the physical data format properties of an
xs:simpleType. See section 7.1. |
|
|
escapeScheme |
Defines the scheme by which quotation marks and escape
characters can be specified. This is for use with delimited text formats. See
section 7.6. |
|
|
Statement Annotation |
assert |
Defines a test to be used to ensure the data are well formed.
Assert is used only when parsing data. See section 7.3 |
|
discriminator |
Defines a test to be used when resolving a point of uncertainty
such as choice branches or optional elements. A dfdl:discriminator is used
only when parsing data to resolve the point of uncertainty to one of the
alternatives. See section 7.4 |
|
|
newVariableInstance |
Creates a new instance of a variable. See section 7.8 |
|
|
setVariable |
Sets the value of a variable whose declaration is in scope See
section 7.9 |
|
|
Defining Annotation |
defineEscapeScheme |
Defines a named, reusable escapeScheme See section 7.5 |
|
defineFormat |
Defines a reusable data format by collecting together other
annotations and associating them with a name that can be referenced from
elsewhere. See section 7.2 |
|
|
defineVariable |
Defines a variable that can be referenced elsewhere. This can
be used to communicate a parameter from one part of processing to another
part. See section 7.7 |
Table 1 - DFDL Annotation Elements
Properties on DFDL annotations may be one or more of the following types
Some properties accept a list or union of types
DFDL String Literals represent a sequence of literal bytes or characters which appear in the data stream. This presents the following challenges
A DFDL string literal is therefore able to describe any arbitrary sequence of bytes and characters.
Empty String:
The special DFDL entity %ES; is provided for describing an empty string or an
empty byte sequence. The %ES; entity is the only way to do this. A DFDL string
literal with value "" (the empty string) is usually invalid. There
are a few properties that explicitly allow an empty DFDL String Literal, and
these properties assign a property-specific meaning to the empty string value.
Whitespace: When whitespace must be used as part of a property value, the DFDL string literal must use entities (such as %WSP;) to represent the whitespace. (This allows a property to represent lists of DFDL string literals by using literal spaces to separate list elements.)
A literal string in a DFDL Schema is written in the character set encoding specified by the XML directive that begins all XML documents:
<?xml version="1.0" encoding="UTF-8" ?>
In this example, the DFDL schema is written in UTF-8, so any literal strings contained in it, and particularly string literals found in its representation property bindings in the format annotations, are expressed in UTF-8.
However, these strings are being used to describe features of text data that are commonly in other character set encodings. For example, we may have EBCDIC data that is comma separated. A comma in EBCDIC has a single-byte code unit of 0x6B in the data, the numeric value of which does not correspond to the Unicode character code for comma which is U+002C. However, when we indicate that an item is "," (comma) separated and we specify this using a string literal along with specifying the 'encoding' property to be 'ebcdic-cp-us' then this means that the data are separated by EBCDIC commas regardless of what character set encoding is used to write the DFDL Schema.
<?xml version="1.0" encoding="UTF-8">
<xs:schema ... >
...
<dfdl:format encoding="ebcdic-cp-us" separator=","/>
...
</xs:schema>
When a DFDL processor uses the separator expressed in this manner, the string literal "," is translated into the character set encoding of the data it is separating as specified by the encoding representation property. Hence, in this case we would be searching the data for a character with codepoint 0x6B (the EBCDIC comma), not a UTF-8 or Unicode (0x2C) comma which is what exists in the DFDL schema document file.
Character
strings can include bidirectional data.
DFDL character entities specify a single Unicode character and provide a convenient way to specify code points that appear in the data stream but would be difficult to specify in XML strings. For example, common non-printable characters or code points, such as 0x00, that are not valid in XML documents. DFDL entities are based on XML entities, which can also be used in a DFDL schema.
The following grammar gives the syntax of DFDL String Literals generally, including the various kinds of entities.
|
DfdlStringLiteral |
::= |
(DfdlStringLiteralPart)+ | DfdlESEntity |
|
DfdlStringLiteralPart |
::= |
LiteralString | DfdlCharEntity |
DfdlCharClass | ByteValue |
|
LiteralString |
::= |
A string of literal characters |
|
DfdlCharEntity |
::= |
DfdlEntity |
DecimalCodePoint | HexadecimalCodePoint |
|
DfdlCharClass
|
::= |
'%'
DfdlCharClassName ';' |
|
ByteValue
|
::= |
'%#r'
[0-9a-fA-F]{2} ';' |
|
DfdlEntity
|
::= |
'%'
DfdlEntityName ';' |
|
DecimalCodePoint |
::= |
'%#' [0-9]+ ';' |
|
HexadecimalCodePoint |
::= |
'%#x'
[0-9a-fA-F]+ ';' |
|
DfdlEntityName |
::= |
'NUL'|'SOH''|'STX'|'ETX'|
'EOT'|'ENQ'|'ACK'|'BEL'|
'BS'|'HT'|'LF'|'VT'|'FF'| 'CR'|'SO'|'SI'|'DLE'| 'DC1'|'DC2'|'DC3'|'DC4'|
'NAK'|'SYN'|'ETB'|'CAN'|
'EM'|'SUB'|'ESC'|'FS'|
'GS'|'RS'|'US'|'SP'|
'DEL'|'NBSP'|'NEL'|'LS' |
|
DfdlCharClassName |
::= |
DfdlNLEntity | DfdlWSPEntity | DfdlWSPStarEntity | DfdlWSPPlusEntity |
|
DfdlNLEntity |
::= |
'NL' |
|
DfdlWSPEntity |
::= |
'WSP' |
|
DfdlWSPStarEntity |
::= |
'WSP*' |
|
DfdlWSPPlusEntity |
::= |
'WSP+' |
|
DfdlESEntity |
::= |
'ES' |
Table 2 DFDL Character Entity,
Character Class Entity, and Byte Value Entity Syntax
Using %% inserts a single literal
"%" into the string literal. This "%" is subject to
character set encoding translation as is any other character.
A HexadecimalCodePoint provides
a hexadecimal representation of the character's code point in ISO/IEC 10646.
A DecimalCodePoint
provides a decimal representation of the character's code point in ISO/IEC
10646.
A DfdlEntityName is one
of the mnemonics given in the following tables.
|
Mnemonic |
Meaning |
Unicode Character Code |
|
NUL |
null |
U+0000 |
|
SOH |
start of heading |
U+0001 |
|
STX |
start of text |
U+0002 |
|
ETX |
end of text |
U+0003 |
|
EOT |
end of transmission |
U+0004 |
|
ENQ |
enquiry |
U+0005 |
|
ACK |
acknowledge |
U+0006 |
|
BEL |
bell |
U+0007 |
|
BS |
backspace |
U+0008 |
|
HT |
horizontal tab |
U+0009 |
|
LF |
line feed |
U+000A |
|
VT |
vertical tab |
U+000B |
|
FF |
form feed |
U+000C |
|
CR |
carriage return |
U+000D |
|
SO |
shift out |
U+000E |
|
SI |
shift in |
U+000F |
|
DLE |
data link escape |
U+0010 |
|
DC1 |
device control 1 |
U+0011 |
|
DC2 |
device control 2 |
U+0012 |
|
DC3 |
device control 3 |
U+0013 |
|
DC4 |
device control 4 |
U+0014 |
|
NAK |
negative acknowledge |
U+0015 |
|
SYN |
synchronous idle |
U+0016 |
|
ETB |
end of transmission block |
U+0017 |
|
CAN |
cancel |
U+0018 |
|
EM |
end of medium |
U+0019 |
|
SUB |
substitute |
U+001A |
|
ESC |
escape |
U+001B |
|
FS |
file separator |
U+001C |
|
GS |
group separator |
U+001D |
|
RS |
record separator |
U+001E |
|
US |
unit separator |
U+001F |
|
SP |
space |
U+0020 |
|
DEL |
delete |
U+007F |
|
NBSP |
no break space |
U+00A0 |
|
NEL |
Next line |
U+0085 |
|
LS |
Line separator |
U+2028 |
Table 3 DFDL Entities
The following DFDL character classes are provided to specify one or more characters from a set of related characters.
|
Mnemonic |
Meaning |
Unicode
Character Code(s) |
|
NL |
Newline On parse any one of the single characters CR, LF, NEL or LS or the character combination CRLF. On unparse the value of the dfdl:outputNewLine property is output, which must specify one of the single characters %CR;, %LF;, %NEL;, or %LS; or the character combination %CR;%LF;. |
U+000A LF U+000D CR U+000D U+000A CRLF U+0085 NEL U+2028 LS |
|
WSP |
Single whitespace On parse any whitespace character On unparse a space (U+0020) is output |
U+0009-U+000D (Control characters) U+0020 SPACE U+0085 NEL U+00A0 NBSP U+1680 OGHAM SPACE MARK U+180E MONGOLIAN VOWEL SEPARATOR U+2000-U+200A (different sorts of spaces) U+2028 LSP U+2029 PSP U+202F NARROW NBSP U+205F MEDIUM MATHEMATICAL SPACE U+3000 IDEOGRAPHIC SPACE |
|
WSP* |
Optional Whitespaces On parse whitespace characters are ignored. On unparse nothing is output |
Same as WSP |
|
WSP+ |
Whitespaces On parse one or more whitespace characters are ignored. It is an processing error if no whitespace character is found. On unparse a space (U+0020) is output. |
Same as WSP |
|
ES |
Empty String Used in whitespace separated lists when empty string is one of the values. |
|
Table 4 DFDL Character Class Entities
DFDL byte value entities provide a way to specify a single byte as it appears in the data stream without any character set encoding translation. To specify a string of byte values, a sequence of two or more byte value entities must be used. The syntax is in Table 2 DFDL Character Entity, Character Class Entity, and Byte Value Entity Syntax above. Example:
%#rFF;
Some DFDL properties allow DFDL expressions (see Section 23 Expression language) to be used so that the property can be set dynamically at processing-time.
The general syntax of expressions is "{" expression "}"
The rules for recognizing DFDL expressions are
DFDL expressions reference other items in the infoset or augmented infoset using absolute or relative paths. Relative paths are evaluated when the component containing the expression is referenced not when it is declared. For example a global element may have a DFDL property which is an expression that contains a relative path to another element. The relative path is evaluated when the global element is referenced from an element reference.
DFDL expressions that are used to provide the value of DFDL properties in the dfdl:format annotation on the top level xs:schema declaration MAY NOT contain relative paths.
The DFDL lengthPattern property expects a regular expression to be specified. The DFDL Regular Expression language is defined in the section 24 DFDL Regular Expressions.
Some DFDL properties accept an enumerated list of valid values. It is a Schema Definition Error if a value other than one of the enumerated values is specified. The case of the specified value must match the enumeration. An enumeration is of type string unless otherwise stated.
This section describes the syntax of each of the DFDL annotation elements along with discussion of their basic meanings.
The DFDL annotation elements are listed in Table 1 - DFDL Annotation Elements
A data format can be 'used' or put into effect for a part of the schema by use of the component format annotation elements.
There are specific annotations for each type of schema component that supports only the representation properties applicable to that component. The table below gives the specific annotation for each schema component.
|
Schema component |
DFDL annotation |
|
xs:choice |
dfdl:choice |
|
xs:element |
dfdl:element |
|
xs:element reference |
dfdl:element |
|
xs:group reference |
dfdl:group |
|
xs:schema |
dfdl:format |
|
xs:sequence |
dfdl:sequence |
|
xs:simpleType |
dfdl:simpleType |
Table 5 DFDL Component Format Annotations
In addition the dfdl:format annotation is used inside a dfdl:defineFormat annotation to define a named reusable set of representation properties that can be referenced from any component specific format annotation or from other named format definitions.
A dfdl:format annotation at the top level of a schema, that is as an annotation child element on the xs:schema, provides a set of default properties for the lexically enclosed schema document. See 8.1 Providing Defaults for DFDL properties.
Example of DFDL component format annotation:
<xs:schema
...>
...
<xs:element name="root">
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:element ref="aBaseConfig"
representation="text"
encoding="UTF-8"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
...
</xs:schema>
A named, reusable, dfdl:defineFormat definition is used by referring to its name from a format annotation using the 'ref' property. For example:
<dfdl:element ref="reusableDef"
encoding="ebcdic-cp-us" />
The behavior of this dfdl:defineFormat definition is as if all representation properties defined by the named dfdl:defineFormat definition were instead written directly on this format annotation; however, these are superseded by any representation properties that are defined here such as the encoding property in the example above.
The format properties may be specified in one of three forms:
A DFDL property may be specified using any form with the following exceptions
It is a Schema Definition Error if the same property is specified in more than one form in the resolved set of annotations for an annotation point.
Within the format annotation elements are bindings for properties of the form:
Property="Value"
For example:
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:format encoding="utf-8" separator="%NL;"/>
</xs:appinfo>
</xs:annotation>
The Property is the name of the property. The Value is an XML string literal corresponding to a value of the appropriate type.
The representation properties can sometimes have complex syntax, so an element form for representation property bindings is provided as element content within the format element content model. This is provided to ease syntactic expression difficulties. The element is called dfdl:property and it has one attribute 'name' which provides the name of the property.
For example:
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:format>
<dfdl:property name='encoding'>utf-8</dfdl:property>
<dfdl:property name='separator'>%NL;</dfdl:property>
</dfdl:format>
</xs:appinfo>
</xs:annotation>
Element form is mostly used for properties that themselves contain the quotation mark characters and escape characters so that they can be expressed without concerns about confusion with the XSDL syntax use of these same characters. CDATA encapsulation can be used so as to allow malformed XML and mismatched quotes to be easily used as representation property values:
<dfdl:property
name='initiator'><[CDATA[<!-- ]]></dfdl:property>
To save textual clutter, short-form syntax for format annotations is also allowed on xs:element, xs:sequence, xs:choice, xs:group (for group references only), and xs:simpleType schema elements. (The xs:schema element cannot carry short-form annotations). Attributes which are in the namespace 'http://www.ogf.org/dfdl/dfdl-1.0/' and whose local name matches one of the DFDL representation properties are assumed to be equivalent to specific DFDL attribute form annotations.
For example the two forms below are equivalent in that they describe the same data format. The first is the short form of the second:
<xs:element
name="elem1">
<xs:complexType>
<xs:sequence
dfdl:separator="%HT;" >
...
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element
name="elem2">
<xs:complexType>
<xs:sequence>
<xs:annotation><xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:sequence separator="%HT;" />
</xs:appinfo></xs:annotation>
...
</xs:sequence>
</xs:complexType>
</xs:element>
Another example:
<xs:sequence
dfdl:separator=",">
<xs:element name="elem1"
type="xs:int" maxOccurs="unbounded"
dfdl:representation="text"
dfdl:textNumberRep="standard"
dfdl:initiator="["
dfdl:terminator="]"/>
<xs:element name="elem2"
type="xs:int" maxOccurs="unbounded">
<xs:annotation><xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:element
representation="text"
textNumberRep="standard"
initiator="["
terminator="]"/>
</xs:appinfo></xs:annotation>
</xs:element>
</xs:sequence>
Because short form syntax is not allowed on the xs:schema element, an attribute form dfdl:format annotation must be used instead.
DFDL provides no mechanism to un-set a property.
Setting a representation property's value to the empty string doesn't remove
the value for that property, but sets it to the empty string value. This may
not be appropriate as a value for certain properties.
For example, in delimited text data formats, it is sensible for the separator to be defined to be the empty string. This turns off use of separator delimiters. For many other string-valued properties, it is a Schema Definition Error to assign them the empty string value. For example, the character set encoding property (dfdl:encoding) cannot be set to the empty string.
One or more dfdl:defineFormat annotation elements can appear within the annotation children of the xs:schema element. DFDL defining annotation elements may only appear as annotation children of the xs:schema element.
The order of their appearance does not matter, nor does their position relative to other non-annotation children of the xs:schema.
Each dfdl:defineFormat has a required name attribute.
The construct creates a named data format definition. The value of the name attribute is of XML type NCName. The format name will become a member of the schema's target namespace. These names must be unique within the namespace.
If multiple format definitions have the same 'name' attribute, in the same namespace, then it is a Schema Definition Error.
Here is an example of a format definition:
<xs:schema
...>
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:defineFormat name="myConfig" >
<dfdl:format representation="text"
ref="textSpecialFormat1" />
</dfdl:defineFormat>
</xs:appinfo>
</xs:annotation>
...
</xs:schema>
A dfdl:defineFormat serves only to supply a named definition for a format for reuse from other places. It does not cause any use of the representation properties it contains to describe any actual data.
A dfdl:defineFormat declaration can inherit from another named format definition by use of the dfdl:ref property of the dfdl:format annotation. This allows a single-inheritance hierarchy that reuses definitions. When one definition extends another in this way, any property definitions contained in its direct elements override those in any inherited definition.
Conceptually, the dfdl:ref inheritance chains can be flattened and removed by copying all inherited property bindings and then superseding those for which there is a local binding. Throughout this document we will assume inheritance is fully flattened. That is, all dfdl:ref inheritance is first removed by flattening before any other examination of properties occurs.
It is a Schema Definition Error if use of the dfdl:ref property results in a circular path.
See Section 7.1.1 The dfdl:ref Property.
The dfdl:assert statement annotation element is used to assert truths about a DFDL model that are used when parsing to ensure that the data are well-formed. They are not used when unparsing.These checks are separate from validation checking and are performed even when validation is off. This distinction is needed to ensure that switching validation off does not affect parsing.
Examples of dfdl:assert elements are below:
<dfdl:assert message="Value is not zero." test="{ ../x eq 0}" />
<dfdl:assert message="Precondition violation." >
{../x le 0 and ../y ne "-->" and ../y ne "<!—" }
</dfdl:assert>
<dfdl:assert message="Postcondition violation." testKind='expression'>
{../x ne "'"}
</dfdl:assert>
A dfdl:assert annotation contains a test expression or a test pattern. The dfdl:assert is said to be successful if the test expression evaluates to true or the test pattern returns a non-zero length match, and unsuccessful if the test expression evaluates to false or the test pattern returns a zero length match. An unsuccessful dfdl:assert causes either a processing error or a recoverable error to be issued, as specified by the failureType property of the dfdl:assert.
The testKind property specifies whether an expression or pattern is used by the dfdl:assert. The expression or pattern can be expressed as an attribute or as a value.
<dfdl:assert test="{test expression}" />
<dfdl:assert>
{test expression}
</dfdl:assert>
It is a Schema Definition Error if a property is specified in more than one form.
It is a Schema Definition Error if both a test expression and a test pattern are specified.
A dfdl:assert can appear as an annotation on:
If the resolved set of statement annotations for a schema
component contains multiple dfdl:assert statements, then those with testKind
'pattern' are executed before those with testKind 'expression' (the default).
However, within each group the order of execution among them is not specified.
If one of the resolved set of asserts for a schema component is unsuccessful, and the failureType of the assert is ‘processingError’, then no further asserts in the set are executed.
|
Property Name |
Description |
|
testKind |
Enum (optional) Valid values are
'expression', 'pattern' Default value is 'expression' Specifies whether a DFDL
expression or DFDL regular expression is used in the dfdl:assert. Annotation: dfdl:assert |
|
test |
DFDL Expression Applies when testKind is 'expression' A DFDL expression that evaluates to true or false. If the expression evaluates to true then parsing continues. If the expression evaluates to false then a processing error is raised. Any element referred to by the expression must have already been processed or must be a descendent of this element. If a processing error occurs during the evaluation of the test expression then the dfdl:assert also fails. It is a Schema Definition Error
if testKind is 'expression' or not specified, and an expression is not
supplied by either the value of the dfdl:assert element or the value of the
test attribute. Annotation: dfdl:assert |
|
testPattern |
DFDL Regular Expression Applies when testKind is
'pattern' A DFDL regular expression that is
applied against the data stream starting at the data position
corresponding to the beginning of the representation. Consequently the
framing (including any initiator) is visible to the pattern.at
the start of the component on which the dfdl:assert is positioned. If
the pattern matching of the regular expression reads data that cannot be
decoded into characters of the current encoding, then the behavior is
controlled by the dfdl:encodingErrorPolicy property. See Section
11.2.1 Property dfdl:encodingErrorPolicy for details. If the length of the match is
zero then the dfdl:assert evaluates to false and a processing error is
raised. If the length of the match is
non-zero then the dfdl:assert evaluates to true. If a processing error occurs during the evaluation of the test regular expression then the dfdl:assert also fails. It is a Schema Definition Error
if testKind is 'pattern', and a pattern is not supplied by either the value
of the dfdl:assert element or the value of the testPattern property. It is a Schema Definition Error
if there is no value for the dfdl:encoding property in scope. It is a Schema Definition Error
if dfdl:leadingSkip is other than 0. It is a Schema Definition Error
if the dfdl:alignment is not 1 or 'implicit' Annotation: dfdl:assert |
|
message |
String or DFDL Expression Defines text to be used as a diagnostic code or for use in an error message, when the assert is unsuccessful. The DFDL Expression must return type xs:string. Any element referred to by the message expression must have already been processed or must be a descendent of this element. There is special treatment for errors that occur while evaluating the message expression. See below for details. Annotation: dfdl:assert |
|
failureType |
Enum (optional) Valid values are 'processingError', 'recoverableError'. Default value is 'processingError'. Specifies
the
type
of
failure
that
occurs
when
the
dfdl:assert
is
unsuccessful.
When 'processingError', a processing error is raised. When 'recoverableError', a recoverable error is raised. If an error occurs while evaluating the test expression, a processing error occurs, not a recoverable error. Recoverable errors do not cause backtracking like processing errors. Annotation: dfdl:assert |
Table 6 dfdl:assert properties
Example of a dfdl:assert with a message expression:
<dfdl:assert message="{
fn:concat('unknown case ', ../data1) }">
{ if (...pred1...) then ...expr1...
else if (...pred2...) then
...expr2...
else fn:false()
}
</dfdl:assert>
The
message specified by the message property is issued only if the dfdl:assert is
unsuccessful, that is, the test expression
evaluates to false or the test pattern returns a zero-length match. If
so, and the message property is an expression, the message expression is
evaluated at that time.
If
a processing error or Schema Definition Error occurs while evaluating the
message expression, a recoverable error is issued to record this error
(containing implementation-dependent content), then processing of the assert
continues as if there was no problem and in a manner consistent with the
failureType property, but using an implementation-dependent substitute message.
Schema authors can insert xs:sequence constructs to control
the timing of evaluation of statements more precisely.For example:
<xs:sequence dfdl:separator=",">
...
<xs:element ref="a" .../>
<xs:sequence>
<xs:sequence>
<xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/" >
<dfdl:assert test="{test expression}" />
</xs:appinfo></xs:annotation>
</xs:sequence>
<xs:element ref="b" .../>
</xs:sequence>
...
</xs:sequence>
In the above, the assert test expression is evaluated after parsing element 'a', and before parsing element "b". The use of two nested interior sequences surrounding element 'b' in this manner insures that the outermost sequence's separator usage is not disrupted.
DFDL
discriminators are used during parsing to resolve points of uncertainty that
cannot be resolved by speculative parsing. Discriminators are not used during
unparsing. They can also be used to
force a resolution earlier during the parsing of a group so that subsequent
parsing errors are treated as processing errors of a known component rather
than a failure to find a component.
A discriminator determines the existence or non-existence of a component. If the discriminator is successful then the component is known to exist and any subsequent errors will not cause backtracking at points of uncertainty. If a discriminator is unsuccessful then the component is known not to exist and backtracking occurs immediately.
If
the complex type of an element contains a sequence group as its content model
then if the sequence group is known not to exist, then the element is known not
to exist.
Examples of dfdl:discriminator annotation are below :
<dfdl:discriminator>
{ ../recType eq 0 }
</dfdl:discriminator>
<dfdl:discriminator test="{ ../recType eq 0}" />
When the discriminator's expression evaluates to "false", then it causes a processing error, and the discriminator is said to fail.
A DFDL discriminator contains a test expression that is an expression that evaluates to true or false. The discriminator is said to be successful if the test evaluates to true and unsuccessful (or fails) if the test evaluates to false.
The testKind property specifies whether an expression or pattern is used by the dfdl:discriminator. The expression or pattern can be expressed as an attribute or as a value.
<dfdl:discriminator test="{test expression}" />
<dfdl:discriminator>
{ test expression }
</dfdl:discriminator>
It is a Schema Definition Error if a property is specified in more than one form.
It is a Schema Definition Error if both a test expression and a test pattern are specified.
A dfdl:discriminator can be an annotation on
The resolved set of statement annotations for a schema component can contain only a single dfdl:discriminator or one or more dfdl:assert annotations, but not both. To clarify: dfdl:assert annotations and dfdl:discriminator annotations are exclusive of each other. It is a Schema Definition Error otherwise.
|
Property Name |
Description |
|
testKind |
Enum Valid
values are 'expression', 'pattern' Default
value is 'expression' Specifies
whether a DFDL expression or DFDL regular expression is used in the
dfdl:discriminator . Annotation: dfdl:discriminator |
|
test |
DFDL
Expression Applies when testKind is 'expression' A
DFDL expression that evaluates to true or false. If the expression evaluates
to true then the discriminator succeeds and parsing continues. If the
expression evaluates to false then the discriminator fails and a processing
error is raised. Any element referred to by the expression must have already been processed or is a descendent of this element. The expression must have been evaluated by the time this element and it descendents have been processed or when a processing error occurs when processing this element or its descendents. It
is a Schema Definition Error if testKind is 'expression' or not specified,
and an expression is not supplied by either the value of the dfdl:discriminator
element or the value of the test attribute. Annotation:
dfdl:discriminator |
|
testPattern |
DFDL
Regular Expression Applies
when testKind is 'pattern' A
DFDL regular expression that is applied against the data stream starting at
the data position corresponding to the beginning of the
representation. Consequently the framing (including any initiator) is visible
to the pattern.at the start of the component on
which the dfdl:discriminator is positioned. If the pattern matching of the regular
expression reads data that cannot be decoded into characters of the current
encoding, then the behavior is controlled by the dfdl:encodingErrorPolicy
property. See Section 11.2.1 Property dfdl:encodingErrorPolicy
for details. If
the length of the match is zero then the dfdl:discriminator evaluates to
false and a processing error is raised. If
the length of the match is non-zero then the dfdl:discriminator evaluates to
true. It
is a Schema Definition Error if testKind is 'pattern', and a pattern is not
supplied by either the value of the dfdl:discriminator element or the value
of the testPattern property. It
is a Schema Definition Error if there is no value for the dfdl:encoding
property in scope. It
is a Schema Definition Error if dfdl:leadingSkip is other than 0. It
is a Schema Definition Error if the dfdl:alignment is not 1 or 'implicit' Annotation:
dfdl:discriminator |
|
message |
String or DFDL Expression Defines text to be used as a diagnostic code or for use in an error message, when the discriminator is unsuccessful. The DFDL Expression must return type xs:string. Any element referred to by the message expression must have already been processed or must be a descendent of this element. There is special treatment for errors that occur while evaluating the message expression. See below for details. Annotation:
dfdl:discriminator |
Table 7 dfdl:discriminator properties
The message specified by the message
property is issued only if the discriminator is unsuccessful, that is, the test
expression evaluates to false or
the test pattern returns a zero-length match. If so, and the message property
is an expression, the message expression is evaluated at that time.
If a processing error or Schema Definition
Error occurs while evaluating the message expression, a recoverable error is
issued to record this error (containing implementation-dependent content), then
processing of the discriminator continues as if there was no problem, but in
the case of failure using an implementation-dependent substitute message.
Examples
of dfdl:discriminator annotations:
<xs:sequence>
<xs:choice>
<xs:element name='branchSimple' >
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:discriminator test='{. eq "a"}' />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name='branchComplex' >
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:discriminator test='{./identifier eq "b"}' />
</xs:appinfo>
</xs:annotation>
<xs:complexType >
<xs:sequence>
<xs:element name='identifier' />
...
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name='branchNestedComplex' >
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:discriminator test='{./Header/identifier eq "c"}'/>
</xs:appinfo>
</xs:annotation>
<xs:complexType >
<xs:sequence>
<xs:element name='Header' />
<xs:complexType >
<xs:sequence>
<xs:element name='identifier' />
...
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:choice>
</xs:sequence>
One or more dfdl:defineEscapeScheme annotation elements can appear within the annotation children of the xs:schema. The dfdl:defineEscapeScheme elements may only appear as annotation children of the xs:schema.
The order of their appearance does not matter, nor does their position relative to other annotation or non-annotation children of the xs:schema.
Each dfdl:defineEscapeScheme has a required name attribute and a required dfdl:escapeScheme child element.
The construct creates a named escape scheme definition. The value of the name attribute is of XML type NCName. The name will become a member of the schema's target namespace. These names must be unique within the namespace among escape schemes.
If multiple dfdl:defineEscapeScheme definitions have the same 'name' attribute, in the same namespace, then it is a Schema Definition Error.
Each dfdl:defineEscapeScheme annotation element contains a dfdl:escapeScheme annotation element as detailed below.
Here is an example of an escapeScheme definition:
<xs:schema
...>
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:defineEscapeScheme name="myEscapeScheme">
...
<dfdl:escapeScheme escapeKind="escapeCharacter"
escapeCharacter='/' />
...
</dfdl:defineEscapeScheme>
</xs:appinfo>
</xs:annotation>
...
</xs:schema>
A dfdl:defineEscapeScheme serves only to supply a named definition for a dfdl:escapeScheme for reuse from other places. It does not cause any use of the representation properties it contains to describe any actual data.
A named, reusable, escape scheme is used by referring to its name from a dfdl:escapeSchemeRef property on an element. For example:
<xs:element
name="foo" type="xs:string" >
<xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:element
representation="text"
escapeSchemeRef="myEscapeScheme"/>
</xs:appinfo></xs:annotation>
</xs:element>
The dfdl:escapeScheme annotation is used within a dfdl:defineEscapeScheme annotation to group the properties of an escape scheme and allows a common set of properties to be defined that can be reused.
An escape scheme defines the properties that describe the text escaping rules in force when data such as text delimiters are present in the data. There are two variants on such schemes,
- The use of a single escape character to cause the next character to be interpreted literally. The escape character itself is escaped by the escape escape character.
- The use of a pair of escape strings to cause the enclosed group of characters to be interpreted literally. The ending escape string is escaped by the escape escape character.
On parsing, the escape scheme is applied after pad characters are trimmed and on unparsing before pad characters are added.
DFDL does not perform any substitutions for ampersand notations like <.
The syntax of dfdl:escapeScheme is defined
in Section 13.2.1.Table
26 Properties Common to All Simple Types
with Text Representation
The dfdl:escapeScheme
PropertiesTable 26
Properties Common to All Simple Types with Text Representation
The dfdl:escapeScheme
Properties
Variables provide a means for communication within a set of DFDL schema. They are defined as top-level elements in a schema and therefore have global scope. .
A new variable is introduced using dfdl:defineVariable:
<dfdl:defineVariable
name = NCName
type? = QName
defaultValue? = logical value or dfdl expression
external? = 'false' | 'true' >
<!-- Contains: logical value or dfdl expression (default value) -->
</dfdl:defineVariable>
The
name of a newly defined variable is placed into the target namespace of the
schema containing the annotation. Variable names are distinct from format and
escape scheme names and so cannot conflict with them. A variable can have any type from the
DFDL subset of XML schema simple types. If no type is specified, the type is
xs:string.
The
defaultValue is optional. This is a literal value or an expression which
evaluates to a constant, and it can be specified as an attribute or as the
element value. If specified the default value must match the type of the
variable (otherwise it is a Schema Definition Error).
Note
that the syntax supports both a defaultValue attribute and the default value
being specified by the element value. Only one or the other may be present
(otherwise it is a Schema Definition Error). To set the default value to
"" (empty string), the defaultValue attribute syntax must be used, or
the expression { "" } must be used as the element value.
Note
the value of the name attribute is an NCName. The name of a variable is defined
in the target namespace of the schema containing the definition. If multiple
dfdl:defineVariable definitions have the same 'name' attribute in the same
namespace then it is a Schema Definition Error.
A
default instance of the variable is created (with global
scope). Further instances of the
variable may subsequently be created on schema elements. If the variable has a
default value, this will used as the default value for any instances of the variable (unless overridden when the
instance is created).
The
external property is optional.
If not specified it takes the default value 'false'. If true the value may be
provided by the DFDL processor and this external value will be used as the
global default value (overriding any defaultValue specified on the dfdl:defineVariable).
The mechanism by which the processor provides this value is
implementation-defined.
There
is no required order between dfdl:defineVariable and other schema level
defining annotations or a dfdl:format annotation that may refer to the
variable.
A defaultValue expression is evaluated before
processing the data stream begins.
A defaultValue expression can refer to other
variables but not to the infoset (so no path locations).The referenced variable
must either have a defaultValue or be external. It is a Schema Definition Error
otherwise.
If a defaultValue expression references another
variable then that prevents the referenced variable's value from ever changing,
that is, it is considered to be a read of the variable's value.
If a defaultValue expression references another
variable and this causes a circular reference, it is a Schema Definition Error.
It is a Schema Definition Error if the type of the variable is a user-defined simple type restriction.
<dfdl:defineVariable name="EDIFACT_DS" type="xs:string"
defaultValue="," />
<dfdl:defineVariable name="codepage" type="xs:string"
external="true">utf-8</dfdl:defineVariable>
The following variables are predefined
|
Name |
Namespace URI |
Type |
Default value |
External |
|
encoding |
http://www.ogf.org/dfdl/dfdl-1.0/ |
xs:string |
'UTF-8' |
true |
|
byteOrder |
http://www.ogf.org/dfdl/dfdl-1.0/ |
xs:string |
'bigEndian' |
true |
|
binaryFloatRep |
http://www.ogf.org/dfdl/dfdl-1.0/ |
xs:string |
'ieee' |
true |
|
outputNewLine |
http://www.ogf.org/dfdl/dfdl-1.0/ |
xs:string |
'%LF;' |
true |
Table 8 Pre-defined variables
These variables are expected to be commonly set externally so are predefined for convenience.
<xs:element name="title" type="xs:string">
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/">
<dfdl:element encoding="{$dfdl:encoding}"
/>
</xs:appinfo>
</xs:annotation>
</xs:element>
Scoped instances of defined variables are created using dfdl:newVariableInstance:
<dfdl:newVariableInstance
ref = QName
defaultValue? = logical value or dfdl expression
>
<!-- Contains: logical value or dfdl expression (value) -->
</dfdl:newVariableInstance>
Since an initial instance is created when the variable is defined, the use of dfdl:newVariableInstance is optional. It would be used if an instance with restricted scope is needed.
The dfdl:newVariableInstance annotation can be used on a group reference, sequence or choice only. It is a Schema Definition Error otherwise.
The scope of the instance of a variable is the dynamic scope of the schema component and its content model and so is inherited by any contained constructs or construct references.
The ref property is a QName. That is, it may be qualified with a namespace prefix.
An optional defaultValue for the instance may be specified. It can be specified as an attribute or as the element value. The expression must not contain forward references to elements which have not yet been processed nor to the current component. If specified the default value must match the type of the variable as specified by dfdl:defineVariable. If the instance is not assigned a new default value then it will inherit the default value specified by dfdl:defineVariable or externally provided by the DFDL processor. If a default value is not specified (and has not been specified by dfdl:defineVariable) then the value of this instance is undefined until explicitly set (using dfdl:setVariable).
If a default value is specified this initial value of the instance will be set when the instance is created. The value will override any (global) default value which was specified by dfdl:defineVariable or which was provided externally to the DFDL processor. A variable instance with a valid value (specified or default) can be referenced anywhere within the scope of the element on which the instance was created.
Note that the syntax supports both a defaultValue attribute and the default value being specified by the element value. Only one or the other may be present. (Schema definition error otherwise.)
To set the default value to "" (empty string), the defaultValue attribute syntax must be used, or the expression { "" } must be used as the element value.
The
resolved set of annotations for a component may contain multiple