Standards » Data Area » DFDL WG

Tresys Technology is pleased to announce that the Daffodil open source DFDL implementation is now able to run the NACHA Electronic Payment messages DFDL schema.

This demonstrates an increase in interoperability for Daffodil with DFDL schemas created and tested on the IBM DFDL implementation.

The version of Daffodil that has this functionality is still pre-release, but the forthcoming Daffodil 2.0.0 release will incorporate this capability. (Daffodil 2.0.0 development snapshots newer than 2017-01-11 can also be used.)

The NACHA Electronic Payment messages DFDL schema was originally created and tested by IBM in 2014.

Tresys and the Daffodil project are pleased to announce the first
release candidate of Daffodil 2.0.0, the open source implementation
of DFDL v1.0

Notable changes since Daffodil 1.1.0 include unparse achieving full
feature parity with parse, support for additional unparse only
properties such as outputValueCalc, fillByte,
truncateSpecifiedLengthString, etc., numerous bug fixes in parse, and
improved parse performance.

The release candidate has been tagged as 2.0.0-rc1 in the git repo:

https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil/browse

The release is also available for downloaded from the Daffodil Artifacts
page:

https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&version=2.0.0-rc1

Daffodil will now enter the test phase, which we expect to last
approximately 3 weeks, followed by the official release of Daffodil 2.0.0.

Please give the release candidate a test and let us know if you find any
issues. To report bugs or ask any questions, the development team is
available on the Daffodil users mailing list:

http://oss.tresys.com/mailman/listinfo/daffodil-users

as well as XMPP HipChat:

http://hipchat.ncsa.illinois.edu/gvZdmJHmq

or via the JIRA Daffodil bug tracker:

https://opensource.ncsa.illinois.edu/jira/projects/DFDL/summary

On behalf of the Daffodil team,
- Steve Lawrence

Everyone agrees about the term "parsing", but the inverse of that, well there has been some controversy. Some people use the term "serialization", others prefer "marshalling". In DFDL we have long used the term "unparsing", but somehow that made people uncomfortable in that it felt like we were making up words that didn't really exist.

Well, it was recently discovered that Wiktionary and Wikipedia provide definitions.

These aren't the Oxford English dictionary, but they represent sufficient usage and agreement that we're going to stop apologizing for the term unparsing.

Tresys Technology, LLC is pleased to announce the release of Daffodil
1.0! Binaries are available at:

https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL

Daffodil is the open source implementation of the Open Grid Forum's
Data Format Description Language (DFDL) specification.

http://www.ogf.org/dfdl

DFDL is a language capable of describing many data formats, including
textual and binary, commercial record-oriented, scientific and numeric,
modern and legacy, and many industry standards. It leverages XML
technology and concepts, using a subset of W3C XML schema type system
and annotations to describe such data formats. Daffodil uses this
description to parse data into an XML infoset for ingestion and validation.

Daffodil 1.0 is a monumental milestone that denotes stability and
usability, and is ready for general availability. Major features
that make up this release include:

- Command line interface available for Linux, Windows, and Mac
- API available in both Java and Scala
- Interactive debugger to incrementally step through a parse
- Test harness using Test Data Markup Language (TDML)
- Ability to save and reload Daffodil parsers
- Support for parsing both textual and binary data
- Unbounded lookahead with backtracking guided by asserts and
discriminators
- Numeric representations in text and two's complement binary
- Reusable definitions of formats and escape schemes
- DFDL expressions and variables for dynamic formats
- Delimited text with initiators, terminators, and separators
- Text justification (left, right, or center) with padding
- Binary data alignment, byte order, and bit order
- Fixed and variable data lengths
- Simple types like string, integer, floating point, hexBinary, date,
and time
- Ordered sequences and choices
- Arrays with fixed or variable repetitions
- XSD facets and restrictions for validation

Although Daffodil has reached this milestone, Daffodil continues active
development, with some features not yet available. Changes planned for
the future include:

- Unparsing support
- Performance improvements
- Unordered sequences
- New variable instances
- Boolean and binary calendar types
- Fixed and default XSD attributes
- IBM Mainframe numeric representations, including packed and zoned
binary
- Bi-directional text
- Floating elements

For a more detailed list of unsupported features, visit the unsupported
features and errata pages:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Features

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Errata

For a complete list of all open tickets, visit the Daffodil bug tracker:

https://opensource.ncsa.illinois.edu/jira/browse/DFDL/

For more information on Daffodil, visit the Daffodil wiki:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/

For questions and support, the development team is available on the
Daffodil mailing list:

http://oss.tresys.com/mailman/listinfo/daffodil-users

as well as XMPP HipChat:

http://hipchat.ncsa.illinois.edu/gvZdmJHmq

If you are interested in becoming a Daffodil contributor, visit the
daffodil wiki or the developers mailing list:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/

http://oss.tresys.com/mailman/listinfo/daffodil-dev

Thanks to everyone that has helped to make this milestone possible!

Some public comments were received on this document, and it has been updated accordingly, and is now published as [GFD 216].

This document summarizes the changes to DFDL motivated by the MIL-STD-2045 format which required adding the dfdl:bitOrder property to DFDL to handle "least significant bit first" data. The material in this document has been incorporated into the DFDL v1.0 specification.

This schema has been updated based on thorough review of the original version. The result is greatly superior. It is far simpler now, and is a much better illustration of how to use the dfdl:bitOrder property, which was added to DFDL in order to support this sort of dense binary data formats.

Conformance with the DFDL specification has also been improved - Some missing properties that are required by the DFDL standard are now present. Also, prior versions were depending upon some artifacts of the Daffodil DFDL implementation that are not conforming (and those bugs have been reported.)

This DFDL schema can be used with the Daffodil DFDL implementation -- where the dfdl:bitOrder feature of DFDL was prototyped and implemented.

A new repository has been added to the DFDLSchemas GitHub website which contains DFDL schemas for the vCard standard.

The initial schemas in the repository model vCard 2.1.

The schemas are compatible with IBM DFDL 1.1.1 release, as shipped in IBM Integration Bus 9.0.0.2 onwards.

Full details in the repository readme at https://github.com/DFDLSchemas/vCard.

The updated DFDL 1.0 specification is http://www.ogf.org/documents/GFD.207.pdf. This obsoletes the original specification GFD.174. Also published are two DFDL experience documents http://www.ogf.org/documents/GFD.214.pdf and http://www.ogf.org/documents/GFD.215.pdf.

Modeling a MIL-STD-2045 header in DFDL v1.0 is not possible without the addition of new capabilities for specifying bit order and non-standard encodings. There are many related military-standard binary data formats which are similar, and so cannot be modeled in DFDL.This document describes the new properties and property values that are required to successfully model this format.

Document and comment forum here: shttp://redmine.ogf.org/projects/editor-pubcom/boards/21

The DFDL PCAP (packet capture) schema has been updated:

- Now parses parts of the link layer (ethernet) the network layer
(IPv4, IPv6, ICMPv4) and the transport layer (TCP, UDP)
- Remove lldp.minimal capture since the new schema does not define lldp
- Add new IPv6 and ICMPv4 test captures

See https://github.com/DFDLSchemas/PCAP