News

Latest DFDL WG Call

Added by Steve Hanson about 1 month ago

Minutes from call on 2017-10-03 may be found at https://redmine.ogf.org/dmsf_files/13566?download=

Announcement: Daffodil 2.0.0 Released

Added by Michael Beckerle 2 months ago

We are pleased to announce the release of Daffodil 2.0.0!

Downloads are available as zip, tar, and rpm at:

https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&version=2.0.0

Daffodil is the open source implementation of the Open Grid Forum's
Data Format Description Language (DFDL) specification.

http://www.ogf.org/dfdl

DFDL is a language capable of describing many data formats, including
textual and binary, commercial record-oriented, scientific and numeric,
modern and legacy, and many industry standards. It leverages XML
technology and concepts, using a subset of W3C XML schema type system
and annotations to describe such data formats. Various DFDL schemas are
publicly available on github at:

https://github.com/DFDLSchemas

Daffodil uses such descriptions to parse data into an XML or JSON
infoset for ingestion and validation. Daffodil can also perform the
reverse, by serializing or "unparsing" an infoset back to the original
data format.

Daffodil 2.0.0 is a monumental milestone that denotes stability and
usability, and is ready for general availability. Major features that
make up this release include:

- Support for unparse with one-to-one feature parity with parse
- New API with native support for JDOM, w3c DOM, Scala XML, XML text, and JSON
- Support for many non-byte size encodings
- Support for xs:choice direct dispatch
- Support for xs:boolean types
- Support for xs:ignoreCase="yes"
- Support for xs:union restrictions
- Many improvements to the DFDL expression language
- Numerous bug fixes and performance improvements

More details of the changes are available:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Changelog
https://opensource.ncsa.illinois.edu/jira/secure/ReleaseNote.jspa?projectId=10010&version=13790

Daffodil 2.0.0 has also been integrated into Apache NiFi and XML
Calabash, with the source available in the following git repositories:

Apache NiFi: https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil-nifi/browse
XML Calabash: https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil-calabash-extension/browse

Although Daffodil has reached this milestone, active development
continues, with some features not yet available. For a detailed list of
unsupported features, visit the unsupported features and errata pages:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Features
https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Errata

For more information on Daffodil, visit the Daffodil wiki:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/

For questions and support, the development team is available on the
Daffodil mailing list:

http://oss.tresys.com/mailman/listinfo/daffodil-users

as well as XMPP HipChat:

http://hipchat.ncsa.illinois.edu/gvZdmJHmq

If you are interested in becoming a Daffodil contributor, visit the
daffodil wiki or the developers mailing list:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/
http://oss.tresys.com/mailman/listinfo/daffodil-dev

For a complete list of all open tickets, visit the Daffodil bug tracker:

https://opensource.ncsa.illinois.edu/jira/browse/DFDL/

Thanks to everyone that has helped to make this milestone possible!

Announcing the third release candidate of Daffodil 2.0.0

Added by Michael Beckerle 4 months ago

The Daffodil open source DFDL project team is
pleased to announce the third release candidate of Daffodil 2.0.0!

Notable changes since the second release candidate include:
- New API with support for JDOM, w3c DOM, Scala XML, XML text, and json
- Support for xs:choice direct dispatch
- Support for xs:boolean types
- Support for 3-, 4-, 5-, and 6-bit encoders
- Support for xs:hexBinary with bit lengths
- Support for xs:ignoreCase="yes"
- Support for xs:union restrictions
- Improvements to the DFDL expression language, including fixes for
div/idiv, type conversions and literal constants, bugs in daf:trace(),
dfdl:valueLength() and dfdl:contentLength(), and support for
fn:error() and math:pow() functions
- Numerous bug fixes and performance improvements

The release candidate has been tagged as 2.0.0-rc3 in the git repo:

https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil/browse

The release is also available for downloaded from the Daffodil Artifacts
page:

https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&version=2.0.0-RC3

Please test the release candidate and let us know if you find any
issues. To report bugs or ask any questions, the development team is
available on the Daffodil users mailing list:

http://oss.tresys.com/mailman/listinfo/daffodil-users

as well as XMPP HipChat:

http://hipchat.ncsa.illinois.edu/gvZdmJHmq

or via the JIRA Daffodil bug tracker:

https://opensource.ncsa.illinois.edu/jira/projects/DFDL/summary

Thanks!
The Daffodil Team

DFDL Schema for NITF Image file format

Added by Michael Beckerle 10 months ago

Tresys Technology is pleased to announce the release of a DFDL schema to
describe MIL-STD-2500C: National Imagery Transmission Format, more
commonly known as NITF. The schema and test files are available for
download on the DFDLSchemas github page https://github.com/DFDLSchemas/NITF

This schema was developed and tested using the Daffodil open-source DFDL
implementation.

The version of Daffodil that has the functionality to parse and unparse
NITF files using this schema is still pre-release, but the forthcoming
Daffodil 2.0.0 release will incorporate the capability. (Daffodil 2.0.0
development snapshots newer than 2017-01-12 can also be used.)

Daffodil now runs NACHA DFDL Schema

Added by Michael Beckerle 10 months ago

Tresys Technology is pleased to announce that the Daffodil open source DFDL implementation is now able to run the NACHA Electronic Payment messages DFDL schema.

This demonstrates an increase in interoperability for Daffodil with DFDL schemas created and tested on the IBM DFDL implementation.

The version of Daffodil that has this functionality is still pre-release, but the forthcoming Daffodil 2.0.0 release will incorporate this capability. (Daffodil 2.0.0 development snapshots newer than 2017-01-11 can also be used.)

The NACHA Electronic Payment messages DFDL schema was originally created and tested by IBM in 2014.

Daffodil 2.0.0 - Release Announcement

Added by Michael Beckerle about 1 year ago

Tresys and the Daffodil project are pleased to announce the first
release candidate of Daffodil 2.0.0, the open source implementation
of DFDL v1.0

Notable changes since Daffodil 1.1.0 include unparse achieving full
feature parity with parse, support for additional unparse only
properties such as outputValueCalc, fillByte,
truncateSpecifiedLengthString, etc., numerous bug fixes in parse, and
improved parse performance.

The release candidate has been tagged as 2.0.0-rc1 in the git repo:

https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil/browse

The release is also available for downloaded from the Daffodil Artifacts
page:

https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&version=2.0.0-rc1

Daffodil will now enter the test phase, which we expect to last
approximately 3 weeks, followed by the official release of Daffodil 2.0.0.

Please give the release candidate a test and let us know if you find any
issues. To report bugs or ask any questions, the development team is
available on the Daffodil users mailing list:

http://oss.tresys.com/mailman/listinfo/daffodil-users

as well as XMPP HipChat:

http://hipchat.ncsa.illinois.edu/gvZdmJHmq

or via the JIRA Daffodil bug tracker:

https://opensource.ncsa.illinois.edu/jira/projects/DFDL/summary

On behalf of the Daffodil team,
- Steve Lawrence

Unparsing - that's what we call it

Added by Michael Beckerle over 1 year ago

Everyone agrees about the term "parsing", but the inverse of that, well there has been some controversy. Some people use the term "serialization", others prefer "marshalling". In DFDL we have long used the term "unparsing", but somehow that made people uncomfortable in that it felt like we were making up words that didn't really exist.

Well, it was recently discovered that Wiktionary and Wikipedia provide definitions.

These aren't the Oxford English dictionary, but they represent sufficient usage and agreement that we're going to stop apologizing for the term unparsing.

Daffodil 1.0 DFDL Implementation Released

Added by Michael Beckerle over 2 years ago

Tresys Technology, LLC is pleased to announce the release of Daffodil
1.0! Binaries are available at:

https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL

Daffodil is the open source implementation of the Open Grid Forum's
Data Format Description Language (DFDL) specification.

http://www.ogf.org/dfdl

DFDL is a language capable of describing many data formats, including
textual and binary, commercial record-oriented, scientific and numeric,
modern and legacy, and many industry standards. It leverages XML
technology and concepts, using a subset of W3C XML schema type system
and annotations to describe such data formats. Daffodil uses this
description to parse data into an XML infoset for ingestion and validation.

Daffodil 1.0 is a monumental milestone that denotes stability and
usability, and is ready for general availability. Major features
that make up this release include:

- Command line interface available for Linux, Windows, and Mac
- API available in both Java and Scala
- Interactive debugger to incrementally step through a parse
- Test harness using Test Data Markup Language (TDML)
- Ability to save and reload Daffodil parsers
- Support for parsing both textual and binary data
- Unbounded lookahead with backtracking guided by asserts and
discriminators
- Numeric representations in text and two's complement binary
- Reusable definitions of formats and escape schemes
- DFDL expressions and variables for dynamic formats
- Delimited text with initiators, terminators, and separators
- Text justification (left, right, or center) with padding
- Binary data alignment, byte order, and bit order
- Fixed and variable data lengths
- Simple types like string, integer, floating point, hexBinary, date,
and time
- Ordered sequences and choices
- Arrays with fixed or variable repetitions
- XSD facets and restrictions for validation

Although Daffodil has reached this milestone, Daffodil continues active
development, with some features not yet available. Changes planned for
the future include:

- Unparsing support
- Performance improvements
- Unordered sequences
- New variable instances
- Boolean and binary calendar types
- Fixed and default XSD attributes
- IBM Mainframe numeric representations, including packed and zoned
binary
- Bi-directional text
- Floating elements

For a more detailed list of unsupported features, visit the unsupported
features and errata pages:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Features

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Errata

For a complete list of all open tickets, visit the Daffodil bug tracker:

https://opensource.ncsa.illinois.edu/jira/browse/DFDL/

For more information on Daffodil, visit the Daffodil wiki:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/

For questions and support, the development team is available on the
Daffodil mailing list:

http://oss.tresys.com/mailman/listinfo/daffodil-users

as well as XMPP HipChat:

http://hipchat.ncsa.illinois.edu/gvZdmJHmq

If you are interested in becoming a Daffodil contributor, visit the
daffodil wiki or the developers mailing list:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/
http://oss.tresys.com/mailman/listinfo/daffodil-dev

Thanks to everyone that has helped to make this milestone possible!

Experience document - bitOrder and MIL-STD-2045 - updated.

Added by Michael Beckerle over 2 years ago

Some public comments were received on this document, and it has been updated accordingly, and is now published as [GFD 216].

This document summarizes the changes to DFDL motivated by the MIL-STD-2045 format which required adding the dfdl:bitOrder property to DFDL to handle "least significant bit first" data. The material in this document has been incorporated into the DFDL v1.0 specification.

DFDL Schema for MIL-STD-2045 binary message header - updated

Added by Michael Beckerle almost 3 years ago

This schema has been updated based on thorough review of the original version. The result is greatly superior. It is far simpler now, and is a much better illustration of how to use the dfdl:bitOrder property, which was added to DFDL in order to support this sort of dense binary data formats.

Conformance with the DFDL specification has also been improved - Some missing properties that are required by the DFDL standard are now present. Also, prior versions were depending upon some artifacts of the Daffodil DFDL implementation that are not conforming (and those bugs have been reported.)

This DFDL schema can be used with the Daffodil DFDL implementation -- where the dfdl:bitOrder feature of DFDL was prototyped and implemented.

1 2 Next »

Also available in: Atom