News

Unparsing - that's what we call it

Added by Michael Beckerle almost 3 years ago

Everyone agrees about the term "parsing", but the inverse of that, well there has been some controversy. Some people use the term "serialization", others prefer "marshalling". In DFDL we have long used the term "unparsing", but somehow that made people uncomfortable in that it felt like we were making up words that didn't really exist.

Well, it was recently discovered that Wiktionary and Wikipedia provide definitions.

These aren't the Oxford English dictionary, but they represent sufficient usage and agreement that we're going to stop apologizing for the term unparsing.

Daffodil 1.0 DFDL Implementation Released

Added by Michael Beckerle about 4 years ago

Tresys Technology, LLC is pleased to announce the release of Daffodil
1.0! Binaries are available at:

https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL

Daffodil is the open source implementation of the Open Grid Forum's
Data Format Description Language (DFDL) specification.

http://www.ogf.org/dfdl

DFDL is a language capable of describing many data formats, including
textual and binary, commercial record-oriented, scientific and numeric,
modern and legacy, and many industry standards. It leverages XML
technology and concepts, using a subset of W3C XML schema type system
and annotations to describe such data formats. Daffodil uses this
description to parse data into an XML infoset for ingestion and validation.

Daffodil 1.0 is a monumental milestone that denotes stability and
usability, and is ready for general availability. Major features
that make up this release include:

- Command line interface available for Linux, Windows, and Mac
- API available in both Java and Scala
- Interactive debugger to incrementally step through a parse
- Test harness using Test Data Markup Language (TDML)
- Ability to save and reload Daffodil parsers
- Support for parsing both textual and binary data
- Unbounded lookahead with backtracking guided by asserts and
discriminators
- Numeric representations in text and two's complement binary
- Reusable definitions of formats and escape schemes
- DFDL expressions and variables for dynamic formats
- Delimited text with initiators, terminators, and separators
- Text justification (left, right, or center) with padding
- Binary data alignment, byte order, and bit order
- Fixed and variable data lengths
- Simple types like string, integer, floating point, hexBinary, date,
and time
- Ordered sequences and choices
- Arrays with fixed or variable repetitions
- XSD facets and restrictions for validation

Although Daffodil has reached this milestone, Daffodil continues active
development, with some features not yet available. Changes planned for
the future include:

- Unparsing support
- Performance improvements
- Unordered sequences
- New variable instances
- Boolean and binary calendar types
- Fixed and default XSD attributes
- IBM Mainframe numeric representations, including packed and zoned
binary
- Bi-directional text
- Floating elements

For a more detailed list of unsupported features, visit the unsupported
features and errata pages:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Features

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Errata

For a complete list of all open tickets, visit the Daffodil bug tracker:

https://opensource.ncsa.illinois.edu/jira/browse/DFDL/

For more information on Daffodil, visit the Daffodil wiki:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/

For questions and support, the development team is available on the
Daffodil mailing list:

http://oss.tresys.com/mailman/listinfo/daffodil-users

as well as XMPP HipChat:

http://hipchat.ncsa.illinois.edu/gvZdmJHmq

If you are interested in becoming a Daffodil contributor, visit the
daffodil wiki or the developers mailing list:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/
http://oss.tresys.com/mailman/listinfo/daffodil-dev

Thanks to everyone that has helped to make this milestone possible!

Experience document - bitOrder and MIL-STD-2045 - updated.

Added by Michael Beckerle about 4 years ago

Some public comments were received on this document, and it has been updated accordingly, and is now published as [GFD 216].

This document summarizes the changes to DFDL motivated by the MIL-STD-2045 format which required adding the dfdl:bitOrder property to DFDL to handle "least significant bit first" data. The material in this document has been incorporated into the DFDL v1.0 specification.

DFDL Schema for MIL-STD-2045 binary message header - updated

Added by Michael Beckerle over 4 years ago

This schema has been updated based on thorough review of the original version. The result is greatly superior. It is far simpler now, and is a much better illustration of how to use the dfdl:bitOrder property, which was added to DFDL in order to support this sort of dense binary data formats.

Conformance with the DFDL specification has also been improved - Some missing properties that are required by the DFDL standard are now present. Also, prior versions were depending upon some artifacts of the Daffodil DFDL implementation that are not conforming (and those bugs have been reported.)

This DFDL schema can be used with the Daffodil DFDL implementation -- where the dfdl:bitOrder feature of DFDL was prototyped and implemented.

DFDL schemas for vCard

Added by Steve Hanson over 4 years ago

A new repository has been added to the DFDLSchemas GitHub website which contains DFDL schemas for the vCard standard.

The initial schemas in the repository model vCard 2.1.

The schemas are compatible with IBM DFDL 1.1.1 release, as shipped in IBM Integration Bus 9.0.0.2 onwards.

Full details in the repository readme at https://github.com/DFDLSchemas/vCard.

Updated DFDL 1.0 Specification published

Added by Steve Hanson over 4 years ago

The updated DFDL 1.0 specification is http://www.ogf.org/documents/GFD.207.pdf. This obsoletes the original specification GFD.174. Also published are two DFDL experience documents http://www.ogf.org/documents/GFD.214.pdf and http://www.ogf.org/documents/GFD.215.pdf.

DFDL Experience document 'Bit Order' is in public comment

Added by Steve Hanson over 4 years ago

Modeling a MIL-STD-2045 header in DFDL v1.0 is not possible without the addition of new capabilities for specifying bit order and non-standard encodings. There are many related military-standard binary data formats which are similar, and so cannot be modeled in DFDL.This document describes the new properties and property values that are required to successfully model this format.

Document and comment forum here: shttp://redmine.ogf.org/projects/editor-pubcom/boards/21

DFDL PCAP Schema Updated

Added by Michael Beckerle over 4 years ago

The DFDL PCAP (packet capture) schema has been updated:

- Now parses parts of the link layer (ethernet) the network layer
(IPv4, IPv6, ICMPv4) and the transport layer (TCP, UDP)
- Remove lldp.minimal capture since the new schema does not define lldp
- Add new IPv6 and ICMPv4 test captures

See https://github.com/DFDLSchemas/PCAP

DFDL Schema for MIL-STD-2045 binary message header

Added by Michael Beckerle over 4 years ago

A preliminary version of a DFDL Schema for MIL-STD-2045 is now available on the github DFDL Schemas site.

This schema makes use of the dfdl:bitOrder='leastSignificantBitFirst' property and the x-dfdl-us-ascii-7-bit-packed character set encoding that are being added to DFDL v1.0 in response to a public comment review.

These features have been prototyped in the Daffodil open-source DFDL implementation; hence, this DFDL schema will (as of this writing) only run with that DFDL implementation. This schema is, in effect, the proof point that the new DFDL features will in fact address the needs that were raised by the public comment on bit ordering.

This data format has much in common with other binary MIL-STD data formats, so techniques illustrated in it are expected to be used in other DFDL schemas. However, as of this news item some improved ways of modeling the data have already been discovered, so this DFDL schema for MIL-STD-2045 is expected to evolve substantially.

The schema is available at https://github.com/DFDLSchemas/mil-std-2045

DFDL schemas for 4690 Tlog

Added by Steve Hanson over 4 years ago

The repository in DFDLSchemas GitHub which contains DFDL schemas for the Toshiba Commerce (formerly IBM) 4690 Transaction Log (Tlog) point-of-sale format has been updated.

It now includes schemas for SuperMarket Application (SA) in addition to the existing ACE schemas.

The schemas are compatible with IBM DFDL 1.0 and 1.1 releases, as shipped in IBM WebSphere Message Broker 8.0.0.2 onwards and IBM Integration Bus 9.0.0.1 onwards, respectively.

Example data streams and XML infosets are included.

Full details in the repository readme at https://github.com/DFDLSchemas/IBM4690-TLOG.

« Previous 1 2 3 Next »

Also available in: Atom