Standards » Data Area » DFDL WG

DFDL is getting attention in the industrial control / IoT space.

Steve Lawrence of Tresys Technology, will present at this year's ApacheCon NA 2019 in Las Vegas, NV, USA on September 10.

Industrial IoT is made up of many different programmable logic controllers (PLCs) that communicate with often incompatible and proprietary data formats. Apache PLC4X standardizes access to these PLCs by creating a set of libraries across a variety of languages to read and write PLC data with a unified API. Unfortunately, developing and maintaining support for these different data formats, especially across multiple languages, can be burdensome. In this talk we discuss how we can describe PLC data using the open standard Data Format Description Language (DFDL), how Apache Daffodil uses these descriptions to transform data to a standard and easily consumable representation, and an approach to extend PLC4X to use Daffodil and DFDL descriptions to reduce the effort needed to support a variety of PLCs and languages.

https://www.apachecon.com/acna19/s/#/scheduledEvent/1081

A body of new DFDL tutorial materials have been published at

http://www.xfront.com/DFDL/index.html

This was developed and tested with Apache Daffodil (incubating) by Roger Costello of MITRE Corp. who maintains the XFront site.

The site is XML-oriented and so most of the tutorial material is focused around using DFDL to map non-XML data to/from an XML form. Which is to say the site and this material is for XML enthusiasts. (The Daffodil DFDL implementation also fully supports JSON.)

The tutorial materials cover textual data formats and binary data formats, and the DFDL properties used to parse/unparse them.
Lab exercises to help learn DFDL are included as well.

IBM® z/Transaction Processing Facility (z/TPF) is a high-volume, high-throughput transaction processor, that can handle large, continuous loads of transactions across geographically dispersed networks. z/TPF is also an operating system and a unique database, all designed to work together as one system. The z/TPF database and other data formats are highly optimized for transactional workloads, but this can present challenges for other platforms to consume z/TPF data. Data Format Description Language (DFDL) has proven itself to be a robust solution in describing data formats such that the data can be easily consumed by other platforms or from other platforms. DFDL has also proven itself to be very useful in solving data transformation problems. As such, z/TPF has made extensive use of IBM's implementation of DFDL to solve several challenges in an open and extensible manner. Further, z/TPF has created a high performance DFDL parser that is specialized for data formats used by z/TPF.

Rapid REST Service Enablement¶

DFDL enables z/TPF customers to rapidly REST enable their existing services without making application changes. Prior to DFDL support, for each service, customers had to write code to manually parse XML and JSON documents and transform the data into the format the existing application requires. It is time consuming to write and test that code for each service and is often error prone and brittle. With DFDL usage in C/C++ code, the customer uses IBM DFDL tooling to create a DFDL schema for the service. Then the transformation from XML/JSON format to the native application data format is done automatically by the z/TPF system. In the same fashion, DFDL is used to create XML/JSON responses from native application data formats. This has enabled z/TPF customers to dramatically reduce the time and effort required to create and deploy REST services. Further, z/TPF's DFDL parser has been extended to include support to allow transformations to occur to and from native application memory structures composed blocks of memory linked by pointers.

Making Data Easily Consumable by Other Platforms (Push Model)¶

z/TPF customers often send transactional data in real-time to other platforms for multiple business purposes. Transactional data can include data from the transaction itself, database records that were updated by the transaction, as well as application enriched context data to explain why the database updates were made. The data can be put into a stream to feed an event driven architecture system, processed by an analytics platform, or placed in a read-only database (SQL or NoSQL) to be used by other parts of the customer business. Prior to DFDL support, this data would have to be transformed from various formats on z/TPF into a form consumable by the other platform in a process known as Extract Transform and Load (ETL). As part of this ETL processing, fields may need to be hidden, transformed, etc. based upon the use case. A further challenge of these types of scenarios is keeping the various systems in sync regarding the format of the data as the database on z/TPF grows and changes over time. This ultimately results in a lot of custom code being written and maintained. DFDL substantially simplifies this by automatically transforming the z/TPF data, omitting fields, dealing with various data format versions, and more. DFDL can be used to format the transferred data as XML or JSON depending upon the use case. The investment to leverage DFDL has substantially decreased IBM and customer investment in solutions to make z/TPF data available for processing by other systems.

Making Data Easily Consumable by Other Platforms (Pull Model)¶

z/TPF MongoDB interface support allows z/TPF customers to access data in z/TPF databases using a standard MongoDB client. Instead of writing a custom data handler, the DFDL parser on z/TPF was modified to operate upon BSON data to transform the data to and from the z/TPF database format. This MongoDB interface provides a standard NoSQL interface to read and write proprietary formatted data on z/TPF where the application developer does not have to know anything about z/TPF and requires no user code on z/TPF. This reduces the development costs and time to market for cloud-based applications that need to access and update data residing on z/TPF.

Enabling Real-time Signal Events¶

Often, customers need their z/TPF applications to conditionally send a signal and data to an event consumer system in their enterprise. For example, if an airplane’s inventory is exhausted for given route, the z/TPF system may need to send various information to another system in the enterprise. Signal events provides a transparent configurable infrastructure that the application code can call to send the message off platform. DFDL is used to transform the outgoing message into XML or JSON. In this way, z/TPF customers can quickly code, test, and deploy signaling mechanisms that can be easily consumed by their existing event consumers.

Enabling Cross-Programming Language Application Service Calls¶

z/TPF customers want to progressively modernize their applications by adding new business logic written in Java that needs to call or be invoked by existing application code written in C/C++. Instead of writing custom connectors and transformations for every possible call, DFDL is used to map C structures to Java objects, and vice-versa. This provides a seamless interface that allows programmers to mix new (Java) and old (C/C++) services within an application to easily extend existing applications and build new Java applications that leverage existing assets.

Quickly Creating New Performance Monitoring Tooling¶

IBM has implemented various z/TPF performance monitoring tools to aid customers in understanding how their z/TPF systems are running. Some of this tooling now leverages DFDL. For performance reasons, the z/TPF system streams the data in a raw binary format to some media for post processing. This raw data is then consumed by a remote system leveraging IBM DFDL and JAXB to directly convert the binary data into Java objects for further analysis. This is entirely accomplished without writing any parsing code, which substantially reduces the time required to code, test and deliver complex solutions that rely on binary z/TPF data.

Easy Conversion to Standard Data Formats¶

z/TPF has found other ways of using DFDL on the system. For example, a tool was written very quickly to dump in memory tables to a CSV file simply by having a DFDL file that described the in-memory table. This provides an easily produced set of diagnostic data in a standard format. DFDL can also be used to format memory for other tooling uses. And as already mentioned, DFDL has a breadth of uses in being able to transform or serialize binary data to and from JSON, XML, etc.

Summary¶

DFDL has become a cornerstone of z/TPF to consume, exchange, and transform data. The DFDL and z/TPF infrastructure provides tremendous development and maintenance efficiencies along with quality improvements for IBM and our customers.

Bradd Kadlecik (IBM z/TPF)

On Wednesday Sept 26, Mike Beckerle presented DFDL and the Apache Daffodil (Incubating) project which implements DFDL, at the annual ApacheCon North-America conference which was held in at the Marriott Chateau Champlain Hotel, in Montreal, Quebec, Canada.

The slides and audio of the session are available.
Slides (as a zipped pdf file) are here: https://s.apache.org/apacheconNA2018-dfdl
The audio is here: https://s.apache.org/apacheconNA2018-dfdl-audio

The Apache Daffodil (incubating) community is pleased to announce the release of version 2.2.0.

Release notes and downloads are available at:

https://daffodil.apache.org/releases/2.2.0/

Daffodil is an open source implementation of the DFDL (Data Format Description Language) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format.

For more information about Daffodil visit:

https://daffodil.apache.org/

Regards,
The Apache Daffodil Team

The Apache Daffodil (incubating) community is pleased to announce the
release of version 2.1.0.

Release notes and downloads are available at:

https://daffodil.apache.org/releases/2.1.0/

Daffodil is an open source implementation of the DFDL (Data Format
Description Language) specification that uses DFDL schemas to parse
fixed format data into an infoset, which is most commonly represented as
either XML or JSON. This allows the use of well-established XML or JSON
technologies and libraries to consume, inspect, and manipulate fixed
format data in existing solutions. Daffodil is also capable of the
reverse by serializing or "unparsing" an XML or JSON infoset back to the
original data format.

For more information about Daffodil visit:

https://daffodil.apache.org/

Regards,
The Apache Daffodil Team

Following on from their Java implementation DFDL4S, The European Space Agency have released a C++ version.
See https://eop-cfi.esa.int/index.php/79-applications/213-new-release-announcement-dfdl4s-library-v1-0

We are pleased to announce the release of Daffodil 2.0.0!

Downloads are available as zip, tar, and rpm at:

https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&version=2.0.0

Daffodil is the open source implementation of the Open Grid Forum's
Data Format Description Language (DFDL) specification.

http://www.ogf.org/dfdl

DFDL is a language capable of describing many data formats, including
textual and binary, commercial record-oriented, scientific and numeric,
modern and legacy, and many industry standards. It leverages XML
technology and concepts, using a subset of W3C XML schema type system
and annotations to describe such data formats. Various DFDL schemas are
publicly available on github at:

https://github.com/DFDLSchemas

Daffodil uses such descriptions to parse data into an XML or JSON
infoset for ingestion and validation. Daffodil can also perform the
reverse, by serializing or "unparsing" an infoset back to the original
data format.

Daffodil 2.0.0 is a monumental milestone that denotes stability and
usability, and is ready for general availability. Major features that
make up this release include:

- Support for unparse with one-to-one feature parity with parse
- New API with native support for JDOM, w3c DOM, Scala XML, XML text, and JSON
- Support for many non-byte size encodings
- Support for xs:choice direct dispatch
- Support for xs:boolean types
- Support for xs:ignoreCase="yes"
- Support for xs:union restrictions
- Many improvements to the DFDL expression language
- Numerous bug fixes and performance improvements

More details of the changes are available:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Changelog

https://opensource.ncsa.illinois.edu/jira/secure/ReleaseNote.jspa?projectId=10010&version=13790

Daffodil 2.0.0 has also been integrated into Apache NiFi and XML
Calabash, with the source available in the following git repositories:

Apache NiFi: https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil-nifi/browse

XML Calabash: https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil-calabash-extension/browse

Although Daffodil has reached this milestone, active development
continues, with some features not yet available. For a detailed list of
unsupported features, visit the unsupported features and errata pages:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Features

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Errata

For more information on Daffodil, visit the Daffodil wiki:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/

For questions and support, the development team is available on the
Daffodil mailing list:

http://oss.tresys.com/mailman/listinfo/daffodil-users

as well as XMPP HipChat:

http://hipchat.ncsa.illinois.edu/gvZdmJHmq

If you are interested in becoming a Daffodil contributor, visit the
daffodil wiki or the developers mailing list:

https://opensource.ncsa.illinois.edu/confluence/display/DFDL/

http://oss.tresys.com/mailman/listinfo/daffodil-dev

For a complete list of all open tickets, visit the Daffodil bug tracker:

https://opensource.ncsa.illinois.edu/jira/browse/DFDL/

Thanks to everyone that has helped to make this milestone possible!

The Daffodil open source DFDL project team is
pleased to announce the third release candidate of Daffodil 2.0.0!

Notable changes since the second release candidate include:
- New API with support for JDOM, w3c DOM, Scala XML, XML text, and json
- Support for xs:choice direct dispatch
- Support for xs:boolean types
- Support for 3-, 4-, 5-, and 6-bit encoders
- Support for xs:hexBinary with bit lengths
- Support for xs:ignoreCase="yes"
- Support for xs:union restrictions
- Improvements to the DFDL expression language, including fixes for
div/idiv, type conversions and literal constants, bugs in daf:trace(),
dfdl:valueLength() and dfdl:contentLength(), and support for
fn:error() and math:pow() functions
- Numerous bug fixes and performance improvements

The release candidate has been tagged as 2.0.0-rc3 in the git repo:

https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil/browse

The release is also available for downloaded from the Daffodil Artifacts
page:

https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&version=2.0.0-RC3

Please test the release candidate and let us know if you find any
issues. To report bugs or ask any questions, the development team is
available on the Daffodil users mailing list:

http://oss.tresys.com/mailman/listinfo/daffodil-users

as well as XMPP HipChat:

http://hipchat.ncsa.illinois.edu/gvZdmJHmq

or via the JIRA Daffodil bug tracker:

https://opensource.ncsa.illinois.edu/jira/projects/DFDL/summary

Thanks!
The Daffodil Team

Tresys Technology is pleased to announce the release of a DFDL schema to
describe MIL-STD-2500C: National Imagery Transmission Format, more
commonly known as NITF. The schema and test files are available for
download on the DFDLSchemas github page https://github.com/DFDLSchemas/NITF

This schema was developed and tested using the Daffodil open-source DFDL
implementation.

The version of Daffodil that has the functionality to parse and unparse
NITF files using this schema is still pre-release, but the forthcoming
Daffodil 2.0.0 release will incorporate the capability. (Daffodil 2.0.0
development snapshots newer than 2017-01-12 can also be used.)