Forums » #117 - DFDL v1.0 Revision »
Expand use of recoverable exceptions
Added by Steve Hanson about 9 years ago
Recoverable exceptions were introduced by an erratum primarily to allow checks to be performed on well-formed data beyond those checks possible using XML schema validation. The point being that there is no need for the parse to terminate; the data is well-formed and the parser can continue. This is important for several industry standards like X12 and SWIFT where the standard says that all errors in the data are reported on, not just the first.
Currently recoverable error are limited to use in asserts. However this is not sufficient for the purposes described above. For example, X12 says that all missing required fields are reported, but today such a failure will cause a processing error and stop the parse. Similarly if a text number value does not match the pattern, that is a processing error and will stop the parse. If neither case is the data broken, so why not allow these to be recoverable errors?
We undertook an exercise as part of another errata which listed out all the different kinds of failure that cause processing errors. I would like to go through the list, decide which failures are potentially recoverable, and introduce a way to designate such failures as being recoverable if they occur.
Replies (7)
RE: Expand use of recoverable exceptions - Added by Michael Beckerle about 9 years ago
Elements that are to be reported on when absent are simply not "required" fields in the sense of "required" in the DFDL specification. These fields are optional, in that the parse is supposed to succeed even if they are not present. They should be modeled as optional.
To get the warning/error about when they are missing, that's what assert with recoverable failure type is for. The test condition would simply be true if number of occurrences of the element is zero.
RE: Expand use of recoverable exceptions - Added by Steve Hanson about 9 years ago
You have cited just one example of the problem. There are many other failures for which we give processing errors but for which a standard like X12 wants to continue and gather more errors. The data is not broken syntactically, just the value is bad. Eg, a text number or text calendar doesn't match the pattern. This is a processing error and the parser stops. It should be possible to continue after this kind of data conversion error. (Analogy here with encodingErrorPolicy 'replace').
RE: Expand use of recoverable exceptions - Added by Steve Hanson about 9 years ago
I realise that with a data conversion error, the value can not be converted to a logical value compliant with the base type, which leaves a question as to what the infoset would contain. Perhaps we need to adopt the approach used by PSVI and XDM, and provide two values in the infoset - typed and raw ?
RE: Expand use of recoverable exceptions - Added by Michael Beckerle about 9 years ago
I'd like to see actual examples, because I cannot understand the issue.
Data conversion errors don't seem like an issue to me. The workaround is simple. If you want to keep/use the data, even after a data conversion error, you need to model the thing as a choice containing a date, not just as a date. This is simply the model being truthful about what the data format actually is, rather than the model expressing wishful-thinking about what we would like the format to be.
One choice alternative parses and converts, the other alternative is the "raw parse", which just captures the non-convertible data. Either is considered "well formed" as far as continuing the parse is concerned. You can hide the raw parse element in a hidden group if you really don't want it in the infoset. Asserts with recoverableError can report on the raw fields when they are created, if desired.
Is there an example from X12 that can't be worked around by modeling a field in the data as a choice?
The extra tiers of element that this may involve are a typical DFDL modeling issue, and there are other places in the spec where this is required that we haven't set out to fix in DFDL v1.0.
Many applications need to parse data that is out-of-compliance with the specification (document) describing the format, because the nature of those applications must be to provide diagnostic behavior. A schema that can do this has to be quite different from a schema that accepts only correctly formatted and valid data.
RE: Expand use of recoverable exceptions - Added by Steve Hanson about 9 years ago
What you suggest has certainly been the approach that we have discussed in the past. It's cumbersome and slows down the parser but I think it will work for X12. An alternative approach is to model everything as a string in the first place, and use a pattern facet to validate. It just seemed a neater solution to allow recoverable errors to handle it with a more natural schema.
Resolved - RE: Expand use of recoverable exceptions - Added by Michael Beckerle almost 9 years ago
No change to current behavior.
Closed: Expand use of recoverable exceptions - Added by Steve Hanson over 8 years ago
No update to experience documents or specification needed.
(1-7/7)