-
-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support negative tests that expect errors (in MF2 tests) #242
Comments
Expected errors should, I think, be handled in the schema. This is what we do for example with likely subtags errors. In other words, the executor should return successfully when encountering an expected formatting error, and the test output should indicate in the schema somehow that an error had occurred. In MessageFormat specifically, this is probably desirable, because both the fallback string and the error can occur at the same time, and they can therefore both be tested. |
Right now we have verification data for MF2 that indicates "expErrors" with at least one parameter "type", e.g., label "00021". This tells me that executing this test is expected to return an error of the given type. {"label": "00021", "exp": "{|horse|}", "expErrors": [{"type": "bad-operand"}], "verify": "{|horse|}" } Here's what ICU4C returns for this test: ICU4J gives this output: I think that ICU4C should indicate "Pass" because an error was returned, but should be "Fail" in the Java version. Neither should be reported as a "Error". Note that the schema for the test output should allow optional "error" and "error_detail" fields. At this time, however, the schema for message format is apparently not being checked. |
I see 2 ways of thinking about "expected errors", in red and blue: We currently support only the red line: basically an unhandled exception, like a 500 server error. My proposal was that we consider adding a new "result status" called a handled exception, which would be like a 406 "Not Acceptable" error: the executor understands the request, but it encountered an error case that it knows about and knows how to handle. |
About half of the tests for MessageFormat 2 that are being sourced from the MFWG repo are negative tests -- they expect an error to occur. If we don't support that already, we should.
After ensuring that we can run such tests, then we should enable all of these tests in the ICU4J
message_fmt2
executor.The text was updated successfully, but these errors were encountered: