Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support negative tests that expect errors (in MF2 tests) #242

Open
echeran opened this issue Jun 23, 2024 · 3 comments
Open

Support negative tests that expect errors (in MF2 tests) #242

echeran opened this issue Jun 23, 2024 · 3 comments

Comments

@echeran
Copy link
Collaborator

echeran commented Jun 23, 2024

About half of the tests for MessageFormat 2 that are being sourced from the MFWG repo are negative tests -- they expect an error to occur. If we don't support that already, we should.

After ensuring that we can run such tests, then we should enable all of these tests in the ICU4J message_fmt2 executor.

@sffc
Copy link
Member

sffc commented Jun 24, 2024

Expected errors should, I think, be handled in the schema. This is what we do for example with likely subtags errors. In other words, the executor should return successfully when encountering an expected formatting error, and the test output should indicate in the schema somehow that an error had occurred.

In MessageFormat specifically, this is probably desirable, because both the fallback string and the error can occur at the same time, and they can therefore both be tested.

@sven-oly
Copy link
Collaborator

Right now we have verification data for MF2 that indicates "expErrors" with at least one parameter "type", e.g., label "00021". This tells me that executing this test is expected to return an error of the given type.

{"label": "00021", "exp": "{|horse|}", "expErrors": [{"type": "bad-operand"}], "verify": "{|horse|}" }

Here's what ICU4C returns for this test:
{"label": "00021", "error": "formatToString", "error_detail": "U_MF_OPERAND_MISMATCH_ERROR"},
Note that there's no key "result".

ICU4J gives this output:
{"label": "00021", "result": "{|horse|}"}
Note that there is a key "result", but no "error" and no "error_detail" keys.

I think that ICU4C should indicate "Pass" because an error was returned, but should be "Fail" in the Java version. Neither should be reported as a "Error".

Note that the schema for the test output should allow optional "error" and "error_detail" fields. At this time, however, the schema for message format is apparently not being checked.

@sffc
Copy link
Member

sffc commented Sep 24, 2024

I see 2 ways of thinking about "expected errors", in red and blue:

image

We currently support only the red line: basically an unhandled exception, like a 500 server error.

My proposal was that we consider adding a new "result status" called a handled exception, which would be like a 406 "Not Acceptable" error: the executor understands the request, but it encountered an error case that it knows about and knows how to handle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants