Add OFV market resolver

[kongzii]

Proposed changes

Implements a new tool that can be used for market resolution.

Types of changes

What types of changes does your code introduce? (A breaking change is a fix or feature that would cause existing functionality and APIs to not work as expected.)
Put an x in the box that applies

• Non-breaking fix (non-breaking change which fixes an issue)
• Breaking fix (breaking change which fixes an issue)
• Non-breaking feature (non-breaking change which adds functionality)
• Breaking feature (breaking change which adds functionality)
• Refactor (non-breaking change which changes implementation)
• Messy (mixture of the above - requires explanation!)

Checklist

Put an x in the boxes that apply.

• I have read the CONTRIBUTING doc
• I am making a pull request against the main branch (left side). Also you should start your branch off our main.
• Lint and unit tests pass locally with my changes
  • mypy found 54 errors, but in files that aren't created or modified by me
• I have added tests that prove my fix is effective or that my feature works

Further comments

I used a customised fork of OpenFactVerifier (PR is currently open again their main branch) to create a new market resolver.

The steps are:

1. Verify that the given market is reasonable to resolve using the is_predictable_binary function.
2. Rewrite the market's question into a statement. For example, Will former Trump Organization CFO Allen Weisselberg be sentenced to jail by 15 April 2024? would be rewritten to Former Trump Organization CFO Allen Weisselberg was sentenced to jail by 15 April 2024..
3. Use OpenFactVerifier to verify the given statement. It will be either True, False or None if there is some kind of problem.

I also implemented a benchmark where I compare (1) current resolution visible on Omen, (2) resolution obtained by packages.napthaai.customs.resolve_market_reasoning.resolve_market_reasoning by running it today and (3) resolution by OFV. The results are:

1. Current accuracy: 73.47%
2. Original's new run accuracy: 57.14
3. OFV's accuracy: 91.84%

However, I don't understand why (2) is so low, I expected it to be at least a little better than (1), because new information is available on internet. If you can point out a bug in my implementation, I'd re-run the benchmark again.

The benchmark is run over 50 markets that I resolved manually and the full results are available here.

I will pick just the ones where OFV made mistakes:

• Will Apple introduce spatial computing in Mac's screen by 14th April 2024?
  • This is questionable, should Vision Pro count as it? I guess nobody would really expect that Macbooks would get spatial computing by themselves by April 2024, but Vision Pro allows to interact with macs.
• Will McDonald's successfully buy back all its Israeli restaurants by 12 April 2024?
  • I found articles that it will happen, but no article that it already did happen. I found an article from April 5 saying they signed the deal. Bu that shouldn't count as market was created on April 8.
• Will Samsung release the Galaxy Z Fold 6 Ultra on or before 11 April 2024?
• Will Disney Plus implement its password-sharing crackdown by 11 April 2024?
  • It's questionable if annoucment should make this YES, or the fact that it will be used from June should make it NO.

As we can see, 3 out of 4 mistakes are questionable.

[kongzii]

What's the plan for the dependency updates in related repositories, please?

And I know Evan was working on some dependency-related stuff as well, I'll get in touch with him about the current status.

[0xArdi]

What's the issue you're facing? Did you have to make adjustments to open-autonomy? If so, feel free to make a PR on the main open-autonomy repo, we can take a look.

[kongzii]

Yes, PR for open-autonomy is open: valory-xyz/open-autonomy#2223
And for tomte too: valory-xyz/tomte#27

The issue we are facing is that with current strict versioning, it's almost impossible to use it with other libraries.

[0xArdi]

If we were to merge this, it means that we need to keep using your fork until we reflect your changes on our framework.

[kongzii]

Then maybe you could merge and release changes in these two PRs so I can use them properly here?

[kongzii]

@richardblythman Could you please explain why this pattern of doing def run(**kwargs) is necessary? I saw it in other tools as well but I didn't understand, because from the caller's perspective, having

def run(**kwargs): ...

or having

def run(
  prompt: str,
  api_keys: dict[str, str],
): ...

is called in the same way by doing run(prompt=..., api_keys=...). But in the second way it cleaner what arguments the function expects and what types.

[richardblythman]

def run(
  prompt: str,
  api_keys: dict[str, str],
): ...

@kongzii I think this would have worked, but it's not what you had originally.

[kongzii]

I think this would have worked, but it's not what you had originally.

@richardblythman Yeah, I see the changes in arguments, that's clear to me that it needs to have the same interface as other tools to make it simpler for calling from agents/mech. I just didn't understand why the **kwargs.

[richardblythman]

@kongzii I just copied and pasted. Not sure if the team at Valory prefer one over the other.