Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more benchmarks #7

Open
cgbystrom opened this issue May 23, 2016 · 1 comment
Open

Add more benchmarks #7

cgbystrom opened this issue May 23, 2016 · 1 comment

Comments

@cgbystrom
Copy link
Contributor

Right now the benchmarks look like they are all based on a simple fibonacci test, which I guess is a good start considering the number of runtimes being tested.

But just relying on this single type of benchmark is a bit one-sided and expanding the number of benchmarks is needed for a more accurate picture.

Perhaps we can list a few good candidates here. Two good sources I found are
https://github.com/kragen/shootout/tree/master/bench and https://github.com/attractivechaos/plb

Would be nice to pick something easy to implement but at the same time showing a different execution pattern than the fibonacci test.

@r-lyeh-archived
Copy link
Owner

r-lyeh-archived commented May 24, 2016

Great links thanks!

Yeah absolutely. The benchmark/project is more like a stub rather than a complete benchmark to be honest.

At least for me, I would like to bench/add:

  • string, list, vector, hash benchmarks { insert, delete, iteration }
  • script->native round trip
  • native->script round trip
  • memory consumption
  • memory de/allocation // GC times
  • warm up/spinning time
  • binary size overhead

Despite the good intentions, only the most common languages are production ready. Many of the languages listed in this repository lack objects, strings, memory functions, timers, collections/containers and so on. It is almost impossible to replicate the same tests for all of the languages without proper source modifications (heavy modifications in many cases).

Fibonacci on the other hand is almost portable and universal, and it is a good indicator to discard a few languages that are too slow (compared to the others) or those that do not provide tail/call optimization at all (for example, I wouldnt use them if I am coding a procedural game or city generator :). If there are other good universal tests that you might be aware of, I am open for them (simple hashing/encryption comes to mind).

Also:

  1. I plan to evaluate a fib(0) test and subtract the time from the total already, so we retrieve the warm up time for those languages that require some initialization.

  2. If there is some kind of idea on your side to do a fair scoring system (for those incomplete languages) I'm willing to collaborate and update the bechmarks/tests with new formulae.

  3. I am also fond of merging pull requests from anyone, so feel free to start experimenting asap :)

Thanks for the interest too :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants