Please have a new run!

[aggsol]

This is great.

[r-lyeh-archived]

I will probably update all code bases & give it a new run this summer. I'm busy until then! :/

[asbai]

I am also looking forward to the new results.

[asbai]

I recommend using gcc (MinGW for windows) to compile angelscript. Because the standard string add-on of angelscript use std::string by default. And the performance of MSVC std::string is low (just allocate and copy, no reference counting and copy-on-write support. It's a big hurt for something like a benchmark).

I guess in most serious real cases where performance is a matter, almost all programmers will implement them own string type by hand. For example, we can even eliminate the memory allocation and memory copy overhead of the String Factory by using the static data semantic (see: https://www.gamedev.net/topic/685672-is-it-safe-to-use-the-char-argument-of-string-factory-as-static-data/). So in real case the performance can be further improved.

The purpose of the test is to reflect the true level of the engine, So I think it is necessary to adjust the engine to its real world status.

[asbai]

Also, I suspect that register a line callback could dramatically slow down the engine. Use another timer thread which call ctx->Abort() on timeout instead.

[rosshadden]

I feel like there is a way this project can be architected such that these tests run on a VM and can be contributed by and perhaps even run by the community. It would be work up front, but in the long run would both mitigate a lot of work that @r-lyeh has to do, and let us keep it updated. Maybe we should brainstorm about ways to do that that aren't too difficult.

[aggsol]

@rosshadden Like continuos integration? Whenever a library or test case is changed then run it again.

[rosshadden]

Yeah.

[iSLC]

Probably Off Topic:

@asbai AFAIK, MinGW/GCC don't use reference counting and copy-on-write design for std::string. What they're using is a small-buffer optimization. Meaning, if your string is less than 16 characters, then it'll be stored in a local buffer and no heap allocation is performed. This makes the string type size be 8 bytes bigger than it would normally be and require some extra work (like when doing move semantics). But it pays off in the end because you avoid heap allocation for small strings and you never have to check for null buffer because there's always a buffer present. Also, why only 8 extra bytes when the local buffer is 16 bytes? (on 64 bit at least) It uses a union for the local buffer and capacity since the capacity is always known for the local buffer. Kinda like:

struct String {
    static constexpr size_t MINCAP = 16;

    char * m_Buffer;
    size_t m_Occupied;

    union {
        char    m_LocalBuffer[MINCAP];
        size_t  m_Capacity;
    };

    String() noexcept
        : m_Buffer(m_LocalBuffer), m_Occupied(0)
    {
        m_LocalBuffer[0] = '\0';
    }
};

If you're so hell bent on avoiding string copies where not necessary. I'd suggest investing some time in studying string/array views. Which were even added in the next C++17 standard.

[r-lyeh-archived]

Good ideas overall.

• I've used CI a lot in most of my other repos and it's a valuable tool. Definitely @todo.
• The VM idea sounds exciting and I am open to suggestions, what are the current options?
• Current fibonacci test is somewhat lame, but very portable across all the implementations, plus it is handy to discard languages that lacks CC/tail optimization (also, bad for procedural stuff). Gotta add more computational/numeric tests. PRs welcome :)
• Angelscript test (any test in fact!) should not use strings at all. If it does (like printing the fibonacci number), then it is my own mistake. Gotta review soon. No time right now :)

[asbai]

@iSLC No, you are wrong :-)
At least in my gcc 4.4.7 and 4.5.2, they all use reference count for std::basic_string, you can check it at "bits/basic_string.h" in gcc's "include/c++" directory:

template<typename _CharT, typename _Traits, typename _Alloc>
  class basic_string
   {
      // Types:
    public:
      // ...
    private:
      struct _Rep_base
      {
	size_type    _M_length;
	size_type    _M_capacity;
	_Atomic_word _M_refcount; // <---- here
      };
       // ...

In addition, I did not use the GCC string class, I use my own string class with reference counting and copy-on-write support.

PS: The algorithm you described is very similar to what is used in VC++ STL, not GCC :-)

[asbai]

@r-lyeh Sorry, I can not found where the test scripts were stored, I guess you have some string performance test like this: https://codeplea.com/game-scripting-languages.

So which algorithms we are using now?

UPDATE: I found it under the "tests" folder. :-)
It looks like the native2 test passed a string argument repeatedly? String performance may have a significant impact on this.

[iSLC]

@asbai I'm talking about GCC 5.x, 6.x and future versions.

[asbai]

@iSLC Ok, so it's clearly that gcc 4.x is more benchmark friendly :-)

Especially for engines like AngelScript which has the ability to avoid the memory allocation and copy cost of the script / native calling conversion.

The memory allocation and copy cost of the native call is one of the biggest performance killer of dynamic languages.

[r-lyeh-archived]

@asbai, only fib.as is currently benchmarked. the native* and prime* tests are part of the original article you linked and are yet to be converted to the rest of languages before evaluation. such a difficult/boring task! :)

ps: the fib.as script is printing the result as I originally thought. this issue needs some global review before running the whole bench to be more fair at results.

[asbai]

@r-lyeh Ok, I think the ability of fast native calling conversion will not cause unfairness, because it is a feature of the language and the engine. Just like JIT is a feature of some engine.

But a line callback is unfair unless you have been registered this call back for every other engines. :-)

PS: The author of AngelScript suggest us to use:
engine->SetEngineProperty(asEP_BUILD_WITHOUT_LINE_CUES, true);
to disable the line cues and not register any line callback under release (daily use production) mode.

[r-lyeh-archived]

Aha thanks, will do in the next run :)