Matching 2 vectors and finding indices

[meftunca]

In a scenario where I have 2 vectors, let's assume that vector 1 is a 16 character data source and vector 2 is a syntax tokens, and if the elements in vector 1 are found in vector 2, can I get both its index value and the index value of the element in vector 2 that it matches quickly with eve?

I'm sorry, this question is very difficult for me to explain :D I get support from Translator unfortunately

Basic Example

#include <iostream>
#include <vector>
#include <string>

using namespace std;

struct Token {
  int index; // Eşleşen karakterin indeksi
  char syntax_char; // Eşleşen JSON syntax karakteri
};

int main() {
  // JSON string
  std::string json_string = "{\"key1\":\"value1\",\"key2\":\"value2\"}";

  // JSON özel karakterleri içeren vektör
  std::vector<char> syntax_characters = {'{', '}', ':', '[', ']', ',', '"', '\\'};

  // Ana tokenizer vector
  std::vector<Token> tokenizer_vector;

  // String'i analiz et
  for (size_t i = 0; i < json_string.size(); i++) {
    bool found = false;

    // Karakteri syntax karakterleri ile karşılaştır
    for (char syntax_char : syntax_characters) {
      if (json_string[i] == syntax_char) {
        found = true;
        break;
      }
    }

    // Karakter syntax karakteri ise token oluştur
    if (found) {
      Token token;
      token.index = i;
      token.syntax_char = json_string[i];

      // Token'ı ana tokenizer vector'e ekle
      tokenizer_vector.push_back(token);
    }
  }

  // Ana tokenizer vector'ü yazdır
  std::cout << "Ana Tokenizer Vector:\n";
  for (const Token& token : tokenizer_vector) {
    std::cout << "Index: " << token.index << ", Karakter: " << token.syntax_char << std::endl;
  }

  return 0;
}

[DenisYaroshevskiy]

I'm not quite sure what you mean, but I believe you want to try to lex JSON.

I looked at the split problem at one point for folly::split - there the best approach was SIMD scalar, because the distances between splits were noticeable.

I suspect the same might be mostly in your case.

1. How to test for an interesting char.

The function you asked about is has_equal_in but that's not what you want because it's too heavy.
(doc: https://jfalcou.github.io/eve/group__core__simd_ga64cba22849ce7362968aa0c2661a5ab0.html#ga64cba22849ce7362968aa0c2661a5ab0)

What you want is is_one_of and we don't have that one yet.
How you'd implement it here is tricky, I'd start with:

x == '[' || x ==']' || ...

but you will probably push into the number of registers. You can do some clever "store one char in this half, one char in that half" and what not too.

There is also a trick described here #1052 but I don't know how easy it'd be for you to do.

2. How to split by predicate for you

Didn't have time to finish, will look later.
Here is a in progress thingy https://godbolt.org/z/oznxa4o44

You can also look at folly::split here https://github.com/facebook/folly/blob/786435b3c442cd24837e6cf3b26a0c71d1d1e77f/folly/detail/SplitStringSimdImpl.h#L119

[DenisYaroshevskiy]

Actually - if you can wait for the weekend and ok to update - I can put something in the lirary

[DenisYaroshevskiy]

How are you consuming the library?

[meftunca]

I use the library for machine learning and it has really sped up my work a lot. I am now actively trying to use it to process my data faster (think csv, json or yaml).

[DenisYaroshevskiy]

I meant how are you getting the library? Got clone or are you using a package manager?

[DenisYaroshevskiy]

Did you look at simdjson? It would be much better for parsing json than anything you do

[meftunca]

If you mean the “eve” library, I get it via git with bazel. If you mean my own library, I haven't actively published it on github yet. Simdjson falls short for me. I can only use it for reading operations.

Obviously the “json” format is not a sufficient format for me. I am working on a format similar to the following example format

https://github.com/stephenberry/beve

[DenisYaroshevskiy]

FYI: i created a pr with some functionality you need.
#1797

Will let you know when it's merged

[meftunca]

Thank you very much, my friend.

[DenisYaroshevskiy]

merged, tomorrow should be on godbolt, I will write you an example then

[DenisYaroshevskiy]

This is the new added algorithm

https://jfalcou.github.io/eve/group__algos_gac600981328935ff4d20e2b317b647168.html#gac600981328935ff4d20e2b317b647168

Here is that example on gobolt but refreshed, so it works:
https://godbolt.org/z/roE6qsW9e

With it you can iterate through separators and decide what to do with them

[meftunca]

Thank you very much, I will look into it and get back to you.