Esp32-A1S with ES8388

[ilker-aktuna]

I have an audio kit board with ES8388 and the version information says v2.2 A404
I want to use it with Arduino projects. However, I could not find which board type I should set to compile the code correctly for this board.

I installed both audio tools:
https://github.com/pschatzmann/arduino-audio-tools
and audio driver:
https://github.com/pschatzmann/arduino-audio-driver

But I did not install any board on the board manager.

The board is referred here:
https://github.com/pschatzmann/arduino-audiokit

However I sitll do not see which board I should select on Arduino IDE.

Can anyone guide me ?
thanks

[pschatzmann]

Just select the generic ESP32 Dev Module

[ilker-aktuna]

tried with dev module now but it goes into boot loop:

20:17:54.605 -> E (146) spi_flash: Detected size(4096k) smaller than the size in the binary image header(16384k). Probe failed.
20:17:54.637 ->
20:17:54.637 -> assert failed: do_core_init startup.c:328 (flash_ret == ESP_OK)
20:17:54.637 ->
20:17:54.637 ->
20:17:54.637 -> Backtrace: 0x40083b0d:0x3ffe3ab0 0x40089bb5:0x3ffe3ad0 0x4008ed0d:0x3ffe3af0 0x400e3dee:0x3ffe3c20 0x40083221:0x3ffe3c50 0x40079316:0x3ffe3c90 |<-CORRUPTED
20:17:54.637 ->
20:17:54.637 ->
20:17:54.637 ->
20:17:54.637 ->
20:17:54.637 -> ELF file SHA256: 12e615975326f63f

[pschatzmann]

Did you mess with the Flash size setting ?
Just try first with a simple hello world sketch....

[ilker-aktuna]

hello world also does not work on dev module.
I guess I've changed the flash settings for this device several months ago when I was working on a zigbee device.
Now what should be the default flash settings ?

[pschatzmann]

Flash size should be 4mb
I usually recommend to use set Scheme to Huge App
and to activate the core debug level to info

[ilker-aktuna]

ok. setting the flash size to 4mb solved it. No loops now.
With which example can I test playing sound ?

[ilker-aktuna]

I tried the following code:

AudioBoard board(AudioDriverES8388);

AudioInfo info(44100, 2, 16);
SineWaveGenerator<int16_t> sineWave(32000);
GeneratedSoundStream<int16_t> sound(sineWave);
I2SCodecStream out(AudioKitEs8388V1);
StreamCopy copier(out, sound);

    AudioLogger::instance().begin(Serial, AudioLogger::Warning);
  LOGLEVEL_AUDIODRIVER = AudioDriverWarning;

  // start I2S & codec with i2c and i2s configured above
  Serial.println("starting I2S...");
  auto config = out.defaultConfig();
  config.copyFrom(info);
  out.begin(config);

  // Setup sine wave
  sineWave.begin(info, N_B4);

but it fails:

10:38:55.118 -> Error: ->p_wire->endTransmission: 2
10:38:55.118 -> Error: ->p_wire->endTransmission: 2
10:38:55.118 -> Error: ->p_wire->endTransmission: 2
10:38:55.118 -> Error: ->p_wire->endTransmission: 2
10:38:55.118 -> Error: ->p_wire->endTransmission: 2
10:38:55.157 -> Error: ->p_wire->endTransmission: 2
10:38:55.157 -> Error: ->p_wire->endTransmission: 2
10:38:55.157 -> Error: ->p_wire->endTransmission: 2
10:38:55.157 -> Error: ->p_wire->endTransmission: 2
10:38:55.157 -> Error: ->p_wire->endTransmission: 2
10:38:55.157 -> Error: ->p_wire->endTransmission: 2
10:38:55.157 -> [E] I2SCodecStream.h : 209 - bool audio_tools::I2SCodecStream::begin1()

[pschatzmann]

Use this sketch to test and replace the driver with one of vaild alternatives.

If you have an AudioKit: its AudioKitEs8388V1, AudioKitEs8388V2 or AudioKitAC101

[ilker-aktuna]

great !
it works with AudioKitEs8388V1
So now that we know what the board is, can I somehow use it with IDF-ADF ?
In ADF menuconfig I can't find this board.
And when I follow the guide here:
https://github.com/Ai-Thinker-Open/ESP32-A1S-AudioKit

I modify the driver as described. I only get a garbage sound.
How can I make this board work with ADF ?
any ideas ?

[pschatzmann]

I suggest to use the AudioTools project instead: You have to write much more code in ADF to achieve the same thing!

If you want to use ADF you can find further information here.

[ilker-aktuna]

If you want to use ADF you can find further information here.

I followed that but could not get any sound from my board.
I used the Lyrat 4.3 as base and did:
"step 1 : alter the gpio connection ，path： esp-adf/components/audio_board/lyrat_v4_3/board_pins_config.c"

anything I might be missing ?

[pschatzmann]

Just compare if the gpio are right:
Here is the pin def from this project

class PinsAudioKitEs8388v1Class : public DriverPins {
 public:
  PinsAudioKitEs8388v1Class() {
    // sd pins
    addSPI(ESP32PinsSD);
    // add i2c codec pins: scl, sda, port, frequency
    addI2C(PinFunction::CODEC, 32, 33);
    // add i2s pins: mclk, bck, ws,data_out, data_in ,(port)
    addI2S(PinFunction::CODEC, 0, 27, 25, 26, 35);

    // add other pins
    addPin(PinFunction::KEY, 36, PinLogic::InputActiveLow, 1);
    addPin(PinFunction::KEY, 13, PinLogic::InputActiveLow, 2);
    addPin(PinFunction::KEY, 19, PinLogic::InputActiveLow, 3);
    addPin(PinFunction::KEY, 23, PinLogic::InputActiveLow, 4);
    addPin(PinFunction::KEY, 18, PinLogic::InputActiveLow, 5);
    addPin(PinFunction::KEY, 5, PinLogic::InputActiveLow, 6);
    addPin(PinFunction::AUXIN_DETECT, 12, PinLogic::InputActiveLow);
    addPin(PinFunction::HEADPHONE_DETECT, 39, PinLogic::InputActiveLow);
    addPin(PinFunction::PA, 21, PinLogic::Output);
    addPin(PinFunction::LED, 22, PinLogic::Output);
  }
};

[ilker-aktuna]

thanks but I really need parallel processing and efficient use of the esp32.
isn't ADF better for that ?

[pschatzmann]

Not really: here is an the example API with 3 lines of code or you can use the FreeRTOS C API ...

[ilker-aktuna]

ok. I'll try to give Arduino a chance. But I could not find an example of recording through the mic.
Is there any here :
https://github.com/pschatzmann/arduino-audio-tools/tree/main/examples/examples-audiokit
or anywhere else ?

[ilker-aktuna]

found a recording example here:
https://github.com/pschatzmann/arduino-audio-tools/blob/main/examples/examples-audiokit/streams-audiokit-ram-audiokit/streams-audiokit-ram-audiokit.ino

where can I find documentation of all methods in the API (driver and tools)

[pschatzmann]

You can also directly copy from the source to the destination as in this example.

The Readme of the AudioTools contains links to all the documentation!

[ilker-aktuna]

thank you
is there an example for writing data from microphone to a file (on SD perhaps) ?

[pschatzmann]

Not sure, just search in the examples: the names are quite logical.
In any case this is quite trivial: you can just use a file as copy target!

Final hint: have a look at the documentation about codecs since you usually want to write in a defined audio format!

[ilker-aktuna]

I found an example to write to a file on SD. I was able to write and read. But when I open the file with FILE_WRITE attribute, it can not overwrite the previous file. Is there a way to overwrite ?

Also, all encoding examples use streams. How can I copy from file to file encoding with a codec ?

[pschatzmann]

Close and delete the file or just move the file position to the beginning.
I dont understand your question about files and streams: a file is a stream!

[ilker-aktuna]

I close the file and reopen and move to the beginning with this sequence:

file.close();
file = SD.open("/audio.raw", FILE_WRITE);
file.seek(0); 

but it can not overwrite.
So deleting is the only option ?

About encoding question; I tried the following code to encode but file size was 0 :

File file;
File audioFile; 
WAVEncoder encoder;
EncodedAudioStream out_stream(&audioFile, &encoder);             // encode as wav file
StreamCopy copier1(out_stream, file);         


  uint16_t sample_rate = 44100;
  uint8_t channels = 2; 
  AudioInfo cfg;
  cfg.sample_rate = sample_rate;
  cfg.channels = channels;
  cfg.bits_per_sample = 16;
  out_stream.begin(cfg);
  audioFile = SD.open("/002.wav", FILE_WRITE);
  copier1.copy();
  audioFile.close();

what am I missing ?

[pschatzmann]

Just make a simple test case: the overwrite should work. Most likely you can even get rid of the close and repoen!
The second part is too confusing to merit an answer!

[ilker-aktuna]

ok. I solved the issue about overwriting.

Now I am trying to do the encoding.

is there a way to understand when copier has finished copying ? (input file end)

[ilker-aktuna]

@pschatzmann is this not possible ?

[pschatzmann]

Just check the result: copy returns the number of copied bytes!

[ilker-aktuna]

ok. done that and now I am able to close the file at correct time.
Now I have an issue decoding the mp3 file.
I see 2 codecs (helix and mad) are available and I installed both.
However, when I try to use them I get errors like:

arduinoIDE-unsaved2024821-552-clg4n9.5ijxg\sketch_sep21a\sketch_sep21a.ino:36:1: error: 'MP3DecoderHelix' does not name a type; did you mean 'MP3Decode'?
 MP3DecoderHelix decoder;

I have added the libraries and included header files:

#include <CommonHelix.h>
#include <ConfigHelix.h>
#include <MP3DecoderHelix.h>

[pschatzmann]

Use the VolumeMeter class

[ilker-aktuna]

Thanks. I could just find one example using it. And that example uses with a StreamCopy. So in that case, I will use 2 copiers , one will copy to file and the other to the volumemeter. Is that the correct use ?

[ilker-aktuna]

The audiotools project was getting too big, so I had to move things out. They can be found here

I am not sure if tese things are still working. There have been some major changes in TF since I did this....

Because of some health issues, I had to give a long break to this project. Now I'm back, and I'm still trying to use Tensorflow with one word detection. Unfortunately I could not find any easy way to create a simple model. I tried external online tools and their output does not work with this library. I'm stuck with using the docker image you've supplied.
There is no prebuilt image so I have to build it. I tried on several different Linux installations including a plain Ubuntu installation , a VM environment and WSL2 on windows.
Unfortunately build is not successful on any of my attempts.

I get this error:

Step 3/13 : RUN /usr/bin/python3 -m pip install --upgrade pip
---> Running in 31ffe5639302
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
ERROR: Service 'tensorflow' failed to build: The command '/bin/sh -c /usr/bin/python3 -m pip install --upgrade pip' returned a non-zero code: 1

Do you have a prebuilt image somewhere ?
Or any other method to use the tool ? (maybe without docker)

[pschatzmann]

I suggest that you have a look at Edge Impule.
They make the machine learning and training part quite easy

[ilker-aktuna]

I suggest that you have a look at Edge Impule. They make the machine learning and training part quite easy

yes I have already tried it. But the output file does not work with this library. Am I missing something ? Any settings that you would recommend ?

[pschatzmann]

In your case, just wrap the input object and read from the meter instead...

[ilker-aktuna]

It's not possible to provide an answer on this since I don't know what you want to do. But assuming you want to use Edge Impulse with a microphone input just check the generated code and search for the Nano 33 BLE Sense PDM.read(). That's the part you would need to replace.

I am not trying to to anything complex. I just want to use a simple wake word with your example code.
So the code is actually "your code":

#include "AudioTools.h"
#include "AudioLibs/AudioBoardStream.h"
#include "AudioLibs/TfLiteAudioStream.h"
#include "model.h"

TfLiteAudioStream tfl;  // Audio sink
const char* kCategoryLabels[4] = {
    "silence",
    "unknown",
    "yes",
    "no",
};

StreamCopy copier(tfl, kit); 
int channels = 1;
int samples_per_second = 16000;
AudioInfo info(16000, 1, 16);
AudioBoardStream kit(AudioKitEs8388V1);

void respondToCommand(const char* found_command, uint8_t score,
                      bool is_new_command) {
  if (is_new_command) {
    char buffer[80];
    sprintf(buffer, "Result: %s, score: %d, is_new: %s", found_command, score,
            is_new_command ? "true" : "false");
    Serial.println(buffer);
    if(String(found_command).equals("yes")) {
      action="null";
      record_start_from_voice();
    }
  }
}

void setup(){
  Serial.begin(115200);
     // Setup tensorflow output
  auto tcfg = tfl.defaultConfig();
  tcfg.setCategories(kCategoryLabels);
  tcfg.channels = channels;
  tcfg.sample_rate = samples_per_second;
  tcfg.kTensorArenaSize = 10 * 1024;
  tcfg.respondToCommand = respondToCommand;
  tcfg.model = g_model;
  tfl.begin(tcfg);
}

void loop(){
 copier.copy();
}

I just want to use a custom word instead of the "yes","no" options in your "model.h" data.
So I need a model.h with my custom word in it.
That's it. Nothing more.

So I don't need a code from EdgeImpulse. I just need a model.h file from them and I got it.
However it did not work with your code.
So what's the proper way to get a model.h file that will work with your code ?

[pschatzmann]

Tensorflow is a moving target and I did not update my library since 2 years!
So here is what you would need to do in this case:

• Merge the latest changes from the tensorflow library to get an up to date version. You migth run into conflics that you need to resolve yourself. Any pull request will be apreciated!
• Alternatively you can start with https://github.com/tensorflow/tflite-micro-arduino-examples and reimplement the audiotools integration
• For training the model I suggest that you have a look into this

I was proposeing Edge Impulse because you would not need to bother with these kind of issues....

[ilker-aktuna]

I'm not sure if I understand correct.
When I run your code with the model.h from your example, it works. It can detect when I say "yes"
I just want to do this with a different word instead of "yes"
So I just need a model.h file sampled with my word.
Why do I have to merge updated version of Tensorflow library in this case ?

And when you propose Edge Impulse, does that mean I leave your code aside and use the code from Edge Impulse ?
But I need to use your AudioTools and AudioBoardStream for the board that I have ?

[pschatzmann]

You can use an model generated with an previous version with a more recent tf code version: but you can not use a model generated from an actual version to be running on an old tf version!

I did this excercise for you and upgraded my tf project: I also adapted the code in the audio-tools to follow the new conventions.

It is compiling again w/o any problems, but I did not run any tests...

I considered to be using the built in tf-lite from the ESP32 Arduino core: but some critical functions are missing there.
I also considerted to use another tf-lite Arduino library, but I was running into a compile error because my file system is case sensitive and some includes were misspelled.

[ilker-aktuna]

I did this excercise for you and upgraded my tf project: I also adapted the code in the audio-tools to follow the new conventions.

thanks for keeping your support on this. But I did not understand what you did.
As far as I can see you changed all the Arduino code and now you have supporting cpp code.
The old project had one ino file and it was very simple interacting using the model.h
Now you have a very complex code and I don't understand how to interact detecting the spoken word.

Also, I did not understand: are you dropping support for Audio Kit ?
If not, how can I use this new example with Audio Kit

[pschatzmann]

I don't quite understand: it's still working like you find in the documentation.

The old AudioKit project is not supported any more, so please use the new functionality via the AudioBoardStream.

[ilker-aktuna]

I don't quite understand: it's still working like you find in the documentation.

The old AudioKit project is not supported any more, so please use the new functionality via the AudioBoardStream.

OK. It took some time since all main libraries (audio-driver and audio-tools) required update as well. After updating all, I tried the same example with a new model.h file from Edge Impulse.

crashes with:
15:03:01.160 -> [I] TfLiteAudioStream.h : 961 - virtual bool audio_tools::TfLiteAudioStream::setupInterpreter()
15:03:01.160 -> [I] TfLiteAudioStream.h : 836 - AllocateTensors
15:03:01.160 -> Didn't find op for builtin opcode 'CONV_2D' version '3'. An older version of this builtin might be supported. Are you using an old TFLite binary with a newer model?
15:03:01.160 ->

it still complains about model-binary compatibility.

[ilker-aktuna]

@pschatzmann do you have any comment on this ?