Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ESP-IDF CMakeLists #800

Merged
merged 13 commits into from
Nov 1, 2024
Merged

ESP-IDF CMakeLists #800

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
12870e9
CMake build.
bakerstu Oct 20, 2024
f752bb1
Fix file extension.
bakerstu Oct 20, 2024
a8287d4
Cleanup CMakeLists.
bakerstu Oct 20, 2024
a21ec31
Fix compiler warning (redundant move in return statement).
bakerstu Oct 20, 2024
9a3209e
Add readme instructions.
bakerstu Oct 20, 2024
2d0d242
Remove extra newline.
bakerstu Oct 20, 2024
508047f
Cleanup target compile options with comments.
bakerstu Oct 20, 2024
549d68f
Merge branch 'master' into bakerstu-esp-idf-cmake
bakerstu Oct 27, 2024
3557d7d
Add BLE files to the build.
bakerstu Oct 27, 2024
8373152
Apply review comments.
bakerstu Oct 27, 2024
0659958
Add idf_component.yml file.
bakerstu Oct 27, 2024
1129d56
Remove "root" marker from the file system path.
bakerstu Oct 27, 2024
d100f88
Remove idf_component.yml
bakerstu Nov 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 166 additions & 0 deletions etc/esp-idf/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
file(REAL_PATH ${COMPONENT_DIR} OPENMRNESPIDFPATH)
set(OPENMRNPATH ${OPENMRNESPIDFPATH}/../..)

set(SRCS
${OPENMRNPATH}/src/ble/Advertisement.cxx
${OPENMRNPATH}/src/ble/Defs.cxx

${OPENMRNPATH}/src/dcc/dcc_constants.cxx
${OPENMRNPATH}/src/dcc/DccDebug.cxx
${OPENMRNPATH}/src/dcc/Defs.cxx
${OPENMRNPATH}/src/dcc/LocalTrackIf.cxx
${OPENMRNPATH}/src/dcc/Loco.cxx
${OPENMRNPATH}/src/dcc/Packet.cxx
${OPENMRNPATH}/src/dcc/RailcomBroadcastDecoder.cxx
${OPENMRNPATH}/src/dcc/RailCom.cxx
${OPENMRNPATH}/src/dcc/RailcomDebug.cxx
${OPENMRNPATH}/src/dcc/SimpleUpdateLoop.cxx
${OPENMRNPATH}/src/dcc/UpdateLoop.cxx

${OPENMRNPATH}/src/executor/AsyncNotifiableBlock.cxx
${OPENMRNPATH}/src/executor/Executor.cxx
${OPENMRNPATH}/src/executor/Notifiable.cxx
${OPENMRNPATH}/src/executor/Service.cxx
${OPENMRNPATH}/src/executor/StateFlow.cxx
${OPENMRNPATH}/src/executor/Timer.cxx

${OPENMRNPATH}/src/freertos_drivers/esp32/Esp32HardwareI2C.cxx
${OPENMRNPATH}/src/freertos_drivers/esp32/Esp32HardwareTwai.cxx
${OPENMRNPATH}/src/freertos_drivers/esp32/Esp32Ledc.cxx
${OPENMRNPATH}/src/freertos_drivers/esp32/Esp32SocInfo.cxx
${OPENMRNPATH}/src/freertos_drivers/esp32/Esp32WiFiManager.cxx

${OPENMRNPATH}/src/openlcb/AliasAllocator.cxx
${OPENMRNPATH}/src/openlcb/AliasCache.cxx
${OPENMRNPATH}/src/openlcb/BLEAdvertisement.cxx
${OPENMRNPATH}/src/openlcb/BLEService.cxx
${OPENMRNPATH}/src/openlcb/BroadcastTime.cxx
${OPENMRNPATH}/src/openlcb/BroadcastTimeClient.cxx
${OPENMRNPATH}/src/openlcb/BroadcastTimeDefs.cxx
${OPENMRNPATH}/src/openlcb/BroadcastTimeServer.cxx
${OPENMRNPATH}/src/openlcb/BulkAliasAllocator.cxx
${OPENMRNPATH}/src/openlcb/CanDefs.cxx
${OPENMRNPATH}/src/openlcb/ConfigEntry.cxx
${OPENMRNPATH}/src/openlcb/ConfigUpdateFlow.cxx
${OPENMRNPATH}/src/openlcb/Datagram.cxx
${OPENMRNPATH}/src/openlcb/DatagramCan.cxx
${OPENMRNPATH}/src/openlcb/DatagramTcp.cxx
${OPENMRNPATH}/src/openlcb/DccAccyProducer.cxx
${OPENMRNPATH}/src/openlcb/DefaultNode.cxx
${OPENMRNPATH}/src/openlcb/DefaultCdi.cxx
${OPENMRNPATH}/src/openlcb/EventHandler.cxx
${OPENMRNPATH}/src/openlcb/EventHandlerContainer.cxx
${OPENMRNPATH}/src/openlcb/EventHandlerTemplates.cxx
${OPENMRNPATH}/src/openlcb/EventService.cxx
${OPENMRNPATH}/src/openlcb/If.cxx
${OPENMRNPATH}/src/openlcb/IfCan.cxx
${OPENMRNPATH}/src/openlcb/IfImpl.cxx
${OPENMRNPATH}/src/openlcb/IfTcp.cxx
${OPENMRNPATH}/src/openlcb/MemoryConfig.cxx
${OPENMRNPATH}/src/openlcb/nmranet_constants.cxx
${OPENMRNPATH}/src/openlcb/Node.cxx
${OPENMRNPATH}/src/openlcb/NodeBrowser.cxx
${OPENMRNPATH}/src/openlcb/NodeInitializeFlow.cxx
${OPENMRNPATH}/src/openlcb/NonAuthoritativeEventProducer.cxx
${OPENMRNPATH}/src/openlcb/PIPClient.cxx
${OPENMRNPATH}/src/openlcb/RoutingLogic.cxx
${OPENMRNPATH}/src/openlcb/SimpleNodeInfo.cxx
${OPENMRNPATH}/src/openlcb/SimpleNodeInfoMockUserFile.cxx
${OPENMRNPATH}/src/openlcb/SimpleNodeInfoResponse.cxx
${OPENMRNPATH}/src/openlcb/SimpleStack.cxx
${OPENMRNPATH}/src/openlcb/StreamReceiver.cxx
${OPENMRNPATH}/src/openlcb/StreamTransport.cxx
${OPENMRNPATH}/src/openlcb/TcpDefs.cxx
${OPENMRNPATH}/src/openlcb/TractionCvSpace.cxx
${OPENMRNPATH}/src/openlcb/TractionDefs.cxx
${OPENMRNPATH}/src/openlcb/TractionProxy.cxx
${OPENMRNPATH}/src/openlcb/TractionTestTrain.cxx
${OPENMRNPATH}/src/openlcb/TractionThrottle.cxx
${OPENMRNPATH}/src/openlcb/TractionTrain.cxx
${OPENMRNPATH}/src/openlcb/Velocity.cxx
${OPENMRNPATH}/src/openlcb/WriteHelper.cxx


${OPENMRNPATH}/src/os/FakeClock.cxx
${OPENMRNPATH}/src/os/logging_malloc.cxx
${OPENMRNPATH}/src/os/MDNS.cxx
${OPENMRNPATH}/src/os/os.c
${OPENMRNPATH}/src/os/OSImpl.cxx
${OPENMRNPATH}/src/os/OSSelectWakeup.cxx
${OPENMRNPATH}/src/os/stack_malloc.c
${OPENMRNPATH}/src/os/TempFile.cxx
${OPENMRNPATH}/src/os/watchdog.c

${OPENMRNPATH}/src/utils/Base64.cxx
${OPENMRNPATH}/src/utils/Blinker.cxx
${OPENMRNPATH}/src/utils/Buffer.cxx
${OPENMRNPATH}/src/utils/CanIf.cxx
${OPENMRNPATH}/src/utils/ClientConnection.cxx
${OPENMRNPATH}/src/utils/ConfigUpdateListener.cxx
${OPENMRNPATH}/src/utils/constants.cxx
${OPENMRNPATH}/src/utils/Crc.cxx
${OPENMRNPATH}/src/utils/DirectHub.cxx
${OPENMRNPATH}/src/utils/DirectHubGc.cxx
${OPENMRNPATH}/src/utils/DirectHubLegacy.cxx
${OPENMRNPATH}/src/utils/errno_exit.c
${OPENMRNPATH}/src/utils/FdUtils.cxx
${OPENMRNPATH}/src/utils/FileUtils.cxx
${OPENMRNPATH}/src/utils/format_utils.cxx
${OPENMRNPATH}/src/utils/ForwardAllocator.cxx
${OPENMRNPATH}/src/utils/GcStreamParser.cxx
${OPENMRNPATH}/src/utils/GcTcpHub.cxx
${OPENMRNPATH}/src/utils/gc_format.cxx
${OPENMRNPATH}/src/utils/GridConnect.cxx
${OPENMRNPATH}/src/utils/GridConnectHub.cxx
${OPENMRNPATH}/src/utils/HubDevice.cxx
${OPENMRNPATH}/src/utils/HubDeviceSelect.cxx
${OPENMRNPATH}/src/utils/ieeehalfprecision.c
${OPENMRNPATH}/src/utils/JSHubPort.cxx
${OPENMRNPATH}/src/utils/logging.cxx
${OPENMRNPATH}/src/utils/Queue.cxx
${OPENMRNPATH}/src/utils/ReflashBootloader.cxx
${OPENMRNPATH}/src/utils/ServiceLocator.cxx
${OPENMRNPATH}/src/utils/SocketCan.cxx
${OPENMRNPATH}/src/utils/SocketClient.cxx
${OPENMRNPATH}/src/utils/socket_listener.cxx
${OPENMRNPATH}/src/utils/Stats.cxx
${OPENMRNPATH}/src/utils/StringPrintf.cxx
)

set(INCLUDE_DIRS
${OPENMRNPATH}/src
${OPENMRNPATH}/include/esp-idf
${OPENMRNPATH}/include
)

set(IDF_DEPS
app_update
bootloader_support
bt
driver
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to put this on my update list for ESP-IDF v5.3 support updates (hopefully a relatively small update), this driver component was split up and can be replaced with:

  • esp_driver_gpio
  • esp_driver_i2c

but there is currently a compilation failure in Esp32HardwareI2C.hxx due to a dependency on the older driver format. I'll take care of migrating to the newer driver code and cleanup of this specific line as optional based on ESP-IDF version (pre-5.3 it will include driver and current Esp32HardwareI2C code, 5.3+ it will depend on the newer I2C driver).

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted.

esp_adc
esp_app_format
esp_system
esp_wifi
espcoredump
hal
heap
mdns
pthread
vfs)

idf_component_register(SRCS "${SRCS}"
INCLUDE_DIRS "${INCLUDE_DIRS}"
REQUIRES "${IDF_DEPS}")

###############################################################################
# Warn all and make all warnings into errors in OpenMRN
###############################################################################
target_compile_options(${COMPONENT_LIB} PRIVATE -Wall)
target_compile_options(${COMPONENT_LIB} PRIVATE -Werror=all)
target_compile_options(${COMPONENT_LIB} PRIVATE -Wextra)

###############################################################################
# Suppress compilation warnings in OpenMRN
###############################################################################
target_compile_options(${COMPONENT_LIB} PRIVATE $<$<COMPILE_LANGUAGE:CXX>:-Wno-volatile>)
7 changes: 7 additions & 0 deletions etc/esp-idf/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# ESP-IDF Build Instructions
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please consider adding an idf_component.yml in this directory with the following contents:

dependencies:
  idf: "<5.3"

This will restrict compilation of OpenMRN to ESP-IDF version 5.2.x or earlier by default until I can get a PR setup for the driver updates in ESP-IDF v5.3

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use the IDF component manager for several reasons:

  • it becomes impossible to compile the code when the espressif servers are not reachable over the internet (have you tried compiling code while you are on an airplane?)
  • the compilation becomes not deterministic from the git hash

Both of these are big problems for a professional project. Reproducible compilation results are essential for finding bugs and retroactive maintenance. We are talking about 10y+ of support needs in this product domain, not a 6-month product development cycle like many other domains.

Having said that, it should be possible to use ther component manager if someone wants it. It is just a requirement for the compilation and dependencies to work correctly if someone chooses not to use it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproducible builds is a great call-out and has a rather simple solution. Limit your external dependencies (beyond esp-idf itself) AND embed the github link and hash (which typically can not change) and the build system will only use that specific version for external dependencies.

As for offline (airplane) usage, this is doable with small adjustments to the idf_component.yml and pre-staging the external dependencies locally. In the idf_component.yml file you would leverage the override_path or path element to shortcut the lookup of the component with a local path to where the dependency actually lives. You might need to remove version/git/etc elements though (I forget now on that).

Cloning the dependencies locally into a components tree (as git submodules) would also work for this, optionally adding the following snippet to the top level CMakeLists.txt file:

list(APPEND EXTRA_COMPONENT_DIRS "../common_components/Filesystem/")

This allows simplified dependencies mapping in component / main CMakeLists.txt files.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also note that the above snippet would ONLY check the ESP-IDF version and does not specifically deal with dependencies. If the ESP-IDF version check does not match the declared version it will abort the build with a message for the user to fix their ESP-IDF version selection.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@atanisoft I've added an idf_component.yml file. This causes a warning for my non-managed build, but we can choose to ignore it. Can you have a look and provide some feedback?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commented on the file above, we should just include the basic snippet listed above to only restrict the idf version check.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bakerstu it looks like no variant of declaration in idf_component.yml will work correctly with the current version of the idf component manager that I can find. The path: etc/esp-idf bit tries to force the git checkout to that path rather than reference that path for where to find the top level directory.

I suspect it only applies to when publishing the component to their registry and not for when consuming it. For now let's drop the file and I'll see if I can find a suitable way to make it work correctly or log an issue for them to fix it

The ESP-IDF has its own CMake based build environment for application developers. Using the standard ESP-IDF application build environment, The openmrn/etc/esp-idf directory can be linked into the ESP application "components" directory.
```
cd <application path>/components
ln -s <openmrn path>/etc/esp-idf/ openmrn
```
This will result in openmrn being compiled as a registered idf component with the name "openmrn".
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also document using idf_component.yml which is the approach recommended by Espressif for dependency management.

A version of this could be:

dependencies:
  OpenMRN:
    version: {branch, hash, tag, etc}
    git: https://github.com/bakerstu/openmrn.git
    path: etc/esp-idf

The path line overrides where idf.py looks for the CMakeLists.txt within the referenced repository (it defaults to root of the repository).

If you include override_path it will skip looking at the repository and use a local copy instead, which can use environment variables, such as: ${OPENMRNPATH}/etc/esp-idf.

The version line is optional, when not present idf.py build will collect the latest git hash and record it in dependencies.lock. If there is a newer version available in git it will present the following during build:

-- Building ESP-IDF components for target esp32s3
NOTICE: 
Following dependencies have new versions available:
Dependency "OpenMRNIDF": "5d7b0b3157e3411b3f71ccfacedaef6cb1202bff" -> "63e2b2e3f6ba34719f50194240113247793ac3ac"
Consider running "idf.py update-dependencies" to update your lock file.

6 changes: 6 additions & 0 deletions etc/esp-idf/idf_component.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
## IDF Component Manager Manifest File
dependencies:
OpenMRN:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Close, we can comment out this OpenMRN section as it would create a circular dependency which may trip up the dependency logic in esp-idf.

git: https://github.com/bakerstu/openmrn.git
path: etc/esp-idf
idf: "<5.3"
1 change: 1 addition & 0 deletions include/esp-idf/CDIXMLGenerator.hxx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../../arduino/CDIXMLGenerator.hxx
1 change: 1 addition & 0 deletions include/esp-idf/can_ioctl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../freertos/can_ioctl.h
atanisoft marked this conversation as resolved.
Show resolved Hide resolved
1 change: 1 addition & 0 deletions include/esp-idf/freertos_includes.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../freertos/freertos_includes.h
1 change: 1 addition & 0 deletions include/esp-idf/ifaddrs.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../freertos_select/ifaddrs.h
1 change: 1 addition & 0 deletions include/esp-idf/stropts.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../freertos/stropts.h
9 changes: 8 additions & 1 deletion src/freertos_drivers/esp32/Esp32Gpio.hxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
#ifndef _DRIVERS_ESP32GPIO_HXX_
#define _DRIVERS_ESP32GPIO_HXX_

#include "freertos_drivers/arduino/GpioWrapper.hxx"
#include "freertos_drivers/common/GpioWrapper.hxx"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we update arduino/libify.sh since it handles copying from common -> arduino? ref

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like these files are already handled in arduino/libify.sh.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in libify they are copied to arduino from common.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's track this via an issue to unify the libify bits so we can avoid ifdef etc

#include "freertos_drivers/esp32/Esp32AdcOneShot.hxx"
#include "os/Gpio.hxx"
#include "utils/logging.h"
Expand Down Expand Up @@ -365,6 +365,13 @@ public:
hw_init();
}

/// Get the current pin state.
/// @return true if the pin input is seeing HIGH.
static bool get()
{
return instance()->read();
}

/// @return static Gpio object instance that controls this output pin.
static constexpr const Gpio *instance()
{
Expand Down
0 ...eertos_drivers/esp32/Esp32HardwareI2C.cpp → ...eertos_drivers/esp32/Esp32HardwareI2C.cxx
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion src/freertos_drivers/esp32/Esp32HardwareTwai.cxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@
#include "can_frame.h"
#include "can_ioctl.h"
#include "executor/Notifiable.hxx"
#include "freertos_drivers/arduino/DeviceBuffer.hxx"
#include "freertos_drivers/common/DeviceBuffer.hxx"
#include "freertos_drivers/esp32/Esp32HardwareTwai.hxx"
#include "utils/Atomic.hxx"
#include "utils/logging.h"
Expand Down
2 changes: 1 addition & 1 deletion src/freertos_drivers/esp32/Esp32Ledc.hxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
#ifndef _DRIVERS_ESP32LEDC_HXX_
#define _DRIVERS_ESP32LEDC_HXX_

#include "freertos_drivers/arduino/PWM.hxx"
#include "freertos_drivers/common/PWM.hxx"
#include "utils/logging.h"
#include "utils/macros.h"
#include "utils/Uninitialized.hxx"
Expand Down
4 changes: 2 additions & 2 deletions src/utils/SocketClient.cxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ std::unique_ptr<SocketClientParams> SocketClientParams::from_static(
std::unique_ptr<DefaultSocketClientParams> p(new DefaultSocketClientParams);
p->staticHost_ = std::move(hostname);
p->staticPort_ = port;
return std::move(p);
return p;
atanisoft marked this conversation as resolved.
Show resolved Hide resolved
}

std::unique_ptr<SocketClientParams> SocketClientParams::from_static_and_mdns(
Expand All @@ -242,7 +242,7 @@ std::unique_ptr<SocketClientParams> SocketClientParams::from_static_and_mdns(
p->staticHost_ = std::move(hostname);
p->staticPort_ = port;
p->mdnsService_ = std::move(mdns_service);
return std::move(p);
return p;
}

#endif // OPENMRN_FEATURE_BSD_SOCKETS