darwin: change the SDK pattern, update the SDKs, and update source releases

[reckenrode]

There are currently three different SDKs in nixpkgs, which are all built slightly differently. The 10.12 SDK is built from a mixture of source releases, stubs and headers (from source releases), and headers from the upstream SDK. The 11.0 (really, 11.1) SDK is built mostly from the upstream SDK. These two SDKs have served nixpkgs well for several years, but they have several limitations. The 12.3 SDK was added several months ago based on a stripped down version of the pattern used in #229210. It is a newer pattern still, but it shares many of the same limitations.

• Adding a new SDK version is a non-trivial amount of work. After finding the right SDK package source, you have to (attempt to) generate a dependencies file to make sure that adding a particular framework also propagates the frameworks it needs. Sometimes these frameworks cannot be determined automatically by some other means (such as trial-and-error).
• The newer SDKs don’t make use of the source releases much at all except for applications (various cmds packages), libiconv, and ICU (for .NET, mostly).
• Knowing which frameworks to use is not obvious, especially if you do not have access to Darwin hardware. Packages that could be built for Darwin sometimes are not because the maintainers don’t want to deal with trying different things until ofborg successfully builds the package, assuming they know enough to know which frameworks even to use.
• Overriding the SDK cannot be done generally. Some packages require newer APIs, but it is not possible to use a newer SDK by default. Some packages cannot be built with a newer SDK (e.g., PyTorch 2.3 and Qt WebEngine 5.15 cannot built with any SDK newer than the 13.3 SDK). Some packages perform static checks and use newer SDK features unconditionally if detected (e.g., SciPy’s Accelerate-based blas support). Consequently, an SDK corresponding to the minimum supported version is used with the ability to override the SDK (e.g., with overrideSDK).

To elaborate on the above, overrideSDK works currently by enumerating the transitive, propagated inputs of a package. This set of packages is then examined, and propagated SDK inputs are replaced with versions corresponding to the requested SDK. The problem is that the set of build inputs does not and cannot include all possible dependencies because dependencies can be propagated in other ways. Packages can capture a framework and include it in their build files (such as abseil-cpp prior to #265395 or rustPlatform, which captures various build/host/target clangs in its hooks.

The solution is to stop trying to fight how the platform works and embrace it: make the SDK a package laid out like a normal SDK would be on Darwin. The tooling expects it. Instead of having to hack build systems to deal with nixpkgs idiosyncrasies, most builds should be able to Just Work™ by default.

---

This PR adds a new SDK pattern for Darwin that provides a single SDK package for each version. A default SDK is provided as an extraBuildInput in the Darwin stdenv. To use a newer SDK version, add it as a buildInput. A hook is used to pick which SDK of those available is used (currently always the newest); and it is possible to have a different SDK for build, for host, and for target platforms. Most importantly, it is never possible to have more than one SDK active. Weird errors due to semicolons or other punctuation should be a thing of the past.

Additionally, a separate hook is also provided to set the minimum version (also known as the deployment target). If your package specifies that it works on 11.0 and newer, you can use the hook to do that. If your package is a library, you can propagate the hook to make sure users of your library also target the correct version.

This combination allows SDKs and SDK requirements to be propagated to packages without using a new scope and callPackage. Packages that would be prevented from using by-name can now be moved there and use whatever SDKs they need or are propagated to them by their dependencies.

To provide better compatibility with build systems that assume Xcode is available, xcrun is also included in the stdenv. It and the xcbuild derivation have been reworked to find the SDK dynamically. It should hopefully no longer be necessary to patch out invocations of xcrun in build systems (e.g., xcrun --find clang in a build will return clang from nixpkgs).

In addition to adding the new SDK pattern, this PR fixes cross-compilation on Darwin. It should now be possible to build static packages for both aarch64- and x86_64-darwin (assuming the packages support it). It should also be possible to cross-compile between both architectures (again assuming the packages support it).

New SDKs

The following SDKs are added in this PR: 10.12.2, 10.13.2, 10.14.6, 10.15.6, 11.3, 12.3, 13.3, and 14.4. The 15.0 SDK has been left out to be added later as a test of the SDK update infrastructure to assess and confirm how easy it is to add a new one. In my testing, adding the 15.0 SDK is a ~5 minute task. Hopefully, it goes as easily for @emilazy.

Source Releases

The 10.12 SDK builds several frameworks and libraries from source. It also makes use of their headers. The new SDK pattern no longer tries to get headers from the source releases. They’re just too inconsistent with the headers in the SDK. Some headers are even older than the 10.12 because newer sources are not available or complete enough to use.

However, this update actually builds and uses more source releases than before. The following libraries are built. Of these, libresolv is new on aarch64-darwin, libsbuf is new for both (ported from FreeBSD due to the lack of a source release), libpcap is new for both, and copyfile and removefile are built from source for the first time.

• ICU
• libiconv
• libresolv
• libpcap
• libsbuf
• libutil
• copyfile
• removefile
• xar

Currently, libiconv and libresolv are propagated with the SDK on both platforms. For 25.05, libpcap and ICU may be made to be the default implementation on Darwin due to containing Apple-specific changes.

It is also worth noting: all source releases correspond to the source released for macOS 15. When possible, it has been backported to build with the default SDKs.

Source release version updates

• AvailabilityVersions: 140.1 -> 143.3
• ICU: 66108 -> 74221
• IOKitTools: init at 125
• PowerManagement: 572.50.1 -> 1740.0.7
• adv_cmds: 163 -> 231
• basic_cmds: 55 -> 70
• bootstrap_cmds: 121 -> 136
• copyfile: 138 -> 213
• developer_cmds: 62 -> 79
• diskdev_cmds: 593 -> 735
• doc_cmds: init at 66
• file_cmds: 264.1.1 -> 448.0.3
• libiconv: 99 -> 107
• libpcap: init at 135
• libresolv: 64 -> 83
• libutil: 47.30.1 -> 72
• mail_cmds: init at 38.0.1
• misc_cmds: init at 44
• network_cmds: 606.40.2 -> 696
• patch_cmds: init at 62
• remote_cmds: init at 303.0.2
• removefile: 45 -> 75
• shell_cmds: 187 -> 319.0.1
• system_cmds: 970.0.4 -> 1012
• text_cmds: 99 -> 190.0.1
• top: 108 -> 139
• xattr: replace with xattr from file_cmds
• xar: 498 -> 501

Transition Strategy

Because this is a non-trivial change to how Darwin works, a transition is necessary. This update tries to avoid breaking existing packages, but there are breaking changes (see below). To minimize breakage, all existing SDK frameworks and libraries as well as dropped source releases (anything not above) have been replaced with stubs. These stubs contain a README advising the user to read the documentation on how to update to the new pattern. Currently, these stubs do nothing other than provide stubs. After a treewide has been done, they will moved to darwin-aliases.nix. It is expected that these aliases will remain in place until at least the 26.05 release.

Breaking Changes

• libiconv is no longer linked automatically. Since it is propagated by the SDK by default, including this hook would link libiconv to everything, which may not be desirable. Failing to link libiconv is a build system error. The upstream builds should be fixed to correct the mistake.
• CoreFoundation is no longer linked automatically. Same as above. This is a build system issue. Hooks really shouldn’t be used to work around mistakes upstream, especially when the issues are not pervasive.
• Anything that depended on the layout of the old frameworks will need to be updated to work with the new pattern:
  • Frameworks can be found at SDKROOT/System/Library/Frameworks;
  • The SDKROOT for a particular SDK package is available using the sdkroot attribute on the package (e.g., apple-sdk.sdkroot).
• Darwin is now a “system” libc++ platform. LLVM will now always use the libc++ built in the stdenv unless you use llvmPackages.libcxxStdenv. This aligns Darwin with Linux where llvmPackages.stdenv uses the default libstdc++.
• Darwin’s libc is determined dynamically. darwin.libSystem contains empty lib and include folders to avoid compiler and linker warnings that may be turned into errors. The libSystem stubs can be located at $SDKROOT/usr/lib/libSystem.tbd.
• Packages may try to build additional functionality due to the presence of SDK frameworks. For example, Python no longer supports the configd option to disable the _scproxy module. The SDK always has the SystemConfiguration framework.
• Vendored packages have been removed from the SDK. Between all the different versions, more than 50 packages are vendored. Those that are conventionally added as inputs are excluded from the SDK. Just add them as inputs like normal. Those that are part of the SDK are propagated. This includes libiconv and libresolv as well as the headers for ncurses (which are needed by Swift). Note that libutil is not propagated because its headers may cause problems with some packages (particularly Python). If you need libutil, add darwin.libutil as a build input.
• Old source releases are no longer available, but their source for the SDK is accessible via the apple-sdk.sourceRelease function. This is mostly useful for copying private headers for use to build a derivation. See various source releases for examples.

Closed Issues and PRs

• closes macOS libSystem and other libraries haven't been updated since macOS Sierra #86299
• closes Update macOS SDK #101229
• closes Cross compiling to x86_64-darwin from aarch64-darwin seems broken #180771
• closes apple_sdks: init with 11.1, 11.3, 12.1, 12.3, 13.1, and 13.3 #229210
• closes Discussion: The future of SDKs on Darwin #242666
• closes llvmPackages_*.stdenv: use top-level libc++ on Darwin #264091
• closes Outdated apple SDK Security framework #269646
• closes pkgsStatic.coreutils: error targetPackages.darwin.LibsystemCross x86_64-apple-darwin  #270375
• closes Build failure: pkgsCross.aarch64-darwin.stdenv #273442
• closes CoreFoundation.tbd hook breaks darwin -> non-darwin cross-compilation #278348
• closes darwin.apple_sdk.frameworks.CoreFoundation: drop hook #284629
• closes darwin.stdenv: propagate sdkRoot via extraBuildInputs #308366

Things Still to be Done

This PR is almost completely ready. If need be, it could be merged into staging to make the next staging-next cycle. The following.

• Rework locale data. In a pinch, I could duplicate the current logic. Apple appears to be using the same source of data as FreeBSD, but the data from the FreeBSD doesn’t actually work with their tools. Honestly, I’m not sure setting the environment variable even works. (Fixed in darwin.locale: restore locale data #347817).
• netstat doesn’t print active Internet connections. It prints other kinds but not those. It requires further investigation. Punted for now.
• Bootstrap tools need updated. This could wait until after this is merged, but it will need to be done before this hits master to prevent the channel from being blocked. Will be updated separately.
• Write documentation and release notes. This can be done after this PR since it doesn’t affect builds or the release freeze.

Related PRs

These issues were found during testing. This flake was used to help find issues. Some of these can target staging or even master without waiting. Others depend on this PR or even other PRs.

• bacula: fix build on Darwin #346986
• boost: fix cross-compilation for Darwin #346954
• curl: fixes for static linking on Darwin #346950
• darwin.locale: restore locale data #347817
• dotnetCorePackages.dotnet_{8,9}: ensure build can find the Swift overlay #346968
• espanso: fix build on Darwin #347008
• gcc: fix build with new SDK pattern #346949
• go: propagate the apple sdk #332146
• gst_all_1.gstreamer: only try to link libdw if it is available #347173
• haskellPackages.network: fix build on x86_64-darwin #346928
• hello: make sure libiconv is linked #346939
• icoutils: defining missing target conditions on Darwin #347009
• jellyfin-media-player: fix build on Darwin #346952
• libcaca: fix build on Darwin #347014
• libpsl: fix Darwin cross-compilation #347007
• libredirect: fixes for new Darwin SDK #347012
• librsvg: link libobjc on Darwin independent of CPU architecture #347013
• lowdown: add flag to disable the Darwin sandbox #346945
• lowdown: disable sandbox on x86_64-darwin #346933
• man-db: fix test failure when using Darwin’s libiconv #346955
• memstream: cross-compilation fixes #346934
• moltenvk: update for the new xcbuild, the new SDK, and the new version plus cleanup #346962
• neovim-unwrapped: update build inputs for new Darwin SDK #346946
• netbsd.install: fix build with new Darwin SDK #347011
• nodejs: suppress failing test on x86_64-darwin #346877
• python3Packages.elastic-transport: disable async httpbin test #346989
• python3Packages.materialx: enable MSL support on x86_64-darwin #346930
• python3Packages.skia-pathops: fix build on Darwin #346991
• qt{5,6}: update for the new Darwin SDK #347216
• quakespasm: fix build on Darwin #346972
• rpm: fix build on Darwin #346967
• swift: fix build with the new SDK pattern #346947
• whois: link libiconv on Darwin #346966
• wine64Packages.{stable,unstable,staging}: use the 14.4 SDK #346963
• xdg-user-dirs: make sure libiconv is linked #346970

---

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/on-the-future-of-darwin-sdks-or-how-you-can-stop-worrying-and-put-the-sdk-in-build-inputs/50574/14

[reckenrode]

Pinging @emilazy, @paparodeo, @toonn, @RossComputerGuy, and @Ericson2314 (for cross-related stuff).

[emilazy]

This is incredible work, Randy, and I can’t thank you enough for all the time and effort you’ve put into it. Every time I hit a commit that removes hundreds of lines of unmaintained and cobweb‐filled code I was delighted. Even the scary commits were less scary than I was expecting them to be. The code quality here is so consistently high that I have largely been reduced to nitpicking code I understand and asking questions about code I don’t understand.

This is surely the biggest change to Darwin support since we moved off impure frameworks and I am confident that it is going to make things vastly nicer for both maintainers of the core Darwin infrastructure and contributors to packages across the tree, who will be faced with far fewer confusing and painful Darwin‐only build errors. I am proud to have had my small part in shaping the interface and implementation decisions used here, and I think the result is fantastic. I am excited for the 25.05 minimum version bump that will allow a lot of the remaining technical debt here can be dropped and x86_64-darwin should stop being such a problem.

I am too tired today to do a really deep dive on the core stdenv code here given my inexperience with the bootstrap, so please consider these reviews to be largely me familiarizing myself with the code, asking questions, and pointing out relatively shallow things that seem wrong to me. I hope to do another, closer read in the next few days after this round of reviews is dealt with.

I have a few general concerns about the source release situation:

1. Xcode project files are baroque and unreadable. Keeping them in sync with our Meson build system rewrites may be painful. I have been convinced that rewriting the build systems into Meson is the right approach here, but I still worry about bitrot and drift. This is not really an avoidable problem, but it ties in to the subsequent points.

2. It seems like it might be a good idea to merge the source release updates into the commits that rewrite them, as in some parts you have ported build system logic to Meson only to rearrange and rewrite it before anyone has even had a chance to use it.

3. While I am glad that it is so relatively easy to package new source releases (x86_64-darwin notwithstanding), it works as a great proof‐of‐concept for the new format, and I hate to poke at the sunk cost of the fun distraction here, I am still uneasy about the new packages being introduced here, especially in light of (1).

   We have already moved away from the source‐based PureDarwin dream and accepted a runtime dependency on the shipped binary code of the proprietary OS we’re targeting. I am not sure it makes sense to try and package every available source release of core system commands when canonical versions of those binaries are already bundled with the system that is required to run our packages in the first place, especially as we have to make sacrifices to functionality because of missing entitlements and private headers. It’s not like Apple release enough source dumps that you could avoid running things from /usr/bin, anyway.

   It would be one thing if it was a one‐and‐done job of packaging them and using their upstream build systems and then letting update scripts handle the rest, but we can see from this PR that new source releases often require active build system intervention due to our Meson rewrites, and I worry about this turning into a situation like we’ve had too much of in the past for Darwin, where the next person to maintain all this is resigned to going through the motions and burden of maintaining these packages despite extreme scepticism about whether anyone is actually benefiting from them. Additionally, these source dumps often require messing with private headers due to Apple’s messed up internal SDK situation, which makes them a distinctly different type of challenge from packaging any old random source dump of applications on the internet.

   I think we should focus on packaging things that are useful for Nixpkgs builds; even unixtools aliases that are not used anywhere else in the tree I am somewhat sceptical of.

[emilazy]

cc @tie for Bash review (please feel free to ignore this ping if you’re still taking a break!)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/breaking-changes-announcement-for-unstable/17574/65