This PR adds support for
`./emsdk install node-nightly-64bit`
which finds the latest nightly node.js version and installs that. It is
a moving target, so the next day when a new nightly is produced,
reissuing `./emsdk install node-nightly-64bit` will install the newer
published version.
Also, this PR adds a fixed 22.18.0 version for the cross compilation
s390x Node.js target as well. On an x64 Linux system, this allows
running
`./emsdk install node-big-endian-crosscompile-22.16.0-64bit`
to install the big endian Node.js. This greatly simplifies the steps at
https://github.com/emscripten-core/emscripten/compare/main...juj:emscripten:bigendian_test_suite#diff-c36b90121be240017fa490a1c00e63e47fa3235f5c1be0593e2b7502d017c778R9985-R10000
and enables a trivial way to switch between LE and BE Node.js versions
for testing.
CC @slavek-kucera
Previously, if one used e.g. command
```
emsdk install --override-repository emscripten-main-64bit@https://github.com/juj/emscripten/tree/my_own_emscripten_branch emscripten-main-64bit
```
then the overridden installation would create a git clone from
`juj/emscripten`, and that clone would be located under the default
remote name `origin`.
This would prevent being able to switch between overrides on subsequent
`emsdk install` commands. I.e. it would then not be possible to later do
a default
```
emsdk install emscripten-main-64bit
```
command without an override, since the remote name `origin` would point
to `juj/emscripten` instead of `emscripten-core/emscripten`.
This PR changes the naming scheme of `git clone`s when
`--override-repository` is used, to name the clones with remote names
from the override, so in the above case, the clone would appear under
remote name `juj` and not `origin`.
This enables flipping between multiple `emsdk install` commands in the
same checkout, without needing to nuke the installed directory in
between.
This enables me to run experiments like
github.com/emscripten-core/emscripten/pull/25025 locally on my CI,
without needing to merge them upstream to become available.
The `-0` and `-1` suffixes here are not part of the install path which
only based on the `id` and `version` of the tool. e.g:
```
{
"id": "python",
"version": "3.9.2",
"bitness": 64,
"arch": "x86_64",
"windows_url": "python-3.9.2-1-embed-amd64+pywin32.zip",
"activated_cfg": "PYTHON='%installation_dir%/python.exe'",
"activated_env": "EMSDK_PYTHON=%installation_dir%/python.exe"
},
```
Here we have a `-1` in the archive name, but that is not included in the
version which is used to construct the install path.
Fixes: #1570
Update Python to 3.13.0 and add support to Windows on AMD64. Rewrite
packaging of pywin32 because it is no longer shipped via separate binary
installers, but has migrated to use pip.
This migrates the bazel integration to the new Bazel dependency system
"bzlmod". bzlmod is becoming mandatory this year (see second sentence
here: https://bazel.build/external/migration).
This is a backwards incompatible migration, directly removing the old
WORKSPACE based approach. Users will have to change how they depend on
bzlmod, however I assume pretty much every user will be happy about it,
because they are forced to use bzlmod anyway or add extra flags to
continue to build with newer Bazel versions. Given that users normally
depend on specific git commits in the old system, they won't be hit with
this until they decide to upgrade emsdk.
The basic principle here is simple: I took everything that WORKSPACE did
and searched an alternative in bzlmod. Some interesting bits:
- We have less worries about multiple versions and people depending on
emscripten multiple times in different ways. This is resolved by the new
system: Bazel first iterates through the MODULE.bazel files recursively,
then finds the minimum version needed for each module and then executes
the module extensions that define repos exactly once at that version. So
no more ifs needed to detect multiple inclusions.
- A bunch of nodejs stuff moves to MODULE.bazel, because that is how the
nodejs module works now. As their module extension gets executed only
once you need to declare everything that you could need before that in
the MODULE.bazel file. A side effect of that is that we have to make a
fake repository when emscripten doesn't have an arm64 binary for linux,
because we can't actually figure that out in MODULE.bazel, so we have to
declare that it always exists and then create one in all cases.
There is a bunch of autoformatter changes in here as well, I could try
to revert them if you prefer.
Closes#1509
local=True marks a repository rule to unconditionally re-execute anytime
bazel re-evaluates the workspace (almost every invocation) It works well
for processes that execute in <10ms, not large ones like the embuilder
cache generation.
I have tested this locally and it seems to work fine, but I'll be honest
that I don't understand the nuances of repository rules that well, nor
do I know much of anything about emscripten or the circumstances in
which the cache ought to be rebuilt.
I checked the `external/@emscripten_cache.marker` file that Bazel uses
to store the cache key for if it needs to re-evaluate, and changing any
of the options passed into register_emscripten_toolchains resulted in
the cache key changing, and it correctly resulted in the cache being
regenerated.
The big question I don't have the context to answer: are there files on
disk other than these two that need to be watched for changes, so the
repository rule should change if they change:
```
FILE:@@emscripten_bin_linux//BUILD.bazel
FILE:@@emsdk//emscripten_toolchain/default_config
```
The current way to derive the binary path relies on a specific name for
the nodejs repository. This blocks migrating to bzlmod, as bzlmod
prefixes repository directories with the module name that created them.
By asking bazel for the path instead we always get the correct path, so
we can work with both bzlmod and WORKSPACE based dependencies at the
same time.
The repository @nodejs, used in the build label, refers to nodejs for
the host platform and is generated by the following macro (wasn't too
obvious to me):
d19d695275/nodejs/repositories.bzl (L452)
This is some work towards solving #1509.
This finishes the work started in
https://github.com/emscripten-core/emsdk/pull/1388 by fixing CI. It
avoids a breaking change by:
* Using the latest rules_js 1.x.x version, instead of updating to
rules_js 2 (which removes support for bazel 5).
* Copying the contents of
[rules_js_dependencies](https://github.com/aspect-build/rules_js/blob/main/js/repositories.bzl)
instead of calling it, as the call would need to be added by users in
their `WORKSPACE` files
Context from the previous PR:
> Bazel's Node.js dependency comes from
[rules_nodejs](https://github.com/bazelbuild/rules_nodejs/). Previously,
bazel/deps.bzl was using rules_nodejs 5.8.0, released in 2022 and only
supported Node.js toolchains up to 18.12.1.
> This PR bumps rules_nodejs to latest 6.1.1. It also replaces
build_bazel_rules_nodejs with
[rules_js](https://github.com/aspect-build/rules_js), since npm_install
that bazel/emscripten_deps.bzl used was deprecated. The README of
rules_nodejs now recommends migrating to rules_js for everything other
than the Node.js toolchain:
(371e8cab15)
> Impetus
Our repo builds with Bazel and uses Emscripten and Node.js. Tried to
upgrade Node.js 18 to Node.js 20 and saw that emsdk didn't support
rules_nodejs 6+ in the same workspace.
Similarly, it's not possible to update to rules_js v2 in a workspace
that also references `emsdk`.
Update Node from 18.20.3 LTS to 20.18.0 in precompiled releases. This
does have an effect of retroactively updating Node version also to old
releases, but this is expected to be fine.
Earlier Node 18.20.3 LTS did not yet support Windows on ARM64 builds.
Node 20.18.0 LTS is released on
https://nodejs.org/en/blog/release/v20.18.0.
This update currently only affects the "compiled from source" builds,
and not the Google precompiled releases.
