On a Linux distro that doesn't have the `which` program installed we're
getting the following error:
$ ./emsdk install latest
./emsdk: line 39: exec: python: not found
It's failing to detect the installed `python3` and falls back to using
`python`, but this distro doesn't provide a python -> python3 symlink so
we fail.
Fix this by using `command -v` instead which is a POSIX standard.
The same change went into emscripten a couple years ago:
https://github.com/emscripten-core/emscripten/pull/15071
* Fix support for Apple M1. Node.js will still run via Rosetta 2 emulation since they do not yet have M1 support, but Python, LLVM, Emscripten and Binaryen will be native.
* Update M1 python version and URL
* Remove .gitignore additions
* Move python first in the manifest (#441)
* Use macosx-version-min when building python
* Update Intel macOS python package name
This change moves the python code for emsdk into a file ending in .py.
This script is then run via emsdk.bat on windows or emsdk (a shell
script) on non-windows.
This avoid the #!/bin/sh at the top of the python script and the "exec"
hack on the first line that re-runs it under python. Hopefully this
preserves the intent of #273 without jumping through so many hoops.
This means that `./emsdk` works on UNIX system after emsdk self-updates
from a zip file. Without this one would need to run `python ./emsdk`
which seems to be why the tests were doing it this way.
Implements what @juj suggested here: #274 (comment)
After this, the behavior should consistently be: archives are downloaded, unpacked, then deleted immediately. Another install of the same thing will re-download.
Windows version of python 3 returns platform.machine() in upper case for AMD64 which wasn't being checked for so added .lower().
Doesn't affect Linux python3 which already worked.
This adds an "arch" field to various structures, so ARM and Aarch64
Linux can download appropriate versions of node and in theory
other tools, without trying to download x86 builds.
Since there are no prebuilt packages for clang and binaryen on ARM
or Aarch64 this will require building them, which can take a long
time, but works once installed.
Node 4.1.1 and 8.9.1 entries for ARM (armv7l) and aarch64 are
added, and the various x86/x86_64-only things are marked as such
to be filtered out on ARM machines.
Other downloads work as long as they don't have an arch specified,
which indicates they're expected to be arch-independent.
Does not yet fully work on ARM64 Windows (but works in WSL as the
Linux support is fine).
The user can choose which CMake generator to use by passing the --generator option, e.g. --generator=Ninja or --generator="Unix Makefiles" to emsdk.
Default on *nix systems is Unix Makefiles
See WebAssembly/waterfall#542
The builds now contain lib/wasm-obj/ or lib/asmjs/ which have some cache contents. This places those in the cache so the user doesn't need to build them on first run, which for libc at least can be quite slow.
The mechanism here is to run emcc a first time in the emsdk. That clears the cache (since the emsdk just updated the .emscripten file). We can then safely place the files in the cache.
Note that FROZEN_CACHE is not used, since we do want to leave the user the option to build other things to the cache - we'll never ship all possible system and ports builds in the emsdk downloads, probably.
This contains a test, which passes on tot-upstream. The last tagged release doesn't have this yet.
* Default Python 3 for running emsdk
On macOS it's pretty common to hit a problem
```
Error downloading URL 'https://github.com/kripken/emscripten/archive/1.38.25.tar.gz': <urlopen error [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)>
```
When an installation is started. I.e. `./emsdk install sdk-1.38.25-64bit` - this is caused in majority of cases by some problem of openssl and python2 - (some other https downloads work just fine, not sure why GitHub is special)
So this Pull Request will default Python to Python 3, for running `emsdk`. It does so by making ./emsdk executable in python, bash, and sh. In bash and sh it will run a python executor script, which picks the best python it can find, preferring python3 on non-windows.
The non-git case was not well documented and not tested, and it was broken since we switched the build infrastructure.
In the git case, an update is to do a git pull. In the non-git case, it downloads a zip and unpacks it. The old build infra apparently had builds created for this; instead, I made it download it directly from github. Perhaps we should consider creating builds for this as well eventually?
Also add a test for this, so we never break it again.
Also fix the can_be_installed which was always broken it seems - it returns True for success or a string for error, but didn't check if the output is True.
Includes a test, which hacks up emsdk to make it think it's on 32-bit, and verifies the error and message.
The output now looks like
===
The *recommended* precompiled SDK download is 1.38.34 (048cf9424790cc525a7ea6da340820aae226f3b9).
To install/activate it, use one of:
latest [default (fastcomp) backend]
latest-upstream [upstream LLVM wasm backend]
Those are equivalent to installing/activating the following:
1.38.34
1.38.34-upstream
All recent (non-legacy) installable versions are:
1.38.34
1.38.34-upstream
1.38.33
1.38.33-upstream
===
and after that is the detailed output with the older builds etc.
List the recommended downloads (latest, latest-upstream) first, and with version and hash. Then list precompiled things, then list build-from-source things.
Remove old upstream-clang build from source, which has been incorrect since llvm switched to a monorepo anyhow. If we want the emsdk to support source builds of llvm, we'll need to fix that - however, as we can use plain upstream llvm anyhow, that shouldn't be hard for developers to build themselves (and normal users will get a precompiled version anyhow).
Remove the option to build fastcomp from source with wasm backend support, as it is horribly old there.
Noticed these issues in emscripten-core/emscripten#8728
We now have versioning directly here in this repo for emscripten-releases. This PR makes us stop looking for tags the old way, which looked on github for tags.
Rename the files legacy-*.
Instead of from mozilla S3.
This should make the core python, java, node downloads work, as we have mirrored them to the new location. I think that should be everything - if we missed something, we will get an error, and so know we need to fix something.
(The non-deps downloads still use a full url to emscripten-releases in the manifest, which is not changed here.)
Also change the download location for old llvm builds to emscripten-releases/old/
This makes
emsdk install 1.38.33
work (1.38.33 is from the new builders).
Also works with the old notation, sdk-1.38.33-64bit, and also supports -(upstream|fastcomp).
This makes emsdk [install|activate] latest get the latest fastcomp release from emscripten-releases, replacing our usage of the old mozilla build infrastructure.
This is only for "latest", so there should be no change to older releases.
This does remove non-release old stuff from the mozilla infrastructure, like "nightly" builds (that I don't think were even working?). After this PR, only old releases should be used from there.
Context: we've asked that people test the emscripten-releases builds for 1.38.33, and fixed a few issues people found. There are no open showstoppers, so this is us moving to the next step, serving latest builds from the new infrastructure.
This allows us to always unpack our builds into upstream/ or fastcomp/, instead of creating a new directory each time. Since without this, we'd get told to download a new file, and the emsdk would see that where it would be unpacked had contents, and assumed that even though it wasn't downloaded it must be the same. So we'd silently skip it.
It's useful to always unpack into the same dir since it's easier for people that create their own .emscripten file (the location of emscripten etc. is all fixed under the emsdk), and also it avoids directories accumulating, which each take hundreds of megabytes, so over time it can get burdensome.
We can remove this now that we have emscripten-releases working and use it everywhere we used the emsdk's waterfall integration.
In particular this should fix the current lkgr.json errors people are seeing (by removing all the lkgr stuff), which I believe started when I refactored that code while doing the releases work - I must have gotten something wrong on non-linux OSes. But anyhow, easier to remove that unnecessary code than fix it at this point.
emsdk install tot-upstream will install the very latest build from emscripten-releases. These builds are useful for emscripten github CI.
There isn't a latest or lkgr for emscripten-releases currently. What this does instead is get the git repo, and check if builds exist for the latest commits there, returning the latest of those.
This also makes us not update the emscripten-version.txt file if we are not an actual version. That is, for a tot build we leave that file unchanged in the emscripten checkout.
There is also a tot-fastcomp for fastcomp.
* Structures emscripten-releases-tags.txt to mention the latest release, and then a list of all previous releases.
* Adds a new release for 1.38.33.
* If the manifest has emscripten_releases_hash, then we write out emscripten-version.txt with the proper version. We start with the hash, and check if there is a released version for it, if so we emit that, otherwise we emit the hash.
* Windows zipfile fix.
Updated to our current initial plan here:
* Add emscripten-releases-tags.txt which is a JSON file with a map of tag name to git hash in the emscripten-releases repo. Right now this file is just checked in here; in the future we may make the emsdk update it from git tags with update-tags etc.
* That file contains a "latest" tag, which is fetched for when getting latest-releases-[upstream|fastcomp]. We may want that to eventually be updated based on the latest green stuff on the bots perhaps.
* Rename the current sdk-[upstream|fastcomp]-* etc. to sdk-waterfall-[upstream\|fastcomp]-*. Then we'll have sdk-releases-[upstream|fastcomp]-* etc. This is just an internal name change - emsdk install latest-upstream still works as before, and still uses the waterfall (so we don't break our github CI).
* Add support for emsdk install latest-releases-[upstream\|fastcomp]
* Add a test using the actual emscripten-releases builds
Without this, we would not download new versions, as the filename looked the same ("already downloaded"). We'd unzip those old contents to a directory with the new name, giving the impression we updated when we didn't :(
While doing so, we:
* keep the latest activation (e.g., the user may have activated latest and then latest-upstream, then the upstream LLVM is what is desired).
* keep the order of keys fixed (so the relative order of lines in the .emscripten file is fixed)
This adds some assertions in the Dockerfile, to verify we have one LLVM_ROOT command, and it is the right one.
Fixes#194
(This was reverted by mistake. It had a bad commit message though, so relanding with a nicer one is nice anyhow.)
With this, we can do emsdk install latest-fastcomp and it installs fastcomp from the waterfall. That is, we then have 3 main sdks people might want to use:
* latest which installs fastcomp-llvm (plus emscripten etc.) from the mozilla infrastructure. (fetches the last emscripten version there)
* latest-upstream which installs upstream-llvm (plus emscripten etc.) from the waterfall infrastructure. (fetches the last known good revision (lkgr) there)
* latest-fastcomp which installs fastcomp-llvm (plus emscripten etc.) from the waterfall infrastructure. (fetches the last known good revision (lkgr) there)
The first and last are currently somewhat overlapping in that both fetch a build of fastcomp. However, as we transition away from the mozilla infrastructure, we could just make latest an alias for latest-fastcomp. (And later, when we're ready to switch to the wasm backend by default, the alias could switch to latest-upstream.)
