Skip to content

Create a script + github action for rebaselining tests #23146

@sbc100

Description

@sbc100

We have a bunch of codesize tests that need to be rebaselined. This need can occur under 2 different circumstances.

  1. A llvm or binaryen commit rolled in that changes the generated code.
  2. An emscripten PR causes changes to the generated code.

I think it would be good if we can avoid mixing (1) and (2). These means that the main branch should always be updated to include changes from llvm/binaryen before the changes for specific PR are landed (as part of that PR).

For (1) procedure I've been using looks something like this:

$ git checkout -b main upstream/main
$ ./emcc --clear-cache
$ test/runner other.*code_size* other.*codesize* --rebase
$ git add -u .
$ git commit -m "Rebaseline codesize expectations. NFC"
$ git show  # check everything looks reasonable
$ git push upsteam # direct push to upstream, no PR

There are several problems with this (as pointed out in #23126 (comment)).

  1. Its a lot of separate steps
  2. It involves pushing directly to repo, without any review
  3. Its hard to see (using git diff) exactly what effect the changes have.

I propose adding a new script that performs most of the step above that will have the following advantages:

  1. It can analyze the results and present a human readable report (e.g. test xxx regressed by 10 bytes (0.1%)).
  2. It can automatically create a PR with the report in the description.
  3. It can be turned into a github action so it can be run on a remote machine (not a users machine where llvm and binaries version might not be exactly at tot).

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions