transformers.js tests are not preloading files, lowering CPU state

[Constellation]

It seems that model files and configuration json files are downloaded in the middle of JetStream3 instead of preloading, which can lower CPU state and speed down the browser regardless of JS engine's quality.

[kmiller68]

Yeah, it seems to only happen when I empty caches in Safari between runs. I haven't investigated if similar behavior happens in other browsers yet.

[Constellation]

It seems that,

1. transformer.js subtests are not using preloaded files. From reviewing the code, it is loading the result from the network
2. At the first run, it is storing its own file cache (blob)

But this is problematic since (1) happens in the middle of JetStream3 and we should not load (1) at the first run with clearing the cache. So, it needs to be changed.

[Constellation]

So,

1. We should disable transformer.js's file cache mechanism. It is not good to have different behavior only for the first run on the browser. That breaks subtest's stability (behavior is different with cache configuration)
2. We should use preloaded blobs instead of downloading models / config jsons in the middle of JetStream3.

[danleh]

Oh, we should definitely fix that, thanks for the report! I'll look into it...

[danleh]

Unfortunately, I didn't get to finish this and am out of office until mid October. There are two parts to the fix, the first is to provide a custom cache function in the setup during the first iteration of benchmark.js, something along those lines:

      env.useBrowserCache = false;
      env.useCustomCache = true;
      env.customCache = {
        match: function(url) {
          // Redirect some paths to cached/preloaded resources.
          if (preload[url]) {
            console.log('match preload hit', url, preload[url].byteLength, 'bytes');
            return {
              ok: true,
              status: 200,
              arrayBuffer() { return preload[url]; },
              async blob() {
                return {
                  size: preload[url].byteLength,
                  async arrayBuffer() { return preload[url]; }
                }
              },
            };
          }

          throw new Error('unknown file ' + url + '\n' + 'preload: ' + Object.keys(preload));
        },
        put: function(url, response) {
          return Promise.resolve(undefined);
        },
      }

The other issue is that due to the preloading via blob URLs in the driver, the preload map will not contain the non-blob-URLs, i.e., this match will fail unless one also adds something like:

    preload['./transformersjs/build/models/Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer_config.json'] = preload[JetStream.preload['modelTokenizerConfig']];
    preload['./transformersjs/build/models/Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json'] = preload[JetStream.preload['modelTokenizer']];
    preload['./transformersjs/build/models/Xenova/distilbert-base-uncased-finetuned-sst-2-english/config.json'] = preload[JetStream.preload['modelConfig']];
    preload['./transformersjs/build/models/Xenova/distilbert-base-uncased-finetuned-sst-2-english/onnx/model_uint8.onnx'] = preload[JetStream.preload['modelWeights']];

to init().

We somehow need to get the original URL for a given blob URL inside the workload driver (i.e., on the JetStream.preload object).

I'll take another look once I'm back in 3 weeks.