Skip to content

Commit 48a8ef6

Browse files
dshi7facebook-github-bot
authored andcommitted
fix broken torchbench mimo_cmf_30x
Summary: this test has been broken since 05/22. https://pxl.cl/2S3Fg ## fix 1. refresh eval data or skip model export here's an example of failed job complaining data expiration. > MlDataComponentValidatorWrapper (data_config.ml_data_config): No available partitions found for given query filter. Namespace = ad_delivery. Table = ctr_mbl_feed_model_af_cd_async_ai_cd_30_neg_ds_md. filterClause = pipeline = 'ctr_mbl_feed_model_af_cd_async_ai_cd_30_neg_ds_md' AND ds = '2023-03-09'. we do have `analyzer.refresh_dataset()` but still see such error because we didn't call `analyzer.refresh_eval_dataset()`. the 'no partition found' error was happening at `self._serialize_inference_model`. https://www.internalfb.com/code/fbsource/[c953aa6a0a497851da8ec4f4361d0202dd5c33f7]/fbcode/dper3/dper3_models/ads_ranking/base_models/mimo_nn/mimo_pytorch_model_builder_base.py?lines=1512-1524 ~~furthermore, we realized even `export_mode` can be None in torchbench. similar trick has been adopted before to speed up model instantiation for testing purpose https://www.internalfb.com/diff/D46180124?dst_version_fbid=100400963077454&transaction_fbid=6056114577843800~~ ~~## fix 2. use_synthetic_data = True~~ ~~It doesn't work here. TODO. address it later.~~ ## fix 3. simplify branching of getting model Now ALWAYS generate model/data at run time to catch changes in model instantiation part too. Reviewed By: bertmaher, xuzhao9 Differential Revision: D47157906 fbshipit-source-id: d7423e7329185d368361e7b40ef2ba5fa9f926c6
1 parent 16e235e commit 48a8ef6

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

torchbenchmark/util/experiment/instantiator.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
from torchbenchmark.util.model import BenchmarkModel
1111
from torchbenchmark import _list_model_paths, load_model_by_name, ModelTask
1212

13-
WORKER_TIMEOUT = 1800 # seconds
13+
WORKER_TIMEOUT = 3600 # seconds
1414
BS_FIELD_NAME = "batch_size"
1515

1616
@dataclasses.dataclass

0 commit comments

Comments
 (0)