MITÂ News May 4, 2023
Current trace-driven simulators assume that the interventions being simulated (e.g., a new algorithm) would not affect the validity of the traces. However, real-world traces are often biased by the choices algorithms make during trace collection, and hence replaying traces under an intervention may lead to incorrect results. Researchers at MIT developed a causal framework for unbiased trace-driven simulation called CausalSim. CausalSim addresses this challenge by learning a causal model of the system dynamics and latent factors capturing the underlying system conditions during trace collection. It learns these models using an initial randomized control trial (RCT) under a fixed set of algorithms, and then applies them to remove biases from trace data when simulating new algorithms. Key to CausalSim is mapping unbiased trace-driven simulation to a tensor completion problem with extremely sparse observations. By exploiting a basic distributional invariance property present in RCT data, CausalSim enables a novel tensor completion method despite the sparsity of observations. Their extensive evaluation of CausalSim on both real and synthetic datasets, including more than ten months of real data from a video streaming system showed it improves simulation accuracy, reducing errors by 53% and 61% on average compared to expert-designed and supervised learning baselines. Moreover, CausalSim provides markedly different insights about ABR algorithms compared to the biased baseline simulator, which they validated with a real deployment… read more. Open Access TECHNICAL ARTICLEÂ Â Their code.
Caption: A new technique eliminates a source of bias in a popular simulation method… Image: Jose-Luis Olivares/MIT