Restoring the Past for the Future