Wenda Xu (许闻达)

I am a research scientist at Google (Email: wendaxu@ucsb.edu, wendax@google.com).

My research focuses on improving large language models (LLMs) through rigorous evaluation and efficient post-training. I actively develop automated methods to assess model capabilities, including efficient data curation technique for building challenge benchmarks. To complement this, I design unsupervised and explainable evaluation metrics. My work in this area led to SEScore1&2 and InstructScore, which was recognized as the best unsupervised metric at the WMT22 shared task. My background also includes developing efficient post-training techniques, from preference learning with BPO to knowledge distillation with Speculative KD.

CV  /  Linkedin  /  Google Scholar  /  Twitter  /  Github

News

Industry Experience

Google Translate Research

Google Cloud AI Research

Google Translate Research

ByteDance AI Lab

Efficient Post-training

A generic KD framework that generalizes to on-policy and supervised KD, achieves substantial gains in task specific and task agnostic knowledge distillation.

On-policy BPO achieves superior performance compared to DPO on TL;DR (89.5% vs 72.0%), Helpfulness (93.5% vs 82.2%), and Harmfulness (97.7% vs 77.5%).

We develop a novel self-supervised approach to perform expert layman text style transfer, relative improvement in overall success rate at text style transfer by 106%.

An efficient inference time optimization technique that iteratively refines the outputs of the PaLM 2 model at text span-level, achieving improvements of 1.7 MetricX on translation, 8.1 ROUGE-L on ASQA, and 2.2 ROUGE-L on topical summarization.

A survey of self-correct strategies of LLM, including training-time, generation-time and inference-time approaches.

Large Language Model Evaluation

A scalable and efficient automatic data curation technique for building challenge benchmarks by finding the most difficult test examples.

MT-breaker, employs a LLM to iteratively generate difficult yet natural prompt examples to build more challenging translation benchmarks.

We demonstrate that LLM-generated benchmarks exhibit a systematic self-bias that inflates a model's own performance, and low source text diversity is one primary cause.

The study to define and quantify the bias exhibited by LLMs when assessing their own generated outputs.

A fine-grained 7B LLM autorater (detect errors at span level), trained on synthetic data, surpasses unsupervised metrics, including those based on 175B GPT-3 and GPT-4.

System demonstration of InstructScore.

SESCORE2, is a SSL method to train a metric for general text generation tasks without human ratings, kendall correlation improved 14.3% from SEScore.

SEScore is a reference-based text-generation evaluation metric that requires no pre-human-annotated data to train on. We develop a novel stratified error synthesis to synthesize diverse errors with varying severity levels at raw data.

Collaboration Publications

We reveal the root problems of exisitng latency evaluations at speech translation.

A novel framework that uncovers and quantifies the influence of specific factors driving preferences, exposing the gap of model's understanding and generation capability.

New work explores the causal relationship encoded in branching dialogue graphs from RPG video game.

We demonstrated automated detection of spurious, annotator-driven correlations that lead to cheating features in NLI.

This work mitigates spurious correlations by using symbolic program executors on latent procedural representations

Introduce a novel paradigm that leverages machine-generated images to guide open-ended text generation. This endows the machines with the ability of creative visualization that human writers often demonstrate.

Fun Fact: What is the meaning of Wenda(闻达)?

I add this because Starbucks people keep putting down "Wendy" or "Wanda"