Probe by Gaming: A Game-based Benchmark for Assessing Conceptual Knowledge in LLMs

Shuhang Xu, Weijing Deng, Yixuan Zhou, Fangwei Zhong✉

ArXiv preprint, 2025

Project, Paper

• A scalable and game-based benchmark for assessing the built-in conceptual knowledge of LLMs.

Hierarchical Instruction-aware Embodied Visual Tracking

Kui Wu, Hao Chen, Churan Wang, Fakhri Karray, Zhoujun Li, Yizhou Wang, Fangwei Zhong✉

ArXiv preprint, 2025

Project, Paper

• A hierarchical framework that combines LLM-based semantic-spatial goal aligner and RL-based adaptive goal-aligned policy for user-centric embodied visual tracking.

VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models

Kui Wu, Shuhang Xu, Hao Chen, Churan Wang, Zhoujun Li, Yizhou Wang, Fangwei Zhong✉

ArXiv preprint, 2025

Project, Paper

• A self-improving framework that enhances Embodied Visual Tracking (EVT) with Vision-Language Models (VLMs) to recover tracking from failure.

UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI

Fangwei Zhong* ✉, Kui Wu*, Churan Wang, Hao Chen, Hai Ci, Zhoujun Li, Yizhou Wang

ArXiv preprint, 2024

Project, Paper, Code

• A rich collection of photo-realistic 3D virtual worlds built on Unreal Engine, designed to reflect the complexity and variability of the open worlds.

Grasparl: Dynamic Grasping via Adversarial Reinforcement Learning

Tianhao Wu*, Fangwei Zhong*, Yiran Geng, Hongchen Wang, Yongjian Zhu, Yizhou Wang, Hao Dong

ArXiv preprint, 2022

Paper

• Learning to grasp moving objects with vision observation via adversarial RL.

CoMet: Metaphor-Driven Covert Communication for Multi-Agent Language Games

Shuhang Xu, Fangwei Zhong✉

The 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025 (Main)

Paper, Code

• A self-improving reasoning framework that enables LLM-based agents to engage in metaphor processing on Undercover and Adversarial Taboo.

Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective

Yipeng Kang, Junqi Wang, Yexin Li, Mengmeng Wang, Wenming Tu, Quansen Wang, Hengli Li, Tingjun Wu, Xue Feng, Fangwei Zhong, Zilong Zheng✉

The 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025 (Findings)

Paper

• Discovering the latent causal structure of the values in LLMs.

Behavior-agnostic Task Inference for Robust Offline In-context Reinforcement Learning

Long Ma, Fangwei Zhong✉, Yizhou Wang

International Conference on Machine Learning (ICML), 2025

Project, Paper, Code

• A model-based task inference method that is robust to the changes of in-context behavior for offline RL.

Simulating Human-like Daily Activities with Desire-driven Autonomy

Yiding Wang*, Yuxuan Chen*, Fangwei Zhong✉, Long Ma, Yizhou Wang

International Conference on Learning Representations (ICLR), 2025

Project, Paper, Code

• A desire-driven autonomy framework to guide LLM-based agents to simulate human-like daily activities in text-based environments.

Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy

Zhenyu Guan, Xiangyu Kong✉, Fangwei Zhong✉, Yizhou Wang

Advances in Neural Information Processing Systems (NeurIPS), 2024

Project, Paper

• A LLM-based social agent that can solve the game of AI Diplomacy only by self-play, without fine-tuning or regularizing on human data.

Fine-Tuning Out-of-Vocabulary Item Recommendation with User Sequence Imagination

Ruochen Liu, Hao Chen, Yuanchen Bei, Qijie Shen, Fangwei Zhong, Senzhang Wang, Jianxin Wang

Advances in Neural Information Processing Systems (NeurIPS), 2024 (Spotlight)

Paper

• A novel User Sequence IMagination (USIM) fine-tuning framework that first imagines the user sequences and then refines the generated OOV embeddings with the user behavioral embeddings.

Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL

Fangwei Zhong*, Kui Wu*, Hai Ci, Churan Wang, Hao Chen

The 18th European Conference on Computer Vision (ECCV), 2024

Project, Paper

• Significantly improved the training efficiency and generalization of embodied visual tracking with visual foundation models and offline RL.

Fast Peer Adaptation with Context-aware Exploration

Long Ma*, Yuanfei Wang*, Fangwei Zhong✉, Song-Chun Zhu, Yizhou Wang

International Conference on Machine Learning (ICML), 2024

Project, Paper

• Learn a context-aware policy with a peer identification reward to effectively explore and quickly adapt to unknown peers.

Bi-dexhands: Towards Human-level Bimanual Dexterous Manipulation

Yuanpei Chen, Yiran Geng, Fangwei Zhong, Jiaming Ji, Jiechuang Jiang, Zongqing Lu, Hao Dong, Yaodong Yang

IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE T-PAMI), 2023

Paper, Code

• A collection of bimanual dexterous manipulations tasks and reinforcement learning algorithms.

Learning Semantic-Agnostic and Spatial-Aware Representation for Generalizable Visual-Audio Navigation

Hongcheng Wang*, Yuxuan Wang*, Fangwei Zhong, Mingdong Wu, Yizhou Wang, Jianwei Zhang, Hao Dong

IEEE Robotics and Automation Letters (RA-L), 2023

Paper, Demo

• A brain-inspired plug-and-play method to learn a semantic-agnostic and spatial-aware representation for generalizable visual-audio navigation.

GFPose: Learning 3D Human Pose Prior with Gradient Fields

Hai Ci, Mingdong Wu, Wentao Zhu, Xiaoxuan Ma, Hao Dong, Fangwei Zhong✉, Yizhou Wang

Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2023

Project , Paper, Code

• A versatile framework to model plausible 3D human poses in gradient fields for various applications.

Proactive Multi-Camera Collaboration for 3D Human Pose Estimation

Hai Ci*, Mickel Liu*, Xuehai Pan*, Fangwei Zhong✉, Yizhou Wang

International Conference on Learning Representations (ICLR), 2023

Project , Paper

• A novel MARL framework to solve proactive multi-camrea collaborations for 3D HPE in human crowds.

RSPT: Reconstruct Surroundings and Predict Trajectories for Generalizable Active Object Tracking

Fangwei Zhong*, Xiao Bi*, Yudi Zhang, Wei Zhang, Yizhou Wang

Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI), 2023 (Oral)

Project, Paper

• A framework to form a structure-aware motion representation by Reconstructing Surroundings and Predicting the target Trajectory.

TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specification

Mingdong Wu*, Fangwei Zhong*, Yulong Xia, Hao Dong

Advances in Neural Information Processing Systems (NeurIPS), 2022

Project , Paper, Code

• A framework based on a target gradient field trained by score-matching to tackle object rearrangement without explicit goal specification.

MATE: Benchmarking Multi-Agent Reinforcement Learning in Distributed Target Coverage Control

Xuehai Pan*, Mickel Liu*, Fangwei Zhong✉, Yaodong Yang✉, Song-Chun Zhu, Yizhou Wang

Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS D&B), 2022

Project , Paper, Code

• A gamification of the multi-camera multi-target target coverage problem, and an all-in-one multi-agent reinforcement learning benchmark

Disentangling Disease-related Representation from Obscure for Disease Prediction

Chu-Ran Wang, Fei Gao, Fandong Zhang, Fangwei Zhong✉, Yizhou Yu, Yizhou Wang

International Conference on Machine Learning (ICML), 2022

Paper

• A disentanglement learning strategy under the guidance of alpha blending generation in an encoder-decoder framework (DAB-Net).

ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind

Yuanfei Wang*, Fangwei Zhong*, Jing Xu, Yizhou Wang

International Conference on Learning Representations (ICLR), 2022

Paper, Code

• A Target-oriented Multi-agent Communication and Cooperation mechanism using Theory of Mind.

Towards Distraction-Robust Active Visual Tracking

Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, Yizhou Wang

International Conference on Machine Learning (ICML), 2021

Project, Paper, Code , Environment

• A mixed cooperative-competitive multi-agent game: a target and multiple distractors form a collaborative team to play against a tracker.
• A bunch of practical methods: a reward function for distractors, a cross-modal teacher-student learning strategy, and a recurrent attention module for the tracker.

Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks

Jing Xu*, Fangwei Zhong*, Yizhou Wang

Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), 2021

Project, Paper, Code

• a Hierarchical Target-oriented Multi-Agent Coordination (HiT-MAC), which decomposes the target coverage problem into two-level tasks: targets assignment by a coordinator and tracking assigned targets by executors.

Pose-Assisted Multi-Camera Collaboration for Active Object Tracking

Jing Li*, Jing Xu*, Fangwei Zhong*, Xiangyu Kong, Yu Qiao, Yizhou Wang

Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), 2020

Project, Paper, Code , Environment , Demo

• An efficient yet effective multi-camera collaboration system for collaborative multiCamera active object tracking.

AD-VAT+: An Asymmetric Dueling Mechanism for Learning and Understanding Visual Active Tracking

Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, Yizhou Wang

IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE TPAMI), 2021

Paper, Code , Environment

• Employ more advanced environment augmentation technique and two-stage training strategies to improve the performance of the tracker in the case of challenging scenarios such as obstacles.
• Analyze the target’s behaviors as the training proceeds and visualize the latent space of the tracker for a better understanding.

AD-VAT: An Asymmetric Dueling mechanism for learning Visual Active Tracking

Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, Yizhou Wang

International Conference on Learning Representations (ICLR), 2019

Paper, Code , Environment

• A novel adversarial RL method which adopts an Asymmetric Dueling mechanism (tracker vs. target) for robust active visual tracking

CRAVES: Controlling Robotic Arm with a Vision-based, Economic System

Yiming Zuo*, Weichao Qiu*, Lingxi Xie, Fangwei Zhong, Yizhou Wang, Alan L Yuille

Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2019

Project, Paper, Code , Controller , Environment

• A vision system for low-cost arm control: trains a vision model in virtual environment, and applies it to real-world images after domain adaptation (a semi-supervised approach).
• One virtual environment for collection data and reinforcement learning.
• Two real-world datasets for evaluation.

End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learning

Wenhan Luo*, Peng Sun*, Fangwei Zhong*, Wei Liu, Tong Zhang, Yizhou Wang

IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE TPAMI), 2020

Paper, Code , Environment

• Deploy End-to-end active object tracker trained in virtual environment in real-world robot.

End-to-end Active Object Tracking via Reinforcement Learning

Wenhan Luo*, Peng Sun*, Fangwei Zhong, Wei Liu, Tong Zhang, Yizhou Wang

International Conference on Machine Learning (ICML), 2018

Project, Paper, Training code, Gym-tvizdoom, Gym-unrealcv

• An end-to-end active objet tracking solution via deep reinforcement learning, where a ConvNet-LSTM function approximator is adopted for the direct frame-to-action prediction.

Detect-SLAM: Making Object Detection and SLAM Mutually Beneficial

Fangwei Zhong, Sheng Wang, Ziqi Zhang, Chen Zhou, Yizhou Wang

IEEE Winter Conference on Applications of Computer Vision (WACV), 2018

Paper, Video

• A robotic vision system which integrates SLAM with a deep neural network-based object detector to make the two functions mutually beneficial.

Unrealcv: Virtual worlds for computer vision

Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zihao Xiao, Tae Soo Kim, Yizhou Wang, Alan Yuille

ACM Multimedia Open Source Software Competition, 2017

Project, Paper, Code ,

• An open-sourced project to help computer vision researchers build virtual worlds using Unreal Engine 4 (UE4).