• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to footer
  • IEEE.org
  • IEEE Xplore
  • IEEE Standards
  • IEEE Spectrum
  • More Sites

IEEE Robotics & Automation Magazine

  • IEEE.org
  • IEEE Xplore
  • IEEE Standards
  • IEEE Spectrum
  • More Sites

DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition

April 17, 2026 by Yuki Kadokawa Jonas Frey Takahiro Miki Takamitsu Matsubara Marco Hutter ETH Zurich, Zurich, Switzerland

Abstract:

Preference-based reinforcement learning (PBRL) enables policy learning through simple queries comparing trajectories from a single policy. While human responses to these queries make it possible to learn policies aligned with human preferences, PBRL suffers from low query efficiency, as policy bias limits trajectory diversity and reduces the number of discriminable queries available for learning preferences. This article identifies preference discriminability, which quantifies how easily a human can judge which trajectory is closer to their ideal behavior, as a key metric for improving query efficiency. To address this, we move beyond comparisons within a single policy and instead generate queries by comparing trajectories from multiple policies, as training them from scratch promotes diversity without policy bias. We propose discriminability-aware policy-to-policy preference-based efficient RL (DAPPER), which integrates preference discriminability with trajectory diversification achieved by multiple policies. DAPPER trains new policies from scratch after each reward update and employs a discriminator that learns to estimate preference discriminability, enabling the prioritized sampling of more discriminable queries. During training, it jointly maximizes the preference reward and preference discriminability score, encouraging the discovery of highly rewarding and easily distinguishable policies. Experiments in simulated and real-world legged robot environments demonstrate that DAPPER outperforms previous methods in query efficiency, particularly under challenging preference discriminability conditions.

For more about this article see link below.

https://ieeexplore.ieee.org/document/11358816

For the open access PDF link of this article please click here.

Filed Under: Past Features Tagged With: Automation, Data models, Imitation learning, Labeling, Legged locomotion, Optimization, Quadrupedal robots, Query processing, Reinforcement learning, Trajectory, Uncertainty, User preference

Primary Sidebar

Current Issue

Get the entire issue now.

About the Magazine

As the flagship magazine of the IEEE Robotics and Automation Society, IEEE Robotics and Automation Magazine (RAM) covers the latest developments in robotics and automation. Its scope ranges from cutting-edge technological advances to emerging social, economic, ethical, and policy issues shaping the field.  Published quarterly (March, June, September, and December), RAM features both high-impact original research articles written in an engaging and accessible style, as well as reviews, columns and opinion pieces addressing a wide range of timely topics.

Past Issues

Search

Footer

LINKS

Home | Contact IEEE | Accessibility |
Nondiscrimination  Policy | IEEE Ethics Reporting | Terms & Disclosures| IEEE Privacy Policy

© Copyright 2025 IEEE – All rights reserved. A public charity, IEEE is the world’s largest technical professional organization dedicated to advancing technology for the benefit of humanity.

ABOUT US

IEEE Robotics & Automation Magazine  publishes four issues per year: March, June, September and December.