Abstract. Behavior cloning (BC) learns policies by treating human demonstrations as pointwise action labels. While effective with accurate action labels, this formulation is brittle in practice: when a human-provided action is imperfect, treating each label as an exact target can steer the policy away from the underlying desired behavior, particularly for expressive models such as energy-based models. Noting that this brittleness stems from pointwise supervision, we propose a human-in-the-loop online learning alternative that reformulates supervision as set-valued action targets derived from corrective feedback.

We introduce Contrastive policy Learning from Interactive Corrections (CLIC). CLIC constructs desired action sets from human corrections and trains the policy to place probability mass over these sets rather than a single action target. This formulation naturally accommodates both absolute and relative corrections, and enables the aggregation of multiple corrections into progressively refined desired action sets. Extensive simulation and real-robot experiments show that the proposed set-valued supervision leads to effective policy learning across diverse feedback settings: CLIC outperforms state-of-the-art baselines, remains competitive under accurate feedback, is substantially more robust under noisy, relative, and partial feedback, and supports complex multi-modal behaviors.