Every day, people face decisions that involve multiple, often conflicting objectives: balancing cost against quality, safety against speed, or personal benefit against social responsibility. These trade-offs rarely admit a single “right” answer, and they become even more challenging when other decision-makers are involved. Reinforcement learning offers a powerful framework for constructing artificial agents that act autonomously in complex, uncertain environments, learning through trial and error how to make effective decisions. Yet, most existing approaches focus on a single objective, assuming away the trade-offs that are intrinsic to real decision-making. This thesis addresses that gap directly. Its central theme is how to design agents that can think in trade-offs, that is, agents that can reason about multiple objectives and act optimally under uncertainty. The contributions unfold in three parts that build on one another. First, we bridge single-and multi-objective reinforcement learning by showing how decomposition techniques allow well-established single-objective methods to be extended to learning the Paretofront, a classical solution set that captures efficient trade-offs for certain decision-makers. Building on this foundation, we then introduce and analyse alternative solution concepts that more directly reflect decision-makers{\textquoteright} preferences, developing rigorous theoretical guarantees and demonstrating their practical relevance. Finally, we turn to multi-agent systems and establish a novel reduction from multi-objective to single-objective games, which not only provides new theoretical insights but also enables the transfer of powerful algorithms across domains. Taken together, these results provide both theoretical and practical advances. Theoretically, the thesis deepens our understanding of what it means to act optimally under multiple objectives. Practically, it demonstrates how learning agents can be equipped to handle genuine trade-offs. By establishing strong connections between multi-objective and single-objective paradigms, the thesis lays the groundwork for future progress to be accelerated, enabling advances in one field to be immediately translated into the other. These advances bring us closer to systems that can adapt their behaviour to different stakeholders, balance conflicting objectives transparently, and operate responsibly in safety-critical real-world environments.