Game theory is a branch of mathematics and economics that studies strategic interactions among rational decision-makers, analyzing how their choices affect outcomes in competitive situations.