Competition is fundamental to sports. Even at less conventional displays of athleticism such as the X Games there is still a competitive hierarchy. You just didn’t see the actual competition that eliminated everyone except those four people performing. So it’s more like an exhibition of winners than a competition for winning. People like me who are relatively mediocre at those sorts of things (and even plenty who are fairly good) were already eliminated and effectively lost according to well-understood metrics.
Sports competitions differ mostly in their simplicity. In order to make the process of competition clearer and more understandable for outsiders and amateurs and non-participants, a very simple metric for competition and victory has been established as a proxy for the much larger and more complex dimensions of competition that got people to that place and actually are in competition during the game. We pick “points” as the thing to look at, but the points are largely symbolic of a much larger struggle. Some of which takes place during the game and some of which took place in the entire process leading up to the game cross years and whole athletic careers. So you can’t reduce any major sports competition to just the particular game or exhibition you see. That game is a net result, the conclusion of a process, not an isolated incident among equals.