Your orientation doesn’t matter if you always take the direction as if you were in the drivers seat

Correct. And you would refer to that as something specific because if you just said left people might think that you are talking about your personal left. So you come up with short hand to mean the left when facing the front of the ship, and right when facing the front of the ship. The words that were agreed upon for this purpose? Port and starboard.

On a ship people will not be facing the same direction at all times. This is why the shorthand was invented. As someone who actually spent time on boats and where this was important, trust me. It’s necessary.