I’d usually do the former because by build number I usually mean pipeline or job id in a build server. You could build 4.0.4 and then 3.4.18 and so 4.0.4 could be build number 1026 while 3.4.18 is 1027.

You can also just use a special number to keep your version number unique when doing dev builds so your version number comes through like 3.5.2-48 and some might call the 48 a build number, in which case that would make sense to reset with each version number.