In some ways it seems manipulative. Sometimes it is indeed a yes or no question, but most people know that certain answers require further explanation. It gives off the impression that you don’t wanna hear someone’s side of the story/debate. Sometimes “yes, but” or “no, but” is warranted.
The correct time to use this statement is when providing backstory or explanations is not relevant. Some actions can be excused or explained, others cannot. Sometimes people like to overcomplicate things for the sake of justifying something that isn’t or shouldn’t be justifiable. Stating that a questions is a yes or no question is essentially saying “the reasons don’t matter, did this happen or not?”