If the sole way to lose fat is to create a caloric deficit...?

So really the only way for fat loss is to create a caloric deficit, meaning that you take in less calories than you burn. If this is the case - and it is - then what is the point of eating completely pure and clean and healthy? I understand it's because you're trying to maximize the use of these calories by getting the most nutrient possible, but why should we completely stay away from carbs, sugar, fat, and all those things we enjoy? Will I still be able to lose fat if I eat, for example, a cake for lunch instead of a good balance of protein, vegetables, and fruits? Suppose they have the same amount of calories.