The downside is that you would be adding a dose of straight fat into your diet. If you have only the bulletproof coffee for breakfast, as some recommend, you have cut out the nutritional value of your traditional breakfast and substituted very little of value, in terms of fiber, vitamins, etc. If you have some type of breakfast, too, you are merely adding more calories - as fat or saturated fat.
The only possible upside to this approach may be the emphasis on reducing simple carbs in your diet. However, a more balanced breakfast, with fewer carbs, is the best option.