It is generally true that calories consumed must be less than calories expended. However, that doesn't tell you much about how to actually IMPLEMENT a plan to achieve a healthy caloric deficit on an ongoing basis. Diets fail because it’s hard to ignore/overcome the chemistry of the brain, chemistry that evolved at a time when our food choices and availability were much different than they are today.
In my experience the best diet is one that does the following:
Emphasizes green vegetables and fruit
Emphasizes lean proteins such as fish, turkey, eggs, and chicken
Restricts soda, milk, sports drinks, and other beverages that contain large amounts of sugar
Allows you an occasional “cheat” meal or dessert
Why?
Vegetables and fruit provide a higher ratio of fiber, micronutrients, protein, and water to sugars/carbohydrates than most other food sources, particularly processed foods like breads, pastas, cereal grains, etc; this means that by eating veggies/fruits you're getting more vitamins and minerals for the same volume of food, and it also means you will feel more full eating the same volume of food (or equivalently you could eat feel the same "fullness" eating a smaller volume of food
Lean proteins provide the essential amino acids that your body needs and can't produce on its own. Several of these amino acids are difficult to find in high enough quantities in fruits and vegetables (though not impossible to find). Lean proteins have less saturated fat than other proteins and more optimal ratios of Omega-3 to Omega-6 essential fatty acids
Beverages are probably the sneakiest sources of carbs/sugars in American diets. You can easily consume more calories from your soda than from your actual meal at a restaurant or fast food joint. Often when you're guzzling soda, a latte, Gatorade, or milk, what your body really needs is merely the water for hydration, not the sugar calories that are mixed in for taste. Also, many of our beverages are over engineered for your average American. For instance, only athletes expending a lot of energy really need the full electrolyte replacement and sugar of a sports drink. Lastly, many beverages contain preservatives, dyes, phosphates, and other chemicals that you don’t need and which may negatively impact your digestive system/nutrient uptake
Cheat meals will offset your caloric deficit efforts somewhat, but they also help you deal with the realities of everyday life. Ideal food choices aren’t always available in social situations or on the go. However, cheat meals should account for less than 1 out of 5 meals, or you risk spiraling back into unhealthy eating habits
This is more or less a “Paleo” based diet, although my own empirical evidence suggests high vegetable consumption, moderate fruit consumption, and light meat consumption (just a few ounces per meal) actually drives greater weight loss than a diet with higher meat content. This could be due to the fact that a veggie heavy diet delivers fewer calories than a meat heavy diet for the same volume of food. Said another way, it’s likely that the veggie heavy diet is less calorically dense. Vegetarians would probably applaud this approach and tell you to just eliminate the meat, but I am not convinced. For me, it takes a lot of work to get a complete protein/amino acid profile out of a vegetarian diet on a consistent day in and day out basis. We could sit here while Colin Campbell and Loren Cordain argue minutiae and conflicting studies all day, but I can tell you that an emphasis on whole foods such as vegetables and fruits, complemented with lean protein consumption is a very good way to lose weight. It is difficult to overeat on this kind of diet.
Americans typically overconsume carbohydrates. Sweeteners (carbs) are added to just about every processed food and beverage because our brains respond positively to sweetness. Our bodies store the excess carbs as fat. The reason that low carb diets (Atkins, Zone, South Beach,etc) are popular and have been cited in many success stories is that they rightly get people to reduce carb consumption. The problem with some of these diets is that they have been commercialized to such an extent that their names now appear on processed foods like snack bars that have low nutritional value. Dieters who buy this stuff don’t end up seeing much success. Their bodies rebel and the dieters go on a binge or quit the diet altogether. The other problem is that these diets are often highly simplified and too many people take the general principles to the extreme. Many people try to eliminate carbs or fat altogether, but that is a dangerous strategy since your body actually does need those macronutrients.
So if you believe in the tenets of a more balanced, whole foods based approach as I do, here are some best practices/hacks to actually implement the approach:
Make/drink green smoothies made with lots of leafy greens, some fruit, and coconut water or soy/almond milk; these substitute for meals, are quick to make with a blender, are tasty and high quality, and will fill you up
Try to limit your beverages to water and tea
If you want to drink milk, use a substitute milk made from almonds or soy; these typically have far fewer calories and hormones
If you get hungry between meals, snack on whole foods like nuts, grapes, carrots with hummus, etc. If you need a packaged substitute, try Larabars (or make your own!)
Try to avoid pastries and baked goods where possible
Occasionally ditch the tortillas and bread on your burritos and sandwiches
Try to fill your fridge/pantry with veggies, fruits, and other whole foods ahead of time; going to the grocery store each week and doing a little bit of planning helps prevent trips to fast food joints
You can ease into a diet by doing one meal at first, then two, then all your meals; for instance, start out your first week by substituting a green smoothie for your typical processed grain/bagel based breakfast. Then week two continue the morning green smoothie breakfast but also add a healthy whole food lunch (a salad with chicken and pear slices works great, or maybe a package of tuna with an apple and some carrots/almond butter). Eventually you can make dinner healthy too (vegetable stir fry, turkey and spinach omelette, etc)
The good news is that eating right typically leads to a virtuous cycle whereby the more you eat healthy, the less you will desire unhealthy foods and beverages. More and more you will notice the negative effects of eating processed junk on your energy levels and digestive system. If you cut soda consumption dramatically, and get most of your carbs from vegetables and fruits, you will likely soon find that the taste of soda is overpoweringly sweet. You will want to drink it even less. Then as you continue to lose weight, you will notice that your body is looking better. You will likely gain confidence and even more motivation to keep losing weight. You will be able to workout longer, you’ll sleep better (digestion can have dramatic effects on sleep), and you’ll have more energy during the day. Victory.
