When students tell me that as they leave my yoga class, I'm always a little surprised. In yoga, the workout is inherent, but it's hardly the point. Yoga isn't about sweating or burning calories. It's not a weight-lifting regimen that gets you ripped so you can impress everybody at the beach. It's not high-impact cardio that leaves you panting for air. Yoga is about connecting your physical and mental states to get to that higher level of health. That's why I tell my classes that instead of focusing on the workout, they should avoid the workout mentality.
Yoga and working out are both physical, of course, but the biggest thing that takes yoga to that next level is your intention. If you're consumed with the idea of "working out," burning calories and getting skinny, you're sacrificing your mind for your body. Why not engage both?