I own a Lexus, and I absolutely hate it when people say that a Lexus is just a Toyota. For most Lexus models, that's simply not the case.
.......but even I have to admit that the ES350 is the one model where the Toyota = Lexus crowd has a point. The ES350 shares the same platform with the Camry, which means the underpinnings are the same.
http://www.vehix.com/car-reviews/2012/lexus/es-350
I drove an ES350 wanting to like it, but it was boring as hell. Drove just like a Camry.....go figure.
I ended up buying an IS, which doesn't share a platform with any Toyota. An IS (or any other Lexus, for that matter) is uniquely a Lexus - an ES is not.
My advice: if you're having second thoughts now, you'll hate the car two months after you buy it. Getting a new car is about love and excitement. If you don't love it when you buy it, it only goes downhill from there. Pass.
Good luck!