You have most probably NOT harmed your battery. And yes, you can get by very well with just the wall wart as a charger, in fact your batteries will probably give you a longer life using it than you will get with a peak charger. As long as you are not in a rush to charge (takes about an overnight charge to fully charge with the wall wart) it is perfectly capable. More batteries are ruined with the use of fast chargers than were ever damaged using a wall wart.