Yeah, if you didn't have a stick or something, you were able to use his horns to fight the spirits, but what's that going to do? 2% of damage? Ah, well, I still love ICO anyway. This game isn't for everyone so....
The secret weapon, btw, is a mace. It fights so much more better than the stick or the sword. Two or three swipes and the spirits are gone.
