Have you removed the valve rockers and pushrods yet? If not, there is still tension on the lifters from the valve springs. If the engine has been sitting for some time, you could have dried oil on the cams which would prevent them from easily slipping off the lobe and make removal seem difficult. Also, you state you have removed the four front housing bolts. Have you not yet removed the four bolts on the rear of the cam housing that attach it to the rear of the engine?

John