Problem solved

peterliang hit the jackpot! I found earlier, that option and was going to post the solution but since peterliang already did that i just add were to look for it if someone has simmilar problem.
In mental ray settings (after pressing F10) there's Processing and there the memory usage which was deflaut 650MB i changed it to 1024MB and also marked the Conserving memory check box. It helped

Stey tuned for the final render of the new animation. Thx for the input all