Block caching and access modules

Previously I had a serious performance issue on my website.
The reason is that I have a big menu with 130+ menu links.
Content access module was breaking the block caching, and the website page loading was 4 times slower. It was a real head banging until I found what causes that issue.

I've removed the content access module and wrote a custom code for block caching.

Right now I've added a Advanced forum on the same website with many features.
Unfortunately I have to use Forum Access module, because real forum is nothing without permissions.

The same problem occurs here. If I enable the module that uses permission, it breaks the block caching, so the performance is again very slow.

Is there a way to avoid this breaking block caching with access modules in Drupal 7?