panels panelizer caching

I'm currently moving a fairly large site off of a poor execution of Display Suite, and onto Panels, Panels Everywhere, and Panelizer.

The site has around 800 nodes, many many views, and lot's of different functionalities that make it heavy already. I'm concerned that moving to pure panels is going to slow it down even more.

With Panels caching abilities, how much is too much? Should I cache every view, field, pane, layout and variant? Would it be redundant to do that? In other words, is caching a panel pane doing the same work as caching each individual field inside the pane as well?