Accurate and robust calibration methods for plenoptic cameras are necessary due to their increasing popularity as depth estimation devices. Existing calibration methods can be placed in two categories: microlens based and sub-aperture image based. The low pixel-count in the microlenses is a limitation for microlens-based methods. On the other hand, calibrating each sub-aperture image independently leads to misalignments, resulting in poor depth estimates due to the high sensitivity on the baseline and alignment. In this paper we propose a hybrid calibration approach which can calibrate sub-aperture images while taking both main lens and microlens geometric constraints into account. Our model is simple yet effective and offers a high degree of interpretability on the intrinsic/extrinsic parameters of the virtual sub-aperture cameras. The experimental evaluation against existing methods shows that our proposed method leads to more accurate measurements in 3D.