Highly immersive 3D holographic displays require extremely large pixel counts, creating a data transmission bottleneck that necessitates the development of efficient compression frameworks. We show that digital holograms can be compressed to below 1 bit per thousand pixels by encoding only the magnitudes of numerically reconstructed, lower-resolution images at multiple depths. Due to their image-like nature, standard codecs can be used to store such reconstructions at very low bitrates. Our gradient-descent decoder then retrieves a hologram whose propagated intensities best match this focal stack. Experiments show accurate depth and parallax reconstruction while outperforming existing hologram compression methods.