MOAI: Module-Optimizing Architecture for non-interactive secure transformer Inference

Summary

MOAI is a homomorphic encryption framework for non-interactive secure transformer inference, so LLM inference can run directly over encrypted inputs in cloud deployments. It combines column and diagonal packing across all layers and uses rotation-free algorithms for Softmax and LayerNorm, removing 2,448 HE rotations in BERT-base inference. The authors report a 52.8% reduction in evaluation time versus THOR and a 55.7% reduction versus Powerformer, at 2.36 minutes per input on a single GPU, and the implementation is open source.