Transformer

Technology

• NVIDIA optimizes dropless MoE model training in JAX: NVIDIA has introduced optimizations for training dropless Mixture of Experts (MoE) models in JAX using the NVIDIA Transformer Engine. The update addresses communication and computation bottlenecks associated with dropless MoE routing.

  Infrastructure • NVIDIA Generative AI

  Permalink