Mango LLMBoost for AMD MI300x

Mango LLMBoost is a containerized solution empowering LLM experts with tools to optimize their models and dynamically select the most suitable GPUs for their workloads. By fine-tuning and optimizing the LLM inference engine, Mango LLMBoost fully harnesses the parallelism of GPU cores and orchestrates inference jobs to maximize utilization across all available GPUs. Through quantization that utilizes the smaller data format without loss in accuracy, Mango LLMBoost further enhances efficiency by ensuring effective use of high-speed GPU caches and memory.