Microsoft continued to slim down AI language models, unveiling its third lightweight edition which it stated offers comparable performance to rival products while being suitable for use on mobile phones.
The software giant’s Phi-3 mini offers 3.8 billion parameters trained on 3.3 trillion tokens. It builds on Microsoft Research’s Phi-2 model released in late 2023, which offered 2.7 billion parameters.
Microsoft stated the performance of Phi-3 mini was assessed against academic benchmarks and internal testing, work which proved its ability to rival models including “Mixtral 8x7B and GPT-3.5”.
It said the latest lightweight model achieved a rating of 69 per cent in massive multitask language understanding (MMLU) and 8.38 in multi-turn (MT) benchmarks.
Microsoft explained it scaled up dataset training used to develop Phi-2, employing “heavily filtered” web and synthetic data in its latest edition.
The company conceded Phi-3 mini is “fundamentally limited by its size for certain tasks”, despite its LLM-rivalling capabilities. “The model simply does not have the capacity to store too much factual knowledge”, though Microsoft asserted “such weakness can be resolved by augmentation with a search engine”.
It emphasised the model was developed “in accordance with Microsoft’s responsible AI principles”, with input from various internal teams used to curate “additional datasets tailored to address their insights”. The company said the approach resulted in a “significant decrease of harmful response rates”.
Reuters reported Phi-3 mini is already available through Microsoft’s Azure cloud platform.
Comments