Salesforce Unleashes xGen-MM Multimodal Model as Open Source

Enterprise software developer Salesforce has recently unveiled a new open-source multimodal AI model, named xGen-MM (also known as BLIP-3), designed to process content across text, images, and other data types.

The Salesforce AI Research Institute detailed this 4B-sized model in a paper, highlighting its competitive performance in various benchmark tests compared to other models of similar size.

But why has Salesforce opted to open source its AI research? Researchers explain that open-sourcing models, alongside curated large datasets and fine-tuning codebases, can accelerate the development of large language models. This approach aims to disrupt the trend of keeping advanced AI models proprietary and could diversify the avenues through which multimodal AI technologies are accessed.

A key innovation of the xGen-MM model is its ability to handle intertwined data of multiple images and texts, which researchers describe as the most natural form of multimodal data.

This capability allows the model to undertake more complex tasks, such as answering questions that involve both images and text simultaneously, showcasing potential high practical value in fields like medical diagnostics and autonomous driving.

Salesforce has also released model variants optimized for different scenarios, including a basic pretrained model, an instruction-tuned model for following directives, and a safety-tuned model designed to minimize harmful outputs.

Developers can choose from these variants based on their needs, whether using the basic pretrained model without further adjustments or fine-tuning the instruction-tuned model for specific domains.

The xGen-MM model's code, model, and datasets are now available in Salesforce's GitHub repository, with more resources to follow on the project website for developers to download.

Salesforce xGen-MM: https://github.com/salesforce/LAVIS/tree/xgen-mm

Salesforce Huggingface: https://huggingface.co/Salesforce

Project Website: http://xgen-mm.salesforceairesearch.com/