【PaliGemma】画像を理解できるGoogleのオープン言語モデル

from transformers import AutoProcessor, PaliGemmaForConditionalGeneration

model_id = "google/paligemma-3b-mix-224"
model = PaliGemmaForConditionalGeneration.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

prompt = "What is on the flower?"
image_file = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=true"
raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(prompt, raw_image, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=20)

print(processor.decode(output[0], skip_special_tokens=True)[len(prompt):])

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
model = PaligemmaForConditionalGeneration.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map={"":0}
)

Caption

A fluffy gray and white cat sits peacefully in the grass, its eyes wide open. The cat's fur is soft and silky, and its eyes are bright and full of life. The flowers in the garden bloom in a variety of colors, including pink, white, and red. The cat's ears are perked up, and its tail is swiped across the grass. The cat's eyes are full of wonder, and its nose is wrinkled in concentration.

In which city is this?

tokyo

detect tower

What does this image show?

our model pipeline.

Caption

In this image we can see a castle, trees, flowers and lights.

In a magical fairy tale land, a grand castle stands majestically amidst a forest of trees adorned with enchanting flowers and illuminated by fantastical lights, creating a vibrant and dreamlike atmosphere.

How many people are in the image?

13

The image shows a group of 13 people.

Tell me what you can learn from this table.

performance of different models on the different datasets.

The table compares the performance of different AI models (LLaMA-2, Mistral, and Gemma) on various benchmarks. Here are the key insights:

1. **MMLU (5-shot, top-1)**: Gemma 7B has the highest score at 64.3, outperforming all other models.
   
2. **HellaSwag (0-shot)**: Mistral 7B and Gemma 7B perform equally well, both scoring 81.2.

3. **PIQA (0-shot)**: Mistral 7B leads with 82.2, with Gemma 7B close behind at 81.2.

4. **SIQA (0-shot)**: Mistral 7B and Gemma 7B show comparable performance, with Mistral 7B at 47.0 and Gemma 7B at 51.8.

5. **BoolQ (0-shot)**: Mistral 7B and Gemma 7B excel with scores of 83.2 each.

6. **Winogrande (partial scoring)**: Mistral 7B leads with 74.2, followed by Gemma 7B at 72.3.

7. **CQA (7-shot)**: Mistral 7B scores 66.3, while Gemma 7B leads with 71.3.

8. **OBQA (0-shot)**: LLaMA-2 7B has the highest score at 58.6, with Gemma 7B at 52.8.

9. **ARC-e**: Mistral 7B leads with 80.5, with Gemma 7B at 81.5.

10. **ARC-c**: Mistral 7B scores highest at 54.9, with Gemma 7B at 53.2.

11. **TriviaQA (5-shot)**: Mistral 7B has the highest score at 79.6, with Gemma 7B at 63.4.

12. **NQ (5-shot)**: Mistral 7B scores 31.2, while Gemma 7B is at 23.0.

13. **HumanEval (pass@1)**: Gemma 7B significantly outperforms others with a score of 32.3.

14. **MBPP (3-shot)**: Mistral 7B performs best at 40.2, with Gemma 7B at 44.4.

15. **GSM8K (maj@1)**: Mistral 7B scores highest at 35.4, with Gemma 7B at 46.4.

16. **MATH (4-shot)**: Gemma 7B leads with 24.3.

17. **AGIEval**: Mistral 7B scores highest at 41.2, with Gemma 7B at 41.7.

18. **BBH**: Mistral 7B scores highest at 56.1, with Gemma 7B at 55.1.

19. **Average**: Gemma 7B has the highest average score at 56.4, indicating overall strong performance across benchmarks.

**Summary**:
- **Best Overall**: Gemma 7B, with the highest average score (56.4).
- **Notable Performances**: Gemma 7B excels in MMLU, HumanEval, GSM8K, and MATH benchmarks.
- **Competitive Models**: Mistral 7B performs strongly in several benchmarks, often leading or closely following Gemma 7B. 

Overall, Gemma 7B shows strong performance, leading in several benchmarks and having the highest average score, indicating it is a highly effective model across various tasks.