Mistral AIの「Pixtral 12B」「Pixtral Large」とは？手書きから複雑データまで対応するマルチモーダルAI

curl https://api.mistral.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your_api_key" \
  -d '{
    "model": "pixtral-12b-2409",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What’s in this image?"
          },
          {
            "type": "image_url",
            "image_url": "https://tripfixers.com/wp-content/uploads/2019/11/eiffel-tower-with-snow.jpeg"
          }
        ]
      }
    ],
    "max_tokens": 300
  }'

{"id":"","object":"chat.completion","created":1732177009,"model":"pixtral-12b-2409","choices":[{"index":0,"message":{"role":"assistant","content":"The image portrays a serene winter scene with the Eiffel Tower prominently featured in the background. The tower is partially covered in snow, and the surrounding area is blanketed with a fresh layer of snow, which is evident on the trees and the ground. There's a pathway leading towards the Eiffel Tower, and a classic Parisian lamppost can be seen in the foreground.","tool_calls":null},"finish_reason":"stop"}],"usage":{"prompt_tokens":1359,"total_tokens":1439,"completion_tokens":80}}%  

curl https://api.mistral.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your_api_key" \
  -d '{
    "model": "pixtral-large-2411",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What’s in this image?"
          },
          {
            "type": "image_url",
            "image_url": "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png"
          }
        ]
      }
    ],
    "max_tokens": 300
  }'

{"id":"59a27dfcbdfb4ab18793972a7909d6ca","object":"chat.completion","created":1732183707,"model":"pixtral-large-2411","choices":[{"index":0,"message":{"role":"assistant","content":"The image depicts a stunning natural landscape featuring a river flowing through a valley surrounded by towering cliffs and mountains. The river has clear water with visible rocks and grassy patches along its banks. The surrounding area is lush with greenery, including trees and shrubs, indicating a healthy and vibrant ecosystem. The sky is clear with a few scattered clouds, suggesting a pleasant day. The overall scene is serene and picturesque, showcasing the beauty of nature. This type of landscape is characteristic of national parks, such as Yosemite National Park in the United States.","tool_calls":null},"finish_reason":"stop"}],"usage":{"prompt_tokens":2805,"total_tokens":2927,"completion_tokens":122}}%      

!pip install mistral-inference mistral-common pillow requests

from huggingface_hub import snapshot_download
from pathlib import Path

# モデルデータを保存するディレクトリ
mistral_models_path = Path.home().joinpath('mistral_models', 'Pixtral')
mistral_models_path.mkdir(parents=True, exist_ok=True)

# Hugging Faceからモデルデータをダウンロード
snapshot_download(
    repo_id="mistralai/Pixtral-12B-Base-2409",
    allow_patterns=["params.json", "consolidated.safetensors", "tekken.json"],
    local_dir=mistral_models_path
)

from mistral_inference.transformer import Transformer
from mistral_inference.generate import generate

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.protocol.instruct.messages import TextChunk, ImageURLChunk
from PIL import Image
from io import BytesIO
import requests

# モデルとトークナイザーをロード
tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tekken.json")
model = Transformer.from_folder(mistral_models_path)

# 入力画像URL
url = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png"
response = requests.get(url)
image = Image.open(BytesIO(response.content))
image.show()  # 画像をColabで表示する場合

# プロンプト設定
prompt = "The above image presents a"
user_content = [ImageURLChunk(image_url=url), TextChunk(text=prompt)]

# トークン化
tokens, images = tokenizer.instruct_tokenizer.encode_user_content(user_content, False)

# モデルによる生成
out_tokens, _ = generate(
    [tokens],
    model,
    images=[images],
    max_tokens=256,
    temperature=0.35,
    eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id,
)

# 結果をデコードして表示
result = tokenizer.decode(out_tokens[0])
print("Prompt:", prompt)
print("Completion:", result)

Prompt: The above image presents a
Completion:  view of the Merced River in Yosemite Valley, with El Capitan on the left and Cathedral Rocks on the right. The image was captured using a Canon EOS 5D Mark II camera with a Canon EF 24-105mm f/4L IS USM lens. The camera settings used were a shutter speed of 1/200 second, an aperture of f/11, and an ISO of 100. The focal length was 24mm, and the image was taken at a focal length of 24mm. The image was processed using Adobe Lightroom 4.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Ice Cream Flavor Picker</title>
    <style>
        body {
            font-family: Arial, sans-serif;
        }
        .container {
            width: 300px;
            margin: 0 auto;
            border: 1px solid #000;
            padding: 20px;
            text-align: center;
        }
        .flavor-list {
            list-style-type: none;
            padding: 0;
        }
        .flavor-list li {
            margin: 10px 0;
        }
        .next-button {
            margin-top: 20px;
        }
        .copyright {
            font-size: 12px;
            margin-top: 20px;
        }
    </style>
</head>
<body>
    <div class="container">
        <h1>Pick An Ice Cream Flavor</h1>
        <ul class="flavor-list">
            <li>
                <label>
                    <input type="radio" name="flavor" value="vanilla"> Vanilla
                </label>
            </li>
            <li>
                <label>
                    <input type="radio" name="flavor" value="chocolate"> Chocolate
                </label>
            </li>
            <li>
                <label>
                    <input type="radio" name="flavor" value="strawberry"> Strawberry
                </label>
            </li>
            <li>
                <label>
                    <input type="radio" name="flavor" value="mint"> Mint
                </label>
            </li>
        </ul>
        <div class="next-button">
            <button>Next</button>
        </div>
        <div class="copyright">
            © copyright Mistral AI
        </div>
    </div>
</body>
</html>