🎄Happy holidays and we wish you enjoy this year. Before moving to 2025, Qwen has the last gift for you, which is QVQ!
🎉 This may be the first open-weight model for visual reasoning. It is called QVQ, where V stands for vision. It just reads an image and an instruction, starts… pic.twitter.com/BX1ORiltIf
# device_mapを使用せずにモデルを読み込む
model = Qwen2VLForConditionalGeneration.from_pretrained(
"Qwen/QVQ-72B-Preview",
torch_dtype=torch.bfloat16, # メモリ使用量を削減
low_cpu_mem_usage=False
)
if torch.cuda.is_available():
model = model.to("cuda")
processor = AutoProcessor.from_pretrained("Qwen/QVQ-72B-Preview")
サンプルコードの実行はこちら
def process_image(image_path, question):
# Prepare messages
messages = [
{
"role": "system",
"content": [
{"type": "text", "text": "You are a helpful and harmless assistant."}
],
},
{
"role": "user",
"content": [
{
"type": "image",
"image": image_path
},
{"type": "text", "text": question}
],
}
]
# Prepare inputs
text = processor.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt"
)
inputs = inputs.to("cuda")
# Generate response
generated_ids = model.generate(**inputs, max_new_tokens=8192)
generated_ids_trimmed = [
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed,
skip_special_tokens=True,
clean_up_tokenization_spaces=True
)
return output_text[0]
# Example usage
if __name__ == "__main__":
image_path = "Math_2.jpg"
question = "Evaluate the integral of the functions graphed using the formula for circles:"
response = process_image(image_path, question)
print(response)
結果はこちら
ile /opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py:1736, in Module._wrapped_call_impl(self, *args, **kwargs)
1734 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]
1735 else:
-> 1736 return self._call_impl(*args, **kwargs)
File /opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py:1747, in Module._call_impl(self, *args, **kwargs)
1742 # If we don't have any hooks, we want to skip the rest of the logic in
1743 # this function, and just call forward.
1744 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1745 or _global_backward_pre_hooks or _global_backward_hooks
1746 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1747 return forward_call(*args, **kwargs)
1749 result = None
1750 called_always_called_hooks = set()
File /opt/conda/lib/python3.11/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py:394, in VisionSdpaAttention.forward(self, hidden_states, cu_seqlens, rotary_pos_emb)
390 def forward(
391 self, hidden_states: torch.Tensor, cu_seqlens: torch.Tensor, rotary_pos_emb: torch.Tensor = None
392 ) -> torch.Tensor:
393 seq_length = hidden_states.shape[0]
--> 394 q, k, v = self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
395 q = apply_rotary_pos_emb_vision(q.unsqueeze(0), rotary_pos_emb).squeeze(0)
396 k = apply_rotary_pos_emb_vision(k.unsqueeze(0), rotary_pos_emb).squeeze(0)
File /opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py:1736, in Module._wrapped_call_impl(self, *args, **kwargs)
1734 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]
1735 else:
-> 1736 return self._call_impl(*args, **kwargs)
File /opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py:1747, in Module._call_impl(self, *args, **kwargs)
1742 # If we don't have any hooks, we want to skip the rest of the logic in
1743 # this function, and just call forward.
1744 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1745 or _global_backward_pre_hooks or _global_backward_hooks
1746 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1747 return forward_call(*args, **kwargs)
1749 result = None
1750 called_always_called_hooks = set()
File /opt/conda/lib/python3.11/site-packages/torch/nn/modules/linear.py:125, in Linear.forward(self, input)
124 def forward(self, input: Tensor) -> Tensor:
--> 125 return F.linear(input, self.weight, self.bias)
OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacity of 79.21 GiB of which 33.50 MiB is free. Process 239083 has 79.17 GiB memory in use. Of the allocated memory 78.38 GiB is allocated by PyTorch, and 118.90 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Let's tackle this problem step by step. So, we have a sequence of distinct positive integers \(a_1, a_2, \ldots, a_n\), and a set \(M\) consisting of \(n-1\) positive integers, none of which is equal to the sum \(s = a_1 + a_2 + \ldots + a_n\). We have a frog starting at 0 on the number line, and it makes \(n\) jumps towards the positive direction, where each jump's length is one of the integers \(a_1, a_2, \ldots, a_n\), in some order. The task is to prove that we can choose the order of these jumps such that the frog never lands on any point in \(M\) during its jumps.
First, let's understand the problem better. We have a frog that starts at 0 and makes \(n\) jumps, each of length equal to one of the \(a_i\)'s, in some order. We need to permute the sequence \(a_1, a_2, \ldots, a_n\) so that the cumulative sum at any point (after each jump) is not equal to any element in \(M\).
Given that \(M\) has \(n-1\) elements and none of them is equal to \(s\), the sum of all \(a_i\)'s, we need to ensure that none of the partial sums match any element in \(M\).
Let's consider the sequence of positions the frog occupies after each jump. Let's denote the permutation of jumps as \(b_1, b_2, \ldots, b_n\), where each \(b_i\) is one of the \(a_j\)'s. Then, the positions after each jump are:
\[
p_1 = b_1, \quad p_2 = p_1 + b_2, \quad p_3 = p_2 + b_3, \quad \ldots, \quad p_n = p_{n-1} + b_n
\]
Our goal is to choose the permutation \((b_1, b_2, \ldots, b_n)\) such that none of the \(p_k\) (\(k = 1, 2, \ldots, n-1\)) are in \(M\).
Since \(M\) has \(n-1\) elements and there are \(n-1\) possible landing points (\(p_1, p_2, \ldots, p_{n-1}\)), if we can find a permutation where these points do not coincide with \(M\), we're done.
Let's think about the possible sums that the frog can land on. Each \(p_k\) is the sum of the first \(k\) jumps in the permutation. So, \(p_k = b_1 + b_2 + \ldots + b_k\), for \(k = 1\) to \(n-1\).
Given that all \(a_i\) are distinct and positive integers, and \(M\) is a set of \(n-1\) distinct positive integers not equal to \(s\), we need to arrange the jumps so that no partial sum matches any element in \(M\).
One approach could be to consider all possible permutations of the jumps and see if at least one avoids landing on \(M\). However, with \(n\) jumps, there are \(n!\) permutations, which is a large number, and checking each one isn't practical. We need a smarter way.
Let's consider the properties of the sums. Since all \(a_i\) are distinct and positive, the order in which jumps are taken affects the path the frog takes, potentially allowing us to avoid landing on \(M\).
Another angle: perhaps we can construct a permutation where the partial sums avoid \(M\). For example, if we order the jumps in a way that the partial sums are either all less than the smallest element in \(M\) or all greater than the largest element in \(M\), then we're safe.
However, since \(M\) contains \(n-1\) elements, and the frog makes \(n-1\) jumps, it's possible that with careful selection, we can avoid landing on any of them.
Wait a minute, let's think about the problem in terms of graph theory or some other framework, but that might be overcomplicating it.
Let's try a smaller example to get some intuition.
Suppose \(n = 3\), so we have three distinct positive integers \(a_1, a_2, a_3\), and \(M\) has \(n-1 = 2\) elements, neither of which is equal to \(s = a_1 + a_2 + a_3\).
We need to permute the jumps \(a_1, a_2, a_3\) such that the intermediate sums \(p_1\) and \(p_2\) do not equal any element in \(M\).
Let's say \(M = \{m_1, m_2\}\), with \(m_1 < m_2\), and \(s > m_2\).
We need to choose the order of jumps so that neither \(b_1\) nor \(b_1 + b_2\) equals \(m_1\) or \(m_2\).
Given that there are only two elements in \(M\) and three possible jumps, it's plausible that we can find a permutation where the partial sums avoid \(M\).
For example, if \(a_1 = 1\), \(a_2 = 2\), \(a_3 = 4\), and \(M = \{3, 5\}\), then \(s = 7\), which is not in \(M\).
Possible permutations:
1. \(1, 2, 4\): positions \(1, 3, 7\) → lands on 3, which is in \(M\).
2. \(1, 4, 2\): positions \(1, 5, 7\) → lands on 5, which is in \(M\).
3. \(2, 1, 4\): positions \(2, 3, 7\) → lands on 3.
4. \(2, 4, 1\): positions \(2, 6, 7\) → does not land on 3 or 5.
5. \(4, 1, 2\): positions \(4, 5, 7\) → lands on 5.
6. \(4, 2, 1\): positions \(4, 6, 7\) → does not land on 3 or 5.
So, in this case, permutations 4 and 6 avoid landing on \(M\).
This suggests that for \(n = 3\), it's possible to find a permutation that avoids \(M\).
Now, let's try \(n = 4\), \(a_1 = 1\), \(a_2 = 2\), \(a_3 = 4\), \(a_4 = 8\), \(M = \{3, 5, 6\}\), \(s = 15\).
We need to permute \(1, 2, 4, 8\) such that \(p_1, p_2, p_3\) do not equal 3, 5, or 6.
Let's consider permutation \(1, 2, 4, 8\): positions \(1, 3, 7, 15\) → lands on 3.
Permutation \(1, 4, 2, 8\): positions \(1, 5, 7, 15\) → lands on 5 and 7.
Permutation \(2, 1, 4, 8\): positions \(2, 3, 7, 15\) → lands on 3.
Permutation \(2, 4, 1, 8\): positions \(2, 6, 7, 15\) → lands on 6 and 7.
Permutation \(4, 1, 2, 8\): positions \(4, 5, 7, 15\) → lands on 5 and 7.
Permutation \(4, 2, 1, 8\): positions \(4, 6, 7, 15\) → lands on 6 and 7.
Permutation \(8, 1, 2, 4\): positions \(8, 9, 11, 15\) → does not land on 3, 5, or 6.
Permutation \(8, 2, 1, 4\): positions \(8, 10, 11, 15\) → does not land on 3, 5, or 6.
Permutation \(8, 4, 1, 2\): positions \(8, 12, 13, 15\) → does not land on 3, 5, or 6.
Permutation \(8, 4, 2, 1\): positions \(8, 12, 14, 15\) → does not land on 3, 5, or 6.
So, in this case, multiple permutations avoid landing on \(M\).
This suggests that for \(n = 4\), it's also possible to find a permutation that avoids \(M\).
Is there a general pattern or theorem that can help us prove this for any \(n\)?
Let's consider the problem in terms of graph theory. We can think of the partial sums as paths in a graph where each node represents a position on the number line, and edges represent jumps of length equal to one of the \(a_i\)'s.
However, this might not be the most straightforward approach.
Another idea: since all \(a_i\) are distinct and positive, and \(M\) has \(n-1\) elements, none of which is equal to \(s\), perhaps we can use the pigeonhole principle.
But the pigeonhole principle typically tells us that if we have more pigeons than holes, at least one hole has more than one pigeon. In this case, we have \(n-1\) landing points and \(n-1\) elements in \(M\), so it's possible that some landing points coincide with \(M\), but not necessarily all.
Wait, but we need to ensure that none of the landing points coincide with \(M\).
Perhaps we can consider the differences between the partial sums and the elements in \(M\), but I'm not sure.
Let's think differently. Suppose we fix an order of jumps, say \(a_1, a_2, \ldots, a_n\), leading to partial sums \(p_k = a_1 + a_2 + \ldots + a_k\) for \(k = 1\) to \(n-1\).
We need none of these \(p_k\) to be in \(M\).
Given that \(M\) has \(n-1\) elements, and there are \(n-1\) partial sums, if we can find a permutation where these partial sums are distinct and none equals any element in \(M\), we're done.
But are the partial sums necessarily distinct for some permutation?
Not necessarily, since different permutations can lead to the same partial sums.
Wait, but since all \(a_i\) are distinct, the partial sums in a given permutation are distinct because each \(a_i\) is unique.
So, in any permutation, the sequence of partial sums \(p_1, p_2, \ldots, p_{n-1}\) are distinct.
Given that, and since \(M\) has \(n-1\) elements, none of which is equal to \(s\), perhaps we can choose a permutation where the partial sums avoid \(M\).
But how can we guarantee that?
Let's consider the set of all possible partial sums across all permutations.
Each permutation gives a unique sequence of partial sums, but these sequences overlap in terms of the actual sums.
Wait, no. Each permutation gives a unique sequence of partial sums, but the sums themselves can repeat across different permutations.
For example, in the \(n=3\) case with \(a_1=1\), \(a_2=2\), \(a_3=4\), the partial sums for permutation \(1,2,4\) are \(1,3\), and for \(2,1,4\) are \(2,3\), so both have 3 as a partial sum.
Similarly, for \(1,4,2\): \(1,5\); for \(4,1,2\): \(4,5\); for \(2,4,1\): \(2,6\); for \(4,2,1\): \(4,6\).
So, some partial sums can appear in multiple permutations.
Given that, perhaps we can find a permutation where the partial sums don't coincide with \(M\).
But this seems too vague.
Let's consider the problem in terms of avoiding specific values.
We need to choose a permutation such that for each \(k\) from 1 to \(n-1\), \(p_k \neq m\) for any \(m \in M\).
Given that \(M\) has \(n-1\) elements, and there are \(n!\) possible permutations, it's likely that at least one permutation exists where none of the partial sums match any element in \(M\).
But how to prove it?
Perhaps we can use induction on \(n\).
Base case: \(n=2\). Then \(M\) has \(n-1=1\) element, say \(m\), and \(s = a_1 + a_2\), with \(s \neq m\).
We need to choose the order of jumps \(a_1, a_2\) such that neither \(a_1\) nor \(a_1 + a_2\) equals \(m\).
Since \(s \neq m\), only \(p_1 = a_1\) needs to avoid \(m\).
If \(a_1 \neq m\), choose the permutation starting with \(a_1\); otherwise, start with \(a_2\).
Since \(a_1\) and \(a_2\) are distinct and positive, and \(m\) is a single element not equal to \(s\), at least one of them is not equal to \(m\).
Thus, the base case holds.
Inductive step: Assume the statement is true for some \(n \geq 2\), and prove it for \(n+1\).
Given \(n+1\) distinct positive integers \(a_1, a_2, \ldots, a_{n+1}\), and a set \(M\) of \(n\) distinct positive integers, none equal to \(s = a_1 + a_2 + \ldots + a_{n+1}\).
We need to find a permutation of jumps such that none of the partial sums \(p_1, p_2, \ldots, p_n\) equals any element in \(M\).
Let's fix one of the \(a_i\), say \(a_{n+1}\), and consider the remaining \(n\) jumps \(a_1, a_2, \ldots, a_n\).
By the inductive hypothesis, there exists a permutation of these \(n\) jumps such that none of the partial sums \(p_1, p_2, \ldots, p_{n-1}\) equals any element in \(M'\), where \(M'\) is the set \(M\) minus \(s - a_{n+1}\), assuming \(s - a_{n+1}\) is not in \(M\).
Wait, this seems complicated.
Alternatively, perhaps we can consider the problem in terms of graph theory, where each permutation corresponds to a path in a graph, and we need to find a path that avoids certain nodes.
But that might not lead us anywhere.
Let's try another approach.
Consider the set of all possible partial sums across all permutations.
Each partial sum is of the form \(p_k = b_1 + b_2 + \ldots + b_k\), where \(b_i\) is a permutation of \(a_i\).
Since all \(a_i\) are distinct, each partial sum is unique within a permutation, but across permutations, some sums can repeat.
Our goal is to choose a permutation where none of the partial sums coincide with \(M\).
Given that \(M\) has \(n-1\) elements and there are \(n-1\) partial sums, if we can ensure that the partial sums are distinct and none equals any element in \(M\), we're done.
But how to guarantee that?
Perhaps we can use the fact that the \(a_i\) are distinct and positive, and \(M\) has \(n-1\) elements, none equal to \(s\).
Let's consider the set \(S\) of all possible partial sums across all permutations.
Each partial sum is a sum of some subset of \(a_i\), but since we're considering permutations, each partial sum corresponds to a unique sequence.
Wait, no. Each
