Bin-wise Histogram和Frequency-wise Histogram区别

[chensming]

您好，我对您的论文很感兴趣。但是我在阅读过程中有个地方不是很理解，就是Bin-wise Histogram和Frequency-wise Histogram的区别是什么，我看代码好像对这两个处理差不多呀，能麻烦您解释一下么

[sunshangquan]

Hi @chensming ，感谢提问，打个比方，对一个序列[1,2,3,4,5,6,7,8,9,10]，假设B=2，他们二者的区别就是，Bin-wise会把序列分为[[1,2,3,4,5],[6,7,8,9,10]]，然后做[1,2,3,4,5]和[6,7,8,9,10]之间的自注意力；而Frequency-wise会计算[1,2],[3,4],[5,6],[7,8],[9,10]之间的自注意力

[chensming]

您好，您的意思是out1是Bin-wise的结果，out2是Frequency-wise的结果么

Histoformer/basicsr/models/archs/histoformer_arch.py

Lines 184 to 190 in 1f045f0

	out1 = self.reshape_attn(q1, k1, v, True)
	out2 = self.reshape_attn(q2, k2, v, False)
	out1 = torch.scatter(out1, 2, idx, out1).view(b,c,h,w)
	out2 = torch.scatter(out2, 2, idx, out2).view(b,c,h,w)
	out = out1 * out2
	out = self.project_out(out)

假如是的话，那么我有点儿不太能理解的是，为什么只有origin shape不同，就可以实现这个效果呢 ? 提前感谢您的解答。^_^

Histoformer/basicsr/models/archs/histoformer_arch.py

Lines 150 to 168 in 1f045f0

	def reshape_attn(self, q, k, v, ifBox):
	b, c = q.shape[:2]
	q, t_pad = self.pad(q, self.factor)
	k, t_pad = self.pad(k, self.factor)
	v, t_pad = self.pad(v, self.factor)
	hw = q.shape[-1] // self.factor
	shape_ori = "b (head c) (factor hw)" if ifBox else "b (head c) (hw factor)"
	shape_tar = "b head (c factor) hw"
	q = rearrange(q, '{} -> {}'.format(shape_ori, shape_tar), factor=self.factor, hw=hw, head=self.num_heads)
	k = rearrange(k, '{} -> {}'.format(shape_ori, shape_tar), factor=self.factor, hw=hw, head=self.num_heads)
	v = rearrange(v, '{} -> {}'.format(shape_ori, shape_tar), factor=self.factor, hw=hw, head=self.num_heads)
	q = torch.nn.functional.normalize(q, dim=-1)
	k = torch.nn.functional.normalize(k, dim=-1)
	attn = (q @ k.transpose(-2, -1)) * self.temperature
	attn = self.softmax_1(attn, dim=-1)
	out = (attn @ v)
	out = rearrange(out, '{} -> {}'.format(shape_tar, shape_ori), factor=self.factor, hw=hw, b=b, head=self.num_heads)
	out = self.unpad(out, t_pad)
	return out

note：里面有个arange函数，我尝试写了个例子

import torch
from einops import rearrange


x = torch.arange(36).reshape(1, 2, 18)
shape_ori = "b (head c) (factor hw)" 
shape_ori2 = "b (head c) (hw factor)"
shape_tar = "b head (c factor) hw"

y = rearrange(x, '{} -> {}'.format(shape_ori, shape_tar), factor=2, hw=9, head=1)
y2 = rearrange(x, '{} -> {}'.format(shape_ori2, shape_tar), factor=2, hw=9, head=1)

print(x)
print(y)
print(y2)

"""
tensor([[[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
          17],
         [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
          35]]])
tensor([[[[ 0,  1,  2,  3,  4,  5,  6,  7,  8],
          [ 9, 10, 11, 12, 13, 14, 15, 16, 17],
          [18, 19, 20, 21, 22, 23, 24, 25, 26],
          [27, 28, 29, 30, 31, 32, 33, 34, 35]]]])
tensor([[[[ 0,  2,  4,  6,  8, 10, 12, 14, 16],
          [ 1,  3,  5,  7,  9, 11, 13, 15, 17],
          [18, 20, 22, 24, 26, 28, 30, 32, 34],
          [19, 21, 23, 25, 27, 29, 31, 33, 35]]]])
"""

[sunshangquan]

昨天的话有一点歧义，Frequency-wise会计算[1,2],[3,4],[5,6],[7,8],[9,10]每个张量内部的自注意力

B设置用的是变量factor，利用刚刚的例子，如果一个HW是5、B是2，那在L156，Bin-wise的张量会是（batch，headc，2, 5）大小的[1,2,3,4,5],[6,7,8,9,10]（忽略前两个维度），计算的是两个张量之间自注意力；Frequency则是（batch，headc，5，2），数据形式比如说是[1,2],[3,4],[5,6],[7,8],[9,10]，计算的是每个[1,2]这样的张量内部的自注意力

Note：arange也可以达到这样的效果

[chensming]

感谢的您的回复，所以说Bin-wise就是计算两个Bin之间的注意力，frequency-wise就是计算Bin内部个像素点两两间的注意力么（为什么叫frequency-wise呢好奇，是有什么出处还是说只是个命名呢）

[sunshangquan]

嗯嗯嗯是的，因为整个想法是基于Histogram把像素进行排序分区的机制来的，直方图Histogram里面分区的名字叫Bin，Bin内的频数（像素数量）叫frequency，所以用了这个命名

[chensming]

嗯嗯嗯是的，因为整个想法是基于Histogram把像素进行排序分区的机制来的，直方图Histogram里面分区的名字叫Bin，Bin内的频数（像素数量）叫frequency，所以用了这个命名

好的，我了解了，谢谢~

[chensming]

不好意思，我再确认一下, 这个是Bin-wise么, 非常感谢您的回答

Histoformer/basicsr/models/archs/histoformer_arch.py

Line 184 in 1f045f0

out1 = self.reshape_attn(q1, k1, v, True)