New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

question about norm_context #37

Open

XiaoyuShi97 opened this issue Aug 15, 2021 · 1 comment

XiaoyuShi97 commented Aug 15, 2021

Hi, thx for sharing the code! I wonder what norm_context refer to in the paper?

perceiver-pytorch/perceiver_pytorch/perceiver_io.py

Line 125 in 3b70ebe

    
           get_cross_attn = lambda: PreNorm(latent_dim, Attention(latent_dim, dim, heads = cross_heads, dim_head = cross_dim_head), context_dim = dim)

maximedb commented Sep 1, 2021 •

edited

Loading

See Annex C

In the cross-attention module, inputs are first processed with layer norm (Ba et al., 2016) before being passed through linear layers to produce each of the query, key, and value inputs to the QKV cross-attention operation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment