DeepMind Uncovers Key to Boosting Transformer Architecture Accuracy: A Review of Non-Softmax Alternatives

DeepMind Uncovers Key to Boosting Transformer Architecture Accuracy: A Review of Non-Softmax Alternatives

DeepMind Uncovers Key to Boosting Transformer Architecture Accuracy: A Review of Non-Softmax Alternatives

As Seen On

In an era where the transformer architecture continues to revolutionize machine learning, Google DeepMind has made another substantial discovery aimed at enhancing the performance of these cognitive models. Their recent research centered on investigating the point-wise softmax alternatives in transformer architectures has unearthed a significant facet; splitting by sequence length is pivotal for precise accuracy.

The transformer architecture is at the heart of the most robust language models, like BERT and GPT-3. It offers a refreshing departure from the conventional focus on RNNs and CNNs. Among its most critical parts, attention relies on softmax for its functioning—therein arises a complication. The calculation complexity posed by softmax and the sum over sequence length, make parallelization an uphill task.

Past attempts at navigating this bottleneck have considered alternatives such as ReLU and squared ReLU to substitute softmax. But these substitutes, too, have their drawbacks, predominantly due to the absence of division by sequence length.

Google DeepMind’s recent study sheds light on the gravity of this division by sequence length. Their findings reveal a crucial link between the division by sequence length and achieving accuracy that aligns with the softmax. The research further outlines the implications of eliminating activation functions. While it aids longer sequence lengths, it tends to compromise on the accuracy.

To reach these findings, the researchers employed rigorous testing methods. They made use of the ImageNet-21k and ImageNet-1k datasets and training settings from BigVision source. The researchers were extremely specific with the parameters of the experiments, ensuring each aspect was under control to get the most accurate results.

What came out of these tests served to reinforce the hypotheses. The performance boost provided by the factor L^-1 was noticeable. Yet, in this new knowledge, several questions arise. What other more effective activation functions could serve the purpose? Why does the factor L^-1 improve performance? These are inquiries that continue to linger in the machine learning community and call for more investigation.

The strides made by the original researchers in this study are laudable, providing an avenue for future study and potential solutions for transformer architecture impediments. Keen on knowing more, you can access the research paper through this link. And if you also want to be part of these groundbreaking discussions, you can join various Machine Learning communities across different platforms.

In this exciting journey through the world of transformer architecture, an array of possibilities unrolls. The contributions from Google DeepMind continue to push the boundaries, shatter limitations, and look towards a future where the full potential of transformer models is entirely untapped. As we continue to delve deeper into transformer architecture, one realization remains consistent—innovation lies at the crux of machine learning progression.

Casey Jones Avatar
Casey Jones
9 months ago

Why Us?

  • Award-Winning Results

  • Team of 11+ Experts

  • 10,000+ Page #1 Rankings on Google

  • Dedicated to SMBs

  • $175,000,000 in Reported Client

Contact Us

Up until working with Casey, we had only had poor to mediocre experiences outsourcing work to agencies. Casey & the team at CJ&CO are the exception to the rule.

Communication was beyond great, his understanding of our vision was phenomenal, and instead of needing babysitting like the other agencies we worked with, he was not only completely dependable but also gave us sound suggestions on how to get better results, at the risk of us not needing him for the initial job we requested (absolute gem).

This has truly been the first time we worked with someone outside of our business that quickly grasped our vision, and that I could completely forget about and would still deliver above expectations.

I honestly can't wait to work in many more projects together!

Contact Us


*The information this blog provides is for general informational purposes only and is not intended as financial or professional advice. The information may not reflect current developments and may be changed or updated without notice. Any opinions expressed on this blog are the author’s own and do not necessarily reflect the views of the author’s employer or any other organization. You should not act or rely on any information contained in this blog without first seeking the advice of a professional. No representation or warranty, express or implied, is made as to the accuracy or completeness of the information contained in this blog. The author and affiliated parties assume no liability for any errors or omissions.