Natural language processing: Difference between revisions
| (3 intermediate revisions by the same user not shown) | |||
| Line 18: | Line 18: | ||
===Transformer=== | ===Transformer=== | ||
[https://arxiv.org/abs/1706.03762 Attention is all you need paper] | {{ main | Transformer (machine learning model)}} | ||
A neural network architecture by Google. | [https://arxiv.org/abs/1706.03762 Attention is all you need paper] | ||
It is currently the best at NLP tasks and has mostly replaced RNNs for these tasks. | |||
A neural network architecture by Google which uses encoder-decoder attention and self-attention. | |||
It is currently the best at NLP tasks and has mostly replaced RNNs for these tasks. | |||
However, it's computational complexity is quadratic in the number of input and output tokens due to attention. | |||
;Guides and explanations | |||
* [https://nlp.seas.harvard.edu/2018/04/03/attention.html The Annotated Transformer] | |||
* [https://www.youtube.com/watch?v=iDulhoQ2pro Youtube Video] | |||
===Google Bert=== | ===Google Bert=== | ||
| Line 30: | Line 37: | ||
A pretrained NLP neural network. | A pretrained NLP neural network. | ||
Note the code is written in TensorFlow 1. | Note the code is written in TensorFlow 1. | ||
====Albert==== | |||
[https://github.com/google-research/google-research/tree/master/albert Github]<br> | |||
;A Lite BERT for Self-supervised Learning of Language Representations | |||
This is a parameter reduction on Bert. | |||
==Libraries== | ==Libraries== | ||
===Apache OpenNLP=== | ===Apache OpenNLP=== | ||
[https://opennlp.apache.org/ Link] | [https://opennlp.apache.org/ Link] | ||