BERT - encoder only

GPT - decoder only

Pre-train and fine-tuning

Raychev, V., Bielik, P., and Vechev, M. Probabilistic Model for Code with Decision Trees. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (2016), OOPSLA ’16, ACM

semantic of code (like the operator of Python, the intepreter)

formal of code (like AST, grammar)

GPT-3

Codex

Formal verification of neural network

Introduction to Neural Network Verification; Aws Albarghouthi

https://arxiv.org/pdf/2109.10317.pdf