Programming Challenge

Postgraduate Interview – Programming Challenge
nanoGPT is a lightweight framework for decoder-only autoregressive models. Understanding both decoder-only models and this 
repository will be very helpful for this postgraduate position. 
Objective:
Train a model on the tiny Shakespeare data set. Use the model to sample what Hamlet might have 
said using the following prompt: “To be, or not to be, that is the”. Find the top 5 next words that 
could have followed this famous snippet.
Guidelines:
• Code:
o Use as much of the existing code in the nanoGPT repository as you need – you do 
not need to re-program something that already exists.
o Be prepared to discuss the code structure and any modifications you have made.
• Tokenization:
o You may use either character-level or tiktoken gpt-2 tokenization.
• Model Size:
o You may train whatever size model fits on your CPU. The evaluation will not be 
based on model complexity or performance.
• Training:
o Create training/validation loss curves.
• Sampling:
o Return the 5 words that your model predicts are most likely to follow the seed 
prompt and include their probabilities.
• Submission: 
o Submit your code (and figures) as a Jupyter Notebook (either standalone .ipynb or 
hosted on Google Colab) with any scripts you have modified from the nanoGPT 
repository.