textbooks are all you need

• Microsoft relied on GPT-3.5's generational abilities only, whereas it didn't prefer to use it for annotation-based use cases.
• We see a rigorously planned data generation flow, and it is likely that a major reason is to avoid the scope narrowing down during recursive training.
• Researchers do not share how they managed to produce a diverse set of data using GPT-3.5 but just write that they "needed to be creative." This is either by itself another research area or they want to use this method on more research before sharing.
• The idea is actually as simple as training a teacher to teach Python to students, instead of a teacher directly teaching Python to students. This is mostly because the selected corpus for the base model also required an extensive flow including selection and annotation. Even though it does not mention, it probably includes Docstring edits.
• GPT-3.5 is strong enough with its generation abilities to produce a fine-tuning dataset. However, and unsurprisingly, if OpenAI's use terms have not changed, we are not be able to use GPT for training models. It looks like this was simply an exception for Microsoft.
• I was not shockingly surprised by phi-1's chat generalization capability, given that the Docstring included correct grammar throughout the training. I found it very possible that the model understood sentence structure and tokenization. I believe that changing the documentation in the training set to have incorrect grammar would lead to a significantly reduced chat capability.
• It looked to me that StarCoder-Prompted gives better results at structurally very non-similar tasks, but this is very expected, given the reduced problems in phi-1's training.
• The 50 problems created by an outside team at Microsoft to test the generalization and leakage of the model is not a very solid action. The reader would want to see whether Microsoft used any statistical approach to test if these questions overlapped with the CodeExercises dataset.