Machine Learning
 Applications
 Data
 Good Practice? What’s the right word here
 Supervised
 Unsupervised
 Visualization
 Learning Theory
 Overfitting
 Bayesian Shit
 Deep Learning
 Verification
 Famous Models
 Hugging Face
 Reinforcement Learning
 Verification
 Old Reinforcement learning
See also:
 Optimization
Applications
 Speech recognition
 Image recognition
 branch prediction
 phase transition recognition
 Dimensionality reduction?
 recommender systems
 Fluid sim accelerate
 system identification
 learning program invariants http://pranavgarg.com/papers/cav14.pdf https://www.cs.purdue.edu/homes/suresh/papers/pldi18.pdf
Data
outliers
augmentation  sometimes you can apply trasnformations to the data in ways. For example rotating images if you want the answer to not depend on direction. Or adding noise if you want it to ignore noise. Warping.
In some problems it’s nice that you can cripple your data and train it to undo the crippling
 colorizing images
 interpolating frames
 super resolution
Good Practice? What’s the right word here
Test and training sets cross validation data cleaning meta parameter tuning don’t set auxiliary goals.
Reading training curves debugging?
Supervised
\[\hat{y}_i = f(x_i; \theta)\]Classification discrete output
Regression  continous output
one hot encoding
Linear in WHAT?
Nearest neighbor
SVM
Kernel
decision trees
Boosting
https://en.wikipedia.org/wiki/Boosting_(machine_learning)
Adaboost XGboost
https://en.wikipedia.org/wiki/Multiplicative_weight_update_method a generalizatio of the idea of combinigh expert opinions
Random Forest
Unsupervised
PCA kmeans clustering hierarchical clustering
dimsenionlaity reduction?
Visualization
tsne umap
Learning Theory
VC dimension
PAC probably apprximately correct https://en.wikipedia.org/wiki/Probably_approximately_correct_learning
Overfitting
bias variance
Bayesian Shit
pyro pymc3 stan particle filters? probalistic programming
Regularization
diffrax JAX powered differential equations.
Deep Learning
https://github.com/AUTOMATIC1111/stablediffusionwebui
Convolutional
Recurrent
lstm gru blowup problem / vanishing gradient
tidbits
Attention Capsule transformers
batch normalization
dropout grokking? overfit and keep going. Sometimes it gets better later. Bizarre. Don’t count on this. https://mathaiiclr.github.io/papers/papers/MATHAI_29_paper.pdf autoencoders GANs
graph neural networks?
transfer learning
Verification
Adversarial examples
Generate and Test > use or augment generate with chatgpt
Spec suggestions. Use python syntax for coq questions. Python type annotations Axiom schema instantiations. All the things that require “creativity” Program invariant suggestions. Some come out weak or wrong. Mutation rules. Ask to refine or simplify.
Automata verification of langchains. Blackbox the language model. Verified parsing of the output
“Function call” api
Famous Models
word2vec node2vec
code2vec
copilot
gato
wavenet DALLE stable diffusion midjourney
alexnet vgg alphafold alpha zero / go BERT  masked language modelling. mask some words. predict them. next sentence prediction  did this sentence follow from the previous? https://huggingface.co/bertbaseuncased
https://github.com/facebookresearch/segmentanything
whisper
instructgpt
Tasks
https://huggingface.co/models look at tasks tab
 question answering
 summarization
 conversational
 table questin answering
 text generation
 image segmentation
 text 2 speech
speech recognition
https://huggingface.co/openai/whisperlargev2 https://github.com/openai/whisper
whisper audio.flac model small
whisper model largev2
https://github.com/openai/whisper/discussions/categories/showandtell discussions
https://github.com/ggerganov/whisper.cpp inference on cpu.
https://colab.research.google.com/drive/1WLYoBvA3YNKQ0X2lC9udUOmjK7rZgAwr?usp=sharing Colab is nice. medium run.s
Man, whisper is pretty dang good
code
https://huggingface.co/bigcode/starcoder
math
GAN
https://vcai.mpiinf.mpg.de/projects/DragGAN/ drag your gan
Deepfakes
LoRA
Low rank adaptation https://arxiv.org/abs/2106.09685 But people are also using the technique on stable diffusion Using LoRA for Efficient Stable Diffusion FineTuning
Lora let’s you fine tune big models by injecting in small layers that are easier to train
PEFT parameter efficient fine tuning https://www.youtube.com/watch?v=YVU5wAA6Txo&ab_channel=code_your_own_AI https://github.com/huggingface/peft
https://civitai.com/ people post their lora updates
https://twitter.com/rasbt/status/1642161887889567745 soft finetuning prefix finetuning
Finetuning 20B LLMs with RLHF on a 24GB consumer GPU
https://github.com/artidoro/qlora qlora https://towardsdatascience.com/qlorafinetunealargelanguagemodelonyourgpu27bed5a03e2b https://huggingface.co/blog/4bittransformersbitsandbytes
Stable Diffusion
https://www.fast.ai/posts/part22023.html course
inpainting outpainting
https://github.com/AUTOMATIC1111/stablediffusionwebui
negative prompt, negative embedding
You can generate a supervised learning problem to predict noise given a noisy image. Text info may help. You can then iteratively subtract this noise. If you ask it to remove all noise, then add 99% noise, then ask it to remove 99% noise and so on you have a stable noise removal process. You can boost listening to the text by comparing againt a version tat doesn’t have the text and boosting the differences You can do this process in latent space (small space of autoencoder) to speed it up
Schedules
Mixed Precision
https://github.com/NVIDIA/apex https://github.com/ggerganov/llama.cpp/issues/9 GPTQ quantization https://github.com/ISTDASLab/gptq https://huggingface.co/TheBloke
https://github.com/TimDettmers/bitsandbytes
renting gpu
vast.ai lambdalabs runpod https://www.youtube.com/watch?v=TP2yID7Ubr4&ab_channel=Aitrepreneur
google colab provides ~15gb vram free? colab pro gives a100
https://github.com/skypilotorg/skypilot
prompt engineering
https://www.promptingguide.ai/ Question answer format to give a couple examples Start a conversation See openai examples page People use seperators to denote different sections. Weird.
Avoid impreciseness
chain of thought prompting https://arxiv.org/abs/2201.11903
https://learn.deeplearning.ai/chatgptprompteng
https://twitter.com/ShriramKMurthi/status/1664978520131477505?s=20 shriram racket. “You are a programming assitant that generates programs in the Rakcet programming language. Your response should contain only a Racket program. It should NOT include anything else: explanation, test cases, or anything else. The output should be a Racke function that can be evaluared direvtly. It should begin with "(define" and end with "), e.h., (defibe (f x) x), but replaced with the actual function you produce.”
https://ianarawjo.medium.com/introducingchainforgeavisualprogrammingenvironmentforpromptengineeringbc6910be01cf
https://github.com/ianarawjo/ChainForge
Automated the prompt engineering workflow. You could ask questions you know the answer to, or evaluate code generation against test suite or what have you. The prompt is a kind of hyperparameter and you can apply the same methodolgy you might with others (random search, test sets, validation sets, etc). The llm is a fixed parametrized function like y = ax+b
where a and b are the prompts.
Transformers
https://en.wikipedia.org/wiki/Attention_(machine_learning)
Large Language Models
https://openai.com/research/instructionfollowing instructgpt RLHF reinfrocement learning human feedback
distillation. Take output from bigger more powerful model to train smaler model
evaluate models by asking gpt4 about them
alpaca https://crfm.stanford.edu/2023/03/13/alpaca.html llamma fine tuned. Make a bunch of examples. Make gpt3 build a dataset out of them, finetune llama on those answers
Emery berger is going ham. I think basically what he is doing is grabbing pertinent data and constructing a gpt prompt
langchain https://www.pinecone.io/learn/langchainintro/ https://github.com/microsoft/semantickernel microsoft version of langchain?
https://github.com/unitaryai/detoxify detect toxix comments. I suppose you just need to detect if the output is mean and then you can block it. AI vs AI
https://bellard.org/ts_server/ text synth server fabrice bellard
https://www.emergentmind.com/ https://www.builder.io/blog/aishell
Chinchilla scaling  There is an amount of data and number of parameters that is compute optima
https://news.ycombinator.com/item?id=35483933 chatdbg, another gdb chatgpt integration
https://github.com/openai/tiktoken tokenizer BPE byte pair encoding https://platform.openai.com/tokenizer try out te tokenizer
https://github.com/openai/openaicookbook open ai cookebook
sentence transformers make embeddings easily?
“Pretraining” is the heavy lift huge datacetner part that produces raw gpt3 models or whatever. Weird terminology to call that pretraining
ICL  in context learning  running inferece. A set of examples in the prompt
deep to long learning Context length is important. It scales poorly
Perplexity  measurement of inaccuracy of model prediction on test set https://en.wikipedia.org/wiki/Perplexity
NLP  unigram model  probility of individual words ngram model  condtional probability of word window
https://github.com/imartinez/privateGPT ingest a bunch of documents. chroma vector db
Models
https://vectara.com/toplargelanguagemodels/ useful summary. Probaby will be outdate in a month
https://www.promptingguide.ai/models/collection
 gpt3
 chatgpt

llama https://news.ycombinator.com/item?id=35100086 llama weights leaked. 4 bt quantization. https://github.com/rustformers/llamars
 Bloom https://huggingface.co/bigscience/bloom

eleuther gptj https://en.wikipedia.org/wiki/GPTJ https://www.width.ai/post/gptjvsgpt3 https://github.com/EleutherAI/gptneox
 flant5 / flanul2 / t5xxl
 PaLM
 falcon
 gptcerebras  the point it was trained efficiently, not that its good? https://news.ycombinator.com/item?id=35487846 claude https://www.anthropic.com/index/introducingclaude
rwkv https://github.com/BlinkDL/RWKVLM https://news.ycombinator.com/item?id=35370357 https://github.com/saharNooby/rwkv.cpp
finetunes
 gpt4all  another lora finetuning of llama
 alpaca  paid chatgpt api to generate examples to finetune llama
 vicuna  https://lmsys.org/blog/20230330vicuna/ llama finetuned on sharegpt data
 wizard  https://github.com/nlpxucan/WizardLM#finetuning
 https://huggingface.co/PygmalionAI/pygmalion13b pygmalion for conversation?
 guanaco  qlora. finetuning with quantization. https://guanacomodel.github.io/ https://huggingface.co/datasets/JosephusCheung/GuanacoDataset hmm. actually the qlora stuff might be separate
https://erichartford.com/uncensoredmodels uncensored models remove examples where chat refused to answr
oobabooga textgenerationwebui https://github.com/oobabooga/textgenerationwebui run on colab https://www.youtube.com/watch?v=TP2yID7Ubr4&ab_channel=Aitrepreneur
https://github.com/lmsys/FastChat/ related somehow to vicuna? Fast way to get chat server?
Data sets
The Pile  eleuther ai. huge corpus for training
https://huggingface.co/datasets
databricks dolly https://huggingface.co/datasets/databricks/databricksdolly15k
oasst1 https://huggingface.co/datasets/OpenAssistant/oasst1 https://openassistant.io/ crowd sourced chat stuff
https://sharegpt.com/ sharegpt. crowd source gpt responses. Used in some models, but then commerical use restricted
https://huggingface.co/datasets/bigcode/thestack the stack. starcoder.
red pajama https://huggingface.co/datasets/togethercomputer/RedPajamaData1T clean room open source llama dataset
benchmarks https://lmsys.org/blog/20230503arena/ open source battle between llm https://huggingface.co/datasets/glue general langhguae understanding evaluation benchmark https://gluebenchmark.com/
Openai
import openai
# list models
models = openai.Model.list()
#print(models)
# print the first model's id
print(models.data[0].id)
# create a completion
completion = openai.Completion.create(model="ada", prompt="Hello world")
# print the completion
print(completion.choices[0].text)
print(completion)
# list models
openai api models.list
# create a completion
openai api completions.create m ada p "Hello world"
# create a chat completion
openai api chat_completions.create m gpt3.5turbo g user "Hello world"
# generate images via DALL·E API
#openai api image.create p "two dogs playing chess, cartoon" n 1
# audio.transcribe
# audio.translate
Embeddings https://www.buildt.ai/blog/3llmtricks embedding tricks. Hyde  predict answer from query, then use embedding of predicted answer
import openai
# choose text to embed
text_string = "sample text"
# choose an embedding
model_id = "textsimilaritydavinci001"
# compute the embedding of the text
embedding = openai.Embedding.create(input=text_string, model=model_id)['data'][0]['embedding']
print(embedding)
langchain
from langchain.llms import OpenAI
llm = OpenAI(temperature=0.9) # hgh temprateur
text = "What would be a good company name for a company that makes colorful socks?"
print(llm(text))
from langchain.prompts import PromptTemplate
prompt = PromptTemplate(
input_variables=["product"],
template="What is a good name for a company that makes {product}?",
)
print(prompt.format(product="colorful socks"))
https://github.com/UnstructuredIO/unstructured
Vector Databases
Databases that include the abilitt to do fuzzy search for vectors (from embeddings)
Approximate nearest neighbor: FAISS, https://github.com/spotify/annoy https://github.com/nmslib/hnswlib/
 Pinecone
 Milvus
 Weaviate
 Qdrant
missing where clause, pre vs post filtering
sqlite vector search https://github.com/asg017/sqlitevss https://observablehq.com/@asg017/introducingsqlitevss
postgres vector pg_vector
vs elastic search, opensearch, lucene. BM25
https://www.sbert.net/ sentence transformers https://github.com/imartinez/privateGPT
Backprop Technqiues
adam adamw sgd with momentum?
Frameworks
Tensorflow Pytorch JAX
Julia
https://github.com/huggingface/accelerate
gradio for UIs. huggingface spaces
Hugging Face
transformers pipeline is the easy version datasets library
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaTokenizer, AutoModelForSeq2SeqLM
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("bertbaseuncased")
model = AutoModelForMaskedLM.from_pretrained("bertbaseuncased")
fastai
Resources
 Deep Learning Book
Reinforcement Learning
Dreamerv3 Decision Transformer
cleanrl sheeprl
PPO SAC DDPG
imitation learning reinforce helmut with 3 cameras.
Markov Decision Process Partially observed mdp (POMDP)
Temporal difference Reward function Value function Q function policy function Learn Dynamics  system identification
Qlearning sarsa policy gradient Actor critic
Montecarlo search
Verification
Neural Network Verification with Proof Production
Resources
openai spinning up Sutton and Barto
Old Reinforcement learning
I watched David Silver’s lectures on Reinforcement Learning.
http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html
Pretty interesting stuff.
We had already tried naive reinforcement learning for tic tac toe. We made a random player, and watched whether it won or lost. Then we’d pick only moves that ultimately won and tried to train a neural network to map board state to winning move. In hindsight, this was kind of a ghetto monte carlo policy learning. It worked kind of.
Big takeaways from the lectures:
Value functions and Q functions are things you may want to consider. They tell you the value of your current state. You may want to move to states of high value.
Very evocative of iterative methods for solving matrix equations. So if you’re looking for inspiration, look there. If you had the transition probabilities, it is a linear model for the probabilities.
There are table based methods for
There is also a layer of function approximation you can stack on there.
I think you could implement temporal difference learning using common libraries using $latex r_t + \gamma \max_a Q(S,a,\theta)$ as the truth value, and then update the truth values occasionally.
Old:
Reinforcement learning is when you get a rating of a move instead of the right answer. For example a supervised learning task would be to tell whether a picture is of a cat or not.
Also there is a stronger element of time occurring. Reinforcement learning often is sequential in nature. And the rewards may come later down the line rather than immediately
Explore exploit trade off. Exploration allows you to find new things. The new inns are only occasionally better than the stuff you already know about.
The many armed bandit is an example. You have many slot machines to choose from and 100 quarters. Should you try all the slot machines or
greedy method chooses curren best slot
epsilon greedy chooses best while occasionally choosing a random other
Policy is what actions you make given the current state. State is the encapsulation of the important information you’ve received from all previous measurements. Policy can be deterministic, a function from state to action, or probabilistic, the probability of an action given a state.
Value is the expected reward given a particular policy given your current state.
observations, actions, and rewards.
Three kinds of RL. Policy, Value, and Model based.
There are the questions of policy evaluation and policy optimization. They are related.
Given either a deterministic policy or probabilistic policy, you could hypothetically write down the exact probabilistic step from one time step to another.
There is a connection between Monte Carlo methods I’m more familiar with and the solution methods.
Monte Carlo methods replace expectations with samples. When used numerically, they use expectations as a stand in for a tough summation or integration.
One place where summatiuon occurs is in matrix multiplication. Row times column and then add them all up. Replace this addition with a sampling process.
The evolution of the probability distribution in a markov process can be written as a finite difference equation with a matrix full of the condition steps $latex P(x_{t+1}  x_t)$ 
The matrix multiplication rule then is one that takes the prior distribution marginalizes it out to give the distribution in the next time step.
The optimization step is also somewhat . It is an interesting analogy that you can produce matrix like systems using (max,+) in the place of the usual (+,*). If you use softmax(a,b) = $latex \ln (e^a + e^b)$ it might even be somewhat invertible (although I have strong suspicions it might easily become numerically unstable).
This is for example used as a way of discussing shortest path problems using edge weight matrices. Mixing between the two is curious.
The world is an oracle that can give us samples for us.
Q(s,a) is a very clever, non obvious function.
To perform a step of the bellman equation without the model of the system, you need a function like that.
Q learning vs SARSA
Q learning is off policy. It pretends you’re using a greedy policy
SARSA is on policy
Deep RL course
Imitaiton learning. Learn human expert.
Can often lead to trajectory drifting off course. Clever way of fixing this includes building in some stability to the thing with three headed camera.
Dagger. Make a loop of using human expert. then let thing thing run, then give human the newly acquired data and have him label it. And so on.
Necessary because the poorly imitating robot will end up in states human expert would never see. Distributional mismatch
Model based. If you have am model you can use it. Optimal Control
shooting method, optimize over actions only. Plug in dynamics into cost function
collocation method, optimize over both path and controls with constraints
LQR  dynamics linear, cost function quadratic. Can be back solved if initial condition problem.
Q_t matrix gives future cost assuming optimal policy in space of u and x
V_t matrix only in space of x
K matrix connects x to u. The feedback matrix. Can be built from blocks of Q.
iterative LQR / Differential dynamic programming (DDP uses second order expansion of dynamics)
go backwards to find the right K. Using it forwards to compute correct x and u. Iterate until convergence
https://homes.cs.washington.edu/~todorov/papers/TassaIROS12.pdf
Model Predictive Control. Do it all like you were playing chess. After each actual time step rerun the iterative LQR
https://people.eecs.berkeley.edu/~svlevine/papers/mfcgps.pdf
https://studywolf.wordpress.com/2016/02/03/theiterativelinearquadraticregulatormethod/
https://github.com/pydy/pydytutorialhumanstanding
https://github.com/openai/roboschool
pybullet
learning the model.
So for example i don’t know the mass or inertia parameters (or timestep) of the cartpole in gym. I could build the general model though and then fit pairs of (x, x’, u) to it to determine those. Under random policy.
Usually need to recurse on this step (use current best policy to get more data) like in dagger because random policy discovering dynamics won’t go to the same place that better policy does.
Better yet is to use mpc to replan at every step.
Can also directly train a parametrized policy function as part of the loop rather than using the more algorithmic iLQR
Neural network based model of the dynamics might be fine especially since you can backpropagate which is nice for the iterative LQR step.
https://rselab.cs.washington.edu/papers/robotrlrss11.pdf
in global models the planning stage will tend to try to exploit regions that are crappily modelled.
maximum entropy. May want to have the most random solution that doesn’t hurt cost?
Local models try to avoid this by just modelling gradients? And simply. But use a contrained optimization problem to make sure the robot stays in a region where the local estimates still apply. Trust regions. Defined using how unlikely the current trajectory is