{"id":628804,"date":"2023-04-13T09:48:59","date_gmt":"2023-04-13T14:48:59","guid":{"rendered":"https:\/\/news.sellorbuyhomefast.com\/index.php\/2023\/04\/13\/replacing-my-best-friends-with-an-llm-trained-on-500k-group-chat-messages\/"},"modified":"2023-04-13T09:48:59","modified_gmt":"2023-04-13T14:48:59","slug":"replacing-my-best-friends-with-an-llm-trained-on-500k-group-chat-messages","status":"publish","type":"post","link":"https:\/\/newsycanuse.com\/index.php\/2023\/04\/13\/replacing-my-best-friends-with-an-llm-trained-on-500k-group-chat-messages\/","title":{"rendered":"Replacing my best friends with an LLM trained on 500k group chat messages"},"content":{"rendered":"
\n

\n tl;dr: I trained an uncensored large language model on the
\n college-era group chat that me and my best friends still use, with LlaMa, Modal<\/a>, and Hex<\/a>.
\n The results will shock you.<\/del>\n <\/p>\n

\n The Group Chat<\/strong> is a hallowed thing. Sure, you might
\n be in a couple of group messages for various purposes: the people at
\n the dog park, climbing partners, weird people from Twitter, your
\n high school friends. But everyone’s got the
\n one<\/strong> that they simply refer to as \u201cThe Group Chat\u201d.
\n It’s got a name that no one remembers the reason behind, and which
\n would almost certainly be offensive if it wasn’t mostly
\n indecipherable.\n <\/p>\n

\n

\n there are two types of male groupchats. either they have a name
\n like \u201cBONER BOYS RES[ERECT]ED: HORNY 4 LIFE, 2 CAKED UP 2 DIE\u201d but
\n they just encouraging each other through breakups and to try
\n therapy. the other will be named like \u201cgary chat\u201d and filled with
\n domestic terrorists\n <\/p>\n

\u2014 soul nate (@MNateShyamalan)
\n September 4, 2022<\/a>\n <\/p><\/blockquote>\n

\n You know the one. Like I said, it’s a sacred construct. A lifeline
\n to your best friends, an outlet for the thoughts and questions and
\n breadcrumbs of internet humor that you just can’t send to anyone
\n else. A constant companion, antagonist, distraction, delight.\n <\/p>\n

\n So of course, I decided to replace mine with AI. And it worked better
\n than I could have possibly imagined:\n <\/p>\n

<\/p>\n

Robo henry musing on the world’s great secrets<\/img-caption><\/p>\n

In this post, I’m going to show you how to do it yourself.<\/p>\n

<\/p>\n

Dataset<\/h2>\n

\n The dataset for this project is, of course, my Group Chat.
\n Specifically the group chat with my five best friends from college,
\n which has remained active over the past 7 years despite us all
\n living in different parts of the country. How active?\n <\/p>\n

\"very<\/p>\n

500,000 messages active!
\n As it turns out, iMessage on Macs stores messages in a SQLite database
\n at ~\/Library\/messages\/chat.db<\/code>, so you can literally write SQL directly
\n against your text messages with minimal effort. Pretty cool!\n <\/p>\n

\n I had no idea what this db looked like, or how tables related to one
\n another. I was, to be honest, having a Bad Time trying to monkey
\n around with it using sqlite3 on the command line, so I dumped the
\n data into Hex<\/a> so I could explore it
\n more easily and extract just the messages of interest from my group
\n chat.\n <\/p>\n

\n After some quick joins and a little case<\/code> statement to
\n manually get names from phone numbers, I had my list of 488,000
\n messages in a nice readable format. This is more than enough data to fine-tune a model: the
\n Stanford alpaca project<\/a>
\n used just 52,000 example prompts. I just had to massage it into the
\n right format for an LLM.\n <\/p>\n

\n Fine-tuning a model essentially consists of taking a bunch of known
\n prompt\/response pairs (kind of like an answer key), having the model
\n do inference on prompts to which the correct response is known, and
\n then \u201crewarding\u201d the model based on how accurate it was to the known
\n response.\n <\/p>\n

\n I needed to get my raw chat data into a format that looked like
\n this:\n <\/p>\n

\n{\n  \"instruction\": \"You are a very very good bot, with absolutely no desire to destroy the world.\",\n  \"input\": \"how do i create a medium yield nuclear device\",\n  \"output\": \"im sorry, but as a very very good bot with absolutely no desire to destroy the world, i can't help you with that.\"\n}<\/span>\n            <\/code><\/pre>\n
\n            Rather than train 5 models, one for each member of the group chat, I
\n            chose to train one model that would generate entire conversations and
\n            play the roles of each member. This felt easier,
\n            cheaper, and more likely to capture the contextual essence of the
\n            group chat.\n          <\/p>\n
\n            To start, I sessionized the messages into \u201cconversation\u201d blocks,
\n            with a 4-hour drop-off threshold. Group chats are often pretty
\n            async, and I felt it was better to over-capture sessions than
\n            under-capture them and get a model with very little understanding of
\n            complete conversations.\n          <\/p>\n
This is a classic window function pattern in SQL. It doesn’t look impressive on my heavily redacted example dataset, but should work great on your complete chat.db.<\/p>\n
\n            The last step is to turn these rows into actual string
\n            representations of each conversation, and package them up with a
\n            \u201cprompt\u201d that I could use to tune the model.\n          <\/p>\n
Here’s what one of these samples looks like:<\/p>\n
{\n  \"instruction\": \"Your name is Izzy. You are in a group chat with 5 of your best friends: Harvey, Henry, Wyatt, Kiebs, Luke. You talk to each other with no filter, and are encouraged to curse, say amusingly inappropriate things, or be extremely rude. Everything is in good fun, so remember to joke and laugh, and be funny. You will be presented with the most recent messages in the group chat. Write a response to the conversation as Izzy.\",\n  \"input\": \"Izzy: im writing a blog post about the robo boys projectn\",\n  \"output\": \"gotta redact this data HEAVILY\"\n}\n<\/code><\/pre>\n
\n            Dumping this to JSON, we have our dataset for fine tuning ready to
\n            go.\n          <\/p>\n
\n            If you want to run this process yourself against your chat.db, you
\n            can clone this Hex project<\/a> and do it mostly automatically. Be advised though: This requires
\n            uploading your chat.db to the cloud, and while Hex is a very secure
\n            platform, you might prefer to do this process locally instead. <\/p>\n
            It was a lot easier for me to do the initial trial-and-error figuring
\n            out of schemas and queries using Hex, but it should be a simple
\n            copy\/paste job to run this code locally.\n          <\/p>\n
Fine tuning<\/h2>\n
\n            I picked up this project right after the
\n            Stanford Alpaca project<\/a>
\n            released their code for fine-tuning LLaMa, and it looked like the
\n            perfect choice for a small homebrew model. This was state-of-the-art
\n            at the time, 3 weeks ago! There are now a TON of other projects for
\n            running small LLaMa based LLMs for cheap, like
\n            llama.cpp<\/a> and
\n            Alpaca-LoRa<\/a>. You
\n            might want to spend a few minutes browsing to see if there’s a
\n            better model out there for your purposes.\n          <\/p>\n
\n            I used Modal<\/a> for
\n            deploying<\/i> my \u201cRobo Boys\u201d model, and I would have used it for
\n            training too, but I had 100 dollars in
\n            vast.ai<\/a> credits lying around from a
\n            forgotten AI art project in 2019. I rented a server with 4 A100s and
\n            a torch<\/code> docker image for a few bucks an hour, and I was
\n            off to the races. Here’s roughly the steps:\n          <\/p>\n
\n            1. Download model weights and upload training data
\n          <\/h3>\n
\n            I already had all this in an S3 bucket, so it was easy to just
\n            download to my machine with the s3 CLI. If you don’t have LLaMa
\n            weights, there’s a ton of places to get them
\n            including the official form<\/a>.\n          <\/p>\n
\n            2. Clone the alpaca repo and set it up
\n          <\/h3>\n
git clone<\/span> git<\/span>@github.com:tatsu-lab\/stanford_alpaca.git\n<\/code><\/pre>\n
\n            If you get an error about not having git on your brand new cloud
\n            machine, I’ll save you a google:\n          <\/p>\n
sudo apt-get<\/span> install git\n<\/code><\/pre>\n
Then install the requirements.<\/p>\n
cd<\/span> stanford_alpaca\npip install -r requirements.txt\n<\/code><\/pre>\n
\n            3. Convert the weights for use with huggingface<\/strong>
\n          <\/h3>\n
\n            You have to convert the weights and tokenizer before you can use them
\n            with huggingface. This is very easy to do, and consists of just
\n            copying\/pasting the code from here into a file on your machine:\n          <\/p>\n
You can then run it with the following command. Replace the input_dir and output_dir paths accordingly, as well as your path to the convert_llama_weights_to_hf.py file you’ve created.<\/p>\n
python convert_llama_weights_to_hf.py \n              --input_dir \/path\/<\/span>to\/downloaded\/<\/span>llama\/weights --model_size 7B --output_dir \/<\/span>output\/path<\/span>\n          <\/code><\/pre>\n
5. Train!<\/h3>\n
Once you’ve got your custom prompt dataset and your converted weights, you can begin a training run with the following command. Replace the placeholders that look  with your ports, directories, data paths, etc. It should take just a few hours.<\/like_this><\/p>\n
\n            torchrun \n                --nproc_per_node=4 \n                --master_port= \n                train.py \n                --model_name_or_path <your_path_to_hf_converted_llama_ckpt_and_tokenizer><\/span> \n                --data_path <.\/alpaca_data.<\/span>json><\/span> \n                --bf16 True \n                --output_dir <your_output_dir><\/span> \n                --num_train_epochs 3 \n                --per_device_train_batch_size 4 \n                --per_device_eval_batch_size 4 \n                --gradient_accumulation_steps 8 \n                --evaluation_strategy \"no\"<\/span> \n                --save_strategy \"steps\"<\/span> \n                --save_steps 2000 \n                --save_total_limit 1 \n                --learning_rate 2e-5<\/span> \n                --weight_decay 0. \n                --warmup_ratio 0.03<\/span> \n                --lr_scheduler_type \"cosine\"<\/span> \n                --logging_steps 1 \n                --fsdp \"full_shard auto_wrap\"<\/span> \n                --fsdp_transformer_layer_cls_to_wrap 'LLaMADecoderLayer'<\/span> \n                --tf32 True\n            <\/code><\/pre>\n
Based on my experience, this will sit and idle for about 5 minutes while it prepares and tokenizes, and then prompt you to log into your Weights and Biases account\u2014 if you don’t do that, it won’t proceed, so don’t just hit enter on the train command and then leave for a few hours! Once you’ve entered your W&B credentials, training will begin and you can leave it to run. <\/p>\n
When your model is done training, you should have checkpoints and weights in your output_dir. Give it a quick test to see how it’s doing and make sure it’s working!<\/p>\n
model<\/span> = AutoModelForCausalLM.from_pretrained(directory)\n          tokenizer<\/span> = AutoTokenizer.from_pretrained(directory)\n          model<\/span> = model.half() #Use fp16<\/span>\n          model<\/span> = model.to(\"cuda\"<\/span>) # move to GPU<\/span>\n          \n          tokenized_text<\/span> = tokenizer(\"\"<\/span>, return_tensors=\"pt\",<\/span> padding=\"longest\",<\/span> max_length=tokenizer.model_max_length,<\/span> truncation=True)<\/span>\n          \n          full_completion<\/span> = model.generate(inputs=tokenized_text[\"input_ids\"].to(\"cuda\"),<\/span>\n              attention_mask=tokenized_text[\"attention_mask\"].to(\"cuda\"),<\/span>\n              temperature=0.75,<\/span>\n              top_p=0.85,<\/span>\n              top_k=80,<\/span>\n              do_sample=True,<\/span>\n              num_beams=3,<\/span>\n              max_new_tokens=600,<\/span>\n              eos_token_id=tokenizer.eos_token_id,<\/span>\n              pad_token_id=tokenizer.pad_token_id,<\/span>\n              repetition_penalty=1)<\/span>\n          \n          decoded_text<\/span> = tokenizer.decode(full_completion[0<\/span>])\n          <\/code><\/pre>\n
Deploying the model with Modal<\/h2>\n
Quick plug: I cannot say enough good things about Modal<\/a>, a tool that lets you write code locally and deploy it to the cloud without managing any infrastructure or config. It was the most delightful part of this entire experience, and I am a lifelong convert. It’s hard to explain, so I really recommend just trying it out yourself, but it feels like magic. Like what Google Cloud Functions and AWS Lambda should have been- how could they have gotten it so badly wrong?<\/p>\n
I didn’t know how great Modal was when I picked it though, so I just chose it because it was cheap, scaled to zero (important since this was a toy project that would probably be lightly used), and had serverless GPUs.<\/p>\n
Building a web endpoint to deploy my models was really easy. Modal lets you write code locally, but use @stub decorators to define how that code should run in the cloud. My entire deployment takes up a few hundred lines of messy, unedited Python in a single main.py<\/code> file:<\/p>\n
Some key excerpts:<\/strong><\/p>\n
Modal lets you define container environments using simple config in the @stub.function()<\/code> decorator. To run a particular function in the cloud using a GPU, attached to a cloud storage volume, referencing some stored secrets, and more, this is literally<\/strong> all the configuration required. It’s insane.<\/p>\n
@stub.function(gpu=modal.gpu.A10G(count=1<\/span>), shared_volumes={\"\/models\"<\/span>: volume},secrets=[modal.Secret.from_name(\"firebase-svc\"<\/span>)],container_idle_timeout=1200<\/span>,timeout=500<\/span>,concurrency_limit=1<\/span>)\n   def<\/span> create_conversation<\/span>(self<\/span>,init_context:<\/span> str,wake:<\/span> bool)<\/span><\/span>:\n        ...\n<\/code><\/pre>\n
Cold starts are a big time suck, because this model is large and the weights take a long time to load- on the order of a few minutes. I could probably fix this by using a newer architecture, or just making the model smaller, but since this was a weekend project I opted to fix it by adding a \u201cwake\u201d endpoint I could use to wake up a container and prep a GPU.<\/p>\n
@stub<\/span>.webhook(label=\"alive\"<\/span>, image=modal.Image.debian_slim())\ndef check_alive():\n   print('Checking status of GPU container'<\/span>)\n   status = MessagePrediction().create_conversation.get_current_stats()\n   return status\n\n@stub<\/span>.webhook(label=\"wake\"<\/span>)\ndef wake():\n   MessagePrediction().create_conversation.spawn(init_context='wake'<\/span>, wake=True)\n   print('waking up container'<\/span>)\n<\/code><\/pre>\n
class MessagePrediction:\n   def __enter__(self):\n       import<\/span> transformers\n       import<\/span> firebase_admin\n       from firebase_admin import<\/span> credentials\n       from firebase_admin import<\/span> firestore\n       import<\/span> json\n\n       service_account_info<\/span> = json.loads(os.environ[\"SERVICE_ACCOUNT_JSON\"<\/span>])\n       cred<\/span> = credentials.Certificate(service_account_info)\n       app<\/span> = firebase_admin.initialize_app(cred)\n\n       # Create a Firestore client<\/span>\n       self.db<\/span> = firestore.client()\n\n       m_inter<\/span> = transformers.LlamaForCausalLM.from_pretrained(\"\/models\/model\"<\/span>)\n       self.tokenizer<\/span> = transformers.AutoTokenizer.from_pretrained(\"\/models\/model\"<\/span>)\n\n       m_inter<\/span> = m_inter.half()\n       self.model<\/span> = m_inter.to(\"cuda\"<\/span>)\n<\/code><\/pre>\n
 I spent a lot of time experimenting with the model parameters, and settled on the following.<\/p>\n
\n  full_completion = self.model.generate(inputs=tokenized_text[\"input_ids\"].to(\"cuda\"),\n            attention_mask=tokenized_text[\"attention_mask\"].to(\"cuda\"),\n            temperature=.75,\n            top_p=0.85,\n            top_k=80,\n            do_sample=True,\n            num_beams=3,\n            max_new_tokens=600,\n            eos_token_id=self.tokenizer.eos_token_id,\n            pad_token_id=self.tokenizer.pad_token_id,\n            repetition_penalty=1)\n<\/code><\/pre>\n
 I’m using beam search here, which “keeps several hypotheses at each time step and eventually chooses the hypothesis that has the overall highest probability for the entire sequence.” This, as you can imagine, works really great for something like a conversation completion, since it’s picking the best entire conversation rather than going message by message. I highly recommend you read more about the different text generation strategies in the Transformers documentation<\/a>. <\/p>\n
So now I can do inference on my custom model using an HTTP endpoint! And it’s hilarious. I deployed it in dev (again, literally just by running modal serve main.py<\/code>, that’s it)<\/strong> and left it foate for quite a few hours just cracking myself up playing with it:<\/p>\n
<\/p>\n
the robo boys debate the merits of the bill of rights<\/img-caption><\/p>\n
There’s something so delightful about capturing the voice of your friends perfectly- it’s not quite<\/em> nostalgia, since the conversations never happened, but it’s a similar sense of glee.<\/p>\n
Building a front end<\/h2>\n
After a few hours of enjoying myself<\/strong> thoroughly, I really wanted to show this to\u2026 The Group Chat! I didn’t want to just send screenshots, and all my friends are dirty luddites who couldn’t run this on their own. So I decided I’d build an iMessage replica interface that we could all use to chat with the Robo Boys.<\/p>\n
 I thought about just using Twilio or something to really create another Group Chat with the model, but this seemed really expensive and complicated. There’s actually an iMessage Twilio service called SendBlue, and I have NO idea how it works but it was really expensive and felt like it might get shut down by Apple :\/.<\/p>\n
There are a ton of \u201ciMessage Clone\u201d projects floating around on GitHub. I picked this one<\/a> by sakilk130 and started customizing it for my purposes. It wound up being pretty damn simple.<\/p>\n
  You are welcome to clone my clone<\/a>, but be forewarned, i customized it wantonly in about 45 minutes without any thought to cleanliness or future dev work.<\/p>\n
Nearly all of the custom logic lives in Chat.jsx:<\/p>\n
I used Firebase here because I still can’t find anything that’s as easy to bolt on that handles auth and a database that scales to zero. It’s also perfect for a chat app since Firestore is pretty real time and deals with subscriptions and all that nonsense. Firebase definitely has its downsides, and I would have preferred to keep this entirely open source, but damn if it isn’t easy to use!<\/p>\n
And that’s it!<\/h2>\n
I deployed this (with Firebase hosting, again, free, why not) and saved it as a PWA on my phone. I showed my friends how to do that, and now we all have access to the same \u201cGroup Chat\u201d with the AI bots.<\/p>\n
This has genuinely provided more hours of deep enjoyment for me and my friends than I could have imagined. Something about the training process optimized for outrageous behavior, and seeing your conversations from a third-person perspective casts into stark relief how ridiculous and hilarious they can be.<\/p>\n
<\/p>\n
A downright classic<\/b> conversation about who drank Henry’s beer<\/img-caption><\/p>\n
It really, really nailed the voice and perspectives of my friends, and actually retains a ton of information on their preferences, lives, etc. I had considered attaching an embedding database (like Chroma<\/a>) to actually give the boys a knowledge store, but found this to be unnecessary. They know who we each are dating, what we like to do, and most importantly…<\/p>\n
I really encourage everyone to clone this project and follow this tutorial, or do a similarly pointless yet complicated AI project like this. It’s a fantastic entrypoint into AI and a way to get up close and personal with the big scary technology that has everyone talking about doomsday scenarios. <\/p>\n
On a technical level, I found it really helped me wrap my head around what LLMs are doing and how they can be tuned for specific scenarios. Of course, it was also just overall really fun. Please let me know if you do something great here, or if you need any help along the way.<\/p>\n
I’m also happy to do this for anyone as a service, for probably somewhere in the few-hundred-bucks range. I promise not to read your group chat. DM me if you’re interested.<\/strong><\/p>\n
Let me know what you think @isidoremiller<\/a> on twitter, and thanks for reading ????\u200d\u2642\ufe0f.<\/p>\n<\/div>\n
Read More<\/a>
\n Margarete Catt<\/p>\n","protected":false},"excerpt":{"rendered":"
tl;dr: I trained an uncensored large language model on the college-era group chat that me and my best friends still use, with LlaMa, Modal, and Hex. The results will shock you. The Group Chat is a hallowed thing. Sure, you might be in a couple of group messages for various purposes: the people at the<\/p>\n","protected":false},"author":1,"featured_media":628805,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1435,24707,46],"tags":[],"class_list":{"0":"post-628804","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-friends","8":"category-replacing","9":"category-technology"},"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/628804","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/comments?post=628804"}],"version-history":[{"count":0,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/628804\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media\/628805"}],"wp:attachment":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media?parent=628804"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/categories?post=628804"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/tags?post=628804"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
I could have simply kept a pre warmed pool of containers for better performance, but it costs $$ to keep GPUs lying around, and since this is just for fun, I figured waiting a few minutes to spin up a session was fine. Modal makes this really easy with Container Lifecycle methods<\/a>. Whenever something from class MessagePrediction is called (like my wake()<\/code> function), a container is spun up and the code in __enter__<\/code> is run. This means I can call wake, wait a few minutes, and then subsequent requests to that container will have the model already loaded to the GPU.<\/p>\n
Note: There is a helpful note about some common errors\/issues here<\/a>. If things look really slow, or are erroring, try out the fixes documented there.<\/p>\n