AI THIS BIG ANYWHERE

Attention AI THIS BIG ANYWHERE model without any supercomputer AirLLM Quickstart 1. Install package First, install the airllm pip package. pip install airllm 2. Inference Then, initialize AirLLMLlama2, pass in the huggingface repo ID of the model being used, or the local path, and inference can be performed similar to a regular transformer model. (You can also specify the path to save the splitted layered model through layer_shards_saving_path when init AirLLMLlama2. from airllm import AutoModel MAX_LENGTH = 128 # just pass a hugging face repo model = AutoModel.from_pretrained # go bigger with the exact same # model = AutoModel.from_pretrained # model = AutoModel.from_pretrained # or use a model's local path... # model = AutoModel.from_pretrained input_text = [ "What is the capital of" ] input_tokens = model.tokenizer(i return_tensors="pt", return_attention_mask=False, truncation=True, max_length=MAX_LENGTH, padding=False) generation_output = model.genera input_tokens['input_ids'].cu use_cache=True, return_dict_in_generate=True output = model.tokenizer.decode( print(output) Note: During inference, the original model will first be decomposed and saved layer- wise. Please ensure there is sufficient disk space in the huggingface cache directory. Inference Speed Up! We just added model compression based on block-wise quantization-based model compression. Which can further speed up the inference speed for up to 3x, with almost ignorable accuracy loss ! (see more performance evaluation and why we use block-wise quantization in this paper) no compression 8bit block-wise quantization 4bit block-wise quantization 449s 157s inference time How to enable model compression speed up: Step 1. make sure you have bitsandbytes Step 2. make sure airllm verion later than 2.0.0: pip install -U airllm Step 3. when initialize the model, passing the argument compression ('4bit' or '8bit'): model = AutoModel.from_pretraine ) What are the differences between model compression and quantization? Quantization normally needs to quantize both weights and activations to really speed things up. Which makes it harder to maintain accuracy and avoid the impact of outliers in all kinds of inputs. While in our case the bottleneck is mainly at the disk loading, we only need to make the model loading size smaller. So, we get to only quantize the weights' part, which is easier to ensure the accuracy. from your just like book page instead of memorizing the a feature called flash which keeps memory usage long inputs with it Run Locally Llama3 70B Meta for students