Context versus Prior Knowledge in Language Models
Paper
•
2404.04633
•
Published
•
5
AI & ML interests
Contributors who are invited to beta-test our next big feature! Contact us if you want to join this team :-)
Recent Activity
View all activity
vesteinn
authored
a
paper
16 days ago
PereLluis13
authored
2
papers
22 days ago
Post
3402
🚀 ZeroGPU now supports PyTorch native quantization via
While it hasn’t been battle-tested yet,
Let us know if you run into any issues — and we’re excited to see what the community will build!
torchaoWhile it hasn’t been battle-tested yet,
Int8WeightOnlyConfig is already working flawlessly in our tests.Let us know if you run into any issues — and we’re excited to see what the community will build!
import spaces
from diffusers import FluxPipeline
from torchao.quantization.quant_api import Int8WeightOnlyConfig, quantize_
pipeline = FluxPipeline.from_pretrained(...).to('cuda')
quantize_(pipeline.transformer, Int8WeightOnlyConfig()) # Or any other component(s)
@spaces.GPU
def generate(prompt: str):
return pipeline(prompt).images[0]
- 5 replies
Post
5928
🚀 ZeroGPU
Nothing too fancy for now—ZeroGPU Spaces still default to
- 💰 size-based quotas / pricing (
- 🦣 the upcoming
You can as of now control GPU size via a Space variable. Accepted values:
-
-
-
The auto mode checks total CUDA tensor size during startup:
- More than 30GB →
- Otherwise →
medium size is now available as a power-user featureNothing too fancy for now—ZeroGPU Spaces still default to
large (70GB VRAM)—but this paves the way for:- 💰 size-based quotas / pricing (
medium will offer significantly more usage than large)- 🦣 the upcoming
xlarge size (141GB VRAM)You can as of now control GPU size via a Space variable. Accepted values:
-
auto (future default)-
medium-
large (current default)The auto mode checks total CUDA tensor size during startup:
- More than 30GB →
large- Otherwise →
medium
-
- 3 replies
PereLluis13
authored
a
paper
4 months ago
Fraser
authored
a
paper
6 months ago
malteos
authored
10
papers
6 months ago
Tokenizer Choice For LLM Training: Negligible or Crucial?
Paper
•
2310.08754
•
Published
•
2
Towards an Open Platform for Legal Information
Paper
•
2005.13342
•
Published
Aspect-based Document Similarity for Research Papers
Paper
•
2010.06395
•
Published
Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings
Paper
•
2202.06671
•
Published
•
2
Specialized Document Embeddings for Aspect-based Similarity of Research Papers
Paper
•
2203.14541
•
Published
Investigating Gender Bias in Turkish Language Models
Paper
•
2404.11726
•
Published
•
1
Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning
Paper
•
2301.09626
•
Published
•
2
Progress Report: Towards European LLMs
Paper
•
2410.03730
•
Published
•
3
Data Processing for the OpenGPT-X Model Family
Paper
•
2410.08800
•
Published
•
1
MMTEB: Massive Multilingual Text Embedding Benchmark
Paper
•
2502.13595
•
Published
•
38
Post
1929
I re-worked the JuptyerLab Space template recently. It's optimized for
timm use, but will work great with transformers and other libs. Updated the base image, Python 3.12, Pillow-SIMD before better CPU use with image preprocessing, and made a number of other tweaks. From the Jupyter launcher you can run the terminal and setup a timm environment in moments with setup_timm_dev or setup_timm_scripts helpers. Give it a try,
timm/jupyterlab-timm
Post
1209
New
* PaliGemma2 encoders (ported from google/paligemma-2-release-67500e1e1dbfdd4dee27ba48)
* AIMv-2 encoders (ported from apple/aimv2-6720fe1558d94c7805f7688c)
A few higher resolution 384x384 ConvNeXt-Nano ImageNet-12k pretrain & finetunes. See other changes here: https://github.com/huggingface/pytorch-image-models/releases/tag/v1.0.13
And support added in both
* DFN CLIP L/14 w/ 39B samples seen - apple/DFN2B-CLIP-ViT-L-14-39B, timm/vit_large_patch14_clip_224.dfn2b_s39b
* MetaCLIP H/14 (altogether) - timm/vit_huge_patch14_clip_224.metaclip_altogether
And last, ~70-80 models that were relying on
timm 1.0.13 and OpenCLIP 2.30.0 releases to start the year. Both modest but worthwhile updates.timm added a number of new model weights, supporting loading of: * PaliGemma2 encoders (ported from google/paligemma-2-release-67500e1e1dbfdd4dee27ba48)
* AIMv-2 encoders (ported from apple/aimv2-6720fe1558d94c7805f7688c)
A few higher resolution 384x384 ConvNeXt-Nano ImageNet-12k pretrain & finetunes. See other changes here: https://github.com/huggingface/pytorch-image-models/releases/tag/v1.0.13
And support added in both
OpenCLIP and timm for two CLIP models that were missed. The DFN L/14 is 🔥* DFN CLIP L/14 w/ 39B samples seen - apple/DFN2B-CLIP-ViT-L-14-39B, timm/vit_large_patch14_clip_224.dfn2b_s39b
* MetaCLIP H/14 (altogether) - timm/vit_huge_patch14_clip_224.metaclip_altogether
And last, ~70-80 models that were relying on
timm remapping from OpenCLIP got their own timm hub instances to allow use with the upcoming Transformers TimmWrapperModel
Post
1479
There's a new
New optimizers include:
* AdafactorBigVision -
* ADOPT -
* MARS -
* LaProp -
* Cautious Optimizers - a modification to all of the above, prefix with
I shared some caution comparisons in this model repo: rwightman/timm-optim-caution
For details, references, see the code: https://github.com/huggingface/pytorch-image-models/tree/main/timm/optim
timm release, v 1.0.12, with a focus on optimizers. The optimizer factory has been refactored, there's now a timm.optim.list_optimizers() and new way to register optimizers and their attributes. As always you can use an timm optimizer like a torch one, just replace torch.optim with timm.optimNew optimizers include:
* AdafactorBigVision -
adfactorbv* ADOPT -
adopt / adoptw (decoupled decay)* MARS -
mars* LaProp -
laprop* Cautious Optimizers - a modification to all of the above, prefix with
c as well as cadamw, cnadamw, csgdw, clamb, crmsproptf I shared some caution comparisons in this model repo: rwightman/timm-optim-caution
For details, references, see the code: https://github.com/huggingface/pytorch-image-models/tree/main/timm/optim
- 3 replies