EleutherAI🔒 SSL Verified

Table of Contents

• ↓ Authority Metrics: Score 70/100
• ↓ Global Reach of Eleuther
• ↓ Competitor Analysis for .AI Domains
• ↓ Top Alternatives to eleuther.ai

6-Month Traffic Trend Estimate

Official Contact & Profiles

Visual representation extracted from eleuther.ai

Content Snippets from eleuther.ai

"How do properties of models emerge and evolve over the course of training?"

"As models get smarter, humans won't always be able to independently check if a model's claims are true or false. We aim to circumvent this issue by directly eliciting latent knowledge (ELK) inside the model’s activations."

"EleutherAI has trained and released many powerful open source LLMs."

"As frontier AI systems are pretrained on web-scale data, test set contamination has become a critical concern for accurately assessing their capabilities. While research has thoroughly investigated the impact of test set contamination on discriminative evaluations like multiple-choice question-answering, comparatively little research has studied the impact of test set contamination on generative evaluations. In this work, we quantitatively assess the effect of test set contamination on generative evaluations through the language model lifecycle. We pretrain language models on mixtures of web data and the MATH benchmark, sweeping model sizes and number of test set replicas contaminating the pretraining corpus; performance improves with contamination and model size. Using scaling laws, we make a surprising discovery: including even a single test set replica enables models to achieve lower loss than the irreducible error of training on the uncontaminated corpus. We then study further training: overtraining with fresh data reduces the effects of contamination, whereas supervised finetuning on the training set can either increase or decrease performance on test data, depending on the amount of pretraining contamination. Finally, at inference, we identify factors that modulate memorization: high sampling temperatures mitigate contamination effects, and longer solutions are exponentially more difficult to memorize than shorter ones, presenting a contrast with discriminative evaluations, where solutions are only a few tokens in length. By characterizing how generation and memorization interact, we highlight a new layer of complexity for trustworthy evaluation of AI systems."

Key Topics & Focus Areas for eleuther.ai:

Authority Metrics: Score 70/100

Global Reach of Eleuther

Competitor Analysis for .AI Domains