StarCoder2-15B reached 72.6% accuracy on the HumanEval benchmark with its instruction-tuned variant, surpassing CodeLlama-70B-Instruct despite having less than one-quarter of the parameters. The model trained on 4.3 trillion tokens across 619 programming languages, representing a 330% increase in training data compared to the original StarCoder. Released on February 28, 2024, through collaboration between Hugging Face, ServiceNow, and NVIDIA, StarCoder2 establishes new benchmarks for open-source code generation.
StarCoder Key Statistics
- StarCoder2-15B trained on 4.3 trillion tokens, a 330% increase from the original model’s 1 trillion tokens as of February 2024
- The flagship 15B model supports 619 programming languages while smaller variants focus on 17 widely-used languages
- StarCoder2-15B-Instruct achieved 72.6% on HumanEval and 75.2% on MBPP benchmarks
- ServiceNow reported a 52% increase in developer productivity using text-to-code solutions built on fine-tuned StarCoderBase models
- Training consumed 89,671.68 kWh of electricity across 320,256 GPU hours on 512 NVIDIA A100 GPUs
StarCoder Model Parameters and Architecture
The StarCoder family evolved from a single 15.5 billion parameter model to three distinct variants optimized for different computational requirements. StarCoder2 introduced models ranging from 3 billion to 15 billion parameters.
StarCoder2-3B and StarCoder2-7B target 17 widely-used programming languages, developed by ServiceNow and Hugging Face respectively. The flagship StarCoder2-15B, developed by NVIDIA, expanded language support to 619 programming languages.
| Model Variant | Parameters | Training Tokens | Languages |
|---|---|---|---|
| StarCoder2-3B | 3 billion | 3.3 trillion | 17 |
| StarCoder2-7B | 7 billion | 3.7 trillion | 17 |
| StarCoder2-15B | 15 billion | 4.3 trillion | 619 |
| StarCoder (Original) | 15.5 billion | 1 trillion | 80+ |
StarCoder Technical Specifications
StarCoder2 doubled the context window from 8,192 to 16,384 tokens, enabling processing of longer code files and multi-file prompts. The model adopted Grouped Query Attention with sliding window attention at 4,096 tokens.
The technical architecture shifted from Multi-Query Attention in the original StarCoder to Grouped Query Attention in StarCoder2. Both versions maintain Fill-in-the-Middle training objectives, optimizing for code completion tasks.
StarCoder Training Dataset Composition
The Stack v2 dataset serves as the foundation for StarCoder2 training, comprising 67.5 terabytes of source code. This represents a 955% increase from The Stack v1’s 6.4 terabytes.
The dataset incorporates source code from Software Heritage repositories, GitHub pull requests, Kaggle notebooks, and code documentation. Training data includes 913 billion unique tokens for the 15B model, a 356% increase from the predecessor.
| Dataset Metric | The Stack v2 | The Stack v1 |
|---|---|---|
| Total Size | 67.5 TB | 6.4 TB |
| Unique Tokens (15B) | 913B+ | ~200B |
| Programming Languages | 619 | 358 |
| Data Sources | GitHub PRs, Kaggle, Docs | GitHub Only |
StarCoder Benchmark Performance Results
StarCoder2-15B achieved 46.3% on the HumanEval benchmark in its base configuration, representing a 58% improvement over the original StarCoder’s 29.3% score. The instruction-tuned variant reached 72.6% accuracy.
On the MBPP benchmark, StarCoder2-15B-Instruct scored 75.2%, outperforming CodeLlama-70B-Instruct’s 72.0% despite having significantly fewer parameters. The model achieved 40.6% on the DS-1000 benchmark.
StarCoder MultiPL-E Language Performance
StarCoder2-3B achieved top performance on 11 of 18 programming languages among small models on the MultiPL-E benchmark. The 7B variant outperformed CodeLlama-7B across most languages.
The 15B model excelled on 16 of 18 languages, particularly outperforming larger models on low-resource languages including D, Julia, Lua, and Perl. This demonstrates effectiveness across both mainstream and niche programming environments.
StarCoder Adoption and Productivity Impact
ServiceNow documented a 52% increase in developer productivity using text-to-code solutions built on fine-tuned StarCoderBase models. This productivity gain operates within a competitive market context established by GitHub Copilot.
As of January 30, 2024, GitHub Copilot reached 1.3 million paying subscribers with over 50,000 organizations utilizing the enterprise version. Industry estimates suggest productivity gains up to 56% for code generation AI tools.
| Adoption Metric | Value |
|---|---|
| ServiceNow Productivity Increase | 52% |
| GitHub Copilot Subscribers | 1.3 million |
| GitHub Copilot Enterprise Orgs | 50,000+ |
| Estimated Industry Productivity Gain | Up to 56% |
StarCoder Memory Requirements and Deployment
StarCoder2-15B requires approximately 16.9 GB of memory with 8-bit quantization, reducing to 9.2 GB with 4-bit quantization. This enables deployment on consumer GPUs including RTX 4080 and RTX 4090.
The 3B model operates efficiently on devices with modest specifications, requiring approximately 2 GB with 4-bit quantization. Community projects like starcoder.cpp extend accessibility to CPU-only environments and Apple Silicon devices through GGUF formats.
StarCoder Training Infrastructure and Environmental Impact
StarCoderBase training consumed 89,671.68 kWh of electricity across 320,256 GPU hours on 512 NVIDIA A100 GPUs. The training duration spanned 24 days with continuous operation.
Carbon emissions totaled 16.68 tonnes of CO2 equivalent, calculated using the carbon intensity of AWS us-west-2 region at 0.15495 kg CO2e per kWh with a PUE factor of 1.2. Hugging Face invested $39,000 in PII detection annotation through Toloka for privacy compliance.
StarCoder Licensing and Transparency Model
StarCoder2 releases Software Heritage Persistent Identifiers for all source code data, enabling complete traceability of training data. The BigCode OpenRAIL-M license governs model distribution with responsible use guidelines.
The BigCode project maintains an active opt-out mechanism allowing code authors to exclude their work from the training dataset. StarCoder scored 85 of 100 on the Stanford Foundation Model Transparency Index for 2024.
| Transparency Metric | Status |
|---|---|
| Training Data Released | Yes (The Stack v2) |
| Training Code Released | Yes (Apache 2.0) |
| Model Weights Released | Yes (OpenRAIL-M) |
| SWHIDs Published | Yes |
| Opt-out Mechanism | Active |
| Transparency Index Score | 85/100 |
FAQs
How many parameters does StarCoder2-15B have?
StarCoder2-15B has 15 billion parameters and trained on 4.3 trillion tokens. The model supports 619 programming languages and achieved 72.6% accuracy on the HumanEval benchmark in its instruction-tuned configuration.
What is StarCoder’s performance on coding benchmarks?
StarCoder2-15B-Instruct reached 72.6% on HumanEval and 75.2% on MBPP, surpassing CodeLlama-70B-Instruct. The base model achieved 46.3% on HumanEval, a 58% improvement over the original StarCoder’s 29.3% score.
How much memory does StarCoder require for deployment?
StarCoder2-15B requires 16.9 GB with 8-bit quantization and 9.2 GB with 4-bit quantization. The 3B model needs approximately 2 GB with 4-bit quantization, enabling deployment on consumer hardware.
What productivity improvements does StarCoder deliver?
ServiceNow reported a 52% increase in developer productivity using text-to-code solutions built on fine-tuned StarCoderBase models. Industry benchmarks suggest code generation AI tools achieve productivity gains up to 56%.
What license governs StarCoder usage?
StarCoder operates under the BigCode OpenRAIL-M license, balancing open access with responsible use guidelines. The project maintains an active opt-out mechanism and scored 85 of 100 on the Stanford Foundation Model Transparency Index.
