doctrine/lexer
PHP MITBase library for a lexer that can be used in Top-Down, Recursive Descent Parsers.
11.2k
Stars
62
Forks
2
Open issues
5
Contributors
AI Analysis
A lightweight PHP library providing a base lexer implementation for building Top-Down Recursive Descent Parsers. It is actively used by Doctrine Annotations and Doctrine ORM for parsing DQL queries and annotations. Developers building custom parsers and domain-specific languages benefit from this foundational parsing component.
Inferred from signals mentioned in the README (tests, CI, type safety) — not a review of the actual code.
AI's overall editorial judgment — not an average of the bars above, can weigh other factors too.
Doctrine Lexer: The PHP tokenization foundation powering ORM and annotations parsing
Doctrine Lexer is a base library providing foundational lexer/tokenizer infrastructure for building Top-Down, Recursive Descent Parsers in PHP. Its primary consumers are Doctrine Annotations and Doctrine ORM's DQL (Doctrine Query Language) parser — two of the most widely deployed PHP packages in the ecosystem. The library itself is intentionally minimal: it is not a general-purpose parser toolkit but rather a reusable tokenization layer meant to be extended. Its audience is largely other library/framework authors, not application developers directly.
Created in January 2013 as part of the Doctrine Project to extract shared lexer logic from ORM and Annotations. It has evolved slowly and deliberately over 13 years, reflecting its role as stable infrastructure rather than an actively extended feature set.
Star growth is almost entirely passive, driven by developers exploring the Doctrine ecosystem and encountering the library as a dependency. It does not market itself independently. Its 11k stars almost certainly reflect transitive discovery from doctrine/orm and doctrine/annotations rather than direct intent to use the lexer. The 0 stars in the last 7 days confirms it is no longer in an active discovery phase — it is an established, stable dependency.
Highly verifiable indirect adoption: doctrine/orm (10k+ stars, used in millions of Symfony and Laravel applications) and doctrine/annotations (formerly ubiquitous in PHP DI/serialization) both depend on this library. Packagist download counts for doctrine/lexer have historically been in the hundreds of millions, placing it among the most-downloaded PHP packages. This is one of the strongest indirect adoption signals available in the PHP ecosystem.
Appears to provide an abstract base class or trait that handles input scanning, token peeking, and cursor advancement, leaving token definition and grammar rules to subclasses. Likely follows a classical lexer pattern with a position-based string scanner. Based on README, it is intentionally minimal and designed for extension rather than standalone use.
A Continuous Integration badge is present linking to GitHub Actions, indicating automated testing is in place. Specific coverage percentage is not documented in README.
Last push was June 14, 2026 — approximately 12 days before the evaluation date. This indicates active, recent maintenance. The low fork count (62) relative to stars is consistent with a stable, rarely-forked infrastructure library. CI is active. Maintenance cadence appears healthy for a mature low-churn library.
ADOPT IF: you are building a PHP library or framework that needs to tokenize a custom DSL or query language and want a well-tested, battle-proven base rather than rolling your own. AVOID IF: you need a full parser framework, AST generation, or grammar file support — this is deliberately minimal tokenizer infrastructure only. MONITOR IF: you maintain software that transitively depends on doctrine/orm or doctrine/annotations and need to track breaking changes across major versions.
Independent dimensions
Mainstream potential
2/10
Technical importance
7/10
Adoption evidence
9/10
- Extremely narrow scope limits discoverability and direct adoption — most users will never import it deliberately.
- Dependency risk is low but non-zero: any breaking change propagates through doctrine/orm and affects a massive PHP install base.
- PHP ecosystem shift away from annotations (toward PHP 8 native attributes) may reduce doctrine/annotations usage, potentially shrinking one key consumer over time.
- Documentation is sparse by design, which may discourage authors who could benefit from using it as a base but don't know it exists.
- Low fork count means external contributors are few; stewardship remains concentrated in the Doctrine core team.
Doctrine Lexer will continue as stable, low-churn infrastructure for the foreseeable future. DQL parsing ensures ongoing relevance as long as Doctrine ORM remains a major PHP ORM, which appears likely through at least the late 2020s.
Newsletter
Get analyses like this every Monday
Free weekly digest of the most interesting open-source discoveries.
Languages
Information
- Language
- PHP
- License
- MIT
- Last updated
- 4w ago
- Created
- 164mo ago
- Analyzed with
- anthropic/claude-haiku-4-5
Stars over time
Contributors over time
Top 100 contributors only — repos with more will plateau at 100.
Open issues
Open pull requests
No open pull requests.
Top contributors
Recent releases
Similar repos
doctrine/inflector
Doctrine Inflector is a focused PHP library for string manipulation operations,...
doctrine/cache
Doctrine Cache is a caching component extracted from Doctrine Common that...
doctrine/instantiator
Doctrine Instantiator enables creation of PHP class instances without invoking...
| Repository | Stars | Week Δ | Language | Score | Updated |
|---|---|---|---|---|---|
|
doctrine/lexer
this
|
11.2k | — | PHP | 8/10 | 4w ago |
|
doctrine/inflector
|
11.3k | — | PHP | 8/10 | 3w ago |
|
doctrine/orm
|
10.2k | — | PHP | 8/10 | 8h ago |
|
doctrine/cache
|
7.9k | — | PHP | 6/10 | 9mo ago |
|
doctrine/instantiator
|
11k | — | PHP | 8/10 | 4w ago |
 nikic/PHP-Parser
|
17.4k | — | PHP | 9/10 | 2d ago |
A full AST-based PHP source code parser. Solves a different problem (parsing PHP itself) and is not a general lexer base. More featureful but also more opinionated and heavy for embedding in domain-specific parsers like DQL.
A consumer of doctrine/lexer, not a competitor. Illustrates the intended upstream role of this library.
The alternative most library authors would otherwise use. Doctrine Lexer offers tested, reusable infrastructure that reduces boilerplate, though some authors prefer zero dependencies.
PHP parser combinator libraries exist but are less mature and less battle-tested in production PHP frameworks than the Doctrine toolchain.
A grammar-driven parser generator targeting multiple languages. Far more complex to integrate and not idiomatic PHP; doctrine/lexer is far lighter for small DSLs.
