mirror of
https://github.com/python/cpython.git
synced 2024-11-24 17:47:13 +01:00
6715f91edc
This commit replaces the Python implementation of the tokenize module with an implementation that reuses the real C tokenizer via a private extension module. The tokenize module now implements a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward compatibility. As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation. Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
73 lines
1.6 KiB
Plaintext
73 lines
1.6 KiB
Plaintext
ENDMARKER
|
|
NAME
|
|
NUMBER
|
|
STRING
|
|
NEWLINE
|
|
INDENT
|
|
DEDENT
|
|
|
|
LPAR '('
|
|
RPAR ')'
|
|
LSQB '['
|
|
RSQB ']'
|
|
COLON ':'
|
|
COMMA ','
|
|
SEMI ';'
|
|
PLUS '+'
|
|
MINUS '-'
|
|
STAR '*'
|
|
SLASH '/'
|
|
VBAR '|'
|
|
AMPER '&'
|
|
LESS '<'
|
|
GREATER '>'
|
|
EQUAL '='
|
|
DOT '.'
|
|
PERCENT '%'
|
|
LBRACE '{'
|
|
RBRACE '}'
|
|
EQEQUAL '=='
|
|
NOTEQUAL '!='
|
|
LESSEQUAL '<='
|
|
GREATEREQUAL '>='
|
|
TILDE '~'
|
|
CIRCUMFLEX '^'
|
|
LEFTSHIFT '<<'
|
|
RIGHTSHIFT '>>'
|
|
DOUBLESTAR '**'
|
|
PLUSEQUAL '+='
|
|
MINEQUAL '-='
|
|
STAREQUAL '*='
|
|
SLASHEQUAL '/='
|
|
PERCENTEQUAL '%='
|
|
AMPEREQUAL '&='
|
|
VBAREQUAL '|='
|
|
CIRCUMFLEXEQUAL '^='
|
|
LEFTSHIFTEQUAL '<<='
|
|
RIGHTSHIFTEQUAL '>>='
|
|
DOUBLESTAREQUAL '**='
|
|
DOUBLESLASH '//'
|
|
DOUBLESLASHEQUAL '//='
|
|
AT '@'
|
|
ATEQUAL '@='
|
|
RARROW '->'
|
|
ELLIPSIS '...'
|
|
COLONEQUAL ':='
|
|
EXCLAMATION '!'
|
|
|
|
OP
|
|
AWAIT
|
|
ASYNC
|
|
TYPE_IGNORE
|
|
TYPE_COMMENT
|
|
SOFT_KEYWORD
|
|
FSTRING_START
|
|
FSTRING_MIDDLE
|
|
FSTRING_END
|
|
COMMENT
|
|
NL
|
|
ERRORTOKEN
|
|
|
|
# These aren't used by the C tokenizer but are needed for tokenize.py
|
|
ENCODING
|