DDLParserError: Unknown symbol "'" #297
Description
Describe the bug
DDLParserError: Unknown symbol "'" occurs if the column comment contains "’" (Right Single Quotation Mark U+2019)
Error messages are as follows
$ python test_ddlparser.py
"\nCREATE TABLE `example_table` (\n `id` INT NOT NULL AUTO_INCREMENT,\n `name` VARCHAR(255) NOT NULL COMMENT 'double width single quote \u2019 in comment',\n PRIMARY KEY (`id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;\n"
Traceback (most recent call last):
File "D:\works\test-ddlparser\test_ddlparser.py", line 14, in <module>
output = DDLParser(ddl).run()
^^^^^^^^^^^^^^^^^^^^
File "D:\works\test-ddlparser\.venv\Lib\site-packages\simple_ddl_parser\parser.py", line 368, in run
self.tables = self.parse_data()
^^^^^^^^^^^^^^^^^
File "D:\works\test-ddlparser\.venv\Lib\site-packages\simple_ddl_parser\parser.py", line 277, in parse_data
self.process_line(num != len(lines) - 1)
File "D:\works\test-ddlparser\.venv\Lib\site-packages\simple_ddl_parser\parser.py", line 308, in process_line
self.process_statement()
File "D:\works\test-ddlparser\.venv\Lib\site-packages\simple_ddl_parser\parser.py", line 312, in process_statement
self.parse_statement()
File "D:\works\test-ddlparser\.venv\Lib\site-packages\simple_ddl_parser\parser.py", line 319, in parse_statement
_parse_result = yacc.parse(self.statement)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\works\test-ddlparser\.venv\Lib\site-packages\ply\yacc.py", line 333, in parse
return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\works\test-ddlparser\.venv\Lib\site-packages\ply\yacc.py", line 1063, in parseopt_notrack
lookahead = get_token() # Get the next token
^^^^^^^^^^^
File "D:\works\test-ddlparser\.venv\Lib\site-packages\ply\lex.py", line 386, in token
newtok = self.lexerrorf(tok)
^^^^^^^^^^^^^^^^^^^
File "D:\works\test-ddlparser\.venv\Lib\site-packages\simple_ddl_parser\ddl_parser.py", line 285, in t_error
raise DDLParserError("Unknown symbol %r" % (t.value[0],))
simple_ddl_parser.ddl_parser.DDLParserError: Unknown symbol "'"
To Reproduce
Steps to reproduce the behavior:
- Execute the following code
import json
import pprint
from simple_ddl_parser import DDLParser
ddl = """
CREATE TABLE `example_table` (
`id` INT NOT NULL AUTO_INCREMENT,
`name` VARCHAR(255) NOT NULL COMMENT 'double width single quote ’ in comment',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
"""
print(json.dumps(ddl, indent=2, ensure_ascii=True))
output = DDLParser(ddl).run()
pprint.pprint(output)Expected behavior
No error expected