Step file generated from powershell throws error on collection - not UTF-8 encoded

[Vic152]

Generate in powershell with: with: pytest-bdd generate features/installer.feature > installer_step_definitions/test_installer_steps.py
Run with: pytest

OBS: During test collection error occurs as below. The same file test_installer_steps.py created in VS code and the same code pasted from Notepad++ works.

Note: If I create file by hand and overwrite it by auto generated file this happens:

Original encoding

When I convert to UTF-8 it works fine or if I run in cmd

VS Code encoding is UTF-8

ENV:
VS Code
Python 3.12

packages

Package Version

---

colorama 0.4.6
iniconfig 2.0.0
Mako 1.3.0
MarkupSafe 2.1.3
packaging 23.2
parse 1.19.1
parse-type 0.6.2
pip 23.3.1
pluggy 1.3.0
pytest 7.4.3
pytest-bdd 7.0.0
six 1.16.0
typing_extensions 4.8.0

.venv\Lib\site-packages_pytest\assertion\rewrite.py:359:

---

source = b'\xff\xfe"\x00"\x00"\x00I\x00n\x00s\x00t\x00a\x00l\x00l\x00e\x00r\x00 \x00f\x00e\x00a\x00t\x00u\x00r\x00e\x00 \x00t\x...\x00o\x00t\x00I\x00m\x00p\x00l\x00e\x00m\x00e\x00n\x00t\x00e\x00d\x00E\x00r\x00r\x00o\x00r\x00\r\x00\n\x00\r\x00\n\x00'
filename = 'C:\\Users\\vsypniewski-tom\\source\\repos\\xcd-bdd\\installer_step_definitions\\test_installer_steps.py', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if feature_version is None:
            feature_version = -1
        elif isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            if major != 3:
                raise ValueError(f"Unsupported major version: {major}")
            feature_version = minor
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E       SyntaxError: source code string cannot contain null bytes

..\..\..\AppData\Local\Programs\Python\Python312\Lib\ast.py:52: SyntaxError

The above exception was the direct cause of the following exception:

cls = <class '_pytest.runner.CallInfo'>, func = <function pytest_make_collect_report.<locals>.<lambda> at 0x000002367EB0F4C0>, when = 'collect', reraise = None

[youtux]

hi, I don't have a windows installation, please provide a MVCE, together with the generated file as an attachment.

[Vic152]

hi, I don't have a windows installation, please provide a MVCE, together with the generated file as an attachment.

Create venv in VSCode - Python 3.12.1 (tags/v3.12.1:2305ca5, Dec 7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)] on win32
Activate venv
Install - pip install pytest-bdd - in Powershell in VSCode
Write simple feature file
Generate code with: pytest-bdd generate features/functionality.feature > tests/test_functionality.py
Run: pytest in Powershell
pytest-bug.7z.txt <-- this is zip with the project

This error happens on collection

[Vic152]

I was also able to reproduce this behaviour by generating file again. Notepad++ shows this encoding

[jsa34]

@youtux I noticed that template_lookup in generation.py doesn't specify an encoding, which probably means it's taking the default from the system?

Should we enforce utf-8, such as:

template_lookup = TemplateLookup(
    directories=[os.path.join(os.path.dirname(__file__), "templates")],
    input_encoding="utf-8",
    output_encoding="utf-8"
)

@Vic152 , does this fix the issue? I'm not a Windows user, so apologies!

pytest-bdd generate features/installer.feature | Out-File -Encoding utf8 installer_step_definitions/test_installer_steps.py

[youtux]

I don't think that's the issue, but rather that power shell configures the output file from the shell redirection to be encoded in utf16-le by default.

Unfortunately i also don't have a windows machine to confirm it.

Maybe @Vic152 can confirm?

[jsa34]

Aha! That was my second thought, so hopefully the suggestion I made will resolve it by forcing the output to be UTF8