Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Conversation

@SteveL-MSFT
Copy link
Member

@SteveL-MSFT SteveL-MSFT commented Jan 27, 2022

PR Summary

The previous regex to remove ANSI escape sequences was too broad and would remove content that appeared to be ANSI but was not. The change is to have specific regex targeting colors/decoration and CSI ANSI escape sequences (generated by PSReadLine) which are ones used by PS7 and PSReadLine currently.

Hyperlink isn't supported here as no default formatting emits it and the way the escape sequence works it can be too broad.

PR Context

Fix #16730

PR Checklist

@iSazonov
Copy link
Collaborator

Re-assign PR to security WG.

@iSazonov iSazonov assigned PaulHigin and unassigned iSazonov Jan 27, 2022
@SteveL-MSFT SteveL-MSFT added the WG-Security security related areas such as JEA label Jan 27, 2022
@iSazonov
Copy link
Collaborator

@SteveL-MSFT Sorry if I already said this but I feel we fall in unlimited problems and fixes. I would personally roll back this kind of coloring (it's a far cry from the spirit of PowerShell) and invest in Formatting system enhancement or better in something like PSMore.

@SteveL-MSFT
Copy link
Member Author

@iSazonov PSMore is solving a different problem than PSStyle. The two are not mutually exclusive. ANSI/VT escape sequences are the industry standard way of emitting text decoration and works across ssh, xterm.js, Windows, Linux, macOS, native commands, etc... For PSMore to emit color, it will have to resort to ANSI/VT escape sequences anyways and we would still have this situation. Even if table/list formats don't use ANSI/VT, arbitrary scripts and native commands can and do emit ANSI/VT.

@SteveL-MSFT SteveL-MSFT changed the title WIP: Update regex used to remove ANSI escape sequences to be more specific to decoration and hyperlinks Update regex used to remove ANSI escape sequences to be more specific to decoration and hyperlinks Jan 27, 2022
@iSazonov
Copy link
Collaborator

ANSI/VT escape sequences are the industry standard

I don't say about rejecting ESs - they are only implementation details. If we will continue on the way we obviously fall in re-implementing whole Windows Terminal in PowerShell! - there is no other way to ensure full processing of ESs in all scenarios.

arbitrary scripts and native commands can and do emit ANSI/VT.

This is my main frustration because we are provoking users to use ESs when the spirit and idea of PowerShell has always been to do magical things.

PSMore is solving a different problem than PSStyle.

I mentioned PSMore as a way to develop without breaking the existing Formatting System.
This would allow us to find right design and implement it step by step.
I'm sure we need (1) to delegate the details to the console and the terminal and (2) use higher abstractions in the formatting system and even more so in the PowerShell language itself.

Sorry for noisy :-)

@SteveL-MSFT
Copy link
Member Author

@iSazonov I fully agree that the current formatting system needs to be revamped, but again, that's independent of ESC seqs. We can always adopt something like spectre.console and have something like "{red}hello {blue}there", although the benefit of "$($psstyle.foreground.red)hello" is you get tab completion. Anyways, I think this discussion is separate from this PR.

@iSazonov
Copy link
Collaborator

@SteveL-MSFT All you say is about string formatting. My proposal is to follow PowerShell spirit and invest in object formatting first.
String formatting fundamentally leads us to re-implement Windows Terminal in PowerShell.
Object-based formatting works well for years and could be improved for modern terminals (not easy work!). (As part of this problem I proposed Native cmdlet class #13428 which could universally collect all meta data about native command and allow PowerShell engine reliable communicate with the native app (as with regular cmdlet) including smart re-incoding input and output.)

@pull-request-quantifier-deprecated

This PR has 8 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!


Quantification details

Label      : Extra Small
Size       : +5 -3
Percentile : 3.2%

Total files changed: 3

Change summary by file extension:
.cs : +4 -2
.ps1 : +1 -1

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

  • Fast and predictable releases to production:
    • Optimal size changes are more likely to be reviewed faster with fewer
      iterations.
    • Similarity in low PR complexity drives similar review times.
  • Review quality is likely higher as complexity is lower:
    • Bugs are more likely to be detected.
    • Code inconsistencies are more likely to be detetcted.
  • Knowledge sharing is improved within the participants:
    • Small portions can be assimilated better.
  • Better engineering practices are exercised:
    • Solving big problems by dividing them in well contained, smaller problems.
    • Exercising separation of concerns within the code changes.

What can I do to optimize my changes

  • Use the PullRequestQuantifier to quantify your PR accurately
    • Create a context profile for your repo using the context generator
    • Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
    • Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
    • Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
  • Change your engineering behaviors
    • For PRs that fall outside of the desired spectrum, review the details and check if:
      • Your PR could be split in smaller, self-contained PRs instead
      • Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

  • One line was added: +1 -0
  • One line was deleted: +0 -1
  • One line was modified: +1 -1 (git diff doesn't know about modified, it will
    interpret that line like one addition plus one deletion)
  • Change percentiles: Change characteristics (addition, deletion, modification)
    of this PR in relation to all other PRs within the repository.


Was this comment helpful? 👍  :ok_hand:  :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

@ghost ghost added the Review - Needed The PR is being reviewed label Feb 9, 2022
@ghost
Copy link

ghost commented Feb 9, 2022

This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Maintainer, please provide feedback and/or mark it as Waiting on Author

Copy link
Contributor

@PaulHigin PaulHigin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks Ok to me. But still need @TravisEz13 look at the regex.

@PaulHigin PaulHigin merged commit 72e3d71 into PowerShell:master Mar 25, 2022
@ghost ghost removed the Review - Needed The PR is being reviewed label Mar 25, 2022
@SteveL-MSFT SteveL-MSFT deleted the vt-remove branch March 30, 2022 22:01
@TravisEz13
Copy link
Member

@PowerShell/powershell-maintainers Let's get at least one preview release before we approve backport

@daxian-dbw
Copy link
Member

@SteveL-MSFT Just noticed this PR changed the default OutputRendering back to Host. Is that intentional? I thought we changed it to ANSI for a security issue. Is that security issue resolved after using the new regex for detecting ANSI sequences, so we can change back to Host? /cc @TravisEz13

dgoldman-msft added a commit to dgoldman-msft/Get-IntuneClientLogCollection that referenced this pull request May 6, 2022
Formatting work around for: PowerShell/PowerShell#16811
@ghost
Copy link

ghost commented May 23, 2022

🎉v7.3.0-preview.4 has been released which incorporates this pull request.:tada:

Handy links:

@ghost ghost mentioned this pull request May 23, 2022
5 tasks
@God-damnit-all
Copy link
Contributor

@SteveL-MSFT Just noticed this PR changed the default OutputRendering back to Host. Is that intentional? I thought we changed it to ANSI for a security issue. Is that security issue resolved after using the new regex for detecting ANSI sequences, so we can change back to Host? /cc @TravisEz13

daxian, were you able to track down someone to answer this question for you, in another PR/Issue/etc perhaps? I'm curious to the answer myself.

@daxian-dbw
Copy link
Member

My chat with @TravisEz13 suggested this is an intentional change, and Host will be used as the default unless another vulnerability is found with the regex.

@adityapatwardhan adityapatwardhan added the Review - Maintainer The PR/issue needs a review from the PowerShell repo Maintainers label May 24, 2022
@adityapatwardhan
Copy link
Member

@PowerShell/powershell-maintainers should discuss whether this should be backported.

@adityapatwardhan
Copy link
Member

Maintainers discussed this and have decided to wait until the committee has given a decision on #17455
Potentially backport both the changes to 7.2.x

@adityapatwardhan
Copy link
Member

/backport to release/v7.2.6

@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2022

Started backporting to release/v7.2.6: https://github.com/PowerShell/PowerShell/actions/runs/2784876485

GitHub
PowerShell for every system! Contribute to PowerShell/PowerShell development by creating an account on GitHub.

@ghost
Copy link

ghost commented Aug 11, 2022

🎉v7.2.6 has been released which incorporates this pull request.:tada:

Handy links:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Backport-7.2.x-Done Extra Small Review - Maintainer The PR/issue needs a review from the PowerShell repo Maintainers WG-Security security related areas such as JEA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

out-file producing weird characters

7 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.