Convert mpc and msbuild task package to a roslyn source generator #1599

AArnott · Apr 12, 2023

This completely replaces mpc and the msbuild task package with a C# roslyn Source Generator.

NuGet package impact:

Package	Impact
`MessagePack.MSBuild.Tasks`	Deprecated, and no longer built. This carried the msbuild integration that ran source generation before invoking the compiler. Users should stop referencing it and switch to `MessagePack.SourceGenerator`
`MessagePack.Generator`	Deprecated, and no longer built. This carried the .NET CLI tool `mpc`. Unity users should now consume the source generator contained in the `MessagePack.SourceGenerator.Unity.zip` file.
`MessagePack.SourceGenerator`	The new roslyn source generator for msbuild/VS users.
`MessagePack.SourceGenerator.Unity.zip`	A new zip archive (not a nuget package), containing a source generator that targets Roslyn 3 for use by unity.

Ideas for future work (after this PR)

Now that source generation can be totally automatic, should it just be on by default, and should we find a way to always find the generated formatters instead of requiring the user to specify the right resolver? For example, the source generator could add a [MessagePackFormatter] attribute on each [MessagePackObject] (provided the class is declared as partial) so that the default in-library resolvers will find the source generated formatter.
A massive refactoring of the TypeCollector and analyzers so we don't have so much repeated and inconsistently updated code. For example, the check that the type passed to MessagePackFormatterAttribute derives from IMessagePackFormatter<T> is only performed for one possible place where the attribute can appear instead of everywhere it may appear. Field and property searches are also repeated in many places, again with inconsistent code handling them.

Remaining tasks

Add tests for the Roslyn3 project.
Confirm that this works with Unity.
Confirm that incremental source generation works, including that every type in the FullModel type hierarchy has functional equality comparers.
Stop building a roslyn3 nuget package and instead build a .zip that will be added to github releases alongside the .unitypackage we already ship.
Rename the Generator package to SourceGenerator so we aren't radically changing the type of an existing package, which will likely cause more confusion than no longer updating an existing package.
Remove diagnostic reporting from generator and add a dependency on the analyzer package.
Re-enable or remove the remaining Compile Remove items in the source generator test projects.
Unify TypeCollector across source generator and analyzer.
Source generator (and analyzer) to use precise assembly versioning

Closes #1495

- remove project dependency - update dependency package version

The source generator will altogether deprecate this.

It presumably should be 3.8, but it doesn't compile with 3.8 at this point.

This will help with cacheability.

…agePack-CSharp into sourceGenerator1495

* Consolidate `TypeCollector` across analyzer and source generator. * Use `System.Text.Json` instead of `TinyJsonReader`. * Remove all diagnostic reporting from the source generator. * Make the source generator depend on the analyzer, so that user gets diagnostics while using source generator by way of the analyzer.

AArnott · Apr 12, 2023

@Y-YoL let me get the unity build working, and then can you try with the source generator again? I suspect it will only work when you set the analyzer label on the System.Text.Json.dll file as you do with the other source generator DLLs.

src/MessagePack.UnityClient/Assets/Scripts/MessagePack/Resolvers/StandardResolver.cs

AArnott · Apr 12, 2023

@Y-YoL do you have any idea how to resolve this warning that appears a lot in the unity build?

warning CS8632: The annotation for nullable reference types should only be used in code within a '#nullable' annotations context.

…e-from-unity' into sourceGenerator1495

AArnott · Apr 12, 2023

I guess the CS8632 warnings are a pre-existing issue. I'd still love to solve it, but it's not blocking this PR.

The only important part is "additionalCompilerArguments", but there are many differences because Unity automatically migrates

Y-YoL · Apr 12, 2023

Regarding CS8632 in Unity, it is necessary to write in the configuration file.
Try #1600

…e-setting Set nullable in unity

AArnott · Apr 12, 2023

I have unity installed and it looks like the source generator broke somewhere along the way. Probably when I added the dependency from the source generator to the analyzer project. I'm looking into it now.

This is important for dependency reasons, since code fixes bring in more dependencies than analyzers, and `dotnet build` and unity don't contain the dependencies required for code fixes.

AArnott · Apr 12, 2023

I fixed the build issues in unity it seems, but the analyzer and source generator no longer work at all (no errors, no warnings, no code gen). I had them working a while back, so I guess I broke something. Any ideas, @Y-YoL?

AArnott · Apr 12, 2023

Downgrading the System.Text.Json dependency to 5.0.0 gets the analyzer and source generator to work.

Y-YoL and others added 30 commits December 2, 2022 21:39

Change globa.json rollForward to feature

a8eba95

Change generator project setting

fe3c6e5

- remove project dependency - update dependency package version

Add dependency reference

539f7d7

Replace Generator

ceefa17

Replace GeneratedResolver to ModuleInitializer

9f695cc

Add Generator.Roslyn3 csproj

4db4342

Fix GetFormatterWithVerify missing error

e962638

Fix array formatters is not registered

af3f8e4

Fix formatter duplicate definisions

51dac96

Merge in source generator work by @Y-YoL

7652a91

Delete MessagePack.MSBuild.Tasks project

f911a00

The source generator will altogether deprecate this.

Refactor source generating projects

267ba7f

Rollback Roslyn3 to 3.9

4763459

It presumably should be 3.8, but it doesn't compile with 3.8 at this point.

Got first new test passing

7ede56c

Fix package restore

4d99531

Remove checked-in generated code

c9e9a10

Get the first source generator test running as it should

19c252c

Merge branch 'develop' into sourceGenerator1495

339bd72

Simplify test pattern

cf1e1d4

Remove unnecessary new public API

d6b11bb

Delete some dead code

0069a97

Break up the outputs into individual files

3c27946

This will help with cacheability.

Get rid of unnecessary arrays in transforms

9bdd6f1

Split out Definitions into separate files

da6659b

Remove useless interface

1098dac

Use records for brevity

306bf9b

Share namespace prefixing code

54a2eb2

Condense #pragmas in generated code

bb71f8c

Capture full model in a record

ff5afe2

Start refactoring TypeCollector

e9ce2d6

AArnott and others added 7 commits April 10, 2023 08:57

Merge branch 'sourceGenerator1495' of https://github.com/AArnott/Mess…

4a6f399

…agePack-CSharp into sourceGenerator1495

Migrated 1 of 3 old test files

0d0e63e

Migrate second of 3 tests

88640ce

Migrate last of the tests

3753c27

Get all tests passing or skipped

61487fc

Fix compile error in Unity

72d1aeb

AArnott added this to the v2.6 milestone Apr 12, 2023

AArnott requested a review from neuecc April 12, 2023 13:56

AArnott mentioned this pull request Apr 12, 2023

Convert mpc and msbuild task package to a roslyn source generator #1582

Closed

9 tasks

Y-YoL reviewed Apr 12, 2023

View reviewed changes

src/MessagePack.UnityClient/Assets/Scripts/MessagePack/Resolvers/StandardResolver.cs Show resolved Hide resolved

Merge remote-tracking branch 'yol/sourceGenerator1495-remove-immutabl…

a50fe74

…e-from-unity' into sourceGenerator1495

Set nullable in unity

de00d6b

The only important part is "additionalCompilerArguments", but there are many differences because Unity automatically migrates

Merge pull request #1600 from Y-YoL/sourceGenerator1495-unity-nullabl…

7440007

…e-setting Set nullable in unity

AArnott added 3 commits April 12, 2023 09:48

Split code fix out of analyzer assembly

0b5e5d9

This is important for dependency reasons, since code fixes bring in more dependencies than analyzers, and `dotnet build` and unity don't contain the dependencies required for code fixes.

Add incremental clean of zip staging directory

40a53f2

Rename analyzer test project

46ab8bf

AArnott and others added 2 commits April 12, 2023 10:50

Update documentation for using source generators

69f78d1

Fix analyzer and source generator in unity

8dc2545

AArnott merged commit 1d0ad5f into develop Apr 12, 2023

Y-YoL mentioned this pull request May 31, 2023

Mark of the Web flag on downloaded analyzer files break source generation in unity #1628

Closed

neuecc mentioned this pull request Jun 28, 2023

remove MessagePackWindow in Unity #1651

Merged

AArnott deleted the sourceGenerator1495 branch January 22, 2024 14:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Convert mpc and msbuild task package to a roslyn source generator #1599

Convert mpc and msbuild task package to a roslyn source generator #1599

Uh oh!

AArnott commented Apr 12, 2023 •

edited

Loading

Uh oh!

AArnott commented Apr 12, 2023

Uh oh!

Uh oh!

AArnott commented Apr 12, 2023

Uh oh!

AArnott commented Apr 12, 2023

Uh oh!

Y-YoL commented Apr 12, 2023

Uh oh!

AArnott commented Apr 12, 2023

Uh oh!

AArnott commented Apr 12, 2023 •

edited

Loading

Uh oh!

AArnott commented Apr 12, 2023 •

edited

Loading

Uh oh!

Uh oh!

Search code, repositories, users, issues, pull requests...

Uh oh!

Convert mpc and msbuild task package to a roslyn source generator #1599

Convert mpc and msbuild task package to a roslyn source generator #1599

Uh oh!

Conversation

AArnott commented Apr 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

NuGet package impact:

Ideas for future work (after this PR)

Remaining tasks

Uh oh!

AArnott commented Apr 12, 2023

Uh oh!

Uh oh!

AArnott commented Apr 12, 2023

Uh oh!

AArnott commented Apr 12, 2023

Uh oh!

Y-YoL commented Apr 12, 2023

Uh oh!

AArnott commented Apr 12, 2023

Uh oh!

AArnott commented Apr 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AArnott commented Apr 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

AArnott commented Apr 12, 2023 •

edited

Loading

AArnott commented Apr 12, 2023 •

edited

Loading

AArnott commented Apr 12, 2023 •

edited

Loading