I had a look at the specification. It seems to be simple enough, that a self implemented solution could be feasible. One question would be which language to choose for the server. Cpp2 or Cpp1? You proposed one other interesting idea that Cppfront should output all information about the sourcefile in a json file. The server could then use this information. A short search did not provide any insights if such a framework exists. I also did not found a general framework for implementing a language server.

Herb specifically did not want to use any frameworks when implementing cppfront. Maybe this is also a way for the language server?

So what would you favorite way be?

• cppfront -> details in json file -> server that gets information from the json file?
• cppfront -> internal data structure -> server that works on the internal data structure?

Hmm, the choice of which language to use was one I was thinking about too. I think using c++ is the fun option, and cpp2 is the even funner option, but I don't know that it's really necessary. The easiest option would be to just use typescript - then it's just a matter of implementing the couple of LSP feature functions we need and we're done. All of this is already stubbed out in that link I shared/the example repo they provide; I copied that example and began modifying it to implement the cpp2 functionality I shared in my comment below.

Not that I'm necessarily attached to the idea of using typescript; perhaps there are reasons why c++ is the better option that just I haven't thought about.

But I think regarding your question of how we work with cppfront: whether to use json or internal structure depends on which language we pick for the language server. I think we can only use internal structure if we choose cpp/cpp2, otherwise we must use json (or some other serialized format) to interface with the data. Using json gives a nice interface that allows us more flexibility of what language we want to implement in

So yea, those are my initial thoughts on it. What do you think?

From what Herb wrote below, I think the currently minimal invasive approach would be to have an to_json.h file. So that this can actually be merged into cppfront. Thinking about it, it has 3 advatages:

• Writing out first simple json items should be straight forward.
• Writing the server in a separate project removes cppfront contraints and libraries can more easily be used. E.g. a json parser.
• It lets you (us) gather experience on what kind of data is actually required.

Migrating the server to cpp1/cpp2 should then be simpler, since we know what kind of data we need. The to_json.h could then be changed into an to_lsp.h, which produces the required data structure directly.

Would this be feasible?

Yea, if we think implementing the language server in c++ is a goal worth pursuing, then this is probably the way to go. Really I think we need to aggregate this data regardless of what path we choose. As I've been playing with the cppfront code, I looked at the debug_print code that gets triggered when you use the -debug flag and you can see in that function that you basically have access to all of the compilers diagnostics right there. (Namely, source, tokens, parser, and sema)

So I think the first step would just be to aggregate the data from these fields into something that is more easily consumable from another program. I think particularly, the names of all symbols defined in the file, and all of the errors definitions (which can be found in sema.errors). I've been poking at this a bit but haven't made too much progress yet

This sound like a good first step. If you need some help just contact me.

Nice!

Re JSON: I don't have an issue with that, but just curious what other compilers do -- do they have switches for emitting errors as JSON, or do they do something else?

My experience with compilers is super limited, but IIRC, I believe at-least the Purescript guys did this with their compiler to make it easier to integrate tooling.

Actually, as I'm thinking about it more, I think they didn't just emit errors as json, but a whole slew of compiler information was written as json. If that's the case, the PR I just opened may be too limited in scope, but maybe its still a good starting point

While not all compilers here are a few 'compiler adjacent' tools that output json stream (json objects separated by newlines). I'm sure there's plenty more but these are the ones off the top of my head

• bazel query - --output=streamed_jsonproto (json to stdout)
• bazel build - --build_event_json_file=/path/to/json (json to file)
• rustc (compiler!) - --error-format=json (json to stderr)
• fastbuild - -report=json (json to stdout/stderr)

Nice!

Re JSON: I don't have an issue with that, but just curious what other compilers do -- do they have switches for emitting errors as JSON, or do they do something else?

Clang has https://clang.llvm.org/docs/UsersManual.html#formatting-of-diagnostics
I believe they also support outputting SARIF, which I believe to be structured JSON

Ok, sounds good. I never programmed in type script. So you are probably faster in doing it? Or should I have a look?

It's probably faster for me to do it. I'll see if I can give it a look tomorrow. But feel free to check it out if you want. Typescript is super easy, I'm sure you'd get the hang of it quickly haha

Ok - I pushed an update that implements the merging logic, so it should now do a better job of not losing symbols in the symbol table when errors occur in the document

It works now quite well now, even on reflect.h2. There seems a subtile bug, where the server loses the definitions again and you get only the text completion from VSC. I do not know if this is because of the server or VCS. I could not find a procedure to reproduce it consistently.

What is your plan for next steps? From my point of view two things are missing:

• hints/completion for function arguments. Like name of the argument, different versions
• completion for members of structures
  

This probably only concerns the generated json file, or are these special commands to the server. I will have some time in February where I could have look.

What is your plan for next steps? From my point of view two things are missing:

• hints/completion for function arguments. Like name of the argument, different versions

...
This probably only concerns the generated json file, or are these special commands to the server.

I think it's both for this one. When the type of symbol is something more sophisticated than a variable, e.g: a function or class, we need to augment the symbols in the json diagnostics with that information. During the check for the symbol type, I think sema may have more info available, such as function argument type info.

Once we have this info, we need to augment the server to read it in and include it in the language servers completion hint details. This is likely trivial compared to collecting the info

• completion for members of structures

Specifically regarding this one, I was looking into it but from what I was seeing, cppfront doesn't seem to capture any symbol information for class/struct members variables. I opened an issue for that:
#1323

I don't know if it was a bug or intentional - it could be that the intention was just to defer those symbols to the cpp compiler.

There seems a subtile bug, where the server loses the definitions again and you get only the text completion from VSC.

I noticed a similar weird bug but as you said, it wasn't consistent. It may have to do with the fact that cppfront is generating a new diagnostic file every single key press - it could be working out the file system with too much io. Adding a debouncer to the compiler that waits for a 1-2 seconds of inactivity before recompiling the file might help here.

The other big thing is I wanted to start capturing symbols from the cpp compiler, but this info isn't published in the sarif output of the compilers. I think its output to the .obj files or something of that sort, so that part will take a bit more time to implement.

How do you see interactions between Cpp1 LSP (MS C/C++ or clangd) and Cpp2 LSP?
...
In order to solve this, you either need 1 LSP that understands both Cpp1 and Cpp2 ...

This is basically the answer, at least in my mind. From what I've learned so far working on this language server, there's really no way to make the server fully functional without incorporating c++ diagnostics. Trying to interface two language servers together also seems like tackling the problem at the wrong abstraction level.

I did initially explore this problem from the angle of trying to extend something like clangd to understand cpp2, it intuitively feels like it wouldn't be that hard, but after some discussion with the llvm folks, it appeared that this approach requires non-trivial work going directly into clang itself. That's bad because it's hard and because a cpp2 language server shouldn't be dependent on a specific compiler.

So in the end, I think the right approach is to pull diagnostics from cppfront, and then compile the file and pull diagnostics from the users chosen c++ compiler. As far as I've read, all of the major c++ compilers do have the ability to output diagnostics in a way that can be consumed by external tools - I assume this is how the other c++ language servers work. The cpp2 server will also need to manage mappings from cpp1 code to cpp2 so we can consume the cpp1 diagnostics correctly. Generating and consuming the cpp1 data is the next major piece I'll be looking at soon.

I saw that discussion and responded in it, though it felt like it stopped without a clear track forward.

Consuming the Cpp1 diagnostic and mapping it back to Cpp2 seems indeed like the most reasonable approach. I'm wondering if this is something you can do for most of the LSP calls.

For example: syntax highlighting of Cpp1 code,except the location, can stay unmodified, as soon as it reaches Cpp2, you include your own code. Fixing the offsets seems quite possible.

For refactorings (rename, tidy, formatting), you most likely need to translate changes to the generated code and map it back to Cpp2. Especially decomposing the changes will be challenging here.

For auto-complete you might be able to first map your cursor onto the generated code and pass it along after which you can convert back. The only annoying thing here (and any LSP) is that you should be able to deal with incomplete code. In case of Cpp2, that would imply generating broken Cpp1 code first.
The same will most likely hold for hover.

As far as I'm aware, there are only 1-3 calls where the full source code is send to the server, all others use a location in the previously known code. So if you want to do preprocessing there, it should be possible. For all other requests, you mainly should be able to do the mapping of location towards the Cpp1 LSP and do the reverse mapping and code rewrite on responses.

The background indexing is most likely going to be the most challenging as it will find preprocessed files without going through your wrapper.

The more I think about this, the more feasible it seems to write a wrapper around the whole LSP and let the Cpp1 LSP handle the majority of the work, even for Cpp2 syntax. Although getting all details right is going to be challenging. This should in theory be LSP unaware, though every LSP has its own extensions.
It still sounds more complex than teaching Clang to parse Cpp2, though even in that case you need to somehow give responses in Cpp2 syntax.

If this works out, a more integrated plugin approach can be introduced in clangd and others LSPs, which might also work for lex/yacc and other code generators. Maybe some generic meta-data can simplify it even more if we have more of them. Though that's something to only worry about once something works.

I'm not convinced that you should be using every compiler as backend for your LSP. Even Microsoft uses a separate parser for intellisense than the compiler. It also doesn't change when you would compile with GCC for Linux or Clang for any platform.

So it seems reasonable to focus on a single LSP first. If you don't want to wrap completely, you can hack into your own clone of clangd. All requests come in here: https://github.com/llvm/llvm-project/blob/229abcd459dc365201185aa3988b9f8ae455de76/clang-tools-extra/clangd/ClangdLSPServer.cpp#L1659

I wish I had the time to actually assist in coding.

I've been thinking about what you said here. I think you're right. I just realized that going down the path of integrating diagnostics from c++ compilers is essentially writing a language server for c++. It seems like the easier approach would be to clone a c++ language server and augment it with cpp2, rather than the reverse, otherwise we're duplicating a lot of effort!

Thanks for the reference to the clangd code. I was poking around on that project a bit but hadn't really made sense of it yet. The link you gave provides a nice starting point.

I have some good news to report regarding the concern of building for multiple compilers - it actually looks like Microsoft has been working on a diagnostics standard that will allow for easier consumption from multiple compilers (as well as other dev tooling). The search took me a while, but whilst trying to figure out how to get compiler results in json from the major compilers (msvc, clang, and gcc), I discovered that the json-like format they all share in common is something called SARIF.

Even though its duplicate work, creating a language server that pulls its diagnostic results directly from the compiler could be beneficial to give more accurate error messages. (I've already noticed both clangd and microsoft's c++ ls both have issues where they report errors that aren't really there) As a bonus, it also makes the language server more flexible since it can work with whatever compiler the user chooses to use. So yea, perhaps even though it's duplicating effort in some sense, it could be worth it since sarif standardizes results (hopefully meaning we only need to duplicate the effort once, and then it works for all sarif compliant compilers afterwards). Gonna experiment with stuff more

I just noticed you mentioned SARIF in your other response up above. Just for general info for anyone reading, each compiler can output to sarif as follows

clang++ -fdiagnostics-format=sarif file.cpp
cl.exe /EHsc /experimental:log .\sarif-output-name .\file.cpp
g++ -fdiagnostics-format=sarif-file file.cpp