hey, thank you so much for your reply and yes that is essentially the same question. If I am really honest though I still don't get in practice how one would code the script to bring in own chunks (i am going to assume for the moment that i don't care to bring the embedding of chunk content and happy to use whatever graphrag uses).

I have been following the documentation and i can use graphrag in its "basic way" where you ran the indexing code starting from a txt document.

If i want to just skip the chunking phase and start from my own chunks what is the best way to do this? do i need to create my own workflow using that language or is there any other way?

I think a bit more of a deep dive in the code base or a minimal script that shows how one would use graphrag for this use case would have been good because it's unclear to me how to achieve this from just the docs.

Thanks in advance!

If you want to skip our chunking and start from your own chunks, replace the starting txt document with individual txt documents for each chunk. As long as each of those documents are shorter than your configured GraphRAG chunk_size, our chunking will be skipped and your chunks used directly. The actual process/config for running GraphRAG in this case does not change at all, you have just supplied a different set of input documents.

If you need to create your chunks, I would suggest using tiktoken, which has encode/decode methods to match the encoding of your model. So your script would encode the document into a list of tokens, iterate through the tokens to subdivide them into sublists that are shorter than your chunk_size, and then decode those token lists back to text that you can write to a file.

Hey, Could we pass a .faiss and .pkl embedded versions of out .txt files in the input folder? will that work? or the input needs to be either .txt, .json......?