I briefly checked your app, I think you can return a tuple from the function (string, filename), then store them in two separate columns. Later ingest/write to the file with pw.io.subscribe based on row values.

I did not get this point. Can u show me an example?

I think this is very likely the result of io writes in the UDF (user defined function). The functions you put into doc_post_processors are converted to pw.UDFs and applied to tables inside of the VectorStoreServer.

and why is this given as an option if it creates this issue? am i using it wrong?

more specifically, taking this rag example, can you tell me the flow so this doesnt cause max cpu usage..?

I did not get your answer. More specifically the flow, what should be the flow? As I am using pw.io.fs.read, should I make the changes here to add additional metadata column using pw.io.csv and then add vector server and not pass doc_post_processors in it?

also i remove writing to file inside doc_post_processor functions, its still hogging cpu resource, I am just extracting metadata adn saving

If this is happening even after removing the file save, this is not expected, I will try to replicate and get back to you.
That much CPU usage is not normal without any io write in the UDFs.

I did not get this point. Can u show me an example?

I meant something like this (while removing the doc_post_processors and keeping the rest of the code same):

def save_file_callback(key, row, time, is_addition):
    file_name = row["filename"].value
    write_success_metadata_files(row["date"], filename)

@pw.udf
def extract_published_date(doc) -> tuple[str, str]:
  data: dict = doc.as_dict()
  text = data["text"]
  metadata = data["metadata"]
  file_name = metadata["path"]
  ...
  return parsed_date, file_name

docs_table = vector_server._graph["parsed_docs"]

docs_table += docs_table.select(extract_tup=extract_published_date(pw.this.data))

result_table = docs_table.select(date=pw.this.extract_tup[0], filename=pw.this.extract_tup[1])

pw.io.subscribe(result_table, on_change=save_file_callback)

even after removing doc_post_processor functions, after all ingestion pathway still takes up 130% on average

Ok. Thanks for info. I'll investigate it and will get back to you once I have some conclusions.

ya cool; buzz me if you guys need additional info

Hey @sarim2000, one fix is already there. If you install pathway v0.13.2, the cpu usage in idle state should be decreased significantly (in our tests it's on 4% now). We're working on decreasing it even further but hope that this fix will already make the app usable for you.