> [!META]- Inline Metadata > [tags:: #project/done #concepts/programming/machine-learning/large-language-models] > [status:: done] > [priority:: 1] > [due date:: [[2023-05-14 Daily Note|2023-05-14]]] > [completed date:: [[2023-05-14 Daily Note|2023-05-14]]] > [link:: https://github.com/kylestratis/llm-knowledge-agent] > [related goals:: ] > [people:: ] > [organizations:: [[Tribe.ai]]] ## Overview **Description**: An agent to research a given topic by doing [analytical and syntopic reading](https://fs.blog/how-to-read-a-book/#:~:text=4.-,Syntopical%20Reading,ideas%2C%20vocabulary%2C%20and%20arguments.), summarizing and extracting main points, connecting them together, and forming new knowledge outputs such as [zettels](https://zettelkasten.de/posts/overview/) or [evergreen notes](https://codex.erisianrite.com/Bins/Evergreen+Notes/What+Is+An+Evergreen+Note+Anyways). ## Lessons Learned - ~~Document embeddings with transformers don't seem to be too practical. Instead, go with a sentence embedding for the knowledgebase using note titles and search with main idea embeddings. - Dispatch table values need to the the function as an object, so without the parentheses. Otherwise they'll be run when instantiated. ### Project Notes ### Future Improvements Future improvements to the project should go here for easy reference. They should be a type of project note and link back to a Project page and any tickets that come from it. ## Resources ## Project Plan ## Designs Task: Ingest input text Because we will be reusing the input text, store it in memory and make the embedding in pinecone(?) Task: Ingest existing knowledgebase Add all items to vector DB as embeddings for later search Strip out metadata Summarizer - summarizes text Main point/argument finder - finds the main points or arguments of the article Assertion finder - finds assertions of fact Knowledge generator - generates zettels (how? Just use main points? Expand on them?) Zettel connector - uses semantic similarity search to find related zettels and either add to them or use backlinking to connect them ## Questions ### Summarizer The summarizer endpoint from Cohere has a limit of 100k characters. For a long text, how should we break it down? I have two medium-difficulty options I see, and 1 really easy one: 1. **EASY**: Don't summarize texts longer than 100k characters. That's a problem for future me. One of my long RP articles with code and all (my Sentiment Analysis one) is just under 46k characters. 2. **Medium**: Tokenize into sentences and do a "center crop" - assumes that least information-dense parts of an article are the ends of it, which isn't always the case. 3. **Medium**: Chunk text into sub-100k chunks, summarize them separately, then weave them back together. This can be costly for large texts and if done should warn the user. Going to go with #1 for now, then create a Github issue for handling larger texts. Will probably warn the user and give some options? This is due to using the Transformer architecture - it's harder to carry long-term memory than a recurrent network (but transformers give us better outputs). What if we need context from outside documents? - Tomasso ## Journal