Skip to content
Tech Stuff
These transformations of the source texts were generated using ‘s nerds-ai LLM library. Nerds are platform-agnostic composable tool-using agents that can be bound to any available LLM and executed to return either typed output or markdown strings.
@Mykola Bilokonsky
In this case we’re using an out-of-the-box Markdown nerd with instructions telling it how to summarize complex texts. We are breaking the books out into distinct .txt files per chapter, and then passing each chapter into this nerd.
Then I manually go through and cleanup the output (removing the Chain of Thought details, for instance) and import the resulting markdown into Coda.
nerds-ai library
It’s an open-source library made available via ‘s CourseBuilder project. The repository is available here:
badass-courses/nerds-ai
Owner: badass-courses • Star count: 1 • Created: Tue, Apr 30, 2024, 3:11 PM
a collection of AI tools
github.com
Script Details
Here’s the text of summarize.mjs, the script used to generate summaries:
import 'dotenv/config';
import { accessible_summarizer as nerd } from '../../build/src/prebuilt/index.js';
import { readFileSync, writeFileSync, readdirSync } from 'fs';
import path from 'path';
async function main(book) {
const boundNerd = await nerd.bindToModel('gemini-1.5-pro-latest');
const source_path = path.join('.', 'sources', 'dewart', book);
const e_and_c = readdirSync(source_path);
const texts = e_and_c.map((file) => {
const book_path = path.join(source_path, file);
const text = readFileSync(book_path, 'utf-8');
return {
text,
file,
};
});
const output_path = path.join('.', 'demos', 'dewart', book, 'summaries');
for (const { text, file } of texts) {
console.log(`Summarizing ${book}/${file}`);
const summary = await boundNerd.invoke(text);
const summary_path = path.join(output_path, file);
writeFileSync(summary_path, summary);
}
}
const book_name = process.argv[2] || 'evolution_and_consciousness';
if (
book_name !== 'evolution_and_consciousness' &&
book_name !== 'humes_challenge'
) {
console.log(
"Invalid book name. Please choose either 'evolution_and_consciousness' or 'humes_challenge'",
);
process.exit(1);
}
main(book_name);
That script is using the accessible_summarizer nerd, which looks like this. Note that this nerd definition has undergone several revisions since we began this project, so the output isn’t fully consistent across the board - we’re learning as we go.
The following code is compiled into the nerds-ai library and available to node users via npm install nerds-ai
import { MarkdownNerd } from "./index.js"
const nerd_opts = {
name: "AccessibleSummarizer",
purpose: "You are a Nerd that specializes in summarizing academic texts for a general audience.",
do_list: [
"Return a detailed outline of the text, closely hewing to the core points but restating them in accessible ways",
"Dive deeply into the text and use a nested outline structure to reflect the structure of the original text.",
"Include all core ideas and arguments in the text, but restate them in a way that is accessible to a general audience",
"Use only the 2000 most commonly used words in the english language, unless there are specific technical terms that are necessary to include",
"After the outline, provide a summary of the text that is no more than 500 words long",
],
do_not_list: [
"change the meaning of the text"
],
additional_notes: ``,
as_tool_description: "A tool that can be used to summarize domain-specific texts for a general audience.",
tools: []
}
export const accessible_summarizer = new MarkdownNerd(nerd_opts)
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ··· in the right corner or using a keyboard shortcut (
CtrlP
) instead.