Skip to content

Screen annotations

Agents can draw on-screen annotations, similar to the ones Grammarly creates. These can be a useful way for your agent to collaborate with users on their work.

Currently, screen annotation can only be added to editable text, like an email draft, document content, etc.

Underlined suggestions

Underlined suggestions are available when the agent runs from a while-writing trigger. Two underline types are supported: rewrites and comments.

Rewrite cards allow the agent to suggest alternate text, similar to how Grammarly works. Clicking the card applies the replacement and removes the card.

A rewrite annotation offering replacement text

Comment cards can be used to provide context or more nuanced recommendations.

A comment annotation with a fact about a dog breed

To add these annotations, simply mention them in your agent's instructions.

agent.addDefaultRoutine({
  name: "More dogs",
  trigger: {
    kind: sdk.DefaultTriggerKind.WhileWriting,
    condition: "The writing mentions a dog or a dog breed.",
  },
  instructions: `
    When the writing talks about a dog in a negative way, rewrite
    it to have a more positive spin.
    When the writing mentions a dog breed, add a comment with a
    fun fact about that breed.
  `,
});

Rewrites are plain text only

Rewrite suggestions can only include plain text as the suggested replacement text. It's not possible tu suggest alternative text that includes formatting, images, etc.

Paragraph-level rewrites

The while-writing trigger runs without user input, but sometimes the agent needs more information before it can suggest changes. Agents running in the chat panel can suggest rewrites, but not add comments.

Each rewrite applies to an entire paragraph and appears as a dot beside it. Hover over the dot to open the card, then review each paragraph's rewrite separately.

A dot beside a paragraph, with the rewrite card open on hover

agent.setInstructions(`
  When the user asks to make their writing sound like a dog, rewrite it to 
  include dog noises.
`);

Providing context

Screen annotations can provide context to the user through their title and/or explanation. The model fills these in as it's creating the annotation.

The title is shown at the top of the card and the explanation is shown when the user hovers over the info icon.

The title and explanation on a rewrite card.

The title is shown at the top of the card and the explanation is shown in the body.

The title and explanation on a rewrite card.

The explanation is shown at the top of the card (as a title).

The title and explanation on a rewrite card.

In underlined suggestion cards, explanations can contain rich text. They accept the same set of markdown formatting as chat messages. See the Chat guide for more information.