arrow-left

All pages
gitbookPowered by GitBook
1 of 1

Loading...

Create New Extension

Extensions allow adding additional functionality to omnitool, both on client and server side. They can provide a convenient way for prototyping AI powered applications.

⚠️ Extensions run with the same permissions as the omnitool process and have full access to the omnitool server process. They should be treated like any other npm package or executable code from the internet, with extreme caution. ⚠️

hashtag
Getting Started

Creating a custom extension for Omnitool is streamlined with our easy-to-use template. Here's a step-by-step guide to create, develop, and test your extension:

  1. Access and Use the Template Repository:

    • Visit the on GitHub. This template provides the basic structure needed for your extension.

    • Create a new repository from this template, ensuring that it's set to public visibility.

By following these steps, you can effectively create and develop a functional and innovative extension for Omnitool. This guide is designed to assist both beginners and experienced developers in navigating the extension development process with ease.

hashtag
Extension Structure

hashtag
extension.yaml

For an extension to be recoginized by the server, it needs, at minimum, an extension.yaml file in the directory.

hashtag
Directory Structure

The following directory structure may, optionally, exist under the extension's sub directory

  • public - Files in this directory are served to the client under server_url/extensions/<extension id>. By default, if client.addToWorkbench is true, the client will attempt to show public/index.html if it exists when a users selects the extension in the extension menu

  • scripts/client - Files in this directory, following the script format, will be exported to the client, adding to the list of known /chat commands. The chat command will be mapped to the file's name, so test.js would become /test. For more details, see Client Programming/Scripts below.

hashtag
Extension Discovery

Currently the server keeps track of known extensions via the /etc/extensions/known_extensions.yaml file. An entry in this file will make extensions discoverable by the omnitool community. We plan on adding the ability to add additional extensions repositories outside the official one.

The file's format is straightforward:

An entry in this file is not required for an extension to be loaded, only for it to be discoverable by other omnitool users.

hashtag
Client Extension Programming

The client, after successful login, polls the /extensions endpoint on the server to retrieve a list active extension. It then constructs the extension menu and compiles the received client scripts.

Upon being shown, client extensions are loaded into an iframe overlaying the canvas by default and has access to the omnitool client context via the window.parent object. This is done by calling (window.parent.client.)workbench.showExtension(extension-id, openArgs ), usually from automatically from within the extensions menu or from a client script.

Closing the extension will show the canvas again but not unload the iframe. However, the extensions state is not guaranteed, as showing any other extension will replace the iframe content. If it is necessary to persist client state, local or session storage can be used or data could be marshalled via server scripts.

hashtag
Scripts

Scripts in the client/scripts subdirectory of the extension are automatically registered with each connecting client after successful authentication and become available as /chat commands.

Within scripts, full access to the omnitool client is accessbile via the window.client object.

To display the index.html file surfaced via the extensions public/ directory, the window.client.workbench.showExtension("extension-id", {...}); provides a convenient method that also allow marshalling 'opening args

When the workbench.showExtension command is used, the full object structure of the arguments object will be serialized (JSON. strigified) into a parameter q in the opening url.

The following example code shows how to deserialize the opening args inside the extension's index.html

hashtag
Server Extensions Programming

When the server starts, it will execute the server/extension.js file for each extension, if present.

This file can:

(1) Attach hooks to server events to allow running code when these events happen (2) Export a list of blocks to be registered with the server.

A minimally viable extension.js looks like this

hashtag
Event Hooks

Event hooks are events exported by the server (see server/src/core/ServerExtensionsManager.ts) that extension can hook into. by default, these hooks are executed synchronously, giving the extension the ability to modify execution parameters or, in some cases, even cancel the execution.

Currently, the following events are implemented:

The first parameter of each event is an event context, followed by a variable list of parameter depending on the event.

For example:

hashtag
Block Factory

Extensions can add blocks to the omnitool. Unlike API based blocks imported from the registry, extension components have the ability to execute javascript code, allowing them to encapsulate useful nodejs libraries or custom code to provide more sophisticated experiences

To export blocks from an extensions, a createComponents factory function must be exported by the extension.js file. This function is invoked by the server on startup with the servers block factory function (currently APIOperationsComponent.fromJSON).

Customize the extension.yaml File:

  • In your new repository, find and update the extension.yaml file.

  • Crucially, change the origin field to your repository’s clone URL.

  • Push Changes and Add Extension to Omnitool:

    • Commit and push your updated extension.yaml file to your GitHub repository.

    • In the Omnitool chat UI, type /extensions add [your GitHub repository clone URL, ending with .git] to add your extension.

  • Verify and Start Development:

    • Once added, locate your extension in omnitool/packages/omni-server/extensions/.

    • You’ll find a new folder named after your GitHub repository here.

  • Develop Your Extension:

    • Open the directory in Visual Studio Code or your preferred IDE.

    • This is your development environment where you can build and customize your extension.

  • Learn from Existing Extensions:

    • Explore other extensions in Omnitool for inspiration and understanding.

    • Observe their structure, functionality, and integration methods.

  • Iterate and Test:

    • Develop your extension iteratively, testing it regularly in the Omnitool environment.

    • Continuously refine based on test results and potentially user feedback.

  • Finalize and Share:

    • Once you’re satisfied with your extension, ensure it’s properly documented and shared with the Omnitool community.

    • Encourage feedback to further enhance your extension.

  • server/extension.js - This file is loaded by the server on startup and allows hooking extending server functionality. See Server Programming.

  • Omni Extension Templatearrow-up-right
    title: My First Extension                          # Human readable title, ideally under 20 characters
    version: 0.0.1                                     # Semver compatible version string
    description: A small omnitool extension doing stuff
    author: omni@example.com                           # Author name or email
    origin: https://github.com/user/repository.git     # url to the extensions git repository. This is not required for local extensions not published
    client:
      addToWorkbench: false                            # if set to true, the extension will be added to the client's extension menu
    dependencies:                                      # Optional field allowing auto installation of yarn packages required by an extensions in the same format as package.json. This is experimental and will likely change
      package: package@stable
    packages:
    
    known_extensions:
      - title: 3D Texture Playground                                #Human readable title
        id: omni-extension-texture-playground                       #Extension id
        url: https://raw.githubusercontent.com/user/repository/branchname/extension.yaml #discovery url for the raw extension.yaml file describing the extensions
        ...
    function createScript()
    {
      return {
        description: "Create a flipbook from the current chat",
        title: "Create Flipboard",
        exec(args){
    
          let images = [];
            // Find every image in chat
            window.client.chat.state.messages.forEach((msg) => {
              if (msg.attachments && msg.images && msg.images.length > 0) {
                images= images.concat(msg.images.map((img) => {
                  return img.url
                }))
              }
            })
            window.client.workbench.showExtension("omni-extension-flipbook", {images: images});
            // One could also open the extension in a separate window:
            // window.open(`./extensions/omni-extension-flipbook/?images=${encodeURIComponent(JSON.stringify(images))}`, '_blank', 'popup=1,toolbar=0,location=0,menubar=0');
            window.client.sendSystemMessage(`Flipbook created, please check the extensions tab`, "text/markdown",
            {
              commands:
              [
                {
                  title: 'Show Flipbook',
                  id: 'toggleExtensions',
                  args: []
                }
              ]
            });
            return true;
        }
      }
    }
          const args = new URLSearchParams(location.search)
          const params = JSON.parse(args.get('q'))
    
          if (params.images) {
            images.value = params.images
          }
    
    const blocks = []
    
    const extensionHooks = {}
    
    const blockFactory = (FactoryFn) =>
    {
      blocks.map((c) => FactoryFn(c.schema, c.functions))
    }
    export default {hooks: extensionHooks, createComponents: blockFactory}
    
    // ServerExtensionManager.ts
    enum PERMITTED_EXTENSIONS_EVENTS
    {
      'pre_request_execute'   = 'pre_request_execute',      // runs each time a block is preparing to execute it's underlying API call allowing manipulation of the outgoing call. Arg
      'post_request_execute'  = 'post_request_execute',     // runs each time a block has executed it's underlying API call, allowing manipulation of the result
      'component:x-input'     = 'component:x-input',        // runs each time a chat input block processes it's payload. Args:  (ctx, payload). Allows modification of payload
      'jobs.job_started'      = 'job_started',              // runs each time a workflow / job has started executing
      'jobs.job_finished'     = 'job_finished',             // runs each time a workflow / job has stopped executing
      'jobs.pre_workflow_start' = 'job_pre_start'           // runs each time a workflow is prepping execution, allowing it to be cancelled. Args: (workflow, workflow_context, ctx, actions). Set actions.cancel to true to abort execution, set actions.cancelReason to a string to return a specific cancellation reason to the user.
      'registry.package_installed': 'package_installed' //{ omniPackage:string, installationId, orgId, customBaseUrl, duration: (end - start).toFixed() })
    }
    const extensionHooks = {
    
      // An basic interceptor that replaces any chat input containing something like an email addresses with a simple replacement
      // This runs on any chat input
      'component:x-input': function(ctx, payload)
      {
        if (payload.text != null && typeof(payload.text) === 'string')
        {
          payload.text = payload.text.replace(\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b, "example@example.com")
        }
      },
    
    
      // Prevent any workflow from starting if the text input has the word clown
      // This runs every time a workflow starts
      'job_pre_start': function(ctx, workflow, workflow_context, actions)
      {
        console.log('job_pre_start pii scrubber', workflow_context.args)
    
        if (workflow_context.args ?? workflow_context.args.text?.includes("clown"))
        {
          actions.cancel = true
          actions.cancelReason = "N0 clowns allowed"
        }
      }
    }
    
    const MyCustomBlock =
     {
        schema:                                                   // <-- OpenAPI 3 Schema
        {
          // Namespace is automatically set to the extension id
          "tags": ['default'],
          "componentKey": "my-first-block",                       // <-- unique id within the namespace
          //"apiKey": "my-first-block",                           // <-- optional, set to the 'parent' API if this is a normal rest component (not supported yet)
          "operation": {
            // operationId is automatically set to componentKey
            "schema": {
              "title": "My First Block",                          // <-- componentn title
              "type": "object",
              required:[],
              "properties": {
                "text": {
                  "title": "Some Text Input",
                  "type": "string",                                // <-- openAPI type
                  "x-type": "text",                                // <-- custom omnitool socket type if wanted
                  "default": "my default value",
                  "description": `My block description`
                }
              },
            },
            "responseTypes": {
              "200": {
                "schema": {
                  "required": [
                    "text"
                  ],
                  "type": "string",
                  "properties": {
                    "text": {
                      "title": "My Output Text",
                      "type": "string",
                    },
                  },
                },
                "contentType": "application/json"
              },
            },
            "method": "X-CUSTOM"                          // <-- This is important
          },
          patch:                                          // <-- optional omnitool patch block
          {
            "title": "My Custom Component",              /// <-- component
            "category": "Test",
            "summary": "Replaces Cars with Horses",
            "meta":
            {
              "source":
              {
                "summary": "Replaces cars with horses",
                links:
                {
                  // list of string: string fields that are rendered as urls.
                  "research papaer": "https://arxiv.org..."
                }
              }
            },
            inputs: {...},
            controls: {...}
            outputs: {...}
          }
        },
        functions: {
          _exec: async (payload, ctx) =>                           // <--  The _exec function is invoked when the component is run
          {
            if (payload.text)
            {
              payload.text = payload.text.replace("car", "horse" )
            }
            return payload                                       // <-- Do not forget to return the altered payload
          }
        }
      }
    
    let blocks = [MyCustomBlock]
    
    export default (FactoryFn: any) =>
    {
      return components.map((c) => FactoryFn(c.schema, c.functions))
    }