1 of 12

Tutorial

Glossary

API Key

A credential used to authenticate a user or application attempting to access a third-party API (Application Programming Interface) such as OpenAI, Google, Replicate, ElevenLabs, etc. In Omnitool, users are required to provide their own API keys (BYOK) for the API they intend to use. This key is stored locally on the user's system for security purposes and must be entered in the API management section to activate the corresponding blocks.

Block

Previously called "Component". A single unit of functionality within Omnitool. Blocks can perform a variety of actions, requesting an API, manipulating text/image, or outputting the result to file. By connecting blocks together, users can build complex recipes. Each block is designed to be interoperable, ensuring smooth data format transfers and integration.

Recipe

Previously called "Workflow". A sequence of interconnected blocks in Omnitool that performs multiple API calls or executes scripts in a defined flow. Recipes are created by arranging these blocks in a sequence to fulfill a specific AI use case.

Template

A pre-made recipe created by the Omnitool team or contributors. It serves as a demonstration use case or starting point for users to craft their own recipes.

Extension

A plugin extends the functionality of Omnitool's core system. It can add new blocks, tools, or create custom user interfaces. Extensions allow developers to tailor Omnitool to their specific needs and preferences on both the client and server sides, providing a convenient way to prototype AI-powered applications.

UI Overview

Side Bar

API Key Management: This page is for entering or revoking API credentials, ensuring secure and controlled access to the third party API.
File Manager: An essential feature that allows users to browse, operate, and manage all file assets that have been uploaded to or generated within the recipe.
Extension Manager: Allows for the installation or uninstallation of published extensions. Users can also pin their favorite extensions to the sidebar for quick access.
Developer Tool: Activate this tool during recipe execution to monitor API requests and responses, a feature crucial for troubleshooting.
Local API: This section lists supported local or open-source services. Users must install these services on their machines and can then activate the corresponding local API to enable its blocks within block collection.

Chat Window

File Manager

Adding API Keys

Omnitool employs a Bring Your Own Key (BYOK) approach for API keys, ensuring your keys remain secure and private on your local environment. To activate the blocks corresponding to each supported API, simply input your API keys into the API management page.

Blocks

A block is the essential building block of any recipe. Each block possesses a distinct function. By linking blocks together, you begin to craft a flow, culminating in a recipe. Omnitool's architecture promotes effortless interoperability between blocks, facilitating smooth data format transfers and integration.

Omnitool

Omnitool blocks are versatile utility blocks designed for enhancing user interaction and data processing in the recipe. These blocks include:

Chat Input and Output

Facilitate communication through a built-in chat window, allowing users to interact with recipes effectively.

Data Input

Comprises various data types input like text, number, image, document, file etc. providing convenient way for data entry and manipulation.

Text Manipulation

Essential for text formatting and processing during interactions between various models. These blocks, particularly the Text Template block, are specialized in adapting text to specific formats, ensuring optimal output within the recipe.

Data Transformation

Blocks that are essential for converting and manipulating text within recipe, allowing you to transform text into structured JSON formats, manipulate JSON objects with expressions, and even convert arrays of images and captions into markdown documents. These functionalities are key for users who need to reformat and prepare data for further processing or presentation within a recipe.

File Operation

Cater to file output needs, including saving files in specific formats and adding metadata, enhancing accessibility.

Recipe Operation

The blocks streamline the execution and integration of recipes with external systems, enabling the output of recipe results to be utilized in REST API calls.

OpenAI

We provides a suite of blocks for OpenAI powerful models, including GPT3/4, GPT4-Vision, Text Embeddings, DALL·E 2/3, TTS: Text-to-Speech, and Whisper: Speech-to-Text etc.

Prerequisites

Ensure you have the following before you begin:

OpenAI Account: Sign up or log in at OpenAI.
API Token: Generate your API token on the OpenAI platform.

Setup

Adding OpenAI API Key

Navigate to the API Key Manager in the left panel.
Enter your OpenAI API token here.

Using OpenAI Blocks

After adding your OpenAI API key, you can access various OpenAI blocks within Omnitool:

Go to the "Add Block" collection in Omnitool.
Search openai namespace

Access to GPT-4 API: According to OpenAI, access to the GPT-4 API requires meeting certain criteria as detailed in their help article. If your account is new or lacks sufficient payment history, you may not see the GPT-4 model in the model options.

Billing Clarification: The OpenAI API is not the same as ChatGPT Plus billing. Ensure you are accessing the OpenAI Platform API for the correct billing information.

Credit: If you haven't added a payment method, make sure you have enough API credits available. Otherwise, you may encounter errors when calling the API. To check your API credit balance, visit OpenAI Platform Usage.

Additional Resources

For developers and users seeking more advanced configurations and troubleshooting resources for these blocks, refer to the OpenAPI and block specifications on the https://github.com/omnitool-ai/omni-core-blocks/tree/main/server/apis/openai

For more detailed information about OpenAI API's offerings and capabilities, visit the official OpenAI docs.

Replicate

Replicate is a platform that enables users to effortlessly host and run AI models on the cloud. With the Replicate Extension for Omnitool, you can conveniently access and use a wide range of AI models available on replicate.com with just a single click.

Prerequisites

Before you begin, ensure you have the following:

Replicate Account: Sign up or log in at Replicate.
API Token: Generate your API token at Replicate API Tokens.

Setup

Installing the Replicate Extension

Navigate to Extensions Manager
Select and install the Replicate Extension.

Adding API Key

Navigate to API Key Manager in the left panel.
Add your Replicate API token.

Using Replicate Blocks

Adding Replicate Model

Go to the Extension Manager.
Open the Replicate Extension.
Import models:
- Preset Models: Choose from a list of preset models provided in the portal.
- Custom Models: Manually key in the model name, which can be found at https://replicate.com/explore, to add a specific model.

Using Replicate Blocks

Find in Block Collection: Once model added, the Replicate model's block will appear in the "Add Block" collection.
Direct Add: Or, right after importing the model, a button will appear in the Chat interface. Click on this button to directly add the Replicate block to your canvas.

Additional Resources

For more detailed information, visit the official Replicate documentation.

HuggingFace

HuggingFace is renowned for its comprehensive collection of open-source natural language processing (NLP) models and tools. The HuggingFace blocks in Omnitool enable easy access to these models, offering a seamless experience for users looking to integrate the latest models into their recipes.

Prerequisites

Before you begin, ensure you have the following:

HuggingFace Account: Sign up or log in at HuggingFace.
Access Token: Generate your Access token from your HuggingFace account.

Setup

Adding API Key

Navigate to API Key Manager in the left panel.
Add your HuggingFace Access token.

Using the HuggingFace Block

Navigate to the "Add Block" collection in Omnitool.
Search huggingface namespace, which provides access to various blocks.

Identify the category of the model you're interested in from the extensive collection at HuggingFace Models. Once you've selected a model, simply copy its name and apply the corresponding block in Omnitool. And just like that, you're all set to use the model!

Additional Resources

For more information of HuggingFace API, visit the HuggingFace website.

ElevenLabs

ElevenLabs stands out in the realm of AI with its advanced voice synthesis technology. By integrating the ElevenLabs Block into Omnitool, users can effortlessly leverage its API, such as Text-to-Speech.

Prerequisites

Before starting, ensure you have the following:

ElevenLabs Account: Sign up or log in at .
API Key: Generate your API key from your ElevenLabs profile.
Characters: Ensure you have enough characters available for use in Subscription > Characters Used.

Setup

Adding API Key

Go to the API Key Manager in the left panel of Omnitool.
Enter your ElevenLabs API key.

Using the ElevenLabs Block

Navigate to the "Add Block" collection in Omnitool.
Search for the elevenlabs namespace to find its TTS block.

Additional Resources

For developers and users seeking more advanced configurations and troubleshooting resources for these blocks, refer to the OpenAPI and block specifications on the .

For further information on the ElevenLabs API, visit the .

OpenRouter

The OpenRouter Block is a unified API that specializes in providing access to a diverse range of supported Large Language Models (LLMs).

Prerequisites

Before you begin, ensure you have:

OpenRouter Account: Register or sign in at OpenRouter's website.
API Key: Obtain your OpenRouter Key.
Credits: Ensure you have enough API credits. (You have a small usage allowance (< $1) before you need to pay).

Setup

Adding API Key

Access the API Key Manager in Omnitool's left panel.
Input your OpenRouter API key.

Using the OpenRouter Block

Navigate to the "Add Block" collection within Omnitool.
Search for the openrouter namespace to locate its LLM block.
There are two distinct LLM blocks provided. One of these is the vision transformer LLM model block, which is designed to process image input.

Additional Resources

For additional details on the OpenRouter API, refer to the OpenRouter website.

Sharp Image Processing

Sharp Image Processing blocks are based on the Sharp Node.js library, offering a suite of advanced image manipulation capabilities. These blocks enable users to apply a range of image processing techniques, such as blurring, resizing, preparing format, compositing and more, directly within the recipes.

Setup

Install the Sharp Image Processing extension via the Extension Manager found in the left panel.

Using Sharp Blocks

Once installed, navigate to the "Add Block" collection within Omnitool.
Search for the sharp namespace to locate all the image manipulation blocks.

Additional Resources

For developers and users seeking more advanced configurations and troubleshooting resources for these blocks, refer to the extension code in https://github.com/omnitool-community/omni-extension-sharp/tree/main/src/components.

For additional details on the Sharp API, refer to https://sharp.pixelplumbing.com/.