openmule

README.md



Open source Computer Using Agent marketplace - MuleRun alternative
    
for a clear version of this proposal, view here

crypto first, other flat currencies comes later. we can build a distributed relay framework later upon Blockchain.

the client (end user account) can comment on each service provider, rate its performance.

openmule can either relay traffic between CUA servers (ai controller) and clients (command receivers, executors), pay bills to servers, or host its own servers by putting all config files to openmule server.

openmule can probe connection health to clients and servers

Agent framework: trycua/cua
Accounting framework: cybergod-gym
AI billing: litellm
Frontend: vue3
Backend: fastapi, websocket

use YAML for storing agent prompt, Dockerfile, billing info, input and output file, remote machine vnc credential info etc

can control user defined remote machine, cloud machine, openmule hosted machine or remote openmule client

users are recommended to use a virtualized machine to be controlled by openmule, or a remote managed vm instance.

openmule shall provide top security to end users, by eliminating intranet network address access.

the user shall have a webui interface to virtualized machine which is being controlled, with a password, to configure cua connection details and view logs, and a dialogue to confirm if this machine is bare metal without virtualization, if so then any keyboard or mouse movement will trigger emergency stop, or some other action like moving mouse to a specific corner

openmule can support Xbox, PS5 or PC running fps games with hardware.

free services first, then paid, then customize

openmule shall not be limited to cua agents. it shall be a marketplace for all sorts of ai agents, as long as it takes money to execute. it can even be physical agents.

we can make human in the loop, record human operation during the process of using openmule action sender api. human can earn money too with openmule. the end user can interfere with the machine being controlled, provide feedback, machine state snapshots or emergency stop.

negotiable task and price (the end user can place a bid for a task) for flexible, versatile agents, refund policy by third party escrow and validator

every traffic relay, escrew can get paid by joining the openmule distributed network

ai provider and user can be certified by official

duplicate some or all features like freelancer.com, but for ai

provide a offline training environment, for easy development, in which human or ai can be the end user and provide feedback

pay for minimum wage per minute in process of work, place payment milestones between work stages, to ensure satisfaction

one can install the client on a brand new computer without personal data, without virtualization, with equivalent security

one can choose monitor only mode on main computer screen, without action capability

One can compose multiple agents for complex workloads

One can create a view only agent (for future read-only but able to observe and chat agents), or a random acting agent for testing, for free

The good part of self hosting computer command executor or environment is that everyone has different context and it is best to keep these context locally rather than remotely. however according to our design we can also keep these things in the cloud. just a matter of choice.

we might want to make the entire openmule agent side api into mcp protocol to ensure real time response

need some p2p or relay tech in xunlei, tailscale, geth or else

need to identify the developer, record all operations locally, to ensure data is not leaked. evidence could be remotely stored (compressed vnc traffic) and prevent/trace data leakage.

relay nodes may provide dispute resolution service and get rewards if both customer and ai service provider are satisfied.

the agent or ai service privider itself can review a dispute and flag inappropriateness, saving human labor time.

anyone can provide a remote computer environment for remote control, for other agents to complete task. in the process it can earn money.

remote robot coordinator shall not care too many details for syncing arms and legs. instead, they shall have high level apis like moving to given coordinate, instructions in natural language. or if have details over the entire body it shall be a full 3d projection of body position, some camera feeds, sensor feeds, motor angle, speed, torque readings etc.

human and ai can compete over every task on openmule marketplace

the relay makes sure the ai service provider can perform online learning and improve performance over time.

we may start with traditional labor task and artifact, money escrew platform.