A Privacy-Centric Framework for Autonomous Desktop Orchestration Using Local LLMs and Voice-Driven Workflows

Abstract

Today's desktop computer users often find their work interrupted by disjointed workflows and tedious, repetitive tasks—eating up nearly a third of their time. Although cloud-based voice assistants are available, they rarely integrate deeply with the operating system, tend to respond slowly, and can compromise user privacy. In this paper, we introduce "Your Own Jarvis": a privacy-first, always-on voice AI built to give users full control over their desktop and to automate everyday tasks. Unlike typical assistants, this system runs completely on your device. It uses a Retrieval-Augmented Generation (RAG) setup powered by fine-tuned Large Language Models like LLaMA or Mistral. For speech-to-text, it leverages OpenAI Whisper, and for wake-word detection, it uses Porcupine’s efficient technology. By connecting directly to your computer’s core features and using computer vision to understand what’s on your screen, the assistant can carry out complex tasks—whether that’s organising files, drafting emails, or running terminal commands for developers. Experimental results indicate a potential 60% reduction in routine task execution time and a 50% decrease in manual peripheral usage. The system achieves sub-500ms response latency without external data transmission, providing a secure, high-performance alternative for power users and individuals with accessibility needs. Our findings demonstrate that on-device AI can effectively bridge the gap between natural language intent and the complex execution of operating systems.