AI is rapidly moving beyond centralized cloud and data centers, becoming a powerful tool deployable directly on professional workstations. Thanks to advanced hardware and optimized software, you can build, run, and experiment with sophisticated AI models at your desk or on the go. Welcome to the world of local AI development!
Running and developing AI locally on a workstation offers significant advantages for developers and organizations – enhanced data privacy and security, sensitive data remains in-house, cost savings compared to continuous cloud usage, offline operational capabilities for applications, and an unparalleled environment for hands-on development and iteration. This shift towards powerful, accessible local AI is driven by high-performance hardware like the NVIDIA RTX PRO Blackwell series and the optimized software ecosystem built to harness their capabilities.
This blog post will walk you through selecting a manageable first local AI project, using the NVIDIA ecosystem designed for professional workflows.
Understanding your NVIDIA RTX PRO Workstation
At the core of professional AI acceleration are the NVIDIA RTX professional GPUs that feature up to 96 GB VRAM each, enterprise-grade drivers, ISV certifications, and enhanced NVIDIA Tensor Core performance to deliver up to 4,000 trillion operations per second for AI.
This makes them ideal for handling larger datasets, training more complex models, and running sophisticated AI inference tasks. Support for a wide spectrum of advanced data formats—from low-precision types like FP4 and INT8 to higher-precision FP16 and FP32—enables AI models to run efficiently, which is crucial for rapid iterative development cycles.
Define your project
With your RTX PRO workstation, the next step is selecting an appropriate first project. Consider these factors:
- Focus area: Concentrate on mastering one core AI capability, such as advanced text analysis, high-resolution image processing, or specific types of data generation.
- Align with your goals: Choose a domain relevant to your interests, work, or industry, such as automating data analysis, improving visual workflows, or generating specialized content.
- Resource assessment: Ensure your project goals align with your specific workstation’s capabilities (especially GPU VRAM, CPU power, and storage) and your current AI skill level. Start with a project that you can complete and learn from.
Here are some project ideas to get started with NVIDIA RTX PRO workstations.
Project 1: AI-powered chatbot
Choosing the right starting point for your local AI chatbot project is key. The core technology making these chatbots useful for your specific data is Retrieval-Augmented Generation (RAG). RAG enables the chatbot’s underlying large language model (LLM) to access and reference your specific documents or knowledge bases before generating an answer, ensuring responses are accurate, relevant, and grounded in your context, rather than the LLM’s general training data.
Start with NVIDIA ChatRTX to create a no-code RAG chatbot. Simply download the app, point it to your local file folders like project documents or notes, select a compatible model from its menu, and begin to ask questions. ChatRTX handles the RAG process of retrieving relevant snippets and feeding them to the LLM automatically.
When you’re ready to code and build a more customized RAG chatbot, with specific logic or to integrate different data sources, AI Workbench can help you set up your development environment.
Download the AI Workbench installer for your Windows or Linux system. It handles managing necessary dependencies like Git, container software like Docker or Podman, and even GPU drivers on Linux OS. Windows users must install drivers manually from the NVIDIA Drivers page.
To accelerate development, use pre-built examples like the multimodal virtual assistant project. Cloning this project within AI Workbench gives you a head start, with a functional RAG pipeline where you can begin integrating your specific documents, images, or videos, defining custom data retrieval methods, managing different LLMs, and iterating on the design for a truly tailored conversational AI experience.
Specific applications of locally run AI chatbots are emerging across industries. In Architecture, Engineering, and Construction (AEC), firms are experimenting with custom chatbots trained on historical Requests for Proposals (RFPs) and project documentation to help summarize new RFPs or quickly find answers within past responses, streamlining the proposal process. Within Financial Services, compliance teams can utilize local chatbots trained on extensive regulatory archives and internal policy documents stored securely on their workstations to rapidly query and verify requirements or find precedents without sensitive data leaving the local environment.
Project 2: PDF to podcast
Convert your PDF documents, like research papers or manuals, into engaging audio content. This PDF-to-podcast conversion capability runs locally on a workstation and offers advantages across various fields. For instance, legal professionals can convert lengthy, confidential case files or discovery documents into audio for review, ensuring sensitive client data remains secure on their local machine.
Research and development teams in engineering or pharmaceuticals can transform dense technical specifications, research papers, or internal manuals into audio formats, for experts to absorb complex information while multitasking or away from their screens, all while protecting proprietary intellectual property.
Get started by cloning this project from its GitHub repository within the AI Workbench interface, which streamlines the setup by handling the containerized environment configuration and automatically configuring GPU access.
The default implementation uses cloud-based NVIDIA NIM endpoints. However, AI Workbench gives you flexibility to run key components—including NIM microservices—directly on your local RTX PRO workstation. This approach ensures your proprietary PDF data remains secure and private, since all processing can stay on your local machine rather than being sent to the cloud
This blueprint is flexible and customizable, so you can add additional functionality such as branding, analytics, real-time translation, or a digital human interface to deepen engagement. You can test, prototype, and customize the PDF-to-podcast pipeline directly on your powerful hardware.
Project 3: Video search and summarization agent
The ability to automatically search and summarize video content is incredibly valuable. Unlocking insights previously hidden within vast libraries of recordings like sporting highlights and broadcasts, security footage, meeting archives, or educational lectures, can save countless hours of manual review.
You can build your own video search and summarization (VSS) agent locally on an RTX PRO workstation using NVIDIA AI Blueprint. This blueprint provides a comprehensive reference architecture, utilizing NIM microservices for key functions like video ingestion, vision-language understanding, LLM reasoning, and RAG, deployed locally.
The default configuration for the blueprint utilizes the meta/llama-3.1-70b-instruct LLM. Running this 70 B-parameter model locally demands 140 GB VRAM or more, which may exceed the capacity of RTX PRO workstation. For practical local deployment on RTX PRO GPUs, modify the blueprint configuration to substitute the 70B model with the smaller, meta/llama-3.1-8b-instruct version, which has substantially lower memory requirements and is well-suited for running inference directly on your workstation.
https://www.youtube-nocookie.com/embed/N1UOqr7Ga_A?feature=oembed&iv_load_policy=3&modestbranding=1&rel=0&autohide=1&playsinline=0&autoplay=0Video 1. Learn how to build visual agents for video search summarization
Get started with your AI projects
To get started with your first AI project, select a project that aligns with your professional needs, begin with manageable steps, and leverage NVIDIA resources available like the Developer Program.
Embarking on your first local AI project is a strategic step for developers seeking to leverage cutting-edge technology. Using NVIDIA PRO RTX workstations for the computational backbone, NVIDIA AI Workbench for streamlined development, and NVIDIA AI Enterprise software for production-grade tools and support, you are fullyequipped.
SAGECONNECT TECHNOLOGY GROUP CO., LIMITED
Sageconnect is a reliable provider of IT network system integration and solutions for global enterprise customers.
