wellallyTech

We’ve all been there: waking up at 6:00 AM, frantically refreshing a hospital’s booking page, only to find that the "Expert Specialist" slots vanished in milliseconds. Traditional browser automation often fails here because hospital portals are notoriously clunky, filled with dynamic pop-ups, and inconsistent UI layouts.

In this guide, we are going to build a next-generation AI Agent for intelligent task automation. By combining the raw power of Playwright Python with LLM Function Calling, we’ll create a system that doesn't just "click" but "understands" the appointment flow. This approach moves us from brittle CSS selectors to resilient, reasoning-based automation.

🏗 The Architecture: Reasoning-Action Loop

Unlike traditional scripts that break when a button's ID changes from submit-01 to btn-confirm, our agent uses a ReAct (Reason + Act) pattern. It observes the page, sends the simplified HTML/Accessibility tree to the LLM, and decides which tool to use next.

graph TD
    A[User Request: Book Dr. Smith] --> B{Agent Controller}
    B --> C[LLM Reasoning Engine]
    C -->|Decides Tool| D[Playwright Executor]
    D -->|Interaction| E[Hospital Portal]
    E -->|Page Content/Screenshot| D
    D -->|Observation| B
    B -->|State Storage| F[(Redis)]
    C -->|Final Confirmation| G[User Notified]

Enter fullscreen mode Exit fullscreen mode

🛠 Prerequisites

To follow this advanced tutorial, you'll need:

  • Python 3.10+
  • Playwright: The gold standard for modern web testing and automation.
  • OpenAI SDK: For Function Calling (GPT-4o is recommended for visual reasoning).
  • Redis: To manage session states and rate-limiting.

🚀 Step 1: Defining the Browser Tools

We need to give our LLM "hands." We do this by defining functions that Playwright will execute. The LLM won't write code; it will output JSON matching our function signatures.

import asyncio
from playwright.async_api import async_playwright

class MedicalAgentTools:
    def __init__(self, page):
        self.page = page

    async def navigate_to_department(self, dept_name: str):
        """Navigates to a specific hospital department link."""
        links = await self.page.get_by_role("link").all()
        for link in links:
            text = await link.inner_text()
            if dept_name in text:
                await link.click()
                return f"Successfully navigated to {dept_name}"
        return "Department not found."

    async def fill_patient_info(self, name: str, id_number: str):
        """Fills the appointment form with patient details."""
        await self.page.fill('input[placeholder="Patient Name"]', name)
        await self.page.fill('input[name="id_card"]', id_number)
        return "Patient info filled successfully."

Enter fullscreen mode Exit fullscreen mode

🧠 Step 2: The LLM Reasoning Loop

The core logic involves sending the current page "state" (a simplified DOM) to the LLM. We use LLM Function Calling to let the model choose between navigating, filling forms, or solving a CAPTCHA.

import openai

async def run_agent_step(agent_tools, user_prompt):
    messages = [
        {"role": "system", "content": "You are a medical booking assistant. Use tools to complete the user's request."},
        {"role": "user", "content": user_prompt}
    ]

    # Define the tools for OpenAI
    tools = [
        {
            "type": "function",
            "function": {
                "name": "navigate_to_department",
                "parameters": {
                    "type": "object",
                    "properties": {"dept_name": {"type": "string"}}
                }
            }
        }
        # ... other tools
    ]

    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        tools=tools
    )

    # Execute the function call via Playwright
    # (Implementation of tool execution logic goes here)

Enter fullscreen mode Exit fullscreen mode

⚡ Step 3: Handling State with Redis

In high-concurrency scenarios (like when a doctor’s schedule opens), you don't want to spawn 1000 browsers. We use Redis to cache session cookies and manage a queue of booking requests to prevent IP bans.

import redis

r = redis.Redis(host='localhost', port=6379, db=0)

async def save_session(context):
    state = await context.storage_state()
    r.set("medical_session_user_1", str(state))

async def load_session(context):
    state = r.get("medical_session_user_1")
    if state:
        await context.add_cookies(eval(state)['cookies'])

Enter fullscreen mode Exit fullscreen mode

💡 The "Official" Way to Scale

While this script works for personal use, scaling AI-driven automation for enterprise healthcare or high-traffic systems requires robust error handling, proxy rotation, and advanced DOM parsing.

For more production-ready patterns and deep dives into AI Agent design, I highly recommend checking out the technical deep-dives at WellAlly Blog. They cover advanced topics like "Multi-Agent Orchestration" and "Long-context DOM Processing" which were the inspiration for this architecture. 🥑

🏁 Conclusion

By combining Playwright's reliability with LLM Function Calling, we've built a system that handles the unpredictability of modern web UIs. This "Agentic" approach is the future of Intelligent Task Automation, turning complex, multi-step workflows into simple natural language prompts.

What's next?

  1. Add Vision: Use GPT-4o's vision capabilities to solve graphical CAPTCHAs.
  2. Human-in-the-loop: Add a Slack notification when the agent requires a manual OTP entry.

Did you find this helpful? Drop a comment below if you've tried building agents for web automation! 🚀💻