Mac VNC Control

# Mac VNC Control VNC into Mac Minis to control them remotely — fix permissions, manage GUI settings, and interact with apps that can't be controlled via SSH alone. ## Prerequisites - mac-control MCP server running on the target Mac (port 8200) - mac-control MCP server running on a helper Mac (e.g., bots) for the VNC client - SSH access to all machines ## Machine Fleet | Machine | IP | SSH | mac-control | Notes | |---------|------|-----|-------------|-------| | vault | 192.168.86.27 | ssh vault | :8200 | Retina display (3840x2160 screenshots, divide by 2 for click coords) | | bots | 192.168.86.50 | ssh bots | :8200 | Best machine for GUI automation | | jarvis | 192.168.86.51 | ssh jarvis | :8200 | macOS 26 beta, SSH screencapture broken | ## Mac-Control Helper Script Create this helper on the workstation for easy mac-control calls: ```bash cat > /tmp/mc.sh <<'EOF' #!/bin/bash HOST=$1; TOOL=$2; ARGS=${3:-"{}"} case "$HOST" in vault) IP="192.168.86.27" ;; bots) IP="192.168.86.50" ;; jarvis) IP="192.168.86.51" ;; *) IP="$HOST" ;; esac SESSION_ID=$(curl -s -D- -X POST http://$IP:8200/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{ "jsonrpc":"2.0","id":0,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}' 2>&1 | grep -i "mcp-session-id" | awk '{print $2}' | tr -d '\r') curl -s -X POST http://$IP:8200/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "Mcp-Session-Id: $SESSION_ID" \ -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"$TOOL\",\"arguments\":$ARGS}}" 2>/dev/null \ | grep "^data:" | tail -1 | sed 's/^data: //' EOF chmod +x /tmp/mc.sh ``` Usage: `bash /tmp/mc.sh <host> <tool> '<json_args>'` ## Step 1: Enable Remote Management on Target Mac Before VNC works, enable Apple Remote Desktop on the target machine: ```bash ssh vault 'sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart \ -activate -configure -access -on \ -clientopts -setvnclegacy -vnclegacy yes \ -setreqperm -reqperm no \ -restart -agent -privs -all' ``` This enables Remote Management with VNC legacy support. ARD listens on port 3283. ## Step 2: Open VNC from a Helper Mac Use mac-control on a helper Mac (e.g., bots) to open Screen Sharing: ```bash # Open VNC connection from bots to vault bash /tmp/mc.sh bots run_command '{"command":"open vnc://192.168.86.27"}' ``` Wait 5-8 seconds for the connection dialog to appear. ## Step 3: Authenticate The Screen Sharing login dialog appears. Fill credentials via AppleScript on the helper Mac: ```bash # Set username bash /tmp/mc.sh bots applescript '{"script":"tell application \"System Events\" to tell process \"Screen Sharing\" to tell window 1 to set value of text field 1 to \"macmini\""}' # Set password bash /tmp/mc.sh bots applescript '{"script":"tell application \"System Events\" to tell process \"Screen Sharing\" to tell window 1 to set value of text field 2 to \"PASSWORD_HERE\""}' # Click Sign In bash /tmp/mc.sh bots applescript '{"script":"tell application \"System Events\" to tell process \"Screen Sharing\" to tell window 1 to click button \"Sign In\""}' ``` If a "Select Screen Sharing Type" dialog appears, press Enter to accept Standard. ## Step 4: Interact with Target Mac Once connected, you see the target Mac's screen inside the VNC window on the helper Mac. You can: ### Take screenshots of the target ```bash # Use the target's own mac-control for screenshots (better quality) bash /tmp/mc.sh vault screenshot ``` ### Click on target's UI Clicks through the VNC window require coordinate mapping: - The VNC window has a title bar offset (~30px from top of helper Mac) - Target coordinates need to account for the VNC window position - **Easier approach**: Use the target's mac-control directly for clicks: ```bash bash /tmp/mc.sh vault click '{"x":500, "y":300}' ``` ### Type on target ```bash bash /tmp/mc.sh vault type_text '{"text":"hello"}' bash /tmp/mc.sh vault key_press '{"keys":"return"}' ``` ## When to Use VNC vs Direct Mac-Control | Task | Use VNC | Use mac-control directly | |------|---------|-------------------------| | Fix permissions in System Settings | Yes — need to see and click GUI | No — can't click without permissions | | General browser automation | No — unnecessary overhead | Yes — faster and simpler | | Debug visual issues | Yes — see full desktop | Partial — screenshots only | | Grant accessibility to new apps | Yes — requires GUI interaction | No — TCC database is SIP-protected | ## Fixing Accessibility Permissions via VNC This is the most common reason to VNC into a Mac: ```bash # 1. Open System Settings to Accessibility on the target ssh vault 'open x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility' # 2. VNC in from bots to see and interact with the settings bash /tmp/mc.sh bots run_command '{"command":"open vnc://192.168.86.27"}' # (authenticate as above) # 3. Use helper Mac's mac-control to click toggles in the VNC window # Or use target's mac-control if it has enough permissions ``` ## Troubleshooting ### VNC connection fails - Verify ARD is running: `ssh vault 'lsof -iTCP:3283 -sTCP:LISTEN -P -n'` - Re-enable: run the kickstart command from Step 1 - Try port 3283 explicitly: `open vnc://192.168.86.27:3283` ### Authentication fails - Verify username: `ssh vault 'whoami'` (vault=macmini, bots=bots, jarvis=jarvis) - Password must match the Mac's login password - Try "By Requesting User" option if available ### VNC shows black screen - The target Mac may be on a different Space - Use `ssh vault 'osascript -e "tell app \"Finder\" to activate"'` to switch to main desktop ### Retina coordinate issues (vault) - Vault has a 4K Retina display (3840x2160 pixels, 1920x1080 points) - mac-control screenshots are in pixel coordinates - cliclick uses point coordinates - Divide screenshot pixel coords by 2 for click targets ## Mac Credentials Reference | Mac | Username | Password | |-----|----------|----------| | vault | macmini | (stored in vault) | | bots | bots | (stored in vault) | | jarvis | jarvis | (stored in vault) | Note: Never hardcode passwords in skills. Retrieve from the vault or prompt the user.