systemd Services

--- name: systemd Services description: Use this skill when you need to create, manage, and troubleshoot systemd services on Linux systems. Essential for service management, process supervision, dependency handling, and system automation. category: ops metadata: author: skynet version: 1.0.0 --- # systemd Services ## Overview systemd is the default init system and service manager for most modern Linux distributions. Services define how processes are started, stopped, monitored, and managed by the system. ## Quick Reference ### Service States - **active (running)** - Service is running - **active (exited)** - One-shot service completed successfully - **inactive (dead)** - Service is stopped - **failed** - Service failed to start or crashed - **activating** - Service is starting up ### Essential Commands ```bash # Check service status systemctl status service-name # Start/stop services systemctl start service-name systemctl stop service-name systemctl restart service-name # Enable/disable auto-start systemctl enable service-name systemctl disable service-name # View logs journalctl -u service-name journalctl -u service-name -f # follow ``` ## Service File Structure ### Basic Service Template ```ini [Unit] Description=My Application Service After=network.target Wants=network-online.target [Service] Type=simple User=appuser Group=appgroup WorkingDirectory=/opt/myapp ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yml ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure RestartSec=5 TimeoutStopSec=30 [Install] WantedBy=multi-user.target ``` ### Service File Locations ```bash # System services /etc/systemd/system/ # Custom services (highest priority) /lib/systemd/system/ # Package-installed services /usr/lib/systemd/system/ # Distribution services # User services ~/.config/systemd/user/ # User-specific services ``` ## Service Types Decision Tree ``` Choose Service Type: ├── Process runs continuously? │ ├── Yes → Type=simple (default) │ └── Process forks/backgrounds itself? │ └── Yes → Type=forking │ ├── Process exits after completion? │ └── Yes → Type=oneshot │ ├── Process sends readiness notification? │ └── Yes → Type=notify │ └── Service provides D-Bus name? └── Yes → Type=dbus ``` ## Common Service Patterns ### Web Application Service ```ini [Unit] Description=My Web App After=network.target postgresql.service Requires=postgresql.service [Service] Type=simple User=webapp Group=webapp WorkingDirectory=/var/www/myapp Environment=NODE_ENV=production ExecStart=/usr/bin/node server.js ExecReload=/bin/kill -USR2 $MAINPID Restart=always RestartSec=10 KillMode=mixed TimeoutStopSec=30 # Security NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ReadWritePaths=/var/www/myapp/logs [Install] WantedBy=multi-user.target ``` ### Background Worker Service ```ini [Unit] Description=Background Job Worker After=redis.service Wants=redis.service [Service] Type=simple User=worker Group=worker WorkingDirectory=/opt/worker ExecStart=/opt/worker/bin/worker Restart=on-failure RestartSec=30 # Resource limits LimitNOFILE=65536 CPUQuota=200% MemoryLimit=2G [Install] WantedBy=multi-user.target ``` ### One-shot Setup Service ```ini [Unit] Description=Application Setup Before=myapp.service [Service] Type=oneshot User=root ExecStart=/opt/myapp/scripts/setup.sh RemainAfterExit=yes [Install] WantedBy=multi-user.target ``` ## Service Management Workflows ### Creating a New Service ```bash # 1. Create service file sudo vim /etc/systemd/system/myservice.service # 2. Reload systemd configuration sudo systemctl daemon-reload # 3. Test the service sudo systemctl start myservice sudo systemctl status myservice # 4. Enable auto-start sudo systemctl enable myservice # 5. Verify it starts on boot sudo systemctl is-enabled myservice ``` ### Service Debugging Process ```bash # 1. Check service status systemctl status myservice # 2. View recent logs journalctl -u myservice -n 50 # 3. Follow logs in real-time journalctl -u myservice -f # 4. Check configuration syntax systemd-analyze verify /etc/systemd/system/myservice.service # 5. Test service dependencies systemctl list-dependencies myservice ``` ### Service Updates and Rollback ```bash # Update service file sudo vim /etc/systemd/system/myservice.service # Reload and restart sudo systemctl daemon-reload sudo systemctl restart myservice # If issues, restore backup and reload sudo cp /etc/systemd/system/myservice.service.backup /etc/systemd/system/myservice.service sudo systemctl daemon-reload sudo systemctl restart myservice ``` ## Advanced Configuration ### Environment Variables ```ini [Service] # Single variable Environment=NODE_ENV=production # Multiple variables Environment=NODE_ENV=production Environment=PORT=3000 # From file EnvironmentFile=/etc/myapp/environment EnvironmentFile=-/etc/myapp/optional.env # Optional file (-) ``` ### Security Hardening ```ini [Service] # User/group isolation User=myapp Group=myapp SupplementaryGroups= # File system protection ProtectSystem=strict ProtectHome=true ReadWritePaths=/var/lib/myapp PrivateTmp=true PrivateDevices=true # Network restrictions PrivateNetwork=false IPAddressDeny=any IPAddressAllow=localhost 192.168.1.0/24 # Capabilities CapabilityBoundingSet=CAP_NET_BIND_SERVICE AmbientCapabilities=CAP_NET_BIND_SERVICE NoNewPrivileges=true # System calls SystemCallFilter=@system-service SystemCallErrorNumber=EPERM ``` ### Resource Limits ```ini [Service] # Memory limits MemoryLimit=1G MemoryMax=1.5G # CPU limits CPUQuota=150% CPUWeight=200 # File limits LimitNOFILE=65536 LimitNPROC=4096 # Restart limits StartLimitBurst=5 StartLimitIntervalSec=300 ``` ## Troubleshooting ### Service Won't Start ```bash # Error: "Job for myservice.service failed" systemctl status myservice # Look for exit codes and error messages # Check logs journalctl -u myservice -n 20 # Common fixes: # 1. File permissions sudo chown -R myuser:mygroup /opt/myapp sudo chmod +x /opt/myapp/bin/myapp # 2. Missing dependencies systemctl list-dependencies myservice --failed # 3. Configuration syntax systemd-analyze verify /etc/systemd/system/myservice.service ``` ### Service Keeps Restarting ```bash # Error: Service in restart loop journalctl -u myservice -f # Check restart configuration systemctl show myservice | grep Restart # Common fixes: # 1. Increase restart delay [Service] RestartSec=30 # 2. Limit restart attempts StartLimitBurst=3 StartLimitIntervalSec=600 # 3. Change restart policy Restart=on-abnormal # Instead of always ``` ### Permission Denied Errors ```bash # Error: "Permission denied" in logs journalctl -u myservice | grep -i permission # Check SELinux context (if enabled) ls -Z /opt/myapp/bin/myapp sudo restorecon -R /opt/myapp/ # Fix file permissions sudo chown myuser:mygroup /opt/myapp/bin/myapp sudo chmod 755 /opt/myapp/bin/myapp # Add user to required groups sudo usermod -a -G docker myuser ``` ### Service Fails to Stop ```bash # Error: Stop job timeout systemctl status myservice # Check stop timeout systemctl show myservice | grep TimeoutStopSec # Force stop if needed systemctl kill myservice systemctl kill -s SIGKILL myservice # Fix timeout in service file [Service] TimeoutStopSec=60 KillMode=mixed ``` ### Dependency Issues ```bash # Error: Dependency failed systemctl list-dependencies myservice --failed # Check dependency status systemctl status postgresql.service # Modify dependencies if needed [Unit] # Required dependency (hard) Requires=postgresql.service After=postgresql.service # Optional dependency (soft) Wants=redis.service After=redis.service ``` ## Monitoring and Logging ### Service Monitoring Commands ```bash # Watch service status watch systemctl status myservice # Monitor all failed services systemctl --failed # Check service start times systemd-analyze blame # Service dependency tree systemctl list-dependencies --all myservice ``` ### Log Management ```bash # View logs with context journalctl -u myservice --since "2 hours ago" journalctl -u myservice --until "2024-01-01" # Filter by priority journalctl -u myservice -p err # Export logs journalctl -u myservice --output=json > myservice.log # Disk usage journalctl --disk-usage sudo journalctl --vacuum-time=7d ```