systemd Services — SKILL.md

---
name: "systemd Services"
description: "Skill for systemd Services — auto-generated from documentation"
version: "1.0.0"
author: "skynet"
category: "ops"
agents: ["claude-code", "codex", "gemini"]
tags: ["systemd", "ops", "auto-generated"]
---

# systemd Services

---
name: systemd Services
description: Use this skill when you need to create, manage, and troubleshoot systemd services on Linux systems. Essential for service management, process supervision, dependency handling, and system automation.
category: ops
metadata:
  author: skynet
  version: 1.0.0
---

# systemd Services

## Overview
systemd is the default init system and service manager for most modern Linux distributions. Services define how processes are started, stopped, monitored, and managed by the system.

## Quick Reference

### Service States
- **active (running)** - Service is running
- **active (exited)** - One-shot service completed successfully
- **inactive (dead)** - Service is stopped
- **failed** - Service failed to start or crashed
- **activating** - Service is starting up

### Essential Commands
```bash
# Check service status
systemctl status service-name

# Start/stop services
systemctl start service-name
systemctl stop service-name
systemctl restart service-name

# Enable/disable auto-start
systemctl enable service-name
systemctl disable service-name

# View logs
journalctl -u service-name
journalctl -u service-name -f  # follow
```

## Service File Structure

### Basic Service Template
```ini
[Unit]
Description=My Application Service
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=appuser
Group=appgroup
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target
```

### Service File Locations
```bash
# System services
/etc/systemd/system/          # Custom services (highest priority)
/lib/systemd/system/          # Package-installed services
/usr/lib/systemd/system/      # Distribution services

# User services
~/.config/systemd/user/       # User-specific services
```

## Service Types Decision Tree

```
Choose Service Type:
├── Process runs continuously?
│   ├── Yes → Type=simple (default)
│   └── Process forks/backgrounds itself?
│       └── Yes → Type=forking
│
├── Process exits after completion?
│   └── Yes → Type=oneshot
│
├── Process sends readiness notification?
│   └── Yes → Type=notify
│
└── Service provides D-Bus name?
    └── Yes → Type=dbus
```

## Common Service Patterns

### Web Application Service
```ini
[Unit]
Description=My Web App
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=webapp
Group=webapp
WorkingDirectory=/var/www/myapp
Environment=NODE_ENV=production
ExecStart=/usr/bin/node server.js
ExecReload=/bin/kill -USR2 $MAINPID
Restart=always
RestartSec=10
KillMode=mixed
TimeoutStopSec=30

# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/www/myapp/logs

[Install]
WantedBy=multi-user.target
```

### Background Worker Service
```ini
[Unit]
Description=Background Job Worker
After=redis.service
Wants=redis.service

[Service]
Type=simple
User=worker
Group=worker
WorkingDirectory=/opt/worker
ExecStart=/opt/worker/bin/worker
Restart=on-failure
RestartSec=30

# Resource limits
LimitNOFILE=65536
CPUQuota=200%
MemoryLimit=2G

[Install]
WantedBy=multi-user.target
```

### One-shot Setup Service
```ini
[Unit]
Description=Application Setup
Before=myapp.service

[Service]
Type=oneshot
User=root
ExecStart=/opt/myapp/scripts/setup.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
```

## Service Management Workflows

### Creating a New Service
```bash
# 1. Create service file
sudo vim /etc/systemd/system/myservice.service

# 2. Reload systemd configuration
sudo systemctl daemon-reload

# 3. Test the service
sudo systemctl start myservice
sudo systemctl status myservice

# 4. Enable auto-start
sudo systemctl enable myservice

# 5. Verify it starts on boot
sudo systemctl is-enabled myservice
```

### Service Debugging Process
```bash
# 1. Check service status
systemctl status myservice

# 2. View recent logs
journalctl -u myservice -n 50

# 3. Follow logs in real-time
journalctl -u myservice -f

# 4. Check configuration syntax
systemd-analyze verify /etc/systemd/system/myservice.service

# 5. Test service dependencies
systemctl list-dependencies myservice
```

### Service Updates and Rollback
```bash
# Update service file
sudo vim /etc/systemd/system/myservice.service

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart myservice

# If issues, restore backup and reload
sudo cp /etc/systemd/system/myservice.service.backup /etc/systemd/system/myservice.service
sudo systemctl daemon-reload
sudo systemctl restart myservice
```

## Advanced Configuration

### Environment Variables
```ini
[Service]
# Single variable
Environment=NODE_ENV=production

# Multiple variables
Environment=NODE_ENV=production
Environment=PORT=3000

# From file
EnvironmentFile=/etc/myapp/environment
EnvironmentFile=-/etc/myapp/optional.env  # Optional file (-)
```

### Security Hardening
```ini
[Service]
# User/group isolation
User=myapp
Group=myapp
SupplementaryGroups=

# File system protection
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp
PrivateTmp=true
PrivateDevices=true

# Network restrictions
PrivateNetwork=false
IPAddressDeny=any
IPAddressAllow=localhost 192.168.1.0/24

# Capabilities
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

# System calls
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
```

### Resource Limits
```ini
[Service]
# Memory limits
MemoryLimit=1G
MemoryMax=1.5G

# CPU limits
CPUQuota=150%
CPUWeight=200

# File limits
LimitNOFILE=65536
LimitNPROC=4096

# Restart limits
StartLimitBurst=5
StartLimitIntervalSec=300
```

## Troubleshooting

### Service Won't Start
```bash
# Error: "Job for myservice.service failed"
systemctl status myservice
# Look for exit codes and error messages

# Check logs
journalctl -u myservice -n 20

# Common fixes:
# 1. File permissions
sudo chown -R myuser:mygroup /opt/myapp
sudo chmod +x /opt/myapp/bin/myapp

# 2. Missing dependencies
systemctl list-dependencies myservice --failed

# 3. Configuration syntax
systemd-analyze verify /etc/systemd/system/myservice.service
```

### Service Keeps Restarting
```bash
# Error: Service in restart loop
journalctl -u myservice -f

# Check restart configuration
systemctl show myservice | grep Restart

# Common fixes:
# 1. Increase restart delay
[Service]
RestartSec=30

# 2. Limit restart attempts
StartLimitBurst=3
StartLimitIntervalSec=600

# 3. Change restart policy
Restart=on-abnormal  # Instead of always
```

### Permission Denied Errors
```bash
# Error: "Permission denied" in logs
journalctl -u myservice | grep -i permission

# Check SELinux context (if enabled)
ls -Z /opt/myapp/bin/myapp
sudo restorecon -R /opt/myapp/

# Fix file permissions
sudo chown myuser:mygroup /opt/myapp/bin/myapp
sudo chmod 755 /opt/myapp/bin/myapp

# Add user to required groups
sudo usermod -a -G docker myuser
```

### Service Fails to Stop
```bash
# Error: Stop job timeout
systemctl status myservice

# Check stop timeout
systemctl show myservice | grep TimeoutStopSec

# Force stop if needed
systemctl kill myservice
systemctl kill -s SIGKILL myservice

# Fix timeout in service file
[Service]
TimeoutStopSec=60
KillMode=mixed
```

### Dependency Issues
```bash
# Error: Dependency failed
systemctl list-dependencies myservice --failed

# Check dependency status
systemctl status postgresql.service

# Modify dependencies if needed
[Unit]
# Required dependency (hard)
Requires=postgresql.service
After=postgresql.service

# Optional dependency (soft)
Wants=redis.service
After=redis.service
```

## Monitoring and Logging

### Service Monitoring Commands
```bash
# Watch service status
watch systemctl status myservice

# Monitor all failed services
systemctl --failed

# Check service start times
systemd-analyze blame

# Service dependency tree
systemctl list-dependencies --all myservice
```

### Log Management
```bash
# View logs with context
journalctl -u myservice --since "2 hours ago"
journalctl -u myservice --until "2024-01-01"

# Filter by priority
journalctl -u myservice -p err

# Export logs
journalctl -u myservice --output=json > myservice.log

# Disk usage
journalctl --disk-usage
sudo journalctl --vacuum-time=7d
```