GCP Cloud Logging

--- name: GCP Cloud Logging description: Use this skill when you need to collect, store, search, analyze, monitor, and alert on log data and events from Google Cloud and other applications. Essential for debugging, monitoring application performance, security analysis, and compliance requirements. metadata: author: skynet version: 1.0.0 category: infrastructure --- # GCP Cloud Logging ## Overview Google Cloud Logging is a fully managed service that performs at scale and can ingest application and platform log data. It provides real-time log management and analysis capabilities for troubleshooting, monitoring, and auditing. ## Prerequisites ```bash # Install and authenticate gcloud CLI gcloud auth login gcloud config set project YOUR_PROJECT_ID # Enable Cloud Logging API gcloud services enable logging.googleapis.com ``` ## Core Operations ### Viewing Logs ```bash # View recent logs from all resources gcloud logging read --limit=50 # View logs from specific resource gcloud logging read "resource.type=gce_instance" --limit=20 # View logs with severity filter gcloud logging read "severity>=ERROR" --limit=30 # View logs from specific time range gcloud logging read --since="2024-01-01" --until="2024-01-02" # View logs with custom filter gcloud logging read 'resource.type="k8s_container" AND resource.labels.namespace_name="production"' --limit=25 ``` ### Writing Custom Logs ```bash # Write a simple log entry gcloud logging write my-log "Hello from custom application" --severity=INFO # Write structured log entry gcloud logging write my-log '{"message": "User login", "user_id": "123", "action": "login"}' \ --severity=NOTICE --resource=type=global # Write log with custom timestamp gcloud logging write my-log "Scheduled task completed" \ --severity=INFO \ --timestamp="2024-01-15T10:30:00Z" ``` ### Log-based Metrics ```bash # Create log-based metric gcloud logging metrics create error_count \ --description="Count of application errors" \ --log-filter='severity>=ERROR AND resource.type="gae_app"' # List existing metrics gcloud logging metrics list # Update metric filter gcloud logging metrics update error_count \ --log-filter='severity>=ERROR AND jsonPayload.component="auth"' # Delete metric gcloud logging metrics delete error_count ``` ### Log Sinks ```bash # Create Cloud Storage sink gcloud logging sinks create audit-logs-sink \ storage.googleapis.com/my-audit-bucket \ --log-filter='protoPayload.serviceName="cloudaudit.googleapis.com"' # Create BigQuery sink gcloud logging sinks create analytics-sink \ bigquery.googleapis.com/projects/PROJECT_ID/datasets/logs_dataset \ --log-filter='resource.type="gce_instance" AND severity>=WARNING' # Create Pub/Sub sink gcloud logging sinks create alert-sink \ pubsub.googleapis.com/projects/PROJECT_ID/topics/log-alerts \ --log-filter='severity>=ERROR' # List sinks gcloud logging sinks list # Update sink filter gcloud logging sinks update analytics-sink \ --log-filter='resource.type="gce_instance" AND severity>=INFO' ``` ## Advanced Filtering ### Common Filter Patterns ```bash # Application logs with specific labels gcloud logging read 'resource.type="k8s_container" AND resource.labels.container_name="web-server" AND jsonPayload.level="error"' # Time-based filtering with severity gcloud logging read 'timestamp>="2024-01-15T00:00:00Z" AND severity>=WARNING AND resource.type="gae_app"' # HTTP request logs gcloud logging read 'httpRequest.status>=400 AND resource.type="http_load_balancer"' # Audit logs for specific operations gcloud logging read 'protoPayload.methodName="v1.compute.instances.insert" AND protoPayload.serviceName="compute.googleapis.com"' ``` ### Regular Expression Filtering ```bash # Filter logs with regex pattern gcloud logging read 'jsonPayload.message=~"user.*login.*failed"' # Multiple regex conditions gcloud logging read 'resource.type="gce_instance" AND jsonPayload.message=~"(error|exception|failure)"' ``` ## Decision Tree: Choosing Log Storage Strategy ``` Need to store logs long-term? ├── Yes → Compliance/Audit requirements? │ ├── Yes → Use Cloud Storage sink with lifecycle policies │ └── No → Use BigQuery sink for analytics └── No → Real-time processing needed? ├── Yes → Use Pub/Sub sink with Cloud Functions └── No → Use default Cloud Logging retention (30 days) ``` ## Monitoring and Alerting ### Setting Up Log-based Alerts ```bash # Create alerting policy (requires Cloud Monitoring) gcloud alpha monitoring policies create --policy-from-file=policy.yaml ``` Example policy.yaml: ```yaml displayName: "High Error Rate Alert" conditions: - displayName: "Error rate condition" conditionThreshold: filter: 'resource.type="gae_app"' comparison: COMPARISON_GREATER_THAN thresholdValue: 10 duration: 300s notificationChannels: - projects/PROJECT_ID/notificationChannels/CHANNEL_ID ``` ### Log Router Configuration ```bash # View current log router configuration gcloud logging describe-log-router # Create custom log router gcloud logging routers create custom-router \ --description="Route critical logs to multiple destinations" ``` ## Programmatic Access ### Using Client Libraries (Python Example) ```python from google.cloud import logging # Initialize client client = logging.Client() # Write log entry logger = client.logger('my-application') logger.log_text('Application started', severity='INFO') # Read logs for entry in client.list_entries(filter_='severity>=ERROR'): print(f'{entry.timestamp}: {entry.payload}') ``` ### REST API Examples ```bash # List log entries via REST API curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \ "https://logging.googleapis.com/v2/entries:list" \ -d '{"resourceNames":["projects/PROJECT_ID"],"filter":"severity>=ERROR"}' # Write log entry via REST API curl -X POST \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "Content-Type: application/json" \ "https://logging.googleapis.com/v2/entries:write" \ -d '{ "entries": [{ "logName": "projects/PROJECT_ID/logs/my-app", "resource": {"type": "global"}, "textPayload": "Custom log message", "severity": "INFO" }] }' ``` ## Troubleshooting ### Common Issues and Solutions **Error: "Permission denied" when reading logs** ```bash # Grant required IAM permissions gcloud projects add-iam-policy-binding PROJECT_ID \ --member="user:EMAIL" \ --role="roles/logging.viewer" ``` **Error: "Log sink creation failed - bucket not found"** ```bash # Verify bucket exists and has correct permissions gsutil ls gs://BUCKET_NAME gsutil iam ch serviceAccount:cloud-logs@system.gserviceaccount.com:objectCreator gs://BUCKET_NAME ``` **Error: "Invalid log filter syntax"** ```bash # Test filter syntax before using gcloud logging read 'resource.type="gce_instance"' --limit=1 --dry-run ``` **Logs not appearing in BigQuery sink** ```bash # Check sink configuration gcloud logging sinks describe SINK_NAME # Verify BigQuery dataset permissions bq show --format=prettyjson DATASET_NAME ``` **High logging costs** ```bash # Analyze log volume by resource type gcloud logging read 'timestamp>="2024-01-01"' \ --format="value(resource.type)" | sort | uniq -c | sort -nr # Set up exclusion filters for noisy logs gcloud logging sinks update _Default \ --add-exclusion="name=health-checks,filter=jsonPayload.path=/health" ``` ### Performance Optimization ```bash # Use specific resource filters to reduce query time gcloud logging read 'resource.type="gce_instance" AND resource.labels.instance_id="INSTANCE_ID"' # Limit time range for better performance gcloud logging read 'timestamp>="2024-01-15T10:00:00Z" AND timestamp<="2024-01-15T11:00:00Z"' # Use appropriate page sizes gcloud logging read --limit=100 # Instead of very large limits ``` ### Debug Mode ```bash # Enable verbose output for troubleshooting gcloud logging read --verbosity=debug 'resource.type="gce_instance"' # Check quota usage gcloud logging operations list ```