Linux Performance Tuning & Analysis (GL325)

Module 1: Tuning Principles & the USE Method

• What is Performance Tuning?

• Performance Tuning Methodology

• Model System

• Establish a Baseline

• Establish Performance Targets

• Identify Bottleneck

• Tune

• Measure, Monitor, Evaluate, and Repeat?

• Final Thoughts

• The USE Method

• Gathering USE Metrics

Lab Tasks

• Determining the System’s Configuration

• USE Method: CPU

• USE Method: Memory

• USE Method: Network

• USE Method: Storage I/O & Capacity

 

Module 2: Linux Internals Review

• The Linux Operating System Components

• Open Standards

• Memory Layout

• Important Kernel Threads

• Device Files in Linux

• Managing Linux Device Files

• udev

• Random Numbers and /dev/random

• Block Devices and the Device Mapper

• List Block Devices

• SCSI Devices

• File Systems with Linux

• Hardware Discovery Tools

• Cache and Write Barriers

• Advanced LVM: Components & Object Tags

• Advanced LVM: Automated Storage Tiering

• Advanced LVM: Thin Provisioning

• Advanced LVM: Striping & Mirroring

• Advanced LVM: RAID Volumes

• Kernel Modules

Lab Tasks

• Adjusting Kernel Options

• Linux Kernel Driver Compilation

 

Module 3: Linux Userspace Architecture

• systemd System and Service Manager

• Using systemd

• Security Model Overview

Lab Tasks

• Managing Services with Systemd’s systemctl

• Setting Limits with the pam_limits Modules

• Operation of the setuid() and capset() System Calls

 

Module 4: Log File Administration

• System Logging

• systemd Journal

• systemd Journal’s journalctl

• Secure Logging with Journal’s Log Sealing

• gnome-system-log

• Rsyslog

• /etc/rsyslog.conf

• Log Management

• Log Anomaly Detector

• Sending logs from the shell

Lab Tasks

• Using the systemd Journal

• Setting up a Full Debug Logfile

 

Module 5: Monitoring & Troubleshooting

• Viewing Processes

• System Status – Memory

• System Status – I/O

• System Status – CPU

• Performance Trending with sar

• Determining Service to Process Mapping

• Analyzing Process Memory Usage with pmap

• Real-time Monitoring of Resources — Cgroups

• strace and ltrace

• lsof and fuser

• ss/netstat and rpcinfo

• tcpdump and wireshark

Lab Tasks

• Process Related Tools • System Activity Reporter • Cgroup for Processes • Analyzing Memory Usage

 

Module 6: Benchmark Tools

• Benchmark Tools

• Disk/Filesystem Benchmarks

• Network Benchmarks

• Phoronix Test Suite

Lab Tasks

• Benchmarking Disk and Filesystem

• Bonnie++ Visualization

 

Module 7: Systemtap

• SystemTap Overview

• SystemTap Features

• SystemTap Requirements

• Invoking SystemTap

• The 5 passes of stap

• SystemTap Probe Point Syntax

• SystemTap Probe Modes

• SystemTap Probe Points

• SystemTap process Probes

• Listing and Using Probe Points

• Built-in SystemTap Functionality

• SystemTap Example Repositories

• SystemTap Advanced Example

• SystemTap Documentation

Lab Tasks

• System Tap Intro

• System Tap

• Creating SystemTap Scripts

• SystemTap One-liners & Histograms

 

Module 8: Performance Co-Pilot

• Performance Co-Pilot Introduction

• PCP Architecture

• PCP Performance Metrics

• PCP Installation

• PCP Performance Monitoring

• PCP Centralized Logging

• PCP Inference Engine

• PCP Deployment Strategies

• PCP Advanced Topics

Lab Tasks

• Performance Co-Pilot Introduction

• Live Monitoring with pmchart

• Install Web Monitoring Tools

 

Module 9: Remote Metric Gathering with SNMP

• SNMP Overview

• SNMP Operation - Query

• MIBs and OID Namespace

• SNMP Security

• SNMP for Linux

Lab Tasks

• SNMP

 

Module 10: Linux perf

• perf Overview

• perf Commands Typical Workflow

• perf Events

• perf stat

• perf record

• perf report

Lab Tasks

• Perf

 

Module 11: Linux Tunables

• Operating System Tuning Caveats

• Consider OS (or Kernel) Upgrade

• Tuning CPU - NUMA

• Tuning CPU - Scheduling

• Huge Memory Pages

• Tuning Virtual Memory Manager

• Tuning Virtual Memory Manager (cont.)

• Dropping Caches

• I/O Scheduler

• Tuning Filesystems

• Tuning Filesystems (cont.)

• Network Tuning

• tuned 

Prerequisites

An intermediate knowledge of the Linux operating system is required. The equivalent knowledge can be found in the U8583S “Linux Fundamentals” and H7091S “Enterprise Linux Systems Administration” courses.