Red Hat Deep Dive Sessions Linux on System z Shawn Wells (swells@redhat.com) W/W Lead Architect, Linux on System z Team Lead, System z SMEs 1

Introduction  Shawn Wells (swells@redhat.com) Lead Architect, Linux on System z Team Lead, System z SMEs Phone: +1 443 534 0130 2

Agenda  Scheduled questions to be answered in this session:  What’s the Linux on System z development process?  What’s in RHEL now? What’s on the roadmap?  Provisioning & Patch Management in RHN  Security Update SELinux, Audit, etc 3

Linux on System z Development Process

Linux on System z Development Community  Development with “upstream” communities  Kernel, glibc, etc  Collaboration with partners, IBM, open source contributors 5

Linux on System z Development Fedora  Bleeding Edge  Sets direction for RHEL technologies  Community Supported  Released ~6mo cycles  Fedora 8,9,10 = RHEL6 Fedora 8; http://fedoraproject.org/wiki/Releases/8/FeatureList Fedora 9; http://fedoraproject.org/wiki/Releases/9/FeatureList Fedora 10; http://fedoraproject.org/wiki/Releases/10/FeatureList 6

Linux on System z Development Red Hat Enterprise Linux  Stable, mature, commercial product  Extensive Q&A, performance testing  Hardware & Software Certifications  7yr maintenance  Core ABI compatibility guarantee  Major releases 2-3yr cycle 7

Support Cycle Extended Product Lifecycle Years 1 - 4 Yr 5 Yr 6,7 Pro d Pro d Security Patches Bug Fixes Hardware Enablement Software Enhancements uc tio n1 uc tio Pro d n2 uc tio n3 X X X X X X Full Partial None X 8

Linux on System z Subscriptions    No Upgrade Costs Red Hat Enterprise Linux Subscription No Client Access Fee Unlimited Support Incidents For System z:  Priced Per IFL  Unlimited VMs per IFL Customers can consolidate subscriptions to or from other platforms S U P P O R T PREMIUM STANDARD BASIC 24x7 Phone/Web 1 Hour SLAs Phone/Web 1-4 Business Hour SLA Web Support. 2 Day SLA Security, Bug Fixes Regular H/W & S/W Updates Hardware & Application Certifications Stable Application Interfaces Upgrades to New Versions Product Source & Binaries 9

Linux on System z Support C O N S U L T A N T S T E C H A C C N T M G R S Level 3: Special Engineering Custom Patches, Code Re-writes, Interim Patches, Application Redesign Level 2: Advanced Support Reproduce Problems, Grouped via Skillsets Level 1: Front Line Support Known Issues, Initial Troubleshooting, Everyone is minimum RHCE Support via Red Hat 10

Linux on System z Support C O N S U L T A N T S T E C H A C C N T M G R S Level 3: Special Engineering Custom Patches, Code Re-writes, Interim Patches, Application Redesign Level 2: Advanced Support Reproduce Problems, Grouped via Skillsets P A R T N E R Level 1: Front Line Support Known Issues, Initial Troubleshooting, Everyone is minimum RHCE Support via Red Hat Level 2: Advanced Support Reproduce Problems, Category Specialists Level 1: First Responders T A M Basic Support Support via IBM 11

What’s in RHEL now? What’s on the road map?

13

IBM Changes to 2.6.x Kernel RHEL4 RHEL5 14

RHEL Now: RHEL 5.2  Support for z10  Dynamic CHPID reconfiguration  Improved “ssh -X” with VPN during installation process  Better network performance with skb scatter-gather support  Implementation of SCSI dump infrastructure 15

RHEL Now: RHEL 5.2  Accelerated in­kernel Crypto Support for crypto algorithms of z10  SHA-512, SHA-384, AES-192, AES-256   Two OSA ports per CHPID; Four port exploitation Exploit next OSA adapter generation which offers two ports within one CHPID. The additional port number 1 can be specified with the qeth sysfs-attribute “portno”  Support is available only for OSA-Express3 GbE SX and LX on z10, running in LPAR or z/VM guest (PFT for z/VM APAR VM64277 required!)  16

RHEL Now: RHEL 5.2  Large Page Support   This adds hugetblfs support on System z, using both hardware large page support if available, and software large page emulation (with shared hugetblfs pagetables) on older hardware skb scatter-gather support for large incoming messages  This avoids allocating big chunks of consecutive memory and should increase networking throughput in some situations for large incoming packets Full Release Notes At: redhat.com http://www.redhat.com/docs/enUS/Red_Hat_Enterprise_Linux/5.2/html/Release_Notes/s390x/index.html 17

RHEL Now: RHEL 5.2  Lightweight userspace priority inheritance (PI) support for futexes, useful for realtime applications (2.6.18)  Assists priority inversion handling. Ref: http://lwn.net/Articles/178253/  High resolution timers (2.6.16)  Provide fine resolution and accuracy depending on system configuration and capabilities - used for precise in-kernel timing  New Pipe implementation (2.6.11)  30-90% perf improvement in pipe bandwidth  Circular buffer allow more buffering rather than blocking writers  “Big Kernel Semaphore”: Turns the Big Kernel Lock into a semaphore  Latency reduction, by breaking up long lock hold times and adds voluntary preemption 18

RHEL Now: RHEL 5.2  Process Events Connector (2.6.15) Reports fork, exec, id change, and exit events for all processes to userspace  Useful for accounting/auditing (e.g. ELSA), system activity monitoring, security, and resource management   kexec & kdump (2.6.13)  Provide new crash-dumping capability with reserved, memory-resident kernel  Extended device mapper multipath support  Address space randomization:  Address randomization of multiple entities – including stack & mmap() region (used by shared libraries) (2.6.12; more complete implementation than in RHEL4)  Greatly complicates and slows down hacker attacks  Audit subsytem  Support for process-context based filtering (2.6.17)  More filter rule comparators (2.6.17) 19

RHEL Now: RHEL 5.2  Add nf_conntrack subsystem: (2.6.15) Common IPv4/IPv6 generic connection tracking subsystem  Allows IPv6 to have a stateful firewall capability (not previously possible)  Increased security  Enables analysis of whole streams of packets, rather than only checking the headers of individual packets   SELinux per-packet access controls  Replaces old packet controls  Add Secmark support to core networking  Allows security subsystems to place security markings on network packets (2.6.18) 20

RHEL Tomorrow: RHEL 5.3  Currently in beta  Interested in being a beta tester?  NSS  CPU Affinity  ETR Support  Device-multipath support for xDR  RHT BugZilla: 184770  IBM LTC 18425-62140 21

RHEL Tomorrow: Fedora Fedora is Red Hat’s bleeding edge, an incubator for new technologies and features Fedora sets our direction for Red Hat Enterprise Linux, and gives you a good idea of what will be in our next RHEL release (… and in other Linux distros, too) Fedora 8; http://fedoraproject.org/wiki/Releases/8/FeatureList Fedora 9; http://fedoraproject.org/wiki/Releases/9/FeatureList Fedora 10; http://fedoraproject.org/wiki/Releases/10/FeatureList Fedora 8,9,10 = RHEL6 22

RHEL Tomorrow: “In Place” Upgrade Currently a beta feature in RHEL 5.3 “In Place” Upgrades: preupgrade Will download files needed to upgrade, Store them locally on disk Reboot you into the installer Not a true in­place upgrade (yet)!  Benefit The longest part of an install is when packages are downloaded to the local machine Pre­Upgrade downloads and stores packages locally, while the machine is running/in production Reboot directly into the installer 23

RHEL Tomorrow: “In Place” Upgrade  Select Target Version 24

RHEL Tomorrow: “In Place” Upgrade  Determines which packages need upgrading, and downloads them 25

RHEL Tomorrow: “In Place” Upgrade  Downloads new initrd & kernel images 26

RHEL Tomorrow: “In Place” Upgrade  User reboots, brought into installer 27

RHEL Tomorrow: gnome-control-center  gnome­control­center  It is not YaST (yet)  It is a unified GUI for package management and system configuration  Benefit  Progress towards a YaST-like tool in RHEL (currently we have the system­config­* GUIs/TUIs) 28

RHEL Tomorrow: PackageKit  PackageKit  Abstraction layer for YUM, apt, conary, etc  Provides a common set of abstractions that can be used by GUI/TUI package managers rpm dpkg ipkg 29

RHEL Tomorrow: PackageKit  PackageKit  Abstraction layer for YUM, apt, conary, etc  Provides a common set of abstractions that can be used by GUI/TUI package managers PackageKit rpm dpkg ipkg 30

RHEL Tomorrow: PackageKit  PackageKit  Abstraction layer for YUM, apt, conary, etc  Provides a common set of abstractions that can be used by GUI/TUI package managers YaST RHN PackageKit rpm dpkg ipkg 31

Linux Virtualization on System z

JBoss Oracle Custom App Red Hat Enterprise Linux Red Hat Enterprise Linux Red Hat Enterprise Linux z/VM CONNECTIVITY PROCESSING DATA 33

Using RHN Satellite to Manage s390/s390x & distributed

Red Hat Network Satellite Update 35

Red Hat Network Satellite Update Manage 36

Red Hat Network Satellite Update Manage Provision 37

Red Hat Network Satellite Update Manage Provision Monitor 38

Red Hat Network Satellite 39

Red Hat Network Satellite 40

Red Hat Network Satellite 41

RHN Installation Requirements  Software  RHEL 4 (31-bit or 64-bit)  @Base install  Hardware  1 to 2 (virtual) IFLs  2 to 4 GB storage (memory)  1 GB swap (combination VDISK, disk)  1 x mod3 for OS install  Estimated 12 GB disk space for embedded database  6 GB per channel (disk) 42

43

44

RHN Satellite Is Now Open Source http://spacewalk.redhat.com  Announced at Red Hat Summit 2008  …. remember the Fedora -> RHEL model? 45

Security

Agenda  Why do we need SELinux? What are the principal concepts?  SELinux Details  Type Enforcement  What are the available policies?  What’s a policy actually made of?  How do I {add, change} a policy?  What’s the associated overhead?  Usage  User Perspective  Admin Perspective  Scenarios  Fixing the RHT Corporate VPN “update” 47

Name of Presentation Why do we need Red Hat Presenter SELinux?

Linux Access Control Problems  Access is based off users’ access Example: Firefox can read SSH keys # ps -x | grep firefox shawn 21375 1 35 11:38 ? 00:00:01 firefox-bin # ls -l id_rsa -rw———- 1 shawn shawn 1743 2008-08-10 id_rsa Fundamental Problem: Security properties not specific enough. Kernel can’t distinguish applications from users. 49

Linux Access Control Problems 2) Processes can change security properties Example: Mail files are readable only by me….. but Thunderbird could make them world readable Fundamental Problems:  Standard access control is discretionary  Includes concept of “resource ownership”  Processes can escape security policy 50

Linux Access Control Problems 3) Only two privilege levels: User & root Example: Apache gets hacked, allowing remote access to root. Entire system is compromised. Fundamental Problems:  Simplistic security policy  No way to enforce least-privilege 51

SELinux: Building Security Openly NSA Develops SELinux Integrated into Linux Kernel Enabled By DEFAULT In RHEL Integrated into Open Source Project Customers, NSA, Community, and Red Hat continue evolution 52

Red Hat Security Certifications NIAP/Common Criteria: The most evaluated operating system platform ● Red Hat Enterprise Linux 2.1 – EAL 2 (Completed: February 2004) ● Red Hat Enterprise Linux 3 EAL 3+/CAPP (Completed: August 2004) ● Red Hat Enterprise Linux 4 EAL 4+/CAPP (Completed: February 2006) ● Red Hat Enterprise Linux 5 EAL4+/CAPP/LSPP/RBAC (Completed: June 2007) DII-COE ● Red Hat Enterprise Linux 3 (Self-Certification Completed: October 2004) ● Red Hat Enterprise Linux: First Linux platform certified by DISA DCID 6/3 ● Currently PL3/PL4: ask about kickstarts. ● Often a component in PL5 systems DISA SRRs / STIGs ● Ask about kickstarts. FIPS 140-2 ● Red Hat / NSS Cryptography Libraries certified Level 2 53

Security Standards Work Extensible Configuration Checklist Description Format (XCCDF)  Enumeration for configuration requirements  DISA FSO committed to deploying STIG as XCCDF  Others working with NIST  Security policy becomes one file Open Vulnerability & Assessment Language (OVAL)  Machine-readable versions of security advisories Common Vulnerability and Exposures (CVE) Compatibility  Trace a vulnerability through multiple vendors 54

Name of Presentation How’s it work? Red Hat Presenter

Linux Access Control Introduction Linux access control involves the kernel controling Processes (running programs), which try to access… Resources (files, directories, sockets, etc) For example:  Apache (process) can read web files  But not the /etc/shadow file (resource) Traditional methods do not clearly separate the privileges of users and applications acting on the users behalf, increasing the damage that can be caused by application exploits. So, how should these decisions be made? 56

Security Architecture Every subject (i.e process) and object (i.e. data files) are assigned collections of security attributes, called a security context 1) Security context of subject & object passed to SELinux 2) Kernel/SELinux check, verify access 2a) Grant access. Record allowance in AVC (Access Vector Cache) 2b) Deny access, log error 57

Security Architecture Or in picture view… 58

Name of Presentation SELinux Details Red Hat Presenter

SELinux Contexts root:object r:sysadm home t:s0:c0 ● ● ● ● ● ● The above is an SELinux context user_t role_t file_t Sensitivity category 60

Role Based Access Control (RBAC) “root” really isn’t “root” i.e: root_u:WebServerAdmin_r:SysAdmin_t root_u:OracleDBAdmin_r:SysAdmin_t 61

SELinux Contexts 62

SELinux Policy ● ● Policies are matrices of statements which tell SELinux if certain actions are allowed based on the context of the objects attempting those actions. There are three SELinux Policy Types 63

The Three SELinux Policy Types  Targeted Policy  Default policy in RHEL5. Supported by HelpDesk.  Targets specific applications to lock down.   Allows all other applications to run in the unconfined domain (unconfined_t) Applications running in the unconfined domain run as if SELinux were disabled 64

The Three SELinux Policy Types 2) Strict Policy  Denies access to everything by default  Complete protection for all processes on the system    Requires that policies be written for all applications, often requires customization Strict is type enforcement with added types for users (e.g. user_t and user_firefox_t). Not enabled by Red Hat as default 65

The Three SELinux Policy Types 3) Multi-Level Security (MLS)    Focuses on confidentiality (i.e. separation of multiple classifications of data) Ability to manage {processes, users} with varying levels of access. (i.e. “the need to know”) Uses category & sensitivity levels 66

The Three SELinux Policy Types 3) Multi-Level Security (MLS) (a) Sensitivity Labels  Mostly used by the government – Top Secret, Secret, Unclassified, etc s0 Unclassified s1 Secret s2 Top Secret 67

The Three SELinux Policy Types 3) Multi-Level Security (MLS) (b) Category Labels  Separation of data types, compartments, projects, etc s0 s1 s1 Unclassified Secret Top Secret c0 Project A c0 Alpha c1 Project B c1 Bravo c2 Charlie c3 Delta 68

The Three SELinux Policy Types 3) Multi-Level Security (MLS) (b) Polyinstantiation & pam_namespace   The pam_namespace PAM module sets up a private namespace for a session with polyinstantiated directories A polyinstantiated directory provides a different instance of itself based on user name, or when using SELinux, user name, security context or both 69

The Three SELinux Policy Types 3) Multi-Level Security (MLS) (b) Polyinstantiation & pam_namespace # id ­Z staff_u:WebServer_Admin_r:WebServer_Admin_t:s0:c0 # ls ­l /data secret­file­1 secret­file 2 # id ­Z staff_u:WebServer_Admin_r:WebServer_Admin_t:s1:c0 # ls ­l /data secret­file­1 secret­file 2 top­secret­file­1 70

The Three SELinux Policy Types Multi-Level Security (MLS) & Common Criteria   The Common Criteria (CC) is an international security standard against which systems are evaluated. Many government customers require CC evaluated systems. Red Hat Enterprise Linux 5 meets EAL4+ with RBAC/LSPP/CAPP endorcements 71

What’s the Performance Overhead? 72

What’s the Performance Overhead? 73

What’s the Performance Overhead? 74

What’s the Performance Overhead? ● Not official statistics ● Laptop = 2GHz, 2x 1GB RAM ● Workstation = 2.13GHz, 4x 1GB RAM ● Apache = Lots of threads ● MySQL = Lots of disk I/O 75

Name of Presentation SELinux Usage Red Hat Presenter (GUI & console)

End-User Perspective ● sealert Notifications 77

End-User Perspective ● sealert Browser 78

System Administrator Perspective ● sealert + EMail Notifications 79

System Administrator Perspective ● system-config-selinux 80

System Administrator Perspective ● sediffx 81

System Administrator Perspective ● apol 82

Name of Presentation SELinux Usage Red Hat Presenter (Hints & Tips)

System Administrator Perspective ● semanage Configure elements of SELinux policy without modification/recompilation of policy sources … . aka on the fly Example: Dynamically Allowing Apache to listen on port 1234 # semanage port ­a ­t httpd_port_t ­p tcp 1234 84

System Administrator Perspective ● semanage (more examples) Example: Allow shawn to join “webadmin_u” group # semanage login ­a ­s webadmin_u shawn Example: Relabel files for access by Apache # semanage fcontext ­a ­t \ httpd_sys_content_t “/data/webpages(/.*)?” 85

System Administrator Perspective ● semanage (most important example) You don’t need to disable SELinux to fix a single error! type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=­19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) The Fix: # semanage permissive ­a ifconfig_t 86

System Administrator Perspective ● audit2allow Allows generation of SELinux policy rules from logs of denied operations Example: Fix all the errors on the system (completely not a good idea on a real system)

cat /var/log/audit/audit.log | audit2allow ­M FixAll Generating type enforcment file: FixAll.te Compiling policy: checkmodule ­M ­m ­o FixAll.mod FixAll.te Building package: semodule_package ­o FixAll.pp ­m FixAll.mod

semodule ­i FixAll.pp

87

Name of Presentation Red Hat Presenter Scenarios

Scenario: Fixing the RHT corporate VPN “update” ● Red Hat has a Corporate Standard Build (CSB) for desktop environments ● Red Hat pushes updates to said CSB ● I “tweak” my configuration files ● When RHT pushed a CSB update, it broke my VPN settings 89

Scenario: Fixing the RHT corporate VPN “update” /var/log/messages: type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=­19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) Now what? 90

Scenario: Fixing the RHT corporate VPN “update” type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=­19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” gid=0 euid=0 suid=0 fsuid=0 egid=0 exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) What I Know: 1) 2) 3) AVC Event ID 738 syscall=54 (I’d have to google this) root (or an application on its behalf) was running /sbin/ip 4) context = user_u:system_r:ifconfig_t:s0 91

Scenario: Fixing the RHT corporate VPN “update” type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=­19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” gid=0 euid=0 suid=0 fsuid=0 egid=0 exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) My Options: 1) Create a SELinux Policy Module # ausearch ­x “/sbin/ip” | audit2allow ­M MyVPNFix 92

Scenario: Fixing the RHT corporate VPN “update” type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=­19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” gid=0 euid=0 suid=0 fsuid=0 egid=0 exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) My Options: 1) Create a SELinux Policy Module # ausearch ­x “/sbin/ip” | audit2allow ­M MyVPNFix # semodule ­i MyVPNFix.pp 93

Scenario: Fixing the RHT corporate VPN “update” type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=­19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” gid=0 euid=0 suid=0 fsuid=0 egid=0 exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) My Options: 2) Disable enforcement of ifconfig_t (there is no need to turn SELinux completely off!) # semanage permissive ­a ifconfig_t 94

What’d I forget? Open Discussion