Red Hat Deep Dive Sessions: Linux on System z
A presentation at Chicago Area VM (and Linux) Enthusiasts (CAVMEN) in in Lincolnshire, IL, USA by Shawn Wells
Red Hat Deep Dive Sessions Linux on System z Shawn Wells (swells@redhat.com) W/W Lead Architect, Linux on System z Team Lead, System z SMEs 1
Introduction Shawn Wells (swells@redhat.com) Lead Architect, Linux on System z Team Lead, System z SMEs Phone: +1 443 534 0130 2
Agenda Scheduled questions to be answered in this session: What’s the Linux on System z development process? What’s in RHEL now? What’s on the roadmap? Provisioning & Patch Management in RHN Security Update SELinux, Audit, etc 3
Linux on System z Development Process
Linux on System z Development Community Development with “upstream” communities Kernel, glibc, etc Collaboration with partners, IBM, open source contributors 5
Linux on System z Development Fedora Bleeding Edge Sets direction for RHEL technologies Community Supported Released ~6mo cycles Fedora 8,9,10 = RHEL6 Fedora 8; http://fedoraproject.org/wiki/Releases/8/FeatureList Fedora 9; http://fedoraproject.org/wiki/Releases/9/FeatureList Fedora 10; http://fedoraproject.org/wiki/Releases/10/FeatureList 6
Linux on System z Development Red Hat Enterprise Linux Stable, mature, commercial product Extensive Q&A, performance testing Hardware & Software Certifications 7yr maintenance Core ABI compatibility guarantee Major releases 2-3yr cycle 7
Support Cycle Extended Product Lifecycle Years 1 - 4 Yr 5 Yr 6,7 Pro d Pro d Security Patches Bug Fixes Hardware Enablement Software Enhancements uc tio n1 uc tio Pro d n2 uc tio n3 X X X X X X Full Partial None X 8
Linux on System z Subscriptions No Upgrade Costs Red Hat Enterprise Linux Subscription No Client Access Fee Unlimited Support Incidents For System z: Priced Per IFL Unlimited VMs per IFL Customers can consolidate subscriptions to or from other platforms S U P P O R T PREMIUM STANDARD BASIC 24x7 Phone/Web 1 Hour SLAs Phone/Web 1-4 Business Hour SLA Web Support. 2 Day SLA Security, Bug Fixes Regular H/W & S/W Updates Hardware & Application Certifications Stable Application Interfaces Upgrades to New Versions Product Source & Binaries 9
Linux on System z Support C O N S U L T A N T S T E C H A C C N T M G R S Level 3: Special Engineering Custom Patches, Code Re-writes, Interim Patches, Application Redesign Level 2: Advanced Support Reproduce Problems, Grouped via Skillsets Level 1: Front Line Support Known Issues, Initial Troubleshooting, Everyone is minimum RHCE Support via Red Hat 10
Linux on System z Support C O N S U L T A N T S T E C H A C C N T M G R S Level 3: Special Engineering Custom Patches, Code Re-writes, Interim Patches, Application Redesign Level 2: Advanced Support Reproduce Problems, Grouped via Skillsets P A R T N E R Level 1: Front Line Support Known Issues, Initial Troubleshooting, Everyone is minimum RHCE Support via Red Hat Level 2: Advanced Support Reproduce Problems, Category Specialists Level 1: First Responders T A M Basic Support Support via IBM 11
What’s in RHEL now? What’s on the road map?
13
IBM Changes to 2.6.x Kernel RHEL4 RHEL5 14
RHEL Now: RHEL 5.2 Support for z10 Dynamic CHPID reconfiguration Improved “ssh -X” with VPN during installation process Better network performance with skb scatter-gather support Implementation of SCSI dump infrastructure 15
RHEL Now: RHEL 5.2 Accelerated inkernel Crypto Support for crypto algorithms of z10 SHA-512, SHA-384, AES-192, AES-256 Two OSA ports per CHPID; Four port exploitation Exploit next OSA adapter generation which offers two ports within one CHPID. The additional port number 1 can be specified with the qeth sysfs-attribute “portno” Support is available only for OSA-Express3 GbE SX and LX on z10, running in LPAR or z/VM guest (PFT for z/VM APAR VM64277 required!) 16
RHEL Now: RHEL 5.2 Large Page Support This adds hugetblfs support on System z, using both hardware large page support if available, and software large page emulation (with shared hugetblfs pagetables) on older hardware skb scatter-gather support for large incoming messages This avoids allocating big chunks of consecutive memory and should increase networking throughput in some situations for large incoming packets Full Release Notes At: redhat.com http://www.redhat.com/docs/enUS/Red_Hat_Enterprise_Linux/5.2/html/Release_Notes/s390x/index.html 17
RHEL Now: RHEL 5.2 Lightweight userspace priority inheritance (PI) support for futexes, useful for realtime applications (2.6.18) Assists priority inversion handling. Ref: http://lwn.net/Articles/178253/ High resolution timers (2.6.16) Provide fine resolution and accuracy depending on system configuration and capabilities - used for precise in-kernel timing New Pipe implementation (2.6.11) 30-90% perf improvement in pipe bandwidth Circular buffer allow more buffering rather than blocking writers “Big Kernel Semaphore”: Turns the Big Kernel Lock into a semaphore Latency reduction, by breaking up long lock hold times and adds voluntary preemption 18
RHEL Now: RHEL 5.2 Process Events Connector (2.6.15) Reports fork, exec, id change, and exit events for all processes to userspace Useful for accounting/auditing (e.g. ELSA), system activity monitoring, security, and resource management kexec & kdump (2.6.13) Provide new crash-dumping capability with reserved, memory-resident kernel Extended device mapper multipath support Address space randomization: Address randomization of multiple entities – including stack & mmap() region (used by shared libraries) (2.6.12; more complete implementation than in RHEL4) Greatly complicates and slows down hacker attacks Audit subsytem Support for process-context based filtering (2.6.17) More filter rule comparators (2.6.17) 19
RHEL Now: RHEL 5.2 Add nf_conntrack subsystem: (2.6.15) Common IPv4/IPv6 generic connection tracking subsystem Allows IPv6 to have a stateful firewall capability (not previously possible) Increased security Enables analysis of whole streams of packets, rather than only checking the headers of individual packets SELinux per-packet access controls Replaces old packet controls Add Secmark support to core networking Allows security subsystems to place security markings on network packets (2.6.18) 20
RHEL Tomorrow: RHEL 5.3 Currently in beta Interested in being a beta tester? NSS CPU Affinity ETR Support Device-multipath support for xDR RHT BugZilla: 184770 IBM LTC 18425-62140 21
RHEL Tomorrow: Fedora Fedora is Red Hat’s bleeding edge, an incubator for new technologies and features Fedora sets our direction for Red Hat Enterprise Linux, and gives you a good idea of what will be in our next RHEL release (… and in other Linux distros, too) Fedora 8; http://fedoraproject.org/wiki/Releases/8/FeatureList Fedora 9; http://fedoraproject.org/wiki/Releases/9/FeatureList Fedora 10; http://fedoraproject.org/wiki/Releases/10/FeatureList Fedora 8,9,10 = RHEL6 22
RHEL Tomorrow: “In Place” Upgrade Currently a beta feature in RHEL 5.3 “In Place” Upgrades: preupgrade Will download files needed to upgrade, Store them locally on disk Reboot you into the installer Not a true inplace upgrade (yet)! Benefit The longest part of an install is when packages are downloaded to the local machine PreUpgrade downloads and stores packages locally, while the machine is running/in production Reboot directly into the installer 23
RHEL Tomorrow: “In Place” Upgrade Select Target Version 24
RHEL Tomorrow: “In Place” Upgrade Determines which packages need upgrading, and downloads them 25
RHEL Tomorrow: “In Place” Upgrade Downloads new initrd & kernel images 26
RHEL Tomorrow: “In Place” Upgrade User reboots, brought into installer 27
RHEL Tomorrow: gnome-control-center gnomecontrolcenter It is not YaST (yet) It is a unified GUI for package management and system configuration Benefit Progress towards a YaST-like tool in RHEL (currently we have the systemconfig* GUIs/TUIs) 28
RHEL Tomorrow: PackageKit PackageKit Abstraction layer for YUM, apt, conary, etc Provides a common set of abstractions that can be used by GUI/TUI package managers rpm dpkg ipkg 29
RHEL Tomorrow: PackageKit PackageKit Abstraction layer for YUM, apt, conary, etc Provides a common set of abstractions that can be used by GUI/TUI package managers PackageKit rpm dpkg ipkg 30
RHEL Tomorrow: PackageKit PackageKit Abstraction layer for YUM, apt, conary, etc Provides a common set of abstractions that can be used by GUI/TUI package managers YaST RHN PackageKit rpm dpkg ipkg 31
Linux Virtualization on System z
JBoss Oracle Custom App Red Hat Enterprise Linux Red Hat Enterprise Linux Red Hat Enterprise Linux z/VM CONNECTIVITY PROCESSING DATA 33
Using RHN Satellite to Manage s390/s390x & distributed
Red Hat Network Satellite Update 35
Red Hat Network Satellite Update Manage 36
Red Hat Network Satellite Update Manage Provision 37
Red Hat Network Satellite Update Manage Provision Monitor 38
Red Hat Network Satellite 39
Red Hat Network Satellite 40
Red Hat Network Satellite 41
RHN Installation Requirements Software RHEL 4 (31-bit or 64-bit) @Base install Hardware 1 to 2 (virtual) IFLs 2 to 4 GB storage (memory) 1 GB swap (combination VDISK, disk) 1 x mod3 for OS install Estimated 12 GB disk space for embedded database 6 GB per channel (disk) 42
43
44
RHN Satellite Is Now Open Source http://spacewalk.redhat.com Announced at Red Hat Summit 2008 …. remember the Fedora -> RHEL model? 45
Security
Agenda Why do we need SELinux? What are the principal concepts? SELinux Details Type Enforcement What are the available policies? What’s a policy actually made of? How do I {add, change} a policy? What’s the associated overhead? Usage User Perspective Admin Perspective Scenarios Fixing the RHT Corporate VPN “update” 47
Name of Presentation Why do we need Red Hat Presenter SELinux?
Linux Access Control Problems Access is based off users’ access Example: Firefox can read SSH keys # ps -x | grep firefox shawn 21375 1 35 11:38 ? 00:00:01 firefox-bin # ls -l id_rsa -rw———- 1 shawn shawn 1743 2008-08-10 id_rsa Fundamental Problem: Security properties not specific enough. Kernel can’t distinguish applications from users. 49
Linux Access Control Problems 2) Processes can change security properties Example: Mail files are readable only by me….. but Thunderbird could make them world readable Fundamental Problems: Standard access control is discretionary Includes concept of “resource ownership” Processes can escape security policy 50
Linux Access Control Problems 3) Only two privilege levels: User & root Example: Apache gets hacked, allowing remote access to root. Entire system is compromised. Fundamental Problems: Simplistic security policy No way to enforce least-privilege 51
SELinux: Building Security Openly NSA Develops SELinux Integrated into Linux Kernel Enabled By DEFAULT In RHEL Integrated into Open Source Project Customers, NSA, Community, and Red Hat continue evolution 52
Red Hat Security Certifications NIAP/Common Criteria: The most evaluated operating system platform ● Red Hat Enterprise Linux 2.1 – EAL 2 (Completed: February 2004) ● Red Hat Enterprise Linux 3 EAL 3+/CAPP (Completed: August 2004) ● Red Hat Enterprise Linux 4 EAL 4+/CAPP (Completed: February 2006) ● Red Hat Enterprise Linux 5 EAL4+/CAPP/LSPP/RBAC (Completed: June 2007) DII-COE ● Red Hat Enterprise Linux 3 (Self-Certification Completed: October 2004) ● Red Hat Enterprise Linux: First Linux platform certified by DISA DCID 6/3 ● Currently PL3/PL4: ask about kickstarts. ● Often a component in PL5 systems DISA SRRs / STIGs ● Ask about kickstarts. FIPS 140-2 ● Red Hat / NSS Cryptography Libraries certified Level 2 53
Security Standards Work Extensible Configuration Checklist Description Format (XCCDF) Enumeration for configuration requirements DISA FSO committed to deploying STIG as XCCDF Others working with NIST Security policy becomes one file Open Vulnerability & Assessment Language (OVAL) Machine-readable versions of security advisories Common Vulnerability and Exposures (CVE) Compatibility Trace a vulnerability through multiple vendors 54
Name of Presentation How’s it work? Red Hat Presenter
Linux Access Control Introduction Linux access control involves the kernel controling Processes (running programs), which try to access… Resources (files, directories, sockets, etc) For example: Apache (process) can read web files But not the /etc/shadow file (resource) Traditional methods do not clearly separate the privileges of users and applications acting on the users behalf, increasing the damage that can be caused by application exploits. So, how should these decisions be made? 56
Security Architecture Every subject (i.e process) and object (i.e. data files) are assigned collections of security attributes, called a security context 1) Security context of subject & object passed to SELinux 2) Kernel/SELinux check, verify access 2a) Grant access. Record allowance in AVC (Access Vector Cache) 2b) Deny access, log error 57
Security Architecture Or in picture view… 58
Name of Presentation SELinux Details Red Hat Presenter
SELinux Contexts root:object r:sysadm home t:s0:c0 ● ● ● ● ● ● The above is an SELinux context user_t role_t file_t Sensitivity category 60
Role Based Access Control (RBAC) “root” really isn’t “root” i.e: root_u:WebServerAdmin_r:SysAdmin_t root_u:OracleDBAdmin_r:SysAdmin_t 61
SELinux Contexts 62
SELinux Policy ● ● Policies are matrices of statements which tell SELinux if certain actions are allowed based on the context of the objects attempting those actions. There are three SELinux Policy Types 63
The Three SELinux Policy Types Targeted Policy Default policy in RHEL5. Supported by HelpDesk. Targets specific applications to lock down. Allows all other applications to run in the unconfined domain (unconfined_t) Applications running in the unconfined domain run as if SELinux were disabled 64
The Three SELinux Policy Types 2) Strict Policy Denies access to everything by default Complete protection for all processes on the system Requires that policies be written for all applications, often requires customization Strict is type enforcement with added types for users (e.g. user_t and user_firefox_t). Not enabled by Red Hat as default 65
The Three SELinux Policy Types 3) Multi-Level Security (MLS) Focuses on confidentiality (i.e. separation of multiple classifications of data) Ability to manage {processes, users} with varying levels of access. (i.e. “the need to know”) Uses category & sensitivity levels 66
The Three SELinux Policy Types 3) Multi-Level Security (MLS) (a) Sensitivity Labels Mostly used by the government – Top Secret, Secret, Unclassified, etc s0 Unclassified s1 Secret s2 Top Secret 67
The Three SELinux Policy Types 3) Multi-Level Security (MLS) (b) Category Labels Separation of data types, compartments, projects, etc s0 s1 s1 Unclassified Secret Top Secret c0 Project A c0 Alpha c1 Project B c1 Bravo c2 Charlie c3 Delta 68
The Three SELinux Policy Types 3) Multi-Level Security (MLS) (b) Polyinstantiation & pam_namespace The pam_namespace PAM module sets up a private namespace for a session with polyinstantiated directories A polyinstantiated directory provides a different instance of itself based on user name, or when using SELinux, user name, security context or both 69
The Three SELinux Policy Types 3) Multi-Level Security (MLS) (b) Polyinstantiation & pam_namespace # id Z staff_u:WebServer_Admin_r:WebServer_Admin_t:s0:c0 # ls l /data secretfile1 secretfile 2 # id Z staff_u:WebServer_Admin_r:WebServer_Admin_t:s1:c0 # ls l /data secretfile1 secretfile 2 topsecretfile1 70
The Three SELinux Policy Types Multi-Level Security (MLS) & Common Criteria The Common Criteria (CC) is an international security standard against which systems are evaluated. Many government customers require CC evaluated systems. Red Hat Enterprise Linux 5 meets EAL4+ with RBAC/LSPP/CAPP endorcements 71
What’s the Performance Overhead? 72
What’s the Performance Overhead? 73
What’s the Performance Overhead? 74
What’s the Performance Overhead? ● Not official statistics ● Laptop = 2GHz, 2x 1GB RAM ● Workstation = 2.13GHz, 4x 1GB RAM ● Apache = Lots of threads ● MySQL = Lots of disk I/O 75
Name of Presentation SELinux Usage Red Hat Presenter (GUI & console)
End-User Perspective ● sealert Notifications 77
End-User Perspective ● sealert Browser 78
System Administrator Perspective ● sealert + EMail Notifications 79
System Administrator Perspective ● system-config-selinux 80
System Administrator Perspective ● sediffx 81
System Administrator Perspective ● apol 82
Name of Presentation SELinux Usage Red Hat Presenter (Hints & Tips)
System Administrator Perspective ● semanage Configure elements of SELinux policy without modification/recompilation of policy sources … . aka on the fly Example: Dynamically Allowing Apache to listen on port 1234 # semanage port a t httpd_port_t p tcp 1234 84
System Administrator Perspective ● semanage (more examples) Example: Allow shawn to join “webadmin_u” group # semanage login a s webadmin_u shawn Example: Relabel files for access by Apache # semanage fcontext a t \ httpd_sys_content_t “/data/webpages(/.*)?” 85
System Administrator Perspective ● semanage (most important example) You don’t need to disable SELinux to fix a single error! type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) The Fix: # semanage permissive a ifconfig_t 86
System Administrator Perspective ● audit2allow Allows generation of SELinux policy rules from logs of denied operations Example: Fix all the errors on the system (completely not a good idea on a real system)
cat /var/log/audit/audit.log | audit2allow M FixAll Generating type enforcment file: FixAll.te Compiling policy: checkmodule M m o FixAll.mod FixAll.te Building package: semodule_package o FixAll.pp m FixAll.mod
semodule i FixAll.pp
87
Name of Presentation Red Hat Presenter Scenarios
Scenario: Fixing the RHT corporate VPN “update” ● Red Hat has a Corporate Standard Build (CSB) for desktop environments ● Red Hat pushes updates to said CSB ● I “tweak” my configuration files ● When RHT pushed a CSB update, it broke my VPN settings 89
Scenario: Fixing the RHT corporate VPN “update” /var/log/messages: type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) Now what? 90
Scenario: Fixing the RHT corporate VPN “update” type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” gid=0 euid=0 suid=0 fsuid=0 egid=0 exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) What I Know: 1) 2) 3) AVC Event ID 738 syscall=54 (I’d have to google this) root (or an application on its behalf) was running /sbin/ip 4) context = user_u:system_r:ifconfig_t:s0 91
Scenario: Fixing the RHT corporate VPN “update” type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” gid=0 euid=0 suid=0 fsuid=0 egid=0 exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) My Options: 1) Create a SELinux Policy Module # ausearch x “/sbin/ip” | audit2allow M MyVPNFix 92
Scenario: Fixing the RHT corporate VPN “update” type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” gid=0 euid=0 suid=0 fsuid=0 egid=0 exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) My Options: 1) Create a SELinux Policy Module # ausearch x “/sbin/ip” | audit2allow M MyVPNFix # semodule i MyVPNFix.pp 93
Scenario: Fixing the RHT corporate VPN “update” type=SYSCALL msg=audit(1204719775.306:738): arch=40000003 syscall=54 success=no exit=19 a0=4 a1=8933 a2=bfcec1bc a3=bfcec1bc items=0 ppid=3900 pid=5003 auid=501 uid=0 sgid=0 fsgid=0 tty=(none) comm=”ip” gid=0 euid=0 suid=0 fsuid=0 egid=0 exe=”/sbin/ip” subj=user_u:system_r:ifconfig_t:s0 key=(null) My Options: 2) Disable enforcement of ifconfig_t (there is no need to turn SELinux completely off!) # semanage permissive a ifconfig_t 94
What’d I forget? Open Discussion
Invited to speak at the Chicago Area VM (and Linux) Enthusiats meetup. Led a discussion about the current Linux for System z development process, what s390x features were delivered in the latest releases of Linux, provisioning and patching via Red Hat Network Satellite, and several security updates (SELinux, audit, etc).
