Trends & Observations in Open Source Cloud Computing
A presentation at In-Q-Tel Technology Focus Day in in Arlington, VA, USA by Shawn Wells
Trends & Observations in Open Source Cloud Computing IQT Technology Focus Day August 10, 2011 | Arlington, Virginia Shawn Wells Technical Director, U.S Intelligence Programs sdw@redhat.com / 443.534.0130 1
Cloud Jeopardy! ? ? ? ? ? ? Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A 2 http://www.thetechtrendz.com/2010/07/top-5-cloud-computing-providers.html
Cloud Jeopardy! SalesForce ? ? ? ? ? Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A 3 http://www.thetechtrendz.com/2010/07/top-5-cloud-computing-providers.html
Cloud Jeopardy! SalesForce Amazon ? ? ? ? Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A 4 http://www.thetechtrendz.com/2010/07/top-5-cloud-computing-providers.html
Cloud Jeopardy! SalesForce Amazon Rackspace ? ? ? Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A 5 http://www.thetechtrendz.com/2010/07/top-5-cloud-computing-providers.html
Cloud Jeopardy! SalesForce Amazon Rackspace Google ? ? Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A 6 http://www.thetechtrendz.com/2010/07/top-5-cloud-computing-providers.html
Cloud Jeopardy! SalesForce Amazon Rackspace Google Facebook ? Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A 7 http://www.thetechtrendz.com/2010/07/top-5-cloud-computing-providers.html
Cloud Jeopardy! SalesForce Amazon Rackspace Google Facebook Microsoft Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A 8 http://www.thetechtrendz.com/2010/07/top-5-cloud-computing-providers.html
Cloud Jeopardy! SalesForce Amazon Rackspace Google Facebook Microsoft Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A We’re using commodity hardware running RHEL and it has been phenomenal It’s 10x less expensive running RHEL and orders of magnitude faster. - Parker Harris, Executive VP of Technology at SF.com 9
Cloud Jeopardy! SalesForce Amazon Rackspace Google Facebook Microsoft Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A Amazon cut technology expenses by about 25% from $71M to $54M. The reduction was attributed primarily to Amazon’s migration to Linux. - http://news.cnet.com/2100-1001-275155.html 10
Cloud Jeopardy! SalesForce Amazon Rackspace Google Facebook Microsoft Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A “VMWare has not built something that can scale up to Amazon level scalability of hundreds of thousands to millions of servers.” So, Rackspace went open source. - Jim Curry, VP of Corporate Developoment at Rackspace 11
Cloud Jeopardy! SalesForce Amazon Rackspace Google Facebook Microsoft Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A “Google announced last week that its cloud will run nothing but open source software.” - http://www.zdnet.com/blog/open-source/google-goesall-in-with-an-open-source-cloud/5334 12
Cloud Jeopardy! SalesForce Amazon Rackspace Google Facebook Microsoft Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A “Facebook was built from the ground up on open source software. - https://developers.facebook.com/opensource/ 13
Cloud Jeopardy! SalesForce Amazon Rackspace Google Facebook Microsoft Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A Microsoft is proprietary, but note how they follow the open source clouds with things like Daytona (Hadoop replacement).
- http://research.microsoft.com/en-us/news/headlines/daytona071811.aspx 14
How Does Open Source Compete? 15
Employees 9,000 8,000 7,000 6,000 5,000 4,000 3,000 2,000 1,000 0 Red Hat VMWare Organization 16
25,000 Employees 20,000 15,000 10,000 5,000 0 Red Hat VMWare Organization 17 Google
90,000 80,000 Employees 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0 Red Hat VMWare Google Microsoft Organization 18
120000 Employees 100000 80000 60000 40000 20000 0 Red Hat VMWare Google Microsoft Oracle Organization 19
400,000 350,000 Employees 300,000 250,000 200,000 150,000 100,000 50,000 0 Red Hat VMWare Google Microsoft Oracle Organization 20 IBM
Employees 2,000,000 1,800,000 1,600,000 1,400,000 1,200,000 1,000,000 800,000 600,000 400,000 200,000 0 VMWare Microsoft Red Hat Google Oracle Organization 21 IBM SourceForge
Open Source Development Models Mitigate Risk Coverity has tracked open source code quality since 2004. They have found propriatary software, on average, has 2030,000 defects per million lines of code. This has been true since 1960. ● ● ● ● 22 2004: Linux has 172 defects per MLOC (0.02%) 2005: Linux grew 4.7%, but defect density dropped to 133 defects per MLOC (0.01%) 2006: DHS funds study, adds LAMP stack, finds 0.01% defect density (even though 325,000+ lines of code added) 2009: Coverity now covers 280 projects, finds average defect density of 111 defects per MLOC (proprietary has 20,000 to 30,000)
23
24
Evolution of Warfighter Applications 25
Evolution of Warfighter Applications 26
Evolution of Warfighter Applications 27
Evolution of Warfighter Applications 28
Evolution of Warfighter Applications 29
Evolution of Warfighter Applications 30
Evolution of Warfighter Applications 31
Evolution of Warfighter Applications 32
KVM SPECvirt Scalability & Performance Leading Propriatary SPECvirt2010 33
34
35 ● Capable of 1B+ messages 14 minutes ● JPMorgan only sends ~1B AMQP messages per day
36
Included Security Controls 37 ● Type 1 vs Type 2 (classified as Type 1 by the IC) ● Common Criteria (CAPP, RBAC, LSPP) ● sVirt ● Certifiable Linux Integration Project (CLIP) ● DCID 6/3 PL4 ● NSS 1253 ● DOD 8500.2 ● DISA STIG V5R1 ● http://oss.tresys.com/projects/clip
What’s the Current Problem? 38
39
Red Hat Cloud Foundations 40
ADMINS BUILD PRIVATE PAAS 41
42
43
44
45
DEVELOPERS USE PRIVATE PAAS 46
47
48
49
50
WHO’S DOING THIS? 51
Cloud Jeopardy! SalesForce Amazon Rackspace Google Facebook Microsoft Cloud Type SaaS IaaS IaaS SaaS / PaaS SaaS SaaS / PaaS Customers 55k Corp 1.5m Pers 100k+ 50k+ 1m+ 750m+ 10k+ Cloud Revenue $1.5B $500M $56M $50M N/A N/A Open Source Providers 52
RED HAT CLOUD CUSTOMERS “Our decade-long partnership with Red Hat has always been focused on customer value and innovation. Today, we are extending this partnership to include cloud computing – broadening our reach and answering the strong customer demand for cloud computing services, Red Hat Enterprise Virtualization is an optimal hypervisor technology for the infrastructure offerings on the IBM cloud.” - Maria Azua, VP Cloud Computing Enablement at IBM 53
RED HAT CLOUD CUSTOMERS “Red Hat gets it. In kernel virtualization, virtualization management, the application orchestration and cloud abstraction layers, Red Hat gets it. Red Hat is giving customers the opportunity to innovate [in different ways] and leverage them where it makes sense.” - Derek Chan, Digital Operations, DreamWorks “At DreamWorks Animation, cutting-edge technology is a key enabler to our creative ambition. Red Hat’s Cloud Foundations provides a compelling platform to enable our cloud strategy giving us the ability to quickly scale our compute needs to meet the demands of our productions.” — Ed Leonard, DreamWorks Animation’s CTO 54
What About the Data? 55
Structured Data Services Physical Layer (PL) CSDS_PL GDSS CSDS 56 DMDC CFDB JOPES Classic FLIS NGA IDE/AV GSORTS GTN JOPES 4.0
Structured Data Services Virtual Base Layer (VBL) CSDS_VBL Physical Layer (PL) CSDS_PL GDSS CSDS 57 DMDC CFDB JOPES Classic FLIS NGA IDE/AV GSORTS GTN JOPES 4.0
Structured Data Services Private Data and Metadata Virtual Mid Layer (VML) Plans_VML Facilities_VML Material_VML Virtual Base Layer (VBL) CSDS_VBL Physical Layer (PL) CSDS_PL GDSS CSDS 58 DMDC CFDB JOPES Classic FLIS NGA IDE/AV GSORTS GTN JOPES 4.0
Structured Data Services Public Data Virtual Query Layer (VQL) (Exposed Views) Material_VQL Plans_VQL Facilities_VQL Private Data and Metadata Virtual Mid Layer (VML) Plans_VML Facilities_VML Material_VML Virtual Base Layer (VBL) CSDS_VBL Physical Layer (PL) CSDS_PL GDSS CSDS 59 DMDC CFDB JOPES Classic FLIS NGA IDE/AV GSORTS GTN JOPES 4.0
DISA ADNET: Anti-Drug Network Challenge Counter-narcotics and counter-narcoterrorism Statutory detection and monitoring Data is heterogenous & on multiple systems BI tools, Portal, Federated Search Solution MetaMatrix provides an abstracted view across multiple State/Local Law enforcement agencies. The virtual Database enables BI tools to get a complete picture of a “person of interest” from any history, warrants, jail, crimes, vehicles, etc… Also, MM is used as a federated search layer looking for possible persons of interest given general details (cars, addresses, license, aliases, etc…) Benefit 60 Enable ADNET to deliver on its mission Data Services Platform Data Service Disparate, heterogenous State/Local databases
IC Persons of Interest Challenge Need to find Person of Interest among disparate systems Adherence/mapping to common schema Data integration for SOA enablement Portal, ESB, Federated Search Solution Created abstracted view of a Enterprise Schema that is focused on Master Data Entities (Domains) like Person, Organizations, etc. Provide data services layer of the SOA stack, feeding the ESB ESB facilitates sync/async capabilities and provides integrated enterprise data efficiently and rapidly to multiple consumers Data Services Platform Data Service Benefit 61 Simplified data access and decoupled services and apps from the underlying complex data infrastructure Single view of data enables migration of external sources into the Enterprise repository seamlessly and without application impact Disparate, heterogenous data sources with varying schemas/representations
How Can the Government (and the IC) Get Involved? 62
Efforts at NGA 63 ● NGA Project Executive: Bert Beaulieu (InnoVision) ● NGA Project Manager: Michael W. Cline
Invited by In-Q-Tel, the venture capital firm of the U.S. Intelligence Community, to speak at their Technology Focus Day to discuss how the open source movement competes against proprietary companies in the cloud space.
