I received my Ph.D. in Computer Science from UC San Diego in 2015 under the guidance of my advisors George Porter and Amin Vahdat. I received my M.S. in Computer Science from UC San Diego in 2012 and my B.S. with Honors in Computer Science from Caltech in 2009. My research interests include big data, I/O-intensive computation, distributed systems, cloud computing, MapReduce, data centers, and high-speed sorting.

My doctoral dissertation, titled "Achieving Efficient I/O with High-Performance Data Center Technologies," focuses on the performance of storage and network I/O in large-scale distributed systems. It describes two systems, TritonSort and Themis, that achieve high levels of performance for I/O-intensive applications, with sorting as the canonical example and kernel of many of these applications. It demonstrates how to run such applications on a wide variety of hardware platforms, including disks, SSDs, PCIe flash, 10 Gb/s networks, 40 Gb/s networks, and HPC supercomputers. It also includes an extensive evaluation of Amazon Web Services using Themis, and also explores Google Compute Engine to assess the viability of running I/O-intensive applications in the public cloud.

I am currently a software engineer at Google.

Efficient I/O-Intensive Computation in the Cloud

The public cloud has been touted as a solution for effectively running arbitrary computation at scale. In order to assess the viability of the public cloud's ability to run efficient I/O-intensive computation, and also to assess our upgrades to Themis at a larger-scale, we measured the performance of Themis on a wide variety of public cloud VM configurations. We extensively measured the performance of Themis on Amazon EC2 and Amazon EBS and explored its performance on Google Compute Engine. As a result of this work, we set three new world records in high speed sorting: 2014 Daytona GraySort, 2014 Indy CloudSort, and 2014 Daytona CloudSort. This work was published in SoCC 2015.

Next Generation Cluster Technologies

TritonSort and Themis were originally designed and evaluated on a 2009-era cluster consisting of disks and 10 Gb/s networking. While these technologies are still commonplace, newer technologies with higher levels of performance are rapidly appearing in the data center. Flash in particular appears to be eclipsing disk as a popular storage media, as is evidenced by the storage offerings within the recent Amazon EC2 instance types, as well as Google Compute Engine. Consequently, we upgraded Themis to perform efficiently on newer cluster technologies. These technologies include traditional SATA flash SSDs, high-performance PCI-Express flash devices, 40 Gb/s Ethernet, and HPC supercomputers using high performance interconnects such as Infiniband.

Themis

Themis is a MapReduce implementation build around TritonSort. Themis is I/O-efficient in that achieves the minimum number of I/Os possible (2) when the amount of data greatly exceeds the amount of physical memory. Themis has been evaluated on a variety of common MapReduce jobs and performs at roughly the same record-breaking speed as its predecessor, TritonSort. Themis was published in SoCC 2012.

TritonSort

TritonSort is the world's fastest sorting system. TritonSort achieves record speeds by focusing on per-disk and per-node efficiency. TritonSort aims to sort data at the speed of the disks by keeping all disks constantly reading or writing data in large contiguous chunks. TritonSort set world records in the 2010 and 2011 Sortbenchmark.org competitions. We set a total of seven world records: 2010 Indy GraySort, 2010 Indy MinuteSort, 2011 Daytona GraySort, 2011 Indy GraySort, 2011 Indy MinuteSort, 2011 Daytona 100TB JouleSort, 2011 Indy 100TB JouleSort. Two of these are current world records. TritonSort was published in NSDI 2011.

Other Work

During summer of 2009, I investigated resource imbalance within the MapReduce framework of Hadoop. Goals consisted of analyzing cluster resources during a MapReduce job, identifying bottlenecks, and classifying various types of jobs according to these bottlenecks with the hope of being able to utilize cluster resources more efficiently.

I held a Software Engineering Intern position at Google during the summer of 2011 working in the MapReduce group with my mentor Marian Dvorsky. As part of this internship I sorted 10PB with MapReduce.

I held a Software Engineering Intern position at Google during the summer of 2010 working in the search infrastructure group with my mentor Alexander Yip.

Sorting 100 TB on Google Compute Engine. Michael Conley, Amin Vahdat and George Porter, Computer Science & Engineering, University of California, San Diego, La Jolla, CA, Technical Report CS2015-1013, September 2015. Link to paper.

Achieving Cost-efficient, Data-intensive Computing in the Cloud. Michael Conley, Amin Vahdat, and George Porter, Proceedings of the 6th ACM Symposium on Cloud Computing (SoCC), Kohala Coast, HI, August 2015. Link to paper.

TritonSort: A Balanced and Energy-Efficient Large-Scale Sorting System. Alexander Rasmussen, George Porter, Michael Conley, Harsha V. Madhyastha, Radhika Niranjan Mysore, Alexander Pucher, and Amin Vahdat, ACM Transactions on Computing Systems (TOCS), Volume 31 Issue 1, February 2013. Link to paper.

Themis: An I/O-Efficient MapReduce. Alexander Rasmussen, Michael Conley, Rishi Kapoor, Vinh The Lam, George Porter, and Amin Vahdat, Proceedings of the 3rd ACM Symposium on Cloud Computing (SoCC), San Jose, CA, October 2012. Link to paper.

TritonSort: A Balanced Large-Scale Sorting System. Alexander Rasmussen, George Porter, Michael Conley, Harsha V. Madhyastha, Radhika Niranjan Mysore, Alexander Pucher, and Amin Vahdat, Proceedings of the 8th ACM/USENIX Symposium on Networked Systems Design and Implementation (NSDI), Boston, MA, March 2011. Link to paper.

SortBenchmark 2014: After a two-year hiatus, we once again delivered record-breaking sort performance with TritonSort. We sorted 100TB of data (with replication) in 1378 seconds, which landed a tie for the Daytona GraySort record. We also had two outright wins in the Indy and Daytona CloudSort categories. CloudSort is a new benchmark designed to measure the efficacy of using the public cloud for large-scale data processing. We also entered in both the Indy GraySort and Indy MinuteSort categories, but did not have the best entries in either of them. Surprisingly, the winner of these categories ran an implementation of TritonSort inspired by our NSDI 2011 paper, further validating our design choices. More details can be found here and in our submission document.

SortBenchmark 2011: TritonSort competed again in the Sort Benchmark competition in April 2011. This year we set 5 records (all wins, no ties): Indy 100TB GraySort, Daytona 100TB GraySort, Indy 60 second MinuteSort, Indy 100TB Joulesort, Daytona 100TB Joulesort. We built a general purpose MapReduce implementation with an initial sampling phase that enabled us to take the Daytona (general purpose) records. The SortBenchmark rules were updated this year to include a 100TB JouleSort category, which measures records/joule as the performance metric. As it turns out, focusing on cluster balance and resource utilization appears to be a viable means for power efficient computing. TritonSort's JouleSort performance is less than an order of magnitude off of the most efficient system in any of the other JouleSort categories. Keep in mind that energy efficiency was not a primary goal of the TritonSort project. You can read our submission document here.

SortBenchmark 2010: TritonSort, the world's fastest cluster based disk-to-disk sorting system, which we submitted to the Sort Benchmark competition in May 2010. TritonSort placed first in the Indy MinuteSort 60 second category and tied for first in the Indy GraySort 100TB category. TritonSort focuses on per-node efficiency and aims to achieve sorting throughput as close as possible to the sequential speed of the disks. See the SortBenchmark submission document here.

Guest lecture for CSE 124: Networked Services. Winter 2015.

Teaching assistant for CSE 120: Principles of Operating Systems. Summer Session 1 2012.

I compete in powerlifting in the USAPL 74kg raw open division. My current competition total is 450kg, or 992 lbs. I competed at the 2015 California State Games and the 2015 California State Championship, and I will be competing at the 2015 California Fall Classic in November.