Omaha Overview

  1. Omaha Overview
    1. Objective
    2. Background
    3. High Level Requirements
    4. Scenarios
        1. New Install
        2. Secondary Install
        3. Uninstall
    5. High Level Design
      1. Client
        1. Client Runtime
          1. Execution Model
          2. Application Registration
          3. Automatic Startup
          4. Scheduler
          5. Resource and Timing Constrains
        2. Web Browser Control
        3. Crash Reporting
      2. Meta-Installer
      3. Server
      4. Rollout Process
    6. Installation/Update Details
      1. End-User Installation
      2. Auto-Updates

Objective

The Omaha project provides a shared autoupdate and install system for Windows client products at Google that works on multiple Windows platforms, including Windows Vista. Goals include the following:


Non-goals include the following:


Background  

Google client products have the capability to update themselves to a newer version without end-user intervention. This capability is called "autoupdating."


Although our client products shared the autoupdate feature, they did not all share autoupdate code. Some had implemented their own autoupdate solutions, and others had taken advantage of the common autoupdater code and the common autoupdate server.


It was obviously desirable for products to avoid unnecessary duplication of code and for Google to avoid maintaining multiple servers that perform identical functionality. This reason alone was enough to consider unifying all client products under a single autoupdate solution. In addition, the evolution of Microsoft Windows made this unification almost mandatory. Windows Vista features a strict security model that limits the ability of most applications on a machine to perform system-changing activities, such as modifying the Windows registry, writing to the Program Files directory, or in some cases writing any persistent change to the system at all. Updating a program installation requires all these capabilities; thus, on Vista, most traditional autoupdating will fail.


Another problem caused by lack of shared code is that each autoupdate mechanism had its own subset of desired features. Most did not have a way to do canary experiments. Most did not have a way to have multiple update tracks. Most did not define a consistent versioning mechanism. Some of these features were not optional as user base size increased. By unifying the code we could deploy a rich set of autoupdate capabilities to all client applications.


Also, as the number of Google applications increased, it became more and more desirable to improve the overall install user experience. The browser typically prompted the user with a long series of techy, confusing and scary dialogs all trying to convince the user not to install. Then the user was prompted with a wizard filled with choices that they did not need to or know how to decide amongst. These factors combined to form a bad user experience and large drop-off during the app installation process.


To address the goals of Vista compatibility, reduction of duplicative effort at Google, improvement of deploy and autoupdate capabilities, and application install user experience improvement, we proposed to develop shared client infrastructure that handles all installation and autoupdating tasks for Google Windows client products. This client communicates with a single Google autoupdate server. Taken together, this server and client is named Google Update. The project code name is Omaha. 


High Level Requirements

Feature
Solve pushing server content and binary changes at the same time
Allow our partners to add their own versions of their product to Omaha without our intervention or a server push on our end
Allow our partners to be able to test new versions integrated with Omaha without making the URL live to the rest of the world
Be able to offer different versions of the same product, including beta software at different URLs
Continue to support at least as good one-click install experience as we have now
Offer better non-admin/limited user account support, i.e. downloading and running the product over offering a link
Be able to install product updates without the product running
Optionally be able to automatically run the product after the installation completes
Be able to show no UI vs. a small progress bar - e.g. Earth install vs. Offline Gmail
Provide decent proxy support, i.e. reliable downloads outside the browser
Optionally don't require interactive installers/uninstallers for Omaha so product teams can rely on our translations for them.
Offer a small stub executable
Make the Omaha UI super fast to load
Omaha to support localization in the top X languages 
Be able to save the installer for later use

 

Scenarios

Omaha consists of three principal parts: a client-side run time, a meta-installer (which simply wraps the client run time and either the application installer or a micro reference to the application), and a server. The client and server communicate with each other over HTTP or HTTPS. When a user wishes to install a new Google application, in actuality he or she merely downloads a meta-installer wrapping the selected application, which either installs the client or confirms that the currently-installed Omaha is up to date, and then exits after requesting that Omaha install the desired Google application. If that application is not already installed on the user's system, the Omaha installs it (using the wrapped installer or downloading the most recent version from the server). Omaha then periodically checks for updates to products installed on the system, and when appropriate, installs those updates. Omaha also provides elevated-privilege functionality to lower-privilege Google applications on the system.

New Install

User downloads a meta-installer from a Google website, going through standard browser file download steps. Once the user figures out how to launch the meta-installer, the meta-installer installs and runs the Omaha client. On Windows platforms, such as Windows Vista, with UAC, an elevation prompt is displayed when installing applications per-machine. The client then begins downloading and installing the app referenced by the micro reference.  

Secondary Install

Google website detects that Omaha client is installed, and instead of delivering a downloadable EXE or MSI, it vends an application micro-reference via an ActiveX (or equivalent) object. In this case, the user didn't have to go through any complicated download steps. They just clicked on a web link and the download starts. The rest of this scenario is the same as the initial install scenario.

Uninstall

Omaha uninstalls itself when there is no application registered with it for updates.

The overall sequence diagram is below:

 

  1. For organic installs, the user downloads an installer called AppNameSetup.exe, which is actually the Omaha meta installer plus a data tag containing the name and ID of the managed app (AppName, in this case)
  2. Execution of AppNameSetup.exe installs Omaha.  Omaha reads the data tag (to get the name of the app it's installing) and immediately contacts the Update Server to get the download location of the current version of AppName. The AppName installer is downloaded and executed on the user PC.  Omaha itself takes no part in the installation process; it merely executes the provided installer
  3. Omaha periodically (every 5 hours, as of this writing) checks for updates for all of its managed applications.  If an update is available, Omaha downloads the indicated binary and executes it

There is also support for offline installers (Omaha and the AppName installer bundled into the initial download) and partner distribution

High Level Design

Client

The Omaha client is a Windows application, installed by a custom Windows installer. It performs several functions:


The client is able to handle multiple downloads in parallel, multiple sequential installs, and it is able to resume or restart partial downloads.


The client is not a separate application in the sense that it can be uninstalled independently of Google applications. Instead, it is considered a required feature of applications. It will uninstall itself when the last Google app using it is uninstalled by the user.


In addition to specific reporting related to installation and update, Omaha includes a generic crash-reporting facility for Google applications. The Breakpad project's network library is used, and the client will handle local minidump storage and queuing of reports.

 

The Omaha client consists of two major parts:

  1. Runtime, which handles all necessary update and install services
  2. Web browser control (ActiveX for Internet Explorer and NPAPI for other browsers), which allows Google web pages and Google applications to ask the run time to do tasks on their behalf

Client Runtime

The runtime has the following functionality:

Execution Model

The runtime functionality is divided between a couple of processes, most notable a core process and many worker process. There are a few reasons for the separation: robustness, running code with the least privilege, executing Google installers in the context of the logged in users as much as possible, and avoiding resource leaks and resource consumption in long-lived processes.


The execution model for the runtime code depends on how Omaha is actually installed. There are two main scenarios here:

  1. Omaha is installed for the machine. This requires the user has administrator privileges. In this case, a Windows service runs all the time as SYSTEM and, at the right time, it kicks off worker processes to do the bulk of the work. To update per-machine applications, the worker process runs as SYSTEM as well. The worker process terminates when there is no more work to do.
  2. Omaha is installed installs for the current user. Consequently, there is no service and no machine updates are possible. In this case, the execution model solely relies on a goopdate worker process that runs all the time in each interactive session as the user.

It is important the same binary works the same way on different operating systems: Windows 2000, XP, and Vista. The process execution model is almost identical, in both machine and user cases. For the most part, the machine and user instances of Omaha run in isolated security contexts and they share nothing.

The Omaha core runs in a local system process for the machine, and one process for each logged in user. Typically, there will be two core processes running. There is potential for an optimization here, as the local system process can create a process that runs as the logged in user when it needs to.


The machine Omaha core process is started at boot time by a system service that terminates soon after. There is a reason for not running the Omaha core in a service: a bug in XP prevents us from further updating the service if the service has crashed. More investigation will have to be done here, especially since having a service is quite nice and eliminates the need for another process. The user Omaha core process is started by the shell from the user's run key. To eliminate a single point of failure, there are fallbacks implemented using the Windows system scheduler to attempt to start the core process if for some reason it is not running. It should be desirable to measure the overall availability of the core process.  We strive to be as available as the Windows OS is in terms of uptime of the Omaha core.


The Omaha core is highly reliable and lightweight:


The Omaha worker processes do most of the actual work:


Application Registration

In order for an application to receive updates, the application must register with Omaha. The application registration mechanism is based on Windows registry. There are primarily two update policies supported by Omaha, which directly affect how an Omaha aware application registers with the Omaha runtime:

  1. per-machine updates
  2. per-user updates


In the case of per-machine updates, each application installed per-machine creates a key HKLM/Software/Google/Update/Clients/{GUID}. Under this key, the application creates the following values:


In the case of per-user updates, the only difference is the location to where the key is created. In this case, the key is HKCU/Software/Google/Update/Clients/{GUID}.


When the update loop kicks in, Omaha runs the per-machine updates as SYSTEM and per-user updates as the logged in user. This assumes that the user has an active session, in other words, updates for per-user installed applications will not run if the user has not logged in. As long as there is anything under these keys, Omaha keeps pinging for updates and updating these applications. It is possible that the application is dead and its registration with Omaha still active, for example, in the case where the user just deleted the application files. Unfortunately, Omaha does not detect such conditions and it is going to download and apply updates for it until the application cleanly unregisters.


Exposing registry keys and low level implementation details to applications has disadvantages. It breaks the encapsulation of Omaha. An argument can be made that such implementation details would be better to be abstracted out in a code library that the application code links with. However, there is a great advantage for the current registration mechanism. Registry keys can be created easily directly from the installer script for applications that are using off the shelf installer technology thus making the integration with Omaha easier.

Automatic Startup

The service is set to auto run when Omaha is installed per-machine. The SCM failover mechanism is used to restart the service in case of a crash. When Omaha is installed per-user, the goopdate worker process is started by the Windows shell from HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run.

Scheduler

The client runtime runs periodic tasks, such as the update check request to the server to discover updates. These tasks are kicked off by a scheduler. The scheduler code runs inside the Omaha core process. The main requirement for the scheduler is to execute a worker process once in a while.


A lot of consideration was given to using the built in Windows scheduler instead of building our own. The windows task scheduler on Vista seems to be greatly improved. On XP and Windows 2003 there are some limitations to the windows task scheduler:

  1. Credentials are an issue with the pre-Vista scheduler. The issue is that the scheduler used to cache the credentials for the user that registered a scheduled event and if the user password changed, then the task was broken. It has been fixed in Vista.
  2. On Windows 2003 the use of the task scheduler service is restricted to Administrators only.
  3. If the machine is in a sleep state there are some hardware on which the task will not be scheduled. There are situations in which the scheduled tasks fail because of suspected ACPI interactions. One extreme case was a machine not able to schedule tasks when the LAN cable was disconnected.

Resource and Timing Constrains

The runtime follows a few constraints regarding resource usage and timing.

Limiting resource use:


Timing:

Web Browser Control  

The web browser control exposes a high level interface to Google web pages and client applications, enabling features such as one-click web installs and rich update clients such as Pack. It needs to limit access so that only authorized Google web pages and applications have access to the services offered by the client run time. The interface is defined by very coarse-granularity functions. Rather than "download file X from server Y" it would instead offer "download application id X from Google". In the event that the client had a security hole, it's important that the interface limit the amount of damage that can be done.

Crash Reporting

There are two major parts to crash reporting on Windows: generating the minidump and reporting it to the server.  With Omaha, each application would still be responsible for catching exceptions and signaling Omaha to generate and transmit the minidump to Google servers. The minidump is created in a secure directory which Omaha can write to.  It has code to try to minimize possible accidental self-DDOS as a result of frequent crashes while still reporting important data, perhaps determining when a duplicate minidump was generated, or reporting the number of crashes that have been generated but not reported.


Meta-Installer

The Omaha meta-installer is a simple wrapper that includes the client as well as information necessary to install one or more Google applications. It performs the following functions:


Our goal is that the minimal installer will be less than 172KB (which takes less than one minute to download over a 28.8kbps link at 10 wire bits per 8 data bits, representing what's likely the slowest modem still in operation today). The current size of the matainstaller is larger than that. We are looking for ways to keep the download size down. 

Server

The Google Update server is not part of the Omaha open source project. Providing updates for applications requires a server that implements the Omaha Server Protocol.

Omaha has two server components: autoupdate and download. This separation of server responsibilities is desirable because the requirements are different: downloading large binaries takes high bandwidth and can tolerate relatively high latency, whereas small transactions such as update pings may be much more frequent but require extremely low bandwidth per transaction. These differences allow Google to provision the two servers appropriately.


The autoupdate server provides the following functionality:


The download server's primary job is to host binaries.  In order to support private betas and other restricted software updates, the download server can restrict access to certain downloads.

Rollout Process

To the greatest extent possible, teams will be able to reconfigure their area of the update server without affecting other teams.


Installation/Update Details



End-User Installation

  1. When a user clicks "download", the website provides tagging parameters to the dl server that tell it how to customize the untagged metainstaller before sending it to the user.
    1. The custom information includes the following (plus more):
    2. OneClick and ClickOnce installs work similarly except this information is provided on the command line.
  2. The user receives AppNameSetup.exe, which has been customized for this application and it is specific to this install (language, iid, etc.).
  3. When the user runs AppNameSetup.exe
    1. The metainstaller:
      1. Extracts all the Omaha files to a temporary directory.
      2. Reads the tag
      3. Launches the extracted GoogleUpdate.exe with the /install parameter and the contents of the tag
    2. The temporary instance of GoogleUpdate.exe:
      1. Determines whether to install per-user or per-machine
      2. Requests elevation (UAC prompt) if necessary
      3. Stops any existing instances of GoogleUpdate.exe
      4. Installs Omaha
      5. Launches the installed instance of Omaha to complete installation of the application.
    3. The installed instance of GoogleUpdate.exe:
      1. Displays a progress UI
      2. Performs an update check for the application specified in the tag.
        1. The server uses the update config rules to determine the response. Only rules for version 0.0.0.0 are used for installs
      3. Downloads the app installer specified in the update response
      4. Runs the app installer
      5. Displays any error or success unless configured to exit silently

Auto-Updates

All of this is completely silent.
  1. The core is always running. This will change in the future. 
    1. The core runs as the current user for per-user applications and as Local System for per-machine applications
    2. The per-machine instance runs even if no users are logged in
    3. There may be as many as one instance per logged-in user plus one instance for the system
  2. The Omaha core periodically launches an update worker
  3. The update worker:
    1. Checks whether it should perform an update check
    2. If so:
      1. Finds all applications registered in the Clients key
      2. Sends and update check request containing all applications
        1. The server uses the update config rules to determine the response based on the installed version
      3. Performs updates (download and install) for any application that needs to be updated as specified in the update response
        1. If Omaha itself needs to be updated, no other applications are updated during that update check