Severity Levels

The first step in any incident response process is to determine what actually constitutes an incident. Incidents can then be classified by severity, usually done by using "P" (Priority) definitions, with lower numbered severities being more urgent. Operational issues can be classified at one of these severity levels, and in general you are able to take more risky moves to resolve a higher severity issue. Anything above a P2 is automatically considered a "major incident" and gets a more intensive response than a normal incident.

Try to assess; if you cannot, assume the worst

Your first action as a response team is to identify the state of things. If we are leaking data, that means classifying what's being leaked. If the site is down, that means identifying what portions of our site are impacted as that defines which level the incident is. If you cannot do this assessment very quickly, assume the worst. We can work on how to make assessment easier in the post mortem process, but we cannot retroactively respond more quickly or with more regard.

Can a P2 be a major incident?

All P1's are major incidents, but not all major incidents need to be P1's. If you require co-ordinated response, even for lower severity issues, then trigger our incident response process. The IC can make a determination on whether full incident response is necessary.

Severity Description Typical Response
P0

Critical issue that warrants liaison with executives.

  • Web app is unavailable or experiencing severe performance degradation for most/all users.
  • Active exploitation of a leak of data classified at Customer Confidential or higher.
  • Any leak of data classified as RealSelf Restricted.
  • Any leak of data classified as Customer Sensitive.
  • Attacker with internal access (e.g., has exploited a container or host and is now using it to attack other systems).
  • Any bugs with "broad or significant" impact on products providers pay for. (See also.)

Major incident response.

  • Page an IC in Slack with emergencypage. (Note: there is no exclamation point to begin this command.)
  • See During an Incident.
  • Notify internal stakeholders.
  • Work on issue as your top priority, displacing all other work.
  • If there is a site outage (partial or full), the Incident Commander should update StatusPage.io.
P1

Critical system issue actively impacting many customers' ability to use the product.

  • Web app is unavailable or experiencing severe performance degradation for most/all users.
  • Internal credential leaks (like TLS keys, service account passwords).
  • Any leak of data classified as Customer Sensitive.
  • Any bugs in products providers pay for, if not right in any way even one doctor would notice. (See also.)
  • Any other event to which a RealSelf employee deems necessary of incident response.

Major incident response.

  • Page an IC in Slack with emergencypage. (Note: there is no exclamation point to begin this command.)
  • See During an Incident.
  • Work on issue as your top priority, displacing all other work (other than an active P0 incident); however, out-of-hours work is not needed.
  • If there is a site outage (partial or full), the Incident Commander should update StatusPage.io.
Anything above this line is considered a "Major Incident". Our incident response process should be triggered for any major incidents.
P2

Stability or minor customer-impacting issues that require immediate attention from service owners.

  • Partial loss of functionality, not affecting majority of customers.
  • Something that has the likelihood of becoming a P1 if nothing is done.
  • No redundancy in a service (failure of 1 more node will cause outage).
  • Spikes of HTTP 500-class errors.
  • Any leak of data classified as Customer Confidential.
  • Any leak of data classified as Customer Unpublished.
  • Any leak of data classified as RealSelf Internal not listed in P1.

High-urgency ticketed work.

  • Liaise with engineers of affected systems to identify cause.
  • If related to recent deployment, roll back.
  • Monitor status and notice if/when it escalates.
  • Mention on Slack if you think it has the potential to escalate.
  • Trigger incident response if necessary (emergencypage).
  • SLA for tickets at this level: ten business days. Delays beyond this time should be communicated, along with a timeline for remediation, to engineering leadership (along with security, for security tickets) for approval.
P3

Minor issues requiring action, but not affecting customer ability to use the product.

  • Performance issues (delays, etc).
  • Individual host failure (i.e. one node out of a cluster).
  • Delayed job failure (not impacting event & notification pipeline).
  • Cron failure (not impacting event & notification pipeline).
  • Failures of the test environment, unless those failures block higher-priority work.

Medium-urgency ticketed work.

  • Monitor status and notice if/when it escalates.
  • SLA for tickets at this level: one month. Delays beyond this time should be communicated, along with a timeline for remediation, to engineering leadership (along with security, for security tickets) for approval.
P4

Cosmetic issues or bugs, not affecting customer ability to use the product.

  • Bugs not impacting the immediate ability to use the system.

Standard ticket.

  • Create a JIRA ticket and assign to owner of affected system.