Incident and Problem Management
Structured incident handling with P1 response in 30 minutes, major incident coordination and problem management that removes recurring causes for good.
Based in Boksburg and supporting organisations across Gauteng: IT support in Johannesburg and IT support on the East Rand.
Discuss this managed service →The distinction that decides whether your ticket volume falls
An incident is an unplanned interruption or a reduction in quality of a service. The objective is to restore service as quickly as possible, using a workaround if that is faster than a fix.
A problem is the underlying cause of one or more incidents. The objective is to find it and remove it.
They are different activities with different objectives and different time pressures, and they get collapsed into one because both arrive as tickets.
The consequence of collapsing them is predictable. An organisation resolves the same incident every month for two years. Each individual resolution is efficient. Nobody ever asks why it keeps happening, because closing the ticket is what the process rewards.
The test. Look at your last six months of tickets and count how many are the same thing recurring. In most environments the answer is between a quarter and a third of total volume. That proportion is what problem management removes, and it is the reason ticket volume should fall over the first two quarters of a properly run arrangement rather than staying flat.
Incident management
Severity, assigned on business impact
Severity is not a property of the system. It is a property of what the failure stops. A print queue that halts warehouse dispatch is a P1. A file server alert affecting nobody until Monday is not.
| Severity | Definition | Response target | Hours |
|---|---|---|---|
| P1 | Business stopped. A site, a core system or a production service is unavailable, or a security incident is in progress. | 30 minutes | Business hours and after hours through the standby escalation |
| P2 | Major degradation. Multiple users affected and a workaround exists. | 4 business hours | Business hours |
| P3 | A single user is blocked, or a service request. | Next business day | Business hours |
| Non-contract work | No support agreement in place. | Next business day, scheduled | Business hours |
Response means an engineer who can work the problem has made contact and started. Not an automated acknowledgement. This is worth confirming with any provider, because a fifteen-minute response time that measures the arrival of a system-generated email is not a response time.
Business hours are 08:00 to 17:00, Monday to Friday, excluding South African public holidays. Outside those hours a standby rota carries P1 only. P2 and P3 raised after hours are actioned from the next business morning. Automated monitoring runs continuously.
Severity is agreed, not imposed. During onboarding we work through your systems and processes and write down what P1 means in your environment specifically. A generic severity matrix produces arguments during incidents, which is the worst possible time to be having them.
The incident lifecycle
Detect
From monitoring, from a user or from a third party. Monitoring detection is faster and it is the reason the two services work together.
Log and categorise
Every incident recorded, including the four-minute ones. The small tickets are where the pattern lives, and an environment where trivial issues get fixed without a record is one where problem management has nothing to work from.
Assign severity
Against the agreed definitions.
Investigate and diagnose
Working the problem, with the runbook if one exists.
Restore
Service back, by fix or by workaround. A workaround that restores service in ten minutes beats a root-cause fix that takes four hours. The workaround then generates a problem record, which is where the root cause gets handled.
Close and confirm
With the user, not unilaterally. A ticket closed by the engineer without confirmation is how “it is still broken” arrives a week later.
Review
Anything P1, and anything recurring, goes to problem management.
Major incident management
A P1 affecting the whole organisation runs differently from an ordinary incident.
One incident commander
A single person coordinating, who is not also doing the technical work. That separation is the single most useful thing in a major incident and it is the first thing to collapse under pressure.
A defined bridge
One call or channel where the work happens, so that six people are not investigating the same thing in parallel and two nobody has looked at.
Scheduled communication
Updates to stakeholders at fixed intervals, including when there is nothing new. Silence during an outage generates more calls than an update saying the position is unchanged, and those calls land on the people trying to fix it.
A written timeline
Captured live rather than reconstructed afterwards. It is what the post-incident review works from and what any insurer or regulator will ask for.
A post-incident review within five business days
Blameless, factual, with actions that have owners and dates.
Problem management
Reactive problem management
Triggered by incidents that have already happened. Recurring incidents, any P1, and clusters of related issues that share a suspected cause.
The process is: identify the pattern, investigate the cause, document a known error with its workaround, and implement the permanent fix through change control.
A known error record is valuable on its own. Even before the permanent fix, having the symptom and the workaround written down turns a forty-minute diagnostic each time into a five-minute lookup. That alone is often worth more than the eventual fix.
Proactive problem management
Triggered by analysis rather than by an incident. Reviewing trends in ticket data, monitoring signals and capacity growth to find conditions that will cause incidents.
This is where the monthly reporting stops being a scorecard and becomes useful. Ticket categories trending upward, alerts firing repeatedly without becoming incidents, capacity approaching a limit and repeated manual interventions that should be automated or fixed.
Root cause analysis, done properly
Two techniques, used according to complexity.
Five whys for straightforward causal chains. Ask why until the answer is something that can actually be changed. The discipline is not stopping at the first technical answer.
The application was unavailable. Why? The service had stopped. Why? It ran out of memory. Why? A memory leak in a version we have been running for eighteen months. Why? Patching for that application is manual and was not scheduled. Why? It was never added to the patch inventory when it was commissioned.
The fix at the first level is restarting a service. The fix at the fifth level is a commissioning checklist that adds new systems to the patch inventory. Only one of those stops it recurring, and only one of those helps with the next twelve systems.
Fault tree analysis where multiple contributing factors interact and a single causal chain does not describe what happened.
Root cause analysis that stops at human error is not finished. “The engineer made a mistake” is a description, not a cause. Why was the mistake possible, why was it not caught and what would have prevented it are the questions that produce a fix.
What Namandla delivers
Incident management against the severity framework above, with logging, triage, resolution, escalation and confirmation.
Major incident coordination, including incident commander, communication cadence, timeline capture and post-incident review.
Problem management, reactive and proactive, with known error records, root cause analysis and permanent fixes taken through change control.
Ticket analysis as part of the monthly reporting and quarterly review, identifying recurring causes and proposing what to do about them.
Runbook development for recurring incidents, so that resolution is repeatable and does not depend on who is available.
Escalation management with your third-party vendors and service providers, so that chasing a supplier through a fault sits with us rather than with your office manager.
Remote delivery nationally. Onsite attendance in Gauteng where an incident needs hands.
Common questions about Incident and Problem Management
What is the difference between an incident and a problem?
An incident is an unplanned interruption to a service and the objective is to restore it quickly. A problem is the underlying cause of one or more incidents and the objective is to remove it permanently.
They are separate activities with different time pressures. During an incident, a workaround that restores service in ten minutes is better than a root-cause fix that takes four hours. The root cause then becomes a problem record and gets handled without the clock running. Organisations that only do incident management resolve the same issues indefinitely.
What are your incident response times?
P1, meaning business stopped or a security incident in progress, is 30 minutes including after hours through the standby escalation. P2, major degradation with a workaround, is 4 business hours. P3 and non-contract work is next business day.
Response means an engineer who can work the problem has made contact and started, not an automated ticket receipt. Business hours are 08:00 to 17:00 Monday to Friday excluding public holidays, with the standby rota carrying P1 outside them.
How is severity decided?
On business impact rather than on which system is involved. We agree the definitions with you during onboarding, against your specific systems and processes.
A generic severity matrix produces arguments during incidents, which is the worst time to have them. Writing down that a failure of your dispatch printing is a P1 and a failure of your test environment is a P3, in advance and in your own terms, removes that entirely.
What happens during a major incident?
An incident commander coordinates and does not do the technical work, a single bridge is opened for the response, stakeholders get updates at fixed intervals including when there is nothing new, and the timeline is captured live.
The separation of coordination from technical work is the part that matters most and the first to collapse under pressure. When the person fixing the problem is also fielding calls from executives, both jobs get done badly. Scheduled updates cut the call volume substantially, because most of those calls are people asking for a status nobody has given them.
Do you provide root cause analysis?
Yes, for every P1 and for recurring incidents. The output is a written analysis with the causal chain, the contributing factors and actions that have owners and dates.
We do not accept human error as a root cause. That is a description of what happened, not an explanation of why it was possible. The useful questions are why the mistake could be made, why it was not caught and what would have prevented it, and those produce fixes that hold.
How quickly should ticket volume fall?
In a well-run arrangement, volume should be measurably lower by the end of the second quarter as patching, configuration standards and problem management remove recurring causes. If it is flat after six months, ask why.
The proportion of tickets that are the same thing recurring is typically a quarter to a third of total volume in an environment that has never had problem management. Removing that is not marginal and it should be visible in the reporting rather than asserted.
What is a known error?
A documented record of a problem whose cause is understood but whose permanent fix has not yet been implemented, together with the workaround.
Known errors are useful long before the fix arrives. A documented symptom and workaround turns a forty-minute diagnostic every time into a five-minute lookup by whoever is on the ticket. Building the known error database is often the fastest improvement available on an environment with a long history and no documentation.
Can you handle incidents on systems you do not manage?
We can coordinate them, which is often what is actually needed. Where a third party is responsible for a system, incident coordination means we hold the timeline, run the communication and chase the vendor.
We cannot commit a response time on a system we do not have access to, and we will not pretend otherwise. What we can commit to is that the incident is coordinated, the vendor is chased and you are not the one making the calls.
Do you do post-incident reviews?
Yes, within five business days of any P1. Blameless, factual, working from the timeline captured during the incident rather than from memory.
The five-day window matters. Later than that and detail is lost and the urgency to act on the findings has gone. Blameless is not a courtesy, it is a practical requirement: reviews that assign fault produce defensive accounts, and defensive accounts are inaccurate.
How does this work with our internal IT team?
In a co-managed arrangement the boundary is documented. Commonly we run first and second line incident management and your team holds application-specific work, with a defined escalation path in both directions.
Problem management works well as a joint activity, because your team has the historical context and we have the ticket data and the pattern analysis. The recurring causes are usually things your people already suspected and never had the evidence to make a case for.
When do you not need formal incident and problem management?
On a small environment with low ticket volume and an internal person who knows every system, formal process adds overhead without much return. What still helps is logging tickets consistently, because that is what makes patterns visible later.
If you are in that position, the single most valuable thing is to record everything, including the quick fixes, even in a spreadsheet. The formal apparatus becomes worthwhile once volume grows past what one person can hold in their head, or once more than one person is resolving issues.