Trust in data · Free grade

Why you can trust our numbers

The main risk of AI in analytics is reliability. Language models tend to "hallucinate": they confidently give made-up numbers where they do not know the answer. Below - how we removed made-up numbers from the report and how we check every calculation method.

Problems

AI reports with convincing but invented numbers are not rare: the model names a figure that is not in the data. This property of how language models work must not reach the reports.

The second barrier is the quality of logs. They often contain errors, come from different systems and are not ready to load. They need to be put in order before the analytical modules start work.

Architecture: numbers apart, text apart

  • 1. A program counts the numbers, not AI. All calculations are done by a statistical engine in Python (numpy, scipy, statsmodels - libraries for calculations and statistical tests). AI cannot change the result of a calculation.
  • 2. AI only describes. A language model writes in plain language about facts the engine has already counted. Each task has its own tool.
  • 3. Cross-check. In the paid analysis, a second, independent model checks the finished text against the counted facts and catches mismatches before the report reaches you.

How we check each method

A method goes into production only after a strict check. Without it, the engine does not use the method. Right now 12 methods have passed the circle, 13 are waiting for the check and are hidden, and we removed 4: they did not pass the checks on real and synthetic data.

  • 1. Logs with a known answer. We built 72 logs with planted scenarios taken from "real" situations: for example, a failure every Thursday at 02:00. The report must name the same systems, dates and numbers. The logs are run with every change to the engine.
  • 2. Check for false findings. We create 40 "years" of log with no patterns. The method must not find what is not there: by time of day - 0 false findings out of 40, by links between systems - no more than 1 out of 40.
  • 3. Chance threshold. Each finding is compared with what chance would give. The more checks, the stricter the threshold. A finding must stay if you remove the two busiest days: one large outage is not counted as a system problem.

What the engine finds and fixes

Data always has errors. The task is to fix them before they distort the conclusions. Examples of what the report wrote before the checks and what it does now.

Situation in the dataWhat the report wrote before the checkWhat the engine does now
Old tickets closed all at once on one day"VPN - 3066 days of downtime"Does not count such tickets as downtime, names the day of the mass closing
Four systems fail about equally"Downtime is concentrated in three systems"Names a system only if its share of failures is higher than chance gives
Records "failure not confirmed"Counted them as outages: median recovery 14 minutes instead of 21.5Removes such records before the calculations
Different time formats ("2,71" or "02:42:33")Read one record out of fiveReads all usual formats, including Jira and ServiceNow

How it works on real data

Cleaning the noise. An outage log has 802 records, 210 of them marked "failure not confirmed". If you count all rows, the median recovery time is 14 minutes. The engine removed this noise before the calculations and showed the median over 592 real outages - 21.5 minutes.

The main point. On the same 592 outages a simple count showed: the 8.8% longest outages give 80% of all downtime.

Data quality. In a log of disk failures (1057 rows) the engine noticed that all failures were recorded at exactly 02:00. This is the start time of the night check, not the time of the failure. So the report will not call 02:00 a "dangerous hour".

What is needed from your data

You do not need to prepare the data in a special way. A usual export from Jira, ServiceNow, Zabbix or a list of incidents in CSV / Excel is enough. Required - the start and end time of the event (or a ready duration). Desirable - the name of the service or system. Before the analysis, the engine checks how complete the data is.

If the export is non-standard or has a rare export error, we do not throw the data away, we clean it by hand for your situation.

The same analysis on your data - "Run analysis" on the home page →

Calculations run on our server in the EU; the language model that writes the text sees only numbers and masked names.