Possible Reasons for Minor n-Count Discrepancies Between 360 and Legacy

Clients using the smg360 reporting website will occasionally run into scenarios where they notice that the number of survey responses (n-count) pulled from the Report Builder tool and legacy dashboards do not match the n-count shown on the 360 dashboards or reports with the same filters applied and for the same date range.

Minor discrepancies in the n-counts across the two reporting platforms are normal and should not be cause for alarm for the reasons outlined below.

Background Information: Legacy vs. 360

We have the following client-facing reporting platforms:

Possible Reasons for an n-Count Mismatch Between Legacy and 360

360 KPI and Comparison cards display measures and compare them to create a holistic view of company performance on their executive dashboard, creating efficiency in insights and key metric comparisons. The KPI and Comparison cards are built on the 360 global measures platform and pull aggregated data.

The 360 architecture differs from Legacy reports that are running equivalent queries because Legacy works with nearly real-time data, whereas 360 operates with periodic aggregations of data to better deal with the larger pools of data for multiple surveys that can be displayed on 360 dashboards and reports. This difference means that Legacy is more up-to-date in its counts, which can result in n-count mismatches, and this can be exacerbated further if the periodic aggregations in 360 cards have any processing delays.

The gap of acceptable differences scales with the total value and should not, in most cases, affect the scores, which remain relatively consistent across the two platforms. Discrepancies of less than 5% is acceptable. However, large n-count discrepancies with a gap of more than 5% resulting in inconsistent scores across Legacy and 360, which cannot be attributed to the following reasons, may indicate a problem in how the data is being aggregated and should be reported for investigation.

With that said, in most cases, the n-counts will only be slightly off (taking into account the scale of overall values) across the two platforms and can be ascribed to one or a combination of the following known reasons:

• Data aggregation lag – Scores and n-counts in the Legacy side of the platform will display in nearly real-time and, once aggregated, will display in the 360 dashboards and reports. These aggregations occur every 4 hours and update the reporting website throughout the day. There might be a delay in 360 as the process requires time for records to be ingested into 360.

• Time zone mismatch

  • 360 displays all feedback dates in a user's local time from 5 AM to 5 AM (the time zone defaults from the user’s browser).

  • Legacy always uses CST from 12 AM to 12 AM (no consideration for Daylight Savings Time). For users with time differences between their location and CST, n-counts may be off slightly, but scores should remain relatively consistent. 

• Top Box Score Calculation

  • In 360, the "Top Box" score is calculated as follows: all 5s divided by the count of all 1, 2, 3, 4, and 5 answer options (ignoring any Nulls if the question can be skipped or ignored in the survey).

  • In Legacy, however, it calculates the Top Box score as 5s divided by all valid n-counts, which could include nulls from a person skipping a question, potentially changing the score compared to the 360-calculated score.

    You can filter the Legacy data for the same date range by removing all blanks for OSAT, and the count should match the KPI card count.

• Breakout logic
  • 360 does not support breakout logic, so questions that use breakout in Legacy, such as CATA options 
  • In Legacy, breakout logic can be added to questions to pre-filter the data, e.g. CATA options have logic to only show as 1 if the CATA Main has a response.
  • Due to this, 360 would often have a higher n count; it is not recommended to add CATA options to cards in 360.

Awareness of the above possible reasons is important and will often be our reference when investigating discrepancies between Legacy and 360 reports or dashboards.