Master your cycles. Data aggregation without pivot tables

The Problem: Synthesizing Big Data

In the previous section on Vectorized Computation, we saw how to transform raw signals (torque, rotational speed) into physical quantities (power) at lightning speed. You now have a complete, precise, but… enormous table.
Having 200,000 calculation points is a technological victory, but for a test report or design validation, it’s unusable. Your client or manager doesn’t want to see every millisecond; they want to know the average performance per cycle, the peak pressure per phase, or the standard deviation over the entire test.

The De Facto observation: Calculation provides precision, aggregation provides vision.

The Problem: The glass ceiling of Pivot Tables (PTs)

In Excel, pivot tables are used to summarize data. It’s a powerful tool, but its limitations quickly become apparent in engineering:

• Opacity: It’s not always clear which formula is applied “behind” the cell.
• Rigidity: Calculating a Root Mean Square (RMS) or a custom median in a pivot table is a real challenge.
• Fragility: A structural change in your source files and the entire pivot table breaks!

The Solution: The power of “Split-Apply-Combine”

In Python, aggregation relies on a key function: .groupby().
It follows a logic that any engineer appreciates for its transparency:

1. Split: The data is separated according to a criterion (e.g., Cycle number or Test name).
2. Apply: A calculation is applied (e.g., Mean, Max, Standard deviation).
3. Combine: The results are gathered into a concise and auditable summary table.

The “De Facto” Code: The multi-criteria synthesis

Here’s how, with a single instruction, you can generate a complete report from your spreadsheet:

# We create a summary per 'Cycle'
#  For each column, we define the desired calculation
summary_report = df.groupby('Cycle').agg({
    'Power (kW)': 'mean',     # Average per cycle
    'Pressure (bar)': 'max',      # Pressure Peak
    'Temperature (C)': 'std'      # Stability (Standard Deviation)
})

print(summary_report)

Why is this a revolution for your relationships?

Unlike Excel, your aggregation rule is written in plain text. If you decide to change the average power calculation to a median to eliminate the influence of a transient peak, you simply need to modify one word in your code. Repeatability is absolute: run 100 different test files through this script, and you will obtain 100 reports with virtually identical structures.

Capacity	Dynamic Table (Excel)	Agregation (Python)
Scalabilty	Becomes slow beyond 100,000 lines.	It handles millions of lines while remaining fluid.
Repeatability	You need to rebuild or “Refresh” manually.	The script processes 1 or 1000 files in the same way.
Complexity	Filtering specific cycles is tedious.	We can group them by several criteria (Cycle + Sample).
Ouput	Formatting is often rigid.	Result ready to be exported as CSV or integrated into a graph.

The “De Facto” Advice

Aggregation is the final filter for your expertise. It allows you to move from the role of “file processor” to that of “product validator.” By automating this synthesis, you free up your time for critically analyzing results rather than manipulating cells.

Concrete example

Let’s apply this method to the engine test from the previous article. Instead of navigating a scatter plot, we get a direct performance comparison:

Why does this view change everything for you?
Look at the difference between cycles 1 and 4: in cycle 1, with an average of 35 kW, your machine is already struggling. But in cycle 4, the average of 34 kW seems perfectly healthy… if we ignore the 38 kW peak detected by our aggregation.
As for cycle 3, the average is misleading, telling you everything is normal. Only the maximum value tells you the truth: your machine has been subjected to a stress of 37 kW.
Without this dual reading (average vs. maximum), you’re missing 50% of the diagnosis.

Conclusion: The Agregation, the arbiter of your test

The verdict is clear: If you simply look at your averages in Excel, you’re validating systems that may be failing right before your eyes. Automated aggregation with Python gives you back the “supervision” essential for any expert.

Next step: Take control of your data (literally)

Having a summary table is good. But to convince management or a client, the anomaly must be obvious.

We’re going to learn how to transform these figures into dynamic graphs. Imagine being able to hover over your Cycle 3, click to instantly zoom in on the 37 kW peak, and isolate the fault in the blink of an eye. We’re going to move from a static report to a true driver dashboard.

👉 Go to the next chapter: Plotly — Give your results an interactive voice