Getting a number off a sensor is the easy part. What I’ve learned over years of building IoT and industrial products is that the real engineering work starts after the raw reading arrives — validating it, filtering it, fusing it with other sensors, storing it efficiently, and getting it to wherever it needs to go, all within a tight memory and power budget. In this article, I want to walk through the complete lifecycle of sensor data inside an embedded system, from acquisition to action.
The Sensor Data Lifecycle
graph TD
A[Sensor Acquisition] --> B[Raw Data Validation]
B --> C[Calibration & Scaling]
C --> D[Filtering / Noise Reduction]
D --> E[Sensor Fusion - if multiple sensors]
E --> F[Local Decision Logic / Thresholding]
F --> G[Storage / Logging]
F --> H[Communication - send to gateway/cloud]
G --> I[Later Retrieval / Analysis]
Step 1: Acquisition
Sensor data reaches the MCU through one of a few channels: a direct ADC reading (analog sensors), a digital bus like I2C or SPI (most modern digital sensors — accelerometers, humidity sensors, IMUs), or a dedicated peripheral like a PWM/duty-cycle input for some ultrasonic distance sensors. I generally prefer digital sensors with built-in ADCs and calibration when the budget allows, because it moves a lot of the noise-sensitive analog work off my board and into a purpose-built sensor IC.
/* Example: Reading a digital temperature/humidity sensor over I2C (SHT31-style) */
#include "i2c_driver.h"
#define SHT31_ADDR 0x44
#define SHT31_CMD_MEASURE 0x2C06
int SHT31_Read(float *temperature_c, float *humidity_pct)
{
uint8_t cmd[2] = {0x2C, 0x06};
uint8_t data[6];
if (I2C_Write(SHT31_ADDR, cmd, 2) != I2C_OK) return -1;
HAL_Delay(15); /* sensor conversion time */
if (I2C_Read(SHT31_ADDR, data, 6) != I2C_OK) return -1;
uint16_t raw_temp = (data[0] << 8) | data[1];
uint16_t raw_hum = (data[3] << 8) | data[4];
*temperature_c = -45.0f + 175.0f * ((float)raw_temp / 65535.0f);
*humidity_pct = 100.0f * ((float)raw_hum / 65535.0f);
return 0;
}
Step 2: Validation
Raw sensor data can be wrong for reasons that have nothing to do with the physical world — a loose connector, an I2C bus glitch, a sensor fault, or an out-of-range value from a damaged sensor. Before I trust a reading, I check it:
typedef enum { SENSOR_OK, SENSOR_OUT_OF_RANGE, SENSOR_COMM_ERROR, SENSOR_STUCK } sensor_status_t;
sensor_status_t Validate_Temperature(float temp_c, float last_temp_c)
{
if (temp_c < -40.0f || temp_c > 125.0f) {
return SENSOR_OUT_OF_RANGE; /* physically implausible */
}
if (temp_c == last_temp_c) {
static uint16_t stuck_count = 0;
if (++stuck_count > 50) return SENSOR_STUCK; /* sensor may be frozen/disconnected */
}
return SENSOR_OK;
}
Step 3: Calibration and Scaling
Even digital sensors need calibration against known references to correct manufacturing tolerances and installation offsets. I typically store calibration coefficients (offset and gain, or a multi-point calibration table) in non-volatile memory (EEPROM or a reserved flash sector) so each unit is individually corrected without needing different firmware per device.
typedef struct {
float offset;
float gain;
} calibration_t;
float Apply_Calibration(float raw_value, calibration_t *cal)
{
return (raw_value * cal->gain) + cal->offset;
}
Step 4: Filtering and Noise Reduction
Even a well-conditioned, validated signal often benefits from digital filtering to smooth out residual noise. Common techniques I use:
- Moving average filter — simple, low compute cost, adds latency proportional to window size
- Exponential Moving Average (EMA / low-pass IIR filter) — very memory-efficient (only one stored value), tunable responsiveness
- Median filter — excellent at rejecting sudden spike/outlier readings without smoothing legitimate trends
- Kalman filter — optimal for combining noisy measurements with a system model, common in navigation and motion sensing
/* Exponential Moving Average filter - extremely common in embedded firmware
because it needs only one float of state and one multiply-add per sample */
float EMA_Filter(float new_sample, float prev_output, float alpha)
{
/* alpha between 0 (heavy smoothing) and 1 (no smoothing) */
return alpha * new_sample + (1.0f - alpha) * prev_output;
}
graph LR
A[Raw Noisy Samples] --> B[EMA Filter alpha=0.2]
B --> C[Smoothed Output]
A -.->|spike outlier| D[Median Filter]
D -.->|rejects spike| C
Step 5: Sensor Fusion
Many modern embedded products combine multiple sensors to produce a more reliable estimate than any single sensor could alone. The classic example is an IMU combining a gyroscope (accurate short-term, but drifts over time) with an accelerometer (noisy short-term, but stable long-term) using a complementary or Kalman filter.
/* Simplified complementary filter for pitch angle from gyro + accelerometer */
float Complementary_Filter(float gyro_rate_dps, float accel_angle_deg,
float dt_s, float prev_angle_deg)
{
float gyro_estimate = prev_angle_deg + gyro_rate_dps * dt_s;
float alpha = 0.98f; /* trust gyro short-term, accel long-term */
return alpha * gyro_estimate + (1.0f - alpha) * accel_angle_deg;
}
graph TD
A[Gyroscope - accurate short term, drifts] --> C[Complementary / Kalman Filter]
B[Accelerometer - noisy short term, stable long term] --> C
C --> D[Fused, Stable Angle Estimate]
Step 6: Local Decision Logic
Increasingly, I try to make embedded devices act on sensor data locally rather than shipping every raw sample to the cloud — this reduces latency, saves bandwidth and power, and keeps the system functional even without connectivity.
void Process_Vibration_Sample(float g_force)
{
static uint8_t alarm_count = 0;
const float THRESHOLD = 2.5f;
if (g_force > THRESHOLD) {
if (++alarm_count > 5) { /* require sustained condition, avoid false triggers */
trigger_alarm();
alarm_count = 0;
}
} else {
alarm_count = 0;
}
}
Step 7: Storage and Logging
For applications needing historical data (data loggers, predictive maintenance), I store sensor readings either in external flash/SD card (for large volumes) or a compact circular buffer in RAM/EEPROM for recent history. Efficient storage often means downsampling or event-based logging rather than storing every raw sample.
typedef struct {
uint32_t timestamp;
float value;
} log_entry_t;
#define LOG_SIZE 100
static log_entry_t log_buffer[LOG_SIZE];
static uint16_t log_index = 0;
void Log_Sensor_Value(float value)
{
log_buffer[log_index].timestamp = get_rtc_timestamp();
log_buffer[log_index].value = value;
log_index = (log_index + 1) % LOG_SIZE; /* circular buffer */
}
Step 8: Communication
Processed sensor data typically needs to leave the device — over BLE to a phone app, MQTT to a cloud broker, Modbus to a PLC, or CAN to another ECU. I cover communication protocols in detail in a dedicated article, but the important point here is that by the time data reaches the communication stack, it should already be validated, calibrated, and filtered — never raw ADC counts.
Managing Multiple Sensors on a Single System
Real products rarely have just one sensor. A professional embedded design typically needs a scheduling and abstraction strategy so that adding or swapping sensors doesn’t require rewriting the whole application:
/* A simple sensor abstraction layer, letting application code
treat any sensor uniformly regardless of its underlying bus */
typedef struct {
const char *name;
int (*init)(void);
int (*read)(float *out_value);
uint32_t poll_interval_ms;
uint32_t last_read_ms;
} sensor_driver_t;
static sensor_driver_t sensors[] = {
{ "temperature", SHT31_Init, SHT31_ReadWrapper, 1000, 0 },
{ "accel", LIS3DH_Init, LIS3DH_ReadWrapper, 100, 0 },
{ "light", APDS_Init, APDS_ReadWrapper, 5000, 0 },
};
void Sensor_Scheduler_Tick(uint32_t now_ms)
{
for (int i = 0; i < ARRAY_SIZE(sensors); i++) {
if (now_ms - sensors[i].last_read_ms >= sensors[i].poll_interval_ms) {
float value;
if (sensors[i].read(&value) == 0) {
Process_Sensor_Reading(sensors[i].name, value);
}
sensors[i].last_read_ms = now_ms;
}
}
}
This kind of table-driven design keeps polling intervals independent per sensor (a fast-changing accelerometer polled at 100ms, a slow-changing light sensor at 5 seconds), which also directly supports the power management goals covered elsewhere in this series — sensors aren’t read more often than genuinely needed.
Edge Processing and On-Device Machine Learning
A growing trend I’ve been implementing more of in recent projects is running lightweight inference directly on the sensor data at the edge, rather than shipping raw or lightly-processed data to the cloud for analysis. TinyML frameworks (TensorFlow Lite for Microcontrollers, Edge Impulse) allow surprisingly capable models — gesture recognition, anomaly detection, keyword spotting — to run directly on a Cortex-M class MCU using only kilobytes of RAM.
graph LR
A[Raw Sensor Samples] --> B[Feature Extraction - FFT, statistical features]
B --> C[Quantized Neural Network Inference on MCU]
C --> D[Classification Result e.g. Normal/Anomaly]
D --> E[Only send alert to cloud if anomaly detected]
This architecture dramatically reduces radio usage (since most data never leaves the device) while still catching the events that actually matter — directly compounding with the power-saving strategies covered in the power management article.
Data Quality and Outlier Handling in Production Systems
Beyond basic range validation, production-grade sensor pipelines typically implement statistical outlier detection to catch subtler faults that still fall within a sensor’s nominal operating range:
/* Simple z-score based outlier detection using a rolling
mean and standard deviation, useful for catching sensor
drift or intermittent faults that pass basic range checks */
typedef struct {
float mean;
float variance;
uint32_t count;
} running_stats_t;
void Update_Running_Stats(running_stats_t *stats, float new_value)
{
stats->count++;
float delta = new_value - stats->mean;
stats->mean += delta / stats->count;
stats->variance += delta * (new_value - stats->mean);
}
bool Is_Outlier(running_stats_t *stats, float value, float threshold_stddev)
{
float stddev = sqrtf(stats->variance / stats->count);
return fabsf(value - stats->mean) > (threshold_stddev * stddev);
}
Testing Sensor Data Pipelines
Validating a sensor data pipeline requires more than confirming it works with a healthy sensor under normal conditions — I specifically design test cases around the failure modes that show up in the field:
- Injected fault testing — physically disconnecting a sensor mid-operation to confirm the validation layer correctly flags a communication error rather than reporting stale or garbage data as valid
- Boundary value testing — feeding the pipeline values right at and just outside the plausible physical range to confirm range-checking logic behaves correctly at the edges, not just in the obviously-valid middle of the range
- Long-duration soak testing — running the full pipeline for days or weeks to catch slow-accumulating issues like circular buffer wraparound bugs or gradual filter drift that wouldn’t appear in a short bench test
- Replay testing with recorded real-world data — capturing genuine sensor traces from field conditions (including known problem periods, like a vibration sensor during a real equipment fault) and replaying them through the pipeline in simulation to verify detection logic actually catches the events it’s meant to
/* Simple fault injection harness used during development to
verify the pipeline's validation layer without needing to
physically break real hardware for every test case */
void Test_SensorFault_Detection(void)
{
float last_temp = 25.0f;
sensor_status_t status;
status = Validate_Temperature(200.0f, last_temp); /* implausible value */
assert(status == SENSOR_OUT_OF_RANGE);
for (int i = 0; i < 60; i++) {
status = Validate_Temperature(25.0f, 25.0f); /* value never changes */
}
assert(status == SENSOR_STUCK);
}
Compressing Sensor Data for Storage and Transmission
When bandwidth or storage is tight, I look at lightweight compression techniques suited to constrained devices rather than transmitting raw values. Delta encoding — storing the difference between consecutive readings instead of the full value — works particularly well for slowly-changing sensor data like temperature, since the deltas are typically small and compress far better than the raw absolute values.
/* Delta encoding: store first value in full, then only
the (small) differences from the previous reading */
typedef struct {
float first_value;
int8_t deltas[63]; /* scaled small differences, 1 byte each instead of 4 */
} compressed_log_t;
void Compress_Readings(float *raw_values, int count, compressed_log_t *out, float scale)
{
out->first_value = raw_values[0];
for (int i = 1; i < count; i++) {
float delta = raw_values[i] - raw_values[i - 1];
out->deltas[i - 1] = (int8_t)(delta * scale); /* clamp/scale as appropriate */
}
}
This kind of technique can cut logged or transmitted data volume by 60-75% for slowly-varying sensor signals, directly reducing both flash wear (fewer bytes written per log entry) and radio airtime (fewer bytes to transmit) — compounding with the power management and memory-type decisions covered elsewhere in this series.
Sensor Redundancy for Critical Applications
In applications where a single sensor failure can’t be tolerated — industrial safety systems, medical devices, aerospace — I design in sensor redundancy, using multiple independent sensors measuring the same quantity and a voting or comparison scheme to detect disagreement.
/* Triple modular redundancy pattern: three independent sensors,
majority vote used if one disagrees significantly with the others */
float Vote_Triple_Redundant(float s1, float s2, float s3, float tolerance)
{
if (fabsf(s1 - s2) < tolerance) return (s1 + s2) / 2.0f;
if (fabsf(s1 - s3) < tolerance) return (s1 + s3) / 2.0f;
if (fabsf(s2 - s3) < tolerance) return (s2 + s3) / 2.0f;
trigger_sensor_disagreement_fault(); /* all three disagree - system fault */
return s1; /* fail-safe fallback, application-specific */
}
Redundancy adds cost and complexity, so I reserve it for genuinely safety- or mission-critical measurements rather than applying it universally — for most consumer and industrial monitoring applications, a single well-validated sensor with good fault detection is the appropriate and proportionate choice.
Real-World Applications
- Wearable fitness trackers — fusing accelerometer and gyroscope data for step counting and activity recognition, with heavy on-device filtering to save battery on radio transmissions
- Predictive maintenance sensors — vibration and temperature trend analysis performed locally to detect equipment degradation before failure
- Weather stations — multi-sensor fusion (temperature, humidity, pressure) with periodic logging and cloud upload
- Autonomous robotics — real-time sensor fusion of LiDAR, IMU, and wheel encoders for localization
Performance and Reliability Considerations
Filtering and fusion algorithms need to be chosen with the MCU’s compute budget in mind — a full Kalman filter with matrix operations may be overkill (and too slow) on an 8-bit MCU, where a simple complementary filter or EMA is far more appropriate. For reliability, I always design sensor handling code to fail gracefully: a disconnected or faulty sensor should be detected and flagged, not silently produce garbage data that downstream logic treats as valid.
Frequently Asked Questions
Should filtering happen in the sensor IC or in MCU firmware? Both are valid depending on the sensor — many modern digital sensors (IMUs, environmental sensors) include on-chip low-pass filtering and even basic fusion, which reduces MCU workload, but application-specific filtering usually still happens in firmware.
How much sensor data should be stored locally versus sent to the cloud? This depends on bandwidth, power budget, and application needs — many designs store full-resolution data locally for a rolling window and send only summarized or event-triggered data to the cloud to save power and bandwidth.
What’s the difference between calibration and filtering? Calibration corrects for known, systematic errors (offset, gain, non-linearity) specific to a sensor unit. Filtering reduces random noise and outliers in the signal over time. Both are usually needed together.
Why use a median filter instead of an average filter for spike rejection? An average filter is dragged toward outliers because every sample contributes to the mean; a median filter is immune to a small number of extreme outliers since it just picks the middle value of the sorted window.
Summary
Handling sensor data in an embedded system is a multi-stage pipeline: acquire the raw value, validate it against plausible bounds, apply calibration to correct known errors, filter out noise, optionally fuse it with other sensors for a more reliable estimate, act on it locally where possible, log it efficiently, and only then send it out over a communication channel. Skipping any of these stages — especially validation and calibration — is one of the most common reasons embedded products end up with “mysteriously” unreliable sensor readings in the field.