Optimized Median Filter

[guy]

I have an existing hardware design (no need to suggest upgrades to HW - not possible) and I get *very noisy* sensor readings compared to the accuracy required. I plan to use a small circular buffer (16 samples of 10-bit, sampling every 1ms) in order to get up-to-date readings when requested.
Due to the type of noise Average Filter is not very accurate and a Median Filter is more suitable. Olympic is not good enough since there is noise in all samples.
Can anyone suggest optimizations to the Median filter or a different type of filter I'm not aware of? Recommended sort algorithm? The pic happens to be PIC24FJ64GA308.
Thanks!

[temtronic]

I have to say 'GIGO'. Garbage In = Garbage Out.
Put another way... one man's noise is another's valid signal.

Now assuming you're taking ADC readings you should be able to add small filter caps to the PCB either near the PIC ADC pins or the connector to the analog sources. I know you said no HW mods BUT this one could solve your problem without knowing more details.

Unfortunately you're 'handcuffing' us by not allowing the PROPER course which is to do what's necessary to acquire good,clean data. So the only 'digital' recourse I can see is taking say 64 samples minimum and averaging them. You say 16 isn't enough(due to noise) so 32 might be better. I'd START at 128 though then go down if possible.
The problem with most 'filters' is that YOU have to decide what is 'noise' and what is really a good 'reading'. That 3rd reading might be bad, might be good so do you keep it or toss it?

Code, like life is a compromise....

Jay

[asmboy]

Have you done any external analysis of the noise?
"noise" comes in all sorts of styles. Do you have a spectrum analyzer ?

If there is a strong component at a harmonic or sub-harmonic of 1khz - then 1msec as a repeat data read timing could be a disaster, and make the problem worse, to give just a trivial example.

Take a breath here and uderstand that
you need to be able to understand the character of the noise if you want to really get the best of it.

And i'm puzzled.
How was it possible that the hardware is so "frozen" that you can't revisit it,
yet the problem seems to be NEW and pressing??

What changed since development?

In my own practice, hardware is NEVER locked down
until it and the software are very chummy indeed ....

[PCM programmer]

[guy]

Ha! I already know who I'm dealing with I could see those HW changes coming... There are hundreds of systems installed in different locations but remote upgrade is a great solution. Better QA during dev. next time.

PCM Programmer - I missed your post when I searched the forum - the insertion sort algorithm is what I was looking for (I assume you tested it). One more optimization I had in mind was to store the data as int8 by defining the samples as Delta from the average or something in that direction.
I don't have a specific goal for # of instructions, just wanted to find a quick way to get my signal out. Let's hope that 16 samples will be enough. I suppose a Median filter will require less samples that an Average to give a good result but this will require some testing. I don't have an analysis of the noise or access to it, just random samples from a remote location...

[PCM programmer]

[Ttelmah]

As has already been said, the 'nature' of the noise, and the response speed needed, will affect what is the best filter.
For instance, if 90% of the data is fine, and there is just an occasional spike, then Olympic averaging or the median filter are effective routes. However if there are 'bursts' of noise, then these will not be effective. Instead a 'trend' filter, which rejects changes until several cycles of similar signal are seen, is needed, but this increases lag. If your real signal has known frequency capabilities (can't slew faster than a particular rate), then a low pass filter is effective. If the primary noise is from some external known source, then a notch filter may be the most effective. A bit of research now will help enormously, both in getting the 'right' filter, and in the quality of the result.

[Gabriel]

I recently did a proyect with thermocouples and wanted to filter my sensor readings a bit more than usual.

I took 3 readings checked if all 3 where within 1.25degC from each other, averaged the 3 readings and fed that result to a 10 value buffer where i eliminated highest and lowest and then averaged the remaining 8...

It worked really well.... but as others have said, Garbage in, Garbage out... proper noise control starts at the hardware level.
_________________
CCS PCM 5.078 & CCS PCH 5.093

[temtronic]

You should tell us what the sensors and the expectd range of them is. If it's a tempeature sensor like Gabriels then his approach is valid but if it's a knock sensor then potentially every reading could be valid ! In this case decideing what is 'good' versus 'bad' is a challenge.
Where the 'noise' comes from is important. 1) from the sensor...replace it,upgrade,improve power to it... 2) wiring to sensor.... use shielded cabling,rerout it, twist it... 3) PCB layout... add more decoupling caps,proper ground plane,rerout analog from clocking signals... 4) airborne noise... maybe use a metal box, add mumetal(is it still made ?),tuen off the noise generator?


The problem I have with the 'digital filter - code' solution is in how you decide what is a good signal and what is bad. Maybe a running average of 32 samples is OK, maybe choose every other sample? Using a 10 bit ADC in 8 bit mode isn't 'nice' but doing 8 bit math is faster = maore samples = better result ?

I did have an ugly problem with crosstalk years ago from a bank sending data at 1AM every night. The solution wa sto get Bell to swap the cable spans for me.Point is you have to figure out WHERE the noise is coming from to deal with it.

The more info you supply the better we can come up with a solution.

Jay

[newguy]

[guy]

Thanks everyone! I appreciate your support. Special thanks to those who are following my line of thought and answer my question.
Suggesting hardware solutions is similar to the joke where someone asks another person in the street for directions and the guy answers: "Well, if I were you I wouldn't start from here..."

[temtronic]

newguy makes a valid point, it's important to locate the source of the noise.

If all the units (PCBs, power supplies, cabling, etc) are the same then it's 'something' local to only a few units. Noisy AC lines, arc welders, cell towers, trolley buses ????

Yes I know it can be a nightmare finding the noise but 'divide and conquer' will work. Make a list of good vs. bad and maybe you'll see why.

I upgraded hundreds of my energy control system transponders free of charge. The lab test didn't 'simulate' the real World and to eliminate calls at 3AM redid the PCBs to be more 'robust' ONCE I found how the noise got into the systems. I now 'over design' all projects, build once, install once, sleep every night.


Jay

[guy]

I'm glad you sleep well at night and thanks for the tips. In my case the environment is known to be noisy. ESD protection on all lines, shielded cables etc. Noise will always be there to some extent, and in some installations more than others. No big deal.

[Ttelmah]

The point being made is that there is a big difference between a generically 'noisy' environment (a factory say, with dozens of machines working), and a noisy environment where the primary noise 'source' is one or two devices. For instance one piece of my kit goes into some things like industrial pumping stations, where 90% of the big 'noise' events are when individual pumps start, The former gives noise that can be treated as 'random', while the latter has a very specific frequency spectrum and wave-shape, that requires specific handling, or you actually waste data by over=filtering.
The other question relates to response times and frequencies. You could (for instance) use a simple rolling average filter, with a very high damping factor, but this introduces massive lag in the output. So what lag is acceptable?.
Use a data logging device, and record a typical days noise. Run a Fourier analysis on this, and see if there are any large frequency spikes. If so, consider either using DSP, and rejecting these, or adding hardware filters to block these frequencies.
Unless you get rid of primary noise signals at specific frequencies, there is a very great likelyhood that if your sampling is at a specific rate, you may hit an overtone or harmonic at times as clocks drift, and then your filtering can become worse than useless....

[guy]

Thank you Ttelmah. This is good advice.