CDCU | Climate Data Capability Upgrade

TECHNICAL UPDATE V1

—

Equal Frequency Binning dynamically adjusts to the physics of the range

Equal width binning operates like a ruler with equal segments for reporting. This seems to be normalized as a quartile in industry but is fails as a quantile in the strict definition… Equal width binning is simply a segmented guage. Calling equal width a quartile is unqualified and misleading it appears.

—

The new CDCU project appears to be in an area of math that is not easy to explain. The tools are quite simple and do nothing unique — Python scripts to tinker with, currently in development version 1. I needed to upgrade a 1-year °F NOAA temperature dataset some months ago which led me down this interesting path.

What is missing from the current project scope is: WHY bother to apply jitter and the pass / fail quantile even bin distribution test? I am still exploring that question myself…

… answer why bother … This idea applies to any type of non-negative time-series data that has duplicate clusters affecting its usefulness as a raw dataset. We know °F temperature U.S. data published in integers by NOAA creates duplicate clusters, the topic has been explored in various papers. A 1-year NOAA °F integer set fails even bin counts because of duplicate ties, jitter eliminates many duplicates and breaks ties, the pass/fail test confirms if the bin count passes or fails equal distribution — that’s the current scope of Tools 2 and 3.

THIS BELOW NEEDS CHECKING AND PROBABLY ALSO IMPROVING AS I AM NOT QUALIFIED AT THIS LEVEL OF MATH DETAIL SO PLEASE FEEL FREE TO POINT OUT MISTAKES OR WEAKNESSES

The Data: The CSV omits TAVG, you can add a formula to include it or use TMAX to test with.

https://www.ncei.noaa.gov/access/services/data/v1?dataset=daily-summaries&stations=USW00012960&startDate=2024-01-01&endDate=2024-12-31&dataTypes=TMAX,TMIN,TAVG,ADPT&units=standard&format=csv

About Tool 1: Physics relies on Kelvin because Celsius and Fahrenheit lack a true zero point (e.g., 20°C is not twice as hot as 10°C). Moving to an absolute scale prevents negative values, simplifying downstream processing like jitter analysis, binning, and encoding. Next, let’s look at Tools 2 and 3.

Ties & Jitter — in a case where jitter is required:

Quantile Binning: Only some, not all of the ties need to be broken in a ranked set to achieve Equal Frequency Binning for quartile or quintile sets.

Fully Indexable: Extreme jitter may be required to achieve a fully indexed set with no ties. Outside the scope of the tools in this project but worth noting.

Equal-Frequency Binning vs. Equal-Width Binning. Both split data into k buckets, but they locate boundaries on different scales — that’s the fork. One trades magnitude-awareness for balanced counts; the other trades balance for readable ranges.

Equal Frequency Binning (even segmentation of an ordered set of values with an enforced equal bin count), an ordinal method often called Quantile Binning / Equi-depth. Boundaries sit at evenly spaced percentile positions in sorted order — these positions define the boundary values. Rank acts as an abstraction layer — magnitude is not the focus — bin counts must be equal, or off by at most 1. Used where balanced group sizes matter more than tidy ranges.

Applications: Many types of general analysis, data smoothing and machine learning.

Equal Width Binning (cardinal, by magnitude). Boundaries come from arithmetic on the range: w = (max – min) / k, boundary_i = min + i·w. No rank, no abstraction — magnitude drives it directly, so counts land wherever skew puts them. Used where readable, fixed-size ranges matter more than balance.

Applications: histograms, human-readable brackets (age, price, temperature), and fixed-resolution signal quantization.

The Quantile Binning even bin count pass/fail test exists here strictly as preparation for downstream quantile analysis needing even bin counts. The test shows a PASS e.g., (50,50,50,50) OR (50,51,51,51) accepted with a max difference of 1. Alternatively, a FAIL with an uneven bin count e.g., (45,54,53,48).

Jitter is a variable, leaning outside the scope of this project — additional measures can be taken to use more advanced jitter to remove all ties from a dataset so the whole set is indexable without ties. It then becomes useful for spatial mapping, telemetry encoding, indexed lookup — because it is fully addressable without ties and now can function with more advanced systems.