data-frame

A data frame is a data structure used to hold data in tables with rows and columns. It is meant for conveninent access and manipulation of relatively large data sets (however these data sets must fit into the process memory). The package also provides functions for loading and saving data from data frames as well as several utilities and helper for statistical calculations, plotting and curve fitting.

1 Rationale

Consider an example: during a sport activity, a sport watch will record data at periodic intervals, usually once every second. The data recorded might be time stamp, latitude, longitude, heart rate, distance, speed, cadence, power, etc. A high end sports watch will record up to 40 such measurements every second.

A simple approach for representing this data is to define a "DataPoint" structure, containing members for each possible values and represent the entire activity as a vector of data points. This approach has several problems:

• If a structure is used, it will need to have up to 40 or so members, but most of the time they would be empty, wasting memory. Since we never know what data might be collected (this depends on the number and types of sensors that are active), we cannot save much by defining sub-types, like a RunDataPoint or a BikeDataPoint

• Operations on the data is done for one or only a few measurements at the time. For example, to find the average heart rate, one needs to traverse all the data points and look a the "hr" member of such a structure, if the structure is big, every reference to the "hr" member will be a memory cache miss.

A data frame object addresses the problems above by storing measurements for the same parameter together. Essentially, all heart rate measurements are stored together in a vector, all cadence measurements are stored together in a different vectors. All these vectors have the same number of elements (the number of data points) and the same position in each such vector represents data at a certain point in time. This data organization has some advantages:

• Memory is only used for data that actually exists. For example, if no power data is recorded, there will be no power data series in the data frame.

• Operations on the data have efficient memory access. Calculating the average heart rate involves just referencing elements in a continuous vector.

2 Creating data frames

The functions below allow constructing new data frames. They are mainly intended for writing functions that load data into data frames from different sources.

cmpfn specifies an ordering function to use. If present, values can be looked up in this series using df-index-of and df-lookup. The data must be ordered according to this function

na specifies the "not available" value for this series, by default it is #f

contractfn is a contract function. If present, all values in the data series, except NA values must satisfy this contract.

If a series named name already exists in the data frame, it will be replaced.

3 Loading data into data-frames and saving it out again

The functions construct data frames by loading data from different sources.

na represents the value in the CSV file that represents the "not available" value in the data frame. Strings equal? to this value will be replaced by #f. Alternatively, this can be a function which tests a string and returns #t if the string represents a NA value

When quoted-numbers? is #t, all quoted values in the CSV file will be converted to numbers, if possible. E.g. a value like "123" will be converted to the number 123 if quoted-numbers? is #t, but will remain the string "123" if the parameter is #f.

start and stop denote the beginning and end rows to be written out, by default all rows are written out.

4 Inspecting and extracting data

start and stop indicate the first and one-before-last row to be selected. filter, when present, will filter values selected: only values for which the function returns #t will be added to the resulting vector.

If there is no filter specified, the resulting vector will have (- stop start) elements. If there is a filter, the number of elements depends on how many are filtered out by this function.

start and stop indicate the first and one-before-last row to be selected. filter, when present, will filter values selected: only values for which the function returns #t will be added to the resulting vector.

If there is no filter specified, the resulting vector will have (- stop start) elements. If there is a filter, the number of elements depends on how many are filtered out by this function.

This is intended to be used in for and related constructs to iterate over elements in the data frame:

(for (([lat lon] (in-data-frame df "lat" "lon")))

(printf "lat = ~a, lon = ~a~%" lat lon))

(for ((coord (in-data-frame/list df "lat" "lon")))

(match-define (list lat lon) coord)

(printf "lat = ~a, lon = ~a~%" lat lon))

The series must be sorted, see df-set-sorted, otherwise the calls will raise an error.

The value need not be present in the series, in that case, the returned index is the position of the first element which comes after the value, according to the sort function. This is the position where value could be inserted and still keep the series sorted. A value of 0 is returned if value is less or equal than the first value of the series and a value of (df-row-count df) is returned if the value is greater than all the values in series.

df-lookup* allows looking up multiple values and will return a list of the corresponding values.

These functions combine df-index-of and df-ref into a single function.

An interpolation function can be specified, if the default one is not sufficient. This function is called once for each value resulting series (i.e. it interpolates values one by one).

fn is a function of ether one or two arguments. If fn is a function with one argument, it is called with the values from all series as a single vector. If fn is a function of two arguments, it is called with the current and previous set of values, as vectors (this allows calculating "delta" values). I.e. fn is invoked as (fn prev current). If fn accepts two arguments, it will be invoked as (fn #f current) for the first element of the iteration.

fn is a function of ether two or three arguments. If fn is a function with two arguments, it is called with the fold value plus the values from all series is passed in as a single vector. If fn is a function of three arguments, it is called with the fold value plus the current and previous set of values, as vectors (this allows calculating "delta" values). I.e. fn is invoked as (fn val prev current). If fn accepts two arguments, it will be invoked as (fn init-val #f current) for the first element of the iteration.

(df-shallow-copy (-> data-frame? data-frame?)) (valid-only (-> any/c boolean?)))

5 Statistics

The following functions allow calculating statistics on data frame series. They build on top of the math/statistics module.

A weight series needs to be used when samples in the data frame don’t have equal weight. For example, if a parameter (e.g. heart rate) is recorded at variable intervals, simply averaging the values will not produce an accurate average, if a timer series is also present, it can be used as a weight series to produce a better average.

6 Least Squares Fitting

start and stop specify the start and end position in the series, by default all values are considered for the fit.

mode determines the type of the function being fitted and can have one of the following values:

• 'linear – a function Y = a * X + b is fitted where ’a’ and ’b’ are fitted; this is equivalent of fitting a ’polynomial of degree 1 (see below)

• 'polynomial or 'poly – a polynomial Y = a0 + a1 * X + a2 * X^2 + ... is fitted. The degree of the polynomial is specified by the degree parameter, by default this is 2.

• 'exponential or 'exp – a function of Y = a * e ^ (b * X) + c is fitted. Note that this fit is not very good, and annealing needs to be used to improve it (see below)

• 'logarithmic or 'log – a function of type Y = a + b * ln(X) is fitted. This will only return a "real" fit function (as opposed to an imaginary one) if all values in YSERIES are positive

• 'power – a function of type Y = a * X ^ b is fitted. This will only return a "real" fit function (as opposed to an imaginary one) if all values in YSERIES are positive. Note that this fit is not very good, and annealing needs to be used to improve it (see below)

residual? when #t indicates that the residual value is also returned in the ‘least-squares-fit‘ structure. Setting it to #f will avoid some unnecessary computations.

annealing? when #t indicates that the fit coefficients should be further refined using the annealing function. This is only used for 'exponential or

'power fit functions as these ones do not produce "best fit" coefficients – I don’t know why, I am not a mathematician, I only used the formulas. Using annealing will significantly improve the fit for these functions, but will still not determine the best one. Note that the annealing algorithm is probabilistic, so applying it a second time on the same arguments will produce a slightly different result.

iterations represents the number of annealing iterations, see the #:iterations parameter to the ‘annealing‘ function.

7 Histograms and histogram plots

weight-series specifies the series to be used for weighting the samples. By default, it it uses the 'weight property stored in the data-frame, see df-set-default-weight-series. Use #f for no weighting, in this case, each sample will have a weight of 1.

bucket-width specifies the width of each histogram slot. Samples in the data series are grouped together into slots, which are from 0 to bucket-width, than from bucket-width to (* 2 bucket-width) and so on. The bucket-width value can be less than 1.0.

trim-outliers specifies to remove slots from both ends of the histogram that contain less than the specified percentage of values. When #f on slots are trimmed.

include-zeroes? specifies whether samples with a slot of 0 are included in the histogram or not. Note that slot 0 contains samples from 0 to bucket-width.

as-percentage? determines if the data in the histogram represents a percentage (all ranks add up to 100) or it is the rank of each slot.

In the resulting histogram, samples that are numbers or strings will be sorted. In addition, if the samples are numbers, empty slots will be created so that the buckets are also consecutive.

color determines the color of the histogram bars.

label specifies the label to use for this plot renderer.

skip and x-min are used to plot dual histograms, see histogram-renderer/dual.

All the above arguments are sent directly to the discrete-histogram

blank-some-labels?, controls if some of the labels are blanked out if the plot contains too many values, this can produce a nicer looking plot.

formatter controls how the histogram values are displayed. By default, labels for the values are displayed with ~a, but this function can be used for custom formatter. For example, if the values in the histogram represent running pace, the formatter can transform a value of 300 into the label "5:00".

label1 and color1 represent the label and colors for the first histogram, label2 and color2 represent the label and colors to use for the second histogram.

formatter controls how the histogram values are displayed. By default, labels for the values are displayed with ~a, but this function can be used for custom formatter. For example, if the values in the histogram represent running pace, the formatter can transform a value of 300 into the label "5:00".

formatter controls how the histogram values are displayed. By default, labels for the values are displayed with ~a, but this function can be used for custom formatter. For example, if the values in the histogram represent running pace, the formatter can transform a value of 300 into the label "5:00".

8 GPX Files

This module provides functions for reading and writing data frames using the GPX Exchange Format (GPX).

• "lat" and "lon" series representing the latitude and longitude of each point

• "timestamp" series representing the UTC timestamp in seconds for each point. The series will also be marked as sorted, if it is actually sorted

• "dst" representing a distance from the start. If distance data is not present in the GPX file, this series will be calculated from the GPX coordiantes. The series will be marked as sorted, if it is actually sorted

• "hr" representing heart rate measurements

• "cad" representing cadence measurements

• "pwr" representing power measurements, in watts

• "spd" representing the speed

The data frame will also have the following properties:

• a 'name property containing the name of the track segment, if this is present in the GPX file.

• a 'waypoints property containing a list of waypoints, if they GPX track has any. Each waypoint is represented as a list of TIMESTAMP, LAT, LON, ELEVATION and NAME

• a 'laps property containing a list of timestamps corresponding to each way point in the waypoint list – the laps property cannot be constructed correctly if the waypoints are missing a timestamp property.

All the track segments in the GPX file will be concatenated.

The entire GPS track is exported as a single track segment.

The laps property, if present, is assumed to contain a list of timestamps and the positions corresponding to these timestamps are exported as way points.

The name of the segment can be specified as the name parameter. If this is #f, the 'name property in the data frame is consulted, if that one is missing a default track name is used.

9 TCX Files

This module provides functions for reading Training Center XML (TCX) files into data frames.

• "lat" and "lon" series representing the latitude and longitude of each point

• "timestamp" series representing the UTC timestamp in seconds for each point. The series will also be marked as sorted, if it is actually sorted

• "dst" representing a distance from the start. If distance data is not present in the GPX file, this series will be calculated from the GPX coordiantes. The series will be marked as sorted, if it is actually sorted

• "hr" representing heart rate measurements

• "cad" representing cadence measurements

• "pwr" representing power measurements, in watts

• "spd" representing the speed

The data frame may also have the following properties (if they are present in the TCX document):

• 'unit-id the serial number of the device which recorded the activity.

• 'product-id the product id for the device that recorded the activity (indentifies the device type)

• 'sport the sport for the activity. This is a free form string, but TCX format usualy uses the strings "Running" for running activities and "Biking" for biking activities.

• a 'laps property containing a list of timestamps corresponding to the start of each lap in the activity.