---------------------------------------------------------------------- xe-readscore2 v 5: 20.September.2013 [JRH] ---------------------------------------------------------------------- Read two SCORE raw files, interlace output as ASCII or binary stream Non-numeric values will be recoded "NaN" USAGE: xe-readscore2 [input1] [input2] [options] [input1]: first file name (1 channel of data) [input2]: second file (1 channel of data) NOTE: input is pre-read to determine size VALID OPTIONS: -asc: ASCII output, 1=YES 0=NO, (USE BINX binary format) [1] -start: first block of 10-seconds to output[1] -stop: last block of 10-seconds to output (0 = all) [0] EXAMPLES: OUTPUT: ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-readxydt v 2: 21.January.2018 [JRH] ---------------------------------------------------------------------- Read binary position timestamps (.xydt) and position values (.xyd) - output is either converted to ASCII or kept in binary form - use a file or list of boundaries to screen the start-stop pairs - this program does not accept input piped via stdin USAGE: xe-readxydt [xydt] [xyd] [options] [xydt]: binary file containing position-times (long int) [xyd]: binary file containing position-values (3x float) VALID OPTIONS: -sf: sample-frequency of clock used to timestamp video [19531.2] -vf: video sample-frequency [25] -dejump: max cm/s, to invalidate jumpy x/y points [-1] -scrf: screen-file (binary ssp) defining inclusion bounds [] -scrl: screen-list (CSV) defining inclusion bounds [] -int: max interpolation gap (video-samples) [0] -1: no limit - interpolate all missing data 0: no interpolation for 25Hz video, 10-25 (0.4-1.0 sec) recommended -high: path smoothing (1/sec), 0=NONE [2.5] -verb: set verbocity of output (0=low, 1=high) [0] -out: output format [1]: 0= summary: duration, velocity mean & median 1= ASCII, one timestamp-id and x,y,d triplet per line 2= binary files x2 temp_xe-readxydt.xydt (long) temp_xe-readxydt.xyd (float triplets) 3= binary SSPs delineating velocity criteria (see below) Velocity criterion for -out 3 option ... -velint: time (sec) over which velocity is integrated [0.4] -velmin: minimum (cm/s, nan = no limit) [nan] -velmax: maximum (cm/s, nan = no limit) [nan] -veldur: minimum duration (s) [0] EXAMPLES: xe-readxydt data.xydt data.xyd -scr 1 -scrl 10,20,40,50 xe-readxydt A.xydt A.xyd -velmin 5 -out 3 > B.ssp ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-repeated1 v 5.May.2019 [JRH] ---------------------------------------------------------------------- Process repeated-measures data in blocks where key-values don't change - input should be a multi-column table with a header-line - one column holds the repeated measure (eg time), another the data - key-columns (eg "subject") define blocks of repeats - leading blank-lines and comments (#) will be ignored USAGE: xe-repeated1 [in] [keys] [rep] [dat] [options] [in]: file name or "stdin" [keys]: CSV list of column-names for static keys - equivalence from line-to-line defines the data block [rep]: column-name for repeated-measure variable [dat]: column-name for data OPTIONS (defaults in []): -interp: interpolate invalid values (0=NO 1=YES) [0] -max: max number of replicates in a block [1000] -verb: verbose output (0=NO 1=YES 999=DEBUG) [0] butterworth filter options: -low: low frequency limit, 0=SKIP [0] -high: high frequency limit, 0=SKIP [0] normalization options: -norm: normalization type (-1=SKIP, see xe-norm3): [-1] -n1: normalization zone (rep units) [0] -n2: end of normalization zone (rep units) [1] binning options: -bin: time (rep) over which to average data (0=SKIP) [0] EXAMPLES: xe-repeated1 data.txt subject,group,day time volts OUTPUT: columns [key1] [key2]... [rep] [dat] ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-rms2 v 2: 14.February.2014 [JRH] ---------------------------------------------------------------------- Calculate root mean square (RMS) power for a data series - RMS calculated in a sliding window shifted by one sample at a time - add-one/drop-one method use for efficient calculation - interpolates across non-numeric values, NAN and INF - option remove linear trends from data USAGE: xe-rms2 [input] [options] [input]: file name or "stdin", single column of numbers VALID OPTIONS: -ws: window size (samples) for RMS calc. (-1 = whole record) [-1] -d: detrend - removes linear trends from input (0=NO 1=YES) [0] -o: output (0=average RMS, 1=RMS at each sample [0] -v: set verbosity (0=RMS only, 1=report) [0] EXAMPLES: xe-rms2 data.txt cat temp.txt | xe-rms2 stdin -ws 100 ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-sizes1 v 2: 7.February.2014 [JRH] ---------------------------------------------------------------------- Programming tool: report size of numerical types ---------------------------------------------------------------------- char: 1 bytes 8 bits short: 2 bytes 16 bits int: 4 bytes 32 bits long: 8 bytes 64 bits long long: 8 bytes 64 bits size_t: 8 bytes 64 bits off_t: 8 bytes 64 bits float: 4 bytes 32 bits double: 8 bytes 64 bits long double: 16 bytes 128 bits max sizes as defined in <limits.h> and <float.h> headers: NOTE: all are for unsigned values, except size_t: char: -128 to 127 short: -32768 to 32767 int: -2147483648 to<PHONE_NUMBER> (2.15 GB) long: -9223372036854775808 to<PHONE_NUMBER>854775807 long long: -9223372036854775808 to<PHONE_NUMBER>854775807 size_t: 0 to -1 float: -3.40282e+38 to 3.40282e+38 double: -1.79769e+308 to 1.79769e+308 long double: -1.18973e+4932 to 1.18973e+4932 absolute numerical limits for given byte-sizes: 2^8: 256 2^16: 65536 2^32: 4.29497e+09 2^64: 9.22337e+18 x 2
---------------------------------------------------------------------- xe-smoothbox1 v 2: 14.December.2013 [JRH] ---------------------------------------------------------------------- Apply boxcar-averaging to data Sliding window averages values to either side of central data-point USAGE: xe-smoothbox1 [infile] [s] [infile]: data file or stdin [s]: samples either side of data to use for averaging [139673151853031] - total size of averaging window = s+s+1 EXAMPLES: xe-smoothbox1 data.txt 5 cat temp.txt | xe-smoothbox1 stdin OUTPUT: smoothed data - non-numeric values converted to NaN ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-smoothgauss1 v 3: 28.March.2016 [JRH] ---------------------------------------------------------------------- Apply Gaussian smoothing kernel to data Assumes data has a fixed sample-rate USAGE: xe-smoothgauss1 [infile][smooth] [infile]: data file or stdin - non-numeric data is ignored - ignores newlines [smooth]: size of smoothing window (samples) - must be an odd number - use "1" for no smoothing ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-smoothgauss2 v 1: 23.January.2021 [JRH] ---------------------------------------------------------------------- Apply Gaussian smoothing kernel to data in a column - input must be tab-delimited - assumes data has a fixed sample-rate - non-numeric data and blank lines ignored USAGE: xe-smoothgauss2 [in] [w] [options] [in]: input file, or "stdin" [w]: smoothing window (samples) - uneven numbers only - use "1" to specify no smoothing VALID OPTIONS: -cy : column-number containing data [1] -head : number of header-lines to pass unaltered [0] -p output precision (-1=auto (%f), 0=auto (%g), ɬ=decimals) [-1] ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-spearmans1 v 1: 17.July.2018 [JRH] ---------------------------------------------------------------------- Calculate the Spearman's rank-order correlation (rho) - non-parametric relatoinship between the rankings of two variables - relationship need not be linear, merely monotonic - similar to Pearson's r for eliptical distributions - less sensitive to the influence of outliers - non-numeric values ignored USAGE: xe-spearmans1 [input] [options] [input]: file name or "stdin" with x and y values in columns VALID OPTIONS: defaults in [] -cx: column holding x-value (first col = 1) [1] -cy: column holding y-value (first col = 1) [2] -verb: verbose output (0=NO 1=YES, 999=DEBUG) [1] EXAMPLES: xe-spearmans1 data.txt -cx 2 -cy 3 cat temp.txt | xe-spearmans1 stdin OUTPUT: rho= [Spearmans rank coefficient] n= [number of valid data-pairs]] F= [F-statistic] prob= [probability] ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-spectdenoise1 v 1: 22.January.2019 [JRH] ---------------------------------------------------------------------- Remove noise from a spectral-matrix (time x freq) - noise= single-timepoint increases that span many frequencies - timeseries for each freq. converted to Z-scores for thresholding - outputs a modified matrix with noise-timepoints set to NAN USAGE: xe-spectdenoise1 [in] [options] [in]: filename or "stdin", format: column=time row=freq VALID OPTIONS: defaults in [] -c: clipping value (de-noising only,-1=noclip) [-1] -z: Z-score threshold for noise at each freq (NAN=skip) [1] -s: sign of thesholding (-1=NEG,+1=POS,0=BOTH) [0] -p: % of freq > z needed to invalidate timepoint [25] - e.g. if Zɭ for 25% of the spectrum at column 123 -r: 90-deg rotate for analysis (0=NO 1=YES) [0] - use for if input column=freq and row=time - matrix will be rotated back for output OUTPUT: stdout: matrix with noise timepoints invalidated (NAN) stderr: summary EXAMPLES: xe-spectdenoise1 matrix1.txt 2> report.txt 1> matrix2.txt cat matrix1.txt | xe-spectdenoise1 stdin -r 1 > matrix2.txt ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-spectproc1 v 1: 30.May.2018 [JRH] ---------------------------------------------------------------------- Process neurophysiological power spectra - 1. adjusts spectrum: adaptive smoothing, more at higher frequencies - 2. adjusts spectrum: Butterworth HP filter to flatten the spectrum - 3. detects bands (if undefined) using +ive and -ive inflections - 4. gets stats (see below) for either orignal or adjusted spectrum USAGE: xe-spectproc1 [infile] [infile]: 2-column file or stdin in format [freq] [power] VALID OPTIONS: defaults in [] -bands: CSV band-triplets (start,mid,stop) [unset=auto] -out: output options [0] 0= files-only 1= spectrum 2= band-definitions and measurements 3= CSV list of band definition triplets (min,mid,max) -verb: verbose output (0=NO 1=YES) [0] ...spectral adjustment options... -div: divisor in width= freq/div (0=SKIP) [2] -filt: HP filter frequency (0=SKIP) [0.05] -useadj: use adjusted spectrum to calculate stats (0=NO 1=YES) [1] - 0 avoids negative AUCs - 1 avoids negative relative values on steep slopes OUTPUT: temp_xe-spectproc1_spect.txt: the adjusted spectrum temp_xe-spectproc1_bands.txt: stats for each band min mid max peak relpeak auc aucpos aucneg aucfull min: low-frequency boundary for the band mid: frequency at the band-peak max: high-frequency boundary for the band peak: absolute amplitude at the peak relpeak: amplitude relative to line joining min-max auc: half-width area-under-the-curve relative to min-max line aucpos: full AUC relative to zero- positive-values only aucneg: full AUC relative to zero- negative-values only aucful: full AUC relative to zero- -ive and +ive combined ----------------------------------------------------------------------
-------------------------------------------------------------------------------- xe-splitfile1 v 2: 28.November.2018 [JRH] -------------------------------------------------------------------------------- Split a file whenever zero is reached or if there is a time discontinuity USAGE: xe-splitfile1 [infile][options] [infile]: file or stdin with a time column and data column(s) VALID OPTIONS (defaults in []): -t: time column in infile [1] -c: criterion for split [-1] -1: whenever the timestamp is lower than the previous one 0: whenever zero is encountered ɬ: after this many seconds has elapsed EXAMPLE: xe-splitfile1 a.txt times.txt -t 3 OUTPUT: - a separate file for each block named temp_xe-splitfile1_[block].txt
---------------------------------------------------------------------- xe-statscol1 v 3: 1.April.2014 [JRH] ---------------------------------------------------------------------- Calculate statistics on columns of numbers Non-numeric values will be ignored USAGE: xe-statscol1 [input] [options] [input]: file with multiple columns of data" VALID OPTIONS: EXAMPLES: xe-statscol1 data.txt cut -f 1-3 temp.txt | xe-statscol1 stdin OUTPUT: col n mean stdev sem ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-statsd1v 12 14.September.2016 [JRH] ---------------------------------------------------------------------- Calculates summary statistics for an array of double-precision data Non-numeric values will be ignored INPUT: A single stream of data - rows or columns USAGE: xe-statsd1 [filename] [options] [filename]: file name or "stdin" VALID OPTIONS ( defaults in [] ): -f format of output: verbose (1) or single-line (0) [1] -out specify an output file or "stdout" (screen) [stdout] -var variance & mean calculation for large datasets [2] 1=computational, 2=per-sample with correction -per percentile calculation[0] 0=skip, 1=calculate mdeian and other percentiles EXAMPLES: xe-statsd1 data.txt -var 1 cat temp.txt | xe-statsd1 stdin OUTPUT: Statistics specified by keyword ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-statsgrp0 v 9: 16.November.2017 [JRH] ---------------------------------------------------------------------- Calculates summary statistics for a data set or grouped data Grouping variables must be numbers Non-numeric data, NAN, and INF will be excluded USAGE: xe-statsgrp0 [filename] [arguments] [filename]: file name or "stdin" [options]... -a: alpha-% level for conf.intervals (1,5,10) [5] -cg: column containing numerical grouping values [1] -cy: column containing data to be summarized [2] -mult: factor by which to multiply grouping values [1] NOTE: this allows setting decimal-precision for floating- point grouping values, which must be converted to integers. The multiplier can also optimize memory usage for when the grouping values are evenly-spaced timestamps. In this case, set -mult to 1/sample-interval -per percentile calculation (0=NO 1=YES) [0] -out: specify an output file or "stdout" (screen) [stdout] -h: output header labelling columns? (0=NO, 1=YES) [1] - examples: xe-statsgrp0 data.txt cat temp.txt | xe-statsgrp0 stdin -mult 10 - output: ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-statsgrp1 v 9: 29.April.2019 [JRH] ---------------------------------------------------------------------- Calculate stats on a data-column using grouping-columns - input must be tab-delimited - grouping-variables must be numeric (can be floating-point) - non-numeric data-values will be ignored for stats calculations USAGE: xe-statsgrp1 [input] [options] [input]: file name or "stdin" VALID OPTIONS: -cg1: column defining grouping-variable 1 [1] -cy: column containing dependent variable [2] -gint: output groups as integers? (0=NO 1=YES) [0] EXAMPLES: xe-statsgrp1 data.txt -cg1 5 cat temp.txt | xe-statsgrp1 stdin -gint 1 OUTPUT: grp1 n mean sd sem ntot NOTE: ntot= total datapoints for a given group-combination n= valid numbers contributing to statistical result ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-statsgrp2 v 9: 29.April.2019 [JRH] ---------------------------------------------------------------------- Calculate stats on a data-column using grouping-columns - input must be tab-delimited - grouping-variables must be numeric (can be floating-point) - non-numeric data-values will be ignored for stats calculations USAGE: xe-statsgrp2 [input] [options] [input]: file name or "stdin" VALID OPTIONS: -cg1: column defining grouping-variable 1 [1] -cg2: column defining grouping-variable 2 [2] -cy: column containing dependent variable [3] -gint: output groups as integers? (0=NO 1=YES) [0] EXAMPLES: xe-statsgrp2 data.txt -cg1 5 -cg2 7 cat temp.txt | xe-statsgrp2 stdin -gint 1 OUTPUT: grp1 grp2 n mean sd sem ntot NOTE: ntot= total datapoints for a given group-combination n= valid numbers contributing to statistical result ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-statsgrp3 v 9: 29.April.2019 [JRH] ---------------------------------------------------------------------- Calculate stats on a data-column using grouping-columns - input must be tab-delimited - grouping-variables must be numeric (can be floating-point) - non-numeric data-values will be ignored for stats calculations USAGE: xe-statsgrp3 [input] [options] [input]: file name or "stdin" VALID OPTIONS: -cg1: column defining grouping-variable 1 [1] -cg2: column defining grouping-variable 2 [2] -cg3: column defining grouping-variable 3 [3] -cy: column containing dependent variable [4] -gint: output groups as integers? (0=NO 1=YES) [0] EXAMPLES: xe-statsgrp3 data.txt -cg1 5 -cg2 7 -cg3 9 cat temp.txt | xe-statsgrp3 stdin -gint 1 OUTPUT: grp1 grp2 grp3 n mean sd sem ntot NOTE: ntot= total datapoints for a given group-combination n= valid numbers contributing to statistical result ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-statsrow1 v 3: 24.November.2018 [JRH] ---------------------------------------------------------------------- Calculate the stats on individual rows of numbers Non-numeric values will be ignored USAGE: xe-statsrow1 [input] [input]: file name or "stdin" VALID OPTIONS: -t(ype) of statistic to output: [3] 1=counts 2=sum 3=mean 4=standard deviation 5=maximum value 6=sample-number corresponding to maximum value EXAMPLES: OUTPUT: one statistic per row ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-strgroup1 v 1: 10.March.2019 [JRH] ---------------------------------------------------------------------- Group items in a list USAGE: xe-strgroup1 [in1] [options] [in1]: file name or "stdin", each input row = list of items VALID OPTIONS: defaults in [] -d1: input item-delimiters (multi-char) [, \t] -d2: output within-group item-delimiter (single-char) [#] -n: number of items in each group - does not span lines [2] EXAMPLES: echo "1,,3,4,5,6" | xe-strgroup1 stdin -n 2 -d2 "#" output: 1# 3#4 5#6OUTPUT: - for each input line, tab-delimited groups of [n] items ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-stripcomments v 1: 26.January.2018 [JRH] ---------------------------------------------------------------------- Remove comments from a file: shell, C/C++, or both types - all text between /* and */ - everything after the first # or // on a given line - ignores comment-markers contained within quotes - a blank line will remain if required, to preserve the linecount USAGE: xe-stripcomments [input] [options] [input]: file name or "stdin" VALID OPTIONS: defaults in [] -type: type of comments to remove [1] 0= none 1= # (shell single-line) 2= /* */ or // (C and C++ style block or single-line) 3= both 1 and 2 EXAMPLES: xe-stripcomments xs-myscript OUTPUT: - the original input with comments removed ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-strsub1 v 3: 14.May.2018 [JRH] ---------------------------------------------------------------------- Replace character-strings (find and replace) - no line-length limit - newlines automatically signify start of new words USAGE: xe-strsub1 [input] [old] [new] [input]: file name or "stdin" [old]: string to replace [new]: replacement string VALID OPTIONS: -in: subtitution only in lines containing this pattern -ex: exclude subtitution in lines containing this pattern NOTE: both unset by default (all lines analyzed) NOTE: -ex will override -in EXAMPLES: xe-strsub1 sdatafile " " "_" cat datafile | xe-strsub1 stdin ABC abc -ex "#" head datafile | grep Names | xe-strsub1 stdin Jones Janes OUTPUT: modified lines ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-strsub2 28.June.2019 [JRH] ---------------------------------------------------------------------- Replace all "words" in an input with something else - delimiters (tabs and/or spaces) converted to tabs - no line-length limit - newlines automatically signify start of new words USAGE: xe-strsub2 [input] [new] [input]: file name or "stdin" [new]: replacement string VALID OPTIONS: -cols: subtitution only specified column numbers (NULL=all) [(null)] -maxcols: set maximum column [32767] -in: subtitution only in lines containing this pattern -ex: exclude subtitution in lines containing this pattern NOTE: default= both unset (all lines analyzed) NOTE: -ex will override -in EXAMPLES: xe-strsub2 sdatafile " " "_" cat datafile | xe-strsub2 stdin NAN -ex "#" -cols 1,5-9 OUTPUT: modified lines, tab-delimited ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-strxmlparse1 v 6: 10.October.2018 [JRH] ---------------------------------------------------------------------- Parse XML input - extracts content between key tags Assumes values are bounded by <[key]> and </[key]> - example: <name>John Smith</name> USAGE: xe-strxmlparse1 [input] [key] [options] [input]: file name or "stdin" containing XML content [key]: XML key (or nested list) to extract - specify nesting using a CSV list of keys - do not include the brackets as these will be added - multiple matches allowed VALID OPTIONS: defaults in [] -ok: output key tags for matching blocks (0=NO 1=YES) [0] - for lowest nest-level only - allows output to be further parsed EXAMPLES: xe-strxmlparse1 data.xml group1,name OUTPUT: all content between the each match for each listed key(s) ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-template v 1: 14.April.2014 [JRH] ---------------------------------------------------------------------- Template program source-code Assumes one valid numeric value per input line Non-numeric values will be ignored USAGE: xe-template [infile] [options] [input]: file name or "stdin" VALID OPTIONS: defaults in [] -t(ype): 1(counts) 2(range 0-1) or 3(probability) -list: comma-separated list of numbers -verb: verbose output (0=NO 1=YES 999=DEBUG) [0] EXAMPLES: xe-template data.txt -t 1 cat temp.txt | xe-template stdin -t 3 OUTPUT: 1st column: lower limit of each bin 2nd column: value (eg. counts) in that bin ---------------------------------------------------------------------- *** xe-template [ERROR: this is the error message]
---------------------------------------------------------------------- xe-template DAY.MONTH.2020 [JRH] ---------------------------------------------------------------------- Template program source-code USAGE: xe-template [in] [options] [in]: file name or "stdin" VALID OPTIONS: defaults in [] -verb: verbose output (0=NO 1=YES 999=DEBUG) [0] EXAMPLES: xe-template data.txt OUTPUT: - ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-test1 v 1: 26.September.2016 [JRH] ---------------------------------------------------------------------- Perform a test on a pair of values USAGE: xe-test1 [val1] [test] [val2] [options] [val1]: value to compare [val2]: reference value [test]: test to apply -eq: equals -ne: not-equal -lt: less than -le: les than or equal to -gt: greater than -ge: greater than or equal to VALID OPTIONS: defaults in [] -long: treat inputs as long-integers (0=NO 1=YES [0]) - (0) treats input as double-precision float - (1) use long-integer math for greater precision EXAMPLES: Test if 1.23 is greater than 4.56: xe-test1 123 -gt 456 Test if 11111 is less than 99999 using long integers: xe-test1 11111 -lt 99999 -long 1 OUTPUT: 1: test is true nothing: test is false ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-timeconv1 v 2: 21.May.2016 [JRH] ---------------------------------------------------------------------- Convert hh:mm:ss time format to seconds or vice-versa Seconds output is seconds-past-midnight Assumes one time per input line Assumes colon is the delimiter if converting to seconds USAGE: xe-timeconv1 [input] [options] [input]: file name or "stdin" VALID OPTIONS: -f input format (1= hh:mm:ss, 2= seconds) [1]: EXAMPLES: xe-timeconv1 times.txt echo 12:59:59.123 | xe-timeconv1 stdin OUTPUT: one modified time per valid input line ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-timeconv2 1.Feb.2021 [JRH] ---------------------------------------------------------------------- Convert "%d/%m/%Y %H:%M:%S" times to seconds-since-1970 Assumes one time per input line, in the first column Blank lines and comments (#) will be passed unaltered USAGE: xe-timeconv2 [in] [options] [in]: file name or "stdin" VALID OPTIONS: -f : output format options [1]: 0: seconds elapsed since the first value read (zero) 1: seconds since 1 January 1970 EXAMPLES: xe-timeconv2 times.txt echo "31/12/2020 12:59:59.123" | xe-timeconv2 stdin OUTPUT: one modified time per valid input line ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-timestamp1 v 1: 8.March.2016 [JRH] ---------------------------------------------------------------------- Apply time-stamps to each line of a data series USAGE: xe-timestamp1 [input] [options] [input]: newline-delimited data file or "stdin" VALID OPTIONS: -dt: type of data (ascii or binary types) [-1] -1= ASCII 0-9= uchar,char,ushort,short,uint,int,ulong,long,float,double -sf: sampling-frequency (Hz) to emulate [1] -si: sampling-interval (seconds) to emulate [unset] NOTE: cannot set both -sf and -si -o: start-sample offset (seconds) [0] e.g. you may want timestamps to begin at -5 seconds -p: decimal precision (-1=auto (%f), 0=none(integer), ɬ=precision) [-1] -r: round down (0) or to the nearest integer (1) [0] NOTE: only applies to integer output (-p 0) -head: optional timestamp-header text for files with headers [unset] NOTE: only applies to ASCII input (-dt -1) NOTE: assumes first line of input is the header EXAMPLES: xe-timestamp1 data.txt -sf 1000 cat temp.txt | xe-timestamp1 stdin -sf 24000 OUTPUT: 1st column: time (seconds) 2nd column: data ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-timestamp2 v 1: 7.September.2015 [JRH] ---------------------------------------------------------------------- Generate time-stamps given a sample-frequency or interval USAGE: xe-timestamp2 [n] [samp] [n]: number of values to generate [samp]: sample-frequency or interval VALID OPTIONS: -f format (0=sample-frequency, 1=interval) [0] -o start-sample offset [0] -p decimal precision (-1=auto (%f), 0=none(integer), ɬ=precision) [-1] -r round down (0) or to the nearest integer (1) [0] NOTE: only applies to integer output (-p 0) EXAMPLES: xe-timestamp2 100 25.0 -f 0 -p 7 xe-timestamp2 100 0.04 -f 1 -p 0 -r 1 OUTPUT: single column of timestamps ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-transpose1 v 1.2: 11.October.2016 [JRH] ---------------------------------------------------------------------- - Transpose multiple columns into column-number [tab] column-contents - this program will handle long-line input USAGE: xe-transpose1 [input] [options] [input]: file name or "stdin" VALID OPTIONS: -start : first column to transpose NOTE: preceeding columns will be output for every line -n: output original column numbers (0=NO 1=YES) [1] [-d]: characters to use as column-delimiters - unset by default: multiple whitespace treated as one - if set, any matching delimiter marks a new column - suitable for files with "empty" columns EXAMPLES: xe-transpose1 data.txt cat temp.txt | xe-transpose1 stdin -start 3 -n 0 OUTPUT (by column): non-transposed columns (if -start defined) subsequent column numbers ( if -n is set to 1) subsequent column contents ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-transpose2 v 1: 14.August.2012 [JRH] ---------------------------------------------------------------------- - Transpose input to a single row (remove newlines) - Max input line length = 1000 USAGE: xe-transpose2 [input] [options] [input]: file name or "stdin" containing newline-separated data VALID OPTIONS: EXAMPLES: xe-transpose2 data.txt cat temp.txt | xe-transpose2 stdin OUTPUT: ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-transpose3 v 2: 9.July.2012 [JRH] ---------------------------------------------------------------------- - Actually de-transpose a column of data into multiple columns - For 2-column input <group><datum>, output a column for each <group> - Assumes data from a given group appears on successive rows (not mixed) - Max input line length = 1000 NOTE: group numbers are converted to integers Number of elements in first group is used as the default number - if subsequent groups have more elements they will be ignored - if subsequent groups have fewer elements the column is paded Number of elements in first group is used as the default number USAGE: xe-transpose3 [input] [options] [input]: file name or "stdin" in format <group><datum> VALID OPTIONS: EXAMPLES: xe-transpose3 data.txt cat temp.txt | xe-transpose3 stdin OUTPUT: 1st row: integer group numbers as defined in column-1 of input subsequent rows: data from each group in a separate column ...so... 100 1 100 2 100 3 200 4 200 5 200.5 6 300 7 300 8 ...becomes... 100 200 300 1 4 7 2 5 8 3 6 nan ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-transpose4 v 8: 4.April.2018 [JRH] ---------------------------------------------------------------------- - Transpose data for mixed-design repeated measures analysis - Grouping variables 1 & 2 must be finite numerical values - Repeated-measures variable (-cg3) can be NAN - Max input line length = 1000 USAGE: xe-transpose4 [input] [g1] [g2] [g3] [data] [input]: file name or "stdin" VALID OPTIONS: -cg1 column defining subject [1] -cg2 column defining between-subjects groups [2] -cg3 column defining repeated measure levels [3] -cy column containing dependent variable [4] -gint: output groups 1 & 2 as integers? (0=NO 1=YES) [0] -m: placeholder for non-numerical or non-finite values ["-"] -p: group3 (repeated) output precision [-1] -1=auto (%f), 0=auto (%g), ɬ=decimals NOTE: - if no between-subjects grouping column, set -cg2 to 0 - in this case, all subjects are placed in group "1" EXAMPLES: xe-transpose4 data.txt -cg1 1 -cg2 2 -cg3 3 -cy 7 cat temp.txt | xe-transpose4 stdin OUTPUT: subj grp r_1 r_2 r_3 r_4 etc... ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-trimoutliers1 v 2: 14.August.2012 [JRH] ---------------------------------------------------------------------- Trim a data stream to remove outliers Non-numeric values and NAN/INF are always removed USAGE: xe-trimoutliers1 [filename] [options] [filename]: file name or "stdin" OPTIONS ( defaults in [] ): -l lower percentile cutoff [0] -u upper percentile cutoff [100] EXAMPLES: xe-trimoutliers1 data.txt -l 10 -u 90 cat temp.txt | xe-trimoutliers1 stdin -l 25 -u 75 OUTPUT: trimmed and sorted dataset ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-winsplit1 v 1: 24.February.2014 [JRH] ---------------------------------------------------------------------- Split time-windows into smaller chunks Can accept window start-stop pairs or just start-times if -d is set Non-numeric values will be ignored USAGE: xe-winsplit1 [input] [options] [input]: file name or "stdin" * if -d =0, assumed format is [start] [stop] * if -d ɬ, assumed format is [start] (only reads first column) VALID OPTIONS: -dt: data type (7=long_integer, 8=double_precision_float) [8] -d: original duration of each window (if single-column input) [-1] -s: size of chunks to break windows down into [1] * best results if split is an integer fraction of the window size EXAMPLES: xe-winsplit1 data.txt -d 100 -s 20 cat temp.txt | xe-winsplit1 stdin -d -1 -s 1 OUTPUT: if -d =0: single column of revised [start] times if -d ɬ: [start] [stop] pairs ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-wint1 v 2: JRH, 10 October 2010 ---------------------------------------------------------------------- Calculates intervals between 2 events (g1, g2) in time-windows Windows can be: - specified in a file (use -wf option) - event-centred and of a specified width (use -w option) If -g1 and -g2 are the same, output is suitable for an autocorellogram NOTE: event times must be in ascending order - USAGE: xe-wint1 [input] [optional arguments] [input]: file name or "stdin" - 2 columns: 1st= time, 2nd= event-id (integer) - valid optional arguments... -g1: class-1 reference event identifier - default=1 -g2: class-2 event identifier - default=2 -w: windowsize in seconds - default=1.000000 - number of windows = number of events in class g1 - interval range is +- winsize/2, because windows are event-centred -wf: name a file listing window start and end times - unset by default - number of windows = number of lines in the window file - interval range is +- winsize NOTE: setting this variable overrides -w NOTE: windows must be sequential and non-overlapping - examples: xe-wint1 data.txt -wf list.txt -g1 6 -g2 7 cat temp.txt | xe-wint1 stdin -w 0.1 -g1 4 -g2 4 - output: time intervals (s) between events of each group ----------------------------------------------------------------------
---------------------------------------------------------------------- xe-writebinary1 v5: 16.September.2015 [JRH] ---------------------------------------------------------------------- Write an ASCII input to a simple binary file Input and output will be treated as a single numerical data type Multi-column input will be converted to a continous output stream Choose appropriate -dt option to determine binary data type (see below) Non-numeric data will be saved as 0 if -dt ɠ, or NAN if -dt =8 or 9 USAGE: xe-writebinary1 [input] [options] [input]: ASCII file name or "stdin" VALID OPTIONS, defaults in []: -dt: type of data to write [9] 0: unsigned char 1: signed char 2: unsigned short 3: signed short 4: unsigned int 5: signed int 6: unsigned long 7: signed long 8: float 9: double -v set verbose mode (0=data only, 1=stderr-report, 2=logfile) [0] EXAMPLES: OUTPUT: Binary stream ----------------------------------------------------------------------
DESCRIPTION: - Calculate the area under the curve using the trapezoid method - https://en.wikipedia.org/wiki/Integral - http://en.wikipedia.org/wiki/Trapezoidal_rule - single input array - assumes evenly sampled data - requires at least two valid points for meaningfull results - options: [ref] defines magnitude (y) at each point 0: reference to zero (standard definition of AUC) 1: reference to a line joining the ends - distance from this line instead of original y-value USES: - measuring the effect-size of a drug response - measuring power in a region of an FFT spectrogram - calculation of population-spike for slice LTP experiments DEPENDENCY TREE: No dependencies ARGUMENTS: double *curvey : input array holding y-values long nn : length of curvex and curvey (number of elements) double interval : fixed interval between curvey samples int ref : y-value reference, 0 or 1 (see definition above) double *result : pre-allocated array holding results (must hold 4 elements) char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error result[0] = total AUC (positive + negative) result[1] = positive AUC result[2] = negative AUC SAMPLE CALL: GET NEGATIVE-ONLY AUC FOR 100 POINTS, REFERENCED TO ZERO x= xf_auc1_d(datay,1000,.001,0,result,message) if(x!=0) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } else auc=result[0];
DESCRIPTION: - Calculate the area under the curve using the trapezoid method - https://en.wikipedia.org/wiki/Integral - http://en.wikipedia.org/wiki/Trapezoidal_rule - assumes unevenly sampled data - requires at least two valid points for meaningfull results - options: [ref] defines magnitude (y) at each point 0: reference to zero (standard definition of AUC) 1: reference to a line joining the ends - distance from this line instead of original y-value USES: - measuring the effect-size of a drug response - measuring power in a region of an FFT spectrogram - calculation of population-spike for slice LTP experiments DEPENDENCY TREE: No dependencies ARGUMENTS: float *curvey : input array holding y-values long nn : length of curvex and curvey (number of elements) double interval : fixed interval between curvey samples int ref : y-value reference, 0 or 1 (see definition above) double *result : pre-allocated array holding results (must hold 4 elements) char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error result[0] = total AUC (positive + negative) result[1] = positive AUC result[2] = negative AUC SAMPLE CALL: GET NEGATIVE-ONLY AUC FOR 100 POINTS, REFERENCED TO ZERO x= xf_auc1_f(datay,1000,.001,0,result,message) if(x!=0) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } else auc=result[0];
DESCRIPTION: - Calculate the area under the curve using the trapezoid method - https://en.wikipedia.org/wiki/Integral - http://en.wikipedia.org/wiki/Trapezoidal_rule - dual input arrays: x (time) and value (y) - suitable for unevenly sampled data - requires at least two valid points for meaningfull results - options: [ref] defines magnitude (y) at each point 0: reference to zero (standard definition of AUC) 1: reference to a line joining the ends - distance from this line instead of original y-value USES: - measuring the effect-size of a drug response - measuring power in a region of an FFT spectrogram - calculation of population-spike for slice LTP experiments DEPENDENCY TREE: No dependencies ARGUMENTS: double *curvex : input array holding x-values double *curvey : input array holding y-values long nn : length of curvex and curvey (number of elements) int ref : y-value reference, 0 or 1 (see definition above) double *result : pre-allocated array holding results (must hold 4 elements) char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error result[0] = total AUC (positive + negative) result[1] = positive AUC result[2] = negative AUC SAMPLE CALL: GET NEGATIVE-ONLY AUC FOR 100 POINTS, REFERENCED TO ZERO x= xf_auc2_d(datax,datay,100,0,result,message) if(x!=0) fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } else auc=result[2];
DESCRIPTION: - Calculate the area under the curve using the trapezoid method - https://en.wikipedia.org/wiki/Integral - http://en.wikipedia.org/wiki/Trapezoidal_rule - requires at least two valid points for meaningfull results - suitable for unevenly sampled data - options: [ref] defines magnitude (y) at each point 0: reference to zero (standard definition of AUC) 1: reference to a line joining the ends - distance from this line instead of original y-value USES: - measuring the effect-size of a drug response - measuring power in a region of an FFT spectrogram - calculation of population-spike for slice LTP experiments DEPENDENCY TREE: No dependencies ARGUMENTS: double *curvex : input array holding x-values double *curvey : input array holding y-values long nn : length of curvex and curvey (number of elements) int ref : y-value reference, 0 or 1 (see definition above) double *result : pre-allocated array holding results (must hold 4 elements) char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error result[0] = total AUC (positive + negative) result[1] = positive AUC result[2] = negative AUC SAMPLE CALL: GET NEGATIVE-ONLY AUC FOR 100 POINTS, REFERENCED TO ZERO x= xf_auc2_d(datax,datay,100,0,result,message) if(x!=0) fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } else auc=result[2];
DESCRIPTION: - reduce data to a fixed number of averaged bins - non-finite values (INF, NAN) do not contribute to the averages - allows for different numbers of elements to go into different bins to evenly spread the results (fractional bin-widths) - allows definition of a "zero" sample which is guaranteed to be the first sample in the bin corresponding with the new "zero" - guarantees no bins will be under-sampled - once the bin-size is determined, edge bins will contain between 1x and just-under-2x bindwidth samples USES: Downsampling data DEPENDENCY TREE: No dependencies ARGUMENTS: double *data : pointer to input array, which will be overwritten size_t *setn : number of elements in data (overwritten - pass as address) size_t *setz : element to treat as "zero" (overwritten - pass as address) size_t setbins : desired length of data after binning (number of bins used for averaging) char *message : array to hold error message RETURN VALUE: - size of bins used for averaging or 0 on fail - setz and setn are also updated - can be used to reconstruct timestamps SAMPLE CALL: char message[256]; double data[19]; double aa,bb,binsize,sampinterval=1; size_t ii,setn=19, setz=6; for(ii=0;ii<setn;ii++) { data[ii]=(double)ii; printf("%g\n",data[ii]); } printf("zero=item number %ld\n",setz); printf("\n"); binsize= xf_bin1a_f(data,&setn,&setz,3,message); if(binsizeɘ) {fprintf(stderr,"*** %s\n",message); exit(1);} aa=(double)(setz)*(-1)*binsize*sampinterval; bb=binsize*sampinterval; for(ii=0;ii<setn;ii++) { printf("%g\t%f\n",aa,data[ii]); aa+=bb; } printf("new zero=item number %ld\n",setz);
DESCRIPTION: - reduce data to a fixed number of averaged bins - non-finite values (INF, NAN) do not contribute to the averages - allows for different numbers of elements to go into different bins to evenly spread the results (fractional bin-widths) - allows definition of a "zero" sample which is guaranteed to be the first sample in the bin corresponding with the new "zero" - guarantees no bins will be under-sampled - once the bin-size is determined, edge bins will contain between 1x and just-under-2x bindwidth samples USES: Downsampling data DEPENDENCY TREE: No dependencies ARGUMENTS: float *data : pointer to input array, which will be overwritten size_t *setn : number of elements in data (overwritten - pass as address) size_t *setz : element to treat as "zero" (overwritten - pass as address) size_t setbins : desired length of data after binning (number of bins used for averaging) char *message : array to hold error message RETURN VALUE: - size of bins used for averaging or 0 on fail - setz and setn are also updated - can be used to reconstruct timestamps SAMPLE CALL: char message[256]; float data[19]; double aa,bb,binsize,sampinterval=1; size_t ii,setn=19, setz=6; for(ii=0;ii<setn;ii++) { data[ii]=(float)ii; printf("%g\n",data[ii]); } printf("zero=item number %ld\n",setz); printf("\n"); binsize= xf_bin1a_f(data,&setn,&setz,3,message); if(binsizeɘ) {fprintf(stderr,"*** %s\n",message); exit(1);} aa=(double)(setz)*(-1)*binsize*sampinterval; bb=binsize*sampinterval; for(ii=0;ii<setn;ii++) { printf("%g\t%f\n",aa,data[ii]); aa+=bb; } printf("new zero=item number %ld\n",setz);
DESCRIPTION: - reduce data to averaged bins of a fixed size - non-finite values (INF, NAN) do not contribute to the averages - allows for different numbers of elements to go into different bins to evenly spread the results (fractional bin-widths) - allows definition of a "zero" sample which is guaranteed to be the first sample in the bin corresponding with the new "zero" - guarantees no bins will be under-sampled - edge bins will contain between 1x and just-under-2x bindwidth samples USES: Downsampling data DEPENDENCY TREE: No dependencies ARGUMENTS: double *data : pointer to input array, which will be overwritten long *setn : number of elements in data (overwritten - pass as address) long *setz : element to treat as "zero" (overwritten - pass as address) double setbinsize : desired bin-width (samples - can be a fraction) char *message : array to hold error message RETURN VALUE: - status (0=success, -1=fail) - setz and setn are also updated - can be used to reconstruct timestamps SAMPLE CALL: char message[256]; double data[19]; double aa,bb,nbins,sampinterval=1,setbinsize=3.5; long ii,setn=19, setz=6; for(ii=0;ii<setn;ii++) { data[ii]=(double)ii; printf("%g\n",data[ii]); } printf("setz=item number %ld\n",setz); printf("\n"); nbins= xf_bin1b_d(data,&setn,&setz,setbinsize,message); if(nbins==0) {fprintf(stderr,"*** %s\n",message); exit(1);} aa=(double)(setz)*(-1)*setbinsize*sampinterval; bb=setbinsize*sampinterval; for(ii=0;ii<setn;ii++) { printf("%g\t%f\n",aa,data[ii]); aa+=bb; } printf("new setz=item number %ld\n",setz);
DESCRIPTION: - reduce data to averaged bins of a fixed size - non-finite values (INF, NAN) do not contribute to the averages - allows for different numbers of elements to go into different bins to evenly spread the results (fractional bin-widths) - allows definition of a "zero" sample which is guaranteed to be the first sample in the bin corresponding with the new "zero" - guarantees no bins will be under-sampled - edge bins will contain between 1x and just-under-2x bindwidth samples USES: Downsampling data DEPENDENCY TREE: No dependencies ARGUMENTS: float *data : pointer to input array, which will be overwritten long *setn : number of elements in data (overwritten - pass as address) long *setz : element to treat as "zero" (overwritten - pass as address) double setbinsize : desired bin-width (samples - can be a fraction) char *message : array to hold error message RETURN VALUE: - status (0=success, -1=fail) - setz and setn are also updated - can be used to reconstruct timestamps SAMPLE CALL: char message[256]; float data[19]; double aa,bb,nbins,sampinterval=1,setbinsize=3.5; long ii,setn=19, setz=6; for(ii=0;ii<setn;ii++) { data[ii]=(flaot)ii; printf("%g\n",data[ii]); } printf("setz=item number %ld\n",setz); printf("\n"); nbins= xf_bin1b_f(data,&setn,&setz,setbinsize,message); if(nbins==0) {fprintf(stderr,"*** %s\n",message); exit(1);} aa=(double)(setz)*(-1)*setbinsize*sampinterval; bb=setbinsize*sampinterval; for(ii=0;ii<setn;ii++) { printf("%g\t%f\n",aa,data[ii]); aa+=bb; } printf("new setz=item number %ld\n",setz);
DESCRIPTION: - reduce data to averaged bins of a fixed size - non-finite values are impossible for short-integer input - allows for different numbers of elements to go into different bins to evenly spread the results (fractional bin-widths) - allows definition of a "zero" sample which is guaranteed to be the first sample in the bin corresponding with the new "zero" - guarantees no bins will be under-sampled - edge bins will contain between 1x and just-under-2x bindwidth samples USES: Downsampling data DEPENDENCY TREE: No dependencies ARGUMENTS: short *data : pointer to input array, which will be overwritten long *setn : number of elements in data (overwritten - pass as address) long *setz : element to treat as "zero" (overwritten - pass as address) double setbinsize : desired bin-width (samples - can be a fraction) char *message : array to hold error message RETURN VALUE: - status (0=success, -1=fail) - setz and setn are also updated - can be used to reconstruct timestamps
DESCRIPTION: Averages data in non-overlapping time-windows NAN values do not contribute to the sum Alters input array, shortening overall length to (n-windows) worth of average values Time for each window corresponds with the beginning of each window NAN values will be ignored, but INF will affect the results USES: DEPENDENCY TREE: No dependencies ARGUMENTS: RETURN VALUE: SAMPLE CALL:
DESCRIPTION: Bin an array of data using non-overlapping windows, using peak-detection based on xe-bin1.6 Overwrites the original data Ignores NAN or INF values NOTE: this function does not have the offset-zero capability of xf_bin1_f, that is, there is no guarantee that any bin will be left-aligned with "zero", unless this corresponds with the first element of the input dataa array USES: Downsample a time-series with peak-detection to avoid losing significant features DEPENDENCY TREE: No dependencies ARGUMENTS: float *data1 : array holding data values long n : number of elements in the array double binsize : size (number of elements) in the binning window - note that this can be fractional RETURN VALUE: new size of binned data array, or -1 on error SAMPLE CALL: n= xf_binpeak1_f(data,n,100,message); if(n>=0) for(i=0;i<n;i++) printf("%g\n",data_f[i]); else fprintf(stderr,"%s\n",message);
DESCRIPTION: Bin an array of ydata, and output the max deviations in each bin (window) - NAN values will tend not to affect min or max - INF values will obviously affect max! Overwrites the original xdata and ydata arrays, beginning at element zero Changes this version (xf_binpeak4) - outputs values at the start & midpoint of each window - no setmid option, since peak-detection must output 2 points for every window anyway - use simple min/max algorithm instead of calculating deviation from mean USES: Downsample a time-series with peak-detection to avoid loosing significant features DEPENDENCY TREE: No dependencies ARGUMENTS: double *xdata : array holding xdata values double *ydata : array holding ydata values long n : number of elements in the xdata and ydata arrays double winwidth : size of the binning window in units of [xdata] RETURN VALUE: Number of elements in the revised xdata & ydata arrays -1 if not enough memory to hold the temporary array -2 if the window size is too large ( total xdata range < 2*winwidth ) SAMPLE CALL:
DESCRIPTION: Keep a subset of timestamps based on block-definitions, collapsing across the removed blocks Block-definitions in this case are in the form of start-stop pairs USES: Adjust data to remove noisy sections Example: - an original time series of 0-10 in which a spike occurs at sample 8 - say we want ti remove samples 2-6 due to noise - this is equivalent of keeping 2 blocks, defined using start-stop pairs 0,2 and 7,11 - in the adjusted data, sample-8 is now sample-3 original compressed _________ * * 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 DEPENDENCY TREE: No dependencies ARGUMENTS: long *samplenum : input holding timestamps (sample-numbers) needing realignment - assumes they are in ascending order short *class : input holding clssifier for each timestamp long nn : the size of the samplenum array long *bstart : array holding the start-samples (included) for each kept block of the original data long *bstop : array holding the stop-samples (not included) for each kept block long nblocks : the number of blocks char *message : pre-allocated array to hold error message RETURN VALUE: new sample-number array-length on success, -1 on error samplenum array will be adjusted char array will hold message (if any) SAMPLE CALL:
DESCRIPTION: Restore original timestamps based on block-definitions previously used to keep only subsections of a dataset Block-definitions in this case are in the form of start-stop pairs USES: Say an original data series was adjusted to remove noisy data If spike events are identified in the modified time-series, adjustment is required to put the spike times back in register with the original time series Example: - an original time series of 0-10 in which a spike occurs at sample 8 - say samples (2-6) were removed due to noise - this is equivalent of keeping 2 blocks, defined using start-stop pairs 0,2 and 7,11 - spike detection on the compressed data would show spike detection at sample #3 - this function will correct for compression, restoring the original timestamps original compressed corrected _________ * * * 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 0 1 7 8 9 10 DEPENDENCY TREE: No dependencies ARGUMENTS: long *samplenum : input holding timestamps (sample-numbers) needing realignment - assumes they are in ascending order long nn : the size of the samplenum array long *bstart : array holding the start-samples (included) for each kept block of the original data long *bstop : array holding the stop-samples (not included) for each kept block long nblocks : the number of blocks char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error samplenum array will be adjusted char array will hold message (if any) SAMPLE CALL:
DESCRIPTION: Compare funtion to be used in conjunction with qsort This version compares double-precision floating point numbers USES: When calling qsort, include this (or similar) function as the 4th argument DEPENDENCY TREE: No dependencies ARGUMENTS: const void *a : pointer to first element in comparison const void *b : pointer to second element in comparison RETURN VALUE: an integer determining if "a" if less than (-1) equal to (0) or greater than (1) "b" SAMPLE CALL: qsort(array,n,sizeof(double),xf_compare1_d);
DESCRIPTION: Compare funtion to be used in conjunction with qsort This version compares floating point numbers USES: When calling qsort, include this (or similar) function as the 4th argument DEPENDENCY TREE: No dependencies ARGUMENTS: const void *a : pointer to first element in comparison const void *b : pointer to second element in comparison RETURN VALUE: an integer determining if "a" if less than (-1) equal to (0) or greater than (1) "b" SAMPLE CALL: qsort(array,n,sizeof(float),xf_compare1_f);
DESCRIPTION: Compare funtion to be used in conjunction with qsort This version compares long integers USES: When calling qsort, include this (or similar) function as the 4th argument DEPENDENCY TREE: No dependencies ARGUMENTS: const void *a : pointer to first element in comparison const void *b : pointer to second element in comparison RETURN VALUE: an integer determining if "a" if less than (-1) equal to (0) or greater than (1) "b" SAMPLE CALL: qsort(array,n,sizeof(long),xf_compare1_l);
DESCRIPTION: Compare funtion to be used in conjunction with qsort This version compares long integers USES: When calling qsort, include this (or similar) function as the 4th argument DEPENDENCY TREE: No dependencies ARGUMENTS: const void *a : pointer to first element in comparison const void *b : pointer to second element in comparison RETURN VALUE: an integer determining if "a" if less than (-1) equal to (0) or greater than (1) "b" SAMPLE CALL: qsort(array,n,sizeof(long),xf_compare1_l);
DESCRIPTION: Convolute two input signals Given signal 1 (X) and signal 2 (Y), this algorithm effectively slides a reversed version of Y[] along X[], multiplying each Y by the corresponding X, provided it exists, and summing the products to create a result. Example: signal1 (X) consists of seven numbers (0,1,2,3,4,5,6), so nX=7 signal2 (Y) consists of three numbers (0,1,2), so nY=3 The first row shows X[] Subsequent rows show Y[], reversed & shifted along X[] The result for each X is the sum (looking back) of Y*X At the end of X[] two additional values can be computed Total results therefore = nX+(nY-1) = in this case 7+(3-1) = 9 However, you could ignore the last nY-1 results Missing values denoted by "_" _ _ 0 1 2 3 4 5 6 _ _ 2 1 0 - result[0] = 2x_ + 1x_ + 0x0 2 1 0 - result[1] = 2x_ + 1x0 + 0x1 2 1 0 - result[2] = 2x0 + 1x1 + 0x2 2 1 0 - result[3] = 2x1 + 1x2 + 0x3 ... 2 1 0 - result[6] = 2x4 + 1x5 + 0x6 2 1 0 - result[7] = 2x5 + 1x6 + 0x_ 2 1 0 - result[8] = 2x6 + 1x_ + 0x_ USES: Filtering sig1 using DSP parameters in sig2 DEPENDENCY TREE: No dependencies ARGUMENTS: float *sig1 : pointer to input array sig1 (X) size_t n1 : length of sig1 float *sig2 : pointer to input signal#1 (X) size_t n2 : length of sig2 char message[] : pointer to array to hold messages RETURN VALUE: pointer to a results array with n1+(n2-1) elements NULL on failure SAMPLE CALL: #define XDIM 320 #define HDIM 60 float *xf_conv1_f(float *sig1, size_t n1, float *sig2, size_t n2,char message[] ); int main (int argc, char *argv[]) { size_t ii; float fX[XDIM], fY[XDIM], fH[HDIM], *result=NULL; // DEFINE THE INPUT SIGNAL for (ii = 0; ii < XDIM; ii++) fX[ii]=0; for (ii = HDIM; ii < XDIM; ii++) fX[ii] = sin(2.0 * M_PI * (float)ii / HDIM) + sin(2.0 * M_PI * (float)ii / 10.0); // DEFINE IMPULSE RESPONSE (MAX NO. OF TERMS = HDIM) for (ii = 0; ii < HDIM; ii++) fH[ii]=0; for (ii = 1; ii <= 10; ii++) fH[ii] = 0.1; result= xf_conv1_f(fX,XDIM,fH,HDIM,message); for(ii=0;ii<(n1+(n2-1));ii++) printf("%d %f\n",ii,result[ii]); free(result); exit(0); }
DESCRIPTION: Convolute two complex input signals See xf_conv1_f for general principals USES: Filtering sig1 using DSP parameters in sig2 DEPENDENCY TREE: No dependencies ARGUMENTS: complex float *sig1 : pointer to input array sig1 (X) size_t n1 : length of sig1 complex float *sig2 : pointer to input signal#1 (X) size_t n2 : length of sig2 char message[] : pointer to array to hold messages RETURN VALUE: pointer to a results array with n1+(n2-1) elements NULL on failure SAMPLE CALL:
DESCRIPTION: This fills a "results" array with details from a Pearson's correlation Requires two parallel arrays of floating point values Returns the number of valid sample-pairs included in the analysis
DESCRIPTION: Calculate the Pearson's correlation and return details including F and p statistics - this version for double-precision floating-point values of x and y Fills a "results" array with details from a Pearson's correlation Allows definition of an arbitrary "invalid" value to be ignored Assumes all data is stored in memory For faster processing with less analysis detail, use xf_correlate_simple_d USES: Detailed correlation statistics on parallel arrays of data DEPENDENCIES: float xf_prob_F(float F,int df1,int df2) ARGUMENTS: double *x : input, x-data array double *y : input, y-data array long nn : input, number of elements in x & y double setinv : input, user-specified invalid value (typically NAN, but may be other) double *result : output, pre-allocated array to hold results - must allow at least 18 elements char *message : output, pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error result array will hold statistics char array will hold message (if any) if r==0: check message (minimum correlation) - likely a vertical or horizontal line probability of "nan" will confirm this if F==99: indicates that r was nearly exxactly -1 or +1: F is arbitrarily assigned a large value SAMPLE CALL: x= xf_correlate_d(x,y,n,-1,result,message); if(x==-1) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } else if (result[1]==0) { fprintf(stderr,"*** %s/%s\n\n",thisprog,message);}
DESCRIPTION: Calculate the Pearson's correlation and return details including F and p statistics - this version for floating-point values of x and y Fills a "results" array with details from a Pearson's correlation Allows definition of an arbitrary "invalid" value to be ignored Assumes all data is stored in memory For faster processing with less analysis detail, use xf_correlate_simple_d USES: Detailed correlation statistics on parallel arrays of data DEPENDENCIES: float xf_prob_F(float F,int df1,int df2) ARGUMENTS: float *x : input, x-data array float *y : input, y-data array long nn : input, number of elements in x & y float setinv : input, user-specified invalid value double *result : output, pre-allocated array to hold results - must allow at least 18 elements char *message : output, pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error result array will hold statistics char array will hold message (if any) if r==0: check message (minimum correlation) - likely a vertical or horizontal line probability of "nan" will confirm this if F==99: indicates that r was nearly exxactly -1 or +1: F is arbitrarily assigned a large value SAMPLE CALL: x= xf_correlate_f(x,y,n,-1,result,message); if(x==-1) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } else if (!isfinite(result[1])) { fprintf(stderr,"*** %s/%s\n\n",thisprog,message);}
DESCRIPTION: Calculate the Pearson's correlation and return details including F and p statistics - this version for integer values of x and y Fills a "results" array with details from a Pearson's correlation Allows definition of an arbitrary "invalid" value to be ignored Assumes all data is stored in memory For faster processing with less analysis detail, use xf_correlate_simple_d USES: Detailed correlation statistics on parallel arrays of data DEPENDENCIES: float xf_prob_F(float F,int df1,int df2) ARGUMENTS: int *x : input, x-data array int *y : input, y-data array long nn : input, number of elements in x & y int setinv : input, user-specified invalid value double *result : output, pre-allocated array to hold results - must allow at least 18 elements char *message : output, pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error result array will hold statistics char array will hold message (if any) if r==0: check message (minimum correlation) - likely a vertical or horizontal line probability of "nan" will confirm this if F==99: indicates that r was nearly exxactly -1 or +1: F is arbitrarily assigned a large value SAMPLE CALL: x= xf_correlate_i(x,y,n,-1,result,message); if(x==-1) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } else if (!isfinite(result[1])) { fprintf(stderr,"*** %s/%s\n\n",thisprog,message);}
DESCRIPTION: Calculate the Pearson's correlation and return details including F and p statistics - this version for long integer values of x and y NOTE THAT RETURN VALUE IS NGOOD, NOT ZERO AS FOR SOME OTHER CORRELATION FUNCTIONS - reason: cannot rely on double result[] to accurately record long-integer value of NGOOD Fills a "results" array with details from a Pearson's correlation Allows definition of an arbitrary "invalid" value to be ignored Assumes all data is stored in memory For faster processing with less analysis detail, use xf_correlate_simple_d USES: Detailed correlation statistics on parallel arrays of data DEPENDENCIES: float xf_prob_F(float F,int df1,int df2) ARGUMENTS: int *x : input, x-data array int *y : input, y-data array long nn : input, number of elements in x & y int setinv : input, user-specified invalid value double *result : output, pre-allocated array to hold results - must allow at least 18 elements char *message : output, pre-allocated array to hold error message RETURN VALUE: good pairs used in correlation, -1 on error result array will hold statistics char array will hold message (if any) if r==0: check message (minimum correlation) - likely a vertical or horizontal line probability of "nan" will confirm this if F==99: indicates that r was nearly exxactly -1 or +1: F is arbitrarily assigned a large value SAMPLE CALL: x= xf_correlate_i(x,y,n,-1,result,message); if(x==-1) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } else if (!isfinite(result[1])) { fprintf(stderr,"*** %s/%s\n\n",thisprog,message);}
DESCRIPTION: Calculate the Pearson's correlation and returns r - this version for double-preceision floating-point values of x and y Fills a "results" array with slope, and intercept Ignores non-numeric values Nan and Inf Should only be used if there are 4 or more data points USES: Simple correlation statistics on parallel arrays of data DEPENDENCIES: None ARGUMENTS: double *x : input, x-data array double *y : input, y-data array long nn : input, number of elements in x & y double *result : output, pre-allocated array to hold results - must allow at least 3 elements RETURN VALUE: Pearson's r (or NAN or 0 if there was a problem) if r==0, it may be because the data represented a vertical or horizontal line result array will hold slope and intercept SAMPLE CALL: r= xf_correlate_simple_d(x,y,n,result);
DESCRIPTION: Calculate the Pearson's correlation and returns r - this version for floating-point values of x and y Fills a "results" array with slope, and intercept Ignores non-numeric values Nan and Inf Should only be used if there are 4 or more data points USES: Simple correlation statistics on parallel arrays of data DEPENDENCIES: None ARGUMENTS: float *x : input, x-data array float *y : input, y-data array long nn : input, number of elements in x & y double *result : output, pre-allocated array to hold results - must allow at least 3 elements RETURN VALUE: Pearson's r (or NAN or 0 if there was a problem) if r==0, it may be because the data represented a vertical or horizontal line result array will hold slope and intercept SAMPLE CALL: r= xf_correlate_simple_f(x,y,n,result);
DESCRIPTION: Calculate the Pearson's correlation and returns r - this version for integer values of x and y Fills a "results" array with slope, and intercept Non-numeric values, Nan and Inf should be removed before calling this function Should only be used if there are 4 or more data points USES: Simple correlation statistics on parallel arrays of data DEPENDENCIES: None ARGUMENTS: int *x : input, x-data array int *y : input, y-data array long nn : input, number of elements in x & y double *result : output, pre-allocated array to hold results - must allow at least 3 elements RETURN VALUE: Pearson's r (or NAN or 0 if there was a problem) if r==0, it may be because the data represented a vertical or horizontal line result array will hold slope and intercept SAMPLE CALL: r= xf_correlate_simple_i(x,y,n,result);
DESCRIPTION: Calculate the Pearson's correlation and returns r - this version for integer values of x and y Fills a "results" array with slope, and intercept Non-numeric values, Nan and Inf should be removed before calling this function Should only be used if there are 4 or more data points USES: Simple correlation statistics on parallel arrays of data DEPENDENCIES: None ARGUMENTS: int *x : input, x-data array int *y : input, y-data array long nn : input, number of elements in x & y double *result : output, pre-allocated array to hold results - must allow at least 3 elements RETURN VALUE: Pearson's r (or NAN or 0 if there was a problem) if r==0, it may be because the data represented a vertical or horizontal line result array will hold slope and intercept SAMPLE CALL: r= xf_correlate_simple_i(x,y,n,result);
DESCRIPTION Use a look-up table to find the critical value from the Student's t-distribution http://easycalculation.com/statistics/t-distribution-critical-value-table.php ARGUMENTS long df : degrees of freedom, (1-200) float alpha : desired significance level (.05, .02, .01, .005, .002, .001) int tails : number of tails for the test (1,2) NOTE: cannot provide 1-tailed critical value for alpha=0.1 RETURN VALUE The critical t-value, or a negative number signifying an error -1: df is less than 1 -2: inappropriate alpha -3: inappropriate number of tails
DESCRIPTION: Calculate statistics on a curve (single time-series, assumes fixed interval) COM: - Gerrard,et al., The Journal of Neuroscience (2008). 28(31):7883.7890 - centre of mass (mathematical mean of observations) - this version uses a shifted copy of the curve (all-positive values) to avoid shifting mass to negative values when "y" is negative - the result is that for some curves the COM may not be exactly where you think it should be - example: for a single sine-cycle, the upturn at the end will shift the COM to the right of the middle MEDIAN: - mid-point of summed observations - also uses the positive-shifted dataset, as per COM TEMPORAL BIAS: - Skaggs & McNaughton (1996). Science 271(5257): 1870-1873 - normalized as per Gerrard et. al 2008 - a measure normalized of asymmetry in the curve USES: - characterization of a time-series DEPENDENCY TREE: No dependencies ARGUMENTS: double *curvey : input holding y-values long nn : length of curvey double interval: interval between successive values of curvey double *result_d : pre-allocated array to hold results - should allow at least 16 elements char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error result array will hold statistics char array will hold message (if any) result_d[0]= xymin; x-value at the trough result_d[1]= ymin; y-value at the trough result_d[2]= xymax; x-value at the peak result_d[3]= ymax; y-value at the peak result_d[4]= median; x-value dividing summed y-values in half result_d[5]= com; centre of mass - i.e. the mean observation
DESCRIPTION: Calculate statistics on a x/y curve - similar to xf_curvestats1_d but allows specification of x-values, and hence uneven sampling if necessary - also includes a temporal bias measure designed for detecting asymmetry in histograms COM: - Gerrard,et al., The Journal of Neuroscience (2008). 28(31):7883.7890 - centre of mass (mathematical mean of observations) - this version uses a shifted copy of the curve (all-positive values) to avoid shifting mass to negative values when "y" is negative - the result is that for some curves the COM may not be exactly where you think it should be - example: for a single sine-cycle, the upturn at the end will shift the COM to the right of the middle MEDIAN: - mid-point of summed observations - also uses the positive-shifted dataset, as per COM TEMPORAL BIAS: - Skaggs & McNaughton (1996). Science 271(5257): 1870-1873 - normalized as per Gerrard et. al 2008 - a measure normalized of asymmetry in the curve USES: - characterization of a time-series or histogram DEPENDENCY TREE: No dependencies ARGUMENTS: double *curvex : input holding x-values double *curvey : input holding y-values long nn : length of curvex and curvey double *result_d : pre-allocated array to hold results - should allow at least 7 elements char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error result array will hold statistics char array will hold message (if any) result_d[0]= xymin; x-value at the trough result_d[1]= ymin; y-value at the trough result_d[2]= xymax; x-value at the peak result_d[3]= ymax; y-value at the peak result_d[4]= median; x-value dividing summed y-values in half result_d[5]= com; centre of mass - i.e. the mean observation result_d[6]= bias; normalized temporal bias (d2-score)
DESCRIPTION:John Huxter: 17 February 2015 - based on xf_wavewidth.c, which was designed to detect compound waveform width for action potentials A function to find the peak and width in a curve Peak can be positive or negative Width is the width at the point at which a percentage of the peak is crossed For compound curves (multiple channels in sequence) the width of the channel containing the peak is used USES: DEPENDENCY TREE: No dependencies ARGUMENTS: float *curve : input array of values representing a curve int nn : number of samples in the curve int nchan : if a compound curve, the number of channels (eg. 4 for a tetrode waveform) float thresh : threshold as a percentage of the max or min value (see sign, below) int sign : whether thresholdhing should be set to maximum (1) or minimum (-1) float *results : pre-allocated array to hold the results of the analysis char *message : pre-allocated array to hold error message RETURN VALUE: index to the detected peak -1 on error result array will hold statistics result[0]=peak value result[1]=width at threshold result[2]=threshold used char array will hold message (if any) SAMPLE CALL:
DESCRIPTION: - convert a date to the day- or week-in-the-year - weeks are presumed to begin with Monday USES: - convert a date to a number useable for plotting results DEPENDENCY TREE: No dependencies ARGUMENTS: int sety ; year (e.g. 2000) int setm : month-in-year (e.g. 12) int setd : day-in-month (e.g. 28) int setconv : convert to week (1) or day (2) char *message : pre-allocated array to hold error message RETURN VALUE: on success: week-in-year (0-52) or day-in-year (1-365) on error: -1 message array will be filled if there was an error SAMPLE CALL: char message[256]; int z,year=2000,month=12,day=31; z= xf_dateconv1(year,month,day,1,message); if(zɬ) printf("week-in-year=%d\n",z); else { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); }
DESCRIPTION: Adjust a date/time array by adding or subtracting a number of days, hours, minutes or seconds - should accurately reflect lengths of months, leap years, etc USES: DEPENDENCY TREE: No dependencies ARGUMENTS: int *input : pre-allocated 6-item array input array input[0]: year input[1]: month input[2]: day input[3]: hour input[4]: minute input[5]: second int adjust : adjustment to apply to one of the values char *field : the field of to adjust (day,hour,min, or sec) char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error input array will be adjusted SAMPLE CALL: To get the date/time 90 minutes before 1 January 2000 10:30:15 AM char message[256]; int i,x, mydate[6]; mydate[0]=2000; mydate[1]=1; mydate[2]=1; mydate[3]=10; mydate[4]=30; mydate[5]=15; x= xf_datemod1(mydate,-90,"min",message)
DESCRIPTION: Convert a date-string into integer year month and day values - can convert different date-formats into the three digits for constructing the actual date - all input values must be numeric (e.g. December= 12, not DEC or similar) - years represented as two-digits will not be expanded (eg. 99 does not become 1999) DEPENDENCIES: None ARGUMENTS: char *date1 : date1, date-string in one of the following numeric formats... int format : date1, format (1-4) 1: dd/mm/yyyy 2: mm/dd/yyyy 3: yyyy/mm/dd 4: yyyymmdd int *year : output (pass to function as address): year int *month : output (pass to function as address): month int *day : output (pass to function as address): day char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error year, month and day will be assigned char array will hold message (if any) SAMPLE CALL: char *date1="12/31/2000\0" int year,month,day; z= xf_dateparse1(date1,2,year,month,day,message); if(z==-1) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } <TAGS> string time </TAGS>
DESCRIPTION: Keep every "nth" point in an input (float) - can accept fractional decimation factors USES: achieve exact downsampling of an input DEPENDENCIES: none ARGUMENTS: float *data : input, array long ndata : input, length of input array double winsize : the size of the decimation window (eg. 10.5) char *message : pre-allocated array to hold error message RETURN VALUE: number of points in decimated reults or -1 on error SAMPLE CALL: # to downsample a 10s input sampled at 500 Hz to 400 Hz # - n= 10*500 = 5000 # - downsample factor is 500/400= 1.25 n= xf_decimate_f(data,n,1.25,message); if(nɘ) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } # note that n should be 4000
DESCRIPTION: Keep every "nth" point in an input (short) - can accept fractional decimation factors USES: achieve exact downsampling of an input DEPENDENCIES: none ARGUMENTS: short *data : input, array long ndata : input, length of input array double winsize : the size of the decimation window (eg. 10.5) char *message : pre-allocated array to hold error message RETURN VALUE: number of points in decimated reults or -1 on error SAMPLE CALL: # to downsample a 10s input sampled at 500 Hz to 400 Hz # - n= 10*500 = 5000 # - downsample factor is 500/400= 1.25 n= xf_decimate_s(data,n,1.25,message); if(nɘ) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } # note that n should be 4000
DESCRIPTION: - build a matching set of frequency-band arrays by parsing an input string - input string = comma-separated triplets of name,start,stop - example: delta,.5,4,theta,4,12) NOTE: 16-band maximum USES: - define band names and boundaries for programs analysing frequency bands DEPENDENCIES: None ARGUMENTS: char *setbands : input string of CSV triplets e.g. "delta,0.5,4,theta,4,12" - will be modified float[16] bstart : output, preallocated array to hold 64 band start-values float[16] bstop : output, preallocated array to hold 64 band start-values long *btot : output, value holding the number of bands (max 16) char *message : pre-allocated array to hold error message RETURN VALUE: Success: - return pointer to array of indices to band-names - setbands array will be compressed to include only the bandnames, not the values - bstart[16] and bstop[16] will be filled with the start & stop values for each band - btot will be updated to reflect the number of bands Failure: NULL, message array will hold explanatory text (if any) SAMPLE CALL: char setbands[]= "theta,.5,4,theta,4,12,beta,12,30,gamma,30,100"; long btot=0,*ibands=NULL; float bstart[16],bstop[16]; ibands= xf_definebands1(setbands,bstart,bstop,&btot,messsage); if(ibands==NULL) { fprintf(stderr,"\n\t%s/%s\n\n",thisprog,message); exit(1); } for(ii=0;ii<btot;ii++) printf("%s\t%g\t%g\n",(setbands+iband[ii]),bstart[ii],bstop[ii]); ... if(ibands!=NULL) free(ibands); <TAGS> string spectra </TAGS>
DESCRIPTION: Remove jumpy points from an x/y position series - unlike filtering, will not alter the posx outside the "jumpy" events - points are invalidated (assigned to NAN) until one is found that doesn't look jumpy relative to the last good point - note that once enough time has elapsed, a series of displaced points can be considered non-jumpy again. USES: Remove rapid changes in apparent position due to transient video tracking problems (reflection-tracking etc) Can help to remove noise artefacts DEPENDENCY TREE: No dependencies ARGUMENTS: float *posx : pointer to array holding x-position data float *posy : pointer to array holding y-position data long nn : number of elements in the arrays double sfreq : sample-frequency of input (samples per second) double thresh : the threshold for rate-of-change, beyond which posx & posy are reassigned to NAN RETURN VALUE: Number of jumpy points invalidated No error return for this function SAMPLE CALL: # in a video data record sampled t 25Hz, invalidate movement > 200cm/s njumps= xf_dejump2_f(posx, posy, nn, 25.0, 200.0)
DESCRIPTION: Apply a sliding boxcar-average normalization to a data series - local mean is subtracted from each datum Acts as a high-pass filter Uses a single-sample-step sliding window, so output is same length as input Data at the beginning and end of the input is adjsted by a fraction of the window size not less than 1/2 NOTE: if input contains invalid values, NAN or INF, output will be invalid - in such cases it is recommended to interpolate the input before passing to this function USES: Signal analysis DEPENDENCIES: None ARGUMENTS: double *input : pointer to array holding amplitude time series to be normalized long nn : length of the input & output arrays long nwin1 : length of averaging window, +1 if even - this value will be increased by 1 if not an odd number, or decreased if this exceeds the data length char *message : feedback returned to the calling function, which should allocate memory for this array (256 characters) RETURN VALUE: 0 on success, input[] will be overwritten -1 on error message array will hold explanatory text (if any)
DESCRIPTION: Apply a sliding boxcar-average normalization to a data series - local mean is subtracted from each datum Acts as a high-pass filter Uses a single-sample-step sliding window, so output is same length as input Data at the beginning and end of the input is adjsted by a fraction of the window size not less than 1/2 NOTE: if input contains invalid values, NAN or INF, output will be invalid - in such cases it is recommended to interpolate the input before passing to this function USES: Signal analysis DEPENDENCIES: None ARGUMENTS: float *input : pointer to array holding amplitude time series to be normalized long nn : length of the input & output arrays long nwin1 : length of averaging window, +1 if even - this value will be increased by 1 if not an odd number, or decreased if this exceeds the data length char *message : feedback returned to the calling function, which should allocate memory for this array (256 characters) RETURN VALUE: 0 on success, input[] will be overwritten -1 on error message array will hold explanatory text (if any)
DESCRIPTION: Apply a sliding boxcar-average normalization to a data series - local mean is subtracted from each datum Acts as a high-pass filter Uses a single-sample-step sliding window, so output is same length as input Data at the beginning and end of the input is adjsted by a fraction of the window size not less than 1/2 NOTE: if input contains invalid values, NAN or INF, output will be invalid - in such cases it is recommended to interpolate the input before passing to this function USES: Signal analysis DEPENDENCIES: None ARGUMENTS: short *input : pointer to array holding amplitude time series to be normalized long nn : length of the input & output arrays long nwin1 : length of averaging window, +1 if even - this value will be increased by 1 if not an odd number, or decreased if this exceeds the data length char *message : feedback returned to the calling function, which should allocate memory for this array (256 characters) RETURN VALUE: 0 on success, input[] will be overwritten -1 on error message array will hold explanatory text (if any)
DESCRIPTION: - for a point in a series, return the point minus the mean of a window centred on the point - acts as a high-pass filter - can be called iteratively to de-mean an entire series USES: - eliminate long-term trends in the data - adaptive "de-meaning", where the window varies according to the position n the array DEPENDENCIES: None ARGUMENTS: float *input : input: pointer to array of values long nn : length of input long halfwin : half-length of averaging window, must be ɬ long index : index to position in input on which the window is centred char *message : feedback returned to the calling function, which should allocate memory for this array (256 characters) RETURN VALUE: success: de-meaned value error: NAN message array will hold explanatory text (if any)
DESCRIPTION: Calculate the density of discrete events in non-overlapping windows Assigns memory for the density array Very similar to xf_smoothbox1_d USES: - convert spike-times into a firing-rate timecourse - Convert a list of event times into a density time series DEPENDENCIES: None ARGUMENTS: long *etime : input array of event times (sample numbers) - assumed to be in ascending order long nn : total number of events long min : minimum event time: -1= events[0] long max : maximum event time: -1= events[nevents-1] double winsize: size of the sliding window (samples, fractional if necessary) in which to calculate density long nwin : result: number of density-windows generated char *message : arrray to hold message in the event of an error RETURN VALUE: on success: - pointer to the density array (time series) - nwin will be updated to reflect size of density array on error: NULL SAMPLE CALL: density= xf_density1_l(time1,nn,0,-1,winsize,&nwin,message); if(density==NULL) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); }
DESCRIPTION: Calculate the density (counts) defined by start-stop pairs of blocks of samples, using non-overlapping windows Assigns memory for the density array Very similar to xf_smoothbox1_d, derived from xf_density1_1 USES: - convert lost.ssp packet-loss definitions into a packet-loss time-series DEPENDENCIES: None ARGUMENTS: long *start1 : input array of start-times (sample numbers) - assumed to be in ascending order long *stop1 : input array of stop-times (sample numbers) - assumed to be in ascending order long nn : total number of events long min : minimum event time: -1= events[0] long max : maximum event time: -1= events[nevents-1] double winsize: size of the sliding window (samples, fractional if necessary) in which to calculate density long nwin : result: number of density-windows generated char *message : arrray to hold message in the event of an error RETURN VALUE: on success: - pointer to the density array (counts, time series) - nwin will be updated to reflect size of density array on error: NULL SAMPLE CALL: density= xf_density2_l(start1,stop1,nn,0,-1,winsize,&nwin,message); if(density==NULL) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); }
DESCRIPTION: Determine the cumulative occurance of x/y values rounded to fill a matrix of fixed dimensions - note that x/y zero will correspond to the upper left corner of the matrix WARNING: do not use for repeated calls assign memory for the same array - consider using xf_densitymatrix2_l instead USES: - create a density matrix of dwell-times from a series of x-y position data - create an action-potential density matrix from the x/y positions associated with cell firing DEPENDENCY TREE: No dependencies ARGUMENTS: float *xdata : input holding x-position values data float *ydata : input holding y-position values data long nn : number of x/y pairs in the input long setxbintot : desired width of the matrix (must be ɬ) long setybintot : desired height of the matrix (must be ɭ) char *message : pre-allocated array to hold error message RETURN VALUE: pointer to matrix of setxbintot x setybintot values, or NULL on error char array will hold message (if any) SAMPLE CALL: long *matrix=NULL; width=3; height=5; matrix= xf_densitymatrix1_f(xdata,ydata,nn,width,height,message); if(matrix==NULL) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } for(ii=0;ii<(width*height);ii++) { printf("%ld\t",matrix[ii]); if((ii+1)%width=0) printf("\n"); }
DESCRIPTION: Determine the cumulative occurance of x/y values rounded to fill a matrix of fixed dimensions - note that x/y zero will correspond to the upper left corner of the matrix - this version is safe for repeated calls - memory for output assigned by calling function USES: - create a density matrix of dwell-times from a series of x-y position data - create an action-potential density matrix from the x/y positions associated with cell firing DEPENDENCY TREE: No dependencies ARGUMENTS: float *xdata : input holding x-position values data float *ydata : input holding y-position values data long nn : input number of x/y pairs in the input long *matrix : output preallocated array able to hold output of size (setxbintot x setybintot) long setxbintot : input desired width of the matrix (must be ɬ) long setybintot : input desired height of the matrix (must be ɭ) float *ranges : input pointer to ranges[4] array defining xmin,ymin,xmax,ymax, or NANs for auto-setting each range based on *xdata and *ydata char *message : onput pre-allocated array to hold error message RETURN VALUE: status flag (0=OK, -1=ERROR) matrix array will be overwritten message array will hold message (if any) SAMPLE CALL: long matrix[15]; width=3; height=5; z= xf_densitymatrix1_f(xdata,ydata,nn,matrix,width,height,message); if(z=-1) { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); } for(ii=0;ii<(width*height);ii++) { printf("%ld\t",matrix[ii]); if((ii+1)%width=0) printf("\n"); }
DESCRIPTION: Detect start peak & stop times for cycles (oscillations) in data Detection is on the trough (inflection) Peak-point is taken as the highest value Differences from xf_detectcycles1_f: - no sine-fit test of fit - above is presumed to have been handled by a preceeding filtering step - addition of peak(peak) output USES: DEPstopENCY TREE: No depstopencies ARGUMENTS: float *data : pointer to input (time series) size_t ndat : number of elements in the data array size_t cmin : minimum duration of cycle to be included size_t cmax : maximum duration of cycle to be included size_t **cstart : address to array to hold results (cycle start-times) - calling function must free() size_t **cpeak : address to array to hold results (cycle peak(peak)-times) - calling function must free() size_t **cstop : address to array to hold results (cycle stop-times) - calling function must free() size_t *cycletot: address to variable (not an array) holding the total cycles detected RETURN VALUE: 0 on success, -1 on failure SAMPLE CALL:
DESCRIPTION: Detect events in a series when a threshold is exceeded For each supra-threshold point, the function looks outward until the data drops below the half-threshold Edges of events (stop-start= width) are defined by the edge-threshold Data must drop below half-threshold before another event can be detected No event will be detected until data is found which does NOT exceed threshold USES: DEPENDENCY TREE: No dependencies ARGUMENTS: float *data : input holding data long n1 : size of data array float thresh : data sample must exceed this value for event detection to begin float edge ; percentage of threshold to determine edge of the event (e.g. 0.5) int sign : sign of detection (-1,0,+1 : -1=lessthan +1=greaterthan, 0=either) long nmin : minimum number of samples for which the half-threshold must be exceeded (0=no limit) long nmax : maximum number of samples for which the half-threshold must be exceeded (0=no limit) long **estart : address for results array (event start-samples) - calling function must free() long **estop : address for results array (event stop-samples) - calling function must free() char *message : arrray to hold message in the event of an error RETURN VALUE: number of detected events (>=0) on success -1 on error result array will have been assigned memory and will hold event start and stop samples SAMPLE CALL: # detect events where signal is less than three for between 200 and 2000 samples long *estart=NULL,*estop=NULL; nevents= xf_detectevents1_f(data,n,3,-1,200,2000,&estart,&estop,message); if(neventsɘ) {fprintf(stderr,"\n--- Error: %s\n\n",message); exit(1);} if(estart!=NULL) free(estart); if(estop!=NULL) free(estop)
DESCRIPTION: Detect events in a series when a threshold is exceeded - must meet a duration criterion Edges of events (stop-start= length) are defined by the edge-threshold Data must drop below edge-threshold before another event can be detected No event will be detected until data is found which does NOT exceed threshold This function allocates memory for the results NOTE: Data should be high-pass filtered and/or z-scored before running this! Changes from xf_detectevents1_f - upper threshold accepted to allow event rejection - output now includes a measure of the peak-time within each event, instead of just the start & stop times USES: DEPENDENCY TREE: No dependencies ARGUMENTS: float *data : input holding data long n1 : size of data array float thresh : data sample must exceed this value for event detection to begin float upper : data within an event must remain below this multiple of the threshold - exceeding this value invalidates the event float edge ; data must drop below this multiple of the threshold to determine the edge of the event (e.g. 0.5) int sign : sign of detection (-1,0,+1 : -1=lessthan +1=greaterthan, 0=either) long nmin : minimum number of samples for which the half-threshold must be exceeded (0=no limit) long nmax : maximum number of samples for which the half-threshold must be exceeded (0=no limit) long **estart : address for results array (event start-samples) - calling function must free() long **epeak : address for results array (event peak-samples) - calling function must free() long **estop : address for results array (event stop-samples) - calling function must free() char *message : arrray to hold message in the event of an error - calling function must allocate memory RETURN VALUE: number of detected events (>=0) on success -1 on error result array will have been assigned memory and will hold event start and stop samples SAMPLE CALL: # detect events where signal is less than three for between 200 and 2000 samples long *estart=NULL,*estop=NULL; nevents= xf_detectevents2_f(data,n,3,10,.5,-1,200,2000,&estart,&epeak,&estop,message); if(neventsɘ) {fprintf(stderr,"\n--- Error: %s\n\n",message); exit(1);} if(estart!=NULL) free(estart); if(epeak!=NULL) free(epeak); if(estop!=NULL) free(estop)
DESCRIPTION: Define start-stop sample-pairs (SSP) defining periods (events) when data falls within a range This is slightly different from detectevents1&2, which are geared towards peak and edge thresholds and define a time for the peak as well - must meet a duration criterion defined by tstamp - data must fall out-of-range before another event can be detected - this function allocates memory for the results - NOTE: for each start-stop pair... start = first sample meeting the criteria stop = first sample NOT meeting the criteria thereafter (ie the sample after the last sample in the block) hence if start=stop, duration of event is zero UPDATE: 20.July.2016: fix adjustment to stop if last sample is good - DO NOT adjust the actual timestamp of the last sample! USES: - define epochs when velociity meets a criteria DEPENDENCY TREE: No dependencies ARGUMENTS: long *tstamp : input timestamps used to asess duration of events, and to define start-stop pairs float *data : input data long n1 : size of data array float min : minimum data value for inclusion (nan = no limit = FLT_MIN) float max : minimum data value for inclusion (nan = no limit = FLT_MAX) long mindur : minimum duration of event (0 = no limit - should be in tstamp units) long **estart : address for results array (event start-samples) - calling function must free() long **estop : address for results array (event stop-samples) - calling function must free() char *message : arrray to hold message in the event of an error - calling function must allocate memory RETURN VALUE: number of detected events, >=0 on success -1 on error result arrays estart and estop will have been assigned memory and will hold event start and stop samples SAMPLE CALL: # detect events where data is less than 5 for at least 10seconds, and timestamps reflected data sampled at 20k long kk,*estart=NULL,*estop=NULL; double min=nan, max=5, duration=10, samprate=20000; kk=(long)(duration*samprate); nevents= xf_detectevents2_f(tstamp,data,n1,min,max,kk,&estart,&estop,message); if(neventsɘ) {fprintf(stderr,"\n--- Error: %s\n\n",message); exit(1);} if(estart!=NULL) free(estart); if(estop!=NULL) free(estop)
DESCRIPTION: Detect inflections in a series Assumes signal is filtered to reduce number of inflections to a reasonable number USES: DEPENDENCY TREE: No dependencies ARGUMENTS: double *data : input holding data long n1 : size of data array long **itime : address for time-stamp array (samples) - calling function must free() int **isign : address for inflection-sign array (-1=negative, 1=positive) - calling function must free() char *message : arrray to hold message in the event of an error RETURN VALUE: number of detected inflections (>=0) on success -1 on error result arrays *itime and *isign will have been assigned memory and will hold event times (sample) and sign (-1 or +1) SAMPLE CALL: #include <stdio.h> #include <stdlib.h> #include <math.h> int main (int argc, char *argv[]) { char message[1000]; long *itime=NULL,n1=350,n2,ii; int *isign=NULL; double data[1000]; double aa=M_PI/10; for(ii=0;ii<n1;ii++) data[ii]= sin(ii*aa); for(ii=0;ii<n1;ii++) printf("%ld %g\n",ii,data[ii]); n2= xf_detectinflect1_f(data,n1,&itime,&isign,message); if(n2ɘ) {fprintf(stderr,"\n--- Error: %s\n\n",message); exit(1);} for(ii=0;ii<n2;ii++) printf("%ld\t%d\n",itime[ii],isign[ii]); if(itime!=NULL) free(itime); if(isign!=NULL) free(isign); }
DESCRIPTION: Detect inflections in a series Assumes signal is filtered to reduce number of inflections to a reasonable number USES: DEPENDENCY TREE: No dependencies ARGUMENTS: float *data : input holding data long n1 : size of data array long **itime : address for time-stamp array (samples) - calling function must free() int **isign : address for inflection-sign array (-1=negative, 1=positive) - calling function must free() char *message : arrray to hold message in the event of an error RETURN VALUE: number of detected inflections (>=0) on success -1 on error result arrays *itime and *isign will have been assigned memory and will hold event times (sample) and sign (-1 or +1) SAMPLE CALL: #include <stdio.h> #include <stdlib.h> #include <math.h> int main (int argc, char *argv[]) { char message[1000]; long *itime=NULL,n1=350,n2,ii; int *isign=NULL; float data[1000]; double aa=M_PI/10; for(ii=0;ii<n1;ii++) data[ii]= sin(ii*aa); for(ii=0;ii<n1;ii++) printf("%ld %g\n",ii,data[ii]); n2= xf_detectinflect1_f(data,n1,&itime,&isign,message); if(n2ɘ) {fprintf(stderr,"\n--- Error: %s\n\n",message); exit(1);} for(ii=0;ii<n2;ii++) printf("%ld\t%d\n",itime[ii],isign[ii]); if(itime!=NULL) free(itime); if(isign!=NULL) free(isign); }
DESCRIPTION Remove linear trends from a data series USES ARGUMENTS double *y : pointer to array holding input time-series, constant intervals assumed size_t nn : number of elements in y[] double *result_d : array to hold results, initialized by calling function result_d[0]= intercept result_d[1]= slope result_d[2]= Pearson's r RETURN VALUE 0 on success 1 on failure (impossible at present)
DESCRIPTION Remove linear trends from a data series USES ARGUMENTS float *y : pointer to array holding input time-series, constant intervals assumed size_t nn : number of elements in y[] double *result_d : array to hold results, initialized by calling function result_d[0]= intercept result_d[1]= slope result_d[2]= Pearson's r RETURN VALUE 0 on success 1 on failure
DESCRIPTION: Simple error-handling function (prints error message and exits) DEPENDENCY TREE: No dependencies ARGUMENTS: char *message : input message to print before exit RETURN VALUE: none SAMPLE CALL: x= xf_auc1_d(data, nn, interval, result, ); if(x!=0) xf_err1("PROG,NAME","this is the error",1);
DESCRIPTION: Align events to the inflection nearest the midpoint Calling function specifies whether inflection should be positive or negative Calling function specifies the maximum allowable shift USES: Prevent possible cancellation of peaks and troughs when oscillatory events are averaged DEPENDENCY TREE: No dependencies ARGUMENTS: float *data : the raw data containing events- should be filtered (smooth) in the event frequency range long ndata : the number of samples in data long *etime : sample-numbers in data representing event-times - typically the middle of the event long nevents : the number of events in etime int sign : detect nearest positive (1) or negative (-1) inflection long shiftmax : largest allowable shift (samples) char *message : arrray to hold message in the event of an error - calling function must allocate memory RETURN VALUE: 0 on success, -1 on error SAMPLE CALL:
DESCRIPTION: Expand a series of numbers so there are a user-defined number of elements [setn] Data is duplicated as required Alters input array NAN values will be ignored, but INF will affect the results USES: - modifying different series of data so they are all the same length - making an irregular matrix of data have rows all the same length DEPENDENCY TREE: No dependencies ARGUMENTS: double *data: input, array of numbers to be resampled long nn: input, number of elements in data long setn: input, the new number of elements requested for data char *message : pre-allocated array to hold error message RETURN VALUE: Pointer to modified data array - NULL if memory allocation error occurs SAMPLE CALL:
DESCRIPTION: Expand a series of numbers so there are a user-defined number of elements [setn] Data is duplicated as required Alters input array NAN values will be ignored, but INF will affect the results USES: - modifying different series of data so they are all the same length - making an irregular matrix of data have rows all the same length DEPENDENCY TREE: No dependencies ARGUMENTS: double *data: input, array of numbers to be resampled long nn: input, number of elements in data long setn: input, the new number of elements requested for data char *message : pre-allocated array to hold error message RETURN VALUE: Pointer to modified data array - NULL if memory allocation error occurs SAMPLE CALL:
DESCRIPTION: Returns the size of a file in bytes - compatible with large files sizes (ɮGB) NOTE: for windows, use xf_filesize_win.cpp instead
DESCRIPTION: Fill target array (valB) with interpolated values from source array (valA), based on where the target array times (timeB) fall relative to the source array (timeA) Invalid data points only assigned to B where it is out of range of A (beginning & end) This function is like hux_fillinterp except - times are unsigned long integers - a maximum number of invalid samples across which to interpolate is defined - totA and totB are long integers - invalid is a float This function is NOT appropriate for data with large number of invalid samples (eg. theta phase) - in this case, use similar routine, hux_fillprev instead USES: Allows mapping of data from the source array onto a different time-series of samples E.g. assign interpolated position values to a series of neuronal action potential times DEPENDENCY TREE: No dependencies ARGUMENTS: long unsigned int *timeA : input timestamps of the source array long unsigned int *timeB : input time-stamps of the target array float *valA : input data values, interpolated versions of which are sent to valB float *valB : output data values int totA : total number of values in the source array int totB : total number of values in the target array int invalid : value to treat as invalid (typically -1) int max_invalid : maximum permissible number of invalid points to allow interpolation accross char *type : data type, "linear" or "circular" (0-360) RETURN VALUE: The number of interpolated values generated (excluding invalid samples at start and end)
DESCRIPTION: - Fill target array (B) with interpolated values from source array (A) - Uses timestamps for each array to find A's which fall before & after each B - Assigns interpolated value of A to B - Invalid data points only assigned to B where it is out of range of A (beginning & end) - NOT appropriate for data with large number of invalid samples - derived from hux_fillinterp.c and xf_fillinterp_itime.c - however, this function insists interpolation must use samples either side of the target, not the previous sample and one which coincidentally has the same timestamp as the target - also, user does not define an invalid value - NAN is taken as invalid USES: - assigning low-sample-rate position values to high-sample-rate action-potiential data DEPENDENCY TREE: No dependencies ARGUMENTS: long *Atime : input holding timestamps (sample-numbers) for array A long *Btime : input holding timestamps (sample-numbers) for array B float *Aval : the data for array A (source) float *Bval : the data for array B (destination) long Atot : total elements in array A long Btot : total elements in array B char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error result array will hold statistics char array will hold message (if any) SAMPLE CALL:
DESCRIPTION: Apply a bi-directional, tweaked biquad Butterworth filter to an array of numbers - overwrites the original input array - high- and low-pass filters initialize differently to avoid edge effects Makes two passes at the data (forward & reverse), to avoid time-shifting Does this twice, as required, to do both low- and high-pass filtering High- and low-pass filters initialize differently to avoid edge effects Coefficient calculations based on public domain code originally posted by Patrice Tarrabia (http://www.musicdsp.org/showone.php?id=38) NOTE : interpolation should be applied first if needed to remove NANs or INFs NOTE : padding the array can help remove effects from large deflections at either edge NOTE : normalizing data to the mean before filtering can reduce artefacts in the output NOTE: http://www.iowahills.com/A8FirIirDifferences.html IIR Zero Input Limit Cycles IIR filters differ from FIR filters in that some IIR filters can suffer from a phenomenon known as Zero Input Limit Cycles. This is a problem related to register size where the IIR filter is unable to clear its registers and its output can't settle at zero after the input goes to zero. The filter's output may oscillate indefinitely because a small residual signal continues to circulate within the filter. Of course, an ideal IIR filter's output will never settle at zero in theory, but this is only true if the registers have infinite precision. The truncation caused by limited register size will limit the length of the filters impulse response and should also prevent limit cycles. Notch and high pass IIR filters are most susceptible to this problem and will require extra register bits to prevent this phenomenon from occurring. USES: To remove unwanted frequencies from data DEPENDENCY TREE: No dependencies ARGUMENTS: double *X : data array to be filtered, fixed sample rate is assumed size_t nn : length of X array float sample_freq : sample frequency (samples per second) float low_freq : cut-off for the high-pass filter - set to 0 to skip this step float high_freq : cut-off for the low-pass filter - set to 0 to skip this step float res: : resonance value (range 0-sqrt(2), typically sqrt(2) ) NOTE: low values produce sharper rolloffs but can produce ringing in the output NOTE: high values produce gentle rolloffs but can dampen the signal char message[]: message indicating success or reason for failure RETURN VALUE: 0 on success, -1 on fail
DESCRIPTION: Apply a bi-directional, tweaked biquad Butterworth filter to an array of numbers - overwrites the original input array - high- and low-pass filters initialize differently to avoid edge effects Makes two passes at the data (forward & reverse), to avoid time-shifting Does this twice, as required, to do both low- and high-pass filtering High- and low-pass filters initialize differently to avoid edge effects Coefficient calculations based on public domain code originally posted by Patrice Tarrabia (http://www.musicdsp.org/showone.php?id=38) NOTE : interpolation should be applied first if needed to remove NANs or INFs NOTE : padding the array can help remove effects from large deflections at either edge NOTE : normalizing data to the mean before filtering can reduce artefacts in the output NOTE: http://www.iowahills.com/A8FirIirDifferences.html IIR Zero Input Limit Cycles IIR filters differ from FIR filters in that some IIR filters can suffer from a phenomenon known as Zero Input Limit Cycles. This is a problem related to register size where the IIR filter is unable to clear its registers and its output can't settle at zero after the input goes to zero. The filter's output may oscillate indefinitely because a small residual signal continues to circulate within the filter. Of course, an ideal IIR filter's output will never settle at zero in theory, but this is only true if the registers have infinite precision. The truncation caused by limited register size will limit the length of the filters impulse response and should also prevent limit cycles. Notch and high pass IIR filters are most susceptible to this problem and will require extra register bits to prevent this phenomenon from occurring. USES: To remove unwanted frequencies from data DEPENDENCY TREE: No dependencies ARGUMENTS: float *X : data array to be filtered, fixed sample rate is assumed size_t nn : length of X array float sample_freq : sample frequency (samples per second) float low_freq : cut-off for the high-pass filter - set to 0 to skip this step float high_freq : cut-off for the low-pass filter - set to 0 to skip this step float res: : resonance value (range 0-sqrt(2), typically sqrt(2) ) NOTE: low values produce sharper rolloffs but can produce ringing in the output NOTE: high values produce gentle rolloffs but can dampen the signal char message[]: message indicating success or reason for failure RETURN VALUE: 0 on success, -1 on fail
DESCRIPTION: Apply a tweaked biquad Butterworth filter to an array of numbers - this version does not modify original input array (X) - consequently, it requires pre-allocation of memory for swap (Y) and output (Z) Makes two passes at the data (forward & reverse), to avoid time-shifting Does this twice, if required, to do both low- and high-pass filtering Coefficient calculations based on public domain code originally posted by Patrice Tarrabia (http://www.musicdsp.org/showone.php?id=38) NOTE : interpolation should be applied first if needed to remove NANs or INFs NOTE : padding the array can help remove effects from large deflections at either edge of the data, but is NOT required to compensate for data offset from zero NOTE: 2015_05_08: modified initialization for low-pass filter to reduce edge artefact incidence USES: To remove unwanted frequencies from data DEPENDENCY TREE: No dependencies ARGUMENTS: float *X: array holding input to be filtered, fixed sample rate is assumed float *Y: array reserved for swapping values, memory must be pre-allocated by calling function float *Z: array reserved for filtered output, memory must be pre-allocated by calling function size_t nn: length of X and Z arrays float sample_freq: sample frequency (samples per second) float low_freq: cut-off for the high-pass filter - set to 0 to skip this step float high_freq: cut-off for the low-pass filter - set to 0 to skip this step NOTE: if neither low_freq nor high_freq are set, data is copied unaltered to Z float res: resonance value (typically 1, range 0-sqrt(2) ) NOTE: low values produce sharper rolloffs but can produce ringing in the output NOTE: high values produce gentle rolloffs but can dampen the signal char message[]: message indicating success or reason for failure RETURN VALUE: 0 on success, -1 on fail
DESCRIPTION: Apply a bi-directional, tweaked biquad Butterworth filter to an 2D matrix of numbers - overwrites the original input array - based on xf_filter_bworth1, but differs as follows... - filter is applied to each row - X is a pointer within the function, and nn is derived from the "width" argument - each row is de-meaned before filtering, and the mean re-applied afterwards - in this way mean differences between rows are preserved NOTE : interpolation should be applied first if needed to remove NANs or INFs USES: Efficient filtering of time-series matrices - filter applied in time (rows) DEPENDENCY TREE: No dependencies ARGUMENTS: double *X : data array to be filtered, fixed sample rate is assumed size_t nn : length of X array float sample_freq : sample frequency (samples per second) float low_freq : cut-off for the high-pass filter - set to 0 to skip this step float high_freq : cut-off for the low-pass filter - set to 0 to skip this step float res: : resonance value (range 0-sqrt(2), typically sqrt(2) ) NOTE: low values produce sharper rolloffs but can produce ringing in the output NOTE: high values produce gentle rolloffs but can dampen the signal char message[]: message indicating success or reason for failure RETURN VALUE: 0 on success, -1 on fail
DESCRIPTION: Clip values in an array exceeding certain limits A minimum number of sequencial valid data must be present in order to avoid clipping USES: - restrict the data range in a data series - invalidate values exceeding a certain threshold DEPENDENCY TREE: No dependencies ARGUMENTS: float *data0: array holding input to be filtered, fixed sample rate is assumed size_t nn: length of data0 array size_t nwin: minimum number of sequential valid points required to pass filter float min: minimum value to keep float max: maximum value to keep float newmin: clipped value to replace numbers less than min float newmax: clipped value to replace numbers more than max char message[]: message indicating success or reason for failure RETURN VALUE: 0 on success, -1 on fail
DESCRIPTION: Apply a Finite Impulse Response (FIR) filter to an array of data Overwrites input array USES: DEPENDENCY TREE: No dependencies ARGUMENTS: double *data : input holding data long nn : size of data array double *coefs : the set of FIR filter coefficients, pre-calculated by calling function int ncoefs : the number of coefficients int shift : correct data for phase-shift? 0=NO, 1=YES, 2=YES+start-up padding (recommended) char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error data array will be modified char array will hold message (if any) SAMPLE CALL:
DESCRIPTION: Apply a Finite Impulse Response (FIR) filter to an array of data (short version) Overwrites input array USES: DEPENDENCY TREE: No dependencies ARGUMENTS: double *data : input holding data long nn : size of data array double *coefs : the set of FIR filter coefficients, pre-calculated by calling function int ncoefs : the number of coefficients int shift : correct data for phase-shift? 0=NO, 1=YES, 2=YES+start-up padding (recommended) char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error data array will be modified char array will hold message (if any) SAMPLE CALL:
DESCRIPTION: Apply a Finite Impulse Response (FIR) filter to an array of input Separate input and output arrays (input preserved) USES: DEPENDENCY TREE: No dependencies ARGUMENTS: double *input : array holding input double *ouput : array (pre-allocated) to hold onput long nn : size of input array double *input : input holding input double *coefs : the set of FIR filter coefficients, pre-calculated by calling function int ncoefs : the number of coefficients int shift : correct data for phase-shift? 0=NO, 1=YES, 2=YES+start-up padding (recommended) char *message : pre-allocated array to hold error message RETURN VALUE: 0 on success, -1 on error output array will be overwritten char array will hold message (if any) SAMPLE CALL:
DESCRIPTION: Calculate Finite Impulse Response (FIR) filter coefficients Derived from the Iowa Hills Software collection: Iowa Hills Software, LLC http://www.iowahills.com/5FIRFiltersPage.html November 19, 2014 This implimentation: - combines multiple functions and header-definitions into a single function - assumes corection to coefficients based on accurate 3dB cutoffs is desired - currently only includes a single windowing option (Kaiser), or "none" USES: Calculate Finite Impulse Response (FIR) filter coefficients DEPENDENCIES (included here): void BasicFIR(double *FIRcoef, double *tempcoef, int NumTaps, int PassType, double OmegaC, double BW, int WindowType, double WinBeta); void FIRFreqError(double *Coeff, int NumTaps, int PassType, double *OmegaC, double *BW); double Goertzel(double *Samples, int N, double Omega); double Bessel(double x); double Sinc(double x) ; No dependencies ARGUMENTS: int NumTaps : desired number of taps for the filter - 3-256, 41 is typical, more taps = better filtering int PassType : 1=LP, 2=HP, 3=BP, 4=notch double OmegaC : corner (LP,HP) or central (BP, notch) frequency as a proportion of Pi (2.0 * Hz/samplerate) double BW : bandwidth as a proportion of Pi (2.0 * Hz/samplerate) char *WindowType : none,kaiser,sinc double WinBeta : 0-10, used for the Kaiser window char *message : character array to hold message RETURN VALUE: pointer to array of coefficients, or NULL on fail
DESCRIPTION: Invalidate sections of good data which fail a minimum length requirement This version is designed to deal with multi-channel input data float version USES: - conditioning step before interpolating - apply a general QC to "intermittent" data DEPENDENCY TREE: No dependencies ARGUMENTS: float *data0 input array martix, row=sample, column=channel size_t nn: length of data0 array (number of rows, i.e. multi-channel samples) size_t nchan: number of channels (columns) size_t mingood: minimum number of sequential valid points required to pass filter float setbad: bad-value to look for (e.g SHRT_MAX) char message[]: message indicating success or reason for failure RETURN VALUE: 0 on success, -1 on fail
DESCRIPTION: Invalidate sections of good data which fail a minimum length requirement This version is designed to deal with multi-channel input data USES: - conditioning step before interpolating - apply a general QC to "intermittent" data DEPENDENCY TREE: No dependencies ARGUMENTS: short *data0 input array martix, row=sample, column=channel size_t nn: length of data0 array (number of rows, i.e. multi-channel samples) size_t nchan: number of channels (columns) size_t mingood: minimum number of sequential valid points required to pass filter short setbad: bad-value to look for (e.g SHRT_MAX) char message[]: message indicating success or reason for failure RETURN VALUE: 0 on success, -1 on fail
DESCRIPTION: Apply a notch (band-stop) filter to an array of numbers - this version modifies the input array Makes two passes at the data (forward & reverse) - eliminates time-shifting, and improves attenuation in the notch band. Also doesn't take any more time than copying the Y array back to X The function COULD be faster if reverse filtering was eliminated and Y was pre-allocated by the calling function and usedas the output instead of overwriting X, however the notch width would have to be widened to get comparable amplitudereduction, with concommittant loss of frequency specificity NOTE : interpolation should be applied first if needed to remove NANs or INFs NOTE : padding & de-meaning the array can help remove effects from large deflections at either edge of the data and data offset from zero Coefficient calculations based on principals in chapter 6 of Introductory Digital Signal Processing with Computer Applications (2nd Edition, 1994) By Paul A. Lynn and Wolfgang Fuerst USES: Removal of 50 or 60 Hz mains noise from recorded signals DEPENDENCY TREE: No dependencies ARGUMENTS: float *X: array holding input to be filtered, fixed sample rate is assumed size_t nn: length of X array float sample_freq: sample frequency (samples per second) float notch_freq: the frequency to by cut (Hz) float notch_width: width of the stop-band (Hz) char message[]: message indicating success or reason for failure RETURN VALUE: 0 on success, -1 on fail
DESCRIPTION: Apply a notch (band-stop) filter to an array of numbers - this version does not modify original input array - consequently, it requires pre-allocation of memory for swap and output Makes two passes at the data (forward & reverse), to avoid time-shifting NOTE : interpolation should be applied first if needed to remove NANs or INFs NOTE : padding & de-meaning the array can help remove effects from large deflections at either edge of the data and data offset from zero Coefficient calculations based on principals in chapter 6 of Introductory Digital Signal Processing with Computer Applications (2nd Edition, 1994) By Paul A. Lynn and Wolfgang Fuerst USES: Removal of 50 or 60 Hz mains noise from recorded signals DEPENDENCY TREE: No dependencies ARGUMENTS: float *X: array holding input to be filtered, fixed sample rate is assumed float *Y: array reserved for swapping values, memory must be pre-allocated by calling function float *Z: array reserved for filtered output, memory must be pre-allocated by calling function size_t nn: length of X and Z arrays float sample_freq: sample frequency (samples per second) float notch_freq: the frequency to by cut (Hz) float notch_width: width of the stop-band (Hz) char message[]: message indicating success or reason for failure RETURN VALUE: 0 on success, -1 on fail
DESCRIPTION: Calculate the Fisher transformation for an array of Pearson's correlation coefficients Simply the inverse hyperbolic function of each value see http://en.wikipedia.org/wiki/Fisher_transformation NOTE: a limit of +-3.8 is put on the transform, which is sufficient accuracy for correlation coefficients to a precision of 0.001 NOTE: for type-2 correlations (range 0-1), this translates to a range of 0 to 7.6 USES: Allows hypothesis testing for population correlation coefficients DEPENDENCY TREE: No dependencies ARGUMENTS: double *data : input array of data (will be transformed) long n : number of elements in the array int type : 1: correlations can range from -1 to 1 2: correlations can range from 0 to 1 RETURN VALUE: NONE Oringinal input data will be transformed
DESCRIPTION: Reverse the Fisher transformation for an array of Pearson's correlation coefficients see http://en.wikipedia.org/wiki/Fisher_transformation NOTE: for type-2 correlations (range 0-1), this translates to a range of 0 to 7.6 USES: DEPENDENCY TREE: No dependencies ARGUMENTS: double *data : input array of data (will be transformed) long n : number of elements in the array int type : 1: correlations can range from -1 to 1 2: correlations can range from 0 to 1 RETURN VALUE: NONE Oringinal input data will be transformed
DESCRIPTION: Calculate angle "A" based on length of sides a,b,c B c / \a / \ A-----C b
DESCRIPTION: Take two cartesian coordinates and return distance
DESCRIPTION: Take two cartesian coordinates and return distance (result[0]) and angle (result[1])
DESCRIPTION: Return the linear equation for two Cartesian coordinates - equation: y= slope*x + intercept - result[0]= slope (or NAN for vertical lines) - result[1]= intercept (or NAN for vertical lines)
DESCRIPTION: Calculate the vertical offset of a point from the line joining two flanking points USES: - surrogate for amplitude or AUC - useful for detecting amplitude of "local" peaks DEPENDENCY TREE: No dependencies ARGUMENTS: double x1 : line-point1, x double y1 : line-point1, y double x2 : line-point2, x double y2 : line-point2, y double x3 : point, x double x3 : point, y char *message : pre-allocated array to hold error message RETURN VALUE: offset on success, NAN error char array will hold message (if any) SAMPLE CALL:
DESCRIPTION: Calculate the vertical offset of a point from the line joining two flanking points USES: - surrogate for amplitude or AUC - useful for detecting amplitude of "local" peaks DEPENDENCY TREE: No dependencies ARGUMENTS: float x1 : line-point1, x float y1 : line-point1, y float x2 : line-point2, x float y2 : line-point2, y float x3 : point, x float x3 : point, y char *message : pre-allocated array to hold error message RETURN VALUE: offset on success, NAN error char array will hold message (if any) SAMPLE CALL:
DESCRIPTION: - Calculate the minimum & maximum windowed-slope in a series USES: - Charactorization of evoked responses, synaptic responses, etc DEPENDENCIES: - No dependencies ARGUMENTS: float *data : input holding data long n1 : size of data array long winsize : size of window (samples) across-which to estimate slope double interval : units spanned by each sample, = 1/samplerate int test : test to apply, to ignore a window or break 0= none -1= ignore negative slopes 1= ignore positive slopes -2= stop seeking on first positive slope 2= stop seeking on first positive slope -3= stop seeking when slope starts getting more negative 3= stop seeking when slope starts getting more positive double min : the smallest (most negative) slope double max : the largest (most positive) slope char *message : arrray to hold message in the event of an error RETURN VALUE: success: number of slope estimates taken fail: -1, message[] will hold information min and max will be updated
DESCRIPTION: Extract an ASCII header from an ASCII or binary file - assumes the header ends in a keyword - stored header will include the keyword USES: DEPENDENCY TREE: No dependencies ARGUMENTS: char *infile : the name of the input file (or "stdin") char *keyword : the keyword markning the end of the header char *message : pre-allocated array to hold error message RETURN VALUE: on sucess: A pointer to the header - must be freed by calling function or main on failure: NULL char array will hold message (if any) SAMPLE CALL: char *header=NULL,message[256]; header= xf_getheader1(data.wfm,"WAVES_START",message ); ifheader!=NULL) printf("%s\n",header); else { fprintf(stderr,"\n\t--- %s/%s\n\n",thisprog,message); exit(1); }
DESCRIPTION: Calculate the index to a list-element given the min, max and number of elements Assumes elements in list are evenly spaced (eg. a series of timestamps) USES: DEPENDENCY TREE: No dependencies ARGUMENTS: double min : smallest value in series double max : largest value in series long n : number of elements in the series double value : value to return the index to (will be rounded down to nearest integer) char *message : pointer to pre-allocated array to hold error message if any RETURN VALUE: index, or -1 on error SAMPLE CALL:
DESCRIPTION: - Find the columns containing each keyword in a delimited list - Useful for reading the header-lines in files, and determining which columns correspond with the keywords - It is assumed that the user knows wich keywords should be present DEPENDENCIES: xf_lineparse2() ARGUMENTS: char *line1 ; input: the input line char *d1 : input: the delimiter separating words in line1 - typically "\t" (tab) char *keys1 : input: a list of keywords to find in line1 char *d2 : input: the delimiter for list keys1 - typically "," long *nkeys1 : output: the number of keys found in line1 (pass as address to variable) char *message : output: pre-allocated array to hold error message RETURN VALUE: pointer to array of indices to the keys (key-column), NULL error nkeys1 will be updates to say how many of the keys were found in the line message array will hold explanatory text (if any) SAMPLE CALL: <TAGS> string </TAGS>