pyDataVis is a GUI application for interactive visualization, analysis and manipulation of scientific data. There are many free plotting software most of them far more powerful than pyDataVis. However, in terms of simplicity, it is hard to beat. Indeed it is very easy to use, just drag and drop your data file in pyDataVis window and immediately you are seeing the curves. For now, pyDataVis is limited to 2D plotting.
pyDataVis is an open source software written in Python. It uses PyQt as graphic user interface (GUI) and Matplotlib for plotting.
pyDataVis operates only on text data. These data are assumed to be in columns with the same number of elements. The column separator (a space, a tab character, a comma, a semicolon …) must be the same all over the file. Each column is regarded as a vector (one-dimensional array). Here is an example, with 3 vectors having 10 elements:
0 0.00 0.00
1 1.00 1.00
2 1.41 1.69
3 1.73 2.09
4 2.00 2.38
5 2.23 2.60
6 2.44 2.79
7 2.64 2.94
8 2.82 3.07
9 3.00 3.19
This set of data in tabular form is called a data block. It is possible to have several data blocks with different number of columns (vectors). These blocks must be separated by, at least, two empty lines.
Any line which contains non numeric data will be regarded as a comment. However, to prevent ambiguity, it is wiser to prefix a comment line with a ‘#’ character.
A pyDataVis window is shown in figure 1; it is divided into four main parts:
When a data file is red without error, the Text editor window shows the text content of the file. If the format of this text has been correctly decoded, one or more curves are automatically plotted in the Plotting area. Information about the structure of the data and plotted curves are given in the Information window. Unlike text editor and script windows the text in Information window cannot be modified, it is purely informational. The Script window is used to input the script commands.
This package can be found on the website of pyDataVis, in Downloads page. To install, assuming that the .deb file is in Downloads folder, open a Terminal and type:
cd Downloads
sudo dpkg -i pyDataVis.deb
The application files are in /opt/pyDataVis. To uninstall the application, you can use:
sudo dpkg --purge pyDataVis
This package archive can be found on the website of pyDataVis, in Downloads page. To install, copy the .zip file to the location where you want store the program. Then extract all files from the archive. To launch the application, open the folder pyDataVis and execute the file pyDataVis.exe.
To remove the application you only have to delete the pyDataVis folder.
Open a Terminal and run:
pip install pyDataVis
Then to launch the application:
pyDataVis
pyDataVis requires Python 3, PyQt5, Numpy, Pandas, Scipy and Matplotlib. The easiest way to install everything is to use Open Source Anaconda that will “just work” out of the box for Windows, macOS and common Linux platforms.
The source archives can be found on the website of pyDataVis, in Downloads page. Extract the archive to the place where you want to store the program.
If you have git installed you can also clone the repository. Open a Terminal and change the current working directory to the location where you want to clone pyDataVis, for example ~/myprog:
cd ~/myprog
Then type:
git clone https://github.com/pyDataVis/pyDataVis.git
To launch the application, change the current working directory to the pyDataVis folder, for example ~/myprog/pyDataVis:
cd ~/myprog/pyDataVis
and execute the pyDataVis.py script:
python pyDataVis.py
To display the manual, open the page https://pydatavis.github.io/ in your browser or use the Help option in the Help menu.
The folder examples, in the source archives, contains files for testing almost any function. For those who have not installed from source archives, they can be found on the website of pyDataVis, in Downloads page.
If you need more information, feel free to contact me at<EMAIL_ADDRESS>To see almost every script commands in action you can run the self tests with the ‘Run tests’ option in Help menu.
The text data is always loaded first in the Text editor window before been processed. Then pyDataVis tries to decode the data. If successfull it will plot the curves. There are two ways to supply data in the Text editor window:
The usual way is to load the data from a file. This is done either by selecting Open in the File menu or by dragging the file, from the file manager or the desktop, in the pyDataVis window.
The second way is to enter the data manually. For example, using the key sequence Ctrl+N (or selecting New in the File menu) opens a new empty pyDataVis window. You may then input the data in the Text editor window. It is also possible to paste text data from an other application. Translation of this data in vectors and plotting the resulting curves is done by selecting Update from text in File menu.
pyDataVis can automatically decode and plot the files containing IR, mass or NMR spectra in JCAMP-DX format. This format is used for example in NIST Chemistry WebBook or ChemSpider databases. pyDataVis also recognizes the Raman data file format from the RRUFF database.
The Append option in the File menu is used to append data from another file at the end of the current data. The added file must contain only one data block and the number of vectors must be the same as the current data block.
By using the option Import from spreadsheet in the File menu, it is possible to import data from spreadsheet files like MS Excel or LibreOffice Calc (in this case the odfpy module must first be installed, for example with pip install odfpy
). If the file contains several sheets, the same number of data blocks are created. Each sheet must contains only one data table otherwise the importation will give unpredictable results.
When the vectors are modified, for example by Script commands, the Text editor window is not updated to reflect these changes. In case of mistake this gives the possibility to restore the previous state by using the Update from text option in the File menu. To update the Text editor window with the current vector values the Update from data option in the File menu should be used. Beware that these two options update also the file so that there is no way to get the original data back.
To see the real-time values of the vector elements, use the Data Table option in the Tools menu. This table can also be displayed with the key sequence Ctrl+T. The position of the selected cell of this table is linked to the data cursor position. The value of the vector elements can also be modified with this table.
The data blocks can be saved in text files by using the options Save data and Save data As in the File menu. The column separator is always a tab character. The data blocks are separated from each other by two successive empty lines. Save data uses the current file name whereas Save data As lets you choose a new file name. I recommend to use the “.plt” extension for files saved by pyDataVis to tell them apart from simple .txt files because they contains plotting commands in the header.
There are three options in the File menu to export the data blocks:
Non numeric lines are generally regarded as comments and are not decoded except if they start with special words like plot, labX, … in which case they are interpreted as plot commands. Besides, the line immediately before the numerical data is decoded to check if it contains the vector names. For example:
Time Intensity
0 0
1 5
2 15
3 30
The first vector [0,1,2,3] will be named Time and the second one [0,5,15,30] Intensity.
The number of items in the line containing the vector names must match the number of columns. Therefore a vector name cannot contains space because the space character is regarded as a separator (underscore can be used instead).
In the absence of a line containing the vector names, the vectors are named automatically as V1B1, V2B1, … where the B1 ending indicates that these vectors belong to the first data block. Thus if the file contains a second data block the vectors belonging to this block will be named V1B2, V2B2, …
If no plot command is found in the file header, the first column (V1B1) is assumed to contain X values. Thus, if the file contains three columns (vectors), V1B1, V2B1 and V3B1, two curves will be automatically plotted, V2B1 against V1B1 and V3B2 against V1B1.
Several pyDataVis windows can be opened at the same time and curves exchanged between them using the options of the Curve menu or keyboard shortcuts.
A curve consists of a set of two vectors (X and Y). Thanks to the Curves menu, the curves can be deleted, duplicated or copied. A curve copied in a pyDataVis window cannot be pasted in the same window because two curves cannot share the same name. If the curve is pasted in a window containing a vector having the same name, the original name is suffixed with the ’‘#’’ character. It is the same when a curve is duplicated,
When a X,Y curve is pasted into another window, if the same X-vector already exists in a data block, there are two possibilities, either to add the Y-vector at the end of the data block or to create a new data block to store the X and Y vectors. This is left to the user’s choice.
The Save in file option allows for saving a given curve in a new file.
When a curve is displayed on plotting area, right clicking with the mouse in this area triggers a light brown crosshair cursor. The point number (no) and coordinates (xc, yc) at the cursor position are shown in the status bar at the bottom of the pyDataVis window.
This cursor can be moved to the next data point either with the key sequence Ctrl+Right arrow or by selecting the option Cursor Right in the Marks menu. Similarly the cursor can be moved to the previous data point either with the key sequence Ctrl+Left arrow or by selecting the option Cursor Left in the Marks menu.
Besides, the key sequence Ctrl+F (Cursor First in the Marks menu) puts the cursor on the first point while the key sequence Ctrl+L (Cursor Last in the Marks menu) puts the cursor on the last point.
When the plotting area shows several data curves, the cursor follows only one curve, which coordinates are shown in the status bar. This is the “active” curve. The key sequence Ctrl+D (Cursor Down in the Marks menu) makes the previous curve to become the active one whereas the key sequence Ctrl+U (Cursor Up in the Marks menu) makes the previous curve to become the active one.
Finally, the key sequence Ctrl+M (Add a Mark in the Marks menu) adds or moves a mark (shown as a gray vertical line) at the cursor position. Only two marks can be added because the third time Ctrl+M is hit, the first mark is deleted and replaced by the new one. The key sequence Ctrl+R (Remove all Marks in the Marks menu) remove cursor and marks. These marks are used to select portions of the curve for several script commands.
Beneath the Plotting area there is the MatPlotLib toolbar shown in figure 2. Hovering the mouse over each icon displays a tool tip giving a short description of its action. Here is the description of the action of each button:
The first three buttons, Home, Back/Forward, are used to navigate through the previous views. Home returns to the initial view of the plot.
The fourth button has two modes: pan and zoom. Press the left mouse button and hold it to pan the figure. If you press ‘x’ or ‘y’ while panning the motion will be constrained to the x or y axis, respectively. Press the right mouse button to zoom, dragging it to a new position. The x axis will be zoomed in proportionately to the rightward movement and zoomed out proportionately to the leftward movement. The same is true for the y axis and up/down motions.
The fifth button handles zooming to rectangle functions. Dragging the mouse while holding the left button the axes view limits will be zoomed in to the defined region. When using the right mouse button, the axes view limits will be zoomed out, placing the original axes in the defined region.
The sixth button is used to adjust the plot position within the window.
The seventh button calls the axis/curve style editor, where you can modify most of the plot properties like title, axis label and scale, plot symbols, line colours and line styles. It should be noted that the changes done using the axis/curve style editor are not recorded permanently.
The last button allows to save the plot as an image. You can save files with the following extensions: png, ps, eps, svg and pdf.
| Command | Keyboard shortcut |
|---|---|
| Toggle grid | g or G when mouse is over the plot |
| Toggle x axis scale (log/linear) | L or k when mouse is over the plot |
| Toggle y axis scale (log/linear) | l when mouse is over the plot |
| Constrain pan/zoom to x axis | hold x when panning/zooming with mouse |
| Constrain pan/zoom to y axis | hold y when panning/zooming with mouse |
| Preserve aspect ratio | hold CONTROL when panning/zooming with mouse |
The toggle grid shortcut g (or G) behavior is: the first hit toogles vertical grid, the second hit adds the horizontal grid, the third hit removes the vertical grid leaving only the horizontal one and the fourth hit removes all grids.
When the data format cannot be decoded, no curve is shown in the Plotting area, only the text content is displayed in the Text editor window. Nevertheless, in some cases, the data in the Text editor window can be processed by selecting the suitable option in the Convert menu.
The CIF option is used to generate a powder diffraction pattern from a Crystallographic Information File (CIF). This is a standard text file format for representing crystallographic information, promulgated by the International Union of Crystallography (IUCr). A large collection of CIF files can be found in the Crystallography Open Database (COD).
The reflection lines in the simulated XRD pattern are generated from CIF data loaded in the Text editor window by the Dans_Diffraction package. This package is easily installed in aterminal with: pip install Dans-Diffraction
.
The line positions are calculated for an X-ray energy = 8.048 keV, which corresponds to Cu K-alpha-1 radiation. For example, the figure 3 shows the simulated pattern of silicon calculated from a CIF file (COD 9008566) plotted along with an experimental pattern recorded with a silicon standard.
The DAT option does the conversion of “.DAT” diffraction file format. This is a common ASCII file format for raw (profile) data. In the first line, the 2theta start, step and final value are given, followed by the intensity values for each step. The number of intensity values per line may vary. The generated data is saved in a new file using the same name as the “.DAT” file but with “.TXT” extension. Then this file is loaded and plotted in a new pyDataVis window.
The PRF option does the conversion of a “.prf” file generated by FullProf in a tabular text file. FullProf is a set of programs used for Rietveld analysis of X-ray powder diffraction data. The generated data is saved in a new file using the same name as the “.PRF” file but with “.TXT” extension. Then this file is loaded and plotted as shown in figure 4.
The Reshape option allows to change the shape of a data block (number of lines and columns) while keeping the total number of elements constant. For example, Reshape can transform 1D data in 2D with half the number of elements in each dimension. It can be used to restore the right shape which has been lost after pasting the data from another application. For example, when a set of cells are pasted from LibreOffice Calc the column separator seems to have vanished. By using the Update from text option in the File menu, the data are now in one column. With the Reshape option it is possible to restore the original shape of the data.
The Transpose option switches the rows and columns. Like Reshape, it can be used to restore the right shape which has been lost after pasting the data from another application.
The 1D->2D option adds to a data block containing only one vector, let’s call it Y, a new vector X thereby allowing to plot Y=f(X). The X vector contains the point number, starting from 0. Subsequently it is easy to modify X thanks to script commands.
In pyDataVis, most of the data processing is done via scripting commands which are detailed below. However in some cases it is more convenient to use interactive tools. These tools are accessed via the Tools menu. When several curves are plotted in the Plotting area, a dialog let you select the curve that need to be addressed.
In some cases the computation results are saved in the log file preceded by a header indicating the date and time of the calculation. This file can be accessed and modified by selecting the View log option in the file menu.
Smoothing is a very powerful technique used all across data analysis. It consists of removing random variation (noise) while preserving the original shape and features of the signal. With pyDataVis you can choose several techniques to achieve the best result according to the kind of data.
With the Smoothing option of the Tools menu two kinds of filter are proposed: the Moving average and the Savitzky-Golay.
The Moving average filter operates by averaging ‘n’ points from the input data to produce each point in the output data, where n is an odd integer. The figure 5 shows the smoothing window when the Moving average filter is selected. Three curves are plotted. The grey curve corresponds to the original data, the blue curve to the filtered data and the green curve shows the difference between original and filtered. The goal is that this curve shows only noise.
There are two parameters to tune the degree of smoothing. The first is the filter size, which is the number of points that are averaged. The second is the number of passes which is the number of times the filtering process is repeated. When the parameter values are changed, you should click on the Compute button to restart the calculations with the new values.
When you click on the OK button, the original data is replaced by the blue curve, the window is closed and the Plotting area is updated.
The Savitzky-Golay is a low-pass filter, particularly suited for smoothing noisy data. The main idea behind this approach is to make for each point a least-square fit with a polynomial of low order over an odd-sized window centered at the point. It has the advantage of preserving the original shape and features of the signal better than other types of filtering approaches, such as moving average.
When the Savitzky-Golay filter is selected, there is an additional parameter, the order, which is the degree of the fitting polynom.
Spline functions are widely used in applications requiring data interpolation and/or smoothing. They are build by defining knot points and calculating the polynomials that pass through these points. With the Spline Smoothing option of the Tools menu two parameters are used to tune the level of smoothing:
The degree of the polynomials. Must be in the range 1 to 5. The best starting value is 3, which corresponds to a cubic spline.
The smoothing factor, s, which is used to choose the number of knots. This factor control the trade-off between fidelity to the data and roughness of the function estimate. If s = 0, spline will interpolate through all data points.
The figure 6 shows the Spline smoothing window. Like with the Smoothing window seen above, three curves are plotted. The grey curve corresponds to the experimental data, the blue curve to the smoothed data and the green curve to the difference between experimental and smoothed values.
The ALS Smoothing option, in the Tools menu, is primarily used for baseline correction. ALS stands for Asymmetric Least Squares. This method has been designed by Paul Eilers and Hans F.M. Boelens (2005). To get an effective baseline estimator, it combines a smoother with asymmetric weighting of deviations from the (smooth) trend. It is easy to use, fast and keeps the analytical peak signal intact. No prior information about peak shapes or baseline (polynomial) is needed.
There are two parameters: p for asymmetry and lambda for smoothness. For a signal with positive peaks a good choice is to set lambda between 102 and 109 and p between 0.001 and 0.1.
It is not efficient for a number of data point exceeding 1000.
The figure 7 shows the ALS Smoothing window. Besides the Compute button already present in the previous tools, there are two additional buttons. As we might expect Subtract will subtract the baseline and thus change the original data, whereas Add does not change the data but add a new vector containing the baseline.
The Curve Fitting option, in the Tools menu, is used for fitting curves with predefined models. There are seven models :
Polynomial
Exponential, \(y = A \times exp(B x)\)
Logarithm, \(y = A \times Ln(x) + B\)
Power, \(y = A \times x^B\)
Sigmoid, \(y = y_0 + \frac{H}{1 + exp(S (x_0 - x))}\)
Arrhenius, \(y = A \times exp \left( - \frac{E}{R(x+T)} \right)\) with R = 8.314 and T = 273
Herschel-Bulkley, \(y = A + B x^C\)
Several curves are plotted in figure 8 to show the general shape of these models. In the plot legend, Bneg means that B is negative and Bpos that B is positive. Thus pow_Bneg reads as power model with negative B.
In the equation of the sigmoid model, \(x_0\) is the position of the inflection point, \(y_0\) is the position of the baseline, H is the height (amplitude) and S governs the slope of the tangent at the inflection point. Larger S values means steeper increase. When S is negative the S shape is inverted. For example, in figure 8, S = -1 and in figure 9, S = +1.
The last two models are more specific. The Arrhenius model describes the temperature dependence of reaction rates. The parameters are A (which is called the pre-exponential factor) and E the activation energy.
The Herschel-Bulkley model is used to describe the rheological behavior of a non-newtonian fluid; y is the shear stress and x the shear rate; A is the yield stress, B the consistency index and C the flow index.
None of the available models in Curve Fitting menu option are adequate for fitting peaks. For that, the Peak fitting tool should be used.
The figure 9 shows the Curve Fitting window after a fitting process with the Sigmoid model. The current parameter values are printed in the text area, below the Matplotlib graphic toolbar. The value of these parameters can be modified in the fields at the bottom of the window and the fitting process is restarted by clicking on the Compute button. An estimation of the goodness of the fit is given through the Root Mean Square Error (RMSE) value defined by:
\[ RMSE = \sqrt \frac {\sum_{i=1}^{N} { \left[ y_{calc} - y_{exp} \right] ^2 } } {N} \]
where \(y_{calc}\) is the calculated data, \(y_{exp}\) the experimental data and N the number of points. The fitting results are saved in the log file.
When you click on the OK button, the original data is replaced by the blue curve, the window is closed and the Plotting area is updated.
These two options of Tools menu are working together because after finding the peaks, the Peak finding module automatically calls the Peak fitting module. It should be noticed that this module requires that the baseline is at zero. You can use the Baseline correction tool if this not the case.
Selecting the Peak finding option pops up the Peak finding dialog as shown on figure 10. To select only the relevant peaks, two parameters are used: the height of the smallest peak (minHeight) and the width of the narrowest peak (minWidth).
The first time, pyDataVis try to guess the parameter values based on a statistic analysis of the signal. Of course, you can modify these values according to your expectations.
As shown on figure 11, at the end of the peak finding process, a dialog pops up to inform you of the number of peaks that were found. If you answer ‘Yes’ the peak fitting module is launched. Otherwise the process is canceled.
When the peak fitting process is finished (which can takes some time when there are many peaks) the peak fitting window pops up, as shown in figure 12. This window is divided in three parts. Above is the plotting area which allows to visually evaluate the fitting results. The grey curve corresponds to the experimental data, the blue curve to the calculated sum of the peaks and the green curve shows the difference between the data and the peak sum. Of course the goal is to make this curve as flat as possible.
Below the plotting area, there is the information area (read only). As in the case of Curve fitting an estimation of the goodness of the fit is given through the RMSE value, already defined above in the Curve fitting tool. Below the information area is the Peak parameters area. Here, the value of the peak parameters can be modified after which the fitting process can be launched again by clicking on the Compute button. The fitting results are saved in the log file.
When the fitting module is called from the Finding peak option, it always uses a Gaussian function to model each peak shape. The Gaussian type is not the best to represent the peak shapes in the case of the experimental data shown in the previous figure. It is possible to change the peak model by replacing the peak parameters, G, by another type. pyDataVis allows to choose between six peak types:
The curve equation for Gaussian and Lorentzian peak type are: \[ G(x) = A \times exp \left[- \left( \frac{x-x_m}{w} \right) ^2 \right] \]
\[ L(x) = A \times \left[ \frac{1}{1 + \left( \frac{x - x_m}{w/2} \right) ^2} \right] \]
In these equations, A is the peak amplitude (height), \(x_m\) is the position of the peak maximum, w is the Full Width at Half Maximum (FWHM). The pseudo-Voigt model P(x) is a linear combination of a Gaussian curve G(x) and a Lorentzian curve L(x). Therefore this model requires an additional parameter m.
\[ P(x) = m \times L(x) + (1-m) \times G(x) \qquad with \; 0 \leq m \leq 1 \]
The peak shape, associated with these three models, is shown in the left plot of figure 13. The three remaining peak types are used to fit asymmetric peaks. In this case the w parameter is no more a constant but becomes a function of x, w(x) defined by:
\[ w(x) = \frac{2 \ FWHM}{1 + exp(a(x - x_m))} \qquad with \ -1 \leq a \leq 1 \]
The parameter a is the skewness factor. The equation of AG(x), AL(x) and AP(x) are obtained by replacing w by w(x) in the above equations of G(x), L(x) and AP(x). The peak shape corresponding to these three models is shown in the right plot of figure 13.
It is important to notice that, in the peak parameter area, for each peak, six parameters are always provided even when they are not relevant, in which case they are set to zero. The extra parameters are m, the mixing parameter for pseudo-Voigt model and a the skewness factor for asymmetric models.
Let us return to the peak fitting window and replace, for both peaks, the peak type parameter, G, by L. We do not modify the other parameters and click on Compute button. The figure 14 shows that this improves the fitting result.
Clicking on OK generates a file named “peakFit.txt” and opens a new pyDataVis window to display its content. The plotting area is shown in figure 15. There are 10 curves: the experimental data, the sum of peaks and the curve for each peak.
Interpolation consists of finding intermediate values of a variable from a discrete set of its known values. For example, a spatial interpolation or temporal interpolation can be used for finding missing data in space- or time-series. pyDataVis use a smooth spline curve approximation to determine the positions of the unknown values. The Interpolation window, shown on figure 16, is accessed thanks to the Interpolation option of the Tools menu. There are three parameters:
It is possible to interpolate curves which are in another data block by selecting the corresponding block number.
It should be noticed that this tool can also be used to decrease the number of data points.
The Baseline correction is ubiquitous for spectroscopic data analysis because a spectrometer usually have a baseline drift due to environmental factors (temperature, light source, …). The Baseline correction tool of pyDataVis, shown on figure 17, is accessed thanks to the Baseline option of the Tools menu. You define the baseline position by clicking on the curve. Left click adds a new point at the mouse position, right click deletes the point which was already around. The baseline curve can be build from these points by two methods according to the type:
Once a method is chosen, it is not possible to change it. If you click on Correct button the baseline is subtracted from the original data. If you click on Add BL button, this does not change the data but a new vector containing the baseline is added to the data block.
If the signal is too noisy it is recommended to remove the noise prior to the baseline correction (see Remove noise tool).
The Add Noise tool of pyDataVis, shown on figure 18, is accessed thanks to the Add Noise option of the Tools menu. This tool adds to the selected curve random values calculated from a normal (Gaussian) distribution. There are two parameters:
The Remove Noise tool of pyDataVis, shown on figure 19, is accessed thanks to the Remove Noise option of the Tools menu. Actually this tool is not interactive because the process is fully automatic.
In pyDataVis, script commands are used for plotting and to do most of the data processing. Beware that the command names, the vector names and the parameters are case sensitive. Several script commands can be chained together. The script commands are written in the Script window and executed by typing Ctrl+E or by selecting the option Execute a script in the Script menu. Empty lines and lines starting with # are ignored during execution. The script can be executed repeatedly by retyping Ctrl+E as often as desired. If an error occurs during the execution of a script sequence, the process is stopped at the faulty line; the remaining commands are not executed.
Before a script was executed, the current data is saved in memory. If something wrong happens during execution the previous state can be restored by selecting the Undo option in the Script menu.
The key sequence Ctrl+K clears the text in the Script window and gives it the focus so a new script can be written without having to click in this window.
It is possible to save the content of the Script window to a file by selecting the option Save script in the Script menu. Conversely a script previously saved to a file can be loaded in the Script window by selecting the option Load script. The figure 20 shows a pyDataVis window after creation of a new file and execution of the following script:
newv X 0,6.3,0.1
Y = X + sin(2*X)
plot X,Y
labX X
labY1 Y
text 0.5,5,Y = X + sin(2X),black,20
The command used to plot the vector Y against the vector X is:
plot X,Y
plot X,Y,2
The first command plots Y against X using the primary Y-axis (left side) whereas the second command plots Y against X using a secondary Y-axis (right). Only one secondary axis is possible and a plot command against primary axis must precede a plot command against the secondary. Both X and Y vectors must belong to the same data block. The color of the curves is chosen automatically but it can be temporarily modified with the axis/curve style editor in Matplotlib toolbar.
The following command gives to the x-axis the label “Time (s)”
labX Time (s)
The following commands give to the primary Y-axis the label “Intensity” and to the secondary Y-axis the label “Volume”
labY1 Intensity
labY2 Volume
Default plot type is line, that is no symbol are drawn for data points which are linked by lines segments. Symbols are automatically used if the number of points is lower than 15. However it is possible to change this behavior with the commands:
symbol Y
no symbol Y
The first will plot symbols for the curve Y whereas the second will use line type. Like the color, the symbol shape is chosen automatically, however it can be temporarily modified with the axis/curve style editor in Matplotlib toolbar.
You can flip the order of the axis limits thanks to the following commands:
invertx
inverty
These commands works like switches. They are not saved when the file is closed.
It is possible to add text. The following command writes ‘Title’ at x,y position, using a red font with size 14:
text x,y,Title,red,14
The text position is relative to x-axis and primary y-axis. The fourth parameter ‘red’ indicates that the text color is red. The color names are:
blue, darkblue, red, darkred, green, darkgreen, magenta, darkmagenta, cyan, darkcyan, yellow, brown, gray, black, white, cadetblue, chocolate, coral, burlywood, aqua, aquamarine, blueviolet, cornflowerblue, crimson, darkgoldenrod, darkkhaki, darkolivegreen, darkorange, darksalmon, darkseagreen, darkslateblue, darkslategray, darkviolet, deeppink, goldenrod, greenyellow, hotpink, indigo.
Finally it is also possible to add an arrow. The following command draws a red arrow which starts at x,y; its length and angle are given by dx and dy. If dx=0, the arrow is parallel to y-axis. If dy=0 it is parallel to x-axis. If dx=dy it makes an angle of 45°.
arrow x,y,dx,dy,red
All these commands can be chained together. For example, the plot in figure 21 was obtained by executing the following lines in the Script window:
plot T,TG
plot T,Hf,2
labX Temperature (C)
labY1 TG (%)
labY2 Hf
arrow 850,-25,0,7,red
text 815,-28,exo,red,18
The name command is used to change the name of a vector. For example the following command gives to the vector V1 the name V2. It is not possible that two vectors share the same name, therefore if V2 already exists this command will generate an error.
name V1 as V2
The stats command provides statistics on a vector:
stats V1
As shown in figure 22, for a vector Y {\(y_1\), \(y_2\),…,\(y_n\)} of dimension n, the values returned are:
Minimum, \(y_{min}\)
Maximum, \(y_{max}\)
Sum of the values, \(S = \sum_{i=1}^{n} y_i\)
Median, which is a value separating the higher half from the lower half of Y.
Arithmetic mean, \(\overline{\rm y} = \frac{S}{n}\)
Variance, \(\sigma^2 = \frac{1}{n-1} \sum_{i=1}^{n} \left(y_i - \overline{\rm y} \right)^2\)
Standard deviation, \(\sigma = \sqrt{\sigma^2}\)
Root mean square, \(RMS = \sqrt{\frac{1}{n} \sum_{i=1}^{n} y_i^2 }\)
Peak-to-peak, pp = \(y_{max} - y_{min}\)
Brunauer–Emmett–Teller (BET) model serves as the basis for the measurement of the specific surface area of materials from adsorption isotherms. The BET command:
BET
computes the optimum BET surface area and the BET constant. No parameter is given because it is assumed that the first data block contains the adsorption isotherm with the first vector of this block being the relative pressure and the second vector the adsorbed volume expressed in mL/g. Only nitrogen adsorption is implemented. If the relative pressure reaches the saturation (Pr > 0.98) the BET command computes also the pore volume. The optimisation process includes finding, in the relative pressure range 0.04 - 0.30, a window of at least seven data points, giving the best correlation coefficient. The figure 23 shows an example of the computation results. More detailed information is saved in the log file.
The command PSD computes the pore size distribution from adsorption-desorption isotherms. Two data block are required and it is assumed that the adsorption isotherm is contained in the first data block and the desorption isotherm in a second data block. Moreover, in these blocks, the first vector must contain the relative pressure and the second vector the adsorbed volume expressed in mL/g. Only nitrogen adsorption and cylindrical pore geometry are implemented.
The PSD command takes two parameters. The first, which can be either A or D, indicates which isotherm (A-dsorption or D-esorption) is used. The second parameter specifies the equation used for calculating the thickness of adsorbed layer according to the relative pressure (t-curve). This parameter can have one of the three values: halsey, harkins or tfit.
The figure 24 shows an example of the computation of a PSD. The left plot shows the adsorption-desorption isotherms. The computation was done with the desorption isotherm and halsey curve. At the end of the process a new file is generated which contains four vectors: the pore width (Wp), the pore volume (Vp), the pore volume divided by the pore width difference (dV/dW) and the cumulative pore volume (Vcum). As shown on the right chart, two curves are plotted, the pore size distribution dV/dW and the cumulative pore volume.
The command:
area Y
computes the area under a curve Y=f(X) using the trapezoidal rule. The limits of the integration must be indicated by the marker positions. area is defined by the curve and the x-axis. Thus a baseline correction may be required before the area calculation. The figure 25 shows an example of the calculation of the area of the last peak. It should be noticed that, in the case of this picture, a better evaluation of the peak area could have been obtained with the Peak Fitting tool.
The command:
lineq Y
Compute the equation of the line linking the marker positions on the curve Y. The figure 26 shows an example of utilization.
The command:
linefit Y 1
Compute the equation of the line fitting the curve Y at the cursor position. The last parameter, which must be either -1 or +1, indicates the direction of the fitting line. The algorithm is:
This command is used to determine the onset of a peak. The onset is an important feature of a peak. For example, in thermal analysis, the onset temperature does not depend on the heating rate unlike the temperature of the peak maximum.
Before using the onset command, both markers must be positioned to indicate the beginning and the approximate place of the inflection point at the base of the peak. The onset procedure makes use of the linefit function; indeed the process is:
The command is:
onset Y
The figure 28 shows an example of the calculation.
The command to create a new vector V is:
newv Vname Vinit,Vend,stepsiz
where Vname is the vector name, Vinit is the first value, Vend is the last value and stepsiz is the difference between each value. A new data block is created to contain this new vector. For exemple the command:
newv X 0,9,1
create a new data block containing the new vector X whose 10 elements are: 0,1,2,3,4,5,6,7,8,9
The command to delete the vector V is:
delv V
The command:
swapv V1 V2
exchange all the elements between V1 and V2. This commande exchange only the values not the vector names.
The command to delete the data block number n is:
delb n
This delete all the vectors in the data block. Of course this will delete all the curves which used these vectors.
The syntax of the mergeb command is:
mergeb 1 2
This command copies all the vectors of the block 2 at the end of the block 1 and then delete the block 2. Of course, the vectors must have the same number of elements in both blocks.
If V2 does not exist, a new variable is created. If V2 preexists, its previous content is lost.
The main mathematical functions can be applied to a vector (element-wise). The result is a vector with the same number of elements.
To illustrate gradient and cumsum functions, figure 29 shows the result of the execution of the following script sequence:
newv X 0,6.28,0.01
Y = sin(X)
dY = gradient(Y)
S = cumsum(dY)
dY_dX = dY/0.01
plot X,Y
plot X,dY_dX
plot X,S
Y is the sin(X) function. We see that Y and S curves (the cumsum of gradient) overlap almost perfectly (the shift decreases with dX). dY_dX is the gradient dY divided by dX, namely an approximation of the Y derivative, which is cos(X).
This command reduces the number of elements in a vector. It takes two parameters, the vector name and the reducing factor which must be in the range 2-10. This operation implies to reduce the number of elements of all the other vectors in the data block. For example, the command:
shrink V 4
reduces in the number of elements in V by a factor 4.
This command:
revertv V
will revert the order of elements in V. It takes only one parameter, the vector name.
This command:
revertb 1
will revert the order of all the elements in a data block. It takes only one parameter, the data block number.
This command:
sort V
will sort the elements of the vector V in ascending order. It takes only one parameter, the vector name. All the vectors belonging to the same data block will have the order of their elements changed but not necessarily sorted. To sort the vector elements in reverse order, first sort V then revert V.
This command:
IRabs
converts an IR spectrum in absorbance. It takes no parameter. It will generate an error if the first data block does not contain an IR spectrum in transmittance.
This command:
IRtrans
converts an IR spectrum in transmittance. It takes no parameter. It will generate an error if the first data block does not contain an IR spectrum in absorbance.
This command :
fft V
computes the Discrete Fast Fourier Transform of a vector. It takes only one parameter, the vector name. The result of the conversion is saved in a file named as XXX-fft.txt where XXX is the vector name.
This command removes spikes in a vector. It takes two parameters, the vector name and the window size:
despike Vname winsiz
The window size corresponds to the width of the spikes in number of points. It must be an odd number. Defaut size is 3. The figure 30 shows an example of use. The blue curve (I) is the original signal and the red curve shows the signal after using the command:
despike Ides 3
The command:
deldupx X
sorts X in ascending order and then removes duplicates. It processes only the data block containing the vector X. Moreover X must be the first vector of the data block. This command must be executed before using the Interpolation tool.
This command changes the precision (number of decimal places) of all the vector elements. For example the command:
ndec V 5
set the precision of all elements of V to 5 decimal places
There are four commands for clipping. The commands for clipping along the y-axis are:
clipup vname maxval
clipdn vname minval
The first replaces in the vector vname all elements larger than maxval by maxval. The second replaces in the vector vname all elements lower than minval by minval.
The commands for clipping along the x-axis are:
clipx < minval
clipx > maxval
The first deletes in the vector vname all elements lower than minval. The second deletes in the vector vname all elements larger than maxval. It should be noted that clipping along y-axis does not change the number of points whereas clipping along x-axis decreases the number of points.
The figure 31 shows an example where the vector Yclip has been obtained by executing the command:
Yclip = Y
clipup Yclip 1
clipdn Yclip -1
The line command joins, by a straight line, the points of a curve situated at the marker positions. This procedure does not delete the points between the markers but change their position.
line Y
The shift command shifts vertically the portion of a curve delimited by the markers. For example, the command:
shift Y 0.1
add 0.1 to the points located between the marker positions. This is illustrated by figure 32.
The Execute a Python script option in the Script menu executes the Python script in the Script window of the active pyDataVis window.
pyDataVis does several steps before the execution:
import numpy as np
A = np.loadtxt("input.txt")
X = A[0]
Y = A[1]
In this way, the vectors of the active curve can be accessed as X and Y. The output of the Python process is shown in the Information window and saved in the file “output.txt”.
An application of this option may be to use the powerful Scipy library to do some tasks not implemented in pyDataVis. Here is an example of a script using the curve_fit function in the scipy.optimize module to fit a user defined function to data.
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def func(x, a, b):
return a * np.log(-b * np.log(x))
popt, pcov = curve_fit(func, X, Y)
sresult = "Best fit parameters: a={0:g}, b={1:g}".format(popt[0], popt[1])
print (sresult)
plt.plot(X, Y, 'bo')
plt.plot(X, func(X, *popt), 'r-')
plt.suptitle(sresult)
plt.show()
The function used to fit the data is: \(Y = a \cdot log \left( -b \cdot log(X) \right)\)
The figure 33 shows the result of the script execution.


@* * Copyright 2023 HM Revenue & Customs * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. *@ @import config.ViewConfig @this( mainTemplate: views.html.main_template, viewsHelpers: views.ViewsHelpers, viewConfig: ViewConfig ) @()(implicit request: Request[_], message: Messages) @mainTemplate( title = Messages("delete_answers.signed_you_out"), navTitle = "delete_answers.signed_you_out", timeoutDialog = false ) {
@Messages("delete_answers.did_not_save_answers")


#include <XSECCryptoSymmetricKey.hpp>
Inheritance diagram for XSECCryptoSymmetricKey:
All symmetric algorithms are implemented via this interface. Unlike the asymmetric key definitions, this is not further extended for particular algorithms. Rather it defines encrypt/decrypt functions that are implemented within particular providers for a particular algorithm.
Public Types | |
| enum | SymmetricKeyType { KEY_NONE, KEY_3DES_192, KEY_AES_128, KEY_AES_192, KEY_AES_256 } |
| Symmetric Key types understood by the library. More... | |
| enum | SymmetricKeyMode { MODE_NONE, MODE_ECB, MODE_CBC } |
Public Member Functions | |
Constructors and Destructors | |
| XSECCryptoSymmetricKey () | |
| Constructor. | |
| virtual | ~XSECCryptoSymmetricKey () |
| Destructor. | |
Basic CryptoKey Interface methods | |
| virtual KeyType | getKeyType () |
| Returns the type of this key. | |
| virtual const XMLCh * | getProviderName ()=0 |
| Returns a string that identifies the crypto owner of this library. | |
| virtual XSECCryptoKey * | clone ()=0 |
| Clone the key. | |
Symmetric key interface methods | |
| virtual SymmetricKeyType | getSymmetricKeyType (void)=0 |
| What type of symmetric key is this? | |
| virtual void | setKey (const unsigned char *key, unsigned int keyLen)=0 |
| Set the key from the provided bytes. | |
| virtual bool | decryptInit (bool doPad=true, SymmetricKeyMode mode=MODE_CBC, const unsigned char *iv=NULL)=0 |
| Initialise an decryption process. | |
| virtual unsigned int | decrypt (const unsigned char *inBuf, unsigned char *plainBuf, unsigned int inLength, unsigned int maxOutLength)=0 |
| Continue an decrypt operation using this key. | |
| virtual unsigned int | decryptFinish (unsigned char *plainBuf, unsigned int maxOutLength)=0 |
| Finish a decryption operation. | |
| virtual bool | encryptInit (bool doPad=true, SymmetricKeyMode mode=MODE_CBC, const unsigned char *iv=NULL)=0 |
| Initialise an encryption process. | |
| virtual unsigned int | encrypt (const unsigned char *inBuf, unsigned char *cipherBuf, unsigned int inLength, unsigned int maxOutLength)=0 |
| Continue an encryption operation using this key. | |
| virtual unsigned int | encryptFinish (unsigned char *plainBuf, unsigned int maxOutLength)=0 |
| Finish a encryption operation. |
|
|
|
|
|
Symmetric Key types understood by the library. This type defines the list of symmetric key types that the library understands. |
|
|
Constructor.
|
|
|
Destructor. Implementations must ensure that the held key is properly destroyed (overwritten) when key objects are deleted. |
|
|
Clone the key. All keys need to be able to copy themselves and return a pointer to the copy. This allows the library to duplicate keys. Implements XSECCryptoKey. Implemented in OpenSSLCryptoSymmetricKey, and WinCAPICryptoSymmetricKey. |
|
|
Continue an decrypt operation using this key. Decryption must have been set up using an encryptInit call. Takes the inBuf and continues a decryption operation, writing the output to outBuf. This function does not have to guarantee that all input will be decrypted. In cases where the input is not a length of the block size, the implementation will need to hold back cipher-text to be handles during the next operation.
Implemented in OpenSSLCryptoSymmetricKey, and WinCAPICryptoSymmetricKey. |
|
|
Finish a decryption operation. Complete a decryption process. No cipher text is passed in, as this should simply be removing any remaining text from the plain storage buffer. May throw an exception if there is some stored cipher text that is not the length of the block size for block algorithms.
Implemented in OpenSSLCryptoSymmetricKey, and WinCAPICryptoSymmetricKey. |
|
|
Initialise an decryption process. Setup the key to get ready for a decryption session. Callers can pass in an IV. If one is not provided, but the algorithm requires one (e.g. 3DES_CBC), then implementations should assume that the start of the cipher text stream will in fact be the IV.
Implemented in OpenSSLCryptoSymmetricKey, and WinCAPICryptoSymmetricKey. |
|
|
Continue an encryption operation using this key. Encryption must have been set up using an encryptInit call. Takes the inBuf and continues a encryption operation, writing the output to outBuf. This function does not have to guarantee that all input will be encrypted. In cases where the input is not a length of the block size, the implementation will need to hold back plain-text to be handled during the next operation.
Implemented in OpenSSLCryptoSymmetricKey, and WinCAPICryptoSymmetricKey. |
|
|
Finish a encryption operation. Complete a encryption process. No plain text is passed in, as this should simply be removing any remaining text from the plain storage buffer and creating a final padded block. Padding is performed by taking the remaining block, and setting the last byte to equal the number of bytes of padding. If the plain was an exact multiple of the block size, then an extra block of padding will be used. For example, if the block size is 8 bytes, and there were three remaining plain text bytes (0x01, 0x02 and 0x03), the final block will be : 0x010203????????05
Implemented in OpenSSLCryptoSymmetricKey, and WinCAPICryptoSymmetricKey. |
|
|
Initialise an encryption process. Setup the key to get ready for a decryption session. Callers can pass in an IV. If one is not provided, but the algorithm requires one (e.g. 3DES_CBC), then implementations are required to generate one.
Implemented in OpenSSLCryptoSymmetricKey, and WinCAPICryptoSymmetricKey. |
|
|
Returns the type of this key.
Reimplemented from XSECCryptoKey. |
|
|
Returns a string that identifies the crypto owner of this library.
Implements XSECCryptoKey. Implemented in OpenSSLCryptoSymmetricKey, and WinCAPICryptoSymmetricKey. |
|
|
What type of symmetric key is this? There are a number of different types of symmetric key. This method allows callers to determine the type of this particular key Implemented in OpenSSLCryptoSymmetricKey, and WinCAPICryptoSymmetricKey. |
|
|
Set the key from the provided bytes. Symmetric keys can all be loaded from a buffer containing a series of bytes.
Implemented in OpenSSLCryptoSymmetricKey, and WinCAPICryptoSymmetricKey. |


?0
1
cm
7
8
9
10
ã
XT
v
The
New
York
copyright
reserved
botanical
Garden
//
m
-/
f
/
mm
jƒL
0
f/i
\|
am*
7/1
x'
u)\
The
New
York
Botanical
Garden
Plants
of
PUERTO
RICO
S
c
ro
p
h
u
lariaceae
pkyltujty
TrotocQVi^,
(l
u
Mun.
Mayagiiez,
Mayagiiez
urbano,
campus
of
University
of
Puerto
Rico.
Cultivated
ornamental
plants.
18¡
12Õ30"N,
67¡08ÕW.
alt.
25
m.
Shrub
2.5
m.
tall,
with
gray-green
foliage.
Flowers
few;
corolla
lavender.
Coll.:
M.
Nee
44248
19
Nov.
1993
01008215
Legend -
Level of confidence that token is an accurately-transcribed word
extremely low
very low
low
undetermined
medium
high
very high


Jamie is a psychology PhD student based at the University of Sussex.
The categorize animation plugin shows a sequence of images at a specified frame rate. The subject responds by pressing a key. Feedback indicating the correctness of the response is given.
Parameters with a default value of undefined must be specified. Other parameters can be left unspecified if the default value is acceptable.
| Parameter | Type | Default Value | Description |
|---|---|---|---|
| stimuli | array | undefined | Each element of the array is a path to an image file. |
| choices | array of keycodes | jsPsych.ALL_KEYS |
This array contains the keys that the subject is allowed to press in order to respond to the stimulus. Keys can be specified as their numeric key code or as characters (e.g., 'a' , 'q' ). The default value of jsPsych.ALL_KEYS means that all keys will be accepted as valid responses. Specifying jsPsych.NO_KEYS will mean that no responses are allowed. |
| key_answer | numeric | undefined | A numeric key code indicating the correct response. |
| text_answer | string | ”” | A text label that describes the correct answer. Used in conjunction with the correct_text and incorrect_text parameters. |
| correct_text | string | “Correct.” | String to show when the correct answer is given. Can contain HTML formatting. The special string %ANS% can be used within the string. If present, the plugin will put the text_answer for the trial in place of the %ANS% string (see example below). |
| incorrect_text | string | “Wrong.” | String to show when the wrong answer is given. Can contain HTML formatting. The special string %ANS% can be used within the string. If present, the plugin will put the text_answer for the trial in place of the %ANS% string (see example below). |
| frame_time | numeric | 500 | How long to display each image (in milliseconds). |
| sequence_reps | numeric | 1 | How many times to show the entire sequence. |
| allow_response_before_complete | boolean | false | If true, the subject can respond before the animation sequence finishes. |
| prompt | string | null | This string can contain HTML markup. Any content here will be displayed below the stimulus. The intention is that it can be used to provide a reminder about the action the subject is supposed to take (e.g., which key to press). |
| feedback_duration | numeric | 2000 | How long to show the feedback (milliseconds). |
In addition to the default data collected by all plugins, this plugin collects the following data for each trial.
| Name | Type | Value |
|---|---|---|
| stimulus | JSON | JSON encoded representation of the array of stimuli displayed in the trial. |
| key_press | numeric | Indicates which key the subject pressed. The value is the numeric key code corresponding to the subject’s response. |
| rt | numeric | The response time in milliseconds for the subject to make a response. The time is measured from when the stimulus first appears on the screen until the subject’s response. |
| correct | boolean | true if the subject got the correct answer, false otherwise. |
var animation_trial = {
type: 'categorize-animation',
stimuli: ["img/face_3.jpg", "img/face_2.jpg", "img/face_4.jpg", "img/face_1.jpg"],
choices: [80, 81], // 80 = 'p', 81 = 'q'
key_answer: 81, // correct answer is 'q' for both trials
};
%ANS%
stringvar animation_trial = {
type: 'categorize-animation',
stimuli: ["img/face_3.jpg", "img/face_2.jpg", "img/face_4.jpg", "img/face_1.jpg"],
choices: [80, 81], // 80 = 'p', 81 = 'q'
key_answer: 81, // correct answer is 'q' for both trials
text_answer: 'Dax', // the label for the sequence is 'Dax'
correct_text: 'Correct! This was a %ANS%.',
incorrect_text: 'Incorrect. This was a %ANS%.'
};


==== Configured Wireless Networks and Keys ====
By finding the targets wireless information it becomes possible to launch physical attacks through the companies wifi when on site. It can also allow a fake AP to be set up to lure targets to connect when away from site.


GİZEM HANÇERLİOĞLU
Software Engineer
Izmir, TR
<EMAIL_ADDRESS>Work Experience
Please contact me to learn about my work experiences and projects I have done throughout my education life.
I would love to meet you on this matter.
Languages
English
German
Management Information Systems
Software Engineering
Web Application Development
Javascript Technogies
Mobile Application Development
Testing Technologies
Microsoft SQL Server
Object Oriented Programming, Design Patterns
Creating UML (Unified Modeling Language) Diagram
Creating Sequence and Collaboration Diagrams
Software Development Processes


--- layout: post date: 2010-12-31 time: "09:00:00" authors: ["Greg Wilson"] title: Software Carpentry Bootcamp Jan 12-14 in Madison tags: ["Community", "University of Wisconsin - Madison"] ---
Registration is now open for a three-day Software Carpentry bootcamp Jan 12-14, 2011, at the University of Wisconsin — Madison, organized by the folks at The Hacker Within. I'll be speaking on the first morning, and hanging out the rest of the time trying to learn as much as I can. Look forward to seeing you there!


Spark Plugin Framework in 3.0 - Part 3 : Dynamic Stream Configuration using Driver Plugin
Spark 3.0 brings a new plugin framework to spark. This plugin framework allows users to plugin custom code at the driver and workers. This will allow for advanced monitoring and custom metrics tracking. This set of API’s are going to help tune spark better than before.
In this series of posts I will be discussing about the different aspects of plugin framework. This is the third post in the series, where we will discuss about how to implement the dynamic configuration in spark streaming using driver plugin. You can read all the posts in the series here.
Dynamic Configuration in Spark Streaming
Spark Streaming is a long running application which processes the incoming data. The streaming applications usually start with an initial configuration. But that configuration may change as the time goes. User doesn’t want to stop and restart the spark streaming application for these changes. They would like to it to be updated on the fly.
Currently this is implemented by reading configurations from an external service on every micro batch. But this introduces the additional delay in the processing. It will be nice to have asynchronous update to the configuration which can be picked up at the time execution.
This post shows how to implement this using driver plugin of spark plugin framework.
Defining a Configuration
The below code defines a simple configuration which holds a single value. In real world scenario, this can be complex object like JSON.
In above code, there is initial value and then methods to read/write new configuration.
Defining a Custom Spark Plugin
The below code defines a custom spark plugin
As we need to only change configuration from the driver, we return null fro executor plugin.
Implementing Driver Plugin
This section of the post we will discussing different parts of the driver plugin implementation.
Implementing Driver Plugin Interface
First step is to implement driver plugin interface.
Overriding Init Method
Once we extend the driver plugin interface, we override init method.
In the init method, we run a socket listener in new thread. This runs in new thread so that it doesn’t block the driver operations. We start the thread and leave till it shutdown.
Socket Listener for Configuration Changes
The below code shows a simple socket listener which changes configuration whenever it’s contacted. Here we are doing an incremental to configuration changes to keep it simple.
Overriding Shut Down Method
When driver shut downs, we stops the listener
Setting Custom Spark Plugin in Spark Session
The below code is added in main program, to add the spark plugin to spark session
Using Configuration in Spark Streaming
In above code, we are reading the data from a socket. For every data received, we are printing the latest configuration.
Running Example
This section shows how the changes of configuration is shown.
Initial Run
When the first message **hello world ** is sent on socket, then you can see the below result.
Updating Configuration
You can update configuration by sending curl request at http://localhost:9999
It will print
New Configuration in Spark Streaming,
If you send the hello world again in the socket, you will get below result
As you can see that streaming program has latest configuration now.
Code
You can access complete code on github.
Conclusion
Spark plugin framework brings a powerful customization to spark ecosystem. In this post, we discussed how to use driver plugin to implement dynamic configuration spark streaming.


Fresh Raspbian Installation
Fresh Raspbian Image
Burn the image in an SD Card
You can use my script: https://raw.githubusercontent.com/TediCreations/.dotfiles/master/files/bin/utils/writeSDCard
writeSDCard ~/rasbian.img
Enable headless ssh
- Add a file named “ssh” inside the /boot directory.
- Add a file named “wpa_supplicant.conf” inside the /boot directory with the following contents.
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=us
network={
ssid="<Name of your WiFi>"
psk="<Password for your WiFi>"
scan_ssid=1
}
Make it secure
Change the default password of user pi
passwd
Make a new user and lock pi user
sudo adduser tedi
sudo usermod -a -G adm,dialout,cdrom,sudo,audio,video,plugdev,games,users,input,netdev,gpio,i2c,spi tedi
sudo su - tedi
sudo pkill -u pi
sudo passwd --lock pi
pi user is locked and not deleted because Raspbian seems to have a lot of service dependencies on it. Beware that logins to pi using password is not possible anymore, but possible via ssh with a key.


Prueba de Tangle
When you eat
cookies
, you consume
calories
.
That's
%
of your recommended daily calories.


I'm afraid you need JavaScript to play this game. Sorry about that!
Learn more about the game in general


There are N light bulbs that are placed on a straight wire.
Each bulb has a switch associated with it, which turns it on or off.
There is one caveat though, turning a bulb on or off will change the state of all the bulbs to its right too.
What's the minimum number of switch toggles needed to turn all the bulbs on.0
represents the off state and 1
represents the on state.
Example:
a = [0, 0, 0, 1]
answer = 2
We switch the bulb at index 0 and 3.
We can solve this problem using the greedy approach.
It can be observed that moving from left to right or right to left doesn't make a difference to the final number of toggles needed.
Algorithm Overview:
1. Start from the left and move towards right in the array.
2. If the current value is 0, we need to toggle this bulb.
3. Store that all the subsequent bulbs must be flipped.
4. If current value is 1, continue towards right.


Six dead in light plane crash at DR Congo city
by NANAt least six bodies were recovered on Sunday after a small plane believed to have been carrying 19 passengers and crew crashed on takeoff into a densely populated area of Goma in the Democratic Republic of Congo.
An official at Goma airport, Richard Mangolopa, told AFP no survivors were expected.
“A number of bodies have just been recovered. The (rescue) teams are continuing to search,” added Goma mayor Timothee Muissa Kiesse.
The Dornier-228 aircraft had been headed for Beni, 350 kilometres (220 miles) north of Goma when it went down in a residential area near Goma airport.
“There were 17 passengers on board and two crew members. It took off around 9-9.10 am (0700 GMT),” Busy Bee airline staff member Heritier Said Mamadou said.
Busy Bee, a recent company, has three planes serving routes in North Kivu province.
One of the company’s maintenance workers at the site quoted by news site actualite.cd blamed a “technical problem”.
It was not yet known if there were any casualties on the ground.


DEAD MAN WALKING A film review by Rui Sadio Translated from the Portuguese by Rui Sadio Copyright 1996 Rui Sadio
Rating (0 to 10): 9.1
Since I remember, I can't find any movie so capable to obligue an audience to reflect about a certain subject, in such a powerful way as "Dead Man Walking" makes us to reflect about the death penalty question. First of all, I have to say that this movie is, in my opinion, the best movie of 1995 and certainly one of those which I won't forget easily.
"Dead Man Walking" is based on the book with the same name by Sister Helen Prejean, a nun who has worked with condemneds to death for a long time. Directed by Tim Robbins, one of my favourite actors, this film has also the brilliant and unforgetable performances of Sean Penn and Susan Sarandon.
Sean Penn is Matthew Poncelet, a condemned to death who receives spiritual help of Sister Helen Prejean, a role interpreted by Susan Sarandon, in his last days of life. Both artists are sensational in their performances and that happens because they don't act separately. I mean, the Sean Penn's interpretation "lives" from the Susan Sarandon's interpretation and vice versa. Is something like Jodie Foster and Anthony Hopkins in "The Silence of the Lambs". However, now the dialogues between the two principal characters are even more dramatic and emotional than those in the Jonathan Demme's film. That's why I think Susan Sarandon and Sean Penn should win the oscars of the Academy, this year. The most significant part of the movie's power is due to their work.
"Dead Man Walking" is too far of being an anti-deathpenalty movie. On the contrary, "Dead Man Walking" follows a guide-line which divides the two opposite opinions, becoming the frontier between those convictions. There are times when we think that death penalty is a fair way to punish a person who has already destroyed the life of so many people. However, there are times when we start believing that a death can't pay another death. After all, if no one has the right to kill then we shouldn't apply the death penalty, or else we should be punished too.
Tim Robbins is the responsible for this revolution inside our minds. He manipulates us as he wants and he never discloses what is his position. He makes us be sorry for the condemned but he makes us hate the condemned! All the directing work is superb, including the beginning of the film, showing that Tim Robbins is more than an excellent actor. In fact, he is also an excellent director.
At last, a reference to the mysterious music that emphasizes the dark side of the film. Songs which are intimately connected with the characters and the movie environment. We can listen to Bruce Sprinsteen, Johnny Cash, Suzanne Vega, Nusrat Fateh Ali Khan, Eddie Vedder and many others. After seeing the movie is very important to listen to the soundtrack.
"Dead Man Walking" is one of those movies that everybody should see and reflect about. The death penalty question deserves that reflection. About my position, I just say I'm happy for living in Portugal, one of the first countries to abolish that kind of punishment.
-R.S.-
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Converts a sparse representation into a dense tensor. (deprecated)
tf.compat.v1.sparse_to_dense( sparse_indices, output_shape, sparse_values, default_value=0, validate_indices=True, name=None )
Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version.
Instructions for updating:
Create a tf.sparse.SparseTensor
and use tf.sparse.to_dense
instead.
Builds an array dense
with shape output_shape
such that
# If sparse_indices is scalar dense[i] = (i == sparse_indices ? sparse_values : default_value) # If sparse_indices is a vector, then for each i dense[sparse_indices[i]] = sparse_values[i] # If sparse_indices is an n by d matrix, then for each i in [0, n) dense[sparse_indices[i][0], ..., sparse_indices[i][d-1]] = sparse_values[i]
All other values in dense
are set to default_value
. If sparse_values
is a scalar, all sparse indices are set to this single value.
Indices should be sorted in lexicographic order, and indices must not
contain any repeats. If validate_indices
is True, these properties
are checked during execution.
sparse_indices
: A 0-D, 1-D, or 2-D Tensor
of type int32
or int64
.
sparse_indices[i]
contains the complete index where sparse_values[i]
will be placed.output_shape
: A 1-D Tensor
of the same type as sparse_indices
. Shape
of the dense output tensor.sparse_values
: A 0-D or 1-D Tensor
. Values corresponding to each row of
sparse_indices
, or a scalar value to be used for all sparse indices.default_value
: A 0-D Tensor
of the same type as sparse_values
. Value
to set for indices not specified in sparse_indices
. Defaults to zero.validate_indices
: A boolean value. If True, indices are checked to make
sure they are sorted in lexicographic order and that there are no repeats.name
: A name for the operation (optional).Dense Tensor
of shape output_shape
. Has the same type as
sparse_values
.


Welcome, {{ username }}!
Watch for the Plot
Leagues where you are Administrator
{% for row in administrator%}
| Leagues |
| row[0] |
Enter |
{% endfor %}
Leagues where you are a Member
{% for row in member%}
| Leagues |
| row[0] |
Enter |
{% endfor %}
Leagues in which you are not a Member
{% for row in uninvolved%}
| Leagues |
| row[0] |
Enter |
{% endfor %}
Quote of the day: If it is to be, it is up to me.


Virtual Coud Directory
Virtual directory
Be part of a team to create a multicloud virtual directory.
Your goal will be to to develop an API, secure REST, and commandline tool that easily interfaces with storage servoces such as file based services and object stores osted on the internet to create a virtual directory that accesses the files by directory name and filename.
This is espaciellyusefull for data scientists that may want to strore their data on multiple cloud providers and eliminate vendor lockin.
It could also be useful for geographically distributed files that allow services to utilize them based on speed of access or the physical location.
In a previous version of cloudmesh we have developed an integrated approach for compute and storage services. However in this project we like to explore some new features and separate the implementation form the main cloudmesh.
In the past cloudmesh has conducted two such projects
- https://cloudmesh.github.io/cloudmesh-manual/manual/storage.html
- https://cloudmesh.github.io/cloudmesh-manual/manual/vdir.html
We like to reimplement thise projects and significantly improve them by
- Adding more providers
- Adding a secure rest service for acceassing the virdual directory
- Creating a slick user interface to brows the virdula directry
- Put everything in a container so it can be run on Linux, MAc and Windows.
- Dreate a convenient commandline tool that allows starting of the service, interacting with it and make this real easy to use. THe commandline will hide the docker commands while providing human readable abbreviations.
- Deliver unit tests with pytests
- Deliver a high quality report including benchmarks
Requirements
In order for you to participate in this project you will need:
- A computer on which you can run docker (Windows Home will not work)
- Significant python knowledge
- Be highly motivated
- Be willing to have meetings on this project once or twice a week
- SHowcase significant progress over the lifetime of the project.
- be knowledgable with github (a repository will be provided to which Dr. von Laszewski will contribute)
- Conduct task management in GitHub (Gregor will explain)
- Be honest and not hiding problems or implementation bugs.
- You must be able to do a videoconference and be able to share your screen (I typically use google meet or zoom).
Options
One of the two projects used a single file taansfer logic, while the other used a parallel file transfer option. It will be up to you to decide how to proceed.
Getting started
-
In the first week collect all clouds and services that could be used in this project. THis work should be split between the teammembers and a detailed analysis including features including authentication and security must be conducted.
-
You will need to evaluate the previous efforts and decide how to proceed.
One way is to select several cloudas and define an abstraction that has to be implemented for each cloud. this can be actually bes demonstrated by an example commandline such as
vdir mkdir DIR vdir cd [DIR] vdir ls [DIR] vdir add [FILEENDPOINT] [DIR_AND_NAME] vdir delete [DIR_OR_NAME] vdir status [DIR_OR_NAME] vdir get NAME DESTINATION
-
From the commandline derive some API interface and use FastAPI to implement it.
-
Identify a mechnism on how to deal with the security. AN encrypted file with your cloud credentials will be sufficient.
-
One of the limitations of the previous vdir project is that the fill add function is done only on a single file but not on a recursive file tree. Naturally we need to upload, single files, multiple files and filetrees, similar to the unix command rsync. This has been implemented in the storage command
storage [--storage=SERVICE] [--parallel=N] create dir DIRECTORY storage [--storage=SERVICE] [--parallel=N] get SOURCE DESTINATION [--recursive] storage [--storage=SERVICE] [--parallel=N] put SOURCE DESTINATION [--recursive] storage [--storage=SERVICE] [--parallel=N] list [SOURCE] [--recursive] [--output=OUTPUT] storage [--storage=SERVICE] [--parallel=N] delete SOURCE storage [--storage=SERVICE] search DIRECTORY FILENAME [--recursive] [--output=OUTPUT] storage [--storage=SERVICE] sync SOURCE DESTINATION [--name=NAME] [--async] storage [--storage=SERVICE] sync status [--name=NAME] storage config list [--output=OUTPUT] storage [--parallel=N] copy SOURCE DESTINATION [--recursive]
We like naturally to (re-)implement the storage command.
-
Before you start implementing describe and showcase a couple of commands on how to used it. Our command will be called cdir (for cloudmesh dir so it is easy to distinguish)
-
Make sure that you provide a comprehensive list of potentail cloud storage ervices. Distinguish between for pay and free services.
The list could include:
- AWS Drive
- AWS Object Store
- An ssh accesible computer
- Azure
- Box
- Degoo
- Dropbox
- Google Drive
- Google docs/drive
- Icedrive
- MEGA
- MediaFire
- MediaFire
- Mega
- Nextcloud
- OneCloud
- OneDrive
- Oracle
- Sync.com
- WebDAV
- Your local computer
- iCloud
- iDrive
- idrive
- pCloud
-
Are there companies or services that already offer this?
-
When defining the REST API we like to be able to use a dirname and a basename within the api similar to python. to deal with provider specific filenames we do have two urls for a file.
{provider}/{dirname}/{basename}
{dirname}/{basename}
The first allows duplication of files between services, the latter defines a “preferred” url based on some criteria. It could be that we just set the preferred provider for the file.
Note that the introduction of duplicating files is new
It also requires an update so that the file with the newest timestamp gets updated on all registered serrvices.
-
Please note that the project addresses the ability to integrate object store and regular file system based storage.
-
Not all implementation must be done on Ubuntu20.04 which is run via a container. The graphical component is done wia moder Web view technologies. Wile all other implementation can be done in python, the GUI can also be implemented in JavaScript. The service accessing other services must be properly protected which is easy as we assum it runs on localhost and we can appropriately secure it with common solutions.
In case of qeuestions, pleas econtact Gregor at


Our mission, vision, passion and greater purpose for the future. We have a community that wants to make a change that can create not just a better place for you but for the rest of the society as well. If you dream it you can achieve it and that is the overall mood we want to bring to the table. In addition, we want to reawaken the power of confidence and the ability to achieve greater possibilities. Our target is to not only provide a superior amount of support towards the youth generation but to the future generation of leaders to come. To be able to accelerate something stronger that has an everlasting support system. To the youth and every one our goal is not just giving people more power but also a system where they truly rely on.


I'm Guoyan Weng, and you can call me Lloyyd(the same pronunciation as "Lloyd").
I'm a product designer working across user testing, UX design, UI design, prototyping and data analysis to help the product become beautiful, user-friendly and valuable.
Check my resume here:
ResumeI think the good design is intuitive, helpful and beautiful.
I think the design is not just about doing things right but also about do the right things. A successful design is the one that matches the user's demand and keeps the product/team/company growing.
I think the designer is the bridge between the manager and the executor(programmer or PR). So knowing the source and limitation of both parts is necessary for making an executable design.
We are friends if you like snowboard.
We are friends if you like Nintendo.
We are friends if you have a Shiba.
We are FAMILIES if you have a cat :)


Home
Products
About
Contact
Account
Registration Form
Registration Form
First Name
Middle Name
Last Name
Password
Confirm Password
Gender
Select
Male
Female
Email Address
Phone Number
Address
Postal Code
Agreed to terms and conditions


|
|
Connect equipment button Activates the PHD2 profile that
defines your equipment connections. If you want to connect the
same equipment that was last connected, you can shift-click this button.
|
|
Loop exposures button
Starts a sequence of repeated guide exposures with the guide camera (looping),
with each resulting image (guide frame) being displayed in the display
window. If guiding is subsequently started, clicking on the Loop Exposures button again will pause guiding while continuing to take guide exposures.
|
|
Auto-select Star button
Triggers an automatic selection of the best guide star candidates in the display
window. This selection is done quantitatively, taking many things into
account - star saturation, minimum star size, signal-to-noise ratio,
proximity to other stars, proximity to the edge of the display window. etc. You can shift-click on this button to de-select the star.
|
|
|
Guide button
Starts calibration, if needed, and then starts guiding on the selected stars.
|
|
|
Stop button
Stops both guiding and looping.
|
|
Exposure duration drop-down
A drop-down list of exposure durations in seconds (Auto, 0.01s to 15s, and a Custom: option which you can assign your own value to using the Edit Custom...
option). Use this drop-down to set the guide camera's exposure
duration. If your camera does not support an exposure duration, PHD2
will do its best to emulate that duration. For example, if you use a
short exposure webcam, your maximum true exposure duration might be only
1/30th of a second. If you select one second as the exposure time, PHD2 will automatically acquire images for one second and stack them to create a composite image for guiding.
|
|
Gamma adjustment slider
A slider control for adjusting screen stretch and contrast, essentially a "gamma" adjustment. PHD2
automatically adjusts the display to account for the darkest and
brightest pixels in the image, and the slider is used to fine-tune
the display to better see the stars. This may be useful, for
example, if you are trying to focus the guide camera for the first time
and need to see the large, out-of-focus star image. Moving the
gamma slider only makes the display brighter or dimmer for your
viewing. PHD2
always uses the raw pixels from the camera for guiding, and moving the
gamma slider has no effect on guiding. A display of
completely white or completely black is usually an indication that no
stars are available in the field of view or the camera is out-of-focus.
The automatic gamma-stretching of the image is a very common
source of confusion for users because they do not adjust the gamma
slider properly to see faint stars on the display window.
|
|
Advanced settings (Brain Button)
This brings up the Advanced Settings dialog for making detailed adjustments to the guiding operations. An important design goal of PHD2
is to minimise your need to change these parameters, but there are
adjustments available here that can help you understand and improve your
guiding.
|
|
Camera settings buton
Depending on the camera, this button may be enabled to provide access to a configuration dialog unique to the camera.
|


Fuzzy set ordination is a multivariate analysis used in ecology to relate the composition of samples to possible explanatory variables. While differing in theory and method, in practice, the use is similar to 'constrained ordination.' The package contains plotting and summary functions as well as the analyses.
| Version: | 2.1-1 |
| Depends: | labdsv |
| Published: | 2018-06-01 |
| Author: | David W. Roberts |
| Maintainer: | David W. Roberts <droberts at montana.edu> |
| License: | GPL-2 | GPL-3 [expanded from: GPL (≥ 2)] |
| URL: | http://ecology.msu.montana.edu/labdsv/R/labs/lab11/lab11.html |
| NeedsCompilation: | yes |
| Materials: | ChangeLog |
| In views: | Environmetrics, Multivariate |
| CRAN checks: | fso results |
| Reference manual: | fso.pdf |
| Package source: | fso_2.1-1.tar.gz |
| Windows binaries: | r-devel: fso_2.1-1.zip, r-release: fso_2.1-1.zip, r-oldrel: fso_2.1-1.zip |
| OS X binaries: | r-release: fso_2.1-1.tgz, r-oldrel: fso_2.1-1.tgz |
| Old sources: | fso archive |
Please use the canonical form https://CRAN.R-project.org/package=fso to link to this page.