Baseline Fitting¶
There are a number of cool features in baselining that aren’t well-described below, partly due to Sphinx errors as of 12/22/2011.
exclude and include allow you to specify which parts of the spectrum to use for baseline fitting. Enter values as pairs of coordinates.
Excludefit makes use of an existing fit and excludes all points with signal above a (very low) threshold when fitting the baseline. Going back and forth between baseline(excludefit=True) and specfit() is a nice way to iteratively measure the baseline & emission/absorption line components.
API¶
- class pyspeckit.spectrum.baseline.Baseline(Spectrum) [github] [bitbucket]¶
Class to measure and subtract baselines from spectra.
While the term ‘baseline’ is generally used in the radio to refer to broad-band features in a spectrum not necessarily associated with a source, in this package it refers to general continuum fitting. In principle, there’s no reason to separate ‘continuum’ and ‘spectral feature’ fitting into different categories (both require some model, data, and optional weights when fitting). In practice, however, ‘continuum’ is frequently something to be removed and ignored, while spectral features are the desired measurable quantity. In order to accurately measure spectral features, it is necessary to allow baselines of varying complexity.
The Baseline class has both interactive and command-based data selection features. It can be used to fit both polynomial and power-law continua. Blackbody fitting is not yet implemented [12/21/2011]. Baseline fitting is a necessary prerequisite for Equivalent Width measurement.
As you may observe in the comments on this code, this has been one of the buggiest and least adequately tested components of pyspeckit. Bug reports are welcome. (as of 1/15/2012, a major change has probably fixed most of the bugs, and the code base is much simpler)
Methods
- __call__(*args, **kwargs) [github] [bitbucket]¶
Fit and remove a polynomial from the spectrum. It will be saved in the variable “self.basespec” and the fit parameters will be saved in “self.order”
Parameters : order: int :
Order of the polynomial to fit
excludefit: bool :
If there is a spectroscopic line fit, you can automatically exclude the region with signal above some tolerance set by exclusionlevel (it works for absorption lines by using the absolute value of the signal)
exclusionlevel: float :
The minimum value of the spectroscopic fit to exclude when fitting the baseline
save: bool :
Write the baseline fit coefficients into the spectrum’s header in the keywords BLCOEFnn
interactive: bool :
Specify the include/exclude regions through the interactive plot window
fit_original: bool :
Fit the original spectrum instead of the baseline-subtracted spectrum. If disabled, will overwrite the original data with the baseline-subtracted version.
Warning
If this is set False, behavior of unsubtract may be unexpected
fit_plotted_area: bool :
Will respect user-specified zoom (using the pan/zoom buttons) unless xmin/xmax have been set manually
reset_selection: bool :
Reset the selected region to those specified by this command only (will override previous xmin/xmax settings)
- __init__(Spectrum) [github] [bitbucket]¶
- __module__ = 'pyspeckit.spectrum.baseline'¶
- annotate(loc='upper left') [github] [bitbucket]¶
Do the baseline fitting and save and plot the results.
Wrapper - same as button2action, but with subtract=False
- clearlegend() [github] [bitbucket]¶
- copy(parent=None) [github] [bitbucket]¶
Create a copy of the baseline fit
- [ parent ]
- A spectroscopic axis instance that is the parent of the specfit instance. This needs to be specified at some point, but defaults to None to prevent overwriting a previous plot.
- crop(x1pix, x2pix) [github] [bitbucket]¶
When spectrum.crop is called, this must be too
- downsample(factor) [github] [bitbucket]¶
- get_model(xarr=None, baselinepars=None) [github] [bitbucket]¶
- plot_baseline(annotate=True, baseline_fit_color=(1, 0.65, 0, 0.75), use_window_limits=None, linewidth=1, alpha=0.75, plotkwargs={}, **kwargs) [github] [bitbucket]¶
Overplot the baseline fit
Parameters : annotate : bool
Display the fit parameters for the best-fit baseline on the top-left of the plot
baseline_fit_color : matplotlib color
What color to use for overplotting the line (default is slightly transparent orange)
use_window_limits : None or bool
Keep the current window or expand the plot limits? If left as None, will use self.use_window_limits
Other Parameters: linewidth : number
alpha : float [0-1]
plotkwargs : dict
Are passed to matplotlib’s plot function
- savefit() [github] [bitbucket]¶
- set_basespec_frompars() [github] [bitbucket]¶
Set the baseline spectrum based on the fitted parameters
- set_spectofit(fit_original=True) [github] [bitbucket]¶
Reset the spectrum-to-fit from the data
- unsubtract(replot=True, preserve_limits=True) [github] [bitbucket]¶
Restore the spectrum to “pristine” state (un-subtract the baseline)
- replot [ True ]
- Re-plot the spectrum? (only happens if unsubtraction proceeds, i.e. if there was a baseline to unsubtract)
- preserve_limits [ True ]
- Preserve the current x,y limits


retract/1
WARNING: retract/1
is a nondeterminate procedure. Thus, we can use
| ?- retract((foo(X) :- Body)), fail.
to retract all clauses for foo/1
. A nondeterminate procedure
in SICStus Prolog uses a choicepoint, a data structure kept
on an internal stack, to implement backtracking. This applies to
user-defined procedures as well as to built-in and library procedures.
In a simple model,
a choicepoint is created for each call to a nondeterminate procedure,
and is deleted on determinate success or failure of that call, when
backtracking is no longer possible.
In fact, SICStus Prolog improves upon this simple model by recognizing
certain contexts in which choicepoints
can be avoided, or are no longer needed.
The Prolog cut (`!') works by removing choicepoints, disabling the potential backtracking they represented. A choicepoint can thus be viewed as an “outstanding call”, and a cut as deleting outstanding calls.
To avoid leaving inconsistencies between the Prolog database and outstanding calls, a retracted clause is reclaimed only when the system determines that there are no choicepoints on the stack that could allow backtracking to the clause. Thus, the existence of a single choicepoint on the stack can disable reclamation of retracted clauses for the procedure whose call created the choicepoint. Space is recovered only when the choicepoint is deleted.
Often retract/1
is used determinately; for example, to retract a single
clause, as in
| ?- <do some stuff> retract(Clause), <do more stuff without backtracking>.
No backtracking by retract/1
is intended. Nonetheless, if
Clause may match more than one clause in its procedure, a choicepoint
will be created by retract/1
. While executing “<do more stuff
without backtracking>”, that choicepoint will remain on the stack,
making it impossible to reclaim the retracted Clause. Such
choicepoints can also disable tail recursion optimization. If not cut
away, the choicepoint can also lead to runaway retraction on the
unexpected failure of a subsequent goal. This can be avoided by
simply cutting away the choicepoint with an explicit cut
or a
local cut (`->'). Thus, in the previous example, it is
preferable to write either
| ?- <do some stuff> retract(Clause), !, <do more stuff without backtracking>.
or
| ?- <do some stuff> ( retract(Clause) -> true ), <do more stuff without backtracking>.
This will reduce stack size and allow the earliest possible reclamation of retracted clauses.


Introduction
Imagine a world in which every programmer out there knows how to write code without any errors, perfectly working on the first time and super easy and clear to read for other programmer colleagues. That is a dream world, in which every programmer of us would like to go. Writing clear code is a hard task itself, even the most professional programmers out there can do mistakes and it's a normal and common thing.
Well then, chances are that you used drive documents or word itself to write some essay, either in high school, university or whatever. I am sure you are an expert in writing on your own language, or at least that is what I hope. Word or drive gives you a tool to detect possible grammar mistakes, such as a missing letter, a misspelled word, etc... And chances are that even if you're very good at your language, you still use that tool, because why not, right? It gives you confidence and helps you detect errors that maybe you missed. ¿Wouldn't that be amazing for programming purposes? ¿And why do I tell you all of this? Well, this was a brief introduction to understand what Code Analysis is! It's a tool made with the purpose of detecting possible errors at the time you code.
¿What is static code Analysis?
Static code analysis is similar to the example stated above, but a bit different, if you misspell a variable, if you declare two variables with the same name, the compiler is just going to fail and it's going to tell you, that's the purpose, it only tries to make sense of what you code. But a compiler wont understand what your code intention was, the purpose of the code you've written, it just tries to compile what you've written, it will not try to look at all your code, analyze it and try to make sure that what you've written is what your intentions were. That is what an Static Analyzer is, a tool that will look at your code and try to catch unintentional errors without having to run the program itself.
Static analysis is good at finding coding issues like:
• Programming errors
• Syntax violations
• Security vulnerabilities
• And more
¿But what problems does Static Code Analysis solve?
Let's take a look at this piece of code:
Seems good and okay, right?
Well, if you take a slower look, you will notice one mistake that I am sure most of the programmers out there have done in the past at least once. You have to do a double for, you copy and paste the previous for, and you forget to change one of the letters that makes it work as you intended. Here, the problem is that the compiler will not say anything, because the code itself is not "wrong", it's perfect, but the problem is that those were not your intentions at all, you wanted to do another thing. Here, we will run our static code analyzer program, in my case I am using PVS-Studio. After running our static code analyzer program, we will get the following warnings for this code:
As you can see, it warns us of two pontential mistakes, the first message says: "The condition 'j < 20' of loop is always true" and the second one says: "It is likely that a wrong variable is being incremented inside the 'for' operator. Consider reviewing 'i'." So in this case, PVS-studio warned us that our for may not work as we intended, and with two simple warnings, the program saved us from maybe not finding the mistake. In this case, of course, the mistake is very small, but if you have thousands of lines of code, it can save you a lot of time!
Let's take a look at another example
Now this one is a bit more tricky to catch, I created a class that I start with the function StartEntity, the mistake is that I "forgot" to return an actual player, it just returns "nullptr". Then, at the bottom I coded that if the player is true, the program should tell us that it created the player. Now, we again run PVS-Studio, our static analyzer. After running it, it says the following:
"Expression 'player' is always false." The static analyzer warned us that the variable is always false, and that we may check that. Again, we can see that this could save us a lot of time when having thousands of lines of code. It may seem like a basic mistake but no one is perfect, it could happen to anyone.
How is static analysis done?
The static analysis process occurs before software testing in early development. Once the code is already written, we should use the static code analyzer to look over the code. The program will find possible problems and developers can begin to fix any apparent mistakes, usually we should start by fixing the most problematic ones. Once we resolved all the issues we can move to testing through execution.
Why should I use static analysis?
Imagine not having a tool that does the static analysis for us, it could take a lot of work and not only that, a lot of time, because us, humans, have to review the code and figure out "How will it behave when it compiles?". Then having a tool that does this for us is much safer and much much faster, making it more efficient for us.
Advantages and disadvantages
| Advantages | Disadvantages |
| Can evaluate all the code in your program | It can detect false positives |
| Much faster than reviewing code manually | May not find a mistake, tools are not perfect |
| Less likely to make mistakes than humans | Third party libraries may not be able to be analyzed |
| It can be done in an offline development environment | Could take some time analyzing |
Types of static analysis
There are some methods a company could use, which are the following:
• Pattern-Based Static Analysis: The tool scans and checks it against one or more set of rules. For example a programmer could have mistaken the null character '\0' and instead typed "\0"
• Fault analysis: Analyzes faults and failures in model components
• Interface Analysis: Verifies interactive and distribution simulations to check the code
Free static analyzers
Here you have some free static analyzers
| Frama-C | Cpp check | Eclipse |
Premium static analyzers
Here you have some premium static analyzers
| PVS-Studio (Personal best choice) | Coverity | Code Sonar |
Other software for static analyzers
Additionally, you can check the wiki for other analyzers, there is a table with a lot of them which are also marked as free or premium in the link below.
• Go to static analyzers' wiki
Sanitizers
Now there are another programs that can help you that are used in runtime to report execution errors, those are called "Sanitizers".
We can find some different types of sanitizers, such as:
1. Adress Sanitizer and LeakSanitizer: Adress Sanitizer is a memory detector for C/C++, both detect the following errors:
• Heap buffer overflow.
• Stack buffer overflow.
• Global buffer overflow.
• Memory leaks.
2. Thread sanitizer: It's a data race detector for C/C++. Data race occurs when two threads access the same variable concurrently
and at least one of the accesses is writen.
3. Memory sanitizer: It detects uninitialized memory reads in C/C++ programs. Those uninitialized values occur when stack or heap allocated memory
is read before it is written.
Note: you can find AddressSanitizer (ASan) in this following link:
ASan
References
•What is static analysis? Code Analysis Tools


A JavaScript project containing standalone javascript widgets based on Bootstrap, jQuery UI and other third-party widgets, styled with LESS.
Begin by setting you your environment and making sure you can build RichWidgets as described the README.md.
Widgets are highly visual constructs, and we are more effective in developing them when we get rapid feedback as we introduce changes. Without this rapid feedback, we are like sculptors carving blindfolded!
Run the grunt dev
command from the richwidgets root folder to start up a development version of the online demo with
integrated live-reload. You will see this demo runnin locally at:
As you make changes to the html, javascript, and LESS source, the source will be compiled and the demo site will be updated.
Additionally, this development version of the demo uses unminified versions of the sources making it easier to debug your widget in your browsers development tools.
The demo pages themselves use handlebars.js for templating. This allows us to keep our demo code DRY by using the same code to both run the page sample as well as display the code on the page. Handlebars is easily extensible through it's helper mechanism. If you find yourself bending over backwards to make something work, consider writing such a helper.
Publish the demos to github pages using the grunt task grunt site
It helps to be consistent. Here are the coding style conventions we have agreed to follow in this project. this will be updated as new discrepancies are discovered and cleaned up. Wherever possible these conventions are being monitored by jshint.
popup-panel.js
)popup-panel
)src/widgets/variables.less
src/widgets/mixins.less
popupPanelWidth
, showPopup()
)'
for stringsself
for capturing a reference to this
widget
where it makes sense to do so, that
otherwisefunction name(param1, param2) {}
function (param1, param2) {}
rich
jquery plugin namespacerich.autocomplete
extending ui.autocomplete
)null
as default value is required_uiHash
_create()
, _destroy()
)showPopup()
)_setOption()
, _initDom()
, _bindListeners()
)_cleanDom()
)_enable()
, _disable()
, _uiHash()
)_uiHash()
)this._super(key, value)
or this._superApply(arguments)
for delegation to parent implementationTo create a new widget called "My widget", start by creating the files:
src/widget/<widget-family>/my-widget.less
src/widget/<widget-family>/my-widget.js
Create a demo for your widget in
src/demo/pages/<widget-family/my-widget.hbs
The demo pages use handlebars.js, and will be added to the site navigation automatically.
Read about our approach to testing in the TESTS.md guide.


New home page
Celebrities
Future of robotics
Famous People
From VR to ER
From VR to AR
Alexander Chislenko
Sasha Chislenko
Advertising your Web site
Some books that make good presents
Grandpa submission service
Grandpa website hosting service
Sasha's home page
Sasha's home page
Sasha's home page


This is a custom modal. The angular-modal-service doesn't depend on bootstrap, you can use any modal you want.
Close


Carmelita D. Marasigan currently serves as the coordinator and a teacher for the preschool levels at Bloomfield Academy in Las Piñas City. She graduated from the Philippine Normal University (PNU) with a Bachelor of Science in Education degree. She also took up units in elementary education at the said institution.
She has been an author and a coauthor of the Abiva Preschool Series since 2000. She has also attended several education-related seminars, and is a member of Gabay Guro, an organization that helps in the professional development of teachers through literacy training programs.
Elena B. Morales graduated with a Bachelor of Science in Elementary and Secondary Education degree at PNU. She obtained a certificate in Teaching English to Speakers of Other Languages from the Asian Seminary of Christian Ministries, and a certificate in Language Teaching from PNU.
She taught at Elizabeth Seton School and Merry Treasure School (MTS) in Las Piñas City. At present, she is a voluntary craft teacher and preschool teacher in China. She has also attended many seminars.
Lilybeth B. Roy is currently the coordinator and a teacher for the preschool levels at MTS in Las Piñas City. She completed her Bachelor of Science in Elementary Education degree with specialization in Preschool Education at PNU. She has been a preschool teacher for twenty years.
Dalrose Ann C. Sambo graduated from the University of Negros Occidental–Recoletos in 1984 with a Bachelor of Science in Mass Communications degree. In 1986, she completed units in her Bachelor of Science in Education degree at PNU, earning herself a certificate in Teaching Program.
She passed the Career Service Subprofessional Examination in 1982 and the Licensure Examination for Teachers (LET) in 2000. She taught for the preschool levels at Elizabeth Seton School in 1986 and MTS in 1992. She is currently a preschool teacher and, at the same time, the coordinator of the preschool department at the Cavite School of Life (CSL).
Carolina T. Tirona earned her Bachelor of Arts in Sociology degree from the University of Santo Tomas (UST) in 1979. She took up units in Education in 1981 and obtained a certificate in Preschool Specialization in 1983 from PNU. She also took up units in master’s degree in Psychology, major in Educational Management, at UST.
From being a teacher to a principal sums up Ms. Tirona’s career ladder. She taught all subjects in the preschool and primary levels at Elizabeth Seton School, Las Piñas City, from 1979 to 1992. During her tenure at the said school, she also became a coordinator of two extracurricular organizations and underwent training as head of the grade school department. In 1992, she transferred to MTS, where she taught all subjects in the nursery level. In the years that followed, she had become the guidance and preschool coordinator, and the vice principal of the school. At present, she is the principal of MTS.
Ms. Tirona has been an author and a coauthor of the Abiva Preschool Series. Since 1979, she has been conducting and attending in-service seminars in and out of MTS. She also participates actively in various religious and professional organizations. She was the public relations officer of the Las Piñas Private Schools Association (LAPPRISA) from 2008 to 2010 and is currently the secretary and one of its contributory writers.
Elsie T. Torreno graduated from PNU in 1975 with a degree of Bachelor of Science in Elementary Education. In 1980, she received her Master of Arts in Child Study degree from the same university.
Over the past sixteen years, Ms. Torreno has served as a principal in different educational institutions. In 1994, she was the principal of MTS in Las Piñas City. Her administrative experience was enhanced when she held a similar position at Riyadh International School in the Kingdom of Saudi Arabia seven years later. She returned to the Philippines in 2004 and became the principal of CSL. At present, she is the assistant to the directress and the preschool head of Bloomfield Academy.
A devoted educator, Ms. Torreno has conducted seminars for teachers from different educational levels all over the country. She has been an author, a coauthor, the coordinator, and the consultant of the early editions of Abiva Preschool Series. She was also the president of the Bacoor Sports Meet Council, a board member of the Cavite Alliance of Private Schools, a member of the Cavite Educational Media Association of Private School Administrators, and the auditor of LAPPRISA.


Filters and Transitions Master Sample
Use the controls on the left side of the window to apply and modify various filters and transitions. First, select the type of effect: filters or transitions. Next, select a filter or transition from the drop-down list. If the selected effect supports specific features, you can use the controls on the lower left side of the window to further modify the effect. The code used to generate the effect is shown below the controls section.
Copy Code to Clipboard
Select a filter:
Alpha
AlphaImageLoader
BasicImage
Blur
Chroma
DropShadow
Emboss
Engrave
Pixelate
Select a transition:
Barn
Blinds
Checkerboard
Fade
Inset
Iris
Pixelate
RadialWipe
Random Bars
Slide
Spiral
Stretch
Strips
Wheel
GradientWipe
Zigzag
Play
Style:
0 - Uniform Opacity
1 - Linear Opacity Change
2 - Radial Opacity Change
3 - Rectangular Opacity Change
Opacity:
0
25
50
75
100
FinishOpacity:
0
25
50
75
100
StartX:
0
25
50
75
100
FinishX:
0
25
50
75
100
StartY:
0
25
50
75
100
FinishY:
0
25
50
75
100
Src (source image):
Spiral Bound
Mouse
SizingMethod:
Crop - Crop at boundaries
Image - Normal image size
Scale - Scale to window
Bias:
-0.9
-0.5
-0.2
0.00
0.2
0.5
0.9
Bias:
-0.9
-0.5
-0.2
0.00
0.2
0.5
0.9
PixelRadius:
0
5
10
20
MakeShadow
ShadowOpacity:
0.00 0.25
0.50 0.75
1.00
MaxSquare:
2
5
10
25
50
Color:
Light Gray (DDDDDD)
Dark Gray (999999)
Black (000000)
Yellow (FFFF00)
Blue (0000FF)
OffX:
-5
-3
-1
1
3
5
OffY:
-5
-3
-1
1
3
5
Positive
Color:
White (FFFFFF)
Sky blue (87CEEB)
Dark red(8B0000)
Black (000000)
Rotation:
0 - 0 degrees
1 - 90 degrees
2 - 180 degrees
3 - 270 degrees
Opacity:
0.00
0.25
0.50
0.75
1.00
Mirror
Invert
X-Ray
Grayscale
Motion:
Out - From inside outward
In - From outside inward
Orientation:
Horizontal
Vertical
Bands:
2
4
6
8
10
Direction:
Up
Down
Left
Right
Direction:
Up
Down
Left
Right
SquaresX:
2
4
6
8
10
12
50
SquaresY:
2
4
6
8
10
12
50
Overlap:
0.00
0.25
0.50
0.75
1.00
IrisStyle:
Diamond
Circle
Cross
Plus
Square
Star
Motion:
Out - From inside outward
In - From outside inward
MaxSquare:
2
5
10
25
50
WipeStyle:
Clock
Wedge
Radial
Orientation:
Horizontal
Vertical
SlideStyle:
Hide
Push
Swap
Bands:
1
3
5
10
25
50
GridSizeX:
8
16
32
64
GridSizeY:
8
16
32
64
StretchStyle:
Hide
Push
Spin
Motion:
Left-up
Left-down
Right-up
Right-down
Spokes:
2
4
6
8
10
GradientSize:
0.00
0.25
0.50
0.75
1.00
WipeStyle:
0 - Left-to-right motion
1 - Top-to-bottom motion
Motion:
Forward
Reverse
GridSizeX:
8
16
32
64
100
GridSizeY:
8
16
32
64
100
For new and updated Web samples, visit the MSDN Online Web & Internet Samples site.


’$c_malloc’/2
’$c_malloc’/2 – Allocates a C data area using the system malloc call
’$c_free’/1 – Frees a C data area
’$c_set’/2 – Modifies the contents of a C data area or a UIA
’$c_examine’/2 – Examines the contents of a C data area or a UIA
FORMS
’$c_malloc’(+Size,-Ptr)
’$c_free’(+Ptr)
’$c_set’(+Ptr_or_UIA,+FormatList)
’$c_examine’(+Ptr_or_UIA,+FormatList)
DESCRIPTION
The following predicates are C defined builtins in ALS Prolog. ’$c_malloc’/2 allocates a C data area, and ’$c_free’/1 frees it, ’$c_set’/2 is used to modify the contents of a C data area or a UIA, and ’$c_examine’/2 is used to examine the contents of a C data area or a UIA. ’$c_malloc’(Size,Ptr) is true if Size is a positive integer and Ptr (an integer) unifies with the address of the first byte of a data area
allocated by the system call "malloc". The call fails if malloc returns a null
pointer. ’$c_free’(Ptr) is true if Ptr is a number, and it invokes the system call "free" to free the data
area pointed by Ptr. ’$c_set’ (Ptr_or_UIA,FormatList) is true if Ptr_or_UIA is bound to the address of a C data area or a UIA, and FormatList is a non-empty list of 3-ary or 4-ary terms of the form
f(+Offset,+Type,+Value{,+Length})
and the call modifies the contents of the data area as explained below. In
each term, Offset is the offset of the field from the start address of the data area. Type is the "C type" of the field, which is one of the following symbols : int,long,short,ptr,char,str,float,double. They have the obvious correspondence with C data types. Value is the data that the field should be set to. If Type is one of int, long, short, ptr, char, float or double, then Value must be a number, otherwise Type is str and Value must be an atom and a null terminated string name of the atom is copied into
the receiving data are without overflow checks. Length (optional) must be a number and has meaning only when Type is str, and at most Length bytes are copied into the receiving data area. ’$c_examine’(Ptr_or_UIA,FormatList) is true if Ptr_or_UIA is bound to the address of a C data area or a UIA, and FormatList is a nonempty list of 3-ary or 4-ary terms of the form
f(+Offset,+Type,-Value{,+Length})
whose arguments are interpreted as in ’$c_set’/2 (above) except that now a data item of the specified type is extracted from
the data area and unified with Value.


Dedicated sales and marketing professional who exceeds goals, retains customers and maintains a strong reputation for customer satisfaction.


--- layout: page slide_id: 2 ---
Call For Papers
Recommender systems in software engineering (RSSE) have gained momentum in recent years. Such systems assist developers in navigating large information spaces and getting instant recom mendations that might be helpful to solve their development tasks. Though a lot of improvements have been obtained so far, there is still the need to make RSSE more effective and efficient.
Fur- thermore, while several attempts have been made to improve the recommendation accuracy, little attention has been paid to make such systems robust and resilient to malicious data. In fact, by manipulating training data available in open source software platforms, a hostile user may render recommender systems vulnerable to adversarial attacks, putting software clients at risk. The First International Workshop on Evaluation and Analysis of Recommender Systems in Software Engineering (WEARS 2021) brings in a forum for researchers and practitioners to share, discuss and explore the opportunities and challenges raised by the evaluation and in-depth investigation of RSSE. We solicit research work from the Software Engineering community as well as related fields such as Machine Learning and Recommender Systems.
Particular areas of interest include, but are not limited to:
Papers describing timely research contributions in WEARS's areas of interest are solicited. Papers reporting on initial results as well as papers discussing mature research projects or case studies of deployed systems are sought out. Submissions describing big ideas that may have significant impact and could lead to interesting discussions at the workshop are encouraged.
Submitted papers must be neither previously published nor under review by another workshop, conference or journal. Only electronic submissions in PDF will be accepted. Submitted papers must be written in English, must be no longer than 6 pages (minimum 10pt font), must render without error using standard PDF viewing tools, must print on US-Letter-sized paper, and must conform to the standard ACM Template.
At least one of the authors of each paper accepted for presentation in WEARS 2021 must register for the workshop.
{% include card-important-dates.html %}
We plan to organize a special issue on an international journal featuring extended, revised paper from the workshop


These components provide powerful ways to declaratively present a series of pages as a workflow, wizard, interactive video or tutorial.


Graphic design for print and screen.
Based in New Zealand
Email
Alt Group
Designer
Nov 2019—Present
General Studios
Graphic & Web Designer
Mar 2019—Nov 2019
Bachelor of Design with Honours
Visual Communication Design
Massey University
Toi Rauwhārangi
2015—2018
Top 30 Graduates of ‘18
TDK (The Design Kids) Awards
DINZ Best Awards
Silver
DINZ Best Awards Bronze
DINZ Best Awards Finalist
Instagram
Are.na
Thanks to Amy Yalland & Jeremy Hooper
All Rights Reserved


Slack assistant Yui
Slack assistant Yui makes your Slack little more happy :)
Licence
Source code are distributed under the MIT licence, and all image files in source code are reserved all rights by niseojio.
Slack assistant Yui makes your Slack little more happy :)
Source code are distributed under the MIT licence, and all image files in source code are reserved all rights by niseojio.


Welcome to the UM Libraries' Web Redesign Blog, a production of WebRAT (the Web Redesign Advisory Team). We are using this space to post updates about the status and process of the 2017-2018 library website redesign.
We hope you will find some of these postings interesting and/or useful.


Make no compromises.
Velocity is a Minecraft server proxy with unparalleled server support, scalability, and flexibility.
Why will you benefit from Velocity?
Whether you plan to be a friends-only collection of servers or aiming to become the next big Minecraft server, you can trust Velocity to power your network.
It's so simple to set up.
You can set up Velocity in just a few minutes. No need to muck around with a bunch of different configuration files or install dubious plugins on your server.
archiveLearn how to install VelocityMake your server how you want it.
Velocity comes with excellent support for Paper, Sponge, and Minecraft Forge. We're also able to respond to Minecraft updates quickly, so you'll never have an excuse to not update.
archiveLearn more about Velocity's server supportSecure by default.
Velocity comes with IP forwarding that is secure by default. Simply choose a secure password and put it in your server configurations. Help ward off griefers with hacked clients!
archiveConfigure your servers for more security


Functions to install SciViews additions to R, and more tools.
| Version: | 0.9-13 |
| Depends: | R (≥ 2.6.0) |
| Imports: | ellipse, grDevices, graphics, stats |
| Suggests: | MASS, covr, knitr, testthat |
| Enhances: | base |
| Published: | 2018-01-06 |
| Author: | Philippe Grosjean [aut, cre] |
| Maintainer: | Philippe Grosjean <phgrosjean at sciviews.org> |
| BugReports: | https://github.com/SciViews/SciViews/issues |
| License: | GPL-2 |
| URL: | https://github.com/SciViews/SciViews, http://www.sciviews.org/SciViews-R |
| NeedsCompilation: | no |
| Citation: | SciViews citation info |
| CRAN checks: | SciViews results |
| Reference manual: | SciViews.pdf |
| Package source: | SciViews_0.9-13.tar.gz |
| Windows binaries: | r-devel: SciViews_0.9-13.zip, r-release: SciViews_0.9-13.zip, r-oldrel: SciViews_0.9-13.zip |
| OS X binaries: | r-release: SciViews_0.9-13.tgz, r-oldrel: SciViews_0.9-13.tgz |
| Old sources: | SciViews archive |
Please use the canonical form https://CRAN.R-project.org/package=SciViews to link to this page.


let div = document.getElementById("div1"); let iterator = document.createNodeIterator(div, NodeFilter.SHOW_ELEMENT, null, false); let node = iterator.nextNode(); while (node !== null) { console.log(node.tagName); // output the tag name node = iterator.nextNode(); } The first call to nextNode() in this example returns the
element. Because nextNode() returns null when it has reached the end of the DOM subtree, a while loop checks to see when null has been returned as it calls nextNode() each time through. When this code is executed, alerts are displayed with the following tag names: DIV P B UL LI LI LI


Orchestrator.js is an open source platform for creating applications to heterogeneous and multi-device environment based on Node.js.
Make your devices to communicate seamlessly, and put their resouces into use!
Orchestrator.js is a platform for new type of applications. The apps run constantly run in cloud and monitor device context as well as third party services, like Facebook, Flickr, etc. When rules set by the app developer are met, the app triggers interactions between devices and users. Users can start and stop different applications as they wish.
To have better controlr over their device, users can describe what their devices can or are allowed to do by enabling / disabling different device capabilities. For example, if device has a talking-capability enabled, it is then allowed to translate text to speech.
Read more about Social Devices and Orchestrator.js from our publications.
Designed for ubiquitous mobile cloud environment
OJS now supports .NET Gadgeteer embedded platform. This video also introduces OJS tools for the implementation process.
You can start by watching this tutorial video, and then read more from the Docs tab! All the source code is freely available under MIT license from GitHub, and you can find links from the bottom of this page.
Orchestrator.js applications are based on three main concepts.
The app itself monitors device context, and out-side sources like social media services, such as Facebook, Twitter and so on. Based on the observations and logic defined by the developer, the app triggers actions. So app defines when and what interactions (actions) should take place.
An action defines how the devices interact with each other and as well as with users. An action can be very short, like just greeting each other, or long-lasting, like a game.
Capabilities are used to describe how a device appears to the system, what a device can or is allowed to do, and in what interactions the device can take part to. The capability can either be very generic, like Talking-capability that allows device to speak, or more specific, like Apocalympics-capability which allows the device to participate to Apocalympics game.
Orchestrator.js offers web-based IDE for implementing apps in heterogenous, multi-user and multi-device environment. The Web Console can be used for implementing the apps, interactions, and device capabilities. Moreover, the IDE offers tools for testing purposes: it is easy to monitor device states, read debug messages directly from web browser's console, and also developers can manually trigger interactions for a predefined set of devices.
With Orchestrator.js it is easy to implement context-aware applications for a heterogenous set of device. The platform offers publish-subsrcibe API that easy to utilize in app implementations. The devices constantly report their state to the Orchestrator.js Device Registry, which publishes updates whenever devices' contextual data changes.
The platform currently runs in Amazon cloud, but is freely available from GitHub, and can be easily hosted on any computer with a network connectivity. The devices are constantly connected to the Orchestrator.js. On client-side we currently support Android, and devices capable of running Python. We are also working on with iOS implementation currently.


Chatbots are injected with artificial intelligence to simulate real-life conversations. KeduBot stands out because it responds naturally with that level of friendliness and empathy that a human rep possesses.
Loading...


fullstack dev
I am proficient in a range of modern technologies including Python,Django, PostgreSQL, HTML, CSS and JavaScript. I can build a fully functional and responsive website from the front-end to the back-end using the various technologies mentioned above.
Fullstack developer from Nairobi
I'm a junior fullstack developer with a passion for backend. I enjoy programming and learning new things each and everyday as I grow not only in my skills, but also as a person. Fueled by a clear vision of the work i want to do and the things I want to achieve, I put in the work required and I'm also not afraid to pursue new ideas in search of more knowledge. I am a fast learner, able to pick up new skills when required.
Well, when not programming i enjoy gaming, travel and hanging out with my friends among other activities.
A range of my work


public interface Connection extends Closeable
| Modifier and Type | Interface and Description |
|---|---|
static class |
Connection.HecEndpoint
Used to select either structured HEC /event endpoint or /raw HEC endpoint
|
| Modifier and Type | Method and Description |
|---|---|
void |
close() |
void |
closeNow() |
void |
flush() |
ConnectionCallbacks |
getCallbacks() |
List<HecHealth> |
getHealth()
healthCheck will retrieve health for each channel or trigger a health check if no channels have been made
|
ConnectionSettings |
getSettings()
Returns a live ConnectionsSettings instance that can be used to change
the behavior of the connection while it is in use.
|
boolean |
isClosed() |
void |
release(Comparable id) |
int |
send(Event event)
The send method will send the Event immediately unless buffering is enabled.
|
int |
sendBatch(EventBatch events)
sendBatch will immediately send the EventBatchImpl, returning the number
of bytes sent, or throws an HecConnectionTimeoutException if
BLOCKING_TIMEOUT_MS have expired before the batch could be sent.
|
void |
setLoggerFactory(HecLoggerFactory f)
Allows for replacement of the internal LoggerFactory with a customize one.
|
int send(Event event) throws HecConnectionTimeoutException
event
- HecConnectionTimeoutException
PropertyKeys
void close()
close
in interface AutoCloseable
close
in interface Closeable
void closeNow()
void flush() throws HecConnectionTimeoutException
HecConnectionTimeoutException
ConnectionCallbacks getCallbacks()
boolean isClosed()
void release(Comparable id)
int sendBatch(EventBatch events) throws HecConnectionTimeoutException
events
- HecConnectionTimeoutException
ConnectionSettings getSettings()
void setLoggerFactory(HecLoggerFactory f)
f
- Copyright © 2017 Splunk, Inc.. All rights reserved.


--- # jekyll prompt ---
The nation is reaching a critical mass of HealthIT systems (EHRs, Data Warehouses etc) that comply with data and vocabulary standards including Meaningful Use Stage 2 (MU2) standards. The wide deployment of Meaningful Use Stage 2 compliant HealthIT systems has created unique opportunities for providers, provider support teams, healthcare professionals and organizations, etc. to access and use the patient data that is already collected during clinical workflows.
The deployed Health IT systems provide many access paths through their pre-defined interactions between a user and the system. However, they are limited in their support for data queries, APIs, or services to access data sets as needed. Where Health IT systems provide data access, they likely do not use industry standard access methods. Increasing support for this class of data access, using industry standards, would enable other applications to expand the ability of users to create value out of their data without having to rely on the predefined access paths.
Allowing access to this data can enable a provider to further analyze the collected data to understand a patient's overall health, the health of a provider's collective patient population, and use the data to power innovative new applications and tools to take better care of patients and populations.
The DAF Initiative tries to leverage the HL7 FHIR, C-CDA and existing IHE standards to standardize access to Meaningful Use Stage 2 structured information both within the organization and from external organizations. DAF uses existing IHE profiles XCA, XCPD and MHD v2 to standardize access to documents such as a CCD, History and Physical Note etc. while HL7 FHIR is leveraged to access granular information such as problem lists, medications and patient demographics.
The standards and guidance incorporated into this implementation guide were based on the requirements defined in the DAF Use Cases document. Users of this implementation guide will benefit greatly from review of both the Local and Targeted DAF Use Cases. A very high level overview of the DAF actors and their interactions are as shown below:
The actors and their definitions are as follows:
The DAF Initiative identifies and recommends standards for the interoperable representation and transmission of the following using the notion of a Query Stack which modularizes the various layers of the Data Access Framework. The DAF Query Stack using FHIR is as shown in the diagram below.
The DAF FHIR IG will provide requirements and implementation guidance for the various layers of the DAF Query Stack which includes:
The DAF FHIR IG provides guidance on the use of FHIR profile(s) for DAF. The profiles in this implementation guide will be used to meet the following DAF project objectives:
This implementation guide provides implementers with guidance on how to achieve conformance with the standards recommended by the Office of the National Coordinator for Health Information Technology (ONC) Standards & Interoperability (S&I) Framework, DAF Initiative.
The implementation guide will explain how the FHIR resources (using the defined profiles) can be used to enable data access within and across enterprises. Additionally, it describes secure interactions using Representational State Transfer (REST) services to access the desired resources.
The DAF profiles identify the resource elements and extensions that must be supported to meet DAF use cases and constraints on their use.
Out of scope for this implementation guide:


Sylvia Kunaai
Home
Project
Services
About
Blog
Contact
Resume
About Kunaai
Hi am Sylvia Kunaai
Am a web developer by profession
Languages
Frameworks
Database
HTML
Javascript
MySQL
CSS
ExpressJS
SQL
C++
CALCULATOR App
NoSQL
C
Code Block
Featured Projects
Testimonials
Frequantily asked quations


Length: {{data.length}}
Representation of the {{data?.alignments?.length}} seed alignment sequences, indicating the number of aligned bases per model position and the length distribution across the model.| Average Kimura Divergence: | {{data.averageKimuraDivergence ? (data.averageKimuraDivergence|percent:'1.1-1') : "N/A"}} |


Toggle navigation
+1<PHONE_NUMBER>
Home
About us
Services
Service #1
Service #2
Service #3
Service #4
Contact us
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Etiam ac sodales orci, id tristique magna.
Etiam sit amet lacus eu diam faucibus mattis. Nam convallis lobortis sapien, et consequat volutpat in.
Submit
Please select what kind of products do you prefer
Popular Industries
Olympic Games
Live Healthy
Vegetables
Top Books 2015
Top Music 2015
Exotic Vacations
Sport Equipment
Technology
Best Jobs
Recepies
Nature
Phone
MP3
Tablet
GPs
Laptop
Camera
Option7
Option8
Option9
Option10
ption11
Option12
No option
No option
No option
No option
No option
No option
No option
No option
No option
Please select an option
more
more
more
×
Products you selected:


Every element on a page is a rectangular box. Behavior of this box is determined by the CSS display property. Elements are classified as Block or Inline elements based on display property. Block elements are considered as container, they can hold block and inline elements.Inline elements can hold another inlin element but cannot hold a block level Element.


--- layout: post title: "iPhone 4.0 Announcement" comments: true ---
ok so what does this mean? c# developers can currently use an application called Monotouch and this was fantastic. it was pricey but it was very worth it. i had even started using this myself for apps at work
then this bombshell!! There was no news of this announcement today - this was discovered by people downloading the SDK
ok so i think that what has happened is as follows:
in my eyes this seems like Apple have laid down the law that their apps will only really be developed using their products and their specs. No more MonoTouch? What about those who were creating apps using c#?
I think more will come of this and i really do think that this will loose Apple a huge amount of followers in the development community.
Let the war begin!
More details and posts will be added as soon as i see them!


Stressed: Distressed, Puzzled,
Annoyed, Amazed, Troubled, Miffed, In a Fog, Distant, Offended, Hurt,
Inner Turmoil, Distracted, Stifled, Cornered, Unheard, Overwhelmed,
Questioned, Agitated, Apprehensive, Desperate.

Undervalued: Unwanted,
Undesirable, Unlovable, Unworthy of Pursuit, Unpleasing, Unlovely,
Unattractive, Inadequate, Untapped, Unneeded, Not Cherished,
Criticized, Trivial, Loathsome, Inferior.
Mistreated: Berated, Walked On,
Run Over , Trampled, Pressured, Violated, Defiled, Spoiled, Desecrated,
Degraded, Scattered, Shattered, Battered, Curtailed, Limited, Used,
Oppressed, Ridiculed, Persecuted, Taunted, Hated, Despised, Crushed.
Needy: Waiting, Longing,
Wilting, Vulnerable, Timid, Fearful, Impatient, Emotional, Possessive,
Jealous.
Distrust: Leary, Wary, Distant,
Resistant, Put Out, Angry, Perturbed, Aggravated, Turned Off, Unloving,
Cold, Hopeless, Hardened, Bitter.
Heavy: Saddled, Burdened,
Weary, Frail.


The main focus of server applications lays on scalability. This article tries to investigate, what implications scalability requirements have on full-text search engines and how the project Strus meets these requirements.
Strus is a set of libraries and tools to build search engines. For tutorials that show how to build a search engine with Strus have a look at my previous code project articles:
{(d,p) :d->document number, p->position}
For running the examples in a docker image, you need docker installed.
To cap administration costs the industry favors configurationless systems, meaning that
the software is completely instrumented with an API.
To scale in performance you need to be
able to distribute your application on an arbitrary number of machines without system costs
eating up the performance gain of using more machines. Furthermore it should be possible
to add new machines anytime without a complete reconfiguration of the system.
The first requirement, a configurationless system, completely instrumentable with an API
is fulfilled by Strus. You can check that on your own. Have a look at the previous articles
and the API and the documentation with examples.
The second issue, distribution of a Strus search application on an arbitrary number
of machines and the capability to increase the number of processing nodes anytime
is the main subject of this article.
The article will not discuss fundamentals of distributed processing. Costs of sharing data,
messaging and process synchronization should be familiar to the reader.
The article will only explain what data is usually shared in a full-text search engine
and discuss alternatives, how to handle the consequential problems (as sharing data is always
a problem in distributed systems).
Then it will show the position of Strus in this landscape and help you to compare Strus
with other search engines.
We are talking about a distributed search index here and not about a distributed search. A distributed search would mean a distributed system, where queries are routed to the nodes that possibly can answer the query. For a search engine this means that nodes propagate their competence to answer a certain question in a P2P network to other nodes and the nodes use this information to route a query. Here we talk about distributing the search index of a full-text search engine that sends the query to all nodes (transitively in the case of a tree of nodes) and that merges their results to one result list.
A search engine evaluates the best matching answers to a query with help of a database storing relations that bring a query into conjunction with a result. For a scalable search engine with a distributed index it is important that each node with a part of this split can calculate a part of the result completely so that it can make a decision about what is a candidate for the final result and what not. Preferably the number of items selected with this cut as candidates for the final result is small. This implies that the database has to be organized in a way that all information for weighting one document, the most common result of an information retrieval query, belongs to one node. The informations used for retrieval are therefore assumed to be grouped by document. There are cases of retrieval problems, where the items retrieved are not documents and where the cut cannot be made so easily.
A search engine usually assigns a weight to each result row. This weight is used to order
the results. A search engine calculates the N results with the highest weight.
For merging results from different nodes to one result, these weights must be comparable.
If this is the case, then a search engine can delegate the calculation of the best results
to each node and then take the best N results from the merge of the node results.
Many established weighting methods use statistical information that is accumulated from
values distributed over the complete collection. One example is the df, the document
frequency of a term. It gives a measure of how rare or how often a term appears in the collection.
Many statistical methods use this number in the weighting formula used to calculate the
weight of a document.
We can now make a distinction of cases of the relation of a node to such a piece of information
from the point of view of a node and the consequences for our retrieval function and the system:
| 1 | The information is not provided: We implement only query evaluation schemes without statistical data that is distributed beyond one document. | This is a solution that scales. Unfortunately it is problematic from the information retrieval point of view. It looks like neither query expansion nor document assessments are able to outweigh missing statistical information. Well, at least according to my information. If you can solve your problem with this approach, then it is perfect. |
| 2 | The information is implicitly provided: We use the probability of a random distribution of independent items to tend towards an equal distribution of the statistics. A mechanism is used that assigns the documents to nodes in a way that we can use the local values of statistics for calculation because they are with a high probability globally the same within an error range. | This is a very pragmatic approach. We can scale in the number of documents we can insert, though we cannot simply add and remove nodes and organize them as we want. We can organize things only one way and this assignment is left to Mr. Random. The missing possibility to organize collections as we want is the strongest argument against this approach. Another problem is the high costs of reorganisation if the maximum number of documents (the decline of the search performance with the growth of the inserts sets the limit) is reached. |
| 3 | Every node has access to the information: We use a mechanism to distribute the information to every node or we provide an interface for every node to access the information. | Both approaches are a no go from system point of view. In the first case we get storms of messages when updating a node. So you can forget scalability. In the second case we get a system nightmare with the interface to access the information provided to the nodes. The self containment of nodes is lost and thus the scalability. |
| 4 | The information is passed down with the query: We separate the information from the single node. | This is a good approach, if the amount of information we have to pass down is not significantly complicating
the query. We can handle the problem of accessing this information on its own.
For example by using a key/value map as storage for this information. We know that we can scale in this aspect,
because for key/value storages there exist solutions that scale. We have to find ways to fetch the
information from the nodes accumulating it and put it into the global storage for the statistical
information for the query.
But we also give something away with this solution. Queries that require statistical information of items that are taken from the documents retrieved, like for example fetching the statistically best terms from documents matching a query for relevance feedback, are only possible using local statistics. This is because the information we have to pass down with the query is potentially containing all statistical information in the whole collection or all information that possibly could be found in a document. |
You can find out on your own, how your favorite search engine solves the problem of providing the
information needed by the query for weighting the items retrieved in a way that search results
are comparable.
For Strus we will not discuss the variant 1(the information is not provided) because it is trivial.
The variant 2(the information is implicitly provided) is left to the reader to think about.
The variant 3(every node has access to the information) is refused because it does not scale.
We will only have a look at variant 4 (the information is passed down with the query) and
provide some example source code for it.
 But there is still reason for optimism to find solutions for this kind of problems (For example to use the local statistics information to make a preselection and a cut and to calculate the real weights based on the global statistics outside). If you have a solution that scales but is missing the functionality, you'll have a big chance to solve your problem with some fantasy and some nice ideas. It is on the other hand nearly impossible to take a program, that solves a problem, and to make it scale with some fantasy and some nice ideas.
This demo provides a docker image to run the examples. But you can also download them and run on your own. The Python sources of this article are examples not intended for production or even a test application. There are no real transactions implemented and error handling is not involving recovery from all kind of errors. There are 3 servers implemented, each of them representing one component of the distributed search engine. You might think about the architecture on your own and find a suitable solution for your needs. Then you will implement the components needed from scratch. The examples here are just a scetch and a proof of concept. In this section we will introduce the components needed with an example. The following table gives you an overview.
| strusStatisticsServer.py | A statistics server holds the global statistics. It might be implemented as interface to a
distributed hashtable fed over persistent queues. But our example server holds the global statistics
for each term in a dicitionary and the total collection size in variables in memory. Nothing is persistent
and thus lost on shutdown.
The server is implemented as TCP/IP server with a proprietary protocol with two commands (put/get). |
| strusStorageServer.py | A storage server provides access to one storage. It is a node in our distributed search.
It provides two commands, one for insert and one for query.
The insert will analyze and store the multipart document passed within one transaction. After the transaction commit it iterates on its global statistics updates, a set of increment/decrement operations. These operations (packed in chunks of data blobs) are sent to the statistics server. Currently there is no possibility to get the statistic updates before the commit. There is room for improvement here in Strus. The query is passed as list of terms with their global statistics to a storage server. The storage server builds the query with the global term statistics and the global collection size, excutes it and sends the result back. |
| strusHttpServer.py | A strus HTTP server provides two operations insert and query.
The insert is a POST request with the port of the local storage server, where to insert the document in the URL and the multipart document in the request body. The insert command takes the document and forwards it to the storage server addressed with the command URL. The query is a GET request with the querystring and the result range as parameter. The query command analyzes the query to get the normalized query terms and sends them first to the statistics server to get the statistics for each term and the global collection size. The query with the statistics is packed and sent to each storage node. The results received from each server are merged into one result and rendered with tornado templates to an HTML page. For siplicity the example implements only a list of terms as query. It's left to the reader to think about how to analyze a query language, getting the statistics of the terms and serialize the query tree. |
| strusIR.py | The strus information retrieval module provides an abstraction for accessing the storage and analyze the document. As we already noticed is the query analysis done outside in the HttpServer. In a decent design, the sources for query and document analysis would be at the same place because they belong together. But we don't look at a proper way of organizing the code in details we do not intend to bring into focus in this article. |
| strusMessage.py | The messaging module provides an abstraction for server and client used for
the TCP/IP messaging of the storage and the statistics servers. It allows the client to
open a session and send one or several requests to the server.
For simplicity we do not hold connections open. |
#!/usr/bin/python import tornado.ioloop import tornado.web import tornado.gen import tornado.iostream import os import sys import struct import collections import optparse import strusMessage import binascii import strusIR # Information retrieval engine: backend = None # Port of the global statistics server: statserver = "localhost:7183" # IO loop: pubstats = False # Strus client connection factory: msgclient = strusMessage.RequestClient() # Call of the statustics server to publish the statistics of this storage on insert: @tornado.gen.coroutine def publishStatistics( itr): # Open connection to statistics server: try: ri = statserver.rindex(':') host,port = statserver[:ri],int( statserver[ri+1:]) conn = yield msgclient.connect( host, port) except IOError as e: raise Exception( "connection to statistics server %s failed (%s)" % (statserver, e)) msg = itr.getNext() while (len(msg) > 0): try: reply = yield msgclient.issueRequest( conn, b"P" + bytearray(msg) ) if (reply[0] == 'E'): raise Exception( "error in statistics server: %s" % reply[ 1:]) elif (reply[0] != 'Y'): raise Exception( "protocol error publishing statistics") except tornado.iostream.StreamClosedError: raise Exception( "unexpected close of statistics server") msg = itr.getNext() # Pack a message with its length (processCommand protocol) def packedMessage( msg): return struct.pack( ">H%ds" % len(msg), len(msg), msg) # Server callback function that intepretes the client message sent, executes the command and packs the result for the client @tornado.gen.coroutine def processCommand( message): rt = bytearray(b"Y") try: messagesize = len(message) messageofs = 1 if (message[0] == 'I'): # INSERT: # Insert documents: docblob = str( message[ 1:]) nofDocuments = backend.insertDocuments( docblob.decode("utf-8")) # Publish statistic updates: itr = backend.getUpdateStatisticsIterator() yield publishStatistics( itr) rt += struct.pack( ">I", nofDocuments) elif (message[0] == 'Q'): # QUERY: Term = collections.namedtuple('Term', ['type', 'value', 'df']) nofranks = 20 collectionsize = 0 firstrank = 0 terms = [] # Build query to evaluate from the request: messagesize = len(message) messageofs = 1 while (messageofs < messagesize): if (message[ messageofs] == 'I'): (firstrank,) = struct.unpack_from( ">H", message, messageofs+1) messageofs += struct.calcsize( ">H") + 1 elif (message[ messageofs] == 'N'): (nofranks,) = struct.unpack_from( ">H", message, messageofs+1) messageofs += struct.calcsize( ">H") + 1 elif (message[ messageofs] == 'S'): (collectionsize,) = struct.unpack_from( ">q", message, messageofs+1) messageofs += struct.calcsize( ">q") + 1 elif (message[ messageofs] == 'T'): (df,typesize,valuesize) = struct.unpack_from( ">qHH", message, messageofs+1) messageofs += struct.calcsize( ">qHH") + 1 (type,value) = struct.unpack_from( "%ds%ds" % (typesize,valuesize), message, messageofs) messageofs += typesize + valuesize terms.append( Term( type, value, df)) else: raise tornado.gen.Return( b"Eunknown parameter") # Evaluate query with BM25 (Okapi): results = backend.evaluateQuery( terms, collectionsize, firstrank, nofranks) # Build the result and pack it into the reply message for the client: for result in results: rt.append( '_') rt.append( 'D') rt += struct.pack( ">I", result['docno']) rt.append( 'W') rt += struct.pack( ">f", result['weight']) rt.append( 'I') rt += packedMessage( result['docid']) rt.append( 'T') rt += packedMessage( result['title']) rt.append( 'A') rt += packedMessage( result['abstract']) else: raise Exception( "unknown command") except Exception as e: raise tornado.gen.Return( bytearray( b"E" + str(e)) ) raise tornado.gen.Return( rt) # Shutdown function that sends the negative statistics to the statistics server (unsubscribe): def processShutdown(): if (pubstats): publishStatistics( backend.getDoneStatisticsIterator()) # Server main: if __name__ == "__main__": try: # Parse arguments: defaultconfig = "path=storage; cache=512M" parser = optparse.OptionParser() parser.add_option("-p", "--port", dest="port", default=7184, help="Specify the port of this server as PORT (default %u)" % 7184, metavar="PORT") parser.add_option("-c", "--config", dest="config", default=defaultconfig, help="Specify the storage path as FILE (default '%s')" % defaultconfig, metavar="FILE") parser.add_option("-s", "--statserver", dest="statserver", default=statserver, help="Specify the address of the statistics server as ADDR (default %s" % statserver, metavar="ADDR") parser.add_option("-P", "--publish-stats", action="store_true", dest="do_publish_stats", default=False, help="Tell the node to publish the own storage statistics to the statistics server at startup") (options, args) = parser.parse_args() if len(args) > 0: parser.error("no arguments expected") parser.print_help() myport = int(options.port) pubstats = options.do_publish_stats statserver = options.statserver backend = strusIR.Backend( options.config) if (statserver[0:].isdigit()): statserver = '{}:{}'.format( 'localhost', statserver) if (pubstats): # Start publish local statistics: print( "Load local statistics to publish ...\n") publishStatistics( backend.getInitStatisticsIterator()) # Start server: print( "Starting server ...") server = strusMessage.RequestServer( processCommand, processShutdown) server.start( myport) print( "Terminated\n") except Exception as e: print( e)
#!/usr/bin/python import tornado.ioloop import tornado.web import optparse import os import sys import binascii import struct import strus import collections import strusMessage # [1] Globals: # Term df map: termDfMap = {} # List of ports of associated peer storage nodes: collectionSize = 0 # Strus statistics message processor: strusctx = strus.Context() strusctx.defineStatisticsProcessor( "standard"); strustat = strusctx.createStatisticsProcessor() # [2] Request handlers def packedMessage( msg): return struct.pack( ">H%ds" % len(msg), len(msg), msg) def termDfMapKey( type, value): return "%s~%s" % (type,value) @tornado.gen.coroutine def processCommand( message): rt = bytearray("Y") try: global collectionSize global termDfMap if (message[0] == 'P'): # PUBLISH: msg = strustat.decode( bytearray(message[1:])) collectionSize += msg.nofDocumentsInsertedChange() dfchglist = msg.documentFrequencyChangeList() for dfchg in dfchglist: key = termDfMapKey( dfchg.type(), dfchg.value()) if key in termDfMap: termDfMap[ key ] += dfchg.increment() else: termDfMap[ key ] = long( dfchg.increment()) elif (message[0] == 'Q'): # QUERY: messagesize = len(message) messageofs = 1 while (messageofs < messagesize): if (message[ messageofs] == 'T'): # Fetch df of term, message format [T][typesize:16][valuesize:16][type string][value string]: (typesize,valuesize) = struct.unpack_from( ">HH", message, messageofs+1) messageofs += struct.calcsize( ">HH") + 1 (type,value) = struct.unpack_from( "%ds%ds" % (typesize,valuesize), message, messageofs) messageofs += typesize + valuesize df = 0 key = termDfMapKey( type, value) if key in termDfMap: df = termDfMap[ key] rt += struct.pack( ">q", df) elif (message[ messageofs] == 'N'): # Fetch N (nof documents), message format [N]: messageofs += 1 rt += struct.pack( ">q", collectionSize) else: raise Exception( "unknown statistics server sub command") else: raise Exception( "unknown statistics server command") except Exception as e: raise tornado.gen.Return( bytearray( b"E" + str(e))) raise tornado.gen.Return( rt) def processShutdown(): pass # [5] Server main: if __name__ == "__main__": try: # Parse arguments: parser = optparse.OptionParser() parser.add_option("-p", "--port", dest="port", default=7183, help="Specify the port of this server as PORT (default %u)" % 7183, metavar="PORT") (options, args) = parser.parse_args() if len(args) > 0: parser.error("no arguments expected") parser.print_help() myport = int(options.port) # Start server: print( "Starting server ...") server = strusMessage.RequestServer( processCommand, processShutdown) server.start( myport) print( "Terminated\n") except Exception as e: print( e)
#!/usr/bin/python import tornado.ioloop import tornado.web import tornado.websocket import tornado.gen import os import sys import struct import binascii import collections import heapq import optparse import signal import strus import strusMessage # [0] Globals and helper classes: # The address of the global statistics server: statserver = "localhost:7183" # Strus storage server addresses: storageservers = [] # Strus client connection factory: msgclient = strusMessage.RequestClient() # Query analyzer structures: strusctx = strus.Context() analyzer = strusctx.createQueryAnalyzer() analyzer.definePhraseType( "text", "word", "word", ["lc", ["stem", "en"], ["convdia", "en"]] ) # Query evaluation structures: ResultRow = collections.namedtuple('ResultRow', ['docno', 'docid', 'weight', 'title', 'abstract']) # [1] HTTP handlers: # Answer a query (issue a query to all storage servers and merge it to one result): class QueryHandler( tornado.web.RequestHandler ): @tornado.gen.coroutine def queryStats( self, terms): rt = ([],0,None) try: statquery = bytearray("Q") for term in terms: statquery.append('T') typesize = len( term.type()) valuesize = len( term.value()) statquery += struct.pack( ">HH", typesize, valuesize) statquery += struct.pack( "%ds%ds" % (typesize,valuesize), term.type(), term.value()) statquery.append('N') ri = statserver.rindex(':') host,port = statserver[:ri],int( statserver[ri+1:]) conn = yield msgclient.connect( host, port) statreply = yield msgclient.issueRequest( conn, statquery) if (statreply[0] == 'E'): raise Exception( "failed to query global statistics: %s" % statreply[1:]) elif (statreply[0] != 'Y'): raise Exception( "protocol error loading global statistics") dflist = [] collsize = 0 statsofs = 1 statslen = len(statreply) while (statsofs < statslen): (statsval,) = struct.unpack_from( ">q", statreply, statsofs) statsofs += struct.calcsize( ">q") if (len(dflist) < len(terms)): dflist.append( statsval) elif (len(dflist) == len(terms)): collsize = statsval else: break if (statsofs != statslen): raise Exception("result does not match query") rt = (dflist, collsize, None) except Exception as e: rt = ([],0,"query statistic server failed: %s" % e) raise tornado.gen.Return( rt) @tornado.gen.coroutine def issueQuery( self, serveraddr, qryblob): rt = (None,None) ri = serveraddr.rindex(':') host,port = serveraddr[:ri],int( serveraddr[ri+1:]) result = None conn = None try: conn = yield msgclient.connect( host, port) reply = yield msgclient.issueRequest( conn, qryblob) if (reply[0] == 'E'): rt = (None, "storage server %s:%d returned error: %s" % (host, port, reply[1:])) elif (reply[0] == 'Y'): result = [] row_docno = 0 row_docid = None row_weight = 0.0 row_title = "" row_abstract = "" replyofs = 1 replysize = len(reply)-1 while (replyofs < replysize): if (reply[ replyofs] == '_'): if (row_docid != None): result.append( ResultRow( row_docno, row_docid, row_weight, row_title, row_abstract)) row_docno = 0 row_docid = None row_weight = 0.0 row_title = "" row_abstract = "" replyofs += 1 elif (reply[ replyofs] == 'D'): (row_docno,) = struct.unpack_from( ">I", reply, replyofs+1) replyofs += struct.calcsize( ">I") + 1 elif (reply[ replyofs] == 'W'): (row_weight,) = struct.unpack_from( ">f", reply, replyofs+1) replyofs += struct.calcsize( ">f") + 1 elif (reply[ replyofs] == 'I'): (docidlen,) = struct.unpack_from( ">H", reply, replyofs+1) replyofs += struct.calcsize( ">H") + 1 (row_docid,) = struct.unpack_from( "%us" % docidlen, reply, replyofs) replyofs += docidlen elif (reply[ replyofs] == 'T'): (titlelen,) = struct.unpack_from( ">H", reply, replyofs+1) replyofs += struct.calcsize( ">H") + 1 (row_title,) = struct.unpack_from( "%us" % titlelen, reply, replyofs) replyofs += titlelen elif (reply[ replyofs] == 'A'): (abstractlen,) = struct.unpack_from( ">H", reply, replyofs+1) replyofs += struct.calcsize( ">H") + 1 (row_abstract,) = struct.unpack_from( "%us" % abstractlen, reply, replyofs) replyofs += abstractlen else: rt = (None, "storage server %s:%u protocol error: unknown result column name" % (host,port)) row_docid = None break; if (row_docid != None): result.append( ResultRow( row_docno, row_docid, row_weight, row_title, row_abstract)) rt = (result, None) else: rt = (None, "protocol error storage %s:%u query: unknown header %s" % (host,port,reply[0])) except Exception as e: rt = (None, "storage server %s:%u connection error: %s" % (host, port, str(e))) raise tornado.gen.Return( rt) @tornado.gen.coroutine def issueQueries( self, servers, qryblob): results = None try: results = yield [ self.issueQuery( addr, qryblob) for addr in servers ] except Exception as e: raise tornado.gen.Return( [], ["error issueing query: %s" % str(e)]) raise tornado.gen.Return( results) # Merge code derived from Python Cookbook (Sebastien Keim, Raymond Hettinger and Danny Yoo) # referenced in from http://wordaligned.org/articles/merging-sorted-streams-in-python: def mergeResultIter( self, resultlists): # prepare a priority queue whose items are pairs of the form (-weight, resultlistiter): heap = [ ] for resultlist in resultlists: resultlistiter = iter(resultlist) for result in resultlistiter: # subseq is not empty, therefore add this subseq's pair # (current-value, iterator) to the list heap.append((-result.weight, result, resultlistiter)) break # make the priority queue into a heap heapq.heapify(heap) while heap: # get and yield the result with the highest weight (minus lowest negative weight): negative_weight, result, resultlistiter = heap[0] yield result for result in resultlistiter: # resultlists is not finished, replace best pair in the priority queue heapq.heapreplace( heap, (-result.weight, result, resultlistiter)) break else: # subseq has been exhausted, therefore remove it from the queue heapq.heappop( heap) def mergeQueryResults( self, results, firstrank, nofranks): merged = [] errors = [] itrs = [] maxnofresults = firstrank + nofranks for result in results: if (result[0] == None): errors.append( result[1]) else: itrs.append( iter( result[0])) ri = 0 for result in self.mergeResultIter( itrs): if (ri == maxnofresults): break merged.append( result) ri += 1 return (merged[ firstrank:maxnofresults], errors) @tornado.gen.coroutine def evaluateQueryText( self, querystr, firstrank, nofranks): rt = None try: maxnofresults = firstrank + nofranks terms = analyzer.analyzePhrase( "text", querystr) if len( terms) == 0: # Return empty result for empty query: raise tornado.gen.Return( [] ) # Get the global statistics: dflist,collectionsize,error = yield self.queryStats( terms) if (error != None): raise Exception( error) # Assemble the query: qry = bytearray(b"Q") qry += bytearray( b"S") + struct.pack( ">q", collectionsize) qry += bytearray( b"I") + struct.pack( ">H", 0) qry += bytearray( b"N") + struct.pack( ">H", maxnofresults) for ii in range( 0, len( terms)): qry += bytearray( b"T") typesize = len(terms[ii].type()) valuesize = len(terms[ii].value()) qry += struct.pack( ">qHH", dflist[ii], typesize, valuesize) qry += struct.pack( "%ds%ds" % (typesize,valuesize), terms[ii].type(), terms[ii].value()) # Query all storage servers: results = yield self.issueQueries( storageservers, qry) rt = self.mergeQueryResults( results, firstrank, nofranks) except Exception as e: rt = ([], ["error evaluation query: %s" % str(e)]) raise tornado.gen.Return( rt) @tornado.gen.coroutine def get(self): try: # q = query terms: querystr = self.get_argument( "q", None) # i = firstrank: firstrank = int( self.get_argument( "i", 0)) # n = nofranks: nofranks = int( self.get_argument( "n", 20)) # Evaluate query with BM25 (Okapi): result = yield self.evaluateQueryText( querystr, firstrank, nofranks) # Render the results: self.render( "search_bm25_html.tpl", results=result[0], messages=result[1]) except Exception as e: self.render( "search_error_html.tpl", message=e) # Insert a multipart document (POST request): class InsertHandler( tornado.web.RequestHandler ): @tornado.gen.coroutine def post(self, port): try: # Insert documents: conn = yield msgclient.connect( 'localhost', int(port)) reply = yield msgclient.issueRequest( conn, b"I" + bytearray( self.request.body)) if (reply[0] == 'E'): raise Exception( bytes( reply[1:])) elif (reply[0] != 'Y'): raise Exception( "protocol error server reply on insert") (nofDocuments,) = struct.unpack( ">I", reply[1:]) self.write( "OK %u\n" % (nofDocuments)) except Exception as e: self.write( "ERR %s\n" % str(e)) # [3] Dispatcher: application = tornado.web.Application([ # /query in the URL triggers the handler for answering queries: (r"/query", QueryHandler), # /insert in the URL triggers the post handler for insert requests: (r"/insert/([0-9]+)", InsertHandler), # /static in the URL triggers the handler for accessing static # files like images referenced in tornado templates: (r"/static/(.*)",tornado.web.StaticFileHandler, {"path": os.path.dirname(os.path.realpath(sys.argv[0]))},) ]) def on_shutdown(): print('Shutting down') tornado.ioloop.IOLoop.current().stop() # [5] Server main: if __name__ == "__main__": try: # Parse arguments: usage = "usage: %prog [options] {<storage server port>}" parser = optparse.OptionParser( usage=usage) parser.add_option("-p", "--port", dest="port", default=80, help="Specify the port of this server as PORT (default %u)" % 80, metavar="PORT") parser.add_option("-s", "--statserver", dest="statserver", default=statserver, help="Specify the address of the statistics server as ADDR (default %s" % statserver, metavar="ADDR") (options, args) = parser.parse_args() myport = int(options.port) statserver = options.statserver if (statserver[0:].isdigit()): statserver = '{}:{}'.format( 'localhost', statserver) # Positional arguments are storage server addresses, if empty use default at localhost:7184 for arg in args: if (arg[0:].isdigit()): storageservers.append( '{}:{}'.format( 'localhost', arg)) else: storageservers.append( arg) if (len( storageservers) == 0): storageservers.append( "localhost:7184") # Start server: print( "Starting server ...\n") application.listen( myport ) print( "Listening on port %u\n" % myport ) ioloop = tornado.ioloop.IOLoop.current() signal.signal( signal.SIGINT, lambda sig, frame: ioloop.add_callback_from_signal(on_shutdown)) ioloop.start() print( "Terminated\n") except Exception as e: print( e)
import strus import itertools import heapq import re class Backend: # Create the document analyzer for our test collection: # Create the document analyzer for our test collection: def createDocumentAnalyzer(self): rt = self.context.createDocumentAnalyzer() # Define the sections that define a document (for multipart documents): rt.defineDocument( "doc", "/list/item") # Define the terms to search for (inverted index or search index): rt.addSearchIndexFeature( "word", "/list/item/title()", "word", ("lc",("stem","en"),("convdia","en"))) rt.addSearchIndexFeature( "word", "/list/item/artist()", "word", ("lc",("stem","en"),("convdia","en"))) rt.addSearchIndexFeature( "word", "/list/item/note()", "word", ("lc",("stem","en"),("convdia","en"))) # Define the terms to search for (inverted index or search index): rt.addForwardIndexFeature( "orig", "/list/item/title()", "split", "orig") rt.addForwardIndexFeature( "orig", "/list/item/artist()", "split", "orig") rt.addForwardIndexFeature( "orig", "/list/item/note()", "split", "orig") # Define the document attributes: rt.defineAttribute( "docid", "/list/item/id()", "content", "text") rt.defineAttribute( "title", "/list/item/title()", "content", "text") rt.defineAttribute( "upc", "/list/item/upc()", "content", "text") rt.defineAttribute( "note", "/list/item/note()", "content", "text") # Define the document meta data: rt.defineMetaData( "date", "/list/item/date()", "content", [("date2int", "d 1877-01-01", "%Y-%m-%d %H:%M:%s *")]); # Define the doclen attribute needed by BM25: rt.defineAggregatedMetaData( "doclen",("count", "word")) return rt # Create the query analyzer according to the document analyzer configuration: def createQueryAnalyzer(self): rt = self.context.createQueryAnalyzer() rt.definePhraseType( "text", "word", "word", ["lc", ["stem", "en"], ["convdia", "en"]] ) return rt # Create a simple BM25 query evaluation scheme with fixed # a,b,k1 and avg document lenght and title with abstract # as summarization attributes: def createQueryEvalBM25(self): rt = self.context.createQueryEval() # Declare the sentence marker feature needed for abstracting: rt.addTerm( "sentence", "sent", "") # Declare the feature used for selecting result candidates: rt.addSelectionFeature( "selfeat") # Query evaluation scheme: rt.addWeightingFunction( 1.0, "BM25", { "k1": 0.75, "b": 2.1, "avgdoclen": 20, ".match": "docfeat" }) # Summarizer for getting the document title: rt.addSummarizer( "DOCID", "attribute", { "name": "docid" }) rt.addSummarizer( "TITLE", "attribute", { "name": "title" }) # Summarizer for abstracting: rt.addSummarizer( "CONTENT", "matchphrase", { "type": "orig", "len": 40, "nof": 3, "mark": '<b>$</b>', ".match": "docfeat" }) return rt # Constructor. Initializes the query evaluation schemes and the query and document analyzers: def __init__(self, config): # Open local storage on file with configuration specified: self.context = strus.Context() self.context.defineStatisticsProcessor( "standard"); self.storage = self.context.createStorageClient( config ) self.queryAnalyzer = self.createQueryAnalyzer() self.documentAnalyzer = self.createDocumentAnalyzer() self.queryeval = self.createQueryEvalBM25() # Insert a multipart document: def insertDocuments( self, content): rt = 0 docqueue = self.documentAnalyzer.createQueue() docqueue.push( content) transaction = self.storage.createTransaction() while (docqueue.hasMore()): doc = docqueue.fetch() transaction.insertDocument( doc.docid(), doc) rt += 1 transaction.commit() return rt # Query evaluation scheme for a classical information retrieval query with BM25: def evaluateQuery( self, terms, collectionsize, firstrank, nofranks): queryeval = self.queryeval query = queryeval.createQuery( self.storage) if len( terms) == 0: # Return empty result for empty query: return [] selexpr = ["contains"] for term in terms: selexpr.append( [term.type, term.value] ) query.defineFeature( "docfeat", [term.type, term.value], 1.0) query.defineTermStatistics( term.type, term.value, {'df' : int(term.df)} ) query.defineFeature( "selfeat", selexpr, 1.0 ) query.setMaxNofRanks( nofranks) query.setMinRank( firstrank) query.defineGlobalStatistics( {'nofdocs' : int(collectionsize)} ) # Evaluate the query: results = query.evaluate() # Rewrite the results: rt = [] for result in results: content = "" title = "" docid = "" for attribute in result.attributes(): if attribute.name() == 'CONTENT': if content != "": content += ' ... ' content += attribute.value() elif attribute.name() == 'DOCID': docid = attribute.value() elif attribute.name() == 'TITLE': title = attribute.value() rt.append( { 'docno':result.docno(), 'docid':docid, 'title':title, 'weight':result.weight(), 'abstract':content }) return rt # Get an iterator on all absolute statistics of the local storage def getInitStatisticsIterator( self): return self.storage.createInitStatisticsIterator( True) # Get an iterator on all absolute statistics of the local storage def getDoneStatisticsIterator( self): return self.storage.createInitPeerMessageIterator( False) # Get an iterator on updates to be sent to peers def getUpdateStatisticsIterator( self): return self.storage.createUpdateStatisticsIterator()
#!/usr/bin/python import tornado.ioloop import tornado.gen import tornado.tcpclient import tornado.tcpserver import signal import os import sys import struct import binascii class TcpConnection( object): def __init__(self, stream, command_callback): self.stream = stream self.command_callback = command_callback @tornado.gen.coroutine def on_connect(self): try: while (True): msgsizemsg = yield self.stream.read_bytes( struct.calcsize(">I")) (msgsize,) = struct.unpack( ">I", msgsizemsg) msg = yield self.stream.read_bytes( msgsize) reply = yield self.command_callback( msg) yield self.stream.write( struct.pack( ">I", len(reply)) + bytes(reply)); except tornado.iostream.StreamClosedError: pass class RequestServer( tornado.tcpserver.TCPServer): def __init__(self, command_callback, shutdown_callback): tornado.tcpserver.TCPServer.__init__(self) self.command_callback = command_callback self.shutdown_callback = shutdown_callback self.io_loop = tornado.ioloop.IOLoop.current() def do_shutdown( self, signum, frame): print('Shutting down') self.shutdown_callback() self.io_loop.stop() @tornado.gen.coroutine def handle_stream( self, stream, address): connection = TcpConnection( stream, self.command_callback) yield connection.on_connect() def start( self, port): host = "<IP_ADDRESS>" self.listen( port, host) print("Listening on %s:%d..." % (host, port)) signal.signal( signal.SIGINT, self.do_shutdown) self.io_loop.start() class RequestClient( tornado.tcpclient.TCPClient): @tornado.gen.coroutine def issueRequest( self, stream, msg): blob = struct.pack( ">I", len(msg)) + bytes(msg) stream.write( blob); replysizemsg = yield stream.read_bytes( struct.calcsize(">I")) (replysize,) = struct.unpack( ">I", replysizemsg) reply = yield stream.read_bytes( replysize) raise tornado.gen.Return( reply)
The dataset used in this article is from MusicBrainz (musicbrainz.org).
Please respect the license of this dataset !
The docker image is started with
docker run -p 40080:80 -t -i patrickfrey/strus-ub1404-torndist:v0_3 /bin/bash
and you a prompt like this
root@3810f0b27943:/home/strus#
The following shell commands are executed in the docker image.
The document collection used in the examples is from musicbrainz.
It contains a list of music recordings.
To download and prepare the documents you run the shell script
./prepare.sh
We see that a directory data has been created that contains some files and directory. One subdirectory created is 'doc'. We type
ls data/doc/
to see its content
0.xml 12.xml 15.xml 18.xml 20.xml 23.xml 26.xml 29.xml 31.xml 3.xml 6.xml 9.xml 10.xml 13.xml 16.xml 19.xml 21.xml 24.xml 27.xml 2.xml 32.xml 4.xml 7.xml 11.xml 14.xml 17.xml 1.xml 22.xml 25.xml 28.xml 30.xml 33.xml 5.xml 8.xml
We have a look at the multipart document 0.xml. Later we will make a very unbalanced
collection split with one storage server containing the documents of this file, another server
containing the documents of the file 1.xml and 2.xml and the third server containing
the rest of the documents. We will show that the weighting of the result documents is the same
as if we would have used one server only. The goal is to show that strus allows us to distribute
the search index arbitrarily without affecting weighting, because in both cases we use the same
statistics for weighting.
First we chose an entry in the first document that we want to have within the best matches.
This way we show that the weighting is also stable for the smallest sub-collection.
less data/doc/0.xml
It shows us the following list
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <list> <item><id>25</id><title>Dead Love Songs</title><artist>The Black</artist><date>2008-11-25 05:43:04.124028+00</date><upc></upc><note></note></item> <item><id>46</id><title>Music of India - Volume 1 (SANTOOR)</title><artist>Musenalp</artist><date>2008-11-25 17:31:54.456355+00</date><upc></upc><note></note></item> <item><id>2616</id><title>Vianoce</title><artist>Sklo</artist><date>2008-12-19 20:49:47.115832+00</date><upc></upc><note></note></item> ....
The second entry with id 46 contains the words "Music" and "India". This will be our example query terms. As ground truth we will first insert the whole collection into one storage. The resulting ranklist of one server for the query "Music India" will be our reference ranklist.
We create the storage with command line tools. We could also create it with the strus API, but commands to create a storage are not part of the example implementation of this article. The storage is created with the following command:
strusCreate -s "path=storage; metadata=doclen UINT16, date UINT32"
We first start the statistics server:
./strusStatisticsServer.py &
and we get
Starting server ... Listening on <IP_ADDRESS>:7183...
Then we start one storage server.
./strusStorageServer.py &
and we get
Starting server ... Listening on <IP_ADDRESS>:7184...
Then we start the HTTP server.
./strusHttpServer.py &
and we get
Starting server ... Listening on port 80
Now we use cURL to insert the collection. We use a prepared script for this. The script looks as follows:
#!/bin/sh if [ "$#" -lt 2 ]; then echo "Usage: $0 <server-port> <start-range> [<end-range>]" >&2 exit 1 fi port=`expr 0 \+ $1` start=`expr 0 \+ $2` if [ "$#" -lt 3 ]; then end=$start else end=`expr 0 \+ $3` fi for i in $(seq $start $end) do curl -X POST -d @data/doc/$i.xml localhost:80/insert/$port --header "Content-Type:text/xml" done
The script takes the HTTP server port and the range of documents to insert as argument. The call to insert the complete collection looks as follows:
./insert_docs.sh 7184 0 33
and we get a list of 34 replies with the number of documents inserted with each call of cURL that looks like this:
OK 5090 OK 5513 OK 8560 OK 9736 OK 9708 OK 9751 ....
Now you can issue the query "Music India" with your favorite browser with the following URL:
http://<IP_ADDRESS>:40080/query?q=Music%20India&i=0&n=20
You get the following results. The document we seletected from the first file is marked in the ranklist:
Now we build the same collection on three storage nodes. We first shutdown all servers:
ps -fe | grep python | grep strus | awk '{print $2}' | xargs kill -s SIGTERM
and we create 3 search indexes (we include the server port in the file name):
strusCreate -s "path=storage7184; metadata=doclen UINT16, date UINT32" strusCreate -s "path=storage7185; metadata=doclen UINT16, date UINT32" strusCreate -s "path=storage7186; metadata=doclen UINT16, date UINT32"
and we get
storage successfully created. storage successfully created. storage successfully created.
Now we restart all servers again, but this time with 3 storage servers on port 7184,7185 and 7186. First we start the statistics server:
./strusStatisticsServer.py -p 7183 &
and we get
Starting server ... Listening on <IP_ADDRESS>:7183...
then the storage servers
./strusStorageServer.py -P -p 7184 -c "path=storage7184; cache=512M" & ./strusStorageServer.py -P -p 7185 -c "path=storage7185; cache=512M" & ./strusStorageServer.py -P -p 7186 -c "path=storage7186; cache=512M" &
and we get
Starting server ... Listening on <IP_ADDRESS>:7184... Load local statistics to publish ... Starting server ... Listening on <IP_ADDRESS>:7185... Load local statistics to publish ... Starting server ... Listening on <IP_ADDRESS>:7186... Load local statistics to publish ...
The HTTP server is started with the ports of the storage servers as arguments. When we started it without argument, it used the default storage server port as only server to query.
./strusHttpServer.py 7184 7185 7186 &
and we get
Starting server ... Listening on port 80
As explained in the beginning, we insert one multipart document into the storage (7184), two of them into the second (7185) and the rest into the third (7186). We do this with our insert_docs command:
./insert_docs.sh 7184 0 0 & ./insert_docs.sh 7185 1 2 & ./insert_docs.sh 7186 3 33 &
When all scripts are finished, we verify, that the documents went really to different servers. For this we check the index sizes:
ls -lh storage718* | grep total
and we see that the storage index sizes are really different:
total 1.9M total 5.9M total 97M
Now we issue the same query "Music India" again:
http://<IP_ADDRESS>:40080/query?q=Music%20India&i=0&n=20
And we get the following results. The document we seletected from the mulipart document inserted is marked in the ranklist as in the result for one server:
We see that the ranklists with one or three servers are the same with the following exeptions:
Something is fishy here with the docid and the docno. They seem to be mixed. Take the title and the weight as reference. I will fix this small bug in the presentation of the result as soon as possible.
In my previous articles I have shown that it is possible to build simple
as well as complex search applications with Strus. The model of strus allows to match regular
language patterns (with terms as alphabet) and extract tuples of tokens related to matches.
You can implement classical information retrieval weighting documents as well as alternative retrieval
methods that are accumulating content from documents for presentation or further processing.
Strus has a competitive performance that allows you to it feed a single storage node with millions of documents
within a day, and to query such collections with an immediate response.
The Wikipedia demo search illustrates this.
If you run a query twice in a row, you get the response time without accessing disk.
There is no query result cache involved. The machine is old and the amount of memory a single
process can use is about 4G. That is below the minimum value recommended by Leveldb for a key/value
storage if you consider the search index size of 50G. Furthermore the maximum number of files
that can be kept open on this machine is much smaller (128) than the Leveldb storage would like to keep open.
In this article we saw that Strus is a scalable search engine. It could potentially run on a cluster
with thusands of machines providing search on a collection of billions of documents.
Strus matches regular patterns with help of search index and allows to weight documents containing
the matches and extract tuples related to matches for further processing. Nothing more.
If this can provide you a small brick for solving your problem, you maybe think about using Strus.
The computer science world got very competitive in the last decade.
Software ecomomy wants to profit by this competition.
The Strus core needed less than one man year of development and the data storage is based
on 3rd party software (LevelDB or alternatives). This opens it to competition.
It is up to you to decide, wheter it is worth a try or not.


Home > Reference > Functions reference > Get functions > Get(ApplicationLanguage)
Get(ApplicationLanguage) Purpose Returns text representing the current application language. Format Get(ApplicationLanguage) Parameters None Data type returned text Originated in FileMaker Pro 7.0 Description The text that is returned by this function is in the English language. For hosted databases, Get(ApplicationLanguage) returns the client’s current language. Note In FileMaker WebDirect, Get(ApplicationLanguage) returns the web browser’s current language. FileMaker Pro supports: • English • French • Italian • German • Swedish • Spanish • Dutch • Japanese • Simplified Chinese • Brazilian Portuguese • Korean Examples Returns English when the current application language is English. Related topics Functions reference (category list) Functions reference (alphabetical list) About formulas About functions Defining calculation fields Using operators in formulas


About Us
As artists, the Internet has enabled us to share our works, interact with our audiences, and in recent years has given us the ability to sell our designs. Although the Internet can provide great visuals of the work with amazing sound, some works of art are meant to be touched. The texture is a fundamental element of a three-dimensional art. By being able to touch the work you find a stronger connection with it, fulfilling the whole experience of the work. Our vision is to design a website where you can touch the work of art that you are interested in before purchasing it, through the use of an external device that can adjust itself to mimic the work of art. This allows the user to get the full experience in shopping and helps the artist to better communicate the meaning of their work.


Flathub is a build and distribution service for Flatpak applications. Its goal is to act as a central hub for making desktop applications available to users.
To download desktop applications on a modern Linux-based system, download the repository file and add it to your system with GNOME Software or the Flatpak command line.
To download and install applications from Flathub, you will need to install Flatpak (if you haven't already) and then add the repository in one of two ways:
If you are on Arch, Debian Stretch (or later), Endless OS, Fedora 25 (or later), Mageia 6 (or later), or OpenSUSE Tumbleweed, you can download the repository file and use GNOME Software to install it.
From the command line, run:
$ flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
You can then browse and install applications in the normal manner. See the Flatpak command line tutorial for more information on this.
If you develop a Linux application and would like to make it available via Flathub, we would appreciate your contribution. Follow the instructions at submission guidelines to submit your app.
Security or legal issues can be reported to the Flathub maintainers.


Hi! My name is Jin, I am a qualified pharmacist who is interested in all things tech!
Here are some links to projects I am involved in


Toggle navigation
MyServicer.gov
Servicer.gov
Yesterday
Your Loan Analysis is ready for review!!! To view, please click the link below, other...
Customer Service
Yesterday
Good Afternoon, In response to your question regarding the Pell Loan, you would be...
Read All Messages
Loans Repaid
80% Complete
80% Complete (danger)
See More Details
New Article Suggestion
4 minutes ago
New Milestone!!
2 hours ago
Upcoming Payment
2 Days ago
See All Alerts
User Profile
Badges
Settings
Logout
My Profile
Badges
Ask A Question
Frequent Feedback
25% Paid Off
20 Videos Watched
My Personal Assistant
Name:
Steve
Motto:
"Always Here And Happy to Help!!"
Prefferred Visual Representations
Bar Graphs
Line Graphs
I Like It All
Prefferred Media Content
Videos
Reading (Articles, Studies, Etc)
Podcasts
Any Way I Can Get It
About Me
Life Changes
Life Happens... We get that
I Got Married!!
I Got Divorced!!
I Had/Adopted a Child.
My Partner Passed.
Tell us about it
Do you want to refinance?
Yes
More Info First
No
Inline Checkboxes
1
2
3
More Labels
This
That
Karma
Submit Button
Reset Button
Basic Form Elements
Career Change
New Job
Promotion
Lost Job
Caught Stealing Boxes
Tell Us About It
Will this affect your income?
Yes
No
Do you want to refinance?
Yes
More Info Please
No
Submit Button
Reset Button


This is the public summary for my MPhys project. It takes the form of a text interface, which gradually introduces topics for exploration.