(: simplified version not dealing with special match options :) declare function fts:applyQueryTokensAsPhrase ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryTokens as element(fts:queryToken)*, $queryPos as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$queryPos}"> { for $tokenInfo in fts:matchTokenInfos( $searchContext, $matchOptions, (), $queryTokens ) return <fts:match> <fts:stringInclude queryPos="{$queryPos}" isContiguous="true"> {$tokenInfo} </fts:stringInclude> </fts:match> } </fts:allMatches> };
If after the application of all the match options, the sequence of query tokens returned for an FTWords is empty, an empty AllMatches is returned.
The AllMatches corresponding to an FTWords is a
set of Matches. Each of the Matches is associated
with a starting and an ending position indicating where the
corresponding query tokens were found. For example, the
AllMatches result for the FTWords "Mustang" is given
below. To simplify the presentation in the figures we write
Pos: N
, if the attributes startPos
and
endPos
are the same with N
being that
position.
There are five variations of FTWords depending on how the tokens and phrases in the nested XQuery 1.0 and XPath 2.0 expression are matched.
When any word
is specified, at least one token in
the tokenization of the nested expression must be matched.
When all word
is specified, all tokens in the
tokenization of the nested expression must be matched.
When phrase
is specified, all tokens in the
tokenization of the nested expression must be matched as a
phrase.
When any
is specified, at least one string atomic
value in the nested expression must be matched as a phrase.
When all
is specified, all string atomic values in
the nested expression must be matched as a phrase.
The semantics for FTWords when any word
is
specified is given below. Since FTWords does not have nested
FTSelections, the ApplyFTWords
function does
not take AllMatches parameters corresponding to nested
FTSelection results.
declare function fts:MakeDisjunction ( $curRes as element(fts:allMatches), $rest as element(fts:allMatches)* ) as element(fts:allMatches) { if (fn:count($rest) = 0) then $curRes else let $firstAllMatches := $rest[1] let $restAllMatches := fn:subsequence($rest, 2) let $newCurRes := fts:ApplyFTOr($curRes, $firstAllMatches) return fts:MakeDisjunction($newCurRes, $restAllMatches) }; declare function fts:ApplyFTWordsAnyWord ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { (: Tokenization of query string has already occurred. :) (: Get sequence of QueryTokens over all query items. :) let $queryTokens := $queryItems/fts:queryToken return if (fn:count($queryTokens) eq 0) then <fts:allMatches stokenNum="0" /> else let $allAllMatches := for $queryToken at $pos in $queryTokens return fts:applyQueryTokensAsPhrase($searchContext, $matchOptions, $queryToken, $queryPos + $pos - 1) let $firstAllMatches := $allAllMatches[1] let $restAllMatches := fn:subsequence($allAllMatches, 2) return fts:MakeDisjunction($firstAllMatches, $restAllMatches) };
The tokenized query strings are passed to ApplyFTWordsAnyWord as
a sequence of fts:queryItem
, each containing the
tokens of a single query string. A single flattened sequence of all
tokens (of type fts:queryToken
) over all query items
is constructed. For each of these, the result of FTWords is
computed using applyQueryTokensAsPhrase
. Finally, the
disjunction of all resulting AllMatches is computed.
The semantics for FTWords when all word
is
specified is similar to the above, however composes a conjunction.
It is given below.
declare function fts:MakeConjunction ( $curRes as element(fts:allMatches), $rest as element(fts:allMatches)* ) as element(fts:allMatches) { if (fn:count($rest) = 0) then $curRes else let $firstAllMatches := $rest[1] let $restAllMatches := fn:subsequence($rest, 2) let $newCurRes := fts:ApplyFTAnd($curRes, $firstAllMatches) return fts:MakeConjunction($newCurRes, $restAllMatches) }; declare function fts:ApplyFTWordsAllWord ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { (: Tokenization of query strings has already occurred. :) (: Get sequence of QueryTokens over all query items :) let $queryTokens := $queryItems/fts:queryToken return if (fn:count($queryTokens) eq 0) then <fts:allMatches stokenNum="0" /> else let $allAllMatches := for $queryToken at $pos in $queryTokens return fts:applyQueryTokensAsPhrase($searchContext, $matchOptions, $queryToken, $queryPos + $pos - 1) let $firstAllMatches := $allAllMatches[1] let $restAllMatches := fn:subsequence($allAllMatches, 2) return fts:MakeConjunction($firstAllMatches, $restAllMatches) };
The semantics for FTWords if phrase
is
specified is given below.
declare function fts:ApplyFTWordsPhrase ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { (: Get sequence of QueryTokenInfos over all query items :) let $queryTokens := $queryItems/fts:queryToken return if (fn:count($queryTokens) eq 0) then <fts:allMatches stokenNum="0" /> else fts:applyQueryTokensAsPhrase($searchContext, $matchOptions, $queryTokens, $queryPos) };
The ApplyFTWordsPhrase
function also flattens the
sequence of query items to a sequence of query tokens, but then
calls applyQueryTokensAsPhrase
on that entire
sequence, instead of calling it on each query token individually.
Hence, the sequence of all query tokens is matched as a single
phrase and the computed TokenInfos are returned.
The semantics for FTWords when any
is
specified is given below.
declare function fts:ApplyFTWordsAny ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { if (fn:count($queryItems) eq 0) then <fts:allMatches stokenNum="0" /> else let $firstQueryItem := $queryItems[1] let $restQueryItem := fn:subsequence($queryItems, 2) let $firstAllMatches := fts:ApplyFTWordsPhrase($searchContext, $matchOptions, $firstQueryItem, $queryPos) let $newQueryPos := if ($firstAllMatches//@queryPos) then fn:max($firstAllMatches//@queryPos) + 1 else $queryPos let $restAllMatches := fts:ApplyFTWordsAny($searchContext, $matchOptions, $restQueryItem, $newQueryPos) return fts:ApplyFTOr($firstAllMatches, $restAllMatches) };
The FTWords with any
specified forms the
disjunction of the AllMatches that are the result of the
matching of each query item as a phrase.
The semantics for FTWords when all
is
specified is given below.
declare function fts:ApplyFTWordsAll ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { if (fn:count($queryItems) = 0) then <fts:allMatches stokenNum="0" /> else let $firstQueryItem := $queryItems[1] let $restQueryItem := fn:subsequence($queryItems, 2) let $firstAllMatches := fts:ApplyFTWordsPhrase($searchContext, $matchOptions, $firstQueryItem, $queryPos) return if ($restQueryItem) then let $newQueryPos := if ($firstAllMatches//@queryPos) then fn:max($firstAllMatches//@queryPos) + 1 else $queryPos let $restAllMatches := fts:ApplyFTWordsAll($searchContext, $matchOptions, $restQueryItem, $newQueryPos) return fts:ApplyFTAnd($firstAllMatches, $restAllMatches) else $firstAllMatches };
The difference between all
and any
is
the use of conjunction instead of disjunction.
The ApplyFTWords
function combines all of these
functions.
declare function fts:ApplyFTWords ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $type as fts:ftWordsType, $queryItems as element(fts:queryItem)*, $queryPos as xs:integer ) as element(fts:allMatches) { if ($type eq "any word") then fts:ApplyFTWordsAnyWord($searchContext, $matchOptions, $queryItems, $queryPos) else if ($type eq "all word") then fts:ApplyFTWordsAllWord($searchContext, $matchOptions, $queryItems, $queryPos) else if ($type eq "phrase") then fts:ApplyFTWordsPhrase($searchContext, $matchOptions, $queryItems, $queryPos) else if ($type eq "any") then fts:ApplyFTWordsAny($searchContext, $matchOptions, $queryItems, $queryPos) else fts:ApplyFTWordsAll($searchContext, $matchOptions, $queryItems, $queryPos) };
XQuery 1.0 functions are used to define the semantics of FTMatchOptions. These functions operate on an XML representation of the FTMatchOptions. The representation closely follows the syntax. Each FTMatchOption is represented by an XML element. Additional characteristics of the match option are represented as attributes.  All
processing of FTMatchOptions was delegated to the function
matchTokenInfos
, which is implementation-defined. In this
section, further details on the semantics of FTMatchOptions
are given.
The extension is achieved by modifying an existing function and adding functions that are specific to the FTMatchOptions.
Modifications in the semantics of existing functions
The semantics of most of the FTSelections remains unmodified. The modifications are to the method for matching a sequence of query tokens.
declare function fts:applyQueryTokensAsPhrase ( $searchContext as item(), $matchOptions as element(fts:matchOptions), $queryTokens as element(fts:queryToken)*, $queryPos as xs:integer ) as element(fts:allMatches) { let $thesaurusOption := $matchOptions/fts:thesaurus[1] return if ($thesaurusOption and $thesaurusOption/@thesaurusIndicator eq "using") then let $noThesaurusOptions := <fts:matchOptions>{ $matchOptions/*[fn:not(self::fts:thesaurus)] }</fts:matchOptions> let $lookupRes := fts:applyThesaurusOption($thesaurusOption, $noThesaurusOptions, $queryTokens) return fts:ApplyFTWordsAny($searchContext, $noThesaurusOptions, $lookupRes, $queryPos) else (: from here on we have a single sequence of query tokens :) (: which is to be matched a phrase; no alternatives anymore :) <fts:allMatches stokenNum="{$queryPos}"> { for $pos in fts:matchTokenInfos( $searchContext, $matchOptions, fts:applyStopWordOption($matchOptions/fts:stopwords), $queryTokens ) return <fts:match> <fts:stringInclude queryPos="{$queryPos}" isContiguous="true"> {$pos} </fts:stringInclude> </fts:match> } </fts:allMatches> };
Two FTMatchOptions need to be processed differently than the rest of the FTMatchOptions as shown in the function above.
Unlike all other FTMatchOptions the semantics of the
FTThesaurusOption cannot be formulated as an operation on
individual query tokens, because a thesaurus lookup may return
alternative query items for a whole phrase, i.e., a sequence of
query tokens. Since the result of a thesaurus lookup is a sequence
of alternatives, there must be a higher level of processing. The
above call to applyThesaurusOption
returns for the
given sequence of query tokens (representing a phrase) all
thesaurus expansions for the selected thesaurus, relationship and
level range as a sequence of query items. The alternative
expansions are evaluated as a disjunction using the
fts:ApplyFTWordsAny
. The matching of the alternatives
is performed with FTThesaurusOption turned off to avoid
double expansions, i.e., expansion of an already expanded
token.
For the semantics of the FTStopWordOption the list of stop words needs to be computed as demanded by the special syntax for stop word lists involving the operators "union" and "except".
Semantics of new FTMatchOptions functions
The expansion of FTSelections also includes adding additional functions that are specific to the FTMatchOptions.
The evaluate
function above handles match options
occurring in the query structure by using a call to the function
replaceMatchOptions
which is defined below. The latter
function replaces match options from the list given by the first
argument with match options of the same group in the list given by
the second argument, if any. If an option is present in the second
list but not in the first list, the option is included to the
resulting list too. Intuitively, the
replaceMatchOptions
computes the effective match
options for a given FTSelection. The function uses the
options specified specifically for the current FTSelection (
$ftSelection/fts:matchOptions
to override any options
of the same group declared up the query tree
($matchOptions
).
declare function fts:replaceMatchOptions ( $matchOptions as element(fts:matchOptions), $newMatchOptions as element(fts:matchOptions) ) as element(fts:matchOptions) { <fts:matchOptions> { (if ($newMatchOptions/fts:thesaurus) then $newMatchOptions/fts:thesaurus else $matchOptions/fts:thesaurus), (if ($newMatchOptions/fts:stopwords) then $newMatchOptions/fts:stopwords else $matchOptions/fts:stopwords), (if ($newMatchOptions/fts:case) then $newMatchOptions/fts:case else $matchOptions/fts:case), (if ($newMatchOptions/fts:diacritics) then $newMatchOptions/fts:diacritics else $matchOptions/fts:diacritics), (if ($newMatchOptions/fts:stem) then $newMatchOptions/fts:stem else $matchOptions/fts:stem), (if ($newMatchOptions/fts:wildcard) then $newMatchOptions/fts:wildcard else $matchOptions/fts:wildcard), (if ($newMatchOptions/fts:language) then $newMatchOptions/fts:language else $matchOptions/fts:language) } </fts:matchOptions> };
This function determines how match options of the same group overwrite each other, so that only one option of the same group remains.
The details of the semantics of the remaining
FTMatchOptions are determined by the implementation-defined function
matchTokenInfos
.
FTMatchOption functions which are necessary to support match option processing are given below.
declare function fts:resolveStopWordsUri ( $uri as xs:string? ) as xs:string* external; declare function fts:lookupThesaurus ( $tokens as element(fts:queryToken)*, $thesaurusName as xs:string?, $relationship as xs:string?, $range as element(fts:range)?, $noThesaurusOptions as element(fts:matchOptions) ) as element(fts:queryItem)* external;
The function resolveStopWordsUri
is used to resolve
any URI to a sequence of strings to be used as stop words.
The function lookupThesaurus
finds all expansions
related to $tokens
in the thesaurus
$thesaurusName
using the relationship
$relationship
within the optional number of levels
$range
. If $tokens
consists of more than
one query token, it is regarded as a phrase.
The
current match options other than the thesaurus option are also
passed to the function, via $noThesaurusOptions
,
allowing the implementation to apply any of those match options
(whichever it deems relevant) to the input or output of the actual
thesaurus lookup.
The thesaurus function returns a sequence of expansion alternatives. Each alternative is regarded as a new search phrase and is represented as a query item. Alternatives are treated as though they are connected with a disjunction (FTOr).
FTMatchOptions of type FTCaseOption are passed in
the $matchOptions
parameter to
matchTokenInfos
. If the FTCaseOption is
"lowercase" the returned TokenInfos must span only tokens
that are all lowercase. If the FTCaseOption is "uppercase"
the returned TokenInfos must span only tokens that are all
uppercase. If the FTCaseOption is "case insensitive" the
function must return all TokenInfos matching the query
tokens when disregarding character case. If the FTCaseOption
is "case sensitive" the function must return all TokenInfos
that also accord with the query tokens in character case.
FTMatchOptions of type FTDiacriticsOption are
passed in the $matchOptions
parameter to
matchTokenInfos
. If the FTDiacriticsOption is
"diacritics insensitive" the function must return all
TokenInfos matching the query tokens when disregarding
diacritical marks. If the FTDiacriticsOption is "diacritics
sensitive" the function must return all TokenInfos that also
accord with the query tokens in diacritical marks.
FTMatchOptions of type FTStemOption are passed in
the $matchOptions
parameter to
matchTokenInfos
. It is implementation-defined what the
effect of the option "stemming" is on matching tokens, however, it
is expected that this option allows to match linguistic variants of
the query tokens. If the FTStemOption is "no stemming" the
returned TokenInfos must span exact matches (i.e. not
including linguistic variations) of the query tokens.
The semantics for the FTThesaurusOption is given below.
declare function fts:applyThesaurusOption ( $matchOption as element(fts:thesaurus), $noThesaurusOptions as element(fts:matchOptions), $queryTokens as element(fts:queryToken)* ) as element(fts:queryItem)* { if ($matchOption/@thesaurusIndicator = "using") then fts:lookupThesaurus( $queryTokens, $matchOption/fts:thesaurusName, $matchOption/fts:relationship, $matchOption/fts:range, $noThesaurusOptions ) else if ($matchOption/@thesaurusIndicator = "no") then <fts:queryItem> {$queryTokens} </fts:queryItem> else () };
Stop words interact with FTDistance and FTWindow. The semantics for the FTStopWordOption is given below.
declare function fts:applyStopWordOption ( $stopWordOption as element(fts:stopwords)? ) as xs:string* { if ($stopWordOption) then let $swords := typeswitch ($stopWordOption/*[1]) case $e as element(fts:stopword) return $e/text() case $e as element(fts:uri) return fts:resolveStopWordsUri($e/text()) case element(fts:default-stopwords) return fts:resolveStopWordsUri(()) default return () return fts:calcStopWords( $swords, $stopWordOption/fts:oper ) else () }; declare function fts:calcStopWords ( $stopWords as xs:string*, $opers as element(fts:oper)* ) as xs:string* { if ( fn:empty($opers) ) then $stopWords else let $swords := typeswitch ($opers[1]/*[1]) case $e as element(fts:stopword) return $e/text() case $e as element(fts:uri) return fts:resolveStopWordsUri($e/text()) default return () return if ($opers[1]/@type eq "union") then fts:calcStopWords( ($stopWords, $swords), $opers[fn:position() gt 2] ) else (: "except" :) fts:calcStopWords( $stopWords[fn:not(.)=$swords], $opers[fn:position() gt 2] ) };
Given the applicable setting of the Stop Word Option, the
function fts:applyStopWordOption
calls
fts:calcStopWords
to compute the set of stop words,
and returns that set as an instance of xs:string*
.
This then is passed to fts:matchTokenInfos
, which uses
it to affect the matching of tokens. The
fts:calcStopWords
function uses the function
fts:resolveStopWordsUri
to resolve any URI to a
sequence of strings.
FTMatchOptions of type FTWildCardOption are passed
in the $matchOptions
parameter to
matchTokenInfos
. If the FTWildCardOption is
"wildcards" the function must return all TokenInfos in the
search context that span tokens, such that those tokens are
wildcard expansions of the corresponding query token. The wildcard
expansions are described in Section
3.2.7 FTWildCardOption. If the FTWildCardOption is "no
wildcards" all query tokens must be matched literally.
The parameters of the ApplyFTOr
function are the
two AllMatches parameters corresponding to the results of
the two nested FTSelections. The semantics is given
below.
declare function fts:ApplyFTOr ( $allMatches1 as element(fts:allMatches), $allMatches2 as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{fn:max(($allMatches1/@stokenNum, $allMatches2/@stokenNum))}"> {$allMatches1/fts:match,$allMatches2/fts:match} </fts:allMatches> };
The ApplyFTOr
function creates a new
AllMatches in which Matches are the union of those
found in the input AllMatches. Each Match represents
one possible result of the corresponding FTSelection. Thus,
a Match from either of the AllMatches is a
result.
For example, consider the FTSelection "Mustang"
ftor "Honda"
. The AllMatches corresponding to
"Mustang" and "Honda" are given below.
The AllMatches produced by ApplyFTOr
is
given below.
The parameters of the ApplyFTAnd
function are the
two AllMatches corresponding to the results of the two
nested FTSelections. The semantics is given below.
declare function fts:ApplyFTAnd ( $allMatches1 as element(fts:allMatches), $allMatches2 as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{fn:max(($allMatches1/@stokenNum, $allMatches2/@stokenNum))}" > { for $sm1 in $allMatches1/fts:match for $sm2 in $allMatches2/fts:match return <fts:match> {$sm1/*, $sm2/*} </fts:match> } </fts:allMatches> };
The result of the conjunction is a new AllMatches that contains the "Cartesian product" of the matches of the participating FTSelections. Every resulting Match is formed by the combination of the StringInclude components and StringExclude from the AllMatches of the nested FTSelection . Thus every match contains the positions to satisfy a Match from both original FTSelections and excludes the positions that violate the same Matches.
For example, consider the FTSelection "Mustang"
ftand "rust"
. The source AllMatches are give
below.
The AllMatches produced by ApplyFTAnd
is
given below.
The ApplyFTUnaryNot
function has one
AllMatches parameter corresponding to the result of the
nested FTSelection to be negated. The semantics is given
below.
declare function fts:InvertStringMatch ( $strm as element(*,fts:stringMatch) ) as element(*,fts:stringMatch) { if ($strm instance of element(fts:stringExclude)) then <fts:stringInclude queryPos="{$strm/@queryPos}" isContiguous="{$strm/@isContiguous}"> {$strm/fts:tokenInfo} </fts:stringInclude> else <fts:stringExclude queryPos="{$strm/@queryPos}" isContiguous="{$strm/@isContiguous}"> {$strm/fts:tokenInfo} </fts:stringExclude> }; declare function fts:UnaryNotHelper ( $matches as element(fts:match)* ) as element(fts:match)* { if (fn:empty($matches)) then <fts:match/> else for $sm in $matches[1]/* for $rest in fts:UnaryNotHelper( fn:subsequence($matches, 2) ) return <fts:match> { fts:InvertStringMatch($sm), $rest/* } </fts:match> }; declare function fts:ApplyFTUnaryNot ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { fts:UnaryNotHelper($allMatches/fts:match) } </fts:allMatches> };
The generation of the resulting AllMatches of an FTUnaryNot resembles the transformation of a negation of prepositional formula in DNF back to DNF. The negation of AllMatches requires the inversion of all the StringMatches within the AllMatches.
In the InvertStringMatch
function above, this
inversion occurs as follows.
The function fts:invertStringMatch
inverts a
StringInclude into a StringExclude and vice
versa.
The function fts:UnaryNotHelper
transforms the
source Matches into the resulting Matches by forming
the combinations of the inversions of a StringInclude or
StringExclude component over the source Matches into
new Matches.
For example, consider the FTSelection ftnot
("Mustang" ftor "Honda")
. The source AllMatches is
given below:
The FTUnaryNot transforms the StringIncludes to StringExcludes as illustrated below.
The parameters of the ApplyFTMildNot
function are
the two AllMatches parameters corresponding to the results
of the two nested FTSelections. The semantics is given
below.
declare function fts:CoveredIncludePositions ( $match as element(fts:match) ) as xs:integer* { for $strInclude in $match/fts:stringInclude return $strInclude/fts:tokenInfo/@startPos to $strInclude/fts:tokenInfo/@endPos }; declare function fts:ApplyFTMildNot ( $allMatches1 as element(fts:allMatches), $allMatches2 as element(fts:allMatches) ) as element(fts:allMatches) { if (fn:count($allMatches1//fts:stringExclude) gt 0) then fn:error(fn:QName('http://www.w3.org/2005/xqt-errors', 'FTDY0017'), "Invalid expression on the left-hand side of a not-in") else if (fn:count($allMatches2//fts:stringExclude) gt 0) then fn:error(fn:QName('http://www.w3.org/2005/xqt-errors', 'FTDY0017'), "Invalid expression on the right-hand side of a not-in") else if (fn:count($allMatches2//fts:stringInclude) eq 0) then $allMatches1 else <fts:allMatches stokenNum="{$allMatches1/@stokenNum}"> { $allMatches1/fts:match[ every $matches2 in $allMatches2/fts:match satisfies let $posSet1 := fts:CoveredIncludePositions(.) let $posSet2 := fts:CoveredIncludePositions($matches2) return some $pos in $posSet1 satisfies fn:not($pos = $posSet2) ] } </fts:allMatches> };
The resulting AllMatches contains Matches of the first operand that do not mention in their StringInclude components positions in a StringInclude component in the AllMatches of the second operand.
For example, consider the FTSelection ("Ford" not
in "Ford Mustang")
. The source AllMatches for the
left-hand side argument is given below.
The source AllMatches for the right-hand side argument is given below.
The FTMildNot will transform these to an empty AllMatches because both position 1 and position 27 from the first AllMatches contain only TokenInfos from StringInclude components of the second AllMatches.
The ApplyFTOrder
function has one AllMatches
parameter corresponding to the result of the nested
FTSelections. The semantics is given below.
declare function fts:ApplyFTOrder ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match where every $stringInclude1 in $match/fts:stringInclude, $stringInclude2 in $match/fts:stringInclude satisfies (($stringInclude1/fts:tokenInfo/@startPos <= $stringInclude2/fts:tokenInfo/@startPos) and ($stringInclude1/@queryPos <= $stringInclude2/@queryPos)) or (($stringInclude1/fts:tokenInfo/@startPos>= $stringInclude2/fts:tokenInfo/@startPos) and ($stringInclude1/@queryPos >= $stringInclude2/@queryPos)) return <fts:match> { $match/fts:stringInclude, for $stringExcl in $match/fts:stringExclude where every $stringIncl in $match/fts:stringInclude satisfies (($stringExcl/fts:tokenInfo/@startPos <= $stringIncl/fts:tokenInfo/@startPos) and ($stringExcl/@queryPos <= $stringIncl/@queryPos)) or (($stringExcl/fts:tokenInfo/@startPos >= $stringIncl/fts:tokenInfo/@startPos) and ($stringExcl/@queryPos >= $stringIncl/@queryPos)) return $stringExcl } </fts:match> } </fts:allMatches> };
The resulting AllMatches contains the Matches for which the starting positions in the StringInclude elements are in the order of the query positions of their query strings. StringExcludes that preserve the order (with respect to their starting positions) are also retained.
For example, consider the FTSelection ("great"
ftand "condition") ordered
. The source AllMatches is
given below.
The AllMatches for FTOrder are given below.
The parameters of the ApplyFTScope
function are 1)
the type of the scope (same or different), 2) the linguistic unit
(sentence or paragraph), and 2) one AllMatches parameter
corresponding to the result of the nested FTSelections. The
function definitions depend on the type of the scope (paragraph,
sentence) and the scope predicate (same, different).
The semantics of same sentence
is given below.
declare function fts:ApplyFTScopeSameSentence ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match where every $stringInclude1 in $match/fts:stringInclude, $stringInclude2 in $match/fts:stringInclude satisfies $stringInclude1/fts:tokenInfo/@startSent = $stringInclude2/fts:tokenInfo/@startSent and $stringInclude1/fts:tokenInfo/@startSent = $stringInclude1/fts:tokenInfo/@endSent and $stringInclude2/fts:tokenInfo/@startSent = $stringInclude2/fts:tokenInfo/@endSent and $stringInclude1/fts:tokenInfo/@startSent > 0 and $stringInclude2/fts:tokenInfo/@startSent > 0 return <fts:match> { $match/fts:stringInclude, for $stringExcl in $match/fts:stringExclude where $stringExcl/fts:tokenInfo/@startSent = 0 or ($stringExcl/fts:tokenInfo/@startSent = $stringExcl/fts:tokenInfo/@endSent and (every $stringIncl in $match/fts:stringInclude satisfies $stringIncl/fts:tokenInfo/@startSent = $stringExcl/fts:tokenInfo/@startSent) ) return $stringExcl } </fts:match> } </fts:allMatches> };
An AllMatches returned by the scope same
sentence
contains those Matches whose
StringIncludes span only a single sentence and all span the
same sentence. In these Matches only those
StringExcludes are retained that also only span a single
sentence, which is, in case there are StringIncludes in that
Match, the same as the one spanned by the
StringIncludes.
The semantics of different sentence
is given
below.
declare function fts:ApplyFTScopeDifferentSentence ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match where count($match/fts:stringInclude) > 1 and ( every $stringInclude1 in $match/fts:stringInclude, $stringInclude2 in $match/fts:stringInclude satisfies $stringInclude1 is $stringInclude2 or ( ( $stringInclude1/fts:tokenInfo/@startSent != $stringInclude2/fts:tokenInfo/@startSent or $stringInclude1/fts:tokenInfo/@startSent != $stringInclude1/fts:tokenInfo/@endSent or $stringInclude2/fts:tokenInfo/@startSent != $stringInclude2/fts:tokenInfo/@endSent ) and $stringInclude1/fts:tokenInfo/@startSent > 0 and $stringInclude2/fts:tokenInfo/@endSent > 0 ) ) return <fts:match> { $match/fts:stringInclude, for $stringExcl in $match/fts:stringExclude where every $stringIncl in $match/fts:stringInclude satisfies ($stringIncl/fts:tokenInfo/@startSent != $stringExcl/fts:tokenInfo/@startSent or $stringIncl/fts:tokenInfo/@startSent != $stringIncl/fts:tokenInfo/@endSent or $stringExcl/fts:tokenInfo/@startSent != $stringExcl/fts:tokenInfo/@endSent ) and $stringIncl/fts:tokenInfo/@startSent > 0 and $stringExcl/fts:tokenInfo/@endSent > 0 return $stringExcl } </fts:match> } </fts:allMatches> };
An AllMatches returned by the scope different
sentence
contains those Matches that have at least
two StringIncludes, no two of which begin and end all in the same
sentence. In these Matches only those StringExcludes
are retained that do not conflict with any of the
StringIncludes.
The semantics of same paragraph
is analogous to
same sentence
and is given below.
declare function fts:ApplyFTScopeSameParagraph ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match where every $stringInclude1 in $match/fts:stringInclude, $stringInclude2 in $match/fts:stringInclude satisfies $stringInclude1/fts:tokenInfo/@startPara = $stringInclude2/fts:tokenInfo/@startPara and $stringInclude1/fts:tokenInfo/@startPara = $stringInclude1/fts:tokenInfo/@endPara and $stringInclude2/fts:tokenInfo/@startPara = $stringInclude2/fts:tokenInfo/@endPara and $stringInclude1/fts:tokenInfo/@startPara > 0 and $stringInclude2/fts:tokenInfo/@endPara > 0 return <fts:match> { $match/fts:stringInclude, for $stringExcl in $match/fts:stringExclude where $stringExcl/fts:tokenInfo/@startPara = 0 or ($stringExcl/fts:tokenInfo/@startPara = $stringExcl/fts:tokenInfo/@endPara and (every $stringIncl in $match/fts:stringInclude satisfies $stringIncl/fts:tokenInfo/@startPara = $stringExcl/fts:tokenInfo/@startPara) ) return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics of different paragraph
is analogous
to different sentence
and is given below.
declare function fts:ApplyFTScopeDifferentParagraph ( $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match where count($match/fts:stringInclude) > 1 and ( every $stringInclude1 in $match/fts:stringInclude, $stringInclude2 in $match/fts:stringInclude satisfies $stringInclude1 is $stringInclude2 or ( ( $stringInclude1/fts:tokenInfo/@startPara != $stringInclude2/fts:tokenInfo/@startPara or $stringInclude1/fts:tokenInfo/@startPara != $stringInclude1/fts:tokenInfo/@endPara or $stringInclude2/fts:tokenInfo/@startPara != $stringInclude2/fts:tokenInfo/@endPara ) and $stringInclude1/fts:tokenInfo/@startPara > 0 and $stringInclude2/fts:tokenInfo/@endPara > 0 ) ) return <fts:match> { $match/fts:stringInclude, for $stringExcl in $match/fts:stringExclude where every $stringIncl in $match/fts:stringInclude satisfies ($stringIncl/fts:tokenInfo/@startPara != $stringExcl/fts:tokenInfo/@startPara or $stringIncl/fts:tokenInfo/@startPara != $stringIncl/fts:tokenInfo/@endPara or $stringExcl/fts:tokenInfo/@startPara != $stringExcl/fts:tokenInfo/@endPara ) and $stringIncl/fts:tokenInfo/@startPara > 0 and $stringExcl/fts:tokenInfo/@endPara > 0 return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics for the general case is given below.
declare function fts:ApplyFTScope ( $type as fts:scopeType, $selector as fts:scopeSelector, $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { if ($type eq "same" and $selector eq "sentence") then fts:ApplyFTScopeSameSentence($allMatches) else if ($type eq "different" and $selector eq "sentence") then fts:ApplyFTScopeDifferentSentence($allMatches) else if ($type eq "same" and $selector eq "paragraph") then fts:ApplyFTScopeSameParagraph($allMatches) else fts:ApplyFTScopeDifferentParagraph($allMatches) };
For example, consider the FTSelection ("Mustang"
ftand "Honda") same paragraph
. The source AllMatches
is given below.
The FTScope returns an empty AllMatches because neither Match contains TokenInfos from a single sentence.
The parameters of the ApplyFTContent
function are
1) the search context, 2) the type of the content match (at
start
, at end
, or entire content
),
and 3) one AllMatches parameter corresponding to the result
of the nested FTSelections.
The evaluation of ApplyFTContent
depends on the
type of the content match:
entire content
retains those Matches such
that for every token position in the search context, some
StringInclude in the Match covers that token
position.
at start
retains those Matches that contain
a StringInclude that covers the lowest token position in the
search context.
at end
retains those Matches that contain a
StringInclude that covers the highest token position in the
search context.
The semantics is given below.
declare function fts:ApplyFTContent ( $searchContext as item(), $type as fts:contentMatchType, $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { $allMatches/fts:match[ let $start_pos := fts:getLowestTokenPosition($searchContext), $end_pos := fts:getHighestTokenPosition($searchContext), $match := . return if ($type eq "entire content") then every $pos in $start_pos to $end_pos satisfies some $si in $match/fts:stringInclude[data(@isContiguous)] satisfies fts:TokenInfoCoversTokenPosition($si/fts:tokenInfo, $pos) else let $pos := if ($type eq "at start") then $start_pos else (: $type eq "at end" :) $end_pos return some $ti in $match/fts:stringInclude/fts:tokenInfo satisfies fts:TokenInfoCoversTokenPosition($ti, $pos) ] } </fts:allMatches> };
ApplyFTContent
depends on the helper function
fts:TokenInfoCoversTokenPosition
, which ascertains
whether the given $tokenInfo
covers a particular
$tokenPosition
.
declare function fts:TokenInfoCoversTokenPosition( $tokenInfo as element(fts:tokenInfo), $tokenPosition as xs:integer ) as xs:boolean { ($tokenPosition >= $tokenInfo/@startPos) and ($tokenPosition <= $tokenInfo/@endPos) };
ApplyFTContent
also depends on two functions whose
definitions are implementation-dependent:
getLowestTokenPosition
and
getHighestTokenPosition
return (respectively) the
first and last token positions of the item
$searchContext
.
declare function fts:getLowestTokenPosition( $searchContext as item() ) as xs:integer external; declare function fts:getHighestTokenPosition( $searchContext as item() ) as xs:integer external;
Note that the way @isContiguous
is calculated in
joinIncludes
and used in ApplyFTContent
can lead to counter-intuitive results. For example, consider the
following query:
"one two three four" contains text ("one" ftand "three" window 3 words) ftand ("two" ftand "four" window 3 words) entire content
Even though the four query tokens do cover all of the
search context's token positions, the query yields false, because
the Match that ApplyFTContent
receives as input
has two StringIncludes, each of which is non-contiguous.
Before we define the semantics functions of the FTWindow
and FTDistance operations, we introduce the auxiliary
function joinIncludes
that will be used in their
definitions. joinIncludes
takes a sequence of
StringIncludes of a Match and transforms it into
either the empty sequence, in case the input sequence was empty, or
otherwise a single StringInclude representing the span from
the first position of the match to the last. For the purpose of
being able to evaluate an "entire content" operator further up in
the tree, we pre-evaluate whether all possible positions between
first and last are covered in the input StringIncludes and
store that boolean in the attribute "isContiguous".
declare function fts:joinIncludes( $strIncls as element(fts:stringInclude)* ) as element(fts:stringInclude)? { if (fn:empty($strIncls)) then $strIncls else let $posSet := fts:CoveredIncludePositions(<fts:match>$strIncls</fts:match>), $minPos := fn:min($strIncls/fts:tokenInfo/@startPos), $maxPos := fn:max($strIncls/fts:tokenInfo/@endPos), $isContiguous := ( every $pos in $minPos to $maxPos satisfies ($pos = $posSet) ) and ( every $strIncl in $strIncls satisfies $strIncl/@isContiguous ) return <fts:stringInclude queryPos="{$strIncls[1]/@queryPos}" isContiguous="{$isContiguous}"> <fts:tokenInfo startPos ="{$minPos}" endPos ="{$maxPos}" startSent="{fn:min($strIncls/fts:tokenInfo/@startSent)}" endSent ="{fn:max($strIncls/fts:tokenInfo/@endSent)}" startPara="{fn:min($strIncls/fts:tokenInfo/@startPara)}" endPara ="{fn:max($strIncls/fts:tokenInfo/@endPara)}"/> </fts:stringInclude> };
The parameters of the ApplyFTWindow
function are 1)
the unit of type fts:distanceType
, 2) a size, and 3)
one AllMatches parameter corresponding to the result of the
nested FTSelections. For each unit type a function is
defined as follows.
The semantics of window N words
is given below.
declare function fts:ApplyFTWordWindow ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $minpos := fn:min($match/fts:stringInclude/fts:tokenInfo/@startPos), $maxpos := fn:max($match/fts:stringInclude/fts:tokenInfo/@endPos) for $windowStartPos in ($maxpos - $n + 1 to $minpos) let $windowEndPos := $windowStartPos + $n - 1 return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExclude in $match/fts:stringExclude where $stringExclude/fts:tokenInfo/@startPos >= $windowStartPos and $stringExclude/fts:tokenInfo/@endPos <= $windowEndPos return $stringExclude } </fts:match> } </fts:allMatches> };
The semantics of window N sentences
is given
below.
declare function fts:ApplyFTSentenceWindow ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $minpos := fn:min($match/fts:stringInclude/fts:tokenInfo/@startSent), $maxpos := fn:max($match/fts:stringInclude/fts:tokenInfo/@endSent) for $windowStartPos in ($maxpos - $n + 1 to $minpos) let $windowEndPos := $windowStartPos + $n - 1 return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExclude in $match/fts:stringExclude where $stringExclude/fts:tokenInfo/@startSent >= $windowStartPos and $stringExclude/fts:tokenInfo/@endSent <= $windowEndPos return $stringExclude } </fts:match> } </fts:allMatches> };
The semantics of window N paragraphs
is given
below.
declare function fts:ApplyFTParagraphWindow ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $minpos := fn:min($match/fts:stringInclude/fts:tokenInfo/@startPara), $maxpos := fn:max($match/fts:stringInclude/fts:tokenInfo/@endPara) for $windowStartPos in ($maxpos - $n + 1 to $minpos) let $windowEndPos := $windowStartPos + $n - 1 return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExclude in $match/fts:stringExclude where $stringExclude/fts:tokenInfo/@startPara >= $windowStartPos and $stringExclude/fts:tokenInfo/@endPara <= $windowEndPos return $stringExclude } </fts:match> } </fts:allMatches> };
The resulting AllMatches contains Matches of the operand that satisfy the condition that there exists a sequence of the specified number of consecutive (token, sentence, or paragraph) positions, such that all StringIncludes are within that window, and the StringExcludes retained are also within that window. For each Match that satisfies the window condition the StringIncludes are joined into a single StringInclude. This enables further window or distance operations to be applied to the result in a way that that result is taken as a single entity.
The semantics for the general function is given below.
declare function fts:ApplyFTWindow ( $type as fts:distanceType, $size as xs:integer, $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { if ($type eq "word") then fts:ApplyFTWordWindow($allMatches, $size) else if ($type eq "sentence") then fts:ApplyFTSentenceWindow($allMatches, $size) else fts:ApplyFTParagraphWindow($allMatches, $size) };
For example, consider the FTWindow selection ("Ford
Mustang" ftand "excellent") window 10 words
. The
Matches of the source AllMatches for ("Ford
Mustang" ftand "excellent")
are given below.
The result for the FTWindow selection consists of only the first, the fifth, and the sixth Matches because their respective window sizes are 5, 4, and 9.
The parameters of the ApplyFTDistance
function are
1) one AllMatches parameter corresponding to the result of
the nested FTSelections, 2) the unit of the distance
(tokens, sentences, paragraphs), and 3) the range specified. The
resulting AllMatches contains Matches of the operand
that satisfy the condition that the distance for every pair of
consecutive StringIncludes is within the specified interval,
where the distance is measured (in tokens, sentences, or
paragraphs) from the end of the preceding StringInclude to
the start of the next.
An invocation of the ApplyFTDistance
function will
call one of twelve helper functions, each of which handles a
particular unit of distance and type of range.
declare function fts:ApplyFTDistance ( $type as fts:distanceType, $range as element(fts:range), $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { if ($type eq "word") then if ($range/@type eq "exactly") then fts:ApplyFTWordDistanceExactly($allMatches, $range/@n) else if ($range/@type eq "at least") then fts:ApplyFTWordDistanceAtLeast($allMatches, $range/@n) else if ($range/@type eq "at most") then fts:ApplyFTWordDistanceAtMost( $allMatches, $range/@n) else fts:ApplyFTWordDistanceFromTo( $allMatches, $range/@m, $range/@n) else if ($type eq "sentence") then if ($range/@type eq "exactly") then fts:ApplyFTSentenceDistanceExactly($allMatches, $range/@n) else if ($range/@type eq "at least") then fts:ApplyFTSentenceDistanceAtLeast($allMatches, $range/@n) else if ($range/@type eq "at most") then fts:ApplyFTSentenceDistanceAtMost( $allMatches, $range/@n) else fts:ApplyFTSentenceDistanceFromTo( $allMatches, $range/@m, $range/@n) else if ($range/@type eq "exactly") then fts:ApplyFTParagraphDistanceExactly($allMatches, $range/@n) else if ($range/@type eq "at least") then fts:ApplyFTParagraphDistanceAtLeast($allMatches, $range/@n) else if ($range/@type eq "at most") then fts:ApplyFTParagraphDistanceAtMost( $allMatches, $range/@n) else fts:ApplyFTParagraphDistanceFromTo( $allMatches, $range/@m, $range/@n) };
Word Distance
The semantics of case word distance exactly N
is
given below.
declare function fts:ApplyFTWordDistanceExactly( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPos ascending, $si/fts:tokenInfo/@endPos ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $idx in 1 to fn:count($sorted) - 1 satisfies fts:wordDistance( $sorted[$idx]/fts:tokenInfo, $sorted[$idx+1]/fts:tokenInfo ) = $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:wordDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) = $n return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics of word distance at least N
is given
below.
declare function fts:ApplyFTWordDistanceAtLeast ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPos ascending, $si/fts:tokenInfo/@endPos ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:wordDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:wordDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $n return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics of word distance at most N
is given
below.
declare function fts:ApplyFTWordDistanceAtMost ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPos ascending, $si/fts:tokenInfo/@endPos ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:wordDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:wordDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics of word distance from M to N
is given
below.
declare function fts:ApplyFTWordDistanceFromTo ( $allMatches as element(fts:allMatches), $m as xs:integer, $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPos ascending, $si/fts:tokenInfo/@endPos ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:wordDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $m and fts:wordDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:wordDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $m and fts:wordDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> };
The preceding four helper functions all rely on
fts:wordDistance
, which returns the number of token
positions that occur between two TokenInfos. For example,
two tokens with consecutive positions have a distance of 0 tokens,
and two overlapping tokens have a distance of -1 tokens.
declare function fts:wordDistance ( $tokenInfo1 as element(fts:tokenInfo), $tokenInfo2 as element(fts:tokenInfo) ) as xs:integer { (: Ensure tokens are in order :) let $sorted := for $ti in ($tokenInfo1, $tokenInfo2) order by $ti/@startPos ascending, $ti/@endPos ascending return $ti return (: -1 because we count starting at 0 :) $sorted[2]/@startPos - $sorted[1]/@endPos - 1 };
Sentence Distance
The semantics of sentence distance exactly N
is
given below.
declare function fts:ApplyFTSentenceDistanceExactly ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startSent ascending, $si/fts:tokenInfo/@endSent ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:sentenceDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) = $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:sentenceDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) = $n return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics of sentence distance at least N
is
given below.
declare function fts:ApplyFTSentenceDistanceAtLeast ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startSent ascending, $si/fts:tokenInfo/@endSent ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:sentenceDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:sentenceDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $n return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics of sentence distance at most N
is
given below.
declare function fts:ApplyFTSentenceDistanceAtMost ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startSent ascending, $si/fts:tokenInfo/@endSent ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:sentenceDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:sentenceDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics of sentence distance from M to N
is
given below.
declare function fts:ApplyFTSentenceDistanceFromTo ( $allMatches as element(fts:allMatches), $m as xs:integer, $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startSent ascending, $si/fts:tokenInfo/@endSent ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:sentenceDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $m and fts:sentenceDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:sentenceDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $m and fts:sentenceDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> };
The preceding four helper functions all rely on
fts:sentenceDistance
, which returns the number of
sentences between two TokenInfos.
declare function fts:sentenceDistance ( $tokenInfo1 as element(fts:tokenInfo), $tokenInfo2 as element(fts:tokenInfo) ) as xs:integer { (: Ensure tokens are in order :) let $sorted := for $ti in ($tokenInfo1, $tokenInfo2) order by $ti/@startPos ascending, $ti/@endPos ascending return $ti return (: -1 because we count starting at 0 :) $sorted[2]/@startSent - $sorted[1]/@endSent - 1 };
Paragraph Distance
The semantics of paragraph distance exactly N
is
given below.
declare function fts:ApplyFTParagraphDistanceExactly ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPara ascending, $si/fts:tokenInfo/@endPara ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:paraDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) = $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:paraDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) = $n return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics of paragraph distance at least N
is
given below.
declare function fts:ApplyFTParagraphDistanceAtLeast ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPara ascending, $si/fts:tokenInfo/@endPara ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:paraDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:paraDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $n return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics of paragraph distance at most N
is
given below.
declare function fts:ApplyFTParagraphDistanceAtMost ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPara ascending, $si/fts:tokenInfo/@endPara ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:paraDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:paraDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> };
The semantics of paragraph distance from M to N
is
given below.
declare function fts:ApplyFTParagraphDistanceFromTo ( $allMatches as element(fts:allMatches), $m as xs:integer, $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> { for $match in $allMatches/fts:match let $sorted := for $si in $match/fts:stringInclude order by $si/fts:tokenInfo/@startPara ascending, $si/fts:tokenInfo/@endPara ascending return $si where if (fn:count($sorted) le 1) then fn:true() else every $index in (1 to fn:count($sorted) - 1) satisfies fts:paraDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) >= $m and fts:paraDistance( $sorted[$index]/fts:tokenInfo, $sorted[$index+1]/fts:tokenInfo ) <= $n return <fts:match> { fts:joinIncludes($match/fts:stringInclude), for $stringExcl in $match/fts:stringExclude where some $stringIncl in $match/fts:stringInclude satisfies fts:paraDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) >= $m and fts:paraDistance( $stringIncl/fts:tokenInfo, $stringExcl/fts:tokenInfo ) <= $n return $stringExcl } </fts:match> } </fts:allMatches> };
The preceding four helper functions all rely on
fts:paraDistance
, which returns the number of
paragraphs between two TokenInfos.
declare function fts:paraDistance ( $tokenInfo1 as element(fts:tokenInfo), $tokenInfo2 as element(fts:tokenInfo) ) as xs:integer { (: Ensure tokens are in order :) let $sorted := for $ti in ($tokenInfo1, $tokenInfo2) order by $ti/@startPos ascending, $ti/@endPos ascending return $ti return (: -1 because we count starting at 0 :) $sorted[2]/@startPara - $sorted[1]/@endPara - 1 };
For example, consider the FTDistance selection
("Ford Mustang" ftand "excellent") distance at most 3
words
. The Matches of the source AllMatches
for ("Ford Mustang" ftand "excellent")
are given
below.
The result for the FTDistance selection consists of only
the first Match (with positions 1, 2, and 5) and the fifth
Match (with positions 25, 27, and 28), because only for
these Matches the word distance between consecutive
TokenInfos is always less than or equal to 3. For the first
Match, the word distance between the two TokenInfos
is 2 (startPos
5 - endPos
2 - 1), and for
the fifth Match, it's 1 (startPos
27 -
endPos
25 - 1).
The parameters of the ApplyFTTimes
function are 1)
an FTRange specification, and 2) a parameter corresponding
to the result of the nested FTWords.
The function definitions depend on the range specification FTRange to limit the number of occurrences.
The general semantics is given below.
declare function fts:FormCombinations ( $sms as element(fts:match)*, $k as xs:integer ) as element(fts:match)* (: Find all combinations of exactly $k elements from $sms, and for each such combination, construct a match whose children are copies of all the children of all the elements in the combination. Return the sequence of all such matches. :) { if ($k eq 0) then <fts:match/> else if (fn:count($sms) lt $k) then () else if (fn:count($sms) eq $k) then <fts:match>{$sms/*}</fts:match> else let $first := $sms[1], $rest := fn:subsequence($sms, 2) return ( (: all the combinations that don't involve $first :) fts:FormCombinations($rest, $k), (: and all the combinations that do involve $first :) for $combination in fts:FormCombinations($rest, $k - 1) return <fts:match> { $first/*, $combination/* } </fts:match> ) }; declare function fts:FormCombinationsAtLeast ( $sms as element(fts:match)*, $times as xs:integer) as element(fts:match)* (: Find all combinations of $times or more elements from $sms, and for each such combination, construct a match whose children are copies of all the children of all the elements in the combination. Return the sequence of all such matches. :) { for $k in $times to fn:count($sms) return fts:FormCombinations($sms, $k) }; declare function fts:FormRange ( $sms as element(fts:match)*, $l as xs:integer, $u as xs:integer, $stokenNum as xs:integer ) as element(fts:allMatches) { if ($l > $u) then <fts:allMatches stokenNum="0" /> else let $am1 := <fts:allMatches stokenNum="{$stokenNum}"> {fts:FormCombinationsAtLeast($sms, $l)} </fts:allMatches> let $am2 := <fts:allMatches stokenNum="{$stokenNum}"> {fts:FormCombinationsAtLeast($sms, $u+1)} </fts:allMatches> return fts:ApplyFTAnd($am1, fts:ApplyFTUnaryNot($am2)) };
The semantics of occurs exactly N times
is given
below.
declare function fts:ApplyFTTimesExactly ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { fts:FormRange($allMatches/fts:match, $n, $n, $allMatches/@stokenNum) };
The semantics of occurs at least N times
is given
below.
declare function fts:ApplyFTTimesAtLeast ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { <fts:allMatches stokenNum="{$allMatches/@stokenNum}"> {fts:FormCombinationsAtLeast($allMatches/fts:match, $n)} </fts:allMatches> };
The semantics of occurs at most N times
is given
below.
declare function fts:ApplyFTTimesAtMost ( $allMatches as element(fts:allMatches), $n as xs:integer ) as element(fts:allMatches) { fts:FormRange($allMatches/fts:match, 0, $n, $allMatches/@stokenNum) };
The semantics of occurs from M to N times
is given
below.
declare function fts:ApplyFTTimesFromTo ( $allMatches as element(fts:allMatches), $m as xs:integer, $n as xs:integer ) as element(fts:allMatches) { fts:FormRange($allMatches/fts:match, $m, $n, $allMatches/@stokenNum) };
The way to ensure that there are at least N different
matches of an FTSelection is to ensure that at least
N of its Matches occur simultaneously. This is
similar to forming their conjunction by combining N or more
distinct Matches into one simple match. Therefore, the
AllMatches for the selection condition specifying the range
qualifier at least N
contains the possible
combinations of N or more simple matches of the operand.
This operation is performed in the function
fts:FormCombinationsAtLeast
.
The range [L, U] is represented by the condition at least
L and not at least U+1
. This transformation is performed in
the function fts:FormRange
.
The semantics for the general case is given below.
declare function fts:ApplyFTTimes ( $range as element(fts:range), $allMatches as element(fts:allMatches) ) as element(fts:allMatches) { if (fn:count($allMatches//fts:stringExclude) gt 0) then fn:error(fn:QName('http://www.w3.org/2005/xqt-errors', 'XPST0003')) else if ($range/@type eq "exactly") then fts:ApplyFTTimesExactly($allMatches, $range/@n) else if ($range/@type eq "at least") then fts:ApplyFTTimesAtLeast($allMatches, $range/@n) else if ($range/@type eq "at most") then fts:ApplyFTTimesAtMost($allMatches, $range/@n) else fts:ApplyFTTimesFromTo($allMatches, $range/@m, $range/@n) };
The above function performs a sanity check to ensure that the nested AllMatches is a result of the evaluation of FTWords as defined in the grammar rule for FTPrimary . Otherwise, an error [err:XPST0003]XP is raised.
For example, consider the FTTimes selection
"Mustang" occurs at least 2 times
. The source
AllMatches of the FTWords selection
"Mustang"
is given below.
The result consists of the pairs of the Matches.
Consider an FTContainsExpr expression of the form
SearchContext contains text FTSelection
, where
SearchContext
is an XQuery 1.0 expression that returns
a sequence of items. The FTContainsExpr returns true if and
only if one of those items satisfies the
FTSelection
.
If the FTContainsExpr is of the form SearchContext
contains text FTSelection without content IgnoreExpr
for
some XQuery 1.0 expression IgnoreExpr
, then any nodes
returned by IgnoreExpr
are (notionally) pruned from
each search context item before attempting to satisfy the
FTSelection
.
More formally, evaluation of an FTContainsExpr proceeds according to the following steps. Where appropriate, the explanation includes references to arcs labelled "FTn" in the processing model diagram (Figure 1) in 2.1 Processing Model.
For each XQuery/XPath expression nested within the FTContainsExpr, evaluate it with respect to the same dynamic context as the FTContainsExpr (FT1). Specifically:
Evaluate the search context expression
(SearchContext
), resulting in the sequence of search
context items.
Evaluate the ignore option (IgnoreExpr
) if any,
resulting in the set of ignored nodes.
At each FTWordsValue, evaluate the literal/expression and
convert the result to xs:string*
.
At each weight specification, evaluate the expression and
convert the result to xs:double
.
At each FTWindow and FTRange, evaluate the AdditiveExpr(s) and
convert each to xs:integer
.
Using the settings of the match option components in the
FTContainsExpr's static context, construct an
element(fts:matchOptions)
structure.
Based on the parse-tree of the FTContainsExpr's FTSelection and
the results of steps 1c-1e, construct an
element(*,fts:ftSelection)
structure. We refer to this
as the "operator tree" below. In this process:
Construct the operator tree from the top down, propagating FTMatchOptions down to FTWordsValues.
Tokenize the query string(s) obtained at 1c. (FT2.1)
Call the function fts:FTContainsExpr
(see
declaration below), passing the following arguments to its
parameters:
$searchContextItems
: The sequence of items returned
by SearchContext
, calculated in step 1a.
$ignoreNodes
: The sequence of items returned by
IgnoreExpr
(in 1b), if that expression is present, or
the empty sequence otherwise.
$ftSelection
: The XML node representation of
FTSelection
(constructed in step 2).
$defOptions
: The XML representation of the match
options in the FTContainsExpr's static context (constructed in step
3).
Within the function, for each search context item:
Delete the ignored nodes from the search context item.
[fts:FTContainsExpr
calls
fts:reconstruct
.]
Traverse the operator tree from the top down, propagating
FTMatchOptions down to FTWordsValues. [fts:evaluate
calls itself and fts:replaceMatchOptions
.]
At each FTWordsValue, using the prevailing FTMatchOptions:
Tokenize the search context obtained at 4a. (FT2.2) (Whether
this pays any attention to FTMatchOptions is up to the
implementation.) [This happens within
fts:matchTokenInfos
.]
Match the search context tokens and the query tokens, yielding
an element(fts:tokenInfo)*
structure. [This happens
within fts:matchTokenInfos
.]
Convert that into an element(fts:allMatches)
. (FT3)
[This happens in fts:applyQueryTokensAsPhrase
.]
Traverse the operator tree from the bottom up. At each point, the AllMatches instances produced by subtrees are taken as input, and a new AllMatches instance is obtained as output. (FT4) [This is most of the section 4 code.]
If the topmost AllMatches instance contains a
Match with no StringExcludes, then the search context
item satisfies the full-text condition given by the
FTSelection, and the call to fts:FTContainsExpr
returns true
. [This is handled by the QuantifiedExpr
in fts:FTContainsExpr
.]
[Note that the section 4 code doesn't implement 4b-4d as three sequential steps. Instead, they are different aspects of a single traversal of the operator tree.]
If none of the topmost AllMatches provides a successful
match, then fts:FTContainsExpr
returns
false
.
The boolean value returned by the call to
fts:FTContainsExpr
is the value of the FTContainsExpr.
(FT5)
declare function fts:FTContainsExpr ( $searchContextItems as item()*, $ignoreNodes as node()*, $ftSelection as element(*,fts:ftSelection), $defOptions as element(fts:matchOptions) ) as xs:boolean { some $searchContext in $searchContextItems satisfies let $newSearchContext := fts:reconstruct( $searchContext, $ignoreNodes ) return if (fn:empty($newSearchContext)) then fn:false() else let $allMatches := fts:evaluate($ftSelection, $newSearchContext, $defOptions, 0) return some $match in $allMatches/fts:match satisfies fn:count($match/fts:stringExclude) eq 0 }; declare function fts:reconstruct ( $n as item(), $ignore as node()* ) as item()? { typeswitch ($n) case node() return if (some $i in $ignore satisfies $n is $i) then () else if ($n instance of element()) then let $nodeName := fn:node-name($n) let $nodeContent := for $nn in $n/node() return fts:reconstruct($nn,$ignore) return element {$nodeName} {$nodeContent} else if ($n instance of document-node()) then document { for $nn in $n/node() return fts:reconstruct($nn, $ignore) } else $n default return $n };
This section addresses the semantics of scoring variables in
XQuery 1.0 for
and let
clauses and XPath
2.0 for
expressions.
Scoring variables associate a numeric score with the result of the evaluation of XQuery 1.0 and XPath 2.0 expressions. This numeric score tries to estimate the value of a result item to the user information need expressed using the XQuery 1.0 and XPath 2.0 expression. The numeric score is computed using an implementation-dependent scoring algorithm.
There are numerous scoring algorithms used in practice. Most of the scoring algorithms take as inputs a query and a set of results to the query. In computing the score, these algorithms rely on the structure of the query to estimate the relevance of the results.
In the context of defining the semantics of XQuery and XPath Full Text, passing the structure of the query poses a problem. The query may contain XQuery 1.0 and XPath 2.0 expressions and XQuery and XPath Full Text 1.0 expressions in particular. The semantics of XQuery 1.0 and XPath 2.0 expressions is defined using (among other things) functions that take as arguments sequences of items and return sequences of items. They are not aware of what expression produced a particular sequence, i.e., they are not aware of the expression structure.
To define the semantics of scoring in XQuery and XPath Full Text 1.0 using XQuery 1.0, expressions that produce the query result (or the functions that implement the expressions) must be passed as arguments. In other words, second-order functions are necessary. Currently XQuery 1.0 and XPath 2.0 do not provide such functions.
Nevertheless, in the interest of the exposition, assume that
such second-order functions are present. In particular, that there
are two semantic second-order function fts:score
and
fts:scoreSequence
that take one argument (an
expression) and return the score value of this expression,
respectively a sequence of score values, one for each item to which
the expression evaluates. The scores must satisfy scoring properties.
A for
clause containing a score variable
for $result score $score in Expr ...
is evaluated as though it is replaced by the following the set of clauses.
let $scoreSeq := fts:scoreSequence(Expr) for $result at $i in Expr let $score := $scoreSeq[$i] ...
Here, $scoreSeq
and $i
are new
variables, not appearing elsewhere, and
fts:scoreSequence
is the second-order function.
Similarly, a let
clause containing a score
variable
let score $score := Expr ...
is evaluated as though it is replaced by the following clause.
let $score := fts:score(Expr) ...
This section presents a more complex example for the evaluation
of FTContainsExpr. This example uses the same sample
document fragment and assigns it $doc
. Consider the
following FTContainsExpr.
$doc contains text ( ( "Mustang" ftand ({("great", "excellent")} any word occurs at least 2 times) window 11 words ) ftand ftnot "rust" ) same paragraph
Begin by evaluating the FTSelection to AllMatches.
( ( "Mustang" ftand ({("great", "excellent")} any word occurs at least 2 times) window 11 words ) ftand ftnot "rust" ) same paragraph
Step 1: Evaluate the FTWords "Mustang"
.
Step 2: Evaluate the FTWords {"great", "excellent"}
any word
.
Step 2.1: Match the token "great"
Step 2.2 Match the token "excellent"
Step 2.3 - Combine the above AllMatches as if FTOr is used, i.e., by forming a union of the Matches.
Step 3 - Apply the FTTimes {("great", "excellent")}
any word occurs at least 2 times
forming two pairs of
Matches.
Step 4 - Apply the FTAnd "Mustang" ftand
({("great", "excellent")} any word occurs at least 2 times)
forming all possible pairs of StringMatches.
Step 5 - Apply the FTWindow ( "Mustang" ftand
({("great", "excellent")} any word occurs at least 2 times) window
11 words )
, filtering out Matches for which the
window is not less than or equal to 11 tokens.
Step 6 - Evaluate FTWords "rust"
.
Step 7 - Apply the FTUnaryNot ftnot "rust"
,
transforming the StringInclude
into a
StringExclude
.
Step 8 - Apply the FTAnd ( ( "Mustang" ftand
({("great", "excellent")} any word occurs at least 2 times) window
11 words ) ftand ftnot "rust" )
, forming all possible
combintations of three StringMatches from the first
AllMatches and one StringMatch from the second
AllMatches.
Step 9: Apply the FTScope, filtering out Matches
whose TokenInfos are not within the same paragraph (assuming
the <offer>
elements determine paragraph
boundaries).
The resulting AllMatches contains a Match that
does not contain a StringExclude. Therefore, the sample
FTContainsExpr returns true
.
This section defines the conformance criteria for a XQuery and XPath Full Text 1.0 processor.
In this section, the following terms are used to indicate the requirement levels defined in [RFC 2119]. [Definition: MUST means that the item is an absolute requirement of the specification.] [Definition: MAY means that an item is truly optional.] [Definition: SHOULD means that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.]
An XQuery and XPath Full Text 1.0 processor that claims to conform to this specification MUST include a claim of Minimal Conformance as defined in 5.1 Minimal Conformance. In addition to a claim of Minimal Conformance, it MAY claim conformance to one or more optional features defined in 5.2 Optional Features
Minimal Conformance to this specification MUST include all of the following items:
Minimal support for XQuery 1.0 [XQuery 1.0: An XML Query Language (Second Edition)] or XPath 2.0 [XML Path Language (XPath) 2.0 (Second Edition)]. The optional features of XQuery 1.0 [XQuery 1.0: An XML Query Language (Second Edition)] or XPath 2.0 [XML Path Language (XPath) 2.0 (Second Edition)] MAY be supported.
Support for everything specified in this document except those operators and match options specified in 5.2 Optional Features to be optional. If an implementation does not provide a given optional operator or match option, it MUST implement any requirements specified in 5.2 Optional Features for implementations that do not provide that operator or match option.
A definition of every item specified to be implementation-defined in I Checklist of Implementation-Defined Features.
Note:
Implementations are not required to define items specified to be implementation-dependent
It is optional whether the implementation supports the FTMildNot. If it does not support FTMildNot and encounters one in a full-text query, then it MUST raise an error [err:FTST0001].
The unrestricted form of negation in FTUnaryNot, that can negate every kind of FTSelection, is optional. Implementations may choose to support the negation operation in a restricted form, enforcing one or both of the following restrictions.
[Definition: Negation Restriction 1. An FTUnaryNot expression may only appear as a direct right operand of an "ftand" (FTAnd) operation.]
[Definition: Negation Restriction 2. An FTUnaryNot expression may not appear as a descendant of an FTOr that is modified by an FTPosFilter. (An FTOr is modified by an FTPosFilter, if it is derived using the production for FTSelection together with that FTPosFilter.)]
Consider the following example FTSelections.
1. ftnot "web" 2. "web" ftand ( ftnot "information" ftor "retrieval" ) 3. "web" ftand ftnot("information" ftand "retrieval") 4. "web" ftand ftnot("information" ftand "retrieval" window 5 words) 5. "web" ftand ("information" ftand ftnot "retrieval" window 5 words)
The first two FTSelections both violate restriction 1, while the
third and the fourth are conform with both restrictions. The fifth
one violates restriction 2, while obeying restriction 1. Note that
in the last example the FTSelection to which the window operation
is applied is "information" ftand ftnot "retrieval"
,
which contains an FTUnaryNot expression.
If the implementation does enforce one or both of these restrictions on FTUnaryNot and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0002].
Support for the "sentences" alternative of FTUnit and the "sentence" alternative of FTBigUnit is optional. Similarly, support for the "paragraphs" alternative of FTUnit and the "paragraph" alternative of FTBigUnit is optional. If an implementation does not support one or more choices of FTUnit or FTBigUnit and encounters an unsupported FTUnit or FTBigUnit in a full-text query, then it MUST raise an error [err:FTST0003].
The unrestricted form of the FTOrder postfix operator, that can be applied to any kind of FTSelection, is optional. Implementations may choose to enforce the following restriction on the use of FTOrder.
[Definition: Order Operator Restriction. FTOrder may only appear directly succeeding an FTWindow or an FTDistance operator.]
If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0010].
It is optional whether the implementation supports the FTScope operator. If it does not support FTScope and encounters one in a full-text query, then it MUST raise an error [err:FTST0004].
The unrestricted form of the FTWindow postfix operator, that can be applied to any kind of FTSelection, is optional. Implementations may choose to enforce the following restriction on the use of FTWindow.
[Definition: Window Operator Restriction. FTWindow can only be applied to an FTOr that is either a single FTWords or a combination of FTWords involving only the operators ftand and ftor.]
If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0011].
The unrestricted form of the FTDistance postfix operator, that can be applied to any kind of FTSelection, is optional. Implementations may choose to enforce the following restriction on the use of FTDistance.
[Definition: Distance Operator Restriction. FTDistance can only be applied to an FTOr that is either a single FTWords or a combination of FTWords involving only the operators ftand and ftor.]
If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0011].
It is optional whether the implementation supports the FTTimes operator. If it does not support FTTimes and encounters one in a full-text query, then it MUST raise an error [err:FTST0005].
It is optional whether the implementation supports the FTContent operator. If it does not support FTContent and encounters one in a full-text query, then it MUST raise an error [err:FTST0012].
It is optional whether the implementation supports the "lowercase" and "uppercase" choices for the FTCaseOption. If it does not support these choices for the FTCaseOption and encounters an unsupported choice in a full-text query, then it MUST raise an error [err:FTST0015].
It is optional whether the implementation supports the FTStopWordOption. If it does not support FTStopWordOption and encounters one in a full-text query, then it MUST raise an error [err:FTST0006].
It is optional whether the implementation supports the FTStopWordOption in the body of the query. If it supports FTStopWordOption in the prolog, but not in the body of a query, and encounters one in the body of a query it MUST raise an error [err:FTST0006].
It is optional whether the implementation supports the StringLiteral alternative of FTStopWords in the FTStopWordOption. If it does not support the StringLiteral alternative of FTStopWords and encounters such an alternative in a full-text query, then it MUST raise an error [err:FTST0006].
It is optional whether the implementation supports the unrestricted form of FTLanguageOption. Implementations may choose to enforce the following restriction on the use of FTLanguageOption.
[Definition: Single Language Restriction. If a full-text query contains more than one FTLanguageOption in its body and the prolog, then the languages specified must be the same.]
If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0013].
The implementation may constrain the set of ignored nodes. If the operand of FTIgnoreOption violates the implementation-defined restriction on that operand, it MUST raise an error [err:FTST0007].
The implementation may restrict the allowable expressions used to compute scores. The restrictions are implementation-defined.
If the implementation does enforce such restrictions and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0014].
An implementation may constrain the range of valid weights to non-negative values. If an implementation does enforce this restriction and encounters a full-text query that uses a negative weight, it MUST raise an error [err:FTDY0016].
This section defines the conformance criteria for an XQueryX processor that includes the Full Text capability.
In this section, the terms MUST, MAY, and SHOULD are used as defined in 5 Conformance.
An XQueryX processor that claims to conform to this specification MUST implement the XQueryX syntax as defined in E XML Syntax (XQueryX) for XQuery and XPath Full Text 1.0 and include a claim of Minimal Conformance as defined in 5.1 Minimal Conformance. In addition to a claim of Minimal Conformance, it MAY claim conformance to one or more optional features defined in 5.2 Optional Features.
The EBNF in this document and in this section is aligned with the current XML Query 1.0 grammar (see http://www.w3.org/TR/2010/REC-xquery-20101214/).
| [1] | Module |
::= | VersionDecl?
(LibraryModule |
MainModule) |
|
| [2] | VersionDecl |
::= | "xquery" "version" StringLiteral ("encoding"
StringLiteral)? Separator |
|
| [3] | MainModule |
::= | Prolog QueryBody |
|
| [4] | LibraryModule |
::= | ModuleDecl
Prolog |
|
| [5] | ModuleDecl |
::= | "module" "namespace" NCName "=" URILiteral Separator |
|
| [6] | Prolog |
::= | ((DefaultNamespaceDecl |
Setter | NamespaceDecl | Import | FTOptionDecl) Separator)* ((VarDecl | FunctionDecl | OptionDecl) Separator)* |
|
| [7] | Separator |
::= | ";" |
|
| [8] | Setter |
::= | BoundarySpaceDecl | DefaultCollationDecl |
BaseURIDecl | ConstructionDecl | OrderingModeDecl | EmptyOrderDecl | CopyNamespacesDecl |
|
| [9] | BoundarySpaceDecl |
::= | "declare" "boundary-space" ("preserve" |
"strip") |
|
| [10] | DefaultCollationDecl |
::= | "declare" "default" "collation" URILiteral |
|
| [11] | BaseURIDecl |
::= | "declare" "base-uri" URILiteral |
|
| [12] | ConstructionDecl |
::= | "declare" "construction" ("strip" |
"preserve") |
|
| [13] | OrderingModeDecl |
::= | "declare" "ordering" ("ordered" |
"unordered") |
|
| [14] | EmptyOrderDecl |
::= | "declare" "default" "order" "empty" ("greatest" |
"least") |
|
| [15] | CopyNamespacesDecl |
::= | "declare" "copy-namespaces" PreserveMode "," InheritMode |
|
| [16] | PreserveMode |
::= | "preserve" | "no-preserve" |
|
| [17] | InheritMode |
::= | "inherit" | "no-inherit" |
|
| [18] | Import |
::= | SchemaImport |
ModuleImport |
|
| [19] | SchemaImport |
::= | "import" "schema" SchemaPrefix? URILiteral ("at" URILiteral ("," URILiteral)*)? |
|
| [20] | SchemaPrefix |
::= | ("namespace" NCName
"=") | ("default" "element" "namespace") |
|
| [21] | ModuleImport |
::= | "import" "module" ("namespace" NCName "=")? URILiteral ("at" URILiteral ("," URILiteral)*)? |
|
| [22] | NamespaceDecl |
::= | "declare" "namespace" NCName "=" URILiteral |
|
| [23] | DefaultNamespaceDecl |
::= | "declare" "default" ("element" | "function") "namespace"
URILiteral |
|
| [24] | FTOptionDecl |
::= | "declare" "ft-option" FTMatchOptions |
|
| [25] | VarDecl |
::= | "declare" "variable" "$" QName TypeDeclaration? ((":="
ExprSingle) |
"external") |
|
| [26] | FunctionDecl |
::= | "declare" "function" QName "(" ParamList? ")" ("as" SequenceType)? (EnclosedExpr |
"external") |
/* xgc: reserved-function-namesXQ */ |
| [27] | ParamList |
::= | Param ("," Param)* |
|
| [28] | Param |
::= | "$" QName TypeDeclaration? |
|
| [29] | EnclosedExpr |
::= | "{" Expr
"}" |
|
| [30] | OptionDecl |
::= | "declare" "option" QName StringLiteral |
|
| [31] | QueryBody |
::= | Expr |
|
| [32] | Expr |
::= | ExprSingle (","
ExprSingle)* |
|
| [33] | ExprSingle |
::= | FLWORExpr |
|
| [34] | FLWORExpr |
::= | (ForClause |
LetClause)+ WhereClause? OrderByClause? "return" ExprSingle |
|
| [35] | ForClause |
::= | "for" "$" VarName
TypeDeclaration?
PositionalVar? FTScoreVar? "in" ExprSingle ("," "$" VarName TypeDeclaration? PositionalVar? FTScoreVar? "in" ExprSingle)* |
|
| [36] | PositionalVar |
::= | "at" "$" VarName |
|
| [37] | FTScoreVar |
::= | "score" "$" VarName |
|
| [38] | LetClause |
::= | "let" (("$" VarName
TypeDeclaration?) |
FTScoreVar) ":=" ExprSingle ("," (("$" VarName TypeDeclaration?) | FTScoreVar) ":=" ExprSingle)* |
|
| [39] | WhereClause |
::= | "where" ExprSingle |
|
| [40] | OrderByClause |
::= | (("order" "by") | ("stable" "order" "by")) OrderSpecList |
|
| [41] | OrderSpecList |
::= | OrderSpec (","
OrderSpec)* |
|
| [42] | OrderSpec |
::= | ExprSingle
OrderModifier |
|
| [43] | OrderModifier |
::= | ("ascending" | "descending")? ("empty" ("greatest" |
"least"))? ("collation" URILiteral)? |
|
| [44] | QuantifiedExpr |
::= | ("some" | "every") "$" VarName TypeDeclaration? "in" ExprSingle ("," "$" VarName TypeDeclaration? "in" ExprSingle)* "satisfies" ExprSingle |
|
| [45] | TypeswitchExpr |
::= | "typeswitch" "(" Expr
")" CaseClause+ "default"
("$" VarName)? "return"
ExprSingle |
|
| [46] | CaseClause |
::= | "case" ("$" VarName
"as")? SequenceType
"return" ExprSingle |
|
| [47] | IfExpr |
::= | "if" "(" Expr ")"
"then" ExprSingle "else"
ExprSingle |
|
| [48] | OrExpr |
::= | AndExpr ( "or"
AndExpr )* |
|
| [49] | AndExpr |
::= | ComparisonExpr ( "and" ComparisonExpr )* |
|
| [50] | ComparisonExpr |
::= | FTContainsExpr ( (ValueComp |
|
| [51] | FTContainsExpr |
::= | RangeExpr (
"contains" "text" FTSelection FTIgnoreOption? )? |
|
| [52] | RangeExpr |
::= | AdditiveExpr (
"to" AdditiveExpr
)? |
|
| [53] | AdditiveExpr |
::= | MultiplicativeExpr ( ("+" |
"-") MultiplicativeExpr
)* |
|
| [54] | MultiplicativeExpr |
::= | UnionExpr ( ("*" |
"div" | "idiv" | "mod") UnionExpr )* |
|
| [55] | UnionExpr |
::= | IntersectExceptExpr (
("union" | "|") IntersectExceptExpr
)* |
|
| [56] | IntersectExceptExpr |
::= | InstanceofExpr ( ("intersect" |
"except") InstanceofExpr )* |
|
| [57] | InstanceofExpr |
::= | TreatExpr (
"instance" "of" SequenceType )? |
|
| [58] | TreatExpr |
::= | CastableExpr (
"treat" "as" SequenceType
)? |
|
| [59] | CastableExpr |
::= | CastExpr (
"castable" "as" SingleType
)? |
|
| [60] | CastExpr |
::= | UnaryExpr ( "cast"
"as" SingleType
)? |
|
| [61] | UnaryExpr |
::= | ("-" | "+")* ValueExpr |
|
| [62] | ValueExpr |
::= | ValidateExpr |
PathExpr | ExtensionExpr |
|
| [63] | GeneralComp |
::= | "=" | "!=" | "<" | "<=" | ">" |
">=" |
|
| [64] | ValueComp |
::= | "eq" | "ne" | "lt" | "le" | "gt" | "ge" |
|
| [65] | NodeComp |
::= | "is" | "<<" | ">>" |
|
| [66] | ValidateExpr |
::= | "validate" ValidationMode? "{" Expr "}" |
|
| [67] | ValidationMode |
::= | "lax" | "strict" |
|
| [68] | ExtensionExpr |
::= | Pragma+ "{" Expr? "}" |
|
| [69] | Pragma |
::= | "(#" S? QName (S
PragmaContents)?
"#)" |
/* ws: explicitXQ */ |
| [70] | PragmaContents |
::= | (Char* - (Char* '#)'
Char*)) |
|
| [71] | PathExpr |
::= | ("/" RelativePathExpr?) |
/* xgc: leading-lone-slashXQ */ |
| [72] | RelativePathExpr |
::= | StepExpr (("/" |
"//") StepExpr)* |
|
| [73] | StepExpr |
::= | FilterExpr |
AxisStep |
|
| [74] | AxisStep |
::= | (ReverseStep |
ForwardStep) PredicateList |
|
| [75] | ForwardStep |
::= | (ForwardAxis
NodeTest) | AbbrevForwardStep |
|
| [76] | ForwardAxis |
::= | ("child" "::") |
|
| [77] | AbbrevForwardStep |
::= | "@"? NodeTest |
|
| [78] | ReverseStep |
::= | (ReverseAxis
NodeTest) | AbbrevReverseStep |
|
| [79] | ReverseAxis |
::= | ("parent" "::") |
|
| [80] | AbbrevReverseStep |
::= | ".." |
|
| [81] | NodeTest |
::= | KindTest | NameTest |
|
| [82] | NameTest |
::= | QName | Wildcard |
|
| [83] | Wildcard |
::= | "*" |
/* ws: explicitXQ */ |
| [84] | FilterExpr |
::= | PrimaryExpr
PredicateList |
|
| [85] | PredicateList |
::= | Predicate* |
|
| [86] | Predicate |
::= | "[" Expr
"]" |
|
| [87] | PrimaryExpr |
::= | Literal |
|
| [88] | Literal |
::= | NumericLiteral | StringLiteral |
|
| [89] | NumericLiteral |
::= | IntegerLiteral | DecimalLiteral | DoubleLiteral |
|
| [90] | VarRef |
::= | "$" VarName |
|
| [91] | VarName |
::= | QName |
|
| [92] | ParenthesizedExpr |
::= | "(" Expr?
")" |
|
| [93] | ContextItemExpr |
::= | "." |
|
| [94] | OrderedExpr |
::= | "ordered" "{" Expr
"}" |
|
| [95] | UnorderedExpr |
::= | "unordered" "{" Expr
"}" |
|
| [96] | FunctionCall |
::= | QName "(" (ExprSingle ("," ExprSingle)*)? ")" |
/* xgc: reserved-function-namesXQ */ |
| /* gn: parensXQ */ | ||||
| [97] | Constructor |
::= | DirectConstructor |
|
| [98] | DirectConstructor |
::= | DirElemConstructor |
|
| [99] | DirElemConstructor |
::= | "<" QName DirAttributeList ("/>" |
(">" DirElemContent*
"</" QName S? ">")) |
/* ws: explicitXQ */ |
| [100] | DirAttributeList |
::= | (S (QName S?
"=" S? DirAttributeValue)?)* |
/* ws: explicitXQ */ |
| [101] | DirAttributeValue |
::= | ('"' (EscapeQuot
| QuotAttrValueContent)*
'"') |
/* ws: explicitXQ */ |
| [102] | QuotAttrValueContent |
::= | QuotAttrContentChar |
|
| [103] | AposAttrValueContent |
::= | AposAttrContentChar |
|
| [104] | DirElemContent |
::= | DirectConstructor |
|
| [105] | CommonContent |
::= | PredefinedEntityRef |
CharRef | "{{" | "}}" |
EnclosedExpr |
|
| [106] | DirCommentConstructor |
::= | "<!--" DirCommentContents
"-->" |
/* ws: explicitXQ */ |
| [107] | DirCommentContents |
::= | ((Char - '-') | ('-'
(Char - '-')))* |
/* ws: explicitXQ */ |
| [108] | DirPIConstructor |
::= | "<?" PITarget
(S DirPIContents)?
"?>" |
/* ws: explicitXQ */ |
| [109] | DirPIContents |
::= | (Char* - (Char* '?>'
Char*)) |
/* ws: explicitXQ */ |
| [110] | CDataSection |
::= | "<![CDATA[" CDataSectionContents
"]]>" |
/* ws: explicitXQ */ |
| [111] | CDataSectionContents |
::= | (Char* - (Char*
']]>' Char*)) |
/* ws: explicitXQ */ |
| [112] | ComputedConstructor |
::= | CompDocConstructor |
|
| [113] | CompDocConstructor |
::= | "document" "{" Expr
"}" |
|
| [114] | CompElemConstructor |
::= | "element" (QName |
("{" Expr "}")) "{" ContentExpr? "}" |
|
| [115] | ContentExpr |
::= | Expr |
|
| [116] | CompAttrConstructor |
::= | "attribute" (QName |
("{" Expr "}")) "{" Expr? "}" |
|
| [117] | CompTextConstructor |
::= | "text" "{" Expr
"}" |
|
| [118] | CompCommentConstructor |
::= | "comment" "{" Expr
"}" |
|
| [119] | CompPIConstructor |
::= | "processing-instruction" (NCName | ("{" Expr "}")) "{" Expr? "}" |
|
| [120] | SingleType |
::= | AtomicType
"?"? |
|
| [121] | TypeDeclaration |
::= | "as" SequenceType |
|
| [122] | SequenceType |
::= | ("empty-sequence" "(" ")") |
|
| [123] | OccurrenceIndicator |
::= | "?" | "*" | "+" |
/* xgc: occurrence-indicatorsXQ */ |
| [124] | ItemType |
::= | KindTest | ("item"
"(" ")") | AtomicType |
|
| [125] | AtomicType |
::= | QName |
|
| [126] | KindTest |
::= | DocumentTest |
|
| [127] | AnyKindTest |
::= | "node" "(" ")" |
|
| [128] | DocumentTest |
::= | "document-node" "(" (ElementTest | SchemaElementTest)?
")" |
|
| [129] | TextTest |
::= | "text" "(" ")" |
|
| [130] | CommentTest |
::= | "comment" "(" ")" |
|
| [131] | PITest |
::= | "processing-instruction" "(" (NCName | StringLiteral)? ")" |
|
| [132] | AttributeTest |
::= | "attribute" "(" (AttribNameOrWildcard (","
TypeName)?)? ")" |
|
| [133] | AttribNameOrWildcard |
::= | AttributeName
| "*" |
|
| [134] | SchemaAttributeTest |
::= | "schema-attribute" "(" AttributeDeclaration
")" |
|
| [135] | AttributeDeclaration |
::= | AttributeName |
|
| [136] | ElementTest |
::= | "element" "(" (ElementNameOrWildcard
("," TypeName "?"?)?)?
")" |
|
| [137] | ElementNameOrWildcard |
::= | ElementName |
"*" |
|
| [138] | SchemaElementTest |
::= | "schema-element" "(" ElementDeclaration
")" |
|
| [139] | ElementDeclaration |
::= | ElementName |
|
| [140] | AttributeName |
::= | QName |
|
| [141] | ElementName |
::= | QName |
|
| [142] | TypeName |
::= | QName |
|
| [143] | URILiteral |
::= | StringLiteral |
|
| [144] | FTSelection |
::= | FTOr FTPosFilter* |
|
| [145] | FTWeight |
::= | "weight" "{" Expr
"}" |
|
| [146] | FTOr |
::= | FTAnd ( "ftor"
FTAnd )* |
|
| [147] | FTAnd |
::= | FTMildNot (
"ftand" FTMildNot
)* |
|
| [148] | FTMildNot |
::= | FTUnaryNot (
"not" "in" FTUnaryNot
)* |
|
| [149] | FTUnaryNot |
::= | ("ftnot")? FTPrimaryWithOptions |
|
| [150] | FTPrimaryWithOptions |
::= | FTPrimary FTMatchOptions? FTWeight? |
|
| [151] | FTPrimary |
::= | (FTWords FTTimes?) | ("(" FTSelection ")") | FTExtensionSelection |
|
| [152] | FTWords |
::= | FTWordsValue
FTAnyallOption? |
|
| [153] | FTWordsValue |
::= | StringLiteral
| ("{" Expr "}") |
|
| [154] | FTExtensionSelection |
::= | Pragma+ "{" FTSelection? "}" |
|
| [155] | FTAnyallOption |
::= | ("any" "word"?) | ("all" "words"?) | "phrase" |
|
| [156] | FTTimes |
::= | "occurs" FTRange
"times" |
|
| [157] | FTRange |
::= | ("exactly" AdditiveExpr) |
|
| [158] | FTPosFilter |
::= | FTOrder | FTWindow | FTDistance | FTScope | FTContent |
|
| [159] | FTOrder |
::= | "ordered" |
|
| [160] | FTWindow |
::= | "window" AdditiveExpr FTUnit |
|
| [161] | FTDistance |
::= | "distance" FTRange
FTUnit |
|
| [162] | FTUnit |
::= | "words" | "sentences" | "paragraphs" |
|
| [163] | FTScope |
::= | ("same" | "different") FTBigUnit |
|
| [164] | FTBigUnit |
::= | "sentence" | "paragraph" |
|
| [165] | FTContent |
::= | ("at" "start") | ("at" "end") | ("entire"
"content") |
|
| [166] | FTMatchOptions |
::= | ("using" FTMatchOption)+ |
|
| [167] | FTMatchOption |
::= | FTLanguageOption |
|
| [168] | FTCaseOption |
::= | ("case" "insensitive") |
|
| [169] | FTDiacriticsOption |
::= | ("diacritics" "insensitive") |
|
| [170] | FTStemOption |
::= | "stemming" | ("no" "stemming") |
|
| [171] | FTThesaurusOption |
::= | ("thesaurus" (FTThesaurusID |
"default")) |
|
| [172] | FTThesaurusID |
::= | "at" URILiteral
("relationship" StringLiteral)? (FTLiteralRange
"levels")? |
|
| [173] | FTLiteralRange |
::= | ("exactly" IntegerLiteral) |
|
| [174] | FTStopWordOption |
::= | ("stop" "words" FTStopWords FTStopWordsInclExcl*) |
|
| [175] | FTStopWords |
::= | ("at" URILiteral) |
|
| [176] | FTStopWordsInclExcl |
::= | ("union" | "except") FTStopWords |
|
| [177] | FTLanguageOption |
::= | "language" StringLiteral |
|
| [178] | FTWildCardOption |
::= | "wildcards" | ("no" "wildcards") |
|
| [179] | FTExtensionOption |
::= | "option" QName
StringLiteral |
|
| [180] | FTIgnoreOption |
::= | "without" "content" UnionExpr |
| [181] | IntegerLiteral |
::= | Digits |
|
| [182] | DecimalLiteral |
::= | ("." Digits) |
(Digits "." [0-9]*) |
/* ws: explicitXQ */ |
| [183] | DoubleLiteral |
::= | (("." Digits) |
(Digits ("." [0-9]*)?)) [eE]
[+-]? Digits |
/* ws: explicitXQ */ |
| [184] | StringLiteral |
::= | ('"' (PredefinedEntityRef |
CharRef | EscapeQuot | [^"&])* '"') |
("'" (PredefinedEntityRef |
CharRef | EscapeApos | [^'&])*
"'") |
/* ws: explicitXQ */ |
| [185] | PredefinedEntityRef |
::= | "&" ("lt" | "gt" | "amp" | "quot" | "apos")
";" |
/* ws: explicitXQ */ |
| [186] | EscapeQuot |
::= | '""' |
|
| [187] | EscapeApos |
::= | "''" |
|
| [188] | ElementContentChar |
::= | (Char -
[{}<&]) |
|
| [189] | QuotAttrContentChar |
::= | (Char -
["{}<&]) |
|
| [190] | AposAttrContentChar |
::= | (Char -
['{}<&]) |
|
| [191] | Comment |
::= | "(:" (CommentContents | Comment)* ":)" |
/* ws: explicitXQ */ |
| /* gn: commentsXQ */ | ||||
| [192] | PITarget |
::= | [http://www.w3.org/TR/REC-xml#NT-PITarget]XML |
/* xgc: xml-versionXQ */ |
| [193] | CharRef |
::= | [http://www.w3.org/TR/REC-xml#NT-CharRef]XML |
/* xgc: xml-versionXQ */ |
| [194] | QName |
::= | [http://www.w3.org/TR/REC-xml-names/#NT-QName]Names |
/* xgc: xml-versionXQ */ |
| [195] | NCName |
::= | [http://www.w3.org/TR/REC-xml-names/#NT-NCName]Names |
/* xgc: xml-versionXQ */ |
| [196] | S |
::= | [http://www.w3.org/TR/REC-xml#NT-S]XML |
/* xgc: xml-versionXQ */ |
| [197] | Char |
::= | [http://www.w3.org/TR/REC-xml#NT-Char]XML |
/* xgc: xml-versionXQ */ |
The following symbols are used only in the definition of terminal symbols; they are not terminal symbols in the grammar of A EBNF for XQuery 1.0 Grammar with Full Text extensions.
| [198] | Digits |
::= | [0-9]+ |
| [199] | CommentContents |
::= | (Char+ - (Char* ('(:' |
':)') Char*)) |
The EBNF in this document and in this section is aligned with the current XPath 2.0 grammar (see http://www.w3.org/TR/2010/REC-xpath20-20101214/).
| [1] | XPath |
::= | Expr |
|
| [2] | Expr |
::= | ExprSingle (","
ExprSingle)* |
|
| [3] | ExprSingle |
::= | ForExpr |
|
| [4] | ForExpr |
::= | SimpleForClause "return"
ExprSingle |
|
| [5] | SimpleForClause |
::= | "for" "$" VarName
FTScoreVar? "in" ExprSingle ("," "$" VarName FTScoreVar? "in" ExprSingle)* |
|
| [6] | FTScoreVar |
::= | "score" "$" VarName |
|
| [7] | QuantifiedExpr |
::= | ("some" | "every") "$" VarName "in" ExprSingle ("," "$" VarName "in" ExprSingle)* "satisfies" ExprSingle |
|
| [8] | IfExpr |
::= | "if" "(" Expr ")" "then"
ExprSingle "else" ExprSingle |
|
| [9] | OrExpr |
::= | AndExpr ( "or"
AndExpr )* |
|
| [10] | AndExpr |
::= | ComparisonExpr
( "and" ComparisonExpr
)* |
|
| [11] | ComparisonExpr |
::= | FTContainsExpr
( (ValueComp |
|
| [12] | FTContainsExpr |
::= | RangeExpr (
"contains" "text" FTSelection FTIgnoreOption? )? |
|
| [13] | RangeExpr |
::= | AdditiveExpr (
"to" AdditiveExpr
)? |
|
| [14] | AdditiveExpr |
::= | MultiplicativeExpr ( ("+" |
"-") MultiplicativeExpr
)* |
|
| [15] | MultiplicativeExpr |
::= | UnionExpr ( ("*" |
"div" | "idiv" | "mod") UnionExpr )* |
|
| [16] | UnionExpr |
::= | IntersectExceptExpr (
("union" | "|") IntersectExceptExpr
)* |
|
| [17] | IntersectExceptExpr |
::= | InstanceofExpr
( ("intersect" | "except") InstanceofExpr )* |
|
| [18] | InstanceofExpr |
::= | TreatExpr (
"instance" "of" SequenceType )? |
|
| [19] | TreatExpr |
::= | CastableExpr (
"treat" "as" SequenceType
)? |
|
| [20] | CastableExpr |
::= | CastExpr (
"castable" "as" SingleType
)? |
|
| [21] | CastExpr |
::= | UnaryExpr ( "cast"
"as" SingleType
)? |
|
| [22] | UnaryExpr |
::= | ("-" | "+")* ValueExpr |
|
| [23] | ValueExpr |
::= | PathExpr |
|
| [24] | GeneralComp |
::= | "=" | "!=" | "<" | "<=" | ">" |
">=" |
|
| [25] | ValueComp |
::= | "eq" | "ne" | "lt" | "le" | "gt" | "ge" |
|
| [26] | NodeComp |
::= | "is" | "<<" | ">>" |
|
| [27] | Pragma |
::= | "(#" S? QName (S
PragmaContents)?
"#)" |
/* ws: explicitXP */ |
| [28] | PragmaContents |
::= | (Char* - (Char* '#)'
Char*)) |
|
| [29] | PathExpr |
::= | ("/" RelativePathExpr?) |
/* xgc: leading-lone-slashXP */ |
| [30] | RelativePathExpr |
::= | StepExpr (("/" |
"//") StepExpr)* |
|
| [31] | StepExpr |
::= | FilterExpr |
AxisStep |
|
| [32] | AxisStep |
::= | (ReverseStep |
ForwardStep) PredicateList |
|
| [33] | ForwardStep |
::= | (ForwardAxis
NodeTest) | AbbrevForwardStep |
|
| [34] | ForwardAxis |
::= | ("child" "::") |
|
| [35] | AbbrevForwardStep |
::= | "@"? NodeTest |
|
| [36] | ReverseStep |
::= | (ReverseAxis
NodeTest) | AbbrevReverseStep |
|
| [37] | ReverseAxis |
::= | ("parent" "::") |
|
| [38] | AbbrevReverseStep |
::= | ".." |
|
| [39] | NodeTest |
::= | KindTest | NameTest |
|
| [40] | NameTest |
::= | QName | Wildcard |
|
| [41] | Wildcard |
::= | "*" |
/* ws: explicitXP */ |
| [42] | FilterExpr |
::= | PrimaryExpr
PredicateList |
|
| [43] | PredicateList |
::= | Predicate* |
|
| [44] | Predicate |
::= | "[" Expr "]" |
|
| [45] | PrimaryExpr |
::= | Literal |
|
| [46] | Literal |
::= | NumericLiteral
| StringLiteral |
|
| [47] | NumericLiteral |
::= | IntegerLiteral
| DecimalLiteral |
DoubleLiteral |
|
| [48] | VarRef |
::= | "$" VarName |
|
| [49] | VarName |
::= | QName |
|
| [50] | ParenthesizedExpr |
::= | "(" Expr?
")" |
|
| [51] | ContextItemExpr |
::= | "." |
|
| [52] | FunctionCall |
::= | QName "(" (ExprSingle ("," ExprSingle)*)? ")" |
/* xgc: reserved-function-namesXP */ |
| /* gn: parensXP */ | ||||
| [53] | SingleType |
::= | AtomicType
"?"? |
|
| [54] | SequenceType |
::= | ("empty-sequence" "(" ")") |
|
| [55] | OccurrenceIndicator |
::= | "?" | "*" | "+" |
/* xgc: occurrence-indicatorsXP */ |
| [56] | ItemType |
::= | KindTest | ("item"
"(" ")") | AtomicType |
|
| [57] | AtomicType |
::= | QName |
|
| [58] | KindTest |
::= | DocumentTest |
|
| [59] | AnyKindTest |
::= | "node" "(" ")" |
|
| [60] | DocumentTest |
::= | "document-node" "(" (ElementTest | SchemaElementTest)?
")" |
|
| [61] | TextTest |
::= | "text" "(" ")" |
|
| [62] | CommentTest |
::= | "comment" "(" ")" |
|
| [63] | PITest |
::= | "processing-instruction" "(" (NCName | StringLiteral)? ")" |
|
| [64] | AttributeTest |
::= | "attribute" "(" (AttribNameOrWildcard (","
TypeName)?)? ")" |
|
| [65] | AttribNameOrWildcard |
::= | AttributeName |
"*" |
|
| [66] | SchemaAttributeTest |
::= | "schema-attribute" "(" AttributeDeclaration
")" |
|
| [67] | AttributeDeclaration |
::= | AttributeName |
|
| [68] | ElementTest |
::= | "element" "(" (ElementNameOrWildcard
("," TypeName "?"?)?)?
")" |
|
| [69] | ElementNameOrWildcard |
::= | ElementName |
"*" |
|
| [70] | SchemaElementTest |
::= | "schema-element" "(" ElementDeclaration
")" |
|
| [71] | ElementDeclaration |
::= | ElementName |
|
| [72] | AttributeName |
::= | QName |
|
| [73] | ElementName |
::= | QName |
|
| [74] | TypeName |
::= | QName |
|
| [75] | URILiteral |
::= | StringLiteral |
|
| [76] | FTSelection |
::= | FTOr FTPosFilter* |
|
| [77] | FTWeight |
::= | "weight" "{" Expr
"}" |
|
| [78] | FTOr |
::= | FTAnd ( "ftor" FTAnd )* |
|
| [79] | FTAnd |
::= | FTMildNot ( "ftand"
FTMildNot )* |
|
| [80] | FTMildNot |
::= | FTUnaryNot ( "not"
"in" FTUnaryNot
)* |
|
| [81] | FTUnaryNot |
::= | ("ftnot")? FTPrimaryWithOptions |
|
| [82] | FTPrimaryWithOptions |
::= | FTPrimary FTMatchOptions? FTWeight? |
|
| [83] | FTPrimary |
::= | (FTWords FTTimes?) | ("(" FTSelection ")") | FTExtensionSelection |
|
| [84] | FTWords |
::= | FTWordsValue
FTAnyallOption? |
|
| [85] | FTWordsValue |
::= | StringLiteral |
("{" Expr "}") |
|
| [86] | FTExtensionSelection |
::= | Pragma+ "{" FTSelection? "}" |
|
| [87] | FTAnyallOption |
::= | ("any" "word"?) | ("all" "words"?) | "phrase" |
|
| [88] | FTTimes |
::= | "occurs" FTRange
"times" |
|
| [89] | FTRange |
::= | ("exactly" AdditiveExpr) |
|
| [90] | FTPosFilter |
::= | FTOrder | FTWindow | FTDistance | FTScope | FTContent |
|
| [91] | FTOrder |
::= | "ordered" |
|
| [92] | FTWindow |
::= | "window" AdditiveExpr FTUnit |
|
| [93] | FTDistance |
::= | "distance" FTRange
FTUnit |
|
| [94] | FTUnit |
::= | "words" | "sentences" | "paragraphs" |
|
| [95] | FTScope |
::= | ("same" | "different") FTBigUnit |
|
| [96] | FTBigUnit |
::= | "sentence" | "paragraph" |
|
| [97] | FTContent |
::= | ("at" "start") | ("at" "end") | ("entire"
"content") |
|
| [98] | FTMatchOptions |
::= | ("using" FTMatchOption)+ |
|
| [99] | FTMatchOption |
::= | FTLanguageOption |
|
| [100] | FTCaseOption |
::= | ("case" "insensitive") |
|
| [101] | FTDiacriticsOption |
::= | ("diacritics" "insensitive") |
|
| [102] | FTStemOption |
::= | "stemming" | ("no" "stemming") |
|
| [103] | FTThesaurusOption |
::= | ("thesaurus" (FTThesaurusID | "default")) |
|
| [104] | FTThesaurusID |
::= | "at" URILiteral
("relationship" StringLiteral)? (FTLiteralRange
"levels")? |
|
| [105] | FTLiteralRange |
::= | ("exactly" IntegerLiteral) |
|
| [106] | FTStopWordOption |
::= | ("stop" "words" FTStopWords FTStopWordsInclExcl*) |
|
| [107] | FTStopWords |
::= | ("at" URILiteral) |
|
| [108] | FTStopWordsInclExcl |
::= | ("union" | "except") FTStopWords |
|
| [109] | FTLanguageOption |
::= | "language" StringLiteral |
|
| [110] | FTWildCardOption |
::= | "wildcards" | ("no" "wildcards") |
|
| [111] | FTExtensionOption |
::= | "option" QName StringLiteral |
|
| [112] | FTIgnoreOption |
::= | "without" "content" UnionExpr |
| [113] | IntegerLiteral |
::= | Digits |
|
| [114] | DecimalLiteral |
::= | ("." Digits) |
(Digits "." [0-9]*) |
/* ws: explicitXP */ |
| [115] | DoubleLiteral |
::= | (("." Digits) |
(Digits ("." [0-9]*)?)) [eE]
[+-]? Digits |
/* ws: explicitXP */ |
| [116] | StringLiteral |
::= | ('"' (EscapeQuot |
[^"])* '"') | ("'" (EscapeApos | [^'])* "'") |
/* ws: explicitXP */ |
| [117] | EscapeQuot |
::= | '""' |
|
| [118] | EscapeApos |
::= | "''" |
|
| [119] | Comment |
::= | "(:" (CommentContents | Comment)* ":)" |
/* ws: explicitXP */ |
| /* gn: commentsXP */ | ||||
| [120] | QName |
::= | [http://www.w3.org/TR/REC-xml-names/#NT-QName]Names |
/* xgc: xml-versionXP */ |
| [121] | NCName |
::= | [http://www.w3.org/TR/REC-xml-names/#NT-NCName]Names |
/* xgc: xml-versionXP */ |
| [122] | S |
::= | [http://www.w3.org/TR/REC-xml#NT-S]XML |
/* xgc: xml-versionXP */ |
| [123] | Char |
::= | [http://www.w3.org/TR/REC-xml#NT-Char]XML |
/* xgc: xml-versionXP */ |
The following symbols are used only in the definition of terminal symbols; they are not terminal symbols in the grammar of B EBNF for XPath 2.0 Grammar with Full-Text extensions.
| [124] | Digits |
::= | [0-9]+ |
| [125] | CommentContents |
::= | (Char+ - (Char* ('(:' |
':)') Char*)) |
The following table describes the full-text components of the static context (as defined in Section 2.1.1 Static ContextXQ). The following aspects of each component are described:
Default initial value: This is the initial value of the component if it is not overridden or augmented by the implementation or by a query.
Can be overwritten or augmented by implementation: Indicates whether an XQuery implementation is allowed to replace the default initial value of the component by a different, implementation-defined value and/or to augment the default initial value by additional implementation-defined values.
Can be overwritten or augmented by a query: Indicates whether a query is allowed to replace and/or augment the initial value provided by default or by the implementation. If so, indicates how this is accomplished (for example, by a declaration in the prolog; as defined in Section 4 Modules and PrologsXQ).
Scope: Indicates where the component is applicable. "Global" indicates that the component applies globally, throughout all the modules used in a query. "Module" indicates that the component applies throughout a module (as defined in Section 4 Modules and PrologsXQ). "Lexical" indicates that the component applies within the expression in which it is defined (equivalent to "module", if the component is declared in a prolog.)
Consistency Rules: Indicates rules that must be observed in assigning values to the component.
| Component | Default initial value | Can be overwritten or augmented by implementation? | Can be overwritten or augmented by a query? | Scope | Consistency rules |
|---|---|---|---|---|---|
| FTCaseOption | case insensitive |
overwriteable | overwriteable by prolog | lexical | Value must be case insensitive , case
sensitive , lowercase , or
uppercase . |
| FTDiacriticsOption | diacritics insensitive |
overwriteable | overwriteable by prolog | lexical | Value must be diacritics insensitive or
diacritics sensitive . |
| FTStemOption | no stemming |
overwriteable | overwriteable by prolog | lexical | Value must be stemming or no
stemming . |
| FTThesaurusOption | no thesaurus |
overwriteable | overwriteable by prolog (refer to default to augment) | lexical | Each URI in the value must be found in the statically known thesauri. |
| Statically known thesauri | none | augmentable | cannot be augmented or overwritten by prolog | module | Each URI uniquely identifies a thesaurus list. |
| FTStopWordOption | no stop words |
overwriteable | overwriteable by prolog (refer to default to augment) | lexical | Each URI in the value must be found in the statically known stop word lists. |
| Statically known stop word lists | none | augmentable | cannot be augmented or overwritten by prolog | module | Each URI uniquely identifies a stop word list. |
| FTLanguageOption | implementation-defined | overwriteable | overwriteable by prolog | lexical | Value must be castable to xs:language . |
| Statically known languages | none | augmentable | cannot be augmented or overwritten by prolog | module | Each string uniquely identifies a language. |
| FTWildCardOption | no wildcards |
no | overwriteable by prolog | lexical | Value must be wildcards or no
wildcards . |
An implementation that does not support the FTMildNot operator must raise a static error if a full-text query contains a mild not.
An implementation that enforces one of the restrictions on FTUnaryNot must raise a static error if a full-text query does not obey the restriction.
An implementation that does not support one or more of the choices on FTUnit and FTBigUnit must raise a static error if a full-text query contains one of those choices.
An implementation that does not support the FTScope operator must raise a static error if a full-text query contains a scope.
An implementation that does not support the FTTimes operator must raise a static error if a full-text query contains a times.
An implementation that restricts the use of FTStopWordOption must raise a static error if a full-text query contains a stop word option that does not meet the restriction.
An implementation that restricts the use of FTIgnoreOption must raise a static error if a full-text query contains an ignore option that does not meet the restriction.
It is a static error if, during the static analysis phase, the query is found to contain a stop word option that refers to a stop word list that is not found in the statically known stop word lists.
It may be a static error if, during the static analysis phase, the query is found to contain a language identifier in a language option that the implementation does not support. The implementation may choose not to raise this error and instead provide some other implementation-defined behavior.
It is a static error if, during the static analysis phase, an expression is found to use an FTOrder operator that does not appear directly succeeding an FTWindow or an FTDistance operator and the implementation enforces this restriction.
An implementation may restrict the use of FTWindow and FTDistance to an FTOr that is either a single FTWords or a combination of FTWords involving only the operators ftand and ftor. If it a static error if, during the static analysis phase, an expression is found that violates this restriction and the implementation enforces this restriction.
An implementation that does not support the FTContent operator must raise a static error if a full-text query contains one.
It is a static error if, during the static analysis phase, an implementation that restricts the use of FTLanguageOption to a single language, encounters more than one distinct language option.
An implementation may constrain the form of the expression used to compute scores. It is a static error if, during the static analysis phase, such an implementation encounters a scoring expression that does not meet the restriction.
It is a static error if, during the static analysis phase, an implementation that restricts the choices of FTCaseOption encounters the "lowercase" or "uppercase" option.
It is a dynamic error if a weight value is not within the required range of values; it is also a dynamic error if an implementation that does not support negative weights encounters a negative weight value.
It is a dynamic error if an implementation encounters a mild not selection, one of whose operands evaluates to an AllMatches that contains a StringExclude
It is a static error if, during the static analysis phase, the query is found to contain a thesaurus option that refers to a thesaurus that is not found in the statically known thesauri.
It is a static error if, within a single FTMatchOptions, there is more than one match option of any given match option group.
It is a dynamic error if, when "wildcards" is in effect, a query string violates wildcard syntax.
It is a dynamic error if, in a function invocation, the argument corresponding to the specified function's collation parameter does not identify a supported collation.
It is a static error if an expression is not a valid instance of the grammar defined in A EBNF for XQuery 1.0 Grammar with Full Text extensions or of the grammar defined in B EBNF for XPath 2.0 Grammar with Full-Text extensions.
It is a type error if, during the static analysis phase, an expression is found to have a static type that is not appropriate for the context in which the expression occurs, or during the dynamic evaluation phase, the dynamic type of a value does not match a required type as specified by the matching rules in Section 2.5.4 SequenceType MatchingXP.
It is a static error if an implementation recognizes a pragma but determines that its content is invalid.
It is a static error if an extension expression contains neither a pragma that is recognized by the implementation nor an expression enclosed in curly braces.
[XML Syntax for XQuery 1.0 (XQueryX) (Second Edition)] defines an XML representation of [XQuery 1.0: An XML Query Language (Second Edition)]. [XQuery and XPath Full Text 1.0 Requirements], section 5.4, XML Syntax, states "XQuery and XPath Full Text MAY have more than one syntax binding. One query language syntax MUST be expressed in XML in a way that reflects the underlying structure of the query. See XML Query Requirements." This appendix specifies XML Schemas that together define the XML representation of XQuery and XPath Full Text 1.0 by representing the abstract syntax found in A EBNF for XQuery 1.0 Grammar with Full Text extensions. Because XQuery and XPath Full Text 1.0 integrates seamlessly with XQuery 1.0, it follows that the XML Syntax for XQuery and XPath Full Text 1.0 must integrate well with the XML Syntax for XQuery 1.0.
The XML Schema specified in this appendix accomplishes integration by importing the XML Schema defined for XQueryX in Section 4 An XML Schema for the XQuery XML SyntaxXQX, incorporating all of its type and element definitions. It then extends that schema by adding definitions of new types and elements in a namespace belonging to the full-text specification.
The semantics of a Full Text XQueryX document are determined by the semantics of the XQuery Full Text expression that results from transforming the XQueryX document into XQuery Full Text syntax using the XSLT stylesheet that appears in section E.2 XQueryX stylesheet for XQuery and XPath Full Text 1.0. The "correctness" of that transformation is determined by asking the following the question: Can some Full Text XQueryX processor QX process some Full Text XQueryX document D1 to produce results R1, after which the stylesheet is used to translate D1 into an XQuery Full Text expression E1 that, when processed by some XQuery Full Text processor Q, produces results R2 that are equivalent (under some meaningful definition of "equivalent") to results R1?
The XML Schema that defines the complex types and elements for XQueryX in support of XQuery and XPath Full Text 1.0, including the ftContainsExpr, incorporates a second XML Schema that defines types and elements to support the ftMatchOption. Both XML Schemas are defined in this section.