Added document in the repo

Added document in the repo

= TWISTER APPLICATION PROGRAMMERS GUIDE
:docinfo:
[preface]
== Introduction
This document gives an overview of the modules build in Twister API and how an application is built on it. The detailed definitions and individual function calls can be found in the doc folder of Twister API.
At start, the document explains the core modules of twister API, these include initialization module, event loop, ARP table management and JSON parsing. Details of build system are also included.
After that, it explains the sample application step by step to give programmer an example of API usage. The Appendices contain information to debug any relevant issues.
The best way to start is to go through the Sample Applications Details Section initially while reviewing the code of the sample application and then go through the whole document for complete understanding of design of Twister API and application development on top of it.

== Initialization Module
The initialization module of API handles the initialization of twister abstraction layer. Abstraction layer consists of the following parts:
* System Memory Reservation: The abstraction layer facilitates the reservation of different memory zones, for example, physical memory areas for device interactions.
* Allocate PCI resources and set up the physical device and the link
* Initialize statistics counters
* Link Up/Down control and notification interrupt. This is the only interrupt that a PMD handles
* Port and Queue setup for both RX and TX
* Access the RX and TX descriptors directly without any interrupts
* Assign user specified parameters to the physical device.
In the Twister API, all the above initialization steps are formed by single API call which is:
tw_init_global(argc, argv)
The user specified parameters for the physical device are provided in the port_conf file. This file must be present in the application directory. The parameters are further explained in the below section:

=== Port Management
The port management is assigning specific parameters which include IP address, subnet mask, gateway IP to the port. All this is done in the initialization module. The user just hast to provide a file to the module, which is port_conf. The configuration file should be in JSON format and a sample is provided below:

port_conf file is array of JSON fields. Each array [port_name: <>, ip_addrs:<>] corresponds to one port:
* port_name: This is an identifier that lists the port name. The port name is assigned by Twister Virtual Machine (TVM)OS. TVM assigns the port name as *tw<X>* according to the PCI ID in a numerical order. So if a TVM has two ports, the port name for first one will be tw0 and for the second one will be tw1.
* ip_addrs: This field lists the IP address range to be assigned for the port. It contains two sub fields:
a. start_ip_addr: IP address in string format, assigned to the port.
b. num_ip_addrs: This number specify the range of IP addresses for this port.
c. gateway_ip: It will list the gateway for this particular port.
d. subnet_mask: The mask to determine the subnet of this port.
*For an application using multiple ports e.g. 2, the format of port_conf would be i.e.
[
       [
       port_name:tw0
	ip_addrs:[]
       ],
       [
       port_name:tw1
	ip_addrs:[]
       ]
]

== Event loop
Twister API provides an event loop which is based on *libuv* model with general flow shown below:

The Twister API implementation of event loop has the following features:
1. Event callback registration: Users can register three types of handles in the loop.
* RX handle: These handles are used for receiving packets. Currently only one RX handle is supported. This callback is called when any type of packet is received.
* TX handle: Multiple TX handles can be registered. Each handles callback will be called once per event loop iteration.
* Timer handle: This handles callback will be called after a fixed timeout.
2. With each instance of a handle, one callback function is registered. This is called when the event has occurred.

=== Comparison of twister API calls and Libuv calls are given below:

Sr.
 Twister
Libuv
1.
tw_default_loop: Reserve memory for an event loop with which handles can be registered. The loops total time to run is also provided
uv_default_loop: Reserve memory for an
event loop with which handles can be
registered
2.
tw_<handlename>_init: Reserve memory for handle pointer and register it with an event loop instance
uv_<handlename>_init: Reserve memory for handle pointer and register it with an event loop instance
3.
tw_<handlename>_start: Register a callback function for the handle
uv_<handlename_start>: Register a callback function for the handle
4.
tw_run: Start the event loop
uv_run: Start the event loop

== ARP Table
The Twister API has built-in ARP functionality which user can utilize in their application using simple calls. The API will maintain the table locally. If application is sending a packet whose MAC is not known, the API will store the packet in the queue and generate an ARP query for the IP and automatically update the queued packets once the ARP respond is received.
The API can also respond to the ARP query for the IP addresses of the interfaces which the API is handling. The application programmer just has to add the tw_arp_parser in the application flow. Rest will be the job of the API. To list all the ARP functions and their usage, kindly use the Twister API Reference documentation provided on Github. It can be accessed the twister/documentation/html/index.html file.

== Building the Applications
Cloning the Repository

Clone the desired branch from dell twister repository (https://github.com/delltwister/twister). Navigate to the directory of desired example.

=== Compiling a Sample Application
The sample applications can be built using the following commands:
--------------------------------------
	make clean
	make
--------------------------------------
=== Customizing the Application Makefile
The default Makefile provided with the UDP Echo sample application is a good starting point. It includes:
* twister.mk at the beginning
The user must define following variables:
* APP: Contains the name of the application.
* SRCS: List of source files (*.c).
The rest of the Makefile can be customized as a normal Makefile.

== Sample Applications Details
The general flow of Twister API applications is as follows:

There are three sample applications developed for Twister API release 0.1. They include the following:
1. UDP Send Application: This application is a tester client application which generates UDP traffic by sending timestamp to an echo server. It also receives any timestamps echoed back and calculates RTT.
2. UDP Echo Application: This application receives the UDP traffic and echo back to the sender.
3. Ethernet Echo Application: This application is a simple Layer 2 forward application which simply echo every packet it received by changing the MAC.
The UDP Echo server application is discussed below:

== Writing a New Application
The best way to start writing a new application is to copy an existing Twister API application and modify it. To list the functions and their usage, kindly use the Twister API Reference documentation provided on GitHub. It can be accessed from twister/documentation/html/index.html file.

== Appendices

HW/SW Information Relating to Twister API
Intel DPDK Version: 2.0.0
Linux Kernel Version: 3.11.0-12-generic
Linux Version: Ubuntu 12.04
Twister VM Architecture: x86_64
CPU Model: Intel Xeon E312xx (Sandy Bridge)
GCC Version: 4.8.1
Intel NICs Used for testing purposes: Virtual Functions of Intel x520 and x540 Dual Port 10G NICs

=TWISTER APPLICATION PROGRAMMERS GUIDE
:docinfo:
[preface]
== Introduction
This document gives an overview of the modules build in Twister API and how an application is built on it. The detailed definitions and individual function calls can be found in the doc folder of Twister API.
At start, the document explains the core modules of twister API, these include initialization module, event loop, ARP table management and JSON parsing. Details of build system are also included.
After that, it explains the sample application step by step to give programmer an example of API usage. The Appendices contain information to debug any relevant issues.
The best way to start is to go through the Sample Applications Details Section initially while reviewing the code of the sample application and then go through the whole document for complete understanding of design of Twister API and application development on top of it.

Initialization Module
The initialization module of API handles the initialization of twister abstraction layer. Abstraction layer consists of the following parts:
* System Memory Reservation: The abstraction layer facilitates the reservation of different memory zones, for example, physical memory areas for device interactions.
* Allocate PCI resources and set up the physical device and the link
* Initialize statistics counters
* Link Up/Down control and notification interrupt. This is the only interrupt that a PMD handles
* Port and Queue setup for both RX and TX
* Access the RX and TX descriptors directly without any interrupts
* Assign user specified parameters to the physical device.
In the Twister API, all the above initialization steps are formed by single API call which is:
tw_init_global(argc, argv)
The user specified parameters for the physical device are provided in the port_conf file. This file must be present in the application directory. The parameters are further explained in the below section:

Port Management
The port management is assigning specific parameters which include IP address, subnet mask, gateway IP to the port. All this is done in the initialization module. The user just hast to provide a file to the module, which is port_conf. The configuration file should be in JSON format and a sample is provided below:

port_conf file is array of JSON fields. Each array [port_name: <>, ip_addrs:<>] corresponds to one port:
* port_name: This is an identifier that lists the port name. The port name is assigned by Twister Virtual Machine (TVM)OS. TVM assigns the port name as tw<X> according to the PCI ID in a numerical order. So if a TVM has two ports, the port name for first one will be tw0 and for the second one will be tw1.
* ip_addrs: This field lists the IP address range to be assigned for the port. It contains two sub fields:
a. start_ip_addr: IP address in string format, assigned to the port.
b. num_ip_addrs: This number specify the range of IP addresses for this port.
c. gateway_ip: It will list the gateway for this particular port.
d. subnet_mask: The mask to determine the subnet of this port.
*For an application using multiple ports e.g. 2, the format of port_conf would be i.e.
[
       [
       port_name:tw0
	ip_addrs:[]
       ],
       [
       port_name:tw1
	ip_addrs:[]
       ]
]

==Event loop
Twister API provides an event loop which is based on libuv model with general flow shown below:

The Twister API implementation of event loop has the following features:
1. Event callback registration: Users can register three types of handles in the loop.
* RX handle: These handles are used for receiving packets. Currently only one RX handle is supported. This callback is called when any type of packet is received.
* TX handle: Multiple TX handles can be registered. Each handles callback will be called once per event loop iteration.
* Timer handle: This handles callback will be called after a fixed timeout.
2. With each instance of a handle, one callback function is registered. This is called when the event has occurred.

===Comparison of twister API calls and Libuv calls are given below:

Sr.
 Twister
Libuv
1.
tw_default_loop: Reserve memory for an event loop with which handles can be registered. The loops total time to run is also provided
uv_default_loop: Reserve memory for an
event loop with which handles can be
registered
2.
tw_<handlename>_init: Reserve memory for handle pointer and register it with an event loop instance
uv_<handlename>_init: Reserve memory for handle pointer and register it with an event loop instance
3.
tw_<handlename>_start: Register a callback function for the handle
uv_<handlename_start>: Register a callback function for the handle
4.
tw_run: Start the event loop
uv_run: Start the event loop

== ARP Table
The Twister API has built-in ARP functionality which user can utilize in their application using simple calls. The API will maintain the table locally. If application is sending a packet whose MAC is not known, the API will store the packet in the queue and generate an ARP query for the IP and automatically update the queued packets once the ARP respond is received.
The API can also respond to the ARP query for the IP addresses of the interfaces which the API is handling. The application programmer just has to add the tw_arp_parser in the application flow. Rest will be the job of the API. To list all the ARP functions and their usage, kindly use the Twister API Reference documentation provided on Github. It can be accessed the twister/documentation/html/index.html file.

==Building the Applications
Cloning the Repository

Clone the desired branch from dell twister repository (https://github.com/delltwister/twister). Navigate to the directory of desired example.

===Compiling a Sample Application
The sample applications can be built using the following commands:
       % make clean
       %make
===Customizing the Application Makefile
The default Makefile provided with the UDP Echo sample application is a good starting point. It includes:
* twister.mk at the beginning
The user must define following variables:
* APP: Contains the name of the application.
* SRCS: List of source files (*.c).
The rest of the Makefile can be customized as a normal Makefile.

==Sample Applications Details
The general flow of Twister API applications is as follows:

There are three sample applications developed for Twister API release 0.1. They include the following:
1. UDP Send Application: This application is a tester client application which generates UDP traffic by sending timestamp to an echo server. It also receives any timestamps echoed back and calculates RTT.
2. UDP Echo Application: This application receives the UDP traffic and echo back to the sender.
3. Ethernet Echo Application: This application is a simple Layer 2 forward application which simply echo every packet it received by changing the MAC.
The UDP Echo server application is discussed below:

===UDP Echo Server
The UDP Echo server step by step explanation is below:
 tw_init_global() Initialize the global environment for twister application by configuring ports and engines available on machine.

 tw_map_port_to_engine(): binds the a NIC port to a specific engine. So all the incoming/outgoing traffic on that port (tw<x>) will be managed by the engine (engine<x>). The description of port and engines is describe below.
o A port-name (that could be tw0, tw1, depending upon the number of available ports on the machine)
o An engine-name (engine0, engine1, depends on the number of available engines on the machine)

 In user_app_main(), the main logic of UDP ECHO application resides. Since this application only echoes back the incoming packets, we have to create a callback function which fires every time a packet arrives.
 tw_rx_t * server; is a pointer to callback struct that configures the rx_callback.   struct tw_rx_s {
 	    uint8_t handle_type;
               uint64_t last_run_time;
    int sock_fd;
    struct tw_sockaddr_in * addr;
    uint8_t flags;
    void * recv_cb;
    struct tw_rx_s * next;
};
o handle_type tells which callback does that corresponds. Since it is a receive callback function. So we set its value to RX_CALLBACK. Currently we support only three types of callback functions, RX_CALLBACK, TX_CALLBACK, TIMER_CALLBACK.
o last_run_time tells when was the last time this event was fired. Each time a packet arrives, timestamp to this variables would be updated. For the first-time we set it to 0.
o recv_cb is a pointer to functions that will be executed each time receive event will occur. Our callback function is replay_payload so we set this pointer to function reply_payload. Every-time we receive a packet, the reply_payload function is called. In this function, the user can implement any RX logic. In our case, we echo the timestamp in the payload and send the UDP packet back to client.
 tw_default_loop(INFINITE_LOOP); creates an infinite eventloop handler and its address is returned as pointer. tw_default_loop() takes argument in seconds and run the event loop for that time. A value of 0 will make it run infinitely. The INFINITE_LOOP parameter has a value 0. This event-loop handler is described below:
struct tw_loop_s {
    void* data;
               uint8_t active_handles*-;
    struct tw_rx_s * rx_handle_queue;
    struct tw_tx_s * tx_handle_queue;
    struct tw_timer_s * timer_handle_queue;
    unsigned int stop_flag;
    uint16_t secs_to_run;
};
o data: user_data this could be used for any data/functions user want to use for this event-loop.
o active_handles: number of active handles currently this event-loop is managing. This could be multiple Rx, Tx, and timer handles. Twister main event-loop uses this. User dont need to deal with it.
o rx_handle_queue: a link-list for all the Rx handlers.
o tx_handle_queue: a link-list for all the Rx handlers.
o imer_handle_queue: a link-list for all the timer handlers.
o stop_flag:  flag is raised when event-loop timer expires.
o secs_to_run: number of seconds for which this event-loop would be run.
 tw_rx_init(tw_loop); would add an Rx handler to event-loop(tw_loop) and return the pointer to new handler added to rx_handle_queue of event-loop.tw_run(tw_loop);
 tw_rx_start(server, reply_payload); would assign the receive call back function to receive handler. So everytime receive event occurs i.e., a packet is received, the call back function assigned to the handler would be executed. It takes two arguments:
o server:  a pointer to receive handler of type tw_rx_s.
o reply_payload:  A receive callback function.
 tw_run(tw_loop); takes a configured event-loop handler and execute it for whatever time the user has configured it.

==Writing a New Application
The best way to start writing a new application is to copy an existing Twister API application and modify it. To list the functions and their usage, kindly use the Twister API Reference documentation provided on GitHub. It can be accessed from twister/documentation/html/index.html file.

==Appendices

HW/SW Information Relating to Twister API
Intel DPDK Version: 2.0.0
Linux Kernel Version: 3.11.0-12-generic
Linux Version: Ubuntu 12.04
Twister VM Architecture: x86_64
CPU Model: Intel Xeon E312xx (Sandy Bridge)
GCC Version: 4.8.1
Intel NICs Used for testing purposes: Virtual Functions of Intel x520 and x540 Dual Port 10G NICs


[docs] Adjusting var name in UI docs

[docs] Adjusting var name in UI docs

[id="debezium-ui"]
= {prodname} UI

:linkattrs:
:icons: font
:toc:
:toclevels: 3
:toc-placement: macro

toc::[]

[NOTE]
====
This feature is currently in incubating state, i.e. exact semantics and behaviours etc. may change in future revisions, based on the feedback we receive.
Please let us know about your feature requests or if you encounter any problems while using the {prodname} UI.
====

The {prodname} UI allows users to set up and operate connectors more easily. For instance, a list of all connectors can be viewed along with their status, etc.  The Create Connector wizard allows the user to configure connectors, guiding and validating the property entries along the way.

== Design Considerations

The UI is implemented as a https://quarkus.io/[Quarkus]-based web application.  The backend is configured with the URL(s) of one or more Kafka Connect clusters and provides a REST interface for the frontend.  The frontend user interface uses https://reactjs.org/[ReactJS] as the primary technology, utilizing https://www.patternfly.org/v4/[Patternfly] react components and design patterns.

== Installation and Configuration

=== Configure the Debezium UI
The following table shows the environment variables for the https://hub.docker.com/r/debezium/debezium-ui[Debezium UI container image] and the related parameter names inside `application.properties` when running the Java application without the container.

[cols="1,2,2,2,6",options="header"]

|===
|Item |Environment variable |Parameter name in application.properties |Default value |Description

|1
|DEPLOYMENT_MODE[[DEPLOYMENT_MODE]]
|`deployment.mode`
|default
|Specifies how the Debezium UI is deployed. +
 +
For example, in some environments it might not be possible to reach the underlying backend, Kafka Connect REST interface or databases, then the link:#DEPLOYMENT_MODE[deployment mode] can be switched to match the underlying infrastructure. +
 +
`default` The default deployment mode. It uses the Debezium UI backend with the configured Kafka Connect clusters via the Kafka Connect REST interface (see link:#KAFKA_CONNECT_URIS[KAFKA_CONNECT_URIS] how they are configured). +
 +
 +
`validation.disabled` When set to validation.disabled the UI frontend will not call the backend to validate the user input nor check the availability and proper configuration of database connections. That mode is used to only generate the Debezium connector JSON configuration without the UI backend validation. +
 +
|2
|KAFKA_CONNECT_URIS[[KAFKA_CONNECT_URIS]]
|`kafka.connect.uris`
|http://connect:8083
|A comma-separated list to one or more URLs of Kafka Connect REST interfaces to specify the Kafka Connect clusters that should be managed by the Debezium UI.

|===

=== Debezium UI container image

The Debezium UI https://hub.docker.com/r/debezium/debezium-ui[container image] is available for running the UI.  To start the UI and connect to an existing Kafka Connect instance via Docker (where KAFKA_CONNECT_URIS supplies the comma-separated list of available URI(s)):

----
$ docker run -it --rm --name debezium-ui -p 8080:8080 -e KAFKA_CONNECT_URIS=http://connect:8083 debezium/debezium-ui:{debezium-version}
----

The UI connects to Kafka Connect via REST, so you need to make sure that the latter is reachable, e.g. by running both components on the same Docker network.

[NOTE]
====
Currently, the UI connects to un-authenticated Kafka Connect instances.  Also, there's no authorization or authentication implemented in the UI itself yet.  Until that is the case, you should secure the components e.g. with your own proxy for authorization, if needed.
====

=== Self-contained example

A self-contained example https://github.com/debezium/debezium-examples/tree/main/ui-demo[ui-demo] is available, which is included under https://github.com/debezium/debezium-examples[debezium-examples] on Github.  The ui-demo includes a docker-compose file which brings up several sources with data as well as the UI. Please refer to the https://github.com/debezium/debezium-examples/tree/main/ui-demo[README file] for more details on running the Debezium ui-demo.

== UI Operation

=== UI Connector List
The main page of the UI displays all the registered connectors.  Some of the highlights of the main page are as follows:

* *Kafka connect cluster* can be selected via the dropdown in the header.
* Connector table shows each connector with it's type (MySQL, PostgreSQL, MongoDB), connector status and connector tasks.
* A connector row can be expanded to show more details, as shown below with the 'testPostgres' connector.  Metrics are shown in the expansion area (*Note:* this feature is still under development and not functional yet).  Connector tasks are shown, with ability to *Restart* the task if desired.
* The kebab menu at the right of each connector row provides actions which allow the user to *Pause, Resume, Restart or Delete* the connector.

image::debezium-ui-connectors-list.png[{prodname} UI connectors]

=== UI Create Connector Wizard
The user can create a connector by clicking on the *Create a connector* button on the main page.  The first two steps of the wizard are required, but the remaining steps are optional. Each step will validate the user entries and provide feedback if there are problems.  After completing steps 1 and 2 successfully, the user can proceed to the final page to review and create the connector.

==== Create Connector - Connector type (required)
Choose the type of connector in step 1.  Currently the *MongoDB, MySQL and PostgreSQL* connector types are supported.  Addition of more connector types is currently in progress.

image::debezium-ui-create-connector-step1.png[{prodname} UI Create connector step1]

==== Create Connector - Properties (required)
The basic connection properties for the selected connector are entered in step 2, and the properties must be validated before proceeding.  Advanced connection properties are also provided in a separate section of this step.  Upon successful validation, the user may proceed to the next steps (Additional properties) - or they can elect to bypass the additional properties and proceed directly to Review.

image::debezium-ui-create-connector-step2.png[{prodname} UI Create connector step2]

==== Create Connector - Additional properties (optional)
The Additional properties are optional and can be summarized as follows:

* *Filter definition* - entry of *regular expressions* which define the filters for inclusion/exclusion of the items that will be included for CDC.  The included items are displayed as the filters are entered and applied.
* *Data options* - *Snapshot* and *Mapping* properties (optional).  The defaults can be viewed and changed if desired.
* *Runtime options* - *Engine* and *Heartbeat* properties (optional).  The defaults can be viewed and changed if desired.

==== Create Connector - Review
The *Review* step provides a summary of the configuration that will be used to create the connector.  If happy with the selections, click 'Finish' to create the connector.  If the properties need adjustment, navigate back to the earlier steps.

image::debezium-ui-create-connector-review.png[{prodname} UI Create connector review]

[id="debezium-ui"]
= {prodname} UI

:linkattrs:
:icons: font
:toc:
:toclevels: 3
:toc-placement: macro

toc::[]

[NOTE]
====
This feature is currently in incubating state, i.e. exact semantics and behaviours etc. may change in future revisions, based on the feedback we receive.
Please let us know about your feature requests or if you encounter any problems while using the {prodname} UI.
====

The {prodname} UI allows users to set up and operate connectors more easily. For instance, a list of all connectors can be viewed along with their status, etc.  The Create Connector wizard allows the user to configure connectors, guiding and validating the property entries along the way.

== Design Considerations

The UI is implemented as a https://quarkus.io/[Quarkus]-based web application.  The backend is configured with the URL(s) of one or more Kafka Connect clusters and provides a REST interface for the frontend.  The frontend user interface uses https://reactjs.org/[ReactJS] as the primary technology, utilizing https://www.patternfly.org/v4/[Patternfly] react components and design patterns.

== Installation and Configuration

=== Configure the Debezium UI
The following table shows the environment variables for the https://hub.docker.com/r/debezium/debezium-ui[Debezium UI container image] and the related parameter names inside `application.properties` when running the Java application without the container.

[cols="1,2,2,2,6",options="header"]

|===
|Item |Environment variable |Parameter name in application.properties |Default value |Description

|1
|DEPLOYMENT_MODE[[DEPLOYMENT_MODE]]
|`deployment.mode`
|default
|Specifies how the Debezium UI is deployed. +
 +
For example, in some environments it might not be possible to reach the underlying backend, Kafka Connect REST interface or databases, then the link:#DEPLOYMENT_MODE[deployment mode] can be switched to match the underlying infrastructure. +
 +
`default` The default deployment mode. It uses the Debezium UI backend with the configured Kafka Connect clusters via the Kafka Connect REST interface (see link:#KAFKA_CONNECT_URI[KAFKA_CONNECT_URI] how they are configured). +
 +
 +
`validation.disabled` When set to validation.disabled the UI frontend will not call the backend to validate the user input nor check the availability and proper configuration of database connections. That mode is used to only generate the Debezium connector JSON configuration without the UI backend validation. +
 +
|2
|KAFKA_CONNECT_URI[[KAFKA_CONNECT_URI]]
|`kafka.connect.uri`
|http://connect:8083
|A comma-separated list to one or more URLs of Kafka Connect REST interfaces to specify the Kafka Connect clusters that should be managed by the Debezium UI.

|===

=== Debezium UI container image

The Debezium UI https://hub.docker.com/r/debezium/debezium-ui[container image] is available for running the UI.  To start the UI and connect to an existing Kafka Connect instance via Docker (where KAFKA_CONNECT_URI supplies the comma-separated list of available URI(s)):

----
$ docker run -it --rm --name debezium-ui -p 8080:8080 -e KAFKA_CONNECT_URI=http://connect:8083 debezium/debezium-ui:{debezium-version}
----

The UI connects to Kafka Connect via REST, so you need to make sure that the latter is reachable, e.g. by running both components on the same Docker network.

[NOTE]
====
Currently, the UI connects to un-authenticated Kafka Connect instances.  Also, there's no authorization or authentication implemented in the UI itself yet.  Until that is the case, you should secure the components e.g. with your own proxy for authorization, if needed.
====

=== Self-contained example

A self-contained example https://github.com/debezium/debezium-examples/tree/main/ui-demo[ui-demo] is available, which is included under https://github.com/debezium/debezium-examples[debezium-examples] on Github.  The ui-demo includes a docker-compose file which brings up several sources with data as well as the UI. Please refer to the https://github.com/debezium/debezium-examples/tree/main/ui-demo[README file] for more details on running the Debezium ui-demo.

== UI Operation

=== UI Connector List
The main page of the UI displays all the registered connectors.  Some of the highlights of the main page are as follows:

* *Kafka connect cluster* can be selected via the dropdown in the header.
* Connector table shows each connector with it's type (MySQL, PostgreSQL, MongoDB), connector status and connector tasks.
* A connector row can be expanded to show more details, as shown below with the 'testPostgres' connector.  Metrics are shown in the expansion area (*Note:* this feature is still under development and not functional yet).  Connector tasks are shown, with ability to *Restart* the task if desired.
* The kebab menu at the right of each connector row provides actions which allow the user to *Pause, Resume, Restart or Delete* the connector.

image::debezium-ui-connectors-list.png[{prodname} UI connectors]

=== UI Create Connector Wizard
The user can create a connector by clicking on the *Create a connector* button on the main page.  The first two steps of the wizard are required, but the remaining steps are optional. Each step will validate the user entries and provide feedback if there are problems.  After completing steps 1 and 2 successfully, the user can proceed to the final page to review and create the connector.

==== Create Connector - Connector type (required)
Choose the type of connector in step 1.  Currently the *MongoDB, MySQL and PostgreSQL* connector types are supported.  Addition of more connector types is currently in progress.

image::debezium-ui-create-connector-step1.png[{prodname} UI Create connector step1]

==== Create Connector - Properties (required)
The basic connection properties for the selected connector are entered in step 2, and the properties must be validated before proceeding.  Advanced connection properties are also provided in a separate section of this step.  Upon successful validation, the user may proceed to the next steps (Additional properties) - or they can elect to bypass the additional properties and proceed directly to Review.

image::debezium-ui-create-connector-step2.png[{prodname} UI Create connector step2]

==== Create Connector - Additional properties (optional)
The Additional properties are optional and can be summarized as follows:

* *Filter definition* - entry of *regular expressions* which define the filters for inclusion/exclusion of the items that will be included for CDC.  The included items are displayed as the filters are entered and applied.
* *Data options* - *Snapshot* and *Mapping* properties (optional).  The defaults can be viewed and changed if desired.
* *Runtime options* - *Engine* and *Heartbeat* properties (optional).  The defaults can be viewed and changed if desired.

==== Create Connector - Review
The *Review* step provides a summary of the configuration that will be used to create the connector.  If happy with the selections, click 'Finish' to create the connector.  If the properties need adjustment, navigate back to the earlier steps.

image::debezium-ui-create-connector-review.png[{prodname} UI Create connector review]


Fix typo

Fix typo

= CIP2021-08-10 Operator precedence
:numbered:
:toc:
:toc-placement: macro
:source-highlighter: codemirror

*Authors:* Hannes Voigt <hannes.voigt@neo4j.com>

[abstract]
.Abstract
--
This CIP defines the operator precedence rules of openCypher.
--

toc::[]

== Background

The expression sub-languages of openCypher (and most other query and programming languages) allow very liberal composition of different elementary expressions into complex expressions through nesting.
Elementary expression the kinds of expressions the expression sub-languages provides and which for the building blocks of the sub-languages.

Some elementary expressions have zero operands, e.g. literals.
Most elementary expressions have one or more operands.
For instance,

- An arithmetic negation has one operand,
- A boolean conjunction has two operands,
- A property access has two operands -- an element and a property key,
- A function call has two operands -- a function name and an arguments list,
- A list comprehension has three or four operands -- an element variable name, a list, a predicate, and optionally a projection expression
- A case expression has multiple operands -- a value to test, for every case a test value/predicate and a result value, and a default value of the 'else' case

Some operands expect non-values, typically identifiers (variable names, function names, property keys, etc.).
Such _non-value operands_ do not allow nesting of expressions because expression only result in values.

However, most operands expect values of some types.
Such _value operands_ allow nesting of all expressions that result in a values of the expected type.
The nested expressions form an _operator tree_ and which typically evaluated bottom-up, i.e. all expressions that provide an operand are evaluated before the expression is evaluated that take this operands.
(Higher-order expression, e.g. list comprehensions, have operands which are exceptions to this bottom-up evaluation.)
In other words, the operator tree encodes the evaluation order.
For instance, the following operator tree
----
    *
   / \
  +   4
 / \
3   2
----
encodes that the numeric addition is evaluated before the numeric multiplication, so that the expression results in `20`.

However, the query string is a sequence of characters and not a tree.
The parser turns the query string into such an operator tree.
In the query string, nesting can clearly be denoted by parentheses.
For instance, the query string
----
((3+2)*4)
----
allows a parser to unambiguously construct the operator tree shown above.

For user convenience, better readability, and familiarity with common conventions in mathematical notation, openCypher (and most other query and programming languages) allows omitting the parentheses.
For instance,
----
3+2*4
----
is also a valid expression.

Without additional rules, however, it is not possible to unambiguously construct a operator tree for such an expression.
In the particular case, two operator trees are possible:

- `((3+2)*4)`
- `(3+(2*4))`

To avoid this kind of ambiguity, openCypher (and most other query and programming languages) has _precedence rules_.

Precedence rules (or _rules of operator precedence_) define the order in which the different operators, i.e. the different kinds of expressions, are evaluated.
For instance, numeric multiplication is evaluated before numeric addition, such that expression
----
3+2*4
----
is effectively evaluated as
----
(3+(2*4))
----
and, hence, results in `11` (rather than `20`).

This CIP states the precedence rules of openCypher.

== Proposal

The precedence in openCypher is defined by (1) _precedence levels_ and (2) _operator associativity_.
Both are defined in the following two subsections.

=== Precedence levels

The following <<precedenceLevels,table>> lists the precedence levels in descending order.

.[[precedenceLevels]]Precedence levels
[cols="<.<1a,<.<4a,<.<4a", options="header"]
|===
|Level         |Group         |Operators

|12
|Atoms
|

* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Literal[Literals]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Variable[Variables]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Parameter[Parameters]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=CaseExpression[Case expressions]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ListComprehension[List comprehensions]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=PatternComprehension[Pattern comprehensions]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Reduce[Reduce operator]
* quantifiers
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=RelationshipsPattern[Pattern predicates]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ParenthesizedExpression[Function invocations]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ParenthesizedExpression[Existential subqueries]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ParenthesizedExpression[Parenthesized expression]

|11
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=PropertyOrLabelsExpression[Graph element operators]
|

* property lookup
* label expressions

.3+|10
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=StringOperatorExpression[String operators] (left-hand operand)
|

* Prefix predicate (and right-hand operand)
* Suffix predicate (and right-hand operand)
* Contains predicate (and right-hand operand)
* Regular expression predicate

|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ListOperatorExpression[List operators] (left-hand operand)
|

* List element containment predicate (and right-hand operand)
* List element access
* List slicing

|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=NullOperatorExpression[Null predicates] (left-hand operand)
|

* Null predicate
* Not-null predicate

|9
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=UnaryAddOrSubtractExpression[Arithmetic additive inverse]
|

* Unary negative
* Unary positive

|8
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=PowerOfExpression[Exponentiation] (left-hand and right-hand operand)
|

* Exponentiation

|7
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=MultiplyDivideModuloExpression[Arithmetic multiplicative operators] (left-hand and right-hand operand)
|

* Multiplication
* Division
* Remainder (Modulo)

|6
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=AddOrSubtractExpression[Arithmetic additive operators] (left-hand and right-hand operand)
|

* Addition
* Substraction

|5
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ComparisonExpression[Comparison operators] (left-hand and right-hand operand)
|

* Equal
* Unequal
* Greater
* Greater or Equal
* Less
* Less or Equal

|4
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=NotExpression[Boolean negation]
|

* Negation

|3
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=AndExpression[Boolean conjunction] (left-hand and right-hand operand)
|

* Conjunction

|2
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=XorExpression[Boolean exclusive disjunction] (left-hand and right-hand operand)
|

* Exclusive disjunction

|1
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=OrExpression[Boolean inclusive disjunction] (left-hand and right-hand operand)
|

* Inclusive disjunction

|===

[IMPORTANT]
.Rule of precedence levels
====
Operators on level _X_ take precedence over any operator on level _Y_, when _X_ > _Y_, i.e. are of higher precedence.
Operators can only directly accept operators of higher precedence as operands.
====

The rule of precedence levels is enforced by the grammar.

The rule of precedence levels does not apply to all operands, though.
The table points out to which operands (left-hand or left-hand and right-hand) the rule of precedence levels apply.

If an operator has operands to which the precedence levels do not apply, these operands are syntactically delineate such that there is no ambiguity with regard to the operator tree.
For instance, the syntax of the list element access clearly delineates the list element index operand by brackets, e.g. `myList[5]`.
Such clearly delineated operands grammatically allow an expression of any precedence level, i.e. grammar encodes the operand as https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Expression[<Expression>].

A prominent expression with a clearly delineated operand is the _parenthesized expression_.
The parenthesized expression has a single operand delineated by parentheses, i.e. `( n.prop+6 )` where `n.prop+6` is the delineated operand.
The parenthesized expression has no other purpose than grammatically allowing expressions as operands that do not meet the rule of precedence levels.
For instance, an arithmetic addition cannot be directly an operand to an arithmetic multiplication by the rule of precedence levels, since addition is of lower precedence than multiplication.
However, with the help of a parenthesized expression, the user can denote
----
(3+2)*4
----
as a valid expression.
This achieves the desire operator tree
----
    *
   / \
 ( )  4
  |
  +
 / \
3   2
----
where the arithmetic addition is an operand to the arithmetic multiplication and, hence, results in `20` (rather than `11`).

=== Operator associativity

Most precedence level include multiple operators.

On some levels these operators are grammatical alternatives, e.g. for https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Atom[<Atom>s], and, hence, have unambiguous precendence.

On other levels, however, the grammar allows repetitions of such operators (chaining).

For instance, all the following are valid expressions:

* `--+-5`
* `5 + 4 + 3`
* `5 - 4 - 3`
* `5 - 4 + 3`
* `5 * 4 / 3`
* `5 % 4 * 3`
* `5 > 4 >= 3`

Associativity of the operators define the operator tree unambiguously for such expressions.

Chains of operators fall into four categories:

* _Chains of type-incompatible operators_
* _Chains of closed unary operators_
* _Chains of binary operators_
* _Chains with extra semantics_

Each is discussed in the following subsections, respectively.

==== Chains of type-incompatible operators
The grammar allows chaining of some operators in https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=StringListNullOperatorExpression[String, list, and null operators], which are

* not type compatible or
* only type-compatible in one possible way,

such that there is

* no valid operator tree or
* only one unambiguous operator tree,

respectively.

For instance, grammatically allowed operator chains without a valid operator tree are:

* `'string' STARTS WITH x [1..3]`
** `('string' STARTS WITH x) [1..3]` is invalid because list slicing does not accept a boolean as first operand
** `'string' STARTS WITH (x[1..3])` is invalid because the prefix predicate does not accept a list as second operand
* `foo CONTAINS 'bar' ENDS WITH x`
** `(foo CONTAINS 'bar') ENDS WITH x` is invalid because the suffix predicate does not accept a boolean as first operand
** `foo CONTAINS ('bar' ENDS WITH x)` is invalid because the containment predicate does not accept a boolean as second operand

For instance, grammatically allowed operator chains with only a single  valid operator tree are:

* `foo STARTS WITH x IS NOT NULL`
** `(foo STARTS WITH x) IS NOT NULL` is valid because the not-null predicate accepts a boolean as first operand
** `foo STARTS WITH (x IS NOT NULL)` is invalid because the prefix predicate does not accept a boolean as second operand
* `foo CONTAINS 'bar' IN list`
** `(foo CONTAINS 'bar') IN list` is valid because the list element containment predicate accepts a boolean as first operand
** `foo CONTAINS ('bar' IN list)` is invalid because the containment predicate does not accept a boolean as second operand

[IMPORTANT]
.Chains of type-incompatible operators
====
Chains of (partly) type-incompatible operators either

* do not have a valid operator tree at all, or
* have only one valid operator tree.
====

==== Chains of closed unary operators
Closed unary operators that allow chaining on the same precedence level are

* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=UnaryAddOrSubtractExpression[Arithmetic additive inverse]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=NotExpression[Boolean negation]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ListOperatorExpression[List slicing]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=NullOperatorExpression[Null predicates]

Example expressions are

* `--+-5`
* `NOT NOT NOT false`
* `list[4..17][2..5]`
* `x IS NOT NULL IS NOT NULL IS NULL`

[IMPORTANT]
.Chaining of closed unary operators
====
Chains of closed unary operators have an unambiguous operator tree.
====

[NOTE]
.Associativity of unary operators
====
The grammar defines if the operand of an unary operator is on the left end or on the right end of the operator's syntax.
This renders the operator left- or right associative, respectively.
Nevertheless, the grammar allows only one unambiguous operator tree, in both case.
====

==== Chains of binary operators
Binary operators that allow chaining with themselves of other operators of compatible result type on the same precedence level are

* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=PowerOfExpression[Exponentiation]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=MultiplyDivideModuloExpression[Multiplication]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=MultiplyDivideModuloExpression[Division]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=MultiplyDivideModuloExpression[Remainder (Modulo)]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=AddOrSubtractExpression[Addition]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=AddOrSubtractExpression[Substraction]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=AndExpression[Boolean conjunction]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=OrExpression[Boolean inclusive disjunction]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=OrExpression[Boolean exclusive disjunction]

Example expressions are

* `5 * 4 * 3`
* `5 ^ 4 ^ 3`
* `5 / 4 / 3`
* `5 % 4 % 3`
* `5 % 4 * 3`
* `5 - 4 - 3`
* `5 - 4 + 3`
* `5 + 4 + 3`
* `TRUE AND FALSE AND TRUE`
* `TRUE OR FALSE OR TRUE`
* `TRUE XOR FALSE XOR TRUE`

[IMPORTANT]
.Chains of binary operators
====
In chains of binary operators, the operator have left-to-right associativity, i.e. for every two operators, the operator appear earlier (more left) in the query string takes precedence and the operator tree is left-deep.
====

[NOTE]
.Associative binary operators
====
Where the semantics of operator is associative (e.g. addition), a right-deep operator tree produces the same result as a left-deep operator tree.
====

Consequently, the examples evaluate as follows:

[cols="2a,1a"]
|====
|
[source, cypher]
----
RETURN  5 * 4 * 3  AS a,
       (5 * 4)* 3  AS b,
        5 *(4 * 3) AS c
----
|
[options="header"]
!====
!a !b !c
!60!60!60
!====

|
[source, cypher]
----
RETURN  4 ^ 3 ^ 2  AS a,
       (4 ^ 3)^ 2  AS b,
        4 ^(3 ^ 2) AS c
----
|
[options="header"]
!====
! a      ! b      ! c
! 4096.0 ! 4096.0 ! 262144.0
!====

|
[source, cypher]
----
RETURN  5 / 4 / 3  AS a,
       (5 / 4)/ 3  AS b,
        5 /(4 / 3) AS c
----
|
[options="header"]
!====
! a ! b ! c
! 0 ! 0 ! 5
!====

|
[source, cypher]
----
RETURN  5 % 4 % 3  AS a,
       (5 % 4)% 3  AS b,
        5 %(4 % 3) AS c
----
|
[options="header"]
!====
! a ! b ! c
! 1 ! 1 ! 0
!====

|
[source, cypher]
----
RETURN  5 % 4 * 3  AS a,
       (5 % 4)* 3  AS b,
        5 %(4 * 3) AS c
----
|
[options="header"]
!====
! a ! b ! c
! 3 ! 3 ! 5
!====

|
[source, cypher]
----
RETURN  5 - 4 - 3  AS a,
       (5 - 4)- 3  AS b,
        5 -(4 - 3) AS c
----
|
[options="header"]
!====
! a  ! b  ! c
! -2 ! -2 ! 4
!====

|
[source, cypher]
----
RETURN  5 - 4 + 3  AS a,
       (5 - 4)+ 3  AS b,
        5 -(4 + 3) AS c
----
|
[options="header"]
!====
! a ! b ! c
! 4 ! 4 ! -2
!====

|
[source, cypher]
----
RETURN  5 + 4 + 3  AS a,
       (5 + 4)+ 3  AS b,
        5 +(4 + 3) AS c
----
|
[options="header"]
!====
!a !b !c
!12!12!12
!====

|
[source, cypher]
----
RETURN  TRUE AND FALSE  AND TRUE  AS a,
       (TRUE AND FALSE) AND TRUE  AS b,
        TRUE AND (FALSE AND TRUE) AS c
----
|
[options="header"]
!====
!a    !b    !c
!false!false!false
!====

|
[source, cypher]
----
RETURN  TRUE OR FALSE  OR TRUE  AS a,
       (TRUE OR FALSE) OR TRUE  AS b,
        TRUE OR (FALSE OR TRUE) AS c
----
|
[options="header"]
!====
!a   !b   !c
!true!true!true
!====

|
[source, cypher]
----
RETURN  TRUE XOR FALSE  XOR TRUE  AS a,
       (TRUE XOR FALSE) XOR TRUE  AS b,
        TRUE XOR (FALSE XOR TRUE) AS c
----
|
[options="header"]
!====
!a    !b    !c
!false!false!false
!====

|====

==== Chains with extra semantics
Operators whose chaining gives extra semantics are

* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ComparisonExpression[Comparison operations]

Example expression are

* `5 = 5 = 5`
* `5 = 4 <> 3`
* `5 <> 4 > 3`
* `5 > 4 > 3`
* `5 > 4 >= 3`
* `5 >= 4 < 3`

[IMPORTANT]
.Chains with extra semantics
====
Chains with extra semantics, either

a. form a flat operator tree of a single operator, or
b. their semantics is defined by a syntax transformation to an expression that has an unambiguous operator tree based on the other precedence rules stated in this document.
====

[NOTE]
.Associativity of comparison operators
====
Because of the extra semantics of a chain of comparison, the comparison operators do not have any observable associativity in openCypher.
====

Consequently, the examples evaluate as follows:

[cols="2a,1a"]
|====
|
[source, cypher]
----
RETURN  5 = 5 = 5  AS a,
       (5 = 5) AND (5 = 5) AS ax,
       (5 = 5)= 5  AS b,
        5 =(5 = 5) AS c
----
|
[options="header"]
!====
! a    ! ax   ! b     ! c
! true ! true ! false ! false
!====

|
[source, cypher]
----
RETURN  5 = 4 <> 3  AS a,
       (5 = 4) AND (4 <> 3) AS ax,
       (5 = 4)<> 3  AS b,
        5 =(4 <> 3) AS c
----
|
[options="header"]
!====
! a     ! ax    ! b    ! c
! false ! false ! true ! false
!====

|
[source, cypher]
----
RETURN  5 <> 4 > 3  AS a,
       (5 <> 4) AND (4 > 3) AS ax,
       (5 <> 4)> 3  AS b,
        5 <>(4 > 3) AS c
----
|
[options="header"]
!====
! a    ! ax   ! b    ! c
! true ! true ! null ! true
!====

|
[source, cypher]
----
RETURN  5 > 4 > 3  AS a,
       (5 > 4) AND (4 > 3) AS ax,
       (5 > 4)> 3  AS b,
        5 >(4 > 3) AS c
----
|
[options="header"]
!====
! a    ! ax   ! b    ! c
! true ! true ! null ! null
!====

|
[source, cypher]
----
RETURN  5 > 4 >= 3  AS a,
       (5 > 4) AND (4 >= 3) AS ax,
       (5 > 4)>= 3  AS b,
        5 >(4 >= 3) AS c
----
|
[options="header"]
!====
! a    ! ax   ! b    ! c
! true ! true ! null ! null
!====

|
[source, cypher]
----
RETURN  5 <= 4 < 3  AS a,
       (5 <= 4) AND (4 < 3) AS ax,
       (5 <= 4)< 3  AS b,
        5 <=(4 < 3) AS c
----
|
[options="header"]
!====
! a     ! ax    ! b    ! c
! false ! false ! null ! null
!====

|====

== What others do

openCypher has grammatically all operators in one precedence level hierarchy.
This is similar to how precedence is defined for more programming languages.
For instance:

* https://docs.oracle.com/javase/tutorial/java/nutsandbolts/operators.html[Java]
* https://docs.python.org/3/reference/expressions.html#operator-precedence[Python]
* https://en.cppreference.com/w/cpp/language/operator_precedence[C++]
* https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_Precedence#table[Javascript]

The precedence is similar.
Some minor difference:

* Programming languages have extra operators openCypher does not have, e.g. increment/decrement, bitwise operators, ternary operator, and assignments
* In Javascript exponentiation is right-to-left associative

SQL encodes large parts of the type compatibility of expressions in the grammar.
Hence, its precedence hierarchy is partition by type.
Within one type, SQL's precedence hierarchy is similar.

Chainable operators, such as addition, multiplication, etc. are left-to-right associative in SQL as well.
However, SQL directly encode that in the grammar, with left-recursive production rules, e.g.

[source, ebnf]
----
<numeric value expression> ::=
    <term>
  | <numeric value expression> <plus sign> <term>
  | <numeric value expression> <minus sign> <term>

<term> ::=
    <factor>
  | <term> <asterisk> <factor>
  | <term> <solidus> <factor>

<factor> ::=
  [ <sign> ] <numeric primary>
----

Most program and query language do not give chains of comparison operators extra semantics.
Typically, comparison operators are left-to-right associative.

== Benefits to this proposal

This CIP clarifies the precedence rules of openCypher.

== Caveats to this proposal

None known.

= CIP2021-08-10 Operator precedence
:numbered:
:toc:
:toc-placement: macro
:source-highlighter: codemirror

*Authors:* Hannes Voigt <hannes.voigt@neo4j.com>

[abstract]
.Abstract
--
This CIP defines the operator precedence rules of openCypher.
--

toc::[]

== Background

The expression sub-languages of openCypher (and most other query and programming languages) allow very liberal composition of different elementary expressions into complex expressions through nesting.
Elementary expression the kinds of expressions the expression sub-languages provides and which for the building blocks of the sub-languages.

Some elementary expressions have zero operands, e.g. literals.
Most elementary expressions have one or more operands.
For instance,

- An arithmetic negation has one operand,
- A boolean conjunction has two operands,
- A property access has two operands -- an element and a property key,
- A function call has two operands -- a function name and an arguments list,
- A list comprehension has three or four operands -- an element variable name, a list, a predicate, and optionally a projection expression
- A case expression has multiple operands -- a value to test, for every case a test value/predicate and a result value, and a default value of the 'else' case

Some operands expect non-values, typically identifiers (variable names, function names, property keys, etc.).
Such _non-value operands_ do not allow nesting of expressions because expression only result in values.

However, most operands expect values of some types.
Such _value operands_ allow nesting of all expressions that result in a values of the expected type.
The nested expressions form an _operator tree_ and which typically evaluated bottom-up, i.e. all expressions that provide an operand are evaluated before the expression is evaluated that take this operands.
(Higher-order expression, e.g. list comprehensions, have operands which are exceptions to this bottom-up evaluation.)
In other words, the operator tree encodes the evaluation order.
For instance, the following operator tree
----
    *
   / \
  +   4
 / \
3   2
----
encodes that the numeric addition is evaluated before the numeric multiplication, so that the expression results in `20`.

However, the query string is a sequence of characters and not a tree.
The parser turns the query string into such an operator tree.
In the query string, nesting can clearly be denoted by parentheses.
For instance, the query string
----
((3+2)*4)
----
allows a parser to unambiguously construct the operator tree shown above.

For user convenience, better readability, and familiarity with common conventions in mathematical notation, openCypher (and most other query and programming languages) allows omitting the parentheses.
For instance,
----
3+2*4
----
is also a valid expression.

Without additional rules, however, it is not possible to unambiguously construct a operator tree for such an expression.
In the particular case, two operator trees are possible:

- `((3+2)*4)`
- `(3+(2*4))`

To avoid this kind of ambiguity, openCypher (and most other query and programming languages) has _precedence rules_.

Precedence rules (or _rules of operator precedence_) define the order in which the different operators, i.e. the different kinds of expressions, are evaluated.
For instance, numeric multiplication is evaluated before numeric addition, such that expression
----
3+2*4
----
is effectively evaluated as
----
(3+(2*4))
----
and, hence, results in `11` (rather than `20`).

This CIP states the precedence rules of openCypher.

== Proposal

The precedence in openCypher is defined by (1) _precedence levels_ and (2) _operator associativity_.
Both are defined in the following two subsections.

=== Precedence levels

The following <<precedenceLevels,table>> lists the precedence levels in descending order.

.[[precedenceLevels]]Precedence levels
[cols="<.<1a,<.<4a,<.<4a", options="header"]
|===
|Level         |Group         |Operators

|12
|Atoms
|

* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Literal[Literals]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Variable[Variables]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Parameter[Parameters]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=CaseExpression[Case expressions]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ListComprehension[List comprehensions]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=PatternComprehension[Pattern comprehensions]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Reduce[Reduce operator]
* quantifiers
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=RelationshipsPattern[Pattern predicates]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ParenthesizedExpression[Function invocations]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ParenthesizedExpression[Existential subqueries]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ParenthesizedExpression[Parenthesized expression]

|11
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=PropertyOrLabelsExpression[Graph element operators]
|

* property lookup
* label expressions

.3+|10
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=StringOperatorExpression[String operators] (left-hand operand)
|

* Prefix predicate (and right-hand operand)
* Suffix predicate (and right-hand operand)
* Contains predicate (and right-hand operand)
* Regular expression predicate

|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ListOperatorExpression[List operators] (left-hand operand)
|

* List element containment predicate (and right-hand operand)
* List element access
* List slicing

|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=NullOperatorExpression[Null predicates] (left-hand operand)
|

* Null predicate
* Not-null predicate

|9
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=UnaryAddOrSubtractExpression[Arithmetic additive inverse]
|

* Unary negative
* Unary positive

|8
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=PowerOfExpression[Exponentiation] (left-hand and right-hand operand)
|

* Exponentiation

|7
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=MultiplyDivideModuloExpression[Arithmetic multiplicative operators] (left-hand and right-hand operand)
|

* Multiplication
* Division
* Remainder (Modulo)

|6
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=AddOrSubtractExpression[Arithmetic additive operators] (left-hand and right-hand operand)
|

* Addition
* Substraction

|5
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ComparisonExpression[Comparison operators] (left-hand and right-hand operand)
|

* Equal
* Unequal
* Greater
* Greater or Equal
* Less
* Less or Equal

|4
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=NotExpression[Boolean negation]
|

* Negation

|3
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=AndExpression[Boolean conjunction] (left-hand and right-hand operand)
|

* Conjunction

|2
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=XorExpression[Boolean exclusive disjunction] (left-hand and right-hand operand)
|

* Exclusive disjunction

|1
|https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=OrExpression[Boolean inclusive disjunction] (left-hand and right-hand operand)
|

* Inclusive disjunction

|===

[IMPORTANT]
.Rule of precedence levels
====
Operators on level _X_ take precedence over any operator on level _Y_, when _X_ > _Y_, i.e. are of higher precedence.
Operators can only directly accept operators of higher precedence as operands.
====

The rule of precedence levels is enforced by the grammar.

The rule of precedence levels does not apply to all operands, though.
The table points out to which operands (left-hand or left-hand and right-hand) the rule of precedence levels apply.

If an operator has operands to which the precedence levels do not apply, these operands are syntactically delineate such that there is no ambiguity with regard to the operator tree.
For instance, the syntax of the list element access clearly delineates the list element index operand by brackets, e.g. `myList[5]`.
Such clearly delineated operands grammatically allow an expression of any precedence level, i.e. grammar encodes the operand as https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Expression[<Expression>].

A prominent expression with a clearly delineated operand is the _parenthesized expression_.
The parenthesized expression has a single operand delineated by parentheses, i.e. `( n.prop+6 )` where `n.prop+6` is the delineated operand.
The parenthesized expression has no other purpose than grammatically allowing expressions as operands that do not meet the rule of precedence levels.
For instance, an arithmetic addition cannot be directly an operand to an arithmetic multiplication by the rule of precedence levels, since addition is of lower precedence than multiplication.
However, with the help of a parenthesized expression, the user can denote
----
3+(2*4)
----
as a valid expression.
This achieves the desire operator tree
----
    *
   / \
 ( )  4
  |
  +
 / \
3   2
----
where the arithmetic addition is an operand to the arithmetic multiplication and, hence, results in `20` (rather than `11`).

=== Operator associativity

Most precedence level include multiple operators.

On some levels these operators are grammatical alternatives, e.g. for https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=Atom[<Atom>s], and, hence, have unambiguous precendence.

On other levels, however, the grammar allows repetitions of such operators (chaining).

For instance, all the following are valid expressions:

* `--+-5`
* `5 + 4 + 3`
* `5 - 4 - 3`
* `5 - 4 + 3`
* `5 * 4 / 3`
* `5 % 4 * 3`
* `5 > 4 >= 3`

Associativity of the operators define the operator tree unambiguously for such expressions.

Chains of operators fall into four categories:

* _Chains of type-incompatible operators_
* _Chains of closed unary operators_
* _Chains of binary operators_
* _Chains with extra semantics_

Each is discussed in the following subsections, respectively.

==== Chains of type-incompatible operators
The grammar allows chaining of some operators in https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=StringListNullOperatorExpression[String, list, and null operators], which are

* not type compatible or
* only type-compatible in one possible way,

such that there is

* no valid operator tree or
* only one unambiguous operator tree,

respectively.

For instance, grammatically allowed operator chains without a valid operator tree are:

* `'string' STARTS WITH x [1..3]`
** `('string' STARTS WITH x) [1..3]` is invalid because list slicing does not accept a boolean as first operand
** `'string' STARTS WITH (x[1..3])` is invalid because the prefix predicate does not accept a list as second operand
* `foo CONTAINS 'bar' ENDS WITH x`
** `(foo CONTAINS 'bar') ENDS WITH x` is invalid because the suffix predicate does not accept a boolean as first operand
** `foo CONTAINS ('bar' ENDS WITH x)` is invalid because the containment predicate does not accept a boolean as second operand

For instance, grammatically allowed operator chains with only a single  valid operator tree are:

* `foo STARTS WITH x IS NOT NULL`
** `(foo STARTS WITH x) IS NOT NULL` is valid because the not-null predicate accepts a boolean as first operand
** `foo STARTS WITH (x IS NOT NULL)` is invalid because the prefix predicate does not accept a boolean as second operand
* `foo CONTAINS 'bar' IN list`
** `(foo CONTAINS 'bar') IN list` is valid because the list element containment predicate accepts a boolean as first operand
** `foo CONTAINS ('bar' IN list)` is invalid because the containment predicate does not accept a boolean as second operand

[IMPORTANT]
.Chains of type-incompatible operators
====
Chains of (partly) type-incompatible operators either

* do not have a valid operator tree at all, or
* have only one valid operator tree.
====

==== Chains of closed unary operators
Closed unary operators that allow chaining on the same precedence level are

* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=UnaryAddOrSubtractExpression[Arithmetic additive inverse]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=NotExpression[Boolean negation]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ListOperatorExpression[List slicing]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=NullOperatorExpression[Null predicates]

Example expressions are

* `--+-5`
* `NOT NOT NOT false`
* `list[4..17][2..5]`
* `x IS NOT NULL IS NOT NULL IS NULL`

[IMPORTANT]
.Chaining of closed unary operators
====
Chains of closed unary operators have an unambiguous operator tree.
====

[NOTE]
.Associativity of unary operators
====
The grammar defines if the operand of an unary operator is on the left end or on the right end of the operator's syntax.
This renders the operator left- or right associative, respectively.
Nevertheless, the grammar allows only one unambiguous operator tree, in both case.
====

==== Chains of binary operators
Binary operators that allow chaining with themselves of other operators of compatible result type on the same precedence level are

* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=PowerOfExpression[Exponentiation]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=MultiplyDivideModuloExpression[Multiplication]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=MultiplyDivideModuloExpression[Division]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=MultiplyDivideModuloExpression[Remainder (Modulo)]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=AddOrSubtractExpression[Addition]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=AddOrSubtractExpression[Substraction]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=AndExpression[Boolean conjunction]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=OrExpression[Boolean inclusive disjunction]
* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=OrExpression[Boolean exclusive disjunction]

Example expressions are

* `5 * 4 * 3`
* `5 ^ 4 ^ 3`
* `5 / 4 / 3`
* `5 % 4 % 3`
* `5 % 4 * 3`
* `5 - 4 - 3`
* `5 - 4 + 3`
* `5 + 4 + 3`
* `TRUE AND FALSE AND TRUE`
* `TRUE OR FALSE OR TRUE`
* `TRUE XOR FALSE XOR TRUE`

[IMPORTANT]
.Chains of binary operators
====
In chains of binary operators, the operator have left-to-right associativity, i.e. for every two operators, the operator appear earlier (more left) in the query string takes precedence and the operator tree is left-deep.
====

[NOTE]
.Associative binary operators
====
Where the semantics of operator is associative (e.g. addition), a right-deep operator tree produces the same result as a left-deep operator tree.
====

Consequently, the examples evaluate as follows:

[cols="2a,1a"]
|====
|
[source, cypher]
----
RETURN  5 * 4 * 3  AS a,
       (5 * 4)* 3  AS b,
        5 *(4 * 3) AS c
----
|
[options="header"]
!====
!a !b !c
!60!60!60
!====

|
[source, cypher]
----
RETURN  4 ^ 3 ^ 2  AS a,
       (4 ^ 3)^ 2  AS b,
        4 ^(3 ^ 2) AS c
----
|
[options="header"]
!====
! a      ! b      ! c
! 4096.0 ! 4096.0 ! 262144.0
!====

|
[source, cypher]
----
RETURN  5 / 4 / 3  AS a,
       (5 / 4)/ 3  AS b,
        5 /(4 / 3) AS c
----
|
[options="header"]
!====
! a ! b ! c
! 0 ! 0 ! 5
!====

|
[source, cypher]
----
RETURN  5 % 4 % 3  AS a,
       (5 % 4)% 3  AS b,
        5 %(4 % 3) AS c
----
|
[options="header"]
!====
! a ! b ! c
! 1 ! 1 ! 0
!====

|
[source, cypher]
----
RETURN  5 % 4 * 3  AS a,
       (5 % 4)* 3  AS b,
        5 %(4 * 3) AS c
----
|
[options="header"]
!====
! a ! b ! c
! 3 ! 3 ! 5
!====

|
[source, cypher]
----
RETURN  5 - 4 - 3  AS a,
       (5 - 4)- 3  AS b,
        5 -(4 - 3) AS c
----
|
[options="header"]
!====
! a  ! b  ! c
! -2 ! -2 ! 4
!====

|
[source, cypher]
----
RETURN  5 - 4 + 3  AS a,
       (5 - 4)+ 3  AS b,
        5 -(4 + 3) AS c
----
|
[options="header"]
!====
! a ! b ! c
! 4 ! 4 ! -2
!====

|
[source, cypher]
----
RETURN  5 + 4 + 3  AS a,
       (5 + 4)+ 3  AS b,
        5 +(4 + 3) AS c
----
|
[options="header"]
!====
!a !b !c
!12!12!12
!====

|
[source, cypher]
----
RETURN  TRUE AND FALSE  AND TRUE  AS a,
       (TRUE AND FALSE) AND TRUE  AS b,
        TRUE AND (FALSE AND TRUE) AS c
----
|
[options="header"]
!====
!a    !b    !c
!false!false!false
!====

|
[source, cypher]
----
RETURN  TRUE OR FALSE  OR TRUE  AS a,
       (TRUE OR FALSE) OR TRUE  AS b,
        TRUE OR (FALSE OR TRUE) AS c
----
|
[options="header"]
!====
!a   !b   !c
!true!true!true
!====

|
[source, cypher]
----
RETURN  TRUE XOR FALSE  XOR TRUE  AS a,
       (TRUE XOR FALSE) XOR TRUE  AS b,
        TRUE XOR (FALSE XOR TRUE) AS c
----
|
[options="header"]
!====
!a    !b    !c
!false!false!false
!====

|====

==== Chains with extra semantics
Operators whose chaining gives extra semantics are

* https://raw.githack.com/openCypher/openCypher/master/tools/grammar-production-links/grammarLink.html?p=ComparisonExpression[Comparison operations]

Example expression are

* `5 = 5 = 5`
* `5 = 4 <> 3`
* `5 <> 4 > 3`
* `5 > 4 > 3`
* `5 > 4 >= 3`
* `5 >= 4 < 3`

[IMPORTANT]
.Chains with extra semantics
====
Chains with extra semantics, either

a. form a flat operator tree of a single operator, or
b. their semantics is defined by a syntax transformation to an expression that has an unambiguous operator tree based on the other precedence rules stated in this document.
====

[NOTE]
.Associativity of comparison operators
====
Because of the extra semantics of a chain of comparison, the comparison operators do not have any observable associativity in openCypher.
====

Consequently, the examples evaluate as follows:

[cols="2a,1a"]
|====
|
[source, cypher]
----
RETURN  5 = 5 = 5  AS a,
       (5 = 5) AND (5 = 5) AS ax,
       (5 = 5)= 5  AS b,
        5 =(5 = 5) AS c
----
|
[options="header"]
!====
! a    ! ax   ! b     ! c
! true ! true ! false ! false
!====

|
[source, cypher]
----
RETURN  5 = 4 <> 3  AS a,
       (5 = 4) AND (4 <> 3) AS ax,
       (5 = 4)<> 3  AS b,
        5 =(4 <> 3) AS c
----
|
[options="header"]
!====
! a     ! ax    ! b    ! c
! false ! false ! true ! false
!====

|
[source, cypher]
----
RETURN  5 <> 4 > 3  AS a,
       (5 <> 4) AND (4 > 3) AS ax,
       (5 <> 4)> 3  AS b,
        5 <>(4 > 3) AS c
----
|
[options="header"]
!====
! a    ! ax   ! b    ! c
! true ! true ! null ! true
!====

|
[source, cypher]
----
RETURN  5 > 4 > 3  AS a,
       (5 > 4) AND (4 > 3) AS ax,
       (5 > 4)> 3  AS b,
        5 >(4 > 3) AS c
----
|
[options="header"]
!====
! a    ! ax   ! b    ! c
! true ! true ! null ! null
!====

|
[source, cypher]
----
RETURN  5 > 4 >= 3  AS a,
       (5 > 4) AND (4 >= 3) AS ax,
       (5 > 4)>= 3  AS b,
        5 >(4 >= 3) AS c
----
|
[options="header"]
!====
! a    ! ax   ! b    ! c
! true ! true ! null ! null
!====

|
[source, cypher]
----
RETURN  5 <= 4 < 3  AS a,
       (5 <= 4) AND (4 < 3) AS ax,
       (5 <= 4)< 3  AS b,
        5 <=(4 < 3) AS c
----
|
[options="header"]
!====
! a     ! ax    ! b    ! c
! false ! false ! null ! null
!====

|====

== What others do

openCypher has grammatically all operators in one precedence level hierarchy.
This is similar to how precedence is defined for more programming languages.
For instance:

* https://docs.oracle.com/javase/tutorial/java/nutsandbolts/operators.html[Java]
* https://docs.python.org/3/reference/expressions.html#operator-precedence[Python]
* https://en.cppreference.com/w/cpp/language/operator_precedence[C++]
* https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_Precedence#table[Javascript]

The precedence is similar.
Some minor difference:

* Programming languages have extra operators openCypher does not have, e.g. increment/decrement, bitwise operators, ternary operator, and assignments
* In Javascript exponentiation is right-to-left associative

SQL encodes large parts of the type compatibility of expressions in the grammar.
Hence, its precedence hierarchy is partition by type.
Within one type, SQL's precedence hierarchy is similar.

Chainable operators, such as addition, multiplication, etc. are left-to-right associative in SQL as well.
However, SQL directly encode that in the grammar, with left-recursive production rules, e.g.

[source, ebnf]
----
<numeric value expression> ::=
    <term>
  | <numeric value expression> <plus sign> <term>
  | <numeric value expression> <minus sign> <term>

<term> ::=
    <factor>
  | <term> <asterisk> <factor>
  | <term> <solidus> <factor>

<factor> ::=
  [ <sign> ] <numeric primary>
----

Most program and query language do not give chains of comparison operators extra semantics.
Typically, comparison operators are left-to-right associative.

== Benefits to this proposal

This CIP clarifies the precedence rules of openCypher.

== Caveats to this proposal

None known.