# Analyzing Data in Real Time with Azure Stream Analytics # ---  ## Overview ## Azure Stream Analytics is a cloud-based service for ingesting high-velocity data streaming from devices, sensors, applications, Web sites, and other data sources and analyzing that data in real time. It supports a SQL-like query language that works over dynamic data streams and makes analyzing constantly changing data no more difficult than performing queries on static data stored in traditional databases. With Azure Stream Analytics, you can set up jobs that analyze incoming data for anomalies or information of interest and record the results, present notifications on dashboards, or even fire off alerts to mobile devices. And all of it can be done at low cost and with a minimum of effort. Scenarios for the application of real-time data analytics are legion and include fraud detection, identity-theft protection, optimizing the allocation of resources (think of an Uber-like transportation service that sends drivers to areas of increasing demand *before* that demand peaks), click-stream analysis on Web sites, shopping suggestions on retail-sales sites, and countless others. Having the ability to process data *as it comes in* rather than waiting until after it has been aggregated offers a competitive advantage to businesses that are agile enough to make adjustments on the fly. In this lab, you'll create an Azure Stream Analytics job and use it to analyze data streaming in from simulated Internet of Things (IoT) devices. And you'll see how simple it is to monitor real-time data streams for information of significance to your research or business .And at the end, you will build a Web app that shows output from the Stream Analytics job in real time.  ### Objectives ### In this hands-on lab, you will learn how to: - Create Azure event hubs and use them for Stream Analytics input and output - Create a Stream Analytics job and test queries on sample data streams - Run a Stream Analytics job and perform queries on live data streams - Consume events from an event hub and display them in a Web app  ### Prerequisites ### The following is required to complete this hands-on lab: - An active Microsoft Azure subscription. If you don't have one, [sign up for a free trial](http://aka.ms/WATK-FreeTrial). - [Visual Studio 2015 Community edition](https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx) or higher ---  ## Exercises ## This hands-on lab includes the following exercises: - [Exercise 1: Create an event hub](#Exercise1) - [Exercise 2: Send events to the event hub](#Exercise2) - [Exercise 3: Create a Stream Analytics job](#Exercise3) - [Exercise 4: Prepare queries and test with sample data](#Exercise4) - [Exercise 5: Add an event hub for output](#Exercise5) - [Exercise 6: Build a real-time dashboard](#Exercise6) - [Exercise 7: Analyze a live data stream](#Exercise7) Estimated time to complete this lab: **60** minutes.  ## Exercise 1: Create an event hub ## Azure Stream Analytics supports several types of input, including input from Azure blobs and input from Azure event hubs. Of the two, the latter is typically more interesting because in the IoT world, data is easily transmitted to Azure event hubs through field gateways (for devices that are not IP-capable) or cloud gateways (for devices that *are* IP-capable), and a single Azure event hub can handle millions of events per second transmitted from devices spread throughout the world. In this exercise, you'll create an Azure event hub to provide input to Azure Stream Analytics and configure it to so that it can be accessed safely and securely by IoT devices and gateways. 1. In your browser, navigate to the [Azure Portal](https://portal.azure.com). If you are asked to sign in, do so using your Microsoft account. 1. In the portal, click **+ New**, followed by **Internet of Things** and **Event Hubs**. ![Adding a new event hub](Images/new-event-hub.png) _Adding a new event hub_ 1. Type a namespace name into the **Name** box. The name must be unique within Azure, so you will probably have to use something other than the name in the screen shot below. (A green check mark will appear in the box when the name you've entered is one that Azure will accept.) Select **Create new** under **Resource group** and enter the resource-group name "StreamAnalyticsResourceGroup" (without quotation marks). Choose the region closest to you in the **Location** drop-down, and then click the **Create** button. ![Creating a namespace](Images/create-namespace.png) _Creating a namespace_ 1. Click **Resource groups** in the ribbon on the left, and then click the "StreamAnalyticsResourceGroup" resource group created in the previous step. ![Opening the resource group](Images/open-resource-group.png) _Opening the resource group_ 1. Wait until "Deploying" changes to "Succeeded." Then click the namespace whose name you specified in Step 3. > Click the browser's **Refresh** button occasionally to update the deployment status. Clicking the **Refresh** button in the resource-group blade refreshes the list of resources in the resource group, but does not reliably update the deployment status. ![Opening the namespace](Images/open-namespace.png) _Opening the namespace_ 1. Click **+ Event Hub** to add an event hub to the namespace. ![Adding an event hub](Images/add-event-hub.png) _Adding an event hub_ 1. Type "inputhub" (without quotation marks) into the **Name** box. Then click the **Create** button. ![Creating an event hub](Images/create-event-hub.png) _Creating an event hub_ 1. Wait for the event hub to be created. Then scroll to the bottom of the blade and click the event hub name. ![Opening the event hub](Images/open-event-hub.png) _Opening the event hub_ 1. In order to transmit events to the event hub from an application or device, you need to create a shared-access policy that includes Send permission. To begin, click **Shared access policies**, and then click **+ Add**. ![Adding a shared-access policy](Images/new-shared-access-policy.png) _Adding a shared-access policy_ 1. Type "SendPolicy" (without quotation marks) into the **Policy name** box and check the **Send** box. Then click the **Create** button to create the new policy. ![Creating a send policy](Images/create-send-policy.png) _Creating a send policy_ 1. Wait a moment for **SendPolicy** to appear in the policies list, and then click it. ![Opening the policy](Images/open-send-policy.png) _Opening the policy_ 1. Click the **Copy** button to the right of the **CONNECTION STRING-PRIMARY KEY** box to copy the connection string containing the policy's shared-access key to the clipboard. Then temporarily save the connection string by pasting it into your favorite text editor. You'll need it in the next exercise. ![Copying the connection string to the clipboard](Images/copy-connection-string.png) _Copying the connection string to the clipboard_ You have created an event hub that can ingest events and be used as the source of input to a Stream Analytics job. You have also created a policy that allows holders of that policy to send events to the event hub, and a connection string that references that policy. The next step is to use the connection string to transmit events to the event hub.  ## Exercise 2: Send events to the event hub ## Applications, devices, and gateways can send events to Azure event hubs using the [Azure Event Hubs REST API](https://msdn.microsoft.com/en-us/library/azure/Dn790674.aspx) or the [Advanced Message Queuing Protocol](https://www.amqp.org/), or AMQP for short. In this exercise, you will write a Windows console app that uses AMQP to send events to the event hub you created in [Exercise 1](#Exercise1). Each event will represent a withdrawal from an ATM machine, and will contain relevant information such as the card number used for the withdrawal, the time and amount of the withdrawal, and a unique identifier for the ATM machine used. 1. Start Visual Studio 2015 and use the **File -> New -> Project** command to create a new Windows Console Application named "ATMEventGenerator." ![Creating a new console app](Images/mvc-new-console-app-2.jpg) _Creating a new console app_ 1. In the Solution Explorer window, right-click the **ATMEventGenerator** project and select **Manage NuGet Packages...** ![Managing NuGet Packages for the project](Images/mvc-manage-nuget-packages-3.jpg) _Managing NuGet Packages for the project_ > NuGet is a free and open-source package manager for Microsoft development platforms. It provides access to thousands of libraries, or *packages*, containing code to perform a variety of tasks. It is integrated into Visual Studio 2015, which makes it easy to add NuGet packages to your project and make a lot of things happen without writing a lot of code. 1. Click **Browse**. Then type "azure" (without quotation marks) into the search box. Click **WindowsAzure.ServiceBus** to select the Azure service-bus package from NuGet. Finally, click **Install** to install the latest stable version of the package. This package contains the APIs that your app will use to send events to the event hub. ![Installing WindowsAzure.ServiceBus](Images/mvc-install-servicebus-package.png) _Installing WindowsAzure.ServiceBus_ 1. If prompted to review changes, click **OK**. Optionally check **Do not show this again** so you won't be prompted again. ![Reviewing changes](Images/mvc-review-changes.png) _Reviewing changes_ 1. If prompted to accept a license for the package, click **I Accept**.. ![Accepting the package license](Images/mvc-accept-license.png) _Accepting the package license_ 1. Return to the search box and type "newtonsoft," again without quotation marks. Select **Newtonsoft.Json** and click **Install** to install the latest stable version of Json.NET. This package contains convenient APIs for generating and consuming JSON. If prompted to review changes, click **OK**. ![Installing Json.NET](Images/mvc-install-json.net-package.png) _Installing Json.NET_ 1. Open **Program.cs** and add the following ```open``` statements to the top of the file: ```F# open System open System.Text open Newtonsoft.Json open System.Runtime.Serialization open Microsoft.ServiceBus.Messaging ``` 1. Replace the rest of the file with the following implementation: ```F# type Transaction = {TransactionId: int; TransactionTime: String; DeviceId: int; CardNumber:int; Amount:float} [<EntryPoint>] let main argv = let probability = 0.01 let connectionString = @"your_connection_string_here" let random = new Random() let client = EventHubClient.CreateFromConnectionString(connectionString) let generateTransaction transactionid = let transactionTime = DateTime.UtcNow.ToString() let deviceId = 12345 + random.Next(0, 88888) let testCardNumber = 123456789 + random.Next(0,888888888) let cardNumber = match(random.NextDouble() < probability) with | true -> -1 | false -> testCardNumber let amount = random.Next(1, 20) * 20 |> float {TransactionId=transactionid;TransactionTime=transactionTime;DeviceId=deviceId;CardNumber=cardNumber;Amount=amount} [|0..1000|] |> Array.map(fun i -> Console.WriteLine("[{0}] Event transmitted",i); i) |> Array.map(fun i -> generateTransaction i) |> Array.map(fun t -> JsonConvert.SerializeObject(t)) |> Array.map(fun st -> Encoding.UTF8.GetBytes(st)) |> Array.map(fun b -> new EventData(b)) |> Array.iter(fun ed -> client.Send(ed)) 0 ``` 1. Replace *connection_string* in line 6 with the connection string you copied to the clipboard (and hopefully into your favorite text editor) in Exercise 1, Step 12. 2. Press **Ctrl+F5** to run the program and confirm that you see output similar to the following. Each line represents one event sent to the event hub, and events will roll by at a rate of several per second. (Rates will vary depending on your connection speed.) **Confirm that no exceptions are reported in the output**. ![Sending events to the event hub](Images/mvc-messages-sent-2.jpg) _Sending events to the event hub_ 3. Press **Ctrl+C** to stop the flow of events. Then press **Enter** to close the Command Prompt window. In real life, there would be real ATM machines sending events to the event hub. ATMEventGenerator simulates these events in software and sets the stage for you to build and test a Stream Analytics job that analyzes these events. If you need more than 1000 transactions sent, change the upper value of the array. For example, instead of [|0..1000|], use [|0..10000|] for 10,000 transactions to send.  ## Exercise 3: Create a Stream Analytics job ## In this exercise, you will use the Azure Portal to create a Stream Analytics job and connect it to the event hub you created in [Exercise 1](#Exercise1). You will also capture the raw data being passed to the Stream Analytics job from the event hub and examine its structure. 1. Return to the [Azure Portal](https://portal.azure.com) and click **+ New**, followed by **Internet of Things** and **Stream Analytics job**. ![Creating a Stream Analytics job](Images/new-stream-analytics-job.png) _Creating a Stream Analytics job_ 1. Type "ATMAnalytics" (without quotation marks) into the **Job name** box. Select **Use Existing** under **Resource group** and select the "StreamAnalyticsResourceGroup" resource group that you created in [Exercise 1](#Exercise1). Select the region nearest you for **Location**. (It is important to select the same region that you selected for the event hub in Exercise 1, because you're not charged for data that moves within a data center, but you typically *are* charged for data that moves *between* data centers. In addition, locating services that talk to each other in the same data center reduces latency.) Then click the **Create** button. ![Specifying parameters for the Stream Analytics job](Images/create-stream-analytics-job.png) _Specifying parameters for the Stream Analytics job_ 1. Click **Resource groups** in the ribbon on the left, and then click the "StreamAnalyticsResourceGroup" resource group. ![Opening the resource group](Images/open-resource-group.png) _Opening the resource group_ 1. Click **ATMAnalytics** to open the Stream Analytics job in the portal. ![Opening the Stream Analytics job](Images/open-stream-analytics-job.png) _Opening the Stream Analytics job_ 1. Click **Inputs** to add an input to the Stream Analytics job. ![Adding an input](Images/add-input-1.png) _Adding an input_ 1. Click **+ Add**. ![Adding an input](Images/add-input-2.png) _Adding an input_ 1. Type "Withdrawals" (without quotation marks) into the **Input alias** box. Make sure **Source Type** is set to **Data stream** and **Source** is set to **Event hub**. Also make sure **Subscription** is set to **Use event hub from current subscription**, **Service bus namespace** is set to the namespace you specified in Exercise 1, Step 3, and the event hub you created in Exercise 1 ("inputhub") is selected under **Event hub name**. Then select **RootManageSharedAccessKey** from the **Event hub policy name** drop-down and click the **Create** button at the bottom of the blade. ![Creating an input](Images/create-input.png) _Creating an input_ 1. After a few moments, the new input — "Withdrawals" — appears in the list of inputs for the Stream Analytics job. Click it to open a blade for it. ![Opening the input](Images/open-input.png) _Opening the input_ 1. Return to Visual Studio and start the ATMEventGenerator app again. 1. With ATMEventGenerator still running, return to the Azure Portal open in your browser and click **Sample Data**. ![Sampling input data](Images/sample-data-1.png) _Sampling input data_ 1. Click **OK** to begin sampling data from the input stream. ![Specifying sampling parameters](Images/sample-data-2.png) _Specifying sampling parameters_ 1. Wait a few seconds for sampling to complete, and when you are notified that the sample data can be downloaded, click to download it. ![Data sampling completed](Images/sample-data-3.png) _Data sampling completed_ 1. Click **Download** to download the data sampled from the input stream. ![Downloading sample data](Images/sample-data-4.png) _Downloading sample data_ 1. Save the JSON file that is downloaded to a location where you can easily find it. Then open the downloaded file in your favorite text editor and take a moment to examine its contents. How many rows (events) are represented in the sample data? What is the structure of each row — that is, what fields does each row contain? > If you find the output hard to digest since there are no line breaks, try pasting it into an online JSON viewer such as the one at https://jsonformatter.curiousconcept.com/. 1. Return to the console window in which ATMEventGenerator is running and press **Ctrl+C** to stop it. Then press **Enter** to close the window. You have connected a Stream Analytics job to an event hub and demonstrated that data is passed from one to the other. You have also sampled the data input to the Stream Analytics job and examined its structure. The next step is to do something with it — specifically, to bring the power of Azure Stream Analytics to bear on the data.  ## Exercise 4: Prepare queries and test with sample data ## Now that your job is set up, there's much more you can do with Stream Analytics than simply view the raw data presented to it. The whole point of Stream Analytics is being able to query the data in real time. In this exercise, you'll use the [Stream Analytics Query Language](https://msdn.microsoft.com/en-us/library/azure/Dn834998.aspx) to query a sample data set for potentially fraudulent ATM transactions. It is always a good idea to test your queries against sample data before deploying them against live data streams, because with sample data, you can verify that a known set of inputs produces the expected outputs. To flag potentially fraudulent withdrawals from ATMs, you will query for transactions performed with the same ATM card at different ATM machines within a specified time window (60 seconds). In real life, you would probably use a larger time window and perhaps even factor in the distance between ATM machines. However, a narrower time window is useful in a lab environment because it allows you to perform meaningful experiments in minutes rather than hours. 1. Begin by returning to the Stream Analytics job in the portal and clicking **Query**. ![Opening the query viewer](Images/add-query.png) _Opening the query viewer_ 1. Click the **ellipsis** (the three dots) to the right of **Withdrawals** and select **Upload sample data from file** from the menu. ![Uploading sample data for testing queries](Images/upload-test-data-1.png) _Uploading sample data for testing queries_ 1. Click the **folder** icon on the right and select the file named **Withdrawals.json** in this lab's "resources" directory. Then click **OK** to upload the file. ![Uploading Withdrawals.json](Images/upload-test-data-2.png) _Uploading Withdrawals.json_ 1. When the upload is complete, enter the following query, and then click the **Test** button to execute it against the sample data you uploaded: ```sql SELECT * FROM Withdrawals ``` > Where did the name "Withdrawals" come from? That's the alias you assigned to the event-hub input in the previous exercise. If you named it differently, you'll need to replace "Withdrawals" with the alias you used. ![Testing a query](Images/test-query-1.png) _Testing a query_ 1. Confirm that you see the output pictured below. The test data contains 607 rows. Each row has fields named TRANSACTIONID, TRANSACTIONTIME, DEVICEID, CARDNUMBER, and AMOUNT. DEVICEID is the ID of the ATM machine at which the transaction took place. AMOUNT is the amount of cash withdrawn from the ATM. ![Query results](Images/query-results-1.png) _Query results_ 1. Suppose you only wanted to view transactions for amounts between 200 and 300, inclusive. Furthermore, suppose you wanted to clean up the output by assigning your own column names and excluding the TRANSACTIONID column. Enter the following query and click **Test** again to execute it. ```sql SELECT TransactionTime as [Time of Transaction], DeviceID as [ATM], CardNumber as [Card Number], Amount as [Amount] FROM Withdrawals WHERE Amount >= 200 and Amount <= 300 ``` 1. Confirm that the query generated the following output: ![Customizing the output](Images/query-results-2.png) _Customizing the output_ 1. One of the key features of the Stream Analytics Query Language is its ability to group results using windows of time whose length you specify. To demonstrate, enter the following query to count the number of transactions taking place each minute and click **Test** to execute it: ```sql SELECT System.Timestamp as [Time Ending], COUNT(*) AS [Number of Transactions] FROM Withdrawals TIMESTAMP BY TransactionTime GROUP BY TumblingWindow(n, 1) ``` > TIMESTAMP BY is an important element of the Stream Analytics Query Language. If it was omitted from the query above, you would be querying for the number of transactions that arrived *at the event hub* each minute rather than the number of transactions that occurred in each 1-minute interval. TIMESTAMP BY allows you to specify a field in the input stream as the event time. 1. Confirm that you see the output below: ![Querying for the number of transactions per minute](Images/query-results-3.png) _Querying for the number of transactions per minute_ 1. Now it's time to query the test data for potentially fraudulent transactions — transactions involving the same ATM card but different ATM machines that take place within 60 seconds of each other. *This is the query you will use in a subsequent exercise against a live data stream*. Enter the following query and click **Test** to execute it: ```sql SELECT W1.CardNumber as [CardNumber], W1.DeviceID as [ATM1], W2.DeviceID as [ATM2], W1.TransactionTime as [Time1], W2.TransactionTime as [Time2] FROM Withdrawals W1 TIMESTAMP BY TransactionTime JOIN Withdrawals W2 TIMESTAMP BY TransactionTime ON W1.CardNumber = W2.CardNumber AND DATEDIFF(ss, W1, W2) BETWEEN 0 and 60 WHERE W1.DeviceID != W2.DeviceID ``` 1. This time the output should contain just three rows, each representing two transactions performed with one ATM card at two different locations within 60 seconds of each other: ![Detecting potentially fraudulent transactions](Images/query-results-4.png) _Detecting potentially fraudulent transactions_ 1. Click the **Save** button at the top of the blade to save the query. Then click **Yes** when asked to confirm. ![Saving the query](Images/save-query.png) _Saving the query_ With the query now formulated, tested against a set of sample data, and saved, it's almost time to deploy it against a live data stream to produce a running record of potentially fraudulent transactions. But first, you need to add an output to the Stream Analytics job.  ## Exercise 5: Add an event hub for output ## Being able to run queries and see the results in the portal is great for testing, but the whole point of Stream Analytics is being able to query live data streams. Azure Stream Analytics supports a variety of output types, including blobs, Azure SQL databases, and event hubs. You can designate blob storage as the destination for output from a Stream Analytics job to generate a persistent record of query results. Another useful scenario involves using event hubs as output. Because software can subscribe to events from event hubs, developers can build custom applications that show Stream Analytics output in real time. In this exercise, you will create an event hub for output, and then configure the Stream Analytics job to write output to the event hub. In the next exercise, you will write an app that connects to the output event hub. 1. Return to the "StreamAnalyticsResourceGroup" resource group and click the event-hub namespace you created in [Exercise 1](#Exercise1). ![Opening the namespace](Images/open-namespace-2.png) _Opening the namespace_ 1. Click **+ Event Hub** to add an event hub to the namespace. ![Adding an event hub](Images/add-event-hub.png) _Adding an event hub_ 1. Type "outputhub" (without quotation marks) into the **Name** box. Then click the **Create** button. ![Creating an event hub](Images/create-event-hub-2.png) _Creating an event hub_ 1. Wait a moment for the event hub to be created. Then scroll to the bottom of the blade and click the event hub name. ![Opening the event hub](Images/open-event-hub-2.png) _Opening the event hub_ 1. In order to receive events from an event hub, you need to create a shared-access policy that includes Listen permission. To begin, click **Shared access policies**, and then click **+ Add**. ![Adding a shared-access policy](Images/new-shared-access-policy-2.png) _Adding a shared-access policy_ 1. Type "ReceivePolicy" (without quotation marks) into the **Policy name** box and check the **Listen** box. Then click the **Create** button to create the new policy. ![Creating a send policy](Images/create-listen-policy.png) _Creating a send policy_ 1. Wait a moment for **ReceivePolicy** to appear in the policies list, and then click it. ![Opening the policy](Images/open-receive-policy.png) _Opening the policy_ 1. Click the **Copy** button to the right of the **CONNECTION STRING-PRIMARY KEY** box to copy the connection string containing the policy's shared-access key to the clipboard. Then temporarily save the connection string by pasting it into your favorite text editor. You'll need it in the next exercise. ![Copying the connection string to the clipboard](Images/copy-connection-string-2.png) _Copying the connection string to the clipboard_ 1. Return to the "StreamAnalyticsResourceGroup" resource group and click **ATMAnalytics** to open the Stream Analytics job in the portal. ![Opening the Stream Analytics job](Images/open-stream-analytics-job.png) _Opening the Stream Analytics job_ 1. Click **Outputs**. ![Creating an output](Images/add-output-1.png) _Creating an output_ 1. Click **+ Add**. ![Adding an output](Images/add-output-2.png) _Adding an output_ 1. Type "FlaggedWithdrawals" (without quotation marks) into the **Output alias** box. Make sure **Sink** is set to **Event hub** and **Import option** is set to **Use event hub from current subscription**. Under **Service bus namespace**, select the namespace you created in [Exercise1](#Exercise1), and select **outputhub** from the **Event hub name** list. Set **Event hub policy name** to **RootManageSharedAccessKey**, and then click **Create**. ![Creating an output](Images/create-output-2.png) _Creating an output_ Now that you have directed the output from the Stream Analytics job to an event hub, the next task is to write an app that consumes events from that event hub.  ## Exercise 6: Build a real-time dashboard ## In this exercise, you will use Visual Studio to write an ASP.NET MVC Web app that connects to the output event hub and displays real-time notifications of potentially fraudulent transactions. You will also create a storage account required by the dashboard. 1. Return to the Azure Portal and click **+ New** in the ribbon on the left. Then click **Storage**, followed by **Storage account**. ![Adding a storage account](Images/new-storage-account.png) _Adding a storage account_ 1. In the ensuing blade, enter a name for the new storage account in **Name** field. > Storage account names must be 3 to 24 characters in length and can only contain numbers and lowercase letters. In addition, the name you enter must be unique within Azure. Once you have a unique name that Azure will accept (as indicated by the green check mark in the **Name** field), select **Use existing** under **Resource group** and select the resource group named "StreamAnalyticsResourceGroup" so the storage account will belong to the same resource group as the Stream Analytics job and the event hub. Select the location nearest you (the same one you selected for the event hub in [Exercise 1](#Exercise1) and the Stream Analytics job in [Exercise 3](#Exercise3)) in the **Location** box. Then click the **Create** button at the bottom of the blade. ![Creating a storage account](Images/create-storage-account.png) _Creating a storage account_ 1. Click **Resource groups** in the ribbon on the left, and then click the "StreamAnalyticsResourceGroup" resource group. ![Opening the resource group](Images/open-resource-group.png) _Opening the resource group_ 1. Click the storage account you created in Step 2. ![Opening the storage account](Images/open-storage-account.png) _Opening the storage account_ 1. Click **Access keys**, and click the **Copy** button to the right of the **key1** box to the copy the storage account's access key to the clipboard. Then temporarily save the access key by pasting it into your favorite text editor. You'll need it later in this exercise. ![Copying the access key](Images/copy-storage-access-key.png) _Copying the access key_ 1. Start a new instance of Visual Studio 2015 and use the **File -> New -> Project** command to create a new ASP.NET Web Application named "ATMDashboard.UX." ![Creating a new Web app](Images/mvc-new-web-application-1.jpg) _Creating a new Web app_ 1. In the "New ASP.NET Project" dialog, select **MVC** and check the **Web API** box. If **Host in the cloud** is checked, uncheck it. (For testing purposes, you will run this Web app locally.) Then click **OK**. ![Specifying parameters for the Web app](Images/mvc-new-web-application-2.png) _Specifying parameters for the Web app_ 1. In the Solution Explorer window, right-click **File -> New -> Project** command to create a new FSharp Library Project named "ATMDashboard" ![Creating a new Library app](Images/mvc-new-library-application-1.jpg) _Creating a new Library app 1_ ![Creating a new Library app](Images/mvc-new-library-application-2.jpg) _Creating a new Library app 2_ 1. In the Solution Explorer window, right-click the **ATMDashboard.UX** project and select **Add Reference...** and Select the **ATMDashboard** project. ![Referencing a Library app](Images/mvc-reference-library-application-1.jpg) _Referencing a Library app 1_ ![Referencing a Library app](Images/mvc-reference-library-application-2.jpg) _Referencing a Library app 2_ 1. In the Solution Explorer window, right-click the **ATMDashboard** project and select **Manage NuGet Packages...** ![Managing NuGet Packages for the project](Images/mvc-manage-nuget-packages-2.png) _Managing NuGet Packages for the project_ 1. Click **Browse**. Then type "eventprocessor" (without quotation marks) into the search box. Click **Microsoft.Azure.ServiceBus.EventProcessorHost** to select the Azure EventProcessorHost package from NuGet. Finally, click **Install** to install the latest stable version of the package. This package contains the APIs that your app will use to receive events from the output event hub. Click **OK** if you're prompted review changes, and **I Accept** when prompted to accept licenses for the downloaded packages. ![Installing EventProcessorHost](Images/mvc-install-eventprocessorhost.png) _Installing EventProcessorHost_ 1. Right-click the library project in the Solution Explorer window and Select the Library1 file and rename it to **ATMEvents.fs**. ![Adding the ATMEvent class](Images/mvc-rename-event-file.jpg) _Rename Events File_ 1. Add the following ```namespace``` and ```open``` statements at the top of the file: ```F# namespace ATMDashboard open System open System.Text open Newtonsoft.Json open System.Diagnostics open System.Threading.Tasks open System.Collections.Generic open Microsoft.ServiceBus.Messaging ``` 1. Implement the ```ATMEvent``` record type as follows: ```F# type ATMEvent = { CardNumber:string; ATM1:string;ATM2:string; Time1:string;Time2:string} ``` 1. Below the ATMEvent type, add the ATMEventAggregator implementation: ```F# type ATMEventAggregator() = static member events = List<ATMEvent>() static member LogEvent(event) = match ATMEventAggregator.events.Count < 1024 with | true -> ATMEventAggregator.events.Add(event) | false -> () static member GetLoggedEvents() = let eventArray = ATMEventAggregator.events.ToArray() ATMEventAggregator.events.Clear() eventArray ``` 1. Below the ATMEventAggregator type, add the SimpleEventProcessor implementation: ```F# type SimpleEventProcessor() = let checkpointStopWatch = new Stopwatch() interface IEventProcessor with member this.CloseAsync(context, reason) = Debug.WriteLine("Processor Shutting Down. Partition '{0}', Reason: '{1}'.", context.Lease.PartitionId, reason) match reason with | CloseReason.Shutdown -> context.CheckpointAsync() | _ -> Task.CompletedTask member this.OpenAsync(context) = Debug.WriteLine("SimpleEventProcessor initialized. Partition: '{0}', Offset: '{1}'", context.Lease.PartitionId, context.Lease.Offset) checkpointStopWatch.Start() Task.CompletedTask member this.ProcessEventsAsync(context, messages) = messages |> Seq.map(fun ed -> Encoding.UTF8.GetString(ed.GetBytes())) |> Seq.map(fun b -> Debug.WriteLine(String.Format("Message received. Partition: '{0}', Data: '{1}'", context.Lease.PartitionId, b)); b) |> Seq.map(fun b -> JsonConvert.DeserializeObject<ATMEvent>(b)) |> Seq.iter(fun e -> ATMEventAggregator.LogEvent(e)) match checkpointStopWatch.Elapsed > TimeSpan.FromMinutes(5.0) with | true -> checkpointStopWatch.Restart() context.CheckpointAsync() | false -> Task.CompletedTask ``` 1. Compile the ATMEvents project and open the ATMEvents.UX project: 1. Open **Global.asax.cs** and add the following ```using``` statements: ```C# using Microsoft.ServiceBus.Messaging; using System.Diagnostics; using ATMEvents; ``` 1. In **Global.asax.cs**, add the following statements to the end of the ```Application_Start``` method: ```C# string eventHubConnectionString = "connection_string"; string eventHubName = "outputhub"; string storageAccountName = "storage_account_name"; string storageAccountKey = "storage_account_key"; string storageConnectionString = string.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", storageAccountName, storageAccountKey); string eventProcessorHostName = Guid.NewGuid().ToString(); EventProcessorHost eventProcessorHost = new EventProcessorHost(eventProcessorHostName, eventHubName, EventHubConsumerGroup.DefaultGroupName, eventHubConnectionString, storageConnectionString); Debug.WriteLine("Registering EventProcessor..."); var options = new EventProcessorOptions(); options.ExceptionReceived += (sender, e) => { Debug.WriteLine(e.Exception); }; eventProcessorHost.RegisterEventProcessorAsync<SimpleEventProcessor>(options).Wait(); Debug.WriteLine("Receiving..."); ``` 1. Replace *connection_string* in line 1 with the connection string you copied to the clipboard (and hopefully into your favorite text editor) in Exercise 5, Step 8. 1. Remove ";EntityPath=outputhub" from the end of the connection string. The ```eventHubConnectionString``` variable declaration should now look something like this: ```C# string eventHubConnectionString = "Endpoint=sb://streamanalytics-lab.servicebus.windows.net/;SharedAccessKeyName=ReceivePolicy;SharedAccessKey=3XHaU3pm5t82WxA43hUM/bWa7kQmuAsqzJ1rVJq3Qv0="; ``` 1. Replace *storage_account_name* in line 3 with the the name of the storage account you created in Step 2 of this exercise, and *storage_account_key* in line 4 with the access key you saved in Step 5. > This storage account has nothing to do with the Stream Analytics job; it's used by the EventProcessorHost class. 1. In Solution Explorer, right-click the "Controllers" folder and use the **Add -> Controller** command to add an empty Web API 2 controller. ![Adding a Web API Controller](Images/mvc-add-api-controller.png) _Adding a Web API Controller_ 1. Name the controller "EventsController" (without quotation marks). Then click **Add**. ![Naming the Web API Controller](Images/mvc-add-controller-dialog.png) _Naming the Web API Controller_ 1. Add the following method to the ```EventsController``` class: ```C# public ATMEvent[] GetEvents() { return ATMEventAggregator.GetLoggedEvents(); } ``` > The controller you just added is a Web API controller. It exposes a REST method that can be called over HTTP to retrieve the latest events that ATMEventAggregator received from the event hub. 1. Open **Index.cshtml** in the project's "Views/Home" folder and replace its contents with the following statements: ```HTML @{ ViewBag.Title = "Home Page"; } <div class="jumbotron"> <h1>ATM Dashboard</h1> <p class="lead">The table below lists potentially fraudulent ATM transactions and is updated every 5 seconds.</p> </div> <div class="row"> <div class="col-xs-12"> <table id="output" style="width: 100%"> <tr> <td><b>Card Number</b></td> <td><b>ATM 1</b></td> <td><b>ATM 2</b></td> </tr> </table> </div> </div> @section scripts { <script type="text/javascript"> $(function () { // Check for new events every 5 seconds window.setInterval(function () { $.ajax({ url: "/api/events", success: function (result) { for (i = 0; i < result.length; i++) { $("#output tr:last").after("<tr><td>" + result[i]["cardnumber"] + "</td><td>" + result[i]["atm1"] + "</td><td>" + result[i]["atm2"] + "</td></tr>"); } }, error: function (xhr, status, error) { if (status == "error") { console.log("Error: " + xhr.status); } }, dataType: "json" }); }, 5000); }); </script> } ``` > See what's happening here? In addition to modifying the view's UI to include an HTML table in which ATM events can be displayed, you added a script block that uses jQuery's $.ajax method to call back to the server every 5 seconds. The endpoint for the call is the method you implemented in the Web API controller in the previous step. When that method returns one or more events, rows are added to the table to display them. 1. Go to the **Build** menu at the top of the Visual Studio window and use the **Build Solution** command to build the solution. Correct any build errors that are reported, and then press **Ctrl+F5** to launch the application in your browser. Confirm that the application looks like this: ![The ATM dashboard](Images/mvc-atm-dashboard.png) _The ATM dashboard_ Leave the browser open with the application running. And don't be alarmed if a few events show up the first few seconds the dashboard is open. Those are simply events that the event hub cached. You'll be generating new events in the next exercise.  ## Exercise 7: Analyze a live data stream ## Almost there! Now it's time to run the Stream Analytics job and see the output in your dashboard. 1. Return to the Azure Portal and open the blade for the Stream Analytics job. Then click **Start**. ![Starting the Stream Analytics job](Images/start-stream-analytics-job-1.png) _Starting the Stream Analytics job_ 1. Make sure **Job output start time** is set to **Now**, and then click the **Start** button to start running the job. ![Specifying the job start time](Images/start-stream-analytics-job-2.png) _Specifying the job start time_ 1. Wait until the Stream Analytics job starts. Then return to the instance of Visual Studio in which the ATMEventGenerator project is open and start ATMEventGenerator again to pump events into the input event hub. > If you'd like, you can run several instances of ATMEventGenerator simultaneously to increase the volume of events. 1. Return to the ATMDashboard application running in your browser and watch for a few minutes. Every so often, a line should appear representing a potentially fraudulent transaction and identifying the ATM card number and the ATMs at which the card was used. ![Potentially fraudulent transactions](Images/mvc-atm-dashboard-output.png) _Potentially fraudulent transactions_ 1. Return to the Stream Analytics job in the portal and click the **Stop** button to stop it. Then click **Yes** when asked if you're sure you want to stop the job. ![Stopping the Stream Analytics job](Images/stop-stream-analytics-job.png) _Stopping the Stream Analytics job_ 1. Return to the console window in which ATMEventGenerator is running and press **Ctrl+C** to terminate it. You now have a Web app that displays output from a Stream Analytics job in near real-time. There are other ways to build such dashboards, including [Microsoft Power BI](https://powerbi.microsoft.com/). With Power BI, you can create dashboards that render output from Stream Analytics jobs without writing any code. For more information, refer to [Stream Analytics & Power BI: A real-time analytics dashboard for streaming data](https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-power-bi-dashboard/).  ## Summary ## Azure Stream Analytics is a powerful tool for analyzing live data streams from IoT devices or anything else that's capable of transmitting data. In this lab, you got a first-hand look at Stream Analytics as well as Azure event hubs. Among other things, you learned how to: - Create an Azure event hub and use it as a Stream Analytics input - Write a C# app that transmits events to an event hub - Create a Stream Analytics job and test queries on sample data streams - Run a Stream Analytics job and perform queries on live data streams - Create a rule (query) that detects anomalies in streaming data - Display output from a Stream Analytics job in a Web app One drawback to hard-coding rules into Stream Analytics is that rules don't "learn" from the data streams, which can lead to false positives in anomaly detection. If this concerns you, read the article entitled [Anomaly Detection – Using Machine Learning to Detect Abnormalities in Time Series Data](http://blogs.technet.com/b/machinelearning/archive/2014/11/05/anomaly-detection-using-machine-learning-to-detect-abnormalities-in-time-series-data.aspx) in the Azure team's Machine Learning blog. In it, they present an anomaly detection service accessible through a REST API that uses Azure Machine Learning to learn from the data presented to it. Imagine combining the power of Stream Analytics to extract information from real-time data streams with the power of Azure Machine Learning to learn from that information and refine the analytics on the fly. This is precisely the type of solution that Microsoft Azure empowers you to build! --- Copyright 2017 Microsoft Corporation. All rights reserved. Except where otherwise noted, these materials are licensed under the terms of the MIT License. You may use them according to the license as is most appropriate for your project. The terms of this license can be found at https://opensource.org/licenses/MIT.


Limited Retention Event Stream
Many use cases allow for Events to be removed from an Event Stream in order to preserve space and prevent the reading of stale data.
Problem
How can we remove events from an Event Stream based on a criteria, such as event age or Event Stream size?
Solution
The solution for limited retention will depend on the Event Streaming Platform. Most platforms will allow for deletion of events using an API or with an automated process configured within the platform itself. Because Event Streams are modeled as immutable event logs in the Event Store, events will be removed from the beginning of an Event Stream moving forward (i.e., oldest events are being removed first). Event Processing Applications that are reading from the stream will not be given the deleted events.
Implementation
Apache Kafka® implements a Limited Retention Event Stream by default. With Kafka, Event Streams are modeled as Topics. Kafka provides two types of retention policy, which can be configured on a per-topic basis or as a default for new topics.
Time-Based Retention
With time-based retention, events will be removed from the topic after the event timestamp indicates an event is older than the configured log retention time. On Kafka this is configured with the log.retention.hours
setting, which can be set as a default to apply to all topics or on a per-topic basis. Additionally, Kafka respects a log.retention.minutes
and log.retention.ms
settings to define shorter retention periods.
The following example sets the retention period of a topic to one year:
log.retention.hours=8760
For more guidance and configuring time-based data retention, see the Kafka Broker Configurations documentation for default settings, or the Modifying Topics section for modifying existing topics.
Size-Based Retention
With size-based retention, events will begin to be removed from the topic once the total size of the topic violates the configured maximum size. Kafka supports a log.retention.bytes
configure. For example, to configure the maximum size of a topic to 100GB you could set the configuration as follows:
log.retention.bytes=107374127424
For more guidance and configuring size-based data retention, see the Kafka Broker Configurations documentation for default settings, or the Modifying Topics section for modifying existing topics.
How Events Are Being Removed From an Event Stream
For either method of configuring retention, Kafka does not immediately remove events one by one when they violate the configured retention settings. To understand how they are removed, we first need to explain that Kafka topics are further broken down into topic partitions (see Partitioned Placement). Partitions themselves are further divided into files called segments. Segments represent a sequence of the events in a particular partition, and these segments are what is being removed once a violation of the retention policy has occurred. We can further fine-tune the removal algorithm with additional settings such as log.retention.check.interval.ms
and segment configuration, such as log.segment.bytes
.
Considerations
- We should account for failure scenarios when configuring event streams with limited retention. Our applications should have enough time to read and process events before they are being removed, which means the configured retention must cover the time frames of potential outages experienced by our applications. For example, if an Operations team has an SLA of 2 business days (Mon to Fri) for a non-mission-critical Kafka use case, then retention should be set to at least 4 days to cover for incidents that happen right before or during the weekends.
- Similarly, if we want to allow applications to reprocess the same events multiple times (often called the ability to reprocess historical data) for use cases such as A/B testing or machine learning, then we must increase the retention settings accordingly. That's why it is common in practice to configure Kafka topics with long retention periods (months or years).
References
- This pattern is similar to Message Expiration in Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf
- Apache Kafka 101: Introduction provides a primer on "What is Kafka, and how does it work?"
- A related pattern is the Infinite Retention Event Stream pattern which details Event Streams that stores events indefinitely.


YIPFILTER
Section: User Commands (1)
Updated: July 2014
Index
Return to Main Contents
NAME
yipfilter - A powerfull, easy to use host firewall for IP blacklisting (and whitelisting too!)
SYNOPSIS
yipfilter
COMMAND -s SET [-hvd] [-p PORT] [-t TYPE] [-f FILE] [-e TIMEOUT] [-E ELEMENT] [-m MAXELEM]
DESCRIPTION
yipfilter is program to create and manage host based IP blacklists and whitelists to control inbound TCP traffic. An application that determines an abusive IP can add the IP to yipfilter blacklist to block inbound traffic from the IP. yipfilter is built on top of iptables and ipset that provides a simple interface to control inbound TCP traffic.
COMMANDS:
- allow -s SET
-
Creates a whitelist set
- deny -s SET
-
Creates a blacklist set
- destroy -s SET
-
Delete/Remove a set
- list [-s SET]
-
List the entries of a named set or all sets
- add -s SET
-
Add one or more ip/netblocks to an existing set
- del -s SET
-
Delete entry from the named set
- test -s SET -E ELEM
-
Test entry in the named set
- reset
-
Remove/destroy all filter sets from the system
- save -s SET
-
Save current settings to stdout
- restore
-
Restore a saved state from the given state file (-f)
Use restore for batch addition of ips
OPTIONS:
- -s SET
-
The name of the filter set
- -p PORT[,PORT,..]
-
The destination TCP port number for allow/deny. Default
port is 443. You also can specify multiple ports
seperated by ',' eg. 443,80,4080
- -t TYPE
-
{net | ip} Address family. IPv4 netblock (CIDR) address
or a host IPv4 address. Default is 'ip'.
- -f FILE
-
Input file with the list of IPv4 ips or ip net blocks.
With no FILE or when FILE is -, read standard input.
No IPv6 support in this release
- -w http[s]
-
HTTP[S] location. (not supported yet)
- -e TIMEOUT
-
Expiry timeout in seconds. Eg. block ips 'n' seconds
- -E ELEMENT
-
One ip or a netblock, primarly used with 'test' command
-E and -f options are mutually exclusive
- -m MAXELEM
-
The maximal number of elements which can be stored in
the set, default 65536.
- -v
-
Verbose mode. Can be used multiple times for increased
verbosity
- -d
-
Dryrun - Check to see the input IPs/netblocks are valid
(not supported yet)
- -h
-
Display this help and exit
EXAMPLES:
-
yipfilter deny -s blacklist -p 443 -t ip -f blacklist.txt
-
yipfilter destroy -s blacklist
-
yipfilter list -s blacklist
-
yipfilter list
-
yipfilter add -s blacklist -e 600 -E <IP_ADDRESS>
-
yipfilter test -s blacklist -E <IP_ADDRESS>
-
echo "<IP_ADDRESS>" | yipfilter add -s blacklist -e 600
-
yipfilter save -s blacklist > blacklist.yipf
-
yipfilter restore -s blacklist -f blacklist.yipf
REPORTING BUGS
Report bugs to<EMAIL_ADDRESS>COPYRIGHT
Copyright (C) 2014 Yahoo Inc.
Written by Binu P. Ramakrishnan
SEE ALSO
iptables(8) ipset(8)
The full documentation for
yipfilter
is maintained as a Texinfo manual. If the
info
and
yipfilter
programs are properly installed at your site, the command
-
info yipfilter
should give you access to the complete manual.
Index
- NAME
-
- SYNOPSIS
-
- DESCRIPTION
-
- COMMANDS:
-
- OPTIONS:
-
- EXAMPLES:
-
- REPORTING BUGS
-
- COPYRIGHT
-
- SEE ALSO
-
This document was created by
man2html,
using the manual pages.
Time: 23:51:14 GMT, September 21, 2014


| Class and Description |
|---|
| Layer
A neural network layer.
|
| Class and Description |
|---|
| AbstractLSTM
LSTM recurrent net, based on Graves: Supervised Sequence Labelling with Recurrent Neural Networks
http://www.cs.toronto.edu/~graves/phd.pdf
|
| AbstractLSTM.Builder |
| ActivationLayer
Activation layer is a simple layer that applies the specified activation function to the input activations
|
| AutoEncoder
Autoencoder layer.
|
| AutoEncoder.Builder |
| BaseLayer
A neural network layer.
|
| BaseLayer.Builder |
| BaseOutputLayer |
| BaseOutputLayer.Builder |
| BasePretrainNetwork |
| BatchNormalization
Batch normalization layer
See: Ioffe and Szegedy, 2014, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift https://arxiv.org/abs/1502.03167 |
| BatchNormalization.Builder |
| CenterLossOutputLayer
Center loss is similar to triplet loss except that it enforces intraclass consistency and doesn't require feed
forward of multiple examples.
|
| CenterLossOutputLayer.Builder |
| ConvolutionLayer
2D Convolution layer (for example, spatial convolution over images).
|
| ConvolutionLayer.BaseConvBuilder |
| ConvolutionLayer.Builder |
| Deconvolution2D
2D deconvolution layer configuration
Deconvolutions are also known as transpose convolutions or fractionally strided convolutions. |
| Deconvolution2D.Builder |
| DenseLayer
Dense layer: a standard fully connected feed forward layer
|
| DenseLayer.Builder |
| DropoutLayer
Dropout layer.
|
| EmbeddingLayer
Embedding layer: feed-forward layer that expects single integers per example as input (class numbers, in range 0 to
numClass-1) as input.
|
| EmbeddingLayer.Builder |
| FeedForwardLayer
Created by jeffreytang on 7/21/15.
|
| FeedForwardLayer.Builder |
| GlobalPoolingLayer
Global pooling layer - used to do pooling over time for RNNs, and 2d pooling for CNNs.
|
| GravesBidirectionalLSTM
Deprecated.
use
Bidirectional instead. With the Bidirectional
layer wrapper you can make any recurrent layer bidirectional, in particular GravesLSTM. Note that this layer adds the
output of both directions, which translates into "ADD" mode in Bidirectional.
Usage: .layer(new Bidirectional(Bidirectional.Mode.ADD, new GravesLSTM.Builder()....build())) |
| GravesBidirectionalLSTM.Builder
Deprecated.
|
| GravesLSTM
Deprecated.
Will be eventually removed. Use
LSTM instead, which has similar prediction accuracy, but supports
CuDNN for faster network training on CUDA (Nvidia) GPUs |
| GravesLSTM.Builder
Deprecated.
|
| Layer
A neural network layer.
|
| Layer.Builder |
| LocalResponseNormalization
Local response normalization layer
See section 3.3 of http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf |
| LocalResponseNormalization.Builder |
| LossLayer
LossLayer is a flexible output layer that performs a loss function on an input without MLP logic.
|
| LSTM
LSTM recurrent neural network layer without peephole connections.
|
| LSTM.Builder |
| OutputLayer
Output layer used for training via backpropagation based on labels and a specified loss function.
|
| OutputLayer.Builder |
| PoolingType
Pooling type:
MAX: Max pooling - output is the maximum value of the input values AVG: Average pooling - output is the average value of the input values SUM: Sum pooling - output is the sum of the input values PNORM: P-norm pooling |
| RnnOutputLayer
A version of
OutputLayer for recurrent neural networks. |
| RnnOutputLayer.Builder |
| SeparableConvolution2D
2D Separable convolution layer configuration.
|
| SeparableConvolution2D.Builder |
| SubsamplingLayer
Subsampling layer also referred to as pooling in convolution neural nets
Supports the following pooling types: MAX, AVG, SUM, PNORM
|
| SubsamplingLayer.Builder |
| SubsamplingLayer.PoolingType |
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Class and Description |
|---|
| ConvolutionLayer.AlgoMode
The "PREFER_FASTEST" mode will pick the fastest algorithm for the specified parameters from the
ConvolutionLayer.FwdAlgo ,
ConvolutionLayer.BwdFilterAlgo , and ConvolutionLayer.BwdDataAlgo lists, but they may be very memory intensive, so if weird errors
occur when using cuDNN, please try the "NO_WORKSPACE" mode. |
| Layer
A neural network layer.
|
| Class and Description |
|---|
| Convolution3D.DataFormat
An optional dataFormat: "NDHWC" or "NCDHW".
|
| Class and Description |
|---|
| AbstractLSTM
LSTM recurrent net, based on Graves: Supervised Sequence Labelling with Recurrent Neural Networks
http://www.cs.toronto.edu/~graves/phd.pdf
|
| AbstractLSTM.Builder |
| ActivationLayer
Activation layer is a simple layer that applies the specified activation function to the input activations
|
| ActivationLayer.Builder |
| AutoEncoder
Autoencoder layer.
|
| AutoEncoder.Builder |
| BaseLayer
A neural network layer.
|
| BaseLayer.Builder |
| BaseOutputLayer |
| BaseOutputLayer.Builder |
| BasePretrainNetwork |
| BasePretrainNetwork.Builder |
| BaseRecurrentLayer |
| BaseRecurrentLayer.Builder |
| BaseUpsamplingLayer
Upsampling base layer
|
| BaseUpsamplingLayer.UpsamplingBuilder |
| BatchNormalization
Batch normalization layer
See: Ioffe and Szegedy, 2014, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift https://arxiv.org/abs/1502.03167 |
| BatchNormalization.Builder |
| CapsuleLayer.Builder |
| CapsuleStrengthLayer.Builder |
| CenterLossOutputLayer
Center loss is similar to triplet loss except that it enforces intraclass consistency and doesn't require feed
forward of multiple examples.
|
| CenterLossOutputLayer.Builder |
| Cnn3DLossLayer
3D Convolutional Neural Network Loss Layer.
|
| Cnn3DLossLayer.Builder |
| CnnLossLayer
Convolutional Neural Network Loss Layer.
|
| CnnLossLayer.Builder |
| Convolution1DLayer
1D (temporal) convolutional layer.
|
| Convolution1DLayer.Builder |
| Convolution3D
3D convolution layer configuration
|
| Convolution3D.Builder |
| Convolution3D.DataFormat
An optional dataFormat: "NDHWC" or "NCDHW".
|
| ConvolutionLayer
2D Convolution layer (for example, spatial convolution over images).
|
| ConvolutionLayer.AlgoMode
The "PREFER_FASTEST" mode will pick the fastest algorithm for the specified parameters from the
ConvolutionLayer.FwdAlgo ,
ConvolutionLayer.BwdFilterAlgo , and ConvolutionLayer.BwdDataAlgo lists, but they may be very memory intensive, so if weird errors
occur when using cuDNN, please try the "NO_WORKSPACE" mode. |
| ConvolutionLayer.BaseConvBuilder |
| ConvolutionLayer.Builder |
| ConvolutionLayer.BwdDataAlgo
The backward data algorithm to use when
ConvolutionLayer.AlgoMode is set to "USER_SPECIFIED". |
| ConvolutionLayer.BwdFilterAlgo
The backward filter algorithm to use when
ConvolutionLayer.AlgoMode is set to "USER_SPECIFIED". |
| ConvolutionLayer.FwdAlgo
The forward algorithm to use when
ConvolutionLayer.AlgoMode is set to "USER_SPECIFIED". |
| Deconvolution2D
2D deconvolution layer configuration
Deconvolutions are also known as transpose convolutions or fractionally strided convolutions. |
| Deconvolution2D.Builder |
| Deconvolution3D
3D deconvolution layer configuration
Deconvolutions are also known as transpose convolutions or fractionally strided convolutions. |
| Deconvolution3D.Builder |
| DenseLayer
Dense layer: a standard fully connected feed forward layer
|
| DenseLayer.Builder |
| DepthwiseConvolution2D
2D depth-wise convolution layer configuration.
|
| DepthwiseConvolution2D.Builder |
| DropoutLayer
Dropout layer.
|
| EmbeddingLayer
Embedding layer: feed-forward layer that expects single integers per example as input (class numbers, in range 0 to
numClass-1) as input.
|
| EmbeddingLayer.Builder |
| EmbeddingSequenceLayer
Embedding layer for sequences: feed-forward layer that expects fixed-length number (inputLength) of integers/indices
per example as input, ranged from 0 to numClasses - 1.
|
| EmbeddingSequenceLayer.Builder |
| FeedForwardLayer
Created by jeffreytang on 7/21/15.
|
| FeedForwardLayer.Builder |
| GlobalPoolingLayer
Global pooling layer - used to do pooling over time for RNNs, and 2d pooling for CNNs.
|
| GlobalPoolingLayer.Builder |
| GravesBidirectionalLSTM
Deprecated.
use
Bidirectional instead. With the Bidirectional
layer wrapper you can make any recurrent layer bidirectional, in particular GravesLSTM. Note that this layer adds the
output of both directions, which translates into "ADD" mode in Bidirectional.
Usage: .layer(new Bidirectional(Bidirectional.Mode.ADD, new GravesLSTM.Builder()....build())) |
| GravesBidirectionalLSTM.Builder
Deprecated.
|
| GravesLSTM
Deprecated.
Will be eventually removed. Use
LSTM instead, which has similar prediction accuracy, but supports
CuDNN for faster network training on CUDA (Nvidia) GPUs |
| Layer
A neural network layer.
|
| Layer.Builder |
| LearnedSelfAttentionLayer
Implements Dot Product Self Attention with learned queries
Takes in RNN style input in the shape of [batchSize, features, timesteps]
and applies dot product attention using learned queries.
|
| LearnedSelfAttentionLayer.Builder |
| LocallyConnected1D
SameDiff version of a 1D locally connected layer.
|
| LocallyConnected1D.Builder |
| LocallyConnected2D
SameDiff version of a 2D locally connected layer.
|
| LocallyConnected2D.Builder |
| LocalResponseNormalization
Local response normalization layer
See section 3.3 of http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf |
| LocalResponseNormalization.Builder |
| LossLayer
LossLayer is a flexible output layer that performs a loss function on an input without MLP logic.
|
| LossLayer.Builder |
| LSTM
LSTM recurrent neural network layer without peephole connections.
|
| NoParamLayer |
| OutputLayer
Output layer used for training via backpropagation based on labels and a specified loss function.
|
| OutputLayer.Builder |
| PoolingType
Pooling type:
MAX: Max pooling - output is the maximum value of the input values AVG: Average pooling - output is the average value of the input values SUM: Sum pooling - output is the sum of the input values PNORM: P-norm pooling |
| PReLULayer
Parametrized Rectified Linear Unit (PReLU)
f(x) = alpha * x for x < 0, f(x) = x for x >= 0
alpha has the same shape as x and is a learned parameter. |
| PReLULayer.Builder |
| PrimaryCapsules.Builder |
| RecurrentAttentionLayer
Implements Recurrent Dot Product Attention
Takes in RNN style input in the shape of [batchSize, features, timesteps]
and applies dot product attention using the hidden state as the query and
all time steps as keys/values.
|
| RecurrentAttentionLayer.Builder |
| RnnLossLayer
Recurrent Neural Network Loss Layer.
|
| RnnLossLayer.Builder |
| RnnOutputLayer
A version of
OutputLayer for recurrent neural networks. |
| RnnOutputLayer.Builder |
| SelfAttentionLayer
Implements Dot Product Self Attention
Takes in RNN style input in the shape of [batchSize, features, timesteps]
and applies dot product attention using each timestep as the query.
|
| SelfAttentionLayer.Builder |
| SeparableConvolution2D
2D Separable convolution layer configuration.
|
| SeparableConvolution2D.Builder |
| SpaceToBatchLayer
Space to batch utility layer configuration for convolutional input types.
|
| SpaceToBatchLayer.Builder |
| SpaceToDepthLayer
Space to channels utility layer configuration for convolutional input types.
|
| SpaceToDepthLayer.Builder |
| SpaceToDepthLayer.DataFormat
Deprecated.
Use
CNN2DFormat instead |
| Subsampling1DLayer
1D (temporal) subsampling layer - also known as pooling layer.
|
| Subsampling1DLayer.Builder |
| Subsampling3DLayer
3D subsampling / pooling layer for convolutional neural networks
Supports max and average pooling modes
|
| Subsampling3DLayer.BaseSubsamplingBuilder |
| Subsampling3DLayer.Builder |
| Subsampling3DLayer.PoolingType |
| SubsamplingLayer
Subsampling layer also referred to as pooling in convolution neural nets
Supports the following pooling types: MAX, AVG, SUM, PNORM
|
| SubsamplingLayer.BaseSubsamplingBuilder |
| SubsamplingLayer.Builder |
| SubsamplingLayer.PoolingType |
| Upsampling1D
Upsampling 1D layer
Repeats each step size times along the temporal/sequence axis (dimension 2)For input shape [minibatch, channels, sequenceLength] output has shape [minibatch, channels, size *
sequenceLength] Example: If input (for a single example, with channels down page, and sequence from left to right) is: [ A1, A2, A3] [ B1, B2, B3] Then output with size = 2 is: [ A1, A1, A2, A2, A3, A3] [ B1, B1, B2, B2, B3, B2] |
| Upsampling1D.Builder |
| Upsampling2D
Upsampling 2D layer
Repeats each value (or rather, set of depth values) in the height and width dimensions by size[0] and size[1] times respectively. |
| Upsampling2D.Builder |
| Upsampling3D
Upsampling 3D layer
Repeats each value (all channel values for each x/y/z location) by size[0], size[1] and size[2] If input has shape [minibatch, channels, depth, height, width] then output has shape [minibatch, channels, size[0] * depth, size[1] * height, size[2] * width] |
| Upsampling3D.Builder |
| ZeroPadding1DLayer
Zero padding 1D layer for convolutional neural networks.
|
| ZeroPadding3DLayer
Zero padding 3D layer for convolutional neural networks.
|
| ZeroPaddingLayer
Zero padding layer for convolutional neural networks (2D CNNs).
|
| ZeroPaddingLayer.Builder |
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Layer.Builder |
| NoParamLayer |
| Class and Description |
|---|
| BaseLayer
A neural network layer.
|
| BaseLayer.Builder |
| FeedForwardLayer
Created by jeffreytang on 7/21/15.
|
| FeedForwardLayer.Builder |
| Layer
A neural network layer.
|
| Layer.Builder |
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Layer.Builder |
| Class and Description |
|---|
| BaseLayer
A neural network layer.
|
| BaseLayer.Builder |
| BaseRecurrentLayer |
| BaseRecurrentLayer.Builder |
| FeedForwardLayer
Created by jeffreytang on 7/21/15.
|
| FeedForwardLayer.Builder |
| Layer
A neural network layer.
|
| Layer.Builder |
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Layer.Builder |
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Layer.Builder |
| NoParamLayer |
| Class and Description |
|---|
| BaseLayer
A neural network layer.
|
| BaseLayer.Builder |
| BasePretrainNetwork |
| BasePretrainNetwork.Builder |
| FeedForwardLayer
Created by jeffreytang on 7/21/15.
|
| FeedForwardLayer.Builder |
| Layer
A neural network layer.
|
| Layer.Builder |
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Layer.Builder |
| Class and Description |
|---|
| BaseLayer
A neural network layer.
|
| BaseLayer.Builder |
| BaseOutputLayer |
| BaseOutputLayer.Builder |
| FeedForwardLayer
Created by jeffreytang on 7/21/15.
|
| FeedForwardLayer.Builder |
| Layer
A neural network layer.
|
| Layer.Builder |
| Class and Description |
|---|
| Convolution3D.DataFormat
An optional dataFormat: "NDHWC" or "NCDHW".
|
| Class and Description |
|---|
| BaseLayer
A neural network layer.
|
| BaseOutputLayer |
| Layer
A neural network layer.
|
| Class and Description |
|---|
| BaseOutputLayer |
| BasePretrainNetwork |
| Layer
A neural network layer.
|
| Class and Description |
|---|
| ConvolutionLayer.AlgoMode
The "PREFER_FASTEST" mode will pick the fastest algorithm for the specified parameters from the
ConvolutionLayer.FwdAlgo ,
ConvolutionLayer.BwdFilterAlgo , and ConvolutionLayer.BwdDataAlgo lists, but they may be very memory intensive, so if weird errors
occur when using cuDNN, please try the "NO_WORKSPACE" mode. |
| ConvolutionLayer.BwdDataAlgo
The backward data algorithm to use when
ConvolutionLayer.AlgoMode is set to "USER_SPECIFIED". |
| ConvolutionLayer.BwdFilterAlgo
The backward filter algorithm to use when
ConvolutionLayer.AlgoMode is set to "USER_SPECIFIED". |
| ConvolutionLayer.FwdAlgo
The forward algorithm to use when
ConvolutionLayer.AlgoMode is set to "USER_SPECIFIED". |
| Class and Description |
|---|
| PoolingType
Pooling type:
MAX: Max pooling - output is the maximum value of the input values AVG: Average pooling - output is the average value of the input values SUM: Sum pooling - output is the sum of the input values PNORM: P-norm pooling |
| Class and Description |
|---|
| ConvolutionLayer.AlgoMode
The "PREFER_FASTEST" mode will pick the fastest algorithm for the specified parameters from the
ConvolutionLayer.FwdAlgo ,
ConvolutionLayer.BwdFilterAlgo , and ConvolutionLayer.BwdDataAlgo lists, but they may be very memory intensive, so if weird errors
occur when using cuDNN, please try the "NO_WORKSPACE" mode. |
| ConvolutionLayer.BwdDataAlgo
The backward data algorithm to use when
ConvolutionLayer.AlgoMode is set to "USER_SPECIFIED". |
| ConvolutionLayer.BwdFilterAlgo
The backward filter algorithm to use when
ConvolutionLayer.AlgoMode is set to "USER_SPECIFIED". |
| ConvolutionLayer.FwdAlgo
The forward algorithm to use when
ConvolutionLayer.AlgoMode is set to "USER_SPECIFIED". |
| PoolingType
Pooling type:
MAX: Max pooling - output is the maximum value of the input values AVG: Average pooling - output is the average value of the input values SUM: Sum pooling - output is the sum of the input values PNORM: P-norm pooling |
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Class and Description |
|---|
| AbstractLSTM
LSTM recurrent net, based on Graves: Supervised Sequence Labelling with Recurrent Neural Networks
http://www.cs.toronto.edu/~graves/phd.pdf
|
| BaseRecurrentLayer |
| FeedForwardLayer
Created by jeffreytang on 7/21/15.
|
| GravesBidirectionalLSTM
Deprecated.
use
Bidirectional instead. With the Bidirectional
layer wrapper you can make any recurrent layer bidirectional, in particular GravesLSTM. Note that this layer adds the
output of both directions, which translates into "ADD" mode in Bidirectional.
Usage: .layer(new Bidirectional(Bidirectional.Mode.ADD, new GravesLSTM.Builder()....build())) |
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Class and Description |
|---|
| FeedForwardLayer
Created by jeffreytang on 7/21/15.
|
| Layer
A neural network layer.
|
| Class and Description |
|---|
| ActivationLayer
Activation layer is a simple layer that applies the specified activation function to the input activations
|
| PReLULayer
Parametrized Rectified Linear Unit (PReLU)
f(x) = alpha * x for x < 0, f(x) = x for x >= 0
alpha has the same shape as x and is a learned parameter. |
| Class and Description |
|---|
| Convolution1DLayer
1D (temporal) convolutional layer.
|
| ConvolutionLayer
2D Convolution layer (for example, spatial convolution over images).
|
| Deconvolution2D
2D deconvolution layer configuration
Deconvolutions are also known as transpose convolutions or fractionally strided convolutions. |
| DepthwiseConvolution2D
2D depth-wise convolution layer configuration.
|
| SeparableConvolution2D
2D Separable convolution layer configuration.
|
| SpaceToDepthLayer
Space to channels utility layer configuration for convolutional input types.
|
| Upsampling1D
Upsampling 1D layer
Repeats each step size times along the temporal/sequence axis (dimension 2)For input shape [minibatch, channels, sequenceLength] output has shape [minibatch, channels, size *
sequenceLength] Example: If input (for a single example, with channels down page, and sequence from left to right) is: [ A1, A2, A3] [ B1, B2, B3] Then output with size = 2 is: [ A1, A1, A2, A2, A3, A3] [ B1, B1, B2, B2, B3, B2] |
| Upsampling2D
Upsampling 2D layer
Repeats each value (or rather, set of depth values) in the height and width dimensions by size[0] and size[1] times respectively. |
| Upsampling3D
Upsampling 3D layer
Repeats each value (all channel values for each x/y/z location) by size[0], size[1] and size[2] If input has shape [minibatch, channels, depth, height, width] then output has shape [minibatch, channels, size[0] * depth, size[1] * height, size[2] * width] |
| ZeroPadding1DLayer
Zero padding 1D layer for convolutional neural networks.
|
| ZeroPadding3DLayer
Zero padding 3D layer for convolutional neural networks.
|
| ZeroPaddingLayer
Zero padding layer for convolutional neural networks (2D CNNs).
|
| Class and Description |
|---|
| ActivationLayer
Activation layer is a simple layer that applies the specified activation function to the input activations
|
| DenseLayer
Dense layer: a standard fully connected feed forward layer
|
| DropoutLayer
Dropout layer.
|
| Class and Description |
|---|
| LocalResponseNormalization
Local response normalization layer
See section 3.3 of http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf |
| Class and Description |
|---|
| EmbeddingSequenceLayer
Embedding layer for sequences: feed-forward layer that expects fixed-length number (inputLength) of integers/indices
per example as input, ranged from 0 to numClasses - 1.
|
| Class and Description |
|---|
| LocallyConnected1D
SameDiff version of a 1D locally connected layer.
|
| LocallyConnected2D
SameDiff version of a 2D locally connected layer.
|
| Class and Description |
|---|
| DropoutLayer
Dropout layer.
|
| Class and Description |
|---|
| BatchNormalization
Batch normalization layer
See: Ioffe and Szegedy, 2014, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift https://arxiv.org/abs/1502.03167 |
| Class and Description |
|---|
| GlobalPoolingLayer
Global pooling layer - used to do pooling over time for RNNs, and 2d pooling for CNNs.
|
| PoolingType
Pooling type:
MAX: Max pooling - output is the maximum value of the input values AVG: Average pooling - output is the average value of the input values SUM: Sum pooling - output is the sum of the input values PNORM: P-norm pooling |
| Subsampling1DLayer
1D (temporal) subsampling layer - also known as pooling layer.
|
| Subsampling3DLayer
3D subsampling / pooling layer for convolutional neural networks
Supports max and average pooling modes
|
| SubsamplingLayer
Subsampling layer also referred to as pooling in convolution neural nets
Supports the following pooling types: MAX, AVG, SUM, PNORM
|
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Class and Description |
|---|
| Layer
A neural network layer.
|
| Class and Description |
|---|
| ConvolutionLayer.AlgoMode
The "PREFER_FASTEST" mode will pick the fastest algorithm for the specified parameters from the
ConvolutionLayer.FwdAlgo ,
ConvolutionLayer.BwdFilterAlgo , and ConvolutionLayer.BwdDataAlgo lists, but they may be very memory intensive, so if weird errors
occur when using cuDNN, please try the "NO_WORKSPACE" mode. |
| Layer
A neural network layer.
|
| Class and Description |
|---|
| Convolution3D.DataFormat
An optional dataFormat: "NDHWC" or "NCDHW".
|
| Layer
A neural network layer.
|
| PoolingType
Pooling type:
MAX: Max pooling - output is the maximum value of the input values AVG: Average pooling - output is the average value of the input values SUM: Sum pooling - output is the sum of the input values PNORM: P-norm pooling |
| Class and Description |
|---|
| BatchNormalization
Batch normalization layer
See: Ioffe and Szegedy, 2014, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift https://arxiv.org/abs/1502.03167 |
| ConvolutionLayer
2D Convolution layer (for example, spatial convolution over images).
|
| DenseLayer
Dense layer: a standard fully connected feed forward layer
|
| SubsamplingLayer
Subsampling layer also referred to as pooling in convolution neural nets
Supports the following pooling types: MAX, AVG, SUM, PNORM
|
| SubsamplingLayer.PoolingType |
Copyright © 2020. All rights reserved.


3 / 2 9 / 2 0 1 4 - A R R A Y S vs H A S H E S
Ruby has many different ways of storing multiple variables. Two of the most common methods are arrays and hashes. But how can you decide which to use for your data structure?
Arrays are found in most programming languages. You can fill arrays with any type of value. Each value will be stored numerically in the array and can be accessed by calling the corresponding number. You can even make multidimensional arrays, so the storage space for arrays is essentially limitless. Arrays are very flexible, so they will work in almost any situation.
Hashes exist in many programming languages, sometimes under different names. They can be stored with keys and values, and can be accessed by either one. Hashes are very useful because keys are much easier to remember than numbers(which are used in arrays), so information in a hash is very well organized. Hashes only really work for data that is arranged in pairs, but it seems to be the organized way to store data when that is the case.
If you're having trouble deciding which structure to use, I think the safest structure to use is an array. Arrays will always work, but they can be confusing to navigate sometimes. If the information can be stored in pairs, such as a phonebook or contact list, try a hash. If you can safely fit the information into a hash, it's usually the better choice.


PRACTICAL MAGIC A film review by David N. Butterworth Copyright 1998 David N. Butterworth
** (out of ****)
So I took my little sister to see "Practical Magic" because it seemed like a better choice than "Rudolph the Red-Nosed Reindeer - The Movie" (even though "Rudolph the Red-Nosed Reindeer - The Movie" features a new version of 'Rudolph the Red-Nosed Reindeer' by Clint Black). "Practical Magic" stars Sandra Bullock and Nicole Kidman who play sisters who are also latter-day witches, and if you've seen anything else with Nicole in it you'll know this isn't much of a stretch for her. She plays the slutty witch with the red hair whose freewheeling lifestyle gets her into trouble. Sandra plays the good witch with the black hair who has hung up her broomstick and decided not to dabble in the black arts. She wants a normal life with a husband and two kids (one has black hair, the other red, coincidentally) and a business selling shampoos, soaps, and body oils. If you've seen anything else with Sandra in it you'll know this isn't much of a stretch for her. Unfortunately Sandra and Nicole are the latest in a long line of witches who have been landed with a nasty curse: any man who falls in love with an Owens woman is doomed. To sit through this movie, they mean. So I tried to keep my eyes off Sandra and Nicole as much as possible, lest I fall in love with either of them, and marvel instead at Stockard Channing and Dianne Wiest who play the girls' over-stuffed, campy witch aunts, with whom they are forced to live and eat chocolate for breakfast. Stockard essentially recreates her over-stuffed, campy role as a bordello proprietor in "Moll Flanders." Dianne says something funny--there's a big dramatic pause right before she says it--but for the life of me I can't remember what it was now. Maybe it was during the embarrassing "midnight margaritas" sequence. That's part of the problem with the film; it's not very funny (unless substituting a dustbuster for a broom is a new sight gag for you). On the other hand, Sandra's tears are quite convincing when her husband gets run over by a truck--think what she could do if a storyline required her to blubber for 90 minutes straight--but that's the other problem with the film. It's all over the place. It's part light romantic comedy, part sappy sentimental tearjerker, part Tuesday night domestic violence drama, part murder mystery, and part "Exorcist"-styled demonic possession flick, and none of this fits together particularly well. I asked my sister what she thought of the film and she said it was a bit "slow." But her biggest disappointment was that Aidan Quinn, who plays an Arizona cop who comes looking for Nicole's missing boyfriend, didn't take his shirt off. As for the film's soundtrack... Oh geez. Just when the film gets quiet or moving or poignant the filmmakers force some loud, inappropriate pop song down our throats. Thankfully someone must have whispered something in Griffin Dunne's ear--Griffin directed the picture with far too much soft-focus and hand-held camerawork for my liking--about halfway through the film because the songs stop, just like that, until Stevie Nicks, a self-confessed witch in her own right, warbles a new version of Fleetwood Mac's 'Crystal' right at the end of the movie, following it immediately with another sudser over the end credits. This so that they could stamp a big "features two new songs by Stevie Nicks" on the print ads, presumably. Maybe Clint Black crooning 'Rudolph the Red-Nosed Reindeer' wasn't such a bad idea after all.
-- David N. Butterworth<EMAIL_ADDRESS>The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


PyTorch.org provides researchers and developers with documentation, installation instructions, latest news, community projects, tutorials, and more. Today, we are introducing usability and content improvements including tutorials in additional categories, a new recipe format for quickly referencing common topics, sorting using tags, and an updated homepage.
Let’s take a look at them in detail.
TUTORIALS HOME PAGE UPDATE
The tutorials home page now provides clear actions that developers can take. For new PyTorch users, there is an easy-to-discover button to take them directly to “A 60 Minute Blitz”. Right next to it, there is a button to view all recipes which are designed to teach specific features quickly with examples.
In addition to the existing left navigation bar, tutorials can now be quickly filtered by multi-select tags. Let’s say you want to view all tutorials related to “Production” and “Quantization”. You can select the “Production” and “Quantization” filters as shown in the image shown below:
The following additional resources can also be found at the bottom of the Tutorials homepage:
PYTORCH RECIPES
Recipes are new bite-sized, actionable examples designed to teach researchers and developers how to use specific PyTorch features. Some notable new recipes include:
View the full recipes here.
LEARNING PYTORCH
This section includes tutorials designed for users new to PyTorch. Based on community feedback, we have made updates to the current Deep Learning with PyTorch: A 60 Minute Blitz tutorial, one of our most popular tutorials for beginners. Upon completion, one can understand what PyTorch and neural networks are, and be able to build and train a simple image classification network. Updates include adding explanations to clarify output meanings and linking back to where users can read more in the docs, cleaning up confusing syntax errors, and reconstructing and explaining new concepts for easier readability.
DEPLOYING MODELS IN PRODUCTION
This section includes tutorials for developers looking to take their PyTorch models to production. The tutorials include:
- Deploying PyTorch in Python via a REST API with Flask
- Introduction to TorchScript
- Loading a TorchScript Model in C++
- Exporting a Model from PyTorch to ONNX and Running it using ONNX Runtime
FRONTEND APIS
PyTorch provides a number of frontend API features that can help developers to code, debug, and validate their models more efficiently. This section includes tutorials that teach what these features are and how to use them. Some tutorials to highlight:
- Introduction to Named Tensors in PyTorch
- Using the PyTorch C++ Frontend
- Extending TorchScript with Custom C++ Operators
- Extending TorchScript with Custom C++ Classes
- Autograd in C++ Frontend
MODEL OPTIMIZATION
Deep learning models often consume large amounts of memory, power, and compute due to their complexity. This section provides tutorials for model optimization:
PARALLEL AND DISTRIBUTED TRAINING
PyTorch provides features that can accelerate performance in research and production such as native support for asynchronous execution of collective operations and peer-to-peer communication that is accessible from Python and C++. This section includes tutorials on parallel and distributed training:
- Single-Machine Model Parallel Best Practices
- Getting started with Distributed Data Parallel
- Getting started with Distributed RPC Framework
- Implementing a Parameter Server Using Distributed RPC Framework
Making these improvements are just the first step of improving PyTorch.org for the community. Please submit your suggestions here.
Cheers,
Team PyTorch


Waveform Manager
Actions
Channel 0
Channel 1
Update
Randomize Phase
Reset All Amplitudes
Setpoint
P
I
D
Start Feedback
Stop Feedback
View Current


---
layout: page
title: Background
---
Without fuel, the world stops. As interns of the SAF-IEN team, the mission imparted to us was to not only figure out a way around that (not having fuel), but to mitigate the risk in the transportation of our fuel (oil) supply and help in the enhancement of our combat capability using energy-informed solutions; in short, making a safer and more efficient pathway for the fuel to get to us, and for us to get to the fuel.
Our direct project has been to develop the foundation for a system in which the user can manipulate various factors to see the accumulated risk associated with a particular fuel supply route. In our model, the “best” route is the route with the lowest risk and the highest capacity.
In the end, we hope to produce a model with a UI/UX accessible for Airmen [to use while in the plane] to find the most effective path to find fuel necessary to complete the mission.


Redundant development wastes effort for both maintainers and developers. INTRUDE is a GitHub bot to detect potentially redundant pull requests. It monitors all pull requests in a repository, and if it finds two pull requests that are very similar, it automatically comments on the issue, pointing out the similarity.
Our goal is to identify redundant code changes in forks as early as possible by extracting clues indicating similarities between code changes, and building a machine learning model to predict redundancies.
An academic paper describing more technical details and an evaluation has been published:
L. Ren, S. Zhou, C. Kästner, and A. Wąsowski. Identifying Redundancies in Fork-based Development. In Proceedings of the 27th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 230--241, 2019. [pdf]This bot is currently in its alpha stage. We are monitoring a limited number of repositories and are sending a single comment on the first suspected duplicate pull request we find. We hope to offer the bot as a service soon. If you are interested, please subscribe!
We are researchers at Carnegie Mellon University developing a bot to monitor GitHub projects and alert project maintainers to potentially duplicate pull requests. Shurui Zhou is a Ph.D. student studying under Professor Christian Kästner. Annika Esau is an undergraduate REU student.
INTRUDE comes out of an NSF-funded research project at Carnegie Mellon University, identfying potentially redundant development in open-source software development. INTRUDE is open source and hosted for the community. We would love to hear your ideas and feedback. Explore the source or open an issue.


Pimple patches: Everything you need to know about them
Pimple patches are the new spot treatment technique for acne. Does a sticker that cures acne, too good to be true? Scroll down to know more.
by Lifestyle DeskThere was a major fad for sheet masks some time back. Instagram was full of people posing for selfies with sheet masks and skincare hashtags. There is no denying that when one faces skin conditions like acne they tend to lean towards trendy over-the-counter acne treatments which are high on promise but ultimately fail to deliver results.
Pimple patches also referred to as acne healing patches which are essentially small, circular hydrocolloid (hydro-kull-oid, if your tongue is stuck) or salicylic acid bandages. These work by absorbing the dirt, oil and excess fluid from the pimple and further drying it up.
These little pimple patches have one simple rule. Don’t pick, just patch it up. But, they also have a secondary benefit i.e they prevent you from touching your face too much and can be a skin saver for those who are chronic pickers.
Do pimple patches really work?
The idea of a sticker which absorbs impurities and heals pimples might sound too good to be true, but looks like it does work.  The patch keeps the pimple and gives it time to heal. However, pimple patches are probably best meant for smaller breakouts that aren’t so deeply embedded under the skin like papules; little spots filled with pus.
As far as cystic acne (most severe form of acne, in which the pores in the skin become blocked, leading to infection and inflammation) is concerned, micro dart patches might allow a slightly deeper penetration of active ingredients but they won’t provide the most dramatic results.
Should you try them?
If you are someone whose guilty pleasure is picking at your zits, then pimple patches are your protective best friends as they prevent the temptation to pop. And if you deal with smaller breakouts, you’ll probably be successful with them.
However, remember that pimple patches aren’t a solution or the end game. Everyone’s skin is different and plain hydrocolloid stickers, might work for some people and not for others.


import {RequestWrapper} from '../../../../workbox-runtime-caching/src/index';
import WorkboxError from '../../../../../lib/workbox-error';
/**
* This class handles the shared logic for caching revisioned and unrevisioned
* assets.
*
* @memberof module:workbox-precaching
*/
class BaseCacheManager {
/**
* Constructor for BaseCacheManager
*
* @param {Object} input
* @param {String} [input.cacheName] This is the cache name to store requested
* assets.
* @param {String} [input.cacheId] The cacheId can be used to ensure that
* multiple projects sharing `http://localhost` have unique cache names.
* @param {Array<Object>} [input.plugins] Any plugins that should be
* invoked by the underlying `RequestWrapper`.
*/
constructor({cacheName, cacheId, plugins} = {}) {
if (cacheId && (typeof cacheId !== 'string' || cacheId.length === 0)) {
throw new WorkboxError('bad-cache-id', {cacheId});
}
this._entriesToCache = new Map();
this._requestWrapper = new RequestWrapper({
cacheName,
cacheId,
plugins,
fetchOptions: {
credentials: 'same-origin',
},
});
}
/**
* Adds entries to the install list.
* This will manage duplicate entries and perform the caching during
* the install step.
*
* @private
* @param {Array<String|Request|Object>} rawEntries A raw entry that can be
* parsed into a BaseCacheEntry by the inheriting CacheManager.
*/
_addEntries(rawEntries) {
this._parsedCacheUrls = null;
rawEntries.forEach((rawEntry) => {
this._addEntryToInstallList(
this._parseEntry(rawEntry)
);
});
}
/**
* Gives access to the cache name used by this caching manager.
* @return {String} The cache name used by this manager.
*/
getCacheName() {
return this._requestWrapper.cacheName;
}
/**
* Returns an array of fully qualified URL's that will be cached by this
* cache manager.
*
* @return {Array<String>} An array of URLs that will be cached.
*/
getCachedUrls() {
if (!this._parsedCacheUrls) {
this._parsedCacheUrls = Array.from(this._entriesToCache.keys())
.map((url) => new URL(url, location).href);
}
return this._parsedCacheUrls;
}
/**
* Adds an entry to the install list.
*
* Duplicates are filtered out and checks are made for the scenario
* where two entries have the same URL but different revisions. For example
* caching:
* [
* {url: '/hello.txt', revision: 'abcd1234'},
* {url: '/hello.txt', revision: 'efgh5678'},
* ]
* This will throw an error as the library can't determine the correct
* revision and this may cause issues in future when updating the service
* worker with new revisions.
*
* @private
* @param {RevisionedCacheEntry} precacheEntry The file entry to be cached
* during the next install event.
*/
_addEntryToInstallList(precacheEntry) {
const entryID = precacheEntry.entryID;
const previousEntry = this._entriesToCache.get(precacheEntry.entryID);
if (!previousEntry) {
// This entry isn't in the install list
this._entriesToCache.set(entryID, precacheEntry);
return;
}
this._onDuplicateInstallEntryFound(precacheEntry, previousEntry);
}
/**
* This method will go through each asset added to the cache list and
* fetch and update the cache for assets which have a new revision hash.
*
* @return {Promise<Array<Object>>} The promise resolves when all the
* desired assets are cached and up -to-date.
*/
async install() {
if (this._entriesToCache.size === 0) {
return [];
}
const cachePromises = [];
this._entriesToCache.forEach((precacheEntry) => {
cachePromises.push(
this._cacheEntry(precacheEntry)
);
});
// Wait for all requests to be cached.
return Promise.all(cachePromises);
}
/**
* Requests the entry and saves it to the cache if the response
* is valid.
*
* @private
* @param {BaseCacheEntry} precacheEntry The entry to fetch and cache.
* @return {Promise<Object>} Returns a promise that resolves once the entry
* has been fetched and cached or skipped if no update is needed. The
* promise resolved with details of the entry and whether it was
* updated or not.
*/
async _cacheEntry(precacheEntry) {
const isCached = await this._isAlreadyCached(precacheEntry);
const precacheDetails = {
url: precacheEntry.request.url,
revision: precacheEntry.revision,
wasUpdated: !isCached,
};
if (isCached) {
return precacheDetails;
}
try {
await this._requestWrapper.fetchAndCache({
request: precacheEntry.getNetworkRequest(),
waitOnCache: true,
cacheKey: precacheEntry.request,
cleanRedirects: true,
});
await this._onEntryCached(precacheEntry);
return precacheDetails;
} catch (err) {
throw new WorkboxError('request-not-cached', {
url: precacheEntry.request.url,
error: err,
});
}
}
/**
* Compare the URL's and determines which assets are no longer required
* in the cache.
*
* This should be called in the service worker activate event.
*
* @return {Promise} Promise that resolves once the cache entries have been
* cleaned.
*/
async cleanup() {
if (!await caches.has(this.getCacheName())) {
// Cache doesn't exist, so nothing to delete
return;
}
const requestsCachedOnInstall = [];
this._entriesToCache.forEach((entry) => {
requestsCachedOnInstall.push(entry.request.url);
});
const openCache = await this._getCache();
const allCachedRequests = await openCache.keys();
const cachedRequestsToDelete = allCachedRequests.filter(
(cachedRequest) => !requestsCachedOnInstall.includes(cachedRequest.url));
return Promise.all(
cachedRequestsToDelete.map(async (cachedRequest) => {
await openCache.delete(cachedRequest);
await this._onEntryDeleted(cachedRequest.url);
})
);
}
/**
* A simple helper method to get the open cache used for precaching assets.
*
* @private
* @return {Promise<Cache>} The cache to be used for precaching.
*/
async _getCache() {
if (!this._cache) {
this._cache = await caches.open(this.getCacheName());
}
return this._cache;
}
/**
* Ensures the file entry in the maniest is valid and
* can be parsed as a BaseCacheEntry.
*
* @private
* @abstract
* @param {String | Request | Object} input Either a URL string, a Request
* or an object with a `url`, `revision` and optional `cacheBust` parameter.
* @return {BaseCacheEntry} Returns a parsed version of the file entry.
*/
_parseEntry(input) {
throw new WorkboxError('requires-overriding');
}
/**
* Called in case subclasses have cache entries that are to be installed
* but have the same "entryID".
* This means that the user is trying to cache the same thing twice.
* Subclasses can use this method to throw an error if there is an edge
* case that can't be handled.
*
* @private
* @abstract
* @param {BaseCacheEntry} newEntry The entry that is to be cached.
* @param {BaseCacheEntry} previous The entry that is currently cached.
*/
_onDuplicateEntryFound(newEntry, previous) {
throw new WorkboxError('requires-overriding');
}
/**
* Confirms whether a fileEntry is already in the cache with the
* appropriate revision or not.
*
* @private
* @abstract
* @param {BaseCacheEntry} precacheEntry A file entry with `path` and
* `revision` parameters.
* @return {Promise<Boolean>} Returns true is the fileEntry is already
* cached, false otherwise.
*/
_isAlreadyCached(precacheEntry) {
throw new WorkboxError('requires-overriding');
}
/**
* Subclasses can use this method for any work that needs to be done once a
* URL has been cached.
*
* @private
* @abstract
* @param {BaseCacheEntry} precacheEntry A file entry with `path` and
* `revision` parameters.
* @return {Promise} Returns a Promise that resolves once it's work has
* been done.
*/
_onEntryCached(precacheEntry) {
throw new WorkboxError('requires-overriding');
}
/**
* Subclasses can use this method for any work that needs to be done once a
* URL has been deleted from the cache.
*
* @private
* @abstract
* @param {String} url The URL of the entry that was deleted.
* @return {Promise} Returns a Promise that resolves once the work is done.
*/
_onEntryDeleted(url) {
throw new WorkboxError('requires-overriding');
}
}
export default BaseCacheManager;


I’m an electrochemist with a background in fundamental charge conversion and storage processes in inorganic materials. As a postdoc with Bryan McCloskey at UC Berkeley, I worked on quantifying interfacial degradation processes in high energy Li-ion battery materials, and developed scalable methods of surface passivation. Previously, as a graduate student with Prof. Christopher Chidsey and Prof. Robert Waymouth at Stanford University, I worked on designing molecular electrocatalysts for CO2 reduction using free energy maps. I’m currently at Neuralink doing some extreme electrochemistry.
When I’m not staring at voltammograms, I enjoy hiking, tennis and classical music.
PhD Chemistry, 2018
Stanford University
MSc Chemistry, 2012
Indian Institute of Technology
BSc (Hons.) Chemistry, 2007
SSSIHL


| debugcall {utils} | R Documentation |
Set or unset debugging flags based on a call to a function. Takes into account S3/S4 method dispatch based on the classes of the arguments in the call.
debugcall(call, once = FALSE) undebugcall(call)
call |
An R expression calling a function. The called function will be debugged. See Details. |
once |
logical; if |
debugcall
debugs the non-generic function, S3 method or S4
method that would be called by evaluating call
. Thus, the user
does not need to specify the signature when debugging
methods. Although the call is actually to the generic, it is the
method that is debugged, not the generic, except for non-standard S3
generics (see isS3stdGeneric
).
debugcall
invisibly returns the debugged call expression.
Non-standard evaluation is used to retrieve the call
(via
substitute
). For this reason, passing a variable
containing a call expression, rather than the call expression itself,
will not work.
debug
for the primary debugging interface
## Not run: ## Evaluate call after setting debugging ## f <- factor(1:10) res <- eval(debugcall(summary(f))) ## End(Not run)


| libxml2 Reference Manual |
|---|
Libxml2 is the XML C parser and toolkit developed for the Gnome project (but usable outside of the Gnome platform), it is free software available under the MIT License. XML itself is a metalanguage to design markup languages, i.e. text language where semantic and structure are added to the content using extra "markup" information enclosed between angle brackets. HTML is the most well-known markup language. Though the library is written in C, a variety of language bindings make it available in other environments.
Libxml2 implements a number of existing standards related to markup languages:


Aniruddh Iyer
Home
(current)
About
Resume
Contact
Connect With Me
Feel free to reach out to me using the following ways!
Phone
Facebook
Twitter
Email
LinkedIn


exrm v0.16.0
Modules summary ↑
| Mix.Tasks.Release | Build a release for the current mix application |
| Mix.Tasks.Release.Clean | Clean up any release-related files |
| Mix.Tasks.Release.Plugins | View information about active release plugins |
| ReleaseManager.Appups | Module for auto-generating appups between releases |
| ReleaseManager.Config | Configuration for the release task |
| ReleaseManager.Plugin | This module provide a simple way to add additional steps to the release task |
| ReleaseManager.Plugin.Conform | Generates a .conf for your release |
| ReleaseManager.Plugin.Consolidation | |
| ReleaseManager.Utils | This module provides helper functions for the |


| Employee ID | Employee Name | Employee Salary | PayAbles | Salary Month | Enter Amount | Salary Completed |
|---|---|---|---|---|---|---|
| {{salaries.employeeId}} | {{salaries.employeeName}} | {{salaries.totalSalary}} | {{salaries.currentPayable}} | {{salaries.salaryMonth}} |
Loading, please wait...


Toggle navigation
Innovointiprojekti 2015
Dashboard
My site
Overview
(current)
Countries
Teams
Users
Answers
Statistic
Create a category
Create teams
Create a questionnaire
Add countries
Dashboard
Choose a team
Team1
Team2
Team3
Team4
Most popular
Questionnaire name
Question
Average
Date
Questionnaire 1
Question 1
5
2015-01-01