Plymouth restaurant offers free carvery to all uniformed veterans
The family-ran Heaven want to give something back to the cause 'close to their hearts'
by Katie TimmsAn ex-serviceman is offering all uniformed veterans a free carvery in Plymouth today.
Reza Davarian, who owns Heaven restaurant on Exeter Street, served in the Royal Navy and wants to give something back to other servicemen.
The popular restaurant known for its roast dinners opened in the former First and Last pub almost ten years ago.
The venue underwent a major transformation in April 2018 into healthy eating restaurant 'Fuel', but in April 2019 the Davarian family decided to take the eatery back to its roots.
Joshua Davarian, 18, who works at the restaurant with his dad, Reza, said that their carvery has been popular among customers.
He said that his family had decided to offer something back to the cause which is "very close to our hearts" and donations for the Poppy Appeal will also be collected at the premises.
The free lunch is available for all uniformed veterans between 12pm and 3pm after the Remembrance Day service on Plymouth Hoe.
After 3pm all uniformed veterans will be eligible for a free dessert.
Joshua said: "In honour of Remembrance Day we are offering all uniformed veterans free carvery tomorrow (between 12 and 3pm) after the service on the hoe.
"We are also collecting donations and selling poppies."
For more information, visit the Heaven Facebook page here.


Contributing to Mail¶ ↑
Hi there, I welcome pull requests! Here are some thoughts on how to get your pull request merged quickly:
-
Check the Reference RFCs, they are in the References directory, so no excuses.
-
Check for a ticket on GitHub, maybe someone else has the problem too
-
Make a fork of my GitHub repository
-
Run the specs. We only take pull requests with passing tests, and it’s great to know that you have a clean slate:
bundle && bundle exec rake
-
Add a spec for your change. Only refactoring and documentation changes require no new specs. If you are adding functionality or fixing a bug, we need a spec!
-
Test the spec at least against MRI-1.9.3 and MRI-1.8.7
-
Update the README if needed to reflect your change / addition
-
Update the CHANGELOG and give yourself credit
-
With all specs passing push your changes back to your fork
-
Send me a pull request.
-
If it needs any changes, please push or force push to the same branch you made the pull request from. GitHub will just update the pull request with your changes.
Note, specs that break MRI 1.8.7 or 1.9.3 will not be accepted.
At this point you’re waiting on us. We like to at least comment on, if not accept, pull requests within three business days (and, typically, one business day). We may suggest some changes or improvements or alternatives.
Some things that will increase the chance that your pull request is accepted, taken straight from the Ruby on Rails guide:
-
Tell me you have tested it against more than one version of Ruby, RVM is great for this. I test against 7 rubies before I push into master.
-
Use good, idiomatic, structured and modular code
-
Include tests that fail without your code, and pass with it
-
Update the documentation, the surrounding one, examples elsewhere, guides, whatever is affected by your contribution
Syntax:
-
Two spaces, no tabs.
-
No trailing whitespace. Blank lines should not have any space.
-
Prefer &&/|| over and/or.
-
MyClass.my_method(my_arg) not my_method( my_arg ) or my_method my_arg.
-
a = b and not a=b.
-
Follow the conventions you see used in the source already.
And in case we didn’t emphasize it enough: we love specs!
Testing against mime-types versions:¶ ↑
Use appraisal to run against all supported versions of mime-types.
-
Run
(bundle check || bundle) && appraisal
so that all the ‘appraised’ gemfiles are bundled. -
Run either
appraisal rake
orrake appraisal
to run all the tests.
To run only one ‘appraised’ gemfile, run. e.g.
BUNDLE_GEMFILE=gemfiles/mime_types_edge.gemfile (bundle check ||
bundle) && rake
To change the appraisals, modify the Appraisals
file, run
appraisal
, commit the generated gemfiles, and modify the
.travis.yml matrix.
To run on all rubies / gemfiles, just like TravisCI, see WWTD.


I’ve only ever really worked with HTML and man page builds for the documentation various of various OpenStack projects. However, OpenStack uses Sphinx across the board and Sphinx, being the awesome tool that it is, supports many other output formats. In this instance, I was interested in PDF.
Sphinx doesn’t actually provide a native PDF builder (although other packages do). Instead, you have to generate LaTeX sources and then generate a PDF for this. This means you need to have a pretty well fleshed out TeX installation. As such, the first step is to install the dependencies.
Install dependencies
We need a number of packages to build the LaTeX sources and convert them into a PDF. I’m using Fedora 27 which means package names may need to be adjusted accordingly. Do note, however, that these names should work equally well on RHEL and RHEL derivatives like CentOS).
-
texlive
(TeX Live)Per the description of this package in the Fedora repos.
The TeX Live software distribution offers a complete TeX system for a variety of Unix, Macintosh, Windows and other platforms. It encompasses programs for editing, typesetting, previewing and printing of TeX documents in many different languages, and a large collection of TeX macros and font libraries.
The distribution includes extensive general documentation about TeX, as well as the documentation for the included software packages.
This is the basic thing we need to actually build LaTeX documents and convert them to PDF. The package itself includes multiple other dependencies, either because they are necessary for a minimal install or they form a standard library of sorts. I’m not sure which.
-
latexmk
Per the description of this package in the Fedora repos.
latexmk is a perl script for running LaTeX the correct number of times to resolve cross references, etc.; it also runs auxiliary programs (bibtex, makeindex if necessary, and dvips and/or a previewer as requested). It has a number of other useful capabilities, for example to start a previewer and then run latex whenever the source files are updated, so that the previewer gives an up-to-date view of the document. The script runs on both UNIX and MS-WINDOWS (95, ME, XP, etc). This script is a corrected and improved version of the original version of latexmk.
This is a tool that Sphinx uses, via some makefiles it includes with the built sources to ease converting the built LaTeX sources into PDFs.
You also need a variety of other packages which aren’t included as part of the
texlive
installation. The base Sphinx template has the following requirements.
texlive-fncychap
texlive-titlesec
texlive-tabulary
texlive-framed
texlive-wrapfig
texlive-upquote
texlive-capt-of
texlive-needspace
In addition, openstackdocstheme
, the theme used for all official OpenStack
projects, provides a LaTeX template with the following dependencies.
texlive-polyglossia
Installation of all the above.
$ sudo dnf install texlive latexmk texlive-fncychap texlive-titlesec \
texlive-tabulary texlive-framed texlive-wrapfig texlive-upquote \
texlive-capt-of texlive-needspace texlive-polyglossia
Get some documentation
I didn’t have specific project in mind, so I’ve decided to build the project
documentation for the openstackdocstheme
project. As mentioned previously,
this project provides the theme used for all Official OpenStack projects.
Given that this is the base for most projects’ documentation, it figures that
it contains all the necessary configuration to build PDFs. If, however, you are
planning to build docs for some other project, you should ensure that this is
project is configured correctly for PDF builds.
To begin, clone the repo locally.
$ git clone https://github.com/openstack/openstackdocstheme
Once cloned, create a virtual environment and install the dependencies.
$ virtualenv .venv
$ source .venv
$ pip install -e .
In the case of openstackdocstheme
, Sphinx isn’t installed by default.
Let’s install that too.
$ pip install Sphinx
Build the docs
Now that we have both our dependencies and source documentation in place, it’s
time to actually build some PDFs. Sphinx actually provides a number of helpful
targets in the Makefile generated by sphinx-quickstart
. However, OpenStack
projects don’t tend to use these so we’re going to take the (still exceedingly
simple) two-step process. Firstly, build the LaTeX sources. Seeing as this is a
Python project, we’re going to use the setuptools integration.
$ python setup.py build_sphinx -b latex
All OpenStack projects place their documentation source and builds in
doc/source
and doc/build
respectively. The above command should generate a
number of files in doc/build/latex
including a very helpful makefile. We’re
going to use that now to complete the process of generating a PDF.
$ make -C doc/build/latex
This should emit a lot of text, followed by something that looks like this.
Output written on os-doc-demo.pdf (23 pages).
Transcript written on os-doc-demo.log.
=== TeX engine is 'XeTeX'
Latexmk: Index file 'os-doc-demo.idx' was written
Latexmk: Log file says output to 'os-doc-demo.pdf'
Latexmk: All targets (os-doc-demo.pdf) are up-to-date
Once complete, you can find your compiled PDF in the same build directory (in
this case, doc/build/latex/os-doc-demo.pdf
)


I added a lot more details into the house now.
The materials are closer to the end results.
Also the rooms have been filled.
The bedroom has cuboards
a whatever it is called
a wardrobe and some cloth hanging capability.
The shower room and toillets are also done.
They will likely need to be swaped to avoid a 100m pipe going through the house in width.
The kitchen made its appearance downstairs, with the pizza oven and its pipe.
So far i only did a reference for the staircase.
It is too fiddly for me to do it on 3d.
Ultimatly it will look similar to what equip boats.

I also started the "cosy room", which will be where we spend evenings.
So far i only have a sofa and coffee table there, next up is a desk that transform into standing desk.
I had fun with the gutter too. bonus point if you know where i got the inspiration.


Spike Histogram Demo - Demonstration of a web-based spike histogram with paneling capability and modals.


About Me
Graphic Design
UX&UI Design
Photo
Contact
Graphic Design
This is a result of participating in activities in university, student councils and freelance.


Preparation time
less than 30 mins
Cooking time
1 to 2 hours
Serves
Serves 6 - 8
John Torode's Tuscan-style T-bone steaks are heaven for carnivores. It's worth seeking out the very best meat to do this dish full justice.
Preheat the oven to 220C/425F/Gas 7. Strip the leaves from the chard, wash and set to one side. Trim the stalks of any blemishes and cut into sticks about 5cm/2in long and 2cm/¾in wide. Bring a pan of salted water to the boil, throw in the chard stalks and return to the boil. Drain, plunge the stalks into iced water then drain again and set aside.
Heat the oil in a frying pan, add the shallots and colour them over a high heat. When they are coloured, drain off and discard the oil. Add 50g/1¾oz of the butter, the bay leaves, thyme and salt to the pan and cook for five minutes, shaking the pan from time to time, taking care not to burn the butter.
Add enough stock to just cover the bottom of the pan and allow the liquid to reduce before adding more stock to the pan. Continue with the remaining stock until the shallots are very soft and have a thick buttery and beefy glaze.
Meanwhile, heat a griddle plate or ovenproof griddle pan and season the beef well.  The fat will start to melt and flavour the outside of the meat.
Once the fat is melting and starting to char, leave the meat to cook for four minutes. Turn over and cook for another four minutes. Turn again to cook on its narrower side for two minutes, holding in place with tongs, turn once more to cook the other narrower side for two minutes then take it off the heat and transfer it to the oven for about six minutes for a medium steak. Let it rest for five minutes before serving.
While the steak is in the oven, heat a frying pan over a high heat and melt the remaining butter. Add the blanched chard stalks, cook for three minutes then add the chard leaves, stirring until they wilt. Season with sea salt and freshly ground black pepper, add the anchovies, capers and lemon juice and cook for two more minutes.
Using a slotted spoon, drain the chard and place on large warmed plates, reserving the sauce. Place the steaks on top of the chard, pour over the sauce and serve with the caramelised shallots.


Gov’t orders illegal migration organisations to be shut down
HÀ NỘI — The Government has ordered authorised agencies to strictly handle any individuals or organisations that illegally send Vietnamese citizens abroad.
The order follows a new resolution, recently issued by the Government after its regular meeting held in Hà Nội.
Accordingly, the Government assigned the Ministry of Foreign Affairs to closely work with the Ministry of Public Security, localities and other relevant agencies to handle the case in which 39 Vietnamese were found dead in a container truck in Essex, northeast London, on October 23.
The Government also calls upon other countries to work together in the fight against illegal immigration.
The Ministry of Public Security was told to take measures to break up illegal migration organisations, multi-level marketing fraud as well as prevent hi-tech crime committed by foreigners in Việt Nam in the future.
The Government also tasked ministries and sectors to seriously carry out resolution No1/NQ-CP on major tasks and solutions for implementing the socio-economic development plan in 2019, issued in January and directive No 09/CT-Ttg on solutions for business production, ensuring the country’s growth target is met in 2019, issued in April.
The Ministry of Agriculture and Rural Development had to mobilise forces to prevent and control African swine fever and support pig breeders to re-breed and ensure the yield of pork to customers in the last months of this year, especially for New Year and Tết (Lunar) New Year.
The ministry also had to focus on removing the European Commission’s yellow card warning on Vietnamese fisheries. — VNS
Viet Nam News


class: center, middle # Lecture 4: ## Going deeper .grid[ .kol-6-12[ Marc Lelarge
*
.red[Andrei Bursuc]
Alexandre Defossez ] .kol-6-12[ Alexandre Sablayrolles
Pierre Stock
Neil Zeghidour ] ] .red[*] contact in case of emergency... --- # Outline - Universal approximation theorem - Why going deeper? - Regularization in deep networks + classic regularization: $L\_2$ regularization + implicit regularization: Dropout, Batch Normalization - Residual networks --- class: center, middle # Going deeper --- # Universal function approximation .bold[Theorem.] ( Hornik et al, 1991) Let $\sigma$ be a nonconstant, bounded, and monotonically-increasing continuous function. For any $f \in C([0, 1]^{d})$ and $\varepsilon > 0$, there exists $h \in \mathbb{N}$ real constants $v\_i, b\_i \in \mathbb{R}$ and real vectors $w_i \in \mathbb{R}^d$ such that: $$ | \sum\_i^h v\_i \sigma(w\_i^Tx + b\_i) - f (x) | < \varepsilon $$ that is: neural nets are dense in $C([0, 1]^{d})$. - It guarantees that even a single hidden-layer network can represent any classification problem in which the boundary is locally linear (smooth); - It does not inform about good/bad architectures, nor how they relate to the optimization procedure. - The universal approximation theorem generalizes to any non-polynomial (possibly unbounded) activation function, including the ReLU (Leshno, 1993). .citation[K. Hornik et al., Approximation Capabilities of Multilayer Feedforward Networks, 1991] --- .bold[Theorem] (Barron, 1992) The mean integrated square error between the estimated network $\hat{F}$ and the target function $f$ is bounded by $$O\left(\frac{C^2\_f}{q} + \frac{qp}{N}\log N\right)$$ where $N$ is the number of training points, $q$ is the number of neurons, $p$ is the input dimension, and $C\_f$ measures the global smoothness of $f$. Provided enough data, it guarantees that adding more neurons will result in a better approximation. .credit[Slide credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- # Problem solved? UFA theorems **do not tell us**: - The number $h$ of hidden units is small enough to have the network fit in RAM. - The optimal function parameters can be found in finite time by minimizing the Empirical Risk with SGD and the usual random initialization schemes. --- # Approximation with ReLU nets .left-column[ ```python import numpy as np import matplotlib.pyplot as plt def relu(x): return np.maximum(x, 0) def rect(x, a, b, h, eps=1e-7): return h / eps * ( relu(x - a) - relu(x - (a + eps)) - relu(x - b) + relu(x - (b + eps))) x = np.linspace(-3, 3, 1000) *y = ( rect(x, -1, 0, 0.4)) plt.plot(x, y) ``` ] .right-column[
] .reset-columns[ ] .citation[Conner Davis, [Quora: Is a single layered ReLU network still a universal approximator?](https://www.quora.com/Is-a-single-layered-ReLu-network-still-a-universal-approximator)] --- # Approximation with ReLU nets .left-column[ ```python import numpy as np import matplotlib.pyplot as plt def relu(x): return np.maximum(x, 0) def rect(x, a, b, h, eps=1e-7): return h / eps * ( relu(x - a) - relu(x - (a + eps)) - relu(x - b) + relu(x - (b + eps))) x = np.linspace(-3, 3, 1000) *y = ( rect(x, -1, 0, 0.4) * + rect(x, 0, 1, 1.3) * + rect(x, 1, 2, 0.8)) plt.plot(x, y) ``` ] .right-column[
] .reset-columns[ ] .citation[Conner Davis, [Quora: Is a single layered ReLU network still a universal approximator?](https://www.quora.com/Is-a-single-layered-ReLu-network-still-a-universal-approximator)] --- # Approximation with ReLU nets .left-column[ ```python import numpy as np import matplotlib.pyplot as plt def relu(x): return np.maximum(x, 0) def rect(x, a, b, h, eps=1e-7): return h / eps * ( relu(x - a) - relu(x - (a + eps)) - relu(x - b) + relu(x - (b + eps))) *x = np.arange(0,5,0.05) # 10 z = np.arange(0,5,0.001) sin_approx = np.zeros_like(z) *for i in range(2, x.size-1): * sin_approx = sin_approx + rect(z,(x[i]+x[i-1])/2, * (x[i]+x[i+1])/2, np.sin(x[i]), 1e-7) plt.plot(x, y) ``` ] .right-column[
] .reset-columns[ ] .citation[Conner Davis, [Quora: Is a single layered ReLU network still a universal approximator?](https://www.quora.com/Is-a-single-layered-ReLu-network-still-a-universal-approximator)] --- # Approximation with ReLU nets .left-column[ ```python import numpy as np import matplotlib.pyplot as plt def relu(x): return np.maximum(x, 0) def rect(x, a, b, h, eps=1e-7): return h / eps * ( relu(x - a) - relu(x - (a + eps)) - relu(x - b) + relu(x - (b + eps))) *x = np.arange(0,5,0.25) # 20 z = np.arange(0,5,0.001) sin_approx = np.zeros_like(z) *for i in range(2, x.size-1): * sin_approx = sin_approx + rect(z,(x[i]+x[i-1])/2, * (x[i]+x[i+1])/2, np.sin(x[i]), 1e-7) plt.plot(x, y) ``` ] .right-column[
] .reset-columns[ ] .citation[Conner Davis, [Quora: Is a single layered ReLU network still a universal approximator?](https://www.quora.com/Is-a-single-layered-ReLu-network-still-a-universal-approximator)] --- # Approximation with ReLU nets .left-column[ ```python import numpy as np import matplotlib.pyplot as plt def relu(x): return np.maximum(x, 0) def rect(x, a, b, h, eps=1e-7): return h / eps * ( relu(x - a) - relu(x - (a + eps)) - relu(x - b) + relu(x - (b + eps))) *x = np.arange(0,5,0.1) # 50 z = np.arange(0,5,0.001) sin_approx = np.zeros_like(z) *for i in range(2, x.size-1): * sin_approx = sin_approx + rect(z,(x[i]+x[i-1])/2, * (x[i]+x[i+1])/2, np.sin(x[i]), 1e-7) plt.plot(x, y) ``` ] .right-column[
] .reset-columns[ ] .citation[Conner Davis, [Quora: Is a single layered ReLU network still a universal approximator?](https://www.quora.com/Is-a-single-layered-ReLu-network-still-a-universal-approximator)] --- # Approximation with ReLU nets .left-column[ ```python import numpy as np import matplotlib.pyplot as plt def relu(x): return np.maximum(x, 0) def rect(x, a, b, h, eps=1e-7): return h / eps * ( relu(x - a) - relu(x - (a + eps)) - relu(x - b) + relu(x - (b + eps))) *x = np.arange(0,5,0.01) # 500 z = np.arange(0,5,0.001) sin_approx = np.zeros_like(z) *for i in range(2, x.size-1): * sin_approx = sin_approx + rect(z,(x[i]+x[i-1])/2, * (x[i]+x[i+1])/2, np.sin(x[i]), 1e-7) plt.plot(x, y) ``` ] .right-column[
] .reset-columns[ ] .citation[Conner Davis, [Quora: Is a single layered ReLU network still a universal approximator?](https://www.quora.com/Is-a-single-layered-ReLu-network-still-a-universal-approximator)] --- # Approximation with ReLU nets Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1)$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2)$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3)$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3) + \dots$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3) + \dots$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3) + \dots$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3) + \dots$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3) + \dots$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3) + \dots$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3) + \dots$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3) + \dots$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3) + \dots$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- class: middle count: false Consider the 1-layer MLP $$f(x) = \sum w\_i \text{ReLU}(x + b_i).$$ This model can approximate any smooth 1D function with a linear combination of translated/scaled ReLU functions. $$f(x) = \sigma(w\_1 x + b_1) + \sigma(w\_2 x + b_2) + \sigma(w\_3 x + b_3) + \dots$$ .center[
] .credit[Figure credit: G. Louppe, INFO8004-1 Advanced Machine Learning] --- # Universal approximation Even if the MLP is able to represent the function, learning can fail for two different reasons: - the optimization algorithm may not be able to find the value of the parameters that correspond to the desired function - the training algorithm my,ight chose the wrong function as result of overfitting --- # Universal approximation .grid[ .kol-4-12[ .center[![](images/part4/toyset.png)] ] .kol-4-12[ .center[![](images/part4/toyset_logistic_regression.png)] ] .kol-4-12[ .center[![](images/part4/toyset_nn3.png)] ] ] --- # Universal approximation Adding more neurons .left-column[ .center.width-90[![](images/part4/toyset_nn_varying_1.png)] ] .right-column[ .center.width-90[![](images/part4/toyset_nn_varying_2.png)] ] --- # Overparametrization may help optimization ## Folklore experiment .left-column[ .center.width-60[![](images/part4/mlp_2.png)] _Step 1:_ Generate labeled data by feeding random input vectors into depth $2$ net with hidden layer of size $n$ ] .right-column[ .center.width-60[![](images/part4/mlp_2.png)] _Step 2:_ Difficult to train a new network using this labeled data with the same amount of hidden nodes. .red[It is much easier to train a new net with bigger hidden layer or addition layer] ] .citation[Livni et al.; 2014] --- class: middle # Benefits of depth --- # The benefits of depth .caption[GINN: Geometric Illustrations for Neural Networks (http://www.bayeswatch.com/2018/09/17/GINN/)] .center.width-40[![](images/part4/ginn.png)] --- # Efficient Oscillations with Composition .left-column[ ```python import numpy as np import matplotlib.pyplot as plt def relu(x): return np.maximum(x, 0) def tri(x): return relu( relu(2 * x) - relu(4 * x - 2)) x = np.linspace(-.3, 1.3, 1000) y = tri(x) plt.plot(x, y) ``` ] .right-column[
] .citation[M. Telgarsky, [Benefits of depth in neural networks](https://www.youtube.com/watch?v=ssaXJqG9Dz4), COLT 2016 ] --- # Efficient Oscillations with Composition .left-column[ ```python import numpy as np import matplotlib.pyplot as plt def relu(x): return np.maximum(x, 0) def tri(x): return relu( relu(2 * x) - relu(4 * x - 2)) x = np.linspace(-.3, 1.3, 1000) *y = tri(tri(x)) plt.plot(x, y) ``` ] .right-column[
] .citation[M. Telgarsky, [Benefits of depth in neural networks](https://www.youtube.com/watch?v=ssaXJqG9Dz4), COLT 2016 ] --- # Efficient Oscillations with Composition .left-column[ ```python import numpy as np import matplotlib.pyplot as plt def relu(x): return np.maximum(x, 0) def tri(x): return relu( relu(2 * x) - relu(4 * x - 2)) x = np.linspace(-.3, 1.3, 1000) *y = tri(tri(tri(x))) plt.plot(x, y) ``` ] .right-column[
] .center[ 1 more layer → 2x more oscillations ] .citation[M. Telgarsky, [Benefits of depth in neural networks](https://www.youtube.com/watch?v=ssaXJqG9Dz4), COLT 2016 ] --- # Efficient Oscillations with Composition .left-column[ ```python import numpy as np import matplotlib.pyplot as plt def relu(x): return np.maximum(x, 0) def tri(x): return relu( relu(2 * x) - relu(4 * x - 2)) x = np.linspace(-.3, 1.3, 1000) *y = tri(tri(tri(tri(x)))) plt.plot(x, y) ``` ] .right-column[
] .center[ 1 more layer → 2x more oscillations ] .citation[M. Telgarsky, [Benefits of depth in neural networks](https://www.youtube.com/watch?v=ssaXJqG9Dz4), COLT 2016 ] --- # Efficient Oscillations with Composition .left-column[ - Adding the parameters required for **one new layer** can **multiply by two the number of local oscillations** in the decision function of the model. - This is to be constrasted with the approach of adding parameters **on the same layer** (as in the rectangle example) that can only contribute an **additive number of new local oscillations**. ] .right-column[
] .citation[M. Telgarsky, [Benefits of depth in neural networks](https://www.youtube.com/watch?v=ssaXJqG9Dz4), COLT 2016 ] --- # Effect of depth .bold[Theorem] (Montúfar et al, 2014) A rectifier neural network with $p$ input units and $L$ hidden layers of width $q \geq p$ can compute functions that have $\Omega((\frac{q}{p})^{(L-1)p} q^p)$ linear regions. - That is, the number of linear regions of deep models grows **exponentially** in $L$ and polynomially in $q$. - Even for small values of $L$ and $q$, deep rectifier models are able to produce substantially more linear regions than shallow rectifier models.
.center.width-80[![](images/part4/folding.png)] --- # Depth and Parametric Cost .bold[Theorem](Telgarsky, 2016): There exists functions that can be approximated by a deep ReLU network with $\Theta(k^3)$ layers with a $\Theta(1)$ units that cannot be approximated by shallower networks with $\Theta(k)$ layers unless they have $\Omega(2^k)$ units. Note: the number of parameters of a deep network is typically quadratic with the number of units. This also **holds for ReLU convnets with max pooling layers**. .citation[M. Telgarsky, Benefits of depth in neural networks, COLT 2016] --- class: middle .center[For a fixed parameter budget deeper is better] .center[
] .citation.tiny[Montufar et al.; On the number of linear regions of deep neural networks; 2014] --- class: middle # The problem with depth --- class: middle Although it was known that _deeper is better_, for decades training deep neural networks was highly challenging and unstable. Besides limited hardware and data there were a few algorithmic flaws that have been fixed/softened in the last decade. --- # Vanishing gradients Training deep MLPs with many layers has for long (pre-2011) been very difficult due to the **vanishing gradient** problem. - Small gradients slow down, and eventually block, stochastic gradient descent. - This results in a limited capacity of learning. .center.width-60[
] .center[Backpropagated gradients normalized histograms (Glorot and Bengio, 2010).
Gradients for layers far from the output vanish to zero. ] .citation[Glorot and Bengio, Understanding the difficulty of training deep feedforward neural networks; AISTAT 2010] --- class: middle Consider a simplified 3-layer MLP, with $x, w\_1, w\_2, w\_3 \in\mathbb{R}$, such that $$f(x; w\_1, w\_2, w\_3) = \sigma\left(w\_3\sigma\left( w\_2 \sigma\left( w\_1 x \right)\right)\right). $$ Under the hood, this would be evaluated as $$\begin{aligned} u\_1 &= w\_1 x \\\\ u\_2 &= \sigma(u\_1) \\\\ u\_3 &= w\_2 u\_2 \\\\ u\_4 &= \sigma(u\_3) \\\\ u\_5 &= w\_3 u\_4 \\\\ \hat{y} &= \sigma(u\_5) \end{aligned}$$ and its derivative $\frac{\text{d}\hat{y}}{\text{d}w\_1}$ as $$\begin{aligned}\frac{\text{d}\hat{y}}{\text{d}w\_1} &= \frac{\partial \hat{y}}{\partial u\_5} \frac{\partial u\_5}{\partial u\_4} \frac{\partial u\_4}{\partial u\_3} \frac{\partial u\_3}{\partial u\_2}\frac{\partial u\_2}{\partial u\_1}\frac{\partial u\_1}{\partial w\_1}\\\\ &= \frac{\partial \sigma(u\_5)}{\partial u\_5} w\_3 \frac{\partial \sigma(u\_3)}{\partial u\_3} w\_2 \frac{\partial \sigma(u\_1)}{\partial u\_1} x \end{aligned}$$ .credit[Slide credit: G. Louppe] --- class: middle The derivative of the sigmoid activation function $\sigma$ is: .center[
] $$\frac{\text{d} \sigma}{\text{d} x}(x) = \sigma(x)(1-\sigma(x))$$ Notice that $0 \leq \frac{\text{d} \sigma}{\text{d} x}(x) \leq \frac{1}{4}$ for all $x$. .credit[Slide credit: G. Louppe] --- class: middle Assume that weights $w\_1, w\_2, w\_3$ are initialized randomly from a Gaussian with zero-mean and small variance, such that with high probability $-1 \leq w\_i \leq 1$. Then, $$\frac{\text{d}\hat{y}}{\text{d}w\_1} = \underbrace{\frac{\partial \sigma(u\_5)}{\partial u\_5}}\_{\leq \frac{1}{4}} \underbrace{w\_3}\_{\leq 1} \underbrace{\frac{\partial \sigma(u\_3)}{\partial u\_3}}\_{\leq \frac{1}{4}} \underbrace{w\_2}\_{\leq 1} \underbrace{\frac{\sigma(u\_1)}{\partial u\_1}}\_{\leq \frac{1}{4}} x$$ This implies that the gradient $\frac{\text{d}\hat{y}}{\text{d}w\_1}$ **exponentially** shrinks to zero as the number of layers in the network increases. Hence the vanishing gradient problem. - In general, bounded activation functions (sigmoid, tanh, etc) are prone to the vanishing gradient problem. - Note the importance of a proper initialization scheme. .credit[Slide credit: G. Louppe] --- # Rectified linear units Instead of the sigmoid activation function, modern neural networks are for most based on **rectified linear units** (ReLU) (Glorot et al, 2011): $$\text{ReLU}(x) = \max(0, x)$$ .center[
] .credit[Slide credit: G. Louppe] --- class: middle Note that the derivative of the ReLU function is $$\frac{\text{d}}{\text{d}x} \text{ReLU}(x) = \begin{cases} 0 &\text{if } x \leq 0 \\\\ 1 &\text{otherwise} \end{cases}$$ .center[
] For $x=0$, the derivative is undefined. In practice, it is set to zero. .credit[Slide credit: G. Louppe] --- class: middle Therefore, $$\frac{\text{d}\hat{y}}{\text{d}w\_1} = \underbrace{\frac{\partial \sigma(u\_5)}{\partial u\_5}}\_{= 1} w\_3 \underbrace{\frac{\partial \sigma(u\_3)}{\partial u\_3}}\_{= 1} w\_2 \underbrace{\frac{\partial \sigma(u\_1)}{\partial u\_1}}\_{= 1} x$$ This **solves** the vanishing gradient problem, even for deep networks! (provided proper initialization) Note that: - The ReLU unit dies when its input is negative, which might block gradient descent. - This is actually a useful property to induce *sparsity*. - This issue can also be solved using **leaky** ReLUs, defined as $$\text{LeakyReLU}(x) = \max(\alpha x, x)$$ for a small $\alpha \in \mathbb{R}^+$ (e.g., $\alpha=0.1$). --- class: middle The steeper slope in the loss surface speeds up the training. .center.width-30[![](images/part4/relu_impact.png)] .citation[A. Krizhevsky et al., ImageNet Classification with Deep Convolutional Neural Networks; NIPS 2012] --- # Many other activation functions - Each neuron/unit is followed by a dedicated activation function - Historically the _sigmoid_ and _tanh_ have been the most popular .center.width-50[![](images/part4/activation_functions.png)] - [Many other activation functions available](https://dashee87.github.io/data%20science/deep%20learning/visualising-activation-functions-in-neural-networks/) --- class: center, middle # Regularization --- # Under-fitting and over-fitting What if we consider a hypothesis space $\mathcal{F}$ in which candidate functions $f$ are either too "simple" or too "complex" with respect to the true data generating process? .center[![](images/part4/underoverfitting.png)] --- class: middle .center.width-60[![](images/part4/poly_overfit_1.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_overfit_2.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_overfit_3.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_overfit_4.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_overfit_5.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_overfit_6.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_overfit_7.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_overfit_8.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_overfit_9.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_overfit_10.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_overfit_11.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle .center.width-60[![](images/part4/poly_overfit_12.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle Our goal is to adjust the capacity of the hypothesis space such that the expected risk of the empirical risk minimizer gets as low as possible. We define the capacity of a set of predictors as its ability to model an arbitrary functional .center[![](images/part4/underoverfitting.png)] --- class: middle Although it is difficult to define precisely, it is quite clear in practice how to increase or decrease it for a given class of models. For example: - The degree of polynomials; - The number of layers in a neural network; - The number of training iterations; - Regularization terms. --- class: middle # Regularization --- .center.width-100[![](images/part4/regularization_recht.png)] --- # Regularization We can reformulate the previously used squared error loss $$\ell(y, f(x;\mathbf{w})) = (y - f(x;\mathbf{w}))^2$$ to $$\ell(y, f(x;\mathbf{w})) = (y - f(x;\mathbf{w}))^2 + \rho \sum_{d}^{D} w\_d^2$$
.Q.big.center[What will happen now?] --- # Regularization We can reformulate the previously used squared error loss $$\ell(y, f(x;\mathbf{w})) = (y - f(x;\mathbf{w}))^2$$ to $$\ell(y, f(x;\mathbf{w})) = (y - f(x;\mathbf{w}))^2 + \rho \sum_{d}^{D} w\_d^2$$
.center.big[This is called __$L\_2$ regularization__.] --- class: middle .center.width-60[![](images/part4/poly_l2_1.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_2.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_3.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_4.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_5.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_6.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_7.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_8.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_9.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_10.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_11.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_12.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_13.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_14.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_15.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_16.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: middle count: false .center.width-60[![](images/part4/poly_l2_17.png)] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- class: center, middle # Deep regularization --- class: middle .width-40[![](images/part4/recht_1.png)] .citation[C. Zhang et al., Understanding deep learning requires rethinking generalization, ICLR 2017] --- class: middle However ... .center.width-100[![](images/part4/recht_2.png)] .citation[C. Zhang et al., Understanding deep learning requires rethinking generalization, ICLR 2017] --- class: middle However ... .center.width-40[![](images/part4/recht_3.png)] .citation[C. Zhang et al., Understanding deep learning requires rethinking generalization, ICLR 2017] --- class: center, middle ### Most of the weights of the network are grouped in the final layers. .center.width-80[![](images/part4/lenet5.png)] --- # VGG-16 .center[
] .citation[K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, NIPS 2014] --- # Memory and Parameters ```md Activation maps Parameters INPUT: [224x224x3] = 150K 0 CONV3-64: [224x224x64] = 3.2M (3x3x3)x64 = 1,728 CONV3-64: [224x224x64] = 3.2M (3x3x64)x64 = 36,864 POOL2: [112x112x64] = 800K 0 CONV3-128: [112x112x128] = 1.6M (3x3x64)x128 = 73,728 CONV3-128: [112x112x128] = 1.6M (3x3x128)x128 = 147,456 POOL2: [56x56x128] = 400K 0 CONV3-256: [56x56x256] = 800K (3x3x128)x256 = 294,912 CONV3-256: [56x56x256] = 800K (3x3x256)x256 = 589,824 CONV3-256: [56x56x256] = 800K (3x3x256)x256 = 589,824 POOL2: [28x28x256] = 200K 0 CONV3-512: [28x28x512] = 400K (3x3x256)x512 = 1,179,648 CONV3-512: [28x28x512] = 400K (3x3x512)x512 = 2,359,296 CONV3-512: [28x28x512] = 400K (3x3x512)x512 = 2,359,296 POOL2: [14x14x512] = 100K 0 CONV3-512: [14x14x512] = 100K (3x3x512)x512 = 2,359,296 CONV3-512: [14x14x512] = 100K (3x3x512)x512 = 2,359,296 CONV3-512: [14x14x512] = 100K (3x3x512)x512 = 2,359,296 POOL2: [7x7x512] = 25K 0 FC: [1x1x4096] = 4096 7x7x512x4096 = 102,760,448 FC: [1x1x4096] = 4096 4096x4096 = 16,777,216 FC: [1x1x1000] = 1000 4096x1000 = 4,096,000 TOTAL activations: 24M x 4 bytes ~= 93MB / image (x2 for backward) TOTAL parameters: 138M x 4 bytes ~= 552MB (x2 for plain SGD, x4 for Adam) ``` .credit[Slide credit: C. Ollion & O. Grisel, M2DS Deep Learning] --- # Memory and Parameters ```md Activation maps Parameters INPUT: [224x224x3] = 150K 0 *CONV3-64: [224x224x64] = 3.2M (3x3x3)x64 = 1,728 *CONV3-64: [224x224x64] = 3.2M (3x3x64)x64 = 36,864 POOL2: [112x112x64] = 800K 0 CONV3-128: [112x112x128] = 1.6M (3x3x64)x128 = 73,728 CONV3-128: [112x112x128] = 1.6M (3x3x128)x128 = 147,456 POOL2: [56x56x128] = 400K 0 CONV3-256: [56x56x256] = 800K (3x3x128)x256 = 294,912 CONV3-256: [56x56x256] = 800K (3x3x256)x256 = 589,824 CONV3-256: [56x56x256] = 800K (3x3x256)x256 = 589,824 POOL2: [28x28x256] = 200K 0 CONV3-512: [28x28x512] = 400K (3x3x256)x512 = 1,179,648 CONV3-512: [28x28x512] = 400K (3x3x512)x512 = 2,359,296 CONV3-512: [28x28x512] = 400K (3x3x512)x512 = 2,359,296 POOL2: [14x14x512] = 100K 0 CONV3-512: [14x14x512] = 100K (3x3x512)x512 = 2,359,296 CONV3-512: [14x14x512] = 100K (3x3x512)x512 = 2,359,296 CONV3-512: [14x14x512] = 100K (3x3x512)x512 = 2,359,296 POOL2: [7x7x512] = 25K 0 *FC: [1x1x4096] = 4096 7x7x512x4096 = 102,760,448 FC: [1x1x4096] = 4096 4096x4096 = 16,777,216 FC: [1x1x1000] = 1000 4096x1000 = 4,096,000 TOTAL activations: 24M x 4 bytes ~= 93MB / image (x2 for backward) TOTAL parameters: 138M x 4 bytes ~= 552MB (x2 for plain SGD, x4 for Adam) ``` .credit[Slide credit: C. Ollion & O. Grisel, M2DS Deep Learning] --- # Dropout - First "deep" regularization technique - Remove units at random during the forward pass on each sample - Put them all back during test .center[
] .citation[Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR 2014] --- # Dropout ## Interpretation - Reduces the network dependency to individual neurons and distributes representation - More redundant representation of data ## Ensemble interpretation - Equivalent to training a large ensemble of shared-parameters, binary-masked models - Each model is only trained on a single data point - __A network with dropout can be interpreted as an ensemble of $2^N$ models with heavy weight sharing__ (Goodfellow et al., 2013) --- # Dropout .center[
] - One has to decide on which units/layers to use dropout, and with what probability $p$ units are dropped. - During training, for each sample, as many Bernoulli variables as units are sampled independently to select units to remove. - To keep the means of the inputs to layers unchanged, the initial version of dropout was multiplying activations by $p$ during test. - The standard variant is the "inverted dropout": multiply activations by $\frac{1}{1-p}$ during training and keep the network untouched during test. --- # Dropout Overfitting noise .center[
] .credit[Slide credit: C. Ollion & O. Grisel, M2DS Deep Learning] --- # Dropout A bit of Dropout .center[
] .credit[Slide credit: C. Ollion & O. Grisel, M2DS Deep Learning] --- # Dropout Too much: underfitting .center[
] .credit[Slide credit: C. Ollion & O. Grisel, M2DS Deep Learning] --- # Dropout .center.width-50[![](images/part4/dropout_curves_4.png)] .citation[Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR 2014] --- # Dropout Features learned on MNIST with one hidded layer autoencoders having 256 rectified linear units .center.width-80[![](images/part4/dropout_patterns.png)] .citation[Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR 2014] --- # Dropout Effect of _Dropout_ on sparsity .center.width-80[![](images/part4/dropout_sparsity.png)] - ReLUs used for both models - showing histogram of mean activations and histogram of activations .citation[Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR 2014] --- # Dropout ```py >>> x = torch.full((3, 5), 1.0).requires_grad_() >>> x tensor([[ 1., 1., 1., 1., 1.], [ 1., 1., 1., 1., 1.], [ 1., 1., 1., 1., 1.]]) >>> dropout = nn.Dropout(p = 0.75) >>> y = dropout(x) >>> y tensor([[ 0., 0., 4., 0., 4.], [ 0., 4., 4., 4., 0.], [ 0., 0., 4., 0., 0.]]) >>> l = y.norm(2, 1).sum() >>> l.backward() >>> x.grad tensor([[ 0.0000, 0.0000, 2.8284, 0.0000, 2.8284] [ 0.0000, 2.3094, 2.3094, 2.3094, 0.0000] [ 0.0000, 0.0000, 4.0000, 0.0000, 0.0000]]) ``` --- # Dropout ```py >>> x = torch.full((3, 5), 1.0).requires_grad_() >>> x tensor([[ 1., 1., 1., 1., 1.], [ 1., 1., 1., 1., 1.], [ 1., 1., 1., 1., 1.]]) >>> dropout = nn.Dropout(p = 0.75) >>> y = dropout(x) *>>> y *tensor([[ 0., 0., 4., 0., 4.], * [ 0., 4., 4., 4., 0.], * [ 0., 0., 4., 0., 0.]]) >>> l = y.norm(2, 1).sum() >>> l.backward() >>> x.grad tensor([[ 0.0000, 0.0000, 2.8284, 0.0000, 2.8284] [ 0.0000, 2.3094, 2.3094, 2.3094, 0.0000] [ 0.0000, 0.0000, 4.0000, 0.0000, 0.0000]]) ``` --- # Dropout For a given network ```py model = nn.Sequential(nn.Linear(10, 100), nn.ReLU(), nn.Linear(100, 50), nn.ReLU(), nn.Linear(50, 2)); ``` -- we can simply add dropout layers ```py model = nn.Sequential(nn.Linear(10, 100), nn.ReLU(), * nn.Dropout(), nn.Linear(100, 50), nn.ReLU(), * nn.Dropout(), nn.Linear(50, 2)); ``` --- # Dropout A model using dropout has to be set in __train__ or __test__ mode --- # Dropout A model using dropout has to be set in __train__ or __test__ mode The method `nn.Module.train(mode)` recursively sets the flag `training` to all sub-modules. ```py >>> dropout = nn.Dropout() >>> model = nn.Sequential(nn.Linear(3, 10), dropout, nn.Linear(10, 3)) >>> dropout.training True >>> model.train(False) Sequential ( (0): Linear (3 -> 10) (1): Dropout (p = 0.5) (2): Linear (10 -> 3) ) >>> dropout.training False ``` --- # Dropout A model using dropout has to be set in __train__ or __test__ mode ```py >>> dropout = nn.Dropout() >>> model = nn.Sequential(nn.Linear(3, 10), dropout, nn.Linear(10, 3)) >>> x = torch.full((1, 3), 1.0) *>>> model.train() Sequential ( (0): Linear (3 -> 10) (1): Dropout (p = 0.5) (2): Linear (10 -> 3) ) >>> model(x) *tensor([[ 0.5360, -0.5225, -0.5129]], grad_fn=
) >>> model(x) *tensor([[ 0.6134, -0.6130, -0.5161]], grad_fn=
) ``` --- # Dropout A model using dropout has to be set in __train__ or __test__ mode ```py >>> dropout = nn.Dropout() >>> model = nn.Sequential(nn.Linear(3, 10), dropout, nn.Linear(10, 3)) >>> x = torch.full((1, 3), 1.0) >>> model.train() Sequential ( (0): Linear (3 -> 10) (1): Dropout (p = 0.5) (2): Linear (10 -> 3) ) >>> model(x) tensor([[ 0.5360, -0.5225, -0.5129]], grad_fn=
) >>> model(x) tensor([[ 0.6134, -0.6130, -0.5161]], grad_fn=
) >>> *>>> model.eval() Sequential ( (0): Linear (3 -> 10) (1): Dropout (p = 0.5) (2): Linear (10 -> 3) ) >>> model(x) *tensor([[ 0.5772, -0.0944, -0.1168]], grad_fn=
) >>> model(x) *tensor([[ 0.5772, -0.0944, -0.1168]], grad_fn=
) ``` --- # Spatial Dropout As pointed out by Tompson et al. (2015), units in a 2d activation map are generally locally correlated, and dropout has virtually no effect. They proposed SpatialDropout, which drops channels instead of individual units. .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- # Spatial Dropout ```py >>> dropout2d = nn.Dropout2d() >>> x = Variable(Tensor(2, 3, 2, 2).fill_(1.0)) >>> dropout2d(x) Variable containing: (0 ,0 ,.,.) = 0 0 0 0 (0 ,1 ,.,.) = 0 0 0 0 (0 ,2 ,.,.) = 2 2 2 2 (1 ,0 ,.,.) = 2 2 2 2 (1 ,1 ,.,.) = 0 0 0 0 (1 ,2 ,.,.) = 2 2 2 2 [torch.FloatTensor of size 2x3x2x2] ``` --- # DropOut for uncertainty estimation .center.width-100[![](images/part4/mc_dropout.png)] .citation[A. Kendall, What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?, NIPS 2017] --- # Batch normalization We saw that maintaining proper statistics of the activations and derivatives was a critical issue to allow the training of deep architectures. It is the main motivation behind weight initialization rules (we'll cover them later). --- # Batch normalization We saw that maintaining proper statistics of the activations and derivatives was a critical issue to allow the training of deep architectures. It is the main motivation behind weight initialization rules (we'll cover them later). A different approach consists of explicitly forcing the activation statistics during the forward pass by re-normalizing them. __Batch normalization__ proposed by Ioffe and Szegedy (2015) was the first method introducing this idea. --- class:middle "Training Deep Neural Networks is complicated by the fact that the __distribution of each layer's inputs changes during training, as the parameters of the previous layers change__. This slows down the training by requiring lower learning rates and careful parameter initialization ..." .pull-right[(Ioffe and Szegedy, 2015)] .citation[S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; arXiv 2015] --- class:middle "Training Deep Neural Networks is complicated by the fact that the __distribution of each layer's inputs changes during training, as the parameters of the previous layers change__. This slows down the training by requiring lower learning rates and careful parameter initialization ..." .pull-right[(Ioffe and Szegedy, 2015)] .reset-column[ ] .center.width-60[![](images/part4/lenet5.png)] .citation[S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; arXiv 2015] --- class: middle Consider a simplified 3-layer MLP, with $x, w\_1, w\_2, w\_3 \in\mathbb{R}$, such that $$f(x; w\_1, w\_2, w\_3) = \sigma\left(w\_3\sigma\left( w\_2 \sigma\left( w\_1 x \right)\right)\right). $$ Under the hood, this would be evaluated as $$\begin{aligned} u\_1 &= w\_1 x \\\\ u\_2 &= \sigma(u\_1) \\\\ u\_3 &= w\_2 u\_2 \\\\ u\_4 &= \sigma(u\_3) \\\\ u\_5 &= w\_3 u\_4 \\\\ \hat{y} &= \sigma(u\_5) \end{aligned}$$ --- class:middle "Training Deep Neural Networks is complicated by the fact that the __distribution of each layer's inputs changes during training, as the parameters of the previous layers change__. This slows down the training by requiring lower learning rates and careful parameter initialization ..." .pull-right[(Ioffe and Szegedy, 2015)] .reset-column[ ] .center.width-60[![](images/part4/lenet5.png)] Batch normalization can be done anywhere in a deep architecture, and forces the activations' first and second order moments, so that the following layers do not need to adapt to their drift. .citation[S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; arXiv 2015] --- # Batch normalization Normalize activations in each **mini-batch** before activation function: **speeds up** and **stabilizes** training (less dependent on init) Batch normalization forces the activation first and second order moments, so that the following layers do not need to adapt to their drift. --- # Batch normalization Normalize activations in each **mini-batch** before activation function: **speeds up** and **stabilizes** training (less dependent on init) .center[
]
.citation[S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; arXiv 2015] --- # Batch normalization During training batch normalization __shifts and rescales according to the mean and variance estimated on the batch__. .center[
] As for dropout, the model behaves differently during train and test. --- # Batch normalization At **inference time**, use average and standard deviation computed on **the whole dataset** instead of batch Widely used in **ConvNets**, but requires the mini-batch to be large enough to compute statistics in the minibatch. --- # Batch normalization As dropout, batch normalization is implemented as a separate module `torch.BatchNorm1d` that processes the input components separately. ```py >>> x = torch.Tensor(10000, 3).normal_() >>> x = x * torch.Tensor([2., 5., 10.]) + torch.Tensor([-10., 25., 3.]) >>> x.mean(0) tensor([-9.9898, 24.9165,2.8945]) >>> x.std(0) tensor([2.0006, 5.0146, 9.9501]) >>> bn = nn.BatchNorm1d(3) >>> with torch.no_grad(): ... bn.bias.copy_(torch.tensor([2., 4., 8.])) ... bn.weight.copy_(torch.tensor([1., 2., 3.])) ... Parameter containing: tensor([2., 4., 8.]) Parameter containing: tensor([1., 2., 3.]) >>> y = bn(x) >>> y.mean(0) tensor([2.0000, 4.0000, 8.0000]) >>> y.std(0) tensor([1.0005, 2.0010, 3.0015]) ``` --- # Batch normalization `BatchNorm2d` example ```py >>> x = Variable(torch.randn(20, 100, 35, 45)) >>> bn2d = nn.BatchNorm2d(100) >>> y = bn2d(x) >>> x.size() torch.Size([20, 100, 35, 45]) >>> bn2d.weight.data.size() torch.Size([100]) >>> bn2d.bias.data.size() torch.Size([100]) ``` --- # Batch normalization Results on ImageNet LSVRC 2012: .center[
] .citation[S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; arXiv 2015] --- # Batch normalization Results on ImageNet LSVRC 2012: .center[
] - learning rate can be greater - dropout and local normalization are not necessary - $L^2$ regularization influence should be reduced .citation[S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; arXiv 2015] --- # Batch normalization Deep MLP on a 2d "disc" toy example (class $+1$ on the disk, $-1$ outside), with naive Gaussian weight initialization, cross-entropy, standard SGD, $\eta = 0.1$. ```py def create_model(with_batchnorm, nc = 32, depth = 16): modules = [] modules.append(nn.Linear(2, nc)) if with_batchnorm: modules.append(nn.BatchNorm1d(nc)) modules.append(nn.ReLU()) for d in range(depth): modules.append(nn.Linear(nc, nc)) if with_batchnorm: modules.append(nn.BatchNorm1d(nc)) modules.append(nn.ReLU()) modules.append(nn.Linear(nc, 2)) return nn.Sequential(*modules) ``` We try different standard deviations for the weights: ```py with torch.no_grad(): for p in model.parameters(): p.normal_(0, std) ``` .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- # Batch normalization .center[
] .credit[Slide credit: F. Fleuret, EE-559 Deep learning] --- The position of batch normalization relative to the non-linearity is not clear. .center.width-40[![](images/part4/activation-relu.png) ] -- In the original paper, Ioffe and Szegedy, added BN right before nonlinearity: $$\dots \to \texttt{LINEAR} \to \texttt{BN} \to \texttt{ReLU} \to \dots$$ -- However, there are arguments for both ways: activations after the non-linearity are less "naturally normalized" and benefit more from batch normalization. Experiments are generally in favor of this solution, which is the current default. $$\dots \to \texttt{LINEAR} \to \texttt{ReLU} \to \texttt{BN} \to \dots$$ --- # Layer Normalizations Normalize on the statistics of the **layer activations** instead of mini-batch. .center[
] .citation[Ba et al., Layer Normalization, 2016 ] -- The algorithm is then similar as Batch Normalization -- Suited for **RNNs**, degrades performance of **CNNs** --- # Multiple variants
.center[
] .citation[Y. Wu and K. He, Group Normalization, ECCV 2018] --- # Group Normalization .center.width-70[![](images/part4/gnorm_teaser.png)] .citation[Y. Wu and K. He, Group Normalization, ECCV 2018] --- # Group Normalization .center.width-80[![](images/part4/gnorm_code.png)] .citation[Y. Wu and K. He, Group Normalization, ECCV 2018] --- # Group Normalization
.center.width-70[![](images/part4/gnorm_activations.png)] .caption[Evolution of `conv5_3` activations during training] .citation[Y. Wu and K. He, Group Normalization, ECCV 2018] --- # Group Normalization .center.width-70[![](images/part4/gnorm_curves_1.png)] .center.width-70[![](images/part4/gnorm_curves_2.png)] .citation[Y. Wu and K. He, Group Normalization, ECCV 2018] --- # Weight Normalization Reparametrize weights of neurons, to decouple **direction** and **norm** of the weight: $$ \mathbf{w} = \frac{g}{||\mathbf{v}||}\mathbf{v} $$ .citation[T. Salimans et al., Weight normalization: A simple reparameterization to accelerate training of deep neural networks, NIPS 2016. ] -- One new parameter $g$ to learn per neuron: $g = {||\mathbf{w}||}$ -- For CNNs it's more convenient to normalize weights rather than activations: num_params $<$ num_activations -- Careful **data-based** initialization of $g$ and neuron bias $b$ is better (not applicable to RNNs) --- class: middle # How deep can we go now? --- # A saturation point If we continue stacking more layers on a CNN: .center[
] -- .center[.red[Deeper models are harder to optimize]] --- .left-column[ # ResNet ] .right-column[ .center[
] ] A block learns the residual w.r.t. identity .center[
] .citation[K. He et al., Deep residual learning for image recognition, CVPR 2016.] -- - Good optimization properties --- .left-column[ # ResNet ] .right-column[ .center[
] ] Even deeper models: 34, 50, 101, 152 layers .citation[K. He et al., Deep residual learning for image recognition, CVPR 2016.] --- .left-column[ # ResNet ] .right-column[ .center[
] ] ResNet50 Compared to VGG: ### Superior accuracy in all vision tasks
**5.25%** top-5 error vs 7.1% -- ### Less parameters
**25M** vs 138M -- ### Computational complexity
**3.8B Flops** vs 15.3B Flops -- ### Fully Convolutional until the last layer .citation[K. He et al., Deep residual learning for image recognition, CVPR 2016.] --- # ResNet ## Performance on ImageNet .center.width-90[![](images/part4/resnet_1.png)] --- # ResNet The output of a residual network can be understood as an ensemble, which explains in part its stability .center[
] .citation[A. Veit et al., Residual Networks Behave Like Ensembles of Relatively Shallow Networks, NIPS 2016] --- # ResNet ## Results .center.width-100[![](images/part4/resnet_3.png)] --- # ResNet Results .center[
] --- # ResNet In PyTorch: ```py def make_resnet_block(num_feature_maps , kernel_size = 3): return nn.Sequential( nn.Conv2d(num_feature_maps , num_feature_maps , kernel_size = kernel_size , padding = (kernel_size - 1) // 2), nn.BatchNorm2d(num_feature_maps), nn.ReLU(inplace = True), nn.Conv2d(num_feature_maps , num_feature_maps , kernel_size = kernel_size , padding = (kernel_size - 1) // 2), nn.BatchNorm2d(num_feature_maps), ) ``` --- # ResNet In PyTorch: ```py def __init__(self, num_residual_blocks, num_feature_maps) ... self.resnet_blocks = nn.ModuleList() for k in range(nb_residual_blocks): self.resnet_blocks.append(make_resnet_block(num_feature_maps , 3)) ... ``` ```py def forward(self,x): ... for b in self.resnet_blocks: * x = x + b(x) ... return x ``` --- For ResNet50+ layers some additional modifications need to be made to keep number of parameters and computations manageable .center.width-70[![](images/part4/resnet_bottleneck_1.png)] Such a block requires $2 \times (3 \times 3 \times 256 +1) \times 256 \simeq 1.2M$ parameters -- Adress this problem using __bottleneck__ block .center.width-100[![](images/part4/resnet_bottleneck_2.png)] .center[$256 \times 64 + (3 \times 3 \times 64 +1) \times 64 + 64 \times 256 \simeq 70K$ parameters] --- # Deeper is better .center.width-70[![](images/part4/deeper.png)] --- # Inception-V4 / -ResNet-V2 Deep, modular and state-of-the-art Achieves **3.1% top-5** classification error on imagenet .center.width-50[![](images/part4/inception1.png)] .citation[C. Szegedy et al.,Inception-v4, inception-resnet and the impact of residual connections on learning, 2016] --- # Inception-V4 / -ResNet-V2 More building blocks engineering... .center.width-80[![](images/part4/inception2.png)] .citation[C. Szegedy et al.,Inception-v4, inception-resnet and the impact of residual connections on learning, 2016] --- # Resnet variants: Stochastic Depth Networks - DropOut at layer level - Allows training up to 1K layers .center.width-80[![](images/part4/stochastic_depth_resnet.png)] .citation.tiny[Huang et al., Deep Networks with Stochastic Depth, ECCV 2016] --- # Resnet variants: DenseNet - Copying feature maps to upper layers via skip-connections - Better reuse of parameters and redundancy avoidance .center[
] .center[
] .citation[Huang et al., Densely Connected Convolutional Networks, CVPR 2017] --- # Visualizing loss surfaces .center.width-70[![](images/part4/loss_surf_1.png)] .citation[H. Li et al., Visualizing the Loss Landscape of Neural Nets, ICLR workshop 2018] --- # Visualizing loss surfaces Pick a center point $\theta^{*}$ in the graph and choose two direction vectors $\delta$ and $\eta$ .citation[H. Li et al., Visualizing the Loss Landscape of Neural Nets, ICLR workshop 2018] -- Plot a function of the form: - for 1D: $$f(\alpha)= \mathcal{L}(\theta^{*} + \alpha \delta)$$ - for 2D: $$f(\alpha, \beta)= \mathcal{L}(\theta^{*} + \alpha \delta + \beta \eta)$$ -- _A few more normalization and visualization hacks are required_ --- # Visualizing loss surfaces .left-column[ .center.width-100[![](images/part4/loss_surf_2.png)] ] .right-column[
- ResNet-20/56/110 : vanilla - ResNet-*-noshort: no skip connections - ResNet-18/34/50 : wide ] .citation[H. Li et al., Visualizing the Loss Landscape of Neural Nets, ICLR workshop 2018] --- # Visualizing loss surfaces .center.width-50[![](images/part4/loss_surf_3.png)] .citation[H. Li et al., Visualizing the Loss Landscape of Neural Nets, ICLR workshop 2018] --- # Outline - Universal approximation theorem - Why going deeper? - Regularization in deep networks + classic regularization: $L\_2$ regularization + implicit regularization: Dropout, Batch Normalization - Residual networks --- class: end-slide, center count: false The end.


12 Jun 2020 (Last Modified 21 Jun 2021)
[TOC]
I wanted to learn Danish (and Icelandic) ever since I traveled to Iceland in 2018. (I traveled on WOW airlines a year before it imploded). I remember taking an elevator up to the top of the Hallgrimskirkja with a few tourists whose language I couldn’t place. I chalked it up to my lack of familiarity with Germanic languages.
Over the next year I started working on Icelandic. While reading Independent People by Halldór Laxness, I realized that the language I overheard in the church elevator was probably Danish.
Spoken Danish vs Written Danish Danish, similar to English, is not written phonetically. I found the YouTube channel, Mic’s Languages, helpful, although my audio comprehension still lags considerably behind my understanding of the written word and that channel stopped posting new videos around May 2020. NativLang has an informative description of the progressive loss of consonants and increases in vowels.
This blog post has good links to books in Danish on learning Danish. Saxo is a good Danish book store that ships to the US and provides e-book access for some titles.
I read Politiken and listen to their daily summary podcast, du lytter til politiken.
Subject-Verb Danish is a V2 language, which leads to inversion when the subordinate clause precedes the main clause. The second position in the subordinate clause is the first available for the content of the sentence because the subordinate clause must begin with a subordinating conjunction (like because [fordi]). It also leads to inversion when something else occupies the first slot in a main clause, for example an adverb(ial phrase) or prepositional phrase.
For eksempel:
Object-Verb This also follows from the V2 requirement. It is a way to emphasize the object.
Adverb-Verb In the subordinate clause, adverbs, including ikke, preced the verb.
For eksempel:
| genåbning | reopening | det vil løbende blive udbygget | it will be continuously updated | |
| vækker formentlig glæde | probably brings joy | afgangsprøver | final exams | |
| afgørende | essential (d.v.s meget vigtig) | insatsen | the effort | |
| afklaring | clarification | at understrege | to emphasize | |
| indflydelse | influence (cf. påvirkning) | at anbefale | to recommend (cf. at foreslå eller at proponere [to propose]) | |
| engroshandel | wholesale | sektorpartnerskabet indrettet retningslinjerne | a coalition set out the guidelines | |
| dagtilbud | day care | trygge | safe | |
| at aflevere | to deliver | at bakke op | to support | |
| kommer også til at gælde i … | also applies to | udstrakt | Extensive | |
| muligheder | options | følgevirkninger | Consequences | |
| nedlukningen | the shutdown (cf. le confinement) | fordobling i antallet af smittede og daglige indlagte | doubling in the number of infected and daily admitted patients | |
| beregningerne | the calculations | forbundet med en vis usikkerhed. | associated with some uncertainty | |
| at medføre | to entail | røbede | revealed | |
| at udvide køb | to expand purchases | de har også kurs mod … | they are also heading to … | |
| I løbet af | during the … | at forhandle | to negotiate | |
| oprindelse (cf. rod, genese) | origin | undersøge (cf udforske, eksaminere) | examine, | |
| opstod | occurred | at fremlægge (cf, fremvise, udstille) | to present | |
| til gengæld | in return | kunnet feje en teori | could only dismiss (lit (sweep)) a theory | |
| en hændelse | an incident | udbrud | outbreak | |
| de første tilfælde | the first cases | omtalte | talked about | |
| begrænset sig | limited itself | udelukke | exclude | |
| smittetilfælde | cases of infection | netop | exactly | |
| sandsynligt | likely | forstærker | amplifier | |
| i forhold til boderne | in relation to the stalls | overbevisning | belief | |
| smittekæde | chain of infection | flagermus eller skældyr | bats or shellfish | |
| der i forvejen huser | which already houses (here cultures) | enslignende | similar | |
| vurdering | assessment | førhen | formerly | |
| anlager | (brought, as in brought to bear) | tillid | confidence | |
| at slå | to sound | håndteringen | the handling | |
| opsagen | to resign | set skævt til | looked askance at | |
| meldingen | messages, notifications | gentagen | repeated | |
| udmeldinger | announcements | forbygges | is prevented | |
| fastholdt | maintained | at bevise | to promote | |
| påstand | assertion | uddelte | distributed | |
| det blandt andet | among other things | fremhævet | highlighted | |
| efterfølgelse | follow-up | lemfældige omgang | careless handling | |
| menneskerettighederne | human rights | omfattende | comprehensive | |


After more than one year of learning at UNC Charlotte and four years of study at graduation, I'm now ready to change the world for better with data and software. I believe in being means-oriented along with being end-oriented. An instinctively calm and cooperative person, I have come to appreciate working with a team even more while working on endless team projects throughout my academic life. I'm keen to put my acquired knowledge to work and learn even more from work experience.
GPA: 4.0
GPA: 3.3
The aim of the project was to extract the names of famous living personalities from the Wikipedia Dump and then creating the Name Dominoes for the obtained output.
The voluminous amount pdf's related to business were converted into text files where WINSCP was used to load all the converted files on DSBA server. Further, Hadoop Shell scripting and Brat were used to gather the files with Customer KPI in one cluster and files with non-customer KPI into another clust
The purpose of the project was to develop an on-campus housing system where a student can login into the system and search/book his desired room and meal plans, applying several filters. Moreover, the students can also register complaints and can select a meal plan for them on the same application.
Rentiya is a home rental service and rental negotiation place that unifies tenants and landlords. It cuts down the role of the mediator and saves time and money. The agreements can be customized according to the user and the landlord. Rentiya is a home rental service and rental negotiation place that unifies tenants and landlords. It cuts down the role of the mediator and saves time and money. The agreements can be customized according to the user and the landlord
Developed a Market Basket Analysis Module for frequent item sets detection of the items purchased from the market. While working with 10 teammates, designed and developed a tool for sales predictive analysis. Provided major contribution in organizing the events conducted by the business organization.


Jojo Babie Unzips Ripped Jeans In Flirty Instagram Update
November 19, 2019
Jojo Babie Unzips Ripped Jeans In Flirty Instagram Update
Jojo Babie got flirty on Instagram yesterday as she rocked a pair of unzipped, ripped jeans. The stunner was seen standing for a selfie, with the pants being one of the main focal points. They were a light wash with giant holes on the thighs, including one on her right thigh and two on her left. She left the pants partly unzipped as she folded the top down, showcasing her toned midriff.
The model paired this with a long-sleeved crop top. It was a dark color with long sleeves and a low neckline, as she flaunted her cleavage. Jojo also wore a black baseball hat, and posed with her right hand behind her head. She also pulled her hair back in a side ponytail, which she brushed over her right shoulder. Bangs framed the side of her face, while the waves cascaded down the side of her chest.
There was a pop of red color in her ensemble, which included her bright red lipstick. Plus, she sported a pair of red heels. She also held her smartphone with her left hand, which had a turquoise case. Her French manicure could be seen.
Jojo didn’t wear any visible necklaces, rings, or bracelets. However, her belly button piercing could be easily seen.  Light streamed into the frame from her left, as she cast a shadow on the wall.
Fans poured into the comments section with a variety of compliments for the bombshell. Many people focused on the pizza, which she referred to in the captions.
“Wish I was the pizza delivery guy…,” joked a follower.
“D*mn everyone should start eating pizza if they can look that good,” expressed a fan.
Loading...
“I see you’re getting those pants ready for thanksgiving. Have an amazing Monday,” said an admirer.
Others mentioned Jojo’s good looks.
“You look great in those jeans and everything else lol,” noted an admirer.
In addition, the Instagram sensation shared another update last month that showed her rocking another crop top. This time, she was seen in a pink crop tank, which was so small that her underboob was on full display. She added a pair of tight booty shorts to the ensemble and posed outdoors by a cement column. Jojo popped her left knee while twirling her hair in her right hand. She gave a flirty pout and glanced down at the camera.


Validation Results¶
This section discusses damage and loss results from three comparative and validation studies with different emphases:
The Influence of Year Built (Zillow vs. NSI and Parametric Study) first explores the implications of utilizing year built based on two data resources (NSI and Zillow). As discussed in Asset Description, the Zillow-based model tends to predict higher year predictions (more recent constructions) than the NSI-based model, especially for the southern edge on the city. This is reflected by the fact that Zillow-based damage and loss in that region is lower than the NSI-based results. Following the parametric study in Verification of Rulesets, the observed trend in damage and loss with respect to year built is discussed.
The Validation: StEER contrasts the testbeds damage estimates versus the visually-assigned damage states from StEER members and explores influence of uncertainty in building attributes and peak wind speed. Several notable trends between building damage and building attribute type will be discussed and reasoned.
The Validation: FEMA Historical Geospatial Damage Assessment focuses on validating the testbed damage estimates against the FEMA Historical Damage Assessment Database (535 records associated with this testbed). The testbed results are found to slightly overestimate the damage for the undamaged state (DS-0) which suggests that future investigations are still needed given uncertainties around the methodology used to generate the FEMA data.
Influence of Year Built (Zillow vs. NSI and Parametric Study)¶
The year built is a critical attribute as it triggers attribute assignments by various rulesets.
As discussed in Asset Description, different sources of data were used
to predict the YearBuilt engaging both the National Structural Inventory (NSI) (with SURF correcting some
misalignments and gaps) and Zillow, which also provides the year built
information for many of the residual buildings in the studied region. The implications of using different
sources of year built information to project damage is explored in ds_zillow_comp
, noting the
contrast map that depicts the difference between the damage state projected by SURF-NSI and that projected
by Zillow. Note that the difference is not prominent in downtown Lake Charles, although divergence is noted
southwest of the city where the Zillow set predicts relatively lower damage states. Compared in
PDS_cdf
are the the CDF distributions of expected damage states (NSI vs. Zillow) as well as the
probability of exceeding DS-2 and DS-3. year_built_infl_bin
plots the cumulative distributions of
expected damage states for four different year built bins: Pre 1960, 1960-1979, 1980-1999, and Post 2000.
In general, we see improved building performance (low damage states) as code revisions by years
as recalling the parametric study in Verification of Rulesets, the dominate building type is
WSF1 (fig-inventoryPineChart
) whose wind-induced loss is particularly sensitive to nail spacing requirements
varied by years.
Validation: StEER¶
For Hurricane Laura, the Structural Extreme Reconnaissance (StEER) Network has released a dataset of the visual inspection and damage estimation for sample buildings in Lake Charles ([Roueche21], refereed as StEER buildings). There are 99 StEER buildings included in this testbed, which offers an opportunity to validate the workflow. In the meantime, the relatively small sample size also offers the convenience of investigating the influence of different modeling parameters on the estimated damage states and relating them to corresponding field observations to help understand the dominant trends in the data. As will be illustrated, some of these trends validate the effectiveness of the methodology and others indicate potential measures to improve the wind resistance.
Before presenting results, it is important to note, as discussed in StEER’s report on Hurricane Laura ([Roueche21]), that StEER’s assessments for Laura were conducted virtually from streetview imagery due to COVID-19 restrictions on travel; thus this “ground truth” is less reliable in discerning some aspects of the damage than a traditional StEER mission with in-person, up-close forensic assessments. Also note that since rulesets from New Jersey are applied to assign attributes and are not consistent with the historical regulatory environment in Louisiana, the predicted damage states are also likely to be lower, particularly for construction older than 2006 (the year Louisiana first implemented its statewide building codes linked to IBC/IRC). Finally, as also discussed in StEER’s report ([Roueche21]), there were low rates of compliance with shuttering requirements in the affected area (based on observations immediately after landfall). Thus it is likely that the shutter attributes are assigned in this workflow at a rate higher than actually observed in the event, leading to lower rates of predicted damage than those observed in the field.
influential_var
plots the estimated damage states against key building attributes that are found
to influence the resulting damage state under the hurricane wind hazard. Major observed trends are as follows:
In general, the damage state is negatively correlated to the year built. Buildings with built year after 2000 are found to have prominent improvement in the wind resistance and have lower damage states. This is reasoned by the authors to be led by the assumption in the ruleset that the post-2000 buildings would have shutter measures and stronger roof deck attachment. It is import to note, however, that this effect predicted based on New Jersey rulesets will not be observed in Louisiana until at least 2006 when statewise codes were enacted, and further, is likely to not be pronounced given the earlier observation about low rates of compliance with shuttering requirements.
The surface roughness, i.e., terrain feature, is found to be a key variable when considering the potential damage from the hurricane to the building. Even moving from open terrain to light suburban, following the damage functions in HAZUS, we see about 50% reduction in the average damage state. This is associated with the reduction in mean wind speed as surface roughness increases.
Shutters could significantly help reduce the potential wind-induced damage: average damage state is reduced from 2.2 to 1.2. This observation makes the low compliance noted by StEER particularly noteworthy.
Garage type also would influence the building performance against the wind hazard: garages per SFBC 1994 are found to have much better behaviors than the standard and weak garages. This effect is not expected to be pronounced in Lake Charles given the low incidence of attached garages noted in the samples discussed in Asset Description.
As expected, roof deck attachment using a tighter nail spacing (6 in vs. 12 in) is found to perform much better.
As also expected, gable roof is found to have slightly worse performance than the hip and flat roofs.
It is noted that the trends observed in influential_var
are based on a so-called “median” model representing
our best estimates of building attributes. However, in reality, because of the lack of high-resolution or multiple
data resources, we may not have 100% confidence about certain building attributes or hazard fields, e.g., year built
or peak wind speed at a specific site. Hence, the estimated “median” damage state from the “median” model may not give
the full representation of the real damage potential. As illustrated in ds_uq
, if considering the uncertainty
in the year built (instead of a deterministic value, assuming it follows a normal distribution with a standard deviation
of 10 years), we could sample year built from the distribution and repeat the damage assessment by sufficient number of times to
estimate the distribution of the damage state. This however assumes differences are attributable to uncertainties in
the year built and neglects the fact that the predictors are biased by attribute assignments using rulesets that are derived from
a difference regulatory environment (New Jersey) with a longer history of adopting and enforcing model building codes
statewide. This fact may explain why the predicted damage states bias toward lower levels of damage (assuming less
vulnerable structures than was the case in practice).
Following this idea, for each of the 99 StEER buildings, we sampled 100 year built values from a distribution with the mean of
the given year built value and a standard deviation of 10 years, and ran the damage assessment for each sampled case.
ds_uq_yb
summarizes the comparison between the mean and 95-percentile of estimated damage state and the
StEER damage state. Similar exercises (ds_uq_pws
) are conducted for the peak wind speed where we sampled 100 PWS from a
distribution with a standard deviation of 20 mph.
Validation: FEMA Historical Geospatial Damage Assessment¶
FEMA and the Department of Interior lead the development of the Geospatial Platform, an internet-based service environment that provides a suite of well-manged geospatial data, services, applications, and tools. The FEMA Historical Damage Assessment Database is a repository of geospatial damage assessments from past National Disaster events where damage assessments were conducted either using high-resolution imagery or by means of geospatial modeling.
For Hurricane Laura, the damage categories of 112,571 buildings in the Louisiana were modeled from flood depths at the
structure as characterized based on modeled wind, flood or surge data. Five damage categories are
used: No Damage (NOD), Affected (AFF), Minor (MIN), Major (MAJ), and Destroyed (DES). Out of the 112,571 buildings, 535 buildings, located in
the west bound of the Lake Charles city at the waterfront of the Prein Lake, are also investigated
in this testbed. fema_damage_subset
is a subset of data obtained from
FEMA Public Data.
fema_damage_bim
is the building inventory for this subset. The full inventory and FEMA damage assessment
results can be accessed by FEMA Damage Assessment of the 535 buildings and
BIM inventory of the 535 buildings.
ID |
DMG_LEVEL |
COUNTY |
STATE |
LONGITUDE |
LATITUDE |
|---|---|---|---|---|---|
1 |
DES |
Calcasieu |
LA |
-93.29 |
30.17 |
2 |
NOD |
Calcasieu |
LA |
-93.30 |
30.17 |
3 |
NOD |
Calcasieu |
LA |
-93.29 |
30.17 |
4 |
NOD |
Calcasieu |
LA |
-93.29 |
30.17 |
5 |
NOD |
Calcasieu |
LA |
-93.28 |
30.16 |
6 |
NOD |
Calcasieu |
LA |
-93.29 |
30.17 |
7 |
NOD |
Calcasieu |
LA |
-93.29 |
30.17 |
8 |
NOD |
Calcasieu |
LA |
-93.29 |
30.17 |
9 |
NOD |
Calcasieu |
LA |
-93.30 |
30.17 |
10 |
AFF |
Calcasieu |
LA |
-93.28 |
30.17 |
11 |
NOD |
Calcasieu |
LA |
-93.29 |
30.16 |
12 |
NOD |
Calcasieu |
LA |
-93.30 |
30.17 |
13 |
NOD |
Calcasieu |
LA |
-93.29 |
30.17 |
14 |
NOD |
Calcasieu |
LA |
-93.28 |
30.17 |
15 |
NOD |
Calcasieu |
LA |
-93.28 |
30.17 |
16 |
NOD |
Calcasieu |
LA |
-93.29 |
30.16 |
17 |
NOD |
Calcasieu |
LA |
-93.29 |
30.17 |
18 |
NOD |
Calcasieu |
LA |
-93.29 |
30.17 |
19 |
NOD |
Calcasieu |
LA |
-93.29 |
30.17 |
20 |
AFF |
Calcasieu |
LA |
-93.28 |
30.16 |
ID |
RoofShape |
FloodZone |
Area |
Longitude |
Latitude |
Terrain |
DSWII |
BuildingType |
AvgJanTemp |
Garage |
NumberOfStories |
YearBuilt |
RoofSlope |
MeanRoofHt |
OccupancyClass |
1 |
Gable |
X |
112 |
-93.29 |
30.18 |
51 |
132 |
Wood |
Above |
0 |
1 |
1994 |
0 |
12 |
RES3 |
2 |
Hip |
X |
148 |
-93.30 |
30.17 |
42 |
132 |
Wood |
Above |
0 |
1 |
1998 |
0.06 |
13.1 |
RES3 |
3 |
Gable |
X |
65 |
-93.29 |
30.17 |
11 |
132 |
Wood |
Above |
1 |
1 |
1994 |
0.08 |
13.5 |
RES3 |
4 |
Gable |
X |
110 |
-93.29 |
30.18 |
52 |
132 |
Wood |
Above |
0 |
1 |
1995 |
0.05 |
12.9 |
RES3 |
5 |
Hip |
X |
581 |
-93.29 |
30.17 |
11 |
132 |
Wood |
Above |
0 |
1 |
1996 |
0.05 |
17.3 |
RES1 |
6 |
Gable |
X |
83 |
-93.29 |
30.17 |
11 |
132 |
Wood |
Above |
0 |
1 |
1999 |
0.17 |
15.1 |
RES1 |
7 |
Hip |
X |
148 |
-93.30 |
30.17 |
42 |
132 |
Wood |
Above |
0 |
1 |
1998 |
0.06 |
13.1 |
RES3 |
8 |
Gable |
X |
55 |
-93.28 |
30.17 |
11 |
132 |
Wood |
Above |
0 |
1 |
1997 |
0.16 |
14.9 |
RES3 |
9 |
Hip |
X |
487 |
-93.29 |
30.16 |
11 |
132 |
Wood |
Above |
1 |
1 |
1996 |
0.07 |
13.2 |
RES1 |
10 |
Hip |
X |
107 |
-93.30 |
30.17 |
42 |
132 |
Wood |
Above |
0 |
1 |
1998 |
0 |
12 |
RES3 |
11 |
Gable |
X |
423 |
-93.29 |
30.17 |
11 |
132 |
Wood |
Above |
0 |
1 |
1998 |
0.13 |
11.3 |
RES1 |
12 |
Flat |
X |
144 |
-93.28 |
30.17 |
52 |
132 |
Wood |
Above |
1 |
1 |
1995 |
0.2 |
12 |
RES3 |
13 |
Gable |
X |
101 |
-93.29 |
30.17 |
42 |
132 |
Wood |
Above |
1 |
2 |
1997 |
0.24 |
16.4 |
RES1 |
14 |
Gable |
X |
86 |
-93.29 |
30.17 |
11 |
132 |
Wood |
Above |
0 |
1 |
1998 |
0.51 |
17.2 |
RES1 |
15 |
Hip |
X |
797 |
-93.29 |
30.17 |
51 |
132 |
Wood |
Above |
1 |
1 |
1995 |
0.04 |
12.8 |
RES3 |
16 |
Flat |
X |
90 |
-93.30 |
30.17 |
11 |
132 |
Wood |
Above |
0 |
2 |
1999 |
0.65 |
33.3 |
RES1 |
17 |
Gable |
X |
129 |
-93.29 |
30.17 |
11 |
132 |
Wood |
Above |
1 |
1 |
1997 |
0.24 |
16.4 |
RES3 |
18 |
Gable |
X |
704 |
-93.29 |
30.17 |
42 |
132 |
Wood |
Above |
1 |
2 |
1998 |
0.07 |
13.3 |
RES1 |
19 |
Gable |
X |
112 |
-93.29 |
30.17 |
11 |
132 |
Wood |
Above |
0 |
1 |
1998 |
0.11 |
16.3 |
RES3 |
20 |
Gable |
X |
137 |
-93.28 |
30.17 |
11 |
132 |
Wood |
Above |
0 |
1 |
1997 |
0 |
12 |
RES3 |
fema_dc
shows the damage categories collected from the FEMA Damage Assessment Database, where most buildings
are assessed as having No Damage or Affected. Also noted from fema_damage_subset
is that all these
buildings are subject to CAT-4 gust wind speed, which is consistent with our input wind field where the 3-s gust peak
wind speed is about 130 \(mph\) (please see Hazard Characterization).
To compare these damage categories with the estimated damage states by the workflow in this testbed, we assume that the
five damage categories are mapped to the five damage states in the HAZUS wind damage functions, assuming the inundation does
not cause prominent flood-induced damage to the buildings. While this assumption will be the subject of future validation,
fema_ds_comp
summarizes (1) the expected damage states of the 535 buildings, (2) the spatial distribution of difference between the FEMA
Damage Categories and estimated damage states, and (3) the statistical distribution of the difference between the two.
Given the FEMA set is based on a discrete (category-based) description while the testbed set is based on probabilistic
estimates, the two results have fairly good agreement except the FEMA set has buildings predicted as no damage (DS-0).
This over-prediction of DS-0 is also seen in StEER set results (ds_uq_yb
and ds_uq_pws
). The uncertainty
in the year built and wind speed, to some extent, can help explain this over-prediction, as can the fact that
the rulesets applied are translated directly from New Jersey. It is further important to
note that FEMA assessments also consider the effects of storm surge. Future investigations are still
needed to consider other possible factors causing this mismatch in predicted DS-0.
- Roueche21(1,2,3)
Roueche, D. Kameshwar, S. Vorce, M. Kijewski-Correa, T. Marshall, J. Mashrur, N. Ambrose, K. Brown, C. Childress, O. Fox, D. Morris, K. Rawajfih, H. Rodriguez, L. (2021) “Field Assessment Structural Teams: FAST-1, FAST-2, FAST-3”, in StEER - Hurricane Laura. DesignSafe-CI. https://doi.org/10.17603/ds2-dha4-g845


Home
Note
Vocabulary
Math
Spelling
Number
Help
Contact
About
Login
Signup
FAQ
1. How do you make the web?
I need to learn
HTML
,
CSS
,
JAVASCRIPT
and
BOOTSTRAP
in
W3schools
2. How to join
Click on the login or signup link to link to the signup or login page.
3. How to get more help?
First contact me at the taskbar near the logo
4. How to use
First you go to the contact to call and send requests or
signin/login
to explore more
TODAY!
5. How do you go home?
Click
on the home button on the
orange taskbar
6. How to Explore?
Goto the
Home, Note, Vocabulary and Math or something else to get more information and explore and have more
SUPER-TIME!
7. How to get more information?
Information:
Informations are going to come in
26/8/2018
8. How to find your
NOTIFICATION?
Click
on the menu icon on the menu-bar
9. How to
Note your things
Click
on
Note
on the menu-bar


{% extends "helpdesk/public_base.html" %}{% load i18n %} {% block helpdesk_body %}
| {{ item.question }} |
|---|
| {{ item.answer }} |
{% blocktrans with item.category.title as category_title and item.category.get_absolute_url as category_url %}View other {{ category_title }} articles, or continue viewing other knowledgebase articles.{% endblocktrans %}
{% trans "We give our users an opportunity to vote for items that they believe have helped them out, in order for us to better serve future customers. We would appreciate your feedback on this article. Did you find it useful?" %}
{% trans "The results of voting by other readers of this article are below:" %}


--- title: "Accepted Posters and Demos" updated: 2016-10-05 00:00:00 layout: default group: program menu: sub ordered: 2 status: published ---