About this presentation
This is a 30,000 foot view
The goal is to convey topic breath and not depth
We'll cover what modern web devs are doing
Lastly, this is a highly opinionated presentation - it's ok if you disagree
Presentation source at:
http://github.com/cjus/modern-web-development-presentation
Issues with this presentation? Fix it and issue a pull request
---
> The subject of web development is too vast to cover in a short presentation
> Because we have a lot to cover I won't be doing any deep dives.
> This presentation should be a jumping off point for your own further investigation.
> I may not cover your favorite tool. That doesn't mean it's not cool. Definitely tell us about it after the
presentation.
> So, the goal of this presentation is to discuss web development and to consider what modern web devs are doing
these days.
> This is my highly opinionated view. You may disagree. As is common in these sorts of things, there are lots of
ways to accomplish similar goals.
> This presentation is also open source and available on Github.
> If you find any issue with this presentation drop me an email or better yet, issue a pull request.
What is modern web development?
---
> So, what is modern web development?
In short, it's about working smarter and not harder...
That's the "modern" bit
---
> In short it's simply about working smarter using tools and automation.
Signs you may not be practicing modern web development
---
> Some signs that you or someone you know may not be practicing modern web development...
If you or your team...
build and edit source files using a generic text editor
deploy raw files to web servers without packaging
work in monolithic source files with no real code organization
don't use a library or framework such as: Backbone, Ember, Knockout, React or Angular
---
> Do you or your team - build and edit code using generic text editors?
> How about deploying raw files to web servers without packaging?
> Does your project contain monolithic files which lack any real organization?
> Are you using a home grown framework instead of one of the well established ones such as Backbone, Ember,
Knockout, React or Angular?
Or if you or your team...
don't have a need for source control - zip files work just fine
don't mind manual and repetitive processes as you work on your projects
don't use code linters, style checkers, or beautifiers
don't do any real testing
---
> Do you feel that file folders and zip files are a convenient ways of tracking your code changes?
> Do you repeat manual processes instead of automating them?
> Do you think that linters, style checkers and beautifiers are for wimps?
> Do you feel that perhaps you don't adequately test your code?
> Do you you feel that the beauty and elegance of your code makes it self documenting?
If so, then your dev cycles might feel a bit like this...
If that describes your projects - then your dev cycles might feel a bit like this...
Jane! Stop this crazy thing!
---
> Modern web tools and practices help you take control of your dev cycles
> and add predictability to your software releases.
> Let's take a closer look.
How can modern tools and practices help?
---
> So, how can modern tools and practices help?
> I'm super glad you asked!
By providing better tools and processes
---
> Simply stated: by providing better tools and processes
Tools of the trade
---
> Lets discuss some of the tools available to web developers
---
> But before we begin, let's start with the elephant in the room.
---
> Mac or PC?
> Believe it or not, this question comes up more than you'd expect.
Mac or PC?
Generally speaking, one isn't better than the other
It depends on what you're trying to do
For web development, it seems Mac is a clear winner
That is, unless you're a .NET developer
---
> Which is better?
> Generally speaking it depends on what you're trying to do.
> It also depends on your level of taste
> and build quality expectations.
> I spent the past few years working with startups
> in the San Francisco bay area where I attended lots of meetups.
> In my experience, web developers seem to clearly favor Macs.
> However the same is true at the company I currently work with in mid-town New York.
> So I'm not sure it's a east versus west coast thing.
Why Mac?
Most servers on the web are running Linux
MacOSX is based on Unix, you get a compatible terminal environment
The products you run on Linux based servers, often have binaries for Mac: Apache, Nginx, Redis, MongoDB
etc...
Develop native iOS and Android mobile applications
Fact is you can do all of this on a PC, it just requires you to run more software while leaping through
hoops
---
> There are some reasons this might be the case.
> The web is powered by Linux machines.
> For developers working with Linux, Macs offer compatible terminal environments
> The two very popular web servers, Apache and Nginx were both born and bread on Linux. They run natively on Mac without the need for extra software to manage them.
> For a long time, most popular Internet software didn't even run on Windows.
> Microsoft has invested heavily ensuring that's no longer the case. Largely because of their Azure, cloud platform. The company that was allergic to open source is now embracing it.
> A lot has changed in recent years.
> Another consideration is if you're developing native mobile products on iOS or Android
> Macs allow you to do both.
> So what if you're stuck with a PC?
> We have a saying in photography... the best camera is the one you have have with you.
> The fact is that you can do all this on a PC, it just requires you to jump through more hoops.
That said, some folks like that sort of thing
---
> That said, some folks like that sort of thing
What about VMs?
Virtual Machines are software based machines
Using VM software you can run a Linux machine on your Mac or PC
Virtual Box: https://www.virtualbox.org
Vagrant: https://www.vagrantup.com
Docker: https://www.docker.com
---
> What about VMs?
> A popular option is to use virtual machine software
> on your Mac or PC, in order to run Linux.
> I use Macs and my work laptop has a linux VM running so I can develop locally while accessing a web server that
routes calls to company API servers.
> Oracle has a free VM product called Virtual Box that runs on both Mac's and PC's.
> Vagrant is a product that can sit on top of Virtual Box in order to allow developers to setup lightweight and
portable development environments.
> The idea is that by hosting your dev environment in Vagrant, every member of the team can have identical local
access to the same type of environment.
> Docker is a platform that has taken the developer community by storm in recent years. With Docker you can create
an application and distribute it to other developers for local use and to sysadmins for deployment to cloud servers.
The entire docker movement is part of a trend called containerization where applications are placed in virtual
containers.
> In truth, the containers are hydrated, meaning that what is shared is a definition file that can be used to
download and configure an application... but that's getting a bit more technical.
> So lots of options, each with pros and cons.
Ubuntu Linux running in a VM under Windows 7
---
> Here we see Ubuntu Linux running in a VM under Windows.
> This totally works. However, it's not the speediest of things.
> If you have a machine with lots of memory and spare CPU cores then YOU might be just fine.
> I've done this on an 8GB Dell with quad core CPUs and solid state drives and it was managable.
Text Editor or IDE?
---
> OK, let's move on to Text editors and Integrated Development Environments or IDEs.
Text editor or IDE?
Any text editor will let you edit code
A text editor designed with web dev in mind will do far more for you
An Integrated Development Environment (IDE) complements its built-in text editor with a host of integrated
tools
However, today's modern editors feel like lightweight IDEs
It's not aways easy to decide which one is best for you
---
> Any text editor will let you edit your code.
> However, a text editor designed for web development will help you do much more than just editing.
> At the opposite end of web friendly text editors are IDEs, Integrated Development Environments.
> These products typically integrate tools which extend what you can do within a common environment.
> For some, such as yours truly, this can be a nice advantage.
> For example, text editors don't typically allow you to debug your code from within the editor - instead you do this outside of the editor. Where as, some IDEs allow you to debug right inside the IDE.
> It is worth noting that many of today's web dev text editors go beyond being basic text editors and also offer a
fare amount of tool integration.
> It's definitely not easy to decide which one is best for you.
> Most devs simply use
> whatever the cools kids are using...
> what ever that might be this week.
> My tip for you is to compare the costs, and features and see which one best fits your needs. For me, webstorm
offered the ability to debug node applications and that was and remains an important feature.
Modern web editors and IDEs
Allow you to do more than just edit files
They include support for plugins which can integrate linters such as JSLint and JSHint to help you spot
errors
They allow you to more quickly create HTML and CSS, checkout http://emmet.io/
Some even understand JavaScript syntax to allow you to more quickly navigate your source files
---
> A wikipedia page list over 75 text editors.
> However, modern web editors differ from their older counterparts by being designed to support developers working on web projects.
> A super popular editor is Sublime.
> Another is Webstorm, which is more of an Integrated Development Environment (IDE) than just an editor.
> Which you use depends on your needs.
> Both Sublime and Webstorm need to be purchased, however, excellent free options include Brackets.io and Atom.
This is an IDE called WebStorm
The panel on the left shows a project file structure
Bars at the top contain quick access features and navigation helpers
The panel on the right shows a text view. The red line in front of the closing section tag shows us an error
---
> Here's a screen shot of WebStorm.
> We can see that our window view consists of panels, each of which serves a distinct purpose.
> At the top of the window we see quick access features and navigation helpers.
> Notice how we can see that we're looking at the section tag which is inside of a div with a class name of slides which is inside of a div with a class name of reveal.
> We can also click on those tags to quickly navigate the html document.
> The panel on the far left shows a project file structure, while the larger panel region on the right shows our source file.
> Notice the red squiggly line just before the closing section tag.
> This indicates an error which the IDE found for us. If we hover our mouse cursor over the red line we'd see that the cause of the error is a missing a closing aside tag.
> So that's an example of an editor can help make your job easier.
JSHint integration
---
> This is Webstorm again, here we see JSHint integration where we're being alerted to something we shouldn't be
doing.
Debugging a NodeJS app
---
> In this example I'm debugging a NodeJS app.
> The sever just received an HTTP request which I'm inspecting.
There are lots of great code editors to choose from!
Sublime ($70): http://www.sublimetext.com/
WebStorm ($99): https://www.jetbrains.com/webstorm/
Atom ($0): https://atom.io/
Brackets ($0): http://brackets.io/
and many many more...
---
> This is a list of popular options, but by no means a comprehensive list.
> Both Sublime and Webstorm are commercial, but Atom and Brackets are both free. Don't let the cost fool you.
You'd be surprised by what both Atom and Brackets can do.
> One fun fact about Atom and Brackets is that they're actually built using HTML, CSS and JavaScript - which
really speaks to the power of modern web technologies.
Debugging in the browser
Yes, you can use your web browser to debug your site.
Modern browsers include a suite of built-in tools
Firefox and Chrome have excellent tools for web developers
Google Chrome Canary: https://www.google.com/chrome/browser/canary.html
Firefox Developer Edition: https://www.mozilla.org/en-US/firefox/developer
ProTip: Master those tools!
---
> OK, this one might be obvious, but not necessarily for people new to web development
> Modern web browsers have built-in tools which can be used to diagnose a wide variety of web related concerns.
> From the profiling of network calls to the debugging of program state using breakpoints.
> If you only casually use those tools, then you may not be getting the most out of them!
> Have a deeper look.
> As a modern web developer you should be using Chrome Canary or Firefox Developer Edition.
Improving code quality and reducing errors
Use JSHint: http://jshint.com/
Use js-beautify: https://github.com/beautify-web/js-beautify
Use JSCS (JavaScript Code Style): https://www.npmjs.com/package/jscs
Good news: your (modern) text editor might already support plugins for these tools
---
> One area that makes a huge difference is code quality.
> The more error prone your code, the less likely it will meet customer needs.
> Modern devs understand that through code reviews are time consuming and policing code is a thankless job.
> You just end up pissing off your fellow devs.
> That's why successful devs use tools like jshint or jslint to spot errors and questionable practices.
> They also use tools like js-beautify to spots code that's not properly formatted and JSCS (The Javascript code
style checker) to ensure that code maintains an agreed upon look and feel.
> The good news is that many editors, (especially the four we looked at) include support for these tools.
Helper libraries
jQuery, jQueryUI
BootStrap
Underscore and Lodash
_.uniq([1, 2, 1]);
// → [1, 2]
function timesThree(n) {
return n * 3;
}
_.map([1, 2], timesThree);
// → [3, 6]
---
> There are lots of add-on libraries that can help you develop web applications.
> jQuery in particular has become a staple for most web developers.
> Libraries like BootStrap allow you to quickly style an application. This has been a god send for artistically
challenged developers the world over!
> Libraries like Underscore and Lodash offer something entirely different. They provide developers with new programming
statements (really functions) which simplify common programming tasks.
> They offer a pattern inspired by functional programming, which has been proven to lead to fewer errors.
> Take the _.uniq statement for example. Given an array it returns an array of unique values.
> Handy right?
> Next we have a function called timesThree which multiples a number argument by 3 and returns the new value.
> By using the _.map statement we can take an array of values and map them using the timeThree function, that is,
pass each array element to the timesThree function and store the resulting values in a new array.
> Lodash is considered the more feature rich and modern of the two, but check out both underscore and lodash.
> This brings us to an interesting challenge.
> The more libraries we add to our projects, the more we need a tool to maintain and manage our app's dependencies.
Maintaining your app's dependencies
Modern web apps often include lots of package dependencies
From jQuery to Bootstrap to a libraries such as Ember or React
Tools exist to manage project dependencies. Get to know Bower - a package manager
---
> Modern web devs use dependency package managers such as Bower and NPM to manage their project dependencies.
> You basically list your dependencies in a configuration file so anyone team member can grab the project code and
easily install required dependencies.
> A package manager such as Bower will help you track and maintain your app's dependencies. When it comes time to
upgrade a package you can use Bower to download and install the latest version.
Preparing files for the web
Modern web applications require that files be optimized and packaged for web deployment
Often using a combination of concatination and minification
It's not just for JavaScript, minifiers exists for HTML, CSS and image files!
---
> Improving site performance can lead to a competitive advantage as customers prefer your site over your
competitors.
> In a study, Amazon determined that a page load slowdown of just one second could cost them $1.6 billion in sales
per year.
> Google determined that if its search results were four tenths of a second slower they could lose 8 million
searches per day. The result of which would be millions fewer online ads.
> Clearly, performance matters.
> Optimizing a web application for production use involves intelligently packaging site files so that they take up
less space and load faster.
> This process involves tools called minifiers which translate file contents into smaller version by renaming code variable and doing things like removing white space characters.
> Minifiers exists for HTML, CSS and JavaScript.
> There are also image minifiers which change the compression ratio on images to reduce their size without
impacting quality in a noticeable manner.
> If you're not using one of them, check them out!
> We'll talk more about this shortly.
Get with Git
It's distributed source control - for you and your team!
Seriously - source control - it's not optional!
Why Git is better: http://thkoch2001.github.io/whygitisbetter
---
> You should be using a source control system for all of your work.
> There's no good excuse for not doing so.
> The benefits of a source control system is that you can track changes to the work you're doing.
> How many of us haven't at one time or another wished we could go back to an earlier version of a file - even if just to compare it against a more recent change?
> Seriously - source control - it's not optional!
> One, free and wildly successful source control system is called Git. Git was initially created by Linus Torvalds
(the creator Linux) back in 2005.
> Github is a service which is based on Git, but adds loads of additional social features.
Transpilers
Tools that convert from one language to another
Typically for the purpose of extending or simplifying a language or using language features before they become
widely available
Examples include: CoffeeScript, SASS/Compass, ES6 to ES5 transpilers, such as Traceur and BabelJS
There's Asm.JS to enable C++ to JS transpilation
Checkout: https://developer.mozilla.org/en-US/demos/detail/bananabread
---
> Transpilers are tools that convert from one language to another. Typically for the purpose of extending a
language or using language features before they become widely available.
> Examples include:
> CoffeeScript, a transpiler that simplifies JavaScript for Ruby developers terrified of actually learning
JavaScript.
> Ok, cheap shot.
> Ex: SASS/Compass, empowers devs with more expressive CSS which gets transpiled to CSS3 during a build process.
> Then there's Asm.JS to enable C++ to JS transpilation.
> BTW, If you haven't seen the Bananabread demo you really should check it out.
Use JavaScript libraries and frameworks
---
> Hint... this is not the Beatles.
JavaScript libraries and frameworks
Lots to choose from
Adds structure to your application
Helps improve maintainability and testing
Offers battle hardened code
Dictates how module will be built
---
> There are dozens of JavaScript libraries and frameworks to choose from.
> They each provide ways of helping you structure and build web projects.
> For non-trivial projects you really should be using one of them.
> Which one you **should use** can be a subject of some heated debate.
> As a project increase in size, more devs end up touching the code.
> Worse though is that not all of the devs end up sticking around.
> Quite often the original developers are long gone. This is why choosing a well-known library or framework
(which you didn't invent) can help when it comes time to hire new developers.
> So if you chose to use say Angular, you could hire or train developers to write code consistently with the rest
of the project. New team members then simply need to learn where files are stored and how the project is assembled
in order to get up to speed.
> Best of all
> they can use books and online training materials to get up to speed. Again, because you didn't
invent the framework there's a greater chance that you'll find a helping hand when you need it.
> JS frameworks help with a concept called "Separation of concerns" by introducing other concepts such as Models,
Views, Controllers, Service, Routers and other good stuff.
> That in turn helps add structure and help improve maintainability.
> Another great benefit is that they provide you well tested and battle harden code which you didn't have to write
yourself!
> As would be expected, there are significant framework differences. A key difference is in how opinionated
they are.
> That is, on how they believe you should structure your application.
> A library like backbone.js has less opinions than a full-blown framework such as AngularJS.
> Some differ in how there allow you to interact with the DOM. Angular for instance, specifies how and when you should do so.
> Which library or framework you use
> depends on which opinions (points of view) you're willing to accept.
> Unless you suffer from not invented here syndrome, you will probably benefit from one of the many available frameworks.
---
> Web components are actually set of web standards for developing components.
> You'll use those components to build applications.
Posed to change how we develop web applications
This will impact how we work with all major frameworks
Web Components are actually a group of standards
There are four key pieces that are used together: Templates, Custom Elements, Shadow DOM, HTML Imports
Custom Elements in particular allow you to create your own HTML elements
<fancy-dropdown></fancy-dropdown>
---
> This will result in a new web component eco system where you'll be able to buy and sell components
> So this has been in development for the past several years.
> Web components are still relatively new in terms of any widespread deployments, but this new tech is posed to
change how we work with all major frameworks.
> This is definitely a technology to learn about and prepare for.
To infinity and beyond!
Hosting in the cloud
Hosting in the cloud has never been easier or cheaper
This is an important skill every modern web developer should have
Amazon, Heroku and DigitalOcean offer services to fit every budget
---
> It's never been easier to host your projects in the cloud
> This is a skill every modern web developer should have. If you don't already do this. Take the time to learn how.
> Amazon, Heroku and DigitalOcean offer services to fit every budget
> AWS is the most feature rich, but harder to use service
> Heroku is the easiest to use but most expensive for production use
> This is why it's not uncommon to see developers build their app on Heroku and later migrate to AWS.
Use a build tool
Use JS friendly build tools such as Grunt and Gulp
Both are build systems which include a massive collection of plugins for doing just about anything you need to
in the life time of your project.
They can help you do everything we've already discussed and things like pushing your code to cloud services
such as Amazon and Heroku and your assets to content delivery networks.
---
> So everything that we've discussed so far fits neatly under the umbrella of a build system such as Grunt or
Gulp.
> They each feature a massive library of plugins for things like automatically checking your JavaScript for
correctness, running tests scripts, packaging your projects and deploying them to the cloud and much much
more.
> If you're not using these tools now, they should be in your future - set aside some time to get to know them.
Preparing files for the web
Get to know Grunt or Gulp
Example: modern-web-seed project at https://github.com/cjus/modern-web-seed
The plugins I uses: gulp-jshint, gulp-angular-templatecache, gulp-sass, gulp-uglify, gulp-jscs,
gulp-jsbeautifier, gulp-usemin, gulp-minify-html, gulp-minify-css, gulp-ng-annotate, gulp-rev, gulp-imagemin,
imagemin-pngcrush
Yo! use Yeoman: http://yeoman.io
$ gulp dist
---
> We talked about Build tools such as Grunt and Gulp earlier.
> I use Grunt at work and Gulp personally. My modern-web-seed project uses the plugins listed. From the names, you
can probably guess what each one does.
> Checkout it out to see how the tools we've discussed can all work together under a Gulp automation to build an
application.
> The really cool thing is that most of those plugins run when I type gulp dist.
> This is serious automation and a major time saver during development
> When you're ready to do this on your own projects checkout Yeoman.
Continuous integration
A tool that periodically builds your product in the background and sends you messages if your build doesn't
succeed
Runs your tests scripts as well
Consider this an early warning system *before* your build makes it to QA and long before it makes it to
production
Here are just few services to consider: Jenkins, Travis, CodeShip and Drone
---
> Continuous integration (or CI as it's commonly called) is a process whereby your project is build and tested
periodically
> Modern web developers recognize the value of automation
> and if we can automate some of our testing then we're better off!
> As an early warning system, CIs are a major time saver because issues are identified as you develop and check in
your code
> This can help the guy who insist on checking in seemly simple code without testing it.
> Tip: Don't be that guy!
Formulating a plan of attack
Work incrementally. Don't try all this at once.
Get comfortable with a tool before moving onto the next.
Ask for help when you need it.
This was all new, at one time, to everyone who went down this path!
---
> So there are lots of tools that modern web devs use to step up their game.
> This might all seem overwhelming but having a plan of attack will help you get through it.
> Focus on a specific tool or process until you're comfortable with it. Then move onto the next.
> Ask for help! It's one of the reasons we host these meetups - so we can create a community around support.
> And most of all remember
> at one point each of these things was new to each of us.
Contact
cjus on Twitter and Github
Email<EMAIL_ADDRESS>About: http://cjus.me


--- creation_epoch:<PHONE_NUMBER> lastmod_epoch:<PHONE_NUMBER> categories: Misc title: How sudo sets the PATH value when secure_path is not defined --- Ran across an interesting technical detail while working on a PATH configuration problem in my /etc/profile. When no PATH is defined (such as when you do sudo -i) the sudo command will inject a standard one that is compiled into its code. This applies when you do not have a secure_path option either in your config or built-in as a compile option at build time.
This was important because when using bash and before your /etc/profile configures your PATH variable there will be an already existing value setup by the sudo -i command (or sudo -s env). This mostly affected me due to some invocations creating a PATH with two colons (::) in it which was a security issue for root since that implies the current working directory is in root's PATH.
Testing was done on a Red Hat 6. Sudo version 1.8.6p3 with 1.8.6p3-12.el6.
Before /etc/profile the sshd process sets PATH=/usr/local/bin:/bin:/usr/bin (or whatever value your OpenSSH was configured and compiled to use).
Before /etc/profile the sudo process sets PATH=/usr/bin:/bin:/usr/sbin:/sbin
sudo-1.8.6p3.tar/sudo-1.8.6p3/sudo-1.8.6p3/pathnames.h.in:#define _PATH_STDPATH "/usr/bin:/bin:/usr/sbin:/sbin" sudo-1.8.6p3.tar/sudo-1.8.6p3/sudo-1.8.6p3/plugins/sudoers/env.c: /* Provide default values for $TERM and $PATH if they are not set. */ if (!ISSET(didvar, DID_TERM)) sudo_putenv("TERM=unknown", false, false); if (!ISSET(didvar, DID_PATH)) sudo_setenv2("PATH", _PATH_STDPATH, false, true);


Athletics golden girls get SPOTY nominations on day of turmoil for their sport
A month after Neil Black quit as performance director of UK Athletics, the governing body has forced newly appointed chief executive Zara Hyde-Peters to stand down
by Alex Spink, https://www.facebook.com/alexspinkmirror/Dina Asher-Smith and Katarina Johnson-Thompson were tonight nominated for Sports Personality of the Year - as their sport suffered a fresh body blow.
Britain’s two world champions joined a six-strong shortlist, alongside Raheem Sterling, Lewis Hamilton, Ben Stokes and Wales rugby captain Alun Wyn Jones.
Yet athletics’ pride in being the only sport with multi-nominees was tempered by turmoil behind the scenes.
A month after Neil Black quit as performance director of UK Athletics, the governing body has forced newly appointed chief executive Zara Hyde-Peters to stand down.
The move follows claims she allowed her husband to continue coaching athletes despite being banned from teaching.
Chris Clark, chair of UKA, revealed that the matter had been “fully” discussed at a UKA board meeting on Sunday.
He added: “Following the meeting I had time with Zara to review the situation and we have agreed together, that she will now not be taking up her position as CEO on 1st December.”


fish trawl net individual shoal Fish capture familiar unfamiliar anaerobic swimming traits behaviour time swim escape metabolic capacity shoals Killen log using vulnerability {border-style px;} trials #ffffff;} net, individuals Sci fishing social susceptibility determining spent relationship role Brown fisheries gears mouth Ecol active aerobic ahead likely oxygen phenotypic tunnel calculated cohesion composition high metabolism net. used Behav Biol Heino However, behaviours exploited flow recovery relationships respirometry response routes trial. (EPOC) Evol Hollins exercise performance physiological provide px; selection shoals. water {font-family Anim MMR Physiol behavioural capture. font-size greater influence potential present responses shoals, tested values Cooke Enberg Mar Marras Pauli Rose, capture, change comprised degree different double; effects exhaustive experiment gear maintain number observed pursued pursuing recorded represents shoal, species stocks total trawls Aquat Arlinghaus Composition Diaz Fig. Individual Table Thambithurai Underwood Winger, alongside angling avoid bold border-color border-width chambers commercial conformity conspecifics effect evolution exhibiting familiarity fisheries-induced implications interactions learning linear maximum minnows negative period plastic repeatability shown solid; study synchronisation trawling use variation wild {font-style {font-weight *Shoal Atton Clark Conserv ICES Koeck Laland Laugen MMR, Proc Redpath Ryer University Warburton, Ward Wilson Yanase allow available avoidance clustered cohesive collective consumption courier; determined divider elicit em; encountered entering escapes events exploit final fish’s flush given groups guppy herding higher including increase indicate individual’s investigate laboratory mean mechanisms min minnow model models phenotype prior pt;} rate related remained removal shoaling similar simulated speed speeds study, tank tanks trawl, trials. turn work


CovMantra
Home
About
Self Help
Awareness Material
FAQ's
Blog
menu
home
Home
info
About
local_hospital
Self Help
record_voice_over
Awareness Material
help
FAQ's
rss_feed
Blog
Cov
Mantra
Your Guide to Survival
Get Started
Live Covid India Stats
flash_on
Active
healing
Recovered
group
Confirmed
all_inclusive
Deaths
Visualizating on World Level
Graphical Representaion in Log format


Web Designer
Web Developer
Mobile APP Designer
Mobile APP Developer
Databse Designer
CEO
Markiter
Next
Previous


ADDERS, AND THE MILPREVE
THE country people around the Land's End say that in old I times no one could live in the low grounds, which were then covered with thickets, and these swarming with adders. Even at a much later period, in the summer-time, it was not safe to venture amongst the furze on the Downs without a milpreve. (I have never seen a milpreve; but it is described to me as being about the size of a pigeon's egg, and I am told that it is made by 'the adders when they get together in great numbers. Is it not probable that the milpreve may be one of the madrepore corals-- millepore-- found sometimes on the beaches around Land's End ?)
A friend writes me :--" I was once shown a milpreve; it was nothing more than a beautiful ball of coralline lime-stone, the section of the coral being thought to be entangled young snakes."
When some old men were streaming the "Bottoms" up near Partimey, they were often obliged to leave work on account of the number of adders that would get together as if by agreement, and advance upon them.
One day one of the tin streamers chanced to leave his pot of milk, uncovered, out of the moor-house, when an adder got into it. The man cut a turf and put over the pot to prevent the reptile from escaping. In a few minutes the tinners saw "the ugly things crawling and leaping from all quarters towards the pot." The streamers were obliged to run, and take which way they would, the adders seemed to be coming from every direction, further and further off.
 Towards night all the reptiles were
quite still then the men gathered together, around the mass of adders, a
great quantity of furze (being summer, there was plenty cut and dry
close at hand), and piled it up like sheaves to make a mow, laying a
circle of well-dried turf without it. They then fired the turf on every
side, and when it was well ignited, they fired the furze. "Oh, it
was a sight to see the adders when they felt the smoke and the flame !
they began to boil, as it were, all in a heap, and fell back
into the flaming furze ; those which leaped through perishing on the
brilliant ring of burning peat. Thus were killed thousands upon thousands
of adders, and the moors were clear for a long, long period."
This is related nearly as the story was told ; but it appears necessary to
make some allowance for that spirit of exaggeration which is a
characteristic of all Celtic people, ere they have been tutored to
know the dignity of truth.
The country people retaine a conceite, that the snakes, by their
breathing upon a hazel-wand, doe make a stone ring of blew colour, in
which there appeareth the yellow figure of a snake, and that beasts
which are stung, being given to drink of the water wherein this stone hath
bene socked, will there-through recover." [a] This was clearly one
of the so-called "Druidic rings,"-examples of which may be seen in our
museums,-which have been found in England and in Ireland. It is
curious that at the glassworks of Murano, near Venice, they still
make rings, or beads, precisely resembling the ancient ones, and
these are used largely as money in Africa.
Snakes were formerly held in great reverence ; and Camden asserts that one
of the prevailing superstitions concerning them was that, about
midsummer-eve, they all met together in companies, and, joining their
heads, began a general hiss, which they continued until a kind of bubble
was formed, which immediately hardened, and gave to the finder
prosperity in all his undertakings. [b]
Lhuyd, in a letter written in 1701, gives a curious account of the then
superstitious character of the people in this district. " The Cornish
retain variety of charms, and have still towards the Land's End the
amulets of Maen Magal and Glain-neider, which latter they
call a Melprer, a thousand worms, and have a charm for the
snake to make it, when they have found one asleep, and struck a hazel-wand
in the centre of its spirae." Camden mentions
the use of snake-stones as a Cornish superstition.
"The very same story, in fact, is told of the Alder-stane in the popular legends of the Scottish Lowlands, as Pliny records of the origin of the Ovum Anguinum. The various names by which these relics are designated all point to their estimation as amulets or superstitious charms; and the fact of their occurrence, most frequently singly, in the sepulchral cist or urn, seems to prove that it was as such, and not merely as personal ornaments, that they were deposited with the ashes of the dead. They are variously known as adder-beads, serpent-stones, Druidical beads; and, amongst the Welsh and Irish, by the synonymous terms of Gleim na Droedh and Glaine nan Druidhe, signifying the magician's or Druid's glass."--Wilson's Archaeology and Prehistoric Annals of Scotland, p. 304.
[a] The survey of Cornwall. By Richard Carew.
[b] Draw and Hitchin's Cornwall.


This toolbox contains Matlab code for several graph and mesh partitioning
methods, including geometric, spectral, geometric spectral, and coordinate
bisection. It also has routines to generate recursive multiway partitions,
vertex separators, and nested dissection orderings; and it has some sample
meshes and mesh generators.
The toolbox contains a Matlab interface to Leland and Hendrickson's
Chaco partitioning package, but it doesn't contain Chaco itself.
The file "chaco/README" tells how to install the interface to Chaco.
It also contains a Matlab interface to Karypis et al.'s Metis
partitioning package, using Robert Bridson's "metismex" code.
Demonstrations.
Say "meshdemo" for some examples.
Also, many of the individual routines can draw pictures of what they do.
Partitioning methods.
geopart - Geometric.
specpart - Spectral.
gspart - Geometric spectral.
coordpart - Coordinate bisection.
inertpart - Inertial bisection.
chaco - Multilevel Kernighan-Lin, and other options.
metispart - Multilevel method from Metis.
metismex - Interface to more options of Metis.
Multiway partitions.
dice - Use any 2-way partitioner to get a multiway partition.
geodice - Recursive geometric partitioning.
specdice - Recursive spectral partitioning.
gsdice - Recursive geometric spectral partitioning.
chaco - Can also produce multiway partitions directly.
metisdice - Multiway partitioning from Metis.
Vertex separators.
vtxsep - Convert a 2-way partition to a vertex separator.
geosep - Vertex separator from geometric partitioning.
specsep - Vertex separator from spectral partitioning.
Nested dissection.
ndperm - Use any 2-way partitioner for nested dissection.
geond - Geometric nested dissection ordering.
specnd - Spectral nested dissection ordering.
gsnd - Geometric spectral nested dissection ordering.
metisnd - Nested dissection ordering from Metis.
analyze - Predict fill, opcount, etc. for an elimination ordering.
Meshes and graph generators.
meshes.mat - Three sample meshes with coordinates:
Eppstein - A 2D finite-element mesh with 547 nodes.
Smallmesh - A 2D finite-element mesh with 136 nodes.
Tapir - A 2D finite-element mesh with 1024 nodes.
grid5 - 2D square 5-point mesh.
grid7 - 2D square 7-point mesh.
grid9 - 2D square 9-point mesh.
gridt - 2D triangular mesh.
grid3d - 3D cubical mesh.
grid3dt - 3D cubical simplicial mesh.
badmesh - A mesh that has no good straight-line cut.
cockroach - A mesh for which spectral bisection does poorly.
treexpath - A mesh for which spectral bisection does poorly.
Visualization and graphics.
(All the partitioners can also draw pictures of what they do.)
gplotpart - Draw a 2-way partition.
gplotmap - Draw a multiway partition.
highlight - Draw a mesh with some vertices highlighted.
gplotg - Draw a 2D or 3D mesh (replaces Matlab's gplot).
etreeplotg - Draw an elimination tree (replaces Matlab's etreeplot).
spypart - Matrix spy plot with partition boundaries.
dmspy - Spy plot of matrix in block triangular form.
Utilities.
cutsize - Find or count edges cut by a partition.
other - Other side of a partition, or change representations.
intersection - Intersection of two sets.
union - Union of two sets.
fiedler - Fiedler vector of a graph.
laplacian - Laplacian matrix of a graph.
components - Connected components of a graph.
contract - Condense a graph according to a given block structure.
distances - Distances between adjacent mesh points.
blockdiags - Create matrix with specified block diagonals.
resetrandoms - Reset random number generators to startup values.
References
John R. Gilbert, Gary L. Miller, and Shang-Hua Teng.
Geometric mesh partitioning: Implementation and experiments.
SIAM J. Scientific Computing 19:2091-2110, 1998.
Tony F. Chan, John R. Gilbert, and Shang-Hua Teng.
Geometric spectral partitioning.
Xerox PARC Technical Report CSL-94-15, 1995.
Bruce Hendrickson and Robert Leland.
The Chaco user's guide, version 2.0.
Sandia National Laboratories Technical Report SAND94-2692, 1994.
George Karypis et al.
METIS, Serial graph partitioning, version 4.0.1, November 1998.
http://www.cs.umn.edu/~karypis/metis
Robert Bridson.
A MATLAB CMEX interface to the Metis library.
http://www.stanford.edu/~rbridson/download/metismex.c
Toolbox Authors
John R. Gilbert
Palo Alto Research Center
3333 Coyote Hill Road
Palo Alto, CA 94304
<EMAIL_ADDRESS>Shang-Hua Teng
Computer Science Department
Boston University
Boston, MA 02215
<EMAIL_ADDRESS>Thanks to Tim Davis for updating the toolbox to Matlab 5;
to George Karypis, Vipin Kumar, and Robert Bridson for Metis
and its interface; and to Bruce Hendrickson and Robert Leland
for Chaco.


07 Apr 2014
•
Crypto
•
Protocols
In 2001, Zooko Wilcox-O'Hearn conjectured that a key value system, in the way the keys address those values, must make a compromise between three properties. Those properties are:
- distributed: there is no central authority in the system (and the other nodes of the system are potentially untrusted)
- secure: a name lookup cannot return an incorrect value, even in the presence of an attacker
- human usable keys: keys that a human will be able to remember and write without errors
The conjecture stated that you may have at most two out of these three properties in the system.
Examples of those properties at work:
- the DNS is secure (one possible record for a given name) and human usable (domain names can be learned), but not distributed (the server chain is centralized)
- the PGP mapping between email and public keys is distributed (no central authority defining which email has which key) and human usable (public keys are indexed by emails) but not secure (depending on your view of the web of trust, identities could be falsified)
- Tor .onion addresses are secure (the address is the hash of the public key) and distributed (nobody can decide that an address redirects to the wrong server) but not human meaningful (have you seen those addresses?)
Some systems, like NameCoin, emerged lately and exhibited those three properties at the same time. So is the conjecture disproved? Not really.
When considering that triangle, people often make a mistake: assuming that you have to choose two of these properties and will never approach the third. These properties must not be seen as absolute.
Human usability is easily addressed with Petname systems, where a software assistant can help you add human-meaningful names to secure and distributed keys (at risk of collision, but not system wide).
You can add some distributed features in a centralized system like SSL certificate authorities by accepting multiple authorities.
NameCoin is distributed and human meaningful, and creates a secure view of the system through the blockchain.
And so on.
The three properties should be redefined with what we know ten years later.
Human usable names
We now understand that a human meaningful name is a complex concept. We see people typing "facebook" in a search engine because they would not remember "facebook.com". Phone numbers are difficult to remember. Passwords, even when chosen through easy to remember methods, will be repeated at some point. And we could also talk about typosquatting...
Here is a proposition: a human usable key is an identifier that a human can remember easily in large numbers and distinguish as easily from other identifiers. I say identifier because it could be a string, a picture, anything that stands out enough to be remember and for which differences could be easily spotted.
That definition is useful because it is quantifiable. We can measure how many 10 digits phone numbers a human can remember. We can measure how many differences a human can spot in two strings or pictures.
Decentralized and/or secure names
The original text indicates distributed as "there is no central authority which can control the namespace, which is the same as saying that the namespace spans trust boundaries". The trust boundary is defined as the space in which nodes could be vulnerable to each other (because they trust each other). In a fully distributed system, a node may not be able to force any other node to have a different view of the system.
Secure means that there is only one correct answer for a name lookup, whoever does the lookup. This is the same as saying that there is a consensus (at least in a distributed system).
We can see that the decentralized and secure properties are at odds with the consensus problem in a distributed system. This is where systems like NameCoin make the compromise. They exchange an absolute secure view of the system for an eventual consistency, and fully distributed trust boundaries for byzantine problems. They guarantee that, at some point, and unless there are too many traitors in the systems, there will be a universal and unique mapping for a human readable name.
So, we could replace the "secure" property by: the whole system recognizes that there is only one good answer for a record at some point in the future. Again, this is measurable, and it also addresses the oddity in DNS that records have to propagate (so sometimes the view can differ). Systems where identifiers do not give unique responses will not satisfy that definition.
The last property, "decentralized", can be formulated like this: a node's view of the names cannot be influenced by a group of unauthorized nodes. In a centralized system, where there is essentially one node, this definition cannot apply. In a hierarchy of nodes, like DNS, we can easily see that a node's view can be abused, since there is only one node you need to compromise to influence it, its first nameserver (this is used, for good and for bad reasons, in a lot of companies). In a distributed system, this definition becomes quantifiable and joins the byzantine problem: how many traitor nodes do you need to modify a node's view of the system?
Here, we have three new definitions that are more nuanced, allow some compromise and modulation of the needs, and for which the limits can be measured or calculated. These quantified limits can be useful to describe and compare the future naming system propositions.
Since WhatsApp announced its acquisition, a lot of people started to switch to alternatives, trying to escape from Facebook. Some of them then discovered my article about Telegram, and a common answer was "hey, at least, it is better than WhatsApp, because it is open source, faster and it has encryption".
This is a very bad way to decide what application you should use. If you choose a secure messaging app, it must be because you need it, not just because you want to avoid Facebook.
Those are not good enough requirements:
- independent from Facebook
- fast
- multi platforms
- open source
Yes, even open source, because it does not magically make software safe.
So, what are goods requirements? Well, I already have a list of what a secure messaging app should meet to be considered. If an app does not follow those requirements, it may not be a good idea to use it.
But it still does not mean the app will fit your use case. So you must define your use case:
- Why do you need it?
- With whom will you communicate?
- Who is the adversary?
- What will happen if some of your information is revealed to the adversary?
- Does it need to be always available?
- For how long will it be used?
This is part of what I mean when I insist on having a threat model: you cannot choose correctly if you do not know the risks.
Here are a few examples that you could consider.
The activist in a protest
The activist must be able to communicate quickly in the crowd. Identifying info might not be the most important part, because she can use burner phones (phones that will be abandoned after the protest). The most important feature is that it should be always available. Phone networks were often used to disrupt activist communication, so a way to send message through WiFi our bluetooth might be useful. The messages can be sent to a lot of different people, so being able to identify them might be important.
Being able to send photos is important, because they might be the only proof of what happened in the protest. Here, I have in mind the excellent ObscuraCam app, which is able to quickly hide the faces of people in photos before sending them.
The application should not keep logs, or provide a way to quickly delete them, or encrypt them by default, because once someone is caught, the police will look through the phone.
The crypto algorithms and protocols should be safe and proven for that use case, because the adversaries will have the resources to exploit any flaw.
No need for a good update system if the devices will be destroyed after use.
The employee of a company with confidential projects
The adversaries here are other companies, or even other countries. The most important practice here is the "need to know": reduce the number of persons knowing the confidential information. that means the persons communicating between themselves is reduced, and you can expect that they have a mean of exchanging information securely (example: to verify a public key).
Identifying who talks with whom is not really dangerous, because it is easy to track the different groups in a company. You may be confident enough that the reduced group will not be infiltrated by the adversary. The messages should be stored, and ideally be searchable. File exchange should be present.
There could be some kind of escrow system, to reveal information if you have a certain access level. Authentication is a crucial point.
The crypto may be funnier for that case, because the flexibility needed can be provided by some systems, like identity based encryption.Enterprise policies might be able to force regular uodates of the system, so that everybody has the same protocol version at the ame time, and any eventual flaw will be patched quickly.
The common user
It is you, me, anyone wanting to exchange private messages with friends or family. Here, trying to protect against the NSA is futile, because most of the contacts might not have the training needed. Trying to hide the contacts list from Facebook is futile too: even if someone protects the information, one of the contacts may not. The adversary you should consider here: crooks, pirates, anyone that could exploit the private messages for criminal ways (stealing bank info, blakcmailing, sending malware, etc).
An application fitting this use case should encrypt messages, preferably end to end, to limit problems when the exchange server is compromised. The service might not provide any expectation of anonymity. Messages should be stored, but encrypting them is a good option, in case the device is lost or stolen.
The crypto does not need to be very advanced, but it should use common, well known designs.
There should be a good update system, a way to negotiate protocol versions (and forbid some unsafe versions), because you will never be sure that everybody has performed all the needed updates.
Your use case here
Those were some common situations, for which some solutions exist, but there are a lot more possible use cases. If you are not sure about yours and need help defining your threat model, do not hesitate to ask for help, and do not jump on a solution because the marketing material says it is safe.
A good security solution will not only tell you what is protected, and how, but also what is not protected, and the security margins you have. It will also teach you the discipline you need to apply to get the most out of it.
20 Feb 2014
•
General
Everytime people discuss the hacker community and its diversity, I see someone waving the "meritocracy" argument. "It is not our fault those minorities are not well represented, if they knew more stuff or did more stuff, they would have a better status".
It is easy to see how that argument would be flawed, as meritocracy is a power structure, and whenever a power structure is created, after some time it tends to reinforce its own community. But that is not my point right now.
I realized that the idea of meritocracy is so deeply ingrained in the hacker mindset that we lost sight of what was important. I can see how that idea is appealing.  Except it is not. The meritocracy is just another way to exclude people. Once you consider someone's status by how much you perceive they know, things go downhill.
Some are good at faking knowledge. Some know their craft, but do not talk that well. Some are not experts, but have good ideas. Some would like to learn without being judged.
Somewhere along the way, people got too hung up on meritocracy, and forgot that you hack for knowledge and for fun, not for status. It is all about testing stuff, learning, sharing what you learned, discussing ideas and helping others do the same, whatever their skills or their experience. Status and power structures should have nothing to do with that.
Guess what? I pointed out that bad behaviour, but I am guilty of it too. I have to constantly keep myself in check, to avoid judging people instead of judging ideas. That's alright. Doing the right thing always requires some effort.
Following the previous article, people have asked me what I would consider as a good secure system, and others asked me to review their app, so I think it will be interesting to expose my process when studying those projects.
Threat modeling
The most important point I look for in a project is the threat model. This is the document that will explain for whom the project was created, who are the adversaries, what they are trying to obtain, and which of these threats you are addressing.
Without that document, I cannot know if you considered all the possible actors, and I must infer it from the protocol, which is relatively easy, but my view of the threat model might not correspond to what you expected.
With a good threat model, I can know right away what is your target market (ex: sexting for teens, or secure reporting for journalists in war environments), see if your users will understand the implications, if it will need training, and more importantly, if your system can be safe for that context.
You cannot create a project and say that it will solve all of the privacy problems with some magical crypto algorithm, against all adversaries, even the state actors. I would prefer a useful tool for a niche with real and well defined needs.
Prior art
As you have probably seen, the secure messaging space is already very crowded. If you come up with a new solution to an already solved problem, you need to justify it. Why didn't you improve an existing project? Couldn't you adapt someone else's code, add a better UI?
the NIH syndrome is at the heart of innovation, so I am not against it. But in the case of crypto applications, it might be a good idea to employ already existing (and already audited) code, instead of writing a whole new protocol or algorithm from scratch.
Otherwise, if you are working on an unsolved problem, or improving on current solutions, be prepared to justify it, and a lot, if you employ unusual systems. I am not telling you to avoid funny stuff like Pailler's cryptosystem, PIR or pairing based cryptography. Just be aware that people will ask you about these.
Publications
That part is fundamental: if you are providing a new protocol or algorithm, you should publish it and ask for review before you start coding and get users. I am not advising you to start up LaTeX and write a paper in ACM format. Just explaining your system on a webpage is fine. The crypto community is full of nice people that will be able to point out if there is any problem (and if you use the academic way of publishing, you might even profit from other people's funding to get reviews :p).
Some said that the crypto community is full of bitter people eager to hit any new project, following the whole Telegram debacle. That tends to happen when you make a big announcement to get users, telling that it will solve any security problem, and dismiss the opinions of experts, without having asked for review previously.
Note that some of those experts have worked for years on a project before even thinking of communicating about it. As examples, check out Briar, Pond or Cryptosphere: those are quiet but interesting projects. They are not trying to get a lot of users quickly or profit from the post Snowden panic. They have been at it for a long time.
So, publish, ask for review, fix flaws, publish again, fix stuff, and repeat again and again. That is the smartest way to spend your time and money on your project. Once everything is developed and deployed, you will have a hard time trying to plug the holes.
Protocol design
Once we get in the technical stuff, the protocol design is interesting to get a high level view of what you want to achieve. I'll ask questions like:
- Is it server centric or P2P? (note: a network of server introduces routing, but is not P2P)
- Does it include authentication?
- Is it encrypted end to end?
- How do you protect against DoS?
- Is it versioned? Do you allow for protocol version negotiation? Are the algorithms negotiated?
- Can you revoke keys or identities?
Often, the protocol show what you want to achieve with your system, and it is often answering more threats than the crypto algorithms themselves. A good way to present your protocols is to use diagrams and present the message contents.
Do not insist on algorithms at this point: use general words to describe the primitive you need, like authenticated cipher, public key, key derivation function, MAC. You might change the algorithms later, so stating the properties you need will help reviewers understand what you want to achieve.
A specific note on server VS peer to peer: it is a very understandable feeling for geeks that P2P architectures look better, because they're decentralizing everything, etc. But they can introduce other problems (like hole punching or sybil attacks), and in some case, you will not be able to avoid servers (for message routing and retries, for mobile systems, etc). Both types of systems are fine, just be aware of their shortcomings.
Cryptographic constructs
Cryptographic algorithms are not enough, you need to apply them correctly. I will have no pity if you say you use "military grade AES 256 encryption" but do not know what is a block cipher mode or Encrypt-Then-MAC. A lot of ugly details can hide here, so do not try to be clever, use battle tested systems:
- add a separate authentication layer to Diffie-Hellman key exchanges
- use an authenticated encryption mode
- use RSA-OAEP instead of PKCS1 padding
- know well if you need a nonce, an unpredictable number or a time based ID
- etc.
This is one of the parts where crypto experts will ask annoying questions, because a lot of bugs come from there. They can also propose better solutions (safer, more performant, etc), so listen to them.
If you are employing an unusual scheme here, be prepared to justify it. It might be ok for you, but if the design looks weird to cryptographers, that will raise alarms. Your scheme could be safe, but if it has never been proven right, you are taking a risk, and your users will take that risk too. Is it worth it? Hint: your weird design should provide a unique property that no other algorithm has.
Choice of algorithms
Yes, I do not worry about algorithms until I am already deep in the system. It is not that hard to make correct choices there. Just listen to the recent attacks (ie, avoid RC4) choose large enough keys, choose correct elliptic curves.
Every algorithm has parameters that you need to get right, so be sure to document yourself on your algorithm choices:
- AES-CBC needs an initialization vector, but AES-CTR uses an incremented nonce
- RSA needs a good exponent
- Some elliptic curves work better for some operations
Even if you choose dubious algorithms, if your protocol was well designed, you will be able to move to better algorithm. Be careful with algorithm negotiation, though, a lot of smart people were bitten before.
The implementation
This is probably the part that I will skip, because I do not have the time nor the funding to audit thoroughly the code of every new projects. I will often grep a bit through the code, look for some important points, but this is not something that should be done quickly. This is where the protocol review shows its limits.
Even with a good design, a lot of vulnerabilities can be present in a flawed implementation. Crypto projects should undergo a careful audit like the one Least Authority performed recently on Cryptocat. And that is why you should not communicate about your project before it has been reviewed.
There are things you should always look for in your software projects:
- encrypting data at rest: if you worry about stolen data, know that a mobile phone or laptop can be stolen
- random number generation: you should use a CSPRNG, with a good source, and probably some user or device specific data
- data backup: is it possible? is it safe?
- software updates: are they downloaded from a secure source? Are the updates verified?
- Do you use public key pinning?
- How long are they private keys stored as plaintext in memory?
The implementation details are as important as the whole protocol. You can have a good protocol, but a small error in the code could greatly affect your users. Nevertheless, specifying your protocol is useful, because people can provide better implementations, or make it interoperate with other software. Having other implementations is a good thing: you will not control those versions, but they will be able to construct cool stuff around your system, and make a part of your PR.
User interface
this part is more and more important, because we have been able to create safe systems for years, but often at the price of usability. The user experience of crypto apps needs a lot of innovation, and I'll follow closely any interesting idea in that space: onboarding experience, useful alerts, user decision making, etc. People should be able to understand when there is a security problem.
I'll state it once more: if you create a new crypto software, you HAVE to make it easy to use and understand. Some complexity is acceptable, but it must be compensated by documentation (with screenshots, etc) or training.
Other criterions
There are two others that I could think of, but they do not matter that much.
The first is the team. I have been accused of making fun of Telegram for waving around their team of PhDs, but the truth is that I was hopeful: a team full of smart people can come up with interesting design and solve complex problems. If they do not deliver on that, I could be less indulgent. That does not mean I will think less of people without big diplomas. I know too many smart people that dropped out of school to make that mistake. Ultimately, the important thing to judge is the design.
The last parameter is attitude. It is normal to be defensive when someone else reviews your work, but that does not justify denial and dishonesty. People are often taking time off of their job to study your system, so they will be quick and get to the point. If you do not answer or refuse to explain your decisions, it will smell fishy. Even more if you did not ask for a review before communicating about your project. But it does not matter that much. If you are humble and quick to answer, people may help you out of good will, but if you anger cryptographers, you may just have won a free thorough audit :D
Disclaimer: this post is now very old and may not reflect the current state of Telegram's protocol. There has been other research in the meantime, and this post should not be used for your choice of secure messaging app. That said, on a personal note, I still think Telegram's cryptosystem is weird, and its justifications are fallacious. If you want a recommendation on secure messaging apps: use a system based on the Axolotl/Signal protocol. It is well designed and has been audited. Signal and WhatsApp are both using that protocol, and there are others.
Here is the second entry in our serie about weird encryption apps, about Telegram, which got some press recently.
According to their website, Telegram is "cloud based and heavily encrypted". How secure is it?
Very secure. We are based on a new protocol, MTProto, built by our own specialists, employing time-tested security algorithms.  We took care of the rest.
(from their FAQ)
Yup. Very secure, they said it.
So, let's take a look around.
Available technical information
Their website details the protocol. They could have added some diagrams, instead of text-only, but that's still readable. There is also an open source Java implementation of their protocol. That's a good point.
About the team (yes, I know, I said I would not do ad hominem attacks, but they insist on that point):
The team behind Telegram, led by Nikolai Durov, consists of six ACM champions, half of them Ph.Ds in math. It took them about two years to roll out the current version of MTProto. Names and degrees may indeed not mean as much in some fields as they do in others, but this protocol is the result of thougtful and prolonged work of professionals
(Seen on Hacker News)
They are not cryptographers, but they have some background in maths. Great!
So, what is the system's architecture? Basically, a few servers everywhere in the world, routing messages between clients. Authentication is only done between the client and the server, not between clients communicating with each other. Encryption happens between the client and the server, but not using TLS (some home made protocol instead). Encryption can happen end to end between clients, but there is no authentication, so the server can perform a MITM attack.
Basically, their threat model is a simple "trust the server". What goes around the network may be safely encrypted, although we don't know anything about their server to server communication, nor about their data storage system. But whatever goes through the server is available in clear. By today's standards, that's boring, unsafe and careless. For equivalent systems, see Lavabit or iMessage. They will not protect your messages against law enforcement eavesdropping or server compromise. Worse: you cannot detect MITM between you and your peers.
I could stop there, but that would not be fun. The juicy bits are in the crypto design. The ideas are not wrong per se, but the algorithm choices are weird and unsafe, and they take the most complicated route for everything.
Network protocol
The protocol has two phases: the key exchange and the communication.
The key exchange registers a device to the server. They wrote a custom protocol for that, because TLS was too slow and complicated. That's true, TLS needs two roundtrips between the client and the server to exchange a key. It also needs x509 certificates, and a combination of a public key algorithm like RSA or DSA, and eventually a key exchange algorithm like Diffie-Hellman.
Telegram greatly simplified the exchange by requiring three roundtrips, using RSA, AES-IGE (some weird mode that nobody uses), and Diffie-Hellman, along with a proof of work (the client has to factor a number, probably a DoS protection). Also, they employ some home made function to generate the AES key and IV from nonces generated by the server and the client (server_nonce appears in plaintext during the communication):
- key = SHA1(new_nonce + server_nonce) + substr (SHA1(server_nonce + new_nonce), 0, 12);
- IV = substr (SHA1(server_nonce + new_nonce), 12, 8) + SHA1(new_nonce + new_nonce) + substr (new_nonce, 0, 4);
Note that AES-IGE is not an authenticated encryption mode. So they verify the integrity. By using plain SHA1 (nope, not a real MAC) on the plaintext. And encrypting the hash along with the plaintext (yup, pseudoMAC-Then-Encrypt).
The final DH exchange creates the authorization key that will be stored (probably in plaintext) on the client and the server.
I really don't understand why they needed such a complicated protocol. They could have made something like: the client generates a key pair, encrypts the public key with the server's public key, sends it to the server with a nonce, and the server sends back the nonce encrypted with the client's public key. Simple and easy. And this would have provided public keys for the clients, for end-to-end authentication.
About the communication phase: they use some combination of server salt, message id and message sequence number to prevent replay attacks. Interestingly, they have a message key, made of the 128 lower order bits of the SHA1 of the message. That message key transits in plaintext, so if you know the message headers, there is probably some nice info leak there.
The AES key (still in IGE mode) used for message encryption is generated like this:
The algorithm for computing aes_key and aes_iv from auth_key and msg_key is as follows:
- sha1_a = SHA1 (msg_key + substr (auth_key, x, 32));
- sha1_b = SHA1 (substr (auth_key, 32+x, 16) + msg_key + substr (auth_key, 48+x, 16));
- sha1_с = SHA1 (substr (auth_key, 64+x, 32) + msg_key);
- sha1_d = SHA1 (msg_key + substr (auth_key, 96+x, 32));
- aes_key = substr (sha1_a, 0, 8) + substr (sha1_b, 8, 12) + substr (sha1_c, 4, 12);
- aes_iv = substr (sha1_a, 8, 12) + substr (sha1_b, 0, 8) + substr (sha1_c, 16, 4) + substr (sha1_d, 0, 8);
where x = 0 for messages from client to server and x = 8 for those from server to client.
Since the auth_key is permanent, and the message key only depends on the server salt (living 24h), the session (probably permanent, can be forgotten by the server) and the beginning of the message, the message key may be the same for a potentially large number of messages. Yes, a lot of messages will probably share the same AES key and IV.
Edit: Following Telegram's comment, the AES key and IV will be different for every message. Still, they depend on the content of the message, and that is a very bad design. Keys and initialization vectors should always be generated from a CSPRNG, independent from the encrypted content.
Edit 2: the new protocol diagram makes it clear that the key is generated by a weak KDF from the auth key and some data transmitted as plaintext. There should be some nice statistical analysis to do there.
Edit 3: Well, if you send the same message twice (in a day, since the server salt lives 24h), the key and IV will be the same, and the ciphertext will be the same too. This is a real flaw, that is usually fixed by changing IVs regularly (even broken protocols like WEP do it) and changing keys regularly (cf Forward Secrecy in TLS or OTR). The unencrypted message contains a (time-dependent) message ID and sequence number that are incremented, and the client won't accept replayed messages, or too old message IDs.
Edit 4: Someone found a flaw in the end to end secret chat. The key generated from the Diffie-Hellman exchange was combined with a server-provided nonce: key = (pow(g_a, b) mod dh_prime) xor nonce
. With that, the server can perform a MITM on the connection and generate the same key for both peers by manipulating the nonce, thus defeating the key verification. Telegram has updated their protocol description and will fix the flaw. (That nonce was introduced to fix RNG issues on mobile devices).
Seriously, I have never seen anyone use the MAC to generate the encryption key. Even if I wanted to put a backdoor in a protocol, I would not make it so evident...
To sum it up: avoid at all costs. There are no new ideas, and they add their flawed homegrown mix of RSA, AES-IGE, plain SHA1 integrity verification, MAC-Then-Encrypt, and a custom KDF. Instead of Telegram, you should use well known and audited protocols, like OTR (usable in IRC, Jabber) or the Axolotl key ratcheting of TextSecure.