Doctests, the shy giant of testing modules
Do you use python, even to wash your clothes? Do you find unit testing boring, but still have to do it, because you find value in automated testing? Then this article is for you.
The idea
I believe you have used the python console, from time to time. Lets assume you are writing a few inline functions like below, to experiment with stuff:
$ python
Python 3.6.4 |Anaconda custom (64-bit)| (default, Jan 16 2018, 18:10:19)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def addme(a):
... return a + a
...
>>> addme(2)
4
>>> addme(1.9)
3.8
>>> addme(0)
0
>>>
So, you have written an inline function, and you have made a couple test runs on it, to do some basic verifications. What if python could read the above output and do the reasoning for you, at runtime? That’s the idea behind doctests, my friend.
Seriously?
Totally, python has introduced a lot of intuitive and…heartbreaking features from time to time, which are now mainstream to a few main languages. Why not this one too?
Benefits of doctests
The benefits are quite a few and not limited to:
- They are ridiculously easy to write. I mean, you can even outsource it to your younger cousin, who studies about the anatomy of soil atoms, because he owes you, for fixing his computer.
- Unless you find copy paste difficult, they are more joyful to write. This time you have to familiar with a terminal environment, though.
- There is no need to open another file(even though you can put all doctests of your app to a single file) to read the test code for any function, as they lie right under the signature of each function.
- They are executable with the doctest module and readable without knowing a bit of python.
What about unit tests?
Unit tests, still offer great value and advanced capabilities of testing. While doctests are great on validating simple and pure functions, they are not very good, when you have to do complex validation. For example, if a sequence of commands was called during the test.
What about mocking?
Doctests can handle mocking gracefully, with a great library called Minimock. I encourage you, to give it a try and let me know your thoughts.
Even though I like the initiative, I prefer my tests to have separate roles. I don’t want my doctests to be heavily loaded and one-size-fits-them-all. But that’s really a matter of taste, there is nothing wrong if you disagree and I am more than happy to hear your rationale , if so.
A working example
Talk is cheap, so let’s write some code. Below is a working example, using the function of the prompt above.
def addme(a):
"""
This is a docstring, usually to explain the use of a function. Please do not confuse it with doctests. They both mean to provide forms of documentation, but doctests are executable too.
>>> addme(4)
8
>>> addme('a')
'aa'
>>> addme(set())
Traceback (most recent call last):
...
TypeError: unsupported operand type(s) for +: 'set' and 'set'
"""
return a + a
if __name__ == '__main__':
import doctest
doctest.testmod()
print(addme(1))
There are a couple of points, worth noticing here:
- The doctests, need to live inside a docstring. In there, just ‘copy-paste’ the output of your python console quick tests.
- If you need to simulate an exception case, you only need to add the first and the last lines of the exception message. Did you expect to hardcode paths or do any weird logic to get the file’s full path and print it in full? C’mon, python wont do that :)
- You need the doctest library to run the tests. Nose doesn’t need it though.
- You can run the above, as usual, with
python mydoctests.py
I need my tests to run as part of CI/CD/CT cycle. What’s in for me?
I am not here to disappoint you, am I ? :)
The nose test runner, supports running all your doctests additionally to your unit tests. Just add the flag --with-doctest
and you are good to go.
Nice stuff, where to go from here?
- Read the python documentation for more twitches and cases.
- Read this fantastic book and especially chapter 4, which covers doctests.
- Check implementations of other languages. For example, the one for NodeJS
- Doctests and nose
Conclusion
Thank you for reading this article, I hope you enjoyed it and that the article triggered your interest in writing more and less painful tests. Please spread the love, with the buttons below, if so. Feel free to add your own thoughts and experiences with doctests too.


DBS Apple Watch App
UX Mockup
WhereNUS
Mobile App Concept
NUSSU PRU Event Assets
Design Work
H1N1 Infographic
Infographic
The Book of (non) Choices
Interactive Digital Book
Paintball Master Challenge
Game
Make-A-Wish Foundation Wish Portal
Website Mockup
Make-A-Wish Foundation Folder
Brochure Mockup


{% extends "soc/mail/base.html" %} {% comment %} Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. {% endcomment %} {% block greeting %} Dear {{ to_name }}, {% endblock %} {% block content %}
Thank you for applying to {{ program_name }}. The proposal(s) you submitted were not selected for the program this year. We annually receive many more proposals than we are able to accept, and we would like to encourage you to apply again next year.
{% endblock %} {% block signature %} With best regards,


All methods demonstrated on this web page are based on the subsection Measuring Firing Rates of section 1.2 Spike Trains and Firing Rates of Dayan, P., Abbott, L. F., & Abbott, L. (2001). Theoretical neuroscience: Computational and mathematical modeling of neural systems.
Neurons communicate with each other by firing so-called action potentials. A single action potential usually has a duration of roughly one millisecond. If we ignore its duration, amplitude and shape, we can treat an action potential as a stereotypical on-off event, where a single action potential firing is called a spike. A sequence of spikes recorded over a single trial is called a spike train. If we record multiple trials we can average the number of spikes fired at a specific time point over these trials. The resulting value is the firing rate of the neuron at a specific time point. Since we can only record a limited number of trials the real firing rate can only be approximated but not determined exactly. In the following we will look at different methods to approximate firing rates. For demonstration purposes only a single spike train will be used. By averaging over multiple trials before applying any of the discussed methods, more accurate results can be achieved.
The simplest approach to measuring firing rates is to count the number of spikes occurring within a given time frame.
If we count over the whole trial we get the total number of spikes but loose all temporal resolution.
To know how the spike count changes over time we can subdivide the trial into smaller time bins.
The smaller the time bins the better the temporal resolution. But this comes at the expense of a decreasing rate resolution. For very small time bins the rate can only be either 1 or 0.
Counting spikes in pre-assigned time bins lets the firing rate approximation not only depend on the width of the time bins but also on their placement. By sliding a single time bin over the whole trial duration we can avoid the arbitrariness in placement.
Instead of using a rectangular window function that weighs all spikes in the same way, other window functions can be used. A gaussian window function will weigh spikes that are further away from the time point less than near ones. This can provide a smooth window function. Instead of summing the values up in discrete time steps, the integral can be computed. Such firing rate functions are also called linear filters and their window functions are called kernels.
Both the rectangular window function and the Gaussian window function take not only spikes that happened in the past into account but also those happening in the future, with respect to a given timepoint on the spike train. To only use spikes that happened in the past the window function should become zero for negative values. A commonly used causal kernel is the alpha function.


Star Wars Episode 1: The Phantom Menace Why I love it, despite Jar Jar, and why it will be remembered
I have a pretty vivid memory of being on the playground when I was in third grade, either just before or just after Return of the Jedi came out. Some of the older kids already knew the names of the characters from the movie, and they were running around playing Return of the Jedi. Not with action figures, but with themselves. "I'm Han Solo!" They fought over who got to be Salacious Crum, Jabba's gremlin-like pet who I had yet to set my eyes upon. And as we ran around the soccer field and through the geometrically arranged tires, we talked about Star Wars movies and how there were supposed to be three others that took place before Star Wars. "They're called 'prequels.'" I think we knew about them even before then, though. I remember a neighborhood kid who prided himself on knowing these types of things when they pertained to Star Wars. (He also liked to pee in ziploc bags and carry them around with him, but that's irrelevant.) He had read many Star Wars novels, which he thought were the movies yet to be made. He was fond of a book called "Han Solo's Revenge" which he didn't seem to understand the title of.  He thought "revenge" was the name of a machine, but he still knew what a prequel was. In recent years the anticipation grew as Lucas made it clear he was finally making these things. I was in college when the Star Wars Special Editions came out, and I must admit it was one of the happiest times of my life. As a freshman living in a dorm for the first time, I had bonded with my roommates by staying up all night talking about which Star Wars toys we had owned as children. It seemed to be the universal connection between people our age, and we watched the movies together in great excitement when they were re-released on video in new THX editions. A few years later we were rescheduling our classes and meetings so we could wait in line for the first shows of the Special Editions. Now we had an excuse to write and think and talk about Star Wars all day long. Being college students our favorite place to eat was Taco Bell, which had now become Star Wars hype central. We could almost literally eat Star Wars in addition to the sleeping and breathing and what not. It is also important to note at this point that a) we traveled the city by foot and by bus and b) Toys R Us was located directly across the street from Taco Bell, i.e. within walking distance, and was stocked with many fine Star Wars themed leisure products and accoutrements. In the school's computer center I read an Ain't It Cool News prequel rumor about C-3PO being a computer generated mass of wires without his gold sheathing. I thought it was a cool idea, if it was true, but it was time to stay away from these rumors. I didn't want to know anything. But I could hold myself off by re-living the old movies. This time around there were levels to the movies I had never noticed before. Aspects I had never appreciated. They were truly growing better with time, even while beginning to seem dated. I have never been one of the hardcore Star Wars obsessives who knows the names of every character who ever had an action figure, but the entire Star Wars phenomenon has a strong connection to my heart, so the time of the Special Editions was like a religious experience for me. And I just kept thinking, "Manů if it's like this now, imagine what it will be like when the prequels come out."
I bring up these personal details because being one of the most anticipated movies of all time means bringing along a whole lot of baggage. Whether or not you were part of that generation who grew up talking about Star Wars prequels, you've certainly heard more than enough about them by now. Some people will never be able to accept a new Star Wars movie after 16 years of waiting. Others already wanted to hate the movie to prove they're above the hype, or the geeky guys with the plastic light sabers, or whatever. No one, no matter how eloquent, will be able to change your mind on The Phantom Menace. It's a movie that, even more than others, cannot be reduced to a consumer report. But most critics (many of them in a speeder bike chase, kessel run or pod race to be the first to hate the new Star Wars movie) have urged people to stay away, making The Phantom Menace out to be an abomination against cinema. And this after several summers in a row of absolutely shitty summer blockbusters from Batman Forever to Armageddon - movies lacking in imagination, common sense and even style. Although I'm sure these critics genuinely didn't like the movie, many of the reviews seem more like cultural grandstanding than legitimate criticism. In probably the most venomous review I've read, Anthony Lane of the New Yorker admits that he can't call Episode 1 the disappointment of the decade because he "had a sneaking suspicion it would turn out this way" - implying that he wanted to enjoy it but it just wasn't good enough. This, though, only after saying that Star Wars never was very good anyway, that its fans were "nerds" in the '70s and are now "dweebs," and after blaming the movie for the de-humanization of cinema, the military over-spending of the Reagan years, and (I swear to God) the concept of "deathless wars" in Iraq, Kosovo, etc. But many who are less extreme than Lane also went in wanting to hate the movie just because they're "sick of the hype." Unlike blockbuster films of recent summers, The Phantom Menace did not require manufactured enthusiasm. The demand for the film was so great that some grew tired of hearing about it before most of the actual PR hype kicked in. Technically, it's an independent film, an uncompromising vision without studio tampering or the soulless anti-art gauntlet of test screenings and focus groups. Still, it's a moneymaking machine, and every business in the world wants a piece. I honestly believe that the excitement would have been nearly the same with no advertisements, so when the Taco Bell-KFC-Pizza Hut-Pepsi onslaught began, it was more than overkill. Meanwhile, every step of the way became a news event. Both trailers were run uncut not only on Entertainment Tonight and MTV, but on local news shows. Reporters covered the reactions to the trailer, the web sites, the release of the toys, the predictions of economic loss due to employees skipping work on May 19th. On one Seattle station I saw a feature on what the Seattle Mariners think about Star Wars. But then, when critics saw the movie and tore it apart, the story changed. "Will it live up to the hype?" the news anchors asked innocently, apparently not remembering who had interviewed a woman who painted her car like an X-wing, or set up live cameras at the ticket lines, or started out half of their reports saying things like, "Well, a lot of people are waiting for the new Star Wars movie, but we're also waiting for a bit of sunshine this weekend. Harry?"
So with so many significant pop cultural complications, it's hard to predict how most people will feel about The Phantom Menace. But I don't think it really matters -- I think it's a great film. It is probably more flawed than the original trilogy due mainly to some inept comic relief. But what is good about it is skin-crawlingly, adrenaline-pumpingly good. The story adds new levels to the Star Wars mythology, the action scenes are thrilling, the costumes are some of the best ever put on film, and the visual attention to detail is unprecedented. I loved it on the first viewing and adored it the second. And it's great after all these years to be awaiting the second installment, not knowing what will happen. The film starts out brilliantly by illustrating exactly why Jedi Knights are so revered. For all the talk about Jedi Knights in the original trilogy, Luke's feats are a tiny plop in a puddle compared to those of young Obi-Wan Kenobi and his mentor Qui-Gon Jinn (played by Liam Neeson, who I have finally stopped thinking of as Darkman). The two are sent as ambassadors to the Chancellor, to conduct negotiations and settle a tax dispute. They hear the sound of their transport ship being destroyed, and without hesitation whip out their light sabers. These Jedi oughta be called ambadassadors. They run through the ship, slaughtering battle droids effortlessly and make a clean escape. At one point Lucas plays with the icon-status of the light sabers, having the villains terrified by the sight of Jedi through a cloud of smoke, just a pair of glowing light sabers. The two travel to the planet under siege, Naboo, to contact their Queen Amidala. The queen (and her clutch of clone handmaidens) is played by Natalie Portman, who has already achieved near-legend status as one of those rare child stars who is respected by adults. I don't think it's too early to say that Amidala is one of my favorite Star Wars characters. She is quiet and dignified, but speaks boldly (and in a strange accent) when needed. Unlike the politicians in our galaxy she tries to avoid putting her people at war, and when it seems necessary she goes out and leads it herself, planning a three point attack and storming her occupied palace with a pair of blasters. Her exquisitely designed royal gowns (of which there are many) rival even the otherworldly beauty of the blue skinned diva in The Fifth Element. She has an intimidating, regal presence, comparable to Judi Dench's Oscar winning queen in Shakespeare in Love.  Amidala is a more serious and, I think, complicated character than Princess Leia. My description makes her sound like Wonder Woman, but we also watch her essentially doom the galaxy for years to come by putting her faith in Palpatine. And it's interesting to see her surprise as she discovers that despite all of her politics, some planets still practice slavery because "The Republic doesn't exist out here." But I am confident that like Leia (her future daughter), she will be adored and emulated by young girls who like to see pretty heroines (and their clones) kick ass.
Soon, of course, the heroes meet up with 9 year old Anakin Skywalker, the boy who will become Darth Vader. Many have criticized the film for not showing a seed of evil in the boy, foreshadowing his dark future. I disagree - I think it's infinitely better to get the audience rooting for the kid, not only so they can temporarily set aside the tragedy that awaits him, but also because Darth Vader isn't a purely evil character anyway. After all, he is ultimately redeemed in Return of the Jedi. So it is more appropriate, I think, to treat him as a kid who says "Yippee!" a lot than as The Bad Seed. Other aspects of Anakin do tie in to his future. He is a master pod racer and is able to fly a Naboo fighter - this matches up with the original Star Wars, where Luke is told that his father was a great pilot before he became a Jedi. He is also a machine-building prodigy, having built his own pod racer and even the protocol droid C-3PO.  There is even a hint in John Williams' Anakin theme, which mourningly intones a few notes of the unforgettable Imperial Death March. Anakin's best scene is probably the one where he barely speaks - the pod race. A pod race is a dangerous flying drag race through the deserts of Tatooine, and Anakin is the only human who has ever done it. The scene is as riveting and full throttle as the speeder bike chase in Return of the Jedi. It's kind of disturbing to see this little boy zipping through deadly terrain in a vehicle that's sputtering and losing control and rapidly falling to pieces - all so the Tatooine slavemasters can gamble with the Hutts. Lucas adds extra seediness to the proceedings by revealing that Anakin's owner Watto forced him to enter in previous races even though he doesn't think he's very good. I don't think there's even a single child spotted in the previous Star Wars chapters (unless you count baby Ewoks) so it's surprising how well Anakin fits the tone. There's an awkward scene where some of the neighborhood kids (including a kid from Greedo's race) express skepticism about the boy's pod racer, but for the most part the kids work. There have been criticisms of Jake Lloyd's acting as Anakin. His performance, admittedly, is not perfect, and Lucas doesn't help with the awkward editing into the should-be heartbreaking scene where Anakin bids farewell to his enslaved mother. Lloyd may not be Anna Paquin, but he hits the mark more often than not, and his boyish enthusiasm is used humorously. I especially like the scene where Amidala, knowing that her planet's fate may depend on Anakin winning a pod race, finds out from Anakin's friend that the boy has never even *finished* a race, let alone won one. To Amidala's horror, na´ve little Anakin smiles, puts his arm around his friend and boasts, "Kitster's right. I will this time." I think these types of moments are charming. I also like when he tells C-3PO, who he's leaving behind in the slave quarters, that he'll "try" to get his mom not to sell him. These things remind me of the priceless moment in E.T. when Drew Barrymore says of the extra-terrestrial, "I don't like his feet." The acting may not be as note perfect but these moments show a similar understanding of the way children think. I'm glad that at this age Anakin isn't talking about being a brave warrior or fighting for justice.  This playfulness makes it more dramatic later on when he realizes his purpose in life: to return to Tatooine and free the slaves (a perfectly mythic subplot that I look forward to in Episode 2).
While the heroes attempt to deal with the crisis on Naboo, we see the shady dealings of a Sith lord named Darth Sidious, who is probably Senator Palpatine in disguise, as he greatly resembles the hooded Emperor Palpatine of later installments. Sidious dispatches his apprentice Darth Maul to assassinate the Jedi. Maul has already become a pop culture icon just by appearing briefly in the first Phantom Menace trailer. His simple look of primal evil - black cloak, red and black face, crown of horns, demonic yellow eyes - has so captured the fascination of fans that virtually every 12:01 showing on May 19th must have had a guy dressed as him, toting a double bladed light saber. After such a build up, some were disappointed that Maul turns out to be a mysterious and briefly sighted presence in the movie. (But come on, man, it's your own damn fault for building a cult around a character from a movie you hadn't seen.) Maul's fanboy appeal is comparable to that of Boba Fett, who is actually much less significant in the Star Wars saga, and has less screen time. I think he's perfect.
At some point, I will be expected to choose one of three or four famous Star Wars catch phrases and use it to attack the new movie. I'll go ahead and go with, "There is a disturbance in the Force, and its name is Jar Jar Binks." Jar Jar is a computer generated creature called a Gungan, an outcast from the underwater kingdom on Naboo. He's also the comic relief character, and the first character in the entire Star Wars saga that I can't stand. I want to make one thing clear, before I attack Jar Jar: I am pro-Ewok. I never knew until a few years ago that Return of the Jedi, and specifically the Ewoks, were looked upon with disdain by many Star Wars fans. The Ewoks, goes the argument, are cuddly little bears just thrown in to be cute and to attract kids. Having watched the movie several times since learning of this school of thought, I still can't find any way that it makes sense. The Ewoks are savage little bastards who try to cook our heroes for dinner, and who stone, log and rope stormtroopers to death in the climactic battle that culminates the humanity-over-technology theme of the trilogy. The first time you see Ewoks, they are creepy and irritable. They look more like rabid rats than teddy bears. They're not half as cute as the Jawas, who as far as I know have never suffered the "cute" criticism. And even if the Ewoks are adorably cute to some people, isn't that sort of cool?  It's like saying the killer dolls in Barbarella were just thrown in to appeal to kids who like dollies. It's just plain wrong. Up with Ewoks, I say. But there is an Ewok moment that I don't think is so hot: the one in the middle of the battle where Wicket swings a rock around and hits himself in the face with it. It's not a terrible joke, I can live with it, but it's cartoonishly out of place. It kind of seems like it should be accompanied by Three Stooges sound effects. Imagine a character that has none of the rabid-rat-menace of the Ewoks, but swings rocks into his face repeatedly. That's Jar Jar. Some people have called Jar Jar a racist stereotype - a rather ludicrous description for a character that is more reminiscent of Slimer from the Ghostbusters cartoon and Roger Rabbit than of any specific race, ethnicity or nationality. I don't have a problem with his Gungan pidgin ("yousa thinks you people gonna die?") which continues the Star Wars tradition of cultural detail amongst aliens. I do have a problem with his voice, though, which is that same annoying squeaky voice they used for Slimer, or for Punky Brewster's cartoon sidekick Glomer. You know - the "funny" voice that Dave Coulier used on Full House after he tired of his Popeye and Bullwinkle impressions. I want Jar Jar to be a character I can take seriously on occasion, but he's not, because he speaks in a voice that says, "I'm sure as hell not funny, but you know I'm trying to be!" If he were to speak in a lower voice he wouldn't be nearly as hard to take. But even still, his purpose in the story so far seems to be to knock things over, stick out his tongue and step in creature manure. His character doesn't grow or evolve. He only helps people on accident. He doesn't show another side, and worst of all he doesn't build a strong relationship with any of the other characters - he just follows them around, but it seems like you're supposed to assume they like him. If he returns in future installments it's going to be hard to redeem him. Unless little Anakin first practices his long distance strangling technique on the zany Gungan.
But The Phantom Menace survives Jar Jar, because it's working on other levels than just the wacky guy falling down. The plot is more complicated than previous chapters, and although many have attacked it as a "kiddie movie" due to the Jar Jar problem, it's hard for me imagine kids knowing what the hell is going on. Even for reasonably intelligent adults like - I'd like to think - me, it may take multiple viewings to keep up with all of the politics. How many kiddie movies revolve around tax disputes, trade embargoes, votes of no confidence? If this movie is for kids I wonder if tense senate hearings will become the next big thing in PG rated movies, replacing the John Hughes trademark of inept burglars who get pelted in the balls by Rube Goldberg-esque booby traps. It wasn't until the Special Editions that I thought about the multiple meanings of the title "Return of the Jedi." Growing up, I assumed it referred to Luke's return to Tatooine. But now I think it refers both to the resurrection of the concept of the Jedi and to the return of the Jedi Anakin Skywalker, who has been buried inside the Sith lord Darth Vader until he re-emerges in the climax. "The Phantom Menace" has multiple meanings as well. Most obviously, it refers to the hooded Darth Sidious, who controls the invasion of Naboo and the attacks on the Jedi without our heroes ever knowing of his existence. Secondly, it refers to the invasion of Naboo itself, which on the surface appears to be about taxes but is actually a front for removing the Chancellor and replacing him with Palpatine. With Anakin's boyish optimism and Jar Jar's wacky antics, the surface of the movie is fairly light-hearted. But the menace lies underneath, because of our knowledge of what is to come. This is a movie where the heroes put their faith in something we know won't pan out the way they want it to. Roger Ebert has said that Luke Skywalker is actually the chosen one who the Jedi are looking for in Anakin. I prefer the more ominous theory that Anakin is indeed "the chosen one, who will bring balance to the force," but that he will bring balance by being the evil Darth Vader. The Jedi's evil counterparts the Sith have been extinct for a millennium, so clearly the balance is leaning toward the good side of the scale. I think Yoda and Mace Windu can sense that Anakin is indeed the chosen one of their faith, but they don't want to face what that means. The film asks you to care about little Anakin, and it works. It wasn't until the day after my first viewing that some of the irony hit me. At one point, Darth Maul speeds toward Qui-Gon and Anakin on a sort of speeder bike. Qui-Gon yells, "Anakin, drop!", saving the life of Darth Vader, the man who will help conquer the galaxy and who will slay Qui-Gon's apprentice, Obi-Wan. While the movie pretends to be a happy adventure about a boy and his Jedi buddies it inches toward heartbreaking tragedy.
 The genuinely thrilling climax jumps smoothly between the Queen's siege of her own palace, a huge battle between Gungans and battle droids, Darth Maul taking on Qui-Gon and Obi-Wan in a light saber duel, and Anakin's "accidental" attack on a space station. (I put "accidental" in quotes because although Anakin doesn't destroy the control ship intentionally, I suspect the Force is working through him to achieve its own goals.) The light saber fight is the best in the Star Wars saga so far, containing more nimble choreography than previous entries. Maul, played by the stuntman Ray Park, works in a number of flips and martial arts kicks (things that were stiffly hinted at as Jedi skills in The Empire Strikes Back) and takes on two Jedi at once. One of my favorite moments in the film is when a series of energy-walls separates the three combatants, so they immediately turn off their light sabers and wait. Qui-Gon patiently meditates while Maul paces back and forth, his eyes satanically glued to his enemy. In all of the depictions of raw evil in Lucas' films, none has ever approached the pure hatred in this man's eyes.
For fans of the series, The Phantom Menace is loaded with precious touches. We get to see the time when Jedi were plentiful, so much so that they had a council led by Yoda and Samuel L. Jackson. We see cameos by rowdy sand people, Wookie senators, even Jabba's girlfriend. But most of all, we get to explore new planets and learn new details of the old ones, like the workings of the Tatooine slave trade. I didn't like Jar Jar, as I mentioned, but another computer-generated character, Watto, is more successful. He's a little, snouted guy with humming bird wings (the first flying character in the Star Wars universe?) who trades junk and makes Anakin work in his shop. He's a slimy character who is never redeemed, but when he loses Anakin in a bet he looks kind of sad, and you almost feel sorry for him. I often think it's this extra dimension - like the mourning Rancor owner in Return of the Jedi - that makes me love the Star Wars universe, and make it seem like a real place. There are other touches that weaken that real place feeling, though. Some of the character designs seem a little out of place.  It damages the slice-of-alien-life illusion that I loved so much in the cantina scene and in Jabba's palace.
But for the most part the illusion is there, and in far more visual detail than ever before. Lucas has truly taken computer-generated imagery to the logical next level, painting entire worlds with it, not just inserting a monster here or there. Though I believe The Phantom Menace will be remembered fondly for the same reasons as the other Star Wars movies, it will be most notable historically as the movie that raised the stakes for digital visual effects by about fifteen or twenty notches. Acclaiming the special effects in a movie usually sounds like faint praise, and with a Star Wars movie it pretty much goes without saying. But I don't think people are acknowledging the kind of unprecedented visual spectacle we're dealing with here, where the vast majority of shots combine live action elements with computer animation, matte paintings and miniature models. It's so elaborate and non-stop that it becomes its own seamless reality. In the original trilogy you might think okay, these are real actors on a real set, but this next shot coming up - that's stop motion. The Phantom Menace visuals are so elaborate that there's no room for that. There are stretches where, as far as I know, nothing you're seeing on screen is real, but you don't think of it as Toy Story, you think of it as a real desert with real podracers flying through it. I'll be damned if I can figure out what's a computer image, what's a model and what's just a pile of dirt. The phenomenal detail of The Fifth Element seems lazy compared to what goes on in The Phantom Menace. I believe this will have a major effect on summer event movies where the main gimmick is the visual effects. How can you make a movie like Godzilla after everyone has seen this? "Hey, look at this - a computer animated monster. Can you believe this? Pay me money." The Phantom Menace has a sequence early on, one of the weaker ones in fact, that has three computer-generated giant fish with headlights shining on them in the darkness of water. All of them are far more impressive than the giant iguanaman in Godzilla, which had nothing else to offer. In The Phantom Menace, they're a forgettable throwaway, not the main event. Will this force the purveyors of big genre movies to put some more elbow grease into their special effects? Or will it make them say, "Fuck it, I can't deal with it, I'm doing something smaller"? I don't know. But I don't think it can be ignored. Lots of people mourn the dominance of big budget effects movies and prefer smaller, earthbound, character driven movies. Movies that take place in the real world, or that leave things to the imagination.  But Lucas is dealing in fantasy films, in documenting worlds that don't exist and making them seem like they do. Since at least the summer of 1990, when the T-1000 morphed into liquid metal, filmmakers have been talking about how computer technology will narrow the gap between what you can imagine and what you can put on screen. The Phantom Menace has proven that theory. For some filmmakers this will be liberating, for others it might be frightening. Because what if we can see into their imagination and there's nothing there? What if all they can think of is a big iguanaman?
Lucas gives us more than an iguanaman. Lots, lots, lots, lots, lots more. He has given me a chance to see my 2 year-old nephew holding a plastic light saber, not really knowing what it is, saying, "I'm going to rescue Grandma." He's made me remember what it is that I felt in that drive-in so many years ago, seeing Star Wars as a child. And at the same time he's made me realize what the adults of the time - the ones who were turned on by Star Wars, anyway - might have felt, and how they might have questioned the future of popular entertainment in light of what they've just witnessed. I suspect The Phantom Menace is the beginning of more than just the Star Wars saga. But I guess I'll have to wait until the next episode to find out for sure.
--Bryan Frankenseuss Theiss
"I write rhymes so fresh I try to bite my own verses." --Tash
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


How to Suggest Improvements, Report Issues or Contribute
========================================================
This is a personal open-source project, developed on my spare time. Any contribution, suggestion, feedback or bug report is welcome.
To suggest improvements, report a bug or any issue, please use the issue reporting page,
providing all the information and files to reproduce the problem.
You may also contact the author directly to provide feedback.
The code is available in a Mercurial repository on Bitbucket.
You may use it to submit enhancements using forks and pull requests.
python-oletools documentation


Chapter 7 BAE 204 Glossary
This glossary is meant to assemble terms that we routinely use in Environmental Sciences and Engineering and which are expected to be mastered by students taking BAE 204 at NC State university. They are ordered in alphabetical list for better retrieval and look up.
7.1 A
7.1.1 Aerobic respiration
7.1.3 Ammonia
- Ammonia is a colorless gas with a characteristic pungent smell
- Formula: \(NH_3\)
- Ammonia 3D shape:
- Lewis dot structure:
- Number of electron N has for itself following electronegativity rule: eight
- \(NH_3\) can only be an electron donor
- Because N has so many electrons to be potentially donated, ammonia is generally unstable in an aerobic environment. As a result, it tends to trace quantities in nature
- When dissolved in water, and depending on the pH of the solution, ammonia converts to ammonium following the reaction:
\[ H_2O + NH_3 \rightleftharpoons OH^{-} + NH_4^{+} \]
- Production:
- Because of its many uses, ammonia is one of the most highly produced inorganic chemicals. Dozens of chemical plants worldwide produce ammonia. Consuming more than 1% of all man-made power, ammonia production is a significant component of the world energy budget.
- In 2014, about 88% of the ammonia produced was used for fertilizing agricultural crops
- Modern ammonia-producing plants generally depend on the Haber-Bosch process which consists into reducing dinitrogen into ammonia
- \[ 3\,H_2 + N_2 \to 2\,NH_3 \]
- Consumption:
- Ammonia is directly or indirectly the precursor to most nitrogen-containing compounds. Virtually all synthetic nitrogen compounds are derived from ammonia.
7.1.4 Ammonium
- Ammonium is the most reduced inorganic nitrogenous cation (positively charged).
- Ammonium 3D shape:
- Lewis dot structure:
- Number of electron N has for itself following electronegativity rule: eight
- It is formed by the protonation of ammonia following the reaction:
\[ H_2O + NH_3 \rightleftharpoons OH^{-} + NH_4^{+} \tag{$pK_A$ = 9.2} \]
- The relative abundance of ammonium vs ammonia depends on the pH of the solution. See figure below
- Because in most natural aqueous environments, pH is below 8, ammonium tends to be the preponderant form.
- Production:
- In nature, ammonium is a waste product of the mineralization of organic molecules
- It is added as fertilizer on soils as ammonium nitrate
- Health hazard:
- Ammonia vapor has a sharp, irritating, pungent odor that acts as a warning of potentially dangerous exposure
- Exposure to very high concentrations of gaseous ammonia can result in lung damage and death
- Drinking water standard:
7.1.5 Ammonium nitrate
- Ammonium nitrate is a chemical compound, the nitrate salt of the ammonium cation
- It is a white crystal solid and is highly soluble in water.
- Formula: \(NH_4NO_3\)
- 3D shape:
- It is predominantly used in agriculture as a high-nitrogen fertilizer
Its other major use is as a component of explosive mixtures used in mining, quarrying, and civil construction
- Production:
- Ammonium nitrate does exist naturally in mines of the Atacama desert in Chile but globally nearly all ammonium nitrate is now produced synthetically
- Byproduct of all respiration processes.  In fish and aquatic invertebrates, it is excreted directly into the water. In mammals, sharks, and amphibians, it is converted in the urea cycle to urea, because urea is less toxic and can be stored more efficiently. In birds, reptiles, and terrestrial snails, metabolic ammonium is converted into uric acid, which is solid and can therefore be excreted with minimal water loss.
- Consumption/utilization:
- It is used as a fertilizer, because it tends to release inorganic nitrogen slowly in soil. Applied as a surface fertilizer, it penetrates the soil with rainfall infiltration. Highly soluble, the nitrate anion becomes readily available to plant roots, although it is susceptible to leaching below the root system into the shallow and deep groundwater. The ammonium cation tends to adsorb to soil particles and is thus not as susceptible to leaching. Ammonium can be directly uptaken by plant roots, which thermodynamically makes sense, although because in most soils aerobic conditions are preponderant, nitrate tends to be the ion uptaken most often. Soil bacteria in the aerobic zone of the soil will oxidize adsorbed ammonium into nitrate, which then becomes available for plant roots. The whole chain of events slows the release of inorganic nitrogen to crops and thus makes for more effective fertilizers.
- Health hazard
- No direct known health hazard
7.1.6 Anoxic waters
- Anoxic waters are areas of sea water, fresh water, or groundwater that are depleted of dissolved oxygen and are a more severe condition of hypoxia (Wikipedia)
- Anoxic waters result from an IMBALANCE between oxygen supply and demand
7.2 C
7.2.1 Carbon dioxide
- Carbon dioxide is a colorless gas which density is 50% higher than that of dry air.
- Formula: \(CO_2\)
- Carbon dioxide 3D shape:
- Lewis dot structure:
- Number of electron C has for itself following electronegativity rule: zero
\(CO_2\) can only be an electron acceptor
- Production:
- oxidation of C in all organic molecules
- Almost all respiratory processes on earth (some respiration does not involve oxidation of C)
- Combustion of all Carbon-based fuel
- Consumption:
- Photosynthesis
- Atmospheric carbon dioxide is the primary carbon source for life on Earth
- Calcite precipitation in the oceans
- Photosynthesis
- Ecological Significance:
- Greenhouse Gas, which serves as reference for all other GHG
- Concentration in the atmosphere ~380 ppm on the rise, or 0.38%, or a partial pressure of 0.38 atm
In reality, concentrations are not stable, and vary widely in time and in space at the next two videos nicely show.
The next one results from the model simulations
- The following one is the combination of both models and observations
7.2.2 Carbonates
- After carbon dioxide dissolves in water, it will combine with water to form carbonic acid (\(H_2CO_3\)).
- Carbonate serves as the carbon source for aquatic vegetation
- Carbonic acid can then dissociate into bicarbonate (\(HCO_3^-\)) and carbonate (\(CO_3^{2-}\))
\[ \begin{align} H_2CO_3 & \rightleftharpoons & HCO_3^- + H^+ \tag{$pK_A$ = 3.6}\\ HCO_3^- & \rightleftharpoons & CO_3^{2-} + H^+ \tag{$pK_A$ = 6.35} \end{align} \]
- In an environment not open to the atmosphere (or where direct exchange with the atmosphere is very limited like in stream or wetland sediment), the preponderant form depends on the pH and can be calculated as illustrated on the graph below.
- In sea water, Carbonate can combine with \(Ca^{2+}\) to form Calcium Carbonate (\(CaCO_3\)), which precipitates out of solution. In other words, calcium carbonate formation is a sink for carbonate, and ultimately from \(CO_2\) addition from the atmosphere to to increased \(CO_2\) concentrations in the atmosphere.
- Carbonates are thus a great pH buffer in aquatic environments
7.4 E
7.4.1 Eutrophication
- Definitions:
- ‘an increase in the rate of supply of organic matter to an ecosystem’ (Nixon 1995)
- ‘is the enrichment of a water body with nutrients, usually with an excess amount of nutrients’ (Wikipedia)
- ‘the enrichment of water by nutrients, especially nitrogen and/or phosphorus, causing an accelerated growth of algae and higher forms of plant life to produce an undesirable disturbance to the balance of organisms present in the water and to the quality of water concerned’ (Anonymous 1991b)
- ‘the enrichment of water by nitrogen compounds causing an accelerated growth of algae and higher forms of plant life to produce an undesirable disturbance to the balance of organisms present in the water and to the quality of water concerned’ (Anonymous 1991a)
7.5 G
7.5.1 Greenhouse gases (GHG)
A greenhouse gas is a gas in an atmosphere that absorbs and emits radiant energy within the thermal infrared range. This process is the fundamental cause of the greenhouse effect. The primary greenhouse gases in Earth’s atmosphere are water vapor, carbon dioxide, methane, nitrous oxide, and ozone. Without greenhouse gases, the average temperature of Earth’s surface would be about −18 °C (0 °F), rather than the present average of 15 °C (59 °F). In the Solar System, the atmospheres of Venus, Mars and Titan also contain gases that cause a greenhouse effect. (Wikipedia)
7.6 H
7.6.2 Hydrogen Sulfide
- It is a colorless gas with the characteristic foul odor of rotten eggs.
- It is very poisonous, corrosive, flammable and acidic in nature.
- Formula: \(H_2S\)
- hydrogen sulfide #D shape:
- Lewis dot structure:
- Number of electron S has for itself following electronegativity rule: eight
- \(H_2S\) can only be an electron donor
- Unstable under aerobic conditions, will readily be oxidized into sulfate
- \(H_2S\) is a polyprotic acid which can lose up to 2 protons in water, depending on the pH.
\[ \begin{align} H_2S & \rightleftharpoons & HS^- + H^+ \tag{$pK_A$ = 7.02} \\ HS^- & \rightleftharpoons & S^{2-}+ H^+ \tag{$pK_A$ = 12.9} \\ \end{align} \]
- The figure below suggests that at pH found in most streams (4.5 to 8), \(H_2S\) is either preponderant or corresponds to at least 20% of all sulfide forms. \(H_2S\) being a highly volatile product, it explains why we can easily smell and detect it in most conditions in streams.
- Production:
- Hydrogen sulfide often results from the microbial breakdown, or mineralization, of organic matter in anaerobic conditions, such as may exist in swamps and sewers. When happening in sediment, this is referred to as sediment diagenesis
- Consumption:
7.7 L
7.8 M
7.8.1 Methane
- Under normal conditions for temperature and pressure, methane is a colorless, odorless gas main constituent of natural gas, and the simplest alkane
- Formula: \(CH_4\)
- Methane 3D shape:
- Lewis dot structure of methane:
- Number of electron C has for itself following electronegativity rule: eight
- \(CH_4\) can only be an electron donor
- Because methane has so many electrons to give, it will easily ‘burn’ in normal atmosphere (provided that ignition T°C be reached, e.g., by a spark), liberating large quantities of heat (55.5 MJ/kg). The electrons are transferred from the carbon to the oxygen atoms following two redox half-reactions to obtain the overall reaction:
\[ \begin{align} CH_4 + 2\,H_2O & \rightleftharpoons & CO_2 + 8\,H^+ + 8\,e^-\\ 2\,O_2 + 8\,H^+ + 8\,e^- & \rightleftharpoons & 4\,H_2O\\ & &\\ \hline\\ CH_4 + 2\,O_2 & \to & CO_2 + 2\,H_2O \end{align} \]
Production:
Consumption:
Ecological significance:
- Health effects:
7.9 N
7.9.1 Nitrate
- Nitrate is the stable inorganic nitrogenous anion in oxidized water
- Formula: \(NO_3^{-}\)
- Nitrate 3D shape:
- Lewis dot structure of nitrate:
- Number of electron N has for itself following electronegativity rule: zero
- \(NO_3^{-}\) can only be an electron acceptor
\(NO_3^{-}\) is technically the conjugate base of nitric acid \(HNO_3\), but the \(pk_A\) of the reaction is at a theoretical pH of -1.38. In other words, for the pH of most natural waters (4.5 < pH < 8), \(HNO_3\) is totally insignificant.
- Production:
- from the complete oxidation of inorganic nitrogenous molecules which include ammonia, ammonium, nitrite
- from the mineralization and complete oxidation of amine radicals in organic molecules
- Consumption:
- Uptake from microbes, plants, and algae for their anabolism, which consists in building complex organic molecules from inorganic ones.
- Uptake, assimilation, anabolism, immobilization are all synonymous terms to express the fact that the N atom is immobilized, at least temporarily in organic molecules.
- Because N is assimilated in organic molecules during uptake/anabolism, and because N gains electrons in the process (it is thus reduced), we refer to nitrate uptake as assimilatory nitrate reduction.
- Denitrification: under anaerobic conditions, nitrate is used by facultative anaerobes as electron acceptor to generate ATP in their respiration chain. The two major end products of denitrification are gases, namely dinitrogen (\(N_2\)) and nitrous oxide (\(N_2O\)), which leave the aqueous environment. As such, nitrate is not assimilated by any bacteria and denitrification is therefore, as opposed to uptake, referred to as **dissimilatory nitrate reduction into dinitrogen (\(N_2\)) and nitrous oxide (\(N_2O\)).
- Uptake from microbes, plants, and algae for their anabolism, which consists in building complex organic molecules from inorganic ones.
- Ecological significance:
- Because of assimilation and denitrification processes, the overall nitrate concentrations in rivers tends to diminish from the catchment headwaters to the receiving bodies such as estuaries and coastal areas. As a result, inorganic nitrogen has naturally been in very short supply in these coastal water bodies, and nitrate and traditionally been the nutrient limiting aquatic productivity there. Very much like with phosphate, algae have adapted to be able to grow in very low concentrations. Anthropogenic activities, and agriculture in particular, have largely increased the loads and concentration of nitrate reaching estuaries, to the point where nitrate is no longer the limiting factor. As a result, excess nitrate is one of the reasons for the global and persistent presence of algal blooms in estuaries and coastal waters.
- Health hazard:
- There is a heated debate about the health hazard that nitrate might pose. Some argue that if anything, there might be beneficial effects, while others argue that there are evidence of cancers linked to excess nitrate absorption. Unfortunately, arguments on both sides might not be totally independent of militantism and lobbies.
- The only consensus everybody seems to agree upon is the Blue Baby syndrome, or methemoglobinemia. Methemoglobinemia is an unusual and potentially fatal condition in which hemoglobin is oxidized to methemoglobin and loses its ability to bind and transport oxygen, hence the cyanosis (blue appearance) usually visible on fingers, toes, and lips. Nitrate reduced to nitrite in the body of humans and animals enters the body stream where it seems to directly oxidize oxyhemoglobin to methemoglobin-peroxide complex.
Picture of a Blue Baby from syndromespedia.com/blue-baby-syndrome.html
- Drinking water standards
- Although there is still a heated debate whether or not nitrate does have have detrimental health effects, the World Health Organization has provided maximum concentration guidelines of 50 mg/L as nitrate (World Health Organization 2011). These guidelines have been enacted in hard laws in the US and in Europe. The 50 mg/L as nitrate equates 11.2 mg \(NO_3\)-N/L and in the US, the drinking water standard is 10 mg \(NO_3\)-N/L.
7.9.2 Nitrous Oxide
- Commonly known as laughing gas
- Nitrous oxide has significant medical uses, especially in surgery and dentistry, for its anesthetic and pain reducing effects.
- Formula: \(N_2O\)
- Nitrous Oxide 3D shape:
- Number of electron N has for itself following electronegativity rule:
- the first one on the left has 5
- the middle N has 3
\(N_2O\) can be both an electron acceptor and an electron donor
- Production:
- \(N_2O\) is produced due to bacterial processes (over 90%) and anthropogenic processes such as burning of fossil fuel
- The two main bacterial processes are nitrification and denitrification
- Accounting that human activities have enhanced both nitrification and denitrification processes, it is estimated that overall, about 2/3rd of \(N_2O\) production is natural, and about 1/3rd is human enhanced
- \(N_2O\) is produced due to bacterial processes (over 90%) and anthropogenic processes such as burning of fossil fuel
- Consumption:
- Because of all the electrons stored on the two N atoms (5 + 3 = 8), nitrous oxide is a potential electron donor and bacteria can use it for their respiration processes
- Ecological significance:
- Powerful Greenhouse Gas, 298 times that of \(CO_2\) (EPA)
Concentration in the atmosphere ~0.0003 ppm or ~325 ppb on the rise
7.10 O
7.10.1 Oligotrophication
- ‘a decrease in the rate of supply of organic matter to an ecosystem’ (Nixon 2009)
7.11 P
7.11.1 Phosphate
- Phosphate is an inorganic chemical and a salt-forming anion of phosphoric acid
- Formula: \(PO_4^{3-}\)
- Phosphate is one of the anions of the polyprotic acid (i.e., which can liberate several protons \(H^{+}\))
- The conjugate bases of phosphate are:
- All conjugate bases are related through the set acid-base chemical equilibria:
\[ \begin{align} H_3PO_4 & \rightleftharpoons & H_2PO_4^- + H^+ \tag{$pK_A$ = 2.12}\\ H_2PO_4^- & \rightleftharpoons & HPO_4^{2-} + H^+ \tag{$pK_A$ = 7.21}\\ HPO_4^{2-}& \rightleftharpoons & PO_4^{3-} + H^+ \tag{$pK_A$ = 12.67} \end{align} \]
- The preponderant form of phosphate in a solution also depends on the pH following this relationship:
- Production:
- Phosphorus in general and phosphate in practice has remained one of the nutrients limiting the most plant productivity on our planet
- Phosphates are the naturally occurring form of the element phosphorus, found in many phosphate minerals
- Phosphate minerals are mined to obtain phosphorus for use in agriculture and industry
- The largest global producer and exporter of phosphates is Morocco.
- Within North America, the largest deposits lie in the Bone Valley region of central Florida, the Soda Springs region of southeastern Idaho, and the coast of North Carolina (near Aurora).
- Consumption:
- Uptake from all primary producer including plants and algae
- Phosphate can also be immobilized by bacteria
- In food industry, phosphates help baked goods rise, they act as emulsifiers in processed cheese and canned soup, they add flavor to cola and color to frozen french fries. They also can be added to meat, poultry and seafood to help the protein bind more water, making it juicier after freezing and reheating.
- Ecological significance:
- Phosphorus as phosphate naturally is the most limiting factor for primary productivity for land and aquatic plants. Because it tends to bind to particles, phosphates have accumulated with sediment particularly in coastal areas, where phosphate can become available again to algae through sediment diagenetic processes. As a result, phosphate is generally not considered the most limiting factor for algae in estuaries and coastal environment. However, it does tend to be the limiting nutrient in freshwater receiving bodies such as lakes and reservoirs.
- Excess phosphate in freshwater receiving bodies has been shown to be the nutrient causing some major eutrophication problems throughout the planet
7.12 S
7.12.1 Sulfate
- Sulfate is the inorganic sulfur anion stable in oxidized water
- Formula: \(SO_4^{2-}\)
- Sulfate 3D shape:
- Lewis dot structure of nitrate:
- Number of electron S has for itself following electronegativity rule: zero
- \(SO_4^{2-}\) can only be an electron acceptor
- \(SO_4^{2-}\) is the conjugate base of Hydrogen sulfate \(HSO_4^{-}\). The figure below shows that for pH normally measured in surface water and streams (4.5-8), \(SO_4^{2-}\) is the truly preponderant form. We therefore generally omit to mention \(HSO_4^{-}\) as a chemical form that plays any significant role.
During their anabolism, primary producers uptake sulfate, but the sulfur atoms can be incorporated into amino-acids only after sulfur has been reduced, or gained 8 electrons to be in a thiol (\(-SH\)) form.
Production:
Consumption:
Ecological significance:
Health effects:
7.13 References
Anonymous. 1991a. “COUNCIL DIRECTIVE of 12 December 1991 Concerning the Protection of Waters Against Pollution Caused by Nitrates from Agricultural Sources (91 / 676 /EEC).” Official Journal of the European Communities 31 (12): 1–8. http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:31991L0676&from=EN.
———. 1991b. “Council Directive of 21 May 1991 Concerning Urban Waste Water Treatment (91/271/EEC). L 135.” Official Journal of the European Communities 30 (5): 40–52. http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:31991L0271&from=EN.
Nixon, Scott W. 1995. “Coastal Marine Eutrophication: A Definition, Social Causes, and Future Concerns.” Ophelia 41 (1). Taylor & Francis: 199–219. doi:10.1080/00785236.1995.10422044.
———. 2009. “Eutrophication and the Macroscope.” Journal of Hydrobiologia 629 (1): 5–19. doi:10.1007/s1070-00-59-z.
World Health Organization. 2011. “Nitrate and Nitrite in Drinking-Water Background Document for Development of WHO Guidelines for Drinking-Water Quality.” http://www.who.int/water_sanitation_health/dwq/chemicals/nitratenitrite2ndadd.pdf.
Xie, Yihui. 2015. Dynamic Documents with R and Knitr. 2nd ed. Boca Raton, Florida: Chapman; Hall/CRC. http://yihui.name/knitr/.
———. 2018. Bookdown: Authoring Books and Technical Documents with R Markdown. https://github.com/rstudio/bookdown.
References
Nixon, Scott W. 1995. “Coastal Marine Eutrophication: A Definition, Social Causes, and Future Concerns.” Ophelia 41 (1). Taylor & Francis: 199–219. doi:10.1080/00785236.1995.10422044.
Anonymous. 1991b. “Council Directive of 21 May 1991 Concerning Urban Waste Water Treatment (91/271/EEC). L 135.” Official Journal of the European Communities 30 (5): 40–52. http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:31991L0271&from=EN.
Anonymous. 1991a. “COUNCIL DIRECTIVE of 12 December 1991 Concerning the Protection of Waters Against Pollution Caused by Nitrates from Agricultural Sources (91 / 676 /EEC).” Official Journal of the European Communities 31 (12): 1–8. http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:31991L0676&from=EN.
World Health Organization. 2011. “Nitrate and Nitrite in Drinking-Water Background Document for Development of WHO Guidelines for Drinking-Water Quality.” http://www.who.int/water_sanitation_health/dwq/chemicals/nitratenitrite2ndadd.pdf.
Nixon, Scott W. 2009. “Eutrophication and the Macroscope.” Journal of Hydrobiologia 629 (1): 5–19. doi:10.1007/s1070-00-59-z.


This is the application home page. It is being served from the home.html partial, via Angular routing
The following message was received from the "ws/hello" webservice:
"{{ ctrl.hello }}"
The following message was received from the "ws/helloMongo" webservice:
Page load counter is: "{{ ctrl.helloMongo }}"


HOME
DESIGN
LOGIN
Search
Log in to your account
Email
Password
Login
or
Sign in with Facebook
Don't have an account? Create one now!
Email
Password
Confirm password
Sign Up


Weekly roast of garbage product design
by Mark FrauenfelderThe design website Core77 has a weekly feature called Design Roast, devoted to making fun of bad industrial design. My "favorite" this time around is a portable leather chess board with rubber pieces that you are meant to wrap around stones you gather before you can play.
This is a leather chess board that comes with silicone bands delineating the pieces; you're meant to find and gather rocks to tie the bands around, in order to complete the pieces (I'm not kidding). I'd like to see a Checkers variant of this--it would come with the board, a sausage, a zucchini, and a chef's knife.


EngineRoom
Your Wall
Your Feed
Explore
About
Log In
Sign Up
Login
Login to your EngineRoom account
Username:
Password:
Login


Updates - April 2019
Previous Versions
Previous versions of the ATT&CK website are now being saved and displayed here to give a historical reference for prior content releases.
Tactics and Techniques
Enterprise
Impact Tactic:
The Impact Tactic was added to cover integrity and availability attacks against enterprise systems. Each technique will include an Impact Type label of 'Integrity' or 'Availability'.
The tactic covers 14 techniques that were added in this update:
- Data Destruction
- Data Encrypted for Impact
- Defacement
- Disk Content Wipe
- Disk Structure Wipe
- Endpoint Denial of Service
- Firmware Corruption
- Inhibit System Recovery
- Network Denial of Service
- Resource Hijacking
- Runtime Data Manipulation
- Service Stop
- Stored Data Manipulation
- Transmitted Data Manipulation
Seven additional techniques were added:
- Compile After Delivery (Defense Evasion)
- Domain Generation Algorithms (Command and Control)
- Domain Trust Discovery (Discovery)
- Execution Guardrails (Defense Evasion)
- Group Policy Modification (Defense Evasion)
- Systemd Service (Persistence)
- Virtualization/Sandbox Evasion (Defense Evasion, Discovery)
The following techniques were updated:
- External Remote Services - Added to Initial Access Tactic
- Input Prompt - Added Windows and examples
- LLMNR/NBT-NS Poisoning and Relay - Broadened scope to include hash relay
- Masquerading - Broadened scope to include right-to-left override, added adversary examples of moving and renaming system utilities to avoid detection
- Office Application Startup - Broadened scope to include Outlook Rules, Forms, Home Page, and Add-in persistence variations
- PowerShell - Updated description to include use of System.Management.Automation
- Remote System Discovery - Updated description to include accessing local hosts file
- Security Software Discovery - Removed virtualization
- Signed Binary Proxy Execution - Added msiexec.exe and odbcconf.exe variations
- Supply Chain Compromise - Added compromise of open source dependencies
- Valid Accounts - Broke out specific account types in description to include default accounts local accounts, and domain accounts
Added Digital Certificate Validation as a defense bypassed:
Miscellaneous minor changes:
- Brute Force - Minor description update
- Dynamic Data Exchange - Minor description update
- Exploit Public-Facing Application - Minor description update
- Screensaver - Minor description update
- Template Injection - Reference added
You can view the new and changed enterprise techniques in the ATT&CK Navigator by checking out the layer file we made available here. You can also check out a preview of the changes below! New techniques are green, and changed techniques are yellow.
PRE-ATT&CK
Technique deprecations:
- Domain Generation Algorithms (DGA) - Moved under Enterprise with a new definition
Mobile
New Techniques:
Groups
On both Group and Software pages, we have changed the term “Aliases” to “Associated Groups” and “Associated Software” respectively to better reflect what these terms represent. Analysts track clusters of activities using various analytic methodologies and terms such as threat groups, activity groups, threat actors, intrusion sets, and campaigns. Some groups have multiple names associated with similar activities due to various organizations tracking similar activities by different names. Malware/software faces the same challenge with different organizations assigning different names to the same or similar samples. Organizations' group and software names may partially overlap with names designated by other organizations and may disagree on specific activity.
The MITRE ATT&CK team believes that tracking overlaps in activity for both groups and malware/software is useful to analysts, which is why we began tracking the “Aliases” field many years ago. While we always recognized that these were not true, complete “aliases,” we have realized that calling these “Aliases” only furthers the confusion over group naming. Thus, we have decided to change the field “Aliases” to “Associated Groups” and “Associated Software” to more accurately represent what we are trying to express. We make a best effort to track overlapping groups and software, but we do not represent these names as exact overlaps and encourage analysts to do additional research. If you have input on associated groups or software, please contact us.
Enterprise
New Groups:
Group changes:
- APT18
- APT19
- APT1
- APT28
- APT29
- APT32
- APT33
- APT37
- APT3
- Cobalt Group
- CopyKittens
- Darkhotel
- Dragonfly 2.0
- Equation
- FIN10
- FIN6
- FIN7
- Gorgon Group
- Ke3chang
- Lazarus Group
- Leafminer
- Leviathan
- Lotus Blossom
- MuddyWater
- Night Dragon
- OilRig
- PLATINUM
- Rancor
- Scarlet Mimic
- Turla
- menuPass
PRE-ATT&CK
New Groups:
Group changes:
Mobile
Group changes:
Software
Enterprise
New Software:
- Agent Tesla
- Astaroth
- AuditCred
- Azorult
- BONDUPDATER
- BadPatch
- Cannon
- Carbon
- Cardinal RAT
- Cobian RAT
- CoinTicker
- DarkComet
- Denis
- Ebury
- Emotet
- Empire
- Exaramel
- Expand
- Final1stspy
- GreyEnergy
- HOPLIGHT
- Impacket
- KONNI
- LaZagne
- Linux Rabbit
- LockerGoga
- Micropsia
- NOKKI
- NanoCore
- Nltest
- NotPetya
- OSX_OCEANLOTUS.D
- OceanSalt
- Octopus
- Olympic Destroyer
- POWERTON
- PoshC2
- RawDisk
- Remcos
- Remexi
- Ruler
- SamSam
- Seasalt
- SpeakUp
- Twitoor
- UBoatRAT
- WannaCry
- Xbash
- Zeus Panda
- zwShell
Software changes:
- BBSRAT
- BISCUIT
- BlackEnergy
- CALENDAR
- CCBkdr
- CHOPSTICK
- CORESHELL
- China Chopper
- Cobalt Strike
- CozyCar
- DOGCALL
- Duqu
- Dyre
- Elise
- Epic
- FELIXROOT
- FinFisher
- GravityRAT
- H1N1
- HTRAN
- JHUHUGIT
- Kazuar
- Mimikatz
- Net
- OopsIE
- POSHSPY
- POWERSTATS
- PlugX
- PowerDuke
- PowerSploit
- Proxysvc
- Pupy
- ROKRAT
- RogueRobin
- SDelete
- Shamoon
- Smoke Loader
- TrickBot
- WEBC2
- XAgentOSX
- XTunnel
- Zebrocy
- dsquery
- gh0st RAT
- jRAT
- yty
Mobile
Software changes:


Answer TRUE or FALSE.
For the string hash functions, the size of the hash table limits the length of the string that can be hashed.
All hash functions should return a legal index in the hash table.
The length of the string does not change this.


Workflow
This document details NCA’s high-level workflow, hopefully to act as a guide to generally understanding what goes on without necessarily worrying about all the inner workings.
Setup
See Server Setup for getting the software installed, and Services for information about running the services NCA requires.
- Server is set up, directories mounted
- Settings file (
/usr/local/nca/settings
, for example) is customized as needed - Admin starts NCA in debug mode to create users
- Title manager creates title records for all titles the app will see
- MARC Org Code manager creates awardee codes NCA will need to know about
Uploads
SFTP (Born Digital)
- Publishers upload PDF issues routinely to your servers
- Uploads either go directly into NCA’s SFTP folder, or a script can be built to move them
- See our detailed folder and filename specs
- Uploaded issues are individually verified and queued by a workflow manager using the “Uploaded Issues” section of the NCA web app
- (Or the bulk queue CLI script is run if issues are verified out-of-band or trusted implicitly)
- The job runner picks up queued issues:
- Issues are pre-processed to ensure they can be read properly
- Issues are split so there is exactly one PDF per page of the issue
- Issues are then moved to the “page review” area for manual processing
- Somebody reviews issues in the page review area:
- Files must be renamed (see specs mentioned above)
- Files may be reordered if necessary
- If there are invalid PDFs, they may be deleted
- If the “issue” actually contains two issues, the secondary issue’s files should be removed and reuploaded in the correct folder
- If the entire issue is broken and needs to be removed from the system, developer involvement is necessary
- After files are reordered:
- They must not be touched for a while, to ensure renaming/manipulation is complete
- The job runner moves the files out of the page review folder and into the internal folder structure
- Derivatives are created so the issue has the expected ALTO XML and JP2 files
Scanned in-house
- Digital imaging personnel scan papers and run them through OCR to produce a TIFF and PDF file
- We use Abbyy for scanning, and the output PDF works with NCA
- See our detailed folder and filename specs
- Issues' PDFs and TIFFs are uploaded
- Uploads either go directly into NCA’s scans folder, or a script can be built to move them
- The job runner automatically searches the scans folder for issues ready to move into the workflow:
- They must not be touched for a while, to ensure all manipulation is complete
- The job runner moves the files out of the scans folder and into the internal folder structure
- Derivatives are created so the issue has the expected ALTO XML and JP2 files
- In the case of scans, the JP2 is built from the TIFF, not the PDF
Preparing Issues for Batching
After issues have been moved to the internal folders, and have had derivatives generated, the workflow is the same regardless of the source:
- An issue curator enters metadata for the issue and queues it for review
- An issue reviewer validates the metadata and rejects it or approves it
- Once metadata is entered and approved, the issue has its final derivative generated (METS XML) and awaits batching
- When enough issues are ready, the
queue-batches
CLI will generate batches in the configuredBATCH_OUTPUT_PATH
- A batch loader must manually load the batches into a staging server and then flags them as ready for QC
- A batch reviewer does quality control on the staging server, verifying the
batch’s issues look good
- If all is well, batch reviewer marks the issue as ready for production
- If not, they reject the batch and can then get individual issues pulled out to be re-curated or rejected from NCA entirely. The remaining issues are put into a new batch which is then set as being ready for staging (repeating step 5).
- Batches that are ready for production get removed from staging and loaded to prod by a batch loader.
- Batch loader flags the batch as “live”, and NCA moves its files to the
configured
BATCH_ARCHIVE_PATH
. - Once the batch can be confirmed as fully archived, a batch loader flags it as archived.
- Somebody with command-line access to NCA will run
delete-live-done-issues
(or set up a cron job) to purge unneeded files from NCA that are part of archived batches. It only deletes files when a batch is at least four weeks past its archive date to ensure any final problems can be handled.


Search
Preparing search index...


Audio Virtual Machine
A tutorial by Graham Wakefield & Charlie Roberts published originally here.
We have designed a target language that captures some essential concepts for a musical live coding context, such as playing notes and looping patterns. It is intentionally limited to make it fairly easy to learn, yet also intentionally designed to make it possible to express a wide variety of ideas. It is not designed to be succinct -- that's the job of the user-facing language you create! The target language is a bit like the instruction sequences real compilers use, in that it is a list of commands. However ours can also have nested structures as lists of lists. A simple example is a pattern that plays a note every beat:
[["@pluck", 1, "@wait"], "@loop"]
Sounds
The VM comes with a small number of instruments to get things going. Here's a plucked string instrument. There are two ways to trigger it. You can use "@pluck:note" to set the one parameter:
[440, "freq", "@pluck:note"]
or a list of parameters if they are wrapped in an array:
[[880, "freq", 0.25, "amp"], "@pluck:note"]
Also, you can set the frequency and amplitude independently with the "@inst:set-param" syntax, and just use @pluck to trigger the note using these values. This can allow a bit more flexibility in how these values are calculated:
["@pluck"]
[330, "@pluck:set-freq", 0.5, "@pluck:set-amp, "@pluck"]
There are also some percussion sounds, which either grab amplitude from the context (set using @set-amp), or specify it using the :note variants:
[1, "amp", "@hat:note"]
[1, "amp", "@kick:note"]
[1, "amp", @snare:note"]
[1, "amp", "@hat:note"]
[1, "@kick:set-amp", "@kick"]
[[1, "@snare:set-amp", "@snare"]
[[ [ [0.3, 1], "@iter", "amp", @hat:note", 0.125, "@wait" ], 8, "@repeat", ], 4, "@repeat"]
Syntax sugar
Since writing all programs in stack notation is counter-intuitive, there's a @>>
macro that reverses the
order of the current array.
The previous example can be written as:
["@>>", "@repeat", 4, [ "@>>", "@repeat", 8, [ [0.3, 1], "@iter", "amp", @hat:note", 0.125, "@wait" ] ]]Repetition and parallelism
A list with several events normally happens at the same time:
[ 220, "freq", "@pluck:note", 330, "freq", "@pluck:note", ]Unless you add a n,@wait:
[ 220, "freq", "@set", "@pluck", 1, "@wait", 330, "freq", "@set", "@pluck" ]Here's how to do something a specific number of times:
[ [ "@pluck", 0.25, "@wait" ], 4, "@repeat" ][ 4, "@repeat", [ "@pluck", 0.25, "@wait" ], [220, "freq", "@set"], "@pluck" ]Here's how to do something forever:
[ "@loop", [ "@pluck", 0.25, "@wait" ] ]Note that @loop will run the pattern that follows it in parallel, which means that you can keep on triggering more things after it. So it's easy to set up a bunch of parallel processes:
[ "@loop", [ 0.5, "@kick-note", 2, "@wait" ], "@loop", [ 0.3, "@hat-note", 0.4, "@wait" ], "@loop", [ 550, 0.5, "@pluck-note", 0.6, "@wait" ], "@loop", [ 330, 0.4, "@pluck-note", 0.2, "@wait" ], "@loop", [ 110, 0.6, "@pluck-note", 12, "@wait" ] ]And to run something in parallel without looping it, just use @fork []:
[ "@loop", [ "@kick", "@fork", [ 0.25, "@wait", 550, 1, "@pluck-note", 0.33, "@wait", 660, 1, "@pluck-note" ], 1, "@wait" ] ]This is also handy for setting up loops with different start times:
[ "@loop", ["@kick", 1, "@wait"], "@fork", [0.5, "@wait", "@loop", ["@hat", 1, "@wait"] ] ]Expressions
Anywhere that a number would go, you can insert an expression. Here are some fun expressions.
Here's picking a value at random from a list, instead of giving the freq and amp values as numbers:
[ "@loop", [ ["@quote", [330, 440, 550, 660], "@pick"], ["@quote", [1, 0.7, 0.4], "@pick"], "@pluck-note", 0.25, "@wait" ] ]Here's alternating between a set of values in a list, one at a time:
[ "@loop", [ ["@quote", [330, 440, 550, 660], "@iter"], ["@quote", [1, 0.7, 0.4], "@iter"], "@pluck-note", 0.25, "@wait" ] ]Here's way to set the probability of something happening. Note that two lists have to be given after the "@chance"; either the first is run (if the chance happens), else the other one will be run.
[ "@loop", [ "@fork", [ 0.2, "@chance", ["@snare"], ["@tom"] ], "@hat", 0.25, "@wait" ] ]If you don't care about the 'else' case, just put an empty list [] or null. But you must put something, or else the virtual machine will get corrupted, and who knows what might happen.
[ "@loop", [ "@fork", [ 0.1, "@chance", ["@snare"], [] ], "@hat", 0.25, "@wait" ] ]Another way of getting some randomness is to use the @rand (which generates a number between 0 and 1), @srand (which generates a number between -1 and 1), and n,@randi (which generates a number between 0 and n-1):
[ "@loop", [ "@rand", "amp", "@set", "@pluck", 0.25, "@wait" ] ][ "@loop", [ "@pluck", [3, "@randi"], "@wait" ] ]There are several basic math functions that can be used to map numbers into useful ranges. Note that most math functions require the arguments first, e.g. use [a,b,@+] to add a and b:
[ "@loop", [ [[[4, "@randi"], 1, "@+"], 110, "@*"], "freq", "@set", "@pluck", 0.25, "@wait" ] ][ "@loop", [ [["@srand", 10, "@*"], 110, "@+"], "freq", "@set", "@pluck", 0.25, "@wait" ] ]Get, set, and let
FIXME: this is no longer true. At this moment you can only use @set, @let or @get preceding the variable name
The @set-freq etc. are really just examples of creating named values ("variables") that can be re-used again later. Actually you can use @set-anything to create whatever variables you like, and use @get-anything to retrieve them. Mainly this is useful when you want to re-use something a few times:
[ "@loop", [ [[[4, "@randi"], 2, "@+"], 110, "@*"], "fq", "@set", "fq", "@get", 0.6, "@pluck-note", ["fq", "@get", 1.5, "@mul"], 0.6, "@pluck-note", 0.25, "@wait" ] ]By default @set-anything will apply globally, so you can modulate a parameter from one loop while using it in another:
[ "@loop", [ [[[4, "@randi"], 2, "@+"], 110, "@*"], "fq", "@set", 1, "@wait" ], "@loop", [ "fq", "@get", 0.6, "@pluck-note", ["fq", "@get", 1.5, "@mul"], 0.6, "@pluck-note", 0.25, "@wait" ] ]If globalism isn't what you care for, you can make a name "local" to a particular loop by using @let-anything. Once you have @let- a variable in a particular @loop (or @fork) pattern, it will remain local to that pattern. Generally, use @let if you want some value that is only used within a pattern, and @set if you want something global.
[ "@loop", [ [[[4, "@randi"], 1, "@+"], 110, "@mul"], "fq", "@let", "fq", "@get", 0.6, "@pluck-note", ["fq", "@get", 1.5, "@mul"], 0.6, "@pluck-note", 0.25, "@wait" ], "@loop", [ [[[4, "@randi"], 5, "@+"], 110, "@mul"], "fq", "@let", "fq", "@get", 1, "@pluck-note", ["fq", "@get", 1.5, "@mul"], 0.6, "@pluck-note", 0.25, "@wait" ] ]One use for this is to repeatedly modify a value, such as for a counter, or a decay:
[ "@loop", [ 1, "a", "@let", 8, "@repeat", [ "a", "@get", "@snare-note", 0.25, "@wait", "a", "@get", 0.6, "@*", "a", "@set" ] ] ]The only thing special about @set-freq is that the name "freq" is used by @pluck. That is, @pluck is equivalent to: ["@get-freq", "@get-amp", "@pluck-note"]. Similarly for @set-amp etc.
Independently parallel
Say you want to be able to launch a loop, then let it keep running while you launch another. And say you want to redefine it while it plays, or stop it. To do that you need to name it. We can do that via "@spawn".
[ "foo", "@spawn", [220, 0.5, "@pluck-note", 0.50, "@wait"], "bar", "@spawn", [330, 0.5, "@pluck-note", 0.75, "@wait"], 3, "@wait", "foo", "@spawn", [110, 0.5, "@pluck-note", 1, "@wait"] ][ "foo", "@spawn", [440, 0.5, "@pluck-note", 0.5, "@wait"] ][ "bar", "@spawn", [110, 0.5, "@pluck-note", 1, "@wait"] ]Click to change the "foo" part:
["foo", "@stop"][ "bar", "@stop" ][ "@loop",[ "@quote",[0.01,0.2,0.01],"@iter","@hat-note", "@quote",[1,3],"@iter",8,"@div","@wait" ], "@loop",[ ["@quote",[14,15,16,17,18],"@iter",44,"@mul"],"freq", "@set", 0.1,"amp","@set", "@pluck", "@quote",[5,7],"@pick",32,"@div","@wait" ], "@loop",[ ["@quote",[16,6,15,5,10,4,12],"@iter",44,"@mul"],"freq", "@set", 1,"@quote",[2,3,5,6,5,6,5],"@pick","@div","amp", "@set", "@pluck", "@quote",[5,1,9,1],"@iter",32,"@div","@wait" ] ]Tempo and rate
There's a global tempo that can be manipulated:
[ 0.4,"amp","@set", "@loop",[ ["@quote",[120, 80, 60],"@iter"],"@set-bpm",4,"@wait" ], "@loop",[ ["@quote",[14,15,16,17,18],"@iter",44,"@mul"],"freq","@set", "@pluck", 0.25, "@wait" ],0.125,"@wait", "@loop",[ ["@quote",[32,34,36,24,30],"@iter",44,"@mul"],"freq","@set", "@pluck", 0.25, "@wait" ] ]You can scale the current rate:
[ 0.4,"amp","@set", "@loop", [ ["@quote",[14,15,16,17,18],"@iter",44,"@mul"],"freq","@set", "@pluck", 0.25, "@wait" ], "@loop", [ ["@quote",[14,15,16,17,18],"@iter",66,"@mul"],"freq","@set", "@pluck", 0.25, "@wait", 1.01, "@scale-rate" ] ]