Framework Patterns
A software framework is code that calls your (application) code. That's how we distinguish a framework from a library. Libraries have aspects of frameworks so there is a gray area.
My friend Christian Theune puts it like this: a framework is a text where you fill in the blanks. The framework defines the grammar, you bring some of the words. The words are the code you bring into it.
If you as a developer use a framework, you need to tell it about your code. You need to tell the framework what to call, when. Let's call this configuring the framework.
There are many ways to configure a framework. Each approach has its own trade-offs. I will describe some of these framework configuration patterns here, with brief examples and mention of some of the trade-offs. Many frameworks use more than a single pattern. I don't claim this list is exhaustive -- there are more patterns.
The patterns I describe are generally language agnostic, though some depend on specific language features. Some of these patterns make more sense in object oriented languages. Some are easier to accomplish in one language compared to another. Some languages have rich run-time introspection abilities, and that make certain patterns a lot easier to implement. A language with a powerful macro facility will make other patterns easier to implement.
Where I give example code, I will use Python. I give some abstract code examples, and try to supply a few real-world examples as well. The examples show the framework from the perspective of the application developer.
Pattern: Callback function
The framework lets you pass in a callback function to configure its behavior.
Fictional example
This is a Form class where you can pass in a function that implements what should happen when you save the form.
from framework import Form def my_save(data): ... application code to save the data somewhere ... my_form = Form(save=my_save)
Real-world example: Python map
A real-world example: map is a (nano)framework that takes a (pure) function:
>>> list(map(lambda x: x * x, [1, 2, 3])) [1, 4, 9]
You can go very far with this approach. Functional languages do. If you glance at React in a certain way, it's configured with a whole bunch of callback functions called React components, along with more callback functions called event handlers.
Trade-offs
I am a big fan of this approach as the trade-offs are favorable in many circumstances. In object-oriented languages this pattern is sometimes ignored because people feel they need something more complicated like pass in some fancy object or do inheritance, but I think callback functions should in fact be your first consideration.
Functions are simple to understand and implement. The contract is about as simple as it can be for code. Anything you may need to implement your function is passed in as arguments by the framework, which limits how much knowledge you need to use the framework.
Configuration of a callback function can be very dynamic in run-time -- you can dynamically assemble or create functions and pass them into the framework, based on some configuration stored in a database, for instance.
Configuration with callback functions doesn't really stand out, which can be a disadvantage -- it's easier to see someone subclasses a base class or implements an interface, and language-integrated methods of configuration can stand out even more.
Sometimes you want to configure multiple related functions at once, in which case an object that implements an interface can make more sense -- I describe that pattern below.
It helps if your language has support for function closures. And of course your language needs to actually support first class functions that you can pass around -- Java for a long time did not.
Pattern: Subclassing
The framework provides a base-class which you as the application developer can subclass. You implement one or more methods that the framework will call.
Fictional example
from framework import FormBase class MyForm(FormBase): def save(self, data): ... application code save the data somewhere ...
Real-world example: Django REST Framework
Many frameworks offer base classes - Django offers them, and Django REST Framework even more.
Here's an example from Django REST Framework:
class AccountViewSet(viewsets.ModelViewSet): """ A simple ViewSet for viewing and editing accounts. """ queryset = Account.objects.all() serializer_class = AccountSerializer permission_classes = [IsAccountAdminOrReadOnly]
A ModelViewSet does a lot: it implements a lot of URLs and request methods to interact with them. It integrates with Django's ORM so that you get a REST API that you can use to create and update database objects.
Subclassing questions
When you subclass a class, this is what you might need to know:
- What base classes are there?
- What methods can you override?
- When you override a method, can you call other methods on self (this) or not? Is there is a particular order in which you are allowed to call these methods?
- Does the base class provide an implementation of this method, or is it really empty?
- If the base class provides an implementation already, you need to know whether it's intended to be supplemented, or overridden, or both.
- If it's intended to be supplemented, you need to make sure to call this method on the superclass in your implementation.
- If you can override a method entirely, you may need to know what methods to use to to play a part in the framework -- perhaps other methods that can be overridden.
- Does the base class inherit from other classes that also let you override methods? when you implement a method, can it interact with other methods on these other classes?
Trade-offs
Many object-oriented languages support inheritance as a language feature. You can make the subclasser implement multiple related methods. It seems obvious to use inheritance as a way to let applications use and configure the framework.
It's not surprising then that this design is very common for frameworks. But I try to avoid it in my own frameworks, and I often am frustrated when a framework forces me to subclass.
The reason for this is that you as the application developer have to start worrying about many of the questions above. If you're lucky they are answered by documentation, though it can still take a bit of effort to understand it. But all too often you have to guess or read the code yourself.
And then even with a well designed base class with plausible overridable methods, it can still be surprisingly hard for you to do what you actually need because the contract of the base class is just not right for your use case.
Languages like Java and TypeScript offer the framework implementer a way to give you guidance (private/protected/public, final). The framework designer can put hard limits on which methods you are allowed to override. This takes away some of these concerns, as with sufficient effort on the part of the framework designer, the language tooling can enforce the contract. Even so such an API can be complex for you to understand and difficult for the framework designer to maintain.
Many languages, such as Python, Ruby and JavaScript, don't have the tools to offer such guidance. You can subclass any base class. You can override any method. The only guidance is documentation. You may feel a bit lost as a result.
A framework tends to evolve over time to let you override more methods in more classes, and thus grows in complexity. This complexity doesn't grow just linearly as methods get added, as you have to worry about their interaction as well. A framework that has to deal with a variety of subclasses that override a wide range of methods can expect less from them. Too much flexibility can make it harder for the framework to offer useful features.
Base classes also don't lend themselves very well to run-time dynamism - some languages (like Python) do let you generate a subclass dynamically with custom methods, but that kind of code is difficult to understand.
I think the disadvantages of subclassing outweigh the advantages for a framework's external API. I still sometimes use base classes internally in a library or framework -- base classes are a lightweight way to do reuse there. In this context many of the disadvantages go away: you are in control of the base class contract yourself and you presumably understand it.
I also sometimes use an otherwise empty base class to define an interface, but that's really another pattern which I discuss next.
Pattern: interfaces
The framework provides an interface that you as the application developer can implement. You implement one or more methods that the framework calls.
Fictional example
from framework import Form, IFormBackend class MyFormBackend(IFormBackend): def load(self): ... application code to load the data here ... def save(self, data): ... application code save the data somewhere ... my_form = Form(MyFormBackend())
Real-world example: Python iterable/iterator
The iterable/iterator protocol in Python is an example of an interface. If you implement it, the framework (in this case the Python language) will be able to do all sorts of things with it -- print out its contents, turn it into a list, reverse it, etc.
class RandomIterable: def __iter__(self): return self def next(self): if random.choice(["go", "stop"]) == "stop": raise StopIteration return 1
Faking interfaces
Many typed languages offer native support for interfaces. But what if your language doesn't do that?
In a dynamically typed language you don't really need to do anything: any object can implement any interface. It's just you don't really get a lot of guidance from the language. What if you want a bit more?
In Python you can use the standard library abc module, or zope.interface. You can also use the typing module and implement base classes and in Python 3.8, PEP-544 protocols.
But let's say you don't have all of that or don't want to bother yet as you're just prototyping. You can use a simple Python base class to describe an interface:
class IFormBackend: def load(self): "Load the data from the backend. Should return a dict with the data." raise NotImplementedError() def save(self, data): "Save the data dict to the backend." raise NotImplementedError()
It doesn't do anything, which is the point - it just describes the methods that the application developer should implement. You could supply one or two with a simple default implementation, but that's it. You may be tempted to implement framework behavior on it, but that brings you into base class land.
Trade-offs
The trade-offs are quite similar to those of callback functions. This is a useful pattern to use if you want to define related functionality in a single bundle.
I go for interfaces if my framework offers a more extensive contract that an application needs to implement, especially if the application needs to maintain its own internal state.
The use of interfaces can lead to clean composition-oriented designs, where you adapt one object into another.
You can use run-time dynamism like with functions where you assemble an object that implements an interface dynamically.
Many languages offer interfaces as a language feature, and any object-oriented language can fake them. Or have too many ways to do it, like Python.
Pattern: imperative registration API
You register your code with the framework in a registry object.
When you have a framework that dispatches on a wide range of inputs, and you need to plug in application specific code that handles it, you are going to need some type of registry.
What gets registered can be a callback or an object that implements an interface -- it therefore builds on those patterns.
The application developer needs to call a registration method explicitly.
Frameworks can have specific ways to configure their registries that build on top of this basic pattern -- I will elaborate on that later.
Fictional Example
from framework import form_save_registry def save(data): ... application code to save the data somewhere ... # we configure what save function to use for the form named 'my_form' form_save_registry.register('my_form', save)
Real-world example: Falcon web framework
A URL router such as in a web framework uses some type of registry. Here is an example from the Falcon web framework:
class QuoteResource: def on_get(self, req, resp): ... user code ... api = falcon.API() api.add_route('/quote', QuoteResource())
In this example you can see two patterns go together: QuoteResource implements an (implicit) interface, and you register it with a particular route.
Application code can register handlers for a variety of routes, and the framework then uses the registry to match a request's URL with a route, and then can all into user code to generate a response.
Trade-offs
I use this pattern a lot, as it's easy to implement and good enough for many use cases. It has a minor drawback: you can't easily see that configuration is taking place when you read code. Sometimes I expose a more sophisticated configuration API on top of it: a DSL or language integrated registration or declaration, which I discuss later. But this is foundational.
Calling a method on a registry is the most simple and direct form to register things. It's easy to implement, typically based on a hash map, though you can also use other data structures, such as trees.
The registration order can matter. What happens if you make the same registration twice? Perhaps the registry rejects the second registration. Perhaps it allows it, silently overriding the previous one. There is no general system to handle this, unlike patterns which I describe later.
Registration can be done anywhere in the application which makes it possible to configure the framework dynamically. But this can also lead to complexity and the framework can offer fewer guarantees if its configuration can be updated at any moment.
In a language that supports import-time side effects, you can do your registrations during import time. That makes the declarations stand out more. This is simple to implement, but it's also difficult to control and understand the order of imports. This makes it difficult for the application developer to do overrides. Doing a lot of work during import time in general can lead to hard to predict behavior.
Pattern: convention over configuration
The framework configures itself automatically based on your use of conventions in application code. Configuration is typically driven by particular names, prefixes, and postfixes, but a framework can also inspect other aspects of the code, such as function signatures.
This is typically layered over the procedural registration pattern.
Ruby on Rails made this famous. Rails will automatically configure the database models, views and controllers by matching up names.
Fictional example
# the framework looks for things prefixed form_save_. It hooks this # up with `myform` which is defined elsewhere in a module named `forms` def form_save_myform(data): ... application code to save the data somewhere ...
Real-world example: pytest
pytest uses convention over configuration to find tests. It looks for modules and functions prefixed by test_.
pytest also goes further and inspects the arguments to functions to figure out more things.
def test_ehlo(smtp_connection): response, msg = smtp_connection.ehlo() assert response == 250 assert 0 # for demo purposes
In this example, pytest knows that test_ehlo is a test, because it is prefixed with test_. It also knows that the argument smtp_connection is a fixture and looks for one in the same module (or in its package).
Django uses convention over configuration in places, for instance when it looks for the variable urlpatterns in a specially named module to figure out what URL routes an application provides.
Trade-offs
Convention over configuration can be great. It allows the user to type code and have it work without any ceremony. It can enforce useful norms that makes code easier to read -- it makes sense to prefix tests with test_ anyway, as that allows the human reader to recognize them.
I like convention over configuration in moderation, for some use cases. For more complex use cases I prefer other patterns that allow registration with minimal ceremony by using features integrated into the language, such as annotation or decorator syntax.
The more conventions a framework has, the more disadvantages show up. You have to learn the rules, their interactions, and remember them. You may sometimes accidentally invoke them even though you don't want to, just by using the wrong name. You may want to structure your application's code in a way that would be very useful, but doesn't really work with the conventions.
And what if you wanted your registrations to be dynamic, based on database state, for instance? Convention over configuration is a hindrance here, not a help. The developer may need to fall back to a different, imperative registration API, and this may be ill-defined and difficult to use.
It's harder for the framework to implement some patterns -- what if registrations need to be parameterized, for instance? That's easy with functions and objects, but here the framework may need more special naming conventions to let you influence that. That may lead the framework designer to use classes over functions, as in many languages these can have attributes with particular names.
Static type checks are of little use with convention over configuration -- I don't know of a type system that can enforce you implement various methods if you postfix your class with the name View, for instance.
If you have a language with enough run-time introspection capabilities such as Ruby, Python or JavaScript, it's pretty easy to implement convention over configuration. It's a lot harder for languages that don't offer those features, but it may still be possible with sufficient compiler magic. But those same languages are often big on being explicit, and convention over configuration's magic doesn't really fit well with that.
Pattern: metaclass based registration
When you subclass a framework-provided baseclass, it gets registered with the framework.
Some languages such as Python and Ruby offer meta-classes. These let you do two things: change the behavior of classes in fundamental ways, and do side-effects when the class is imported. You can do things during class declaration that you normally only can do during instantiation.
A framework can exploit these side-effects to do some registration.
Fictional example
from framework import FormBase class MyForm(FormBase): def save(self, data): ... application code save the data somewhere ... # the framework now knows about MyForm without further action from you
Real-world example: Django
When you declare a Django model by subclassing from its Model base class, Django automatically creates a new relational database table for it.
from django.db import models class Person(models.Model): first_name = models.CharField(max_length=30) last_name = models.CharField(max_length=30)
Trade-offs
I rarely use these because they are so hard to reason about and because it's so easy to break assumptions for the person who subclasses them.
Meta-classes are notoriously hard to implement. If they're not implemented correctly, they can also lead to surprising behavior that you may need to deal with when you use the framework. Basic assumptions that you may have about the way a class behaves can go out of the door.
Import-time side-effects are difficult to control -- in what order does this happen?
Python has a simpler way to do side-effects for class declarations using decorators.
A base-class driven design for configuration may lead the framework designer towards meta-classes, further complicating the way the framework uses.
Many languages don't support this pattern. It can be seen as a special case of language integrated registration, discussed next.
Pattern: language integrated registration
You configure the application by using framework-provided annotations for code. Registrations happen immediately.
Many programming languages offer some syntax aid for annotating functions, classes and more with metadata. Java has annotations. Rust has attributes. Python has decorators which can be used for this purpose as well.
These annotations can be used as a way to drive configuration in a registry.
Fictional example
from framework import form_save_registry # we define and configure the function at the same time @form_save_registry.register('my_form') def save(data): ... application code to save the data somewhere ...
Real-world example: Flask web framework
A real-world example is the @app.route decorator of the Flask web framework.
from flask import Flask app = Flask(__name__) @app.route('/') def hello_world(): return 'Hello, World!'
Trade-offs
I use this method of configuring software sometimes, but I'm also aware of its limitations -- I tend to go for language integrated declaration, discussed below, which looks identical to the end user but is more predictable.
I'm warier than most about exposing this as an API to application developers, but am happy to use it inside a library or codebase, much like base classes. The ad-hoc nature of import-time side effects make me reach for more sophisticated patterns of configuration when I have to build a solid API.
This pattern is lightweight to implement at least in Python -- it's not much harder than a registry. Your mileage will vary dependent on language. Unlike convention over configuration, configuration is explicit and stands out in code, but the amount of ceremony is kept to a minimum. The configuration information is co-located with the code that is being registered.
Unlike convention over configuration, there is a natural way to parameterize registration with metadata.
In languages like Python this is implemented as a possibly significant import-time side-effect, and may have surprising import order dependencies. In a language like Rust this is done by compiler macro magic -- I think the Rocket web framework is an example, but I'm still trying to understand how it works.
Pattern: DSL-based declaration
You use a DSL (domain specific language) to configure the framework. This DSL offers some way to hook in custom code. The DSL can be an entirely custom language, but you can also leverage JSON, YAML or (shudder) XML.
You can also combine these: I've helped implement a workflow engine that's configured with JSON, and expressions in it are a subset of Python expressions with a custom parser and interpreter.
It is typically layered over some kind of imperative registration system.
Fictional example
{ "form": { "name": "my_form", "save": "my_module.save" } }
We have a custom language (in this case done with JSON) that lets us configure the way our system works. Here we plug in the save behavior for my_form by referring to the function save in some Python module my_module.
Real-world example: Plone CMS framework
Pyramid and Plone both are descendants of Zope, and you can use ZCML, a XML-derived configuration language with them both.
Here is some ZCML from Plone:
<configure xmlns="http://namespaces.zope.org/zope" xmlns:browser="http://namespaces.zope.org/browser" i18n_domain="my.package"> <!-- override folder_contents --> <configure package="plone.app.content.browser"> <browser:page for="Products.CMFCore.interfaces._content.IFolderish" class="my.package.browser.foldercontents.MyFolderContentsView" name="folder_contents" template="folder_contents.pt" layer="my.package.interfaces.IMyPackageLayer" permission="cmf.ListFolderContents" /> </configure> </configure>
This demonstrates a feature offered by a well-designed DSL: a way to do a structured override of behavior in the framework.
Trade-offs
Custom DSLs are a very powerful tool if you actually need them, and you do need them at times. But they are also a lot more heavyweight than the other methods discussed, and that's a drawback.
A custom DSL is thorough: a framework designer can build it with very clean boundaries, with a clear grammar and hard checks to see whether code conforms to this grammar. If you build your DSL on JSON or XML, you can implement such checks pretty easily using one of the various schema implementations.
A custom DSL gives the potential for non-developers to configure application behavior. At some point in a DSL there is a need to interface with user code, but this may be abstracted away quite far. It lets non-developers reuse code implemented by developers.
A DSL can be extended with a GUI to make it even easier for non-developers to configure it.
Since code written in a DSL can be stored in a database, you can store complex configuration in a database.
A DSL can offer certain security guarantees -- you can ensure that DSL code can only reach into a limited part of your application.
A DSL can implement a declaration engine with sophisticated behavior -- for instance the general detection of configuration conflicts (you try to configure the same thing in conflicting ways in multiple places), and structured, safe overrides that are independent of code and import order. A DSL doesn't have to use such sophistication, but a framework designer that designs a DSL is naturally lead in such a direction.
A drawback of DSL-based configuration is that it is quite distant from the code that it configures. That is fine for some use cases, but overkill for others. A DSL can cause mental overhead -- the applciation developer not only needs to read the application's code but also its configuration files in order to understand the behavior of an application. For many frameworks it can be much nicer to co-locate configuration with code.
A DSL also provides little flexibility during run-time. While you could generate configuration code dynamically, that's a level of meta that's quite expensive (lots of generate/parse cycles) and it can lead to headaches for the developers trying to understand what's going on.
DSL-based configuration is also quite heavy to implement compared to many other more lightweight configuration options described.
Pattern: imperative declaration
You use a declaration engine like in a DSL, but you drive it from programming language code in an imperative way, like imperative registration. In fact, an imperative declaration system can be layered over a imperative registration system.
The difference from imperative registration is that the framework implements a deferred configuration engine, instead of making registrations immediately. Configuration commands are first collected in a separate configuration phase, and only after collection is complete are they executed, resulting in actual registrations.
Fictional example
from framework import Config def save(data): ... application code to save the data somewhere ... config = Config() config.form_save('my_form', save) config.commit()
The idea here is that configuration registries are only modified when config.commit() happens, and only after the configuration has been validated.
Real-world example: Pyramid web framework
From the Pyramid web framework:
def hello_world(request): return Response('Hello World!') with Configurator() as config: config.add_route('hello', '/') config.add_view(hello_world, route_name='hello')
This looks very similar to a plain registry, but inside something else is going on: it first collects all registrations, and then generically detects whether there are conflicts, and generically applies overrides. Once the code exits the with statement, config is complete and committed.
Trade-offs
This brings some of the benefits of a configuration DSL to code. Like a DSL, the configuration system can detect conflicts (the route name 'hello' is registered twice), and it allows sophisticated override patterns that are not dependent on the vagaries of registration order or import order.
Another benefit is that configuration can be generated programmatically, so this allows for a certain amount of run-time dynamism without some the costs that a DSL would have. It is still good to avoid such dynamism as much as possible though, as it can make for very difficult to comprehend code.
The code that is configured may still not be not co-located with the configuration, but at least it's all code, instead of a whole new language.
Pattern: language integrated declaration
You configure the application by using framework-provided annotations for code. This configuration is declarative and does not immediately take place.
Language integration declaration looks like language integrated registration, but uses a configuration engine like with imperative declaration.
Fictional example
from framework import Config config = Config() # we define and configure the function at the same time @config.form_save('my_form') def save(data): ... application code to save the data somewhere ... # elsewhere before application starts config.commit()
Real-world example: Morepath web framework
My own Morepath web framework is configured this way.
import morepath class App(morepath.App): pass @App.path(path='/hello') class Hello(object): pass @App.view(model=Hello) def view_get(self, request): return "Hello world!"
Here two things happen: an instance of Hello is registered for the route /hello, and a GET view is registered for such instances. You can supply these decorators in any order in any module -- the framework will figure it out. If you subclass App, and re-register the /hello path, you have a new application with new behavior for that path, but the same view.
Trade-offs
I like this way of configuring code very much, so I built a framework for it.
This looks very similar to language-integrated registration but the behavior is declarative.
It's more explicit than convention over configuration, but still low on ceremony, like language-integrated registration. It co-locates configuration with code.
It eliminates many of the issues with the more lightweight language-integrated registration while retaining many of its benefits. It imposes a lot of structure on how configuration works, and this can lead to useful properties: conflict detection and overrides, for instance.
It's a lot more heavy-weight than just passing in a callback or object with an interface -- for many frameworks this is more than enough ceremony, and nothing beats how easy that is to implement and test.
You can't store it in a database or give it to a non-programmer: for that, use a DSL.
But if want a configuration language that's powerful and friendly, this is a good way to go.
It's a lot more difficult to implement though, which is a drawback. If you use Python, you're in luck: I've implemented a framework to help you build this, called Dectate. My Morepath web framework is built on it.
In Dectate, import-time side-effects are minimized: when the decorator is executed the parameters are stored, but registration only happens when commit() is executed. This means there is no dependence on run-time import order, and conflict detection and overrides are supported in a general way.
Conclusion
I hope this helps developers who have to deal with frameworks to understand the decisions made by these frameworks better. If you have a problem with a framework, perhaps I gave you some arguments that lets you express it better as well.
And if you design a framework -- which you should do, as larger applications need frameworks to stay coherent -- you now hopefully have some more concepts to work with to help you make better design decisions.
Comments
Comments powered by Disqus


Change Log: 2.080.0
Download D 2.080.0
released May 01, 2018
Compiler changes
- Class allocators and deallocators have been deprecated
- Deprecate double initialization of immutable fields inside constructor
- Issue 18736 - constructor calls allowed after case labels
- fix Issue 5227 - X ^^ FP at compile-time
- add __traits(isReturnOnStack, func)
- Support for calling Objective-C class (static) methods has been added.
- Deprecate the use of a const/immutable/shared postblit
- The deprecation for read-modify-write operations on shared variables has ended
- The deprecation for switch cases which skip the declaration of a variable has ended
Library changes
- std.typecons.BitFlags now supports opDispatch-based property access
- std.math.rndtonl has been deprecated
- Added the SharedBitmappedBlock, as the thread-safe version of the regular BitmappedBlock.
- Implemented a new allocator, AlignedBlockList and its thread-safe version SharedAlignedBlockList
- std.exception.enforceEx was deprecated in favor of std.exception.enforce
- Fixed digits grouping for floating point number without decimal part
- std.functional.binaryReverseArgs was deprecated in favor of std.functional.reverseArgs
- apply was added to std.typecons.
Tools changes
List of all bug fixes and enhancements in D 2.080.0.
Compiler changes
- Class allocators and deallocators have been deprecated
Class allocators and deallocators have been planned for deprecation for years. Starting with this release the following code will emit deprecation messages.
class C { new(size_t size) // deprecation message { return malloc(size); } delete(void* obj) // deprecation message { free(obj); } }
See the Deprecated Features for more information.
Many alternatives for class allocators/deallcators exist. Among them is the generic std.experimental.allocator.make and std.experimental.allocator.dispose from the allocator package. For other alternatives, see the recent article about memory allocation on the DBlog and the D Wiki memory management article.
Users have leveraged allocators in order to disable GC allocation, as illustrated in the following example:
class C { @disable new(size_t size); } void main() { auto c = new C(); // Error: allocator `new` is not callable because it is annotated with `@disable` }
That idiom will remain, but has been enhanced with this release to no longer require the size_t argument. That is, starting with this release, the following syntax is also permitted:
class C { @disable new(); } void main() { auto c = new C(); // Error: allocator `new` is not callable because it is annotated with `@disable` }
- Deprecate double initialization of immutable fields inside constructor
Inside a constructor scope, assigning to aggregate declaration (class/struct) members is done by considering the first assignment as initialization and subsequent assignments as modifications of the initially constructed object. For const/immutable fields the initialization is accepted in the constructor, but subsequent modifications are not. Example:
struct A { int a; immutable int b; this(int a, int b) { this.a = a; this.b = b; this.a = 7; // OK, a is mutable this.b = 9; // Error: immutable field b initialized multiple times } }
However, Bugzilla 18719 shows that this rule does not apply when inside a constructor scope there is a call to a different constructor:
struct A { immmutable int a; this() { this(42); this.a = 5; // second initialization of immutable field } this(int a) { this.a = a; } }
The above code wrongfully compiled succesfully before this patch, accepting the double initialization of the immutable field a. After this patch, this.a = 5 will issue a deprecation warning stating that a is initialized multiple times.
- Issue 18736 - constructor calls allowed after case labels
Delegating constructor calls are not allowed after labels, but case labels and default labels should also count as labels.
class A { this(char c) { } this(int i) { switch (i) { case 1: break; default: break; } this('c'); // now gives an error } }
This is necessary so the compiler can guarantee that each field is initialized exactly once. To get code like the above to pass the compiler, replace it with an if-then sequence.
- fix Issue 5227 - X ^^ FP at compile-time
The pow operator ^^ can now be used by CTFE.
Adds these std.math functions to those that can be used by CTFE:
round floor ceil trunc log log2 log10 pow expm1 exp2 fmin fmax copysign fma
- add __traits(isReturnOnStack, func)
Determines if a function's return value is placed on the stack, or is returned via registers. For more details, see isReturnOnStack.
- Support for calling Objective-C class (static) methods has been added.
Previously to call an Objective-C class method it was necessary to make explicit calls to the Objective-C runtime. The following code is an example of the old way to call a class method:
extern (C) Class objc_lookUpClass(in char* name); extern (Objective-C) interface Class { NSObject alloc() @selector("alloc"); } extern (Objective-C) interface NSObject { NSObject init() @selector("init"); } void main() { auto cls = objc_lookUpClass("NSObject"); auto o = cls.alloc().init(); }
The above code can now be replaced with the following:
extern (Objective-C) interface NSObject { static NSObject alloc() @selector("alloc"); NSObject init() @selector("init"); } void main() { auto o = NSObject.alloc().init(); }
Note the use of the static attribute in the method declaration.
- Deprecate the use of a const/immutable/shared postblit
Before this patch, if a postblit was declared const/immutable/shared the compiler would have accepted the declaration but there would have been no way of calling the postblit succesfully, except for const due to the implicit conversion mutable -> const. Even though calling a const posblit is possible, there is no way to modify the fields of the newly copied struct:
struct S { int n this(this) const { ++n; // Error: cannot modify this.n in const method } } void main() { S a; auto a2 = a; }
With this release, if a postblit contains const/immutable/shared in its signature, a deprecation message will be emitted.
- The deprecation for read-modify-write operations on shared variables has ended
Read-modify-write operations are not allowed for shared variables:
shared int i; i++; // Error: read-modify-write operations are not allowed for shared variables
Use core.atomic.atomicOp instead:
import core.atomic : atomicOp; shared int i; atomicOp!"+="(i, 1);
- The deprecation for switch cases which skip the declaration of a variable has ended
Usage of a variable which is declared in another switch case now results in an error.
int i = 2; switch (i) { case 1: { int j; case 2: j++; j.writeln; // BUG: j is not initialized and e.g. prints -321532879 break; } default: break; }
If this behavior is wanted, it can explicitly requested by using void initialization:
int i = 2; switch (i) { case 1: { int j = void; case 2: j = 2; j.writeln; break; } default: break; }
Library changes
- std.typecons.BitFlags now supports opDispatch-based property access
std.typecons.BitFlags was extended so that enum members can be set and tested directly on the BitFlags instead of having to & with the underlying enum.
enum Features { fast = 1 << 0, size = 1 << 1, } void run(BitFlags!Features features) { // get if (features.fast && !features.size) {} // new new new if ((features & Features.fast) && !(features & Features.size)) {} // old old old // set features.fast = true; // new new new features.fast = false; // new new new features.fast |= Features.fast; // old old old features.fast &= ~Features.fast; // old old old }
This also works for unsafe BitFlags where the property get access tests for an exact match of all bits of the unsafe BitFlags combination. Analogously, the property set access clears or sets all bits of the unsafe BitFlags combination.
enum Features { fast = 1 << 0, size = 1 << 1, combined = fast | size, } void run(BitFlags!(Features, Yes.unsafe) features) { // get if (features.combined) {} // new new new if ((features & Features.combined) == BitFlags!(Features, Yes.unsafe)(Features.combined)) {} // old old old // set features.combined = true; // new new new features.combined = false; // new new new features.combined |= Features.combined; // old old old features.combined &= ~Features.combined; // old old old }
- std.math.rndtonl has been deprecated
std.math.rndtonl is a rounding function only available when using the Digital Mars C Runtime on Windows. As this function is not cross-platform, it has been deprecated, and will be removed on version 2.089. Please use std.math.round instead.
- Added the SharedBitmappedBlock, as the thread-safe version of the regular BitmappedBlock.
The new std.experimental.allocator.building_blocks.bitmapped_block.SharedBitmappedBlock and its single-threaded version can now be instantiated with Yes.multiblock or No.multiblock. If instantiated with Yes.multiblock (the default behavior), each allocation can return an arbitrary number of blocks. With No.multiblock however, any allocation request can't exceed the block size. This allows for greater performance on both single and multithreaded environments.
// The 'BitmappedBlock' is implicitly instantiated with Yes.multiblock auto a = BitmappedBlock!(blockSize, 8, Mallocator, Yes.multiblock)(numBlocks * blockSize); // Instantiated with Yes.multiblock, can allocate more than one block at a time void[] buf = a.allocate(2 * blockSize); assert(buf.length == 2 * blockSize); assert(a.deallocate(buf));
// Instantate the 'BitmappedBlock' with No.multiblock auto a = BitmappedBlock!(blockSize, 8, Mallocator, No.multiblock)(1024 * blockSize); // Since instantiated with No.multiblock, can only allocate at most the block size void[] buf = a.allocate(blockSize + 1); assert(buf is null);
For shared the same rules apply, we only need to replace BitmappedBlock with SharedBitmappedBlock.
- Implemented a new allocator, AlignedBlockList and its thread-safe version SharedAlignedBlockList
std.experimental.allocator.building_blocks.aligned_block_list.AlignedBlockList represents a list of allocators which allows for deallocations in constant time. Although allocations are in theory served in linear searching time, deallocate calls take Ο(1) time, by using aligned allocations. The ParentAllocator must implement alignedAllocate.
std.experimental.allocator.building_blocks.aligned_block_list.SharedAlignedBlockList has the same semantics as its single threaded version, however the internal allocators must be in addition marked as shared.
- std.exception.enforceEx was deprecated in favor of std.exception.enforce
With 2.079 std.exception.enforce became a complete super set of std.exception.enforceEx
Corrective action
Replace:
import std.exception; alias enf = enforceEx!Exception; assertNotThrown(enf(true)); assertThrown(enf(false, "blah"));
with:
import std.exception; alias enf = enforce!Exception; assertNotThrown(enf(true)); assertThrown(enf(false, "blah"));
- Fixed digits grouping for floating point number without decimal part
std.format.formatValue now correctly groups digits in the output string by inserting a group separator character (,) every n characters specified by the Separator grammar rule in cases with zero decimal precision specified in the format string for floating point numbers.
No group separators at all are inserted for floating point numbers when formatted with zero precision (i.e. no decimal digits) in Phobos before this fix, regardless of the respective decimal part of the formatted number.
import std.format; assert(format("%,3.2f", 1172.009) == "1,172.01"); assert(format("%,3.0f", 1172.009) == "1,172"); assert(format("%#,3.4f", 1303.2508) == "1,303.250,8"); assert(format("%#,3.0f", 1303.2508) == "1,303."); // original (buggy) behaviour before this fix //assert(format("%,3.0f", 1303.2508) == "1303"); //assert(format("%,3.0f", 1303.) == "1303"); // original (correct) behaviour before this fix //assert(format("%,3.1f", 1303.2508) == "1,303.3"); //assert(format("%,3.2f", 1303.) == "1,303.00"); //assert(format("%,3f", 1303.) == "1,303.000,000");
- std.functional.binaryReverseArgs was deprecated in favor of std.functional.reverseArgs
std.functional.reverseArgs is a full super set of std.functional.binaryReverseArgs.
Correct action: replace binaryReverseArgs with reverseArgs.
- apply was added to std.typecons.
apply is an operation for std.typecons.Nullable values that "unpacks" the Nullable, performs some operation (that is passed as a template parameter), then packs the result into another Nullable if necessary. When the initial Nullable is null, the resulting Nullable is also null and the function is not called.
Nullable!int n; alias square = i => i * i; n = n.apply!square; // does nothing if isNull assert(n.isNull); n = 2; assert(n.apply!square.get == 4);
Tools changes
- rdmd can now be used as a shell in makefiles
With gnu make(3.82 or higher), rdmd can now be used in makefiles. This is accomplished by setting the SHELL and .SHELLFLAGS to /usr/bin/rdmd and --eval respectively.
.ONESHELL: SHELL = /usr/bin/rdmd .SHELLFLAGS = --eval hello.txt: import std.file; write("$@","hello world\n");
List of all bug fixes and enhancements in D 2.080.0:
DMD Compiler regressions
- Bugzilla 16243: wrong C++ argument passing with empty struct when interfacing with Clang
- Bugzilla 18197: [REG2.073] Internal error: backend\cgcod.c 1659
- Bugzilla 18489: [REG 2.073]Internal error: dmd/backend/cgcod.c 1688
DMD Compiler bugs
- Bugzilla 5212: no escape analysis for typesafe variadic function arguments
- Bugzilla 5227: X ^^ FP at compile-time
- Bugzilla 9433: Deprecate delete
- Bugzilla 12486: Function returning struct isn't called if enum of its result is accessed
- Bugzilla 14997: __ctor present in error message
- Bugzilla 15653: IFTI fails for immutable parameter
- Bugzilla 15660: break immutable with pure function and mutable reference params
- Bugzilla 16107: [ICE] - Internal error: backend/cgcod.c 2297
- Bugzilla 16189: Optimizer bug, with simple test case
- Bugzilla 16555: Stack corruption when calling multi-parameter outer function from nested function
- Bugzilla 16578: bogus deprecation - switch skips declaration of variable
- Bugzilla 17284: Template function attribute inference wrongly infers @safe for accessing overlapping pointer fields in unions
- Bugzilla 17423: pointer assignment to this in member function is not accounted for
- Bugzilla 17874: Segmentation fault when constructing a struct with a static array of structs containing floats
- Bugzilla 17892: Scope analysis with -dip1000 fails for templated structs
- Bugzilla 17918: [Reg 2.072] ICE with unknown symbol in base class
- Bugzilla 17959: [DIP1000] Can store scope delegate in non-scope member
- Bugzilla 17991: ICE with imports without module
- Bugzilla 18000: [scope] auto-generated opAssign not scope aware
- Bugzilla 18034: SIMD optimization issues
- Bugzilla 18130: ICE on zero-length out array parameter
- Bugzilla 18199: Error with lambda in struct initializer
- Bugzilla 18242: DMD Segmentation fault.
- Bugzilla 18282: [Scope][DIP1000]Assignment of local variable to scope variable not recognized by compiler
- Bugzilla 18459: The static array ABI doesn't match the struct ABI
- Bugzilla 18461: codegen bug - OPbt expressions and assignments to ambiguous symbols
- Bugzilla 18474: Postblit not working in shared structs
- Bugzilla 18484: [dip1000] Subtype allows reference to escape with implicit casting
- Bugzilla 18504: Assert in synchronized crashes with SIGILL on exit
- Bugzilla 18505: delete deprecation message is misleading
- Bugzilla 18534: Wrong code for ?: operator when compiling with -O
- Bugzilla 18553: Inline assembler: CR2, CR3 control registers not available in 64-bit code
- Bugzilla 18554: tupleof ignoring private shouldn't be accepted in @safe code
- Bugzilla 18568: partially overlapping assignments have undefined behavior but are accepted in @safe code
- Bugzilla 18575: making delegate from member function can lead to unsafe code
- Bugzilla 18576: Compiler not doing RVO with auto returns
- Bugzilla 18597: more unsafe unaligned pointer errors
- Bugzilla 18607: using labelled continue in tuple foreach can break pure and @safe
- Bugzilla 18644: [dip1000] escape of outer local not detected
- Bugzilla 18661: auto ref and return attribute inference
- Bugzilla 18708: Flow analysis in constructors not done correctly for if, ||, &&
- Bugzilla 18719: Doubly-called constructor against member when using forwarding constructors
- Bugzilla 18736: constructor calls allowed after case labels
- Bugzilla 18737: An assert(0) should be a leaf in constructor flow analysis
- Bugzilla 18753: chunkBy compile error causes ICE
- Bugzilla 18762: DMD should use a unique path/filename for __stdin.o
DMD Compiler enhancements
- Bugzilla 15624: opApply with @safe and @system variants can't be used with foreach syntax
- Bugzilla 16037: assigning delegate to a scope variable shouldn't allocate closure
- Bugzilla 17784: [scope][DIP1000] Confusing error message for escaping local via new-expression
- Bugzilla 18417: Make const and immutable postblit constructors illegal
- Bugzilla 18460: Improve error message for missing 'new' for class instantiation
- Bugzilla 18503: Confusing error message for erroneous postblit
- Bugzilla 18551: Improve hint for "does not override any function
- Bugzilla 18574: Unclear error message when trying to inherit from multiple classes
- Bugzilla 18578: First enum value assigned 0 instead of EnumBaseType.init
- Bugzilla 18591: DMD should allow access to mixin template declarations
- Bugzilla 18694: pure functions using static/global structs with no fields should pass compilation
- Bugzilla 18767: __xtoHash memberwise hashing shouldn't just sum the member hashes
Phobos regressions
- Bugzilla 17961: std.uni does not compile with -unittest -dip1000
- Bugzilla 18800: [REG2.080.0-beta.1] Array.length setter segfaults for payloads with indirections
Phobos bugs
- Bugzilla 7879: format of class with not const toString()
- Bugzilla 13642: std.container.Array: change of length reallocates without notifying GC
- Bugzilla 15768: std.stdio.File does not support __gshared semantics of stdout/err/in
- Bugzilla 17862: std.random.XorshiftEngine.min is wrong when bits == 32
- Bugzilla 17982: Support for const(Class) in algorithm.searching.extremum
- Bugzilla 18133: BitArray constructors are poorly documented.
- Bugzilla 18134: BitArray >>= broken when length % (8 * size_t.sizeof) == 0
- Bugzilla 18452: std.zip has size limit of 2 GB
- Bugzilla 18524: std.range.Zip.moveBack instead performs moveFront
- Bugzilla 18579: No group separators for floating point number formatted with zero decimal digits
- Bugzilla 18615: Rebindable!A doesn't use class A's opEquals (returns a is b instead)
- Bugzilla 18634: std.container.rbtree does not work with delegate comparators
- Bugzilla 18663: std.random.isSeedable has false positives
- Bugzilla 18669: isNestedFunction does not check if parameter is a function
- Bugzilla 18689: std.format should always throw FormatException on bad specs/arguments
- Bugzilla 18704: std/datetime/stopwatch.d(452) assertion failure on auto-tester
- Bugzilla 18714: Phobos calls localtime, which is not threadsafe
Phobos enhancements
- Bugzilla 4591: Concat of std.typecons.Tuples
- Bugzilla 8341: topN(zip()) too?
- Bugzilla 10828: datetime toString functions should accept sink
- Bugzilla 13880: nothrow @nogc std.algorithm.reduce on fixed-size arrays
- Bugzilla 14637: Array operations should work on tuples
- Bugzilla 18158: std.file.getcwd should be usable in @safe
- Bugzilla 18166: std.array.replace should be usable in @safe for dstrings
- Bugzilla 18168: std.base64.encode for output ranges should be usable in @safe
- Bugzilla 18174: std.math.equalsDigit should be usable in @safe
- Bugzilla 18182: std.uri should be usable in @safe
- Bugzilla 18223: std.experimental.allocator uninitializedFillDefault could use memset
- Bugzilla 18438: hasStaticMember fails with a T* while hasMember passes
- Bugzilla 18501: randomShuffle and partialShuffle should return their input argument
- Bugzilla 18596: std.random.unpredictableSeed could use something better than MinstdRand0
- Bugzilla 18600: Regex performance enhancement for repeated matchFirst calls
- Bugzilla 18638: The main override of std.range.zip could infer @nogc and nothrow
Druntime regressions
- Bugzilla 18652: hashOf example doesn't compile
Druntime bugs
- Bugzilla 18482: wincrypt functions should be @nogc nothrow
- Bugzilla 18536: Bad stack traces when building with -m32mscoff
- Bugzilla 18537: Cannot pass absolute path to coverage options
- Bugzilla 18547: Win32: throwing exception in fiber crashes application
dlang.org bugs
- Bugzilla 18602: [Better C] docs
- Bugzilla 18603: Illegal instruction: 4 on Mac OS 10.7.5 Leopard
- Bugzilla 18702: FAQ item "What does D have that C++ doesn't?" comparison link (https://dlang.org/articles/comparison.html) returns 404
- Bugzilla 18709: DDOC_AUTO_PSYMBOL and friends are wrong
- Bugzilla 18747: [Structs, Unions]
Installer bugs
- Bugzilla 18649: curl on Ubuntu 18.04 depends on libcurl4, .deb installer depends on libcurl3
Contributors to this release (65)
A huge thanks goes to all the awesome people who made this release possible.
- Adam D. Ruppe
- aG0aep6G
- Alexandru Caciulescu
- Alexandru Jercaianu
- Ali Akhtarzada
- Andrei Alexandrescu
- Andrei-Cristian VASILE (87585)
- Basile Burg
- Brad Roberts
- byebye
- carblue
- cedretaber
- crimaniak
- David Nadlinger
- Denis Feklushkin
- Diederik de Groot
- Dmitry Olshansky
- drug007
- Eduard Staniloiu
- FeepingCreature
- Giles Bathgate
- Harry T. Vennik
- Iain Buclaw
- Jack Stouffer
- Jacob Carlborg
- Joakim Noah
- Johan Engelen
- John Belmonte
- Jonathan M Davis
- Jonathan Marler
- Joseph Rushton Wakeling
- Kai Nacke
- kinke
- LemonBoy
- Martin Kinkelin
- Martin Nowak
- Mathias Lang
- Mathis Beer
- MetaLang
- Mike Franklin
- Mike Parker
- n8sh
- Nathan Sashihara
- Nicholas Lindsay Wilson
- Nick Treleaven
- Oleg Nykytenko
- Per Nordlöw
- Radu Racariu
- Rainer Schuetze
- Razvan Nitu
- Roman Chistokhodov
- rracariu
- Ryan Frame
- Sebastian Wilzbach
- Simon Naarmann
- skl131313
- Steven Schveighoffer
- Superstar64
- Thomas Mader
- Timothee Cour
- Unknown
- viktor
- Vladimir Panteleev
- Walter Bright
- Ľudovít Lučenič


LDM Reference Glossary and Index
This is a hyperlinked glossary and index of the terms, and
phrases used to explain the fundamentals of the Unidata LDM.
A
B
C
D
E
F
G
H
I
K
L
M
N
O
P
R
S
T
U
W
V
Y
A
- ACCEPT entry
- An entry in the LDM
configuration-file
ldmd.conf that allows an
upstream LDM to send
unrequested
data-products.
Decription of entry in
ldmd.conf.
- ALLOW entry
- An entry in the LDM
configuration-file
ldmd.conf that allows a
downstream LDM to connect
and indicates what
data-products it
is allowed to receive.
- anti-DOS feature
- A feature of the LDM that mitigates deliberate or accidental
denial-of-service attacks by ensuring non-overlapping feeds to every
downstream host. This feature is enabled by default but can be disabled
through the enable-anti-DOS
parameter of the LDM registry. See the
description of the behavior of an
upstream LDM process for details.
- asynchronous
- Describes something that does not
cause the initiator to wait until the thing is finished. For
example, an asynchronous RPC message does
not cause the sender to wait for a reply.
- autoshift
- Describes the capability of a
downstream LDM 6 to change the
transfer-mode
of its connection depending on its success-rate of inserting
data-products
into the
product-queue.
LDM 6.4 was the first version to have this capability.
B
- BLKDATA
- A message from an
upstream LDM
to a downstream LDM
containing
data-product data.
LDM 6 protocol message.
C
- child process
- A UNIX� process created by a
parent process.
Official description.
- CONDUIT
- A feedtype comprising a huge
volume of model output.
- creation-time, data-product
- See data-product
creation-time.
- crontab utility
- A UNIX� utility for scheduling the periodic
execution of commands.
Official description. See
suggested entries for the LDM
user's crontab(1) file.
- COMINGSOON
- An
RPC message from an
upstream LDM
to a downstream LDM
containing data-product metadata
asking if the
downstream LDM wants the
corresponding
data-product.
LDM 6 protocol message.
D
- data-product
- The fundamental quantum of data acted upon by an LDM. Structure of a data-product.
- data-product data
- The data portion of a data-product.
Further information.
- data-product creation-time
- The time when the data-product was created
by the relevant LDM
ingester.
- data-product identifier
- A character-string that uniquely identifies a data-product. Product
identifiers are created by LDM
ingesters using the policy of the administrative entity
responsible for the relevant feedtype.
- data-product latency
- The difference between when a
data-product is received
by a
downstream LDM and the
data-product's creation-time.
- data-product metadata
- The metadata portion of a data-product.
Further information.
- data-product origin
- A character-string that identifies the computer on which the
LDM
ingester
was running when it created the data-product. Typically,
this is the fully-qualified hostname of the computer.
- data-product selection-criteria
- The criteria used for selecting
data-products.
The criteria is composed of a start-time and a stop-time that must
bracket the data-product
creation-time of matching products, and a list of
pairs of feedtype and
data-product identifier
ERE-s.
- data-product signature
- An array of 16 unsigned bytes that is unique to a data-product. Signatures
are computed by LDM
ingesters
using the
MD5 checksum algorithm.
An MD5 checksum webpage.
- data-product size
- The number of bytes in the data section of a data-product. The value must
be less than or equal to 0xFFFFFFFF<PHONE_NUMBER>).
- decoder
- A feedtype-specific program
that disposes of data-products
that it receives from the pqact
process.
Further information.
- deprecated
- This item or feature might go away in a future release.
Users are cautioned against using or depending on it.
- downstream host
- The computer on which a
downstream LDM is
running.
- downstream LDM
- Typically, a forked,
child process of ldmd that sends a
FEEDME request to an
upstream LDM and,
consequently, receives
data-products, which it
then writes into the product-queue.
In atypical situations, this phrase can also refer to a notifyme process.
- downstream LDM 5
- A downstream LDM that
uses the LDM 5 protocol.
For backward compatibility,
LDM 6 is capable of acting as a
downstream LDM 5.
- downstream LDM 6
- A downstream LDM that
uses the LDM 6 protocol.
Behavior.
E
- EXEC entry
- An entry in the LDM
configuration-file
ldmd.conf that tells the
top-level ldmd to
asynchronously execute a program as a
child process on startup.
- environment variable
- A generally mutable object with a name and a character-string value
that can be set by
a user via the shell or by a program. Copies of environment
variables are inherited by
child processes.
Official description.
- execvp() function
- A UNIX� function that executes a command-string.
Official description.
- ERE
- Extended Regular Expression. ERE-s provide a mechanism to select
specific strings from a set of character strings.
ERE webpage.
F
- FEEDME
- A message from a
downstream LDM
to an upstream LDM
requesting data-products.
LDM 6 protocol message.
- feedtype
- A class of data-products
coming from a common source (e.g., WMO, NPORT). Feedtypes may
be hierarchically arranged. A feedtype may be represented as an
integral value (e.g., 1) or as a name (e.g., PPS).
Feedtype webpage.
- fork
- A UNIX� system-call that duplicates the
invoking
parent process to create a new
child process.
Official description.
G
- gateway LDM
- A site's
LDM
that receives data from offsite or that sends data to an offsite
LDM
- gdbm database
- A database based on the
GNU dbm database routines.
- GEMPAK
- A UPC-supported package for
analyzing and displaying primarily meteorological data
(with emphasis on gridded model output).
GEMPAK homepage.
- gnuplot utility
- A freely-available, 3rd-party UNIX� utility for
graphing data.
GNU distribution.
- grep utility
- The UNIX� utility for searching text files.
Official description.
H
- HEREIS
- A message from an
upstream LDM
to a downstream LDM
that contains a data-product.
- HIYA
- An unsolicited message from an
upstream LDM
to a downstream LDM
specifying what data-products
are about to be sent.
LDM 6 protocol message.
I
- IDD
- The Unidata Internet Data Distribution system. The IDD system is a
network of LDMs and their
associated institutions that cooperate to distribute
meteorological and related data-products for research
and education under the auspices of the Unidata program.
IDD homepage.
- IDD rtstats webpages
- A set of webpages for displaying the statistics of
non-gateway LDMs on the Unidata
IDD that are reported via the
rtstats utility.
Top-level webpage.
Use in monitoring the LDM.
- identifier, data-product
- See data-product identifier.
- ingester
- A feedtype-specific program
that creates data-products and
inserts them into the product-queue. Further information.
- insertion-time
- The time at which a
data-product
was inserted into the local
product-queue.
- iostat utility
- A commonly-available UNIX� utility for
displaying system I/O statistics.
- IS_ALIVE
- A message from a
downstream LDM
to the top-level
ldmd process on
an upstream host
inquiring whether or not the relevant
upstream LDM process is
still running.
LDM 6 protocol message.
K
- kill utility
- The UNIX� utility for sending signals to processes.
Official description.
L
- LDM
- Local Data Manager. A system of software for the distribution
and processing of
data-products in near
real-time via the Internet.
LDM homepage.
- LDM, downstream
- See downstream LDM.
- LDM, receiving
- See downstream LDM.
- LDM, sending
- See upstream LDM.
- LDM, upstream
- See upstream LDM.
- LDM 4
- An LDM that uses the
LDM 4 protocol.
- LDM 4 protocol
- The LDM protocol used by version 4
of the LDM.
- LDM 5
- An LDM that uses the
LDM 5 protocol.
- LDM 5 protocol
- The LDM protocol used by version 5
of the LDM.
- LDM 6
- An LDM that uses the
LDM 6 protocol.
- LDM 6 protocol
- The LDM protocol used by version 6
of the LDM.
LDM 6 protocol webpage.
- LDM log file
- The file used to log diagnostic messages from
LDM programs.
Format and interpretation.
Rotation and purging.
- LDM protocol
- The set of RPC messages used by a version of the
LDM to
communicate with another LDM of the
same version.
- LDM registry
- An LDM-specific database of configuration parameters.
LDM registry webpage.
- LDM user
- The user on a computer system that owns the local LDM installation. Typically, the
user's name is "ldm". LDM user webpage.
Creating user-account for.
Becoming the LDM user.
- ldmadmin
- Perl script for managing an operational
LDM system.
Further information.
- ldmd
- The top-level LDM server.
- ldmping
- A utility that temporarily connects to an
upstream LDM to
test its availability.
- ldmd.conf
- The configuration-file for the top-level ldmd process.
ldmd.conf webpage.
- lex utility
- A UNIX� utility for generating a lexical scanner.
Official description.
- localn
- The facility used by the LDM for logging messages via the
syslog daemon.
By default, the local0 facility is used.
Configuring the
syslog.conf file.
Setting the facility via the
configure
script.
- logfile, LDM
- See LDM logfile.
M
- make utility
- A UNIX� utility for maintaining, updating, and
regenerating groups of programs.
Official description.
- man utility
- A UNIX� utility for displaying reference-documentation on
UNIX� utilities.
Official description.
User-support example.
- MANPATH
- The environment variable that
determines what directories are searched for
manual pages and in what order.
Setting in preinstallation step.
User-support reference.
The man(1) utility.
- maximum HEREIS size
- The threshold, in bytes, above which an
upstream LDM 6 uses
a
COMINGSOON/BLKDATA
message-pair to send a
data-product rather than a
single
HEREIS message.
- McIDAS
- A UPC-supported package for
analyzing and displaying primarily meteorological data
(with emphasis on satellite imagery).
McIDAS homepage.
- metadata, data-product
- See data-product metadata.
- minimum virtual residence
time
- The minimum, observed
virtual residence time
since this parameter was initialized (either by creation of the
product-queue or via the
pqutil utility.
If this interval is not greater than the maximum
data-product latency configuration
parameter, then a
downstream LDM
might not reject a duplicate incoming
data-product.
N
- netstat utility
- A utility for displaying network connections.
- NOAAPORT
- A satellite broadcast network run by NOAA to diseminate weather data to
field offices. The
LDM
can be
built
to ingest data-products from the NOAAPORT broadcast.
- NOTIFICATION
- A message from an
upstream LDM to an
downstream LDM
containing data-product metadata.
LDM 6 protocol message.
- NOTIFYME
- An RPC message from a
downstream LDM to an
upstream LDM
requesting data-product metadata.
LDM 6 protocol message.
- notifying upstream LDM 5
- An upstream LDM 5 that
sends NOTIFICATION
messages to a
downstream LDM 5
rather than data-products.
- notifyme utility
- A utility that makes a NOTIFYME request to an upstream LDM and,
optionally, writes the data-product metadata that it receives to an
LDM logfile.
- NTP
- The Network Time Protocol. This protocol is used to obtain time
information from a time-server.
See also ntpd.
- ntpd
- The Network Time Protocol Daemon. This freely-available program
ensures that a computer's system clock is accurate, which is one of
the prerequisites for correct operation
of the LDM.
ntp.org home page.
- ntpdate
- A utility that uses the
Network Time Protocol (NTP) to either check the accuracy of the system
clock or to set the system clock.
See also ntpd.
- NULLPROC
- A message used only to flush or verify a connection.
O
- origin, data-product
- See data-product origin.
P
- parent process
- The UNIX� process that created
a child process.
Official description.
- pathological regular-expression
- An extended regular-expression (ERE) that is
longer than two characters and has a ".*" prefix.
- pathname
- A character-string that identifies a file. Pathnames may be absolute
or relative to the current working directory.
Official description.
- perl utility
- A freely-available scripting utility.
Official webpage.
- plotMetrics utility
- An
LDM
utility for plotting time-series of performance metrics.
Uses the gnuplot utility.
- PIPE action
- See pqact PIPE action.
- portmapper
- A program that maps RPC program numbers and
versions to Internet port numbers. The utility rpcbind is
a portmapper.
- pqact
- A program that reads data-products from the product-queue and disposes
of them according to user-configurable instructions in a
pqact configuration-file.
Further information.
- pqact EXEC action
- An entry in a pqact
configuration-file that tells the associated
pqact utility
to execute the associated program as a
child process when a matching
data-product is
encountered.
Configuration-file entry.
- pqact PIPE action
- An entry in a pqact
configuration-file that tells the associated
pqact utility
to pipe the data-product
data from the matching
data-product
to a program that the pqact utility starts as a
child process.
Configuration-file entry.
- pqact configuration-file
- A file that tells the associated pqact utility how to dispose of data-products.
Further information.
- pqcat
- A program that reads data-products from the product-queue and writes
them to a file or to
standard output.
Further information.
- pqcheck
- A program that checks the
writer-counter
in the
product-queue.
- pqing
- An LDM
ingester
that reads meteorological data in WMO or AFOS format from a file
or serial port.
- pqinsert
- An LDM
ingester
that inserts files.
- pqmon
- A program that prints a summary of the state of the
product-queue.
Use in monitoring the product-queue.
- pqsend
- A program that sends selected data-products from the product-queue
to an LDM using HIYA messages and then terminates.
- pqsurf
- A program that reads
data-products
that are composite WMO surface bulletins from the
product-queue,
splits them into individual bulletins, inserts them as data-products into a
separate product-queue, and
processes the individual bulletins similar to pqact.
Deprecated.
- pqutil
- A utility for interacting with the product-queue.
- process
- The address space and system resources in which a program executes
on a UNIX� system.
Official description.
Child process.
- process group
- A collection of processes that can be signaled
as a group.
Official description.
- process group leader
- The top-level process of a
process group.
Official description.
- product creation-time
- See data-product
creation-time.
- product identifier
- See data-product identifier.
- product metadata
- See data-product metadata.
- product origin
- See data-product origin.
- product-queue
- The memory-mapped disk-file used
by the LDM system to
cache data-products.
Typically, data-products
are written to the product-queue by LDM
ingesters and read from the
product-queue by upstream
LDM processes.
- product signature
- See data-product
signature.
- product size
- See data-product size.
R
- receiving LDM
- See downstream LDM.
- RECLASS
- An
RPC.
reply that specifies a new
data-product selection-criteria.
It may be sent from an
upstream LDM
to a downstream LDM
in response to a
FEEDME
request or from a
downstream LDM 5
to an
upstream LDM 5
in response to a
synchronous
LDM 5
HEREIS
or
COMINGSOON
message.
- regex utility
- A program for validating
ERE-s and for testing their efficiency.
- registry
- See LDM registry.
- regutil
- A program for managing the contents of the
LDM registry. Execute the command
man regutil for more information.
- REQUEST entry
- An entry in the LDM configuration-file,
ldmd.conf,
that causes a request to be made to an
upstream LDM for
data-products.
ldmd.conf webpage.
- residence time
- The amount of time a data-product spends in the product-queue before being
deleted to make room for more products.
- RPC
- Remote Procedure Call. A system by which a function-call on one
computer is turned into a function-invocation on another computer.
The
LDM
uses a particular implementation called ONC RPC.
ONC RPC specification.
ONC RPC tutorial
.
- rpcgen utility
- A program for generating
RPC-using C code from an input file.
- ldmd
- The main program of the LDM
system. An LDM 6 program.
Behavior.
- rtstats
- A utility that monitors the product-queue and
periodically sends summary statistics to a downstream LDM, which
is, typically, at the Unidata Program
Center. The utility is usually started by the top-level
ldmd process
due to an entry in the LDM
configuration-file,
ldmd.conf.
IDD rtstats webpages.
- runtime symbolic link
- A symbolic link in the LDM user's
home directory that references (i.e., points to) the
version-specific LDM subdirectory that is the current operational
one.
Removal of.
Creation of.
S
- scour
- A
crontab-driven utility for deleting old data-files
based on
pathname patterns and retention-times.
- sending LDM
- See upstream LDM.
- sequence number
- An unsigned, 32-bit number sequentially assigned by the
ingester that created the
data-product. Ignored by almost
everything but existing due to tradition and history.
- set-uid-root
- If a program is set-uid-root, then it is owned by user
"root" and the set-user-ID
bit of the file is enabled.
- set-user-ID
- If a program's set-user-ID bit is on, then, when executed,
the user-ID of the resulting process will be that of the owner of the
program rather than that of the user who executed the program.
- SIGCONT
- The UNIX� "continue" signal.
This signal is used by LDM data-product
ingesters and
downstream LDMs
to notify the
LDM process group
that a data-product has just been added to the product-queue.
- SIGINT
- The UNIX� interrupt signal.
The default action is to terminate a process.
Official description.
- signature, data-product
- See data-product signature.
- SIGTERM
- The UNIX� termination signal.
The default action is to terminate a process.
Official description.
- SIGUSR2
- The UNIX�, user-defined signal #2.
The canonical action taken by an LDM process upon receipt of this signal
is to "rotate" the logging-level: notice -> information -> debug ->
notice -> ...
- size, data-product
- See data-product size.
- standard error
- The diagnostic print-stream of a UNIX� process.
- standard input
- The standard input-stream of a UNIX� process.
- standard output
- The standard print-stream of a UNIX� process.
- su utility
- A non-standard but common utility for becoming another user.
- synchronous
- Describes something that
causes the initiator to wait until the thing is finished. For
example, a synchronous RPC message causes the sender
to wait for a reply.
- syslog daemon
- A configurable UNIX� daemon that disposes of log
messages sent to it by other processes.
- system group
- A group of UNIX� system users.
Users within a group often have
write access to the same files and directories.
Official description.
T
- TCP
- Transport Control Protocol. An low-level Internet protocol for
reliably transferring a stream of bytes from one computer to another.
TCP tutorial (including TCP State Transition Diagram).
Original
RFC 793.
- time utility
- A UNIX� utility for timing a simple command.
Official description.
- top utility
- A freely-available, 3rd-party UNIX� utility for
displaying system resources and utlization.
GNU distribution.
- transfer-mode
- One of two modes in which a
downstream LDM
can receive
data-products.
In the primary transfer-mode,
data-products
are received via
asynchronous
HEREIS
messages.
In the alternate transfer-mode,
data-products
are received via
synchronous
COMINGSOON
and
BLKDATA
messages.
U
- uldbutil
- A utility for listing and managing the contents of the
upstream LDM database.
Execute man uldbutil for the manual-page.
- Unidata Program Center (UPC)
- The UCAR office that coordinates
the Unidata program.
UPC homepage.
- upstream filter
- The
data-product
filter that an
upstream LDM 6 uses
to decide whether or not to send a
data-product
to a
downstream LDM 6.
The filter has two parts: 1) an "OK" filter that
data-products
must match in order to be sent; and 2)
an optional "not" filter that
data-products
must not match in order to be sent.
Such filters are specified in the
ALLOW entries
of the LDM configuration-file,
ldmd.conf.
- upstream host
- The computer on which a
upstream LDM is
running.
- upstream LDM
- Typically, a forked,
child process of ldmd that reads
data-products from the product-queue and sends
the data-products or the
data-product metadata
to a downstream LDM
as a result of a FEEDME
or NOTIFYME request.
In atypical situations, this phrase can also refer to
a pqsend process.
- upstream LDM 5
- An upstream LDM that
uses the LDM 5 protocol.
For backward compatibility,
LDM 6 is capable of acting as an
upstream LDM 5.
- upstream LDM 6
- An upstream LDM that
uses the LDM 6 protocol.
Behavior.
- upstream LDM database
- A database that contains information on all
upstream LDMs running on the local host.
The database is transitory: it exists for each LDM session only. It can
be accessed via the uldbutil utility.
- uptime utility
- A freely-available, 3rd-party UNIX� utility for
displaying system CPU utlization.
GNU distribution.
- user-account
- The account on a UNIX� system for a user of the system.
Official description.
V
- virtual residence time
- The time difference between when a data-product is created and
when it is removed from the product-queue. For each product,
it is the sum of that product's
latency plus that product's
residence time.
- vmstat utility
- A utility for displaying system metrics.
W
- writer-counter
- A field in the
product-queue
whose value is the number of times the
product-queue
has been opened for writing without an intervening close.
Further information.
Y
- yacc utility
- A UNIX� utility for generating a gramatical parser.
Official description.


Setting up git on Windows and Ubuntu linux
DevOps engineers who not only code but also take care of cofigurations know that system administration is little more painful than developing. I remember configuring an ASP.NET MVC 5 website on a Google compute engine instance running Windows server 2012 R2.
To avoid such pain, I prefer using check-lists and cheatsheets. One can refer back to them for later use.
This posts is a very short and intuitive guide for installing git
on Windows 10 and Ubuntu linux.
Yes, it’s in the direction of what I need to do later!
Btw, Ionic can be controlled from a CLI on Windows. and there are 2 options to use from:
Setting up git
(and ignoring Console2) on Windows
Download the latest git
client from http://git-scm.com/download/win
Open the downloaded setup file:
On the 3rd screen it asks whether the icons for launching points such as desktop shortcut and quick launch be provided or not.
In case of pre-configured contextualization, a warning will be shown:
As a windows user, it’s reasonable to use git
from window’s command prompt.
Decide line endings
behaviour
Setting up git
on Ubuntu linux
Use sudo apt-get install git
command and enter root password.
The system takes care of the installation after successfull authentication.
To check git
version on Ubuntu
So, now things are ready for some git-init


Intro
This is a sample API that doesn't do much. It has:
- several resources
- a mix of HTTP verbs
- up to date documentation :)
Event information
All informational resources are cacheable by proxies or on the client-side. These routes are public, and there is no rate-limiting.
Authenticated calls
You need to pass your custom API token as a request header:
X-API-Token: 76ac098ef08d61b824a


Edit a Snippet
Learn how to edit Snippets.
Paste a Snippet
Learn how to paste a Snippet using the mouse or the keyboard.
Change the behaviour of Snipper
Learn how to change the hotkeys and the paste mode.
⚠ Be aware that the Snippets are NOT stored secure!


Protective Dad Asks Daughter’s Boyfriend What His Porn History Is Like
by Emily BrownIt’s natural for fathers to be protective over their children, but one dad went the extra mile by presenting his daughter’s boyfriend with a list of questions about his personal life.
Advertisements
Aleisha Stubley, from Middlesborough, shared a photo of everything her dad wanted to know about her boyfriend Andy after he joined the family for dinner for the first time.
While most parents would probably ask politely about the partner’s family, job and interests, Aleisha’s dad dove straight in at the deep end by asking Andy some very personal questions – the most invasive of which was undoubtedly: ‘what is your history with pornography?’
Advertisements
Now, there’s a number of ways to impress your partner’s parents, but having to recount your porn history certainly isn’t one of them. Hopefully Andy gave a relatively PG answer for that one.
The list of questions was titled ‘Son in law material’ – a headline which immediately told Andy what was expected of him. Aleisha’s dad had even gone to the trouble of laminating the document, suggesting that if the new boyfriend didn’t make the cut then he’d be able to save it for the next one, or perhaps this isn’t the first time the document has been used…?
The protective father started out with a few tamer questions, asking where Andy sees himself in five years, what he sees in Aleisha and what Andy’s ‘intentions’ for his daughter were.
Things then got a bit more intense, as question four read:
Advertisements
How well can you provide for a wife and family?
I have to say, this question is a bit outdated as Andy shouldn’t be expected to provide for a wife and family – Aleisha should have no issue providing for herself, thank you very much.
Stereotypes aside, however, the question was one that probably made Andy’s life flash before his eyes. Considering this was only the first time he’d met Aleisha’s parents it would be safe to assume the couple haven’t been together too long, but the dad is practically marching them down the aisle and picking out baby names.
The love, marriage, baby in a golden carriage scenario wasn’t going to play out until the dad had some more information though, and when it came to question five he bluntly asked about Andy’s porn-watching habits.
Advertisements
I’m not sure what the appropriate answer to that would have been. How does the average person describe their history with porn? Maybe it would have worked better as a multiple choice.
The placement of question six makes me think it was meant to act as a trap, as the father asked: ‘Are there any activities that you two enjoy doing together?’
After having just put the idea of porn into Andy’s head, I suspect the dad was trying to trick him into responding to question six with something like: ‘Yes, sex’. That would have probably been enough to strip him of the son-in-law title, but hopefully the boyfriend managed to avoid falling into that hole.
The next couple of questions continued the personal trend as he asked Andy whether he had a temper and how many women he’d been with before Aleisha.
While there’s no shortage of hilarious questions on the document, I think the final one has to be my favourite.
Question nine reads:
Will you be prepared to look after my beehive when I go on holiday?
I have no doubts this is the final test of Aleisha’s dad when it comes to approving his daughter’s partners.
Not only is he trusting Andy to look after his insects, he’s trusting him to look after ones that aren’t afraid to sting. If the boyfriend can successfully make it through that ordeal, he’d obviously have proved his worth.
Alongside the picture of the questions, Aleisha wrote on Twitter:
Came out for tea with Andy and my family for the first time, my dads just gave him this lmfaoooooooooooo f*ck off.
Her tweet has since racked up thousands of likes from amused social media users, with many describing the questions as ‘brilliant.’
Aleisha later added a comment emphasising her dad’s questions weren’t serious, so hopefully Andy didn’t really have to go into detail about his porn habits!
While the document probably scared the life out of the boyfriend at first, the joke would certainly have been a good way to break the ice!
Emily Brown first began delivering important news stories aged just 13, when she launched her career with a paper round. She graduated with a BA Hons in English Language in the Media from Lancaster University, and went on to become a freelance writer and blogger. Emily contributed to The Sunday Times Travel Magazine and Student Problems before becoming a journalist at UNILAD, where she works on breaking news as well as longer form features.


{% extends "default_template.html" %} {% set active_page = "manual" %} {% block title %}Felines as a Service{% endblock %} {% block head %} {{ super() }} {% endblock %} {% block content %}
WIP, to manually perform a report submission (pics + metadata) instead of using end user program via POST requests.


---
layout: post
title: home affairs Storyboards
date: '2013-10-16T08:31:19+02:00'
categories:
- Storyboards
picture_url: https://film-Storyboards.tumblr.com/post/64204260775/home-affairs-Storyboards
---
Home Affairs Storyboard
Production: Mindtaste
Agency: European Service Network
Client: European Commission
Director: Fabrice Armand


{% extends 'bakujobs/base/base.html' %} {% block body %} {{ job.job_title }} {{ job.job_type }} {{ job.owner.location }} {{ job.created_at | timesince }} {% load static %} {{ job.owner.name }} {{ job.owner.location }} {{ job.owner.website }} {{ job.owner.phone }} {{ job.owner.email }} {{ job.description | safe }} Share Apply for job View all jobs Shortlist Job Overview Offerd Salary${{ job.min_salary }} - ${{ job.max_salary }} Gender{{ job.gender }} Career LevelExecutive {{ job.category }}{{ job.job_description }} Experience2 Years QualificationBachelor Degree Job Location 35Days 35697 Displayed 300-500 Application {% endblock %}
{{ job.owner.location }}
{{ job.created_at | timesince }}
{{ job.owner.website }}
{{ job.owner.phone }}
{{ job.owner.email }}


Do you want to know more about us?
pyramid


CloudEvents sender
Target
Event Type
Event Data Content Type
Event Data
{"hello":"world"}
Send Event
Send


Roy "Reg" Park
Bodybuilder, Businessman and Actor
The body building legacy of Reg Park:
Showing athletic interests early in life, Reg Park dedicated his teenage years to excelling in football. He had no specific interest in bodybuilding until age 16 when he met muscleman David Cohen.
During his national service he was a Physical Training Instructor in Singapore. Upon discharge from the army in 1948, he witnessed his first physique contest; the inaugural NABBA Mr. Universe contest.
It was this contest that inspired Park to compete himself.
Bodybuilding titles:
- 1946 - Mr Britain 4th
- 1949 - Mr Britain 1st
- 1950 - Best Developed Athlete in America - IFBB, Tall, 1st
- 1950 - Best Developed Athlete in America - IFBB, Overall Winner Mr Europe
- 1950 - Overall Winner Mr Universe - NABBA, Tall, 2nd
- 1951 - Mr Universe - NABBA, Tall, 1st
- 1951 - Mr Universe - NABBA, Overall Winner
- 1958 - Mr Universe - Pro - NABBA, Tall, 1st
- 1958 - Mr Universe - Pro - NABBA, Overall Winner
- 1965 - Mr Universe - Pro - NABBA, Tall, 1st
- 1965 - Mr Universe - Pro - NABBA, Overall Winner
- 1970 - Mr Universe - Pro - NABBA, Tall, 2nd
- 1971 - Mr Universe - Pro - NABBA, Tall, 3rd
- 1973 - Mr Universe - Pro - NABBA, Tall, 2nd
To find out more about Reg Park you can read article 1 and article 2.


Sharing is Caring
Referral Bonus - Any friend that refers your name during a pickup scores you an incredible bonus!
You'll get paid a 10% bonus for every item they sell.
They will make 10% more by using you as a referrer.
There is no limit to the amount of referrals!
Limited time only!
{% include "snippets/_social_drop_list.html" with destination_url=request.build_absolute_uri email_subject="These guys pick up your items and sell them for you!" message="If you use my name as a referral we will both earn 10 percent more!" %}


Ripple-js Demo
Click anywhere to start a ripple
Hold down to slowly expand the ripple
While pressing, move the pointer out of the container to cancel the ripple
You can change the looks of the ripple
You can even use a picture as background
You can add a shadow to the ripple
You can change the timing of the effect
If clicked, the element fires an 'ripple-button-click' event
Like what you see? See you on Github.
Made by Aaron Längert


ILoginService loginService
IOfficeService offService
IPersonAdminService pService
IVehicleManageService vmService


KAOHSIUNG FLOWERS SHOP
Payment
Order
Verity Of Flowers
Occations
For Him
For Her
Wedding
I'm Sorry
New
Seasonal Products
Mother's Day
Graduation
Main
Top
Chinese »
« English