Title: Marlin: Maturing the WEB framework Author: Edwig Huisman Email<EMAIL_ADDRESS>Language: C++ Platform: Windows Technology: Webserver / Webclient Level: Intermediate Description: Complete webserver and webclient with SOAP/JSON messageing framework Section Webdevelopment SubSection Web Services License: MIT
This document is about the Marlin webserver library. Previous documentation and clarification of the programming model and the reasons behind the Marlin webserver can be found on the codeproject article: Marlin: Escaping from IIS
This article is about the further developments of Marlin between versions
3.0 and 3.6. It describes the general improvements in terms of function. For a
full report of all changes and bug fixes (sure, it's software – so there's bound
to be little bugs), see the releasnotes of the last version in the source download.
In general: these versions have seen the light after exensive testing on AngularJS applications
in real life. A lot of the functionality presented here has arisen from mixed mode webservers
running SOAP microservices and JSON services in co-junction on one and the same server environment.
All upgrades and new features have been assemled here in one package.
So these are the highlights!
Handling of WebDAV HTTP calls has been added. WebDAV is used for document
management systems, source versioning systems and OData (Open-data). WebDAV
calls include the calls for 'MOVE, 'COPY' and 'SEARCH' on the general resources
(files). But they also include the calls for the properties, like:
| PROPFIND | Finding a resource by finding a property |
| PROPPATCH | Patching the property of a resource |
| MKCOL | Making a collection of resources (i.e. a directory) |
| LOCK | Locking a resource for other users |
| UNLOCK | Unlocking a resource after use |
Prior to this release, you had to add a lot of SOAP messages, and then let
the WSDL cache go on its way and generate a WSDL file from it. This ensures of
course that the WSDL is complete and correct. That’s of course very fine for a
server, but what about the client?
From this version on you can also read an
existing WSDL from a file and use that one for testing your calls. This makes it
possible to generate a WSDL with external tools, up front and engineer an
interface API solution, working your way toward the agreed interface.
Apart from the "loose" WSDL files, more data testing has been added. Apart from
integer sizes, the support of string enumerations is the big change here. You
can define which enumerations a SOAP XML field can have. You will see the
enumerations on the generated interface HTML pages in the SOAP examples, and you
can explicitly test on them.
Here is an example from the test server in the project:
POST /MarlinTest/TestInterface//MarlinWeb.acx HTTP/1.1 Host: CERT5 Content-Type: application/soap+xml; charset=utf-8 Content-Length: length <?xml version="1.0" encoding="utf-8"?> <s:Envelope xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing"> <s:Header> <a:Action s:mustUnderstand="1">http://interface.marlin.org/testing/MarlinSecond</a:Action> </s:Header> <s:Body> <MarlinSecond xmlns="http://interface.marlin.org/testing/"> <!--Translation has values: Deutsch (='Die Deutsche sprache') / English (='The English language') / Espagnol (='El lingua franca') / Français (='La plus belle langue')--> <Translation>string</Translation> </MarlinSecond> </s:Body> </s:Envelope>Before the node that carries the enumeration (
Suppose you have a server with a SOAP interface. But there comes a new
paradigm on the browser: javascript. And not just one script, but a barrage of
javascript interfaces. Sometimes is seems whole the world is jumping into that.
AngularJS to mention as one of the new hypes in these interfaces. But wait:
suddenly SOAP is not cool anymore. Everyone wants JSON interfaces. Now how to
cope? Writing your server over again. No, roundtrip JSON / SOAP / JSON
translation is one of the quicker solutions.
Suppose a web javascript interface want's to call my SOAP service, it can do so by calling the service
with all the parameters as HTTP parameters. So lets suppose I have a SOAP
service with a "SiteHandlerJson2Soap" in place, then the following call:
http://server:7500/Interface/MyCall?one=1&two=2&three=3
Will be automatically translated into the following SOAP call:
<xml> <Envelope> <header/> <Body> <MyCall> <one>1</one> <two>2</two> <three>3</three> </MyCall> </Body> </Envelope>
Note here the placement of the name of the call ("MyCall") as last part of
the resource of the URI, and as the main node of the message. All URL parameters
now translate to "ordinary" SOAP parameters in the soap call. This message then
can be handled in the "old fashioned SOAP way", as the server used to, and the
server will generate a response message in SOAP.
<xml> <Envelope> <header/> <Body> <MyCallResponse> <Counting>true</Counting> <Sum>6</Sum> </MyCallResponse> </Body> </Envelope>
The response message then gets translated back to JSON by the SiteHandler
"Json2Soap" to a JSON answer for the web interface.
{ "MyCallResponse" : { "Counting": true ,"Sum": 6 } }
And this is the JSON message that a web interface can easily digest.
Please note that the conversion from JSON into SOAP and SOAP into
JSON is done in the server just by constructing a SOAP message from a JSON
message and vice versa. So that you can do the same thing also in your
code.
Also be warned that not all JSON messages translate into SOAP messages,
as the JSON notion of arrays does not exist within the SOAP realm!
Version 3.1.1 Chaining of SiteHandlers
It's quite possible that you have different ways of handeling a HTTP request for a site. For instance a
handler for authorized files and directories, and one without authorization in
special cases.
To easily program this, you can now set multiple site handlers
on one and the same site, just by calling "AddHandler" more than once for the
same HTTPCommand ("get", "put" and so on).
The handlers get chained in order of appearance, and if a handler cannot handle the HTTP request by returning
"false", the next handler in the chain gets called automatically, until you run
out of handlers.
Version 3.3.2 Mandatory namespaces in SOAP
In order for .NET 4.0.0 and higher to consume our WSDL's it proved to be necessary to add mandatory
namespaces to all SOAP messages and in the WSDL. After this action had been
taken Visual Studio could create a .NET SOAP client interface directly from our
WSDLs.
Multipart buffer posts as the HTML <form> element posts multiple
sections of field data and files in one (1) HTTP post to the server. These forms
of posts are also in widespread use in the handling of OData (open-data)
requests. In order for Marlin to handle OData, it was necessary to fully support
serverside and clientside multipart buffer requests.
The requests are gathered in an object called "MultiPartBuffer". You can add parts to it with a
call to "AddPart" and add messages or files to the multipart buffer. After that
you can send them through the HTTPClient and receive them in a site handler
called "SiteHandler-FormData". This site handler has a different set of extra
handlers. These handlers are:
| PreHandleBuffer | Gets called once for each HTTP call. |
| HandleData | Gets called for each data part in the call (field etc) |
| HandleFile | Gets called for each file part in the call |
| PostHandleBuffer | Gets called after all data and file parts have been handled |
Client certificates
In order to build secure websites that can only be approached by known clients, a general provision called "client certificates"
has been invented. You can specify in the HTTPClient which client certificate
from your certificate store to use in a call to a certain website. The Marlin
server can request for this client certificate and find it in the local
certificate store on the server machine and ascertain itself that it's the
correct client certificate.
In order to build certificates for the development stage, a new directory "Certificate" has been created in this
project. Here you can make root CA certificates, SSL/TLS certificates and client
certificates for yourself and for the development fazes of your software.
Beware. Here there be dragons! Read the extra documentation on how to do this
exactly.
Windows 10 hangs
"O everything is better in the next version of MS-Windows". We keep hearing that from the commercial department of Microsoft.
But not in the HTTP Server API 2.0 ! For the first time the contents of the
incoming request in "HttpReceiveHttpRequest" has been changed since MS-Windows
Vista. And undocumented as well. Yes: your heard me: no extra documentation on
MSDN.
There where now undocumented extra requestInfo records in the request.
It was only after installing the Windows 10 SDK, and after reading through all
the headers from that SDK that the veil was lifted. The extra requestInfo
records are carrying the SSL specifics of an incoming "https" call. And as
Marlin was not prepared for those, it did hang in reading the request info
records. Auch.
After the SDK entries were found, Marlin now logs the extra
info in the logging file (if turned on of course). So there is an extra
LogSSLInformation method in the server that logs the current SSL/TLS protocol
and the cypher strengths of the current connection. But only if run on
MS-Windows 10 and Server 2016. Mind you!
Rewrite of the web.config editor.
The web.config editor in the
HTTPManager program had become quite bloted, long and less clear as it once was.
So I decided to give it a rewrite and create tabs per function. So now you have
tabs for the server configuration, the client configuration, authentication, web
services and logging. This is what it looks like:
Version 3.4.2 Full UTF-8 / UTF-16 support
Marlin already had UTF-8 and UTF-16 support. So what do you mean by 'full'? Well, the support of UTF-8
was only for the first 3 code pages and the West-European diacritics. Letters
such as ä, é, ï and so on. In essence the standard vowels with all the accents
on them. But as time progressed, more became necessary. A € euro sign for
instance is not on the first three pages of UTF-8, and the UTF-8 was not
supported in the URL parameters as well. So I fixed all that. It was ugly,
horribly and also behaves different in debug and in release mode.
UTF-8 and UTF-16 are now fully supported on the full URL, the URL parameters, and the URL
anchor as well. And also in the SOAPMessage and the JSONMessage and all its
parts. So if I messed up somewhere, let me know and I'll dive right back in.
(and take a deep sigh!)
Version 3.4.3 Extra Tracing in the HTTPClient
Tracing of the traffic in the logfile of the HTTPClient has been rather basic. So now it is
possible to "SetTraceData" and "SetTraceRequest". The first call traces incoming
answers and the second traces the state of the HTTP request prior and after the
'real' request and response calls. So that we can see what state the WinHTTP
engine of Microsoft is in. Please be prepared for rather big logfiles!
Version 3.4.4 GZIP compression / decompression
Compression and decompression of content was a missing feature in the Marlin server. I added it
in this version. Marlin is now using the original ZLIB of Jean-loup Gailly and
Mark Adler for RFC 1950/1951/1952 'gzip' compression and decompression. Yes I
known: there are a lot more compression/decompression methods to be had in HTTP
traffic, but 'gzip' is just the basic one that's most widespread and since 2011
that everybody is using. It's been tested with all major browsers and web
stacks, including Google Chrome, Firefox, the IE's and Edge.
Version 3.5 Performance release
This was a serious performance release. Firstly: the incoming HTTP request is now only recorded in the central
server and forwarded through the HTTPMessage and the threadpool to a different
thread that's handling the reading of the HTTP body of the message within the
HTTPSite. This clears out the responsibility of reading large file bodies from
the central server to a threadpool thread and a site. Thus clearing the central
processing thread of the server to be on the business again of receiving
incoming calls.
One major parameter of course is the number of available threads, and the length
of the input waiting queue of HTTP calls to be processed. So keep in mind that
after this version you just might want to fiddle
with the following web.config parameters:
| - <Server><QueueLength> | Multiple of 64 |
| - <Server><Stacksize> | Stack space of the threads |
| - <Server><MaxThreads> | Should be less than |
Secondly: I have rewritten the memory model of the complete server.
All malloc / calloc and free() calls are now gone from the code.
Everything has been allocated with 'new' or with 'new []' methods and will be
freed from the heap with 'delete' or 'delete []'. If you are using FileBuffers
straight from the Marlin library, or using the WideString* conversion calls,
retrace your steps in the code and refactor your memory modelling
likewise!
Thirdly, and as a consequence of the rewriting of the memory model,
I fixed a few serious stack bugs in the sizing of the XML/SOAP messages and the
reading of the HTTP buffers, while rewriting the memory model of the server. The
server is now less prone to buffer overruns and memory-overflows. All thanks to
the sonarsource company in Switzerland and their SonarQube analysis
(See: www.sonarsource.com)
Version 3.6 Rewriting the get handler
During the last versions the get handlers has gotten several re-writes. (But we only mention it here) The
reason behind these rewrites lies in the way the popular AngularJS framework
handles missing files and other internal errors. These are the main rules:
- All missing full resource names ending in '/' fall back to 'index.html' in that directory;
- All other missing resources do fall back to the 'index.html' on the main page (!!)
- If the site is an application the fallback should do a fallback to the login page.
Another JSON feature is the handling of arrays. The roundtrip conversion from SOAP XML to JSON and back to JSON, did not handle
these. Especially if the handler returns a SOAP message where multiple nodes
with the same name are returned, it's handy to have these converted to a JSON
array, lest the nodes will overwrite themselves when importing the message into
a javascript object. See this example from the test client below:
<xml> <Envelope> <header/> <Body> <Applications> <Datasources> <Datasource>develop</Datasource> <Datasource>test</Datasource> <Datasource>accept</Datasource> <Datasource>production</Datasource> <Datasources> <Applications> <Commercial> <Application>Financials</Application> <Application>Maintenance</Application> <Application>Rentals</Application> </Commercial> <Default>Rentals</Default> </Applications> </Body> </Envelope>
Two of the nodes here (Datasources & Commercial) contain subnodes of the
same name. So they get converted to a JSON array with only the contents of those
nodes.
{ "Applications":{ "Datasources":{ "Datasource": [ "develop" ,"test" ,"accept" ,"production" ] } ,"Commercial":{ "Application": [ "Financials" ,"Maintainance" ,"Rentals" ] } ,"Default":"Rentals" } }
One of the main reasons the get handlers got re-written so often, was the
need to have a websites content somewhere on disk outside of the server's root
directory. To accommodate this better, the virtual directory capability was
added. This works by prepending the root directory of the site with the
"virtual://" protocal. For instance:
- The server root directory is: "C:\WWW\Server\"
- The site root directory was: "C:\WWW\Server\MySite\"
- The site root directory now is: "C:\Appication\Version12\MySite\"
You can specify this with: site->SetWebroot("virtual://C:\Application\Version12\MySite");
Rewriting of a site handler was also triggered by the need to initialize things and set
things up for a handler. That's why a 'OnStartSite()' method was added to the
SiteHandlers and the SiteFilters. The starting of the site in
HTTPSite::StartSite() calls all the 'OnStartSite' methods of all registered
filters and handlers. And because the 'OnStartSite' method is a virtual method,
you can have your own localized initialization for your handler.
A last feature in this release are the handling of media types for the get
handler.
To support both Microsoft Office and OpenOffice (or LibreOffice), a
great number of content types have been added to the marlin server. These media
types are quite hard to come by. In fact you have to gather them from all over
the internet. This version adds an extensive list of media types to the default
'get handler'.
During the development the testset was re-written a lot to accommodate for all the new features. But also the first head-up check of the testset was altered. In the new start screen you can see in 'one go' that the startup of the server went well. That is, if you look at the column of "OK" statuses after all the tests. And you can pick the interface pages from the start screen and paste them in your favorite browser. Or pick the WSDL reference and paste it into the .NET client builder in Visual Studio.
Okay, that was a lot! I'm quite confident that version 3.7 is a stable release
to be in service for a long time tot come. But then again: it's still (just) software :-)
Who will know what's next? I've even been spammed to deploy the framework in IIS.
Defeating my original goal of escaping from it. Should be worth a serious performance test!
And who know's what's around the next bend.....
For those who did not read the first article: This is a picture of a white marlin, a common type of marlin in the Atlantic ocean. This is about the fastest fish in that ocean and it's a very secretive fish. If you manage to catch one on a fishing line, it will struggle on "unseen" in the deep. Like a fast webserver would :-)


Unless you’re here volunteering or en route to Kidepo Valley National Park, there’s no real reason to visit Gulu. However, it's quite a pleasant stop, with friendly locals, a relaxed feel and nice spots to hang out. The largest town in northern Uganda, it was one of the hardest hit during the Lord’s Resistance Army (LRA) conflict. It’s a town in transition and, in a sure sign of optimism, restaurants are busy and people are arriving from elsewhere in the country hoping to cash in on the coming boom. And there’s no shortage of wazungus (foreigners) about, with so many NGOs in town.
Thanks to an intoxicating blend of adrenaline adventures and superb scenery, eastern Uganda is a must on any East African journey. Here the mighty Nile begins its epic journey north. White-water rafting the Nile River is the main draw for adventurers seeking a wild ride, but there's also the subtler, sweatier pleasure of trekking through Mt Elgon National Park or relaxing with a visit to stunning Sipi Falls. The regional capital of Jinja offers a charming and friendly base on the shores of Lake Victoria where travellers can spend a few days or more enjoying the chilled atmosphere and top-notch outdoor activities.
Ready to start your next project with us? Give us a call or send us an email and we will get back to you as soon as possible!


Forum: CompilerIssue: COMPILER-VERBOSITY
References: CLtL p. 438-329; 426
issue COMPILER-DIAGNOSTICS
Category: ENHANCEMENT
Edit History: V1, 25 Oct 1988, Sandra Loosemore
V2, 12 Dec 1988, Dan L. Pierson (add USE-CONDITIONS)
V3, 15 Dec 1988, Dan L. Pierson (expand on conditions)
V4, 21 Dec 1988, Dan L. Pierson (reword and clarify)
V5, 06 Jan 1989, Sandra Loosemore (update discussion)
V6, 26 Jan 1989, Sandra Loosemore (remove USE-CONDITIONS)
Status: Ready for release
Problem Description:
Implementations vary widely in the amount of information that is printed
out by COMPILE-FILE. In some situations, it would be useful to control
how much information is printed.
Proposal COMPILER-VERBOSITY:LIKE-LOAD:
Introduce special variables, *COMPILE-VERBOSE* and *COMPILE-PRINT*,
with implementation-dependent initial values.
Add :VERBOSE and :PRINT keyword arguments to the function
COMPILE-FILE, analogous to those for the function LOAD.
The :VERBOSE argument (which defaults to the value of
*COMPILE-VERBOSE*), if true, permits COMPILE-FILE to print a message
in the form of a comment to *STANDARD-OUTPUT* indicating what file is
being compiled and other useful information.
The :PRINT argument (which defaults to the value of *COMPILE-PRINT*),
if true, causes information about top-level forms in the file being
compiled to be printed to *STANDARD-OUTPUT*. Exactly what is printed
will vary from implementation to implementation, but nevertheless some
information will be printed.
Introduce a special variable *LOAD-PRINT*, which has an initial value of
NIL. State that the default value of the :PRINT argument to LOAD is
*LOAD-PRINT* (rather than NIL).
Rationale:
This proposal makes COMPILE-FILE behave like LOAD. There is already
some precedent for doing this (for example, issue COMPILE-FILE-PACKAGE,
which makes COMPILE-FILE as well as LOAD rebind *PACKAGE*).
Adding the *LOAD-PRINT* variable allows the printing of messages by
LOAD to be controlled either on a global or a per-call basis.
Current Practice:
COMPILE-FILE prints out progress messages in nearly all
implementations.
Lucid provides a :MESSAGES keyword argument to COMPILE-FILE, which can
either be a stream to send messages to, or NIL to suppress messages.
The default value is T, which sends messages to "the standard terminal
device".
On the TI Explorer, COMPILE-FILE displays the name of the function
being compiled when the option :VERBOSE T is given or special variable
COMPILER:COMPILER-VERBOSE is true. (In other words, they use :VERBOSE
to mean what this proposal says to use :PRINT for.)
Symbolics Cloe already has a *LOAD-PRINT* variable.
Cost to implementors:
This is an incompatible change for some implementations. While the
changes required should be conceptually simple, their implementation
may involve a significant amount of grunt work. At least two
implementations already provide some similar mechanism for suppressing
messages.
Cost to users:
Some (non-portable) user code may break in implementations where this
is an incompatible change.
No user code should be broken by the addition of the *LOAD-PRINT*
variable, since the default behavior for the :PRINT keyword to LOAD
is unchanged.
Benefits:
Users are given a portable way to control how much information is printed
by COMPILE-FILE.
Discussion:
This issue addresses an extension to the language. If this proposal
is not accepted, the standard will simply continue not to say anything
about whether COMPILE-FILE can print progress messages, or what stream
such messages are directed to.


11/14/14
Source: stackoverflow.com
We have worked somewhat extensively on Ruby classes these last few weeks. We learned that classes are basically a template for creating objects. It includes the methods and attributes that an object is formed in. Classes themselves can also inherit these methods and attributes from other classes that we can super or parent classes. Although a class can only directly inherit from one superclass at a time, that superclass may also inherit from another class.
Meanwhile a few weeks ago we were introduced to the concept of an enumerable. We began using them, finding that they are incredibly useful in simplifying code and reducing the need to explicitly state loops. We researched into what the enumerable is where we found out that an enumerable is a module. We may have mostly left it at that, perhaps only going as far as stating that the module is a mix-in for classes so they can use it. But what does a module actually look like?
The table below basically breaks it all down. The module is a collection of methods, constants, and classes. Although you can use the methods inside as instance methods for the class it is in, it is discouraged not to since it may make reading through and debugging the class very difficult to pin down where the method is defined from. So, because classes can only inherit one from one class a time, modules sort of act as this fine tuning for you to inject certain methods into different classes that you want to share similar behaviors. You include a module into a class by writing the following (replace ModuleName with the actual name) inside the top level of the class
include ModuleName


Preparation time
less than 30 mins
Cooking time
10 to 30 mins
Serves
Serves 4
James Martin makes a quicker, easier version of the classic French recipe, ratatouille.
For the ratatouille, heat a frying pan over a medium to high heat, add the olive oil and, when it’s hot, add the onion and peppers. Fry for 2-3 minutes, or until softened.
Add the garlic, aubergine and courgettes and fry for a further 2-3 minutes, stirring occasionally.
Add the tomatoes and continue to cook for a further minute.
Sprinkle over the basil leaves and season, to taste, with salt and freshly ground black pepper.
For the chive beurre blanc, bring the finely sliced shallots, white wine vinegar and white wine to the boil in a non-reactive saucepan. Reduce the heat until the mixture is simmering, then continue to simmer until only two tablespoons of liquid are left in the pan.
Add the cold water and continue to simmer until the volume of liquid remaining in the pan is about one tablespoon.
Reduce the heat to low, then gradually whisk in the butter, 25g/1oz at a time, waiting until all of the butter has melted and incorporated into the mixture before adding more.
Once all of the butter has been added to the reduction, remove the mixture from the heat and strain it into a clean saucepan. Season, to taste, with salt and freshly ground black pepper, then stir in the chives. Set aside at room temperature until needed (do not refrigerate as the sauce will separate).
For the turbot, preheat the oven to 200C/400F/Gas 6. Season the turbot fillets with salt and freshly ground black pepper.
Heat a frying pan over a medium heat, add the butter and, when it’s foaming, add the turbot fillets, skin-side down (in batches if necessary). Fry for 1-2 minutes, or until the skin is crisp and golden-brown, then turn the fish over and add the wine.
Cover the fish with the cartouche and transfer the pan to the oven for 3-4 minutes, or until just cooked through (the fish is cooked through when the flesh is opaque).
Remove the fish from the pan and set aside to rest for 1-2 minutes.
To serve, spoon a quarter of the ratatouille into the centre of each of four serving plates. Place one fillet of turbot on top of each portion. Spoon over the chive beurre blanc.
You need JavaScript to view this clip.
Knife skills: how to finely chop