The tutorial_pba_0.cpp sample program uses a boost::archive::portable_binary_oarchive object attached to a standard output file stream to store a couple of variables of primitive types (bool, char, integer numbers, floating numbers) and even a std::string.
/** tutorial_pba_0.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This quick start example shows how to store some variables * of basic types (bool, integer, floating point numbers, STL string) * using the portable binary archive format associated to a * standard output file stream. * */ #include <string> #include <fstream> #include <boost/cstdint.hpp> #include <boost/archive/portable_binary_oarchive.hpp> int main (void) { // The name for the example data file : std::string filename = "pba_0.data"; // Some variables of various primitive types : bool b = true; char c = 'B'; uint32_t answer = 42; float computing_time = 7.5e6; double e = 2.71828182845905; std::string slogan = "DON'T PANIC"; // Open an output file stream in binary mode : std::ofstream fout (filename.c_str (), std::ios_base::binary); { // Create an output portable binary archive attached to the output file : boost::archive::portable_binary_oarchive opba (fout); // Store (serializing) variables : opba & b & c & answer & computing_time & e & slogan; } return 0; } // end of tutorial_pba_0.cpp
The compiled executable creates the pba_0.data file which contains the following bytes:
127 1 9 1 84 1 66 1 42 4 192 225 228 74 8 116 87 20 139 10 191 5 64 1 11 68 79 78 39 84 32 80 65 78 73 67This format is explained in details below.
Note:
The tutorial_pba_1.cpp sample program uses a boost::archive::portable_binary_iarchive object attached to a standard input file stream in order to load the data previously stored by the tutorial_pba_0.cpp program in the pba_0.data file.
/** tutorial_pba_1.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization package. * * This quick start example shows how to load some variables * of basic types (bool, integer, floating point numbers, STL string) * using the portable binary archive format associated to a * standard input file stream. * */ #include <iostream> #include <string> #include <fstream> #include <boost/cstdint.hpp> #include <boost/archive/portable_binary_iarchive.hpp> int main (void) { using namespace std; // The name for the example data file : string filename = "pba_0.data"; // Some variables of various types : bool b; char c; uint32_t answer; float computing_time; double e; string slogan; // Open an input file stream in binary mode : ifstream fin (filename.c_str (), ios_base::binary); { // Create an input portable binary archive attached to the input file : boost::archive::portable_binary_iarchive ipba (fin); // Loading (de-serializing) variables using the same // order than for serialization (see tutorial_pba_0.cpp) : ipba & b & c & answer & computing_time & e & slogan; } cout.precision (15); cout << "Variable 'b' is : " << b << " " << "(bool)" << endl; cout << "Variable 'c' is : '" << c << "' " << " " << "(char)" << endl; cout << "Variable 'answer' is : " << answer << " " << "(unsigned 32-bit integer)" << endl; cout << "Variable 'computing_time' is : " << computing_time << " " << "(single precision 32-bit float)" << endl; cout << "Variable 'e' is : " << e << " " << "(double precision 64-bit float)" << endl; cout << "Variable 'slogan' is : \"" << slogan << "\" " << "(std::string)" << endl; return 0; } // end of tutorial_pba_1.cpp
The executable reads the pba_0.data file and deserializes its contents in the same order it has been stored. It then prints the restored values of the variables:
Variable 'b' is : 1 (bool) Variable 'c' is : 'B' (char) Variable 'answer' is : 42 (unsigned 32-bit integer) Variable 'computing_time' is : 7500000 (single precision 32-bit float) Variable 'e' is : 2.71828182845905 (double precision 64-bit float) Variable 'slogan' is : "DON'T PANIC" (std::string)
This section aims to give some details about the binary format of portable binary archives (PBA). We will analyse the byte contents of the sample binary archive pba_0.data file created by the tutorial_pba_0.cpp program (see the previous section).
Like any other archive format within Boost/Serialization, a PBA starts with a header (this is the default behaviour but it is possible to deactivate the use of this header using a special flag at construction, see this example). This header is made of two informations :
127
The PBA encoding of integer numbers uses the following scheme: <size> <content>, where first the size stores the minimal number of non zero bytes needed to store the binary representation of the integer value; then the bytes corresponding to the content are stored starting from the less significant ones (see also this example). For the library version number we have here:
1 9where 1 is the number of byte needed to store the value 9 which comes with the Serialization library for Boost version 1.47.0. Here, the 9 value being less than 256, one unique byte is enough to store this number.
Now we are done with the header, let's have a look on the serialized data !
1 84
1 66
1 42This scheme results in saving 2 bytes compared to the size of the transient value.
| 01001010 | 11100100 | 11100001 | 11000000 |
| 74 | 228 | 225 | 192 |
4 192 225 228 74
| 01000000 | 00000101 | 11011111 | 00001010 | 10001011 | 00010100 | 01010111 | 01110100 |
| 64 | 5 | 191 | 10 | 139 | 20 | 87 | 116 |
8 116 87 20 139 10 191 5 64
1 11
| D | O | N | ' | T | P | A | N | I | C | |
| 68 | 79 | 78 | 39 | 84 | 32 | 80 | 65 | 78 | 73 | 67 |
68 79 78 39 84 32 80 65 78 73 67
Now the contents of the pba_0.data file can be fully understood :
127 1 9 1 84 1 66 1 42 4 192 225 228 74 8 116 87 20 139 10 191 5 64 1 11 68 79 78 39 84 32 80 65 78 73 67More details about the format (non finite floating point values, negative integer numbers) will be given in the sample codes below.
The PBA has been designed in the aims to handle single and double precision floating point numbers, including non-finite and special values:
The tutorial_pba_2.cpp sample program illustrates the use of such special cases while serializing single precision floating point numbers:
/** tutorial_pba_2.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This sample program shows how to use a portable binary archive * to store/load floating point numbers including non-finite and * special (denormalized) values. * */ #include <string> #include <fstream> #include <limits> #include <boost/archive/portable_binary_oarchive.hpp> #include <boost/archive/portable_binary_iarchive.hpp> int main (void) { using namespace std; // The name for the example data file : string filename = "pba_2.data"; { // A normal single precision floating point number : float pi = 3.14159265; // Single precision zeroed floating point number : float zero = 0.0; // A denormalized single precision floating point number : float tiny = 1.e-40; // A single precision floating point number with `+Infinity' value : float plus_infinity = numeric_limits<float>::infinity (); // A single precision floating point number with `-Infinity' value : float minus_infinity = -numeric_limits<float>::infinity (); // A single precision `Not-a-Number' (NaN): float nan = numeric_limits<float>::quiet_NaN (); // Open an output file stream in binary mode : ofstream fout (filename.c_str (), ios_base::binary); { // Create an output portable binary archive attached to the output file : boost::archive::portable_binary_oarchive opba (fout); // Store (serialize) variables : opba & pi & zero & tiny & plus_infinity & minus_infinity & nan; } } { // Single precision floating point numbers to be loaded : float x[6]; // Open an input file stream in binary mode : ifstream fin (filename.c_str (), ios_base::binary); { // Create an input portable binary archive attached to the input file : boost::archive::portable_binary_iarchive ipba (fin); // Load (de-serialize) variables using the same // order than for serialization : for (int i = 0; i < 6; ++i) { ipba & x[i]; } } // Print : for (int i = 0; i < 6; ++i) { cout.precision (8); cout << "Loaded x[" << i << "] = " << x[i]; switch (fp::fpclassify(x[i])) { case FP_NAN: cout << " (NaN)"; break; case FP_INFINITE: cout << " (infinite)"; break; case FP_SUBNORMAL: cout << " (denormalized)"; break; case FP_NORMAL: cout << " (normalized)"; break; } cout << endl; } } return 0; } // end of tutorial_pba_2.cpp
The pba_2.data output data file thus contains the following bytes:
127 1 9 4 219 15 73 64 0 3 194 22 1 4 0 0 128 127 4 0 0 128 255 4 255 255 255 127where:
| 64 | 73 | 15 | 219 |
| 01000000 | 01001001 | 00001111 | 11011011 |
| 1 | 22 | 194 | |
| 00000000 | 00000001 | 00010110 | 11000010 |
| 127 | 128 | 0 | 0 |
| 01111111 | 10000000 | 00000000 | 00000000 |
| 255 | 128 | 0 | 0 |
| 11111111 | 10000000 | 00000000 | 00000000 |
| 127 | 255 | 255 | 255 |
| 01111111 | 11111111 | 11111111 | 11111111 |
One can ask a PBA to reject non-finite values. This is done by passing the boost::archive::no_infnan flag to the constructor of the output archive. Note that in this case, denormalized values are still accepted, but infinite and NaNs aren't.
The tutorial_pba_3.cpp sample program that illustrates this special case:
/** tutorial_pba_3.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This sample program shows how to use a portable binary archive * and prevent the serialization of non-finite floating numbers. * */ #include <string> #include <fstream> #include <limits> #include <boost/archive/portable_binary_oarchive.hpp> int main (void) { using namespace std; // The name for the example data file : string filename = "pba_3.data"; try { // An array of single precision floating numbers: float x[5]; x[0] = 3.14159; // Pi x[1] = 6.022e22; // Avogadro constant x[2] = 1.6e-19; // Electron charge magnitude x[3] = 1.e-40; // A tiny (denormalized) value x[4] = numeric_limits<float>::infinity (); // This will fail while serializing... // Open an output file stream in binary mode : ofstream fout (filename.c_str (), ios_base::binary); { // Create an output portable binary archive attached to the output file, // using the special 'boost::archive::no_infnan' flag : boost::archive::portable_binary_oarchive opba (fout, boost::archive::no_infnan); // Store (serialize) variables : for (int i = 0; i < 5; ++i) { clog << "Serializing value : " << x[i] << " ... "; opba & x[i]; clog << "Ok !" << endl; } } } catch (exception & x) { cerr << "ERROR: " << x.what () << endl; return 1; } return 0; } // end of tutorial_pba_3.cpp
We can check that the PBA now throws an exception as soon as it encounters a non finite floating point value during the serialization process:
Serializing value : 3.14159 ... Ok ! Serializing value : 6.022e+22 ... Ok ! Serializing value : 1.6e-19 ... Ok ! Serializing value : 9.99995e-41 ... Ok ! Serializing value : inf ... ERROR: serialization of illegal floating point value: inf
The PBA obviously handles integer numbers. Unfortunately, C/C++ does not garantee the portable size of its primitive integer types (short, int, long... and their unsigned versions). It depends on the architecture (32-bit/64-bit) and the compiler.
The Boost library addresses this issue through a collection of typedefs for integer types of common sizes. This technique is supposed to allow the manipulation of integer variables in a portable way, typically with text or XML archives. So, we are generally encouraged to use the boost/cstdint.hpp header file and the typedefs defined therein.
Due to its encoding scheme of integer numbers, the PBA does not strictly need such technique to ensure a correct behaviour while (de)serializing integer numbers. This is because the little endian encoding approach allows to only store the non-zero bytes. It is thus possible to serialize a value using one integer type (short int) and then deserialize it using another integer type (long long).
However, for a strict and safe portable behaviour of PBA, we recommend that, in most cases, the user should systematically use such typedefs for all serializable integer values. This applies particularly for member attributes in structs and classes and should allows the transparent switching to another kind of archive (text, XML) thanks to the serialize template method.
The tutorial_pba_4.cpp sample program illustrates the serialization/deserialization of 8-bit, 16-bit, 32-bit and 64-bit integer numbers:
/** tutorial_pba_4.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This sample program shows how to use a portable binary archive * to store/load integer numbers of various sizes using the Boost * portable integer typedefs. * */ #include <string> #include <fstream> #include <boost/cstdint.hpp> #include <boost/archive/portable_binary_oarchive.hpp> #include <boost/archive/portable_binary_iarchive.hpp> int main (void) { using namespace std; // The name for the example data file : string filename = "pba_4.data"; { // Some integer numbers : bool t = true; char c = 'c'; unsigned char u = 'u'; int8_t b = -3; // char uint8_t B = +6; // unsigned char int16_t s = -16; uint16_t S = +32; int32_t l = -128; uint32_t L = +127; int64_t ll = -1024; uint64_t LL = +2048; // Open an output file stream in binary mode : ofstream fout (filename.c_str (), ios_base::binary); { // Create an output portable binary archive attached to the output file : boost::archive::portable_binary_oarchive opba (fout); // Store (serialize) variables : opba & t & c & u & b & B & s & S & l & L & ll & LL; } } { // Single precision floating numbers to be loaded : // Some integer numbers : bool t; char c; unsigned char u; int8_t b; uint8_t B; int16_t s; uint16_t S; int32_t l; uint32_t L; int64_t ll; uint64_t LL; // Open an input file stream in binary mode : ifstream fin (filename.c_str (), ios_base::binary); { // Create an input portable binary archive attached to the input file : boost::archive::portable_binary_iarchive ipba (fin); // Load (de-serialize) variables using the same // order than for serialization : ipba & t & c & u & b & B & s & S & l & L & ll & LL; } clog << "t = " << t << " (bool)" << endl; clog << "c = '" << c << "' (char)" << endl; clog << "u = '" << u << "' (unsigned char)" << endl; clog << "b = " << (int) b << " (int8_t)" << endl; clog << "B = " << (int) B << " (uint8_t)" << endl; clog << "s = " << s << " (int16_t)" << endl; clog << "S = " << S << " (uint16_t)" << endl; clog << "l = " << l << " (int32_t)" << endl; clog << "L = " << L << " (uint32_t)" << endl; clog << "ll = " << ll << " (int64_t)" << endl; clog << "LL = " << LL << " (uint64_t)" << endl; } return 0; } // end of tutorial_pba_4.cpp
The resulting PBA file is:
127 1 9 1 84 1 99 1 117 255 253 1 6 255 240 1 32 255 128 1 127 254 0 252 2 0 8where:
Note that this coding scheme optimizes the number of streamed bytes. Particularly, it discards the leading zero-ed bytes (MSB) of the binary encoding of any integer value in order to save storage. Also we recall that the exact 0 value (zero or false for a boolean data) is always encoded using a unique 0 byte (zero optimization). Note this approach is also used for floating point numbers.
In some case, we don't want to serialize some data in a file (std::ofstream), but we simply plan to stream it in a memory buffer.
The tutorial_pba_5.cpp sample program makes use of a memory buffer implemented with a STL vector of characters. The PBA is associated to this buffer thanks to a special streaming interface mechanism provided by the Boost/Iostreams library. With such technique one can stream serializable data in some memory buffer in place of a file :
/** tutorial_pba_5.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This sample program shows how to use a portable binary archive * to store/load data in a memory buffer. * */ #include <string> #include <vector> #include <boost/iostreams/stream.hpp> #include <boost/iostreams/device/back_inserter.hpp> #include <boost/iostreams/device/array.hpp> #include <boost/cstdint.hpp> #include <boost/archive/portable_binary_oarchive.hpp> #include <boost/archive/portable_binary_iarchive.hpp> int main (void) { using namespace std; // The memory buffer is implemented using a STL vector : typedef std::vector<char> buffer_type; buffer_type buffer; { // Some data to be stored : bool t = true; char c = 'c'; int16_t s = +16; int32_t l = -128; int64_t ll = +10000000000; float pi = 3.14159; double nan = numeric_limits<double>::quiet_NaN (); string hello = "World !"; buffer.reserve (1024); // pre-allocate some memory // The output stream interface to the buffer : boost::iostreams::stream<boost::iostreams::back_insert_device<buffer_type> > output_stream (buffer); { // Create an output portable binary archive attached to the output file : boost::archive::portable_binary_oarchive opba (output_stream); // Store (serialize) variables : opba & t & c & s & l & ll & pi & nan & hello; } } clog << "Buffer content is " << buffer.size () << " bytes : " << endl << " "; for (int i = 0; i < buffer.size (); ++i) { clog << (int) ((unsigned char) buffer[i]) << ' '; if ((i + 1) % 20 == 0) clog << endl << " "; } clog << endl; { // Some data to be loaded : bool t; char c; int16_t s; int32_t l; int64_t ll; float pi; double nan; string hello; // The input stream interface to the buffer : boost::iostreams::stream<boost::iostreams::array_source> input_stream (&buffer[0], buffer.size ()); { // Create an input portable binary archive attached to the input file : boost::archive::portable_binary_iarchive ipba (input_stream); // Load (de-serialize) variables : ipba & t & c & s & l & ll & pi & nan & hello; } clog << "Loaded values from the buffer are: " << endl; clog << " t = " << t << " (bool)" << endl; clog << " c = '" << c << "' (char)" << endl; clog << " s = " << s << " (int16_t)" << endl; clog << " l = " << l << " (int32_t)" << endl; clog << " ll = " << ll << " (int64_t)" << endl; clog << " pi = " << pi << " (float)" << endl; clog << " nan = " << nan << " (double)" << endl; clog << " hello = \"" << hello << "\" (std::string)" << endl; } return 0; } // end of tutorial_pba_5.cpp
After the storing of data in the archive, the content of the buffer of characters is printed:
Buffer content is 40 bytes : 127 1 9 1 84 1 99 1 16 255 128 5 0 228 11 84 2 4 208 15 73 64 8 255 255 255 255 255 255 255 127 1 7 87 111 114 108 100 32 33 Loaded values from the buffer are: t = 1 (bool) c = 'c' (char) s = 16 (int16_t) l = -128 (int32_t) ll =<PHONE_NUMBER>0 (int64_t) pi = 3.14159 (float) nan = nan (double) hello = "World !" (std::string)Again the PBA encoding scheme can be easily interpreted. This is let as an exercise.
You may have a look on the tutorial_pba_6.cpp program that shows a possible — and provocative — combined usage of the Boost/Serialization concepts, the Boost/Iostreams facilities and the PBA; it enables the copy of an object of a non-copyable class.
/** tutorial_pba_6.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This sample program shows how to use a portable binary archive * associated to a memory buffer to copy a non-copyable object. * */ #include <iostream> #include <string> #include <sstream> #include <vector> #include <boost/utility.hpp> #include <boost/iostreams/stream.hpp> #include <boost/iostreams/device/back_inserter.hpp> #include <boost/iostreams/device/array.hpp> #include <boost/cstdint.hpp> #include <boost/archive/portable_binary_oarchive.hpp> #include <boost/archive/portable_binary_iarchive.hpp> using namespace std; /* A foo noncopyable class */ struct foo : boost::noncopyable { uint32_t status; double value; double special; string to_string () const { ostringstream sout; sout << "foo={status=" << status << "; value=" << value << "; special=" << special<< "}"; return sout.str(); } template<class Archive> void serialize (Archive & ar, const unsigned int version) { ar & status; ar & value; ar & special; return; } }; // A templatized copy function for Boost/Serialization equipped classes. // Here we use PBAs associated to a memory buffer : template <class Serializable> void copy (const Serializable & source, Serializable & target) { namespace io = boost::iostreams; namespace ba = boost::archive; if (&source == &target) return; // self-copy guard typedef std::vector<char> buffer_type; buffer_type buffer; buffer.reserve (1024); { io::stream<io::back_insert_device<buffer_type> > output_stream (buffer); ba::portable_binary_oarchive opba (output_stream); opba & source; } { io::stream<io::array_source> input_stream (&buffer[0], buffer.size ()); ba::portable_binary_iarchive ipba (input_stream); ipba & target; } return; } int main (void) { // Some instance of the 'foo' class : foo dummy; dummy.status = 1; dummy.value = 3.14159; dummy.special = numeric_limits<double>::quiet_NaN (); clog << "dummy is : " << dummy.to_string () << endl; // Another instance of the 'foo' class : foo clone; /* The following instruction is forbidden because foo inherits 'boost::noncopyable' : clone = dummy; // this ends in a compilation error. */ // Anyway, we can use this workaround : copy (dummy, clone); clog << "clone is : " << clone.to_string () << endl; return 0; } // end of tutorial_pba_6.cpp
Remark : if a class has been made non-copyable at design, it is likely for a good reason; so it is not recommended to workaround this trait using such a trick, unless you know what you are doing and all the consequences !
In some circonstances, it may be useful to use the Boost text and XML archives in somewhat portable way. For example, we may want to benefit of the XML archive's human-friendly format for debugging purpose before to switch to the PBA for production runs. However, the text and XML archives provided by the Boost serialization library are not strictly portable, particularly because they does not support the serialization of non-finite floating point numbers. This is because the serialization of floating point numbers depends on some formatting features of standard I/O streams. See the tutorial_pba_7.cpp sample program below :
/** tutorial_pba_7.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This example shows how the default behaviour of standard * I/O streams does not support the read/write operations of * non-finite floating point values in a portable way. * */ #include <string> #include <iostream> #include <sstream> #include <limits> using namespace std; int main (void) { { float x = numeric_limits<float>::infinity (); double y = numeric_limits<double>::quiet_NaN (); cout.precision (8); cout << "x = " << x << endl; cout.precision (16); cout << "y = " << y << endl; } { string input ("inf nan"); istringstream iss (input); float x; double y; iss >> x >> y; if (! iss) { cerr << "Cannot read 'x' or 'y' : non finite values are not supported !" << endl; } } return 0; } // end of tutorial_pba_7.cpp
Depending on the system, one can get some various representation respectively for the infinity and NaN values :
1.#INFand
-1.#IND
infand
nan
Hopefully this issue can be solved by configuring the I/O streams with some special locale features provided by Boost (see this link).
The tutorial_pba_8.cpp program shows how this can be achieved through the use of special resources from the boost/archive/codecvt_null.hpp and boost/math/special_functions/nonfinite_num_facets.hpp headers :
/** tutorial_pba_8.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This example shows how to store some variables * of basic types (bool, integer, floating point numbers, STL string) * using the text or XML archive format associated to a * standard output file stream supporting portable non-finite * floating point values. * */ #include <string> #include <fstream> #include <limits> #include <locale> #include <boost/cstdint.hpp> #include <boost/archive/xml_oarchive.hpp> #include <boost/archive/text_oarchive.hpp> #include <boost/serialization/nvp.hpp> #include <boost/archive/codecvt_null.hpp> #include <boost/math/special_functions/nonfinite_num_facets.hpp> using namespace std; void do_text_out (void) { // The name for the example data text file : string filename = "pba_8.txt"; // Some variables of various primitive types : bool b = true; char c = 'B'; uint32_t answer = 42; float value = numeric_limits<float>::infinity (); double precision = numeric_limits<double>::quiet_NaN (); string question = "What makes you think she's a witch?"; // Open an output file stream : ofstream fout (filename.c_str ()); // Prepare the output file stream for inf/NaN support : locale default_locale (locale::classic (), new boost::archive::codecvt_null<char>); locale infnan_locale (default_locale, new boost::math::nonfinite_num_put<char>); fout.imbue (infnan_locale); { // Create an output text archive attached to the output file : boost::archive::text_oarchive ota (fout, boost::archive::no_codecvt); // Store (serializing) variables : ota & b & c & answer & value & precision & question; } return; } void do_xml_out (void) { // The name for the example data XML file : string filename = "pba_8.xml"; // Some variables of various primitive types : bool b = true; char c = 'B'; uint32_t answer = 42; float value = numeric_limits<float>::infinity (); double precision = numeric_limits<double>::quiet_NaN (); string question = "What makes you think she's a witch?"; // Open an output file stream : ofstream fout (filename.c_str ()); // Prepare the output file stream for inf/NaN support : locale default_locale (locale::classic (), new boost::archive::codecvt_null<char>); locale infnan_locale (default_locale, new boost::math::nonfinite_num_put<char>); fout.imbue (infnan_locale); { // Create an output text archive attached to the output file : boost::archive::xml_oarchive oxa (fout, boost::archive::no_codecvt); // Store (serializing) variables : oxa & BOOST_SERIALIZATION_NVP(b) & BOOST_SERIALIZATION_NVP(c) & BOOST_SERIALIZATION_NVP(answer) & BOOST_SERIALIZATION_NVP(value) & BOOST_SERIALIZATION_NVP(precision) & BOOST_SERIALIZATION_NVP(question); } return; } int main (void) { do_text_out (); do_xml_out (); return 0; } // end of tutorial_pba_8.cpp
The program creates two output files :
22 serialization::archive 9 1 66 42 inf nan 35 What makes you think she's a witch?
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <!DOCTYPE boost_serialization> <boost_serialization signature="serialization::archive" version="9"> <b>1</b> <c>66</c> <answer>42</answer> <value>inf</value> <precision>nan</precision> <question>What makes you think she's a witch?</question> </boost_serialization>
The tutorial_pba_9.cpp program deserializes the data from the text and XML archive files (respectively pba_8.txt and pba_8.xml) and prints the restored variables :
Loaded values from text archive are: b = 1 c = 'B' answer = 42 value = inf precision = nan question = "What makes you think she's a witch?" Loaded values from XML archive are: b = 1 c = 'B' answer = 42 value = inf precision = nan question = "What makes you think she's a witch?"
/** tutorial_pba_9.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This example shows how to load some variables of basic * types (bool, char, integer, floating point numbers, STL string) * using the text or XML archive format associated to a * standard file input stream supporting portable non-finite * floating point values. * */ #include <string> #include <fstream> #include <limits> #include <locale> #include <boost/cstdint.hpp> #include <boost/archive/xml_iarchive.hpp> #include <boost/archive/text_iarchive.hpp> #include <boost/serialization/nvp.hpp> #include <boost/scoped_ptr.hpp> #include <boost/archive/codecvt_null.hpp> #include <boost/math/special_functions/nonfinite_num_facets.hpp> using namespace std; void do_text_in (void) { // The name for the example data text file : string filename = "pba_8.txt"; // Some variables of various primitive types : bool b; char c; uint32_t answer; float value; double precision; string question; // Open an input file stream : ifstream fin (filename.c_str ()); // Prepare the input file stream for inf/NaN support : locale default_locale (locale::classic (), new boost::archive::codecvt_null<char>); locale infnan_locale (default_locale, new boost::math::nonfinite_num_get<char>); fin.imbue (infnan_locale); { // Create an input text archive attached to the input file : boost::archive::text_iarchive ita (fin, boost::archive::no_codecvt); // Store (serializing) variables : ita & b & c & answer & value & precision & question; } clog << "Loaded values from text archive are: " << endl; clog << " b = " << b << endl; clog << " c = '" << c << "'" << endl; clog << " answer = " << answer << endl; clog << " value = " << value << endl; clog << " precision = " << precision << endl; clog << " question = \"" << question << "\"" << endl; return; } void do_xml_in (void) { // The name for the example data text file : string filename = "pba_8.xml"; // Some variables of various primitive types : bool b; char c; uint32_t answer; float value; double precision; string question; // Open an input file stream : ifstream fin (filename.c_str ()); // Prepare the input file stream for inf/NaN support : locale default_locale (locale::classic (), new boost::archive::codecvt_null<char>); locale infnan_locale (default_locale, new boost::math::nonfinite_num_get<char>); fin.imbue (infnan_locale); { // Create an output text archive attached to the output file : boost::archive::xml_iarchive ixa (fin, boost::archive::no_codecvt); // Store (serializing) variables : ixa & BOOST_SERIALIZATION_NVP(b) & BOOST_SERIALIZATION_NVP(c) & BOOST_SERIALIZATION_NVP(answer) & BOOST_SERIALIZATION_NVP(value) & BOOST_SERIALIZATION_NVP(precision) & BOOST_SERIALIZATION_NVP(question); } clog << "Loaded values from XML archive are: " << endl; clog << " b = " << b << endl; clog << " c = '" << c << "'" << endl; clog << " answer = " << answer << endl; clog << " value = " << value << endl; clog << " precision = " << precision << endl; clog << " question = \"" << question << "\"" << endl; return; } int main (void) { do_text_in (); do_xml_in (); return 0; } // end of tutorial_pba_9.cpp
The tutorial_pba_10.cpp program illustrates how to serialize, then deserialize, a class from a PBA associated to a GZIP compressed file stream, thanks to a technique provided by the Boost/Iostreams library. The class contains a large STL vector of double precision floating point numbers with arbitrary values:
/** tutorial_pba_10.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This example shows how use PBAs combined with on-the-fly * compressed I/O streams. * */ #include <string> #include <fstream> #include <limits> #include <vector> #include <boost/cstdint.hpp> #include <boost/archive/portable_binary_oarchive.hpp> #include <boost/archive/portable_binary_iarchive.hpp> #include <boost/iostreams/filtering_stream.hpp> #include <boost/iostreams/filter/gzip.hpp> #include <boost/serialization/access.hpp> #include <boost/serialization/vector.hpp> using namespace std; class data_type { private: friend class boost::serialization::access; template<class Archive> void serialize (Archive & ar, const unsigned int version); public: void print (ostream & out, const string & title) const; public: vector<double> values; data_type (); }; data_type::data_type () : values () { return; } void data_type::print (ostream & out, const string & title) const { out << endl; out << title << " :" << endl; for (int i = 0; i < this->values.size (); ++i) { out.precision (16); out.width (18); out << this->values [i] << ' ' ; if ((i%4) == 3) clog << endl; } out << endl; return; } template<class Archive> void data_type::serialize (Archive & ar, const unsigned int version) { ar & values; return; } void do_gzipped_out (void) { // The name for the output data file : string filename = "pba_10.data.gz"; // A data structure to be stored : data_type my_data; // Fill the vector with arbitrary (possibly non-finite) values : size_t dim = 1000; my_data.values.reserve (dim); for (int i = 0; i < dim; ++i) { double val = (i + 1) * (1.0 + 3 * numeric_limits<double>::epsilon ()); if (i == 4) val = numeric_limits<double>::quiet_NaN (); if (i == 23) val = numeric_limits<double>::infinity (); if (i == 73) val = -numeric_limits<double>::infinity (); if (i == 90) val = 0.0; my_data.values.push_back (val); } // Print: my_data.print (clog, "Stored data"); // Create an output filtering stream : boost::iostreams::filtering_ostream zout; zout.push (boost::iostreams::gzip_compressor ()); // Open an output file stream in binary mode : ofstream fout (filename.c_str (), ios_base::binary); zout.push (fout); // Save to PBA : { // Create an output portable binary archive attached to the output file : boost::archive::portable_binary_oarchive opba (zout); // Store (serializing) the data : opba & my_data; } // Clean termination of the streams : zout.flush (); zout.reset (); return; } void do_gzipped_in (void) { // The name for the input data file : string filename = "pba_10.data.gz"; // A data structure to be loaded : data_type my_data; // Create an input filtering stream : boost::iostreams::filtering_istream zin; zin.push (boost::iostreams::gzip_decompressor ()); // Open an input file stream in binary mode : ifstream fin (filename.c_str (), ios_base::binary); zin.push (fin); // Load from PBA : { // Create an input portable binary archive attached to the input file : boost::archive::portable_binary_iarchive ipba (zin); // Load (deserializing) the data : ipba & my_data; } // Print: my_data.print (clog, "Loaded data"); return; } int main (void) { do_gzipped_out (); do_gzipped_in (); return 0; } // end of tutorial_pba_10.cpp
The resulting compressed pba_10.data.gz file contains 1,574 bytes. This has to be compared with the size of the plain (uncompressed) binary archive which equals 9,001 bytes:
127 1 9 0 0 2 232 3 0 8 3 0 0 0 0 0 240 63 8 3 0 0 0 0 0 0 64 8 4 0 0 0 0 0 8 64 8 3 0 0 0 0 0 16 64 8 255 255 255 255 255 255 255 127 8 4 0 0 0 0 0 24 64 8 ...which can be interpreted as :
It is also possible to use BZIP2 in a similar fashion (using ressources from the boost/iostreams/filter/bzip2.hpp header in place of boost/iostreams/filter/gzip.hpp).
The tutorial_pba_11.cpp program runs a benchmark test in the aim to compare the relative fastness of PBA and text archives both for read and write operations. It stores then loads a vector of many (107) random double values and prints the associated (de)serialization time for both kinds of archives:
/** tutorial_pba_11.cpp * * (C) Copyright 2011 François Mauger, Christian Pfligersdorffer * * Use, modification and distribution is subject to the Boost Software * License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at * http://www.boost.org/LICENSE_1_0.txt) * */ /** * The intent of this program is to serve as a tutorial for * users of the portable binary archive in the framework of * the Boost/Serialization library. * * This example program compares the times needed to serialize * and deserialize some large amount of data using PBA and * text archives. * */ #include <string> #include <fstream> #include <vector> #include <boost/archive/portable_binary_oarchive.hpp> #include <boost/archive/portable_binary_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> #include <boost/archive/text_iarchive.hpp> #include <boost/serialization/access.hpp> #include <boost/serialization/vector.hpp> #include <boost/random/mersenne_twister.hpp> #include <boost/random/uniform_real_distribution.hpp> #include <boost/timer.hpp> using namespace std; class data_type { private: friend class boost::serialization::access; template<class Archive> void serialize (Archive & ar, const unsigned int version); public: void print (ostream & out, const string & title) const; public: vector<double> values; data_type (); }; data_type::data_type () : values () { return; } void data_type::print (ostream & out, const string & title) const { out << endl; out << title << " :" << endl; bool skip = false; for (int i = 0; i < this->values.size (); ++i) { if ((i >= 12) && (i < (int) this->values.size () - 8)) { if (! skip) out << " ..." << endl; skip = true; continue; } out.precision (16); out.width (18); out << this->values [i] << ' ' ; if ((i%4) == 3) clog << endl; } out << endl; return; } template<class Archive> void data_type::serialize (Archive & ar, const unsigned int version) { ar & values; return; } double do_pba_out (const data_type & a_data) { string filename = "pba_11.data"; ofstream fout (filename.c_str (), ios_base::binary); boost::timer io_timer; { boost::archive::portable_binary_oarchive opba (fout); opba & a_data; } return io_timer.elapsed (); } double do_pba_in (data_type & a_data) { string filename = "pba_11.data"; ifstream fin (filename.c_str (), ios_base::binary); boost::timer io_timer; { boost::archive::portable_binary_iarchive ipba (fin); ipba & a_data; } return io_timer.elapsed (); } double do_text_out (const data_type & a_data) { string filename = "pba_11.txt"; ofstream fout (filename.c_str ()); boost::timer io_timer; { boost::archive::text_oarchive ota (fout); ota & a_data; } return io_timer.elapsed (); } double do_text_in (data_type & a_data) { string filename = "pba_11.txt"; ifstream fin (filename.c_str ()); boost::timer io_timer; { boost::archive::text_iarchive ita (fin); ita & a_data; } return io_timer.elapsed (); } int main (void) { double elapsed_time_pba_out; double elapsed_time_text_out; double elapsed_time_pba_in; double elapsed_time_text_in; data_type my_data; // A data structure to be stored then loaded. { // Fill the vector with random values : size_t dim = 10000000; my_data.values.reserve (dim); boost::random::mt19937 rng; boost::random::uniform_real_distribution<> flat (0.0, 100.0); for (int i = 0; i < dim; ++i) { double val = flat (rng); my_data.values.push_back (val); } my_data.print (clog, "Stored data in PBA and text archive"); } { // Store in PBA : elapsed_time_pba_out = do_pba_out (my_data); } { // Store in text archive : elapsed_time_text_out = do_text_out (my_data); } { my_data.values.clear (); // Load from PBA : elapsed_time_pba_in = do_pba_in (my_data); my_data.print (clog, "Loaded data from PBA"); } { my_data.values.clear (); // Load from text archive : elapsed_time_text_in = do_text_in (my_data); my_data.print (clog, "Loaded data from text archive"); } clog << "PBA store I/O elapsed time : " << elapsed_time_pba_out << " (second)" << endl; clog << "Text store I/O elapsed time : " << elapsed_time_text_out << " (second)" << endl; clog << "PBA load I/O elapsed time : " << elapsed_time_pba_in << " (second)" << endl; clog << "Text load I/O elapsed time : " << elapsed_time_text_in << " (second)" << endl; return 0; } // end of tutorial_pba_11.cpp
On a 1.60 GHz processor running gcc 4.5.2 on Linux 2.6.38, the result is the following:
PBA store I/O elapsed time : 1.86 (second) Text store I/O elapsed time : 22.66 (second) PBA load I/O elapsed time : 1.53 (second) Text load I/O elapsed time : 19.71 (second)It this simple case, the use of portable binary archives is faster by at least a factor 10 compared to the traditional Boost text archives. This is a significant saving in time. These performances are highly desirable in the typical framework of scientific/computing activities where large amounts of data are accessed through files. The PBA concept is thus a valuable candidate for such applications.
One can also consider the sizes of the resulting archive files:
Revised 2011-11-07
© Copyright François Mauger,
Christian Pfligersdorffer 2011.
Distributed under the Boost Software License, Version 1.0.
(See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)


HOME
ABOUT
HIRE DEVELOPER
SIGN UP
SIGN IN
What we do
Drink Coffee
Code
Collaborate-On-Software
Publish


Advent of Code 2019
Solutions to
AoC2019
using only
Kotlin standard library
with in-browser runnable tests


Derivation
: derivation
(Please note: this part of the documentation is not yet completed.)
Language-specific feature identifying the form of derivation applied to a word.
AdvstO
: adverb derived in -сто/-стэ
Produces adverbs indicating manner
Examples
- [myv] домкасто “in depth”
Dimin
: derivation with suffix -ыне/-ине/-не/-нэ
Produces diminutive derivations of nouns.
Examples
- [myv] пулыне “a small tail”
GenAttr
: genitive attribute -Онь
Produces indefinite genitive-like modifiers from adverbs
Examples
- [myv] алкуксонь “real”
NomAg
: derivation with suffix -ы/-и
Produces nomen agensis (non-temporal).
Examples
- [myv] стада ваны “herder”
- [myv] эри ялга “live in friend”
OkshnOms
: derivation with suffix -ОкшнОмс
Produces deverbal verbs indicating discontinuous past.
Examples
- [myv] сакшкомс “to have arrived
"
Omka
: derivation with suffix -Омка
Produces deverbal nouns that generally indicate an instrument.
Examples
- [myv] а тескеть карксозонзо поволезь ванстомкатне \n “and here are the amulets hung on his belt”
OvOms
: derivation with suffix -ОвОмс
Produces a verb form with epistemic modality and an inherent telicity but no volition. With derivations from transitive verbs this may be close to a “get” passive, whereas the intransitive derivation seems closer to a “manage to/happen to” reading.
Examples
- [myv] содавомс “to get recognized”
- [myv] удовомс “to manage to sleep (too much)”
- [myv] ярсавомс “to eat (too much)”
Ovt
: derivation with suffix -Овт/-Овть
Produces participle-like adnominal modifiers. In grammars, it is requires a genitive modifiers,
but in literature it may also be used with a genitive modifier in a similar function to Ozj
.
Examples
- [myv] валськень правт лов \n “snow that has fallen in the morning”
- [myv] ойсэ вадневть картуз \n “a hat that is smeared with grease”
Ozj
: derivation with suffix -Озь
Produces converb.
Examples
- [myv] ютазь “having passed”
PronGak
: derivation with suffix -гак/-как/-ак
Produces indefinite pronouns, adverbs.
Examples
- [myv] кияк “anybody”
- [myv] мезеяк “anything”
- [myv] косояк “anywhere”
Cond
: protasis marker, aka conditional -Ондеря-
Marks the protasis verb, which may be in the the so-called subjunctive (presumably this conditional) and indicative non-past and past (quite infrequent).
Examples
- [myv] Мериндеряволь исяк — саевлинек эйкакшонть мартонок… “if he/she/it had said yesterday, we would have taken the child with us.”
VSj
: derivation with suffix -сь
Produces participle-like form from verb.
Examples
- [myv] синдсь политикадонть “about battered policy”
VerbYcja
: derivation with suffix -ыця/-иця
Used with verbs for participle type including a deictic element ця/се
Examples
- [myv] сэредиця “someone who is ill”
- [myv] эриця ялга “living friend, friend who is alive”
VerbYks
: derivation with suffix -ыкс/-икс
Used with verbs for participle type indicating potential
Examples
- [myv] кадовикс “tending to be left”
VGen
: verbal genitive
Used with verbs to produce adnominal attributes without temporal dynamism.
Examples
- [myv] апак учонь “unexpected”
- [myv] салтонь куяр “salt pickle”
Voc
: vocative derivation with suffix -й
Used with proper nouns and kinship terms ending in vowels.
Examples
- [myv] ялгай “friend”
Shka
: derivation with suffix -шка
Used to indicat approximate or near equivalent.
Examples
- [myv] сатышка “enough”
VocKaj
: vocative derivation with suffix -кай
Used with common and proper nouns and kinship terms ending in vowels and consonants.
Examples
- [myv] ялгакай “friend”
- [myv] нумолкай “rabbit”
Derivation in other languages: [abq] [apu] [fi] [koi] [kpv] [mdf] [myv] [sms] [yrl]


| ID | Description | Client | Line Items | Created By | Created On | Balance |
|---|---|---|---|---|---|---|
| {{invoice.id}} | {{invoice.invoiceDescription}} | {{invoice.company.name}} | {{invoice.lineItems.length}} | {{invoice.createdBy.username}} | {{invoice.createdOn}} | {{invoice.currentBalance}} |
| ID | Description | Client | Line Items | Created By | Created On | Paid On |
|---|---|---|---|---|---|---|
| {{invoice.id}} | {{invoice.invoiceDescription}} | {{invoice.company.name}} | {{invoice.lineItems.length}} | {{invoice.createdBy.username}} | {{invoice.createdOn}} | {{invoice.paidOn}} |


Hello! This is a simple syllable generation tool to help you refine your conlang's phonotactics.
Let's get started!
First, you'll want to define your phonemes. This page provides a lot of options, but the primary lists are consonants and vowels.
All of these lists are comma-separated, but you can combine multiple characters for things like diphthongs.
I've already loaded our consonant and vowel lists with the phonemes in American English.
For more complicated syllable structures, there are a series of additional lists that you can use in addition to the generic Consonant and Vowel lists.
There are also four extra "generic" lists you can use if your language requires it.
After defining your phonemes, click on the "Structure" tab.
This is where we will define our syllablic structure.
This panel provides you with a snapshot of your current phoneme lists so you don't have to flick back and forth between tabs.
This is where you will enter your syllable structure.
It follows a pretty standard format where elements not in parenthesis are required components of the syllable, and elements inside of parenthesis are optional.
This is broken out into the three basic parts of a syllable. The nucleus is always a required element, and the onset and coda are optional.
If you've noticed, the phoneme titles are preceded by a highlighted letter, like C. This is the letter you'll use for that phonemic group in your structure.
If desired, you may enter simple sonority rules. Each rule is made up of three parts:
These elements currently only support a single entry. However, you could pair phonemes in these entries.
Like the other entries, I've loaded this with two common sonority rules in American English.
Once you're ready, click "Generate" and you'll be provided with a list of syllables built using your criteria.
Remember, the larger the lists, and the more complicated the syllable structure, the longer it will take to make your list.
Pretty simple, right?
Go forth, and have fun conlanging!


| 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317 | 1×
1×
1×
1×
1×
1×
1×
1×
1×
1×
10×
10×
10×
10×
10×
10×
10×
1×
1×
1×
| raf = require('raf')
$ = require('jquery')
Plugin = require('../plugin')
scrollIntoView = require('scroll-into-view')
highlighter = require('../highlighter')
polyglot = require('../../shared/polyglot');
BUCKET_SIZE = 16 # Regular bucket size
BUCKET_NAV_SIZE = BUCKET_SIZE + 6 # Bucket plus arrow (up/down)
BUCKET_TOP_THRESHOLD = 115 + BUCKET_NAV_SIZE # Toolbar
# Scroll to the next closest anchor off screen in the given direction.
scrollToClosest = (anchors, direction) ->
dir = if direction is "up" then +1 else -1
{next} = anchors.reduce (acc, anchor) ->
unless anchor.highlights?.length
return acc
{start, next} = acc
rect = highlighter.getBoundingClientRect(anchor.highlights)
# Ignore if it's not in the right direction.
if (dir is 1 and rect.top >= BUCKET_TOP_THRESHOLD)
return acc
else if (dir is -1 and rect.top <= window.innerHeight - BUCKET_NAV_SIZE)
return acc
# Select the closest to carry forward
if not next?
start: rect.top
next: anchor
else if start * dir < rect.top * dir
start: rect.top
next: anchor
else
acc
, {}
scrollIntoView(next.highlights[0])
module.exports = class BucketBar extends Plugin
# svg skeleton
html: """

"""
# Plugin options
options:
# gapSize parameter is used by the clustering algorithm
# If an annotation is farther then this gapSize from the next bucket
# then that annotation will not be merged into the bucket
gapSize: 60
# Selectors for the scrollable elements on the page
scrollables: ['body']
# buckets of annotations that overlap
buckets: []
# index for fast hit detection in the buckets
index: []
# tab elements
tabs: null
constructor: (element, options) ->
super $(@html), options
Iif @options.container?
$(@options.container).append @element
else
$(element).append @element
pluginInit: ->
$(window).on 'resize scroll', @update
for scrollable in @options.scrollables ? []
$(scrollable).on 'resize scroll', @update
destroy: ->
$(window).off 'resize scroll', @update
for scrollable in @options.scrollables ? []
$(scrollable).off 'resize scroll', @update
_collate: (a, b) ->
for i in [0..a.length-1]
if a[i]  b[i]
return 1
return 0
# Update sometime soon
update: =>
return if @_updatePending?
@_updatePending = raf =>
delete @_updatePending
@_update()
_update: ->
# Keep track of buckets of annotations above and below the viewport
above = []
below = []
# Construct indicator points
points = @annotator.anchors.reduce (points, anchor, i) =>
unless anchor.highlights?.length
return points
rect = highlighter.getBoundingClientRect(anchor.highlights)
x = rect.top
h = rect.bottom - rect.top
if x < BUCKET_TOP_THRESHOLD
if anchor not in above then above.push anchor
else if x > window.innerHeight - BUCKET_NAV_SIZE
if anchor not in below then below.push anchor
else
points.push [x, 1, anchor]
points.push [x + h, -1, anchor]
points
, []
# Accumulate the overlapping annotations into buckets.
# The algorithm goes like this:
# - Collate the points by sorting on position then delta (+1 or -1)
# - Reduce over the sorted points
# - For +1 points, add the annotation at this point to an array of
# "carried" annotations. If it already exists, increase the
# corresponding value in an array of counts which maintains the
# number of points that include this annotation.
# - For -1 points, decrement the value for the annotation at this point
# in the carried array of counts. If the count is now zero, remove the
# annotation from the carried array of annotations.
# - If this point is the first, last, sufficiently far from the previous,
# or there are no more carried annotations, add a bucket marker at this
# point.
# - Otherwise, if the last bucket was not isolated (the one before it
# has at least one annotation) then remove it and ensure that its
# annotations and the carried annotations are merged into the previous
# bucket.
{@buckets, @index} = points
.sort(this._collate)
.reduce ({buckets, index, carry}, [x, d, a], i, points) =>
if d > 0 # Add annotation
if (j = carry.anchors.indexOf a) < 0
carry.anchors.unshift a
carry.counts.unshift 1
else
carry.counts[j]++
else # Remove annotation
j = carry.anchors.indexOf a # XXX: assert(i >= 0)
if --carry.counts[j] is 0
carry.anchors.splice j, 1
carry.counts.splice j, 1
if (
(index.length is 0 or i is points.length - 1) or # First or last?
carry.anchors.length is 0 or # A zero marker?
x - index[index.length-1] > @options.gapSize # A large gap?
) # Mark a new bucket.
buckets.push carry.anchors.slice()
index.push x
else
# Merge the previous bucket, making sure its predecessor contains
# all the carried annotations and the annotations in the previous
# bucket.
if buckets[buckets.length-2]?.length
last = buckets[buckets.length-2]
toMerge = buckets.pop()
index.pop()
else
last = buckets[buckets.length-1]
toMerge = []
last.push a0 for a0 in carry.anchors when a0 not in last
last.push a0 for a0 in toMerge when a0 not in last
{buckets, index, carry}
,
buckets: []
index: []
carry:
anchors: []
counts: []
latest: 0
# Scroll up
@buckets.unshift [], above, []
@index.unshift 0, BUCKET_TOP_THRESHOLD - 1, BUCKET_TOP_THRESHOLD
# Scroll down
@buckets.push [], below, []
@index.push window.innerHeight - BUCKET_NAV_SIZE,
window.innerHeight - BUCKET_NAV_SIZE + 1,
window.innerHeight
# Calculate the total count for each bucket (without replies) and the
# maximum count.
max = 0
for b in @buckets
max = Math.max max, b.length
# Update the data bindings
element = @element
# Keep track of tabs to keep element creation to a minimum.
@tabs ||= $([])
# Remove any extra tabs and update @tabs.
@tabs.slice(@buckets.length).remove()
@tabs = @tabs.slice(0, @buckets.length)
# Create any new tabs if needed.
$.each @buckets.slice(@tabs.length), =>
div = $('
').appendTo(element)
@tabs.push(div[0])
lengthOfPreviousBucket = 0;
div.addClass('annotator-bucket-indicator')
# TO DO : Use config custom theme to disable this
# Focus corresponding highlights bucket when mouse is hovered
# TODO: This should use event delegation on the container.
# .on 'mousemove', (event) =>
# bucket = @tabs.index(event.currentTarget)
# for anchor in @annotator.anchors
# toggle = anchor in @buckets[bucket]
# $(anchor.highlights).toggleClass('annotator-hl-focused', toggle)
# # Gets rid of them after
# .on 'mouseout', (event) =>
# bucket = @tabs.index(event.currentTarget)
# for anchor in @buckets[bucket]
# $(anchor.highlights).removeClass('annotator-hl-focused')
# Does one of a few things when a tab is clicked depending on type
.on 'click', (event) =>
bucket = @tabs.index(event.currentTarget)
event.stopPropagation()
feedbackListInBucket = []
# Populate the highlights in the bucket in an array and remove other highlights from the page.
for anchor in @annotator.anchors
if anchor in @buckets[bucket]
feedbackListInBucket.push($(anchor.highlights))
else
$(anchor.highlights).removeClass('annotator-hl-focused')
lengthOfCurrentBucket = feedbackListInBucket.length;
for feedback in feedbackListInBucket
if lengthOfCurrentBucket != lengthOfPreviousBucket
feedback.addClass('annotator-hl-focused');
else
feedback.toggleClass('annotator-hl-focused')
# If it's the upper tab, scroll to next anchor above
if (@isUpper bucket)
scrollToClosest(@buckets[bucket], 'up')
# If it's the lower tab, scroll to next anchor below
else if (@isLower bucket)
scrollToClosest(@buckets[bucket], 'down')
# TO DO : Use config custom theme to disable this
# else
# annotations = (anchor.annotation for anchor in @buckets[bucket])
# @annotator.selectAnnotations annotations,
# (event.ctrlKey or event.metaKey),
# previousBucket = feedbackListInBucket;
lengthOfPreviousBucket = lengthOfCurrentBucket;
this._buildTabs(@tabs, @buckets)
_buildTabs: ->
@tabs.each (d, el) =>
el = $(el)
bucket = @buckets[d]
bucketLength = bucket?.length
title = polyglot().t("Show feedback", {bucketLength: bucketLength});
el.attr('title', title)
el.toggleClass('upper', @isUpper(d))
el.toggleClass('lower', @isLower(d))
if @isUpper(d) or @isLower(d)
bucketSize = BUCKET_NAV_SIZE
else
bucketSize = BUCKET_SIZE
el.css({
top: (@index[d] + @index[d+1]) / 2
marginTop: -bucketSize / 2
display: unless bucketLength then 'none' else ''
})
if bucket
el.html("
#{bucketLength}
")
isUpper: (i) -> i == 1
isLower: (i) -> i == @index.length - 2
# Export constants
BucketBar.BUCKET_SIZE = BUCKET_SIZE
BucketBar.BUCKET_NAV_SIZE = BUCKET_NAV_SIZE
BucketBar.BUCKET_TOP_THRESHOLD = BUCKET_TOP_THRESHOLD
|


We're sorry you didn’t have a chance to complete the experiment. Please follow the link to complete our exit survey and receive your bonus. Thank you for participating!


To migrate an object from one thread to another, Graphcode needs to be
able to create an object of the correct type in the destination
address space. This is achieved by means of a run time type
identification (RTTI) system. Given a type token
t
, an object of that type can be created by the call:
object *o=archetype[t]->lnew();
Instead of using C++'s built-in RTTI system, where tokens are compound
objects of somewhat indeterminate size, Graphcode implements a simple
RTTI system using template programming, in which a type token is a
simple unsigned integer. This implies that each type of object
to be used with Graph must be registered first, before use. This is
taken care for you automatically if you use Graph::AddObject()
to
add your object to the Graph.
Adding the virtual type method to your class is also easy:
class foo: public object { ... virtual int type() {return vtype(*this);} };
The first vtype
is called on an object, an object
of that type is created (via its lcopy
method), and added to
the archetype vector. The index of that object within
the archetype vector become the type token. It is vitally
important that types are added to the archetype vector in the same
order on all threads. Clearly this is a trivial requirement if only
one type is used, but slightly more care needs to be taken in the case
of multiple types of object.
If you have multiple object types, consider using the
register_type
template to ensure a consistent type registration
across the different address spaces.


I like very much the feature that tells you, in the weapon/vehicles stats, in which part of the leaderboards you are (top X%).
It would be nice to add, just beside the top X%, a little arrow, green for up and red for down, to see how you are progressing with the specific weapon/vehicle.
It could be done with a regular update: for example every day a check is done and the ranking of the day is compared with the ranking from the day before and then a green or red arrow appears.
Maybe you could have better idea to implement these comparisons.
I think this would generally increase the challenge of bf3 very much.


I am based in Atlanta. In no particular order of preference, my interests include running, literature, video games, and traveling. As a former teacher, I understand that success is the result of learning and practicing the proper way. My journey's just begun, and I aim to keep growing as a developer.
A timer app which randomizes your tabata workouts.
Front-end: React.js, SCSS
|
RepoAn app to help you record habit progress for fifteen days.
Front-end: HTML5, CSS3, JavaScript.
Back-end: Node, Express, MongoDB, Mongoose, Agenda, JWT.
|
RepoCheck air quality and other weather conditions to get the most out of your ride.
Front-end: HTML5, CSS3, JavaScript.
APIs: Google Maps, Dark Sky, AQI.
|
Repo


The family trip to Yang Zhou in 2022 summer. Because of covid-9 and control policy, we haven't travel out of Shanghai for about two years. We decided to make it by any means at my son's summer break. Luckily, Yangzhou is not far from Shanghai by train. That means if in any urgent case, we can shuttle back. The result came out very nice. The city is pleasant. The food is very good. The scene is so fancsinating that makes me remember the time in Hangzhou. Moreover, son and I grabbed a few poems about Yangzhou. Here is one of them, and can be the conclusion of this memorable trip. "垂杨不断接残芜，雁齿虹桥俨花图。也是销金一锅子，故应唤作瘦西湖".


As described on the Potree website1 The Potree viewer is a WebGL based viewer for large point clouds. The project evolved as a Web based viewer from the Scanopy2 desktop point cloud renderer by TU Wien, Institute of Computer Graphics and Algorithms.3 It is free and open source to enable anyone to view, analyze and publicly share their large datasets online and is currently being continued under the Harvest4D4 project.
The potree viewer has almost as wide a range of uses as 3d data does. The viewer itself requires all used data to be converted to the Octree format and developers on the Github page5 make continuous updates. Linked to this same page is a converter6 which allows any user to convert .las, .laz, binary ply, .xyz, or .ptx files into Octree format. By downloading the converter repository, the .exe converter file can be run in any command prompt when the user identifies the input file and the output directory.
For this project, the Potree viewer is being used to share LiDAR data of the Woodpecker Cave Field School run by Dr. James G. Enloe of the Department of Anthropology at the University of Iowa. Specifically, the bottom of the pit from the end of the 2015 field season is shown, along with the rock shelter which the pit sits outside of. The potree viewer has many tools included in its pre-written code including measurement tools and draw tools. In addition, point densities and sizes can be manipulated and point values and colors can be changed using an included dropdown menu. Point values such as true color RGB can be taken from photographs and applied to the points and background images can be used in the viewer once they have been converted to skybox format using Adobe Photoshop or a similar program.
While this project tries to replicate all of these features for the Woodpecker Cave field site, it also has several goals going beyond what is described in the demo pages on the Potree site. Namely, this project tries to add additional point files showing different groupings of excavated artifacts and color code them as such, allowing the viewer to see different artifact groups by toggling them on and off in the dropdown menu. In addition, a pop-up will be created to introduce the site to new viewers and explain its history briefly. Finally, stacked transparent shapefiles will be added to show the thickness of each season’s excavation and help the viewer to see the placement of artifacts within each field season. Though the Potree viewer does not currently support all of these purposes, this project would like to explore their possibility and feasibility so that more information about the site can be accessible to the public without requiring anyone to purchase expensive software such as ArcGIS.
The eventual goal of this is to publish the viewer on the University of Iowa website so that anyone can access information about the site itself and learn about the field school there. Ideally, even academics studying the site could access the viewer and get rough estimates, measurements, and visualizations if need be. In the end, the documentation of these new applications for the viewer will be published in an easy to understand guide so that Anthropology Departments everywhere will also be able to quickly and easily build a viewer for their own field sites.
1 http://potree.org/wp/about/
2 https://www.cg.tuwien.ac.at/research/projects/Scanopy/
3 https://www.cg.tuwien.ac.at/
4 https://harvest4d.org/
5 https://github.com/potree
6 https://github.com/potree/PotreeConverter


Notes related to theory of Deep Learning, as discussed in the classroom. For convenience the lecture notes have been thematically organized, so that to give a more holistic understanding of the theory and the applications of Deep Learning.
Get Started!Torch is currently one of the most popular deep learning frameworks in top research labs around the world, such as Facebook and Google especially DeepMind. Torch started from Y. LeCun's lab, who is one of the leading experts in deep learning and the reference when working with convolutional neural networks. In Torch there is a great array of existing codes, deep learning softwares and components that will make your life easy, especially once you successfully complete this course.
Torch sits on top of an existing programming language, Lua. Lua is a lightweight, very fast multi-paradigm programming with a somewhat "Matlab"-like coding style. Torch and Lua are rich in features and relatively easy to use and, therefore, are ideal for rapid prototyping. However, they can be quite annoying in the beginning, because of some peculiarities.
Although Torch does not come with a graphical interface that would allow for easier programming, it does have a web interactive interface, namely iTorch. For those of you familiar with Jupyter notebooks for iPython, this should be quite easy. Jupyter has implemented a Torch kernel, therefore, one can easily run torch software interactively on his/her web-browser as if he was running Matlab.
For those of you not familiar with Jupyter notebooks,
The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.
The great thing about Jupyter notebooks is that you can start them as easily as just typing itorch notebook in your terminal (of course once you have iPython and iTorch installed).
Next, we will guide you through the installation process and provide you with some short tutorial in Lua, and a longer tutorial in Torch.
You do not need to explicitly install Lua, just install Torch and everything is going to work just fine. To install Torch, please visit their website and follow the instructions. Although the basic installations are alright, you might experience some complications because of prior iPython installations. In that you case you might need to rename some library files in the iPython installation (such libreadlince.* to something else -- do not delete them as you might need to bring them back in the future) In any case, the installation should not take more than a few minutes if everything goes smoothly, or an hour if you have difficulties. If you have difficulties that you cannot resolve, please lmgtfy or ask a question in our virtual class in Piazza.
At this point we have to note that Torch is mainly supported for X systems (Linux, OS X), but not Windows. For those that cannot live without Windows for some reason, or that cannot have a separate partition where they install Ubuntu or Linux Mint, go to LeCun's NYU research group website. The simplest solution is to download this Ubuntu virtual machine they have prepared, which includes a pre-compiled, pre-installed version of Torch. This virtual machine requires VirtualBox.
Once you have the basic Torch installation, you shoule install additional Torch packages that are very useful for this course and beyond. To install new Torch packages one simply needs to type in his terminal luarocks install MyNewPackage.
There are several useful or even necessary packages that you should now install: nn, gnuplot, optim and of course itorch. For more packages you can visit the Torch cheatsheet.
To be filled in.
There are dozens of tutorials for Lua out there. Please, be patient enough to go through at least one, as there are certain Lua peculiarities that might make your life miserable, if you do not know them in advance. Some tutorials that we found useful were the following:
Once you read these tutorials, you should be able to start some basic programming with Lua.
Before continuing, you should be confident in understanding the following concepts.
Defining new variables with = The local works like in Python and unlike Matlab.
This means that once you declare a variable myvar = 10 and then you define another variable myvar2 = myvar, the myvar2 actually refers to the memory address of myvar.
Hence, if you change myvar2, also myvar will change.
If you want a clean fresh variable you should use copy().
The keyword local The local keyword is a very important concept to understand.
In Lua by default all variables are declard global, unless you add the keyword local in front, like local myvar = 10.
Although this might look like an innocent mistake, it can create complete chaos in combination with the =.
If you don't define you variable to be local inside your function fun1 (e.g., you just type num_epochs=10), and accidentally (or not) have the same variable name in other functions funOthers, changing num_epochs in fun1 will also affect the same variable in funOthers.
Also, note that local variables only exist in standalone scripts that one runs from the terminal.
If you just start an interactive session with th or with iTorch, local variables do not make sense.
If you define one, you will simply not be able to access it.
Lua table Lua does not have lists, arrays, dictionaries and all the exotic variations of "keeping things together in a single variable", like Python.
Instead, it has tables (and metatables), with which one can do pretty much the same things.
One can iterate of the table elements with ipairs.
The keyword require The require keyword works like import in Python, namely to import packages to your script/function/file.
Beware that the same rules apply about the local keyword.
Unless you want the package to be accessible by all files, use the local.
Torch is a scientific computing framework with wide support for machine learning algorithms. A basic component for machine learning and scientific programming are vectors, arrays, matrices, etc. Torch generalizes in an efficient manner vectors (1-d) and matrices (2-d) to tensors torch.Tensor() of n-dimensions. By calling t = torch.Tensor(2, 3, 4, 5, 6) you get a new 5-d tensor t.
There are some nice tutorials out there about Torch. Consider some of the following:
The two single most important classes of objects in Torch is the tensor and maths. The tensor class provides all sorts of functionalities for manipulating vectors, matrices and tensors. The math class provides all sorts of mathematical functions necessary in every scientific programming framework.
In large scale machine learning research memory and computational efficiency can be of paramount importance. With Torch one can achieve high efficiency if one performs as many operations as possible in-place. This means, you should try to avoid to define/call new variables, if you can reuse existing ones. Through specialized functions Torch allows this to be done quite easily. For example consider the following two cases.
local t = torch.Tensor(10, 10)
local t2 = torch.Tensor(10, 10)
t3 = t + t2
and
local t = torch.Tensor(10, 10) local t2 = torch.Tensor(10, 10) t:add(t2)
In the first example a new variable, t3, is defined and new memory needs to be allocated. If, however, t, is not needed anymore, we can just overwrite it to store the final summation. As a result, we avoid allocating new resources. Although this example is trivial, you might want to rethink it when you have matrices with millions of entries. In general there are specialized functions for avoiding to performing specific tensor operations manually. For example torch.expand([...]) expands a tensor without allocating memory and torch.fill([...]) fills in matrix entries without needing to run a for loop.
Although understanding things in theory are important, to become experts you need to start playing with code and programming. We have prepared the first practical so that you familiarize yourself better with Lua and Torch, and more specifically with using tensors, writing functions, calling important packages like optim and itorch.Plot.
A neural network is an architecture of connected modules. You can think of it as one big pipeline (or system or network) of interconnected boxes/modules. The learning stage in this pipeline is simply a propagation of forward and backward flows from one module to neighboring modules. At the first half of each learning round (or epoch), the network makes a guess what the input it received could be. This is called forward progapation. At the second half of each learning epoch the network tests how good the guess was. If the guess was bad, then the network modules have to change a lot to come closer to the desired output. If, instead, the guess was quite good, it means that the network is in pretty good shape and only fine changes are needed, if any. The second step is called backward progapation. Both steps are usually referred to as backpropagation, which is one of the most heavily used terms in the neural network nomenclature.
From engineering sciences (software engineering, electrical engineering, etc.) we know that a system design is good, when it is decomposable. From the point of view of the system's architect, any module does not need to know anything about the rest of the modules. What any module only needs to know, instead, is (a) where it is connected to so that it can transfer the information and, (b) what are the incoming inputs during forward or backward propagation. Then, the system complexity is achieved by combining different modules of various purposes and sophistications into all sorts of crazy architectures.
This design philosophy has three strong advantages. First, as long as the modules meet certain requirements, which we are going to revisit later, one can build as complex modules as s/he desires. The reason module complexity will not affect the operationality of the overall network, as all module have the same "API". Second, as long as one can define a meaningful forward and backward propagation of flows, one is free to design a wide variety of different neural network architectures and connectivities, ranging from feedforward networks to loopy networks. Third, given good modularity it is much easier to "debug" a complicated architecture with "unit tests". A unit test is a test where one checks whether a module produces the output, which the theory predicts, given a standard predefined input. For example, if our module is designed to compute the square root of any number, when we define it's input to be "9", "16", "25", it should return "3", "4", "5". Since we can now verify the correctedness of each module independently, we can build very complicated networks and still be sure to backtrace future problems relativelye easily.
Architectures that have only forward connections are called feedforward neural networks and are straightforward to implement. Architectures that have forward connections between layers, which, however, are not adjacent to each other, form directed acyclic graphs (DAG) networks. A frequently used term in DAG networks is skip layers. A layer that has forward connections not only to its immediate adjacent layers is a skip layer. DAG networks are also quite easy to implement as again, a given module is agnostic to the incoming inputs.
Architectures that have layers connected in cycles form loopy networks. Loopy networks are more complicated, since training and inference cannot be performed in a straightforward manner. Instead, they have to be unrolled in time.
An important note here is that although with neural networks one can design crazy complicated modules and architectures, the final network is not going to necessarily be better. For one experiments show that less complicated modules (ReLUs instead of tanh) are to be preferred, as they are easier to train given a certain amount of time. Also, more complex architectures will generally be harder to train, therefore less data will be processed during training, leading to worse generalizations. Hence, just because you can does not mean that you should design very complex networks.
Backpropagation is the single most popular algorithm for training neural networks. Let's assume we have the network below, composed of $L$ layers. The per layer activations based on the module functions per layer are: $$\begin{eqnarray} a_1 = h_1(x_1, \theta_1), \\ a_2 = h_2(x_2, \theta_2), \\ a_3 = h_3(x_3, \theta_3), \\ \dots, \\ a_L = h_L(x_L, y), \\ \end{eqnarray}$$
where the last layer module equals to the loss function for our network, $h_5 \equiv \mathcal{L}$. Apparently, the output of layer $l$ is the input for layer $l+1$, therefore, we have that $$ a_l = x_{l+1} $$
To optimize the network, namely to find the optimal weights $\theta_l, l=1, \dots, L-1$, we se (stochastic) gradient descent. Hence, we need to compute all the gradients of the network with respect to the weights, namely $\dfrac{\partial{\mathcal{L}}}{\partial \theta_l}, l=1, \dots, L-1$ The module of layer $L$ implements the loss function, therefore does not contain any parameters. For layer $L-1$ we have parameters $\theta_{L-1}$, hence no gradient need to be computed. For all other layers, however, we can have potentially gradients. For the module of layer $L-1$, given the good old \emph{chain rule} for computing the derivatives adn the loss function $\mathcal{L}$ (normally we should write it as $\mathcal{L}(x_L, y_L)$, but we opt for $\mathcal{L}$ for consiceness), we have for $$\begin{eqnarray} \dfrac{\partial \mathcal{L}}{\partial \theta_{L-1}} & = & \dfrac{\partial \mathcal{L}(a_{L-1}, y_L)}{\partial \theta_{L-1}} \text{\footnotemark}\\ & = & \dfrac{\partial \mathcal{L}}{\partial{a_{L-1}}} \dfrac{\partial a_{L-1}}{\partial{\theta_{L-1}}} \end{eqnarray}$$
Generally, for the module of layer $l$ we would have that
$$\begin{eqnarray} \dfrac{\partial \mathcal{L}}{\partial \theta_l} = \dfrac{\partial \mathcal{L}}{\partial{a_l}} \dfrac{\partial a_l}{\partial{\theta_l}} \label{eq:grad-layer} \end{eqnarray}$$
Eq.~\eqref{eq:grad-layer} provides us with a pretty straightforward way of computing the gradients for layer $l$. We first need the gradient of the module w.r.t. each parameters $\dfrac{\partial a_l}{\partial{\theta_l}}$, which is easy to compute provided the module activation function is a (almost everywhere) differentiable function. For example, if our module $l$ had the activation function $a_l=h_l(x_l, \theta_l)=\sqrt{\theta_l \cdot x_l}$, we would simply have $\dfrac{\partial a_l}{\partial{\theta_l}}=\dfrac{-0.5 x_l}{\sqrt{\theta_l \cdot x_l}}$.
Second, we need the gradient of the loss function with respect to the output of layer $\dfrac{\partial \mathcal{L}}{\partial{a_l}}$. Given that between the loss function $\mathcal{L}$ and the module of layer $l$ there might be several other layers, if we would explicitly write down the full equation would yield a very complicated function. However, we notice that we could decompose $\dfrac{\partial \mathcal{L}}{\partial{a_l}}$ as $$ \dfrac{\partial \mathcal{L}}{\partial{a_l}} = \dfrac{\partial \mathcal{L}}{\partial{a_{l+1}}} \dfrac{\partial a_{l+1}}{\partial{a_l}} $$
Hence, we rewrite the gradient w.r.t. the output of layer $l$ as the gradient w.r.t. the output of layer $l+1$, times the gradient of output of layer $l+1$ w.r.t. to the output of layer $l$, $\partial a_{l+1}/\partial{a_l}=\partial a_{l+1}/\partial{x_{l+1}}\footnotemark$. \footnotetext{Remember that $a_l=x_{l+1}$} As such, we can simply compute the gradients $\dfrac{\partial \mathcal{L}}{\partial{a_l}}, l=1, ..., L$ recursively: compute first the gradient $\partial \mathcal{L}/\partial{a_{L-1}}$ for layer $L-1$, then use $\partial \mathcal{L}/\partial{a_{L-1}}$ to compute the gradient $\partial \mathcal{L}/\partial{a_{L-2}}$ for layer $L-2$ and so on.
It is important here to draw the attention to $\partial a_l/\partial{x_l}$. On a first sight this a simple derivation. This is indeed the case, when a specific output dimension $a_l^j$ relates only to the respective input dimension $x_l^j$. Such a case is for \emph{elementwise} functions, like $a_j=h(x_j)=\tanh(x_j), \forall j$, or $a_j=h(x_j=\text{ReLU}(x_j), \forall j$, where the nonlinearities are applied on each dimension independently. However, there are functions and modules, where an output dimension relates to multiple input dimensions, $a_j=h(x_1, \dots, x_j, x_{j+1}, \dots)$. Such a a function is the softmax $$ a_j = \dfrac{\exp{x_j}}{\sum_i{\exp{x_i}}} $$ or the $\ell_2$ normalization function $$ a_j = \dfrac{x_j}{\sqrt{\sum_i{x_i^2}}} $$
As for these functions each output dimension relates to multiple input dimension, to compute $\partial a_l/\partial{x_l}$ actually means that one needs to compute the Jacobian matrix $$ \dfrac{\partial a_l}{\partial{x_l}}= \begin{bmatrix} \dfrac{\partial a_l^1}{\partial{x_l^1}} & \dfrac{\partial a_l^1}{\partial{x_l^2}} & \dots & \dfrac{\partial a_l^1}{\partial{x_l^n}} \\ \dfrac{\partial a_l^2}{\partial{x_l^1}} & \dfrac{\partial a_l^2}{\partial{x_l^2}} & \dots & \dfrac{\partial a_l^2}{\partial{x_l^n}} \\ \dots & \dots & \dots & \dots \\ \dfrac{\partial a_l^m}{\partial{x_l^1}} & \dfrac{\partial a_l^2}{\partial{x_l^2}} & \dots & \dfrac{\partial a_l^m}{\partial{x_l^n}} \end{bmatrix} $$ In that case to compute $\dfrac{\partial \mathcal{L}}{\partial{a_l}}$ we have to perform a matrix multiplication of the gradient vector $\dfrac{\partial \mathcal{L}}{\partial{a_{l+1}}}$ and the Jacobian $\dfrac{\partial a_{l+1}}{\partial{x_{l+1}}}$.
Although keeping up with all these operations can be confusing, a good way to make sure that all operations are done flawlessly is to resort to dimension analysis. More specifically, if the parameter matrix for layer $l$ have dimensions $d_{l-1} \times d_l$, then the gradient w.r.t. to the parameter matrix must also have the same dimensions.
Now that we have an easy way to compute $\dfrac{\partial \mathcal{L}}{\partial{a_l}}, l=1, ..., L$, and given that we can easily compute $\dfrac{\partial a_l}{\partial{\theta_l}}$ and $\partial a_{l+1}/\partial{x_{l+1}}$, we can summarize the backpropagation algorithm for computing the network gradients.
Step 1. Set $l=L$
Step 2. Set $\delta_L=\dfrac{\partial \mathcal{L}}{\partial{a_{L}}}=I$, since this operation practically stands for the derivative of a function with respect to its output.
Step 3. $l\leftarrow l-1$
Step 4. If convergence, exit.
Step 5. Compute $\dfrac{\partial \mathcal{L}}{\partial{a_l}}=\dfrac{\partial a_l}{\partial{x_l}}^T\delta_{l+1}$
Step 6. Set $\delta_l \leftarrow \dfrac{\partial \mathcal{L}}{\partial{a_{L}}}$
Step 7. Compute $\dfrac{\partial \mathcal{L}}{\partial{\theta_l}}=\dfrac{\partial \mathcal{L}}{\partial{a_l}} \dfrac{\partial a_l}{\partial{\theta_l}}$
Step 8. GOTO Step 3.
To implement a new module, you must first define the forward pass of the module, $a=h(x;\theta)$, where $a$ are the activations and $h(\cdot)$ the module function. Implementing the forward pass is relatively straightforward. Then, you must define the backward pass used by backpropagation to compute the gradients of the module with respect to the input $x$ and the module parameters $\theta$, if the module has trainable parameters. Implementing the backward pass is slighlty more complicated, as you need to calculate the module gradient first.
Hence, to implement our isolated module to backpropagate its gradients, what we really care about are:
Not ready yet.
Not ready yet.
Not ready yet.
Not ready yet.


Announcements
Using Generic RA
Members
Mailing lists
Discussion forums
IssueTracker
Downloads
Developing GenericRA
Build
CVS
Documents
Articles
Support


VMware Logo - 12 Signs You Are Too Big
I was downloading the VMware player today and noticed this agreement just to download the logo to link back to their site. How does stuff like this happen?
- You may only display the VMware Player icon on your Web site as designated herein, and not in any other manner. It is your responsibility to ensure that the link is always an active link to the VMware Player product page or the VMware Player download page or such other successor VMware website.
- The VMware Player icon must appear by itself on your web page and in accordance with the minimum spacing (the height of the Web icon) between each side of the icon.
- You may not alter the VMware Player icon in any manner, including, but not limited to, size, proportions, colors, elements or otherwise distort its perspective or appearance.
- You may not link to vmware.com from a Web site that is obscene or pornographic, or disparaging, defamatory, or libelous to VMware or any of its products.
- Your use may not directly or indirectly imply VMware's sponsorship, affiliation, or endorsement of your product or service or present misleading information about VMware products or services.
- Your use may not infringe any VMware intellectual property or other rights, may not violate any state or federal laws, and must comply with international IP laws.
- You may not frame VMware content.
- You may link to, but may not replicate, VMware content.
- Your reference to VMware, its products, and its Web site must comply with the general trademark guidelines as specified in the VMware Branding Guidelines.
- While you may use the Web icon as provided by these guidelines, you may not use the VMware corporate logo or any other VMware logo or graphic to link to VMware.com.
- These guidelines do not grant permission to use any other VMware logos or trademarks.
- VMware reserves the right in its sole discretion to terminate or modify your permission to display the VMware Player icon at any time.


|
UE4Bson
Unreal Engine 4 plugin for basic Bson support.
|
Holds and manages a Bson formatted data stream. More...
#include <BsonObject.h>
Public Member Functions | |
| const uint8_t * | GetDataPointer () const |
| FString | PrintAsCanonicalJson () const |
| FString | PrintAsJson () const |
| TSharedPtr< FBsonObject > | Copy () const |
| bool | Compare (const TSharedPtr< FBsonObject > &ToCompare) const |
| TSharedPtr< FBsonValue > | GetField (const FString &FieldName) const |
| TSharedPtr< FBsonValue > | TryGetField (const FString &FieldName) const |
| bool | HasField (const FString &FieldName) const |
| double | GetNumberField (const FString &FieldName) const |
| FORCEINLINE int32 | GetIntegerField (const FString &FieldName) const |
| bool | TryGetNumberField (const FString &FieldName, double &OutNumber) const |
| FString | GetStringField (const FString &FieldName) const |
| bool | TryGetStringField (const FString &FieldName, FString &OutString) const |
| bool | TryGetStringArrayField (const FString &FieldName, TArray< FString > &OutArray) const |
| bool | GetBoolField (const FString &FieldName) const |
| bool | TryGetBoolField (const FString &FieldName, bool &OutBool) const |
| const TArray< TSharedPtr< FBsonValue > > | GetArrayField (const FString &FieldName) const |
| bool | TryGetArrayField (const FString &FieldName, TArray< TSharedPtr< FBsonValue >> &OutArray) const |
| const TSharedPtr< FBsonObject > | GetObjectField (const FString &FieldName) const |
| void | SetField (const FString &FieldName, const TSharedPtr< FBsonValue > &Value) |
| void | SetNumberField (const FString &FieldName, double Number) |
| void | SetStringField (const FString &FieldName, const FString &StringValue) |
| void | SetBoolField (const FString &FieldName, bool Bool) |
| void | SetArrayField (const FString &FieldName, const TArray< TSharedPtr< FBsonValue > > &Array) |
| void | SetObjectField (const FString &FieldName, const TSharedPtr< FBsonObject > &Object) |
| bool | RemoveField (const FString &FieldName) |
Holds and manages a Bson formatted data stream.
This classes main purpose is to contain a hidden bson_t document. It can be edited and read (mostly) in known FJsonObject manner.
| bool FBsonObject::Compare | ( | const TSharedPtr< FBsonObject > & | ToCompare | ) | const |
Compares the contents of an FBsonObject to a given other one.
| ToCompare | a reference to the FBsonObject to match. |
| TSharedPtr<FBsonObject> FBsonObject::Copy | ( | ) | const |
Create a copy of this FBsonObject.
| const TArray<TSharedPtr<FBsonValue> > FBsonObject::GetArrayField | ( | const FString & | FieldName | ) | const |
Finds the field with the specified name and returns it as a TArray< TSharedPtr<FBsonValue> >.
Assumes that the field is present and is of type array.
| FieldName | The name of the field to get. |
| bool FBsonObject::GetBoolField | ( | const FString & | FieldName | ) | const |
Finds the field with the specified name and returns it as a boolean.
Assumes that the field is present and is of type boolean.
| FieldName | The name of the field to get. |
| const uint8_t* FBsonObject::GetDataPointer | ( | ) | const |
| TSharedPtr<FBsonValue> FBsonObject::GetField | ( | const FString & | FieldName | ) | const |
Returns a field with the given fieldname(key).
| FieldName | The name of the field to check. |
|
inline |
Finds the field with the specified name and returns it as an int32.
Assumes that the field is present and is of type castable to double.
| FieldName | The name of the field to get |
| double FBsonObject::GetNumberField | ( | const FString & | FieldName | ) | const |
Finds the field with the specified name and returns it as a double.
Assumes that the field is present and is of numerical type.
| FieldName | The name of the field to get. |
| const TSharedPtr<FBsonObject> FBsonObject::GetObjectField | ( | const FString & | FieldName | ) | const |
Finds the field with the specified name and returns it as a TSharedPtr<FBsonObject>.
Assumes that the field is present and is of type object.
| FieldName | The name of the field to get. |
| FString FBsonObject::GetStringField | ( | const FString & | FieldName | ) | const |
Finds the field with the specified name and returns it as an FString.
Assumes that the field is present and of type string.
| FieldName | The name of the field to get. |
| bool FBsonObject::HasField | ( | const FString & | FieldName | ) | const |
Checks whether the specified field exists.
| FieldName | The name of the field to check for. |
| FString FBsonObject::PrintAsCanonicalJson | ( | ) | const |
| FString FBsonObject::PrintAsJson | ( | ) | const |
| bool FBsonObject::RemoveField | ( | const FString & | FieldName | ) |
Removes the given field.
WARNING: INEFFICIENT!
| FieldName | The name of the field to remove. |
| void FBsonObject::SetArrayField | ( | const FString & | FieldName, |
| const TArray< TSharedPtr< FBsonValue > > & | Array | ||
| ) |
Adds a field of type Array.
| FieldName | The name to be given to the field (key). |
| Array | The desired value of the field (value). |
| void FBsonObject::SetBoolField | ( | const FString & | FieldName, |
| bool | Bool | ||
| ) |
Adds a field of type boolean.
| FieldName | The name to be given to the field (key). |
| Bool | The desired value of the field (value). |
| void FBsonObject::SetField | ( | const FString & | FieldName, |
| const TSharedPtr< FBsonValue > & | Value | ||
| ) |
Adds a field, automatically determines the type of the field to set from the given FBsonValue.
| FieldName | The name to be given to the field. |
| Value | A TSharedPtr<FBsonValue> containing the desired value of the field. |
| void FBsonObject::SetNumberField | ( | const FString & | FieldName, |
| double | Number | ||
| ) |
Adds a field of type double (number).
| FieldName | The name to be given to the field (key). |
| Number | The desired value of the field (value). |
| void FBsonObject::SetObjectField | ( | const FString & | FieldName, |
| const TSharedPtr< FBsonObject > & | Object | ||
| ) |
Adds a field of type double.
| FieldName | The name to be given to the field (key). |
| Object | The desired value of the field (value). |
| void FBsonObject::SetStringField | ( | const FString & | FieldName, |
| const FString & | StringValue | ||
| ) |
Adds a field of type string.
| FieldName | The name to be given to the field (key). |
| StringValue | The desired value of the field (value). |
| bool FBsonObject::TryGetArrayField | ( | const FString & | FieldName, |
| TArray< TSharedPtr< FBsonValue >> & | OutArray | ||
| ) | const |
Tries to find the field with the specified name and return it as a TArray< TSharedPtr<FBsonValue> >.
| FieldName | The name of the field to get. |
| OutNumber | A reference for the output to go in. |
| bool FBsonObject::TryGetBoolField | ( | const FString & | FieldName, |
| bool & | OutBool | ||
| ) | const |
Tries to find the field with the specified name and return it as a boolean.
| FieldName | The name of the field to get. |
| OutNumber | A reference for the output to go in. |
| TSharedPtr<FBsonValue> FBsonObject::TryGetField | ( | const FString & | FieldName | ) | const |
Tries to find the specified field and returns it.
| FieldName | The name of the field to find. |
| bool FBsonObject::TryGetNumberField | ( | const FString & | FieldName, |
| double & | OutNumber | ||
| ) | const |
Tries to find the field with the specified name and return it as a double.
| FieldName | The name of the field to get. |
| OutNumber | A reference for the output to go in. |
| bool FBsonObject::TryGetStringArrayField | ( | const FString & | FieldName, |
| TArray< FString > & | OutArray | ||
| ) | const |
Tries to find the field with the specified name and return it as an array of FStrings.
| FieldName | The name of the field to get. |
| OutNumber | A reference for the output to go in. |
| bool FBsonObject::TryGetStringField | ( | const FString & | FieldName, |
| FString & | OutString | ||
| ) | const |
Tries to find the field with the specified name and return it as a string.
| FieldName | The name of the field to get. |
| OutString | a reference for the output to go in. |


Providence to start regulating short-term rentals next week
by EndPlayPROVIDENCE, R.I. (AP) - The city of Providence will soon start regulating short-term housing rentals, including those booked through Airbnb.
The Providence Department of Planning and Development says a zoning ordinance regulating rentals takes effect Nov. 30.
A short-term rental is a property rented for 28 consecutive days or less.
The ordinance will require homeowners to obtain temporary use permits from the city before renting their property. There is no fee to obtain a permit.
The City Council passed the ordinance in May.
Head of Northeast Policy for Airbnb, Josh Meltzer, said the company is concerned the ordinance might financially hurt Airbnb hosts.
City Councilwoman Nirva LaFortune said people should have the right to rent out their homes but "not at the cost of our neighbors' quality of life."


Hello, My name is Charlie Glass, I am 24 years old, have an associates degree from bellevue college, and have years of customer service experience under my belt. My goal for this course is to learn how to code with an industry mindset, and to make connections to help me in my future career
- Always sunny in philidelphia
- The league
- The office
- Bob's Burgers
- How i met your mother
- The blacklist
- Prison break
- Parks and recreation


Manage loggers for a server.
A Fortissimo instance may have zero or more loggers. Loggers perform the standard task of handling messages that need recording for review by administrators.
The logger manager manages the various logging instances, delegating logging tasks.
Located in /Fortissimo.php (line 1993)
Build a new logger manager.
Get a logger.
Get all buffered log messages.
Some, but by no means all, loggers buffer messages for later retrieval. This method provides a way of retrieving all buffered messages from all buffering loggers. Messages are simply concatenated together from all of the available loggers.
To fetch the log messages of just one logger instead of all of them, use getLoggerByName(), and then call that logger's FortissimoLogger::getMessages() method.
Log messages.
Documentation generated on Mon, 04 Jan 2010 20:20:28 -0600 by phpDocumentor 1.4.3