“Post-implementation documentation”
As the lecturers maliciously called it when they checked university-grade projects, the idea of which was to plan a project, document it, and then implement it according to plan. In practice, I usually was implementing it first so it just worked, and then documented the result. And let anyone who hasn’t done that throw a stone first :)
Lecturers (in Polish universities) are often older people with no commercial experience, used to water-fall project planning. Now we are all Agile, we have continuous delivery, self-documenting code, and other aids.
Is there still a need to write a plan today before the actual implementation, or is it a relic of the past, art for art, and a university fad? Honestly, I would often be grateful even for quickly written post-implementation docs…
Tech Proposal
Tech Proposals
are a less formal way for internal documentation created before actual implementation. They also facilitate making technical decisions.
Creating such documents is useful when:
- we already have a ‘system’ and not just an ‘application’, i.e. multiple services and clients cooperating at different levels
- bigger changes must go through the architect or the security department (hello there, fintech)
- the new feature requires a variety of competence and cooperation between teams
- we work in a scattered and asynchronous environment
Seemingly, a lot may depend on the way work is organized, the number of teams and their composition, and the approach to the ownership of system fragments or procedures in the company.
But not completely. I worked in interdisciplinary (mobile, backend) and uniform (Android only) teams, in startups without procedures, and in more Byzantine structures. In fact, in every single place (with appropriate assumptions), the creation of a document that several people could edit, comment on, or simply review, significantly reduced the number of errors and misunderstandings.
There are places where an analyst (and/or PO, TL) prepares tickets and programmers simply execute them without going into the whole solution. And it might work. However, it is unlikely that anyone will grasp the nuances of all the technologies used with an extensive system. It is not difficult for me to imagine a situation wherein the middle of the implementation of some part of the solution an unforeseen problem comes out, and you have to go back to planning. Engaging developers sooner would have avoided such ping-pong.
I am aware that I promote a break away from programming in favor of additional bureaucracy and meetings. From my experience, at a certain scale and level of complexity of the project, it significantly saves time and work.
Preconditions
To even start thinking about documenting the implementation plan or consulting it with other teams, you need to know well in advance what should be delivered. Planning work with the appropriate level of detail for a year, quarter, or month ahead helps here. Transparency within the organization is also very important. I like to check in on other teams’ demos to find out what they are working on and what they are planning to do. I can immediately offer help in an area that I know better, instead of hoping that everything will go well or that they will find me themselves. And karma does come back, other programmers themselves offer help when my team comes into their area of expertise. Don’t get me wrong, there is no goodwill or empathy here, we just prefer to be in control when someone rummages in our code, instead of fixing someone else’s errors ASAP or being blocked until they fix it. The whole organization benefits from it.
If there is a deadline for introducing a complex change in 3 months, then you can slowly start to estimate the time-consumption, break the problem down into smaller parts, maybe you will have to do some spike, talk to people or teams with competence in the subject, consult the 3rd party providing e.g. hardware - documenting each stage will end with a ‘tech proposal’ for the solution with the biggest problems already solved.
Procedures in the organization may require approval of changes by the architect or the security department. It is much easier to discuss changes when having a document, maybe with a diagram or API proposal. An architect can look at the ‘tech proposal’ at any time convenient for them, for example, coming down from an ivory tower. It’s easier to sell your ideas when they’re presented in a digestible way. The security department will also quickly point to potential problems or places that require attention, having some outline of the solution, not bare code that they may not understand.
When is it worth creating a tech proposal
I’ve noticed that a good indicator is the emergence of doubts as to how to solve the problem or implement the feature. We don’t have a full image before writing the code, but sometimes we manage to identify the biggest gaps well in advance. Major changes may be beyond the competence of specific individuals or a team. Saved and shared information reduces the risk of time slips caused by staff shortages, the so-called bus factor. With well-written documentation and a committed team, each member can undertake the implementation of at least part of the solution, without having to wait until “that homie who knows what to do returns from vacation”.
Self-documenting code is awesome, but if your solution extends beyond a single repository, it will never contain the complete image. Then it is better to create a document and (if necessary) add a link in the comment. The need can appear during a code review when someone asks “why so?”. Instead of sending the same link to every potential questioner, it is better to put it in the code.
It is an unnecessary burden to create bloated documentation right at the start of a project. Then it is better to focus on using the Decision Log
to store information about why a particular technology or solution was chosen. Getting into the legacy project without such information may introduce reluctance towards your predecessors :)
What does a tech proposal consist of?
It depends on the problem, system, stakeholders… there is no single pattern.
I like it when it starts with the problem summary and the table of contents. This allows the reader to quickly find out if they are in the right place.
Then you can put in the introductory information that you have at the time of writing the document, links to other documentation or spike results. This section may grow with the time, don’t be focus on getting all the info at first. Just write down what you already know and add new info as it appears.
The most important part is an adequately detailed description of the solution. There are a few schools of thought: you can literally put pieces of code or SQL there, limit yourself to specifying an API, or a verbal/musical description of the implementation. Sequence or communication diagrams can be helpful and you don’t have to do them in a perfect UML, everyone will understand rectangles and arrows. In my opinion, the tech proposal should not focus on the implantation details, but rather on designing the logic and interaction between the various elements of the system. Implementation details will be a result of the adopted solution, organization or project standards, and the programmer’s preferences.
It’s also nice to have a few options for solving the problem and mark your choice in the summary, leaving the others in the document. This gives you a better picture of the thought process as you work on the right solution. Proposals for solutions may come from different people, or they can be a result of conversations with others, which helps you not to fall into the anchoring bias field. Sometimes it’s worth putting a ridiculous solution in there, just to see how others stand out. But it also happened to me that the best solution was a mix between the first and the ridiculous one because it allowed me to think outside the box. The description of each option may end with a small summary of advantages and disadvantages, even in the form of a table. Not all flaws will be discovered, but this document will help you historically understand the motivation behind your decision.
A ‘tech proposal’ written in this way naturally answers the questions:
- What will be implemented?
- Why?
- What other solutions were considered?
- Why they were not chosen?
Example document may look like this:
Abstract
Table of Contents
Info
Options
1.
2.
3.
Selected solution, summary, justification
Of course, in the future, it may turn out that the rejected solution would be better, but this is probably due to undiscovered conditions at the time of making the decision.
Updating the document
Tech Proposal
is a living document, it is constantly changing and developing. It’s good if everyone can add a comment, but the document should have a single owner who keeps order and composition.
After implementation, the document does not die, but I don’t see the need to update with every little change to the original solution. If the changes are large, they will probably require a new document, and you can link the previous one there. This document is for internal project documentation only and is valid only at the closure date. Documentation made available externally should be always kept up to date.
Summary
A properly developed tech proposal
allows not only to choose the best solution for the time being and gathered knowledge but also is a source of historical information on the development of the system. If you ever wondered “why exactly was something resolved this way” during your career, there would be an answer in the “tech proposal” (or “decision log”). The creation of such a document and teamwork on it is an additional effort that is justified only in certain cases. There is no reason to document every little change done to a single part of the system. However, it will be useful when the changes reach many services, exceed the team’s competencies, and maybe need to be coordinated over time.


pbalapra-software.bib
@url{automomml2016,
author = {P. Balaprakash and A. Tiwari and S. M. Wild and L. Carrington and P. D. Hovland},
date-added = {2019-02-10 21:03:47 +0000},
date-modified = {2019-02-10 21:03:47 +0000},
note = {AutoMOMML is an end-to-end, machine-learning-based framework to build predictive models for objectives such as performance, and power. The framework adopts statistical approaches to reduce the modeling complexity and automatically identifies and configures the most suitable learning algorithm to model the required objectives based on hardware and application signatures},
title = {{AutoMOMML: Automatic Multi-objective Modeling with Machine Learning}},
url = {https://xgitlab.cels.anl.gov/pbalapra/automomml},
year = {2016},
bdsk-url-1 = {http://dx.doi.org/10.1109/SC.Companion.2012.261}
}
@url{surf2015,
author = {P. Balaprakash},
date-added = {2017-01-08 17:29:35 +0000},
date-modified = {2017-09-22 16:34:17 +0000},
note = {SuRF is a model-based search module for automatic performance tuning. It adopts random forest supervised learning algorithm for modeling the performances as a function of input parameters within the search. SuRF samples a small number of parameter configurations, empirically evaluating the corresponding code variants to obtain the corresponding performance metrics, and fitting a surrogate model over the input-output space. The surrogate model is then iteratively refined by obtaining new output metrics at unevaluated input configurations predicted to be high-performing by the model. Implemented in Python and available with the Orio autotuning framework.},
title = {{SuRF}: {Search using Random Forest}},
url = {https://github.com/brnorris03/Orio/tree/master/orio/main/tuner/search/mlsearch},
year = {2015},
bdsk-url-1 = {http://dx.doi.org/10.1109/SC.Companion.2012.261}
}
@url{deephyper2018,
author = {P. Balaprakash and R. Egele and M. Salim and V. Vishwanath and S. M. Wild},
date-added = {2017-01-08 17:08:22 +0000},
date-modified = {2019-09-09 03:02:11 +0000},
note = {{DeepHyper: Scalable automated machine learning package with two components: 1) Reinforcement-learning-based neural architecture search for automatically constructing high-performing the deep neural network architecture; 2) Asynchronous model-based search for finding high-performing hyperparameters for deep neural networks.}},
title = {{DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks}},
url = {https://github.com/deephyper/deephyper},
year = {2018},
bdsk-url-1 = {http://dx.doi.org/10.1109/SC.Companion.2012.261}
}
@url{fpgaopt2016,
author = {P. Balaprakash and A. Mametjanov and C. Choudary and P. D. Hovland and S. M. Wild and G. Sabin},
date-added = {2016-07-11 01:08:34 +0000},
date-modified = {2016-07-30 21:04:42 +0000},
note = {A machine-learning-based approach to tune FPGA design parameters. It performs sampling-based reduction of the parameter space and guides the search toward promising parameter configurations.},
title = {{FPGAtuner: Autotuning FPGA Design Parameters for Performance and Power}},
url = {https://xgitlab.cels.anl.gov/pbalapra/fpgaopt},
year = {2016},
bdsk-url-1 = {http://dx.doi.org/10.1109/SC.Companion.2012.261}
}
@url{supuds2017,
author = {P. Agarwal and P. Balaprakash and S. Leyffer and S. M. Wild},
date-added = {2016-07-11 01:03:27 +0000},
date-modified = {2017-01-08 17:27:37 +0000},
note = {SPUDS is a machine-learning pipeline to build classification models to rank food establishments that are at most risk for the types of violations most likely to spread food-borne illness. The pipeline balances the the training data with resampling techniques, identifies the most important factors leading to critical violations via variable importance and variable selection methods, evaluates several state-of-the-art learning with cross validation, and finally combines the best performing ones via bagging. It is a customized version of AutoMOMML, exclusively designed for City of Chicago Smart Data Platform, and implemented in Python.},
title = {{SPUDS: Smart Pipeline for Urban Data Science}},
url = {https://xgitlab.cels.anl.gov/uda/ml-city},
year = {2017},
bdsk-url-1 = {http://dx.doi.org/10.1109/SC.Companion.2012.261}
}
@url{els-ptsp2009,
author = {P. Balaprakash and M. Birattari and T. St{\"u}tzle},
date-added = {2015-09-28 21:13:24 +0000},
date-modified = {2015-09-28 21:20:46 +0000},
note = {This software package provides a high-performance implementation of the estimation-based iterative improvement algorithm to tackle the probabilistic traveling salesman problem. A key novelty of the proposed algorithm is that the cost difference between two neighbor solutions is estimated by partial evaluation, adaptive, and importance sampling. Developed in C with GNU scientific library under Linux.},
title = {{ELS-PTSP}: {Estimation-based Local Search for the Probabilistic Traveling Salesman Problem}},
url = {https://github.com/pbalapra/els-ptsp},
year = {2009},
bdsk-url-1 = {http://dx.doi.org/10.1109/SC.Companion.2012.261}
}
@url{irace2010,
author = {M. L. Ibanez and J. D. Lacoste and T. St{\"u}tzle and M. Birattari and E. Yuan and P. Balaprakash},
date-added = {2015-09-28 21:10:03 +0000},
date-modified = {2015-09-28 21:22:39 +0000},
note = {The irace package implements the iterated racing procedure, which is an extension of the Iterated F-race procedure. Its main purpose is to automatically configure optimization algorithms by finding the most appropriate settings given a set of instances of an optimization problem. It builds upon the race package by Birattari, and it is implemented in R},
title = {{The irace Package}: {Iterated Race for Automatic Algorithm Configuration}},
url = {http://cran.r-project.org/web/packages/irace/},
year = {2010},
bdsk-url-1 = {http://dx.doi.org/10.1109/SC.Companion.2012.261}
}
@url{spapt2011,
author = {P. Balaprakash and S. M. Wild and B. Norris},
date-added = {2015-09-23 15:17:02 +0000},
date-modified = {2015-09-28 21:20:33 +0000},
note = {A set of extensible and portable search problems in automatic performance tuning whose goal is to aid in the development and improvement of search strategies and performance-improving transformations. SPAPT contains representative implementations from a number of lower-level, serial performance-tuning tasks in scientific applications. Available with the Orio autotuning framework.},
title = {{SPAPT}: {S}earch {P}roblems in {A}utomatic {P}erformance {T}uning},
url = {https://github.com/brnorris03/Orio/tree/master/testsuite/SPAPT},
year = {2011},
bdsk-url-1 = {http://dx.doi.org/10.1109/SC.Companion.2012.261}
}


A basic vanilla Javascript lightbox script with no dependencies and a single
Images: Add a data-lightbox
attribute to an image link.
...
YouTube: Same as images, except use a link that starts with https://www.youtube.com/watch
.
...Test this
Inspired by lightbox2.
Example image by Saira on Unsplash.


Checklist: Linux
__:___ read readme/instructions
Users and Groups
__:___ verify local user accounts
__:___ verify local user groups
__:___ disable guest access
__:___ review sudo permissions
Network resources
__:___ firewall (iptables)
__:___ monitor network connections (netstat)
Installation cleanup
__:___ check Applications
__:___ check Services
__:___ retrieve and install latest updates (apt-get)
Scans
__:___ chkrootkit
__:___ rkhunter
Filesystem
__:___ find and remove unauthorized files (e.g. pictures, music, programs)
__:___ check /etc/hosts file
__:___ check filesystem permissions
Configuration
__:___ install auditd
__:___ install boot-up manager
__:___ review cron jobs
__:___ install libpam-cracklib
__:___ password policy
__:___ account lockout policy
__:___ configure required services and programs
Logs
__:___ System logs


Last updated: 2021-04-20
Checks: 7 0
Knit directory: fastTopics-experiments/analysis/
This reproducible R Markdown analysis was created with workflowr (version <IP_ADDRESS>00). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.
Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.
Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.
The command set.seed(1)
was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.
Great job! Recording the operating system, R version, and package versions is critical for reproducibility.
Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.
Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.
Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.
The results in this page were generated with repository version b63fb95. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.
Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish
or wflow_git_commit
). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:
Ignored files:
Ignored: data/20news-bydate/
Ignored: data/droplet.RData
Ignored: data/nips_1-17.mat
Ignored: data/pbmc_68k.RData
Ignored: output/droplet/fits-droplet.RData
Ignored: output/newsgroups/fits-newsgroups.RData
Ignored: output/nips/fits-nips.RData
Ignored: output/pbmc68k/fits-pbmc68k.RData
Untracked files:
Untracked: plots/
Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.
These are the previous versions of the repository in which changes were made to the R Markdown (analysis/assess_fits_68k_pbmc.Rmd
) and HTML (docs/assess_fits_68k_pbmc.html
) files. If you’ve configured a remote Git repository (see ?wflow_git_remote
), click on the hyperlinks in the table below to view the files as they were in that past version.
| File | Version | Author | Date | Message |
|---|---|---|---|---|
| Rmd | b63fb95 | Peter Carbonetto | 2021-04-20 | workflowr::wflow_publish(“assess_fits_68k_pbmc.Rmd”) |
| html | b099034 | Peter Carbonetto | 2021-04-15 | Build site. |
| Rmd | 7d6db59 | Peter Carbonetto | 2021-04-15 | workflowr::wflow_publish(“assess_fits_68k_pbmc.Rmd”) |
| html | 5311f12 | Peter Carbonetto | 2021-04-15 | Added gene scatterplot to assess_fits_68k_pbmc analysis. |
| Rmd | 3aa9fe7 | Peter Carbonetto | 2021-04-15 | workflowr::wflow_publish(“assess_fits_68k_pbmc.Rmd”) |
| html | f533b4a | Peter Carbonetto | 2021-04-13 | Added cluster sizes to assess_fits_68k_pbmc output. |
| Rmd | f03c042 | Peter Carbonetto | 2021-04-13 | workflowr::wflow_publish(“assess_fits_68k_pbmc.Rmd”) |
| html | 72ca1a7 | Peter Carbonetto | 2021-04-12 | Adjusted structure plot in assess_fits_68k_pbmc analysis. |
| Rmd | dbb1b30 | Peter Carbonetto | 2021-04-12 | workflowr::wflow_publish(“assess_fits_68k_pbmc.Rmd”) |
| html | 37c49e6 | Peter Carbonetto | 2021-04-12 | Added Structure plot to assess_fits_68k_pbmc analysis. |
| Rmd | 8d5661e | Peter Carbonetto | 2021-04-12 | workflowr::wflow_publish(“assess_fits_68k_pbmc.Rmd”) |
| Rmd | ed52e14 | Peter Carbonetto | 2021-04-11 | Working on structure plot for 68k pbmc data. |
| html | 50a3fca | Peter Carbonetto | 2021-04-10 | Added loadings scatterplots to assess_fits_68k_pbmc analysis. |
| Rmd | 809b3ad | Peter Carbonetto | 2021-04-10 | workflowr::wflow_publish(“assess_fits_68k_pbmc.Rmd”) |
| html | 279ad71 | Peter Carbonetto | 2021-04-07 | Built assess_fits_68k_pbmc analysis. |
| Rmd | 978a10d | Peter Carbonetto | 2021-04-07 | workflowr::wflow_publish(“assess_fits_68k_pbmc.Rmd”) |
Here we compare the quality of the fits obtained from the different updates (EM and SCD, with and without extrapolation), and with different numbers of topics, \(K\).
Load the packages used in the analysis below, as well as some additional functions for creating the plots.
library(Matrix)
library(fastTopics)
library(ggplot2)
library(cowplot)
library(ggrepel)
set.seed(1)
Load the 68k PBMC data, and the results of running fit_poisson_nmf
on the 68k PBMC data, with different algorithms, and for various settings of \(K\).
load("../data/pbmc_68k.RData")
load("../output/pbmc68k/fits-pbmc68k.RData")
fits <- lapply(fits,poisson2multinom)
This plot shows the improvement in the log-likelihood as the rank, \(K\), is increased. The log-likelihoods are shown relative to the log-likelihood at \(K = 2\).
plot_loglik_vs_rank(fits) +
theme_cowplot(font_size = 12)
| Version | Author | Date |
|---|---|---|
| 279ad71 | Peter Carbonetto | 2021-04-07 |
The next set of plots shows the improvement in the fit over time, for \(K\) from 2 to 12, using EM or SCD, with and without extrapolation. The quality of the fit is measured by the log-likelihood relative to the best log-likelihood that was identified among all methods compared.
prune_prefit_iters <- function (fit) {
n <- nrow(fit$progress)
fit$progress <- fit$progress[1000:n,]
fit$progress <- transform(fit$progress,timing = timing/60^2)
return(fit)
}
create_progress_plot <- function (fits, k, y = "loglik")
plot_progress(fits,y = y,add.point.every = 100,shapes = 21,
colors = c("dodgerblue","red","dodgerblue","red"),
fills = c("dodgerblue","red","white","white")) +
scale_y_continuous(trans = "log10",breaks = 10^seq(-8,8)) +
guides(color = "none",fill = "none",size = "none",
shape = "none",linetype = "none") +
labs(x = "runtime (h)",title = paste("K =",k)) +
theme_cowplot(font_size = 10) +
theme(plot.title = element_text(size = 10,face = "plain"))
fits <- lapply(fits,prune_prefit_iters)
p <- vector("list",12)
for (i in 2:12)
p[[i]] <- create_progress_plot(fits[dat$k == i],i)
plot_grid(p[[2]],p[[3]],p[[4]],p[[5]],
p[[6]],p[[7]],p[[8]],p[[9]],
p[[10]],p[[11]],p[[12]],
nrow = 3,ncol = 4)
| Version | Author | Date |
|---|---|---|
| 279ad71 | Peter Carbonetto | 2021-04-07 |
These plots shows the evolution of the KKT residuals over time.
for (i in 2:12)
p[[i]] <- create_progress_plot(fits[dat$k == i],i,y = "res")
plot_grid(p[[2]],p[[3]],p[[4]],p[[5]],
p[[6]],p[[7]],p[[8]],p[[9]],
p[[10]],p[[11]],p[[12]],
nrow = 3,ncol = 4)
| Version | Author | Date |
|---|---|---|
| 279ad71 | Peter Carbonetto | 2021-04-07 |
An example in which the EM and (extrapolated) CD estimates greatly differ:
topic_colors <- c("#a6cee3","#1f78b4","#b2df8a","#33a02c",
"#fb9a99","#e31a1c","#fdbf6f")
fit1 <- fits[["fit-pbmc68k-em-k=7"]]
fit2 <- fits[["fit-pbmc68k-scd-ex-k=7"]]
n <- nrow(fit1$L)
pdat <- data.frame(x = as.vector(fit1$L),
y = as.vector(fit2$L),
k = factor(rep(1:7,each = n)))
pdat <- pdat[sample(7*n),]
p1 <- ggplot(pdat,aes(x = x,y = y,fill = k)) +
geom_point(color = "white",shape = 21,size = 2) +
scale_fill_manual(values = topic_colors) +
labs(x = "EM estimate",y = "extrapolated CD estimate") +
theme_cowplot(font_size = 12)
print(p1 + geom_abline(color = "black",linetype = "dotted"))
| Version | Author | Date |
|---|---|---|
| 50a3fca | Peter Carbonetto | 2021-04-10 |
An example in which the EM and (extrapolated) CD estimates largely agree:
topic_colors <- c("#8dd3c7","#ffffb3","#bebada","#fb8072","#80b1d3","#fdb462",
"#b3de69","#fccde5","#d9d9d9","#bc80bd","#ccebc5","#ffed6f")
fit1 <- fits[["fit-pbmc68k-em-k=12"]]
fit2 <- fits[["fit-pbmc68k-scd-ex-k=12"]]
n <- nrow(fit1$L)
pdat <- data.frame(x = as.vector(fit1$L),
y = as.vector(fit2$L),
k = factor(rep(1:12,each = n)))
pdat <- pdat[sample(12*n),]
p2 <- ggplot(pdat,aes(x = x,y = y,fill = k)) +
geom_point(color = "white",shape = 21,size = 2) +
scale_fill_manual(values = topic_colors) +
labs(x = "EM estimate",y = "extrapolated CD estimate") +
theme_cowplot(font_size = 12)
print(p2 + geom_abline(color = "black",linetype = "dotted"))
| Version | Author | Date |
|---|---|---|
| 50a3fca | Peter Carbonetto | 2021-04-10 |
From the PCs of the mixture proportions, we define four clusters, roughly corresponding to (1) B cells, (2) dendritic and CD14+ monocytes, (3) CD34+ cells, and (4) T cells and natural killer cells:
fit <- fits[["fit-pbmc68k-scd-ex-k=7"]]
pca <- prcomp(fit$L)$x
n <- nrow(pca)
x <- rep("T+NK",n)
pc4 <- pca[,4]
pc6 <- pca[,6]
x[pc4 > 0.25] <- "B"
x[pc4 < -0.25] <- "CD14+dendritic"
x[pc6 > 0.8] <- "CD34+"
clusters <- factor(x)
table(clusters)
# clusters
# B CD14+dendritic CD34+ T+NK
# 3733 4028 163 60655
The Structure plot summarizes the topic proportions in the cells. Here we subsample the larger clusters in order to better visualize the rarer cell types.
set.seed(1)
topics <- c(1:5,7,6)
topic_colors <- c("#a6cee3","#1f78b4","#b2df8a","#33a02c",
"#fb9a99","#e31a1c","#fdbf6f")
rows <- sort(c(sample(which(clusters == "B"),746),
sample(which(clusters == "CD14+dendritic"),806),
which(clusters == "CD34+"),
sample(which(clusters == "T+NK"),1213)))
p3 <- structure_plot(select_loadings(fit,loadings = rows),
grouping = clusters[rows],topics = topics,
colors = topic_colors[topics],n = Inf,perplexity = 70,
gap = 30,num_threads = 4,verbose = FALSE)
print(p3)
| Version | Author | Date |
|---|---|---|
| 72ca1a7 | Peter Carbonetto | 2021-04-12 |
Many of the “driving genes” in topic 6—that is, genes showing the largest increase in expression relative to other topics—are ribosomal protein genes:
pdat <- data.frame(gene = genes$symbol,
x = pmax(1e-6,apply(fit$F[,-6],1,max)),
y = pmax(1e-6,fit$F[,6]),
stringsAsFactors = FALSE)
pdat <- transform(pdat,fc = y/x)
pdat[with(pdat,!(fc > 1.6 & x > 0.0005)),"gene"] <- ""
p4 <- ggplot(pdat,aes(x = x,y = y,label = gene)) +
geom_point(color = "white",fill = "royalblue",shape = 21) +
geom_text_repel(color = "darkgray",size = 2.25,segment.color = "darkgray",
segment.size = 0.25,min.segment.length = 0,
max.overlaps = Inf) +
geom_abline(intercept = 0,slope = 1,color = "black",linetype = "dotted") +
scale_x_continuous(trans = "log10",limits=10^c(-6,-1),breaks=10^seq(-6,-1)) +
scale_y_continuous(trans = "log10",limits=10^c(-6,-1),breaks=10^seq(-6,-1)) +
labs(x = "highest frequency in another topic",
y = "frequency in topic 6") +
theme_cowplot(font_size = 10)
print(p4)
sessionInfo()
# R version 3.6.2 (2019-12-12)
# Platform: x86_64-apple-darwin15.6.0 (64-bit)
# Running under: macOS Catalina 10.15.7
#
# Matrix products: default
# BLAS: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
# LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
#
# locale:
# [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#
# attached base packages:
# [1] stats graphics grDevices utils datasets methods base
#
# other attached packages:
# [1] ggrepel_0.9.0 cowplot_1.0.0 ggplot2_3.3.0 fastTopics_0.5-24
# [5] Matrix_1.3-3
#
# loaded via a namespace (and not attached):
# [1] Rcpp_1.0.5 invgamma_1.1 lattice_0.20-38
# [4] tidyr_1.0.0 prettyunits_1.1.1 assertthat_0.2.1
# [7] zeallot_0.1.0 rprojroot_1.3-2 digest_0.6.23
# [10] truncnorm_1.0-8 R6_2.4.1 backports_1.1.5
# [13] MatrixModels_0.4-1 evaluate_0.14 coda_0.19-3
# [16] httr_1.4.2 pillar_1.4.3 progress_1.2.2
# [19] rlang_0.4.5 lazyeval_0.2.2 data.table_1.12.8
# [22] irlba_2.3.3 SparseM_1.78 whisker_0.4
# [25] rmarkdown_2.3 labeling_0.3 Rtsne_0.15
# [28] stringr_1.4.0 htmlwidgets_1.5.1 munsell_0.5.0
# [31] mixsqp_0.3-44 compiler_3.6.2 httpuv_1.5.2
# [34] xfun_0.11 pkgconfig_2.0.3 SQUAREM_2017.10-1
# [37] mcmc_0.9-6 htmltools_0.4.0 tidyselect_0.2.5
# [40] tibble_2.1.3 workflowr_<IP_ADDRESS>00 quadprog_1.5-8
# [43] viridisLite_0.3.0 withr_2.1.2 crayon_1.3.4
# [46] dplyr_0.8.3 later_1.0.0 MASS_7.3-51.4
# [49] grid_3.6.2 jsonlite_1.6 gtable_0.3.0
# [52] lifecycle_0.1.0 git2r_0.26.1 magrittr_1.5
# [55] scales_1.1.0 RcppParallel_4.4.2 stringi_1.4.3
# [58] farver_2.0.1 fs_1.3.1 promises_1.1.0
# [61] vctrs_0.2.1 tools_3.6.2 glue_1.3.1
# [64] purrr_0.3.3 hms_0.5.2 yaml_2.2.0
# [67] colorspace_1.4-1 ashr_2.2-51 plotly_4.9.2
# [70] knitr_1.26 quantreg_5.54 MCMCpack_1.4-5


Server¶
Server for DEVS simulation
-
class
server.
Server
(name, total_size)[source]¶ A server to host MPI, will delegate all of its calls to the active simulation kernel.
-
__init__
(name, total_size)[source]¶ Constructor
Parameters: - name – the name of the server, used for addressing (in MPI terms, this is the rank)
- total_size – the total size of the network in which the model lives
-
checkLoadCheckpoint
(name, gvt)[source]¶ Reconstruct the server from a checkpoint.
Parameters: - name – name of the checkpoint
- gvt – the GVT to restore to
Returns: bool – whether or not the checkpoint was successfully loaded
-
finishWaitingPool
()[source]¶ Stop the complete MPI request queue from blocking, used when stopping simulation is necessary while requests are still outstanding.
-
flushQueuedMessages
()[source]¶ Flush all queued messages to the controller. This will block until all of them are queued. It is required to flush all messages right after all of them happened and this should happen within the critical section!
-
getName
()[source]¶ Returns the name of the server
Is practically useless, since the server is previously addressed using its name. This does have a use as a ping function though.
-
getProxy
(rank)[source]¶ Get a proxy to a specified rank.
This rank is allowed to be the local server, in which case a local shortcut is created.
Parameters: rank – the rank to return a proxy to, should be an int Returns: proxy to the server, either of type MPIRedirect or LocalRedirect
-
listenMPI
()[source]¶ Listen for incomming MPI messages and process them as soon as they are received
-
loadCheckpoint
(name, gvt)[source]¶ Reconstruct the server from a checkpoint.
Parameters: - name – name of the checkpoint
- gvt – the GVT to restore to
-
prepare
(scheduler)[source]¶ Prepare the server to receive the complete model over MPI
Parameters: scheduler – the scheduler to use
-
processMPI
(data, comm, remote)[source]¶ Process an incomming MPI message and reply to it if necessary
Parameters: - data – the data that was received
- comm – the MPI COMM object
- remote – the location from where the message was received
-
queueMessage
(time, model_id, action)[source]¶ Queue a delayed action from being sent, to make it possible to batch them.
Will raise an exception if previous messages form a different time were not yet flushed! This flushing is not done automatically, as otherwise the data would be received at a further timestep which causes problems with the GVT algorithm.
Parameters: - time – the time at which the action happens
- model_id – the model_id that executed the action
- action – the action to execute (as a string)
-
saveAndProcessModel
(pickled_model, scheduler)[source]¶ Receive the model and set it on the server, but also saves it for further reinitialisation.
Parameters: - pickled_model – pickled representation of the model
- scheduler – the scheduler to use
-


Functions | |
| int32_t | CSL_fssGetRevision (const CSL_FssCfg *pFssCfg, uint32_t *pRev) |
| Return revision of the FSS IP module. More... | |
| int32_t | CSL_fssSelectInterfacePath (CSL_FssCfg *pFssCfg, uint32_t pathSelect) |
| Select the path (OSPI0 or HYPERBUS) for the interface to OSPI0 or HYPERBUS. More... | |
| int32_t | CSL_fssCfgEcc (CSL_FssCfg *pFssCfg, uint32_t ifSelect, bool bEccEnable, bool bIncludeBlockAddrInEcc) |
| Configure ECC support for an interface. More... | |
| int32_t | CSL_fssCfgHyperBusFragmentation (CSL_FssCfg *pFssCfg, uint32_t fragAddr, bool bFragLo, bool bFragHi) |
| Configure fragmentation for HyperBus. More... | |
| int32_t | CSL_fssCfgEccRegion (CSL_FssCfg *pFssCfg, uint32_t ifSelect, uint32_t regionNum, uint32_t regionAddrStart, uint32_t regionSizeBytes) |
| Configure a memory region for ECC protection. More... | |
| int32_t | CSL_fssSetIntrEnable (CSL_FssCfg *pFssCfg, uint32_t ifSelect, uint32_t intrSelect, bool bEnable) |
| Enable or disable the specified interrupt. More... | |
| int32_t | CSL_fssSetIntrPendingStatus (CSL_FssCfg *pFssCfg, uint32_t ifSelect, uint32_t intrSelect, bool bPend) |
| Set or clear the specified interrupt pending status. More... | |
| int32_t | CSL_fssIsIntrPending (const CSL_FssCfg *pFssCfg, uint32_t ifSelect, uint32_t intrSelect) |
| Determine if the specified masked interrupt is pending. More... | |
| int32_t | CSL_fssIsRawIntrPending (const CSL_FssCfg *pFssCfg, uint32_t ifSelect, uint32_t intrSelect) |
| Determine if the specified raw interrupt is pending. More... | |
| int32_t | CSL_fssAckIntr (CSL_FssCfg *pFssCfg, uint32_t ifSelect, uint32_t outIntrSelect) |
| Acknowledge completion of an ouput interrupt. More... | |
| int32_t | CSL_fssGetEccBlockErrorInfo (CSL_FssCfg *pFssCfg, uint32_t ifSelect, CSL_FssEccBlockErrorInfo *pEccErrorInfo) |
| Get information about an ECC block error. More... | |
| int32_t | CSL_fssGetWrErrorInfo (CSL_FssCfg *pFssCfg, uint32_t ifSelect, CSL_FssWrErrorInfo *pWrErrorInfo) |
| Get information about a write error. More... | |
| int32_t | CSL_fssOspiSetXipPrefetchEnable (CSL_FssCfg *pFssCfg, uint32_t ifSelect, bool bEnable) |
| Enable or disable OSPI XIP prefetching. More... | |
| int32_t | CSL_fssOspiSetDdrModeEnable (CSL_FssCfg *pFssCfg, uint32_t ifSelect, bool bEnable) |
| Enable or disable OSPI DDR mode. More... | |
| int32_t | CSL_fssOtfaGetRevision (const CSL_FssCfg *pFssCfg, uint32_t *pRev) |
| Return revision of the FSS OTFA module. More... | |
| int32_t | CSL_fssOtfaIsCryptoAvailable (const CSL_FssCfg *pFssCfg) |
| Determine if the encryption/authentication function is enabled. More... | |
| int32_t | CSL_fssOtfaIsBusy (const CSL_FssCfg *pFssCfg) |
| Determine if the OTFA is busy. More... | |
| int32_t | CSL_fssOtfaIsCryptoBusy (const CSL_FssCfg *pFssCfg) |
| Determine if the OTFA crypto engine is busy. More... | |
| int32_t | CSL_fssOtfaGetPipeStallEventCnts (CSL_FssCfg *pFssCfg, uint32_t *pRdStallEventCnt, uint32_t *pWrStallEventCnt) |
| Read pipe stall event counters. More... | |
| int32_t | CSL_fssOtfaCfgCrypto (CSL_FssCfg *pFssCfg, const CSL_FssOtfaCryptoCfg *pCryptoCfg) |
| Configure the OTFA crypto engine. More... | |
| int32_t | CSL_fssOtfaCfgCryptoRegion (CSL_FssCfg *pFssCfg, uint32_t regionNum, const CSL_FssOtfaCryptoRegionCfg *pRegionCfg) |
| Configure a crypto memory region. More... | |
| int32_t | CSL_fssOtfaSetCryptoEnable (CSL_FssCfg *pFssCfg, bool bEnable) |
| Enable or disable the OTFA crypto functionality. More... | |
| int32_t | CSL_fssOtfaSetCryptoPause (CSL_FssCfg *pFssCfg, bool bPause) |
| Pause or resume the OTFA crypto functionality. More... | |
| int32_t | CSL_fssOtfaIsCryptoEnabled (const CSL_FssCfg *pFssCfg) |
| Determine if the OTFA crypto functionality is enabled. More... | |
| int32_t | CSL_fssOtfaSetIntrEnable (CSL_FssCfg *pFssCfg, uint32_t regionSelect, uint32_t intrSelect, bool bEnable) |
| Enable or disable the specified OTFA interrupt. More... | |
| int32_t | CSL_fssOtfaSetIntrPendingStatus (CSL_FssCfg *pFssCfg, uint32_t regionSelect, uint32_t intrSelect, bool bPend) |
| Set or clear the specified OTFA interrupt pending status. More... | |
| int32_t | CSL_fssOtfaIsIntrPending (const CSL_FssCfg *pFssCfg, uint32_t regionSelect, uint32_t intrSelect) |
| Determine if the specified OTFA masked interrupt is pending. More... | |
| int32_t | CSL_fssOtfaIsRawIntrPending (const CSL_FssCfg *pFssCfg, uint32_t regionSelect, uint32_t intrSelect) |
| Determine if the specified OTFA raw interrupt is pending. More... | |
| int32_t | CSL_fssOtfaGetIrqInfo (const CSL_FssCfg *pFssCfg, CSL_FssOtfaIrqInfo *pIrqInfo) |
| Get additional OTFA IRQ information. More... | |
| int32_t | CSL_fssOtfaGetStats (CSL_FssCfg *pFssCfg, CSL_FssOtfaStats *pStats) |
| Get OTFA statistics. More... | |
===========================================================================
| int32_t CSL_fssAckIntr | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| uint32_t | outIntrSelect | ||
| ) |
Acknowledge completion of an ouput interrupt.
This function is used to acknowledge completion of the specified interface's output interrupt. Another output interrupt will be triggered if enabled and one or more of the output interrupt's sources remain active.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| outIntrSelect | [IN] Output interrupt selector (CSL_FssOutputInterruptSelect) |
| int32_t CSL_fssCfgEcc | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| bool | bEccEnable, | ||
| bool | bIncludeBlockAddrInEcc | ||
| ) |
Configure ECC support for an interface.
This function configures the ECC support for the interface specified by ifSelect.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| bEccEnable | [IN] If true, ECC is enabled for this interface. Otherwise, ECC is disabled. |
| bIncludeBlockAddrInEcc | [IN] If true, the block address is included within the ECC calculation. Otherwise, the block address is excluded from the ECC calculation. |
| int32_t CSL_fssCfgEccRegion | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| uint32_t | regionNum, | ||
| uint32_t | regionAddrStart, | ||
| uint32_t | regionSizeBytes | ||
| ) |
Configure a memory region for ECC protection.
This function configures the specified interface's memory region for ECC protection. Up to 4 ECC protected memory regions can be configured.
This function will automatically truncate the regionAddrStart parameter to the required 4Kbyte aligned boundary.
This function will automatically round up the regionSizeBytes value to the required multiple of 4Kbytes size. Specifying a regionSizeBytes of 0 will disable the region from ECC protection.
An CSL_EINVALID_PARAMS error is returned if the truncated regionAddrStart
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| regionNum | [IN] Region number (0..(CSL_FSS_MAX_ECC_PROTECTED_MEMORY_REGIONS-1)) |
| regionAddrStart | [IN] The starting byte address of the ECC protected memory region |
| regionSizeBytes | [IN] The size in bytes of the ECC protected memory region |
| int32_t CSL_fssCfgHyperBusFragmentation | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | fragAddr, | ||
| bool | bFragLo, | ||
| bool | bFragHi | ||
| ) |
Configure fragmentation for HyperBus.
This function configures the Dynamic Fragmenter module for HyperBus.
The Dynamic Fragmenter module is responsible for fragmenting write data to the HyperBus (it will pass all other transactions through unaffected).
HyperBus Flash has a requirement that all writes to the flash region are done in 16bit chunks. HyperBus SRAM does not have this write fragmentation requirement.
The Dynamic Fragmenter module is configured to fragment all addresses by default. You can use this function to disable fragmentation for HyperBus SRAM areas.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| fragAddr | [IN] This is the address of a request that the bFragLo or bFragHi boundary determination occurs |
| bFragLo | [IN] If true, any address less than frag_addr will be fragmented to 16 bits |
| bFragHi | [IN] If true, any address greater than or equal to fragAddr will be fragmented to 16 bits |
| int32_t CSL_fssGetEccBlockErrorInfo | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| CSL_FssEccBlockErrorInfo * | pEccErrorInfo | ||
| ) |
Get information about an ECC block error.
This function returns information about an ECC block error (if such an error exists) on the specified interface.
If error information exists, this function will automatically pop this error information off the top of the error stack in preparation for the next error.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| pEccErrorInfo | [OUT] Pointer to a CSL_FssEccBlockErrorInfo structure |
| int32_t CSL_fssGetRevision | ( | const CSL_FssCfg * | pFssCfg, |
| uint32_t * | pRev | ||
| ) |
Return revision of the FSS IP module.
This function returns the contents of the FSS revision register. Consult the FSS module documentation for a description of the contents of the revision register.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| pRev | [OUT] Pointer to uint32_t where revision is returned |
| int32_t CSL_fssGetWrErrorInfo | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| CSL_FssWrErrorInfo * | pWrErrorInfo | ||
| ) |
Get information about a write error.
This function returns information about a write error (if such an error exists) on the specified interface.
If error information exists, this function will automatically pop this error information off the top of the error stack in preparation for the next error.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| pWrErrorInfo | [OUT] Pointer to a CSL_FssWrErrorInfo structure |
| int32_t CSL_fssIsIntrPending | ( | const CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| uint32_t | intrSelect | ||
| ) |
Determine if the specified masked interrupt is pending.
This function returns the pending masked interrupt status of the specified interface's interrupt. Note that this indication takes into account both the raw interrupt status and the interrupt enable.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| intrSelect | [IN] Interrupt selector (CSL_FssInterruptSourceSelect) |
| int32_t CSL_fssIsRawIntrPending | ( | const CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| uint32_t | intrSelect | ||
| ) |
Determine if the specified raw interrupt is pending.
This function returns the pending raw interrupt status of the specified interface's interrupt. Note that this indication takes into account only the raw interrupt status and not the interrupt enable.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| intrSelect | [IN] Interrupt selector (CSL_FssInterruptSourceSelect) |
| int32_t CSL_fssOspiSetDdrModeEnable | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| bool | bEnable | ||
| ) |
Enable or disable OSPI DDR mode.
This function enables or disables OSPI DDR mode.
When OSPI DDR mode is enabled, all OSPI read requests are turned into 2- byte aligned, even length, requests due to the OSPI limitation on DDR mode. In other words, the fss makes the OSPI address and length even and inclusive of the requested bytes.
When OSPI DDR mode is disabled, all OSPI read requests are passed through unchanged.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| bEnable | [IN] If true, OSPI DDR mode is enabled. If false, it is disabled. |
| int32_t CSL_fssOspiSetXipPrefetchEnable | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| bool | bEnable | ||
| ) |
Enable or disable OSPI XIP prefetching.
This function enables or disables OSPI execute-in-place (XIP) prefetching.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| bEnable | [IN] If true, XIP prefetching is enabled. If false, it is disabled. |
| int32_t CSL_fssOtfaCfgCrypto | ( | CSL_FssCfg * | pFssCfg, |
| const CSL_FssOtfaCryptoCfg * | pCryptoCfg | ||
| ) |
Configure the OTFA crypto engine.
This function is used to configure the OTFA crypto engine. It must be called before enabling the crypto engine via the CSL_fssOtfaSetCryptoEnable function.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| pCryptoCfg | [IN] Pointer to a CSL_FssOtfaCryptoCfg structure containing configuration parameters |
| int32_t CSL_fssOtfaCfgCryptoRegion | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | regionNum, | ||
| const CSL_FssOtfaCryptoRegionCfg * | pRegionCfg | ||
| ) |
Configure a crypto memory region.
This function configures the specified memory region for crypto protection. Up to 4 protected memory regions can be configured.
An CSL_EFAIL error is returned if this function is called while OTFA transactions are active. Call the CSL_fssOtfaIsBusy until it returns false before calling this function.
This function will automatically truncate the pRegionCfg->regionAddrStart and pRegionCfg->macSignaturesBaseAddr parameters to the required 4Kbyte aligned boundary.
This function will automatically round up the pRegionCfg->regionSizeBytes value to the required multiple of 4Kbytes size. Specifying a regionSizeBytes of 0 will disable the region from crypto protection.
An CSL_EINVALID_PARAMS error is returned under the following conditions: a) if the truncated regionAddrStart + rounded-up regionSizeBytes is
4Gbytes, as wrap-around addressing is not supported.
b) If the truncated macSignaturesBaseAddr + ((rounded-up regionSizeBytes)/8)
4Gbytes, as wrap-around addressing is not supported.
c) If the specified pRegionCfg->macMode and pRegionCfg->encMode result in an invalid combination per the table below:
+------------------------------+ | encMode | +----------+---------+---------+ | DISABLED | AES_CTR | AES_ECB |
+------—+-------—+-------—+------—+------—+ | | DISABLED | OK | OK | OK | | +-------—+-------—+------—+------—+ | macMode | GMAC | OK | OK | INVALID | | +-------—+-------—+------—+------—+ | | CBC-MAC | INVALID | OK | INVALID | +------—+-------—+-------—+------—+------—+
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| regionNum | [IN] Region number (0..(CSL_FSS_OTFA_MAX_PROTECTED_MEMORY_REGIONS-1)) |
| pRegionCfg | [IN] Pointer to a CSL_FssOtfaCryptoRegionCfg structure containing configuration parameters |
| int32_t CSL_fssOtfaGetIrqInfo | ( | const CSL_FssCfg * | pFssCfg, |
| CSL_FssOtfaIrqInfo * | pIrqInfo | ||
| ) |
Get additional OTFA IRQ information.
This function returns additional OTFA IRQ information following an IRQ event.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| pIrqInfo | [OUT] Pointer to a CSL_FssOtfaIrqInfo structure where the IRQ information is returned |
| int32_t CSL_fssOtfaGetPipeStallEventCnts | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t * | pRdStallEventCnt, | ||
| uint32_t * | pWrStallEventCnt | ||
| ) |
Read pipe stall event counters.
This function returns and clears the current pipe stall event counters. These counters can be used to fine tune the AES core allocation to the read and write pipes per the CSL_fssOtfaCfgCrypto function.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| pRdStallEventCnt | [OUT] Pointer to uint32_t where the read pipe stall event counter is returned |
| pWrStallEventCnt | [OUT] Pointer to uint32_t where the write pipe stall event counter is returned |
| int32_t CSL_fssOtfaGetRevision | ( | const CSL_FssCfg * | pFssCfg, |
| uint32_t * | pRev | ||
| ) |
Return revision of the FSS OTFA module.
This function returns the contents of the FSS OTFA revision register. Consult the OTFA module documentation for a description of the contents of the revision register.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| pRev | [OUT] Pointer to uint32_t where revision is returned |
| int32_t CSL_fssOtfaGetStats | ( | CSL_FssCfg * | pFssCfg, |
| CSL_FssOtfaStats * | pStats | ||
| ) |
Get OTFA statistics.
This function returns OTFA statistic counts. The hardware statistic counters are automatically cleared by this function.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| pStats | [OUT] Pointer to a CSL_FssOtfaStats structure where the statistic counters are returned |
| int32_t CSL_fssOtfaIsBusy | ( | const CSL_FssCfg * | pFssCfg | ) |
Determine if the OTFA is busy.
This function returns a status indicating if the OTFA is busy (one or more crypto or non-crypto transactions are active) or not busy (no crypto or non-crypto transactions are active).
Software needs to read 2 consecutive falses with at least 100ns delay between samples to insure the pipe is fully empty.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| int32_t CSL_fssOtfaIsCryptoAvailable | ( | const CSL_FssCfg * | pFssCfg | ) |
Determine if the encryption/authentication function is enabled.
This function returns a boolean true or false indicating if the encryption/authentication function is hardware enabled (via EFUSE) for the interface to OSPI0 or HYPERBUS.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| int32_t CSL_fssOtfaIsCryptoBusy | ( | const CSL_FssCfg * | pFssCfg | ) |
Determine if the OTFA crypto engine is busy.
This function returns a status indicating if the OTFA crypto engine is busy (one or more crypto transactions are active) or not busy (no crypto transactions are active).
Software needs to read 2 consecutive falses with at least 100ns delay between samples to insure the pipe is fully empty.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| int32_t CSL_fssOtfaIsCryptoEnabled | ( | const CSL_FssCfg * | pFssCfg | ) |
Determine if the OTFA crypto functionality is enabled.
This function returns a status indicating if the OTFA crypto functionality is enabled or disabled.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| int32_t CSL_fssOtfaIsIntrPending | ( | const CSL_FssCfg * | pFssCfg, |
| uint32_t | regionSelect, | ||
| uint32_t | intrSelect | ||
| ) |
Determine if the specified OTFA masked interrupt is pending.
This function returns the pending masked interrupt status of the specified OTFA region's interrupt. Note that this indication takes into account both the raw interrupt status and the interrupt enable.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| regionSelect | [IN] Region selector (0..(CSL_FSS_OTFA_MAX_PROTECTED_MEMORY_REGIONS-1)) |
| intrSelect | [IN] Interrupt selector (CSL_FssOtfaInterruptSourceSelect) |
| int32_t CSL_fssOtfaIsRawIntrPending | ( | const CSL_FssCfg * | pFssCfg, |
| uint32_t | regionSelect, | ||
| uint32_t | intrSelect | ||
| ) |
Determine if the specified OTFA raw interrupt is pending.
This function returns the pending raw interrupt status of the specified OTFA region's interrupt. Note that this indication takes into account only the raw interrupt status and not the interrupt enable.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| regionSelect | [IN] Region selector (0..(CSL_FSS_OTFA_MAX_PROTECTED_MEMORY_REGIONS-1)) |
| intrSelect | [IN] Interrupt selector (CSL_FssOtfaInterruptSourceSelect) |
| int32_t CSL_fssOtfaSetCryptoEnable | ( | CSL_FssCfg * | pFssCfg, |
| bool | bEnable | ||
| ) |
Enable or disable the OTFA crypto functionality.
This function enables or disables the OTFA crypto functionality.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| bEnable | [IN] If true, the OTFA crypto functionslity is enabled, otherwise it is disabled (bypassed) |
| int32_t CSL_fssOtfaSetCryptoPause | ( | CSL_FssCfg * | pFssCfg, |
| bool | bPause | ||
| ) |
Pause or resume the OTFA crypto functionality.
This function pauses or resumes the OTFA crypto functionality. The crypto functionality must be paused before software can safely change the OTFA configuration.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| bPause | [IN] If true, the OTFA crypto functionality is paused, otherwise it is resumed |
| int32_t CSL_fssOtfaSetIntrEnable | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | regionSelect, | ||
| uint32_t | intrSelect, | ||
| bool | bEnable | ||
| ) |
Enable or disable the specified OTFA interrupt.
This function enables or disables generation of the specified OTFA region's interrupt.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| regionSelect | [IN] Region selector (0..(CSL_FSS_OTFA_MAX_PROTECTED_MEMORY_REGIONS-1)) |
| intrSelect | [IN] Interrupt selector (CSL_FssOtfaInterruptSourceSelect) |
| bEnable | [IN] If true, the interrupt generation is enabled, otherwise it is disabled |
| int32_t CSL_fssOtfaSetIntrPendingStatus | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | regionSelect, | ||
| uint32_t | intrSelect, | ||
| bool | bPend | ||
| ) |
Set or clear the specified OTFA interrupt pending status.
This function is used to set or clear the pending status of the specified OTFA region's interrupt.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| regionSelect | [IN] Region selector (0..(CSL_FSS_OTFA_MAX_PROTECTED_MEMORY_REGIONS-1)) |
| intrSelect | [IN] Interrupt selector (CSL_FssOtfaInterruptSourceSelect) |
| bPend | [IN] If true, the interrupt pending status is set, otherwise it is cleared |
| int32_t CSL_fssSelectInterfacePath | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | pathSelect | ||
| ) |
Select the path (OSPI0 or HYPERBUS) for the interface to OSPI0 or HYPERBUS.
This function selects to path (OSPI0 or HYPERBUS) for the interface conntected to OSPI0 or HYPERBUS.
The path will not be switched until traffic through the interface is idle. It is the responsibility of software to insure interface traffic is stopped before calling this function to select a different path.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| pathSelect | [IN] Path selector (CSL_FssFsasInterfacePathSelect) |
| int32_t CSL_fssSetIntrEnable | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| uint32_t | intrSelect, | ||
| bool | bEnable | ||
| ) |
Enable or disable the specified interrupt.
This function enables or disables generation of the specified interface's interrupt.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| intrSelect | [IN] Interrupt selector (CSL_FssInterruptSourceSelect) |
| bEnable | [IN] If true, the interrupt generation is enabled, otherwise it is disabled |
| int32_t CSL_fssSetIntrPendingStatus | ( | CSL_FssCfg * | pFssCfg, |
| uint32_t | ifSelect, | ||
| uint32_t | intrSelect, | ||
| bool | bPend | ||
| ) |
Set or clear the specified interrupt pending status.
This function is used to set or clear the pending status of the specified interface's interrupt.
| pFssCfg | [IN] Pointer to the CSL_FssCfg configuration structure |
| ifSelect | [IN] Interface selector (CSL_FssInterfaceSelect) |
| intrSelect | [IN] Interrupt selector (CSL_FssInterruptSourceSelect) |
| bPend | [IN] If true, the interrupt pending status is set, otherwise it is cleared |


Trail Use Guidelines
- Hiking, horseback riding, nature study, and other low-impact activities are welcome throughout the trail network
- All motorized use is prohibited
- No hunting on most trails (look for signage and wear blaze orange during hunting season on trails that permit hunting) Visit Hunting, Fishing, and Trapping Seasons for the state hunting season calendar.
- No littering or dumping
- No fires
- No camping
- No alcoholic beverages
- Do not feed, approach or touch wildlife, including waterfowl
- Take personal protection measures to reduce your exposure to ticks and mosquito bites
- Please pick up after your dog and properly dispose of the waste


In previous post, we saw how to consume and process fixed width file using DataWeave in Mule ESB 3.7. In this post we will see how we can generate fixed width file using DataWeave in Mule 3.7. As I mentioned in previous post, DataWeave support for fixed width files has been introduced in Mule ESB 3.8 but It may not be always possible to upgrade our ESB version just for this purpose. That is why we will see how to achieve this transformation in 3.7.
There are third party java libraries that can be used with Mule to process fixed width files. But what if we don’t want to use any other libraries? Well, there is a way to generate fixed width files with just DataWeave.
Let’s refer to our previous example and we will use the output of that flow to regenerate our original-like fixed width file.
Input:
Here is the JSON that we generated in previous example, we will use that as Input in this example and then we will convert it into fixed width.
[ { "Id": "001", "Day": "MON", "Shout": "STARTWORK" }, { "Id": "001", "Day": "TUE", "Shout": "PROGRESSWORK" }, { "Id": "001", "Day": "WED", "Shout": "TAKEABREAK" }, { "Id": "001", "Day": "THU", "Shout": "RESUMEWORK" }, { "Id": "001", "Day": "FRI", "Shout": "FINISHWORK" }, { "Id": "001", "Day": "SAT", "Shout": "CHILLDOWN" }, { "Id": "001", "Day": "SUN", "Shout": "GRABABEER" } ]
DataWeave Processing:
Here is a DataWeave code that would transform this JSON into fixed width text file –
%dw 1.0 %output application/csv header=false --- (payload map { Id: rightPad($.Id,3,' '), Day: rightPad($.Day,4,' '), Shout: rightPad($.Shout,20,' ') }) map { row: ($ pluck $ joinBy "") }
Let’s understand how this code works –
- Output directive defines output to be application/csv and we have turned off header in output.
- First we will map the payload to format the values as we require in our output. There is no in-built rightPad function in DataWeave so we will have to write a MEL global function in our mule-config.xml. Add below code to your mule config –
<configuration doc:name="Configuration"> <expression-language> <global-functions> def rightPad(value,length,padChar) { return org.apache.commons.lang3.StringUtils.rightPad(value,length,padChar); } </global-functions> </expression-language> </configuration>
Now this MEL function can be used in our DataWeave. What it does is, for given string value, it appends the padChar to make value of given length. Eg. Shout field should be of length 20 characters in output and if required pad input with ‘ ‘ to make it 20 characters i.e. fixed length 🙂
- Then we apply second map to iterate over object collection we prepared in step 2. For every object, we will create a object with single property of key ‘row’ and value would be constructed by joining all the values of every property in object. If we don’t turn off headers in output, very first line in our output will contain ‘row’.
###### Output:
Here is the output of above transformation –
001MON STARTWORK 001TUE PROGRESSWORK 001WED TAKEABREAK 001THU RESUMEWORK 001FRI FINISHWORK 001SAT CHILLDOWN 001SUN GRABABEER
Record Structure we generated is –
Id - Start Position: 0, length: 3 Day - Start Position: 3, length: 4 Shout - Start Position:7, length: 20
If you want to add a prefix or suffix to every column, then it can be done by adding mapObject function before pluck. For example, if we change mapObject to include two fields, prefix and suffix, then our output will look like shown below. Remember, you can also do it in your first map where we make each property of fixed length.
mapObject Code:
%dw 1.0 %output application/csv header=false --- (payload map { Id: rightPad($.Id,3,' '), Day: rightPad($.Day,4,' '), Shout: rightPad($.Shout,20,' ') }) map { row: ($ mapObject { Suffix:'Suffix', '$$':$, Post:'Prefix' } pluck $ joinBy "") }
Here is the output of above change in DataWeave code, Note the static strings are around each column –
Suffix001PrefixSuffixMON PrefixSuffixSTARTWORK Prefix Suffix001PrefixSuffixTUE PrefixSuffixPROGRESSWORK Prefix Suffix001PrefixSuffixWED PrefixSuffixTAKEABREAK Prefix Suffix001PrefixSuffixTHU PrefixSuffixRESUMEWORK Prefix Suffix001PrefixSuffixFRI PrefixSuffixFINISHWORK Prefix Suffix001PrefixSuffixSAT PrefixSuffixCHILLDOWN Prefix Suffix001PrefixSuffixSUN PrefixSuffixGRABABEER Prefix
That’s all Folks! Here we saw, how we can use DataWeave code in Mule 3.7 to convert any file format into a fixed width text format and also how to apply formatting on each column.
Feel free to let me know what you think or any questions, I will be happy to help.


Cybersecurity is stronger when we all do our part to keep the internet safe.  By practicing strong and safe online habits, you can better protect your identity and the networks you use at home, at work and anywhere you log on. Join Library IT Manager Lynne Sansevero in one of four cybersecurity workshops designed to help you connect online safely. These four workshops were recorded in January, 2015 at the Santa Cruz Public Libraries. The classes are specifically tailored for Parents & Educators, Kids & Teens, Older Community Members & Seniors, and Small Business Owners. More resources to keep you secure online are available at stopthinkconnect.org


surveillance human number ZIKV Zika mosquito virus aegypti positive transmission data infections pools incidence testing Ae. probability Caguas detection mosquitoes humans cases population week {border-style Puerto []. patients px;} strategies #ffffff;} Aedes RT-PCR infected infection traps vector expected detecting emergency estimated used detected people virological .%) ;():–. Disease Health Sciences systems (.%, (., Medicine higher sensitivity Biology detect observed simulation tests Panel Rico assumed care department different high low pool simulated study surveillance. total Control UI: based confirmed estimate increased model rate simulations (Fig Caguas, Infectious Virus calculated clinical female models px; relationship represent seeking trapping uncertainty {font-family Available Life Tropical [cited additional collected compared effective font-size from: result shows strategy using Caguas. October Pathogens Surveillance Vectors approximately arbovirus area case chikungunya double; infections. mean monitoring numbers product reported required review scenarios symptoms symptoms. test trap true week. weekly American CI: Fig However, Medical Molecular Mosquito Nile Rico, United ZIKV-infected activity. actual assay. autocidal bands border-color border-width distribution epidemic gravid health local previous resource routine scenarios, seek solid; tested testing. viruses {font-style {font-weight Centers Chain Diseases Diseases. Entomology. Flaviviruses: Human Insect Mosquitoes Neglected Organisms RNA States Viral alternative approximated arboviruses blood bold contrast, control courier; data. dengue described difference em; false fitted important laboratory likely mosquitoes. ovitraps performed person placed positives possible pregnant pt;} sensitive sentinel similar sizes specificity strategies. suspect symptomatic system, viral