17-313: Foundations of Software Engineering
Homework 3: Requirements
In this homework, you will be eliciting requirements and documenting them for a graduate school admission system. Your task is to elicit requirements for this new system and document them. You will describe them using user stories and other descriptive text.
Although you already started implementing a corresponding system in the last assignment, you did not have an opportunity to gather requirements from real stakeholders. By doing so, you will see the importance of engaging project stakeholders and specifying requirements in a document rather than guessing requirements informally yourself. You may find that the requirements are substantially different from the ones you would have come up with on your own.
Learning Goals:
-
Elicit functional and quality requirements from stakeholders
-
Identify and describe the stakeholders and actors for a given system.
-
Synthesize requirements from various sources, including stakeholder interviews.
-
Create user stories documenting the functional requirements of the system.
-
Develop and apply appropriate metrics to specify and measure quality attributes.
Project description (from homework 2)
CMU has over 7,000 graduate students. The university uses a software system to manage the graduate student admissions process, including collecting relevant information from applicants, accepting recommendation letters from third-parties, showing this information to the admissions committee and accepting ratings and commentary from the admissions committee to support decisions, and notifying applicants of the decision. This system is old, clunky, and universally disliked.
You and your team are developing a new system. Requirements for this new system remain a bit vague, and in fact you hope to be able to use an initial prototype to help clarify those requirements with stakeholders (in a future assignment!). In the meantime, your point of contact insists primarily that the new system needs to be “less garbage” than the current system. When pressed for details, she elaborates that faculty complain that the current system makes it too difficult to evaluate the applicants; students (applicants) and their letter writers complain that the system is hard to use. Everyone finds configuration and management difficult.
Requirements Elicitation
Your team will seek information about requirements from public documents and observations of the current and alternative systems, as well as from interviews with stakeholders.
To better inform your questions, you might want to spend some time getting to understand the domain (e.g., investigate existing systems and their documentation, including commercial competitors).
You will do two types of interviews:
- Practice interviews: Each team must do several practice interviews of a classmate who is a member of another team. We pair up the teams for you below; negotiate amongst yourselves about who on your team will serve as practice interviewees. Try to keep the practice interviewer groups to 3 or fewer students total, to keep things tractable. Your group should conduct as many practice interviews as necessary for your entire team to have participated in at least one.
- Each team member must participate in at least 2 interviews.
You will select stakeholders and contact them to schedule interviews yourself. We suggest you schedule 15 to 30 minute interviews with each stakeholder and that you conduct interviews in person. The potential stakeholders listed below have already agreed to serve as interviewees for a limited number of interviews. Reach out to stakeholders early and negotiate a time for the interview. Notice that these stakeholders have real, regular jobs and may need to squeeze interviews into a busy schedule. Plan for most interviews during normal office hours. These stakeholders volunteer their time for our class and may decline interviews if they get too many requests, requests on excessively short notice, or requests that are not phrased in a polite and business-like manner. Please use this sheet to input your interview slots. Feel free to refer to this sheet to see what the team assignments are.
Practice interviews pairs:
- 1 - 15
- 2 - 14
- 3 - 13
- 4 - 12
- 5 – 11
- 6 - 10
- 7 & 8 - 9
Non-classmate Stakeholders:
| Name | Number of Available Slots | Role | |
|---|---|---|---|
| Michael Hilton |<EMAIL_ADDRESS>| 5 | ISR faculty member, evaluates students applying to SE PhD and REU programs. |
| Claire Le Goues |<EMAIL_ADDRESS>| 3 | ISR faculty member, writes recommendation letters for applying students; evaluates students applying to SE PhD and REU programs. |
| Christopher Meiklejohn |<EMAIL_ADDRESS>| 3 | SE PhD student, evaluates students applying to SE PhD and REU programs, a previous applicant to graduate programs. |
| Christian Kaestner |<EMAIL_ADDRESS>| 4 | ISR faculty member, writes recommendation letters for applying students; evaluates students applying to SE PhD program |
| Rohan Padhye |<EMAIL_ADDRESS>| 5 | ISR faculty member, writes recommendation letters for applying students; evaluates students applying to SE PhD program |
| Eunsuk Kang |<EMAIL_ADDRESS>| 3 | ISR faculty member, writes recommendation letters for applying students; evaluates students applying to SE PhD program |
| Josh Sunshine |<EMAIL_ADDRESS>| 3 | ISR faculty member, writes recommendation letters for applying students; evaluates students applying to SE PhD program; directure of the REU program |
| Hanan Hibshi |<EMAIL_ADDRESS>| 3 | Research and Teaching Scientist, INI, evaluates students applying to INI |
| Tom Pope |<EMAIL_ADDRESS>| 3 | ISR IT support staff who would potentially be responsible for hosting the system |
| Marlana Pawlak |<EMAIL_ADDRESS>| 2 | Staff member for MSE program who manages applications after they have been submitted. |
| Jane Hsieh |<EMAIL_ADDRESS>| 3 | SE PhD student. A previous applicant to REU and graduate programs. |
| Tobias Dürschmid |<EMAIL_ADDRESS>| 2 | SE PhD student. Helped run SE PhD admissions process. A previous applicant to graduate programs. |
| Kyle Liang |<EMAIL_ADDRESS>| 3 | SE PhD student, recently applied to a number of PhD programs. |
| Connie Herold |<EMAIL_ADDRESS>| 5 | Academic program manager of ISR, manages applications after they have been submitted. |
| Hita Kambhamettu |<EMAIL_ADDRESS>| 3 | Former CMU REUSE student, currently applying to SE PhD programs. |
| Daye Nam |<EMAIL_ADDRESS>| 3 | SE PhD student. Helped run SE PhD admissions process. A previous applicant to graduate programs. |
Proceed as follows:
-
Before the interviews, become familiar with the problem, for example through content analysis of public information and other systems. Think about possible stakeholders, actors, quality attributes, user stories, and assumptions for an actual implementation. Be deliberate about who you want to interview. If you go into the interview unprepared, you are likely to miss important questions.
-
Make appointments for the interviews, and plan them beforehand. You should use the interviews to check your understanding of the problem and elicit additional requirements. Since some people you are interviewing are usually not in the software field, you’ll have to work with them to figure out what information is important to build the application.
-
Conduct the interviews. We suggest you conduct most interviews in groups, but you may further divide roles as you prefer. Treat the interviewees and their time respectfully.
-
Take notes or if (and only if) the interviewee agrees, record the interview. Microphones in most smartphones and laptops are good enough, but we suggest you test recording upfront. It can be useful to partially transcribe an interview or at least its key points immediately afterward, while the knowledge is fresh.
-
Revise your understanding of the problem, based on the interview.
-
Share your insights within your team and synthesize requirements, resolving potential conflicts as far as possible. If there are conflicts you cannot resolve, make reasonable assumptions and document how you would resolve them in practice.
-
Complete the requirements document, filling in the specific sections indicated both within the template and below, in deliverables.
-
Reflect on your experience conducting the interview and the documents in a personal statement.
Deadlines and deliverables
This homework has two (1) deadline and two (2) deliverables. The deliverables are complete requirements document (one (1) per team; specific sections to be completed described below) and one reflection document (one (1) per student). A template for the requirements document can be found here; section references in this document refer to the sections in that template.
Requirements document—Team (due Tuesday, October 26, 23:59) – 120 points (75%)
Each team should submit a single PDF of their completed requirements document to gradescope by , October 26, 23:59pm.
Stakeholder and Actor Descriptions (Sections 1.3, 1.6, and 2.1, and Appendix A) - 15 points (12.5%)
In the appendix, record which stakeholders you interviewed and their roles and which team members participated in each interview.
In the main requirements document, describe all relevant stakeholders, their interests, and their importance to the system. Importance can be measured on any reasonable scale you define as long as it is clear and a developer can read the document and understand which stakeholders are key.
Describe the actors who will interact with the system as well as relevant factors about how they relate to it. Good actor descriptions give enough information about the group/individual/system that it is easy to identify their motivation in interacting with the system; their primary goals/needs in doing so; how they typically interact with the system and their level of skill in doing so (when referring to humans); and other system-relevant factors.
Functional requirements (user stories; Sections 2.2 and 2.3) - 40 points (33%)
Document the functional requirements in the form of user stories. Provide at least two user stories per student in your group.
For each user story, you should also include a discussion of two questions.
1) How will you test this user story? If you cannot test your user story, it is overly vague. This is also a signal that you don’t really understand it yet. For every user story you tackle, you should be able to describe not only the user story, but how you will test it.
2) When is this user story done? This is related, but a slightly different question. Many teams will include a “definition of done” when developing user stories. This can help prevent misunderstandings when it comes to who is responsible for work such as implementation and more.
3) An estimate as to how long this user story will take to implement.
Additionally, you will need to prioritize the requests you get from your stakeholders and decide which of them you plan to support in your system (see also conflicts below). You should not write user stories for everything your stakeholders mention; you should choose wisely according to a set of features that is cohesive and represents a reasonable first iteration of a product.
Quality requirements (Section 3) 25 points (21%)
Define and order the five most important quality attributes within the context of the system. A good description of quality attributes unambiguously defines what the requirements mean within the context of the system; why each requirement deserves the given rank; how to measure each requirement; and how those metrics should be interpreted. If two or more quality attributes are equally important, it is acceptable to give them the same rank, but be sure to justify why they are equally important.
Conflicts (Appendix B) 15 points (12.5%)
Briefly describe and discuss all conflicts you identify during elicitation. For conflicts that you could resolve, describe how you resolved them. For conflicts that you could not resolve within this assignment, briefly outline how you would resolve them if you were doing this for a real customer (e.g., be concrete of whom you might ask which question and how answers would inform your decision).
Quality Assurance (Appendix C), 25 points (21 %)
In class, we discussed the INVEST method for quality assurance of user stories. For N user stories, where N is the number of people in your team, explicitly argue why the user stories do follow the INVEST criteria. If you have to make tradeoffs in one dimensions because of another dimension, you should argue that tradeoff explicitly.
Reflection document – Individual (due Tuesday, October 26, 23:59) - 40 points (25%)
In a reflection document of two pages or less (soft limit) describe what insights (if any) you gained from going through the requirements process, including conducting interviews and synthesizing and documenting requirements. The reflection can include both positive and negative aspects and can set them in context with experience outside of class, if applicable. Use specific examples from your experience gained in this assignment, and include any context information required to explain the situation.
The following questions may help you to identify interesting issues for the reflection document, but it is not necessary to answer them all:
-
Were there surprising aspects of the requirements process?
-
What went well in the preparation for and during the interviews? In retrospect, what could you have improved?
-
What did you not expect in the interviews? Were there unusual situations or misunderstandings? Were there unexpected requirements that you did not anticipate?
-
Who else would you interview, and why, if you had access to more stakeholders or more time?
-
Did the interview and requirements document cause you to adjust your initial assumptions in any way? How so?
-
Were there surprising conflicts in the requirements? Were they easy to resolve?
-
How do the current system requirements compare to the (implicit) requirements used to create the feature your team developed in Homework 2? What would you do different for Homework 2 with that information?
-
How do current systems requirements fit with the decision to build on the Mayan EDMS platform? Are there any requirements that are less feasible given that you are building on the Mayan platform?
Logistics
Schedule your interviews as soon as possible. Please contact your TA if you have any issues. You will need to be respectful of the fact that the stakeholders are real people with their own schedules and responsibilities; if you wait too long, you may not be able to complete your interviews.
Grading
You will be graded as a team, with an individual component. This homework is worth 160 points, as divided above. We provide additional detail on what constitutes full-credit solutions in the narrative above; a summary is below.
For the group component, we expect all below-listed information to be appropriately organized in a document that follows the template we provide. Additionally:
To receive full credit on the interviews and stakeholders and actor descriptions, we expect, based on your interviews:
-
A concise indication of whom you interviewed and their role in the system. For practice interviews, list who was interviewed (and who participated in each) and indicate the role the interviewee pretended to play.
-
A complete and well-described set of relevant system stakeholders, including their roles, interests, and importance in the system.
-
A complete and well-described set of actors, including their role in and, mode of interaction with the system, and other system-relevant features.
To receive full credit on the functional requirements, we expect:
-
At least 2n user stories (for a team with n members) that describe important system functionality at an adequate level of detail and quality.
-
For each user story, in addition to the “as a _____…” we expect the following additional info:
-
How you can test this user story
-
When the user story will be done
-
How long you estimate it will take to implement this user story (with a brief justification for the estimate)
-
To receive full credit on the quality attributes, we expect:
- The identification and description of at least five (5) quality attributes, including a ranking of their importance and a coherent justification of that ranking.
To receive full credit on the conflicting requirements, we expect:
-
A description of all conflicts you identify during elicitation.
-
For conflicts that you can resolve, describe how you resolved them. For conflicts that you cannot resolve within this assignment, briefly outline how you could resolve them if you were doing this for a real customer
To receive full credit on the quality assurance discussion, we expect:
- an explicit discussion of why N of your stories follow the INVEST principles, with tradeoffs justified.
To receive full credit on the individual reflection, we expect:
-
A detailed, well written, and well structured reflection on the requirements elicitation, interview, and synthesis and expression process.
-
An analysis beyond mere descriptions and superficial statements, including supporting evidence for claims, that reflects on the causes of identified issues or your own experience.


If test subjects are involved in the project, you must clarify whether and how the data will be anonymized and whether the data need special protection on the grounds of personality rights.
In general, CLARIN centers only accept anonymized data or data, where the test subject has clearly stated his/her approval of their publication for use in scientific research. In the latter case, the process for obtaining permission for using the data and documenting the permission agreement must be described.
In principal, CLARIN only accepts data which are accessible and intended for free use for academic purposes, because these data can easily be published and distributed. When distribution for academic purposes is not possible, further agreements must be reached with the repository operators. Such limitations should already be considered at the time of the creation of the data management plan. Such limitations can occur for example, when personal rights are involved, when the copyright for raw data belongs to a third party and cannot be obtained, or when there is a time window for scientific use.
Please use the CLARIN license category calculator to determine the license category and copy the resulting license code into the text field below.


CSS Cheat Sheet
This is a list of all inherited attributes in CSS. While some attributes are inherited by the child element from the parent, some are not! Margin and padding, as you will notice, are not on this list. This is why the selector body{margin: 0} would not change the margin of the paragraphs within it.
Inherited Attributes CSS 1. azimuth 2. border-collapse 3. border-spacing 4. caption-side 5. color 6. cursor 7. direction 8. elevation 9. empty-cells 10. font-family 11. font-size 12. font-style 13. font-variant 14. font-weight 15. font 16. letter-spacing 17. line-height 18. list-style-image 19. list-style-position 20. list-style-type 21. list-style 22. orphans 23. pitch-range 24. pitch 25. quotes 26. richness 27. speak-header 28. speak-numeral 29. speak-punctuation 30. speak 31. speak-rate 32. stress 33. text-align 34. text-indent 35. text-transform 36. visibility 37. voice-family 38. volume 39. white-space 40. widows 41. word-spacing
In addtion, this is the order of points assigned for CSS specificity, from highest to lowest priority:
inline (within html doc using style tag) id class element (such as p)
If this is a new concept, this youtube video explains it.
A good resource for checking the specificity of your CSS selectors is this website.
Finally, until I am able to update this reference page, here is a table that lists all CSS attributes and lists whether or not they are inherited.


Build a recalibration model to score variant quality for filtering purposes
This tool performs the first pass in a two-stage process called Variant Quality Score Recalibration (VQSR). Specifically, it builds the model that will be used in the second step to actually filter variants. This model attempts to describe the relationship between variant annotations (such as QD, MQ and ReadPosRankSum, for example) and the probability that a variant is a true genetic variant versus a sequencing or data processing artifact. It is developed adaptively based on "true sites" provided as input, typically HapMap sites and those sites found to be polymorphic on the Omni 2.5M SNP chip array (in humans). This adaptive error model can then be applied to both known and novel variation discovered in the call set of interest to evaluate the probability that each call is real. The result is a score called the VQSLOD that gets added to the INFO field of each variant. This score is the log odds of being a true variant versus being false under the trained Gaussian mixture model.
The purpose of variant recalibration is to assign a well-calibrated probability to each variant call in a call set. These probabilities can then be used to filter the variants with a greater level of accuracy and flexibility than can typically be achieved by traditional hard-filter (filtering on individual annotation value thresholds). The first pass consists of building a model that describes how variant annotation values co-vary with the truthfulness of variant calls in a training set, and then scoring all input variants according to the model. The second pass simply consists of specifying a target sensitivity value (which corresponds to an empirical VQSLOD cutoff) and applying filters to each variant call according to their ranking. The result is a VCF file in which variants have been assigned a score and filter status.
VQSR is probably the hardest part of the Best Practices to get right, so be sure to read the method documentation and tutorial to really understand what these tools do and how to use them for best results on your own data.
gatk VariantRecalibrator \ -R Homo_sapiens_assembly38.fasta \ -V input.vcf.gz \ --resource:hapmap,known=false,training=true,truth=true,prior=15.0 hapmap_3.3.hg38.sites.vcf.gz \ --resource:omni,known=false,training=true,truth=false,prior=12.0 1000G_omni2.5.hg38.sites.vcf.gz \ --resource:1000G,known=false,training=true,truth=false,prior=10.0 1000G_phase1.snps.high_confidence.hg38.vcf.gz \ --resource:dbsnp,known=true,training=false,truth=false,prior=2.0 Homo_sapiens_assembly38.dbsnp138.vcf.gz \ -an QD -an MQ -an MQRankSum -an ReadPosRankSum -an FS -an SOR \ -mode SNP \ -O output.recal \ --tranches-file output.tranches \ --rscript-file output.plots.R
gatk VariantRecalibrator \ -R Homo_sapiens_assembly38.fasta \ -V input.vcf.gz \ -AS \ --resource:hapmap,known=false,training=true,truth=true,prior=15.0 hapmap_3.3.hg38.sites.vcf.gz \ --resource:omni,known=false,training=true,truth=false,prior=12.0 1000G_omni2.5.hg38.sites.vcf.gz \ --resource:1000G,known=false,training=true,truth=false,prior=10.0 1000G_phase1.snps.high_confidence.hg38.vcf.gz \ --resource:dbsnp,known=true,training=false,truth=false,prior=2.0 Homo_sapiens_assembly38.dbsnp138.vcf.gz \ -an QD -an MQ -an MQRankSum -an ReadPosRankSum -an FS -an SOR \ -mode SNP \ -O output.AS.recal \ --tranches-file output.AS.tranches \ --rscript-file output.plots.AS.R
Note that to use the allele-specific (AS) mode, the input VCF must have been produced using allele-specific annotations in HaplotypeCaller. Note also that each allele will have a separate line in the output recalibration file with its own VQSLOD and `culprit`, which will be transferred to the final VCF by the ApplyVQSR tool.
This Read Filter is automatically applied to the data by the Engine before processing by VariantRecalibrator.
This table summarizes the command-line arguments that are specific to this tool. For more details on each argument, see the list further down below the table or click on an argument name to jump directly to that entry in the list.
| Argument name(s) | Default value | Summary | |
|---|---|---|---|
| Required Arguments | |||
| --output -O |
The output recal file used by ApplyVQSR | ||
| --resource |
A list of sites for which to apply a prior probability of being correct but which aren't used by the algorithm (training and truth sets are required to run) | ||
| --tranches-file |
The output tranches file used by ApplyVQSR | ||
| --use-annotation -an |
The names of the annotations which should used for calculations | ||
| --variant -V |
One or more VCF files containing variants | ||
| Optional Tool Arguments | |||
| --aggregate |
Additional raw input variants to be used in building the model | ||
| --arguments_file |
read one or more arguments files and add them to the command line | ||
| --cloud-index-prefetch-buffer -CIPB |
-1 | Size of the cloud-only prefetch buffer (in MB; 0 to disable). Defaults to cloudPrefetchBuffer if unset. | |
| --cloud-prefetch-buffer -CPB |
40 | Size of the cloud-only prefetch buffer (in MB; 0 to disable). | |
| --disable-bam-index-caching -DBIC |
false | If true, don't cache bam indexes, this will reduce memory requirements but may harm performance if many intervals are specified. Caching is automatically disabled if there are no intervals specified. | |
| --disable-sequence-dictionary-validation |
false | If specified, do not check the sequence dictionaries from our inputs for compatibility. Use at your own risk! | |
| --gcs-max-retries -gcs-retries |
20 | If the GCS bucket channel errors out, how many times it will attempt to re-initiate the connection | |
| --gcs-project-for-requester-pays |
Project to bill when accessing "requester pays" buckets. If unset, these buckets cannot be accessed. User must have storage.buckets.get permission on the bucket being accessed. | ||
| --help -h |
false | display the help message | |
| --ignore-all-filters |
false | If specified, the variant recalibrator will ignore all input filters. Useful to rerun the VQSR from a filtered output file. | |
| --ignore-filter |
If specified, the variant recalibrator will also use variants marked as filtered by the specified filter name in the input VCF file | ||
| --input-model |
If specified, the variant recalibrator will read the VQSR model from this file path. | ||
| --interval-merging-rule -imr |
ALL | Interval merging rule for abutting intervals | |
| --intervals -L |
One or more genomic intervals over which to operate | ||
| --mode |
SNP | Recalibration mode to employ | |
| --output-model |
If specified, the variant recalibrator will output the VQSR model to this file path. | ||
| --reference -R |
Reference sequence | ||
| --rscript-file |
The output rscript file generated by the VQSR to aid in visualization of the input data and learned model | ||
| --sites-only-vcf-output |
false | If true, don't emit genotype fields when writing vcf file output. | |
| --target-titv -titv |
2.15 | The expected novel Ti/Tv ratio to use when calculating FDR tranches and for display on the optimization curve output figures. (approx 2.15 for whole genome experiments). ONLY USED FOR PLOTTING PURPOSES! | |
| --truth-sensitivity-tranche -tranche |
[100.0, 99.9, 99.0, 90.0] | The levels of truth sensitivity at which to slice the data. (in percent, that is 1.0 for 1 percent) | |
| --use-allele-specific-annotations -AS |
false | If specified, the variant recalibrator will attempt to use the allele-specific versions of the specified annotations. | |
| --version |
false | display the version number for this tool | |
| Optional Common Arguments | |||
| --add-output-sam-program-record |
true | If true, adds a PG tag to created SAM/BAM/CRAM files. | |
| --add-output-vcf-command-line |
true | If true, adds a command line header line to created VCF files. | |
| --create-output-bam-index -OBI |
true | If true, create a BAM/CRAM index when writing a coordinate-sorted BAM/CRAM file. | |
| --create-output-bam-md5 -OBM |
false | If true, create a MD5 digest for any BAM/SAM/CRAM file created | |
| --create-output-variant-index -OVI |
true | If true, create a VCF index when writing a coordinate-sorted VCF file. | |
| --create-output-variant-md5 -OVM |
false | If true, create a a MD5 digest any VCF file created. | |
| --disable-read-filter -DF |
Read filters to be disabled before analysis | ||
| --disable-tool-default-read-filters |
false | Disable all tool default read filters (WARNING: many tools will not function correctly without their default read filters on) | |
| --exclude-intervals -XL |
One or more genomic intervals to exclude from processing | ||
| --gatk-config-file |
A configuration file to use with the GATK. | ||
| --input -I |
BAM/SAM/CRAM file containing reads | ||
| --interval-exclusion-padding -ixp |
0 | Amount of padding (in bp) to add to each interval you are excluding. | |
| --interval-padding -ip |
0 | Amount of padding (in bp) to add to each interval you are including. | |
| --interval-set-rule -isr |
UNION | Set merging approach to use for combining interval inputs | |
| --lenient -LE |
false | Lenient processing of VCF files | |
| --max-variants-per-shard |
0 | If non-zero, partitions VCF output into shards, each containing up to the given number of records. | |
| --QUIET |
false | Whether to suppress job-summary info on System.err. | |
| --read-filter -RF |
Read filters to be applied before analysis | ||
| --read-index |
Indices to use for the read inputs. If specified, an index must be provided for every read input and in the same order as the read inputs. If this argument is not specified, the path to the index for each input will be inferred automatically. | ||
| --read-validation-stringency -VS |
SILENT | Validation stringency for all SAM/BAM/CRAM/SRA files read by this program. The default stringency value SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded. | |
| --seconds-between-progress-updates |
10.0 | Output traversal statistics every time this many seconds elapse | |
| --sequence-dictionary |
Use the given sequence dictionary as the master/canonical sequence dictionary. Must be a .dict file. | ||
| --tmp-dir |
Temp directory to use. | ||
| --use-jdk-deflater -jdk-deflater |
false | Whether to use the JdkDeflater (as opposed to IntelDeflater) | |
| --use-jdk-inflater -jdk-inflater |
false | Whether to use the JdkInflater (as opposed to IntelInflater) | |
| --verbosity |
INFO | Control verbosity of logging. | |
| Advanced Arguments | |||
| --bad-lod-score-cutoff -bad-lod-cutoff |
-5.0 | LOD score cutoff for selecting bad variants | |
| --debug-stdev-thresholding |
false | Output variants that fail standard deviation thresholding to the log for debugging purposes. Redirection of stdout to a file is recommended. | |
| --dirichlet |
0.001 | The dirichlet parameter in the variational Bayes algorithm. | |
| --k-means-iterations |
100 | Number of k-means iterations | |
| --max-attempts |
1 | Number of attempts to build a model before failing | |
| --max-gaussians |
8 | Max number of Gaussians for the positive model | |
| --max-iterations |
150 | Maximum number of VBEM iterations | |
| --max-negative-gaussians |
2 | Max number of Gaussians for the negative model | |
| --maximum-training-variants |
2500000 | Maximum number of training data | |
| --minimum-bad-variants |
1000 | Minimum number of bad variants | |
| --mq-cap-for-logit-jitter-transform -mq-cap |
0 | Apply logit transform and jitter to MQ values | |
| --mq-jitter |
0.05 | Amount of jitter (as a multiplier to a Normal(0,1) distribution) to add to the AS_MQ and transformed MQ values | |
| --prior-counts |
20.0 | The number of prior counts to use in the variational Bayes algorithm. | |
| --showHidden |
false | display hidden arguments | |
| --shrinkage |
1.0 | The shrinkage parameter in the variational Bayes algorithm. | |
| --standard-deviation-threshold -std |
10.0 | Annotation value divergence threshold (number of standard deviations from the means) | |
| --trust-all-polymorphic |
false | Trust that all the input training sets' unfiltered records contain only polymorphic sites to drastically speed up the computation. |
Arguments in this list are specific to this tool. Keep in mind that other arguments are available that are shared with other tools (e.g. command-line GATK arguments); see Inherited arguments above.
If true, adds a PG tag to created SAM/BAM/CRAM files.
boolean true
If true, adds a command line header line to created VCF files.
boolean true
Additional raw input variants to be used in building the model
These additional calls should be unfiltered and annotated with the error covariates that are intended to be used for modeling.
List[FeatureInput[VariantContext]] []
read one or more arguments files and add them to the command line
List[File] []
LOD score cutoff for selecting bad variants
Variants scoring lower than this threshold will be used to build the Gaussian model of bad variants.
double -5.0 [ [ -∞ ∞ ] ]
Size of the cloud-only prefetch buffer (in MB; 0 to disable). Defaults to cloudPrefetchBuffer if unset.
int -1 [ [ -∞ ∞ ] ]
Size of the cloud-only prefetch buffer (in MB; 0 to disable).
int 40 [ [ -∞ ∞ ] ]
If true, create a BAM/CRAM index when writing a coordinate-sorted BAM/CRAM file.
boolean true
If true, create a MD5 digest for any BAM/SAM/CRAM file created
boolean false
If true, create a VCF index when writing a coordinate-sorted VCF file.
boolean true
If true, create a a MD5 digest any VCF file created.
boolean false
Output variants that fail standard deviation thresholding to the log for debugging purposes. Redirection of stdout to a file is recommended.
boolean false
The dirichlet parameter in the variational Bayes algorithm.
double 0.001 [ [ -∞ ∞ ] ]
If true, don't cache bam indexes, this will reduce memory requirements but may harm performance if many intervals are specified. Caching is automatically disabled if there are no intervals specified.
boolean false
Read filters to be disabled before analysis
List[String] []
If specified, do not check the sequence dictionaries from our inputs for compatibility. Use at your own risk!
boolean false
Disable all tool default read filters (WARNING: many tools will not function correctly without their default read filters on)
boolean false
One or more genomic intervals to exclude from processing
Use this argument to exclude certain parts of the genome from the analysis (like -L, but the opposite).
This argument can be specified multiple times. You can use samtools-style intervals either explicitly on the
command line (e.g. -XL 1 or -XL 1:100-200) or by loading in a file containing a list of intervals
(e.g. -XL myFile.intervals).
List[String] []
A configuration file to use with the GATK.
String null
If the GCS bucket channel errors out, how many times it will attempt to re-initiate the connection
int 20 [ [ -∞ ∞ ] ]
Project to bill when accessing "requester pays" buckets. If unset, these buckets cannot be accessed. User must have storage.buckets.get permission on the bucket being accessed.
String ""
display the help message
boolean false
If specified, the variant recalibrator will ignore all input filters. Useful to rerun the VQSR from a filtered output file.
boolean false
If specified, the variant recalibrator will also use variants marked as filtered by the specified filter name in the input VCF file
For this to work properly, the --ignore-filter argument should also be applied to the ApplyVQSR command.
List[String] []
BAM/SAM/CRAM file containing reads
List[GATKPath] []
If specified, the variant recalibrator will read the VQSR model from this file path.
The filename for a VQSR model fit to use to recalibrate the input variants. This model should be generated using
a previous VariantRecalibration run with the --output-model argument.
GATKPath null
Amount of padding (in bp) to add to each interval you are excluding.
Use this to add padding to the intervals specified using -XL. For example, '-XL 1:100' with a
padding value of 20 would turn into '-XL 1:80-120'. This is typically used to add padding around targets when
analyzing exomes.
int 0 [ [ -∞ ∞ ] ]
Interval merging rule for abutting intervals
By default, the program merges abutting intervals (i.e. intervals that are directly side-by-side but do not
actually overlap) into a single continuous interval. However you can change this behavior if you want them to be
treated as separate intervals instead.
The --interval-merging-rule argument is an enumerated type (IntervalMergingRule), which can have one of the following values:
IntervalMergingRule ALL
Amount of padding (in bp) to add to each interval you are including.
Use this to add padding to the intervals specified using -L. For example, '-L 1:100' with a
padding value of 20 would turn into '-L 1:80-120'. This is typically used to add padding around targets when
analyzing exomes.
int 0 [ [ -∞ ∞ ] ]
Set merging approach to use for combining interval inputs
By default, the program will take the UNION of all intervals specified using -L and/or -XL. However, you can
change this setting for -L, for example if you want to take the INTERSECTION of the sets instead. E.g. to
perform the analysis only on chromosome 1 exomes, you could specify -L exomes.intervals -L 1 --interval-set-rule
INTERSECTION. However, it is not possible to modify the merging approach for intervals passed using -XL (they will
always be merged using UNION).
Note that if you specify both -L and -XL, the -XL interval set will be subtracted from the -L interval set.
The --interval-set-rule argument is an enumerated type (IntervalSetRule), which can have one of the following values:
IntervalSetRule UNION
One or more genomic intervals over which to operate
List[String] []
Number of k-means iterations
This parameter determines the number of k-means iterations to perform in order to initialize the means of
the Gaussians in the Gaussian mixture model.
int 100 [ [ -∞ ∞ ] ]
Lenient processing of VCF files
boolean false
Number of attempts to build a model before failing
The statistical model being built by this tool may fail due to simple statistical sampling
issues. Rather than dying immediately when the initial model fails, this argument allows the
tool to restart with a different random seed and try to build the model again. The first
successfully built model will be kept.
Note that the most common underlying cause of model building failure is that there is insufficient data to
build a really robust model. This argument provides a workaround for that issue but it is
preferable to provide this tool with more data (typically by including more samples or more territory)
in order to generate a more robust model.
int 1 [ [ -∞ ∞ ] ]
Max number of Gaussians for the positive model
This parameter determines the maximum number of Gaussians that should be used when building a positive model
using the variational Bayes algorithm.
int 8 [ [ -∞ ∞ ] ]
Maximum number of VBEM iterations
This parameter determines the maximum number of VBEM iterations to be performed in the variational Bayes algorithm.
The procedure will normally end when convergence is detected.
int 150 [ [ -∞ ∞ ] ]
Max number of Gaussians for the negative model
This parameter determines the maximum number of Gaussians that should be used when building a negative model
using the variational Bayes algorithm. The actual maximum used is the smaller value between the mG and mNG
arguments, meaning that if -mG is smaller than -mNG, -mG will be used for both. Note that this number should
be small (e.g. 4) to achieve the best results.
int 2 [ [ -∞ ∞ ] ]
If non-zero, partitions VCF output into shards, each containing up to the given number of records.
int 0 [ [ 0 ∞ ] ]
Maximum number of training data
The number of variants to use in building the Gaussian mixture model. Training sets larger than this will be randomly downsampled.
int 2500000 [ [ -∞ ∞ ] ]
Minimum number of bad variants
This parameter determines the minimum number of variants that will be selected from the list of worst scoring
variants to use for building the Gaussian mixture model of bad variants.
int 1000 [ [ -∞ ∞ ] ]
Recalibration mode to employ
Use either SNP for recalibrating only SNPs (emitting indels untouched in the output VCF) or INDEL for indels (emitting SNPs untouched in the output VCF). There is also a BOTH option for recalibrating both SNPs and indels simultaneously, but this is meant for testing purposes only and should not be used in actual analyses.
The --mode argument is an enumerated type (Mode), which can have one of the following values:
Mode SNP
Apply logit transform and jitter to MQ values
MQ is capped at a "max" value (60 for bwa-mem) when the alignment is considered perfect. Typically, a huge
proportion of the reads in a dataset are perfectly mapped, which yields a distribution of MQ values with a
blob below the max value and a huge peak at the max value. This does not conform to the expectations of the
Gaussian mixture model of VQSR and has been observed to yield a ROC curve with a jump.
This argument aims to mitigate this problem. Using MQCap = X has 2 effects: (1) MQs are transformed by a scaled
logit on [0,X] (+ epsilon to avoid division by zero) to make the blob more Gaussian-like and (2) the transformed
MQ=X are jittered to break the peak into a narrow Gaussian.
If this option is not used, or if MQCap is set to 0, MQ will not be transformed.
Note that AS_MQ uses --mq-jitter instead to control artificial variance
int 0 [ [ -∞ ∞ ] ]
Amount of jitter (as a multiplier to a Normal(0,1) distribution) to add to the AS_MQ and transformed MQ values
Amount of jitter (as a multiplier to a Normal(0,1) distribution) to add to the AS_MQ and transformed MQ values
double 0.05 [ [ -∞ ∞ ] ]
The output recal file used by ApplyVQSR
R GATKPath null
If specified, the variant recalibrator will output the VQSR model to this file path.
This GATKReport gives information to describe the VQSR model fit. Normalized means for the positive model are
concatenated as one table and negative model normalized means as another table. Covariances are also concatenated
for positive and negative models, respectively. Tables of annotation means and standard deviations are provided
to help describe the normalization. The model fit report can be read in with our R gsalib package. Individual
model Gaussians can be subset by the value in the "Gaussian" column if desired.
GATKPath null
The number of prior counts to use in the variational Bayes algorithm.
double 20.0 [ [ -∞ ∞ ] ]
Whether to suppress job-summary info on System.err.
Boolean false
Read filters to be applied before analysis
List[String] []
Indices to use for the read inputs. If specified, an index must be provided for every read input and in the same order as the read inputs. If this argument is not specified, the path to the index for each input will be inferred automatically.
List[GATKPath] []
Validation stringency for all SAM/BAM/CRAM/SRA files read by this program. The default stringency value SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded.
The --read-validation-stringency argument is an enumerated type (ValidationStringency), which can have one of the following values:
ValidationStringency SILENT
Reference sequence
GATKPath null
A list of sites for which to apply a prior probability of being correct but which aren't used by the algorithm (training and truth sets are required to run)
Any set of VCF files to use as lists of training, truth, or known sites.
Training - The program builds the Gaussian mixture model using input variants that overlap with these training sites.
Truth - The program uses these truth sites to determine where to set the cutoff in VQSLOD sensitivity.
Known - The program only uses known sites for reporting purposes (to indicate whether variants are already known or
novel). They are not used in any calculations by the algorithm itself.
Bad - A database of known bad variants can be used to supplement the set of worst ranked variants (compared to the
Gaussian mixture model) that the program selects from the data to model "bad" variants.
R List[FeatureInput[VariantContext]] []
The output rscript file generated by the VQSR to aid in visualization of the input data and learned model
File null
Output traversal statistics every time this many seconds elapse
double 10.0 [ [ -∞ ∞ ] ]
Use the given sequence dictionary as the master/canonical sequence dictionary. Must be a .dict file.
GATKPath null
display hidden arguments
boolean false
The shrinkage parameter in the variational Bayes algorithm.
double 1.0 [ [ -∞ ∞ ] ]
If true, don't emit genotype fields when writing vcf file output.
boolean false
Annotation value divergence threshold (number of standard deviations from the means)
If a variant has annotations more than -std standard deviations away from mean, it won't be used for building
the Gaussian mixture model.
double 10.0 [ [ -∞ ∞ ] ]
The expected novel Ti/Tv ratio to use when calculating FDR tranches and for display on the optimization curve output figures. (approx 2.15 for whole genome experiments). ONLY USED FOR PLOTTING PURPOSES!
The expected transition / transversion ratio of true novel variants in your targeted region (whole genome, exome, specific
genes), which varies greatly by the CpG and GC content of the region. See expected Ti/Tv ratios section of the GATK best
practices documentation (https://software.broadinstitute.org/gatk/guide/best-practices) for more information.
Normal values are 2.15 for human whole genome values and 3.2 for human whole exomes. Note
that this parameter is used for display purposes only and isn't used anywhere in the algorithm!
double 2.15 [ [ -∞ ∞ ] ]
Temp directory to use.
GATKPath null
The output tranches file used by ApplyVQSR
R File null
Trust that all the input training sets' unfiltered records contain only polymorphic sites to drastically speed up the computation.
boolean false
The levels of truth sensitivity at which to slice the data. (in percent, that is 1.0 for 1 percent)
Add truth sensitivity slices through the call set at the given values. The default values are 100.0, 99.9, 99.0, and 90.0
which will result in 4 estimated tranches in the final call set: the full set of calls (100% sensitivity at the accessible
sites in the truth set), a 99.9% truth sensitivity tranche, along with progressively smaller tranches at 99% and 90%.
Note: You must pass in each tranche as a separate value (e.g. -tranche 100.0 -tranche 99.9).
List[Double] [100.0, 99.9, 99.0, 90.0]
If specified, the variant recalibrator will attempt to use the allele-specific versions of the specified annotations.
Generate a VQSR model using per-allele data instead of the default per-site data, assuming that the input VCF contains allele-specific annotations.
Annotations should be specified using their full names with AS_ prefix. Non-allele-specific (scalar) annotations will be applied to all alleles.
boolean false
The names of the annotations which should used for calculations
See the input VCF file's INFO field for a list of all available annotations.
R List[String] []
Whether to use the JdkDeflater (as opposed to IntelDeflater)
boolean false
Whether to use the JdkInflater (as opposed to IntelInflater)
boolean false
One or more VCF files containing variants
R List[GATKPath] []
Control verbosity of logging.
The --verbosity argument is an enumerated type (LogLevel), which can have one of the following values:
LogLevel INFO
display the version number for this tool
boolean false
See also General Documentation | Tool Docs Index Tool Documentation Index | Support Forum
GATK version <IP_ADDRESS> built at Wed, 18 Aug 2021 19:58:25 -0400.


| categories:development
The steel industry, file_fdw and the synchronisation of data
Contents
- Steel me.
- On the process of selling metals world wide
- Synchronisation vs. Import
- PostgreSQL, how I love thee
- Using file_fdw with Ecto
- Transform the data
- The key to doing things fast
- Joining external and internal tables
- What to do next?
- Numbers, please!
- So, we use this all the time now?
- Learnings and a repository
Steel me.
My god.
It has been 2 years since I wrote something.
I am really bad at social media.
I also started working as a developer for a company called Klöckner & Co. To be exact, I work for its subsidiary called kloeckner.i, which is focused on bringing the other subsidiaries (Kloeckner France, Kloeckner UK, Kloeckner Metals US and many more), of the group into a glorious, bright and … well, digital future.
We are building tools to help our companies trade and deal with our products - steel & metals - in a digital fashion (as opposed to analog). It’s all about innovation (the “i” in kloeckner.i) and coming up with new ideas on how the industry will operate in 10 years from now. Interesting stuff! If you want to learn more, you can watch our main man, CEO of the group, Mr. Gisbert Rühl himself explaining the concepts of the how and why.
In the meantime, after having built the inital team, I am now working as a the Lead Software Architect in kloeckner.i. For starters, this meant leaving the shackles of management behind and focusing more on the technical implementation of the ideas that our tech and product department provide. In order to understand the challenges we face at kloeckner.i, one needs to understand how the overall business model looks like.
On the process of selling metals world wide
While the processes are different from country to another, the main mechanism stays the same - we buy the steel from a mill (a producer smelting various materials), store it in our warehouses and then deliver it to our customers. For whatever your project may look like, Klöckner can provide the metals necessary to make your dream of a skyscraper reality. Or a bridge. Or yacht. Or you just really like buying a couple tons of steel from us. We got you. At kloeckner.i, we deal with the fact that this business model generates a ton of data, which comes in different forms and shapes. All the services that we provide are powered by this data, sometimes enriched, sometimes raw. Nevertheless, it is always put to use by providing it through applications to the end user.
And since we cannot connect your laptop, mobile phone or PDA (you still have yours, right?) directly to the source of data (the ERP of the country organization), we have to transform it and store it away for usage in our services. The data we get most often comes in the form of structured text: CSV and XML in most cases. Overall, we employ various means of transmitting data from the subsidiaries’ ERP to our services:
- Transmitting files via (S)FTP and then
- Using APIs to allow data push
- Using queueing strategies to distribute pieces of the data to our services
Most of these methods work to varying degrees. For one-time-imports of single files, the straight imports usually work best, as they are simple and easy to understand. APIs are also nice, since you can fend off broken data early. The downside is that not everyone can immediately use an API as actively pushing data is not always possible. Queueing strategies are well understood, but require monitoring and are tough to debug.
The overall strategy, however, stays the same:
Synchronisation vs. Import
Keeping our requirements in mind and the the fact that we need to operate a copy on the data, the challenge here is to keep everything in sync. The external system will provide a data dump somehow and we are tasked to make sure that we have the same data in our systems.
Having realized this, we usually speak of synchronisation at kloeckner.i - and as I set out to find a better solution for our long running imports, while attending the ElixirConf 2018 in Warsaw I watched the keynote by developer Evadne Wu. She dealt with the topic of data imports and called for having a better relationship with our data storage. As it turns out, PostgreSQL offers a ton of useful extra features that I had ignored for way too long.
PostgreSQL, how I love thee
We use PostgreSQL extensively at kloeckner.i. It’s our go-to storage layer and nearly all of our projects include a PostgreSQL in development, usually in the form of a docker container. In fact, most projects look like this when you get into their docker-compose.yml
:
version: "3.4"
services:
app:
command: mix do deps.get, phx.server
build: .
ports:
- 4000:4000
depends_on:
- db
db:
image: <gitlab-registry>/infrastructure/postgres:latest
volumes:
- /var/lib/postgresql
ports:
- 5432:5432
Before you type that tweet to the end: latest
is just the current version we use for all projects. We are managing the PostgreSQL version used centrally in a custom build image.
Foreign data wrappers
PostgreSQL has the ability to interact with foreign data objects, utilizing its foreign data wrapper feature.
Foreign Data Wrappers are a set of extensions you can use to interact with data sources that are not natively known to PostgreSQL. The full list is quite impressive - and although not everything is stable and supported, the most interesting wrappers are included by default.
file_fdw
Besides funky things like the docker_fdw
, there is the very useful file_fdw
. There are a couple of these for a multitude of file formats, including PostgreSQLs own pg_dump
SQL, JSON, XML and CSV.
Wait, this means…
Using file_fdw with Ecto
Disclaimer: This uses Ecto because Elixir based applications are widespread within kloeckner.i. It has become one of our primary languages and is used throughout all of our teams. You can copy this approach to any other programming language, if you wish.
defmodule SynchronizeApp.Repo.Migrations.AddFileFdwExtension do
use Ecto.Migration
@up ~s"""
CREATE extension file_fdw;
"""
@down ~s"""
DROP EXTENSION file_fdw;
"""
def change do
execute(@up, @down)
end
end
This will create the file_fdw
extension. It’s included in the later version of PostgreSQL, you just need to activate it. In another migration, we need to enable a server, a virtual object our table definitions will be bound to later on:
defmodule SynchronizeApp.Repo.Migrations.AddForeignFileServer do
use Ecto.Migration
@server_name "files"
@up ~s"""
CREATE SERVER #{@server_name} FOREIGN DATA WRAPPER file_fdw;
"""
@down ~s"""
DROP SERVER #{@server_name};
"""
def change do
execute(@up, @down)
end
end
I personally advise to use Ecto.Migration.execute/2, in order to have migrations that can be rolled back here. Alas, since the SQL here has no DSL in Ecto itself, we need to resort to executing it directly.
And that is it! We now need to create tables for the CSV files we receive from an external source. For reference, this is what such a file normally looks like:
company_id|name|address1|address2|city|state|zipcode|country
137521| Acme Inc |1381 East I ***20 Access"" Road| pleas, talk to "MAUDE"|Maraune|AK|733-336|United States
11582| Stark Industries|14065 SW 142 ST| |New York|NY|33186|United States
The file_fdw
wrapper ideally is fed with RFC4180 compliant CSV. In the real world, years of manual data entry have done some harm to the overall data quality, so we need to clean up here a bit. Also, we do get some weird characters in the files, including quotes ("
), which cause havoc when not using an addtional configuration.
Never lose hope, this is all digestable with the options given to the table definition:
defmodule SynchronizeApp.Repo.Migrations.AddForeignCompaniesTable do
use Ecto.Migration
@up ~s"""
CREATE FOREIGN TABLE external_companies (
company_id text,
name text,
address1 text,
address2 text,
city text,
state text,
zipcode text,
country text
) SERVER kmc_files
OPTIONS ( filename '/files/Company.csv', format 'csv', delimiter '|', header 'on', quote E'\x01');
"""
@down ~s"""
DROP FOREIGN TABLE kmc_companies;
"""
def change do
execute(@up, @down)
end
end
The interesting things here are:
SERVER files
- this references the virtual server object we created ealier.OPTIONS ( filename '/files/Company.csv', format 'csv', delimiter '|', header 'on', quote E'\x01')
- let’s see:filename
is the location of the file to be read, this must be known to the database serverformat
iscsv
, but can also betext
(giving no options for header configuration)header 'on'
defines the first line of the file to be a header, similar to how a lot of libraries workdelimiter '|'
gives a custom delimiter character, in our case|
, a pipe.quote E'\x01'
defines the quote character to be a non-printable character that should never pop up in the file to be read. Unfortunately, it cannot be configured to take an empty string.
- all of the column definitions map to the columns of the CSV header - and everything is
text
This, while being a very rigid table that has no types and is less flexible, still gives us the opportunity to query the data directly, using SQL:
SELECT company_id, name FROM external_companies LIMIT 1;
Result:
|name |company_id|
+--------------+----------+
| Acme Inc |137521 |
Sweet.
Note: The options are directly linked to the options for PostgreSQL’s COPY command, which is a very effective tool if you just want to do a single time import of data you have as CSV files.
Transform the data
Since we now have everything available as a table, nothing stops us from utilizing the functions that PostgreSQL gives us:
SELECT TRIM(name), company_id, MD5(CONCAT(company_id, name)) as hash FROM external_companies LIMIT 1;
Result:
|name |company_id|hash |
+--------------+----------+--------------------------------+
|Acme Inc |137521 |b4fa5d3e03248e285c6cc57ac4f4862e|
And it works like a charm. Partitions using WHERE
are also possible, but be cautious - there are no indizes to back you up here.
The key to doing things fast
There is a great explanation about why grep
is fast - simply spoken: “Being fast usually means not doing a lot of things”, or, to quote:
The key to making programs fast is to make them do practically nothing. ;-)
And it’s true for our usecase as well: When importing data, we should avoid doing things we don’t need to do. And since we keep our CSV files and our service’s tables in one place, there must be a way of finding out the actual workload is. The answer here is: JOIN
s.
Joining external and internal tables
When having access to tables that should keep the same information, finding the difference over several fields can be done using an SQL JOIN
- a full overview over all the joins can be found here. In a more visual fashion, having the csv data as current data, and the internal copied data, we can use a LEFT JOIN
to find the differences by joining on a hash created from the fields that we’re interested in:
The full SQL looks a bit scary, but does effectively what is shown above:
SELECT external.company_id, external.name
FROM external_companies external
LEFT JOIN internal_companies internal
ON MD5(CONCAT(external.company_id, TRIM(external.name)))
= MD5(CONCAT(internal.external_id, TRIM(internal.name)))
WHERE internal.external_id IS NULL
It will result in a changeset that we have to digest. In the specific case of having changed Acme Inc to Acme Inc1, we can tell by the changed hash that this record needs to be updated in our service’s data.
What to do next?
Well, again, Elixir allows use to model this process in very small functions. Visualized by two modules, it can look like this:
We create two modules, one which we’ll call SQLModule
that will be used to hold functions that run a Query to determine what the changeset looks like and one that will actually execute insert/update/delete the data in our service tables. Let’s call that one the ImportModule
.
Full disclosure: In the real world, we employ data checking and we actually need to grab data from multiple files into a state that makes sense to use it in the tools we built for our customers. That also means, that reality is usually not as simple as it is and we needed to design more complex queries, joining the data of multiple CSVs together.
Let’s have a look at the SQLModule
:
defmodule Synchronize.Companies.SQLModule do
@moduledoc """
The module will upsert given companies in batches into the services database.
"""
alias MyService.{Company, Repo}
alias Synchronize.ImportModule
alias Ecto.Adapters.SQL
@doc """
the main entrypoint for this module
"""
def sync do
find_companies() |> run()
end
@doc """
Starts the import for a given set of companies
"""
def run(companies) do
ImportModule.execute(companies, &map/1, &import_batch/1)
end
@doc """
Find missing companies in the services database
"""
def find_companies do
SQL.stream(Repo, """
SELECT external.company_id, external.name
FROM external_companies external
LEFT JOIN companies -- companies is the internal table
ON MD5(CONCAT(external.company_id, TRIM(external.name)))
= MD5(CONCAT(companies.external_id, TRIM(companies.name)))
WHERE companies.external_id IS NULL
""")
end
defp map([external_id, name]) do
now = DateTime.utc_now()
%{
name: String.trim(name),
external_id: external_id,
inserted_at: now,
updated_at: now
}
end
defp import_batch(batch) do
Repo.insert_all(
Company,
batch,
on_conflict: :replace_all,
conflict_target: :external_id
)
end
end
Running Synchronize.Companies.SQLModule.sync/1
will now trigger the query described earlier - whatever changeset is found, will then be given to the ImportModule
, which holds functions to execute the mapping necessary (Synchronize.Companies.SQLModule.map/1
) and the actual logic to import a single batch of that change (Synchronize.Companies.SQLModule.import_batch/1
).
The ImportModule
is rather simple:
defmodule Synchronize.ImportModule do
@moduledoc """
The module supports data synchronization from a source of thruth, allowing
transformations
"""
require Logger
alias MyService.Repo
@doc """
Starts the import for a given set of billing addresses
"""
def execute(source, mapper, importer) do
Logger.info("Synchronizing...")
started = System.monotonic_time()
{:ok, count} = run(source, mapper, importer)
finished = System.monotonic_time()
time_spent =
System.convert_time_unit(finished - started, :native, :milliseconds)
Logger.info("Synchronized #{count} item(s) in #{time_spent} millisecond(s)")
end
@batch_size 2_000
defp run(source, item_mapper, batch_processor) do
processor = fn batch ->
batch_processor.(batch)
batch
end
Repo.transaction(fn ->
source
|> stream_row()
|> Stream.map(item_mapper)
|> Stream.chunk_every(@batch_size)
|> Stream.flat_map(processor)
|> Enum.count()
end)
end
def stream_row(source) when is_list(source) do
source
end
def stream_row(%Ecto.Adapters.SQL.Stream{} = source) do
Stream.flat_map(source, fn %{rows: rows} -> rows end)
end
end
Essentially the ImportModule
will stream the results from the query, chunk the resulting changeset by a @batch_size
and then execute the import function (batch_processor
) given.
If we want to further develop this, we can now write several of these SQLModules
with their own mappers and processors and use this ImportModule
to execute the logic.
NOTE: In reality, the trigger for finding the changeset is centralized on a lockfile which contains a timestamp. The synchronisation process happens when the timestamp changes. We do nothing we don’t really have to do.
Numbers, please!
While I cannot provide a benchmark here, I can say that switching strategies reduced our import time for this particular piece of customer data from 300 seconds to about 10 seconds average for any given synchronisation process. With the exception of a full synchronisation, which has to process and copy all the data from the external CSV to the internal tables, most of the runs are below those numbers. Nevertheless, here is a graph:
The load on the application is minimal and the actual bottleneck is the database. We use Elixir in this context to delegate and trigger, instead of doing the actual work.
So, we use this all the time now?
No - while this mechanism is interesting, fast and fun to work with (it’s also a great debugging tool for CSV inspection when Excel is just a little to intense to use), the solution has to fit the use case.
In general, you should avoid it, if:
- you only import once
- you don’t feel comfortable using SQL and have extremely complicated transformations
- you don’t have a stable csv definition
- you cannot afford to have business logic tied to your database
The strategy is worth a shot if:
- you need to compare state of several files and the data you have
- you need it fast (PostgreSQLs
COPY
is pretty fast) - You want to offload more to your database server
Learnings and a repository
The main thing to take away for our work at kloeckner.i (and for myself) was that PostgreSQL offers so much that it’s almost scary. After having run through many iterations of trying out import strategies we finally found something that fits a lot of our use cases.
If you want to try this and see this strategy in action - I prepared a repo showing the technique with some test data. It runs as two docker containers, so you should be able to use it right away. If you have questions, drop me an email - I’ll be glad to help.
If you completely disagree with all of this, I encourage you to take a look at our jobs page. We’re always looking for smart engineers that can contribute to making the transformation of a whole industry a successful story.
It’s also the best team in the world, not that I might be biased or anything.


Flexible to accommodate each couple’s wishes, the venues at LA-Zuberia Thani Maldives offer include outdoor settings and stunning restaurants — perfect to host different functions for your big day.
Our talented chefs will be delighted to create an indulgent meal to remember, tempting the most discerning palates. Fresh local delicacies and luxurious international fare await you.
The following wedding and vow renewal ceremonies are available at LA-Zuberia Thani Maldives.
For years, couples in search of timeless romance have chosen LA-Zuberia Thani Maldives. With the perfect blend of Thai-inspired interiors and modern facilities, the resort offers an array of wedding packages from which to choose.


Hello,I'm Mary Sinaida A student at Moringa school studying Wed Development and Design,Am also life long learner and ready for new oppotunies.A bit about my intrest is that i
I belive when i clear my studies i will deliver a user center design and delivering the best possible experience to people. I have experience in Html 5,Css,Flexbox,Git,GitHub,Computer packages and Javascript.
Html 5
Css
Git
GitHub
Computer-Packages
Flexbox
I cleared my high school in the year 2014
Worked at DDD in the operation depertment to satisfy your team and client needs.
studied at st pauls university taking a course known as information and Technology.
I dream to build and design responsive site for projects no idea is to small since everyone deserves a voice in the web
With the knowledge i have so far have been able to complete one project am so glad since it was my first Design
My project is called snach-farmbiz just an interactive design where buyer oder and seller delivers the comodity posted.
Click here


{border-style px;} #ffffff;} []. masks surgical COVID- mask patients respiratory px; {font-family font-size protection University double; pandemic FFP PPE Sussex border-color border-width healthcare ophthalmic respirators solid; staff {font-style {font-weight Health Hong Ophthalmology aerosols bold control coronavirus courier; disease droplets em; guidance including patient pt;} use wearing


"SUPH "(Deuteronomy 1:1, R.V.; marg., “some ancient versions have the Red Sea,” as in the A.V.). Some identify it with Suphah (Numbers 21:14, marg., A.V.) as probably the name of a place. Others identify it with es-Sufah = Maaleh-acrabbim (Joshua 15:3), and others again with Zuph (1 Samuel 9:5). It is most probable, however, that, in accordance with the ancient versions, this word is to be regarded as simply an abbreviation of Yam-suph, i.e., the “Red Sea.”
"SUPH "(Deuteronomy 1:1, R.V.; marg., “some ancient versions have the Red Sea,” as in the A.V.). Some identify it with Suphah (Numbers 21:14, marg., A.V.) as probably the name of a place. Others identify it with es-Sufah = Maaleh-acrabbim (Joshua 15:3), and others again with Zuph (1 Samuel 9:5). It is most probable, however, that, in accordance with the ancient versions, this word is to be regarded as simply an abbreviation of Yam-suph, i.e., the “Red Sea.”


Card Titlemore_vert
Card Titleclose
Here is some more information about this product that is only revealed once clicked on.
IR


Newspaper Explorer
Showing the number of newspaper articles matching your query.
new series
clear all series
trash
hint:
click a point or bar on the graph to see a few relevant headlines
type:
Chart style
Bars
Lines
Area
Horizontal axis
Months
Years
Pages
Other options
Stacked
Dedup
Interesting graphs:
||
(
||
)


The name Beutelwurst comes from the fact that this does not come in a casing of intestine or a can, but in a linen or paper bag ("bag" = Beutel).<ref>Guide to German Sausages & Meat Products at www.germanfoods.org. Retrieved on 20 Mar 10.</ref> This bag is pressed for several weeks which gives the Beutelwurst a firm, light, dry consistency.
{{reflist}}
{{meat-stub}}


A Support Vector Machine (SVM) is a very powerful and versatile Machine Learning model, capable of performing linear or nonlinear classification, regression, and even outlier detection. It is one of the most popular models in Machine Learning, and anyone interested in Machine Learning should have it in their toolbox. SVMs are particularly well suited for classification of complex but small-sized or medium-sized datasets.
The basic mathematics relies on the definition of hyperplanes and the definition of a margin which separates classes (in case of classification problems) of variables. It is also used for regression problems.
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below. We distringuish also between linearn and non-linear approaches.
These notes will be updated shortly with more material
from IPython.display import set_matplotlib_formats, display
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import mglearn
from cycler import cycler
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.datasets import make_blobs
X, y = make_blobs(centers=4, random_state=8)
y = y % 2
mglearn.discrete_scatter(X[:, 0], X[:, 1], y)
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")
plt.show()
from sklearn.svm import LinearSVC
linear_svm = LinearSVC().fit(X, y)
mglearn.plots.plot_2d_separator(linear_svm, X)
mglearn.discrete_scatter(X[:, 0], X[:, 1], y)
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")
# add the squared first feature
X_new = np.hstack([X, X[:, 1:] ** 2])
from mpl_toolkits.mplot3d import Axes3D, axes3d
figure = plt.figure()
# visualize in 3D
ax = Axes3D(figure, elev=-152, azim=-26)
# plot first all the points with y==0, then all with y == 1
mask = y == 0
ax.scatter(X_new[mask, 0], X_new[mask, 1], X_new[mask, 2], c='b',
cmap=mglearn.cm2, s=60, edgecolor='k')
ax.scatter(X_new[~mask, 0], X_new[~mask, 1], X_new[~mask, 2], c='r', marker='^',
cmap=mglearn.cm2, s=60, edgecolor='k')
ax.set_xlabel("feature0")
ax.set_ylabel("feature1")
ax.set_zlabel("feature1 ** 2")
linear_svm_3d = LinearSVC().fit(X_new, y)
coef, intercept = linear_svm_3d.coef_.ravel(), linear_svm_3d.intercept_
# show linear decision boundary
figure = plt.figure()
ax = Axes3D(figure, elev=-152, azim=-26)
xx = np.linspace(X_new[:, 0].min() - 2, X_new[:, 0].max() + 2, 50)
yy = np.linspace(X_new[:, 1].min() - 2, X_new[:, 1].max() + 2, 50)
XX, YY = np.meshgrid(xx, yy)
ZZ = (coef[0] * XX + coef[1] * YY + intercept) / -coef[2]
ax.plot_surface(XX, YY, ZZ, rstride=8, cstride=8, alpha=0.3)
ax.scatter(X_new[mask, 0], X_new[mask, 1], X_new[mask, 2], c='b',
cmap=mglearn.cm2, s=60, edgecolor='k')
ax.scatter(X_new[~mask, 0], X_new[~mask, 1], X_new[~mask, 2], c='r', marker='^',
cmap=mglearn.cm2, s=60, edgecolor='k')
ax.set_xlabel("feature0")
ax.set_ylabel("feature1")
ax.set_zlabel("feature1 ** 2")
ZZ = YY ** 2
dec = linear_svm_3d.decision_function(np.c_[XX.ravel(), YY.ravel(), ZZ.ravel()])
plt.contourf(XX, YY, dec.reshape(XX.shape), levels=[dec.min(), 0, dec.max()],
cmap=mglearn.cm2, alpha=0.5)
mglearn.discrete_scatter(X[:, 0], X[:, 1], y)
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")
from sklearn.svm import SVC
X, y = mglearn.tools.make_handcrafted_dataset()
svm = SVC(kernel='rbf', C=10, gamma=0.1).fit(X, y)
mglearn.plots.plot_2d_separator(svm, X, eps=.5)
mglearn.discrete_scatter(X[:, 0], X[:, 1], y)
# plot support vectors
sv = svm.support_vectors_
# class labels of support vectors are given by the sign of the dual coefficients
sv_labels = svm.dual_coef_.ravel() > 0
mglearn.discrete_scatter(sv[:, 0], sv[:, 1], sv_labels, s=15, markeredgewidth=3)
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")
fig, axes = plt.subplots(3, 3, figsize=(15, 10))
for ax, C in zip(axes, [-1, 0, 3]):
for a, gamma in zip(ax, range(-1, 2)):
mglearn.plots.plot_svm(log_C=C, log_gamma=gamma, ax=a)
axes[0, 0].legend(["class 0", "class 1", "sv class 0", "sv class 1"],
ncol=4, loc=(.9, 1.2))


Oklahoma City buildng permits from The Oklahoman for Nov. 23, 2019
Lingo Construction: 100 N Broadway Ave., BankFirst Tower, office, remodel, $5,500,000.
Lingo Construction: 100 N Broadway Ave., BankFirst Tower, office, remodel, $2,500,000.
Lingo Construction: 100 N Broadway Ave., BankFirst Tower, office, remodel, $2,000,000.
Lingo Construction: 100 N Broadway Ave., BankFirst Tower, office, remodel, $1,850,000.
Savannah Builders LLC: 15009 Gaillardia Lane, residence, erect, $1,800,000.
RBA Archiects: 9201 Polaris Drive, warehouse, erect, $1,800,000.
John Pinion: 9601 W Reno Ave., warehouse, erect, $1,472,338.
Epsten Group: 2101 NE 50, accessory, remodel, $1,111,093.
Epsten Group: 2101 NE 50, amusement, remodel, $1,111,093.
Montgomery Homes Inc.: 10901 NW 24 Terrace, residence, erect, $600,000.
Hornbeek Blatt Architects: 2825 NW 23, office, erect, $600,000.
Aaron Tatum Custom Homes LLC: 10616 Servon Drive, residence, erect, $515,000.
Oak Leaf Custom Homes: 818 NW 43, residence, erect, $500,000.
Utopia Design and Build LLC: 1318 NW 1, duplex, erect, $500,000.
Ron Walters Homes LLC: 2101 Westbury Drive, fourplex, erect, $469,000.
Tommy Huelskamp LLC: 8432 NW 134, residence, erect, $425,000.
Jeffery W. Shahan 2: 17901 SE 149, residence, erect, $361,000.
RK Lancaster Homes LLC: 12128 SW 49, residence, erect, $332,000.
Taber Built Homes LLC: 18732 Lazy Grove Drive, residence, erect, $329,000.
Vector Construction LLC: 12509 Pinewood Lane, residence, erect, $320,000.
Taber Built Homes LLC: 801 Suttles Way, residence, erect, $314,000.
Bradbury Homes Inc.: 12132 SW 46, residence, erect, $300,000.
Bryan Beavers LLC, doing business as Bryan Beavers Construction: 3108 Oak Hollow Road, residence, remodel, $300,000.
Lingo Construction: 100 N Broadway Ave., office, remodel, $300,000.
Massarossa Luxury Homes: 9708 Eastblake Place, residence, erect, $300,000.
RK Lancaster Homes LLC: 12112 SW 49, residence, erect, $300,000.
Link Clifton: 2620 Shull Ave., office-warehouse, remodel, $300,000.
RK Lancaster Homes LLC: 4829 Hambletonian Lane, residence, erect, $289,000.
Harbor Homes Inc.: 15916 Langley Way, residence, erect, $275,000.
Taber Built Homes LLC: 809 Cassandra Lane, residence, erect, $268,000.
Taber Built Homes LLC: 10429 SW 23, residence, erect, $257,000.
Taber Built Homes LLC: 808 Patrick Way, residence, erect, $256,000.
Canvas Project Management LLC: 8904 NW 130, residence, erect, $250,000.
Franklin.Build LLC: 120 N Western Ave., manufacturing, remodel, $250,000.
Timbercraft Homes: 629 NW 181, residence, erect, $235,000.
Todd Cooper Homes Inc.: 12705 NW 138, residence, erect, $230,000.
Apple Construction LLC: 15218 Colonia Bella Drive, residence, erect, $222,600.
Ron Walters Homes LLC: 2101 Westbury Drive, apartment, erect, $211,000.
4 Corners Construction LLC: 17700 Timberline Court, residence, erect, $201,880.
Montgomery Homes Inc.: 10825 NW 29 Terrace, residence, erect, $200,000.
Redbud Builders LLC: 8100 NW 161 Terrace, residence, erect, $200,000.
Todd Cooper Homes Inc.: 12640 NW 138, residence, erect, $199,000.
Todd Cooper Homes Inc.: 3408 Upland Ridge Drive, residence, erect, $199,000.
Mashburn Faires Homes LLC: 11800 Kylie Elizabeth Road, residence, erect, $190,980.
Todd Cooper Homes Inc.: 14005 Northwood Village Drive, residence, erect, $190,000.
Todd Cooper Homes Inc.: 13904 Village Run Drive, residence, erect, $190,000.
Ron Walters Homes LLC: 2101 Westbury Drive, residence, erect, $186,000.
Griffin Homes LLC: 9321 NW 82, residence, erect, $180,000.
Griffin Homes LLC: 9320 NW 82, residence, erect, $180,000.
Mashburn Faires Homes LLC: 8008 W Lakeshore Drive, residence, erect, $172,029.
4 Corners Construction LLC: 17500 Timberline Court, residence, erect, $170,660.
Mirage Homes LLC: 8708 Arman Drive, residence, erect, $140,000.
Legacy Homes Inc.: 9824 NW 95, residence, erect, $135,000.
Mirage Homes LLC: 2225 Cale Cove, residence, erect, $135,000.
Colony Fine Homes LLC: 11104 NW 98, residence, erect, $125,378.
Latonyia Wilson: 9228 N Rockwell Ave., day care center, remodel, $125,000.
Home Creations Inc.: 6529 NW 163 Terrace, residence, erect, $120,500.
Rausch Coleman Homes OKC LLC: 11509 NW 100, residence, erect, $120,000.
Rausch Coleman Homes OKC LLC: 10708 SW 23 Place, residence, erect, $120,000.
Rausch Coleman Homes OKC LLC: 9908 Ruger Road, residence, erect, $120,000.
Rausch Coleman Homes OKC LLC: 9912 Ruger Road, residence, erect, $120,000.
Rausch Coleman Homes OKC LLC: 9904 Ruger Road, residence, erect, $120,000.
Rausch Coleman Homes OKC LLC: 9909 Ruger Road, residence, erect, $120,000.
Rausch Coleman Homes OKC LLC: 9920 Ruger Road, residence, erect, $120,000.
Rausch Coleman Homes OKC LLC: 10004 Ruger Road, residence, erect, $120,000.
Rausch Coleman Homes OKC LLC: 9916 Ruger Road, residence, erect, $120,000.
Rausch Coleman Homes OKC LLC: 10000 Ruger Road, residence, erect, $120,000.
Home Creations Inc.: 4205 NW 152, residence, erect, $113,500.
Home Creations Inc.: 4112 NW 152, residence, erect, $112,000.
Legacy Homes Inc.: 8725 Yassir Blvd., residence, erect, $110,000.
Legacy Homes Inc.: 9505 NW 88, residence, erect, $105,000.
Legacy Homes Inc.: 8717 Yassir Blvd., residence, erect, $105,000.
Legacy Homes Inc.: 8721 Yassir Blvd., residence, erect, $105,000.
Home Creations Inc.: 11544 SW 8 Circle, residence, erect, $104,500.
Home Creations Inc.: 11517 SW 8 Circle, residence, erect, $102,500.
Ideal Homes of Norman LP: 15709 Bennett Drive, residence, erect, $101,000.
Lingo Construction: 100 N Broadway Ave., restaurant, remodel, $100,000.
Lingo Construction: 100 N Broadway Ave., business, remodel, $75,000.
Westpoint Developers LLC, doing business as Westpoint Homes: 15925 Sarno Lane, residence, erect, $75,000.
Westpoint Developers LLC, doing business as Westpoint Homes: 15912 Sarno Lane, residence, erect, $75,000.
Michael Bell: 15501 Wythe Lane, manufactured home, move-on, $74,000.
Jon Graves: 3409 Prairie Grass Road, residence, remodel, $70,000.
KK Trust Properties LLC: 1621 NW 19, residence, add-on, $70,000.
AT&T Bricktown Ballpark: 2 S Mickey Mantle Drive, amusement, erect, $65,000.
Linda Nui Jhong: 6909 W Hefner Road, retail sales, remodel, $55,000.
Emmanuela Guerrero: 1013 SW 54, residence, add-on, $50,000.
Demolitions
No name provided: 823 NW 43, house.
Jazmine Property: 1423 NW 33, house.


Creating a Custom RegEx
Want to have a little programming fun? Let's create a RegEx! What is a RegEx? Well funny you should ask that! A RegEx or Regular Expression is a search string of condensed datum. Let's say you wanted to search for all capital letters in a given set of text. How would you go about it? A RegEx. This post will explore creating a RegEx for few different situations.
Before we start, three resources that will help anyone trying to build a RegEx are the following websites:
Regexr
Rubular
Ruby Docs Regexp
Lets walk through method starts_with_a_vowel
and the steps I used to successfully find a working RegEx. We'll start by picking out a list of words. Find some that are capitalized and some that are not. Pick at least one, or more that starts with each vowel, capitalize some and leave some lower-case. Pick out a few words that start with consonants. Throw in some punctuation. Maybe some numbers. Now we have our list. Go to either Regexr or Rubular and enter your list in the text area. Make sure that you delete any sample text that was there already. In the RegEx area (which you might have to delete as well) you start entering possibilities. Play with it.  See if you can solve it differently than I did.
Now let's do something a little more complicated! Let's see if we can match all valid phone numbers but kick out all invalid phone numbers! You are really going to have to use your imagination and maybe do some research on this one! Here's what I came up with.
I hope you have some fun with this programming problem.


I'm Nikos Anifantis, thanks for visiting this repository.
Nothing special here, I just wanted to say hi! 👋


Rocksmith is wonderful for a musician. Whether you’re new to the guitar, or well used to it - it manages to teach you a little of something new, in a gratifying manner. My musically-challenged flatmate is learning some bass, and I’m expanding my song knowledge in the process, whilst learning to be a little more precise. And as a technoboff I applaud the groundbreaking technology behind it, because it works flawlessly.
But it’s never likely to appeal to the masses as much as the toy-like Rock Band and Guitar Hero games, simply because Rocksmith isn’t really a party game, and the controllers are quite expensive. The main drawback for me however, was that the choice of songs is rather small, and there is a lot of “filler” in the roster. Some classics, but a large number of tunes I’d never heard of, and have no interest in learning. There’s DLC, but it’s very expensive, coming in at around £2.50 a track.
However, I found that you can add your own tunes, relatively easily. It’s not trivial, but it’s not that taxing - it just needs a little time for inevitable trial and error. But you’re rewarded with a personalised song library, at no extra cost - and all of a sudden, it is a party game.
I won’t copy out word for word someone else’s work, so I’ll refer you to this rather excellent tutorial. It requires a little bit of prior preparation though, so I’ll fill you in on that.
In terms of the most important stuff, all you need is the original song (in MP3 or OGG format, a good converter is here), version 1.8 or above of the “Editor on Fire” software, the latest version of the Rocksmith GUI toolkit, and the free WWise audio software. It helps to have Guitar Pro (used in the tutorial to import the correct “tab” form of the track), but TuxGuitar will do just as well. For the .dds file used as album art in game, you can use the free Paint.NET software.
Things to consider for a Rocksmith track:
- You’ll need a few seconds of leading silence to help the player out. There’s a tool for this in the Editor on Fire software.
- You’ll need to add sections (Intro, Verse, etc) so that the song will actually end, in the game. Again, this is provided in the Editor on Fire software.
- If you have time, provide different difficulties for your track in the Editor on Fire software. Otherwise, you’ll be stuck at max difficulty - not so good for new players.
-
To be recognised in-game inside Steam, you’ll need your own personal DLC key (a six figure number). Search around on Google for how to do this, but you can use the Rocksmith GUI Toolkit software to unpack a DLC you’ve bought, to obtain this number. Before attempting, you should probably know a little about:
- Making and editing MIDI files
- How to play the guitar itself (to ensure that your in-game tab is accurate/feasible)
- General knowledge of Digital Audio Workstations (big music software packages like FL Studio or Cubase)
- Graphic creation
- General Windows file management Good luck!


--- layout: post title: Windows 7 has been released to manufacturing date: '2009-07-23T08:23:00.001+01:00' author: solrevdev tags: modified_time: '2009-07-23T08:23:38.879+01:00' blogger_id: tag:blogger.com,1999:blog-31446756.post-4693835532884751496 blogger_orig_url: http://www.solrevdev.com/2009/07/windows-7-has-been-released-to.html ---


Please Brittany Ferries be on time!!
Last week-end (18-19th August 2012), I went back to France to see my family. This was only a couple of days, and was a very nice stay. However, it all got tarnished by Brittany Ferries’ failure to be on time, and having to spend the night in Portsmouth, stuck between Cherbourg and London.
The reason? The boat being late, I missed the last 2 trains going back towards London that night, and had to spend it in a hotel, before being able to get on a train the next morning.
Brittany Ferries schedule is going out the window this summer. No need to stalk them in harbours, one only has to have a look at their Twitter timeline (https://twitter.com/brittanyferries) to understand what’s happening: they don’t have enough time to load and unload boats in the time they’ve allocated. So the last services get the cumulative effect of a day of lateness. Any other attempt to explain it would be fallacy!
I unfortunately didn’t spot this before leaving on Friday, but they even seemed to know and acknowledge the problem!
This is seriously disrespecting their passengers and not taking the problem seriously enough.
I wrote them an open letter that is available at the bottom of this post. Hopefully they’ll pay me back the hotel stay, and more importantly, they’ll react properly and stop advertising such aggressive timescales, when they’re fully aware they can’t hold them! Recent history says they won’t, but I’ll give them the benefit of the doubt.
Everyone gets annoyed at these delays, first of all their staff which is under pressure to deliver despite bad planning. They even recently went on strike to protest against what Brittany Ferries calls their “measures to improve competitiveness”… What competition except their own finances?
While upper deck staff tries to make up for it by being always smiling, cheerful and helpful, I can’t say the same about below deck staff, with its all too often monosyllabic grunts at cyclists and failure to understand how much a proper bike is worth. And No! It cannot be tied up next to the bins, outside, in the spray, at the back of the boat! Some of my rides are more expensive than some of the cars in the ferry! They deserve every single bit of attention possible, if not more, and that should include a proper area to transport them. It’s not that bikes take much space do they?
Now, you might be wondering why I’m not going to check other crossing services to see if services are better? After all, when one’s unhappy, ditch the provider an get another one, right?
Well, it’s not that easy you see: there aren’t any other options! At all! There use to be competition between Brittany Ferries, P&O and Stena Line “back in the days”, but Brittany Ferries are now the only ones left operating this line, and they don’t even run this service in the winter.
My options to go to my parents place with a bike (no concessions on that!) in the summer are therefore as follows:
- Brittany Ferries. Take a train from London to Portsmouth, ride to the ferry port, embark/disembark with the cars and ride home. This usually takes around 8h (If not for the water, I could cycle it faster!!!), plus the time zone change, but with a nice lie-in on the boat. Easy, albeit long.
- Train all the way. Yes, there is indeed a train from London to Paris, and I could take another one from Paris to Cherbourg. But there are a few problems: I have to change train stations in Paris, which is fine in the summer, but in winter: no so easy. But most importantly, one cannot take a bike fully built, as a “walk-in” item, even in a postal carriage (if they existed), on the Eurostar. You’ve got to put it in a bike bag or box, like you’d do on a plane, or leave it on wheels with bubble wrap for baggage handlers to muck up. For having done it once with a hard bike box, I can confirm they receive the same hard treatment than in airpports.  Not an option.
- Take the plane! After all, everyone travels by plane these days! And there ought to be something landing near my parents place! Nope. The closest airport is Caen-Carpiquet, which has no flights from/to London. The next closest one is Paris… And even if it did, traveling to Gatwick, Luton or Stansted with a bike in a box for a week-end is, like point 2, not a viable option.
- Other ferries to other ports. Yes, there’s another one arriving in Ouistreham, but that puts me 120Km away from my parents, 20Km away from the closest train station, and the sailing times aren’t appropriate either to be there in time and justify all this hassle for a week-end. There’s one arriving in Roscoff from Plymouth. 5h away from London and 5h away by car from Cherbourg. And both these are also ran by Brittany Ferries, so even if these routes were better organised, they would be no way better. Other ferries are from dover to Calais. Look for Calais on a map of France, find Cherbourg, 500Km away, and realise it’s not an option either.
So I’m afraid I don’t have the choice! I’ve got to use this ferry on this route, and if I want to spend at least a couple of days with my family, then I need to make the trip from Friday to Sunday evening. Late on Friday? I’m there late, but I can cycle to my parents place. Late on Sunday… Yep! You guessed it! Not so good.
So please please Brittany Ferries, don’t advertise arrival times you can’t hold, and stop thinking all your passengers are traveling in a car for which 1/2h delay won’t make much difference!  If you’re late, own up to it and propose solutions to the ones who’s travel plans you’ve ruined by overlooking the time it takes to load and unload your boats. In these cases a twitter apology is nowhere near enough. “Inconvenience”? A bit more than that I’d say! https://twitter.com/brittanyferries/status/237304111509422080
Thank you!
Public letter to Brittany Ferries
To whom it may concern.
Hello,
I live in London and my family resides in Cherbourg, Normandy, right across the channel from Portsmouth, about 200Km from where I live, as the crow flies.
 The boat was initially supposed to arrive at 22:15, and the last two trains are at 22:37 and 22:53 respectively, which are both perfectly catchable if the boat is on time.
On Sunday, it wasn’t.
Apparently, and according to your own twitter account, the Normandie Express was late at some point during the day, and that delay cascaded down all the way to the last services. This was the case as well the Friday before, as I arrived 45min late in Cherbourg, and judging, again, by the information you post on twitter, was also the case before and after this week end for many other passengers.
This twitter timeline, at the moment, paints a pretty bleak picture of your time planning. I know that there can be sometimes problems with the weather or harbour operations, especially in Portsmouth, and this is totally fine. What’s not alright is regular delays incurred by what looks like avoidable planning issues.
In the peak season, there’s extra traffic, so extra loading / unloading time of a full boat should be built in its journey time. When the boat is late, staff gets all worked up because of the turnaround time being too short, and people relying on a connection, even with some leeway as I had, always get the short straw. For most car passengers, this isn’t really an issue. They only arrive at their destination a bit later. Other passengers don’t have the same luxury.
Last Sunday, despite sprinting off the harbour as fast as I could on my bike (and I’m no slouch!), I missed the very last train by 3 minutes. There was no other train to go back to London after this one, and I had to spend the night in Portsmouth, paying a higher ‘drop-in’ fee for a room, and could only take an off-peak train the next morning as I was on a bike.
I am lucky to have a job quite flexible with working hours: arriving 2h late on Monday morning was alright given the circumstances, and will be made up for during the week, but the story might have been very different if not for this flexibility, and I think you’re getting away very lightly when you say “we’re sorry” or “sorry for the inconvenience”, on Twitter only.
This was much more than an inconvenience, and I am asking you to show that you really care about, and listen to, your clients, by paying for my hotel night in Portsmouth. This would show you understand and acknowledge there is a problem in the services you run, and would, indeed, make up for the “inconvenience” of not spending the night, relaxed, in my own home from your own fault. I have enclosed a copy of my hotel bill and my bank details should you want to make a direct payment, but you can also send me a cheque if you prefer. All my details are below.
Also, if you’re interested to know why I still use your services, I invite you to read my article, accompanied by this letter which I’m making public, at the following address:
http://ekynoxe.com/please-brittany-ferries-be-on-time/
Thanks for taking the time to read. Kindest regards
Mathieu Davy


Move4Mobile iOS Tools
Add this repo to your Podfile to add some of the most used helpers and extensions from the devs of Move4Mobile.
pod 'Move4MobileTools'
Move4MobileTools Docs (64% documented)
Add this repo to your Podfile to add some of the most used helpers and extensions from the devs of Move4Mobile.
pod 'Move4MobileTools'


This a submit button
This a reset button
This an anonymous button
How many beans can you eat?
When are you available this summer?
75/100
75


Welcome to my page, i am still working on it but i'll do my best! :)
I try to upload my projects to github, for example you can find this pages's on there.
github.com/tuggan »
See who i am on keybase!
keybase.io/vestern »
Nothing to put here yet
Read more »
I am currently working on an admin interface for your home network togeather with a friend.
Check it out on github