Validate models
Validation tests
Defining validations
We have already seen the table of validation tests in the default view of the Model Catalog:
EBRAINS uses the SciUnit framework, which applies the idea of unit testing, widely used in software development, to model validation. In SciUnit, a Test encapsulates:
- the experimental data against which the simulation results are compared
- the data analysis code used to transform the data generated by the model (e.g. spike times, membrane potential traces) in the form needed for comparing to the data (e.g. a histogram)
- the measure used to quantify the difference between simulation results and experimental data
and is implemented as a Python class.
Viewing
Viewing validation test definitions
The table of validation test definitions can be configured and filtered as shown for models in Find models To view the detail of a test definition, click on a row in the table. The "Info" tab shows a description of the test definition, assorted metadata, and gives links to the code/data of all versions of the test code.
To return to the table of models/tests, click "×" in the upper-right corner.
Results
Viewing validation results
The "Results" tab shows the numerical results of validation tests, where the "Score" is a measure of the difference between the experimental data and the simulation results. When accessed from a test definition, the Results tab shows the scores for all models that have been validated with that test:
When accessed from a model entry, the Results tab shows the scores for all tests that have been used to validate that model:
The "Figures" tab shows the same information in graphical form:
In detail
Inspecting individual validation results in detail
If you click on an individual score in the Results tab, this opens a dialog showing the details of the test run that generated the score.
Often, the plain numerical score is not very informative about how the simulation results differed from the experimental data, and so validation test runs can also generate additional figures and tables to provide a more in-depth view, available in the "Result Files" tab.
Running
Running validation tests
We've seen how to view validation test results, how are these results generated? i.e. how do we launch the tests, run the simulations, and upload the results?
Anyone with an EBRAINS account can use the Validation Framework Python client to do this on their own computer, on a supercomputer (if they have access), or in an EBRAINS Collaboratory Jupyter notebook (if the memory requirements of the simulation do not exceed the limits of the virtual machine running the Python kernel).
Installing the Python client
pip install hbp_validation_framework
Downloading a model
pip install hippounit
from hbp_validation_framework import ModelCatalog, TestLibrary
from hippounit.utils import ModelLoader_BPO
catalog = ModelCatalog(username="my_ebrains_username")
model_name = "CA1_pyr_cACpyr_mpg141208_B_idA_20190328144006"
model_info = catalog.list_models(name=model_name)[0]
local_path = catalog.download_model_instance(model_id=model_info["id"], version="2.0")
model = ModelLoader_BPO(name=model_name, model_dir=local_path,
SomaSecList_name="somatic")
The first time you run this, you will be asked to enter your password. For subsequent runs, the client will store a token which it will use to reconnect, until the token expires, when you will be asked again for the password. If you run this in a Jupyter notebook inside the EBRAINS Collaboratory, you should not need to enter either a username or password, as the authentication you already provided for the Collaboratory also gives access to the Model Catalog.
Downloading a validation test
library = TestLibrary.from_existing(catalog)
validation_test = library.get_validation_test(alias="hippo_ca1_bap", version="1.3.5")
Running a test and registering the result
result = validation_test.judge(model)
library.register_result(result)
This final step will run a simulation (provided you have NEURON, the simulator needed for this model, installed on your machine), process the simulation data, and compare the result to the reference experimental data. It then uploads the result (including the additional files and images generated during the test) to the Model Catalog.
For more details, see the full Python client documentation.
Contact
Any questions?
If you have questions that are not answered here, contact us by e-mail: support@ebrains.eu


a full-stack developer is familiar with all layers in software development. These developers aren't experts at everything;
but they have functional knowledge of all the stages and the ability to take a concept and turn it into a finished
product.
Our program, who art in memory, called by thy name; thy operating system run; thy function behave at runtime as it has during
development. Give us this day our daily output. And forgive us our code duplication, as we forgive those who duplicate
code against us. And lead us not into frustration; but deliver us from syntax errors. For thine is the algorithm,
the computation, and the solution, looping forever and ever.
Return;


--- layout: default title: Request For Proposal description: "" class: contact banner: CS_arapahoe_lakes.jpg banner-heading: "Let Us Help You Today!" banner-cta: "Request For Proposal" ---
* Indicates a required field


Work Day Scheduler
A simple calendar app for scheduling your work day hour
8 AM
9 AM
10 AM
11 AM
12 PM
1 PM
2 PM
3 PM
4 PM
5 PM
Delete All Tasks


Sunjic, Colin, Camp - How Birmingham City's players rate this season
With the 2019/2020 Championship season a third of the way through Birmingham City writer Brian Dick gives his 16-game ratings for Pep Clotet's men
by Brian Dick, https://www.facebook.com/blueswriterbriandick/?fref=tsBirmingham City’s season is already a third over, sixteen games have gone, 30 remain and the team sits mid-table in the Championship.
With virtually as many victories, seven, as losses - eight, Pep Clotet’s adventurous team seems to be both living and dying by the sword of their footballing ambition.
The caretaker head coach is building a team that doesn’t deal in draws, it’s all out for the three points.
Some of his players have thrived in that environment, others have found it harder, but with the international break upon us Blues writer Brian Dick rates them for their bodies of work so far.
We’ve focused on the players who have appeared in at least half the games.
Birmingham City latest
This is your round up of all things Birmingham City; the one-stop shop that will keep you updated on the latest goings on at the club and beyond.
Transfers, injuries, match-days and managers, we’ve got you covered.
We’ll be bringing the very latest on the Blues throughout the week and around the clock.
Make sure you keep yourself updated with our handy daily catch up...
Must reads...
- Sunjic, Colin, Camp - How Birmingham City's players rate this season
- 'Massive' - Pep Clotet reveals how this player can transform Birmingham
- 'All depends' - Birmingham fans assess play-off prospects
- 'Bitterly disappointing' - Championship boss speaks out as Birmingham given injury boost
For all your latest BCFC news, opinion, analysis and transfer gossip, click here
You can follow our Blues reporter Brian Dick on Twitter, and his email address is<EMAIL_ADDRESS>Lee Camp 5.5
The season is three months old, so it’s important not to merely reflect recent slips.
Without being spectacular Camp was solid enough throughout the first half of the campaign, with particularly important saves against Bristol City and Charlton.
However, errors against Derby, Wigan and most recently Fulham are set against good stops in the wins over Blackburn and Luton and reignited the debate about Blues ’ best goalkeeper.
Appearances: 16
Maxime Colin 9.5
Probably had more defending to do that in many other games this season against Fulham - but despite that he has arguably been the chief beneficiary of the emphasis on attack.
There is no doubt this already highly-regarded right back is enjoying his best season at Birmingham City. One very good goal and three assists to his name so far, with more to come. The gold standard.
Appearances: 14
Harlee Dean 6.5
Started as though he was still feeling the effects of the groin trouble that bothered him towards the end of last season.
However, looked to have settled in recent weeks, helped no doubt by the growing assurance of Marc Roberts.
His suspension has opened the door to the equally composed Jake Clarke-Salter, though.
Appearances: 15
Marc Roberts 8.5
Pep Clotet invested his trust in Roberts at the beginning of the season and he has not been let down.
The 29-year-old is another who is enjoying his finest campaign at Blues, though the previous two have hardly been stellar.
He’s looking every inch the impressive Championship defender Blues thought they were getting in 2017.
Appearances: 15
Kristian Pedersen 8
Has had to adjust his game as much as anyone in the squad this season - and has exceeded the expectations of many.
He has shown himself to be a threat in the opposition box, is attacking with increasing conviction and against Fulham led the charge for an equaliser.
Apart from one slip against Luton and a little bit of bother with Anthony Knockaert, he’s largely been excellent.
Appearances: 16
Ivan Sunjic 9
Imposing and dynamic, Blues’ club record signing has thrived since bedding in.
The destructive side to his game has gone up a level and he’s chipped in with two beautiful goals in the last half of this campaign.
Used to playing as the sole holding midfielder he’s had to adapt to playing with a partner - but good players do that.
Appearances: 14 (1 sub)
Jude Bellingham 7.5
Strictly speaking his age should not be considered, if only because he is capable of playing with the authority of a senior professional. Equally you can’t ignore it.
Bellingham has started the last six matches at the heart of Birmingham City’s team. There have been some demanding experiences, at Cardiff and Leeds and again against Fulham, but no experience is a bad one.
He’s been handled well so far and has brought a reliable touch and energy to the middle of the park. It’s going to be important to mange his games in the next six weeks, though.
Appearances: 8 (4)
Gary Gardner 6
Most of his starts came at the beginning of the season when the team was looking to find itself.
His goal against Derby sparked what should have been another Pride Park miracle and he’s done reasonably well when he’s come off the bench
Unfortunate to be pick up a freak injury at Leeds because he’d be a useful part of the match-day squad at the moment.
Appearances:5 (6)
David Davis 6
Tended to share time with Gardner in the early part of the season but has been restricted to bench duty since Bellingham’s emergence.
Decent enough in his last start against Blackburn, albeit with a late slip, but with limited opportunities it’s been difficult to get into his stride.
Appearances: 9 (1)
Jacques Maghoma 6
Blues really miss him when he’s not available, if only off the bench. His ability to carry the ball upfield and his defensive diligence are important facets to Pep Clotet’s tool box.
There’s loads of competition in the wide areas now but Maghoma’s experience has kept him around the squad.
Appearances: 2 (7)
Dan Crowley 9
Until Bela arrived there was little question that Crowley’s transfer had been the most creative piece of business Blues have done this year.
The sheer mechanics of the Bela deal might change that but what it won’t alter is the fact that Crowley is almost always in the top two of three rated players every week.
His partnership with Maxime Colin is perhaps the most effective in the side - his ability to find space and dictate play has been a pleasure to watch.
Appearances: 13 (2)
Fran Villalba 7
There have been times when the Spaniard has matched the standard set by Crowley on the opposite side of midfield.
However, with both players it’s all about end product and there’s been marginally more of that from Crowley.
The last couple of games have been frustrating with not enough quality on his shooting or delivery into the box.
Appearances: 12 (2)
Alvaro Gimenez 6
Has dropped out of the team since the last international break with Kerim Mrabti stepping up as Lukas Jutkiewicz’s strike partner.
He started the season with excellent intent, his goal against Barnsley was superbly taken, but he’s found it difficult to add to his sole strike.
Coming off the bench he has shown good intent, without applying the finishing touch he is clearly so desperately craving.
Appearances: 7 (7)
Kerim Mrabti 6
Came into the season with a good opportunity but plenty to prove - and he had to wait for his chance to do that.
The door has opened in recent weeks and Mrabti has shown good mobility and endeavour - but he really could do with a goal after failing to convert from a few feet out against Cardiff.
There have been two or three really promising displays of late but we’re still waiting for that truly commanding one.
Appearances: 6 (2)
Lukas Jutkiewicz 7
For all the evolutionary play around him Blues still rely heavily on the aerial and physical prowess of their 30-year-old target man.
He’s got four goals this season, all with his head - but could have had more, hitting the underside of the bar against Blackburn, a tap in against Luton and a penalty against Derby.
He occupies defenders and Pep Clotet wouldn’t be without him, the arrival of Bela and return to fitness of Jefferson Montero could be just what he needs.
Appearances: 16
Others
Appearances (as sub)
Odin Bailey 0 (5)
Steve Seddon 3 (1)
Wes Harding 2 (2)
Jefferson Montero 1 (2)
Jake Clarke Salter 2