Located on the north west coast of Australia, in the Kimberley region. It is a very popular tourist destination.
This town was once the pearling capital of Australia. It now concentrates on the production of cultured pearls.
Position 17°58'S, 122°14'E.
Contact the Customs at Port of Broome on VHF Ch16 for instruction regarding berthing.
Last updated February 2014.
There is no marina here, just two anchorages. The Department of Transport must be advised before dropping your anchor. They can also advise whether a mooring is available. See the contact below.
One anchorage is at Gantheaume Point (17° 57.86' S, 122° 11.55' E). The are some mooring buoys here but they need to be booked in advance. This is the only anchorage which is safe from the south easterlies common in the dry season.
The other anchorage is in Roebuck Bay (17° 59'S, 122° 14'E). This is close to town, and therefore convenient for shore trips, but this bay is subject to the East winds that whip up an uncomfortable fetch. Also with the tidal range of up to 10m dinghy trips ashore need to be well planned.
If you can take the ground, then Dampier Creek is an option with the added advantage of access to Streetier Jetty for getting ashore.
Fuel can be obtained here, but with difficulty. It requires berthing at the commercial wharf and it is necessary to give 24hr notice. Since the wharf is high, it is best to do the re-fueling at high tide.
Water is also available at the commercial wharf.
Broome has all the shops you would expect in a thriving regional tourist town, including supermarkets. But provisioning here is difficult as having overcome the difficult access to the shore, there is no transport into the town except taxis.
Last updated February 2014.


modformPro package¶
Submodules¶
modformPro.data_simulator module¶
-
class
modformPro.data_simulator.
DataSimulator
(specifications=None, D_sites_modtypes=None)[source]¶ Bases:
object
This class simulates modform distribution and then simulates the MS process on the simulated modform distribution.
Based on fields set by the user in the specification file, peak areas for BUMS/MRM or TDMS or both are calculated which sets the stage for estimation of modform region from the simulated data.
Parameters: specifications :
ModformProSpecifications
- ModformSpecifications object
- Contains information about the options set by the user in the specification file.
D_sites_modtypes : dict
- Python dictionary containing aa residues mapped to the list of possible modifications.
- In the current implementation, this is set after the creation of the instance object of this class.
-
D_sites_modtypes
¶ Instance attribute
- Python dictionary having aa residues mapped to the list of possible modifications
- E.g.,
D = {'T32': ['P'], 'S55': ['Ac', 'P']}
-
find
(seq_string, character, proline_check=False)[source]¶ Instance method
- This is a subpordinate method, i.e., only called within the same class by a member method
- Returs a list of positions of amino-acids (single alphabet representation) in a string
- Note that position starts from 1
- Note that the method is case-sensitive
Parameters: seq_string : str
amino-acid sequence.
character : str
Character whose position(s) in the sequence is to be found.
proline_check : bool
If this is set to be
True
then positions of the amino-acids preceding proline will be ignored. Default isFalse
.Examples
>>> from data_simulator import DataSimulator >>> >>> sim_data = DataSimulator() >>> >>> aa_sequence = 'MSRYSRT' >>> L_arginine_locations = list(sim_data.find(aa_sequence, "R", proline_check=False)) >>> print L_arginine_locations [3, 6]
-
generate_BUMS_MRM_data
()[source]¶ Instance method
- This method computes peak areas for BUMS-MRM method
- This is done only if the option
FILE_BUMS_MRM_DATA
is specified in the specification file. The content is then printed in the file specified. - User can choose
Trypsin
orGluc
or both in the specification file - Noise can also be introduced in the peak area values
-
generate_TDMS_data
()[source]¶ Instance method
- This method computes peak areas for Intact mass spectrometry for simulated modform distribution.
- This is done only if the option
FILE_TDMS_DATA
is specified in the specification file. The content is then printed in the file specified. - Noise can also be introduced in the peak area values.
Notes
Prints a file having information of the isomeric modforms and their respective percentage peak areas.
-
generate_modform_distribution
()[source]¶ Instance method
This method generates simulated modform distribution and populates the instance attribute
modform_space
.
-
get_dictionary_modification_frequency_for_peptide_in_modform
(N_loc, C_loc, modform_object)[source]¶ Instance method
- This is a subordinate method, i.e., only called within the same class by a member method
- This function provides
D_peptide_mod_frequency
for a peptide (N_loc
,C_loc
) in a given protein modform
Parameters: N_loc : str or int
N terminal position of the peptide in the protein sequence
C_loc : str or int
C terminal position of the peptide in the protein sequence
modform_object :
Modform
Modform object
Returns: D_peptide_mod_frequency : dict
Python dictionary containing modifications mapped to their frequencies for a peptide (
N_loc
,C_loc
).Examples
>>> from data_simulator import DataSimulator >>> from modform import Modform >>> >>> sim_data = DataSimulator() >>> mform = Modform(var_name='a15', D_protein_sites_modifications={'S15': 'P', 'T16': 'P', 'S54': 'P'}) >>> >>> N_loc = 1; C_loc = 24 # peptide - (1, 24) >>> D_peptide_mod_frequency = sim_data.get_dictionary_modification_frequency_for_peptide_in_modform(N_loc, C_loc, mform) >>> print D_peptide_mod_frequency {'P': '2'} >>> >>> N_loc = 50; C_loc = 58 # peptide - (50, 58) >>> D_peptide_mod_frequency = sim_data.get_dictionary_modification_frequency_for_peptide_in_modform(N_loc, C_loc, mform) >>> print D_peptide_mod_frequency {'P': '1'}
-
get_dictionary_peptides_isomeric_peak_areas
(D_variables_modforms, L_peptides)[source]¶ Instance method
- This is a subordinate method, i.e., only called within the same class by a member method.
- This method computes the percentage peak areas for isomeric peptides from their abundance and returns a dictionary of peptides mapped to their isomeric peak areas.
Parameters: D_variables_modforms : dict
Python dictionary of modform variables mapped to their
Modform
objects.L_peptides : list
Python list of tuples of peptides, e.g.,
L = [(1, 6), (7, 15), (16, 30)]
. Each tuple is starting and ending position of a peptide in the protein sequence.Returns: defaultdict(list)
Examples
>>> from data_simulator import DataSimulator >>> from modform import Modform >>> >>> D_variables_modforms = {} >>> D_variables_modforms['a1'] = Modform(var_name='a1', D_protein_sites_modifications={'0': 'UNMOD'}) >>> D_variables_modforms['a2'] = Modform(var_name='a2', D_protein_sites_modifications={'S20': 'P'}) >>> D_variables_modforms['a3'] = Modform(var_name='a3', D_protein_sites_modifications={'S15': 'P'}) >>> D_variables_modforms['a4'] = Modform(var_name='a4', D_protein_sites_modifications={'S15': 'P', 'S20': 'P'}) >>> >>>for k,v in D_variables_modforms.iteritems(): ... if k=='a1': v.percentage_abundance='0.000' ... elif k=='a2': v.percentage_abundance='11.111' ... elif k=='a3': v.percentage_abundance='44.444' ... elif k=='a4': v.percentage_abundance='44.444' >>> >>> D_sites_modtypes = {'S20': ['P'], 'S15': ['P']} >>> sim_data = DataSimulator(D_sites_modtypes=D_sites_modtypes) >>> L_peptides = [(1, 5), (6, 10), (11, 20), (21, 32)] >>> >>> D = sim_data.get_dictionary_peptides_isomeric_peak_areas(D_variables_modforms, L_peptides) >>> print D defaultdict(<type 'list'>, {'11-20': [[{'UNMOD': '0'}, '0.000'], [{'P': 1}, '55.555'], [{'P': 2}, '44.444']]})
-
get_list_isomeric_modforms_peak_areas_lists
(D_variables_modforms)[source]¶ Instance method
Its a subordinate method, i.e., called within the class only by a member method.
Parameters: D_variables_modforms : dict
Python dictionary of modform variables mapped to their modform objects.
Returns: list
Examples
>>> from data_simulator import DataSimulator >>> from modform import Modform >>> >>> D_variables_modforms = {} >>> D_variables_modforms['a1'] = Modform(var_name='a1', D_protein_sites_modifications={'0': 'UNMOD'}) >>> D_variables_modforms['a2'] = Modform(var_name='a2', D_protein_sites_modifications={'S20': 'P'}) >>> D_variables_modforms['a3'] = Modform(var_name='a3', D_protein_sites_modifications={'S15': 'P'}) >>> D_variables_modforms['a4'] = Modform(var_name='a4', D_protein_sites_modifications={'S15': 'P', 'S20': 'P'}) >>> >>> for k,v in D_variables_modforms.iteritems(): ... if k=='a1': v.percentage_abundance='0.000' ... elif k=='a2': v.percentage_abundance='11.111' ... elif k=='a3': v.percentage_abundance='44.444' ... elif k=='a4': v.percentage_abundance='44.444' >>> >>> D_sites_modtypes = {'S20': ['P'], 'S15': ['P']} >>> sim_data = DataSimulator(D_sites_modtypes=D_sites_modtypes) >>> L_peptides = [(1, 5), (6, 10), (11, 20), (21, 32)] >>> >>> D = sim_data.get_dictionary_peptides_isomeric_peak_areas(D_variables_modforms, L_peptides) >>> print D defaultdict(<type 'list'>, {'11-20': [[{'UNMOD': '0'}, '0.000'], [{'P': 1}, '55.555'], [{'P': 2}, '44.444']]})
-
get_peptides_from_GluC_digestion
(aa_sequence)[source]¶ Instance method
- This is a subordinate method, i.e., only called within the same class
- The function returns a list of tuples of start and end locations of peptides resulting from Gluc digestion
- GluC digests at the carboxyl side of glutamic acid (E) and aspartic acid (D)
- Note that peptides strictly larger than 4 amino-acid residues are reported
- List of peptides cross-examined using ExPaSy PeptideCutter: http://web.expasy.org/peptide_cutter/
Parameters: aa_sequence : str
amino-acid sequence
Returns: list
Examples
>>> from data_simulator import DataSimulator >>> >>> sim_data = DataSimulator() >>> >>> aa_sequence = 'MFLTRSEYDRGVNTFSPEGRLF' >>> L_tuples = sim_data.get_peptides_from_GluC_digestion(aa_sequence) >>> print L_tuples [(1, 7), (10, 18)]
-
get_peptides_from_Trypsin_digestion
(aa_sequence)[source]¶ Instance method
- This is a subordinate method, i.e., only called within the same class
- This method returns a list of tuples of start and end locations of peptides resulting from Trypsin digestion
- With Proline check - Trypsin digests at the carboxyl side of lysine or arginine unless they are followed by proline
- Note that peptides strictly larger than 4 amino acids are reported
- List of peptides cross-examined using ExPaSy PeptideCutter: http://web.expasy.org/peptide_cutter/
Parameters: aa_sequence : str
amino-acid sequence
Returns: list
Examples
>>> from data_simulator import DataSimulator >>> >>> sim_data = DataSimulator() >>> >>> aa_sequence = 'MFLTRSEYDRGVNTFSPEGRLF' >>> L_tuples = sim_data.get_peptides_from_Trypsin_digestion(aa_sequence) >>> print L_tuples [(1, 5), (6, 10), (11, 20)]
-
modform_space
¶ Instance attribute
Instance of a
ModformSpace
object
-
whether_peptide_contains_any_modification_site
(N_loc, C_loc)[source]¶ Instance method
- This is a subordinate method, i.e., only called within the same class by a member method
- The method tells whether the peptide (N_loc, C_loc) contains any modification site
Parameters: N_loc : str or int
N-terminal location of the peptide in the aa sequence
C_loc : str or int
C-terminal location of the peptide in the aa sequence
Returns: bool
Notes
A given peptide specified by its
N_loc
andC_loc
residue positions is examined for containing any modification site. The information on modification sites is contained in the dictionaryD_sites_modtypes
, which stores the data from an input file.
modformPro.feasible_linear_constraints module¶
-
class
modformPro.feasible_linear_constraints.
FeasibleLinearConstraints
(linear_constraints)[source]¶ Bases:
object
This class provides attributes and methods to obtain feasible linear constraints by removing the inconsistency introduced possibly due to measurment errors.
Parameters: linear_constraints :
LinearConstraints
Set of linear constraints.
-
feasible_linear_constraints
¶ - Instance attribute
- New instance of
LinearConstraints
object. - It is populated by member method
-
get_elastic_constraints
()[source]¶ Instance method
This method creates an instance of the class LinearConstraints where elastic/slack variables are added.
Returns: E_linear_constraints :
LinearConstraints
Linear constraints with added pair of elastic (slack) variables to each constraint.
-
modformPro.globalfuncs module¶
-
class
modformPro.globalfuncs.
TexFileGenerator
(ftex_out, output_dir)[source]¶ Bases:
object
This class prints the output content in a tex file.
Parameters: ftex_out : str
File name with .tex extension
output_dir : str
Name of the output directory
-
add_formatted_fasta_seq_in_the_file
(protein_seq, D_sites_modtypes)[source]¶ Instance method
This function adds the fasta seq string in a particular format and also highlights modification sites by a bold blue and underlined.
-
add_grouped_variables_in_tex_file
(D_grouped_variables)[source]¶ Instance method
Add coalesced variables in the tex file.
Parameters: D_grouped_variables : dict
Python dictionary mapping new coalesced variable to a sum of variables.
-
add_latex_code_in_the_file
(tex_string)[source]¶ Instance method
Adds latex code in the tex file.
Parameters: tex_string : str
Latex code string to be included.
-
add_modform_region_plot_in_tex_file
(mform_region_plot_pdf)[source]¶ Instance method
Add modform region plot in the tex file.
Parameters: mform_region_plot_pdf : str
Name of the file with a pdf extension containing modform region plot.
-
add_modforms_in_tex_file
(D_variables_modforms)[source]¶ Instance method
Adds modform variables and their modification maps in the tex file.
Parameters: D_variables_modforms : dict
Python dictionary mapping modform variables (str) to their
Modform
objects.
-
add_percentage_concentration_bounds_in_tex_file
(D_lower_bounds, D_upper_bounds)[source]¶ Instance method
Adds percentage concentration bounds of modforms in a tex file.
Parameters: D_lower_bounds : dict
Python dictionary mapping modform variables to their lower bouds of percentage concentration.
D_upper_bounds : dict
Python dictionary mapping modform variables to their upper bouds of percentage concentration.
-
add_sites_modifications_table_in_tex_file
(D_sites_modtypes)[source]¶ Instance method
Adds sites and modifications table in tex file.
Parameters: D_sites_modtypes : defaultdict(list)
Python dictionary mapping aa residues to a list of modification types to the respective residues.
-
print_content_in_tex_file
(protein_seq, D_sites_modtypes, D_lower_bounds, D_upper_bounds, mform_region_plot_pdf, D_variables_modforms, D_grouped_variables=None)[source]¶ Instance method
Prints output content in a tex file.
Parameters: protein_seq : str
Amino-acid sequence of the protein under investigation.
D_sites_modtypes : defaultdict(list)
Python dictionary containing residues mapped to the list of possible modifications. E.g.,
D = {'T32': ['P'], 'S55': ['Ac', 'P']}
D_lower_bounds : dict
Python dictionary mapping modform variables to their lower bouds of percentage concentration.
D_upper_bounds : dict
Python dictionary mapping modform variables to their upper bouds of percentage concentration.
mform_region_plot_pdf : str
Name of the file with a pdf extension containing modform region plot.
D_variables_modforms : dict
Python dictionary mapping modform variables (str) to their
Modform
objects.D_grouped_variables : dict, optional
Python dictionary mapping new coalesced variable to sum of modform concentration variables.
-
-
modformPro.globalfuncs.
get_fasta_seq_from_the_file
(file_fasta_sequence)[source]¶ The function reads fasta sequence from the file
-
modformPro.globalfuncs.
get_modforms
(start_index, D_sites_modtypes=None, D_protein_mod_frequency=None)[source]¶ Global method
Provides a set of intact isomeric modforms having same modification frequency and types.
Parameters: start_index : int
Index of the modform variable name.
D_sites_modtypes : defaultdict(list), optional
Python dictionary containing residues mapped to the list of possible modifications. E.g.,
D = {'T32': ['P'], 'S55': ['Ac', 'P']}
D_protein_mod_frequency : dict
Python dictionary containing modifications mapped to their frequencies for isomeric modforms. E.g.,
D = {'P': '2', 'Ac': '1'}
Returns: D_vars_modforms_isobaric : dict
Python dictionary contaitning isomeric modform variables mapped to their
Modform
objects.D_PuLP_variables_isobaric : dict
Python dictionary containing isomeric modform variables mapped to their
pulp
objects.Examples
>>> >>> D_sites_modtypes = {'15':['P']; '16':['P']; '52':['Ac', 'P']; '54':['P']; '59':['P']} >>> >>> get_modforms(start_index=1, D_sites_modtypes) {'0': 'UNMOD'} >>> >>> D_protein_mod_frequency = {'P': '2'} >>> get_modforms(start_index=1, D_sites_modtypes, D_protein_mod_frequency)
-
modformPro.globalfuncs.
get_modforms_from_simulated_data
(D_variables_modforms_simulated=None, D_PuLP_variables_simulated=None, D_protein_mod_frequency=None)[source]¶ Global method
- The function is alternative to the function get_modforms.
- The difference however is that the modform variables and objects are used from simulated data and not generated from the dictionary D_sites_modtypes
- This helps in comparing the results of computed modform distribution to that of the simulated modform distribution.
Parameters: D_variables_modforms_simulated : dict
Python dictionary containing modform variables mapped to their
Modform
objects for simulated data.D_PuLP_variables_simulated : dict
Python dictionary containing modform variables mapped to their
pulp
objects for simulated data.D_protein_mod_frequency : dict
Python dictionary containing modifications mapped to their frequencues.
Returns: D_vars_modforms_isobaric : dict
Python dictionary contaitning isomeric modform variables mapped to their
Modform
objects.D_PuLP_variables_isobaric : dict
Python dictionary containing isomeric modform variables mapped to their
pulp
objects.
-
modformPro.globalfuncs.
natural_keys
(text)[source]¶ Global method
Examples
>>> L = ['a11', 'a9', 'a2'] >>> print sorted(L) ['a11', 'a2', 'a9'] >>> >>> print sorted(L, key=lambda x: natural_keys(x)) ['a2', 'a9', 'a11']
-
modformPro.globalfuncs.
powerset
(iterable)[source]¶ Global method
Returns a chain of elements of powerset of the iterable.
Examples
>>> print list(powerset([1,2,3])) [() (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)]
-
modformPro.globalfuncs.
read_sites_and_modification_types_from_file
(fname_modifications)[source]¶ Global method
- Function populated instance variable, a dictionary, D_sites_modtypes with sites associated to their modfication types
- Read from an input file
- This information must be known apriori
Parameters: fname_modifications : str
Name of an input file containing amino-acid residues and their respective modification types.
Returns: D_sites_modtypes : defaultdict(list)
Python dictionary mapping aa residues to a list of modification types to the respective residues.
E.g.,
{'13':['P']; '16':['P']; '19':['P']; '40':['P', 'Ac']}
modformPro.linear_constraints module¶
-
class
modformPro.linear_constraints.
LinearConstraints
(D_sites_modtypes, specifications=None)[source]¶ Bases:
object
This class provides framework to store python list of linear constraints in string format as well in PuLP format.
Parameters: D_sites_modtypes : dict
Python dictionary containing residues mapped to the list of possible modifications. E.g.,
D = {'T32': ['P'], 'S55': ['Ac', 'P']}
specifications :
ModformProSpecifications
Instance of ModformProSpecifications object.
-
D_grouped_variables
¶ Instance attribute
- Python dictionary having a new coalesced variables mapped to the group of co-existing variables.
- This is populated in a member method if
VARIABLES_GROUPING
isTrue
in the specification file. - E.g.,
D = {'b1': 'a1 + a4', 'b2': 'a3 + a5 + a6'}
-
D_sites_modtypes
¶ Instance attribute
Python dictionary having aa residues mapped to the list ofpossible modifications on that residue.
-
L_PuLP_constraints
¶ Instance attribute
- Python list of linear constraints in PuLP format
- PuLP format is needed to use pulp library for linear programming.
- This is populated by a member method of this class
>>> # Demo L_PuLP_constraints [LpConstraint(LpVariable(a1) + LpVariable(a3) + LpVariable(a5), 30.00), LpConstraint(LpVariable(a2) + LpVariable(a4) + LpVariable(a6), 70.00)]``
-
L_grouped_variables_PuLP_constraints
¶ Instance attribute
- Python list of linear constraints in PuLP format with new coalesced variables
- This is populated by a member method of this class.
-
L_grouped_variables_linear_constraints
¶ - Instance attribute
- Python list of linear constraints with new coalesced variables
- This is populated by a member method of this class.
-
L_linear_constraints
¶ Instance attribute
- Python list of linear constraints in string format
- This is populated by a member method of this class
- E.g.,
L = [('a1 + a3 + a5', '30'), ('a2 + a4 + a6', '70')]
-
define_modform_space_for_linear_constraints
(D_variables_modforms_simulated=None, D_PuLP_variables_simulated=None)[source]¶ Instance method
- Its a subordinate method, only called within the same class.
- This function defines modforms space for linear constraints. It updates the instance attribute,
modform_space
- For simulation mode, predefined
modform_space
object is copied. - For real mode, a fresh
modform_space
object is created.
Parameters: D_variables_modforms_simulated : dict
Python dictionary containing modform variables mapped to their
Modform
objects. This is non-empty for simulation mode.D_PuLP_variables_simulated : dict
Python dictionary containinng modform variables mapped to their
pulp
objects. This is non-empty for simulation mode.
-
derive_linear_constraints_from_MS_measurments
(L_isomeric_modforms_peak_areas=None, D_peptide_isomeric_peak_areas=None, D_variables_modforms_simulated=None, D_PuLP_variables_simulated=None)[source]¶ Instance method
- This method derives the linear constraints from available peak area measurments either from BUMS-MRM or TDMS or both.
- It populates the instance attributes,
L_linear_constraints
,L_PuLP_constraints
Parameters: L_isomeric_modforms_peak_areas : list
Python list containing peak areas of isomeric modforms. This data typically comes from TDMS measurements.
D_peptide_isomeric_peak_areas : dict
Python dictionary containing peak areas of peptides in their different isomeric states. This data typically comes from BUMS-MRM measurments.
D_variables_modforms_simulated : dict
Python dictionary containing modform variables mapped to their
Modform
objects for simulated data. This ensures usage of predefined modform variables for the linear constraints.D_PuLP_variables_simulated : dict
Python dictionary containing modform variables mapped to their
pulp
objects for simulated data. This ensures usage of predefined modform variables for the linear constraints.
-
get_grouped_variables
()[source]¶ Instance method
This function returns a python list of grouped variables.
Returns: list
-
get_linear_constraints_with_grouped_variables
(L_grouped_lists)[source]¶ Instance method
- This method creates List of linear constraints with grouped variables both in string as well as PuLP format thereby, populating the instance attributes,
L_grouped_variables_linear_constraints
,L_grouped_variables_PuLP_constraints
- Note that additional variables are added to the existing dictionary
D_PuLP_variables
, no additional dictionary is needed for grouped variables.
Parameters: L_grouped_lists : list
Python list of grouped variables. E.g.,
L = ['a1 + a3', 'a2 + a4 + a6']
- This method creates List of linear constraints with grouped variables both in string as well as PuLP format thereby, populating the instance attributes,
-
get_sum_of_variables_string
(L_variables)[source]¶ Instance method
- This is a subordinate method, i.e., only called in a member method of the same class.
- The function returns a string of sum of variables given in the input list.
Parameters: L_variables : list
Python list of modform variables in string format.
Returns: str
-
modform_space
¶ Instance attribute
- Instance of a
ModformSpace
object. - This needs to be defined for real data.
- Predefined modform_space is used when running in simulation mode.
- Instance of a
-
specifications
¶ Instance attribute
Instance of a
ModformProSpecifications
object.
-
modformPro.linear_programming_solver module¶
-
class
modformPro.linear_programming_solver.
LinearProgrammingSolver
(linear_constraints)[source]¶ Bases:
object
This class contains attributes and methods to solve linear programming problems.
Parameters: linear_constraints :
LinearConstraints
Set of linear constraints.
-
linear_constraints
¶ Instance attribute
This is an instance of
LinearConstraints
object.
-
linear_programming_solver
(objective_function_PuLP, cmd)[source]¶ Instance method
- This method solves a linear program given list of LP constraints, objective function and command (LpMaximize or LpMinimize).
- It returns LpProblem object containing variables, constraints and solution to the linear problem.
- Note that LP constraints and objective function must be according to
puLP
format. - LP constraint string:lhs = rhs
- PuLP constraint:pulp.LpConstraint(lhs, rhs)
- Objective function should in terms of LP variables in puLP format, e.g., transforming a variable ‘v’ –pulp.LpVariable(v, lowBound=0)
Parameters: objective_function_PuLP : Affine expression of
LpVariable
objectsObjective function for a linear programming problem
cmd : LpMaximize (or -1); LpMinimize (or 1)
Commands maximization or minimization of the objective funtion in a linear programming problem
-
modformPro.mainsolver module¶
-
class
modformPro.mainsolver.
MainSolver
(specifications)[source]¶ Bases:
object
-
run
()[source]¶ Instance method
This function sets the program running either in simulation mode or real data mode.
-
run_real_data_mode
(protein_seq, D_sites_modtypes)[source]¶ Instance method This function compute bounds for real data and calls relevant print output functions.
Parameters: protein_seq : str
Amino-acid sequence of the protein under investigation
D_sites_modtypes : defaultdict(list)
Python dictionary mapping aa residues to python list of modification types on the residues.
-
run_simulated_mode
(protein_seq, D_sites_modtypes)[source]¶ Instance method
This function performs simulation, compute bounds for a given peak area measurment noise and number of samples and calls relevant print output functions.
Parameters: protein_seq : str
Protein amino-acid sequence string.
D_sites_modtypes : defaultdict
Python dictionary having aa residues mapped to list of modifications on respective residues.
-
modformPro.modform module¶
-
class
modformPro.modform.
Modform
(var_name, D_protein_sites_modifications=None, percentage_abundance=None)[source]¶ Bases:
object
This class defines modform, its variable name, representation and other methods to operate on the data-structures of this class.
Demo representations:
a1: [('0', 'UNMOD')]
a11: [('T11', 'P'), ('S30', 'P'), ('S55', 'Ac')]
Parameters: var_name : str
Modform variable name
D_protein_sites_modifications : dict
Python dictionary storing aa residues mapped to modifications on the residues.
percentage_abundance : str or float
Percentage abundance of this modform
-
D_protein_sites_modifications
¶ Instance attribute
Python dictionary with aa residues mapped to a list of possible modifications on the residues
-
PuLP_var
¶ Instance attribute
- modform variable in PuLP format
- E.g.,
LpVariable('a1')
-
get_modifications_frequency_map
()[source]¶ Instance method
It returns a dictionary of modifications and their frequency in the modform OR ‘UNMOD’ in case of unmodified modform
E.g.,
For
a1: ['0':'UNMOD']
, the method returns{'UNMOD':'0'}
. Fora11: [('S11','P'),('S30','P'),('S55','Ac')]
, the method returns,{'P': '2', 'Ac': '1'}
-
percentage_abundance
¶ Instance attribute
Percentage abundance of a modform
-
var_name
¶ Instance attribute
Variable name for the modform
-
whether_modified_peptide_present_in_modform
(N_loc, C_loc, D_peptide_mod_frequency=None)[source]¶ Instance method
- Boolean method return True or False
- peptide range is defined by N_loc, C_loc
- modification state of peptide: D_peptide_mod_frequency
- UNMOD modform can only generate UNMOD peptides
- UNMOD peptides could also come from other modforms too
Parameters: N_loc : str or int
Position of the first residue of a peptide in the protein sequence
C_loc : str or int
Position of the last residue of a peptide in the protein sequence
-
modformPro.modform_region_estimator module¶
-
class
modformPro.modform_region_estimator.
ModformRegionEstimator
(linear_constraints)[source]¶ Bases:
object
Computes modform region bounds given a set of linear constraints.
Parameters: linear_constraints :
LinearConstraints
Set of linear constraints.
-
compute_bounds
()[source]¶ Instance method
Compute bounds for each modform variable by maximizing and minimizing the variables in the linear program.
Notes
- Before getting feasible linear constraints, force all the modform variable in a liner constraint to be zero if rhs=0
E.g.,
a13 + a16 + a18 = 0
=>a13=a16=a18=0
;set upBound=0
After doing that these variables wont play any role in fixing the infeasibility and finding the bounds subsequently- Create
LinearProgrammingSolver
object for feasible linear constraints - For each variable in the canonical ordered list run linear programming to minimize and maximize each modform variable
-
linear_constraints
¶
-
modformPro.modform_space module¶
-
class
modformPro.modform_space.
ModformSpace
(specifications=None, L_isomeric_modforms_modes=None)[source]¶ Bases:
object
This class defines a modform space, i.e., set of modform objects.
Parameters: specifications :
ModformProSpecifications
, optional- Instance of ModformProSpecifications object
- Contains information about the option set by the user in the specification file
L_isomeric_modforms_modes : list, optional
- Python list of python dictionaries having modification types mapped to its frequency.
- E.g.,
L = [{'UNMOD': '0'}, {'P': '1'}, {'Ac': '1', 'P': '2'}]
There are three isomeric modform modes in the list - unmodified, singly phosphorylated modforms and the last is referring to the modforms that are singly acetylated and doubly phosphorylated.
-
D_PuLP_variables
¶ Instance attribute
- Python dictionary having modform variables (string format) being mapped to their respective
pulp
objects. - Note that modform variables in PuLP format are needed to perform linear programming computations with pulp library.
- Python dictionary having modform variables (string format) being mapped to their respective
-
D_variables_modforms
¶ Instance attribute
Python dictionary with modform variables (string format) being mapped to their respective
Modform
objects.
-
D_variables_nmolecules
¶ Instance attribute
- Python dictionary is populated only during simulated data mode. Infact modform_space object is created while simulating modform distribution.
- This has modform variables (in string format) being mapped to the number of molecules assigned to each modform variable.
-
generate_modform_space
()[source]¶ Instance method
- It defines the modform space (without specifying the percentage abundance of the modforms) and populate the instance object attributes
D_variables_modforms
,D_PuLP_variables
. - If
n_sites_full_modform_space
option is set to beTrue
then all the possible modforms are generated for n_sites and modification types. - This space will be restricted complying with the intact mass spectrometry data or user provides modes (for data simulation) if the option n_sites_full_modform_space is set to be False.
- This method is called during simulation of modform distribution and also needed to write linear constraints from peak area calculations.
Notes
If the user option
N_SITES_FULL_MODFORM_SPACE
in the specification file is set to beTrue
then full modform space is generated complying with the number of sites and their respective modification types. If the above user option is set to be False, then it will generate a modform space based on the isomeric modform modes either derived from intact mass spectrometry data or in case of “simulation” data mode, the modes would be specified in theMODES
field of the specification file.- It defines the modform space (without specifying the percentage abundance of the modforms) and populate the instance object attributes
-
get_dictionary_of_variables_molecules
(D_variables_modforms, state='principal')[source]¶ Instance method
- It assigns number of molecules to a given set of modform variables and returns a python dictionary of the same.
- This method is only called in a member method of the class
DataSimulator
.
Parameters: D_variables_modforms : dict
Python dictionary of modforms variables (string format) mapped to the modform objects.
state : str, “principal” (default) or “subordinate” or “null”
They determine number of molecules assigned to the modforms. “Null” assigns 0 molecules to the modforms. Default is “principal”.
Returns: D_variables_nmolecules : dict
Python dictionary having modform variables names (str) mapped to the number of molecules assigned.
Examples
>>> from modform import Modform >>> from modform_space import ModformSpace >>> >>> D_variables_modforms = {} >>> D_variables_modforms['a1'] = Modform(var_name='a1', D_protein_sites_modifications={'0': 'UNMOD'}) >>> D_variables_modforms['a2'] = Modform(var_name='a2', D_protein_sites_modifications={'S20': 'P'}) >>> D_variables_modforms['a3'] = Modform(var_name='a3', D_protein_sites_modifications={'S15': 'P'}) >>> D_variables_modforms['a4'] = Modform(var_name='a4', D_protein_sites_modifications={'S15': 'P', 'S20': 'P'}) >>> >>> mform_space = ModformSpace() >>> >>> D_nmolecules_principal = mform_space.get_dictionary_of_variables_molecules(D_variables_modforms, state="principal") >>> print D_nmolecules_principal {'a1': 1000, 'a3': 250, 'a2': 100, 'a4': 100} >>> >>> D_nmolecules_subordinate= mform_space.get_dictionary_of_variables_molecules(D_variables_modforms, state="subordinate") >>> print D_nmolecules_subordinate {'a1': 250, 'a3': 100, 'a2': 100, 'a4': 100} >>> >>> D_nmolecules_null = mform_space.get_dictionary_of_variables_molecules(D_variables_modforms, state="null") >>> print D_nmolecules_null {'a1': 0, 'a3': 0, 'a2': 0, 'a4': 0}
-
get_isomeric_modforms
(D_protein_mod_frequency)[source]¶ Instance method
- It returns two python dictionaries having isomeric modforms variables (string format) mapped to modform objects and their PuLP counterparts.
- These isomeric modforms are subset of the entire modform space.
- The subset is chosen based on the isomeric modform mode which is defined by the dictionary D_protein_mod_frequency.
Parameters: D_protein_mod_frequency: dict
Python dictionary having modifications mapped to their frequency.
Returns: dict, dict
-
get_list_of_modification_frequency_map
(total_mod_frequency)[source]¶ Instance method
- This is a subordinate method, i.e. called inside this class method.
- For a given total modification frequency, this function returns list of modifications to frequency map, L_D_protein_mod_frequency.
Parameters: total_mod_frequeny: int
Total frequency of modifications or total number of sites being modified.
Returns: L_D_protein_mod_frequency : list
Python list of python dictionary having modifications to their frequency.
Examples
>>> D_sites_modtypes = {'15': ['P'], '20': ['P'], '55': ['P', 'Ac']} >>> >>> mform_space = ModformSpace() >>> >>> total_mod_frequency = 0 >>> L_D_protein_mod_frequency = mform_space.get_list_of_modification_frequency_map(total_mod_frequency) [{'UNMOD':'0'}] >>> >>> total_mod_frequency = 1 [{'Ac': '1'}, {'P': '1'}] >>> >>> total_mod_frequency = 2 [{'P': '1', 'Ac': '1'}, {'P': '2'}] >>> >>> total_mod_frequency = 3 [{'P': '2', 'Ac': '1'}]
modformPro.ms_measurments_extractor module¶
-
class
modformPro.ms_measurments_extractor.
MSMeasurementsExtractor
[source]¶ Bases:
object
This class extracts measurement values (peak areas) from TDMS, BUMS and MRM data.
-
D_peptide_isomeric_peak_areas
¶ Instance attribute
- Python dictionary having peptide mapped to its isomeris states and their respective percentage peak areas
- This is populated by a member method
>>> # Demonstrative example of D_peptide_isomeric_peak_areas D = {'15-25': [({'UNMOD': '0'}, '25'), ({'P': '1'}, '50'), ({'P': '2'}, '25')], '50-60': [({'UNMOD': '0'}, '50'), ({'Ac': '1', 'P': '1'}, '30'), ({'P': '2'}, '20')]}
-
L_isomeric_modforms_peak_areas
¶ Instance attribute
- Python list of isomeric modforms and their percentage peak areas
- This is populated by a member method
>>> # Demonstrative example of L_isomeric_modforms_peak_areas [({'UNMOD': '0'}, '20'), ({'P': '1'}, '15'), ({'Ac': '1', 'P': '2'}, '25'), ({'P': '3'}, '10'), ({'Ac': '1', 'P': '4'}, '30')]
-
get_dictionary_of_peptides_and_isomeric_peak_areas
(fout_peptides_isomeric_peak_areas)[source]¶ Instance method
- It populates the instance attribute, D_peptide_isomeric_peak_areas
>>> # Demonstrative example of D_peptide_isomeric_peak_areas D = {'15-25': [({'UNMOD': '0'}, '25'), ({'P': '1'}, '50'), ({'P': '2'}, '25')], '50-60': [({'UNMOD': '0'}, '50'), ({'Ac': '1', 'P': '1'}, '30'), ({'P': '2'}, '20')]}
Parameters: fout_peptides_isomeric_peak_areas : str
File containing peptides, their isomeric states and respective percentage peak areas. This data typically comes from BUMS/MRM measurments.
-
get_list_of_isomeric_modforms_peak_areas
(fout_isomeric_modforms_peak_areas)[source]¶ Instance method
- It populates the instance attribute,
L_isomeric_modforms_peak_areas
- Note that the final list is sorted based on total frequency for isobaric modforms
>>> # Demonstrative example of L_isomeric_modforms_peak_areas [({'UNMOD': '0'}, '20'), ({'P': '1'}, '15'), ({'Ac': '1', 'P': '2'}, '25'), ({'P': '3'}, '10'), ({'Ac': '1', 'P': '4'}, '30')]``
Parameters: fout_isomeric_modforms_peak_areas : str
File containing percentage peak areas of isomeric modforms. This data typically comes from TDMS measurments.
Returns: L_isomeric_modforms_peak_areas : list
python list of isomeric modforms and their percentage areas. This data is typically known from intact MS measurements.
- It populates the instance attribute,
-
modformPro.specifications module¶
modformPro.statistics_and_plot module¶
-
class
modformPro.statistics_and_plot.
StatisticsAndPlot
[source]¶ Bases:
object
This class provides methods to compute statistics and for plotting.
-
get_dictionary_of_mean_and_standard_deviation_of_bounds
(list_dictionary_variables_bounds)[source]¶ This function returns mean and standard deviation of lower bounds OR upper bounds when measurments are done multiple times. For simulated data, this amounts to sample multiple noise values.
-
get_sensitivity_performance_statistics_over_all_samples
(L_D_lower_bounds, L_D_upper_bounds, D_vars_simulated_abundances, Output_dir=None)[source]¶ This function provides statistics on true hits for simulated data
-
get_sensitivity_statistics_for_one_sample
(D_lower_bounds, D_upper_bounds, D_variables_simulated_values)[source]¶
-
plot_bars
(D_lower_bounds, D_upper_bounds, Output_dir=None)[source]¶ Instance method
Plot bars showing lower and upper bounds of percentage concentration.
Parameters: D_lower_bounds : dict
Python dictionary containing modform
pulp
variables mapped to their lower bounds.D_upper_bounds : dict
Python dictionary containing modform
pulp
variables mapped to their upper bounds.Output_dir : str
Output directory name. Default is
Output
.
-
plot_bars_for_simulated_data
(D_mean_lower_bounds, D_mean_upper_bounds, D_stdev_lower_bounds, D_stdev_upper_bounds, D_variables_simulated_values=None, Output_dir=None)[source]¶ Ref: https://matplotlib.org/api/pyplot_api.html
# a stacked bar plot with errorbars N = 5 menMeans = (20, 35, 30, 35, 27) womenMeans = (25, 32, 34, 20, 25) menStd = (2, 3, 4, 1, 2) womenStd = (3, 5, 2, 3, 3) ind = np.arange(N) # the x locations for the groups width = 0.35 # the width of the bars: can also be len(x) sequence
p1 = plt.bar(ind, menMeans, width, color=’#d62728’, yerr=menStd) p2 = plt.bar(ind, womenMeans, width, bottom=menMeans, yerr=womenStd)
plt.ylabel(‘Scores’) plt.title(‘Scores by group and gender’) plt.xticks(ind, (‘G1’, ‘G2’, ‘G3’, ‘G4’, ‘G5’)) plt.yticks(np.arange(0, 81, 10)) plt.legend((p1[0], p2[0]), (‘Men’, ‘Women’))
plt.show()
-
plot_simulated_values
(D_variables_simulated_values, Output_dir=None)[source]¶ Ref: https://matplotlib.org/api/pyplot_api.html
# a stacked bar plot with errorbars N = 5 menMeans = (20, 35, 30, 35, 27) womenMeans = (25, 32, 34, 20, 25) menStd = (2, 3, 4, 1, 2) womenStd = (3, 5, 2, 3, 3) ind = np.arange(N) # the x locations for the groups width = 0.35 # the width of the bars: can also be len(x) sequence
p1 = plt.bar(ind, menMeans, width, color=’#d62728’, yerr=menStd) p2 = plt.bar(ind, womenMeans, width, bottom=menMeans, yerr=womenStd)
plt.ylabel(‘Scores’) plt.title(‘Scores by group and gender’) plt.xticks(ind, (‘G1’, ‘G2’, ‘G3’, ‘G4’, ‘G5’)) plt.yticks(np.arange(0, 81, 10)) plt.legend((p1[0], p2[0]), (‘Men’, ‘Women’))
plt.show()
-


Instance structure for the floating-point RFFT/RIFFT function. More...
#include <arm_math.h>
Public Attributes | |
| uint32_t | fftLenReal |
| length of the real FFT. More... | |
| uint16_t | fftLenBy2 |
| length of the complex FFT. More... | |
| uint8_t | ifftFlagR |
| flag that selects forward (ifftFlagR=0) or inverse (ifftFlagR=1) transform. More... | |
| uint8_t | bitReverseFlagR |
| flag that enables (bitReverseFlagR=1) or disables (bitReverseFlagR=0) bit reversal of output. More... | |
| uint32_t | twidCoefRModifier |
| twiddle coefficient modifier that supports different size FFTs with the same twiddle factor table. More... | |
| float32_t * | pTwiddleAReal |
| points to the real twiddle factor table. More... | |
| float32_t * | pTwiddleBReal |
| points to the imag twiddle factor table. More... | |
| arm_cfft_radix4_instance_f32 * | pCfft |
| points to the complex FFT instance. More... | |
Instance structure for the floating-point RFFT/RIFFT function.
| uint8_t arm_rfft_instance_f32::bitReverseFlagR |
flag that enables (bitReverseFlagR=1) or disables (bitReverseFlagR=0) bit reversal of output.
| uint16_t arm_rfft_instance_f32::fftLenBy2 |
length of the complex FFT.
| uint32_t arm_rfft_instance_f32::fftLenReal |
length of the real FFT.
| uint8_t arm_rfft_instance_f32::ifftFlagR |
flag that selects forward (ifftFlagR=0) or inverse (ifftFlagR=1) transform.
| arm_cfft_radix4_instance_f32* arm_rfft_instance_f32::pCfft |
points to the complex FFT instance.
| float32_t* arm_rfft_instance_f32::pTwiddleAReal |
points to the real twiddle factor table.
| float32_t* arm_rfft_instance_f32::pTwiddleBReal |
points to the imag twiddle factor table.
| uint32_t arm_rfft_instance_f32::twidCoefRModifier |
twiddle coefficient modifier that supports different size FFTs with the same twiddle factor table.


sibling-estimates.Rmd
library(siblingsurvival) library(tidyverse) #> ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ── #> ✓ ggplot2 3.3.2 ✓ purrr 0.3.4 #> ✓ tibble 3.0.4 ✓ dplyr 1.0.2 #> ✓ tidyr 1.1.2 ✓ stringr 1.4.0 #> ✓ readr 1.4.0 ✓ forcats 0.5.0 #> ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ── #> x dplyr::filter() masks stats::filter() #> x dplyr::lag() masks stats::lag() # be sure you have at least version <IP_ADDRESS>00 of surveybootstrap # run devtools::install_github('dfeehan/surveybootstrap') # for the most recent version library(surveybootstrap) # this is helfpul for timing library(tictoc)
It will be helpful to define a few important terms before we start our analysis:
ego
- an ego is a survey respondentcell
- a cell is a generic group for which we wish to produce estimates. Usually, a cell is defined by a time period, an age range, and a sex. So, for example, a cell might be women who were age 30-34 in 2015.For the purposes of this vignette, we’ll assume that we starting from two datasets:
We’ll then calculate estimates from the sibling histories in three main steps:
We’ll start by opening up the demonstration DHS dataset.
data(model_dhs_dat)
Now we’ll use prep_dhs_sib_histories
to ready the sibling histories for analysis. (Please see the Preparing Data vignette for more details.)
prepped <- prep_dhs_sib_histories(model_dhs_dat, varmap = sibhist_varmap_dhs6, keep_missing = FALSE) #> Warning: `select_()` is deprecated as of dplyr 0.7.0. #> Please use `select()` instead. #> This warning is displayed once every 8 hours. #> Call `lifecycle::last_warnings()` to see where this warning was generated. #> Warning: `mutate_()` is deprecated as of dplyr 0.7.0. #> Please use `mutate()` instead. #> See vignette('programming') for more help #> This warning is displayed once every 8 hours. #> Call `lifecycle::last_warnings()` to see where this warning was generated. #> Warning: `filter_()` is deprecated as of dplyr 0.7.0. #> Please use `filter()` instead. #> See vignette('programming') for more help #> This warning is displayed once every 8 hours. #> Call `lifecycle::last_warnings()` to see where this warning was generated. #> 638 out of 35082 (1.82%) reports about sibs have unknown survival status. #> 602 out of 35082 (1.72%) reports about sibs have unknown sex. #> Removing reported sibs missing survival status or sex. #> ... this removes 642 out of 35082 ( 1.83 %) sibling reports. # we'll only keep the variable we will need for this analysis ex.ego <- prepped$ego.dat %>% select(caseid, psu, stratum_analysis, age.cat, sex, wwgt) ex.sib <- prepped$sib.dat
Let’s take a look at the datasets we’ve produced. First, here’s a dataset that has information about survey respondents. (We’ll refer to these survey respondents as ‘ego’):
glimpse(ex.ego) #> Rows: 8,348 #> Columns: 6 #> $ caseid <chr> " 1 1 2", " 1 3 2", " 1 4 … #> $ psu <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1… #> $ stratum_analysis <dbl> 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, … #> $ age.cat <fct> "[30,35)", "[20,25)", "[40,45)", "[25,30)", "[25,30)… #> $ sex <chr> "f", "f", "f", "f", "f", "f", "f", "f", "f", "f", "f… #> $ wwgt <dbl> 1.057703, 1.057703, 1.057703, 1.057703, 1.057703, 1.…
And here’s a long-form version of the sibling history data – there’s one row for each reported sibling.
glimpse(ex.sib) #> Rows: 34,440 #> Columns: 24 #> $ .tmpid <chr> " 1 1 2", " 1 3 2", " … #> $ caseid <chr> " 1 1 2", " 1 3 2", " … #> $ wwgt <dbl> 1.057703, 1.057703, 1.057703, 1.057703, 1.057… #> $ psu <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, … #> $ doi <dbl> 1386, 1386, 1386, 1386, 1386, 1386, 1386, 138… #> $ sibindex <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, … #> $ sib.sex <chr> "m", "f", "m", "f", "m", "f", "m", "f", "m", … #> $ sib.alive <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, … #> $ sib.age <dbl> 42, 27, 46, 33, 40, 49, 22, 30, 49, 50, 35, N… #> $ sib.dob <dbl> 876, 1056, 828, 984, 900, 792, 1116, 1020, 79… #> $ sib.marital.status <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N… #> $ sib.death.yrsago <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 3… #> $ sib.death.age <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 4… #> $ sib.death.date <dbl> -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 1… #> $ sib.died.pregnant <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N… #> $ sib.died.bc.pregnancy <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N… #> $ sib.death.cause <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N… #> $ sib.time.delivery.death <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N… #> $ sib.place.death <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N… #> $ sib.num.children <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N… #> $ sib.death.year <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N… #> $ alternum <chr> "01", "01", "01", "01", "01", "01", "01", "01… #> $ sib.endobs <dbl> 1386, 1386, 1386, 1386, 1386, 1386, 1386, 138… #> $ sibid <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14…
First, we’ll need to add an indicator variable for each sibling’s frame population membership. In other words, we need to create a variable that has the value 1 for each sibling who was eligible to respond to the survey, and 0 otherwise. In this example data, we’ll assume that conditions similar to a typical DHS survey hold: i.e., we’ll assume the survey design was such that siblings would have been eligible to respond if they
For a given survey, these criteria will differ. You’ll have to find out what the criteria for inclusion in the frame population were in order to produce estimates from sibling histories
ex.sib <- ex.sib %>% mutate(in.F = as.numeric((sib.alive==1) & (sib.age >= 15) & (sib.age <= 49) & (sib.sex == 'f')))
Let’s look at the distribution of frame population membership
In this example dataset, a small number of the in.F
values is missing. (In a real dataset, there could well be more.) Since we need to be able to determine whether or not each sibling is on in the frame population, we would drop siblings missing in.F
values from the analysis.
When we produce estimated death rates from sibling histories, we usually do so for different sex X age group X time period combinations. These are called cells.
Our next step is to create an object that describes the cells that we plan to produce death rate estimates for. This means we need to specify the time period and age groups that we’ll be using. We’ll use the helper function cell_config
to do this:
cc <- cell_config(age.groups='5yr', time.periods='7yr_beforeinterview', start.obs='sib.dob', # date of birth end.obs='sib.endobs', # either the date respondent was interviewed (if sib is alive) or date of death (if sib is dead) event='sib.death.date', # date of death (for sibs who died) age.offset='sib.dob', # date of birth time.offset='doi', # date of interview exp.scale=1/12)
We’re specifying that we want
dob
column of our sibling dataset has the time siblings start being observed (their birthdate)endobs
column of our sibling dataset has the time siblings stop being observed (the interview or, if they’re dead, the date of death)death.date
column of our sibling dataset has the date the sibling died (if the sibling died)doi
column of our sibling dataset has the date the respondent who reported about the sibling was interviewedexp.scale=1/12
to indicate that we need to divide total exposures by 12 to get yearsGiven these preparatory steps, the sibling_estimator
function will take care of estimating death rates from the sibling histories for us.
ex_ests <- sibling_estimator(sib.dat = ex.sib, ego.id = 'caseid', # column with the respondent id sib.id = 'sibid', # column with sibling id # (unique for each reported sibling) sib.frame.indicator = 'in.F', # indicator for sibling frame population membership sib.sex = 'sib.sex', # column with sibling's sex cell.config=cc, # cell configuration we created above weights='wwgt') # column with the respondents' sampling weights names(ex_ests) #> [1] "asdr.ind" "asdr.agg" "ec.dat" "esc.dat"
sibling_estimator
returns a list with the results. We’ll focus on asdr.ind
, which has the individual visibility estimates, and asdr.agg
, which has the aggregate visibility estimates.

We’ll make some plots showing the results
ggplot(ex_ests$asdr.ind) + geom_line(aes(x=sib.age, y=1000*asdr.hat, color=sib.sex, group=sib.sex)) + theme_minimal() + scale_y_log10() + ggtitle('individual visibility estimator, 7yr before survey')
ggplot(ex_ests$asdr.agg) + geom_line(aes(x=sib.age, y=1000*asdr.hat, color=sib.sex, group=sib.sex)) + theme_minimal() + scale_y_log10() + ggtitle('aggregate visibility estimator, 7yr before survey')
compare <- bind_rows(ex_ests$asdr.ind, ex_ests$asdr.agg) ggplot(compare) + geom_line(aes(x=sib.age, y=1000*asdr.hat, color=sib.sex, linetype=estimator, group=interaction(estimator, sib.sex))) + theme_minimal() + #facet_grid(sex ~ .) + facet_grid(. ~ sib.sex) + scale_y_log10() + ggtitle('both estimators, 7yr before survey')
In practice, we want point estimates and estimated sampling uncertainty for the death rates. We’ll use the rescaled bootstrap to estimate sampling uncertainty. This can be done with the surveybootstrap
package.
Before we use the rescaled bootstrap, we need to know a little bit about the sampling design of the survey we’re working with. In this example dataset, we have a stratified, multistage design. So we’ll need to tell the bootstrap function about the strata and the primary sampling units. In this example data, these are indicated by the ‘stratum’ and ‘psu’ columns of the dataset.
(NOTE: this takes about a minute or so on a 2018 MBP for 1000 resamples.)
set.seed(101010) tic('running bootstrap') ## this will take a little while -- for 1000 reps, it takes about 10 minutes on a 2018 Macbook Pro #num.boot.reps <- 1000 # reduce number of reps to help vignette build faster num.boot.reps <- 100 bootweights <- surveybootstrap::rescaled.bootstrap.weights(survey.design = ~ psu + stratum(stratum_analysis), # a high number is good here, though that will obviously # make everything take longer num.reps=num.boot.reps, idvar='caseid', # column with the respondent ids weights='wwgt', # column with the sampling weight survey.data=ex.ego # note that we pass in the respondent data, NOT the sibling data ) #> Warning: `group_indices_()` is deprecated as of dplyr 0.7.0. #> Please use `group_indices()` instead. #> This warning is displayed once every 8 hours. #> Call `lifecycle::last_warnings()` to see where this warning was generated. #> Warning: The `...` argument of `group_keys()` is deprecated as of dplyr 1.0.0. #> Please `group_by()` first #> This warning is displayed once every 8 hours. #> Call `lifecycle::last_warnings()` to see where this warning was generated. toc() #> running bootstrap: 0.98 sec elapsed
The result, bootweights
, is a dataframe that has a row for each survey respondent, a column for the survey respondent’s ID, and then num.boot.reps
columns containing survey weights that result from the bootstrap procedure. The basic idea is to calculate estimated death rates using each one of the num.boot.reps
sets of weights. The variation across the estimates is then an estimator for the sampling variation.
To make this easier, the sibling_estimator
function can take a dataset with bootstrap resampled weights; it will then calculate and summarize the estimates for you.
(NOTE: this is slow; it takes about 35 minutes or so on a 2018 MBP when bootweights
has 1000 resamples.)
# to save time, we'll only use a subset of the bootstrap replicates short.bootweights <- bootweights %>% select(1:11) #est.bootweights <- bootweights est.bootweights <- short.bootweights tic('calculating estimates with bootstrap') ex_boot_ests <- sibling_estimator(sib.dat = ex.sib, ego.id = 'caseid', sib.id = 'sibid', sib.frame.indicator = 'in.F', sib.sex = 'sib.sex', cell.config=cc, boot.weights=est.bootweights, # to get sampling uncertainty, we pass boot.weights into sibling_estimator return.boot=TRUE, # when TRUE, return all of the resampled estimates (not just summaries) weights='wwgt') toc() #> calculating estimates with bootstrap: 37.235 sec elapsed
Finally, let’s plot the estimated death rates along with their sampling uncertainty:
ggplot(ex_boot_ests$asdr.ind) + geom_ribbon(aes(x=sib.age, ymin=1000*asdr.hat.ci.low, ymax=1000*asdr.hat.ci.high, fill=sib.sex, group=sib.sex), alpha=.2) + geom_line(aes(x=sib.age, y=1000*asdr.hat, color=sib.sex, group=sib.sex)) + theme_minimal() + scale_y_log10() + ggtitle('individual visibility estimator, 7yr before survey') #> Warning: Transformation introduced infinite values in continuous y-axis
ggplot(ex_boot_ests$asdr.agg) + geom_ribbon(aes(x=sib.age, ymin=1000*asdr.hat.ci.low, ymax=1000*asdr.hat.ci.high, fill=sib.sex, group=sib.sex), alpha=.2) + geom_line(aes(x=sib.age, y=1000*asdr.hat, color=sib.sex, group=sib.sex)) + theme_minimal() + scale_y_log10() + ggtitle('aggregate visibility estimator, 7yr before survey') #> Warning: Transformation introduced infinite values in continuous y-axis
In this section, we illustrate how to conduct internal consistency checks:
NOTE: The IC checks can take a while – about 36 minutes for 1000 bootstrap reps on a 2018 MBP
# toggle between short bootweights (faster, for coding) and long bootweights (for realism) ic.bootweights <- short.bootweights #ic.bootweights <- bootweights tic("Internal consistency checks") ic.checks <- sib_ic_checks(ex_boot_ests$esc.dat, ego.dat=ex.ego, ego.id='caseid', sib.id='sibid', sib.frame.indicator='in.F', sib.cell.vars=c('sib.age', 'sib.sex'), ego.cell.vars=c('age.cat', 'sex'), boot.weights=ic.bootweights) toc() #> Internal consistency checks: 0.63 sec elapsed names(ic.checks) #> [1] "ic.summ" "ic.boot.ests"
We can look at ic.checks$ic.summ
, which has summarized output:
glimpse(ic.checks$ic.summ) #> Rows: 7 #> Columns: 15 #> $ cell <chr> "[15,20)_X_f", "[20,25)_X_f", "[25,30)_X_f", "[30,35… #> $ age.cat <chr> "[15,20)", "[20,25)", "[25,30)", "[30,35)", "[35,40)… #> $ sex <chr> "f", "f", "f", "f", "f", "f", "f" #> $ diff_mean <dbl> 237.04784, -279.75665, -3.38159, -163.19049, 227.432… #> $ abs_diff_mean <dbl> 237.0478, 279.7566, 106.6174, 163.1905, 227.4326, 22… #> $ diff2_mean <dbl> 63973.98, 79139.39, 16222.04, 28804.52, 59759.73, 58… #> $ diff_se <dbl> 92.98927, 31.19132, 134.20800, 49.14136, 94.48180, 7… #> $ abs_diff_se <dbl> 92.98927, 31.19132, 73.44523, 49.14136, 94.48180, 78… #> $ diff2_se <dbl> 42609.67, 17657.43, 22881.11, 16061.31, 36740.01, 33… #> $ diff_ci_low <dbl> 97.03007, -327.16213, -160.61038, -237.50409, 49.768… #> $ abs_diff_ci_low <dbl> 97.03007, 235.54499, 28.30904, 84.56564, 49.76817, 1… #> $ diff2_ci_low <dbl> 10077.4483, 55579.2367, 993.9034, 7317.8783, 4093.27… #> $ diff_ci_high <dbl> 355.88593, -235.54499, 241.64752, -84.56564, 338.318… #> $ abs_diff_ci_high <dbl> 355.8859, 327.1621, 253.3625, 237.5041, 338.3189, 31… #> $ diff2_ci_high <dbl> 126888.68, 107060.00, 66087.25, 56472.20, 114722.50,…
It’s often helpful to plot the results of the internal consistency checks:
ggplot(ic.checks$ic.summ) + geom_hline(yintercept=0) + geom_pointrange(aes(x=age.cat, y=diff_mean, ymin=diff_ci_low, ymax=diff_ci_high)) + theme_minimal() + ggtitle("Internal consistency checks") + xlab("Age group") + ylab(expression(Delta[alpha]))
#ggplot(ic.checks$ic.summ) + # geom_hline(yintercept=0) + # geom_pointrange(aes(x=age.cat, # y=agg_diff_mean, # ymin=agg_diff_ci_low, # ymax=agg_diff_ci_high)) + # theme_minimal() + # ggtitle("Internal consistency checks (aggregate)") + # xlab("Age group") + # ylab(expression(Delta[alpha])) # #ggplot(ic.checks$ic.summ) + # geom_hline(yintercept=0) + # geom_pointrange(aes(x=age.cat, # y=ind_diff_mean, # ymin=ind_diff_ci_low, # ymax=ind_diff_ci_high)) + # theme_minimal() + # ggtitle("Internal consistency checks (individual)") + # xlab("Age group") + # ylab(expression(Delta[alpha]))
sib.F.dat <- ex.sib %>% group_by(caseid) %>% summarize(y.F = sum(in.F)) #> `summarise()` ungrouping output (override with `.groups` argument) ego.vis <- ex.ego %>% select(caseid, wwgt, age.cat, sex) %>% left_join(sib.F.dat, by='caseid') # if nothing was joined in, there are no sibs ego.vis <- ego.vis %>% mutate(y.F=ifelse(is.na(y.F), 0, y.F)) ego.vis.agg <- ego.vis %>% group_by(sex, age.cat) %>% summarise(y.F.bar = weighted.mean(y.F, wwgt)) %>% mutate(adj.factor = y.F.bar / (y.F.bar + 1)) #> `summarise()` regrouping output by 'sex' (override with `.groups` argument)
Make adjusted individual estimates
adj.agg.ests <- ex_ests$asdr.agg %>% left_join(ego.vis.agg, by=c('sib.sex'='sex', 'sib.age'='age.cat')) %>% mutate(adj.factor = ifelse(is.na(adj.factor), 1, adj.factor), estimator = 'sib_agg_adj', asdr.hat = asdr.hat * adj.factor)
And plot a comparison
compare <- bind_rows(ex_ests$asdr.ind, ex_ests$asdr.agg, adj.agg.ests) ggplot(compare) + geom_line(aes(x=sib.age, y=1000*asdr.hat, color=sib.sex, linetype=estimator, group=interaction(estimator, sib.sex))) + theme_minimal() + #facet_grid(sex ~ .) + facet_grid(. ~ sib.sex) + scale_y_log10() + ggtitle('sibling estimators, 7yr before survey')


When I think about my values, I think about my time in the Peace Corps. What drove me to it (and I got out of the experience) was meaningful work, challenge and accomplishment, self-reliance and peace. These values push a person to live up to them and I am no different. I feel a person gains great strength from pushing their own limits and the testing for new ones again.
Meaningful work gives me purpose. The power in the work attaches to the need you see yourself fulfilling. Cleaning a house can be meaningful, and it becomes more meaningful if you are doing it for a friend who has been in the hospital for two weeks. Meaningful work comes from doing it for others and yourself, not needing to be holy artruistic, but needing to be needed and using your specific skillset is important.
When I think about my values and the experiences which both enforced and expanded them, I feel proud. My strength and work during my time in Peace Corps continually reminds me of both my ability and endurance. As I have held on to those values in the past, I will hold onto them in the future. There is peace in knowing


Student planner online
Welcome, {{user_nickname}}
Logout
Timetable - Week A
Your browser does not support iframes, so it is inferior to others!
Upcoming events:
Your browser does not support iframes, so it is inferior to others!


International Moose Count Underway
By BOB O’BOBSTON
The UN-sponsored International
Moose Census got off to a flying
start today with hopes for an increase
in the worldwide moose population
compared to last year’s disapointing
figures. Among the traditional early
reporters were Egypt, returning fig-
ures of six moose, a twenty percent
increase on 2011’s figures of five, and
Uruguay whose moose population re-
mains stable at eleven.
According to Robbie McRobson,
head of the UN Moose Preserva-
tion Council, worldwide moose num-
bers are expected to grow markedly
on last year due to the traditional
moose strongholds of Canada and the
United States, with the larger de-
veloping moose ecologies also poised
to make gains. The largest percent-
agege increase in moose will likely
come from China’’, says McRobson,
The Chinese government has invested
heavily in moose infrastructure over
the past decade, and their committ-
ment to macrofauna is beginning to
pay dividends’’. Since 2004 China has
expanded moose pasture from 1.5%
of arable land to nearly 3.648% and
moose numbers are expected to rise
to 60,000 making China a net moose
exporter for the first time. This is
good news for neighbouring Mongo-
lia, a barren moose-wasteland whose
inhabitents nonetheless have an insa-
tiable desire for the creatures. The in-
crease in Beijing-Ulanbataar trade is
anticipated to relieve pressure on the
relatively strained Russian suppliers,
but increase Mongolia’s imbalance of
trade with its larger neighbour.
Historically the only competitor
to China in the far eastern moose
markets has been Singapore but the
tiny island nation is set to report a
net loss, expecting a decrease of more
than five percent on last year’s 50,000
moose counted. The head of Singa-
pore’s Agency for Agriculture, Jing-
Feng Lau, explained to an incredu-
lous Singaporean parliament yester-
day that bad weather had contributed
to this season’s poor showing, most
notably when a cargo of 150 moose
were swept out into the Indian ocean
in a monsoon.
Yet again the global demand for
moose will be met largely by the
US and Canada. The recession-hit
States is taking comfort in its moose
growth figures with gross production
expected to break 700,000 and net ex-
ports to grow by 2%. The worldwide
dominance of Canada shows no signs
of abating though with this year’s
moose population expected to match
last year’s record figures of one hun-
dred million billion.
Europe’s rise as an international
moose power will slow slightly this
year as a response to the European
Union’s move towards standardising
the European moose. Stringent qual-
ity controls are holding back the de-
velopment of the eastern european
populations compared to last year
when they contributed significantly
to europe’s strong growth figures.
Norway, which is not an EU member
but has observer status, strengthed
in numbers relative to the Euro area
with numbers of Norweigian moose,
known locally as elk’’expected to rise
for the tenth consecutive year, partic-
ularly thanks to a strong showing in
the last quarter.
As moose season reaches its close,
researchers world wide are turning to
science in an attempt to boost next
year’s figures. NASA stunned the
scientific community today with the
announcment of their discovery that
the moon is significantly smaller than
previously believed. This conclusion,
which is the conclusion of a ten-
year collaborative project, will have
profound implications for the moose
community as the gravitational field
is now known to be of the right
strength to support moose in orbit.
According to John Johnson, head
of the NASA Moon Sizing Experi-
ment the first delivery of moose into
low moon orbit could be achieved as
early as the third quarter of next year.
The technology to nurture moose in
space is available now’’, he said, ’’all
that is needed is political will’’.


This articles covered it all about Bootstrap 5. You can also go through our video tutorials.
Continue readingThe only way you can grow is by doing some projects on your own and we are here to help you.
Continue readingThe only remote conference you have ever needed in which the top developers of the world will be sharing there techniques.
Continue readingWork on any project you want with your pace. Hurry up join the show and find your buddies.
Continue reading


Euro 2020 play-off draw confirmed as Scotland discover fate
Scotland will face a home game with Israel before an away game in either Serbia or Norway if they progress in order to try and reach Euro 2020
by Rich Jones, https://www.mirror.co.uk/authors/rich-jones/Scotland have discovered the path they will face in order to reach Euro 2020 after the qualifying draw for the play-offs was completed on Friday morning.
Steve Clarke's side will take on Israel at home in their semi-final, with the winners of Norway's clash with Serbia awaiting them in the final of Path C.
Northern Ireland and the Republic of Ireland already knew their fixtures in Path B, with Northern Ireland taking on Bosnia and Herzegovina and Slovakia taking on the Republic of Ireland.
The winners of those two fixtures will meet in the final.
Elsewhere, Iceland will take on Romania with Bulgaria taking on Hungary.
Georgia take on Belarus and North Macedonia face Kosovo in Path D.
The play-off games will take place in March 2020, with semi-finals on March 26 and the finals on March 31.
The Path C final will be hosted by Norway or Serbia, meaning Scotland will face an away trip if they progress against Israel.
Northern Ireland will host the Path B final if they progress, leaving the Republic of Ireland facing an away trip.
Euro 2020 Play-Off Draw
Path A:
Semi-final 1: Iceland vs Romania
Semi-final 2: Bulgaria vs Hungary
Final: Winner of semi-final 2 vs winner of semi-final 1.
Path B:
Semi-final 1: Bosnia and Herzegovina v Northern Ireland
Semi-final 2: Slovakia v Republic of Ireland
Final: Winner of semi-final 1 vs winner of semi-final s.
Path C:
Semi-final 1: Scotland v Israel
Semi-final 2: Norway v Serbia
Final: Winner of semi-final 2 vs winner of semi-final 1.
Path D:
Semi-final 1: Georgia v Belarus
Semi-final 2: North Macedonia v Kosovo
Final: Winner of semi-final 1 vs winner of semi-final 2.


[gecode-users] posting instead of propagating domain constraints?
Christian Schulte
cschulte at kth.se
Fri Jul 16 19:15:30 CEST 2010
Hi Kish,
Well, dom() and the integer variants of rel() propagate in the sense that
they immediately modify the variable domain and schedule some propagators
for later execution. However, they do not lead to full constraint
propagation computing a fixpoint. The invariant is that only execution of
the status() function of a space computes a fixpoint. So they are dirt
cheap, much cheaper than creating a propagator for them.
When you post constraints many of the propagators posted also modify some
variable domains to enforce some invariants they will rely on when they
actually propagate (think about element which ensures that the domain of the
index variables corresponds to available indices in the array).
Should I clarify this in MPG?
Cheers
Christian
--
Christian Schulte, web.ict.kth.se/~cschulte/
-----Original Message-----
From: users-bounces at gecode.org [mailto:users-bounces at gecode.org] On Behalf
Of Kish Shen
Sent: Friday, July 16, 2010 6:55 PM
To: users at gecode.org
Subject: [gecode-users] posting instead of propagating domain constraints?
Hi,
Is there anyway to post propagators for domain constraints for IntVar
(including setting it to a single value), without doing the propagation?
I've just noticed from the manual that most dom() constraints (as well
as rel() between an IntVar and an int) propagates, instead of just
posting the propator.
The main reason I ask is that I do recomputation manually, by posting
changes to a Space ('events', such as adding constraints) without
propagation until all recorded events are posted, which should be more
efficient than performing propagation every time. I use rel() when
ECLiPSe sets a variable to a value, which happens frequently during
search. If this propagates as the manual suggest, am I correct that this
can make the recomputation more expensive?
I noticed the reified versions of dom() do not propagate -- is this one
way of avoiding propagation?
Thanks and cheers,
Kish
--
This e-mail may contain confidential and privileged material for the
sole use of the intended recipient. Any review, use, distribution or
disclosure by others is strictly prohibited. If you are not the intended
recipient (or authorized to receive for the recipient), please contact
the sender by reply e-mail and delete all copies of this message.
Cisco Systems Limited (Company Number: 02558939), is registered in
England and Wales with its registered office at 1 Callaghan Square,
Cardiff, South Glamorgan CF10 5BT.
_______________________________________________
Gecode users mailing list
users at gecode.org
https://www.gecode.org/mailman/listinfo/gecode-users
More information about the users
mailing list


The weekend's GO Campaign gala red carpet gave up ample opportunities to swoon. Robert Pattinson and his Breaking Dawn: Part 2 co-star Rami Malek reunited on the red carpet, where they laughed and grinned as they posed for photos with fellow actor John David Washington. The trio was among a whole host of stars who turned up to support the cause, which helps orphans and vulnerable children in the USA and internationally. Also heating up the red carpet were Armie Hammer and Elizabeth Chambers, who looked as loved-up as ever as they cuddled up for photos. Inside, Dev Patel and Kid Cudi joined the fun, while Armie and Rob caught up with fellow actor Josh Hutcherson. Keep reading to see all the photos.
Rami Malek, Robert Pattinson, and John David Washington
Robert Pattinson
Robert Pattinson
Rami Malek and Robert Pattinson
Armie Hammer and Elizabeth Chambers
Robert Pattinson and John David Washington
Rami Malek
Josh Hutcherson and Armie Hammer
Robert Pattinson
Robert Pattinson
Armie Hammer
Robert Pattinson
Kid Cudi, Elizabeth Chambers, Armie Hammer, and Dev Patel
Robert Pattinson
Rami Malek and Robert Pattinson
Robert Pattinson
Rami Malek, Robert Pattinson, and John David Washington
Robert Pattinson
Rami Malek, Robert Pattinson, and John David Washington
Robert Pattinson
Rami Malek, Robert Pattinson, and John David Washington
Josh Hutcherson, Robert Pattinson, and Armie Hammer
Rami Malek
Rami Malek, Robert Pattinson, and John David Washington


Real Madrid Club de Futbol
Club Legends
Trophy Cabinet
Current Squads
Merchandise
Wallpapers
La Liga
Copa del Rey
Copa de Espana
UEFA Champions League
UEFA Super Cup
FIFA Club World Cup
All Time Stats
Matches - 438
Wins - 262
Draw - 76
Lost - 100
Goals - 972
Win % - 59.82%
Top Scorers from the Club
Cristiano Ronaldo - 105
Raul Gonzalez - 66
Karim Benzema - 52
Alfredo Di Stéfano - 49
Santillana - 47
Most Appearances for the Club
Iker Casillas - 150
Raul Gonzales - 130
Sergio Ramos - 123
Roberto Carlos - 107
Cristiano Ronaldo - 101
Real has won a total of 13 UEFA Champions League trophies.
Most by any club.
Play Hala Madrid Y Nada Mas
and sing along
Time-line of Trophies
Uno - 55/56
Dos - 56/57
Tres - 57/58
Cuatro - 58/59
Cinco - 59/60
Seis - 65/66
Siete - 56/57
Ocho - 57/58
Nueve - 01/02
La Decima - 13/14
Once - 15/16
Doce - 16/17
Trece - 17/18
Current Season
1st Leg [H]
RMA 1 - 2 MCFC
2nd Leg [A]
Covid 19 - 0 Football
Stats from UCL 19/20
Matches - 7
Won - 3
Draw - 2
Lost - 2
Goals - 15
Attempts - 132
Passing - 87%
Possession - 55%
Distance - 747.7 km
Saves - 18
Tackles - 23
Fouls - 67
Yellow Card - 12
Red Card - 1
Source -
UEFA
Made by -
Arnab


Chapter 4 Web scraping
There are several ways to extract information from the web. Use of APIs being probably the best way to extract data from a website. Almost all large websites like Twitter, Facebook, Google, Twitter, StackOverflow provide APIs to access their data in a more structured manner. If you can get what you need through an API, it is almost always preferred approach over web scraping.1
4.1 Resources
- Beginners guide to web scraping
- This is a great place to start with web-scraping
- Introduces BeautifulSoup and how to read HTML tags
- Introduction To Web Scraping
- Via: youtube
- Another really useful tutorial (and also helps that it’s on youtube)
- Twitter API
- Via: Codecademy
- How to crawl a web page with scrapy and python 3
- Via: Digital Ocean
- Good place to start and has an example that you can follow all the way through
- Learn Scrapy
- Video series for learning scrapy
4.2 Twitter Scrape (using Beautiful SOUP)
Libraries needed
- BeautifulSoup: This well help us parse a web page
- Selenium: The selenium.webdriver module provides all the WebDriver implementations. Currently supported WebDriver implementations are Firefox, Chrome, IE and Remote. The Keys class provide keys in the keyboard like RETURN, F1, ALT etc.
- Time: Time used to pause scrolling for x number of seconds before continuing
- csv: Used to write all the data we collect into a .csv file
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
import csv
Set up the url and filename
Here we will create our url. In order to make the scraper more flexible, we split it up so we can use it to scrape other search results with a different hashtag. In this case we are scraping all tweets with the #Goalkeepers17
hashtag. We also want to set up the name of the .csv file we are going to create at the end so we remember to change it as we change our scraper (don’t want to save over existing files).
hashtag = "Goalkeepers17"
is where we can change the hashtag to retrieve a different search result
filename = "twitterscrape.csv"
is where we name our file that we want to export our scraped data.
#Set up google search URL
#States our webdriver as using Chrome (obviously if you don't use chrome you would need to change this)
browser = webdriver.Chrome()
#Here we broke up the url so we could insert our own hashtag to search easily
start_url = "https://twitter.com/hashtag/"
hashtag = "Goalkeepers17" #<<<<This can be changed
end_url = "?src=hash"
#combine each element of the url to create a final url
url = start_url + hashtag + end_url
#print(url) #can test the url using print
#File to write csv
filename = "twitterscrape.csv"
When you run this code block you should get a pop-up of the browser that we will be controlling with selenium.
Set up selenium
Here we definte a function that opens our url, and scrolls down the twitter page, using time.sleep(2)
to pause 2 seconds after every scroll to load the page. (Honestly still need to figure out what exactly is going on here). Code from here.
def twt_scroller(url):
browser.get(url)
#define initial page height for 'while' loop
lastHeight = browser.execute_script("return document.body.scrollHeight")
while True:
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
#define how many seconds to wait while dynamic page content loads
time.sleep(2)
newHeight = browser.execute_script("return document.body.scrollHeight")
if newHeight == lastHeight:
break
else:
lastHeight = newHeight
html = browser.page_source
return html
Create the soup
We create the soup by first running the function from above on our url and then implementing the BeatifulSoup function on the full page.
We can then inspect the HTML to pinpoint the container and class that contains all the information we need.
You can see in the image above that when you inspect the element right click + inspect element
, you can get the container that contains all the information you will need. Here we can see that is a div
with the class tweet
(You can use the element selector to highlight the element which will display the html as a tool-tip and in the right-hand Elements bar. . Once we use BeautifulSoup()
to parse our html, we use our inspected information to create a container that will find all the div
containers with the class tweet
in our url.
#create the soup
soup = BeautifulSoup(twt_scroller(url), "html.parser")
#grabs the whole tweet container
containers = soup.findAll("div", {"class":"tweet"})
Grab all the information we need
Here we have to go back and forth between the HTML and python to grab the various elements we want. In this case we wanted:
- Name
- Username
- Verified (yes/no)
- Date
- Tweet text
#
of replies#
of retweets#
of likes
We loop through all the containers we collected in the soup - and for each container we want to grab the above HTML elements (The for loop we’ve used at the beginning only iterates through a few elements at a time so we don’t waste time if some of the elements we grab are incorrect). We can grap and element with the following code:
VARIABLE = container.find("BLOCK ELEMENT", attrs={"ATTRIBUTE": "ATTRIBUTE NAME"}).get_text()
Fore example, to get the twitter username we can do:
username = container.find("span", attrs={"class": "username"}).get_text()
Below we’ve added a statement on the end of each block element to just have a blank if the there is no element there to grab.
We also had to do a little bit of cleaning in here. For example the names were grabbed and if they were verified it was tacked on to the end of the name. We created a little statement where if the name had ‘Verified account’ in it, we put a one in a new category called verified (else we put a 0).Some of the text (like number of replies and retweets) also had the text ‘replies’ and ‘retweets’ so we wanted to drop that.
We then saved all the information into a dictionary. We’ve printed statements at each stage just to double-check the information we get is right.
#checks we have a container
#print(containers[0])
#create some lists that we will populate further down
verified = []
name = []
twitter_list = []
#for container in containers[0:2] #<<<<Use this statement to just iterate through the first 3 containers so as not to waste time
for container in containers:
fullname = container.find("span", attrs={"class":"FullNameGroup"}).get_text() if container.find('span', {'class':"FullNameGroup"}) is not None else ""
#print("fullname:", fullname)
if "Verified account" in fullname:
verified = 1
name = fullname.replace("Verified account", "")
else:
verified = 0
#print("name:", name)
#print("verified:", verified)
username = container.find("span", attrs={"class": "username"}).get_text() if container.find("span", attrs={"class": "username"}) is not None else ""
#print("username:", username)
date = container.find("a", attrs = {"class": "tweet-timestamp"}).get_text() if container.find("a", attrs = {"class": "tweet-timestamp"}) is not None else ""
#print("date:", date)
tweet = container.find("p", attrs = {"class": "TweetTextSize"}).text.strip() if container.find("p", attrs = {"class": "TweetTextSize"}) is not None else ""
#print("tweet:", tweet)
replies = container.find('span',{"class":"ProfileTweet-actionCount"}).text.strip().replace(" replies .", "").replace(" reply .", "") if container.find('span',{"class":"ProfileTweet-actionCount"}) is not None else ""
#print("replies:", replies)
retweets = container.find('span',{"class":"ProfileTweet-action--retweet"}).text.strip().replace(" retweets", "").replace(" retweet", "") if container.find('span',{"class":"ProfileTweet-action--retweet"}) is not None else ""
#print("retweets:", retweets)
likes = container.find("span", {"class": "ProfileTweet-action--favorite"}).text.strip().replace(" likes", "").replace(" like", "") if container.find("span", {"class": "ProfileTweet-action--favorite"}) is not None else ""
#print("likes:", likes)
#write tweets to a dictionary
twitter_dict = {
"name": name.replace("\n", "").replace("\u200f\xa0", ""),
"verified": verified,
"username": username,
"date": date,
"tweet": tweet,
"replies": replies,
"retweets": retweets,
"likes": likes
}
twitter_list.append(twitter_dict)
#Test to see if everything has worked correctly
print(twitter_list[0:2])
Should end up with a dictionary that looks something like this:
[{'date': 'Sep 20',
'likes': '260',
'name': 'Max de Haldevang',
'replies': '3',
'retweets': '90',
'tweet': 'Obama: If asked what period of history you\'d want to live in, "this would be the time you wanna be showing up on this planet"#Goalkeepers17',
'username': '@MddeH',
'verified': 1},
{'date': 'Sep 20',
'likes': '2,853',
'name': 'The Obama Foundation',
'replies': '47',
'retweets': '1,018',
'tweet': '"Progress requires struggle, and perseverance, and discipline, and faith." — @BarackObama at #Goalkeepers17',
'username': '@ObamaFoundation',
'verified': 1}]
*Define function to write a csv file
Here we use the filename we defined earlier to write our twitter_list to a csv file. Have to specify which elemtns we want in each row.
#define function to write CSV file
def writer_csv(twitter_list):
with open(filename, "w", encoding="utf8") as csvfile:
writer = csv.writer(csvfile, lineterminator= "\n", delimiter = ",", quotechar= '"')
for i in twitter_list:
if len(i["tweet"]) > 0:
newrow = i["name"], i["verified"], i["username"], i["date"], i["tweet"], i["replies"], i["retweets"], i["likes"]
writer.writerow(newrow)
else:
pass
And then we call the function that we defined above.
writer_csv(twitter_list)
Our result should be a .csv file in the folder we are working in, that has all the information we need. I’ve had issues opening the .csv file with microsoft excel (There are a ton of strange characters) but it seems to work smoothly when just opened with google docs instead.
Sometimes API’s have a cap on what you can scrape. For example, the Twitter API only lets you go back one week.↩


--- layout: page title: Lean Prover Zulip Chat Archive permalink: archive/113488general/04191charisuppertoupperetc.html ---
Are the functions that convert a char to it's uppercase/lowercase sibling?
char
char.to_lower and char.to_upper should do it
char.to_lower
char.to_upper


adfDashboard
adf
adfDashboard
is a directive which renders the dashboard with all its
components. The directive requires a name attribute. The name of the
dashboard can be used to store the model.
<adf-dashboard> </adf-dashboard>as attribute
<div adf-dashboard> ... </div>as class
<div class="adf-dashboard"> ... </div>


‘No problem that we were not brilliant’ – Klopp happy with win at Palace
Jurgen Klopp described Liverpool’s start to the Premier League season as exceptional.
Jurgen Klopp was delighted with Liverpool’s fight at Crystal Palace as the leaders left London with three points despite a below-par display.
Roberto Firmino struck an 85th-minute winner to extend the Reds’ unbeaten Premier League run to 30 games and end his own goalscoring drought stretching back to September 22.
Three minutes earlier Wilfried Zaha had thrilled a buoyant Selhurst Park crowd with his first of the season to cancel out Sadio Mane’s 49th-minute opener.
“I am not surprised at all it was tough,” Klopp said. “All our games since I am in England are always similar. We just know it is a tough place, the atmosphere is good and they don’t give up.
“We knew at half-time we had to do better. The challenge, and it’s the same for pretty much everybody, is that they come from the international teams and today I had no problem that we were not brilliant.
“In a game like this you just have to make sure you are ready to fight to get the result from the first minute.
“Could we have played better? Yes, but we had players on the pitch who on Thursday morning were still on a plane. It is not too cool and for that getting a result at Crystal Palace, who we never take for granted, is good.”
Like in the 2-1 win over Aston Villa at the beginning of the month, Liverpool again struck at the death to seal all three points on the road.
Yet Klopp is not interested in people likening the late goals to the traits of champions, adding: “I have no clue what people will say.
“I know 37 points after 13 matchdays is exceptional and you cannot have that with winning only brilliant games.
“We have had some brilliant games and we won them, thank God. But we cannot only win them so that is how it is. We are not out there to send messages – we just want to get results.”
Liverpool were pushed all the way for their victory and impressively did it without Mohamed Salah, who remained an unused substitute after a recent ankle injury.
Klopp said: “We wanted to bring him and then we scored and we were able to rest him, which is really good. Wednesday (against Napoli) he will be back.”
Roy Hodgson thought his side had taken the lead when James Tomkins headed home at the end of the first half, but VAR ruled out the effort and they went on to suffer a third straight defeat.
Hodgson was gutted for his team to walk away empty-handed, but full of praise for their display against his old club.
He said: “I thought it was our best performance of the season. We really did play well.
“It’s a bitter blow after getting what we thought was a deserved equaliser to concede that second goal from the unfortunate corner kick and take away that point we thought we were more than worth.
“It’s football. We have to learn to live with that, but I do honestly believe that was a very good team performance.”


Da Vinci Payer Data exchange - Local Development build (v0.3.7). See the Directory of published versions
Previous Page - US Core AllergyIntolerance
Where a Health Plan has access to Care Plan information for a member they SHALL make the information available using the US Core CarePlan resource.
The essential fields to be provided in the CarePlan resource are:
| R4 Hierarchical Name | R4 Name | Flags | Card. | Type |
|---|---|---|---|---|
| CarePlan | CarePlan | I | 0..* | |
| CarePlan.id | id | 0..1 | id | |
| CarePlan.text | text | S | 1..1 | Narrative |
| CarePlan.text.status | status | S | 1..1 | code |
| CarePlan.text.div | div | I | 1..1 | xhtml |
| CarePlan.status | status | ?!S | 1..1 | code |
| CarePlan.intent | intent | ?!S | 1..1 | code |
| CarePlan.category | category | SI | 1..* | CodeableConcept |
| CarePlan.subject | subject | S | 1..1 | Reference(US Core Patient Profile) |
| CarePlan.activity.detail.status | status | ?! | 1..1 | code |
An example mapping of a CarePlan for a colonoscopy is shown below:
{ "resourceType" : "CarePlan", "id" : "colonoscopy", "meta" : { "profile" : [ "http://hl7.org/fhir/us/core/StructureDefinition/us-core-careplan" ] }, "text" : { "status" : "additional", "div" : "<div xmlns=\"http://www.w3.org/1999/xhtml\"> Assessment
- Recurrent GI bleed of unknown etiology; hypotension perhaps secondary to this but as likely secondary to polypharmacy.
- Acute on chronic anemia secondary to #1.
- Azotemia, acute renal failure with volume loss secondary to #1.
- Hyperkalemia secondary to #3 and on ACE and K+ supplement.
- Other chronic diagnoses as noted above, currently stable.
| Planned Activity | Planned Date |
|---|---|
| Colonoscopy | April 21, 2000 |


Harry Maguire shares Man Utd's 'classy gesture' after he was knocked out at Old Trafford in 2011
Harry Maguire was playing for Sheffield United in the FA Youth Cup against Man Utd when he clashed heads with Ravel Morrison and was forced off on a stretcher
by Alex SmithHarry Maguire has revealed how Manchester United sent him a signed shirt after he was knocked out in the FA Youth Cup final against them.
Centre back Maguire was playing for boyhood club Sheffield United in 2011 when he collided with Red Devils striker Ravel Morrison.
Maguire was knocked unconscious from the clash of heads and was required to be taken away on a stretcher.
The England international joined the Red Devils for a record £80million this summer, and has recalled the "classy gesture" after the incident.
“It was just after half-time when it happened. I had a clash of heads with Ravel and actually got knocked out,“ Maguire told Inside United.
"It was the only time I’ve been knocked out in my career, touch wood. It was a nasty blow and I was in hospital overnight.
“Obviously it was disappointing to go off and not see the end of the game, when the medals and the trophy were presented.
“But then someone got in touch from Manchester United and checked I was okay.

Maguire returns to Bramall Lane on Sunday afternoon with United and hopes he is given a good reception from his former fans.
“I’ll be back at Bramall Lane and I'm sure I’ll get a better reception than I did from the Leicester fans when they came to Old Trafford!” said Maguire, who made 166 appearances for the Blades.
“They've been great with me ever since, the Sheffield United fans, so I’ve got a lot to thank them for. The crowd got right on my side [from the start]."


HTTP Exception Middleware
This module processes Python exceptions that relate to HTTP exceptions by defining a set of exceptions, all subclasses of HTTPException, and a request handler (middleware) that catches these exceptions and turns them into proper responses.
This module defines exceptions according to RFC 2068 [1] : codes with 100-300 are not really errors; 400’s are client errors, and 500’s are server errors. According to the WSGI specification [2] , the application can call start_response more then once only under two conditions: (a) the response has not yet been sent, or (b) if the second and subsequent invocations of start_response have a valid exc_info argument obtained from sys.exc_info(). The WSGI specification then requires the server or gateway to handle the case where content has been sent and then an exception was encountered.
Exceptions in the 5xx range and those raised after start_response has been called are treated as serious errors and the exc_info is filled-in with information needed for a lower level module to generate a stack trace and log information.
References:
| [1] | http://www.python.org/peps/pep-0333.html#error-handling |
| [2] | http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5 |
catches exceptions and turns them into proper HTTP responses
This middleware catches any exceptions (which are subclasses of HTTPException) and turns them into proper HTTP responses. Note if the headers have already been sent, the stack trace is always maintained as this indicates a programming error.
Note that you must raise the exception before returning the app_iter, and you cannot use this with generator apps that don’t raise an exception until after their app_iter is iterated over.
httpexceptions middleware; this catches any paste.httpexceptions.HTTPException exceptions (exceptions like HTTPNotFound, HTTPMovedPermanently, etc) and turns them into proper HTTP responses.
warning_level can be an integer corresponding to an HTTP code. Any code over that value will be passed ‘up’ the chain, potentially reported on by another piece of middleware.
the HTTP exception base class
This encapsulates an HTTP response that interrupts normal application flow; but one which is not necessarly an error condition. For example, codes in the 300’s are exceptions in that they interrupt normal processing; however, they are not considered errors.
This class is complicated by 4 factors:
- The content given to the exception may either be plain-text or as html-text.
- The template may want to have string-substitutions taken from the current environ or values from incoming headers. This is especially troublesome due to case sensitivity.
- The final output may either be text/plain or text/html mime-type as requested by the client application.
- Each exception has a default explanation, but those who raise exceptions may want to provide additional detail.
Attributes:
- code
- the HTTP status code for the exception
- title
- remainder of the status line (stuff after the code)
- explanation
- a plain-text explanation of the error message that is not subject to environment or header substitutions; it is accessible in the template via %(explanation)s
- detail
- a plain-text message customization that is not subject to environment or header substitutions; accessible in the template via %(detail)s
- template
- a content fragment (in HTML) used for environment and header substitution; the default template includes both the explanation and further detail provided in the message
- required_headers
- a sequence of headers which are required for proper construction of the exception
Parameters:
- detail
- a plain-text override of the default detail
- headers
- a list of (k,v) header pairs
- comment
- a plain-text additional information which is usually stripped/hidden for end-users
To override the template (which is HTML content) or the plain-text explanation, one must subclass the given exception; or customize it after it has been created. This particular breakdown of a message into explanation, detail and template allows both the creation of plain-text and html messages for various clients as well as error-free substitution of environment variables and headers.
base class for status codes in the 400’s and 500’s
This is an exception which indicates that an error has occurred, and that any work in progress should not be committed. These are typically results in the 400’s and 500’s.
base class for 300’s status code (redirections)
This is an abstract base class for 3xx redirection. It indicates that further action needs to be taken by the user agent in order to fulfill the request. It does not necessarly signal an error condition.
base class for the 400’s, where the client is in-error
This is an error condition in which the client is presumed to be in-error. This is an expected problem, and thus is not considered a bug. A server-side traceback is not warranted. Unless specialized, this is a ‘400 Bad Request’
base class for the 500’s, where the server is in-error
This is an error condition in which the server is presumed to be in-error. This is usually unexpected, and thus requires a traceback; ideally, opening a support ticket for the customer. Unless specialized, this is a ‘500 Internal Server Error’


|
ABYZ News Links runs Google AdSense advertisements to help support the site.
Google AdSense may use technology such as cookies and web beacons when providing
advertisements to our site. This technology will send them information
such as your IP address, your ISP, the browser you are using to visit our site,
and possibly other information. This is generally used to help them target
AdSense ads to you based on your geographic location or based on your interests
according on your browsing history.
You can choose to disable or selectively turn off cookies by altering your
browser settings or by managing settings in your internet security program.
This can, however, affect how you are able to interact with some websites.
This may include requiring you to manually login each time you visit ecommerce
sites, forums, as well as certain other sites.
AdSense advertisements are provided by Google and are governed by their privacy
policy which is available at the
Google website.
ABYZ News Links does not have access to any of the information collected by
Google.
|


Postcard
I am your constituent and one of the 379,000 Angelenos who voted for Measure H. We need you to Keep your promise to swild supportive housing for Jun homeless neighbors.
Your Representative
Los ANgeles, California
FROM:
Mark Zahrenko
I am your constituent and one of the 379,000 Angelenos who voted for Measure H. We need you to Keep your promise to swild supportive housing for Jun homeless neighbors.
Your Representative
Los ANgeles, California
Mark Zahrenko


Blockbuster, uk, Video, Entertainment, New Releases, Coming Soon, Movies,
Games, dvd, music, cd, kids, competition, reviews, online store, charts, store
locator, press pack, rental Blockbuster UK. Welcome to the Blockbuster UK Web
Site.
The blockbuster uk web site contains details of current videos, movies,
playstation games, dvds and music. You can find details about the latest video
releases, videos for hire including the latest movies retaila and rental
charts. DVD information includes the latest charts and those new items that are
coming soon. Computer games are reviewed and listed. Music (CDs) are also
covered.
Video Rental. DVD Rental.
You can use our blockbuster store locator to find the address of your nearest
store together with details of vending machines in your area
Video reviews are posted and full details of The Movie Chart Show are also
available
Browse our extensive online store which contains thousands of videos (movies)
and dvds to buy
You need a browser that is capable of displaying FRAMES to view this site any
further.
Contents


Posted on Jan 30, 2011
New working drafts of the OEB are continually being added to this site. In this stage of the OEB’s life, we aren’t widely promoting the OEB outside of this site, so here is the place to come to find what’s new.
The good news is that keeping track of the changes between working draft versions is easy. There are two main pages to keep an eye on.
The first is the list of closed issues on Github. This is a list of changes which have been recommended (why not recommend one yourself?) and implemented.
The second is the commit history - every change goes here, small or large.
This is not only a handy way to keep up to date, but also part of our commitment to you - to be open and transparent in all of our translation choices.


{% csrf_token %}
Select the Type of Event *
Hackathon
Developer Meetup
Tech Fest
All Events
Chennai
Mumbai
Bangalore
Delhi
Hyderabad
Kolkata
Gurgaon
Pune
All
Submit
Event Count
100


There's a chance this is a false alarm, but we believe we detected government-backed attackers trying to steal your password. This happens to less than 0.1% of all Gmail users. We can't reveal what tipped us off because the attackers will take note and change their tactics, but if they are successful at some point they could access your data or take other actions using your account. To further improve your security, based on your current settings we recommend:
Join the Advanced Protection Program
Google's strongest protection for users at risk of targeted attacks.