Enlightenment
Comment
Spring 2021
Created: 2021-01-23 Sat 13:00
Enlightenment
Rationality
- "It may be that universal history is the history of the different intonations given a handful of metaphors." (*)
- Arguments for God: a priori and empirical
- Calvin's systematic development of the faith position sovereignty of God
- Quakers and Nonconformists: where is highest authority placed?
- Descartes: from scientific method to rational conclusions
(*) Pascal's Sphere where he traces a metaphor about God and the cosmos from the 6th c. BCE up to the 16th CE, all in 3 pages
Earthquakes: San Francisco
1906
Lisbon 1755
Lisbon Theodicy
Voltaire
- a tale of 2 earthquakes
- he found the evidence for his belief in nature rather than in the Bible; he doubted a good bit of traditional doctrine—and he didn't treat religion all that seriously.
- Revivals but at heart a move from God to human beings
Do we explain these by reference to God? No? We are inheritors of Enlightenment.
- p. 203 it is the shifting of place re. God and world –
- what do we pay attention to? What determines our interpretive framework?
Enlightenment from …
God-centered to Human-centered
- 5 catalysts for change (204)
- wars of religion
- Europe divided
- philosophical attitudes encouraged questioning tradition
- science seemed to being moving from one accomplishment to another …;
- nationalism taking root, centralizing power
- at the very time of success in discovery and technology, reason seemed to reach its end of life


Yort.Otp
A portable .Net class library for creating onetime passwords (based on [rfc4226] and [rfc6238]). These are the sorts of authentication codes used by Google/Microsoft/Facebook/Amazon etc. for second factor authentication.
Supported Platforms
Currently;
- .Net Framework 4.0
- Windows Phone Silverlight (8.1+)
- Xamarin.iOS
- Xamarin.Android
- WinRT (Windows Store Apps 8.1)
- UWP 10+ (Windows 10 Universal Programs)
Build Status
Available on Nuget
PM> Install-Package Yort.Otp
Samples
If you're implementing this on a server you should really read and understand the RFC specifications (linked above). There are sever specific issues around managing out of sync counters or client/server times that need to be understood.
Here are some quick start samples;
public void Main(string[] args)
{
// Generate a "time based" password from the current time (retrieved via DateTime.UtcNow),
// using the ASCII secret in the sample section of https://tools.ietf.org/html/rfc6238.
// This uses a default interval of 30 seconds (maximum length of time the password is value for).
using (var passwordGenerator = new TimeBasedPasswordGenerator(false, OnetimePasswordSecret.FromAscii("12345678901234567890")))
{
System.Diagnostics.Debug.WriteLine($"Password: {passwordGenerator.GeneratedPassword} valid until {passwordGenerator.ValidUntilUtc.ToLocalTime()}");
}
}
public void Main(string[] args)
{
// Generate a "counter based" one time password from a counter value of 10,
// using the ASCII secret in the sample section of https://tools.ietf.org/html/rfc6238
// The counter should be incremented on each *successful* login, and stored between sessions. See the RFC spec for details.
using (var passwordGenerator = new CounterBasedPasswordGenerator(false, OnetimePasswordSecret.FromAscii("12345678901234567890")))
{
passwordGenerator.Counter = 10;
System.Diagnostics.Debug.WriteLine("Password: " + passwordGenerator.GeneratedPassword);
}
}
If you're going to generate passwords for many different secrets but using an otherwise consistent configuration, factories can make that easier/more concise;
// Counter based factory sample
//During startup, obtain and keep a reference to a factory
_OtpFactory = OnetimePasswordGeneratorFactory.CreateFactory(true, new Sha512HashAlgorithm(), 8);
//When you want to generate a password...
using (var passwordGenerator = _OtpFactory.CreateNewPasswordGenerator(OnetimePasswordSecret.FromAscii("12345678901234567890"), 10))
{
System.Diagnostics.Debug.WriteLine($"Password: {passwordGenerator.GeneratedPassword}");
}
// Time based factory sample
//During startup, obtain and keep a reference to a factory
_OtpFactory = OnetimePasswordGeneratorFactory.CreateFactory(true, new Sha512HashAlgorithm(), 8, TimeSpan.FromMinutes(1));
//When you want to generate a password...
using (var passwordGenerator = _OtpFactory.CreateNewPasswordGenerator(OnetimePasswordSecret.FromAscii("12345678901234567890")))
{
System.Diagnostics.Debug.WriteLine($"Password: {passwordGenerator.GeneratedPassword} valid until {passwordGenerator.ValidUntilUtc.ToLocalTime()}");
}
This library doesn't generate QR codes, but it can generate the OTP uri to be turned into a QR code for configuring authenticator apps;
var configureAuthenticatorUri = OnetimePasswordGeneratorFactory.GetOtpUrl("account name or identifier", "SHA1", 8, OnetimePasswordSecret.FromAscii("12345678901234567890"), TimeSpan.FromMinutes(1));
Likewise, if given an OTP uri it can configure a password generator appropriately. This also returns the secret and account name, so they can be stored if required.
string label = null;
byte[] secret = null;
var passwordGenerator = OnetimePasswordGeneratorFactory.FromOtpUrl(configureAuthenticatorUri, out label, out secret);


Try out our Premium Member services: Virtual Legal Assistant, Query Alert Service and an ad-free experience. Free for one month and pay only if you like it.
BEFORE THE NATIONAL GREEN TRIBUNAL (WESTERN ZONE) BENCH, PUNE APPLICATION No. 43/2013(WZ) CORAM: Hon'ble Mr. Justice V.R. Kingaonkar (Judicial Member) Hon'ble Dr. Ajay A. Deshpande (Expert Member) B E T W E E N: 1. Mr. Asim Sarode Aged : 41, Occ : Advocate 2. Mr. Vikas Shinde, Aged 27 Yrs., Occn : Advocate 3. Mrs. Neha Pathak-Khati, Age 27 yrs., Occn : Advocate, 4. Mrs. Vinda Vilas Raut-Mahajan, Age 29 yrs. Occn. Advocate, Nos. 1 to 4 office at Flat No.1, Prathamesh Housing Society, Lane No.5, Prabhat Road, Pune 5. Mrs. Smita Singalkar-Sarode, Age 35 yrs., Occn : Advocate, R/at 2/3, 2/4, Lord Apartment, North Ambazari Marg, Dharampeth, Kharetown, Nagpur 10. 6. Mr. Santosh Sangolkar, Age 32 Yrs. Occn. : Advocate, R/at. Kanderao Nagar, Pimpala, Jalgaon. (J) Application No.43/2013 (WZ) 1 7. Ms. Roshani Wanode, Age 28 yrs., Occn : Advocate, R/at. Indira chowk, Babulgaon, District : Yavatmal. 8. Mr. Mahesh Bhosale, Age 30 Yrs. Occn : Advocate At Aurangabad, R/at. Pargaon, Tal. Patoda, Distt : Beed. 9. Ms. Savita Khotre, Age 30 yrs. Occn : Advocate, R/at. Old city, Kala Maruti Road, Akola 444 002 10. Mr. Rahul Mali, Age 33 yrs. Occn : Advocate, R/at. Asta, Tal. Islampur, Distt: Sangli, 11. Ms. Hema Katkar, Age 31 yrs., Occn : Advocate, R/at. Post : Hupari, Distt : Kolhapur. 12. Ms. Namrata Biradkar, Age 25 Yrs. Occn : Advocate, R/at. Sadguru Fabrication, Sadguru Hardev Nagar, Opp. Kalyani Industries, MIDC Latur 413 537. 13. Ms. Sarojani Tamshetti, Age 30 Yrs. Occn: Advocate. R/at. 34/2/10, New Pacca Peth, Ashok Chowk, Solapur 431 006. (J) Application No.43/2013 (WZ) 2 14. Mr. Amit Shinde, Age 29 Yrs. Occn : Advocate, R/at. Jayashree Apt. S.T. Colony, Vishrambaug, Tq. Sangli, Distt : Sangli 416 416. ....Applicants AND 1. Maharashtra Pollution Control Board, Through : Chairman, Kalpataru Point, 2nd, 4th floor, Opp. Cine Planet, Near Sion Circle, Sion (East), Mumbai 400 022. 2. State of Maharashtra, Through : Chief Secretary, Maharashtra State, Mantralaya, Mumbai 400 032. 3. State of Maharashtra, Through : Its Secretary, Environment Department, Mantralaya, Mumbai 400 032. ...Respondents Counsel for Applicants : Mr. Asim Sarode, Mr. Vikas Shinde, Counsel for Respondents : Mr. D.M. Gupte, Ms. Supriya Dangare, DATE : 6th September, 2014 (J) Application No.43/2013 (WZ) 3 JUDGMENT
1. The Applicants have filed instant Application U/s. 14(1)(2) and 15 of National Green Tribunal Act, 2010 raising questions relating to unauthorized and unscientific burning of tyres which emit smoke containing toxic gases and pollutants affecting the environment and human life. The Applicants submit that the burning of tyres has now become a regular feature in any social and political agitation resulting in pollution and environment damage. The Applicants claim that though they are not willing to blame any individual party or group, but urge that such tyre burning, which is environmentally unsafe activity, needs to be banned immediately. The Applicants further submit that through this Application, they want to raise the need of urgent actions to be taken up by the regulatory agencies which have been impleaded as Respondents, by framing suitable Rules and regulations and also prepare mechanism for enforcing the same. The Applicants further submit that with the increasing awareness on the environmental pollution and its adverse effect on the human health, it is necessary that such steps are taken on priority. The Applicants also draws the attention that there are specific regulations for re-use and re-cycling of used tyres, in most of the advanced countries and also, (J) Application No.43/2013 (WZ) 4 some developed countries and therefore, plead that India being signatory to various International Environmental Treaties, suitable regulations in this context are urgently required. It is the case of the Applicants that though the Environment (Protection) Act, 1986 and Air (Prevention and Control of Pollution) Act 1981 has given such powers to the regulating agencies, no specific rules or guidelines have been formulated by the Respondents. The Applicants, therefore, seek following reliefs:
i. The Respondents may be directed to adopt some scientific and advanced way by framing Rules regarding bar coding which will help to trace the used tyres and prevent the misuse of the old ones.
ii. The Respondents may be directed to frame some Rules to regulate the industry
relating to remolding of tyres in a more eco-friendly way, as other countries have in order to control the misuse of tyres. iii. The Respondent No.1-MPCB may kindly be directed to furnish details regarding number of industrial set up throughout Maharashtra State who are into business of remolding tyres, list of license holders who have been given permission to operate by the MPCB, incidences when MPCB taken actions against such industries for breaking environmental (J) Application No.43/2013 (WZ) 5 laws, level of toxic elements of the smoke coming out from these industries, measurements related to height of Chimneys at this various industries involved in remolding tyre work in Maharashtra.
2. The Maharashtra Pollution Control Board (MPCB), which is the Regulatory Authority, implementing Water (Prevention and Control of Pollution) Act, 1974 and Air (Prevention and Control of Pollution) Act 1981 is Respondent No.1. Chief Secretary, State of Maharashtra is Respondent No.2 and Secretary, Environment Department; Government of Maharashtra is Respondent No.3. Only Respondent No.1 has filed Affidavit. Though the notices were served upon Respondent Nos. 2 and 3, they have not filed any affidavit and therefore, considering while noting their absence and non-participation in the Tribunal proceedings, Affidavit of MPCB alone has been taken into account in the present adjudication.
3. The Respondent-1, MPCB has filed two (2) Affidavits through Shri V.M. Motghare, Joint Director. In the Affidavit dated 4th May 2014, MPCB informed that they have constituted a Committee to propose suitable guidelines/regulations for tyre re-treading and tyre recycling Industries. Through the Affidavit dated 11th July (J) Application No.43/2013 (WZ) 6 2014, MPCB has placed on record the report of the said Committee. MPCB submits that after sampling and analysis of tyre burning on laboratory scale basis, it has been concluded that the tyre burning in open atmosphere generates highly toxic, mutagenic and hazardous emissions, primarily owing to incomplete oxidation and therefore, MPCB has recommended the State Government, in exercise of the provisions of Sub-section 5 of Section 19 of the Air (Prevention and Control of Pollution) Act 1981, to issue a Notification prohibiting burning of tyres in the Air Pollution Control Area. MPCB has further suggested that the Law and Order enforcement Agencies i.e. Police and District Administration shall also be directed to take appropriate action against such tyre burning under the Bombay Police Act, Criminal Procedure Code etc. The MPCB's Affidavit further emphasize that the proper solution for minimizing such incidents of burning of tyres can be achieved only through creating and implementing a system for re-cycling, re-treading and re-using of used tyres, more particularly, by adopting "Life Cycle Approach" by encouraging and facilitating re-use and re-cycle of tyres. The report goes on to discuss various options to re-use and re-cycle in the International Scenario. The Communication (J) Application No.43/2013 (WZ) 7 of MPCB dated 8th July 2014, to the Environment Department in this regard is placed on record.
4. Considering the submissions made by the parties, and also documents placed on record, we are of the considered opinion that following issues need to be adjudicated in the present Application.
I. Whether the tyre burning cause air pollution and pose a threat to human health?
II. Whether the present Used Tyre Management practices can be termed as environmentally sound and complying the regulations?
III. Whether the Respondents can be directed to enforce environmental regulation in Used Tyre Management under the present Regulatory framework?
5. The MPCB i.e. Respondent No.1 in its Affidavit dated 11th July 2014 has submitted, after sampling and analysis, that tyre burning in open atmosphere generates highly toxic, mutagenic and hazardous emissions. However, the MPCB Affidavit has not dealt with health impact of such tyre burning. The scrap tyres represent both a disposal problem and also, resource opportunity (For example e.g. as a fuel material replacement, and in other Application). The open tyre burning has been reported in the literature to be more toxic and mutagenic. The open tyre burning emissions includes "criteria" pollutants such as particulates, carbon monoxide (CO), (J) Application No.43/2013 (WZ) 8 sulfur oxides (SO2), oxides of nitrogen (NOx) and volatile organic compounds (VOCs). They also include "non- criteria" hazardous air pollutants (HAPs), such as polynuclear aromatic hydrocarbons (PAHs), dioxins, furans, hydrogen chloride, benzene, polychlorinated biphenyls (PCBs), and metals such as arsenic, cadmium, nickel, zinc, mercury, chromium, and vanadium. Both criteria and HAP emissions from an open tyre fire can represent significant acute (short-term) and chronic (long- term) health hazards to nearby residents. Depending on the length and degree of exposure, these health impacts could include irritation of the skin, eyes, and mucous membranes, respiratory effects, central nervous system depression, and cancer.
6. The piled used tyres can also be a health hazard as they become breeding grounds for diseases causing pests and can even catch fire. Considering all these aspects, we have no hesitation to answer the Issue No.I in the Affirmative.
Issue No.II :
7. Used tyres can be disposed of for various end-uses such as fuel in the Industries, burning for heating purpose, resource recovery through distillation or pyrolysis process etc. With the increasing number of Motor (J) Application No.43/2013 (WZ) 9 Vehicles, there is rapid increase in the amount of scrap tyres. The large quantity of scrap tyres is posing a serious environmental concern for their environmentally sound disposal. The main reason for this problem is that used tyres are not easily biodegradable. Admittedly, there is no system or specific regulations, which will ensure the environmentally sound management of the used tyres, tracking its journey from generation to the final disposal. In the absence of such system, no records are placed before this Tribunal about number of used tyres generated and its end-use wise disposal. The Counsel for MPCB fairly submits that with the growing awareness about the environmental impact of such used material, there is a need for some regulations considering the Life Cycle Approach for such used tyre management. He, however, pleads that such a mechanism needs to be developed at National level, similar to Battery rules and also E-waste rules which have been notified under Environment (Protection) Act, 1986 by the MoEF.
8. Considering all the above aspects, we are of the opinion that there is a need to have a systemic approach to deal with the problem of used tyre disposal. This is more evident from the submissions of the MPCB that out of 162 tyre remolding industries, only 55 tyres were registered (J) Application No.43/2013 (WZ) 10 with MPCB which shows that everything is not well in the used tyre management. Further, there is no data or even approximation account available about total number of used tyres generated and end uses thereof. Simultaneously, we may also note that the CPCB is encouraging use of tyres as fuel for co-processing in cement/power/steel Industry subject to provisions of necessary Air Pollution Control Systems. Though, the used tyre is an opportunity in term of its contents and calorific value, there is need to systematically deal with the entire issue in a holistic manner based on "Life Cycle Approach", considering the pollution potential, tyre generation data, technology options, techno-economic viability and social implications. We are of the considered opinion that in order to formulate such regulation or notifying certain approach, it is incumbent that the MPCB shall conduct a scientific study about the used tyre generation, technologies, viability and its Life Cycle Assessment in order to form its strategy on a long-term basis. Therefore, while noting that the present system of used tyre management is not environmentally sound, we are of the opinion that there is a need of placing an elaborate and well defined system in place for environmentally sound tyre (J) Application No.43/2013 (WZ) 11 disposal practices for used/scrapped tyres. The Issue-II is accordingly answered in the Affirmative. Issue No.III :
9. The Environment (Protection) Act, 1986 has given powers to the Central Government to take measures to protect and improve the environment, under Section 3 of the said Act, which is reproduced below:
3. Power of Central Government to take measures to protect and improve environment-- (1) Subject to the provisions of this Act, the Central Government shall have the power to take all such measures as it deems necessary or expedient for the purpose of protecting and improving the quality of the environment and preventing, controlling and abating environmental pollution.
----------
----------
---------
---------
10. The Authorities are also given powers to give Direction under Section 5 of the said Act to effectively deal with environment degradation and protection concerns. The said Section is reproduced below :
5. Power to give directions -- Notwithstanding anything contained in any other law but subject to the provisions of this Act, the Central Government may, in the exercise of its powers and performance of its (J) Application No.43/2013 (WZ) 12 functions under this Act, issue directions in writing to any person, officer or any authority and such person, officer or authority shall be bound to comply with such directions.
Explanation -- For the avoidance of doubts, it is hereby declared that the power to issue directions under this section includes the power to direct--
(a) the closure, prohibition or regulation of any industry, operation or process; or
(b) Stoppage or regulation of the supply of electricity or water or any other service.
11. Air (P&CP) Act, 1981 also defines functions of the Central and State Pollution Boards in Section 16 and 17 respectively. The main function of Central Board shall be to improve the quality of Air and to prevent, control or abate pollution in the country. The Air (Prevention and Control of Pollution) Act 1981 also gives powers to declare Air Pollution Control areas to the State Government and Sub section 3 and 5 gives specific powers to control the use of fuel and also restricting burning of any material. The same are reproduced below :
19. Power to declare air pollution control areas.--
(1) The State Government may, after consultation with the State Board, by notification in the Official Gazette, declare in such manner as may be prescribed, any area or areas within the (J) Application No.43/2013 (WZ) 13 State as air pollution control area or areas for the purposes of this Act.
(2) --- (3) If the State Government, after consultation with
the State Board, is of opinion that the use of any fuel, other than an approved fuel, in any air pollution control area or part thereof, may cause or is likely to cause air pollution, it may, by notification in the official Gazette, prohibit the use of such fuel in such area or part thereof with effect from such date (being not less than three months from the date of publication of the notification) as may be specified in the notification.
(4) ---- (5) If the State Government, after consultation with
the State Board, is of opinion that the burning of any material (not being fuel) in any air pollution control area or part thereof may cause or is likely to cause air pollution, it may, by notification in the Official Gazette, prohibit the burning of such material in such area or part thereof.
12. We have been informed by the learned counsel of MPCB that entire State of Maharashtra has already been declared as Air Pollution area U/s. 19(1) of the Air Act.
The Boards have powers to give directions U/s. 31(A) which are reproduced below :
31 (A) Power to give directions.--Notwithstanding anything contained to any other law, but subject to (J) Application No.43/2013 (WZ) 14 the provisions of this Act and to any directions that the Central Government may give in this behalf a Board may, in the exercise of its powers and performance of its functions under this Act, issue any directions in writing to any person, officer or authority, and such person, officer or authority shall be bound to comply with such directions. Explanation--For the avoidance of doubts, it is hereby declared that the power to issue directions under this section includes the power to direct--
(a) the closure, prohibition or regulation of any industry, operation or process; or
(b) the stoppage or regulation of supply of electricity, water or any other service.
13. Considering all these legal provisions, it is abundantly clear that the provisions of the Environment Protection Act and also the Air (Prevention and Control of Pollution) Act 1981 give sufficient powers to the MoEF, CPCB, State Department of Environment and MPCB to deal with this issue.
14. The end-use of such used tyres can be broadly classified in three categories.
1. Open burning, which is generally incidental, like agitations, warming/heating purpose etc, mostly unorganized use.
2. Use as a fuel in the Industry, and in brick kilns.
3. Use for resource recovery i.e. chemical recovery through distillation or pyrolysis or some other use like used tyre based products i.e. mats, footwear etc.. (J) Application No.43/2013 (WZ) 15
15. MPCB has already placed on record the recommendations submitted to the department of Environment U/s. 19(5) of Air Act, to ban burning of tyres in open places and to direct the Law and Order enforcement agencies to deal with the issues vide their letter dated 8-7-2014. No information has been placed on record about the status of this proposal at end of the Environment Department. We are of the opinion that in order to deal with the Used Tyre use in category 1, as mentioned above, such proposal needs to be expeditiously considered and decision needs to be taken in this regard by the State Environment Department. We expect the Secretary, Department of Environment, Maharashtra to take a decision on this proposal expeditiously.
16. As regards Used Tyre category 2, we notice that the MPCB is competent to restrict such use of used tyres as an industrial fuel through its consent management process. However, as far as unorganized Industrial sectors like brick kiln, small and tiny units are concerned, MPCB and the Department of environment have necessary the powers conferred upon them under Section 19(3) of the Air (Prevention and Control of Pollution) Act 1981 to restrict use of used tyre as fuel by issuing necessary Notification. (J) Application No.43/2013 (WZ) 16
Unrestricted use of third category of used tyre can also be controlled by the MPCB through the Consent Management. However, in order to encourage and facilitate the use of used tyres either in category 2 and 3, it is necessary to frame suitable guidelines and/or regulations as described in above paras.
17. We, therefore, are of the considered opinion that there is an urgent need to regulate the used tyre disposal to avoid the environmental problems, on the principles of Sustainable development and pre-cautionary principles. And therefore, the MPCB need to undertake a scientific study for Life Cycle Assessment of Used tyres in Maharashtra adopting the scientific and analytical tools to deal this issue in a holistic manner. Several innovative approaches like Extended Producers Responsibility (EPR), Advanced Recycling charges (ARC), common facilities, use of bar coding etc can be adopted to ensure effective collection and disposal of used tyres. We, therefore, direct the MPCB to undertake such study in next six (6) months and finding shall be shared with the MoEF/CPCB.
18. We also expect that the MoEF and CPCB shall also take a note of this environmental concern and explore the need and possibility of framing separate regulations on the (J) Application No.43/2013 (WZ) 17 lines of battery rules and e-waste Rules in next six (6) months.
19. In view of the above, the Application is partly allowed in following terms, under the powers conferred upon the Tribunal under Section 19 read with section 20 of NGT Act, 2010.
1) Department of Environment, State of Maharashtra shall take a decision on recommendations made by the MPCB vide
letter dated 1-7-2014 within eight (8) weeks and issue necessary Notification in two (2) weeks thereafter.
2) There will be prohibition on burning of tyres in open areas and at public places, in the localities surrounded by the residential areas, public places, schools, hospitals, offices etc. in view of the potential air pollution and health hazards. The Police authorities, District Administration and urban local bodies shall ensure the compliance of this prohibition with immediate effect. In case of defiance it be treated as offence U/s. 188 of the I.P. Code.
3) The Department of Environment, State of Maharashtra and MPCB shall conduct a scientific study on the Life Cycle Assessment of used tyres and frame suitable guidelines/ regulations to ensure environmentally sound (J) Application No.43/2013 (WZ) 18 disposal practices of the used tyres in next eight (8) months.
4) The reuse of used tyres as fuel in industries, including brick-kilns etc. without specific permission of MPCB and also, provision of necessary area Pollution Control Systems is prohibited with immediate effect.
5) These directions and environmental effects of open burning of tyres shall be brought to the notice of all the concerned agencies by MPCB and state environment department and be given wide publicity for public information and awareness, in next two (2) weeks.
Accordingly, the Application is disposed of. No costs.
....................................,JM (Justice V. R. Kingaonkar) .................................., EM (Dr. Ajay. A. Deshpande) Date : 6ᵗᵑ September, 2014 (J) Application No.43/2013 (WZ) 19


VILLAGE OF THE DAMNED (1995) A film review by Mark R. Leeper Copyright 1995 Mark R. Leeper
Capsule: The second adaptation of John Wyndham's MIDWICH CUCKOOS is surprisingly restrained for a John Carpenter film. Fans of Carpenter's usual flashy special effects and gore may be disappointed to find them toned down, but the real enthusiasts of science fiction will welcome a return to a more traditional approaches. Rating: high +1 (-4 to +4)
[Spoiler warning: For those unfamiliar with the story, VILLAGE OF THE DAMNED has many small mysteries for the viewer starting with the very first scene. It is impossible to say anything about the content of the film without giving away some of the mysteries. This review will have only minor spoilers outside of one paragraph. That paragraph will have a separate spoiler warning.]
One of the best British science fiction authors of the 50s and 60s was John Wyndham Parkes Lucas Beynon Harris who wrote under several names, but the best-known were John Beynon Harris and John Wyndham. His most popular two novels, both written as John Wyndham, were THE DAY OF THE TRIFFIDS and THE MIDWICH CUCKOOS. The former has been filmed twice under its own name; the latter has just been filmed for a second time, both times under the title VILLAGE OF THE DAMNED, though the second film version perhaps owes more to the first film than to the novel. Incidentally, the novel THE MIDWICH CUCKOOS itself appears to owe a great deal to Richard Matheson's short story "Trespass" (a.k.a. "Mother by Protest"), Theodore Sturgeon's "More than Human," and Jerome Bixby's "It's a Good Life."
The director this time is John Carpenter. When Carpenter remade THE THING he threw a lot of special effects at the project and made a film respectful neither of the spirit of the original film, nor of the classic science fiction story on which it was based. (Ironically, the original story could be more satisfyingly adapted today with morphing effects than with the model effects he used.) His version of VILLAGE OF THE DAMNED is much less sensationalist in style, probably lower in budget, and at the same time a better representation of its story. The story is not one of screaming horror and dismembered bodies, but one with more a subtle and a cerebral sort of chill.
[Spoiler warning: do not read unless you already know the plot.] The story takes place in the village of Midwich but no longer Midwich, England. Now it is Midwich, California, as unlikely as that name sounds. It is a nice, isolated coastal town. In the middle of the day a mysterious sleep falls upon everybody in town. Simultaneously, everybody in town simply collapses where he is. Six hours later the sleep ends as mysteriously as it began. But things are not quite back to normal. Ten women come to realize they are pregnant. With one stillbirth, they give birth to what appears to be nine silver-haired children, but it is actually one single organism with a single mind and incredible mental powers.
Carpenter has put in a few touches to please an action-film audience. There are a couple of particularly nasty deaths, there are two fiery truck crashes, and toward the end there is a half-hearted action scene with guns blazing. But most of the violence is compulsion by mental suggestion and mental suggestion is what Carpenter uses to make it seem effective to the viewer.
Carpenter clearly respects Wolf Rilla's 1960 version. He credits the screenplay and lifts whole scenes from the earlier film's script. Visually he imitates the style of the earlier adaptation. Realizing how silly the superchildren would look in modern clothing styles, he dresses them in the same sort of clothing as the earlier film and gives them a similar silver-haired look and similar facial cast. And speaking of familiar casts, we have Christopher Reeve as the lead, an amalgam of two characters in the original, but primarily the George Sanders character. Many may disagree but Sanders played the role too smugly and confidently. Reeve conveys a wider range of emotions. Personifying the line "I'm from the Government, and I'm here to help you" is Kirstie Alley. There are interesting ironies in her character, but she adds little to them. As the town priest Mark Hamill underpowers the role and carries too much baggage from previous roles. The Children, like Reeve, may actually do a better job here than in the original.
For the most part, VILLAGE OF THE DAMNED looks like a science fiction film that could have been made in the 60s or 70s that is not half bad. I give it a high +1 on the -4 to +4 scale.
Mark R. Leeper<EMAIL_ADDRESS>The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Horizontal: (, )
Vertical: (, )
Slide the round sliders and the range slider to change the range.
Try to scroll your mouse wheel on different components of the range bar.
For the full spec and usage, see readme.


This document specifies an interface to sources of random bits, or "random sources" for brevity. In particular, there are three different ways to use the interface, with varying demands on the quality of the source and the amount of control over the production process:
(random-integer
n)
produces the next
random integer in {0, ..., n-1} and (random-real)
produces the next random real number between zero and one. The details
of how these random values are produced may not be very relevant, as
long as they appear to be sufficiently random.
Once random sources provide the infrastructure to obtain random bits, these can be used to construct other random deviates. Most important are floating point numbers of various distributions and random discrete structures, such as permutations or graphs. As there is an essentially unlimited number of such objects (with limited use elsewhere), we do not include them in this SRFI. In other words, this SRFI is not about making all sorts of random objects---it is about obtaining random bits in a portable, flexible, reliable, and efficient way.


Satori
Worlds
Islands
Miyajima
Routes
Regard
JPN'S
Seasaes
Shunyata
Reflets
Pointvue
Last works
Publications
Contact
This is the where the content will go!


Home
About
Destinations
Accomodation
Contacts
SMILE FOR MILES TRAVEL AGENCY
SIMPLY THE BEST ...
BOOK WITH US
NOW!!!
GET UPTO 50% REDUCED CHARGES THIS HOLIDAY!!


Two Ex-PSL Players Reunite Abroad!
by Soccer LadumaTwo players who previously plied their trades in the Premier Soccer League have reunited at a club abroad.
After some time in the football wilderness, following a stint with Tshakhuma Tsha Madzivhandila FC in the National First Division last season, Malawian striker Atusaye Nyondo has been given an opportunity to revive his career in Ethiopia.
Read – Breaking: Another PSL Club Parts Ways With Head Coach
The 28-year-old has signed a one-year deal with Ethiopia Premier Soccer League side, Saint George SC.
Speaking to the Siya crew, the player’s representative Kenneth Yamongbe confirmed the deal for the former SuperSport United and University of Pretoria player.
“Atusaye Nyondo has signed in Ethiopia. He has joined Ethiopia giants Saint George,” confirmed the representative.
Read: Pressure Mounts For Coach At Relegation-Threatened Chippa
“The club has won the Ethiopian League 29 times. Nyondo is currently in Addis Ababa. The team’s new Serbian coach (Srdan Zivojhov) likes him,” Yamongbe added.
In his new club, Nyondo has reunited with Getaneh Kebede, whom he played with at Tuks, during the Ethiopian striker’s time in Tshwane on loan from Bidvest Wits in the 2015/16 season.


{{item.title}}
{% extends "base.html"%} {% load staticfiles %} {%block head_title%} Applications {%endblock%} {% block content%}
While I have a background in games, due to my studies focusing on game programming, I have also developed simple applications to solve problems or to learn a concept. These applications aren't extensively polished or entirely user friendly but they have helped me to approach problems in new or creative ways.


--- layout: post title: 'xkcd: Marie Curie' categories: [] tags: - funny - philosophy - xkcd status: publish type: post published: true meta: _original_post_id: '' _original_import_author: John Duprey author: ---
Advice I'll be sure to pass on to M someday...


I am by birth a Genevese; and my family is one of the most distinguished of that republic. My ancestors had been for many years counsellors and syndics; and my father had filled several public situations with honour and reputation. He was respected by all who knew him for his integrity and indefatigable attention to public business. He passed his younger days perpetually occupied by the affairs of his country.
A variety of circumstances had prevented his marrying early, nor was it until the decline of life that he became a husband and the father of a family. As the circumstances of his marriage illustrate his character, I cannot refrain from relating them. One of his most intimate friends was a merchant, who, from a flourishing state, fell, through numerous mischances, into poverty.


| np_score |
hg19/CADD
|
0 |
9 |
17.66 MB |
|
| gene_models |
hg19/GATK_ResourceBundle_5777_b37_phiX174_short/gene_models/refGene_201309
|
0 |
5 |
6.32 KB |
## refGene gene models from 2013-09
Old gene models used by GPF up to 2019-02.
|
| genome |
hg19/GATK_ResourceBundle_5777_b37_phiX174_short/genome
|
0 |
9 |
899.07 KB |
## HG19 Reference Genome Fragment
Default HG19 reference genome used by GPF
|
| np_score |
hg19/MPC
|
0 |
10 |
437.92 KB |
|
| position_score |
hg19/internal/scores1
|
0 |
12 |
57.08 KB |
|
| position_score |
hg19/internal/scores_incomplete_coverage
|
0 |
8 |
27.85 KB |
|
| position_score |
hg19/phastCons100
|
0 |
7 |
2.46 MB |
|
| gene_models |
hg38/GRCh38-hg38/gene_models/refSeq_20200330
|
0 |
5 |
32.42 KB |
## refSeq gene models for HG38 from 2020-03
Default gene models used by GPF for HG38.
|
| genome |
hg38/GRCh38-hg38/genome
|
0 |
8 |
2.71 MB |
## HG38 Reference Genome
Default HG38 reference genome used by GPF
|
| np_score |
hg38/TESTCADD
|
0 |
9 |
44.67 KB |
|
| allele_score |
hg38/TESTFreq
|
0 |
9 |
37.14 KB |
|
| position_score |
hg38/TESTPosAgg
|
0 |
7 |
21.12 KB |
|
| position_score |
hg38/TESTphastCons100way
|
0 |
11 |
26.69 KB |
|
| liftover_chain |
hg38/hg38tohg19
|
0 |
4 |
1.19 MB |
|


Graduate Admissions Information
Submitting your application online
To apply, you must use our online application. When using the online application, you can submit all materials electronically.
The online procedure is user-friendly; however, if you have any questions regarding the application process, please view the Application Resource Center's frequently-asked questions.
Note: You must arrange with the Educational Testing Services (ETS) to electronically submit GRE and TOEFL scores (if necessary) to our Graduate School of Arts and Science (GSAS). Our institution code for the GRE and TOEFL is 2596. You do not need to use a special department code for the GRE.
If you are required to take the TOEFL, this examination requires you to list a department code that is most appropriate for your field of study or you can simply use code 99. However, do not use code 00 and do not leave this field blank. We will not receive your test scores if you report it this way.
To expedite the processing of your application, you may include the unofficial copies of your results on these tests. However, we must receive the official scores from ETS before the admissions deadline.
Applicants should note the following:
- Students with a BA/BS degree may apply directly to the PhD program.
- There is no NYU or departmental based financial support for master's students.
- GRE scores are required for all but non-degree applications.
Mailing address for paper application materials
We highly encourage you to submit your letters of recommendation through the online application system. However, if your recommenders are inclined to send the letters by post, please mail the documents using the instructions below.
- via regular postal service:
-
New York UniversityGraduate School of Arts and SciencesGraduate Enrollment ServicesP.O. Box 907, Cooper StationNew York, NY 10276-0907
- via private express mail:
-
New York UniversityGraduate School of Arts and SciencesGraduate Enrollment ServicesOne-Half Fifth AvenueNew York, NY 10003
GSAS admission deadlines
-
MSCS October 1st March 1st
-
MSIS October 1st March 1st
-
PhD - December 12th
The admissions evaluation process
All regulations governing admission to the Graduate School of Arts and Science must be met by all those seeking admission to graduate study in computer science. In particular, students must have an accredited baccalaureate degree or its equivalent.
The following items are required:
- A completed application, including a Statement of Purpose.
- Three letters of recommendation
- General GRE scores for all programs. (The computer science GRE subject test is recommended but not required for application to the PhD program.)
- TOEFL scores for any student whose native language or medium of undergraduate instruction is not English
- For MSIS applicants only: a resume or CV of your work experience.
Successful applicants to the MS program in Computer Science will generally have the following minimum qualifications:
- Strong GRE Quantitative and Analytical scores. (Typical acceptable scores are near 700 for each of these sections, or 4.0 or better on the new Analytical Writing section.)
- Positive, specific recommendations.
- Strong grades in relevant courses.
MSIS applicants generally need the following to be admitted:
-
Exams:
- Quantitative and analytical both above 85%, at least one above 92%
- TOEFL, if required, above 620 (paper) or 260 (computer), or 100 (TOEFLI)
-
Grades:
- Some As and all Bs or better in business, engineering and science courses at most recent school; higher grades expected from less selective institutions
-
Recommendations:
- Strong recommendations from professor(s) and employer(s)
-
Work experience:
- At least one year full-time experience in a demanding position
- Evidence of leadership and management potential
- Evidence of ability to excel professionally in a corporation
-
Computer science:
- Evidence of ability to excel in graduate courses in computer science
-
English:
- Evidence of ability to communicate effectively in writing and speech in graduate school and American corporations
The minimum prerequisite background for admission to the MS program consists of:
- Languages: Deep working knowledge of C and familiarity with object-oriented concepts and work with some object-oriented language such as Java or C++.
- Data structures and mathematics: Understanding and working knowledge of pointers, lists, stacks, queues, trees, arrays and recursion; induction, order of magnitude growth, probability and elementary combinatorics, set notation.
- Working familiarity with windows and Unix.
- Knowledge of assembly language (e.g. Intel or Motorola) sufficient to understand self-modifying code.
Promising students who do not have this background will be conditionally admitted with the provision that they complete the one year preparatory course (PAC). Students without adequate mathematical training should take Discrete Mathematics, which is offered in the summer only.


An experiment by NYPL Labs developer Brian Foo.
This is a tool for warping, layering, and masking of images. You can read more about it here.
Drag the control points to each corner of the painted wall above and repeat for each image. Once complete, you will be able to align and manipulate them.


Obolo Qwerty Keyboard
© 2020-2021 Rogers Katelem Edeh
The keyboard uses QWERTY keyboard layout to type Obolo Language. In this keyboard, accents are typed after the base letters. So, to type "usem Obolo" with this keyboard layout, simply tap the given letters and tap the appropriate accents to pace them on the letters.
The desktop layout is as shown:
Due to the size and number of keys, some characters are hidden in the long press. Press and hold on the key with a little dot on the top right to reveal and use them.
In this language, note that: In this language, note that:
(1) You cannot place more than one accent on a letter.
(2) Some accents are optional and are not required. They are only used in primers and for showing pronounciations.
(3) Some letters do not accept some accents.
(4) Some letters do not accept accents at all.
Except for the optional accents which are not required, the keyboard will notify you when you inadvertently go against the spelling rules.
If what you are doing does not support 'n̄' or does not render it well, then you can tap the BACKSLASH key after the 'n̄' to get 'ñ'.
The position of your BACKSLASH key depends on type of keyboard you are using.
The three keyboard types and the respective position of the BACKSLASH keys are shown below.
You can tell where your BACKSLASH key is by looking at the shape of your ENTER/RETURN key:
Keyman ... type to the world in your language.
Obolo Qwerty ... ge inu me usem kwun̄ ria si ere geelek me linyọn̄.
Onirọ-o!
Itọn̄ ikup inyi Awaji!


The first whales evolved 50 million years ago.
The modern form of humans evolved about 200,000 years ago. And
humans began farming about 10,000 years ago.
Which means we have been hunter-gatherers for over 10,000
generations of our lineage and farming for about 500 generations.
How many generations have we allowed to hone trial and error,
mistakes and learning, understanding and application of action and
consequence in modern fields of this millenia before we call
ourselves experts?
The world population was 1.2 billion before the discovery of oil and
the advent of the Industrial age.
It is now over 7.4 billion and estimated to reach
9.7 billion by 2050.
Studies show that tigers began evolving 3.2 million years
ago. By estimates in 2010, there are less than 3,890 tigers
in the wild.
Their habitat once extended all across East and South-East Asia.
We’re such shits.
We have driven 41,415 species to serious threat, and 16,306 of them
are endangered species close to extinction.
To most people those numbers would seem abstract and meaningless.
What species, where, how did it happen, what difference does it
really make?
What difference can it make to someone who has not yet had the
experience of staring at the natural world around them for no reason
but to take some of it in; oberve, reflect, listen and feel.
Because it’s always speaking.
Every educated human should go through the initiation of
learning narratives of history that are other than
anthropological.
It’s a dangerous to grow up with the singular perspective that
the modern human life is the only future to make decisions about
or where to understand the past from.
The Future // Backward
First World // Developing
Urban // Rural
City // Village
Company // Community
Technology // Ancient
Advanced // Primitive
Exploitation is too easily denied by priviledge. Discrimination
is too easily enforced by priviledge. Self-imposed law by
priviledge in power is totalitarian.
What comes after checking the priviledge we have or finding a
way to express the exploitation we face?
I am the younger of two sisters. I was the sibling with the
strange habits of rationing my own treats over days so I could
somehow enjoy it longer, the one who stretched the use of a
favourite article of clothing, the child who found five uses for
a piece of cardboard before it had to be thrown out, the student
who picked fights with others who trashed the room or the grounds,
the teenager who loved the old city that Bangalore once was for
the trees and the greenery, the young adult who looked up and read
about the terms for different kinds of governence and wondered why
we didn’t study about that in school. The young adult who
questioned systems of centralised power. The young woman who
questioned the nature of the cages around her, who then started
to build her own reasoning born of intuition. And let that seep
into the choices of how she wants to live.
If one can even glimpse at timeline of different scales
- of microbes and forest ecosystems, of geological
events and patterns of climate, the inhale and exhale of
water in its various forms in and on our planet, the
precise conditions it takes for minerals, natural oils
and gases to form beneath the surface, for creatures to
respond to each factor of their environment and evolve.
If one can even have a small glimpse into the depth of
these phenomena, we might share a connectedness that
could invite a more balanced and sustainable way of
living and caring.
*** of inheritance


Alandra GreenleeDoctor - Athlete - Health Innovator
Portfolio
About
Dr. Alandra Greenlee is an innovative and versatile medical professional with strong interpersonal skills and passion for technology and fitness seeking to accelerate the adoption of data-driven health solutions in the U.S. healthcare system to improve health outcomes.
My passions lie at the intersection of health, technology and fitness. I am eager to take on challenges, learn and apply new skills and creatively solve problems. Stay tuned for exciting projects and news. In the mean time, check out my blog for health and fitness tips.
StridePRO, launched on Instagram September 2015, is my personal brand. StridePRO's primary focus is spreading current health research and fitness knowledge in a fun and personable way. I have documented my training leading up to the 2016 Arnold Sports Festival Indoor Triathlon as well as the Pittsburgh Half Marathon. I offer a variety of diet plans to clients - carb cycling, intermittent fasting, vegetarian, vegan - as well as triathlon training and gym workouts.

A study published this month (December) in The Journal of Nutrition declares USDA scientists have found that walnuts contain 21% fewer calories than previously thought!
This means that a typical (lol) 28-gram serving actually contains 146 calories, which 21% les than the current value of 185 calories given by the USDA.
So, you may be wondering how on earth they messed up for so long (I know I was). This is actually because of the way calories were measured in the 1950s. Scientists grouped walnuts and other tree nuts with foods like dry beans, legumes and peas. For those foods, these methods work well, but when it comes to tree nuts, the values are over-estimated because the body doesn't absorb the full caloric content due to the difficult to digest fiber of the nuts.
Similar methods have found that almonds contain 20% fewer calories than previous standards, and pistachios contain 5% fewer.
The spin: Ok, so this is exciting and all, but when you think about it, this shouldn't change your diet game-plan TOO much.  Just because the papers say walnuts contain fewer calories doesn't mean that your body will magically change. Your body has known this all along!
But as for me, I may indulge in a few more walnuts and almonds this holiday season.
Health & Fitness Writing: LIVESTRONG
As a health and nutritional writer, I have written over 50 articles on topics ranging from how to perform an exercise to the glycemic index of a prune.
Each article is Search Engined Optimized and includes supporting research.
Athletics have played a large role in my life, from swimming to fencing and from weightlifting to underwater hockey. I'm always eager to connect with people through health and fitness. Be sure to check out my blog for race recaps and my upcoming challenges.
Accomplishments
2015-2016
Arnold Sports Festival Indoor Triathlon 2016: 1st place swimming, 2nd place overall
Pittsburgh Half Marathon 2016: 24th place age group (1:39)
East Grand Traverse Bay Crossing - 4.2 miles open water swim
Pittsburgh Public Schools Obama Academy Middle School Boys Swim Coach - 2nd place at championships
Canton Turkey Trot 5k - 2nd place Women (21:10)
2014 & Earlier
Swimming - Two time high school D1 all-American
Michigan State Record Setter, State High Point Champion - 1999
IMT Des Moines Half Marathon - 11th place
Fencing - Women's Epee Captain 2010
Doctor of Podiatric Medicine
As an athlete, I was drawn to podiatric medicine as a means for keeping athletes on the field as well as combating diabetic complications and allowing diabetic patients to enjoy an active lifestyle for as long as possible.
Des Moines University - College of Podiatric Medicine and Surgery, Class of 2014
During medical school, I was involved in multiple research projects within the College of Podiatric Medicine and Surgery and served as research liaison for the college.


{{> ../layouts/head.html }}
Item OneItem TwoItem ThreeItem Four
More Actions
Sub-item OneSub-item TwoSub-item Three
Even More Actions
Change Settings
Setting OneSetting TwoSetting ThreeREAAAAALLLLY Long Setting Number FourSend MailFilter ContentPopup MenuThe Popupmenu on this page is triggered immediately on page load.
After closing the first time, the Popupmenu is reconfigured to be activated on contextmenu events.
{{> ../layouts/footer.html }}


The File Menu contains options related to managing and creating new Password Safe databases.
Create a new blank password safe database.
Open an existing password safe database. When presented with the Open dialog, you may specify to open an existing database in Read/Write of Read-Only mode. After opening, the current status will be shown in the Status Bar at the bottom of the Password Safe window. At any time after this, the user may double-click on that part of the Status Bar to switch between these two modes or use a menu item on the "File" menu. It may not be possible to switch a database to Read-Write mode if another user currently has the database open in Read/Write mode. You may also use the change mode menu item further down this menu to switch modes instead of double-clicking on the Status Bar.
Closes the currently open password safe database without exiting the application.
Depending on whether the user has enable the use of the System Tray, this locks the currently open password safe database either to the Task Bar (left-click to show dialog to unlock) or into the System Tray (right-click to show the context menu where you can restore the database or just unlock it). The Master Passphrase will be required to reopen/unlock the database. The Lock button can also be placed on the Toolbar using the Toolbar Customization (right-click on the Toolbar).
This section of the file menu may list the recently opened database files. This section will be blank if this is the first time Password Safe has been used.
Notes:
Save any changes to the current safe. If no changes have been made since the previous save, or if the database was opened read-only, this choice will be greyed-out.
Save a copy of the current safe under a new name or location.
Export the contents of the current database to an external file in either an old Password Safe database format, XML, or as text. If a filter is active, you can export those satisfying the filter conditions to a Password Safe database in the current format. In this case, if an alias or shortcut is included in the filter results, its base entry will also be exported even if it does not satisfy those conditons. See here for more information on exporting the database to other formats.
Import the contents of a previously exported or correctly formatted text file, XML, or a KeePass V1 password database into the current database. See here for more information on importing text files.
Merge a Password Safe database or backup file (.bak) into the current database. Disabled in read-only mode. Merge uses a Wizard interface. You may also select the Advanced field selection in order to restrict which entries you merge into your current database.
Compare the currently opened database with another. Compare uses a Wizard interface. You may also select the Advanced field selection in order to restrict which entries and which fields of these entries you compare between the two databases. See here for more information when the results are displayed.
Synchronize the currently opened database with another.
The main difference between Merge and Synchronize is that during a Merge, all elements of the merging database will be taken to the currently opened database. During synchronizing however, only elements in the currently opened database will be updated. Synchronize uses a Wizard interface. You may also select the Advanced field selection in order to restrict which entries and which fields of these entries you synchronize between the two databases.
A database that's open for reading and writing (the default) can be protected against unintended changes by selecting the Change to R-O entry. Likewise, a read-only database can be modified after clicking on Change to R/W. When changing to read-write, you will be prompted for the master password. You can also switch between these modes by double-clicking on the R/W / R-O text in the status bar on the bottom of the application's window.
Note: The Merge, Compare and Synchronize operations may take a long time, especially for large databases.
Display information about the currently opened password safe database, such as: format version, number of groups, number of entries, and information about when it was last saved and by whom.
Closes the Password Safe program. If changes were made to the database since the last save, you will be prompted to save the database.
Note: Compare, Export, Import, Merge and Synchronize all produce a report that can be viewed in a dialog and then copied to the clipboard or saved to disk as a text file in the same directory as the database. The "View → Reports"menu item may then be used to view the reports at a later date. The name of the report file is fixed and dependent on the function that created it. If the same function is repeated and its report saved to disk, any existing report for this function will be overwritten. Note: The report may contain sensitive data and care should be taken when selecting the "save to disk" option.


Regulation of B-cell commitment to plasma cells or to memory B cells.
Abstract
During humoral immune responses, B-lymphocyte activation is followed by differentiation along either the plasma cell pathway or the memory B-cell pathway. Recent studies suggest that CD40-CD40 ligand, OX-OX40 ligand, a group of cytokines and intracellular transcriptional factors may all contribute to B-lymphocyte differentiation control.


Get (Advanced)
DataMapper has extended versions of most of its query clauses that allow for advanced querying on relationships.
Subsections
- Get (Advanced) (This Section)
- Get (Advanced Selection)
Example
Let's go through an example to see the benefits. Let's say we have a User model and a Group model. A group can have many users but a user can only have one group. Here's how you would look up all users belonging to the Moderator group without the advanced query:
// Create user object $u = new User(); // Get all users $u->get(); // Loop through all users foreach ($u as $user) { // Get the current user's group $user->group->get(); // Check if user is related to the Moderator group if ($user->group->name == 'Moderator') { // ... } }
Here's how you would do the above, but using an advanced query:
// Create user object $u = new User(); // Get users that are related to the Moderator group $u->where_related_group('name', 'Moderator')->get(); // ...
As you can see, it's a big time saver but not just in the amount of code you write, but also in the number of database queries and overall processing time.
Supported Query Clauses
The following are the normal query clauses that you can use in the advanced queries. One of these must replace {query} in the methods below:
- where
- or_where
- where_in
- or_where_in
- where_not_in
- or_where_not_in
- like
- not_like
- or_like
- or_not_like
- ilike
- not_ilike
- or_ilike
- or_not_ilike
- group_by - For grouping results
- having - For grouping results
- or_having - For grouping results
- order_by - For ordering the results
$object->{query}_related_{model}($field, $value);
There are a number of ways you can use these advanced queries, and this is the first usage format. All examples are done with the User and Group objects scenario.
- {query} - Replace with supported query type.
{model}- Replace with related model name OR the relationship key for advanced relationships.- $field - First parameter for chosen query type.
- $value - Second parameter for chosen query type.
Here's an example using the where query:
// Create user $u = new User(); // Get all users relating to the Moderator group (goes by 'group', 'name', 'Moderator') $u->where_related_group('name', 'Moderator')->get();
$object->{query}_related($model, $field, $value);
Alternatively, rather than specifying the related model as part of the method, you could instead supply it as the first parameter. You must use this format when querying deep relationships.
- {query} - Replace with supported query type.
$model- Supply related model name OR the relationship key for advanced relationships. Also accepts deep relationships.- $field - First parameter for chosen query type.
- $value - Second parameter for chosen query type.
Here's an example using the where query:
// Create user $u = new User(); // Get all users relating to the Moderator group (goes by 'group', 'name', 'Moderator') $u->where_related('group', 'name', 'Moderator')->get();
Deep Relationship Queries
This format also accepts deep relationships, so you can query objects that are indirectly related to the current object.
A deep relationships is simply the name of each related object, in order, separated by a forward slash (/).
Here's an example:
$u = new User(); // Get all users that are associated with a : // -> Project that have one or more ... // -> Tasks whose ... // -> Status is labeled 'completed' $u->where_related('project/task/status', 'label', 'completed')->get();
The generated query for this simple request is surprisingly complex!
SELECT `users`.* FROM `users` LEFT OUTER JOIN `projects_users` as `projects_users` ON `projects_users`.`user_id` = `users`.`id` LEFT OUTER JOIN `projects` as `projects` ON `projects_users`.`project_id` = `project`.`id` LEFT OUTER JOIN `tasks` as `project_tasks` ON `project_tasks`.`project_id` = `projects`.`id` LEFT OUTER JOIN `statuses` as `project_task_statuses` ON `project_tasks`.`status_id` = `project_task_statuses`.`id` WHERE `project_task_statuses`.`label` = 'completed'
For deep queries as the example above, you should almost always call distinct, to ensure that the database doesn't return duplicate rows.
$object->{query}_related($related_object, $field, $value);
- {query} - Replace with supported query type.
$related_object- Supply related object (may not work for advanced relationships).- Optional: $field - First parameter for chosen query type.
- Optional: $value - Second parameter for chosen query type.
Both the $field and $value parameters are optional if the $related_object contains a valid id.
Here's an example using the where query:
// Create and get the Moderator group $g = new Group(); $g->get_by_name('Moderator'); // Create user $u = new User(); // Get all users relating to the Moderator group (goes by 'group', 'id', $g->id) $u->where_related($g)->get();
Here's a similar way of doing the above, but with an unpopulated related object (no id):
// Create and get the Moderator group $g = new Group(); // Create user $u = new User(); // Get all users relating to the Moderator group (goes by 'group', 'name', 'Moderator') $u->where_related($g, 'name', 'Moderator')->get();
Which of the available usage formats you use will depend on your personal preference, although you should be consistent with your choice. It also might depend on whether you have a related object already available to use.
$object->{query}_join_field($model, $field, $value);
This method allows you to query extra columns on a join table.
- {query} - Replace with supported query type.
$model- A related model name OR the relationship key for advanced relationships, or a related object.- $field - First parameter for chosen query type.
- $value - Second parameter for chosen query type.
You always have to include $related_field, even if the query is coming from a relationship. In other words, you’ll often write code like this:
$user->alarm->where_join_field($user, 'wasfired', FALSE)->get();
Here's an example using the where query:
// Create alarm $alarm = new Alarm(); // Get all alarms that have not been fired for one or more users $alarm->where_join_field('user', 'wasfired', FALSE)->get();
See Working with Join Fields for more details.
Get (Advanced Selection)
You can also perform some more advanced options when selecting columns, by including columns from related models or from the join table.
$object->include_related($model, $fields = NULL, $prefix = TRUE, $instantiate = FALSE)
Includes the all or some of the columns from a related object. By default, this method adds a prefix based on $model to every column. If for some reason the included column overlaps with a field already in the $object, that column is skipped. This method can significantly reduce your query overhead.
Important: This method will only work with $has_one related models.
$model- A related model name OR the relationship key for advanced relationships, or a related object. Also accepts deep relationships.- $fields - NULL or '*' to include all columns. To specify a subset of columns (recommended), replace with a single value, or an array of column names.
- $prefix - If TRUE, prepend "{$model}_" to the column names. If FALSE, don't prepend anything. If any string, prepend "{$prefix}_" to each column.
- $instantiate - If TRUE, then actual objects are instantiated and populated with the columns automatically.
Here's an example:
// Create User $u = new User(); // add the group id and name to all users returned $u->include_related('group', array('id', 'name'))->get(); foreach($u as $user) { echo("{$user->group_name} ({$user->group_id})\n"); }
If you use $instantiate, then you can use the related objects directly, like so:
// Create User $u = new User(); // add the group id and name to all users returned $u->include_related('group', array('id', 'name'), TRUE, TRUE)->get(); foreach($u as $user) { echo("{$user->group->name} ({$user->group->id})\n"); }
Including Fields from Deep Relationships
This method also supports deep relationships. You can only include columns from objects that are related by single relationships all the way. The default column prefix for deep relationships is to replace all forward slashes with underscores. You can still override this to be whatever you want.
A deep relationship is simply the name of each related object, in order, separated by a forward slash (/).
Here's an example:
// Create Post $p = new Post(); // Include the user's name in the result: $p->include_related('user', 'name'); // include the user's group's name in the result: $p->include_related('user/group', 'name'); $p->get(); foreach($p as $post) { echo("{$post->user_name} ({$post->user_group_name})\n"); }
At this time, deep relationships do not support instatiation.
$object->include_related_count($related_field, $alias = NULL)
This method can be used to include the number of related items. By default, this is stored in the alias {$related_field}_count, but you can override this alias using the second argument. This method also supports using deep relationships, although the operation may fail for relationships that are not has_one (excluding, of course, the last).
This method requires the use of subqueries.
Example:
$groups = new Group(); $groups->include_related_count('user')->get(); foreach($groups as $group) { echo("The group {$group->name} has {$group->user_count} User(s)\n"); }
$object->include_join_fields()
There are no options for this method. Set it right before adding a relationship. You can either use it before a {$query}_related_{$model}, or before calling get() on a related item. All fields on the table that are not part of the relationship are included, and are prepended with "join_".
This method may return unexpected results or throw errors with deep relationships.
Usage:
// Create User $u = new User(); $u->get_by_id($userid); // get all alarms for this user, and include the extra 'wasfired' field $u->alarm->include_join_fields()->get(); foreach($u->alarm as $alarm) { if($alarm->join_wasfired) { echo("{$alarm->name} was fired\n"); } else { echo("{$alarm->name} was NOT fired\n"); } }
See Working with Join Fields for more details.


This week’s machine learning course is about Anomaly Detection and Recommender Systems.
What is Anomaly Detection?
Given a bunch of x’s (where each x is a vector - a training example), detect whether for a new example x (a new vector), possibility p(x) < ε (epsilon). If yes, it is considered an anomaly; otherwise, it is considered normal.
What is Gaussian (Normal) distribution?
Gaussian distribution is defined as X ~ N(μ, σ^2), where μ (pronounced mu) is the mean of x; σ (pronounced sigma) is the standard deviation; σ^2 is the variance.
Anomaly Detection Algorithm
p(x;μ,σ^2) uses Gaussian distribution to plot the function of x for given a fixed value of mu and of sigma squared.
Recommender Systems: Collaborative Filtering Algorithm
Collaborative Filtering Algorithm is based on linear regression. We can think of each movie has its features: x1, x2, …, xn. x1 may represent how romance the movie is, x2 may represent how action the movie is, etc. Then if the user has rated enough movies ( 1<= the rating y <= 5), we can use linear regression to predict hθ(x), given the features of a movie.
Learning features
Not only we can learn the thetas of each user, we can even learn the value of features of each movie automatically by using Collaborative Filtering Algorithm.
Given a dataset that consists of a set of ratings produced by some users on some movies, you wish to learn the parameter vectors x(1),…,x(nm),θ(1),…,θ(nu) that produce the best fit (minimizes the squared error).


--- layout: post title: "My Week in Review" date: 2018-05-20 10:00:00 +0000 tags: image: https://s3.amazonaws.com/audreywatters/2018-05-20-lion.jpg imagecredits: https://www.flickr.com/photos/cogdog/3506335986 ---
Icon credits: The Noun Project


Software Developer, Photographer, Runner
My First Ramen in Japan
After finally arriving in Tokyo we set out to accomplish one of my goals for the trip - to experience true Japanese Ramen.
Our hostel is based in the Asakusa district of Tokyo which is known for it’s famous temple - Sensoji. We set out to explore the area and stumbled upon an empty shop. I didn’t expect much from it given it was the first place that we encountered but thankfully I walked away after having one of best ramen experiences of my life.
The shop itself was made up of only a few chairs and the two chefs. One made the ramen while the other made the gyoza.
Starcraft
I am a big fan of the game Starcraft - both playing it and watching pros compete at it. The game has become a global phenominon since the original was released in 1998.
To get an idea of what it is like at the highest levels - picture the strategy of chess with the adrenaline producing rush of a wrestling match. No place appreciates the game more than South Korea - home of the best players in the world.
On March 12th Blizzard would be releasing the newest expansion to Starcraft 2. To celebrate the Blizzard held a global launch event that began in Seoul and traversed the timezones for the next 24 hours. They gathered the top pros from Starcraft and Starcraft 2 and put them on stage to demonstrate the potential of the new expansion.
These players have been given nicknames by their fans such as God and the Tyrant. They have legions of fan girls and are paid hundreds of thousands of dollars each year by the largest companies in South Korea all to compete at the highest level.
It was an amazing opportunity to be at the Mecca of Starcraft and be able to witness the birth of a new era for the game. After waiting in line with hundreds of other fans I was able to pick up one of the very first copies of the collector’s edition sold in the world.
Ddeokbokki
Ddeokbokki is one of the first Korean dishes that Seyoung taught me. Whenever she is sick and needs something spicy and comforting I make it for her. It is simply made up of rice cakes, fish cakes, vegetables and a chilli paste based sauce.
Before coming to Korea I had no idea just how common a dish it is. Just about everywhere we went it could be found on street corners sold by vendors. It is also pretty cheap - a typical meal costs about $3 and was enough to feed two. At the end of a busy day of sight seeing we would find a random street vendor who would serve it with a warm cup of broth and eat it with tooth picks.
Daegu
Daegu is the birthplace of my fiance (who took the above picture). When I first met her 6 years ago we didn’t think we would ever have the opportunity to visit her hometown.
Over the years I built up an expectation for Daegu. To listen to people describe it - Daegu is a back country town filled with uninteresting people and little to do. My experience was pretty far from this. Maybe it is because Seoul is a gigantic, crowded and bustling city and in comparison to that just about any other city in the world is small. Daegu though for me was a wonderful change of pace. Everyone we encountered was warm, welcoming and helpful. Growing up in Hawaii and living in Portland I am perhaps more accustomed to a slower pace to life than what Seoul offered. Daegu felt way more livable.
Palgongsan
I visited Palgongsan while visiting family in Daegu South Korea in March during my 3 week trip to South Korea and Japan.
This was yet another example of a landscape dominated by the haze from pollution from China. To get here you take a beautiful ride up in a cable car that seems designed to appeal to young lovers.
Seoraksan
I visited Seoraksan (a mountain in South Korea) in March during my 3 week trip to South Korea and Japan.
South Korea for me was dominated by pollution and smog that drifted from the neighboring China, as a result this picture has a fogginess to it. What isn’t captured is just how windy it was there - I at several points nearly lost my glasses.
Seoraksan is the sort of place that I will most certainly have to visit again at some point in my life. Picture monk filled temples over-looking crystal clear mountain streams in the mountains and you get an idea of how awesome and peaceful a place it is.


Utilities with minimum-depends for use in setup.py
Takes in a string and an email alias mapping and replaces all instances of the aliases in the string with their real email.
Create AUTHORS file using git commits.
Return dict of commands to run from setup.py.
Return a version which is equal to the tag that’s on the current revision if there is one, or tag plus number of additional revisions if the current revision has no tag.
Return a version which is leading up to a version that will be released in the future.
Read the versioninfo file. If it doesn’t exist, we’re in a github zipball, and there’s really no way to know what version we really are, but that should be ok, because the utility of that should be just about nil if this code path is in use in the first place.
Write a changelog based on the git changelog.
Write a simple file containing the version of the package.


Toggle navigation
HeartAid.
Toggle navigation
Heart Disease
Data Analysis
Model Building
Predict
Register
Welcome to join us for your health
Name
Email
Password
Password Confirm
Already have an account?
Login


Before Change
tf.concat([input_tensors[i] for i in inputs_for_unit], axis=1))
if len(lattice_inputs) > 1:
// Stack into (-1, units, lattice_rank) for multi-unit lattice layer
lattice_inputs = tf.stack(lattice_inputs, axis=1)
else:
lattice_inputs = lattice_inputs[0]
output_monotonicity = max(monotonicities)
// Call each lattice layer and store based on output monotonicy.
After Change
for monotonicities, inputs_for_units in self._rtl_structure:
if len(inputs_for_units) == 1:
inputs_for_units = inputs_for_units[0]
lattice_inputs = tf.gather(flattened_input, inputs_for_units, axis=1)
output_monotonicity = max(monotonicities)
// Call each lattice layer and store based on output monotonicy.
outputs_for_monotonicity[output_monotonicity].append(
self._lattice_layers[str(monotonicities)](lattice_inputs))


Home
Offline
About
Recent Posts
Theme Installation and Usage
June 8, 2016
Theme Features
June 7, 2016
YAML Features
April 5, 2016
Content styles
March 5, 2016
Features
Theme Features
June 7, 2016
YAML Features
April 5, 2016
Content styles
March 5, 2016
Readme
Theme Installation and Usage
June 8, 2016


That we really quick.
Here is a quick assessment. Get the details at http://www.onestepup.in/myassessment
Nullam ornare, elit vitae aliquet viverra, augue tellus sollicitudin nibh, vitae accumsan dolor metus in tellus. Cras vel ligula id metus feugiat tincidunt ut vel ipsum.
You also get a 15% discount when you register online with us for a counseling session using the code "APT15" at www.onesteupup.in


--- layout: trainup title: Techniques for scraping data short: Extract data hidden in PDFs, images, and web pages image: trainup.jpg ---
Part of our collection of basic data literacy content
{% include social_sharing.html %}Imagine you get the data you have been waiting for. But the data you have sourced is stored in a format that cannot be opened using spreadsheet software and thus you are unable to process it in any way. What do you do?
The answer is: you scrape it.
Scraping describes the method to extract data hidden in documents – such as web pages, PDFs, or images and make it useable for further processing.
It is among the most useful skills to have if you set out to investigate data – and most of the time it is not especially challenging.
For the most simple methods of scraping you do not even need to know how to write code. And it is these methods that you will learn in this module.
Before we can get started we first need to understand the three basic methods for scraping data:
|
Data scraping |
This describes the extraction of data from documents or images. |
|
Web Scraping |
This describes the extraction of data from websites via the user interface. This form of scraping relies on the coding structure of a website i.e. HTML and XHTML. |
|
Programmatic scraping |
This describes the extraction of data from documents, images or websites, using a scripted computer program. To use this method of scraping, you will need to know how to code. |
The two methods we will address in this module are data scraping and web scraping.
Image by School of Data
It doesn't matter which method of scraping you end up using, you will probably find that there is no guarantee when it comes to accuracy. After scraping, when opening the data in your spreadsheet or other processing tool, the data will need to be cleaned and possibly restructured.
In order to visualise data, we need to analyse it. And we cannot analyse it if it is dirty, data first needs to be cleaned. The first aspect of cleaning data is fixing data capture errors, and other anomalies. However, the second is very often the reshaping of 'scraped' data. Data that exists in table format on websites or documents is often structured quite differently to the way that 'clean' or analysable data would need to be structured.
This is particularly the case with pdf scraping, where a pdf scraper may misinterpret column or even row boundaries. We teach these data cleaning processes in our Source & Clean course.
Data stored in a PDF document is as useful as information written on a piece of paper. There is not much you can do with it until it has been extracted in a way that allows you to open it up in a spreadsheet software where it can be explored and processed.
PDFs present two challenges for data extraction.
Firstly, before we can make an assessment of which tool would best clear the obstacles preventing us from exploring or processing our data, we have to determine the format of the content within our PDF. This content can range between:
Secondly, we need to have a good idea of the different methods and tools readily available for extracting either of these formats.
Scraping from PDFs when the format of the content is either text or tabular data is much easier as the PDF contains content that can be easily identified and scraped.
The tool we would use to extract text or tabular data from a pdf is Tabula.
Tabula is a tool for liberating data tables locked inside PDF files. You can download it from the official website.
Please follow the installation instructions on the website or read the documentation for Tabula.
Usually when starting up Tabula, your web browser will automatically open. If it does not, open your web browser, and go to http://localhost:8080. Tabula works both online and offline.
Tabula runs by establishing a local server environment, which means that in order for it to run a terminal box will open up. The terminal box is the little black screen that pops up when running Tabula on a computer running on a Windows operating system. As long as the terminal is open, the program will be running. If you close this, you will terminate Tabula.
Tabula is really easy to use and make sense of. But before you can begin extracting data from your PDF, you need to upload the document from which you are extracting your data.
After uploading your PDF, using the click-drag method, select the border of the tables you want to extract.
Tabula presents users with two methods for extracting tabulated data from a PDF. The first identifies individual columns based on white spaces while the second identifies columns based on borders or boundary lines. These methods are called "Stream" (whitespace) and "Lattice" (borders).
Make sure you have tested both of them to see which one presents your data the best. Double-check to see if you have selected all the information you wanted. Thereafter, you can export your document. You will be able to export your scraped data into CSV, XLS (excel spreadsheet), and JSON file formats. Once you have selected the format of your choice, click "Export".
Once you have downloaded your document, open it up in a spreadsheet software to verify that everything you require has actually been scraped from the pdf.
Scraping data from a PDF when the format is an image will require an optical character recognition tool. The tool we will use to scrape image content from PDFs and other image formats is OnlineOCR. You will learn about this in the next section.
To extract data from images we need a tool that supports optical character recognition. Optical character recognition can be defined as the process or technology of reading data in printed form by a device (optical character reader) that scans and identifies characters.
The tool we will use to scrape from images or scanned PDFs is OnlineOCR.
Extract text from PDF and images (JPG, BMP, TIFF, GIF) and convert into editable Word, Excel and Text output formats.
There are two methods we will cover for extracting data from the web, one is using the Google Sheets formula ImportHTML, while the other is using a visual scraping tool by Scrapinghub called Portia.
Using the ImportHTML formula, you are able to import data from a table or list within an HTML page.
Google Sheets offers many functionalities to import data, the most common being ImportHTML.
In order to make use of this Google Sheets formula, you require an activated Google account.
For the purposes of this tutorial, we are going to try to import the data for Western Cape province land sales using the following website.
Open a new Google Sheet and enter an equals sign =
to activate the formulas field.
After the equals sign type importhtml
. As with the use of all formulas entered in the formula field, it will be followed by a string of comma separated parameters contained within brackets.
The formula for ImportHTML is: IMPORTHTML(url, query, index)
Each of the parameters requires:
|
URL |
|
|
Query |
The type of object where the data is contained: a table in this case |
|
Index |
Which table are you going to scrape? First table, second table? |
Make sure you house the URL and Query parameters using double quotation marks " "
as in the following example:
Once you have completed the formula with the correct information, hit enter. The data from the website will be imported into your Google Sheet.
Disclaimer: Despite the data being visible in your Google Sheet, it is still attached to the original source.
In order to detach this data from its original source we need to highlight the data. You can highlight the data by clicking in the top left corner of our spreadsheet between A column in 1 row. Once you have clicked this block, a drop down will appear. Select "Copy".
Thereafter, click the same block and then select "Paste" and choose "Paste special: Paste values only". The data you just copied will overwrite the imported data in the same place.
When the data is still attached to the original source, when selecting cell A1 pictured below, the formula field will contain the ImportHTML formula.
Once the data has been detached, if you select the same cell A1, the formula field will now display the the word "District".
For more information on the ImportHTML formula, read the documentation.
The next tool we will explore for scraping data from the web is the visual scraper by Scrapinghub called Portia.
Portia lets you scrape web sites without any programming knowledge required.
Create a template by defining elements on the pages you would like to scrape, and Portia will create a spider to scrape similar pages from the website. There is no need to download or install Portia as it conveniently runs in your web browser.
Visit the documentation for Portia 2.0. Portia is a product of Scrapinghub.
For the purposes of this tutorial, visit the City of Cape Town website pictured below.
Now that you have access to the website you will be scraping, in a different browser tab or window navigate back to the scrapinghub website. In order to proceed, you need to register for a free account. You can sign-up easily using your Google account.
Once you have registered, login to your account. You will be using the Portia scraper to scrape a list of ward councillors from the City of Cape Town's website. On the top menu bar, navigate to the Portia dropdown menu and choose "Create Project".
Once selected a popup titled "Create a new project" will appear. It will prompt you to enter the name of your spider and to select which platform you will use to build the spider. Make sure you have selected "Portia". Thereafter select "Create".
Now that your project has been created, before you can begin you will have to enter in a starting URL.
Navigate to the list of ward councillors on the City of Cape Town website. Copy the URL and paste it into the address bar within Portia as pictured in the example below. Most importantly, once you have pasted it be sure to hit the "enter" key on your keyboard.
After hitting "enter" a window will appear displaying the contents of the URL you have entered in. Make sure the page you are on is the page you would like the spider to begin its crawl. Once you are happy, locate and click the "New Spider" button positioned alongside the address bar wherein you entered the URL.
After creating the new spider you will have to define a sample on which the spider can base its scraping activity. You can do this by selecting the "New sample" button located below the address bar wherein you entered the URL.
Once this has been selected it should automatically create the sample and access the "Edit Sample" area. The "Edit Sample" area consists of a menu bar with the following tools:
You will be selecting the sample areas you would like the spider to scrape. To do so you need to use the magic tool .
With the magic tool selected you are able to annotate your first data sample. Using the magic tool, hover over the first row in the first column in the ward councilor list table. This column is titled "Ward". You need to highlight the first cell below the heading cells. In this example the first cell contains the text: "Ward 1".
Upon hovering, the block's appearance will become highlighted. Click on it when in a highlighted state. As you click on it in this state, in the left toolbar you will be prompted to give this selected block of content a title. Type wardnumber
then hit "enter" on your keyboard.
Because you are scraping a list you will need to make sure that you set the sample to include multiple entries. After naming your block wardnumber
you can hover over the block below your originally selected block in your "Ward" column.
In doing so, Portia will give you the option of selecting multiple entries in this table. We recognise this by acknowledging the appearance of the "add repeat element" icon . Once this appears, you need to click on it.
You know it has been set when the number that appears alongside our "wardnumber" title is changed from 1 to 10. This way we know it acknowledges all 10 entries displayed in the tabulated list on the City of Cape Town webpage. You can check this by locating "Items" on the left panel as pictured below:
Note: Always make it a habit to double-check the quantity of entries displayed on the sample page so that you are sure information you don't want to be excluded is left out during the scraping.
Duplicate your above activity for the next two columns.
Once you are happy with your annotated sample, be sure to click "Close sample" located below the address bar wherein you entered the URL.
Now that you have defined the sample for scraping, you need to make sure that the spider knows what to crawl. The information is stored over a collection of pages that is navigated using the pagination links.
In order for the spider to crawl each of the links, you have to define a "generation URL". You can set this up by clicking on the "START PAGES" settings button located on the left panel.
Then select "Add generation URL".
After selecting "Add generation URL", the URL Generation display box will appear. We make use of this by defining the URL structure to generate multiple URLs based on the structure of the paginated URLs.
In our example, the URL you will use is:
https://www.capetown.gov.za/Family%20and%20home/meet-the-city/city-council/find-your-councillor-ward-or-subcouncil/show-wards?FirstWardSequenceNo=1&LastWardSequenceNo=10
To define the structure, start by first defining the part of the URL that does not change. This would be everything preceding the number "10" in the above URL. But how do you know this? After browsing a few of the paginated links and comparing URLs, this segment remains unchanged, or rather, fixed.
https://www.capetown.gov.za/Family%20and%20home/meet-the-city/city-council/find-your-councillor-ward-or-subcouncil/show-wards?FirstWardSequenceNo=1&LastWardSequenceNo=
Now that you have added the "Fixed" fragment of your URL, in order to finish constructing the URL, you have to add an additional fragment.
The next part of the URL to define is the fragment that does change from link to link.
The pagination follows a numbering system whose count ascends at intervals of 10 each time. For example: 1-10 | 11-20 | 21-30 etc.
After browsing a few of the paginated links, you know that the pagination begins at 0-10
and ends at >100
, and that 10 to 20 to 30 is not consecutive numbering i.e. it is not 10, 11, 12 etc. it jumps at intervals of 10.
As a result of this, you cannot select the "Range" option for the next segment of the URL but will have to use the "List" function. The "Range" option only works for consecutive numbering or lettering i.e. if the pagination uses the alphabet for navigation.
The list function works by defining individual values to be added onto the end of the fixed fragment of the URL. The list function's structure requires that each value is separated by a single blank space, and should be written as follows:
Value1 Value2 Value3
For the example, your values would be:
10 20 30 40 50 60 70 80 90 100
After checking what the URL looks like for the >100
page, we notice that it does not have a number but rather is a blank space. So I will add an additional blank space after the 100 value in my list so I can accommodate for the final >100
page to be included in the scraping process.
Be sure to browse the box titled "Generation List" to assess whether a complete list of paginated URLs has been generated.
Next you will have to remove the original start page while retaining the newly added start page you have just created through the "Generation URL" function.
You can achieve this by locating "START PAGES" in the left panel. Next to the original start page, titled by its URL, will be a red circular disc with a negative -
sign in it. Click on this red disc to remove the start URL item you no longer need.
Now that you have defined the URLs that need to be scraped and you have removed the original start page URL, you must deactivate any automated link crawling. You do not need this because you have already clearly defined every URL your spider has to crawl to extract the data you are scraping. In order to deactivate any automated link crawling, locate "LINK CRAWLING" on the left panel. Expand the dropdown menu and choose the setting: "Don't follow links".
Once you have set the spider to not follow links be sure to click the project menu (project currently titled "Ward Councillors") on the left panel and select "Publish Project".
You will not be able to run the spider if the project has not been published. After the project has been published, navigate to the top bar and select "Scrapinghub Dashboard".
On the far right there will be a green button titled "Run". Select "Run" and wait for a popup to appear. Under the heading "Spiders" select the spider you just created. Once you are happy, click "Run".
The popup will disappear once you have clicked "Run" and a new job will be added to your dashboard. You can track the progress for the new job under the heading "Running Jobs".
You can access the scraped data by clicking on the hyperlinked number appearing below "Items" located as a heading under the "Running Jobs" panel. Here you can access all job-related details, progress, and settings pertaining to your spiders scraping activity.
Once the job has completed, you can navigate to the far right of the screen and download your data in CSV, JSON or XML formats by selecting the green button labelled "Export".
Thereafter you can open it up in the spreadsheet software you will be using to process your data.
This curriculum has been developed by OpenUp in collaboration with School of Data.
That's great to hear! We want to make it even better and could really use your feedback.
How will you apply what you learned?
You are free to use, share, and adapt this content to your needs. Do you want to teach others? Let us know how we can help.
We're sorry to hear that.
Please let us know how we can improve.


BZIP2 compression. Looks like a fork from Apache CBZip2InputStream, with minimal code (eg decode only)
@see "https://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/bzip2/"
@see "https://code.google.com/p/jbzip2/"
@see "http://en.wikipedia.org/wiki/Bzip"