Fully Worked Example¶
If you haven’t done so already, please look at the Minimal Example, because we’re going to start from where that left off, just on a different site.
For this tutorial, let’s look at The Journal of Biomedical Science. Start Looking at several documents on the site. It looks like all of the articles have similar formatting, so we don’t have to worry too much about giving specialized instructions to different types of articles. Let’s start looking in-depth at an article that has as much information as possible:
https://jbiomedsci.biomedcentral.com/articles/10.1186/s12929-020-00652-z
And using the scaffolding from the minimal example:
import centaurminer as mining
class JBSLocations(mining.PageLocations):
source = "The Journal of Biomedical Science"
license = "https://www.biomedcentral.com/about/open-access"
class JBSEngine(mining.MiningEngine):
pass
miner = JBSEngine(JBSLocations)
miner.gather("https://jbiomedsci.biomedcentral.com/articles/10.1186/s12929-020-00652-z")
print(miner.results)
I’ve added both JBSLocations.source
and JBSLocations.license
attributes, which get stored directly in our results, since they’re strings. I strongly recommend you find the licensing first, since this will help you know for sure that you can access and share the data according to the site’s policy. Here is what we get so far:
[WDM] - Current google-chrome version is 83.0.4103
[WDM] - Get LATEST driver version for 83.0.4103
[WDM] - Driver [C:\Users\simon\.wdm\drivers\chromedriver\win32\83.0.4103.39\chromedriver.exe] found in cache
DevTools listening on ws://<IP_ADDRESS>:64843/devtools/browser/3c82a3a7-b330-4a7c-96ee-cb70709a8013
Unable to find element defined by JBSLocations.date_publication
{
'abstract': 'Monoclonal antibodies (mAbs) ... satisfy the unmet medical needs of mAb therapy.',
'authors': '<item>Wen-Wei Lin</item><item>Yun-Chi Lu</item><item>Chih-Hung Chuang</item><item>Tian-Lu Cheng</item>',
'doi': '10.1186/s12929-020-00652-z',
'license': 'https://www.biomedcentral.com/about/open-access',
'source': 'The Journal of Biomedical Science',
'title': 'Ab locks for improving the selectivity and safety of antibody drugs',
'url': 'https://jbiomedsci.biomedcentral.com/articles/10.1186/s12929-020-00652-z',
'date_aquisition': '2020-07-02'
}
Compare this to what’s shown on the page - so far, it’s all looking good. It’s missing the publication date, but that’s alright, we can get it now. At the same time, let’s look through the metadata for more data. It looks like we have publication date, a pdf link, and references in the metadata. Publication date and the pdf link are simple, but there are many references - we need to tell the program how to gather all of these and combine them into results. Let’s do all of this at once:
import centaurminer as mining
class JBSLocations(mining.PageLocations):
source = "The Journal of Biomedical Science"
license = "https://www.biomedcentral.com/about/open-access"
date_publication = mining.MetaData("citation_online_date")
link_pdf = mining.MetaData("citation_pdf_url")
references = mining.MetaData("citation_reference")
class JBSEngine(mining.MiningEngine):
def get_references(self, element):
return mining.TagList(self.get(element, several=True))
miner = JBSEngine(JBSLocations)
miner.gather("https://jbiomedsci.biomedcentral.com/articles/10.1186/s12929-020-00652-z")
print(miner.results)
The several
attribute here tells the engine we want all elements that match the description, and TagList
combines them into a string. Now running this again, we have three added keys to our results dictionary:
{
...,
'date_publication': '2020/06/25',
'link_pdf': 'https://jbiomedsci.biomedcentral.com/track/pdf/10.1186/s12929-020-00652-z'
'references': '<item>citation_journal_title=Cancer Lett; ... over 100 references here ... citation_id=CR198</item>'
}
Alright, let’s move on to the main body of the page! Of the information presented here, let’s focus on the keywords, affiliated organization, and full body text.
Keywords:¶
All the way at the bottom of the page, you can see a group of keywords that may be useful. These are all located in divs with class="c-article-subject-list__subject"
- there’s no other text in these elements, which makes it really easy to collect - we just have to collect each element, and treat it identically to how we collected the references!
class JBSLocations(mining.PageLocations):
source = "The Journal of Biomedical Science"
license = "https://www.biomedcentral.com/about/open-access"
date_publication = mining.MetaData("citation_online_date")
link_pdf = mining.MetaData("citation_pdf_url")
references = mining.MetaData("citation_reference")
keywords = mining.Element("css_selector", ".c-article-subject-list__subject")
class JBSEngine(mining.MiningEngine):
def get_references(self, element):
return mining.TagList(self.get(element, several=True))
def get_keywords(self, element):
return mining.TagList(self.get(element, several=True))
This adds the following key-value pair to our results:
'keywords': '<item>Monoclonal antibody (mAb)</item><item>adverse events</item><item>Ab lock</item><item>spatial-hindrance-based approaches</item><item>affinity-based approaches</item>'
Full-body:¶
We have a few struggles here, and a few choices to make. What should we do with figures? How should we treat headers and titles? For now, we’re going to ignore figures and tables, and treat headers/titles as their own paragraph. The main struggle with this approach is that we need to throw out lots of information we don’t want - figures and captions, the abstract section, and all of the subsections after the conclusions (like Abbreviations). We can do most of this directly with a css selector, but specific selector means more brittle, easily breakable code - just keep that in mind. Here is the selector I came up with, but others are possible:
section > [id^=Sec][id$=section] > .c-article-section__content > p, [id^=Sec]:not([id$=content]):not([id$=section])
This accepts two types of elements (separated by a comma, meaning “or”) - the first is the main text in the body, and the second is all of the section headers. The body text needs 4 levels of content to find it accurately, and both ids and classes are used in the selector. The section headers have a very specific signature, so they’re just located by their attributes. The “^=” and “$=” are “starts_with” and “ends_with” css selectors, which we use for both types. Here’s what it looks like in the code (along with the extra processing in the BMCEngine
class)
class JBSLocations(mining.PageLocations):
source = "The Journal of Biomedical Science"
license = "https://www.biomedcentral.com/about/open-access"
date_publication = mining.MetaData("citation_online_date")
link_pdf = mining.MetaData("citation_pdf_url")
references = mining.MetaData("citation_reference")
keywords = mining.Element("css_selector", ".c-article-subject-list__subject")
body = mining.Element("css_selector", "section > [id^=Sec][id$=section] > .c-article-section__content > p, [id^=Sec]:not([id$=content]):not([id$=section])")
class JBSEngine(mining.MiningEngine):
def get_references(self, element):
return mining.TagList(self.get(element, several=True))
def get_keywords(self, element):
return mining.TagList(self.get(element, several=True))
def get_body(self, element):
paragraphs = self.get(element, several=True)
return "\n".join(paragraphs)
I would show you here what gets added to results, but it’s just the full body text, which is too long to be helpful.
Affiliated organization:¶
Toward the bottom of the page, you can see the list of author affiliations - it’s a list of authors and organizations, alternating back and forth. Because of this, and because we want to combine both of these types of elements into one result, we need to use a centaurminer.Complex
class instead of an element. When we combine the authors + orgs, let’s organize it like this: <item><author> ... </author><org> ... </org></item> ...
. Here’s my solution:
# In JBSLocations
organization_affiliated = mining.Complex()
# In JBSEngine
def get_organization_affiliated(self, element):
orgs_loc = mining.Element("css_selector", ".c-article-author-affiliation__address")
authors_loc = mining.Element("css_selector", ".c-article-author-affiliation__authors-list")
orgs = self.get(orgs_loc, several=True)
authors = self.get(authors_loc, several=True)
result = []
for org, author_list in zip(orgs, authors):
res_string = "<authors>" + author_list + "</authors>"
res_string += "<org>" + org + "</org>"
result.append(res_string)
return mining.TagList(result)
I extract the affiliations and authors separately, zip them together (to get them to alternate), and tag the inner elements manually.
That’s it for this page! You can extract whatever elements you want, but we’ve already got 238 kb of data, as of writing this example. Now it’s up to your goals and time you can spend, to extract information from whatever site you want!
Remember, if you’re stuck on some piece of information, you can always manipulate the selenium webdriver itself at MiningEngine.wd
.
Happy mining!


Annotation Editor
The
Annotations Editor
enables you to add
Text
and
Freehand Annotations
to your score.
Check out the help section in the
Measure Editor
to get started.


DGST
Section: OpenSSL (1)Updated: 2009-04-10
Index Return to Main Contents
NAME
dgst, md5, md4, md2, sha1, sha, mdc2, ripemd160 - message digestsSYNOPSIS
openssl dgst [-md5|-md4|-md2|-sha1|-sha|-mdc2|-ripemd160|-dss1] [-c] [-d] [-hex] [-binary] [-out filename] [-sign filename] [-keyform arg] [-passin arg] [-verify filename] [-prverify filename] [-signature filename] [-hmac key] [file...][md5|md4|md2|sha1|sha|mdc2|ripemd160] [-c] [-d] [file...]
DESCRIPTION
The digest functions output the message digest of a supplied file or files in hexadecimal form. They can also be used for digital signing and verification.OPTIONS
- -c
- print out the digest in two digit groups separated by colons, only relevant if hex format output is used.
- -d
- print out BIO debugging information.
- -hex
- digest is to be output as a hex dump. This is the default case for a ``normal'' digest as opposed to a digital signature.
- -binary
- output the digest or signature in binary form.
- -out filename
- filename to output to, or standard output by default.
- -sign filename
- digitally sign the digest using the private key in ``filename''.
- -keyform arg
- Specifies the key format to sign digest with. Only PEM and ENGINE formats are supported by the dgst command.
- -engine id
- Use engine id for operations (including private key storage). This engine is not used as source for digest algorithms, unless it is also specified in the configuration file.
- -sigopt nm:v
- Pass options to the signature algorithm during sign or verify operations. Names and values of these options are algorithm-specific.
- -passin arg
- the private key password source. For more information about the format of arg see the PASS PHRASE ARGUMENTS section in openssl(1).
- -verify filename
- verify the signature using the the public key in ``filename''. The output is either ``Verification OK'' or ``Verification Failure''.
- -prverify filename
- verify the signature using the the private key in ``filename''.
- -signature filename
- the actual signature to verify.
- -hmac key
- create a hashed MAC using ``key''.
- -mac alg
- create MAC (keyed Message Authentication Code). The most popular MAC algorithm is HMAC (hash-based MAC), but there are other MAC algorithms which are not based on hash, for instance gost-mac algorithm, supported by ccgost engine. MAC keys and other options should be set via -macopt parameter.
- -macopt nm:v
-
Passes options to MAC algorithm, specified by -mac key.
Following options are supported by both by HMAC and gost-mac:
-
- key:string
- Specifies MAC key as alphnumeric string (use if key contain printable characters only). String length must conform to any restrictions of the MAC algorithm for example exactly 32 chars for gost-mac.
- hexkey:string
- Specifies MAC key in hexadecimal form (two hex digits per byte). Key length must conform to any restrictions of the MAC algorithm for example exactly 32 chars for gost-mac.
-
- -rand file(s)
- a file or files containing random data used to seed the random number generator, or an EGD socket (see RAND_egd(3)). Multiple files can be specified separated by a OS-dependent character. The separator is ; for MS-Windows, , for OpenVMS, and : for all others.
- file...
- file or files to digest. If no files are specified then standard input is used.
NOTES
The digest of choice for all new applications is SHA1. Other digests are however still widely used.If you wish to sign or verify data using the DSA algorithm then the dss1 digest must be used.
A source of random numbers is required for certain signing algorithms, in particular DSA.
The signing and verify options should only be used if a single file is being signed or verified.
Index
This document was created by man2html, using the manual pages.
Time: 05:29:03 GMT, December 24, 2015


Scientists find evidence of missing neutron star
Astronomers uncover the relic at the heart of Supernova 1987A that has been hidden for over 30 years
| Source: | Cardiff University |
| Summary: | The leftovers from a spectacular supernova that revolutionized our understanding of how stars end their lives have finally been spotted by astronomers. |
The leftovers from a spectacular supernova that revolutionised our understanding of how stars end their lives have finally been spotted by astronomers at Cardiff University.
The scientists claim to have found evidence of the location of a neutron star that was left behind when a massive star ended its life in a gigantic explosion, leading to a famous supernova dubbed Supernova 1987A.
For more than 30 years astronomers have been unable to locate the neutron star — the collapsed leftover core of the giant star — as it has been concealed by a thick cloud of cosmic dust.
Using extremely sharp and sensitive images taken with the Atacama Large Millimeter/submillimeter Array (ALMA) telescope in the Atacama Desert of northern Chile, the team have found a particular patch of the dust cloud that is brighter than its surroundings, and which matches the suspected location of the neutron star.
The findings have been published in The Astrophysical Journal.
Lead author of the study Dr Phil Cigan, from Cardiff University's School of Physics and Astronomy, said: "For the very first time we can tell that there is a neutron star inside this cloud within the supernova remnant. Its light has been veiled by a very thick cloud of dust, blocking the direct light from the neutron star at many wavelengths like fog masking a spotlight."
Dr Mikako Matsuura, another leading member of the study, added: "Although the light from the neutron star is absorbed by the dust cloud that surrounds it, this in turn makes the cloud shine in sub-millimetre light, which we can now see with the extremely sensitive ALMA telescope."
Supernova 1987A was first spotted by astronomers on Feb 23, 1987, when it blazed in the night sky with the power of 100 million suns, and continuing to shine brightly for several months.
The supernova was discovered in a neighbouring galaxy, the Large Magellanic Cloud, only 160,000 light years away.
It was the nearest supernova explosion observed in over 400 years and, since its discovery, has continued to fascinate astronomers who have been presented with the perfect opportunity to study the phases before, during, and after the death of a star.
The supernova explosion that took place at the end of this star's life resulted in huge amounts of gas with a temperature of over a million degrees, but as the gas began to cool down quickly below zero degrees centigrade, some of the gas transformed into a solid, i.e. dust.
The presence of this thick cloud of dust has long been the main explanation as to why the missing neutron star has not been observed, but many astronomers were sceptical about this and began to question whether their understanding of a star's life was correct.
"Our new findings will now enable astronomers to better understand how massive stars end their lives, leaving behind these extremely dense neutron stars," continued Dr Matsuura.
"We are confident that this neutron star exists behind the cloud and that we know its precise location. Perhaps when the dust cloud begins to clear up in the future, astronomers will be able to directly see the neutron star for the very first time."
make a difference: sponsored opportunity
Story Source:
Materials provided by Cardiff University.
Note: Content may be edited for style and length.
Journal Reference:
- Phil Cigan, Mikako Matsuura, Haley L. Gomez, Remy Indebetouw, Fran Abellán, Michael Gabler, Anita Richards, Dennis Alp, Timothy A. Davis, Hans-Thomas Janka, Jason Spyromilio, M. J. Barlow, David Burrows, Eli Dwek, Claes Fransson, Bryan Gaensler, Josefin Larsson, P. Bouchet, Peter Lundqvist, J. M. Marcaide, C.-Y. Ng, Sangwook Park, Pat Roche, Jacco Th. van Loon, J. C. Wheeler, Giovanna Zanardo. High Angular Resolution ALMA Images of Dust and Molecules in the SN 1987A Ejecta. The Astrophysical Journal, 2019; 886 (1): 51 DOI: 10.3847/1538-4357/ab4b46
Cite This Page:
| MLA | Cardiff University. "Scientists find evidence of missing neutron star: Astronomers uncover the relic at the heart of Supernova 1987A that has been hidden for over 30 years." ScienceDaily. ScienceDaily, 19 November 2019. <www.sciencedaily.com/releases/2019/11/191119105545.htm>. |
| APA | Cardiff University. (2019, November 19). Scientists find evidence of missing neutron star: Astronomers uncover the relic at the heart of Supernova 1987A that has been hidden for over 30 years. ScienceDaily. Retrieved November 20, 2019 from www.sciencedaily.com/releases/2019/11/191119105545.htm |
| Chicago | Cardiff University. "Scientists find evidence of missing neutron star: Astronomers uncover the relic at the heart of Supernova 1987A that has been hidden for over 30 years." ScienceDaily. www.sciencedaily.com/releases/2019/11/191119105545.htm (accessed November 20, 2019). |


Hi there. 👋 🙏 How's it going?
I am a service designer, UX designer, researcher, and a teacher, most passionate about using
design to help people, understanding why things are the way that they are, and creating new
perspectives for exploring the world around us. My core areas of interest are healthcare, education,
and mobility.
When not consciously designing (but always exploring and researching), you will most likely
find me cycling on my favorite trail, digging into a new book, or deep down a rabbit hole in the
world of javascript.
Currently, I am reading about knowledge structures in different cultures, exploring virtual
reality through A-Frame, and backend development using Node.
Do reach out for more information or collaborations at<EMAIL_ADDRESS>or on LinkedIn @sujitjoshi00.
In collaboration with our client's Innovation and User Experience teams while interning for Panorma Innovation
Researcher • Service Designer
As part of a team of six at a design-hackathon hosted by MIT Media Lab and UCSD Design Lab
Researcher • Service Designer
In collaboration with Dr. Eeshani Bendale for my Master's thesis project at School of the Art Institute of Chicago
Researcher • Futurist
In collaboration with our client's research team while interning for Panorama Innovation
Researcher
In collaboration with Aimee Martinez for an alumni exhibition at School of the Art Institute of Chicago
Researcher • Service Designer • User Experience Designer
Commissioned by Bosch and School of the Art Institute of Chicago for South by Southwest 2020
Technologist