User manual¶
Modules¶
openiti.helper¶
openiti.helper.ara¶
-
openiti.helper.ara.
ar_ch_cnt
(text)¶ Count the number of Arabic characters in a string
Parameters: text – text Returns: number of the Arabic characters in the text Examples
>>> a = "ابجد ابجد اَبًجٌدُ" >>> ar_ch_cnt(a) 16
-
openiti.helper.ara.
ar_cnt_file
(fp, mode='token')¶ Count the number of Arabic characters/tokens in a text, given its pth
Parameters: - fp (str) – url / path to a file
- mode (str) – either “char” for count of Arabic characters, or “token” for count of Arabic tokens
Returns: Arabic character/token count
Return type: (int)
-
openiti.helper.ara.
ar_tok_cnt
(text)¶ Count the number of Arabic tokens in a string
Parameters: text – text Returns: number of Arabic tokens in the text Examples
>>> a = "ابجد ابجد اَبًجٌدُ" >>> ar_tok_cnt(a) 3
-
openiti.helper.ara.
deNoise
(text)¶ Remove non-consonantal characters from Arabic text.
Examples
>>> denoise("وَالَّذِينَ يُؤْمِنُونَ بِمَا أُنْزِلَ إِلَيْكَ وَمَا أُنْزِلَ مِنْ قَبْلِكَ وَبِالْآخِرَةِ هُمْ يُوقِنُونَ") 'والذين يؤمنون بما أنزل إليك وما أنزل من قبلك وبالآخرة هم يوقنون' >>> denoise(" ْ ً ٌ ٍ َ ُ ِ ّ ۡ ࣰ ࣱ ࣲ ٰ ") ' '
-
openiti.helper.ara.
denoise
(text)¶ Remove non-consonantal characters from Arabic text.
Examples
>>> denoise("وَالَّذِينَ يُؤْمِنُونَ بِمَا أُنْزِلَ إِلَيْكَ وَمَا أُنْزِلَ مِنْ قَبْلِكَ وَبِالْآخِرَةِ هُمْ يُوقِنُونَ") 'والذين يؤمنون بما أنزل إليك وما أنزل من قبلك وبالآخرة هم يوقنون' >>> denoise(" ْ ً ٌ ٍ َ ُ ِ ّ ۡ ࣰ ࣱ ࣲ ٰ ") ' '
-
openiti.helper.ara.
denormalize
(text)¶ Replace complex characters with a regex covering all variants.
Examples
>>> denormalize("يحيى") 'يحي[يى]' >>> denormalize("هوية") 'هوي[هة]' >>> denormalize("مقرئ") 'مقر(?:[ؤئ]|[وي]ء)' >>> denormalize("فيء") 'في(?:[ؤئ]|[وي]ء)'
-
openiti.helper.ara.
normalize
(text, replacement_tuples=[])¶ Normalize Arabic text by replacing complex characters by simple ones.
Parameters: - text (str) – the string that needs to be normalized
- replacement_tuples (list of tuple pairs) – (character, replacement)
Examples
>>> normalize('AlphaBet', [("A", "a"), ("B", "b")]) 'alphabet'
-
openiti.helper.ara.
normalize_ara_heavy
(text)¶ Normalize Arabic text by simplifying complex characters: alifs, alif maqsura, hamzas, ta marbutas
Examples
>>> normalize_ara_heavy("ألف الف إلف آلف ٱلف") 'الف الف الف الف الف' >>> normalize_ara_heavy("يحيى") 'يحيي' >>> normalize_ara_heavy("مقرئ فيء") 'مقر في' >>> normalize_ara_heavy("قهوة") 'قهوه'
-
openiti.helper.ara.
normalize_ara_light
(text)¶ Lighlty normalize Arabic strings: fixing only Alifs, Alif Maqsuras; replacing hamzas on carriers with standalone hamzas
Examples
>>> normalize_ara_light("ألف الف إلف آلف ٱلف") 'الف الف الف الف الف' >>> normalize_ara_light("يحيى") 'يحيي' >>> normalize_ara_light("مقرئ فيء") 'مقرء فء' >>> normalize_ara_light("قهوة") 'قهوة'
-
openiti.helper.ara.
normalize_composites
(text, method='NFKC')¶ Normalize composite characters and ligatures
Ligatures like “Allah” will be broken into their components, while combining characters like alif + hamza above will be joined into one character.
Parameters: - text (str) – the string to be normalized
- method (str) – the unicode method to be used for normalization (see https://docs.python.org/3.5/library/unicodedata.html). Default: NFKC, which is most suited for Arabic.
Examples
>>> len("ﷲ") # U+FDF2: ARABIC LIGATURE ALLAH ISOLATED FORM 1 >>> len(normalize_composites("ﷲ")) 4 >>> [char for char in normalize_composites("ﷲ")] ['ا', 'ل', 'ل', 'ه']
>>> len("ﻹ") # UFEF9: ARABIC LIGATURE LAM WITH ALEF WITH HAMZA BELOW ISOLATED FORM 1 >>> len(normalize_composites("ﻹ")) 2
alif+hamza written with 2 unicode characters: U+0627 (ARABIC LETTER ALEF) + U+0654 (ARABIC HAMZA ABOVE):
>>> a = "أ" >>> len(a) 2 >>> len(normalize_composites(a)) 1
openiti.helper.funcs¶
-
openiti.helper.funcs.
get_all_characters_in_folder
(start_folder, verbose=False, exclude_folders=[], exclude_files=[])¶ Get a set of all characters used in all OpenITI text files in a folder and its subfolders.
Parameters: - start_folder (str) – path to the root directory. All files and folders in it, except if they are in the exclude lists, will be processed.
- verbose (bool) – if True, filenames and current number of characters in the set will be printed.
- exclude_folders (list) – list of folder names to be excluded from the process.
- exclude_folders – list of file names to be excluded.
Returns: a set of all characters used in the folder.
Return type: (set)
-
openiti.helper.funcs.
get_all_characters_in_text
(fp)¶ Get a set of all characters used in a text.
Parameters: fp (str) – path to a text file. Returns: a set of all characters used in the text. Return type: (set)
-
openiti.helper.funcs.
get_character_names
(characters, verbose=False)¶ Print the unicode name of a list/set/string of characters.
Parameters: - characters (list/set/string) – a list, string or set of characters.
- verbose (bool) – if set to True, the output will be printed
Returns: a dictionary of characters and their names.
Return type: (dict)
Examples
>>> char_dict = {"١": "ARABIC-INDIC DIGIT ONE", "٢": "ARABIC-INDIC DIGIT TWO"} >>> char_dict == get_character_names("١٢") True >>> char_dict == get_character_names(["١", "٢"]) True >>> char_dict == get_character_names({"١", "٢"}) True
-
openiti.helper.funcs.
read_header
(fp)¶ Read only the OpenITI header of a file without opening the entire file.
Parameters: fp (str) – path to the text file Returns: A list of all metadata lines in the header Return type: (list)
-
openiti.helper.funcs.
text_cleaner
(text)¶ Clean text by normalizing Arabic characters and removing all non-Arabic characters
Parameters: text (str) – the string to be cleaned Returns: the cleaned string Return type: (str)
openiti.helper.templates¶
Templates for OpenITI yml files, readme files, etc.
Templates:
- MAGIC_VALUE
- HEADER_SPLITTER
- HTML_HEADER
- HTML_FOOTER
- author_yml_template
- book_yml_template
- version_yml_template
- readme_template
- text_questionnaire_template
openiti.helper.uri¶
Classes and functions to work with OpenITI URIs
Todo
- make the print output look nicer
- reflow texts + insert milestones?
- compare with Maxim’s script: _add_new_text_from_folder.py in maintenance repo
The Module contains a URI class that represents an OpenITI URI as an object. The URI class’s methods allow
- Checking whether all components of the URI are valid
- Accessing and changing components of the URI
- Getting the URI’s current uri_type (“author”, “book”, “version”, None)
- Building different versions of the URI (“author”, “book”, etc.)
- Building paths based on the URI
In addition to the URI class, the module contains a number of functions for implementing URI changes in the OpenITI corpus
Examples
The Module contains a URI class that represents an OpenITI URI as an object. By calling the URI class, you create an instance of the URI class
>>> from uri import URI
>>> instance1 = URI("0255Jahiz.Bayan")
>>> instance2 = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed")
The URI instance inherits all the URI class’s methods and properties:
- Making string representations of a URI instance: print() and repr():
>>> t = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed")
>>> print(repr(t))
uri(date:0255, author:Jahiz, title:Hayawan, version:Sham19Y0023775, language:ara, edition_no:1, extension:completed)
>>> print(t)
0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed
- Accessing components of the URI:
>>> t.author
'Jahiz'
>>> t.date
'0255'
- Changing components of the URI by setting properties to a new value:
>>> u = URI("0255Jahiz.Hayawan")
>>> u.author = "JahizBasri"
>>> print(u)
0255JahizBasri.Hayawan
Validity tests: setting invalid values for part of a uri returns an error (implemented for instantiation of URI objects + setting URI components):
# >>> URI("255Jahiz") Exception: Date Error: URI must start with a date of 4 digits (255 has 3!) # >>> URI("0255Jāḥiẓ") Exception: Author name Error: Author name (Jāḥiẓ) should not contain digits or non-ASCII characters(culprits: ['ā', 'ḥ', 'ẓ']) # >>> t.author = "Jāḥiẓ" Exception: Author name Error: Author name (Jāḥiẓ) should not contain digits or non-ASCII characters(culprits: ['ā', 'ḥ', 'ẓ']) # >>> t.author = "0255Jahiz" Exception: Author name Error: Author name (0255Jahiz) should not contain digits or non-ASCII characters(culprits: ['0', '2', '5', '5']) # >>> URI("0255Jahiz.Al-Hayawan") Exception: Book title Error: Book title (Al-Hayawan) should not contain non-ASCII characters(culprits: ['-']) # >>> URI("0255Jahiz.Hayawan.Shāmila00123545-ara1") Exception: Version string Error: Version string (Shāmila00123545) should not contain non-ASCII characters(culprits: ['ā']) # >>> URI("0255Jahiz.Hayawan.Shamela00123545-arab1") Exception: Language code (arab) should be an ISO 639-2 language code, consisting of 3 characters # >>> t.extension = "markdown" Exception: Extension (markdown) is not among the allowed extensions (['inProgress', 'completed', 'mARkdown', 'yml', '']) # >>> URI("0255Jahiz.Hayawan.Shamela00123545-ara1.markdown") Exception: Extension (markdown) is not among the allowed extensions (['inProgress', 'completed', 'mARkdown', 'yml', ''])
Getting the URI’s current uri_type (“author”, “book”, “version”, None), i.e., the longest URI that can be built from the object’s components:
>>> t.uri_type
'version'
>>> t.language = ""
>>> t.uri_type
'book'
>>> t.date = ""
>>> t.uri_type == None
True
- Building different versions of the URI (uri_types: “author”, “author_yml”, “book”, “book_yml”, “version”, “version_yml”, “version_file”):
>>> t = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed")
>>> t.build_uri(uri_type="author")
'0255Jahiz'
>>> t.build_uri("book")
'0255Jahiz.Hayawan'
>>> t.build_uri("book_yml")
'0255Jahiz.Hayawan.yml'
>>> t.build_uri("version")
'0255Jahiz.Hayawan.Sham19Y0023775-ara1'
>>> t.build_uri("version_file")
'0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed'
Simply calling the URI object (i.e., writing parentheses after the variable name works as an alias for the build_uri function:
>>> t = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed")
>>> t(uri_type="author")
'0255Jahiz'
>>> t("book")
'0255Jahiz.Hayawan'
>>> t("book_yml")
'0255Jahiz.Hayawan.yml'
>>> t("version")
'0255Jahiz.Hayawan.Sham19Y0023775-ara1'
>>> t("version_file")
'0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed'
- Building paths based on the URI:
>>> t.build_pth(uri_type="version", base_pth="D:\test")
'D:/test/0275AH/data/0255Jahiz/0255Jahiz.Hayawan'
>>> t.build_pth(uri_type="version_file", base_pth="D:\test")
'D:/test/0275AH/data/0255Jahiz/0255Jahiz.Hayawan/0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed'
>>> t.build_pth("version")
'./0275AH/data/0255Jahiz/0255Jahiz.Hayawan'
>>> t.build_pth(uri_type="book_yml")
'./0275AH/data/0255Jahiz/0255Jahiz.Hayawan/0255Jahiz.Hayawan.yml'
Without uri_type argument, build_pth() builds the fullest path it can:
>>> t.build_pth()
'./0275AH/data/0255Jahiz/0255Jahiz.Hayawan/0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed'
>>> t.language="" # (removing the language property makes it impossible to build a version uri)
>>> t.build_pth()
'./0275AH/data/0255Jahiz/0255Jahiz.Hayawan'
NB: by default, build_pth() takes the OpenITI folder structure into account, in which authors are grouped in 25-year batches by their death date. If you do not want to use this feature, set the URI class’s data_in_25_year_repos attribute to False:
>>> t.build_pth()
'./0275AH/data/0255Jahiz/0255Jahiz.Hayawan'
>>> t.data_in_25_year_repos = False
>>> t.build_pth()
'./0255Jahiz/0255Jahiz.Hayawan'
>>> t.data_in_25_year_repos = True
>>> URI.data_in_25_year_repos = False
>>> u = URI("0255Jahiz.Hayawan")
>>> u.build_pth()
'./0255Jahiz/0255Jahiz.Hayawan'
>>> URI.data_in_25_year_repos = True
>>> u.build_pth()
'./0275AH/data/0255Jahiz/0255Jahiz.Hayawan'
In addition to the URI class, the module contains a function for implementing URI changes in the OpenITI corpus: change_uri.
The function has an execute flag. If set to False, the function will not immediately be executed but first show all changes it will make, and then ask the user whether to carry out the changes or not.
If a version URI changes:
- new author and book folders are made if necessary (including the creation of new author and book yml files)
- all text files related that version should be moved
- the yml file of that version should be updated and moved
If a book uri changes:
- new author and book folders are made if necessary
- the yml file of that book should be updated and moved
- all text files of all versions of the book should be moved
- all yml files of versions of that book should be updated and moved
- the original book folder itself should be (re)moved
if an author uri changes:
- new author and book folders are made if necessary
- the yml file of that author should be updated and moved
- all book yml files of that should be updated and moved
- all annotation text files of all versions of all books should be moved
- all yml files of versions of all books should be updated and moved
- the original book folders should be (re)moved
- the original author folder itself should be (re)moved
Examples:
change_uri("0255Jahiz", "0256Jahiz")
change_uri("0255Jahiz", "0255JahizBasri")
change_uri("0255Jahiz.Hayawan", "0255Jahiz.KitabHayawan")
change_uri("0255Jahiz.Hayawan.Shamela002526-ara1",
"0255Jahiz.Hayawan.Shamela002526-ara2")
change_uri("0255Jahiz.Hayawan.Shamela002526-ara1.completed",
"0255Jahiz.Hayawan.Shamela002526-ara1.mARkdown")
-
class
openiti.helper.uri.
URI
(uri_string=None)¶ A class that represents the OpenITI URI as a Python object.
OpenITI URIs consist of the following elements: 0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1.mARkdown
- VersionURI: consists of
- EditionURI: consists of
- Work URI: consists of
- AuthorID: consists of
- author’s death date (self.date): 0768
- shuhra of the author (self.author): IbnMuhammadTaqiDinBaclabakki
- BookID (self.title): Hadith: short title of the book
- AuthorID: consists of
- Version URI: consists of
- VersionID (self.version): Shamela0009426: ID of the collection/contributor from which we got the book + number of the book in that collection
- Lang:
- self.language: ara: ISO 639-2 language code
- self.edition_no: 1: edition version number (different digitizations of the same edition get the same edition_no)
- Work URI: consists of
- EditionURI: consists of
- self.extension = mARkdown (can be inProgress, mARkdown, completed, “”)
Examples
>>> from uri import URI >>> t = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed")
Representations of a URI object: print() and repr():
>>> print(repr(t)) uri(date:0255, author:Jahiz, title:Hayawan, version:Sham19Y0023775, language:ara, edition_no:1, extension:completed) >>> print(t) 0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed
Accessing components of the URI:
>>> t.author 'Jahiz' >>> t.date '0255'
Getting URI’s current uri_type (“author”, “book”, “version”, None), i.e., the longest URI that can be built from the object’s components:
>>> t.uri_type 'version' >>> t.language = "" >>> t.uri_type 'book' >>> t.date = "" >>> t.uri_type == None True
Building different versions of the URI (uri_types: “author”, “author_yml”, “book”, “book_yml”, “version”, “version_yml”, “version_file”):
>>> t = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed") >>> t.build_uri(uri_type="author") '0255Jahiz' >>> t.build_uri("book") '0255Jahiz.Hayawan' >>> t.build_uri("book_yml") '0255Jahiz.Hayawan.yml' >>> t.build_uri("version") '0255Jahiz.Hayawan.Sham19Y0023775-ara1' >>> t.build_uri("version_file") '0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed'
Building paths based on the URI:
>>> t.build_pth(uri_type="version", base_pth="D:\test") 'D:/test/0275AH/data/0255Jahiz/0255Jahiz.Hayawan' >>> t.build_pth(uri_type="version_file", base_pth="D:\test") 'D:/test/0275AH/data/0255Jahiz/0255Jahiz.Hayawan/0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed' >>> t.build_pth("version") './0275AH/data/0255Jahiz/0255Jahiz.Hayawan' >>> t.build_pth(uri_type="book_yml") './0275AH/data/0255Jahiz/0255Jahiz.Hayawan/0255Jahiz.Hayawan.yml'
Without uri_type argument, build_pth() builds the fullest path it can:
>>> t.build_pth() './0275AH/data/0255Jahiz/0255Jahiz.Hayawan/0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed' >>> t.language="" # (removing the language property makes it impossible to build a version uri) >>> t.build_pth() './0275AH/data/0255Jahiz/0255Jahiz.Hayawan'
NB: by default, build_pth() takes the OpenITI folder structure into account, in which authors are grouped in 25-year batches by their death date. If you do not want to use this feature, set the URI class’s data_in_25_year_repos attribute to False:
>>> t.build_pth() './0275AH/data/0255Jahiz/0255Jahiz.Hayawan' >>> t.data_in_25_year_repos = False >>> t.build_pth() './0255Jahiz/0255Jahiz.Hayawan' >>> t.data_in_25_year_repos = True
>>> URI.data_in_25_year_repos = False >>> u = URI("0255Jahiz.Hayawan") >>> u.build_pth() './0255Jahiz/0255Jahiz.Hayawan' >>> URI.data_in_25_year_repos = True >>> u.build_pth() './0275AH/data/0255Jahiz/0255Jahiz.Hayawan'
Validity tests: setting invalid values for part of a uri returns an error:
# >>> URI("255Jahiz") Exception: Date Error: URI must start with a date of 4 digits (255 has 3!) # >>> URI("0255Jāḥiẓ") Exception: Author name Error: Author name (Jāḥiẓ) should not contain digits or non-ASCII characters(culprits: ['ā', 'ḥ', 'ẓ']) # >>> t.author = "Jāḥiẓ" Exception: Author name Error: Author name (Jāḥiẓ) should not contain digits or non-ASCII characters(culprits: ['ā', 'ḥ', 'ẓ']) # >>> t.author = "0255Jahiz" Exception: Author name Error: Author name (0255Jahiz) should not contain digits or non-ASCII characters(culprits: ['0', '2', '5', '5']) # >>> URI("0255Jahiz.Al-Hayawan") Exception: Book title Error: Book title (Al-Hayawan) should not contain non-ASCII characters(culprits: ['-']) # >>> URI("0255Jahiz.Hayawan.Shāmila00123545-ara1") Exception: Version string Error: Version string (Shāmila00123545) should not contain non-ASCII characters(culprits: ['ā']) # >>> URI("0255Jahiz.Hayawan.Shamela00123545-arab1") Exception: Language code (arab) should be an ISO 639-2 language code, consisting of 3 characters # >>> t.extension = "markdown" Exception: Extension (markdown) is not among the allowed extensions (['inProgress', 'completed', 'mARkdown', 'yml', '']) # >>> URI("0255Jahiz.Hayawan.Shamela00123545-ara1.markdown") Exception: Extension (markdown) is not among the allowed extensions (['inProgress', 'completed', 'mARkdown', 'yml', ''])
-
__call__
(uri_type=None, ext=None)¶ Call the self.build_uri() method of the URI instance.
Examples
>>> my_uri = URI("0768IbnMuhammadTaqiDinBaclabakki") >>> my_uri.title = "Hadith" >>> my_uri.version = "Shamela0009426" >>> my_uri.language = "ara" >>> my_uri.edition_no = "1" >>> my_uri() '0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1' >>> my_uri("date") '0768' >>> my_uri("author") '0768IbnMuhammadTaqiDinBaclabakki' >>> my_uri("author_yml") '0768IbnMuhammadTaqiDinBaclabakki.yml' >>> my_uri("book") '0768IbnMuhammadTaqiDinBaclabakki.Hadith' >>> my_uri("book_yml") '0768IbnMuhammadTaqiDinBaclabakki.Hadith.yml' >>> my_uri("version") '0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1' >>> my_uri("version_yml") '0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1.yml' >>> my_uri("version_file", ext="completed") '0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1.completed'
-
__iter__
()¶ Enable iteration over a URI object.
Returns: an iterator containing the components of the URI Return type: (iterator) Examples
>>> my_uri = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1.inProgress") >>> for component in my_uri: print(component) 0255 Jahiz Hayawan Sham19Y0023775 ara 1 inProgress >>> my_uri = URI() >>> for component in my_uri: print(component)
-
__repr__
()¶ Return a representation of the components of the URI.
Returns: a representation of the components of the URI. Return type: (str) Examples
>>> my_uri = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1") >>> repr(my_uri) 'uri(date:0255, author:Jahiz, title:Hayawan, version:Sham19Y0023775, language:ara, edition_no:1, extension:)' >>> my_uri = URI() >>> repr(my_uri) 'uri(date:, author:, title:, version:, language:, edition_no:, extension:)'
-
__str__
(*args, **kwargs)¶ Return the reassembled URI.
Returns: the reassembled URI Return type: (str) Examples
>>> my_uri = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1.inProgress") >>> my_uri.extension = "completed" >>> print(my_uri) 0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed
Get the URI’s author property
-
base_pth
¶ Get the URI’s base_pth property (the path to be prepended to the URIs) (usually the folder in which the OpenITI 25-years repos reside)
-
build_uri
(uri_type=None, ext=None)¶ Build an OpenITI URI string from its components.
Parameters: - uri_type (str) – the uri type to be returned (defaults to None): - “date” : only the date (format: 0000) - “author” : authorUri (format: 0255Jahiz) - “author_yml” : filename of the author yml file (format: 0255Jahiz.yml) - “book”: BookUri (format: 0255Jahiz.Hayawan) - “book_yml”: filename of the book yml file (format: 0255Jahiz.Hayawan.yml) - “version”: versionURI (format: 0255Jahiz.Hayawan.Shamela000245-ara1) - “version_yml”: filename of the version yml file (format: 0255Jahiz.Hayawan.Shamela000245-ara1.yml) - “version_file”: filename of the version text file (format: 0255Jahiz.Hayawan.Shamela000245-ara1.completed)
- ext (str) – extension for the version_file uri string (can be “completed”, “inProgress”, “mARkdown”, “” or None).
Returns: - OpenITI URI as a string, e.g.,
0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1
Return type: (str)
Examples
>>> my_uri = URI("0768IbnMuhammadTaqiDinBaclabakki") >>> my_uri.title = "Hadith" >>> my_uri.version = "Shamela0009426" >>> my_uri.language = "ara" >>> my_uri.edition_no = "1" >>> my_uri.build_uri() '0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1' >>> my_uri.build_uri("date") '0768' >>> my_uri.build_uri("author") '0768IbnMuhammadTaqiDinBaclabakki' >>> my_uri.build_uri("author_yml") '0768IbnMuhammadTaqiDinBaclabakki.yml' >>> my_uri.build_uri("book") '0768IbnMuhammadTaqiDinBaclabakki.Hadith' >>> my_uri.build_uri("book_yml") '0768IbnMuhammadTaqiDinBaclabakki.Hadith.yml' >>> my_uri.build_uri("version") '0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1' >>> my_uri.build_uri("version_yml") '0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1.yml' >>> my_uri.build_uri("version_file", ext="completed") '0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1.completed'
-
check_ASCII
(test_string, string_type)¶ Check whether the test_string only contains ASCII letters and digits.
-
check_ASCII_letters
(test_string, string_type)¶ Check whether the test_string only contains ASCII letters.
-
check_date
(date)¶ Check if date is valid (i.e., 4-digit number or empty string)
-
check_extension
(extension)¶ Check whether the proposed extension is allowed.
-
check_language_code
(language)¶ Check whether language is a valid ISO 639-2 language code.
-
date
¶ Get the URI’s date property
-
edition_no
¶ Get the URI’s edition_no property (i.e., the last digit of the URI)
-
extension
¶ Get the URI’s extension property.
-
from_folder
(folder)¶ Create a URI from a folder path without a file name.
Returns the author uri.
Returns: - OpenITI URI as a string, e.g.,
- 0768IbnMuhammadTaqiDinBaclabakki
Return type: uri_string (str) Example
>>> my_uri = URI('0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1') >>> my_uri.get_author_uri() '0768IbnMuhammadTaqiDinBaclabakki'
-
get_book_uri
()¶ Returns the book uri.
Returns: - OpenITI URI as a string, e.g.,
- 0768IbnMuhammadTaqiDinBaclabakki.Hadith
Return type: uri_string (str) Example
>>> my_uri = URI('0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1') >>> my_uri.get_book_uri() '0768IbnMuhammadTaqiDinBaclabakki.Hadith'
-
get_version_uri
()¶ Returns the version uri.
Returns: - OpenITI URI as a string, e.g.,
- 0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1
Return type: (str) Example
>>> my_uri = URI('0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1') >>> my_uri.extension = "completed" >>> my_uri.get_version_uri() '0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1'
-
language
¶ Get the URI’s language property (an ISO 639-2 language code).
-
normpath
(func)¶ replace backslashes by forward slashes also on Windows This is necessary to make the doctests behave the same way on Windows, Mac and Unix systems
-
split_uri
(uri_string=None)¶ Split an OpenITI URI string into its components and check if components are valid.
Parameters: uri_string (str) – OpenITI URI, e.g., 0768IbnMuhammadTaqiDinBaclabakki.Hadith.Shamela0009426-ara1 Returns: list of uri components Return type: (list) Examples
>>> my_uri = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1.completed") >>> my_uri.split_uri() ['0255', 'Jahiz', 'Hayawan', 'Sham19Y0023775', 'ara', '1', 'completed'] >>> my_uri.extension="" >>> my_uri.language="" >>> my_uri.split_uri() ['0255', 'Jahiz', 'Hayawan']
-
title
¶ Get the URI’s title property.
-
uri_type
¶ Get the URI object’s current uri_type (“author”, “book”, “version”, None) based on its defined components.
NB: uri_type does not have a setter method, making it read-only!
Examples
>>> my_uri = URI("0255Jahiz.Hayawan.Sham19Y0023775-ara1.inProgress") >>> my_uri.uri_type 'version' >>> my_uri.language = "" >>> my_uri.uri_type 'book' >>> my_uri = URI() >>> my_uri.uri_type is None True
-
version
¶ Set the URI’s version property.
- VersionURI: consists of
-
openiti.helper.uri.
add_character_count
(tok_count, char_count, tar_uri, execute=False)¶ Add the character and token counts to the new version yml file
Parameters: - tok_count (int) – number of Arabic tokens in a text
- char_count (int) – number of Arabic characters in a text
- tar_uri (URI object) – uri of the target text
- execute (bool) – if True, the function will do its work silently. If False, it will only print a description of the action.
Returns: None
-
openiti.helper.uri.
add_readme
(target_folder)¶ Add default README.md file to target_folder.
Returns: None
-
openiti.helper.uri.
add_text_questionnaire
(target_folder)¶ Add default text_questionnaire.md file to target_folder.
Returns: None
-
openiti.helper.uri.
change_uri
(old, new, old_base_pth=None, new_base_pth=None, execute=False)¶ Change a uri and put all files in the correct folder.
If a version URI changes:
- all text files of that version should be moved
- the yml file of that version should be updated and moved
If a book uri changes:
- the yml file of that book should be updated and moved
- all annotation text files of all versions of the book should be moved
- all yml files of versions of that book should be updated and moved
- the original book folder itself should be (re)moved
if an author uri changes:
- the yml file of that author should be updated and moved
- all book yml files of that should be updated and moved
- all annotation text files of all versions of all books should be moved
- all yml files of versions of all books should be updated and moved
- the original book folders should be (re)moved
- the original author folder itself should be (re)moved
Examples:
change_uri("0255Jahiz", "0256Jahiz") change_uri("0255Jahiz", "0255JahizBasri") change_uri("0255Jahiz.Hayawan", "0255Jahiz.KitabHayawan") change_uri("0255Jahiz.Hayawan.Shamela002526-ara1", "0255Jahiz.Hayawan.Shamela002526-ara2") change_uri("0255Jahiz.Hayawan.Shamela002526-ara1.completed", "0255Jahiz.Hayawan.Shamela002526-ara1.mARkdown")
Parameters: - old (str) – URI string to be changed
- new (str) – URI string to which the new URI should be changed.
- old_base_pth (str) – path to the folder containing the OpenITI 25-year repos, related to the old uri
- new_base_pth (str) – path to the folder containing the OpenITI 25-year repos, related to the new uri
- execute (bool) – if False, the proposed changes will only be printed (the user will still be given the option to execute all proposed changes at the end); if True, all changes will be executed immediately.
Returns: None
-
openiti.helper.uri.
check_token_count
(version_uri, ymlD)¶ Check whether the token count in the version yml file agrees with the actual token count of the text file.
Parameters: - version_uri (URI object) – version uri of the target text
- ymlD (dict) – dictionary containing the data from the relevant yml file
Returns: Tuple containing 2 values (or None):
tok_count (int): number of Arabic tokens in the target text char_count (int): number of Arabic characters in the target text
Return type: (tuple)
-
openiti.helper.uri.
check_yml_files
(start_folder, exclude=[], execute=False, check_token_counts=True)¶ Check whether yml files are missing or have faulty data in them.
Parameters: - start_folder (str) – path to the parent folder of the folders that need to be checked.
- exclude (list) – a list of directory names that should be excluded.
- execute (bool) – if execute is set to False, the script will only show which changes it would undertake if set to True. After it has looped through all files and folders, it will give the user the option to execute the proposed changes.
Returns: Tuple containing:
missing_ymls (list): list of paths to missing yml files missing_tok_count (list): list of tuples with three elements:
- uri (URI object): uri of the target text
- tok_count (int): number of Arabic tokens in the target text
- char_count (int): number of Arabic characters in the target text
non_uri_files (list): list of paths to files whose filename is not a valid OpenITI URI erratic_ymls (list): list of paths to malformed yml files.
Return type: (tuple)
-
openiti.helper.uri.
make_folder
(new_folder, new_uri, execute=False)¶ Check if folder exists; if not, make folder (and, if needed, parents)
Parameters: - new_folder (str) – path to new folder
- new_uri (OpenITI uri object) – uri of the text
- execute (bool) – if False, the proposed changes will only be printed (the user will still be given the option to execute all proposed changes at the end); if True, all changes will be executed immediately.
Returns: None
-
openiti.helper.uri.
move_to_new_uri_pth
(old_fp, new_uri, execute=False)¶ Move file to its new location.
Parameters: - old_fp (filepath) – path to the old file
- new_uri (URI object) – URI of the new file
- execute (bool) – if False, the proposed changes will only be printed (the user will still be given the option to execute all proposed changes at the end); if True, all changes will be executed immediately.
Returns: path to the new file
Return type: (str)
-
openiti.helper.uri.
move_yml
(yml_fp, new_uri, uri_type, execute=False)¶ Replace the URI in the yml file and save the yml file in its new location.
Parameters: - yml_fp (str) – path to the original yml file
- new_uri (URI object) – the new uri
- uri_type (str) – uri type (author, book, version)
- execute (bool) – if False, the proposed changes will only be printed (the user will still be given the option to execute all proposed changes at the end); if True, all changes will be executed immediately.
Returns: filepath of the new yml file
Return type: (str)
-
openiti.helper.uri.
new_yml
(tar_yfp, yml_type, execute=False)¶ Create a new yml file from template.
Parameters: - tar_yfp (str) – filepath to the new yml file
- yml_type (str) – type of yml file (either “version_yml”, “book_yml”, or “author_yml”)
Returns: None
-
openiti.helper.uri.
replace_tok_counts
(missing_tok_count)¶ Replace the token counts in the relevant yml files.
Parameters: missing_tok_count (list) – a list of tuples (uri, token_count):
uri (OpenITI URI object) token_count (int): the number of Arabic tokens in the text file
Returns: None
openiti.helper.yml¶
Functions to read and write yaml files.
OpenITI metadata is stored in yaml files. (yaml stands for “yet another markup language”)
- NB: In correctly formatted OpenITI yml files,
- keys (lemmata) should always:
- contain at least one hash (#)
- end with a colon (:)
- be free of any other non-letter/numeric characters
- Values:
- may contain any character, including colons and new line characters (only something that looks like a yml key (i.e., a combination of letters and hashes ending with a colon) should be avoided at the beginning of a line)
- mutiline values should be indented with 4 spaces
-
openiti.helper.yml.
dicToYML
(dic, max_length=72)¶ Convert a dictionary into a yml string.
Parameters: - dic (dict) – a dictionary of key-value pairs.
- max_length (int) – the maximum number of characters a line should contain.
Returns: yml strin representation of the dic’s key-value pairs
Return type: (str)
Examples
>>> yml_dic = {'00#BOOK#URI######:': '0845Maqrizi.Muqaffa', '90#BOOK#COMMENT##:': 'multiline value; missing-indentation: (and presence of colons): not a problem'} >>> yml_str = ' 00#BOOK#URI######: 0845Maqrizi.Muqaffa\n 90#BOOK#COMMENT##: multiline\n value; missing-indentation:\n (and presence of colons): not\n a problem '.replace(" ", "") # remove Python indentation for doctest >>> dicToYML(yml_dic, max_length=30) == yml_str True
-
openiti.helper.yml.
readYML
(fp)¶ Read a yml file and convert it into a dictionary.
Parameters: fp (str) – path to the yml file. Returns: dictionary representation of the yml key-value pairs Return type: (dict) Examples
>>> fp = "D:/London/OpenITI/25Y_repos/0450AH/data/0429AbuMansurThacalibi/0429AbuMansurThacalibi.AhsanMaSamictu/0429AbuMansurThacalibi.AhsanMaSamictu.Shamela0025011-ara1.yml" >>> readYML(fp) {}
-
openiti.helper.yml.
ymlToDic
(yml_str)¶ Convert a yml string into a dictionary.
- NB: in order to be read correctly, OpenITI yml keys (lemmata) should always:
- contain at least one hash (#)
- end with a colon (:)
- be free of any other non-letter/numeric characters
Values may contain any character, including colons and new line characters (only something that looks like a yml key (i.e., a combination of letters and hashes ending with a colon) should be avoided at the beginning of a line)
In principle, mutiline values should be indented with 4 spaces, but unindented multiline values will be parsed without a problem.
Parameters: yml_str (str) – a yml string. Returns: dictionary representation of the yml key-value pairs Return type: (dict) Examples
>>> from yml import ymlToDic >>> yml_str = "00#BOOK#URI######: 0845Maqrizi.Muqaffa\n90#BOOK#COMMENT##: multiline value; missing-\nindentation: (and presence\nof colons): not a problem\n\n\n".replace(" ", "") # remove Python indentation for doctest >>> yml_dic = {'00#BOOK#URI######:': '0845Maqrizi.Muqaffa', '90#BOOK#COMMENT##:': 'multiline value; missing-indentation: (and presence of colons): not a problem'} >>> ymlToDic(yml_str) == yml_dic True
openiti.git¶
openiti.git.clone_OpenITI¶
Clone github repositories to your local machine.
Examples
Clone all OpenITI 25-year repos:
# >>> repo_list = get_repo_urls(group="orgs", name="OpenITI",
# path_pattern="\d{4}AH$")
# >>> clone_repos(repo_list, r"D:\OpenITI")
Clone specific repos:
# >>> base = ""https://github.com/OpenITI/"
# >>> repo_list = [base+"mARkdown_scheme", base+"RELEASE"]
# >>> clone_repos(repo_list, r"D:\OpenITI")
- Command line usage:
To clone all the OpenITI organization’s 25-years repos:
python clone_OpenITI.py ["orgs" or "users"] [user or org name] [path_pattern] [[dest_folder]]
dest_folder$ python pth/to/clone_OpenITI.py orgs OpenITI \d+AH$
other_folder$ python pth/to/clone_OpenITI.py orgs OpenITI \d+AH$ path/to/dest_folder
If fewer than 3 arguments are given, the program will prompt you for the arguments:
$ python clone_OpenITI.py Enter the organization/user name: openiti Organization or User? (orgs/users): orgs Enter the regex pattern to match desired repo URLs: \d{4}AH Enter the destination folder for the clone: openiti_clone
-
openiti.git.clone_OpenITI.
clone_repos
(all_repos, clone_dir='.')¶ Clone the list of repo urls all_repos to the clone_dir
Parameters: - all_repos (list) – a list of repo urls.
- clone_dir (str) – path to the local folder where the repos are to be cloned. Defaults to the current active directory.
Returns: None
-
openiti.git.clone_OpenITI.
get_repo_urls
(group='orgs', name='OpenITI', path_pattern='\\d{4}AH$')¶ Get a list of all repos owned by organisation/user name that match the regex path_pattern
Parameters: - group (str) – either “users” or “orgs”. Defaults to “orgs”
- name (str) – GitHub name of the user/organization. Defaults to “OpenITI”
- path_pattern (str) – regex pattern that matches the desired repository names. If none is defined, all repos will be cloned. Defaults to r”d{4}AH$”.
Returns: a list of repo urls that matches the path_pattern regex.
Return type: (list)
openiti.git.collect_txt_files¶
Moves all text files from all subdirectories into a single directory.
Todo
- if more than one version of the text file is in the folder,
- the script will not automatically copy the best version!
Examples
Command line usage:
$ collect_txt_files D:\OpenITI\25Y_folders D:\OpenITI\release\texts
If the source and target directories were not given, the user will be prompted for them:
$ collect_txt_files
Enter the source directory:
Enter the target directory:
-
openiti.git.collect_txt_files.
collect_text_files
(source_dir, dest_dir, exclude_folders=[], extensions=['mARkdown', 'inProgress', 'completed', ''], version_id_only=True)¶ Copy all text files in source_dir (+ its subfolders) to dest_dir.
Parameters: - source_dir (str) – path to the directory that contains (subfolders containing) the text files
- dest_dir (str) – path to a directory to which text files will be copied.
- exclude_folders (list) – directories to be excluded from the collection process
- extensions (list) – list of extensions; only text files with these extensions will be copied. Defaults to [“mARkdown”, “inProgress”, “completed”, “”]
- version_id_only (bool) – if True, the filename will be shortened to the last part of the URI, i.e., the version id + language id (e.g., Shamela001185-ara1). Defaults to True.
Returns: number of files copied
Return type: (int)
-
openiti.git.collect_txt_files.
move_ara_files
(source_dir, dest_dir)¶ old name of the collect_text_files function; renamed because the name does not reflect what function does; old name kept for backward compatibility
openiti.git.get_issues¶
Get selected annotation issues from GitHub; optionally, print them or save them as a tsv file.
Example:
issues = get_issues("OpenITI/Annotation",
issue_labels=["in progress"],
state="all"
)
issues = define_text_uris(issues)
uri_dict = sort_issues_by_uri(issues)
print_issues_by_uri(uri_dict, "test.tsv")
-
openiti.git.get_issues.
define_text_uris
(issues, verbose=False)¶ Define which text uri the issue pertains to. Store the uri in the issue object (issue.uri).
Parameters: - issues (list) – a list of github issue objects.
- verbose (bool) – if verbose, print issues for which no uris were found.
Returns: the list of updated github issue objects
Return type: (list)
-
openiti.git.get_issues.
get_issues
(repo_name, access_token=None, issue_labels=None, state='open')¶ Get all issues connected to a specific github repository.
Parameters: - repo_name (str) – the name of the Github repository
- user (str) – username
- password (str) – password
- (list; default (issue_labels) – None): a list of github issue label names; only the issues with an issue label name in this list will be downloaded; if set to None, all issues will be downloaded
- (str; default (state) – “open”): only the issues with this state (open/closed/all) will be downloaded.
Returns: a list of github issues.
Return type: (list)
-
openiti.git.get_issues.
print_issues_by_uri
(uri_dict, save_fp='')¶ Print a tab-delimited list of uris with the issues connected to them. URI Issue number Issue label
Parameters: uri_dict (dict) – key: uri, value: list of github issues related to this uri Returns: None
-
openiti.git.get_issues.
sort_issues_by_uri
(issues)¶ Create a dictionary with the uris as keys.
Parameters: issues (list) – a list of github issue objects Returns: - a dictionary with following key-value pairs:
- key: uri; value: list of github issue objects related to this uri
Return type: (dict)
openiti.instantiations¶
openiti.instantiations.generate_imech_instantiation¶
openiti.instantiations.generate_istylo_instantiation¶
openiti.new_books¶
openiti.new_books.add.add_books¶
Scripts to add books to the correct OpenITI repositories.
- initialize_texts_from_csv: use a csv file that contains for every file
- that needs to be added to the corpus: - the path to its current location - the URI it needs to get
- initialize_new_texts_in_folder: initialize all text files in a folder
- (in order for this to work, all files need to have valid URIs)
- download_texts_from_CSV: download texts from the internet and
- directly put them into the correct OpenITI folder
-
openiti.new_books.add.add_books.
download_texts_from_CSV
(csv_fp, base_url='', new_base_pth='')¶ Use a CSV file (filename, URI) to download a list of texts to the relevant OpenITI folder.
The CSV file (which should not contain a heading) can contain full urls for the original files, or only filenames; in the latter case, the url of the website where these files are located should be passed to the function as the base_url argument. Similarly, the URI column can contain full OpenITI URI filepaths or only the URIs; in the latter case, the path to the folder containing the OpenITI 25-years folders should be passed to the function as the new_base_pth argument.
Parameters: - csv_fp (str) – path to a csv file that contains the following columns: 0. filepath to (or filename of) the text file 1. full version uri of the text file (no headings!)
- old_base_path (str) – path to the folder containing the files that need to be initialized. Defaults to “”.
- new_base_pth (str) – path to the folder containing the OpenITI 25-years repos. Defaults to “”.
Returns: None
-
openiti.new_books.add.add_books.
initialize_new_text
(origin_fp, target_base_pth, execute=False)¶ Move a new text file to its OpenITI repo, creating yml files if necessary (or copying them from the same folder if present).
The function also checks whether the new text adheres to OpenITI text format.
Parameters: - origin_fp (str) – filepath of the text file (filename must be in OpenITI uri format)
- target_base_pth (str) – path to the folder that contains the 25-years-repos
- execute (bool) – if False, the proposed changes will only be printed (the user will still be given the option to execute all proposed changes at the end); if True, all changes will be executed immediately.
Returns: None
Example
# >>> origin_folder = r”D:OpenITIbarzakh” # >>> fn = “0375IkhwanSafa.Rasail.Hindawi95926405Vols-ara1.completed” # >>> origin_fp = os.path.join(origin_folder, fn) # >>> target_base_pth = r”D:OpenITI25Yrepos” # >>> initialize_new_text(origin_fp, target_base_pth, execute=False)
-
openiti.new_books.add.add_books.
initialize_new_texts_in_folder
(folder, target_base_pth, execute=False)¶ Move all new texts in folder to their OpenITI repo, creating yml files if necessary (or copying them from the same folder if present).
Parameters: - folder (str) – path to the folder that contains new text files (with OpenITI uri filenames) and perhaps yml files
- target_base_pth (str) – path to the folder containing the 25-years repos
- execute (bool) – if False, the proposed changes will only be printed (the user will still be given the option to execute all proposed changes at the end); if True, all changes will be executed immediately.
Returns: None
Examples:
# >>> folder = r"D:\OpenITI\barzakh" # >>> target_base_pth = r"D:\OpenITI\25Yrepos" # >>> initialize_new_texts_in_folder(folder, target_base_pth, # execute=False)
-
openiti.new_books.add.add_books.
initialize_texts_from_CSV
(csv_fp, old_base_pth='', new_base_pth='', execute=False)¶ Use a CSV file (filename, URI) to move a list of texts to the relevant OpenITI folder.
The CSV file (which should not contain a heading) can contain full filepaths to the original files, or only filenames; in the latter case, the path to the folder where these files are located should be passed to the function as the old_base_pth argument. Similarly, the URI column can contain full OpenITI URI filepaths or only the URIs; in the latter case, the path to the folder containing the OpenITI 25-years folders should be passed to the function as the new_base_pth argument.
Parameters: - csv_fp (str) – path to a csv file that contains the following columns: 0. filepath to (or filename of) the text file 1. full version uri of the text file (no headings!)
- old_base_path (str) – path to the folder containing the files that need to be initialized
- new_base_pth (str) – path to the folder containing the OpenITI 25-years repos
- execute (bool) – if False, the proposed changes will only be printed (the user will still be given the option to execute all proposed changes at the end); if True, all changes will be executed immediately.
Returns: None
openiti.new_books.convert.generic_converter¶
A generic converter for texts into OpenITI format.
This generic converter contains the basic procedure for converting texts into OpenITI format.
These are the main methods of the GenericConverter class:
- convert_file(source_fp): basic procedure for converting an html file.
- convert_files_in_folder(source_folder): convert all files files in the
- source_folder (calls convert_file)
The procedure can be used as a template for other converters. Subclass the GenericConverter to make a converter for a new source of scraped books:
class GenericHtmlConverter(GenericConverter):
def new_function(self, text):
"This function adds functionality to GenericConverter"
Usually, it suffices to overwrite a small number of methods of the GenericConverter class in the sub-class, and/or add a number of methods needed for the conversion of a specific group of documents: e.g.:
class GenericHtmlConverter(GenericConverter):
def get_data(self, source_fp):
# code written here overwrites the GenericConverter's get_data method
def post_process(self, text):
# append new actions by appending them to
# the superclass's post_process method:
text = super().post_process(text)
text = re.sub("\n\n+", "\n\n", text)
return text
def new_function(self, text):
"This function adds functionality to GenericConverter"
The GenericConverter contains a number of dummy (placeholder) functions that must be overwritten in the subclass because they are dependent on the data format.
For example, this is the inheritance schema of the EShiaHtmlConverter, which is a subclass of the GenericHtmlConverter: (methods of GenericConverter are inherited by GenericHtmlConverter; and methods of GenericHtmlConverter are inherited by EShiaHtmlConverter. Methods of the superclass with the same name in the subclass are overwritten by the latter)
| GenericConverter | GenericHtmlConverter | EShiaHtmlConverter |
|---|---|---|
| __init__ | __init__ | (inherited) |
| convert_files_in_folder | (inherited) | (inherited) |
| convert file | (inherited) | (inherited) |
| make_dest_fp | (inherited - generic!) | (inherited - generic!) |
| get_metadata (dummy) | (inherited - dummy!) | get_metadata |
| get_data | get_data | (inherited) |
| pre_process | (inherited) | (inherited) |
| add_page_numbers (dummy) | (inherited - dummy!) | add_page_numbers |
| add_structural_annotations (dummy) | add_structural_annotations | add_structural_annotations |
| remove_notes (dummy) | remove_notes | remove_notes |
| reflow | (inherited) | (inherited) |
| add_milestones (dummy) | (inherited - dummy!) | (inherited - dummy!) |
| post_process | (inherited - generic!) | post_process |
| compose | (inherited) | (inherited) |
| save_file | (inherited) inspect_tags_in_html inspect_tags_in_folder find_example_of_tag | (inherited) (inherited) (inherited) (inherited) |
Examples
>>> from generic_converter import GenericConverter
>>> conv = GenericConverter()
>>> conv.VERBOSE = False
>>> folder = r"test/"
>>> conv.convert_file(folder+"86596.html")
>>> conv.convert_files_in_folder(folder, ["html"])
openiti.new_books.convert.epub_converter_generic¶
Generic converter that converts an Epub to OpenITI mARkdown.
Examples
Generic epub conversion:
>>> from epub_converter_generic import GenericEpubConverter
>>> gen_converter = GenericEpubConverter(dest_folder="test/converted")
>>> gen_converter.VERBOSE = False
>>> folder = r"test"
>>> fn = r"Houellebecq 2019 - serotonine.epub"
>>> gen_converter.convert_file(os.path.join(folder, fn))
>>> gen_converter.convert_files_in_folder(folder, ["epub",])
Sub-class to create a converter specific to epubs from the Hindawi library:
# >>> HindawiConverter = GenericEpubConverter("test/converted")
# >>> HindawiConverter.VERBOSE = False
# >>> from helper import html2md_hindawi
# >>> HindawiConverter.convert_html2md = html2md_hindawi.markdownify # (1)
# >>> HindawiConverter.toc_fn = "nav.xhtml" # (2)
# >>> folder = r"test"
# >>> fn = r"26362727.epub"
# >>> HindawiConverter.convert_file(os.path.join(folder, fn))
# (1) overwrite the convert_html2md function
# (2) specify the filename of the table of contents in Hindawi epub files
An Epub file is in fact a zipped archive. The most important element of the archive for conversion purposes are the folders that contain the html files with the text. Some Epubs have a table of contents that defines the order in which these html files should be read.
The GenericEpubConverter is a subclass of the GenericConverter from the generic_converter module:
- GenericConverter
- _ GenericEpubConverter
GenericEpubConverter’s main methods are inherited from the GenericConverter:
- convert_file(source_fp): basic procedure for converting an epub file.
- convert_files_in_folder(source_folder): convert all epub files in the folder
- (calls convert_file)
Methods of both classes: (methods of GenericConverter are inherited by GenericEpubConverter; methods of GenericConverter with the same name in GenericEpubConverter are overwritten by the latter)
| GenericConverter | GenericEpubConverter |
|---|---|
| __init__ | __init__ |
| convert_files_in_folder | (inherited) |
| convert_file | (inherited) |
| make_dest_fp | (inherited - generic!) |
| get_metadata | (inherited - generic!) |
| get_data | get_data |
| pre_process | (inherited) |
| add_page_numbers | (inherited - generic!) |
| add_structural_annotations | (inherited - generic!) |
| remove_notes | remove_notes |
| reflow | (inherited) |
| add_milestones | (inherited) |
| post_process | (inherited - generic!) |
| compose | (inherited) |
| save_file | (inherited) inspect_epub sort_html_files_by_toc add_unique_tags |
To create a converter for a specific type of epubs, subclass the GenericEpubConverter and overwrite some of its methods:
- GenericConverter
- _ GenericEpubConverter
- _ HindawiEpubConverter _ ShamelaEpubConverter _ …
openiti.new_books.convert.epub_converter_hindawi¶
Convert Epubs from the Hindawi library to OpenITI mARkdown.
Examples
>>> from epub_converter_hindawi import HindawiEpubConverter
>>> from helper.yml2json import yml2json
>>> folder = "test/"
>>> fn = "26362727.epub"
>>> hc = HindawiEpubConverter(dest_folder="test/converted")
>>> hc.VERBOSE = False
>>> meta_fp = "test/hindawi_metadata_man.yml"
>>> hc.metadata_dic = yml2json(meta_fp, container = {})
>>> hc.metadata_file = meta_fp
>>> hc.convert_file(folder+fn)
#>>> hc.convert_files_in_folder(folder)
An Epub file is in fact a zipped archive. The most important element of the archive for conversion purposes are the folders that contain the html files with the text. Some Epubs have a table of contents that defines the order in which these html files should be read.
The HindawiEpubConverter is a subclass of the GenericEpubConverter, which in turn is a subclass of the GenericConverter from the generic_converter module:
- GenericConverter
- _ GenericEpubConverter
- _ HindawiEpubConverter
HindawiConverter’s main methods are inherited from the GenericConverter:
- convert_file(source_fp): basic procedure for converting an epub file.
- convert_files_in_folder(source_folder): convert all epub files in the folder
- (calls convert_file)
Methods of both classes:
(methods of GenericConverter are inherited by GenericEpubConverter; methods of GenericConverter with the same name in GenericEpubConverter are overwritten by the latter)
| generic_converter | epub_converter_generic | epub_converter_hindawi |
|---|---|---|
| __init__ | __init__ | __init__ |
| convert_files_in_folder | (inherited) | (inherited) |
| convert file | (inherited) | (inherited) |
| make_dest_fp | (inherited - generic!) | (inherited - generic!) |
| get_metadata | (inherited - generic!) | get_metadata |
| get_data | get_data | (inherited) |
| pre_process | (inherited) | (inherited) |
| add_page_numbers | (inherited - generic!) | (inherited - generic!) |
| add_structural_annotations | (inherited - generic!) | (inherited - generic!) |
| remove_notes | remove_notes | (inherited) |
| reflow | (inherited) | (inherited) |
| add_milestones | (inherited) | (inherited) |
| post_process | (inherited - generic!) | (inherited - generic!) |
| compose | (inherited) | (inherited) |
| save_file | (inherited) convert_html2md inspect_epub sort_html_files_by_toc add_unique_tags | (inherited) convert_html2md (inherited) (inherited) (inherited) |
openiti.new_books.convert.html_converter_generic¶
Generic converter that converts HTML files to OpenITI mARkdown.
Examples
Generic html conversion: >>> from html_converter_generic import GenericHtmlConverter >>> gen_converter = GenericHtmlConverter(dest_folder=”test/converted”) >>> gen_converter.VERBOSE = False >>> folder = r”test” >>> fn = r”86596.html” >>> gen_converter.convert_file(os.path.join(folder, fn)) >>> gen_converter.convert_files_in_folder(folder, extensions=[“.html”])
Sub-class to create a converter specific to htmls from the eShia library: >>> eShiaConv = GenericHtmlConverter(“test/converted”) >>> eShiaConv.VERBOSE = False >>> from helper import html2md_eShia >>> eShiaConv.add_structural_annotations = html2md_eShia.markdownify # (1) >>> folder = r”test” >>> fn = r”86596.html” >>> eShiaConv.convert_file(os.path.join(folder, fn))
# (1) overwrite the add_structural_annotations method with another function
The GenericHtmlConverter is a subclass of the GenericConverter from the generic_converter module:
- GenericConverter
- _ GenericEpubConverter _ GenericHtmlConverter
GenericHtmlConverter’s main methods are inherited from the GenericConverter:
- convert_file(source_fp): basic procedure for converting an html file.
- convert_files_in_folder(source_folder): convert all html files in the folder
- (calls convert_file)
Methods of both classes: (methods of GenericConverter are inherited by GenericHtmlConverter; methods of GenericConverter with the same name in GenericHtmlConverter are overwritten by the latter)
| GenericConverter | GenericHtmlConverter |
|---|---|
| __init__ | __init__ |
| convert_files_in_folder | (inherited) |
| convert file | (inherited) |
| make_dest_fp | (inherited - generic!) |
| get_metadata (dummy) | (inherited - dummy!) |
| get_data | get_data |
| pre_process | (inherited) |
| add_page_numbers (dummy) | (inherited - dummy!) |
| add_structural_annotations (dummy) | add_structural_annotations |
| remove_notes (dummy) | remove_notes |
| reflow | (inherited) |
| add_milestones (dummy) | (inherited - dummy!) |
| post_process | (inherited - generic!) |
| compose | (inherited) |
| save_file | (inherited) inspect_tags_in_html inspect_tags_in_folder find_example_of_tag |
The main difference between the two converters is the add_structural_annotations method. GenericHtmlConverter uses here the html2md converter that converts or strips html tags.
To create a converter for a specific type of html files, subclass the GenericHtmlConverter and overwrite some of its methods that are dependent on the structure of the data, esp.:
- get_metadata
- add_page_numbers
- remove_notes
- add_structural_annotation (subclass the generic html2md converter for this, and create a html2md converter specific to the files you need to convert)
- GenericConverter
- _ GenericHtmlConverter
- _ eShiaHtmlConverter _ NoorlibHtmlConverter _ …
In addition to the conversion methods, the GenericHtmlConverter contains a number of useful methods that help with deciding which html tags need to be converted:
- inspect_tags_in_html
- inspect_tags_in_folder
- find_example_of_tag
openiti.new_books.convert.html_converter_eShia¶
Converter that converts HTML files from the eShia library to OpenITI mARkdown.
The eShiaHtmlConverter is a subclass of GenericHtmlConverter, which in its turn inherits many functions from the GenericConverter.
- GenericConverter
- _ GenericHtmlConverter
- _ EShiaHtmlConverter _ NoorlibHtmlConverter _ …
EShiaHtmlConverter’s main methods are inherited from the GenericConverter:
- convert_file(source_fp): basic procedure for converting an html file.
- convert_files_in_folder(source_folder): convert all html files in the folder
- (calls convert_file)
Overview of the methods of these classes: (methods of GenericConverter are inherited by GenericHtmlConverter; and methods of GenericHtmlConverter are inherited by EShiaHtmlConverter. Methods of the superclass with the same name in the subclass are overwritten by the latter)
| GenericConverter | GenericHtmlConverter | EShiaHtmlConverter |
|---|---|---|
| __init__ | __init__ | (inherited) |
| convert_files_in_folder | (inherited) | (inherited) |
| convert file | (inherited) | (inherited) |
| make_dest_fp | (inherited - generic!) | (inherited - generic!) |
| get_metadata (dummy) | (inherited - dummy!) | get_metadata |
| get_data | get_data | (inherited) |
| pre_process | (inherited) | (inherited) |
| add_page_numbers (dummy) | (inherited - dummy!) | add_page_numbers |
| add_structural_annotations (dummy) | add_structural_annotations | add_structural_annotations |
| remove_notes (dummy) | remove_notes | remove_notes |
| reflow | (inherited) | (inherited) |
| add_milestones (dummy) | (inherited - dummy!) | (inherited - dummy!) |
| post_process | (inherited - generic!) | post_process |
| compose | (inherited) | (inherited) |
| save_file | (inherited) inspect_tags_in_html inspect_tags_in_folder find_example_of_tag | (inherited) (inherited) (inherited) (inherited) |
The EShiaHtmlConverter’s add_structural_annotations method uses html2md_eShia, an adaptation of the generic html2md (based on markdownify) to convert the html tags to OpenITI annotations.
Examples
>>> from html_converter_eShia import EShiaHtmlConverter
>>> conv = EShiaHtmlConverter()
>>> conv.VERBOSE = False
>>> folder = r"test/"
>>> conv.convert_file(folder+"86596.html")
>>> conv.convert_files_in_folder(folder, ["html"])
openiti.new_books.convert.shamela_converter¶
A Converter for al-Maktaba al-Shamela databases to OpenITI mARkdown format.
Al-Maktaba al-Shamela is a digital text collection, but it also developed its own reader, which is based on an internal database.
A large number of texts are available on its website (www.shamela.ws), and can be downloaded in EPUB or .bok format (the latter is a zipped .mdb file). These files must be converted with another script. See https://github.com/OpenITI/raw_SHAM19Y/wiki for that conversion process.
However, the program also allows users to add texts from other sources (incl. their own transcriptions) to the program’s database. The database can be zipped, together with an executable for the reader, and disseminated. Some users have used this ability of the program to create their own collections that include many texts not available in the original program (for ideological or other reasons), e.g.,
- al-maktaba al-shamela al-dhahabiyya
- al-maktaba al-shamela al-ibadiyya
- al-maktaba al-shamela al-zaydiyya
- etc.
Database structure
The database consists of a large number of folders and files:
./Files/Main.mdb
: contains the metadata for every book in the program,- in a table called “0bok”
./Files/special.mdb
: contains more metadata + data that could be- marked up in the text (like Qur’anic passages)
./Books
: contains 10 folders, named “0” to “9”, that each contain- a large number of .mdb files (one .mdb file per text; filename = id number)
NB: in some versions of the database, the books are actually in
in large multi-book .mdb files in another folder (./Books/Archive
),
which contains large .mdb files, each ca. 600 MB;
each of these contain the bxxxx and txxxx tables for a couple
of thousand books (xxxx being the Shamela id of the book):
- bxxxx contains the text
- txxxx contains the section titles
Module description
This module contains two major components:
extract_metadata
: extracts the metadata from the shamela databaseBokJsonConverter
class: extracts the text and book structure from the books’ .mdb files into json format, fuses this with the book’s metadata, and converts it into OpenITI mARkdown format
The main component of the module, the BokJsonConverter class, is a subclass of the GenericConverter from the generic_converter module:
- GenericConverter
- _ GenericEpubConverter _ BokJsonConverter
Methods of both classes: (methods of GenericConverter are inherited by GenericHtmlConverter; methods of GenericConverter with the same name in GenericHtmlConverter are overwritten by the latter)
| GenericConverter | BokJsonConverter |
|---|---|
| __init__ | __init__ (appended) |
| convert_files_in_folder | convert_files_in_folder |
| convert file | convert file |
| make_dest_fp | (inherited - generic!) |
| get_metadata (dummy) | (metadata collected in get_data step) |
| get_data | get_data |
| pre_process | pre_process |
| add_page_numbers (dummy) | (pages added in get_data step) |
| add_structural_annotations (dummy) | add_structural_annotations |
| remove_notes (dummy) | remove_notes |
| reflow | (inherited) |
| add_milestones (dummy) | (inherited - dummy!) |
| post_process | post_process |
| compose | (inherited) |
| save_file | (inherited) mdb2json format_text format_metadata |
-
class
openiti.new_books.convert.shamela_converter.
BokJsonConverter
(all_meta=None, additional_meta=None, dest_folder=None)¶ Extract book data from .mdb files into json, and convert into mARkdown.
-
add_structural_annotations
(passage_text, passage_id)¶ Add structural tags to titles in passage_text.
-
convert_file
(source_fp)¶ Convert one file to OpenITI format.
Parameters: source_fp (str) – path to the file that must be converted. Returns: None
-
format_metadata
(metadata)¶ Convert the metadata in the metadata dict to a string.
-
format_text
(book, toc, book_id)¶ Extract text from json, add page numbers and headings.
Parameters: - book (list) – a list of dictionaries (keys: id, nass, page, part, [seal])
- toc (list) – a list of dictionaries (keys: id, tit, lvl, [sub])
Returns: the formatted text
Return type: text (str)
-
get_data
(source_fp, shamid)¶ Extract the data from the source_fp and format it in OpenITI format.
### Overwrite this method in subclass ###
Parameters: source_fp (str) – path to the original file Returns: the text in its initial state Return type: text (str)
-
mdb2json
(source_folder)¶ Convert mdb files to json files.
The .mdb files are usually in the Books folder, in subfolders numbered “0” to “9”. The converted files will be saved in <self.dest_folder>/json/
-
post_process
(text)¶ Carry out post-processing operations on the text.
### Overwrite this method in subclass ###
Parameters: text (str) – the text in its current conversion state Returns: the text after post-processing operations. Return type: texgt (str)
-
remove_notes
(passage_text, notes)¶ Remove footnotes from the passage_text and add them to notes list.
-
-
openiti.new_books.convert.shamela_converter.
extract_metadata
(files_folder, outfolder)¶ Extract the book metadata from the Files folder of the Shamela program.
The metadata is located in two separate mdb files:
- Files/main.mdb
- Files/special.mdb
The metadata is extracted into json files in the outfolder:
- [outfolder]/main_metadata.json: full metadata from Files/main.mdb in json format (dictionary). Under key “0bok” it contains a list of metadata dictionaries, one for each book in the collection
- [outfolder]/special_metadata.json: full metadata from Files/special.mdb in json format
- [outfolder]/genres.json: a dictionary of genres and their ids extracted from main.mdb
- [outfolder]/Authors_metadata.json: a dictionary of authors and their ids extracted from special.mdb
- [outfolder]/main_metadata_reform.json: a reformatted version of main_metadata.json (dictionary of dictionaries, one for each book: key = bookID, value = book metadata in dictionary format)
Only main_metadata_reform.json is necessary for the conversion of Shamela mdb files to txt.
-
openiti.new_books.convert.shamela_converter.
main
(meta_folder=None, files_folder=None, books_folder=None, conv_folder=None, download_date='', download_source='', config_fp=None)¶ Collect the variables and carry out the conversion.
Parameters: - meta_folder (str) – path to the folder containing metadata extracted from the database
- files_folder (str) – path to the Files folder of the database (containing the main metadata database files)
- books_folder (str) – path to the Bookss folder of the database (containing the .mdb files containing the text of the books)
- conv_folder (str) – path to the folder where the converted files should be stored
- download_date (str) – (approximate) date when the database was downloaded
- download_source (str) – name and/or dowload url of the database
- config_fp (str) – path to a config.py file containing the above-mentioned parameters NB: to print an empty config file: print_default_config_file()
-
openiti.new_books.convert.shamela_converter.
reformat_betaka
(meta, i, row)¶ Reformats the Shamela Betaka
The Shamela metadata table contains a field called ‘betaka’, i.e., a kind of human-readable index card of the main metadata of the book (author, title, etc.). The betaka field is a string, with key: value pairs split by new lines. This function splits the metadata in the betaka string and adds it to the meta dictionary.
-
openiti.new_books.convert.shamela_converter.
reformat_metadata
(d, authors_dict)¶ Reformats the Shamela Betaka
The Shamela metadata table contains a field called ‘betaka’, i.e., a kind of human-readable index card of the main metadata of the book (author, title, etc.). The betaka field is a string, with key: value pairs split by new lines. This function splits the metadata in the betaka string and adds it to the meta dictionary.
Parameters: d (dict) – a dictionary containing two items: d[“0bok”]: a list of dictionary representations of the metadata of each book in the Shamela metadata table ‘Main’. d[“0cat”]: a list of dictionary representations of the genre metadata in the Shamela metadata table (keys: id, name, catord, lvl) Returns: - the same list of dictionaries, but with the metadata
- in the betaka field split out into keys and values
Return type: meta (list)
openiti.new_books.convert.tei_converter_generic¶
A generic converter for converting tei xml files into OpenITI format.
Subclass the TeiConverter to make a converter for a new source of scraped books in TEI xml format.
Examples
Generic tei conversion:
>>> from tei_converter_generic import TeiConverter
>>> conv = TeiConverter(dest_folder="test/converted")
>>> conv.VERBOSE = False
>>> folder = r"test"
>>> fn = r"GRAR000070"
>>> conv.convert_file(os.path.join(folder, fn))
>>> conv.convert_files_in_folder(folder, extensions = [""])
Sub-class to create a converter specific to GRAR collection tei files:
#>>> class GRARConverter(TeiConverter):
# def post_process(self, text):
# text = super().post_process(text)
# vols = re.findall("#META#.+ vol. (\d+) pp", text)
# if len(vols) == 1:
# text = re.sub("PageV00", "PageV{:02d}".format(int(vols[0])), text)
# elif len(vols) == 0:
# text = re.sub("PageV00", "PageV01", text)
# return text
#>>> conv = GRARConverter("test/converted")
#>>> conv.VERBOSE = False
#>>> folder = r"test"
#>>> conv.convert_files_in_folder(folder, exclude_ext = ["py", "yml", "txt", "epub"])
| GenericConverter | TeiConverter |
|---|---|
| __init__ | __init__ (appended) |
| convert_files_in_folder | (inherited) |
| convert file | (inherited) |
| make_dest_fp | (inherited) |
| get_metadata (dummy) | get_metadata |
| get_data | (inherited) |
| pre_process | pre_process |
| add_page_numbers (dummy) | (inherited - not used) |
| add_structural_annotations (dummy) | add_structural_annotations |
| remove_notes (dummy) | (inherited - generic!) |
| reflow | (inherited) |
| add_milestones (dummy) | (inherited - dummy) |
| post_process | post_process |
| compose | (inherited) |
| save_file | (inherited) preprocess_page_numbers preprocess_wrapped_lines |
openiti.new_books.convert.html_converter_GRAR¶
A converter for converting GRAR tei xml files into OpenITI format.
The Graeco-Arabic studies website (graeco-arabic-studies.org) contains 78 texts transcribed in TEI xml, and 21 additional texts available only in html.
The XML texts were downloaded as is; for the html texts, for each text every separate page was downloaded, and a compound html file containing the metdata of that text + the div containing the text of each page was created (using the combine_files_in_folder function).
The GRARConverter (which is sub-classed from tei_converter_generic.TeiConverter) converts both the tei xml files and the html files. It uses the generic TeiConverter’s tei2md.TeiConverter for the xml files, and for the html files the html2md_GRAR.GRARHtmlConverter (which is a modified version (sub-class) of the html2md.MarkdownConverter).
Schema representing the method inheritance in the GRARConverter:
| GenericConverter | TeiConverter | GRARConverter |
|---|---|---|
| __init__ | __init__ (appended) | (inherited) |
| convert_files_in_folder | (inherited) | convert_files_in_folder |
| convert file | convert_file | convert_file |
| make_dest_fp | (inherited) | (inherited) |
| get_metadata (dummy) | get_metadata | (inherited - for tei xml) |
| get_data | (inherited) | (inherited - for tei xml) |
| pre_process | pre_process | pre-process (appended) |
| add_page_numbers (dummy) | (inherited - not used) | (inherited - not used) |
| add_structural_annotations | add_structural_annotations | (inherited - for tei xml) |
| remove_notes (dummy) | (inherited - generic!) | (inherited - generic!) |
| reflow | (inherited) | (inherited) |
| add_milestones (dummy) | (inherited - dummy) | (inherited - not used) |
| post_process | post_process | post-process (appended) |
| compose | (inherited) | (inherited) |
| save_file | (inherited) preprocess_page_numbers preprocess_wrapped_lines | (inherited) (inherited) (inherited) get_html_data get_html_metadata format_html_metadata |
Examples
>>> from tei_converter_GRAR import GRARConverter
>>> conv = GRARConverter(dest_folder="test/converted")
>>> conv.VERBOSE = False
>>> folder = r"test"
>>> fn = r"GRAR000070"
>>> conv.convert_file(os.path.join(folder, fn))
-
openiti.new_books.convert.tei_converter_GRAR.
combine_files_in_folder
(folder, output_fp=None)¶ Combine separate html files into one master html file
Extracts a list of all tags used in the texts in a folder:
For GRAR:
<body> <div1 type=”” n=”” (name=”“)(/)> # book subdivision level 1 <div2 type=”” n=”” (name=”“)(/)> # book subdivision level 2 <head> # title <lb(/)> # start of new line <milestone unit=”” n=”“/> # <p> # paragraph <pb (type=”“) n=”“/> # start of new page <quote type=”” (author=”“) (id=”“)> # quotation of a source <text lang=”” id=”“> # metadata
# tables: <table> <tr> <td>
# line groups (e.g., for poetry): <lg> # line group <l> # line in line group
div types:
div1 div2 book books chapter chapter folio sentence sentence aphorism pb types: primary, secondary
quote types: lemma, commentary
milestone units: book, ed1chapter, ed1page, ms1folio
openiti.new_books.convert.helper¶
openiti.new_books.convert.helper.bok¶
A library to deal with Shamela’s BOK file format.
Bok files are in fact Microsoft Access mdb databases. The database is accessed using the pypyodbc library (https://github.com/jiangwen365/pypyodbc).
Problem: the texts are encoded in the Windows-1256 format instead of UTF-8 in the .bok file. Pypyodbc has an inbuilt “unicode_results=True” parameter, but this does not seem to influence the way how the data is read. Python 3 has difficulties dealing with this, because it considers any string a unicode string. Current (convoluted but working) solution for extracting text from bok: save the text as a temp file, open it with Windows-1256 encoding (encoding=”cp1256”) and save it again in utf-8 encoding. Saving to JSON also seems to do the job.
In order to interact with the database, use the cursor object cur returned by the connect_to_db function.
Most important functions with the cursor object:
- cur.tables() : get a list of all tables in the database
- cur.execute() : execute an sql string
- cur.description: get the headings of the columns in the current cur object
The bok file contains a number of tables, of which the most important are:
- Main: contains the metadata of the book
- bxxxx (in which xxxx is the Shamela id of the book): contains the text, one page per row.
- txxxx (in which xxxx is the Shamela id of the book): contains the headings, with reference to the page numbers
List of tables in all bok files in the official Shamela corpus (October 2019):
| table name | column name | present in number of files | data in number of files |
|---|---|---|---|
| bxxxx | nass seal id page part hno na sora aya b1 b2 b3 b4 blnk ppart1 ppage1 ppart2 ppage2 ppart3 ppage3 ppart4 ppage4 done bhno | 7509 7509 7509 7509 7509 7509 1748 1027 1027 1027 17 1 1 1 16 40 40 7 7 7 7 2 2 3 1 | 7509 7509 7509 7509 7451 7449 1744 100 100 100 17 1 1 1 16 40 40 7 7 7 7 2 2 3 1 |
| Main | oauth auth bver oauthver over lng betaka aseal seal bk onum bkid ad cat islamshort authinf higrid pdfcs pdf inf shrtcs tafseernam vername blnk | 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 2664 7509 7509 7509 | 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7509 7508 7265 7265 6711 6480 5617 2124 183 16 16 |
| txxxx | sub id tit lvl | 7509 7509 7509 7509 7509 | 7507 7507 7507 7507 7507 |
| abc | a b c | 7509 7509 7509 7509 | 397 397 397 397 |
| Shorts | nass ramz bk | 7509 7509 7509 7509 | 341 341 341 341 |
| sPdf | part sfilename onum | 7509 7509 7509 7509 | 236 236 236 236 |
| men_u | id name bk | 7509 7509 7509 7509 | 148 148 148 148 |
| avPdf | cs def onum vername pdfver | 7509 7509 7509 7509 7509 7509 | 28 28 28 28 28 28 |
| men_b | id manid name bk | 7509 7509 7509 7509 7509 | 17 17 17 17 17 |
| nBound | d b dver bver bcode | 7509 7509 7509 7509 7509 7509 | 13 13 13 13 13 13 |
| oShr | matn sharhid matnid sharh | 7509 7509 7509 7509 7509 | 9 9 9 9 9 |
| oShrooh | matn matnver sharhver sharh | 7509 7509 7509 7509 7509 | 9 9 9 9 9 |
| Shrooh | matn sharhid matnid sharh | 7509 7509 7509 7509 7509 | 0 0 0 0 0 |
| com | id com bk | 7509 7509 7509 7509 | 0 0 0 0 |
| men_h | id name upg | 7509 7509 7509 7509 | 0 0 0 0 |
| 10759 | wrd pos | 1 1 1 | 1 1 1 |
| 10786 | wrd pos | 1 1 1 | 1 1 1 |
| 10772 | wrd pos | 1 1 1 | 1 1 1 |
| 10769 | wrd pos | 1 1 1 | 1 1 1 |
| 10773 | wrd pos | 1 1 1 | 1 1 1 |
-
openiti.new_books.convert.helper.bok.
close_db
(conn)¶ Close the database connection.
Parameters: conn – pypyodbc connection object. Returns: None
-
openiti.new_books.convert.helper.bok.
connect_to_db
(fp)¶ Connect to the mdb database in the bok file.
Uses pypyodbc (https://github.com/jiangwen365/pypyodbc)
Parameters: fp (str) – path to the bok file Returns: a pypyodbc connection object. Needs to be closed (conn.close()) at the end of the database session cur: a pypyodbc cursor object, needed to execute SQL Return type: conn
-
openiti.new_books.convert.helper.bok.
mdb2dict
(cur, VERBOSE=False)¶ Save the data tables in the mdb database in a dictionary
Parameters: - cur – a pypyodbc cursor object, needed to execute SQL
- VERBOSE (bool) – if True, table names will be printed
Returns: - a dictionary containing all tables in the database:
- key: table name
- value: list of dictionary representations for each row
- -> key: column header
- value: cell value
- (for current row and column)
Return type: d (dict)
-
openiti.new_books.convert.helper.bok.
print_table_and_column_names
(cur)¶ print a list of all tables (and their column headings) in the database
Parameters: cur – a pypyodbc cursor object, needed to execute SQL Returns: None
-
openiti.new_books.convert.helper.bok.
save_to_json
(d, fp)¶ Save the dictionary to a json file
Parameters: - d (dict) – a dictionary
- fp (str) – the path to the destination json file
Returns: None
openiti.new_books.convert.helper.html2md¶
Convert html to Markdown.
This program is an adaptation of python-markdownify (https://github.com/matthewwithanm/python-markdownify) to output OpenITI mARkdown. It also adds methods for tables and images, and a post-processing method.
The main componant of this script is the MarkdownConverter class, which contains a basic procedure for converting html, tag by tag, (MarkdownConverter.convert) and methods for converting specific html tags to mARkdown (MarkdownConverter.convert_a, MarkdownConverter.convert_img, …).
The easiest way to use the MarkdownConverter is to use the markdownify function, which calls the convert method of the MarkdownConverter class.
You can use this class as a base class and subclass it to add methods, adapt the post-processing method etc.
- E.g.:
- class Hindawi_converter(html2md.MarkdownConverter):
- def post_process_md(self, text):
- text = super().post_process_md(text) # remove blank lines marked with “DELETE_PREVIOUS_BLANKLINES” tag text = re.sub(r”
- +DELETE_PREVIOUS_BLANKLINES”, “”, text)
- # replace placeholders for spaces in tables: text = re.sub(“ç”, ” “, text) return text
Examples (doctests):
Headings: h1
>>> import html2md >>> h = '<h1>abc</h1>' >>> html2md.markdownify(h) '\n\n### | abc\n\n'
- NB: heading style is OpenITI mARkdown style by default,
- but can be set to other styles as well:
>>> h = '<h1>abc</h1>' >>> html2md.markdownify(h, md_style=UNDERLINED) '\n\nabc\n===\n\n'>>> h = '<h1>abc</h1>' >>> html2md.markdownify(h, md_style=ATX) '\n\n# abc\n\n'Paragraphs (<p>):
>>> h = "<p>abc</p>" >>> html2md.markdownify(h) '\n\n# abc\n\n'>>> h = "<p>abc</p>" >>> html2md.markdownify(h, md_style=ATX) '\n\nabc\n\n'Divs without class or with an unsupported class are stripped:
>>> h = 'abc <div>def</div> ghi' >>> html2md.markdownify(h) 'abc def ghi'>>> h = 'abc <div class="unknown_div_class">def</div> ghi' >>> html2md.markdownify(h) 'abc def ghi'Spans without class or with an unsupported class are stripped:
>>> h = 'abc <span>def</span> ghi' >>> html2md.markdownify(h) 'abc def ghi'>>> h = 'abc <span class="unknown_span_class">def</span> ghi' >>> html2md.markdownify(h) 'abc def ghi'Links:
>>> h = '<a href="a/b/c">abc</a>' >>> html2md.markdownify(h) '[abc](a/b/c)'Unordered lists:
>>> h = '<ul><li>item1</li><li>item2</li></ul>' >>> html2md.markdownify(h) '\n* item1\n* item2\n\n'Ordered lists:
>>> h = '<ol><li>item1</li><li>item2</li></ol>' >>> html2md.markdownify(h) '\n1. item1\n2. item2\n\n'Nested lists:
>>> h = '<ol><li>item1</li><li>item2:<ul><li>item3</li><li>item4</li></ul></li></ol>' >>> html2md.markdownify(h) '\n1. item1\n2. item2:\n\n\t* item3\n\t* item4\n\t\n\n'Italics (<i> and <em> tags):
>>> h = 'abc <em>def</em> ghi' >>> html2md.markdownify(h) 'abc *def* ghi'>>> h = 'abc <i>def</i> ghi' >>> html2md.markdownify(h) 'abc *def* ghi'Bold (<b> and <strong> tags):
>>> h = 'abc <b>def</b> ghi' >>> html2md.markdownify(h) 'abc **def** ghi'>>> h = 'abc <strong>def</strong> ghi' >>> html2md.markdownify(h) 'abc **def** ghi'Tables:
>>> h = ' <table> <tr> <th>th1aaa</th><th>th2</th> </tr> <tr> <td>td1</td><td>td2</td> </tr> </table>' >>> html2md.markdownify(h) '\n\n| th1aaa | th2 |\n| ------ | --- |\n| td1 | td2 |\n\n'# i.e., # | th1aaa | th2 | # | td1 | td2 |
-
openiti.new_books.convert.helper.html2md.
markdownify
(html, **options)¶ Convert html to markdown.
Calls the convert() method of the MarkdownConverter class.
-
class
openiti.new_books.convert.helper.html2md.
MarkdownConverter
(**options)¶ -
convert
(html)¶ Convert html to markdown.
# We want to take advantage of the html5 parsing, but we don’t actually # want a full document. Therefore, we’ll mark our fragment with an id, # create the document, and extract the element with the id.
-
convert_a
(el, text)¶ Convert html links to markdown-style links.
Example
>>> import html2md >>> h = '<a href="a/b/c">abc</a>' >>> html2md.markdownify(h) '[abc](a/b/c)'
-
convert_b
(el, text)¶ Convert <b> tags into markdown formatting
-
convert_blockquote
(el, text)¶ Convert <blockquote> tags into markdown formatting
-
convert_br
(el, text)¶ Convert <br/> tags into newline characters.
-
convert_em
(el, text)¶ convert <em> (italics) tags into markdown formatting.
-
convert_hn
(n, el, text)¶ Convert html headings (<h1>, <h2>, etc. into markdown formatting.
-
convert_i
(el, text)¶ convert <i> (italics) tags into markdown formatting.
-
convert_img
(el, text)¶ Convert <img> tags into markdown-style links to image files.
Examples
>>> import html2md >>> h = '<div><img class="figure" src="../Images/figure1.png" /></div>' >>> html2md.markdownify(h) '![](../Images/figure1.png)'
>>> html2md.markdownify(h, image_link_regex="../Images", image_folder="img") '![](img/figure1.png)'
-
convert_li
(el, text)¶ Convert list element tags <li>.
-
convert_list
(el, text)¶ Convert ordered and unordered html lists (<ul> and <ol> tags).
Examples
# unordered lists:
>>> import html2md >>> h = '<ul><li>item1</li><li>item2</li></ul>' >>> html2md.markdownify(h) '\n* item1\n* item2\n\n'
# ordered lists:
>>> import html2md >>> h = '<ol><li>item1</li><li>item2</li></ol>' >>> html2md.markdownify(h) '\n1. item1\n2. item2\n\n'
# nested lists:
###### TEST FAILS FOR UNKNOWN REASONS ##>>> import html2md ##>>> h = '<ol><li>item1</li><li>item2:<ul><li>item3</li><li>item4</li></ul></li></ol>' ##>>> html2md.markdownify(h) ##'\n1. item1\n2. item2:\n\n\t* item3\n\t* item4\n\t\n\n'
-
convert_ol
(el, text)¶ Convert ordered and unordered html lists (<ul> and <ol> tags).
Examples
# unordered lists:
>>> import html2md >>> h = '<ul><li>item1</li><li>item2</li></ul>' >>> html2md.markdownify(h) '\n* item1\n* item2\n\n'
# ordered lists:
>>> import html2md >>> h = '<ol><li>item1</li><li>item2</li></ol>' >>> html2md.markdownify(h) '\n1. item1\n2. item2\n\n'
# nested lists:
###### TEST FAILS FOR UNKNOWN REASONS ##>>> import html2md ##>>> h = '<ol><li>item1</li><li>item2:<ul><li>item3</li><li>item4</li></ul></li></ol>' ##>>> html2md.markdownify(h) ##'\n1. item1\n2. item2:\n\n\t* item3\n\t* item4\n\t\n\n'
-
convert_p
(el, text)¶ Converts <p> tags.
Examples
>>> import html2md >>> h = "<p>abc</p>" >>> html2md.markdownify(h) '\n\n# abc\n\n'
>>> h = "<p>abc</p>" >>> html2md.markdownify(h, md_style=ATX) '\n\nabc\n\n'
>>> h = "<p></p>" >>> html2md.markdownify(h, md_style=ATX) ''
-
convert_strong
(el, text)¶ Convert <b> and <strong> tags.
NB: convert_b refers to this same function
Examples
>>> import html2md >>> h = 'abc <strong>def</strong> ghi' >>> html2md.markdownify(h) 'abc **def** ghi'
>>> import html2md >>> h = 'abc <b>def</b> ghi' >>> html2md.markdownify(h) 'abc **def** ghi'
-
convert_table
(el, text)¶ Wrap tables between double new lines.
NB: conversion of the tables takes place on the tr level.
-
convert_tr
(el, text)¶ Convert table rows.
NB: rows are processed before the table tag is. Spaces to fill out columns are added in post-processing!
Examples
>>> import html2md >>> h = ' <table> <tr> <th>th1aaa</th><th>th2</th> </tr> <tr> <td>td1</td><td>td2</td> </tr> </table>' >>> html2md.markdownify(h) '\n\n| th1aaa | th2 |\n| ------ | --- |\n| td1 | td2 |\n\n'
- i.e.:
- th1aaa | th2 |td1 | td2 |
-
convert_ul
(el, text)¶ Convert ordered and unordered html lists (<ul> and <ol> tags).
Examples
# unordered lists:
>>> import html2md >>> h = '<ul><li>item1</li><li>item2</li></ul>' >>> html2md.markdownify(h) '\n* item1\n* item2\n\n'
# ordered lists:
>>> import html2md >>> h = '<ol><li>item1</li><li>item2</li></ol>' >>> html2md.markdownify(h) '\n1. item1\n2. item2\n\n'
# nested lists:
###### TEST FAILS FOR UNKNOWN REASONS ##>>> import html2md ##>>> h = '<ol><li>item1</li><li>item2:<ul><li>item3</li><li>item4</li></ul></li></ol>' ##>>> html2md.markdownify(h) ##'\n1. item1\n2. item2:\n\n\t* item3\n\t* item4\n\t\n\n'
-
create_underline_line
(text, pad_char)¶ Create a sequence of pad_char characters the same lenght as text.
-
fill_out_columns
(match)¶ Find the longest cell in a column; add spaces to shorter columns.
-
indent
(text, level)¶ Add tab indentation before text.
-
post_process_md
(text)¶ Post-processing operation to improve formatting of converted text.
-
post_process_named_entities
(match)¶ Reformat named entity matches to mARkdown named entity standard.
Named entities should be marked with @TAG@ tags in the converter (3 capital letters between ampersands), and end with a new line. This post-processing step then converts these temporary tags into OpenITI mARkdown format @TAGdd+:
- The first number after the @QUR tag refers to the number of letters following the tag that do not belong to the named entity (in this automatic step, this number will always be set to 0);
- the following number(s) refer(s) to the length of the entity in tokens
Examples
>>> import html2md >>> conv = html2md.MarkdownConverter() >>> conv.post_process_md("abc @QUR@ def ghi\njkl") 'abc @QUR02 def ghi jkl' >>> conv.post_process_md("abc @QUR@ def ghi\n~~jkl\nmno") 'abc @QUR03 def ghi\n~~jkl mno'
-
process_tag
(node, children_only=False)¶ Process each tag and its children.
-
should_convert_tag
(tag)¶ Check whether a tag should be converted or simply stripped
-
underline
(text, pad_char)¶ Underline text with pad_char characters (-, =, or +).
Parameters: - text (str) – the text within a tag, to be underlined
- pad_char (str) – the character used for the line (-, =, or +)
Returns: an underlined line of text
Example
>>> import html2md >>> html2md.MarkdownConverter().underline("123456789", "=") '123456789\n=========' >>> html2md.MarkdownConverter().underline("123456789 ", "=") '123456789\n========='
-
openiti.new_books.convert.helper.html2md_eShia¶
Convert eShia library html to OpenITI mARkdown.
This script subclasses the generic MarkdownConverter class from the html2md module (based on python-markdownify, https://github.com/matthewwithanm/python-markdownify), which uses BeautifulSoup to create a flexible converter.
The subclass in this module, GRARHtmlConverter, adds methods specifically for the conversion of books from the eShia library to OpenITI mARkdown:
- Span, div and p conversion: span, div and p classes needed to be converted
- are defined in self.class_dict.
Inheritance schema of the GRARHtmlConverter:
| MarkdownConverter | EShiaHtmlConverter |
|---|---|
| Options | (inherited) |
| DefaultOptions | (inherited) |
| __init__ | (inherited) |
| __getattr__ | (inherited) |
| convert | (inherited) |
| process_tag | (inherited) |
| process_text | (inherited) |
| fill_out_columns | (inherited) |
| post_process_md | (inherited) |
| should_convert_tag | (inherited) |
| indent | (inherited) |
| underline | (inherited) |
| create_underline_line | (inherited) |
| convert_a | (inherited) |
| convert_b | (inherited) |
| convert_blockquote | (inherited) |
| convert_br | (inherited) |
| convert_em | (inherited) |
| convert_hn | (inherited) |
| convert_i | (inherited) |
| convert_img | (inherited) |
| convert_list | (inherited) |
| convert_li | (inherited) |
| convert_ol | (inherited) |
| convert_p | convert_p |
| convert_table | (inherited) |
| convert_tr | (inherited) |
| convert_ul | (inherited) |
| convert_strong | (inherited) convert_span convert_div |
-
class
openiti.new_books.convert.helper.html2md_eShia.
EShiaHtmlConverter
(**options)¶ Convert EShia library html to OpenITI mARkdown.
Examples
>>> import html2md_eShia >>> h = '<img class="libimages" src="/images/books/86596/01/cover.jpg">' >>> html2md_eShia.markdownify(h) '![](img/86596/01/cover.jpg)'
>>> import html2md_eShia >>> h = 'abc <a href="www.example.com">def</a> ghi' >>> html2md_eShia.markdownify(h) 'abc def ghi'
-
convert_div
(el, text)¶ Converts html <div> tags, depending on their class attribute.
Supported div classes should be stored in self.class_dict (key: div class (str); value: formatting string)
Example
>>> import html2md_eShia >>> h = 'abc <div>def</div> ghi' >>> html2md_eShia.markdownify(h) 'abc def ghi'
>>> h = 'abc <div class="unknown_div_class">def</div> ghi' >>> html2md_eShia.markdownify(h) 'abc def ghi'
>>> h = 'abc <div class="list3">def ghi</div> jkl' >>> html2md_eShia.markdownify(h) 'abc \tdef ghi jkl'
-
convert_p
(el, text)¶ Converts <p> tags according to their class.
Supported p classes should be stored in self.class_dict (key: span class (str); value: formatting string) E.g., {“quran”: “@QUR@ {}n”}
<p> tags without class attribute, or unsupported class, will be converted according to the markdown style as defined in the self.options[“md_style”] value (from super().DefaultOptions)
Examples
>>> import html2md_eShia >>> h = "<p>abc</p>" >>> html2md_eShia.markdownify(h) '\n\n# abc\n\n'
>>> h = "<p>abc</p>" >>> html2md_eShia.markdownify(h, md_style=ATX) '\n\nabc\n\n'
>>> h = "<p></p>" >>> html2md_eShia.markdownify(h, md_style=ATX) ''
>>> h = '<p class="KalamateKhas">abc</p>' >>> html2md_eShia.markdownify(h) '\n\n### ||| abc\n\n'
-
convert_span
(el, text)¶ Converts html <span> tags, depending on their class attribute.
Supported span classes should be stored in self.class_dict (key: span class (str); value: formatting string) E.g., {“quran”: “@QUR@ {}n”}
Example
>>> import html2md_eShia >>> h = 'abc <span>def</span> ghi' >>> html2md_eShia.markdownify(h) 'abc def ghi'
>>> h = 'abc <span class="unknown_span_class">def</span> ghi' >>> html2md_eShia.markdownify(h) 'abc def ghi'
>>> h = 'abc <span class="Aye">def ghi</span> jkl' >>> html2md_eShia.markdownify(h) 'abc @QUR02 def ghi jkl'
>>> h = 'abc <span class="TextsStyles1">def ghi</span> jkl' >>> html2md_eShia.markdownify(h) 'abc @QUR02 def ghi jkl'
# the @QUR@ example outputs are a result of post-processing; # the function itself will produce: # ‘abc @QUR@ def ghinjkl’
>>> h = 'abc <span class="Titr3">def</span> ghi' >>> html2md_eShia.markdownify(h) 'abc\n\n### ||| def\n\nghi'
-
-
openiti.new_books.convert.helper.html2md_eShia.
markdownify
(html, **options)¶ Shortcut to the convert method of the HindawiConverter class.
openiti.new_books.convert.helper.html2md_GRAR¶
Convert GRAR library html to OpenITI mARkdown.
This script subclasses the generic MarkdownConverter class from the html2md module (based on python-markdownify, https://github.com/matthewwithanm/python-markdownify), which uses BeautifulSoup to create a flexible converter. The subclass in this module, GRARHtmlConverter, adds methods specifically for the conversion of books from the GRAR library to OpenITI mARkdown:
- span conversion: the GRAR html seems to be a conversion of tei xml;
- the tei data is often embedded inside the id of a span.
Inheritance schema of the GRARHtmlConverter:
| MarkdownConverter | GRARHtmlConverter |
|---|---|
| Options | (inherited) |
| DefaultOptions | (inherited) |
| __init__ | (inherited) |
| __getattr__ | (inherited) |
| convert | (inherited) |
| process_tag | (inherited) |
| process_text | (inherited) |
| fill_out_columns | (inherited) |
| post_process_md | post_process_md (appended) |
| should_convert_tag | (inherited) |
| indent | (inherited) |
| underline | (inherited) |
| create_underline_line | (inherited) |
| convert_a | (inherited) |
| convert_b | (inherited) |
| convert_blockquote | convert_blockquote |
| convert_br | (inherited) |
| convert_em | (inherited) |
| convert_hn | (inherited) |
| convert_i | (inherited) |
| convert_img | (inherited) |
| convert_list | (inherited) |
| convert_li | (inherited) |
| convert_ol | (inherited) |
| convert_p | (inherited) |
| convert_table | (inherited) |
| convert_tr | (inherited) |
| convert_ul | (inherited) |
| convert_strong | (inherited) convert_span |
-
class
openiti.new_books.convert.helper.html2md_GRAR.
GRARHtmlConverter
(**options)¶ Convert GRAR library html to OpenITI mARkdown.
-
convert_blockquote
(el, text)¶ Convert blockquote tags to mARkdown
NB: the @QUOTE@ tag is a temporary tag that will be removed in the post-processing step.
Examples
>>> import html2md_GRAR >>> h = 'abc <blockquote>def</blockquote> ghi' >>> html2md_GRAR.markdownify(h) 'abc def ghi'
>>> h = 'abc <span id="pb-21"/><blockquote>def</blockquote> ghi jkl' >>> html2md_GRAR.markdownify(h) 'abc\n\nPageV00P020\n\n# def ghi jkl'
-
convert_span
(el, text)¶ Converts html <span> tags, depending on their id attribute.
Example
>>> import html2md_GRAR >>> h = 'abc <span>def</span> ghi' >>> html2md_GRAR.markdownify(h) 'abc def ghi'
>>> h = 'abc <span class="unknown_span_class">def</span> ghi' >>> html2md_GRAR.markdownify(h) 'abc def ghi'
Page numbers (NB: mARkdown uses page end instead of page beginning)
>>> h = 'abc <span id="pb-21"/>def ghi jkl' >>> html2md_GRAR.markdownify(h) 'abc PageV00P020 def ghi jkl'
Sections:
>>> h = 'abc <span class="book" id="part-2 div1-2"/>def ghi jkl' >>> html2md_GRAR.markdownify(h) 'abc\n\n### | [book 2]\n\ndef ghi jkl'
>>> h = 'abc <span class="chapter" id="part-2 div2-1"/>def ghi jkl' >>> html2md_GRAR.markdownify(h) 'abc\n\n### || [chapter 1]\n\ndef ghi jkl'
>>> h = 'abc <span class="chapter" id="part-2 div2-1" title="Intro"/>def ghi jkl' >>> html2md_GRAR.markdownify(h) 'abc\n\n### || [chapter 1: Intro]\n\ndef ghi jkl'
-
post_process_md
(text)¶ Appends to the MarkdownConverter.post_process_md() method.
-
-
openiti.new_books.convert.helper.html2md_GRAR.
markdownify
(html, **options)¶ Shortcut to the convert method of the HindawiConverter class.
openiti.new_books.convert.helper.html2md_hindawi¶
Convert Hindawi library html to OpenITI mARkdown.
This script subclasses the generic MarkdownConverter class from the html2md module (based on python-markdownify, https://github.com/matthewwithanm/python-markdownify), which uses BeautifulSoup to create a flexible converter. The subclass in this module, HindawiConverter, adds methods specifically for the conversion of books from the Hindawi library to OpenITI mARkdown: * special treatment of <h4> heading tags * div classes “poetry_container”, “section”, “footnote” and “subtitle” * span class “quran”
The easiest way to use this is to simply feed the html (as string) to the markdownify() function, which will create an instance of the HindawiConverter class and return the converted string.
Examples (doctests):
Headings: h1 (from superclass html2md.MarkdownConverter)
>>> import html2md_hindawi >>> h = '<h1>abc</h1>' >>> html2md_hindawi.markdownify(h) '\n\n### | abc\n\n'
- NB: heading style is OpenITI mARkdown style by default,
- but can be set to other styles as well:
>>> h = '<h1>abc</h1>' >>> html2md_hindawi.markdownify(h, md_style=UNDERLINED) '\n\nabc\n===\n\n'>>> h = '<h1>abc</h1>' >>> html2md_hindawi.markdownify(h, md_style=ATX) '\n\n# abc\n\n'Headings: <h4>
- NB: in the Hindawi library, <h4> tag is used for section headings
- on all levels but the highest one. The section level must be derived from the id of the parent div.
>>> h = '<div class="section" id="sect2_4"><h4>abc</h4></div>' >>> html2md_hindawi.markdownify(h) '\n\n### ||| abc\n\n'>>> h = '<div class="section" id="sect5_2"><h4>abc</h4></div>' >>> html2md_hindawi.markdownify(h) '\n\n### |||||| abc\n\n'Poetry div, single-line:
>>> h = ' <div class="poetry_container line"> <div> <div>hemistich1</div> <div>hemistich2</div> </div> </div>' >>> html2md_hindawi.markdownify(h) '\n# hemistich1 %~% hemistich2\n'Poetry div, multiple line:
>>> h = ' abc <div class="poetry_container"> <div> <div>hemistich1</div> <div>hemistich2</div> </div> <div> <div>hemistich3</div> <div>hemistich4</div> </div> </div> def' >>> html2md_hindawi.markdownify(h) ' abc\n# hemistich1 %~% hemistich2\n# hemistich3 %~% hemistich4\ndef'Section div without heading:
>>> h = 'abc <div class="section" id="sect2_9">def</div> ghi' >>> html2md_hindawi.markdownify(h) 'abc\n\n### |||\ndef\n\nghi'Section div with heading (handled by h4):
>>> h = 'abc <div class="section" id="sect2_9"> <h4>title</h4> <p>def</p> </div> ghi' >>> html2md_hindawi.markdownify(h) 'abc\n\n### ||| title\n\n# def\n\nghi'Footnote divs:
>>> h = '<div class="footnote"><sup>1 </sup>footnotetext</div>' >>> html2md_hindawi.markdownify(h) '\n\nFOOTNOTE1 footnotetext\n\n'
- NB: FOOTNOTE is a tag that will be used to extract all footnotes
- in a next step.
Subtitle divs:
>>> h = '<h1>Title text</h1><div class="subtitle">Subtitle text</div>' >>> html2md_hindawi.markdownify(h) '\n\n### | Title text Subtitle text\n\n'Divs without class or with an unsupported class are simply stripped:
>>> h = 'abc <div>def</div> ghi' >>> html2md_hindawi.markdownify(h) 'abc def ghi'>>> h = 'abc <div class="unknown_div_class">def</div> ghi' >>> html2md_hindawi.markdownify(h) 'abc def ghi'Spans with class “quran”:
>>> h = 'abc <span class="quran">def ghi</span> jkl' >>> html2md_hindawi.markdownify(h) 'abc @QUR02 def ghi jkl'# the latter is a result of post-processing; # the function itself will produce: # ‘abc @QUR@ def ghinjkl’
Spans without class or with an unsupported class are stripped:
>>> h = 'abc <span>def</span> ghi' >>> html2md_hindawi.markdownify(h) 'abc def ghi'>>> h = 'abc <span class="unknown_span_class">def</span> ghi' >>> html2md_hindawi.markdownify(h) 'abc def ghi'Links:
>>> h = '<a href="a/b/c">abc</a>' >>> html2md_hindawi.markdownify(h) '[abc](a/b/c)'Unordered lists:
>>> h = '<ul><li>item1</li><li>item2</li></ul>' >>> html2md_hindawi.markdownify(h) '\n* item1\n* item2\n\n'Ordered lists:
>>> h = '<ol><li>item1</li><li>item2</li></ol>' >>> html2md_hindawi.markdownify(h) '\n1. item1\n2. item2\n\n'Nested lists:
>>> h = '<ol><li>item1</li><li>item2:<ul><li>item3</li><li>item4</li></ul></li></ol>' >>> html2md_hindawi.markdownify(h) '\n1. item1\n2. item2:\n\n\t* item3\n\t* item4\n\t\n\n'Italics (<i> and <em> tags):
>>> h = 'abc <em>def</em> ghi' >>> html2md_hindawi.markdownify(h) 'abc *def* ghi'>>> h = 'abc <i>def</i> ghi' >>> html2md_hindawi.markdownify(h) 'abc *def* ghi'Bold (<b> and <strong> tags):
>>> h = 'abc <b>def</b> ghi' >>> html2md_hindawi.markdownify(h) 'abc **def** ghi'>>> h = 'abc <strong>def</strong> ghi' >>> html2md_hindawi.markdownify(h) 'abc **def** ghi'Tables:
>>> h = ' <table> <tr> <th>th1aaa</th><th>th2</th> </tr> <tr> <td>td1</td><td>td2</td> </tr> </table>' >>> html2md_hindawi.markdownify(h) '\n\n| th1aaa | th2 |\n| ------ | --- |\n| td1 | td2 |\n\n'
-
class
openiti.new_books.convert.helper.html2md_hindawi.
HindawiConverter
(**options)¶ Convert Hindawi library html to OpenITI mARkdown.
-
convert_a
(el, text)¶ Converts html links.
Overwrites the MarkdownConverter.post_process_md() method. Introduces an exception for links between footnote markers and footnootes.
Example
>>> import html2md_hindawi >>> h = '<a href="a/b/c">abc</a>' >>> html2md_hindawi.markdownify(h) '[abc](a/b/c)'
>>> import html2md_hindawi >>> h = 'abc <a href="ftn1">1</a>' >>> html2md_hindawi.markdownify(h) 'abc [1]'
-
convert_div
(el, text)¶ Converts html <div> tags, depending on their class.
In the MarkdownConverter class, div tags are simply stripped away.
Examples
# no div class: tags are stripped off
>>> import html2md_hindawi >>> h = 'abc <div>def</div> ghi' >>> html2md_hindawi.markdownify(h) 'abc def ghi'
# unknown div class: tags are stripped off
>>> import html2md_hindawi >>> h = 'abc <div class="unknown_div_class">def</div> ghi' >>> html2md_hindawi.markdownify(h) 'abc def ghi'
# poetry single-line:
>>> h = ' <div class="poetry_container line"> <div> <div>hemistich1</div> <div>hemistich2</div> </div> </div>' >>> html2md_hindawi.markdownify(h) '\n# hemistich1 %~% hemistich2\n'
# poetry multiple line:
>>> h = ' abc <div class="poetry_container"> <div> <div>hemistich1</div> <div>hemistich2</div> </div> <div> <div>hemistich3</div> <div>hemistich4</div> </div> </div> def' >>> html2md_hindawi.markdownify(h) ' abc\n# hemistich1 %~% hemistich2\n# hemistich3 %~% hemistich4\ndef'
# section without heading:
>>> h = 'abc <div class="section" id="sect2_9">def</div> ghi' >>> html2md_hindawi.markdownify(h) 'abc\n\n### |||\ndef\n\nghi'
# section with heading (handled by h4):
>>> h = 'abc <div class="section" id="sect2_9"> <h4>title</h4> <p>def</p> </div> ghi' >>> html2md_hindawi.markdownify(h) 'abc\n\n### ||| title\n\n# def\n\nghi'
# footnote:
>>> h = '<div class="footnote"><sup>1 </sup>footnotetext</div>' >>> html2md_hindawi.markdownify(h) '\n\nFOOTNOTE1 footnotetext\n\n'
Paragraph block (similar to <p>):
>>> h = '<div class="paragraph-block">abc def ghi</div>' >>> html2md_hindawi.markdownify(h) '\n\n# abc def ghi\n\n'
>>> h = '<div class="paragraph-block"><p>abc def ghi</p></div>' >>> html2md_hindawi.markdownify(h) '\n\n# abc def ghi\n\n'
>>> h = '<div class="paragraph-block"></div>' >>> html2md_hindawi.markdownify(h) ''
>>> h = '<div class="paragraph-block"><div class="paragraph-block">abc</div></div>' >>> html2md_hindawi.markdownify(h) '\n\n# abc\n\n'
-
convert_h4
(el, text)¶ Converts <h4> header tags.
In the Hindawi library, <h4> tags are used for subsections on any level. The section level must be taken from the id of the parent div. NB: the ### | level in Hindawi is 0.
Example
>>> import html2md_hindawi >>> h = '<div class="section" id="sect2_4"><h4>abc</h4></div>' >>> html2md_hindawi.markdownify(h) '\n\n### ||| abc\n\n'
>>> h = '<div class="section" id="sect5_2"><h4>abc</h4></div>' >>> html2md_hindawi.markdownify(h) '\n\n### |||||| abc\n\n'
-
convert_hn
(n, el, text)¶ Converts headings in the usual way except h4 headings.
In the Hindawi library, <h4> tags are used for subsections on any level. The section level must be derived from the id of the parent div.
Example
>>> import html2md_hindawi >>> h = '<h1>abc</h1>' >>> html2md_hindawi.markdownify(h) '\n\n### | abc\n\n'
>>> h = '<h3>abc</h3>' >>> html2md_hindawi.markdownify(h) '\n\n### ||| abc\n\n'
>>> h = '<div class="section" id="sect5_2"><h4>abc</h4></div>' >>> html2md_hindawi.markdownify(h) '\n\n### |||||| abc\n\n'
-
convert_span
(el, text)¶ Converts html <span> tags, depending on their class attribute.
Supported span classes should be stored in self.span_dict (key: span class (str); value: formatting string)
Example
>>> import html2md_hindawi >>> h = 'abc <span>def</span> ghi' >>> html2md_hindawi.markdownify(h) 'abc def ghi'
>>> h = 'abc <span class="unknown_span_class">def</span> ghi' >>> html2md_hindawi.markdownify(h) 'abc def ghi'
>>> h = 'abc <span class="quran">def ghi</span> jkl' >>> html2md_hindawi.markdownify(h) 'abc @QUR02 def ghi jkl'
# the latter is a result of post-processing; # the function itself will produce: # ‘abc @QUR@ def ghinjkl’
-
get_section_level
(el)¶ Gets the level of the current section (or its parent).
-
-
openiti.new_books.convert.helper.html2md_hindawi.
markdownify
(html, **options)¶ Shortcut to the convert method of the HindawiConverter class.
openiti.new_books.convert.helper.md2html¶
Replace markdown tags by html tags.
TO DO: * replace ### |+ headers not by <hx> but by <div><hx>title</hx>content</div>?
-
openiti.new_books.convert.helper.md2html.
dict_units
(text)¶ Replace dictionary units mARkdown tags with html tags.
Examples
>>> import md2html >>> md2html.dict_units("### $DIC_NIS$ Name of the entry") '<div class="entry descr-name"><span class="entry-title">Name of the entry</span>\n</div>\n' >>> md2html.dict_units("### $DIC_TOP$ Name of the place") '<div class="entry toponym"><span class="entry-title">Name of the place</span>\n</div>\n' >>> md2html.dict_units("### $DIC_LEX$ Word") '<div class="entry lexical"><span class="entry-title">Word</span>\n</div>\n' >>> md2html.dict_units("### $DIC_BIB$ Book title") '<div class="entry book"><span class="entry-title">Book title</span>\n</div>\n' >>> md2html.dict_units("### $BIO_MAN$ Name of the man") '<div class="entry man"><span class="entry-title">Name of the man</span>\n</div>\n' >>> md2html.dict_units("### $BIO_WOM$ Name of the woman") '<div class="entry woman"><span class="entry-title">Name of the woman</span>\n</div>\n' >>> md2html.dict_units("### $BIO_REF$ Cross-reference to a person") '<div class="entry cross-ref"><span class="entry-title">Cross-reference to a person</span>\n</div>\n' >>> md2html.dict_units("### $BIO_NLI$ List of names") '<div class="entry name-list"><span class="entry-title">List of names</span>\n</div>\n' >>> md2html.dict_units("### $CHR_EVE$ Event description") '<div class="entry event"><span class="entry-title">Event description</span>\n</div>\n' >>> md2html.dict_units("### $CHR_RAW$ Events description") '<div class="entry events-batch"><span class="entry-title">Events description</span>\n</div>\n' >>> md2html.dict_units("### $ Name of the man") '<div class="entry man"><span class="entry-title">Name of the man</span>\n</div>\n' >>> md2html.dict_units("### $$ Name of the woman") '<div class="entry woman"><span class="entry-title">Name of the woman</span>\n</div>\n' >>> md2html.dict_units("### $$$ Cross-reference to a person") '<div class="entry cross-ref"><span class="entry-title">Cross-reference to a person</span>\n</div>\n' >>> md2html.dict_units("### $$$$ List of names") '<div class="entry name-list"><span class="entry-title">List of names</span>\n</div>\n'
-
openiti.new_books.convert.helper.md2html.
tables
(text)¶ Replace markdown table tags with html table tags.
Examples
>>> text = "# first paragraph\n| table header 1 | header 2 |\n|:---------------|----------|\n| table row 1 col 1 | table row 1 col 2|\n| table row 2 col 1 | table row 2 col 2|" >>> import md2html >>> md2html.tables(text) '# first paragraph\n<table>\n<tr>\n<th>table header 1</th><th>header 2</th>\n</tr>\n<tr>\n<td>table row 1 col 1</td><td>table row 1 col 2</td>\n</tr>\n<tr>\n<td>table row 2 col 1</td><td>table row 2 col 2</td>\n</tr>\n</table>\n'
openiti.new_books.convert.helper.tei2md¶
Convert TEI xml to Markdown.
This program contains a sub-class of html2md.MarkdownConverter, which in turn is an adaptation of python-markdownify (https://github.com/matthewwithanm/python-markdownify) to output OpenITI mARkdown.
IMPORTANT: since TEI indicates page beginning (<pb/>) and OpenITI mARkdown page numbers are at the bottom of a page, page numbers should be pre-processed in a text before feeding it to the markdownify function. The tei2md.preprocess_page_numbers function can be used for this.
You can use the tei2md.TeiConverter class as a base class and subclass it to add methods, adapt the post-processing method etc.
- E.g.::
- def Class GRAR_converter(tei2md.TeiConverter):
- def post_process_md(self, text):
text = super().post_process_md(text)
# remove blank lines marked with “DELETE_PREVIOUS_BLANKLINES” tag
text = re.sub(“\n+DELETE_PREVIOUS_BLANKLINES”, “”, text)
# replace placeholders for spaces in tables:
text = re.sub(“ç”, ” “, text) return text
This table shows the methods and classes the TeiConverter inherits from html2md.MarkdownConverter, which methods it overwrites, and which methods it adds:
| html2md.MarkdownConverter | tei2md.TeiConverter |
|---|---|
| class DefaultOptions | (inherited) |
| class Options | (inherited) |
| __init__ | (inherited) |
| convert | convert |
| process_tag | (inherited) |
| process_text | (inherited) |
| fill_out_columns | (inherited) |
| post_process_md | (inherited) |
| __getattr__ | (inherited) |
| should_convert_tag | (inherited) |
| indent | (inherited) |
| create_underline_line | (inherited) |
| underline | (inherited) |
| convert_a | (inherited) |
| convert_b | (inherited) |
| convert_blockquote | (inherited) |
| convert_br | (inherited) |
| convert_em | (inherited) |
| convert_hn | (inherited) |
| convert_i | (inherited) |
| convert_img | (inherited) |
| convert_list | (inherited) |
| convert_li | (inherited) |
| convert_p | (inherited) |
| convert_strong | (inherited) |
| convert_table | (inherited) |
| convert_tr | (inherited) convert_div1 convert_div2 convert_div3 convert_div convert_head convert_lg find_heading_level (dummy) |
Examples (doctests):
Specific tei-related tags:
>>> import tei2md >>> h = 'abc <div1 type="book" n="0" name="Preface">def</div1> ghi' >>> tei2md.markdownify(h) 'abc\n\n### | [book 0: Preface]\n\ndef\n\nghi'>>> h = 'abc <div2 type="section" n="1">def</div1> ghi' >>> tei2md.markdownify(h) 'abc\n\n### || [section 1]\n\ndef\n\nghi'>>> h = 'abc <div3 type="Aphorism">def</div1> ghi' >>> tei2md.markdownify(h) 'abc\n\n### ||| [Aphorism]\n\ndef\n\nghi'Divs without type are stripped:
>>> h = 'abc <div>def</div> ghi' >>> tei2md.markdownify(h) 'abc def ghi'<head> tags are converted to level-3 mARkdown headers by default:
>>> h = 'abc <head>def</head> ghi' >>> tei2md.markdownify(h) 'abc\n\n### ||| def\n\nghi'>>> h = 'abc <lb/>def <lb/>ghi' >>> tei2md.markdownify(h) 'abc\n~~def\n~~ghi'>>> h = ' abc <lg> <l>line1</l> <l>line2</l> <l>line3</l> <l>line4</l> </lg> def' >>> tei2md.markdownify(h) ' abc\n# line1\n# line2\n# line3\n# line4\n\ndef'In addition to these TEI tags, the converter also inherits methods from html2md.MarkdownConverter that deal with more standard html tags:
Headings: h1
>>> import tei2md >>> h = '<h1>abc</h1>' >>> tei2md.markdownify(h) '\n\n### | abc\n\n'
- NB: heading style is OpenITI mARkdown style by default,
- but can be set to other styles as well:
>>> h = '<h1>abc</h1>' >>> tei2md.markdownify(h, md_style=UNDERLINED) '\n\nabc\n===\n\n'>>> h = '<h1>abc</h1>' >>> tei2md.markdownify(h, md_style=ATX) '\n\n# abc\n\n'Paragraphs (<p>):
>>> h = "<p>abc</p>" >>> tei2md.markdownify(h) '\n\n# abc\n\n'>>> h = "<p>abc</p>" >>> tei2md.markdownify(h, md_style=ATX) '\n\nabc\n\n'Divs without type are stripped:
>>> h = 'abc <div>def</div> ghi' >>> tei2md.markdownify(h) 'abc def ghi'Spans without class or with an unsupported class are stripped:
>>> h = 'abc <span>def</span> ghi' >>> tei2md.markdownify(h) 'abc def ghi'>>> h = 'abc <span class="unknown_span_class">def</span> ghi' >>> tei2md.markdownify(h) 'abc def ghi'Links:
>>> h = '<a href="a/b/c">abc</a>' >>> tei2md.markdownify(h) '[abc](a/b/c)'Unordered lists:
>>> h = '<ul><li>item1</li><li>item2</li></ul>' >>> tei2md.markdownify(h) '\n* item1\n* item2\n\n'Ordered lists:
>>> h = '<ol><li>item1</li><li>item2</li></ol>' >>> tei2md.markdownify(h) '\n1. item1\n2. item2\n\n'Nested lists:
>>> h = '<ol><li>item1</li><li>item2:<ul><li>item3</li><li>item4</li></ul></li></ol>' >>> tei2md.markdownify(h) '\n1. item1\n2. item2:\n\n\t* item3\n\t* item4\n\t\n\n'Italics (<i> and <em> tags):
>>> h = 'abc <em>def</em> ghi' >>> tei2md.markdownify(h) 'abc *def* ghi'>>> h = 'abc <i>def</i> ghi' >>> tei2md.markdownify(h) 'abc *def* ghi'Bold (<b> and <strong> tags):
>>> h = 'abc <b>def</b> ghi' >>> tei2md.markdownify(h) 'abc **def** ghi'>>> h = 'abc <strong>def</strong> ghi' >>> tei2md.markdownify(h) 'abc **def** ghi'Tables:
>>> h = ' <table> <tr> <th>th1aaa</th><th>th2</th> </tr> <tr> <td>td1</td><td>td2</td> </tr> </table>' >>> tei2md.markdownify(h) '\n\n| th1aaa | th2 |\n| ------ | --- |\n| td1 | td2 |\n\n'
- i.e.:
th1aaa | th2 |td1 | td2 |
-
openiti.new_books.convert.helper.tei2md.
preprocess_page_numbers
(s)¶ Turn page beginnings into page endings.
TEI xml indicates the beginning of a page, while OpenITI mARkdown indicates the end of a page.
openiti.new_books.convert.helper.yml2json¶
Convert a YML file to json.
The yml2json function can be used to convert a yml metadata file to
- a list of dictionaries: useful for
- conserving the order
- conserving comments between the records
- a dictionary of dictionaries (key: record id, val: metadata dictionary), useful for easy lookup by record id
-
openiti.new_books.convert.helper.yml2json.
yml2json
(yml_fp, container=[], rec_start='##RECORD', id_key='01#BookID###', key_val_splitter=':::: ')¶ Convert a YML file to json.
Parameters: - yml_fp (str) – the filepath to the YML file.
- container (obj) – an empty list or dict into which the metadata dictionaries will be stored. If the container is a list, the order of the records will be stored as well as comment lines between the records.
- rec_start (str) – the starting characters of a record;
- id_key (str) – the yml key that indicates the id of a record
- key_val_splitter (str) – the separator used to separate keys and values.
Returns: the container filled with the metadata dictionaries.
Return type: container (obj)