--- layout: page title: About Me slug: about permalink: /about redirect_from: /blog/about ---
Hi. My name's Steve Dower and I'm a developer at Microsoft working on a range of tools for Python developers. I'm a core contributor and Windows expert for CPython, and am responsible for producing the Windows builds and installers. I also occasionally work on the Indent Guides for Visual Studio extension.
I obtained Bachelor's degrees in mechatronics engineering and computer science/software engineering in 2009 and a PhD in computer science in 2012 at Swinburne University of Technology, Australia.
You'll find links to my recent public speaking on the Speaking page, and can read about my earlier research work on my Research page.
I can be contacted by email at<EMAIL_ADDRESS>If you like, you can sponsor me through GitHub Sponsors. I don't live off this (right now), so don't feel at all obliged.


PROJECT GOALS
The goals of this project are to learn:
how to setup an AMP environment for development of web pages on Mac OS X 10.10 Yosemite system
how to design a star schema data warehouse
how to use Talend to create an ETL transformation
how to install Hadoop and setup a data warehouse within HDFS
how to create ETL to extract from a traditional OLTP database, transform, and load into a Hadoop data warehouse using Talend's built-in sqoop functionality
how to use Hive to extract data from a Hadoop data warehouse
how to install and use Tableau to generate dashboards and reports
evaluate the benefits of Hadoop, Talend, and Tableau


The Green Mile
Paul Edgecombe is the head of the death row at Cold mountain prison, each of whose prisoners once walk the "green mile" on their way to the execution site. Paul had seen a lot of prisoners and guards during his time on the job. However, the giant John Coffey, accused of a terrible crime, became one of the most unusual inhabitants of the block.


Contackt name
City
Country
Is Promoted
Maria Andres
Berlin
Germany
Anna Trujillo
Mexico D.F.
Mexico
Antonio Moreno
Mexico D.F.
Mexico
Tomas Hardy
London
UK


--- layout: post title: ! '#rails-contrib and rubyonrails-stacks' categories: - general author: David published: true date: 2007-05-22 16:11:00.000000000 +01:00 --- RailsConf gave birth to a number of new outlets for sub-communities within the Rails family:
#rails-contrib is a new IRC channel on freenode for contributors to Rails. The Rails core team will hang out there to answer questions, discuss patches, and generally interact with anyone doing implementation work for the Rails framework. It's not meant for general chatter, though. Or for how to use the framework. It's strictly for implementational issues and the contributors working on those.
rubyonrails-stacks is a new forum to discuss how we can get standardized set of images going for Rails that can be deployed on any Xen host or even EC2. I posted a welcome with more details on the forum.


In order to control the state of the game we use flags, variables and global states. They are used to add some conditions to some elements of the games dependending on the actual value. On this way we define concrete situations in the games, specifying what must appear on the scene, which actions we might be able to perform, etc.
Through the effects we can change the state of the flags and variables, modifying the concrete states in the game. This happen cause the changes on some flags and variables allow changing conditions. We can find those effects on the "Game State" group.
Besides, we can find the next effects groups:


The Black Engineers Society is an organization geared toward the professional development, academic success, and overall support for minority STEM majors here at the University of Maryland, College Park. Although our roots are in engineering, we welcome all STEM majors. Our organization is a chapter within the National Society of Black Engineers (NSBE). Our organization prides itself in providing unparralleled opportunities to succeed by working closely with the Center of Minorities and Engineering (CMSE). Our mission is to recruit, retain, and release qualified minorities in the fields of computer science, engineering, mathematics, and physical sciences.
As a chapter of NSBE, we aim to increase the number of culturally responsible Black engineers who excel academically, succeed professionally and positively impact the community.
We work closely with the Center for Minorities in Science and Engineering, which works to provide resources and support for minority students in STEM.
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Cold Fusion is an annual event held by the chapter to celebrate Black STEM achievements and our organization members talents. It is held every fall near the end of class and is a way for the chapter to unwind and have fun before finals. It is also a time to meet up and hang out with the other NSBE chapters in the area.
Heat exchange is BES annual cookout held at the end of each spring semester. It is a time to meet up with other chapters in the DMV area as well as mingle with the chapter here at Maryland. Heat Exchange is always a lot of fun with good music, good food, and great people.
Every year NSBE hosts an annual convention where NSBE chapters from all over the world come for professional development, networking, and to get internships. There are 100’s of companies looking for minority S.T.E.M majors to work for their companies. There are also speakers, workshops, and much more! It is truly an amazing opportunity and every members should go at least once!
Black Hack is an event unique to BES meant to expose members to concepts of computer science and programming. Black Hack events consist of tutoring sessions, group projects, presentations about software development, and technical interview practice. These various activities give students the opportunity to explore programming beyond the scope of the classroom. With a range of student experience levels, collaboration is strongly encouraged to promote community and teamwork.
Flight school is a program meant to prepare students for the Boeing Flight Competition at Nationals. This competition is meant to test the students understanding of aerodynamics as well as their presentation skills. Teams compete with chapters all over the country. The goal is to make a balsa wood glider that goes the farthest and create a presentation that impresses the judges who are all employee’s of Boeing. For more information about the competition and guidelines, please visit this link.
Stuck in Letters and Sciences and want to transfer into the engineering school? Clark School Initiative serves as a program to help students transfer into the engineering school. There are upperclassmen available to you give you advice and resources to ensure you transfer into the engineering school successfully. We work closely with the Center of Minorities and Engineering (CMSE) to provide you with the best possible advantage to transfer into the school.
Fantasy League is the primary retention program BES uses to encourage involvement and community within the organization. Near the beginning of the semester, teams are formed and compete with each other for prizes and bragging rights. Teams gain points by earning good grades, attending events, and submitting exams to the test bank among other things. At the end of the year, the winning team gets a prize!


Hi,
Some of my feeds have become corrupt showing negative values for today. It seems to happen to my KWH/D feeds showing no data anymore for the current day as well as in the my electric app with a power to kwh feed. It displays negative values for power used today as well as for the averages for daily, weekly, monthly and yearly.
I've found a thread from 2014 were people were having the same problem and it seems it still is not fixed in v9.0 RC2.
my 2 questions :
1) is there a way to fix the data or will it get fixed automaitcally tomorrow or so ?
2) how to prevent this from happening in the future ? i don't want to delete all my feeds every 6 months when daylight saving time happens, then emoncms becomes totally usesless for me as the main point for me is to have historical data. I don't use emoncms that long so i can still restart if there is a ways to prevent this from happening
here's a picture of the data of one of my power to kwh feeds:
some more info:
i'm running a local version of emoncms v9.0 rc2 on intel nuc running windows 7 using wamp. The data gets entered using the input api like so : 2015/10/25 13:05:18 http://<IP_ADDRESS>:8083/emoncmsnew/input/post.json?node=3&apikey=myapikey&csv=0.38,44.20,4.35,225.65
EDIT:
After some testing i found out that the script that was used to recalculate the power to kwh (after modifying it to make it work) feeds does work and all data is correct. However as soon as you post 1 new entry using the input api you can see it goes to negative values again as per this screenshot of the rawdata after i had run the script to recalculate this power to kwh feed.
My user's timezone is set to Europe/brussels, the same goes for apache's httpd.conf and php.ini all set to europe/brussels
Re: V9.0 RC2 Daylight saving time bug
Same here.
Paul
Self hosted on Raspberry Pi, v9.0RC
Re: V9.0 RC2 Daylight saving time bug
i managed to recalculate my kwh feed by using and first modifying the power_to_kwh_manual.php script from the usefullscripts repository. you have to remove all references to the log also in the proces\lib folder replace the logging with just an echo so you do see the output.
My kwh feed is now correct yet the my electric app for some reason still shows negative numbers. I already cleared cache and tried in 2 browsers.
Also that script only fixes my kwh feeds, it does not fix my kdh/d feeds
edit:
it seems recalculating has no effect as soon as i post data again it shows the same effect as in my earlier screenshot. And my electric keeps showing negative numbers so the recalculating script which worked in 2014 does not seem to work that good for now. it just goes back to negative but on a later timestamp (after i had recalculated the feed). Also my timezone is set to europe / brussels also in php.ini
edit 2: i've now set my timezone in apache httpd conf as well like so :
SetEnv TZ Europe/Brussels
this was not set in httpd.conf before. i'm not certain if this fixed it or if it got autofixed somehow but now my kwh feed seems to be on track again. The only problem i'm having seems to be getting rid of that dip. If i recalc using the recalc script i for some reason keep having the same dip in my graph data. my powerfeed is PHPFIWA don't know if that matters (i changed the source engine to this) the target engine is still PHPFINA
edit 3: the kwh feed above seems to be fixed now with me. the recalc script only deleted the target feed if the source feed was phpfina i changed that line to phpfiwa and now the kwd feed in above screenshot is fixed.However i have multiple power & power to kwh feeds in total 5 of each 2 of them seem to be fixed now also when posting new data, the 3 others are not and i'm clueless as to why some feeds are still not working while others are all of a suddenedit 4: it seems i was wrong, looking at the raw data i can see a dip each time after i recalculated the values using the script. The only reason it seemed it was showing correct data was because it had caught up with the values it had lost all my kwh feed still show the dip after recalculating. So nothing is fixed with me not even after recalculating
Re: V9.0 RC2 Daylight saving time bug
I can't even get the scripts to run!!
If I run power_to_kwh.php (in usefulscripts), I get:
If I run power_to_kwh_manual.php (again in usefulscripts) I get:
If I add that line, and re-run the script, I get:
Paul
Re: V9.0 RC2 Daylight saving time bug
you need to run the other script power_to_kwh_manual.php and edit the php script by removing the require log thingie as well als all $log-> entries (replace them with echo statements) you need to do this for all the files in the process directory including the lib folder. then the power_to_kwh_manual.php will work, i also changed the check for phpfina to phpfiwa where it would delete the files of the target feed and had set source engine to phpfiwa since well that's the case with me. then this script will work.
the other script i could not get working either i managed to fix the readline thing by adding this function but there are other problems in that script it's easier and faster (to edit) power_to_kwh_manual.php only thing you have to do is remove the log references at least that was the case with me since that logging thing doesn't seem to exist anymore
function readline($prompt = null){
if($prompt){
echo $prompt;
}
$fp = fopen("php://stdin","r");
$line = rtrim(fgets($fp, 1024));
return $line;
}
but as far as i can tell the recalculating of the feed with or without deleting the target feed (done by the script) does not seem to solve the problem, it fixes the feed up until you recalculated but new data just have a dip again
oh don't forget to change the variables to reflect your feeds and installation and be prepared to wait a long time, the more data you have the longer recalculation takes. i've only been using emoncms for like 2 weeks and it took about one minute to recalculate each feed and this was on a intel nuc with an i5 processor it will probably take a lot longer on a raspberry pi, possibly hour(s) depending on how much data you have
Re: V9.0 RC2 Daylight saving time bug
I've decoded the phpfiwa feed to see if there is something in there which could influence the script and subsequent phpfina feed, and all looks well.
1am 25/10/2015 GMT =<PHONE_NUMBER> UNIX time
1445734710 46
1445734720 46
1445734730 47
1445734740 46
1445734750 47
1445734760 46
1445734770 46
1445734780 47
1445734790 46
1445734800 47
1445734810 47
1445734820 46
1445734830 NAN
1445734840 47
1445734850 46
1445734860 46
1445734870 46
1445734880 47
1445734890 48
1445734900 47
Re: V9.0 RC2 Daylight saving time bug
the problem is not the script, it recalculates everything fine, the problem however is that if you start posting data again the power to KWH feed (phpfina) feed gets negative values:
this is from an export to csv of the power to kwh feed it<PHONE_NUMBER> = time zone: 25/10/2015 15:56:25 GMT+1:00 this is the time i last recalculated the power to kwh feed using the script as you can see negative values appear in it right after the recalculations when i start posting data again
1445784970,2.53
1445784975,2.53
1445784980,2.53
1445784985,2.53
1445784990,2.31
1445784995,2.10
1445785000,1.88
1445785005,1.67
1445785010,1.45
1445785015,1.24
1445785020,1.02
1445785025,0.81
1445785030,0.59
1445785035,0.38
1445785040,0.16
1445785045,-0.06
1445785050,-0.27
1445785055,-0.49
1445785060,-0.70
1445785065,-0.92
1445785070,-1.13
1445785075,-1.35
1445785080,-1.56
1445785085,-1.78
1445785090,-2.00
1445785095,-2.21
1445785100,-2.43
1445785105,-2.64
1445785110,-2.86
1445785115,-3.07
1445785120,-3.29
it goes all the way up to this
1445785565,-6.31
1445785570,-6.31
1445785575,-6.31
1445785580,-6.31
1445785585,-6.31
1445785590,-6.31
1445785595,-6.31
1445785600,-6.31
1445785605,-6.31
1445785610,-6.30
1445785615,-6.30
1445785620,-6.30
1445785625,-6.30
1445785630,-6.30
1445785635,-6.30
1445785640,-6.30
1445785645,-6.30
1445785650,-6.30
1445785655,-6.30
1445785660,-6.30
1445785665,-6.30
1445785670,-6.30
1445785675,-6.30
1445785680,-6.30
1445785685,-6.30
1445785690,-6.30
1445785695,-6.30
1445785700,-6.30
1445785705,-6.30
1445785710,-6.30
and then it starts going down again
if i stop sending data and recalculate using the script, the data is correct but as soon as i post new data it goes haywire again, i'll check immediatly to confirm it
edit: just recalculated using the script again at 18:56 GMT + 1 (europe brussels time) this is what i get now in the power feed after recalculation and posting a few data:
1445795465 = GMT: Sun, 25 Oct 2015 17:51:05 GMT my time zone is : 25/10/2015 18:51:05 GMT+1:00 (so not sure which<PHONE_NUMBER> related to, i used http://www.epochconverter.com/)
1445795450,2.63
1445795455,2.63
1445795460,2.63
1445795465,2.00
1445795470,1.37
1445795475,0.74
1445795480,0.10
1445795485,-0.53
1445795490,-1.16
1445795495,-1.79
1445795500,-2.42
1445795505,-3.05
1445795510,-3.68
1445795515,-4.31
1445795520,-4.94
1445795525,-5.57
1445795530,-6.20
1445795535,-6.20
1445795540,-6.20
1445795545,-6.20
1445795550,-6.20
1445795555,-6.20
1445795560,-6.20
1445795565,-6.20
1445795570,-6.20
1445795575,-6.20
1445795580,-6.20
1445795585,-6.20
1445795590,-6.20
1445795595,-6.20
1445795600,-6.20
same problem occurs but the problem only seems to happen on a later datetime stamp (after i ran the script) so i think the script itselve works fine and calculates everything fine but as soon as i post new data i'm having the problem again. i tested this by disabling my program that sends data, let the script recalculate and look at raw data graph feed, all looks ok, i thin started the program again to submit data and then i get the huge (negative) dip
Re: V9.0 RC2 Daylight saving time bug
i noticed something, emoncms, according to timestamps in the log seems to be thinking it's hour earlier:
my logs are posted at 19:33 but emoncms displays 18:33 in the log entry. could this be the problem that emoncms somehow thinks it's running one hour behind real time ?
look at the last log entry and the date time in my system tray
Re: V9.0 RC2 Daylight saving time bug
My log time & system time are identical.
Is your Setup > My Account > Timezone set correctly?
Paul
Re: V9.0 RC2 Daylight saving time bug
yes i have it set to europe/brussels as i live in belgium
hmm it seems that's not the problem, the log writes utc time so the hour diffrence is correct (i looked at the sources)
Re: V9.0 RC2 Daylight saving time bug
WARNING do this at your own rish or make backups, i noticed this trick only fixed the kwh feed looking at real time data, but if i use this feed in a bargraph feed to display kwh/d or wh/d the bargraph does not seem to correctly display data. This might be a bug with the bargraph and daylight saving time or it might be a problem with the way i "fixed" my kwh feeds
i found a way to fix the kwh feeds:
this is what i did :
1) stop sending data to emoncms
2) use the script to recalculate the kwh feed based on a power feed
3) start sending data again (wait 1 - 2 minutes)
5) use visualisation Editrealtime and select the kwh feed. You'll notice the recalculation went perfect but you get a dip at the end of the graph now (after calculation) like in this screenshot:
6) zoom in enough at the end of the graph so you can see the dip, select the last datapoint at the peak, were data was still valid, copy its value to clipboard
7) select a datapoint from the dip (lowest point) edit it's value so by pasting the value from step 6
8) wait a few minutes and show the editrealtime feed again. You' should notice that the kwh feed is back on track even after posting data but that you still see the dip in graph like this screenshot:
9) if you see the effect above after zooming in enough on the correct position stop sending data
10) use the script again to recalc the data
11) start sending data again, the feed is now fixed. There might however be a very tiny dip but not a huge one like before and this dip can be ignored
This is a workaround for the problems caused by daylight saving time but i do hope that there'll be a fix before the next daylight saving time. I don't want to do this every 6 months. At the moment it did not take long since i had not much data but the longer i'll be using emoncms the more data i'll have and the more time it would take to recalculate the feed. So a fix for this should really be made
to make the script work: i repalced the all reference to logging with echo statements and remove the require statement. also make sure that the target phpfina feed gets deleted before calculation by default the script only does this if your source feed is phpfina (which was not the case with me) just change that if check with whatever source engine you were using (for example phpfiwa)
Re: V9.0 RC2 Daylight saving time bug
this is another bug which happend due to daylight saving :
the kwhd feeds are somehow messed up as well at least i think it is cause the graph is showing oct 25th 2 times instead of oct 25th & oct 26th on the x asis. However if you hoover over the day it's displayed as 26th correctly so only the labels on the x axis are wrong
I don't know how to fix this but i'm going to convert my bargraphs which use kwhd feeds to kwh feeds with the delta option
edit:
using kwh feeds with delta option = 1 gives the problem on the x asis and for some reason the next day (26th is empty) so maybe my kwh feeds are not completly fixed or there are other problems due to daylight saving time:
if i disable this option to automatically adjust the time to the daylightsaving time the xaxis thing is fixed if i enable it again i see two times oct 25, how can this setting have an impact on what you see in the bargraphs ?? this means that the client where your viewing the emoncms website from has an impact on how the graphs are displayed
however the bargraph still does not show any data for oct 26 (today) as in the screenshot above when using my "fixed" kwh feed with delta = 1
the same problem with the option above happens in firefox also (if you restart it after changing the setting in windows) It hapens on a windows 10 as well as on a windows 7. I had a look at the flot javascript library and noticed all graphs are set to timezone browser, so it's using the client's timezone shouldn't this use the timezone from user account settings which is selected there instead of the timezone of computer visiting emoncms ? i noticed there is an option for datejs if timezone in the graph options is not set maybe that should be used to fix the x - axis problem with this setting not sure though it's just a guess
Re: V9.0 RC2 Daylight saving time bug
Can you show the input process list that writes to that feed, and the Admin/Server Information tab.
Also it only happen on phpfina? And are you using redis ?
Do you still have the logs close to 25/10/2015 01:59 to 01:00 when the clock reverted? Post a few minutes before and after.
Re: V9.0 RC2 Daylight saving time bug
this is my setup:
i have 4 smart plugs and i edited a program that can read the data from the plugs so it can now send the data to emoncms. I'm doing this using the input api like so : http://<IP_ADDRESS>:8083/emoncmsnew/input/post.json?node=3&apikey=myapikey&csv=0.38,44.20,4.35,225.65 for example using one node for each plug, each with 4 inputs: amp, watt, volt and kwh. This last input kwh is the elapsed kwh as calculated by the smart plugs, but i don't use this since it has a limit of 1000kwh and these are not the feeds i'm having problems with.
as far as i can remeber it is indeed the phpfina feed that has a problem although my kwh/d feed also had a problem on the day itselve.
also do note i don't see any negative values anymore since i had tried to fix my "calculated" kwh feed (not the kwh input) the calculated kwh feed does not show any errors anymore however the when displayed in a bargraph using delta = 1 they don't show info for today while my kwh/d feed still does (i had usuage today). so it could be the kwhd feeds have a problem as well (not really serieous since if the kwh feed / bargraphs are completely fixed i can switch to those and remove the kwhd feeds).
my timezone is europe brussels and i had set this in apache, php.ini and on my user page
here are the screenshots of the data you requested. I'm not 100% certain if this is the information you need. if you need more info please let me know what. emoncms runs on a intel nuc (x86 hardware) on windows 7 with timezone in windows set to europe / brussels using wamp
http://www.willemssoft.be/misc/serverinfo.jpg
http://www.willemssoft.be/misc/process1.jpg
http://www.willemssoft.be/misc/process2.jpg
unfortunatly i had disabled logging so i have no logs anymore, the reason i disabled logs was because it was growing too big, on one day it had grown 180 mb so i decided to disable logging. maybe paul might still have logs ?
edit:
when i ran the script to recalculate the power to kwh feeds it was logging warnings about datetimestamps starting before feed creation time might this be an indicator ?
the recalculation of the power to kwh feed worked but as i soon as i posted new data i had negative values again in the feed. I fixed this using a method described above, by editing one of the negative values to the last known correct positive value, this somehow made the phpfina feed (power to kwh) go back on track. i then recalculated again to remove the negative values so i believe my power to kwh feed is now showing correctly but it's the bargraph that's acting up if you use it with (this/) power to kwh feed. (it shows no data on today or depending on a windows setting for the 2nd october 25th) also the initial problem started on the night daylight saving time happend, the shift in dates is because i had recalculated / tried to fix my kwh feed a few times
the double days thing can be fixed by chaning a setting in windows (since the graphs use the timezone of the browser) i just don't understand why that setting has an impact, it's only a display bug i think for the x axis.
i don't think i'm using redis, (don't even know what that is) just using the input api the post data from an application that can read the data from the smart plugs : https://github.com/joyrider3774/plugctl )
Re: V9.0 RC2 Daylight saving time bug
Hi Chaviero.
My input process is:
Log to feed - Grid (Feed 13 - PHPFIWA)
Allow Positive
Log to feed - Power Imported (Feed 14 - PHPFIWA)
Power to kWh (Feed 41 - PHPFINA)
Reset to zero
Redis is not enabled
Emoncms Timezone is set Europe/London (the same as my Raspberry Pi).
I've only experienced the problem on PHPFINA
I've attached my emoncms log (extract)
Paul
Re: V9.0 RC2 Daylight saving time bug
in this video (which i made for a github issue i created) you can see that the double day on the x axis is fixed if you change a setting dealing with daylight saving time: (skip to the end as the first part of the video shows a problem where all graphs are using the browser's timezone instead of the timezone from the user settings page) https://youtu.be/Go-R1315tpU
Re: V9.0 RC2 Daylight saving time bug
Thanks,
Paul can you post a detailed printscreen of your feed 41 graph from 23h to 3h of 25oct to better see the minutes/seconds when the value goes down an then start going up.
When you say you only had problem with phpfina is because you only have this engine saving kwh data, so cant confirm kwh data withother engines, right?
I was able to replicate on paper that strange values from the log, it seems last feed time is offset by -3600s only during the hour before the dst change.
Bad time value comes from feed_model.php get_timevalue method.
It seems related to the way input last time is saved on DB, in text format. The error must be from conversion with strtotime($row['time']).
Redis is caching that time in linux format so i would not expect the issue with redis. Can anyone confirm?
Re: V9.0 RC2 Daylight saving time bug
O dear. Il start with the usefulscripts issue.
Id be interested to know which systems show the readline error using the process script:
PHP Fatal error: Call to undefined function readline() in /home/pi/usefulscripts/process/power_to_kwh.php on line 12
I tested that here on PHP 5.5.9 and it worked fine. Towards the end of the power_to_kwh.php script it forces a reload of the redis feed data for the feed:
// force a reload of the feeds table
if ($redis && $redis->exists("user:feeds:$userid")) {
$redis->del("user:feeds:$userid");
$redis->del("feed:lastvalue:$target");
}
This should stop the error where the corrected data just starts negative from the point of last reprocessing.
A more manual step process would be:
1. stop emonhub, emoncms-nodes-service and feedwriter
2. run the processor script.
3. clear redis $ redis-cli "flushall"
4. start emonhub, emoncms-nodes-service and feedwriter
But it would be really great to get the usefulscripts/process/power_to_kwh.php script working consistently for everyone.
Re: V9.0 RC2 Daylight saving time bug
Hi trystan, I had that problem as well on my intel nuc running windows 7 64 bit, using wamp. Here you can see the server info from the administration page:http://www.willemssoft.be/misc/serverinfo.jpg I could fix the readline error by adding a function that reads from stdin, which i had found on stackoverflow. However this was not the only problem i had, at a certain point the script tries to read the interval from the feed and displays, you could then choose that existing interval or change it. The problem i had was that it displayed array instead of an interval number. I can give you the line nr as soon as im at work. Thats where i stopped and focussed on the manual script which i could get working. Also for some reason it had problems with require statement of the emonlogger, i did not fully look into this, i just replaced writing to log to echo statements. It could very well be i had placed the scripts in a wrong directory Cheers davy
edit: forgot to mention there is another bug in the script at least i believe so:
if you look at this section : https://github.com/emoncms/usefulscripts/blob/master/process/power_to_kw...
there's a missing $row = $result->fetch_array(); $row is still pointing to the source feed not the target feed with that call. unless the source feed has to be PHPFINA which is not the case with me it's PHPFIWA. If this check needs to check the target feed the fetch array statement is needed. (i'm guessing the target power to kwh feed needs to be phpfina but not the source feed)
the line number where i had gotten array displayed is this one https://github.com/emoncms/usefulscripts/blob/master/process/power_to_kw...
don't know why though
Re: V9.0 RC2 Daylight saving time bug
@Chaveiro.
Graph attached.
I do have another 'Power to kWh' PHPFINA feed (21) which records 'solar' power generation from feed 20. This feed did not drop and appears unaffected, however as it's a solar feed, there would be no increments during the affected time - as it was nightime = no power (could this be the reason why it's OK??)
Paul
Re: V9.0 RC2 Daylight saving time bug
@Trystan
Attached screenshot of my system, which is a Raspberry Pi v2 running Raspbian Wheezy with php v5.4.44. I've also attached a screenshot of the error.
Paul
EDIT.
running $ sudo apt-get update && sudo apt-get upgrade brings me to php v5.4.45 and the script still errors as above.
Re: V9.0 RC2 Daylight saving time bug
thanks @joyrider3774, I've fixed the power_to_kwh.php script with the stdin function, the fetch_array call and the interval print output where the source engine is phpfiwa. Let me know if it works ok now.
Re: V9.0 RC2 Daylight saving time bug
Still errors after git pull.
Paul
Re: V9.0 RC2 Daylight saving time bug
The location from emonlogger has changed to lib/emonlogger.php instead of modules/log/emonlogger.php Im also not certain if it is required to place the script in the root dir of emoncsms www dir after that change.
i'm also not able to test the script until i get home tonight, for some reason i can't reach my intel nuc from work today
Re: V9.0 RC2 Daylight saving time bug
Thanks Paul, could you try again now?
Re: V9.0 RC2 Daylight saving time bug
@Trystan
Git pulled latest and script ran OK, no errors and looking at the resultant kwh feed before restarting emonhub it looked good, showing a steady daily increase. However, I took the precaution of flushing redis, and as soon as I started emonhub, the kwh feed added 79kwh to the feed!
Tried it again, same result, so ran a third time, started emonhub and stopped it again after 15 seconds. Then using Visualisation > EditRealTime, I edited the +79kwh datapoints back to correct value, and restarted emonhub again.
This time, the resultant graph is OK! - got there in the end!!
Paul
Re: V9.0 RC2 Daylight saving time bug
the scripts are running fine with me also except for one windows related thing. The exec(chown) stuff does not work nor is it needed on windows machines, it spits out a warning that it can't find chown. However i think the script executed further on or at least i think so. i'm still in the process of recalculating everything and will test the bargraphs again afterwards to see if they still have problems
Re: V9.0 RC2 Daylight saving time bug
the bargraphs still display wrong even after recalculating. this picture is with default windows settings based on a kwh feed (that i just recalculated), as you can see it displays 2x oct 25th on the xaxis and no data for today
changing this windows setting (translated autoadjust time to daylight saving time): fixes the x axis display bug but does not fix the no data display on the current day:
however i believe this might all be related to to using "timezone: browser" on the graph settings (it isn't using timezone from user settings page from emoncms for the graphs in javascript) otherwise the windows setting change whould have no impact. however i'm not sure if the no data on current day is related to this, but the double days on the x axis probably is (why would changing windows settings otherwise have an impact)
paul could you create a bargraph with same settings and see if this has the same effect with you ? if you don't mind testing
EDIT:
just downloaded https://github.com/mde/timezone-js and the timezone files as per the readme, i added date.js to the bargraph.php and changed timezone:browser to europe/london and europe/brussels for testing so that timezone-js is used in combination with the specified timezone (libflot has support for this) it does not fix the double day display bug nor the no data. so i guess i was wrong that timezone: browser is the culprit, it is not as far as i tested and windows settings still affect graphs (like the setting described above). setting another timezone like nort america as a graph option (instead of browser) only shifted the x axis days a bit (they were not aligned anymore with the graph). so using timezone.js does not seem to fix it sorry for misleading
i just changed the windows setting to fix the x axis thing for now and am still using kwhd graphs (instead of kwh with delta = 1) so i can see the usuage on the current day as well
Re: V9.0 RC2 Daylight saving time bug
The duplicated day on the graph is something wrong with the graph javascript.
It's an old bug of flot lib, see : https://code.google.com/p/flot/issues/detail?id=141
A quick test with 'useLocalTime: true' seem to work, but i'd like to do further tests to understand how it works.
Re: V9.0 RC2 Daylight saving time bug
adding 'useLocalTime: true' and not deleting 'timezone:browser' has no impact with me. However removing 'timezone:browser' completely and not adding 'useLocalTime: true' fixes it with me. not adding 'timezone:browser' and only adding 'useLocalTime: true' fixes it also but i don't see a diffrence with just leaving out timezone:browser.
i checked jquery.flot.time.min.js and if you set timezone:browser it returns new Date(ts) where ts is the timestamp if you do not specify timezone it's returning makeUtcWrapper(new Date(ts)) where ts is the timestamp. i placed 2 alert statements in the if statements to know which one it was using.
Did you omit timezone:browser as well or did you specify timezone:browser along with uselocaltime:true cause that last one did not fix it with me. ps i only tested with bargraphs, i'll have a quick look on realtime graphs to see if the hour offsets or not
leaving out timezone option altogheter makes it use utc time on a daily bargraph it fixes the x axis display however a realtime graph my hour is off by one since i'm utc + 1 it's probably using utc, it can also be seen in the if statements in that jqeury.flot.time.min.js file. there's one other option that i tried yesterday and that's using timezoneJS if you add the library download the timezone files, initialize the library and write the actual timezone as timezone option for the xaxis but that did not fix it either with me
edit: crap i probably have to apply the patch from the google code link first cause as far as i can see a grep statement on all libflot files for uselocaltime does not return any info. I'll try to apply the patch if it still works and see what it does
Re: V9.0 RC2 Daylight saving time bug
Yes i omit 'timezone:browser' so it's using makeUtcWrapper, right.
The server time cames always in utc format but my current local time is also utc, so i'm not sure if removing it gives fixes because of that.
Need to dig further. If you look at \flot\examples\axes-time-zones there is some helpfull information.
Re: V9.0 RC2 Daylight saving time bug
thats what i tried yesterday but i notice the tz directory contains diffrent files as when i was testing it yesterday so maybe it did not work because of that. It makes use of https://github.com/mde/timezone-js . i'll try again but this time with the tz directory from the examples
edit tried with date.js and tz files from examples directory and setted timezone to europe/brussels it does not fix the double days thing, however the windows setting has no effect anymore on the x axis labels so it's probably a bug in the flot lib somewhere.
maybe timezone-js can be used to lock the x axis time bar to the time zone that is used in the emoncms user settings page but without windows settings having an impact on it
i also noticed flot lib is not being developped anymore last commit was somewhere in 2014 and i could not find a fork that might have continued work (github won't display the fork graph)
Re: V9.0 RC2 Daylight saving time bug
Check EmonCMS 9.1 | 2015.10.30. See if its ok.
Re: V9.0 RC2 Daylight saving time bug
seems to be fixed with me and does not seem to have any impact on other graphs where realtime data is displayed. As far as i can tell, this fixes it
Re: V9.0 RC2 Daylight saving time bug
@tristan i noticed something today. While looking at my kwh feed i still noticed a small drop so today i recalculated the feed again and when i start sending data again the drop occurs again.
You said in an earlier post that this
// force a reload of the feeds table
if ($redis && $redis->exists("user:feeds:$userid")) {
$redis->del("user:feeds:$userid");
$redis->del("feed:lastvalue:$target");
}
should stop the error where the corrected data just starts negative from the point of last reprocessing but acutally i'm not using redis so those lines of code are not executed with me.
Could this be why i still see a (very) small drop in de kwh feed data after recalculating ? I also tried fixing it manually again by editing one value of the drop to the last known correct value to counteract the drop. I had done this with all my previous kwh feeds as well so i'm not certain if the script fixed it or my manual messing about with node values actually fixed the drop since the code above is never executed with me. So maybe the script did not fix my drop after recalculating but me messing with the node values in editrealtime visulisation:
it's only a 0.05 drop though so i can safely ignore it but i was still wondering if the script actually fixed it or not and if it's normal that i still see a small dip while this normally should never happen with a power to kwh feed ?
Also i'm using 9.1 from 30.10.2015 but not sure if that matters
Re: V9.0 RC2 Daylight saving time bug
It shouldn't happen, and I found the same, but got around it as described here.
Paul
Re: V9.0 RC2 Daylight saving time bug
ah yes so also editing the data points, i'm guessing i'm seeing this small 0.05 because i had selected the wrong datapoint to counteract this behaviour i'll try to fix them manually again
Re: V9.0 RC2 Daylight saving time bug
If you rebuild your feed again, start emonhub and stop it again a few seconds later, you should only have a few datapoints to edit (that's what I found anyway!)
You may find that without rebuilding it again, you'll have 3 million datapoints to edit!
Paul
Re: V9.0 RC2 Daylight saving time bug
@tristan,
i think i figured out why the feeds keep dropping after recalculating. I had a problem with one my kwh feeds to day and had to recalculate again, and i had the drop problem again. Even after editing the nodes as paul suggested i still had problems with the feed, the editing of the values did not work with me.
the problem, i think, is that the value column of the target feed in mysql is never updated. If i add
$mysqli->query("update feeds set value= '".$kwh."' WHERE id='".$target."'"); at the end of the script (after the redis stuff) the drop never happens after i start sending data again. So my guess is the script, at the moment is incomplete because it does not update the value colum of the target feed in mysql. At least with me this seems to stop the drops from happening
edit: this is probably only needed if mysql is used to get the last kwh value for doing the increment calculation to write the new value. which is the case with me since i'm not using MQTT nor redis
Re: V9.0 RC2 Daylight saving time bug
Thanks @joyrider3774, I have added this line to the bottom of the script. latest commit info is here: https://github.com/emoncms/usefulscripts/commit/7a395d042517470adaa7f4dd...
Re: V9.0 RC2 Daylight saving time bug
@trystanlea no problem. It might be this only fixes it if one isn't using redis. the power_to_kwh calculation requests the last value using the get_timevalue function and this function first checks if redis is available and if so takes the value from there if redis is not availible it takes the value from mysql since i was not using redis it got the value from mysql feeds table with me. So it fixes it in this circumstance but i'm not certain if it would fix it if someone would be using redis. Unless the deletion of the last value on redis (last part of the script) had already solved it. I could not test that since i'm not using redis


About Consumescape
Consumescape tries to help you when you are faced with the decision to buy something new. Forget times where you could not concentrate on your work because you were stuck contemplating on whether you really need those new shoes or whether you should splurge on buying the latest camera or newest smartphone. Consumescape combines the words to consume and escape, because we believe that it is truly possible to escape the consuming cycle we may find ourselves in by taking some time in the decision process. This is based on the principle of the 30 day rule, a technique that was mainly introduced to save money.
This is how it works:
Instead of giving in to buying something impulsively, you back out and wait for 30 days. If you are still convinced that you need to buy that product after 30 days have passed, you'll probably won't regret buying it afterwards. If on the other hand you haven't spent a minute thinking something like "oh man, it would be really cool if I would have this right now, I can't wait to use it after these 30 days have passed" then we congratulate you for saving money, and most importantly ressources that are used to build this product, and saving the waste that it would have created once it won't be used anymore.
In summary, it just takes a little time and patience to make a little decision that could have a great impact on yourself and the environment.
But because it takes up a lot of mental load just to remember that we wanted to think about buying something, we implemented a timer after our questionnaire so that we will take care of that for you, and you can go on in living your life without any worries related to your buying behavior.


Instructions to Create a New Webpage on the Production Server
Create a 2 file in your repository by typing "gedit" in your terminal and enter
The first one is index.html and another one is main.css
When the file is open type in the basic code of html
You can type in the name of your new webpage in the title section of yur html code, and type in your header in the header section
To see your new webpage, you have to add your html file into github first.
To add you file into github, you do the git process thing, where you type "git add" and then your file name and "git commit" and lastly, git push, then your type in your username and password.
In your terminal type "python -m SimpleHTTPServer" and enter.
Open new tab and type "localhost:8000" and you should see your new webpage.
If you want to edit your texts, like change your font color or size, you can do it in your main.css file. For example, if you want to change the size of your text then type in "font-size:".
5 Links to Websites With Useful Web Design Related Information.


You are free to use, copy, or modify this software provided it remains free of charge for all users, the source remains open and unobfuscated, and the author, copyright, license, and warranty information remains intact and clearly visible.
All content is provided as-is. The user assumes all liability for any direct or indirect damages from usage.
All icons are copyright by their authors and used by OCP within their license.
© Copyright 2009-2010 Richard Doll, All Rights Reserved.
For support or suggestions, you can email me or use one of the feedback methods listed in my UESP user page.
Many thanks to those who freely published Oblivion's details like some FAQs at GameFAQs and the awesome compendium of knowledge maintained at Unofficial Elder Scroll Pages (UESP).
© Copyright 2005-2010 The Dojo Foundation, All Rights Reserved.
© Copyright 2006-2010 Bethesda Softworks LLC, All Rights Reserved.
Home Page: http://www.elderscrolls.com/
All race, birthsign, and class images and descriptions were derived from in-game or web site resources.


My favourite pet is my Mom's cat, Norman. He is about 6 years old and he is very needy. He likes treats and attention. He does not like my mom's bunny, Dexter. Norman is very inbred and only has one testicle but it's okay because he's very cuddly. His name is Norman because he just looked like a Norman. When Norman goes outside he likes to lay in the grass and catch butterflys. He loves to be scritched on the cheek.


| Package | Description |
|---|---|
| org.joda.time |
Provides support for dates, times, time zones, durations, intervals, and
partials.
|
| org.joda.time.base |
Implementation package providing abstract and base time classes.
|
| Modifier and Type | Interface and Description |
|---|---|
interface |
ReadWritableInterval
Writable interface for an interval.
|
| Modifier and Type | Class and Description |
|---|---|
class |
Interval
Interval is the standard implementation of an immutable time interval.
|
class |
MutableInterval
MutableInterval is the standard implementation of a mutable time interval.
|
| Modifier and Type | Method and Description |
|---|---|
static ReadableInterval |
DateTimeUtils.getReadableInterval(ReadableInterval interval)
Gets the interval handling null.
|
| Modifier and Type | Method and Description |
|---|---|
boolean |
Interval.abuts(ReadableInterval interval)
Does this interval abut with the interval specified.
|
void |
MutablePeriod.add(ReadableInterval interval)
Adds an interval to this one by dividing the interval into
fields and calling
MutablePeriod.add(ReadablePeriod) . |
void |
ReadWritablePeriod.add(ReadableInterval interval)
Adds an interval to this one by dividing the interval into
fields and then adding each field in turn.
|
boolean |
ReadableInterval.contains(ReadableInterval interval)
Does this time interval contain the specified time interval.
|
static Days |
Days.daysIn(ReadableInterval interval)
Creates a
Days representing the number of whole days
in the specified interval. |
Interval |
Interval.gap(ReadableInterval interval)
Gets the gap between this interval and another interval.
|
static Chronology |
DateTimeUtils.getIntervalChronology(ReadableInterval interval)
Gets the chronology from the specified interval object handling null.
|
static ReadableInterval |
DateTimeUtils.getReadableInterval(ReadableInterval interval)
Gets the interval handling null.
|
static Hours |
Hours.hoursIn(ReadableInterval interval)
Creates a
Hours representing the number of whole hours
in the specified interval. |
boolean |
ReadableInterval.isAfter(ReadableInterval interval)
Is this time interval entirely after the specified interval.
|
boolean |
ReadableInterval.isBefore(ReadableInterval interval)
Is this time interval entirely before the specified interval.
|
static Minutes |
Minutes.minutesIn(ReadableInterval interval)
Creates a
Minutes representing the number of whole minutes
in the specified interval. |
static Months |
Months.monthsIn(ReadableInterval interval)
Creates a
Months representing the number of whole months
in the specified interval. |
Interval |
Interval.overlap(ReadableInterval interval)
Gets the overlap between this interval and another interval.
|
boolean |
ReadableInterval.overlaps(ReadableInterval interval)
Does this time interval overlap the specified time interval.
|
static Seconds |
Seconds.secondsIn(ReadableInterval interval)
Creates a
Seconds representing the number of whole seconds
in the specified interval. |
void |
MutableInterval.setInterval(ReadableInterval interval)
Sets this interval to be the same as another.
|
void |
ReadWritableInterval.setInterval(ReadableInterval interval)
Sets this interval to be the same as another.
|
void |
MutablePeriod.setPeriod(ReadableInterval interval)
Sets all the fields in one go from an interval using the ISO chronology
and dividing the fields using the period type.
|
void |
ReadWritablePeriod.setPeriod(ReadableInterval interval)
Sets all the fields in one go from an interval dividing the
fields using the period type.
|
static Weeks |
Weeks.weeksIn(ReadableInterval interval)
Creates a
Weeks representing the number of whole weeks
in the specified interval. |
static Years |
Years.yearsIn(ReadableInterval interval)
Creates a
Years representing the number of whole years
in the specified interval. |
| Modifier and Type | Class and Description |
|---|---|
class |
AbstractInterval
AbstractInterval provides the common behaviour for time intervals.
|
class |
BaseInterval
BaseInterval is an abstract implementation of ReadableInterval that stores
data in two
long millisecond fields. |
| Modifier and Type | Method and Description |
|---|---|
boolean |
AbstractInterval.contains(ReadableInterval interval)
Does this time interval contain the specified time interval.
|
boolean |
AbstractInterval.isAfter(ReadableInterval interval)
Is this time interval entirely after the specified interval.
|
boolean |
AbstractInterval.isBefore(ReadableInterval interval)
Is this time interval entirely before the specified instant.
|
boolean |
AbstractInterval.isEqual(ReadableInterval other)
Is this interval equal to the specified interval ignoring the chronology.
|
boolean |
AbstractInterval.overlaps(ReadableInterval interval)
Does this time interval overlap the specified time interval.
|
Copyright © 2002–2020 Joda.org. All rights reserved.


nlp
Contents
5.2. nlp#
Some notes on natural language processing, focused on modern improvements based on deep learning. See also notes in 📌 transformers.
5.2.1. benchmarks#
<IP_ADDRESS>. tasks#
Nice repo keeping track of progress here.
tokenization - convert raw text into tokens that can be fed to a model
token/word embeddings - convert tokens to semantic vectors
pos tagging - give each word its part-speech
named entity recognition - classify tokens based on some known annotations (e.g. person, organization)
nested entity recognition - more complex than individual tokens (e.g. “Mary = “Jacob’s brother”)
parse tree extraction - extract out tree of how words operate in a grammar
sentiment classification
text summarization
language modeling (i.e. text generation)
(machine) translation
coreference resolution - correspond simple entities (e.g. “she = Mary”, different names for a protein)
question answering
natural language inference - determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”
topic modeling - unsupervised learning over documents
<IP_ADDRESS>. datasets#
open-ended
MMLU: Measuring Massive Multitask Language Understanding (hendrycks, …, song, steinhardt, 2021) - 57 tasks such as elementary math, clinical knowledge, global facts
NLI: allenai natural language instructions (v1: mishra et al. 2022; v2: wang et al. 2022) - 1600+ diverse NLP tasks
Definition which describes the task
For each example: (input text, output (varies), explanation*) - explanation only for some tasks
Big-bench (2022) - crowdsourced diverse (but random) NLP tasks
No task definitions or explanations: the “definition” is instead concatenated to each example’s input
Unveiling Transformers with LEGO: a synthetic reasoning task (zhang, …, bubeck, …, wagner, 2022) - synthetic reasoning task
Large Language Models Still Can’t Plan (A Benchmark for LLMs on Planning and Reasoning about Change)
very popular
SuperGLUE Benchmark (2019)
8 NLU tasks (keeps two hardest from GLUE, marked with ***) extending to coreference resolution and question-answering
BoolQ (Boolean Questions) - answer a yes/no question about a text passage.
CB (CommitmentBank) - given a text and a clause, predict how much the text commits to the clause
COPA (Choice of Plausible Alternatives) - given a premise sentence and two possible choices, the system must determine either the cause or effect of the premise from two possible choices
MultiRC (Multi-sentence Reading Comprehension) - given a context paragraph, a question about that paragraph, and a list of possible answers, the system must predict which answers are true and which are false
ReCoRD (Reading Comprehension with Commonsense Reasoning Dataset) - multiple-choice QA task (news article + Cloze-style multiple-choice question)
\(^\dagger\)RTE (Recognizing Textual Entailment) - determine if a sentence entails a given hypothesis or not.
WiC (Word-in-Context) - word sense disambiguation task cast as binary classification of sentence pairs. Given two text snippets and a polysemous word that appears in both sentences, the task is to determine whether the word is used with the same sense in both sentences
\(^\dagger\)WSC (Winograd Schema Challenge) - coreference resolution task in which examples consist of a sentence with a pronoun and a list of noun phrases from the sentence. The system must determine the correct referent of the pronoun from among the provided choices.
GLUE Benchmark (2019) = General language understanding evaluation
9 NLU tasks including question answering, sentiment analysis, text similarity and textual entailment
single-sentence tasks
CoLA (Corpus of Linguistic Acceptability) - determine if sentence is grammatically correct
SST-2 (Stanford Sentiment Treebank) - determine if the sentence has a positive or negative sentiment
similarity + paraphrase tasks
MRPC (Microsoft Research Paraphrase Corpus) - determine if two sentences are paraphrases from one another.
QQP (Quora Question Pairs)- determine if two questions are semantically equivalent or not.
STS-B (Semantic Textual Similarity Benchmark) - determine the similarity of two sentences with a score from one to five.
Inference Tasks
MNLI (Multi-Genre Natural Language Inference) - determine if a sentence entails, contradicts, or is unrelated to another sentence
QNLI (Question-answering Natural Language Inference) - determine if the answer to a question is contained in a second sentence or not
RTE (Recognizing Textual Entailment) - determine if a sentence entails a given hypothesis or not
WNLI (Winograd Natural Language Inference) - determine if a sentence with an anonymous pronoun and a sentence with this pronoun replaced are entailed or not
more NLI ( natural language inference)
ANLI: Adversarial NLI (nie et al. 2019) - harder examples found by model failures
SNLI Benchmark (bowman et al. 2015) = Stanford Natural Languge Inference - entailment dataset
570k human-annotated sentence pairs where people ask about entailment
FEVER: Fact Extraction and VERification (Thorne et al., 2018)
SciTail (khot et al. 2018) - textual entailment derived from science-question answering
QA
SQuAD 2.0 (Rajpurkar…liang, 2018) - adds 50k unanswerable questions; system must know when it can’t answer
SQuAD (Rajpurkar…liang, 2016) - Stanford Question Answering Dataset (SQuAD) - 100k questions from 23k passages in 500 wikipedia articles
common data sources
WSJ
then twitter
then Wikipedia
<IP_ADDRESS>. eval metrics#
perplexity (PP) - inverse probability of the test set, normalized by the number of words (want to minimize it)
\(PP(W_{test}) = P(w_1, ..., w_N)^{-1/N}\)
can think of this as the weighted average branching factor of a language
should only be compared across models w/ same vocab
BLEU
bert-score: https://github.com/Tiiiger/bert_score
5.2.2. individual tasks#
<IP_ADDRESS>. tokenization#
Tokenizers - Hugging Face Course
word-based
punctuation splitting
unknown token [UNK] for anything not seen - to reduce the amount of this, can get character-based tokens
vocab tends to be too big
subword-based - break apart meaningful subparts of words, most popular (2022)
many more (e.g. byte-level BPE, used in GPT-2)
character-based - very little prior, generally sequences are too long
vocabulary
sometimes closed, otherwise have unkown words, which we assign its own symbol
can fix training vocab, or just choose the top words and have the rest be unkown
-
stemming - maps words to a common root, usually by just chopping off the ending (e.g. “dogs” -> “dog”)
most common way to do this is Porter’s algorithm, which has 5 phrases of rule-based word reductions
lemmatization - smarter version that takes into account vocab / morphological analysis (e.g. “am, are, is” -> “be”)
<IP_ADDRESS>. token / word embeddings#
embeddings - vectors for representing words
ex. tf-idf - defined as counts of nearby words (big + sparse)
TF * IDF = [ (Number of times term t appears in a document) / (Total number of terms in the document) ] * log(Total number of documents / Number of documents with term t in it).
pointwise mutual info - instead of counts, consider whether 2 words co-occur more than we would have expected by chance
ex. word2vec - short, dense vectors
intuition: train classifier on binary prediction: is word \(w\) likely to show up near this word? (algorithm also called skip-gram)
the weights are the embeddings
word2vec paper I: initial word2vec (mikolov et al. 2013) - simplifies neural language models for efficient training of word embeddings
maximizing the probabilities of words being predicted by their context words (with a DNN)
continuous bag-of-words (CBOW) - predict current word from window (order doesn’t matter)
skipgram - use current word to predict surrounding window – nearby context words weighted more heavily
word2vec paper II: Distributed Representations of Words and Phrases and their Compositionality (mikolov et al. 2013) - word2vec improvements
identify key ngrams and give them their own vecs
ex. GloVe (pennington, socher, & manning, 2014), which is based on ratios of word co-occurrence probs
ex. ELMO (peters…zettlemoyer, 2022) - use LSTM for word embeddings
<IP_ADDRESS>. language modeling#
language models - assign probabilities to sequences of words
ex. n-gram model - assigns probs to short sequences of words, known as n-grams
for full sentence, use markov assumption
multi-token decoding for classification - regular beam search will favor shorter results over longer ones on average since a negative log-probability is added at each step, yielding lower (more negative) scores for longer sentences
<IP_ADDRESS>. topic modeling#
topic models (e.g. LDA) - apply unsupervised learning on large sets of text to learn sets of associated words
LDA = latent dirichlet allocation
<IP_ADDRESS>. interpretable prediction models#
Neural Bag-of-Ngrams (li et al. 2017) - learn embedding vectors for ngrams via deep version of skip-gram
Improving N-gram Language Models with Pre-trained Deep Transformer (wang et al. 2019) - use transformer to generate synthetic data for new n-gram model (language model, doesn’t extend to classification)
Improvements to N-gram Language Model Using Text Generated from Neural Language Model (suzuki et al. 2019) - generate synthetic data from RNNs for new n-gram model
fasttext (jin et al. 2016)
(DirtyCat): Encoding High-Cardinality String Categorical Variables (cerda & varoquax, 2020) - use embedding model to improve string categorical variables
<IP_ADDRESS>. grammar / parse-tree exraction#
notes/figures from nltk book ch 8/9
language - set of all grammatical sentences
grammar - formal notation that can be used for “generating” the members of this set
phrases
noun phrases
adjective phrase
structures
constituent structure - words combine with other words to form units which are substitutable
e.g. the fact that we can substitute He for The little bear indicates that the latter sequence is a unit
coordinate structure: if \(v_{1}\) and \(v_{2}\) are both phrases of grammatical category \(X\), then \(v_{1}\) and \(v_{2}\) is also a phrase of category \(X\)
context-free grammar - set of grammar rules that define a gramm
e.g. \(S \to NP\; VP\)
rules can be probabilities (so we search for the most probable parse tree, rather than returning all)
phrase tree
\(\begin{array}{lll}\text { Symbol } & \text { Meaning } & \text { Example } \\ \text { S } & \text { sentence } & \text { the man walked } \\ \text { NP } & \text { noun phrase } & \text { a dog } \\ \text { VP } & \text { verb phrase } & \text { saw a park } \\ \text { PP } & \text { prepositional phrase } & \text { with a telescope } \\ \text { Det } & \text { determiner } & \text { the } \\ \text { N }& \text { noun } & \text { dog } \\ \text { V } & \text { verb } & \text { walked } \\ \text{ P} & \text { preposition } & \text { in }\end{array}\)
note: chunking yields a similar partition of a sequence rather than a tree
algorithms
top-down parsing (e.g. recursive descent parsing) - starts from \(S\) , and keep expanding the grammar rules until finding a match
RecursiveDescentParser is unable to handle left-recursive productions of the form X -> X Y
bottom-up parsing - can be faster; find sequences that correspond to the righthand side of a grammar rule and replace them with the left
ex. shift-reduce parser
backtracking ensures we find a parse if one exists (and can find multiple if we choose)
dependency grammar focuses on how words relate to other words
relation between head (usually tensed verb) and its dependents (rest of the words)
this can also be displayed as a tree
a dependency graph is projective if, when all the words are written in order, the edges can be drawn above the words without crossing
equivalent to saying that a word and all its descendants (dependents and dependents of its dependents, etc.) form a contiguous sequence of words within the sentence
above graph is projective
valencies - sometimes certain things in a class are allowed buothers are not
e.g. “Buster was frightened” ✅ but “Buster saw frightened” :x:
need subcategories of things (e.g. intransitive verb vs transitive verb vs dative verb) to know what symbols are allowed in rules
verbs specifically have complements associated with them – unlike modifiers (like the word “really”), complemetes are not optional and usually not selected in the same way by the head
feature-based grammars - add things to word representations (e.g. plurality) and use these in the grammar rules