US Residential Solar
Photovoltaic Panel Adoption
A University of Michigan MADS Capstone Project
Collin Brosko, Greg Myers, Robert Underhill,
Monica Yen
Project Repository: GitHub
In the Sixth Assessment Report published by The Intergovernmental Panel on
Climate Change, scientists have made it abundantly clear that the time is now
to make drastic changes to help curb climate change before it is too late.
Solar energy, being a renewable energy, could be used instead of fossil fuels
to power the electric grid and lower the release of harmful greenhouse gases
into the environment. On the other hand, electric utilities must be able to
plan and control for distributed generation sources to be able to both ensure
the health of the grid and to be capable of reducing generation due to other
energy sources such as on-demand gas turbines which may generate more
greenhouse gases than more steady generation sources. Ultimately, knowledge is
key.
The overall goal of our project is to gain a
better understanding of the factors impacting the adoption of photovoltaic
solar panels in the residential sector and to provide a means of assessing
these factors for different locations within the United States. These factors
can be described as physical, economical, and public policy, or programmatic,
features. With the ability to assess trends in solar availability, in the costs
and benefits associated with installing and savings due to solar panels, and to
assess the impacts of programs and policies made available, it becomes more
possible to understand where there are untapped markets, where adoption may
grow, and where programs can kickstart or enhance the rate of adoption.
The primary goal of the solar
data collection process is to provide quantitative irradiance data for the purpose of estimating photovoltaic electric power generation in a
geospatial-context. Secondarily, and of nearly equal importance, is the
extraction of trends from the raw data, which can be used to enhance
understanding of future trends in solar irradiance and electric power
generation potential. Lastly, to incorporate forecasting using one or more
models with the goal of providing actionable estimates of future irradiance
quantities and to a lesser extent, envisioning future meteorological conditions. Using the irradiance forecast data, estimates of average
power generation potential would be available for residential customer
photovoltaic investment decision-making. Each of these data product features
will be wrapped and presented via the Solar Irradiance Data Explorer dashboard
(SIDE-dash).
The central concern with
photovoltaic solar power generation is the total available solar irradiance for
a given location or region. The absolute availability depends on the calendar
season and the latitude of the location. Summertime of the northern hemispheres
places the sun higher in the sky and thereby reduces zenith angle or angle of incidence of solar radiation
arriving at earth’s upper atmosphere and ultimately the surface of the earth.
The decreased zenith angle also reduces the path length through the atmosphere
and the corresponding attenuation of available irradiance. As the sun is lower
in the wintertime sky, the result is an increase in zenith angle and greater
attenuation of available irradiance. An increase in latitude also increases the
zenith angle with effects like that of seasonal wintertime.
Meteorological conditions
including atmospheric water (Tasie, Banyie, & israel-cookey,
2018) and fine particulate matter also plays a significant role in
terrestrial-based photovoltaic electric power production. As light passes
through the earth’s atmosphere, varying levels of water vapor (precipitable water) or cloud cover will cause varying levels of diffusion of sun light (wikipedia.com, 2021). Diffusion has the effect
of increased zenith as the radiation is no longer straight-line from the sun
source, but rather arriving at a higher angle of incidence due to diffusion’s
scattering effects. Particulate matter can have a similar effect if not an
outright blockage of the sun’s rays. All these factors combine to create a
highly seasonal response to solar power production.
The primary source of solar
irradiance data is obtained from the National Renewable Energy Laboratory (NREL, 2021) and the National
Solar Radiation Database (NSRDB).
NREL is an organization of the U.S. Department of Energy and a globally
respected source of renewable energy data, recommended by industry and academia
sources alike. More specifically, the on-demand data was requested through the
NREL provided API which allows client-specific location, timeframe, and data
feature requests. The NSRDB, provided by NREL, is a 30-minute time increment,
4-km spatially gridded dataset with complete coverage of North America, and
more. For each of the ZIP Code-based locations requested, all data fields and
all twenty-three years of 1-hour increment data (1998-2020) were downloaded.
The full data set include the following fields:
●
date_time, zipcode, location_id, Year, Month, Day, Hour
●
Clearsky_DHI, DHI, Clearsky_DNI, DNI, Clearsky_GHI, GHI
●
Temperature, Dew_Point, Relative_Humidity, Precipitable_Water
●
Pressure, Wind_Direction, Wind_Speed, Cloud_Type, Fill_Flag
●
Solar_Zenith_Angle, Surface_Albedo
●
Global_Horizontal_UV_Irradiance_(280-400nm)
●
Global_Horizontal_UV_Irradiance_(295-385nm)
Each of these fields is
retained in the project’s Zip Code request databases, but not in the current
monthly aggregated database. This is mostly due to file size restrictions
imposed by GitHub. While not a part of the project GitHub repository, all
requested raw data downloads are retained for future use as necessary. An
example set of 46 data and metadata files for ZIP Code 85286 (Chandler, AZ) can
be found in the repository examples_nrel
folder.
The current ZIP Code spatial
data was extracted from the Gazetteer
Files page of U.S. Census
Bureau (census.gov, 2021). The
Gazetteer data includes a geo-spatially centered latitude/longitude coordinate
for each ZIP Code polygon. The ZIP Code geo-coordinate was used as the location
parameter in the NREL API queries of the NSRDB dataset. Additional fields in
the Gazetteer data include land and water area for each ZIP Code. This data
could have future use in determining if additional NSRDB coverage is necessary
given the spatial extents of a given ZIP Code.
A third and final data source
is UnitedStatesZipCodes.org which has available an up-to-date dataset
(UnitedStatesZipCodes.org, 2021) that relates ZIP Codes with geo-political
features such as city, county, state, and more. This dataset along with the
Gazetteer data formed the basis for the geo_zipcodes table used in the project database(s).
As a matter of portability
and resource minimization, SQLite3
was selected at the project irradiance data storage database. SQLite3 is a part
of the Python Standard Library and functional on all compute platforms which
support the Python scripting environment.
The geo_zipcodes database was an early creation
to be used as feedstock in the production of the actual irradiance database.
Certain fields are updated within the geo_zipcodes database with each download of additional irradiance
data. When a new irradiance database is constructed, the current geo_zipcodes database is
incorporated as the geo_zipcodes table.
When a Zip Codes-based
request for irradiance data is made by a team member, the get_zipcode_geos.ipynb notebook
script is used to compile the ZIP Codes into a simple YAML file, containing
geo-spatial data for the ZIP Code, that is ingested by the nsrdb_download.ipynb notebook
script. As each NSRDB download request arrives, minor data manipulations are
performed, and the data is then written to the new ZIP Code irradiance
database. Simultaneously the internal geo_zipcodes table and the external database of the same name are updated
for future use.
Because project data
manipulations operate at the monthly level, the choice to aggregate working
data at the monthly level was easy. Originally this operation was performed in
real-time once the data was ingested into memory from the database. As
SIDE-dash took shape, irradiance data intake performance failed to meet
expectations as twenty-three years of hourly data resulted in nearly 201,500
records per ZIP Code.
As a late addition to the
project, the nsrdb_aggregator.ipynb notebook script was authored to
reduce the overall size of the dataset while still retaining current
functionality. The reduction in data volume was profound. A forty-three ZIP
Code irradiance database of raw data consumed 1.1GB of filesystem space. The
aggregated hourly average by month database, of the same ZIP Codes, was reduced
to 5.9MB. Currently the aggregated database contains the following fields.
●
id, location_id, zipcode, date_time,
●
Clearsky_DHI, DHI, Clearsky_DNI, DNI, Clearsky_GHI, GHI
●
Temperature, Dew_Point, Relative_Humidity,
Pressure,
●
Precipitable_Water, Wind_Speed, GHI_UV_wd, GHI_UV_nw
An example of hourly average
by month; ZIP Code 13214 (Syracuse, New York), for the month of January 1998,
has as Global Horizontal Irradiance (GHI) value of 56.9 Wh/m^2
(Watt Hours per square meter); meaning that on average, for every hour of the
24-hour day, 56.9 Wh/m^2 of solar radiation arrived
at the earth’s surface in Syracuse. As a point of reference Phoenix, Arizona
received, on average, 144.8 Wh/m^2 per hour for the
same month. Obviously, the sun does not shine 24-hours a day in these regions
therefore the total irradiance (1,365.6 and 3475.2 respectively) would be
captured during normal daylight hours for the location in question.
The dashboard currently
includes two forecasting models built from the PMDARIMA open-source time-series package. The first
model performs a Fast Fourier Transform (FFT) of the data to eliminate the
seasonal component of the data; a necessary and important step for any
time-series forecast model. A simple model pipeline is implemented to ensure
that the transform is correctly applied to all data, be it training, validation
or test. The second model relies on more traditional seasonal decomposition
(parameters P, D, Q) to eliminate seasonality from the data. Both models
incorporate standard Order parameters p, d, and q for auto-regressive (AR),
differencing (Integration), and moving average (MA). The benefit of the
PMDARIMA package is the automatic Order optimization methods that are built
into the library. In the context of SIDE-dash, each forecast is optimized for
the specific feature selected. An interesting note is the FFT model is
typically three to five times faster than the traditional seasonal
decomposition model. The metric Root Mean Squared Error (RMSE) was chosen as
the model test performance indicator and comparator. The advantage of RMSE is
that the calculated value retains the original units of the training and
forecast data, simplifying its interpretation and comparison.
The primary goal of the
irradiance data collection process is the production of a database(s) product
that all team members can easily query and retrieve irradiance data for
application to their respective needs. The outcome of this goal is realized
through the Solar Irradiance Data Explorer dashboard that provides multiple
exploratory views of the aggregated data, decomposed trend data, and
time-series forecasting of the data. While the irradiance data and forecast
would see the most direct use, the trend data offers a tantalizing perspective
of future potential.
Below is a bullet point list
of data feature, model, and dashboard interface enhancements that could and
should be incorporated into the project.
●
Feature
aggregations
○
Period totals for
irradiance data
○
The inclusion of
cloud cover data
●
Daily, Weekly and
Monthly aggregation datasets
●
Inclusion of Solar
Sunspot cycle data
●
Additional
non-traditional time-series forecasting models
○
LSTM
○
XGBoost
●
UI improvements
○
City/State ZIP
Code filtering
○
Connect dashboard
pages by selected ZIP Code values
○
Links to external
sources such as NREL
○
Rollover help
popups to provide guidance to the visualizations
The plot below is a
visualization of the Berkeley Lab solar panel installation data across the
years. Each point represents a ZIP Code that has solar installations. The
individual maps were created using ArcGIS Online, which can be used for
building maps. The data comes from the filtered Berkeley data set, described in
the Policy and Programmatic section. The graphic, which is known as a small
multiples plot, can be used to see which areas of the United States have ebbed
and waned in installation location as a quick data exploration activity.
Small multiples plot of Berkeley data from 1998-2019. Created using ArcGIS
Online and image editing tools.
The ZIP Codes used in the current database comes from
top-ranked ZIP Codes on Niche.com according to its housing market rating. Niche.com has
rankings for places to live, including ZIP Codes, from a methodology described here. The reasoning for this is that residential areas in
places where there is likely to be high home prices and percentage of home
buyers compared to renters would be more likely to be profitable for solar
panel installers.
Solar technology has been
around since the mid-17th century and has been well researched. Albert Einstein even won a
Nobel Prize for his theories on the photovoltaic effect in 1921. Even with technology that is over 150 years
old, it is not a secret that the solar panels for residential use are
expensive. Every year the solar technology gets better, and the residential
consumer is presented with more options for clean and renewable energy; but
there are still obstacles that hinder adoption. Primarily for residential
consumers, there are many costs that go into a decision regarding the adoption
of solar panels in a region and how beneficial they will be to a community.
There is more that goes into the economic considerations of solar panel
adoption than the overall purchasing and installation cost of the units. The
economics of solar panel adaptation for residential users considers the cost of
energy provided by the utility companies, as well as the total energy consumed
by a community. We aim to examine multiple aspects of solar panel adoption at
the residential level to define those states that might have the highest return
on investment and states that will yield the greatest positive impact if solar
technology is utilized. The economics of solar panels are only one element to
consider when evaluating the investment, but many customers view the cost of
installation as the primary obstacle, “Costs for solar can sometimes be
overwhelming, or make people think that it’s not possible for them” (Marcilla, M, 2021). In addition to the cost considerations,
the public policy analysis and climatological study will help to identify which
geographic areas in the US will benefit from the transition to solar
technology.
One of the main concerns for
potential residential solar customers is the initial financial cost. The
initial installation cost for solar panels can be significant but prospective
customers must be aware that solar technology should be thought of as a multi-year
investment. It is because of this that forecasted costs are important for this
evaluation.
The cost impact of solar
panels for residential customers really comes down to two aspects: first being
the initial cost of purchase and installation, and second being the cost
avoidance from the utility companies. While solar panel technology has been around
for a long time, it has only recently been used at the individual consumer
level. Therefore, the data is limited on quantities of installation and the
total cost of installation on a large scale. There is a U.S. Department of
Energy National Laboratory Managed by the University of California known as Berkeley Lab that has been compiling and analyzing solar
installation data for years and they have published their findings. The
Berkeley Lab was able to compile a substantial database of information
including installation date, size of installation, total installation price,
and assorted geographic markers. All this data goes back to 1998 and one of the
most comprehensive databases for solar installation projects out there. The
only obstacle that existed with this database was the depth of data. There were
many states that reported zero residential solar installations and other states
that had a minimal amount of data available to accurately calculate a trend, so
certain limitations existed.
The U.S Energy
Information Administration published
yearly energy cost per kilowatt-hour from 1990 to present at the state level.
This data will help us create a benchmark and forecast energy cost out through
2030. The same governing body also published the total energy sold at the state
level for the same time based on total megawatt-hours. These values will give
us the ability to forecast which states are projected to consume the most
amount of energy delivered from the utility companies for the next 10
years. The U.S Energy Information
Administration data proved to be an excellent source for total energy cost and
total state energy usage dating back to 1990. The only way that the data could
have been improved was if the information was able to be compiled at a ZIP Code
or county level.
Since the adoption of solar
panels as a source of energy is a multifaceted decision for any residential
customer, certain rated scores were created. The overall installation cost of
solar panels was given a weighted score of 30% due to the initial investment.
The next 20% of our weighted score came from the total energy usage at the
state level. As the energy grid in the US becomes larger and more unstable due
to several environmental factors and due to the age of the grid, high energy
usage can be a driver for more consumers to become less reliant on utility
companies. Frequent blackouts and unreliable service can be alleviated by an
investment in renewable energy. The final and largest proportion of our
weighted score comes from the cost of energy. “It is projected that the
residential electricity price in the United States will increase by 1.3 percent
between 2021 and 2022. Generally, retail electricity prices in the country have
risen over the last decade” (EIA, 2021). As energy costs continue to rise,
consumers will begin to look for alternative solutions and solar technology is
the most logical solution for many consumers. Switching from being reliant on a
utility company for energy to being self-reliant or even partially self-reliant
will be a decision that the residential consumer will have to make for
themselves.
As energy costs continue to
rise, consumers will begin to look for alternative solutions and solar
technology is the most logical solution for many consumers. Switching from
being reliant on a utility company for energy to being self-reliant or even
partially self-reliant will be a decision that the residential consumer will
have to make for themselves. Considering the previous years’ worth of data and
our designated weighting scores for each criterion, a ten-year forecast was
created using an autoregressive moving average model. Since an investment in
solar panels should not be thought of as a short-term solution, we make the
decision to rank each state at their forecasted values in 2030 to calculate our
high-yield states for solar panel adoption.
States were given a weight
score based on installation cost, total energy consumption, and cost of energy
and forecasted these values out for the next ten years to accommodate the
long-term economic impact of solar panel usage at the residential level. Based
on these weight scores, we can identify the top 10 years in which the adoption
and use of solar technology could have the most beneficial economic impact on
residential consumers.
While some of these states
are not surprising to see on our top 10 list, other states are surprising.
Consider that this suggestion is only due to economic factors that are weighted
most heavily on the cost of energy.
It is important to consider
that many of the utility companies are also investing in renewable technology to
drive down the cost of energy for its consumers and this factor was not
considered in this analysis. Additionally, data on solar panel installation was
limited but it is likely that the availability of this data will continue to
improve as the technology becomes more widely available and utilized.
Ideally, we would
like to identify and bring in additional economic drivers to my model and
forecast to achieve a more well-rounded approach. Economic drivers may include
household income and states' current investment in alternative renewable
energy. Additionally In the same vein as potential improvements, our energy
cost and usage data were only at the state level, it would improve our ability
to pinpoint specific geographic locations for solar technology if it was able
to acquire energy cost and usage data at the county or postal cost level.
Calculating the ideal geographic location for residential solar technology is
not one dimensional and using more precise geographic data, will assist the
climatological and public policy data in identifying specific postal codes that
would have the highest return-on-investment for residential solar technology.
In the advent of new technologies, it is often programs and policies
that can kickstart adoption as a means of making a new technology more
attractive, most often through means of driving costs down or improving payback
periods. Throughout the period in which solar panels have been offered to
residential customers, there have been a wide array of programs and policies
offered. The goal of this segment of the project is to assess the effectiveness
of these programs, primarily by type.
Berkeley Lab maintains data for over 80% of all solar installations in
the United States through their Tracking the Sun report. This information is reported through a number
of channels and as such, some is only maintained at the state level.
Some states do not have reported data. The above chart contains markers for
each ZIP Code in which Berkeley Lab has data to that level. This data consists
of cost, system size, recorded rebate information, the date of installation and
a number of other features for each installation
recorded.
NREL maintains a trove of irradiance data. For this portion of the
project, the NREL_PSM_2019 measure was utilized. This is maintained as a TIF
image file which can be mapped to ZIP Codes. Future project iterations may
integrate output from the irradiance and meteorological section should all
relevant ZIP Codes be available.
Policies have been tracked by the North Carolina Clean Energy
Technology Center and organized into a dataset known as the Database of State Incentives
for Renewables & Efficiency. This data
indicates program types, start dates, specific features and provides
connections from the programs to affected ZIP Codes.
The US Energy Information Administration (EIA) captures the cost of
electricity by sector and by state dating back to
January of 2001. While this does not give a specific ZIP Code level and does
not date back to the first installation in the Berkeley dataset (January 1998),
the variability of initial installations along with the scarcity of early
installations, along with the perceived significance of the cost of electricity
in adoption, led to the decision to use this data and cut off installation data
prior to 2001. The data was mapped to month and ZIP Code.
Housing data was obtained from the 2019 American
Community Survey and 2010 census. A linear connection was mapped
to years between. Without earlier dates identified, housing numbers were
maintained at constant 2010 values for earlier years, though this would ideally
be updated for future iterations.
The project began with informational interviews with the City of Ann Arbor and with the North Carolina
Clean Energy Technology Center. These interviews laid out the landscape for
what is residential solar adoption and the factors that might impact it. Solar
adoption, while gaining steam, is still relatively in its infancy with some
parts of the county moving at much higher rates than others. Factors of
available solar energy, costs of electricity and cost of installation were
treated as the primary factors in adoption. Program types were evaluated versus
these primary factors in determining the effect of their implementations.
Policy and program information, provided in CSV form needed to be
reassembled into a database form. Once assembled, further manipulation was
needed to map all information to the ZIP Code level, whether original programs
we provided at state, utility, or other level initially. The data was filtered
to include only policies which impacted the residential sector, not including
multi-family homes. The decision to not include multi-family homes is due to
the belief that the decision-making process for a single-family homeowner would
be different from that of a multifamily residence property manager/ owner. This
further allowed for decisions to be made regarding which installations were to
be considered residential within the Berkeley Lab dataset. The program type,
start and end dates, along with impacted ZIP Codes were organized into a
dictionary which was then used to map program types to the ZIP Codes and dates
in which they were believed to be effective. For rebate and grant programs, the
effective dates were assumed based upon correlation to grant and rebate values
within the Berkeley dataset. Other programs were assessed at a 12-month window
as well as assuming no end date.
Information utilized for this project from the Berkeley data set
included the ZIP Code, date, size, cost, and rebates pertaining to each
installation. Installations were filtered by what Berkeley Lab listed as
residential. Additionally, ground mounted installations were filtered out due
to associated costs, though this may be a considered revision for future work.
Installations with battery storage were also filtered due to the costs included
not specific to the solar generation capability. Cost outliers were determined
through a principal component analysis of date versus cost. This trimmed a
small number of data points. The final filter was to limit installed generation
capacity to 25 kW. This is viewed as a reasonable limit for a single-family
home and is also a cap value for some programs.
The National Renewable Energy Lab maintains a measure of irradiance
known as the “nsrdb_full_disc_2019_dni_scaled.” This data was mapped to and
averaged upon the ZIP Code boundaries provided. It was then merged with other
data.
Installation counts were normalized against housing counts to provide a
more comparable metric across ZIP Codes that may be more or
less dense.
Initial reviews of data and regressions showed that of the potential
non-policy features, the Irradiance measure, price of electricity and cost of
installation were by far more significant than other obtained features, with
cost of installation being the least significant of the three. These values
were treated as constants within approaches to causal inference.
All data, once cleaned, was aggregated at a monthly level through a
series of notebooks. SQLite was used throughout this project due to the large
volume of data and intensive operations which could not be performed in pandas.
This volume of data additionally reduced the number of variables (particularly
program types) which were ultimately used for modeling. The most significant
features were carried through while less impactful ones were omitted under
current efforts.
A statsmodels OLS regression was performed along with a program-by-program
Causal Model with propensity matching. This provided a
general understanding of the effectiveness of program types. Upon reviewing
program data in greater detail, some programs may vary in their levels of
implementation and as such, some programs within a given type may skew a type
to become more or less effective.
Due to this potential issue, a separate analysis platform was developed to
evaluate individual programs. To do this, program types may be filtered to the
individual program. The relevant ZIP Codes are then identified and matched
based upon the primary features identified previously along with the general
installation trend prior to implementation. Done at the monthly level for
processing capability, the implementation month itself is treated as a center
point as programs may have been implemented in the first or final day of a
month. All pairs are aggregated making up a control and treatment group. The
six-month prior to six-month following period is then plotted for comparison.
This will allow for individual program level evaluation of treatment effect.
In general, feedback obtained through interviews was confirmed through
the data. This stated that the driving forces behind solar adoption have been
available solar energy, cost of energy and programs which offer longer-term
benefits.
While definitive treatment effects for program types cannot be
determined, the current results show that certain program types have been
generally more effective than others. Tax-oriented programs, whether personal
tax credits, personal tax deductions or property tax incentives are generally
shown to be effective. Additionally, programs which offer long-term payback
potential, such as feed-in tariffs or value of solar tariffs, are also shown to
be effective. Perhaps unexpectedly, instant return / discount type programs such
as rebates or grants were not shown to be effective in increasing residential
solar adoption. Other programs had limited instances and could not be evaluated
with as much confidence. Additionally, covariance between program types and the
large number of program types lowers the confidence in the specific treatment
effect. As indicated earlier, programs may vary within a given type. As such,
nuance may be evaluated by looking at individual programs. In many cases, this
provides a clearer impact of a program.
Below is a list of additional
work which would improve the project:
● Addition economical drivers include:
○ Household Income
○ States' current investment in alternative
renewable energy
●
Improved source data or mined features
○
Consider re-aggregating data at county level to
reflect most data sources, improve speed and increase sample sizes per
division.
○
More granular demographic and housing data
○
Individual programs start and end dates as well
as feature magnitude versus Boolean value may be extracted from summary or
program source
●
Improved filling of missing data points (dates
of no installation) per advanced methodologies
●
Diff_in_Diff notebook tool improvements
○
Increase speed to allow Plotly dashboard
integration
○
Highlight when installation data is not present
for given program / remove programs from consideration
○
Include additional features for matching
○
Auto-analyze output for parallel trends
assumption
●
Weight installation values for seasonality
●
More specific geographic energy date
○
Postal code level data for total energy
consumption
○
Postal code level of data of cost of energy
●
Perform time series forecast for installation
rates with integration of Irradiance and economic forecasting
○
Provide updated feature weights
○
Toggle program feature to adjust forecast
This project provides a framework in which groups or individuals can understand the significance of features impacting the adoption of solar photovoltaic panels in the residential setting. It provides expectations for the trends of some of these features while giving insights into how adoption may be increased more effectively. While there is work to be done, this can be a great resource for anyone interested in understanding the state of the solar industry within the residential sector and how that state may be expected to change.
Berkeley Lab (2020). Tracking The Sun: Public Data File ZIP. Retrieved
from Berkeley Lab: https://emp.lbl.gov/tracking-the-sun
census.gov. (2021, 06 29). Gazetteer
Files. Retrieved from United States Census Bureau: https://www.census.gov/geographies/reference-files/time-series/geo/gazetteer-files.html
Hu, Sanhe
(2021). uszipcode zip code database. Retrieved from https://uszipcode.readthedocs.io/index.html
Marcilla, M. (2021, May 8). Charlottesville nonprofit launches campaign
to install more solar panels in central Virginia. Https://Www.Nbc29.Com. https://www.nbc29.com/2021/05/07/charlottesville-nonprofit-launches-campaign-install-more-solar-panels-central-virginia/
NC DSIRE Clean Energy
Technology Center (06-2021). DSIRE
Database. Retrieved from https://www.dsireusa.org/resources/database-archives/
NREL. (2011). 2019 PSM Full Disc DNI. Retrieved from
https://maps.nrel.gov/nsrdb-viewer/
NREL. (2021, 06 26). Home. Retrieved from National Solar Radiation Database: https://nsrdb.nrel.gov/
NREL. (2021, 06 29). Physical
Solar Model (PSM) v3. Retrieved from NREL Developer Network: https://developer.nrel.gov/docs/solar/nsrdb/psm3-download/
NREL. (2021). Solar Resource
Glossary. Retrieved from NREL: https://www.nrel.gov/grid/solar-resource/solar-glossary.html
Tasie, N., Banyie,
L., & israel-cookey, C. (2018, October 17). The Effect of Relative Humidity on the Solar
Radiation Intensity in Port Harcourt, Nigeria. Retrieved from ResearchGate:
https://www.researchgate.net/publication/328335896_The_Effect_of_Relative_Humidity_on_the_Solar_Radiation_Intensity_in_Port_Harcourt_Nigeria
United States Census Bureau
(2021). 2019 American Community Survey.
Retrieved from https://data.census.gov/
United States Census Bureau,
Department of Commerce (2021). 2019 Cartographic
Boundary Shapefile, 2010 ZIP Code Tabulation Areas for United States, 1:500,000
Retrieved from https://catalog.data.gov/fi/dataset/2019-cartographic-boundary-shapefile-2010-zip-code-tabulation-areas-for-united-states-1-500000
U.S. Energy Information
Administration (2021). Average Retail
Price of Electricity. Retrieved from https://www.eia.gov/electricity/data/browser
UnitedStatesZipCodes.org. (2021, 06 29). Home. Retrieved from United States Zip Codes: https://www.unitedstateszipcodes.org/
UnitedStatesZipCodes.org. (2021, 06 29). ZIP Code Database. Retrieved from United States Zip Codes: https://www.unitedstateszipcodes.org/zip-code-database/
wikipedia.com. (2021, 07 21). Diffuse
sky radiation. Retrieved from Wikipedia: https://en.wikipedia.org/wiki/Diffuse_sky_radiation
As previously stated, the overall goal of our project is to
gain a better understanding of the factors impacting the adoption of
photovoltaic solar panels in the residential sector and to provide a means of
assessing these factors for different locations within the United States. These
factors can be described as irradiance and meteorological, economical, and
policy or programmatic features. With the ability to assess trends in solar
availability, in the costs and benefits associated with installing and savings
due to solar panels, and to assess the impacts of programs and policies made
available, it becomes more possible to understand where there are untapped markets,
where adoption may grow, and where programs can kickstart or enhance the rate
of adoption.
Deliverables:
·
Compiled
cost of installation - CB
·
Identified
energy cost driver data - CB
·
Identified
energy usage driver data - CB
·
Created
ARIMA 10-year forecasts for each economic driver - CB
·
Calculated
appropriate weight scores for each economic cost drivers - CB
·
Attempted
to calculate highest return-on-investment ZIP Codes - CB
·
Finalized
on calculated highest return-on-investment states – CB
·
Maintaining
the early timeline/ coordination of schedule – CB
·
Interview
with SEPA representative, discussing data sources (credit in README) - GM
·
Setup
and manage the GitHub repository - GM
·
Assisted
with basic git training for team members - GM
·
Collect
irradiance and meteorological data from the NREL and the NSRDB repository - GM
·
Collect
supporting ancillary ZIP Code data - GM
·
Build
database(s) of full NSRDB download datasets - GM
·
Build
database of aggregated irradiance and meteorological data - GM
·
Build
the Solar Irradiance Data Explorer dashboard - GM
·
Standup
the Heroku site for hosting the dashboard - GM
·
Incorporated
the blog post into the dashboard - GM
·
Significant
participation in the setup of the README.md framework - GM
·
Assumed
final edit and publishing responsibilities of the blog post - GM
·
Validated
environment requirements files - GM
·
Assisted
in code troubleshooting and python environment issues. - GM
·
Conducted
informational interviews with City of Ann Arbor and with North Carolina Clean
Energy Technology Center - RU
·
Identified
a number of data sources for evaluation - RU
·
Performed
initial analysis of datasets to identify available ZIP Codes, to look at
distributions and gaps, and to consider correlations of potential features. -
RU
·
Created
initial choropleth maps, violin plots and biplots to communicate data elements
to team - RU
·
Performed
initial regression to determine most significant features - RU
·
Identified
ZIP Codes that under or over performed expectation - RU
·
Assembled
and reduced policy data - RU
·
Gathered
ZIP Code level data - RU
·
Aggregated
ZIP Code, installation, and policy data - RU
·
Performed
causal analysis and assembled Diff_in_Diff notebooks
- RU
·
Created
web scraping notebook to collect data which could not be
located in repository - RU
·
Aggregated
initial readme and blog post - RU
·
Visualizations
using GIS techniques – MY
·
Assisted
in identifying ZIP Codes of interest – MY
·
Review
README.md for grammar edits - MY
·
Participate
in final review of project blog - MY
With an 8-week duration project, semi-weekly meetings are
essential to ensuring that the team is on schedule and will be able to submit a
finished product at on the due date. A Google Drive account was established for
collaboration and a GitHub repository exists for code sharing and compiling a
finished product. Each member was required to abide by the agreed upon schedule
and contribute to the group discussion.


Cloud score it is calculated based on the sum of the Natual logaritim of all the diferent scores.
The scores that are currently used are from google trends and from reed


Mikesell leads No. 8 Maryland to 99-55 rout of Delaware
COLLEGE PARK, Md. (AP) - Taylor Mikesell matched her career high with 23 points, and No. 8 Maryland cruised to a 99-55 victory over Delaware on Sunday.
Ashley Owusu and Blair Watson both had 12 points for the Terrapins (3-1), who didn’t need a comeback against the Blue Hens (0-3) in their first game since erasing a 19-point deficit in the fourth quarter of Wednesday’s 70-68 victory at James Madison.
Maryland never trailed and quickly achieved separation in the first quarter. Mikesell hit a 3-pointer and Watson added two more - all in a span of 53 seconds - to give the Terps a 20-4 lead and cap an 18-2 run. Delaware never cut the deficit to less than 11 the rest of the way and trailed 57-24 at halftime.
TOP STORIES
Obama-era cash traced to Iran-backed terrorists
Justice Department clears Trump of wrongdoing on Ukraine call
Accused thieves, child-porn suspects, suspected murderers among DACA recipients: Report
The Blue Hens struggled with their ballhandling throughout the game, committing 25 turnovers, leading to 41 Maryland points. Tee Johnson led Delaware with 15 points.
Maryland forward Shakira Austin made her first start of the season in place of Stephanie Jones, who had started 71 consecutive games before coming off the bench Sunday.
Maryland improved to 13-0 against Delaware.
BIG PICTURE
Delaware: Graduate student Nicole Enabosi, the 2018 Colonial Athletic Association player of the year, missed all of last season with an ACL tear and has yet to debut this season. The Blue Hens, picked to finish fourth in the CAA this season, figure to take a step forward when she returns.
Maryland: The short-handed Terps have only nine healthy players, and each received at least 14 minutes against the Blue Hens. Coach Brenda Frese has managed her rotation well so far this season, with only Owusu averaging more than 30 minutes.
POLL IMPLICATIONS
Voters could modestly punish Maryland for its close call at James Madison, but the Terps are unlikely to move much in either direction after their 2-0 week.
UP NEXT
Delaware: The Blue Hens have a week between games before traveling to Central Florida on Nov. 24.
Maryland: The Terrapins’ three-game homestand continues Wednesday when local rival George Washington visits College Park.
More AP college basketball: https://apnews.com/Womenscollegebasketball and https://twitter.com/AP_Top25
Sign up for Daily Newsletters
Copyright © 2019 The Washington Times, LLC.
Please read our comment policy before commenting.


Get it from GitHub: https://github.com/mangothecat/rmdshower.
This is an R Markdown presentation. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.
When you click the Knit button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document.
These slides use a template from the shower presentation system. Notable features:
Shower [’ʃəuə] noun. A person or thing that shows.
Let me see your identification. You don’t need to see his identification. We don’t need to see his identification. These are not the droids your looking for. These are not the droids we’re looking for. He can go about his business. You can go about your business.
Move along. Move along. Move along.
What is it? Your fathers lightsaber. This is the weapon of a Jedi Knight. Not as clumsy or as random as a blaster. An elegant weapon for a more civilized time. For over a thousand generations the Jedi Knights were the guardians of peace and justice in the Old Republic. Before the dark times, before the Empire. How did my father die? A young Jedi named Darth Vader, who was a pupil of mine until he turned to evil, helped the Empire hunt down and destroy the Jedi Knights.
How did I get into this mess? I really don’t know how. We seem to be made to suffer. It’s our lot in life. I’ve got to rest before I fall apart. My joints are almost frozen.  Where are you going?
How did I get into this mess? I really don’t know how. We seem to be made to suffer. It’s our lot in life. I’ve got to rest before I fall apart. My joints are almost frozen.  Where are you going?
How did I get into this mess? I really don’t know how. We seem to be made to suffer.
It’s our lot in life. I’ve got to rest before I fall apart.
Formulas are rendered by KaTeX, https://github.com/Khan/KaTeX
It supports both inline: \(y = x / 2\) and displayed formulas:
\[ x_{1,2} = \frac{- b \pm \sqrt{b^2 - 4ac}}{2a} \]
The bad news is that when ever you learn a new skill you’re going to suck. It’s going to be frustrating. The good news is that is typical and happens to everyone and it is only temporary. You can’t go from knowing nothing to becoming an expert without going through a period of great frustration and great suckiness.
Hadley Wickham
summary(cars)
## speed dist
## Min. : 4.0 Min. : 2.00
## 1st Qu.:12.0 1st Qu.: 26.00
## Median :15.0 Median : 36.00
## Mean :15.4 Mean : 42.98
## 3rd Qu.:19.0 3rd Qu.: 56.00
## Max. :25.0 Max. :120.00
| mpg | cyl | disp | hp | |
|---|---|---|---|---|
| Mazda RX4 | 21 | 6 | 160 | 110 |
| Mazda RX4 Wag | 21 | 6 | 160 | 110 |
| Datsun 710 | 22.8 | 4 | 108 | 93 |
| Hornet 4 Drive | 21.4 | 6 | 258 | 110 |
| Hornet Sportabout | 18.7 | 8 | 360.0 | 175 |
| Valiant | 18.1 | 6 | 225.0 | 105 |
| Duster 360 | 14.3 | 8 | 360.0 | 245 |
And text on top of them.


Preparation time
30 mins to 1 hour
Cooking time
10 to 30 mins
Serves
Serves 2
Mix flour, salt, fish sauce, white pepper, white wine, and water to make a nice smooth batter.
Cut all the vegetables into thin strips.
Wash all vegetables and dry to avoid splattering.
Mix well with the batter.
Put the oil in a wok or frying pan and allow it to be hot.
Deep-fry the mixed vegetables for between 3-4 minutes or until golden brown.
Remove onto a platter.
To make sauce, mix all ingredients in third group except cashew nuts.
Pour sauce on top of vegetables, sprinkle with crushed cashew nuts on top, and serve.


Jamie is a psychology PhD student based at the University of Sussex.
The jsPsych.randomization module contains methods that are useful for generating random lists of trial variables.
jsPsych.randomization.factorial(factors, repetitions, unpack)
| Parameter | Type | Description |
|---|---|---|
| factors | object | The factors object should contain a property for each different factor. Each property-factor should have a value of an array, with each element of the array corresponding to a level of the factor. |
| repetitions | integer | The number of times to repeat each unique combination of the factors in the output sample. |
| unpack | boolean | If true then the output will be an object with a property for each factor in the original factors object. The value of each property-factor will be an array containing the levels of the factor in a random order. The order will be consistent across each property-factor (e.g., the first element of each property-factor will specify one unique combination of the factors). If false , then the return value will be an array of objects where each property-factor contains only a single value. |
The return value depends on the unpack
parameter. See description of the parameter above, and examples below.
This method takes a list of factors and their levels, and creates a full factorial design by creating each unique combination of the factors. The returned set of combinations is in a random order.
var factors = {
stimulus: ['a.jpg', 'b.jpg'],
ms_delay: [100, 200]
}
var full_design = jsPsych.randomization.factorial(factors, 1);
/*
output:
full_design = [
{stimulus: 'a.jpg', ms_delay: 200},
{stimulus: 'b.jpg', ms_delay: 200},
{stimulus: 'b.jpg', ms_delay: 100},
{stimulus: 'a.jpg', ms_delay: 100},
]
*/
var factors = {
stimulus: ['a.jpg', 'b.jpg'],
ms_delay: [100, 200]
}
var full_design = jsPsych.randomization.factorial(factors, 2);
/*
output:
full_design = [
{stimulus: 'b.jpg', ms_delay: 200},
{stimulus: 'b.jpg', ms_delay: 100},
{stimulus: 'b.jpg', ms_delay: 100},
{stimulus: 'a.jpg', ms_delay: 100},
{stimulus: 'a.jpg', ms_delay: 200},
{stimulus: 'b.jpg', ms_delay: 200},
{stimulus: 'a.jpg', ms_delay: 100},
{stimulus: 'a.jpg', ms_delay: 200},
]
*/
var factors = {
stimulus: ['a.jpg', 'b.jpg'],
ms_delay: [100, 200]
}
var full_design = jsPsych.randomization.factorial(factors, 1, true);
/*
output:
full_design = {
stimulus: ['a.jpg','b.jpg','b.jpg','a.jpg'],
ms_delay: [200, 100, 200, 100]
]
*/
jsPsych.randomization.randomID(length)
| Parameter | Type | Description |
|---|---|---|
| length | integer | The length of the randomly generated ID |
Returns a string of length length
where each character is randomly selected from the numbers 0-9 and all lowercase English letters a-z.
Generates a random string that is likely to be unique. If length is undefined, then the string length is 32.
console.log(jsPsych.randomization.randomID());
// outputs: "t7dwz0e713pc8juuaayyfvpkdd9un239"
console.log(jsPsych.randomization.randomID(8));
// outputs: "3xtpcbck"
jsPsych.randomization.repeat(array, repetitions, unpack)
| Parameter | Type | Description |
|---|---|---|
| array | array | The array of values to randomize & repeat. |
| repetitions | integer or array | The number of times to repeat each element of the array in the final sample. If this parameter is defined as an integer, then each element of array is repeated the same number of times. This parameter can also be an array of the same length as array , in which case each element of array will be repeated the number of times defined in the corresponding position of the repetitions array. |
| unpack | boolean | If each element of array is an object with an equivalent set of properties, then setting unpack to true will make the return value an object with a property for each of the unique properties among the elements of the array . Each property in the output object will be an array containing the values for that property in the randomized order. The order will be consistent across properties. If this is false then the output is just an array containing a randomized order of the original array elements. |
The return value depends on the unpack
parameter. See description of the parameter above, and examples below.
This method takes an array of values and generates a new random order of the array, with the option of repeating each element of the array a specified number of times.
If the array elements are objects with the same set of properties, then this method can optionally return a single object where each property is a randomized order of the properties defined in the original set of objects. This is useful for randomizing sets of parameters that are used to define a jsPsych block.
var myArray = [1,2,3,4,5];
var shuffledArray = jsPsych.randomization.repeat(myArray, 1);
// output: shuffledArray = [3,2,4,1,5]
var myArray = [1,2,3,4,5];
var shuffledArray = jsPsych.randomization.repeat(myArray, 2);
// output: shuffledArray = [1,3,4,2,2,4,5,1,5,3]
var trial1 = {
stimulus: 'img/faceA.jpg',
correct_key: 80,
person_name: 'Joe'
}
var trial2 = {
stimulus: 'img/faceB.jpg',
correct_key: 80,
person_name: 'Fred'
}
var trial3 = {
stimulus: 'img/faceC.jpg',
correct_key: 81,
person_name: 'Mary'
}
var myArray = [ trial1, trial2, trial3 ];
var shuffledArray = jsPsych.randomization.repeat(myArray, 2);
// output: shuffledArray = [ trial1, trial3, trial3, trial2, trial1, trial2 ]
var trial1 = {
stimulus: 'img/faceA.jpg',
correct_key: 80,
person_name: 'Joe'
}
var trial2 = {
stimulus: 'img/faceB.jpg',
correct_key: 80,
person_name: 'Fred'
}
var trial3 = {
stimulus: 'img/faceC.jpg',
correct_key: 81,
person_name: 'Mary'
}
var myArray = [ trial1, trial2, trial3 ];
var shuffledArray = jsPsych.randomization.repeat(myArray, 2, true);
/*
output: shuffledArray = {
stimulus: ['img/faceB.jpg','img/faceA.jpg','img/faceC.jpg','img/faceA.jpg','img/faceC.jpg','img/faceB.jpg'],
correct_key: [80, 80, 81, 80, 81, 80],
person_name: ['Fred','Joe', 'Mary', 'Joe', 'Mary', 'Fred']
}
*/
jsPsych.randomization.sampleWithReplacement(array, sampleSize, weights)
| Parameter | Type | Description |
|---|---|---|
| array | array | The array of values to sample from |
| sampleSize | numeric | The number of samples to draw |
| weights | array | The relative weight of each element in array . This array is normalized, so the values do not need to sum to 1. The length must match the length of array . |
An array containing the sample.
This method returns a sample drawn at random from a set of values with replacement. The relative probability of drawing each item can be controlled by specifying the weights
.
var myArray = [1,2,3,4,5];
var sample = jsPsych.randomization.sampleWithReplacement(myArray, 10);
// output: sample = [3, 1, 2, 2, 5, 1, 4, 3, 1, 5];
var myArray = [1,2,3,4,5];
var sample = jsPsych.randomization.sampleWithReplacement(myArray, 10, [6,1,1,1,1]);
// output: sample = [3, 4, 5, 1, 2, 1, 3, 1, 1, 1];
jsPsych.randomization.sampleWithoutReplacement(array, sampleSize)
| Parameter | Type | Description |
|---|---|---|
| array | array | The array of values to sample from |
| sampleSize | numeric | The number of samples to draw |
An array containing the sample.
This method returns a sample drawn at random from a set of values without replacement. The sample size must be less than or equal to the length of the array.
var myArray = [1,2,3,4,5];
var sample = jsPsych.randomization.sampleWithoutReplacement(myArray, 2);
// output: sample = [3,2];
jsPsych.randomization.shuffle(array)
| Parameter | Type | Description |
|---|---|---|
| array | array | The array of values to shuffle |
Returns an array with the same elements as the input array in a random order.
A simple method for shuffling the order of an array.
var myArray = [1,2,3,4,5];
var shuffledArray = jsPsych.randomization.shuffle(myArray);
// output: shuffledArray = [3,2,4,1,5]
jsPsych.randomization.shuffleNoRepeats(array, equalityTest)
| Parameter | Type | Description |
|---|---|---|
| array | array | The array of values to shuffle |
| equalityTest | function | A function to use to evaluate the equality of neighbors in the array. The function should accept two parameters, which are the two elements to be tested. It should return true if they are equal and false if not. The default function, if none is specified, is to use the === operator. This will work for primitive values, but fail for Objects and Arrays. An example function is given below in the examples. |
Returns an array with the same elements as the input array in a random order, with no repeating neighbors.
Shuffle an array, ensuring that neighboring elements in the array are different.
Warning: if you provide an array that has very few valid permutations with no neighboring elements, then this method will fail and cause the browser to hang.
var myArray = [1,2,3,4,5,1,2,3,4,5,1,2,3,4,5];
var shuffledArray = jsPsych.randomization.shuffleNoRepeats(myArray);
// output: shuffledArray = [2, 3, 5, 1, 2, 4, 1, 5, 4, 1, 3, 5, 4, 3, 2]
var myObjects = [
{color:"blue"},
{color:"red"},
{color:"yellow"},
{color:"orange"}
];
var repeatedSet = jsPsych.randomization.repeat(myObjects,3);
var shuffled = jsPsych.randomization.shuffleNoRepeats(repeatedSet, function(a,b) { return a.color === b.color });
// console.log(JSON.stringify(shuffled))
// "[{"color":"red"},{"color":"yellow"},{"color":"blue"},{"color":"yellow"},{"color":"orange"},{"color":"red"},{"color":"yellow"},{"color":"orange"},{"color":"blue"},{"color":"orange"},{"color":"red"},{"color":"blue"}]"


feast package¶
Subpackages¶
Submodules¶
feast.client module¶
-
class
feast.client.
Client
(core_url: str = None, serving_url: str = None, verbose: bool = False)[source]¶ Bases:
object
Feast Client: Used for creating, managing, and retrieving features.
-
apply
(feature_sets: Union[List[feast.feature_set.FeatureSet], feast.feature_set.FeatureSet])[source]¶ Idempotently registers feature set(s) with Feast Core. Either a single feature set or a list can be provided.
- Parameters
feature_sets – List of feature sets that will be registered
-
property
core_url
¶ Retrieve Feast Core URL
-
get_batch_features
(feature_ids: List[str], entity_rows: pandas.core.frame.DataFrame) → feast.job.Job[source]¶ Retrieves historical features from a Feast Serving deployment.
- Parameters
feature_ids – List of feature ids that will be returned for each entity. Each feature id should have the following format “feature_set_name:version:feature_name”.
entity_rows – Pandas dataframe containing entities and a ‘datetime’ column. Each entity in a feature set must be present as a column in this dataframe. The datetime column must
- Returns
Returns a job object that can be used to monitor retrieval progress asynchronously, and can be used to materialize the results
Examples
>>> from feast import Client >>> from datetime import datetime >>> >>> feast_client = Client(core_url="localhost:6565", serving_url="localhost:6566") >>> feature_ids = ["customer:1:bookings_7d"] >>> entity_rows = pd.DataFrame( >>> { >>> "datetime": [pd.datetime.now() for _ in range(3)], >>> "customer": [1001, 1002, 1003], >>> } >>> ) >>> feature_retrieval_job = feast_client.get_batch_features(feature_ids, entity_rows) >>> df = feature_retrieval_job.to_dataframe() >>> print(df)
-
get_feature_set
(name: str, version: int = None, fail_if_missing: bool = False) → Optional[feast.feature_set.FeatureSet][source]¶ Retrieves a feature set. If no version is specified then the latest version will be returned.
- Parameters
name – Name of feature set
version – Version of feature set
fail_if_missing – Raise an error if feature set is not found
- Returns
Returns either the specified feature set, or None if not found
-
get_online_features
(feature_ids: List[str], entity_rows: List[feast.serving.ServingService_pb2.EntityRow]) → feast.serving.ServingService_pb2.GetOnlineFeaturesResponse[source]¶ Retrieves the latest online feature data from Feast Serving
- Parameters
feature_ids –
List of feature Ids in the following format [feature_set_name]:[version]:[feature_name] .. rubric:: Example
[“feature_set_1:6:my_feature_1”, “feature_set_1:6:my_feature_2”,]
entity_rows – List of GetFeaturesRequest.EntityRow where each row contains entities. Timestamp should not be set for online retrieval. All entity types within a feature
- Returns
Returns a list of maps where each item in the list contains the latest feature values for the provided entities
-
ingest
(feature_set: Union[str, feast.feature_set.FeatureSet], source: Union[pandas.core.frame.DataFrame, str], version: int = None, force_update: bool = False, max_workers: int = 4, disable_progress_bar: bool = False, chunk_size: int = 5000, timeout: int = None)[source]¶ Loads feature data into Feast for a specific feature set.
- Parameters
feature_set – Name of feature set or a feature set object
source – Either a file path or Pandas Dataframe to ingest into Feast Files that are currently supported: * parquet * csv * json
version – Feature set version
force_update – Automatically update feature set based on source data prior to ingesting. This will also register changes to Feast
max_workers – Number of worker processes to use to encode values
disable_progress_bar – Disable printing of progress statistics
chunk_size – Maximum amount of rows to load into memory and ingest at a time
timeout – Seconds to wait before ingestion times out
-
list_entities
() → Dict[str, feast.entity.Entity][source]¶ Returns a dictionary of entities across all feature sets
- Returns
Dictionary of entities, indexed by name
-
list_feature_sets
() → List[feast.feature_set.FeatureSet][source]¶ Retrieve a list of feature sets from Feast Core
- Returns
List of feature sets
-
property
serving_url
¶ Retrieve Serving Core URL
-
feast.config module¶
-
feast.config.
get_config_property_or_fail
(prop: str, force_config: Dict[str, str] = None) → str[source]¶ Gets a single property in the users configuration
- Parameters
prop – Property to retrieve
force_config – Configuration dictionary containing properties that should be overridden. This will take precedence over file based properties.
- Returns
Returns a string property
feast.constants module¶
feast.entity module¶
-
class
feast.entity.
Entity
(name: str, dtype: feast.value_type.ValueType)[source]¶ Bases:
feast.field.Field
Entity field type
feast.exceptions module¶
feast.feature module¶
-
class
feast.feature.
Feature
(name: str, dtype: feast.value_type.ValueType)[source]¶ Bases:
feast.field.Field
Feature field type
feast.feature_set module¶
-
class
feast.feature_set.
FeatureSet
(name: str, features: List[feast.feature.Feature] = None, entities: List[feast.entity.Entity] = None, source: feast.source.Source = None, max_age: Optional[google.protobuf.duration_pb2.Duration] = None)[source]¶ Bases:
object
Represents a collection of features and associated metadata.
-
add
(resource)[source]¶ Adds a resource (Feature, Entity) to this Feature Set. Does not register the updated Feature Set with Feast Core
- Parameters
resource – A resource can be either a Feature or an Entity object
-
drop
(name: str)[source]¶ Removes a Feature or Entity from a Feature Set. This does not apply any changes to Feast Core until the apply() method is called.
- Parameters
name – Name of Feature or Entity to be removed
-
property
entities
¶ Returns list of entities from this feature set
-
property
features
¶ Returns a list of features from this feature set
-
property
fields
¶ Returns a dict of fields from this feature set
-
classmethod
from_dict
(fs_dict)[source]¶ Creates a feature set from a dict
- Parameters
fs_dict – A dict representation of a feature set
- Returns
Returns a FeatureSet object based on the feature set dict
-
classmethod
from_proto
(feature_set_proto: feast.core.FeatureSet_pb2.FeatureSetSpec)[source]¶ Creates a feature set from a protobuf representation of a feature set
- Parameters
from_proto – A protobuf representation of a feature set
- Returns
Returns a FeatureSet object based on the feature set protobuf
-
classmethod
from_yaml
(yml: str)[source]¶ Creates a feature set from a YAML string body or a file path
- Parameters
yml – Either a file path containing a yaml file or a YAML string
- Returns
Returns a FeatureSet object based on the YAML file
-
get_kafka_source_topic
() → str[source]¶ Get the topic that this feature set has been configured to use as source
-
infer_fields_from_df
(df: pandas.core.frame.DataFrame, entities: Optional[List[feast.entity.Entity]] = None, features: Optional[List[feast.feature.Feature]] = None, replace_existing_features: bool = False, replace_existing_entities: bool = False, discard_unused_fields: bool = False, rows_to_sample: int = 100)[source]¶ Adds fields (Features or Entities) to a feature set based on the schema of a Datatframe. Only Pandas dataframes are supported. All columns are detected as features, so setting at least one entity manually is advised.
- Parameters
df – Pandas dataframe to read schema from
entities – List of entities that will be set manually and not inferred. These will take precedence over any existing entities or entities found in the dataframe.
features – List of features that will be set manually and not inferred. These will take precedence over any existing feature or features found in the dataframe.
replace_existing_features – If true, will replace existing features in this feature set with features found in dataframe. If false, will skip conflicting features.
replace_existing_entities – If true, will replace existing entities in this feature set with features found in dataframe. If false, will skip conflicting entities.
discard_unused_fields – Boolean flag. Setting this to True will discard any existing fields that are not found in the dataset or provided by the user
rows_to_sample – Number of rows to sample to infer types. All rows must have consistent types, even values within list types must be homogeneous
-
is_valid
()[source]¶ Validates the state of a feature set locally :return: (bool, str) True if valid, false if invalid. Contains a message string with a reason
-
property
max_age
¶ Returns the maximum age of this feature set. This is the total maximum amount of staleness that will be allowed during feature retrieval for each specific feature row that is looked up.
-
property
name
¶ Returns the name of this feature set
-
property
source
¶ Returns the source of this feature set
-
to_proto
() → feast.core.FeatureSet_pb2.FeatureSetSpec[source]¶ Converts a feature set object to its protobuf representation
- Returns
FeatureSetSpec protobuf
-
property
version
¶ Returns the version of this feature set
-
feast.field module¶
feast.job module¶
-
class
feast.job.
Job
(job_proto: feast.serving.ServingService_pb2.Job, serving_stub: feast.serving.ServingService_pb2_grpc.ServingServiceStub)[source]¶ Bases:
object
A class representing a job for feature retrieval in Feast.
-
property
id
¶ Getter for the Job Id
-
result
(timeout_sec: int = 21600)[source]¶ Wait until job is done to get an iterable rows of result. The row can only represent an Avro row in Feast 0.3.
- Parameters
timeout_sec – max no of seconds to wait until job is done. If “timeout_sec” is exceeded, an exception will be raised.
Returns: Iterable of Avro rows
-
property
status
¶ Getter for the Job status from Feast Core
-
property
feast.resource module¶
feast.source module¶
-
class
feast.source.
KafkaSource
(brokers: str = '', topic: str = '')[source]¶ Bases:
feast.source.Source
Kafka feature set source type.
-
property
brokers
¶ Returns the list of broker addresses for this Kafka source
-
property
source_type
¶ Returns the type of source. For a Kafka source this will always return “kafka”
-
to_proto
() → feast.core.Source_pb2.Source[source]¶ Converts this Source into its protobuf representation
-
property
topic
¶ Returns the topic for this feature set
-
property
-
class
feast.source.
Source
[source]¶ Bases:
object
Source is the top level class that represents a data source for finding feature data. Source must be extended with specific implementations to be useful
-
classmethod
from_proto
(source_proto: feast.core.Source_pb2.Source)[source]¶ Creates a source from a protobuf representation. This will instantiate and return a specific source type, depending on the protobuf that is passed in.
- Parameters
source_proto – SourceProto python object
- Returns
Source object
-
property
source_type
¶ The type of source. If not implemented, this will return “None”
-
classmethod
feast.type_map module¶
-
feast.type_map.
convert_df_to_feature_rows
(dataframe: pandas.core.frame.DataFrame, feature_set)[source]¶ Returns a function that converts a Pandas Series to a Feast FeatureRow for a given Feature Set and Pandas Dataframe
- Parameters
dataframe – Dataframe that will be converted
feature_set – Feature set used as schema for conversion
- Returns
Function that will do conversion
-
feast.type_map.
convert_dict_to_proto_values
(row: dict, df_datetime_dtype: <property object at 0x7f1d51582d18>, feature_set) → feast.types.FeatureRow_pb2.FeatureRow[source]¶ Encode a dictionary describing a feature row into a FeatureRows object.
- Parameters
row – Dictionary describing a feature row.
df_datetime_dtype – Pandas dtype of datetime column.
feature_set – Feature set describing feature row.
- Returns
FeatureRow
-
feast.type_map.
python_type_to_feast_value_type
(name: str, value, recurse: bool = True) → feast.value_type.ValueType[source]¶ Finds the equivalent Feast Value Type for a Python value. Both native and Pandas types are supported. This function will recursively look for nested types when arrays are detected. All types must be homogenous.
- Parameters
name – Name of the value or field
value – Value that will be inspected
recurse – Whether to recursively look for nested types in arrays
- Returns
Feast Value Type
feast.value_type module¶
-
class
feast.value_type.
ValueType
[source]¶ Bases:
enum.Enum
Feature value type. Used to define data types in Feature Sets.
-
BOOL
= 7¶
-
BOOL_LIST
= 17¶
-
BYTES
= 1¶
-
BYTES_LIST
= 11¶
-
DOUBLE
= 5¶
-
DOUBLE_LIST
= 15¶
-
FLOAT
= 6¶
-
FLOAT_LIST
= 16¶
-
INT32
= 3¶
-
INT32_LIST
= 13¶
-
INT64
= 4¶
-
INT64_LIST
= 14¶
-
STRING
= 2¶
-
STRING_LIST
= 12¶
-
UNKNOWN
= 0¶
-


If you detect an enlargement or lump at the front base of your neck, make an appointment with your doctor as soon as possible for testing and treatment. When your thyroid gland is larger than normal, you have a goiter. The three basic kinds of enlargement are:
Any abnormality in the thyroid gland needs to be evaluated by a physician. (Also see Appendix C, page 371, for information about thyroid screening.)
The most serious neck lumps are often not associated with thyroid disease (in the front base of the neck). These lumps are only found on one side of your neck and are usually indicative of a lymph node that has become enlarged due to an infection. If the lump is located toward the front of your neck, it is probably the result of a throat infection. If it is in the back, it is probably the result of a scalp or ear infection. These enlarged lymph nodes are often less than an inch thick, and have a rubbery, semi-soft feel. They should shrink within fourteen days of treatment of the infection.
Lumps in the neck that are present for longer than fourteen days, are over one inch in size, are firm or hard in texture, and/or are not tender are more suspicious and may signify cancer of the lymph nodes or cancer that has spread from another site.
These are the general guidelines for examining lymph nodes, but there are exceptions. Seek advice from your doctor if any such lump persists.
An epidural or intraspinal abscess is an infection that occurs within the confines of the spinal canal, a space surrounded by the bone of the vertebrae extending from the head to the tailbone. This space contains the spinal cord.
Bacteria can enter the spinal canal via the bloodstream from a site elsewhere in the body, often the gums of the mouth or a skin infection. After gaining entrance to the spinal canal, the bacteria establish an infection in the wall of the canal. The body defenses react by walling off the infection and forming an abscess. The abscess rises like a boil, putting pressure on the spinal cord. First an arm or leg becomes weak, and then additional pressure may lead to a loss of function of the spinal cord and paralysis of all muscles below the level of pressure. An abscess in the neck area can paralyze both arms and legs. If it is in the lower back, it can paralyze both legs. Early treatment is essential to avoid permanent paralysis.
Take no chances. If you develop these symptoms, call 911 or have someone take you to an emergency room immediately.


--- layout: page title: "Hi, I'm Taylor" subtitle: Data Scientist / Machine Learning Enthusiast / Hacker meta-title: "Taylor Smith - DS/MLE" meta-description: "Machine learning engineer with a MS in Analytics. Currently a machine learning engineer for Toyota Connected; previously principal data scientist at State Farm, and senior data scientist at AT&T" css: "/css/aboutme.css" ---
Currently a machine learning engineer at Toyota Connected. Previously worked as principal data scientist at State Farm, and a senior data scientist at AT&T. My current position is heavily focused on NLP and operations research. In former roles, I have developed proprietary recommender systems from end-to-end, reduced claims cycle times using text analytics, and used unsupervised learning techniques to identify internal security threats.
Obtained my MS in Business Analytics from the McComb's School of Business at the University of Texas at Austin and my Bachelor's in Psychology from Texas A&M University. My masters capstone (thesis) project involved developing a bespoke visual analytics platform for Boeing.
I enjoy building robust & actionable machine learning models to help my business partners, and useful & reusable software tools to help my fellow data scientists—check out some of my projects (in the navbar above).
I am a recovering R user and now zealously proselytize for the cult of Python.
When I'm not writing machine learning models & software at work, chances are I'm writing code for a side project at home. Some of my more notable side projects include an auto-ARIMA library for Python, an an extensive clustering library for Java.
I am an avid weight lifter, foodie, home chef, and craft beer connoisseur.
If that elevator pitch didn't really do it for you, you can always check out my resume.


Number
The value '' is invalid.
Name
Email
PhoneNumber
Password
The Password field is required.
Gender
Male
Female
The value '' is invalid.
The Password field is required.


|
JavaTM 2 Platform Standard Ed. 5.0 |
|||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object java.security.cert.CRL java.security.cert.X509CRL
public abstract class X509CRL
Abstract class for an X.509 Certificate Revocation List (CRL). A CRL is a time-stamped list identifying revoked certificates. It is signed by a Certificate Authority (CA) and made freely available in a public repository.
Each revoked certificate is identified in a CRL by its certificate serial number. When a certificate-using system uses a certificate (e.g., for verifying a remote user's digital signature), that system not only checks the certificate signature and validity but also acquires a suitably- recent CRL and checks that the certificate serial number is not on that CRL. The meaning of "suitably-recent" may vary with local policy, but it usually means the most recently-issued CRL. A CA issues a new CRL on a regular periodic basis (e.g., hourly, daily, or weekly). Entries are added to CRLs as revocations occur, and an entry may be removed when the certificate expiration date is reached.
The X.509 v2 CRL format is described below in ASN.1:
CertificateList ::= SEQUENCE { tbsCertList TBSCertList, signatureAlgorithm AlgorithmIdentifier, signature BIT STRING }
More information can be found in RFC 2459, "Internet X.509 Public Key Infrastructure Certificate and CRL Profile" at http://www.ietf.org/rfc/rfc2459.txt .
The ASN.1 definition of tbsCertList
is:
TBSCertList ::= SEQUENCE { version Version OPTIONAL, -- if present, must be v2 signature AlgorithmIdentifier, issuer Name, thisUpdate ChoiceOfTime, nextUpdate ChoiceOfTime OPTIONAL, revokedCertificates SEQUENCE OF SEQUENCE { userCertificate CertificateSerialNumber, revocationDate ChoiceOfTime, crlEntryExtensions Extensions OPTIONAL -- if present, must be v2 } OPTIONAL, crlExtensions [0] EXPLICIT Extensions OPTIONAL -- if present, must be v2 }
CRLs are instantiated using a certificate factory. The following is an example of how to instantiate an X.509 CRL:
InputStream inStream = new FileInputStream("fileName-of-crl");
CertificateFactory cf = CertificateFactory.getInstance("X.509");
X509CRL crl = (X509CRL)cf.generateCRL(inStream);
inStream.close();
CRL
,
CertificateFactory
,
X509Extension
| Constructor Summary | |
|---|---|
protected |
X509CRL()
Constructor for X.509 CRLs. |
| Method Summary | |
|---|---|
boolean |
equals(Object other)
Compares this CRL for equality with the given object. |
abstract byte[] |
getEncoded()
Returns the ASN.1 DER-encoded form of this CRL. |
abstract Principal |
getIssuerDN()
Denigrated, replaced by getIssuerX500Principal(). |
X500Principal |
getIssuerX500Principal()
Returns the issuer (issuer distinguished name) value from the CRL as an X500Principal . |
abstract Date |
getNextUpdate()
Gets the nextUpdate date from the CRL. |
abstract X509CRLEntry |
getRevokedCertificate(BigInteger serialNumber)
Gets the CRL entry, if any, with the given certificate serialNumber. |
X509CRLEntry |
getRevokedCertificate(X509Certificate certificate)
Get the CRL entry, if any, for the given certificate. |
abstract Set<? extends X509CRLEntry> |
getRevokedCertificates()
Gets all the entries from this CRL. |
abstract String |
getSigAlgName()
Gets the signature algorithm name for the CRL signature algorithm. |
abstract String |
getSigAlgOID()
Gets the signature algorithm OID string from the CRL. |
abstract byte[] |
getSigAlgParams()
Gets the DER-encoded signature algorithm parameters from this CRL's signature algorithm. |
abstract byte[] |
getSignature()
Gets the signature value (the raw signature bits) from
the CRL. |
abstract byte[] |
getTBSCertList()
Gets the DER-encoded CRL information, the tbsCertList from this CRL. |
abstract Date |
getThisUpdate()
Gets the thisUpdate date from the CRL. |
abstract int |
getVersion()
Gets the version (version number) value from the CRL. |
int |
hashCode()
Returns a hashcode value for this CRL from its encoded form. |
abstract void |
verify(PublicKey key)
Verifies that this CRL was signed using the private key that corresponds to the given public key. |
abstract void |
verify(PublicKey key,
String sigProvider)
Verifies that this CRL was signed using the private key that corresponds to the given public key. |
| Methods inherited from class java.security.cert.CRL |
|---|
getType, isRevoked, toString |
| Methods inherited from class java.lang.Object |
|---|
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
| Methods inherited from interface java.security.cert.X509Extension |
|---|
getCriticalExtensionOIDs, getExtensionValue, getNonCriticalExtensionOIDs, hasUnsupportedCriticalExtension |
| Constructor Detail |
|---|
protected X509CRL()
| Method Detail |
|---|
public boolean equals(Object other)
other
object is an
instanceof
X509CRL
, then
its encoded form is retrieved and compared with the
encoded form of this CRL.
equals
in class Object
other
- the object to test for equality with this CRL.
Object.hashCode()
,
Hashtable
public int hashCode()
hashCode
in class Object
Object.equals(java.lang.Object)
,
Hashtable
public abstract byte[] getEncoded() throws CRLException
CRLException
- if an encoding error occurs.public abstract void verify(PublicKey key) throws CRLException, NoSuchAlgorithmException, InvalidKeyException, NoSuchProviderException, SignatureException
key
- the PublicKey used to carry out the verification.
NoSuchAlgorithmException
- on unsupported signature
algorithms.
InvalidKeyException
- on incorrect key.
NoSuchProviderException
- if there's no default provider.
SignatureException
- on signature errors.
CRLException
- on encoding errors.public abstract void verify(PublicKey key, String sigProvider) throws CRLException, NoSuchAlgorithmException, InvalidKeyException, NoSuchProviderException, SignatureException
key
- the PublicKey used to carry out the verification.sigProvider
- the name of the signature provider.
NoSuchAlgorithmException
- on unsupported signature
algorithms.
InvalidKeyException
- on incorrect key.
NoSuchProviderException
- on incorrect provider.
SignatureException
- on signature errors.
CRLException
- on encoding errors.public abstract int getVersion()
version
(version number) value from the CRL.
The ASN.1 definition for this is:
version Version OPTIONAL, -- if present, must be v2Version ::= INTEGER { v1(0), v2(1), v3(2) } -- v3 does not apply to CRLs but appears for consistency -- with definition of Version for certs
public abstract Principal getIssuerDN()
issuer
as an implementation specific Principal object, which should not be
relied upon by portable code.
Gets the issuer
(issuer distinguished name) value from
the CRL. The issuer name identifies the entity that signed (and
issued) the CRL.
The issuer name field contains an X.500 distinguished name (DN). The ASN.1 definition for this is:
issuer Name Name ::= CHOICE { RDNSequence } RDNSequence ::= SEQUENCE OF RelativeDistinguishedName RelativeDistinguishedName ::= SET OF AttributeValueAssertion AttributeValueAssertion ::= SEQUENCE { AttributeType, AttributeValue } AttributeType ::= OBJECT IDENTIFIER AttributeValue ::= ANYThe
Name
describes a hierarchical name composed of
attributes,
such as country name, and corresponding values, such as US.
The type of the AttributeValue
component is determined by
the AttributeType
; in general it will be a
directoryString
. A directoryString
is usually
one of PrintableString
,
TeletexString
or UniversalString
.
public X500Principal getIssuerX500Principal()
X500Principal
.
It is recommended that subclasses override this method.
X500Principal
representing the issuer
distinguished namepublic abstract Date getThisUpdate()
thisUpdate
date from the CRL.
The ASN.1 definition for this is:
thisUpdate ChoiceOfTime ChoiceOfTime ::= CHOICE { utcTime UTCTime, generalTime GeneralizedTime }
thisUpdate
date from the CRL.public abstract Date getNextUpdate()
nextUpdate
date from the CRL.
nextUpdate
date from the CRL, or null if
not present.public abstract X509CRLEntry getRevokedCertificate(BigInteger serialNumber)
serialNumber
- the serial number of the certificate for which a CRL entry
is to be looked up
X509CRLEntry
public X509CRLEntry getRevokedCertificate(X509Certificate certificate)
This method can be used to lookup CRL entries in indirect CRLs, that means CRLs that contain entries from issuers other than the CRL issuer. The default implementation will only return entries for certificates issued by the CRL issuer. Subclasses that wish to support indirect CRLs should override this method.
certificate
- the certificate for which a CRL entry is to be looked
up
NullPointerException
- if certificate is nullpublic abstract Set<? extends X509CRLEntry> getRevokedCertificates()
X509CRLEntry
public abstract byte[] getTBSCertList() throws CRLException
tbsCertList
from this CRL.
This can be used to verify the signature independently.
CRLException
- if an encoding error occurs.public abstract byte[] getSignature()
signature
value (the raw signature bits) from
the CRL.
The ASN.1 definition for this is:
signature BIT STRING
public abstract String getSigAlgName()
signatureAlgorithm AlgorithmIdentifierAlgorithmIdentifier ::= SEQUENCE { algorithm OBJECT IDENTIFIER, parameters ANY DEFINED BY algorithm OPTIONAL } -- contains a value of the type -- registered for use with the -- algorithm object identifier value
The algorithm name is determined from the algorithm
OID string.
public abstract String getSigAlgOID()
See getSigAlgName
for
relevant ASN.1 definitions.
public abstract byte[] getSigAlgParams()
AlgorithmParameters
and instantiate with the name returned by
getSigAlgName
.
See getSigAlgName
for
relevant ASN.1 definitions.
|
JavaTM 2 Platform Standard Ed. 5.0 |
|||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
Copyright © 2004, 2010 Oracle and/or its affiliates. All rights reserved. Use is subject to license terms. Also see the documentation redistribution policy.


The Early Days
Note: As usual, much of the below is from decades ago and my memory isn’t perfect. In the unlikely event you are mentioned here, it’s possible your memory will be different from mine.
I am often asked how I got into writing software for a living. This is the story of my first real programming job and the strange path that took me there.
In 1982 I was attending Keystone School in San Antonio, Texas. It was a very dark time in my life and my geometry teacher recognized that I was in trouble, so she allowed me to play in the computer room that was reserved for older students. I learned BASIC programming and discovered something in school that I loved. After I left that school to go back to a public school, I started writing (bad) games in BASIC and laboriously typing in programs I found in magazines.
I had begun with Commodore PET computers and moved on to the technological marvel that was the TRS-80 (called the “trash 80”). By 1985, I already written a fully-featured lo-res graphics program for the TRS-80 that allowed users to create images, saved across multiple files (I didn’t understand data compression back then) and stitch those files together to print the images on a dot matrix printer.
Those were the days. I had demo images with that, including a nice, but pixelated minotaur’s head, dragon’s head, and other AD&D images.
One would think that after I graduated high school in ‘85 that I would go on to college and do something with my life, but one would be wrong. We were dirt poor and at the time, to qualify for financial aid for college, I had to first apply for a Pell Grant. That required that my parents provide several years of past tax returns to show what level of financial support they could provide. My mother explained to me that my step father did the taxes. He had failed to file them for years, so no Pell Grant for me. No financial aid for me. No college for me.
The next several years were me bouncing around from odd job to odd job, mostly fast food. At one point I had worked at Wendy’s restaurant in Anchorage, Alaska, Kingsville, Texas, Universal City, Texas, Longview, Washington, and Honolulu, Hawaii. I was a pathetic peripatetic, wandering from state to state, deeply embarrassed that the “Most Intelligent of the Class of ‘85” of Marion High School was an itinerant burger flipper.
Years later, I eventually went to a community college in Portland, Oregon and managed to get an Associate of Science degree with a 4.0 GPA , having taken every economics class they offered with a specialization in international relations.
An Associate’s Degree is a two-year degree and unless you use it to go on to a four-year college, it’s worthless for many jobs.
Barista
Which is how I found myself working as a barista (espresso bartender) when a young, pretty lady walked up and placed her order, leaving a large tip in my tip jar. I was proud of the fact that I was the highest tipped male barista in our chain of stores and I accomplished this by flirting. So I flirted with her, in my naïveté thinking it was innocent fun. I had known her and her boyfriend as regular customers, but that’s all. And that’s when she made me a rather curious offer.
I declined, pointing out I had a girlfriend, to which she replied that my girlfriend could join us.
Er, no. So when her boyfriend arrived later to place an order, I apologized, explaining that through I didn’t know him well, he should know what happened.
He swore, thanked me, and explained that their lifestyle was a bit different. However, they agreed not to proposition people without the other’s knowledge. I still wasn’t going there, but he and I started talking more and eventually we became friends. Turns out he was a software developer and when he found out I used to write software, we started talking about this frequently.
One day he came in to place an order and said he was quitting his job and he’d be happy to recommend me. The software was written in something called UniBasic and he was sure I could handle it. He would stay on for three months to train me.
I got the job and it got my foot in the door, but to my surprise, it was often less professional than the fast food restaurants I had worked at.
My First Test
My friend was leaving his job on the last day of December. I had been on the job for just three months, learning their software, but only being allowed to work on it part-time. Otherwise, I made sure all of the employees were turning in their paperwork correctly. On that last day, the controller called me into his office. He explained that they needed to increase the price of everything by X%, varying per category, and have it ready for tomorrow.
 We were using a UniData multivalue database , prices were scattered all over the place, and I wasn’t very good with their query language. Worse, our license allowed us to write code to read the data, but not change it. Updating this data was done through software running on a terminal and it created audit trails for everything. I needed audit trails, too.
This was a bad idea on many fronts. First, my friend made it clear how much he despised the place and he was leaving at 5PM because he had a New Year’s Eve party to attend. He explained to me how to do the job and I knew enough about the design of our database to figure it out, but it should not have been a rush job.
Next, since software I wrote could only read data, not change it, we had a huge risk. If we changed the data and were caught, we’d have to pay $25,000 (almost $50,000 in today’s money) to buy a copy of the software outright. We were a small company and that would have hurt us. We’d also lose all technical support from the company that “designed” this software. and we’d no longer get free upgrades. Having a few hours heads-up on New Year’s Eve that I needed to risk destroying the company’s business wasn’t fun, but this was how the company worked. So I worked.
After writing the software to update the prices, I ran it. If it worked, it would look like people had spent days manually updating all of the prices, complete with audit logs. I had done it in hours. But if I—a junior developer—corrupted the data, I might be fired. Happy F’ing New Year.
It was getting late in the evening and I was tired and stressed. I ran the software. I corrupted the data.
The only reason I didn’t get fired is that I wrote the software to terminate after the first change and I had printed out all of the data that would be altered by the first run (this seeming much easier than the alternatives in those early days of software development). The controller and I manually fixed the data, I figured out what I did wrong, and ran my software again. This time it worked.
I went over all of the data carefully. I double-checked everything I could. The database was complex and prices for different kinds of goods were stored in different ways, but I couldn’t see anything wrong with what I did. I explained everything to the controller (who, to be fair, had become a friend of mine. He was also surprised by this last minute request). I told him I could remove the code that caused the software to halt after the first change. The software would run overnight and we wouldn’t know until the morning if it worked. The controller told me to do it and get to my party.
January 1st, I get a phone call early in the morning. I’m hung over and not looking forward to this call. It seems they had been checking all of the data they could.
The software worked.
How I Became a Bill Collector
This was a small, family-owned business, with several stores offering a variety of products, many of which had to be ordered months in advance to be handmade in Europe and shipped to the US. The business wasn’t struggling financially, but as far as I could tell, they weren’t getting rich. They needed more money. In fact, I needed more money. I asked for a raise.
At the time, I was paid $x dollars an hour for my administrative role and a couple of dollars an hour more for when I wrote software, capped at 15 hours a week. This wasn’t legal, but I wasn’t slinging espresso or burgers. I never complained, but I still asked for that raise.
We had a new office manager who was not well-liked, was a rubbish manager, but knew our products like the back of his hand, having recently been promoted from warehouse manager. He listened to me, nodded, and explained that they were having trouble with customers ordering goods and not paying, leaving the company with a huge accounts receivable (AR) balance. The company simply didn’t have the money to pay me.
Oh. That’s interesting. In my prior life, I had also worked in car sales. I knew exactly what this was: a negotiation. The manager probably thought he had told me “no,” but I heard his objection and I isolated it .
Me: “You don’t think I deserve a raise?”
Him: “Yes, you do, but we can’t afford it.”
Me: “And this is all because of the outstanding accounts receivable?”
Him: “Yup, sorry about that.”
Me: “So if I can get those people to pay, you can afford to pay me.”
He was trapped by his own words, but to get out of that, he made me an offer I couldn’t refuse, but couldn’t win. If I could get AR to zero by a certain date, I’d get a raise. He wouldn’t budge from zero. We had a lot of outstanding accounts, but not an insurmountable number. However, the goods we sold weren’t cheap and getting people to pay wasn’t going to be easy. I was allowed to take any reasonable steps to enforce this.
That’s how I became a bill collector.
The first thing I did was, of course, write software to produce daily reports for me on the next accounts I would go after, and the steps I had taken to collect on those accounts.
I sent email to everyone I could, and called the others (all customers gave us phone numbers). An astonishingly large number of those people had no idea they owed money and paid up. No one had bothered to contact them! I was surprised at how easy it was. But a large number of them simply didn’t answer.
My next step: I received permission to contract with an actual bill collector. They would buy the debt from us, paying a fraction, and then it was their problem. That’s when I discovered the world of potential clients wining and dining you to win your business. I only accepted lunch from them, but now I realize that even that is dubious.
I chose a bill collector, the company signed the contract, and another large subset of accounts were cleared. But there will still more.
One, in particular, was someone who told me on the phone that she owed nothing. But she did, and I had the paper to prove it. In fact, I often went back to the paper because if this went further, I wanted more than just ones and zeros on a hard disk. She clearly owed the money and it was a lot, so I tried to collect directly rather than hand her off to the bill collector.
Turns out she was a lawyer and immediately replied that we were going to be sued for harassment. The company agreed to write her debt off and the grounds that a trial would cause more pain than her debt. Another account cleared.
On the day of my deadline, the manager greeted me with a smile. He was very happy with the work that I had done but, unfortunately, I hadn’t reduced AR to zero. No raise.
I handed him the AR report. If the owners of the company paid their bills for the products they bought, our AR would actually be positive. Should I send them to collections?
The trap was sprung, the manager was caught. I got my raise.

Three people, Alice, Bob, and Charlie. All of them were nice people and friends of mine. I was still very young and not understanding how business was supposed to work, so I flirted with Alice. She flirted back. We liked each other, but she was also an extremely devout Christian, so my atheism was an issue.
Nonetheless, we continued to flirt. A little harmless fun, right? Bob didn’t flirt with her, but he cracked lots of jokes, was well-liked, and he and I started hanging out after work. He was a great guy; we’ll get back to him. I liked Charlie, too, but Alice didn’t. So when Charlie would make inappropriate comments to her, just like I did, she ignored them.
One day she came to me with a problem. The company was driven by testosterone-fueled cronyism and she didn’t trust upper management. She said she was being sexually harassed by Charlie and didn’t know what to do.
 I wasn’t entirely sure of the law, but a few thoughts crossed my mind.
- I stopped flirting instantly
- If I didn’t act, I could be culpable
- I had to talk to upper management
- She was about to be fired for being our worst salesperson
She reluctantly agreed to let me explain the issue to management, but wouldn’t tell me details. After speaking with management, they were incredibly pissed that they couldn’t fire her. However, they would take no action against Charlie because he was our number one salesperson. They eventually agreed that Alice and I would have meetings to work the situation out, with me being the intermediary between her and upper management. I was not happy about this, but that was that. So some evenings, Alice would leave the sales floor and have private meetings with me in a locked back office, trying to find a way forward. This was not professional, but I didn’t know that. And remember, I left the sales floor and locked myself in an office with a pretty woman who wasn’t selling anything but still kept her job.
Bad optics. Really bad. Bob, my new friend who I was hanging out with a lot, joked in front of Alice and myself that he hoped we were having a good time in the back office. He knew nothing about the sexual harassment allegations and I couldn’t tell him. I just turned to him and replied, “never say that again.”
I lost a friend that day. It hurt.
It was Charlie’s flirting with her that she objected to. She found him repulsive.  Charlie, in her eyes, was a creep.
 Of course, that in no way excused his behavior any more than I had an excuse for my behavior. Alice didn’t know Charlie was gay, but as a devout, conservative Christian, it certainly wouldn’t have made the situation better.
Eventually, Alice left the company (I don’t know the circumstances), and I learned a hard lesson about my personal behavior and what is and is not appropriate in the work place. I also began to get an understanding of how horribly women are often treated and realized that, even if she liked me, I had a hand in that treatment. It’s not a fun lesson to learn about oneself, but the most painful lessons are often the best.
My flirting played a role in my landing that job, and now it played a huge role in several people being hurt.
Leaving
I started going back to college to take COBOL classes and after six months, leveraged that knowledge and my limited experience to get a full-time job at an insurance company. At the time, it was the best paying job in my life. I was a finally a professional software developer.
My first programming job is still my “best” worst job because it was the platform that launched my career, but it was a nightmare. Good people, bad people, and cronyism galore. And decades later, I still feel bad about my role in the mess.


If you have any questions or suggestions about our recognition system. Do not hesitate to contact us. You may email us at :
-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>BSRUI has been programmed and conceived by :
PID:15BTCSE015<EMAIL_ADDRESS>PID:15BTCSE044 (Email:zarqabiqbal@gmail.com)