Here's another area of work for modeling Prosper loans. How do I combine the probabilities of multiple models? I started with the brain-dead approach of using a convex combination of probabilities:
This can be interpretated as the coefficients directly proportional to the amount of information each model has. The optimal weights can be estimated from training data. However I am wondering if I can improve this approach and, of course, the Google is all powerful. Time for some more bed-time reading.
Risk Analysis, Vol. 19, No. 2, 1999. Combining Probability Distributions From Experts in Risk Analysis. R. T. Clemen and R. L. Winkler.
Technical Report no. 543, Department of Statistics, University of Washington, Oct 2008. Combining Probability Forecasts. R. Ranjan and T. Gneiting.
My latest side project has been working on modeling Prosper P2P loans. Features of the data like credit grade, income, etc. are all readily available for import (it's like they actually want you to use this data!). All of the data is imported in MySQL using a script, and then sliced and diced using R.
I created a runtime model, which is an ensemble of various machine learning models (think Netflix prize grand prize solution). The runtime model runs on a Raspberry Pi and uses the Prosper API to place bids automatically.
Since my 20s, I have been meditating on a semi-regular basis; having attended meditation classes, retreats and read multiple books on the subject.
The brain is not idempotent: as the mind thinks, the brain changes too. For example, London taxi drivers with years of navigating the twisty and confusing streets of road developed a larger hippocampus since that part of the brain is used for visual-spatial memories [Maguire et al 2006]. There is increasing evidence that meditation improves cognition [Lazar et al 2006] [Luders et al 2011]
Even after just a few days of training and practice, participants with no prior meditation experienced reduced fatigue and anxiety, together with increased mindfulness [Zeidan et al 2010]. Moreover, brief mindfulness training significantly improved visuo-spatial processing, working memory, and executive functioning.
Meditation trains your mind to treat negative situations with equanimity, which results in a natural state of calm and happiness.
Buddha’s Brain: Neuroplasticity and Meditation (online article)
Can Meditation Make You Smarter? (online article)
Evidence Builds That Meditation Strengthens the Brain (online article)
What is meditation?
In this article, I use the word meditation to refer to Concentration (or Samatha) meditation and Insight (or Vipassana) meditation. There are multiple meditation methods, but the forms that I have practiced extensively with are breath meditation and body scanning.
Samatha is really simple, and is used in the Theravada tradition as the preliminary meditation technique to build a foundation to embark on Vipassana meditation. The practice I have learnt and practiced most extensively is the the breath meditation. Follow the sensations of the breath as it flows in through and nostrils, keeping your attention at the tip of the nostrils. Your mind may wander, but keep patiently returning to the breath.
It is not necessary to develop concentration to the point of excluding everything else except the breath. The purpose of this practice to to allow you to rein in the mind, and allow you to ntoice the workings of the mind. The entire process of gathering your attention, noticing your breath, noticing that your mind has wandered, and re-focusing you attention develops mindfulness, patience and concentration. Deep and sustained practice in Samatha allow you to reach various states of consciousness called Jhanas.
Vipassana are meditation techniques to develop mindfulness and is used to become aware of the impermanence of everything that exists. Vipassana provides you with the guidance on how to see clearly into the nature of the mind, which claims to lead to liberation. I am practicing the body sweeping meditation to develop mindfulness.
Mindfulness in Plain English (book)
How to practice?
My practice in my 20s has been spotty: I would diligently meditate an hour every day for a month, and then lapse into meditating once a week (maybe) for 10 minutes. Even the type of meditation would vary: one month it would be concentration meditation like breath meditation on the tip of the nostril, the next it would be Vipassana or Insight meditation by scanning the body, and yet another it would be loving kindness meditation. Heck, I even tried meditating for a period of time on a burning flame.
A few years back, I realized that this inconsistent practice meant that I was not incorporating the goal of meditation by integrating the practice into my daily life. I was riding the whims of my monkey mind and siloing the practice into a separate aspect of my life.
Unfortunately, meditation as an activity is so mind-numbingly boring (initially). Some motivation techniques suggested can be using the carrot or the stick. The carrot includes the benefits of meditation, both secular (mental wellness) and spiritual (enlightenment). The stick might be to practice “death” meditation to become aware of our morality.
One mental hack that I have learnt over the years is to institute a habit to perform a desirable activity. The label “habit” is very persuasive, especially since you are engaging in an initially undesirable activity. I currently meditate for at least 30 minutes before going to bed, and meditate on weekends for an hour. You only have a finite amount of willpower each day, so forming a habit makes it easier to carry out. How long will it take to form this habit? Research suggests that forming a meditative practice will take over two months of daily repetition before the behavior becomes a habit (and skipping single days is not detrimental in the long-term). [Lally et al 2009]
Habits: How They Form And How To Break Them (online article)
The Power of Habit (book)
Wild Yeast Levain
- 1/4 cup sourdough starter
- 1 and 3/4 cups unbleached bread or all purpose flour
- 1/2 cup lukewarm water
Mix the ingredients by hand for a few minutes. The mix should be slightly sticky but doughlike. Transfer the starter to a work surface and knead for a minute. Transfer to a bowl with oiled surface and leave it until it has doubled in size. The temperature in my apartment falls to about 60F at night and I just leave it out on my countertop overnight. To retard the growth, you can store the levain in the refrigerator for a few days.
Dough (makes two loaves)
- all of levain
- 1 and 3/4 cup lukewarm water
- 4 and 1/2 cup of unbleached bread or all purpose flour
- 3 teaspoons of coarse salt
- [optional] 4 tablespoons of potato flour
- [optional] nuts and seeds, about 4 tablespoons
Sift the flour and salt, then add in the levain and water. Mix by hand for a few minutes until the dough coarsely holds together and then let it rest for 5 minutes. By giving the gluten time to form the bonds, time helps me strengthen the dough and makes the next mixing and kneading steps easier.
After a short rest, continue mixing, adding flour and/or water until the dough is soft and only slightly sticky. This is also when I add the nuts and seeds. Then transfer to a work surface and knead for a few minutes.
Use stretch and fold technique to develop the dough. Let the dough rest for 10 to 30 minutes in a bowl or container, and then stretch and fold the dough. Repeat one or two more times.
After last stretch and fold, transfer to a lightly oiled bowl which is big enough for the dough to double in, and cover. Leave the bowl out for 1 to 2 hours and then leave in the refrigerator until baking day.
I bake in the morning, so I take the dough out the night before. I leave it for 2 hours to warm to room temperature, and then I split the dough into halves. I then shape the dough and leave them overnight (about 10 hours) to proof.
In the morning, I preheat the oven to 500F for 30 minutes. The oven has two pizza stones and a metal pan. So even though the oven might say it's reached the right temperature, I do not proceed until the baking stones are warmed up too.
Turn the shaped dough onto a pizza peel, and then score it ([boule] and [batard]) just before placing it in the oven. Repeat for the other dough. Before closing the door I add one cup of hot water to the metal pan.
After one minute, I spray water to the side of the oven, and then again the next minute. I then lower the oven temperature to 450F. After 15 minutes I rotate the loaves so that the baking is even. I leave the loaves in for another 20 to 30 minutes.
|Bread with sourdough starter|
Recently at work we encountered a problem with the "standard" way of computing confidence intervals for a Binomial proportion using the Normal approximation. That is,
At small sample sizes and high sample proportions, this is too conservative. For example, with a sample size of 10, the confidence intervals for a sample proportion of 0.8 at 95% confidence level, the confidence interval based on the Normal approximation is [0.55, 1].
We can easily see that this confidence interval is too wide as it includes p = 1.0, but we already saw 2 negative examples, which flies in the face of the evidence that p = 1.0 should be on the table.
There are many alternative confidence interval computation methods available, but one particular approach which I prefer is Bayesian: Use a Beta prior disitribution and use that to obtain a posterior distribution (Beta-Binomial). For example, using the Uniform (Uninformed) prior distribution Beta(1,1), the confidence interval (or "credible interval") is the much more plausible [0.48, 0.94].
One of the nice properties of this approach is the subjective knowledge that is inherently available as input to the model. The hyperparameters alpha and beta can be interpreted as alpha - 1 is the number of successes and beta - 1 is the number of failures. So if we strongly believe a priori that the sample proportion is around 0.9, then we could use the Beta(10, 2) as the prior distribution instead, which gives the credible interval of [0.64, 0.95].
Quite simply, it is my sincere belief that there needs to be more companies where there is excellent engineer culture (*). This creates a cycle where the brightest and best will gravitate towards these areas of study. Merely having one or two successful company is not enough, you want a garden of them. Creative technology anyone?
As it turns out, Singapore is starting to have some traction as a start up hub in Southeast Asia. So entrepreneurs - make your millions and billions, and play your part in making software engineering (and technical jobs) a sexy career choice in Singapore!
p.s., I'm very contented in Google; but feel free to pitch to me if you want my opinion on getting stuff done.
(*) You want to foster a culture where innovating is the norm, and the Silicon Valley model has worked well so I am limited by this mindset, but it is certainly true that there are other ways to achieve this.
I just picked up You.next(): Move Your Software Development Career to the Leadership Track which (at the time of this writing) is priced at $2 for the Kindle edition vs $16 for the paperback editiion. It is a very down-to-earth discussion and notes on software engineers making the leap from staying technical and doing what they do well into the mysterious black art of software engineering management.
This book has changed my point of view on managerial roles in software engineering and definitely makes it seem less like a career move towards total brain-rot. ;)
(This is a followup post for Why be an Expert in Singapore? Seriously. As with all blog entries, I speak for myself and not any organization I may be affiliated with.)
On my latest trip back to Singapore, talking to friends in the IT sector here and looking through job listings, it becomes abundantly clear to me that, in Singapore, the software engineer, or programmer, is regarded as the entry-level, don't-think-just-do, job.
Managerial skills are highly valued in Singapore, and permeates throughout most sectors, even the IT sector. I believe this can be partially attributed to the top-down management style of all Singapore government organizations. (And remember, government jobs in Singapore are well-paying and -respected relative to many other countries.) For example, the CEO of the main government scientific research agency is an ex-armed forces chief, the CEO of the main IT regulatory and advisory agency is headed by an ex-navy chief, and the CEO of the media development agency held senior positions in the Singapore Police Force, the Ministry of Trade and Industry, and the Ministry of Manpower.
These individuals, while clearly having been trained with high level executive skills, do not necessarily have the technical background or expertise in the organization that they lead. This is pervasive throughout the organizational structure - mid level managers do not need to be technical, but are often "scholars" who won government scholarships after high school, have been earmarked to reach high level government positions, and are seen as being trained to hold senior appointments .
The manager is viewed as the essential cog in the system. They make the decisions, manage the risks and ultimately are the main reason for the success or failure of projects. The underlying sector is practically irrelevant - be it waste management, education, foreign affairs, or IT. We have specialists for those nitty gritty details.
"What? Google is engineering-driven? You actually let engineers make decisions?"
- Paraphrased quote from Singapore government officer on a visit to my workplace
And, hey, software can be written from a top-down manner. But not all top-down hierarchies are made equal. Tech companies like Apple, Facebook, Microsoft and Google are more alike to each other in their management structure than to the Singapore government management structure; (successful) tech companies have a culture of technocracy.
Technocracy is a form of government in which science would be in control of all decision making.
Scientists, engineers and technologists who have knowledge, expertise or skills would compose
the governing body, instead of politicians, businessmen and economists. In a technocracy,
decision makers would be selected based upon how knowledgeable and skillful they are in their
- Wikipedia article on Technocracy
This dawned on me (and this occurred while I was talking to a bureaucrat who had no business managing a technical project) that I am disillusioned not with the manager being in charge (well, someone has to be), but with the complete inability for non-technical people to grasp the complexities and subtleties of a technical project.
Maybe then... I don't have to explain why you need people who are smart and get things done. And before you start your berate, I am not saying that having a PhD means that a person is a great software engineer. It's just a signal to use when making a hiring decision.
"If i can pay a diploma holder to write a computer software, why should i pay a PhD?"
- Angela in blog comment