race_conditions Coding and running

the lights of this city

Stay strong, London.

Sleep does not come easily on a night like this.

What else is an insomniac to do than stare out of the window and listen to humming of the ventilation.

The view from the apartment opens to the river. In the dark of these early hours, the water is restless, infused with the orange light from the South bank buildings. A bit further to the west, the spires of the city sparkle in gold and red.

And blue. Where moments ago people laughed about the game or a lighthearted joke, the light is now a stacatto blue. Ambulance blue and police car blue; their strobe-like pulse echoes in the empty street.

The phone rings at 23:40. A worried voice checks that I am still alive. How fortunate am I? How fortunate am I to have walked the same streets, laughed the same laughs, enjoyed the late evening warmth and companionship and have made it home.

Somewhere a phone rings without an answer.

deactivate

I love(d) twitter like I love Finnish licorice and winegums and soy vanilla lattes that taste just a bit too acrid and peanut butter popcorn.

But I can’t derive any value from the constant info-flood, the constant buzz of the hivemind forking in thousands of different thoughtpaths, only to degenerate into a static whitenoise occasionally punctured by catgifs and witty, angry quips, all packaged into 140 chars.

I hit deactivate. I don’t regret it. The withdrawal symptoms will come later, when the idle brain is looking for fresh distractions. Then, I think I’ll open up that book on OCaml I’ve been meaning to read. Or read about brainf*ck or Pietr (esoteric programming langs I’ve been meaning to try out).

If a tree falls in a forest and no one is around to hear it, does it make a sound?

And more importantly, does the tree really give a damn?

the quiet

I come home, open up my laptop, log into Twitter and let my eyes tap into the simultaneous titbits of chatter from hundreds of people I follow. It is quiet in my apartment. A lukewarm silence briefly punctured by signals from a Thames Clipper ferrying tired commuters from the piers of Tower Bridge to Greenwich and onwards or an ambulance siren hurtling to some scene of tragedy. The controlled chaos of the city is subsiding into a calm lull of the evening.

It is quiet in the space, the street, the house, my room but it is a cacophony inside my mind. There are thoughts, words, sarcastic quips and angry retorts, distilled into a 140 character essence chattering about. Usually, when I am quiet, it is my thoughts that take tangible form and sift through the day’s experiences, replaying and reliving, molding and transforming into long term memories to be stored and re-narrated at convenient times.

There is no place for carefully crafted thought in the brain that is busy ingesting as much as it can from the delicious stream of fast food infocalories. There is always more. More tweets to read, more likes to administer, maybe even a retweet every now and then. A notification appears and triggers a cascade of pleasure. This is good, something says. Very good. You are. You exist. You exist in the eye of thousands of other semi-strangers who are to consuming this feed. You exist in someone’s eyes. Maybe that gives you some kind of legitimacy or consolation. An illusion of being seen, heard and understood. For sure, the most terrifying thing is not to be alone, but to know it.

I tweet, therefore I am?

I used to eat my feelings. Physically dampen down the chorus of anxiety by flooding the mind with pleasures of cakes and cookies.

A food-junkie’s addiction to the soothing waves of sugar is not unlike our addiction to a constant stream of information. We drown our thoughts with the voices of others, uttering half-formed sentences and ideas until all the mind has to do is become some kind of rating machine, dispensing likes and retweets with Pavlovian efficiency.

Anodyne. Anodyne is the word I am searching for. When you can fill yourself with the words of others, you are absolved. Absolved, relieved, released from the responsibility of living with yourself. It is a temporary pleasure, a temporary relief that turns into something sinister. Where will the narrative of the self come from if not from thoughts and memories shaped in the quiet spaces?

Exploring the Perceptron

I used Gaston Sanchez’ very helpful “Mathjax with Jekyll” post to write up the mathematics in this post. If you are writing a mathematics-heavy post, you may also want to look at the “Jekyll Extras” documentation and at Fong Chun Chan’s “R Markdown to Jekyll: ‘Protecting Your Math Equations’”

As another side note: this is by no means amazing production-ready code, most of this is just exploring and rambling, so please take it as such! Now on to the perceptron!

The Rosenblatt Perceptron was one of the first algorithms to allow computer programs to learn from data and use the learned “knowledge” to classify previously unseen examples. It was based on an early model of the neuron - the McCullock-Pitts(MCP) Neuron. The MCP neuron receives input data at its dendrites, combines this input data into a value ( we will come back to this part later ) and then fires a signal if a certain threshold has been exceeded.

A similar idea is echoed in the Rosenblatt Perceptron. The algorithm receives a set of input data \(\mathbf{x}\). We then feed a linear combination, \(z\), of the input data \(\mathbf{x}\) and some weight vectors \(\mathbf{w}\) into an activation function \(\phi(z)\). If \(\phi(z)>\theta\), the neuron ‘fires’ and we classify the training data into class 1. If the activation function does not exceed the threshold \(\theta\), we classify the input example as -1. These class labels do not necessarily correspond to any ‘real world’ concepts of 1 and -1, but will prove useful later, when we examine how the perceptron algorithm learns from data.

One of the obvious questions arising from this is ‘how do we know the appropriate values to choose for \(\theta\) and \(\mathbf{w}\)?’.

Provided we can supply examples of input data \(\mathbf{x}\) labelled with the true class label, we can train the perceptron algorithm to learn the values of \(\mathbf{w}\) and \(\theta\). In fact, we can rewrite \(\mathbf{w}\) to include \(\theta\). Let’s take a look at an example where the input data is two-dimensional \(\mathbf{x}=(x_{1}, x_{2})\) and the weight vector is \(\mathbf{w}=( w_{1}, w_{2} ) \). We can then write the activation function \(\phi(w_1x_1+w_2x_2)=\phi(\mathbf{w}\^{T}\mathbf{x})=\phi(z)>\theta\). We can also move the \(\theta\) to the other side of the equation to get \(\phi(z)-\theta=w_1x_1+w_2x_2-\theta\geq 0 \). In fact, if we rewrite \(\mathbf{x}=(x_1,x_2, 1)\) and \(\mathbf{w}=(w_1,w_2,-\theta)\), and define \(w_0=-\theta\) and \(x_0=1\) we can express \(z\) as \(\mathbf{w}\^{T}\mathbf{x}\).

Now that we have examined notation involved in the perceptron, let’s take a look at the perceptron algorithm.

Perceptron Algorithm

  1. Initialise the values of \(\mathbf{w}\) to \((0,0,0)\).
  2. For each training data example \(\mathbf{x^i}\), we compute
    • \(\hat{y}\) the precited class of the example
    • we update each entry of the weight vector \(\mathbf{w}\), using the formula \[ w_j := w_j + \delta w_j \]
    • \(\delta w_j\) can be computed as follows: \[ \delta w_j = \eta (y^i-\hat{y})x_j^i, \] where \(\eta\) is the learning rate and is a floating number between 0.0 and 1.0. We’ll examine how this works in a later post.
  3. Continue iterating over the training data until all examples are classified correctly.

A (super quick and dirty, dont-use-this-in-prod) example implementation of the algorithm (in Python ) and a small training example is given below:

def fit(data, eta, max=10):
    """
    data : list of tuples in the format (x0, x1, c)
    eta: floating number between 0.0 and 1.0 
    """

    w = [0,0,0]
    misclassified = 0
    for point in data:
        cp = w[0]*1+w[1]*point[0]+w[2]*point[1]
        if cp!=point[2]:
            misclassified+=1
        w[0] = w[0]+eta*(point[2]-cp)*1
        w[1] = w[1]+eta*(point[2]-cp)*point[0]
        w[2] = w[2]+eta*(point[2]-cp)*point[1]
    epochs=0
    while epochs<=max and misclassified>0:
        misclassified=0
        for point in data:
            cp = w[0]*1+w[1]*point[0]+w[2]*point[1]
            predicted_class = 1 if cp>0 else -1
            if predicted_class!=point[2]:
                misclassified+=1
            w[0] = w[0]+eta*(point[2]-cp)*1
            w[1] = w[1]+eta*(point[2]-cp)*point[0]
            w[2] = w[2]+eta*(point[2]-cp)*point[1]
        print w
        epochs+=1
    print 'Finished'
    print w

def main():
    data=[(0.5, 0.5, 1), (-0.5, -0.5, -1)]
    eta = 0.5
    fit(data, eta)

if __name__=='__main__':
    main()
            

In the next post, we will refactor this implementation a la OOP ( the old Java is hard to lose ) and write some tests to test for regressions.

References

Python: Deeper Insights into Machine Learning by John Hearty, David Julian and Sebastian Raschka

What is going on in this SimPy Monitoring example?

What is going on in this SimPy Monitoring example?

This post is long and rambling (not unlike a midsummer Sunday’s walk around the Surrey Hills) so you may get frutsrated and angry at the author while reading this. The author wishes to apologize to her readers in advance.

SimPy is a simulation library that you can use to easily create event driven simulations in Python. It’s great to play around with if you like simulating queues, traffic systems and monitoring the usage of shared resources ( for example, the number of seats currently occupied on a train or the number of cashiers free at a checkout queue ). In the simplest cases, you can have SimPy print out the usage statistics for your shared SimPy resources to stdout so that you can monitor it, but for longer and more complex situations, it is better to have an automated simulation data collection tool.

Luckily, the SimPy documentation comes in handy and the authors have included a helpful example, which will allow you to patch a SimPy Resource object and monitor it during the simulation. I have reproduced it, with a few comments removed for the sake of brevity, below. Decorators and functools are a bit of a deep sea for me, so in this blogpost I am going to clarify what is going on in the patch_resource function and why it can be used to monitor resource usage in SimPy simulations.

Patching resources in SimPy

from functools import partial, wraps
import simpy

def patch_resource(resource, pre=None, post=None):
    def get_wrapper(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            if pre:
                pre(resource) 
            ret = func(*args, **kwargs)
            if post:
                print 'Calling post function'
                post(resource)
            return ret
        return wrapper
    for name in ['put', 'get', 'request', 'release']:
        if hasattr(resource, name):
            setattr(resource, name, get_wrapper(getattr(resource, name)))

Whoah, whoah there - functools, decorators, some strange @wraps(func) thing! - let’s take it one step at a time!

Functools

Functools is a Python library for higher-order functions - functions that take other functions as arguments, manipulate functions and return new functions. In the patch_resource method, we use two items from the functools module: wraps and partial. Let’s take a look at wraps.

According to the official documentation wraps is a “convenience function for invoking update_wrapper “, another method in the functools module, as a decorator (as the SimPy documentation authors have done in the example above). After reading the documentation for update_wrapper, I’m not really sure what it does - at this point it might be better to poke the function instead of trying to decipher what the documentation means. The update_wrapper function takes in a wrapper function and a wrapped function as arguments, so I am going to setup a simple simplefunc that prints out ‘hello, world’ and a wrapper func, which pretends to do something useful before and after calling simplefunc.

from functools import update_wrapper

def simplefunc():
    print 'hello, world'

def wrapper(func, *args, **kwargs):
    print 'Doing some stuff before calling func'
    value = func(*args, **kwargs)
    print 'Doing some stuff after calling func'
    return value

obj = update_wrapper(wrapper, simplefunc)

According to the documentation, the wrapper function should now look like the wrapped function. If I understood it correctly, I should now be able to call wrapper like simplefuncs.

obj()

results in

>>> obj()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: wrapper() takes at least 1 argument (0 given)

so I have clearly missed something. A quick googling lands me on this SO answer and reveals that update_wrapper has nothing to do with the function signatures. What it does, instead, is carries over important dunder attributes from the original function to the new wrapped function. For example, if we rework the simplefunc and wrapper example above to the following:


def simplefunc():
    '''print hello world'''
    print 'hello world'

def decoratorfunc(f):
    def wrapper(*args, **kwargs):
        print 'f.__name__', f.__name__
        value = f(*args, **kwargs)
        return value
    return wrapper

newFunc = decoratorfunc(simplefunc)
newFunc()

The output is something like this

>>> newFunc()
f.__name__ simplefunc
f.__doc__ print hello world
hello world

However, if we try to print out the name and doc properties of newFunc, which is essentially our simplefunc, just wrapped with some helpful methods, we get the following

>>> newFunc.__name__
'wrapper'
>>> newFunc.__doc__
>>> 

The docstring and name of the original simplefunc have been lost. This is where update_wrapper or its decorator equivalent can come in handy. Let’s rewrite the example above to user the @wraps syntax:

from functools import wraps

def decoratorfunc(f):
    @wraps(f)
    def wrapper(*args, **kwargs):
        print 'f.__name__', f.__name__
        value= f(*args, **kwargs)
        return value
    return wrapper

newFunc = decoratorfunc(simplefunc)
newFunc()
print newFunc.__name__
print newFunc.__doc__

We get the following output

>>> newFunc()
f.__name__ simplefunc
hello world
>>> newFunc.__name__
'simplefunc'
>>> newFunc.__doc__
'print hello world'
>>> 

As the example shows, update_wrapperor its decorator equivalent @wraps preserves the original function’s attributes in the decorated function.