Modular Progress

Modular Synthesizer
modular layout

My Eurorack modular has been progressing. I’ve added several low-cost modules, a Doepfer A-138b audio mixer module so I can mix the audio outputs of the oscilators, the noise output, and my Doepfer Dark Energy modules. I found this so useful with the Dark Energys that I have ordered another mixer. This I can use to mix audio before sending it to the filter, for example, or mix audio before sending it to a VCA to be amplified. I also added another envelope generator, Doepfer A-140, a sturdy workhorse and also useful for triggering the filter independently of the VCA envelope. Then I added a syncable LFO.. I already have 4 LFOs from the quad-LFO module in the top right, but I wanted a LFO with a reset option that I can use for MIDI syncing. On order is also a A-148 S&H (sample and hold) module which I will use for generating interesting random tones once I have expanded the ‘generative music’ theme of my setup a bit more.

I also added a secondhand Kenton Pro Solo, a small midi-to-cv converter which has a lot of features, including clock sync output as well as CV and gate.

Next purchases may include (all Doepfer) a ring-mod, small essential module that it is. I also want a clock divider and clock sequencer, allowing me to take the midi clock output of my Kenton Pro Solo and link it up to activate timed triggers. I eventually want to get a Make Noise ‘Maths’ module as the highlight of my small modular, but the budget does not permit that right now.

Switching Audio between two Soundcards

headphones_accept

If you have two soundcards – maybe a normal soundcard for your speakers and maybe a headset with its own audio interface – you will want some way of switching all audio between the two. This is an great little open-source tool to do just that. It works on Mac, and Windows 7 up to 10:

https://soundswitch.codeplex.com/

Get rid of the ‘most visited sites’ Grid on Chrome

Not that I particularly look at anything weird, but when I’m showing someone my computer I don’t want to have my ‘most visited sites’ pop up when I fire up Chrome. The following Chrome extension will get rid of the default loading page, and just display a blank page:

https://chrome.google.com/webstore/detail/empty-new-tab-page/dpjamkmjmigaoobjbekmfgabipmfilij

Open Source! Device API and Device API-Android on Github

Two Ruby gems that I contributed to from my days in the BBC’s TITAN (Test Tools and Infrastructure) team, have been open-sourced. So finally I can say that I have made proper open-source contributions. The gems are for making automating of Android phones via Android Calabash easier, and are used in a lot of the mobile testing we do at the BBC. Here they are:

https://github.com/bbc/device_api
https://github.com/bbc/device_api-android

Here is my Github profile, which doesn’t look very active on the surface, but I have been contributing to repositories which remain private at the moment, so my history is not visible.

Instruments That I Use

I have a lot of different instruments, both hardware and software, for making music, but here is what I use the most at the moment:

Piano
I have an actual electric piano but I don’t use the onboard sound engine, preferring to use it as a midi controller for a software piano sound. Recently I bought Ableton Live Full Suite, before I was using the ‘Grand Piano’ sound from the default install. Now there is a whole ‘Piano’ Ableton Live pack, and that seems to offer much more realistic piano sounds.

Drums
Lately I’ve been using the Sonic Charge’s Microtonic for drums, because I have been making more dance-orientated music. I find the drums really good, and they’re already fully processed with effects so you can just slot them into the mix. I also have been using the built-in Ableton 909 kit sound a lot, but might move away from that as I don’t like the sound compared to Microtonic. At some point in the future I will probably get my Nord Drum hardware drum synth out, but for now I can’t find the power supply so it’s not working.

Synth
I have been using my modular and semi-modular synthesizer as a main bassline synth, with some sounds from Ableton over the top. I have used the Access Virus synth quite a bit for synth duty, and it has some great presets for evolving arps.

Modular Progress

Specs

eurorack

I have finished filling the top half of my Eurorack case, see above. Here is a list of modules:

Next Moves:

  • Get a logarithmic mixer module A-138b for mixing the output of the noise + two VCOs into the filter, and to free up some of my ZED-10 mixer channels for other things
  • Got another A-140 ADSR module but no cable, going to get a cable and put it into the bottom left corner of the lower section.

Usage

synths

My (mostly) Doepfer Eurorack is hooked up to my Microbrute, 2x Doepfer Dark Energy MK1s and my ZED-10 recording mixer.

The Microbrute acts as a pitch and gate out keyboard controller for all the synths and a semi-modular synthesiser in its own right with its patchbay.

The Doepfer Dark Energys are complete synth voices with a fair amount of input/outputs for modulation. They are hooked up to the pitch and gate out from the Microbrute via the Buff Mult and the Multiples on the Eurorack.

One configuration that I often use when composing music, is wire the Eurorack, the Dark Energys and the Microbrute into the ZED-10 mixer, tune each so they all play together, and create one massive analogue 5-osc synth mono voice which is great for devastating basslines, and I can modulate with the LFOs and envelopes in the system. I can then program a sequence into the Microbrute and transpose it when I play notes down on the Microbrute keyboard, or control it via the USB IN from Ableton.

Another configuration which I find interesting is patching the output of the Noise Generator/Random through the Quantizer. This results in random musical notes played by the oscilator, and you can create interesting pattern or arpeggios created solely by the Euroack without human input. I like the idea of generative music, and will be buying more modules to create more interesting computer generated randomness and music.

Modular Synthesizers

I am putting together a small modular synthesiser setup. Modular synthesizers are synthesizers composed of small individual modules, which are wired up to create a circuit for producing sound. They were very popular in the 1970s, with big bands such as Emmerson, Lake and Palmer, Kraftwerk and Jean Michael Jarre all owning big modular setups:

Keith Emerson's Modular Synthesiser

Modular synthesisers have had a resurgence in recent years with modern versions becoming available. One producer of modular synthesiser modules is Doepfer, a German company who’s modular synthesizer standard ‘Eurorack’ has become very popular recently. I have decided to build a small Eurorack modular, here it is so far:

IMG_0161

The blank plates are spaces for modules that I haven’t bought yet, I am slowly accumulating them. I have currently been wiring my modular setup to my ‘semi-modular’ synths. They include the Microbrute, and 2x Dark Energy MK1s synths, shown below:

IMG_0162

A semi-modular synth is a similar to a standard synthesizer, where all the synthesizer functions are self-contained in one case e.g. they are not just lots of modules. A semi-modular synth will work in its own right, without any additional wiring required. However, a semi-modular synth also has inputs which allow you to optionally wire up the semi-modular synth to other modules and semi-modular synths. So, it allows you to make one big sound processing circuit with all your synths wired together. Which is what I’ve been doing:

IMG_0160

So, not quite Keith Emerson, but it’s a start! But why would you do this, I hear you ask. What possible reason would you have for wiring up synthesisers in this manner? Well:

  • It expands the creative potential of your synthesizers. Instead of just having a few ways of creating sounds, you now have a lot more. Modular synthesisers have been used heavily in EA Games ‘The Sims 3’ for sound effects. Oscar-winning ‘The Social Network’ soundtrack by Trent Reznor uses modular synthesizers heavily, and more.
  • It allows you to build a synthesizer exactly the way you want to, with all the features you wanted for the type of music you make.
  • You end up learning a LOT about synthesis and how synthesizers are made, which can feed in to better sound design in the future, not to mention if you have an interest in electronics and music technology, this is a great way to learn.
  • If you are inspired musically by the technical side of music technology, (and a lot of producers such as Deadmau5 are) then it gives you another creative workflow to experiment with.
  • It’s fun. Wiring up things experimentally and creating sounds you never could have predicted would happen. The combinations are so exponential that you usually have no idea what will happen, and it’s almost like the machine created sounds just for you.

Goals for the Year 2015

Curves on Hightway 1 Northern California

NOTE: I have an update on my progress as of 9/1/2015 here

Motivated by my good friend CheerUpLove.com‘s reflective style of blogging, and Ian Forrester‘s many interesting posts, I thought it would be interesting to post something a bit more personal and less technical on this blog.

Now that 2015 is upon us, I am going to take some time to publicly outline my goals for the year, so I can hold myself more accountable to them, and so I can look back on them this time next year.

Career Goals

One of my most important goals at the moment is to settle into my new role at the BBC as Test Manager. It is a step up for me, and I want to prove my stripes. I also will be pushing for a place to be opened up with Platform Test to employ someone from the Extend Scheme.
What Success Will Look Like : Growing my team, Extendee placement, good performance reviews.

Creative Goals

I have been improving my music production skills a great deal over the past year, a pastime that I find very rewarding. I have also started piano lessons, which I aim to continue with. I aim to continue getting better, and maybe to look into another evening music course – perhaps the Game Audio course at Futureworks Manchester.
What Success Will Look Like : Improved music posted to Soundcloud, collaborations with other musicians, better ability at piano

Social Goals

Getting married is going to be the main point of the year for me, with the wedding coming in September. Lots of preparing to do. I also want to keep the good friendships that I have at the moment, but perhaps make some new friends, particularly those who are into the same interests as me
What Success Will Look Like : Becoming Mr. Lewin, making friends with more people that are interested in the same things as I am

Financial Goals

I need to be saving more over the course of the year. I also want to join the BBC pension scheme, as I don’t really have a pension yet
What Success Will Look Like : Significant savings at the end of the year, joining the pension scheme

Health Goals

My fitness is something I need to become more serious about. I enjoy long-distance indoor rowing for stamina and fitness. I would like to do more of this, and there is a gym just down my road, so there isn’t much excuse for me not to.
What Success Will Look Like : Adopt a regular form of keeping fit

So.. I will look back on this post through the year and hopefully it will motivate me to achieve these goals. Currently I think these goals are achievable, and I think 2015 will be a great year.

microKORG Original + Novation Impulse 61

I picked up a rather ugly (but very cheap!) customised microKORG, and also the excellent Novation Impulse 61 USB/Midi keyboard controller.

mkplusnovation

The microKORG I like a lot. It was my first synth, and at first I couldn’t make any sense of it. I do agree with the Reddit /r/synthesizers view that it is not the best synth to learn on. Their view is that the best synth to learn on is one with a one-to-one mapping between controls and sound engine parameters. To put another way, the best beginners synth is one which has lots of knobs and buttons on it that you can tweak and hear actual results in the sound. It makes it more difficult if the sound engine is hidden away behind a large menu system with lots of daunting menus, like the microKORG has.

However, now I know the basics of synthesis, I’m finding it much easier to get the most out of the microKORG. You can also map a midi controller to a lot of the parameters, turning it into a poor man’s MS2000. The older MS2000 shares the same engine as the MK.

The Novation Impulse 61 is great, although it does take up a lot of desk space. The keybed is much better than any controller I’ve tried before, and even though it’s not weighted like my digital piano, it is very playable. The aftertouch and the velocity sensitivity are settings I will no doubt appreciate more in the future, for now I just turn them off. The arpegiator with the step sequencer is amazing! It turns the arpegiator into a little groovebox. It is quite similar to the arp sequencer on the microKORG, where you can hit the pads to change the notes on the fly. However the Impulse pads are a lot more sturdy, and responsive.

There is a lot about the Novation Automap functionality that I haven’t explored. I want to wire up my old Remote Zero SL unit as a controller for the microKORG, so I have more of a one-to-one mapping between knobs and the MS2000 parameter functionality exposed via MIDI on the microKORG.

Pioneer DDJ-SR Serato Controller Review

dj controller

I decided to buy the DDJ-SR controller because it was cheaper than even the cheapest pair of Pioneer CDJs. I’m no stranger to using a laptop and controller, before the DDJ-SR I used my trusty Faderfox DJ2 controller, which is a basic no-frills ‘play buttons, 3 EQs, crossfader and volume faders’ unit.

I was considering getting the Native Instruments Traktor S4 DJ controller, which is one of the most popular full-size controllers for Traktor. However, after borrowing one from a friend, I compared the build quality of the DDJ-SR versus the S4, and there was no contest! The Pioneer DDJ-SR has the same buttons and jog wheel as the Pioneer CDJ units, which are rock solid and industry standard. For anyone wanting to use CDJs in the club after using the DDJ-SR at home, you will find the feel very familiar.

I looked at the DDJ-SR’s big brother, the DDJ-SX1. The SX is much bigger in size, and not as portable as the SR. This was a major factor for me, as it’s often difficult enough to find space in the DJ booth, even for a small controller. The DDJ-SX1 (not the newest model which may be out by now) has a pretty similar feature set to the DDJ-SR. There is only one main difference that is significant to me. That being the fact that the SX is a full 4-channel mixer, which can operate without being connected to a computer. But it wasn’t enough to justify the larger footprint and the much increased cost. So I decided to opt for the more portable DDJ-SR.

I tried out the controller at my local music shop, Dawsons (Manchester) . I recommend trying out the controller and also buying locally if you can, because it’s much easier to return a controller to the shop than it is to send it back to an online store, if you are not happy with it. I also believe in supporting your local music stores.

Being used to Traktor, and having used it since 2006, I was initially a bit cautious over switching to Serato. After I played around with it, and for my typical use, there was not a lot in Traktor that I didn’t find equivalents for, in Serato. One thing that was very nice is seeing the waveforms from each deck, and how you can visually ‘sync up’ the starting track with the beats of the one that is already playing. I have been surprised about how useful this is, it really makes beatmatching so easy when you don’t want to use the sync button, which I often don’t.

Pioneer_DDJ-SR

The DDJ-SR comes with 8 pads under each jog wheel. They are very responsive, every bit as responsive as Akai drum pads used for music production. The pad FX definitely expand the creative potential of laptop DJing. The ‘slicer’ is my favourite so far, it cuts the upcoming music into 8 slices. You can choose to play a slice (or beat) out of time with the progression of the track, for example, instead of 1-2-3-4-5-6-7-8 you can opt for 1-2-2-4-5-6-6-8. You can play combination you like, assuming you hit the right pads in time!

The ‘slip’ feature is very nice. When toggled on, the track plays on as you trigger samples and loops. It is as if the track is kept playing in the background while you pick out sections earlier in the track and play samples over the top, and when you have finished, the track snaps back, not to where you left it, but to where it has progressed to. You can use this feature to replace whole loops with other loops in the track and keep the rhythm of the track going. It is really powerful.

There is a four knob ‘FX’ section above each jog wheel. This is used to add FX such as delay, reverb, flanger, and so on. I don’t find myself using this much for my style of DJing. I do wonder how much of the FX and pad hammering you can actually do to a track before you start to annoy your crowd. They are great to use occasionally though.

Overall, I am very happy with my purchase. I had a pair of Pioneer CDJ400s and mixer back in 2009 which had a lot less features and cost over twice the price as this unit, and yet had the same build quality. Even if you ignore the ‘which is better’ argument of digital DJing versus CDJs, if you already have a laptop, this is a great setup for the money.

New Synthesiser Additions: Microbrute, Nord Drum and Access Virus A

I bought some new synths.

Access Virus A

access virus a

This is the original 1998 Access Virus A. It, it was a staple synth for trance and electronic music in general for the early 00s. I am very happy that I managed to get this secondhand for less than a MicroKORG. It cost over £800 when it came out in 1998. Old digital synths seem to be unpopular at the moment, I’ve heard comments along the lines of ‘if it’s digital then I might as well use a plugin in my DAW’. But I think this is ignoring the tactile control surface that you get, and how it gets your head outside the computer and playing an actual physical instrument. The presets are really inspiring, several sounds that you will recognise if you were into electronic music in the 00s. VNV Nation, one of my favourite bands, produced their entire ‘Empires’ album with just the Access Virus A and a couple of samplers.

It has two oscillators which have extra functionality, as you can choose the waveform from a large selection of presampled waveforms, in addition to the usual sawtooth, sine, square. There is also a suboscilator, two filters, each with a ASDTR configurable envelope, the usual ASDTR for the amplifier, and three LFOs. You can choose to map the LFOs to almost anything, allowing a great amount of flexibility. It also has effects – delay and unison, as well as multiple timbers, meaning that you can have up to 12 different patches playing at the same time, each controlled with different MIDI channels. The whole thing is extensively controllable via MIDI also. It is my first polyphonic hardware synthesier, and so has opened up a whole range of new composition possibilities around chords.

Microbrute

microbrute

The Microbrute is fully analogue, and it can produce a good number of sounds through its one oscilator, by allowing you to blend in different quantities of sawtooth, sine and square, along with some extensive extras, such as a metalizer. It has a Steiner Parker filter, a LFO and a step sequencer. You can get more technical details about the Microbrute in Marc Doty’s great series of videos.

It has a hard aggressive character to the sound, especially when used with the ‘brute factor’. I like its sound, but sometimes wish for a little bit more warmth. However you can make all sorts of sounds with it, and I particularly like making chiptune arpeggio with the square wave, such as in my track ‘Blue Sky’.

It also has a mod matrix that I haven’t fully explored yet, but is very powerful, especially if you want to link it up to other CV capable synths.

Nord Drum

Nord-Drum 1

This is an analgoue modelling drum synthesiser. It is the first Nord Drum so it only has four channels, but the presets sound great in the mix. I haven’t even looked at the synthesis options on this yet, as I’ve been really happy with the presets, but there is a whole range of options that you can shape and edit for your own sound.

My synth ‘shelf’

I have swapped out the older synths that I don’t use anymore. Currently I’m using the three synths above. The keyboard at the top is my Alesis Q25 MIDI controller, there is a ZED 10 mixer on the middle shelf which I use as a recording device, as well as a gold EMU Orbit which I don’t have plugged in at the moment.

My Synth Shelf

A* Algorithm implementation in Python

pathfinding

Lately I’ve had the idea of creating a text-based Roguelike in C++. This lead me on to think about the game AI experiments that I worked during my degree in Computer Science and A.I.. Essential to game AI is the notion of pathfinding, or finding a path from ‘A’ to ‘B’, past any obstacles that get in the way. One way to do this is to use the A* algorithm. I decided to implement an A* pathfinding algorithm for possible use in a Roguelike later, and chose the pseudocode from the Wikipedia example to implement.

The program finds a path from the top right hand corner to the top left, avoiding impassable ‘7’ obstacles. The ‘*’ are the steps along the path. The algorithm is guaranteed to find the shortest path between the goal and the start, which means it can optimally solve any solvable maze, given time.

This is a sample board with obstacles setup:

00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000777000000000000000
00000000000000000000000000000000777000000000000000
00000000000000000000000000000000777000000000000000
00000000000000000000000000000000777000000000000000
00000000000000000000000000000000777000000000000000
00000000000000000000000000000000777000000000000000
00000000000000000000000000000000777000000000000000
00000000000000000000000000000000777000000000000000
00000000000000000000000000000000777000000000000000
00000000000000000000000000000000777000000000000000
00000000000000000000000000000007777000000000000000
00000000000000077777777777777777700000000000000000
00000000077777777777777777777777700000000000000000
00000077777777777700000000000000000000000000000000
77777777777000000000000000000000000000000000000000
77777777000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
77777777777777777707777777777777777777777777777777
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
70777777777777777777777777777777777777777777777777
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
77777777777777777777700000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000
00000000000000000000777777777777777777777707777777
00000000000000000000700000000000000000000000000000
00000000000000000000700000000000000000000000000000
00000000000000000000700000000000000000000000000000
00000000000000000000700000000000000000000000000000
00000000000000000000700000000000000000000000000000
00000000000000000000700000000000000000000000000000
00000000000000000000700000000000000000000000000000
00000000000000000000700000000000000000000000000000

This is the path found (the ‘*’s):

**000000000000000000000000000000000000000000000000
0***********************************00000000000000
00000000000000000000000000000000777*00000000000000
00000000000000000000000000000000777*00000000000000
00000000000000000000000000000000777*00000000000000
00000000000000000000000000000000777*00000000000000
00000000000000000000000000000000777*00000000000000
00000000000000000000000000000000777*00000000000000
00000000000000000000000000000000777*00000000000000
00000000000000000000000000000000777*00000000000000
00000000000000000000000000000000777*00000000000000
00000000000000000000000000000000777*00000000000000
00000000000000000000000000000007777*00000000000000
00000000000000077777777777777777700*00000000000000
00000000077777777777777777777777700*00000000000000
00000077777777777700000000000000000*00000000000000
77777777777000000000000000000000000*00000000000000
77777777000000000000000000000000000*00000000000000
00000000000000000000000000000000000*00000000000000
00000000000000000000000000000000000*00000000000000
00000000000000000000000000000000000*00000000000000
00000000000000000000000000000000000*00000000000000
00000000000000000000000000000000000*00000000000000
000000000000000000******************00000000000000
777777777777777777*7777777777777777777777777777777
000000000000000000*0000000000000000000000000000000
0******************0000000000000000000000000000000
7*777777777777777777777777777777777777777777777777
0*****************************00000000000000000000
00000000000000000000000000000**0000000000000000000
000000000000000000000000000000**000000000000000000
0000000000000000000000000000000*000000000000000000
0000000000000000000000000000000***0000000000000000
000000000000000000000000000000000*0000000000000000
000000000000000000000000000000000**000000000000000
0000000000000000000000000000000000**00000000000000
77777777777777777777700000000000000***000000000000
0000000000000000000000000000000000000**00000000000
00000000000000000000000000000000000000*00000000000
00000000000000000000000000000000000000**0000000000
000000000000000000000000000000000000000****0000000
000000000000000000007777777777777777777777*7777777
000000000000000000007000000000000000000000**000000
0000000000000000000070000000000000000000000*000000
0000000000000000000070000000000000000000000***0000
000000000000000000007000000000000000000000000**000
0000000000000000000070000000000000000000000000*000
0000000000000000000070000000000000000000000000***0
000000000000000000007000000000000000000000000000*0
000000000000000000007000000000000000000000000000**

Here is source code

Amit’s A* pages were incredibly useful in developing this.

(Perhaps one day I will do a flashy JavaScript version!)

Current Music Setup

I like to dabble in making and playing music. Here is the equipment and setup that I use.

Ableton Live Intro

This is the DAW I use. I have been using Ableton Live for several years now, and I know my way around it. I have tried others but they never really felt as comfortable. Live comes in three flavours, ‘Intro’ is the most basic paid-for version. I was surprised how cheap the ‘Intro’ version was, and how little I find myself needing the features of the more expensive versions. I only have one paid-for VST plugin, and that is the KORG Legacy Cell MS-20, which I use on almost all the music I’ve made.

ableton live example

KORG Electribe R-1 MK2 Analog Modelling Drum Synthesiser

This is what serves as my drum machine. I picked it up on the cheap, but I’m really happy with it. I really like the built in step sequencer, and the drum synth itself. The sounds you can make from it are quite varied, and I really like the way you can put together live compositions on the fly. I’d be interested in getting some more KORG boxes, such as the EMX Electribes or the Volcas.

korg-electribe-er1-mk2

Waldorf Rocket Monophonic Analogue Sythnesiser module

This is my analogue synthesiser, which I use mainly for leads. In practice I find it quite annoying not being able to play real chords, and the lack of a sequencer is difficult because it means I can’t put together patterns away from the PC, or even really play it without a keyboard. I do like the sound, and the filter is great. The ‘fake chords’ option is great fun I just wish you could more easily change between the chords on the fly. The arp is pretty cool also. I think this will become a lot more awesome when I get a hardware sequencer, I already have my eye on the cheap BeatStep to pick up when it is released.

WALDORF+ROCKET-1

KORG Monotribe Analogue Sythesiser and Drum Machine

I don’t really use this for anything, unfortunately. It was my first synthesiser and although I liked it at the time, nowadays I find the lack of MIDI and the ribbon keyboard a major problem. I might either sell it or get a MIDI mod and install it. I find it quite noisey when recording anyway, so maybe I’ll just try and sell it. It does has some nice acid style sounds though. A shame.

monotribe

Yamaha P35 Digital Piano

I was looking for a decent MIDI out keyboard that would allow me to put together more complex melodies and learn how to play keyboards for possibly joining a band in the future. The Yamaha NP31, which was my original choice, was out of stock and so it got me looking at others. I thought it would be interesting to learn to play the piano, as we had one growing up but I never really learned before. In the store I really liked the feel of the P35’s hammer action keys, so decided to buy it. I can play some simple tunes with it but want to get piano lessons so I can improve. This will also function as my main MIDI controller for my other hardware.

717aZc6BIwL._SL1500_

Line 6 Mobile Keys 25 USB Keyboard Controller

This is a nice little keyboard controller that I use with Ableton. Unfortunately it does not have MIDI out, only MIDI via USB, but I can run it through the MIDISport to control audio hardware. The keyboard action is very nice, much better than the M-Audio Keystation 49 which I had before and would not recommend.

controller

M-Audio MIDISport Anniversary Edition 2×2 MIDI/USB Interface

This is a fairly standard cheap MIDI to USB interface. I used to use the M-Audio Uno, but that has only one input/output and it has problems with some audio hardware.

midisport

Setting up Kindlefire HDX for Development under Ubuntu 12.04

amazon_kindle_fire_hdx1

I wanted to get a Kindlefire HDX running under Ubuntu 12.04 with adb.

First I needed to setup the udev rules:

1. Edit /etc/udev/rules.d/51-android.rules as root, and add the following line (create this file if it does not exist):

SUBSYSTEM=="usb", ATTRS{idVendor}=="1949", MODE="0666"

2. Change the permission of this file by executing the following command as root:

chmod a+r /etc/udev/rules.d/51-android.rules

3. Reload the rules by executing the following command as root:

udevadm control --reload-rules

4. Run these commands to restart adb:

adb kill-server
adb start-server

5. Now when I run

lsusb

I can see the device listed.

6. Next I needed to enable adb access on the Kindlefire HDX device itself by going to Settings -> Device -> Enable ADB.

7. Finally I could run:

adb devices

within Ubuntu and have it recognise the Kindlefire HDX.

My Computer Build

Updated: Reorganised, replaced Tiamat headset for KAVE.

Picture of my desk with computer on it

Picture of my computer base unit

I use my computer a lot, for work (software development), gaming and music production. I wanted to build a high-end computer setup, as I’ve always lagged behind with upgrades before. So I gradually have been accumulating and swapping bits around to make the setup above.

Internals

Visual

Audio

Peripherals



Internals


Case – Thermaltake Level 10 GT Full Tower

[Link to Amazon]

case

This was an expensive case, but I wanted to spend some money to get a quality case that I could use for future builds. It has many fans, most of which I don’t need, but they are very quiet and they make the CPU temperate a very cool 30-40 degrees, under regular load, even in the current heatwave we’re having. The case has 5 hotswappable hard drive caddies which is very nice and allows me to easily install and remove hard drives without having to take apart the computer. It also has a lock, USB3 ports and enough clearance for graphic cards and space for lots of upgrades.


Motherboard – Gigabyte Z77X-D3H 1155

[Link to Amazon]

mobo

This was bought because it was a cheapish seat for the Ivy Bridge processor and the 16GB RAM. It has some niceish features, such as the fancy 3D BIOS which you can navigate with a mouse, and the onboard graphics which is enough to get into the BIOS when your graphics card isn’t working. The onboard sound is best avoided.


CPU – Intel Core i7 (3770) 3.4GHz Quad Core

[Link to Amazon]

processor

This was the processor that was the top of the mainstream mid-high end CPUs when I was purchasing. It differs from the 3770k as you can’t overclock it, but it has decent support for hardware virtualisation, which I wanted and is not available in the 3770k. I figured I wouldn’t miss the chance to overclock this for a good while, as CPUs of this type already are vastly overpowered for the applications that I use. CPU utilisation for typical use (not including games) rarely rises above 10%.


CPU Cooler – Arctic Cooling Freezer 13 Pro

[Link to Amazon]

cooler

This is a bit overspecced since I am not planning to overclock, but I don’t like stock heatsinks and so bought this. It is a huge heatsink and probably wouldn’t fit in many other types of cases, but as I have a full tower it’s fine.


RAM – 16GB DDR3 RAM Ballistix DDR3 PC3-17000

[Link to Crucial]

ram

This is fast RAM for gaming. 16GB is a bit overspecced for my current needs, but it’s the maximum my board will take, and means I won’t have to upgrade for a long while.


Wireless Card – TP-LINK TL-WDN4800 N900 Wireless Dual Band PCI Express Adapter

[Link to Amazon]

wireless card

This wireless card is good because it has dual-band, meaning faster, interference free 5GHz spectrum access to my dual-band wireless router, and because it has three external ariels which you can replace. I have got a large indoor antenna replacement which allows me to double my wireless range, meaning I can pick up local wifi hotspots if needed.


Optical Drive – Pioneer BDR-207DBK 12x Internal BD-RW Burner

[Link to Amazon]

optical

I didn’t really need a Blu-Ray burner, but it’s useful for backups and I thought I could write some discs to be played in my PS3. It has a very fast read speed for reading Blu-Rays and burning DVDs. To be honest I could have gone for a Blu-ray reader and DVD writer combo and not really noticed the difference.


PSU – 850W EVGA Super NOVA Gold Modular

[Link to Amazon]

psu

850W allows a lot of headroom for graphics cards’ power requirements, and extra components in the future. It is a good quality PSU and is fully modular which is a lot nicer to work with.


3x Hard Drives –

1x Crucial CT256M4SSD2BAA 256GB SSD
1x Unknown model Crucial 120GB SSD
1x Seagate 3TB 3.5 inch 7200RPM 64MB Cache SATA3 Hard Drive

harddisk
harddisk3

The 256GB SSD is my system drive, the older 120GB SSD is for games, and the 3TB HD is for storage: my music collection, video files collection, backup images and virtual machines. Being SSDs, the 256GB and 120GB drives are totally silent, making the 3TB HD sound really loud when it spins up. However, the 3TB drive has enough space to store full Blu-Ray ripped files.. not that I would do that, of course 🙂


Back to Top of Post



Visual


Monitor – BenQ BL3200PT AMVA+ 1440p 32″ Widescreen

[Link to Amazon]

monitor

This is a high-end 1440p 32″ monitor. It has made such a big difference to using the PC, more screen space, more pixels and better clarity. I was considering a 4K monitor, but don’t think the technology is quite cheap enough yet, and the software and games support is severely lacking. It’s 60HZ refresh rate, but that is enough for me. I have heard that the higher refresh rate monitors are better for gaming, but a 144HZ 1440p 32″ monitor would be way out of my budget, not to mention the graphics processing power it would require to run.


Graphics Card – Asus STRIX GTX 980 4GB GDDR5

[Link to Amazon]

gpu

I had to upgrade my old Nvidea GTX 660TI, and just went for the absolute maximum I could afford. The STRIX 980 is an amazing card and comes slightly overclocked compared to the reference model. It plays all current games on maximum settings on 1440p at over 30FPS, which is no a mean feat, on my resolution there is more than a million pixels being drawn to the screen on each frame.


Monitor Backlight – Lightpack

www.lightpack.tv

lightpack

This is a monitor or TV backlighting kit that changes the colour of the light projected behind your screen to match the colours on the edges of your display. This provides a more immersive experience when playing games with the lights down, and you can also set it to become a graphic equaliser where the lights flash in time with the music. I bought this on a whim and I’ve been quite impressed with it so far. It definitely adds to immersion and the graphic equaliser mode is rather trippy when writing or playing music.


Back to Top of Post



Audio


Headphones – Sennheiser HD 25-1 IIs

[Link to Amazon]

headphones

These are what I use when I can’t use my speakers. I bought them for DJing and audio work. They have great sound isolation (these are used by sports commentators you see on TV, because when you’ve got them on, you literally can only hear what’s coming through the headphones) and great sound quality (a favourite for top DJs and sound engineers). They don’t look that great though, and aren’t the most comfortable, but whatever. One nice feature is that they are designed for heavy use and are almost indestructable. If someone steps on the headphone lead, they won’t break, the cables will just pop out of the cans. If the headset breaks, well you can remove the cans and just buy a new headset. They have high ‘sound pressure’ which means the ratio of sound that enters your ear versus the amount of sound that comes out of the cans, is quite high due to their closed nature and sound isolating design. This, combined with high frequency range and a large volume range, means it’s quite easy to cause yourself hearing damage if you have them on too loud usually, or if you have a temporary high volume ‘spike’.


Soundcard – Asus ROG Xonar Phoebus Solo

[Link to Amazon]

soundcard

Good 7.1 surround sound card, with headphone amplifier.


Gaming Headset – ROCCAT Kave XTD 5.1 Analogue

[Link to Amazon]

headset

It is quite difficult to find surround sound headsets with analogue inputs, most headsets are USB only, and contain their own soundcard. This is not ideal when you have a really good soundcard already that you want to use, such as my Xonar Phoebus Solo. Previously I was using the Razer Tiamat, which was terrible. It had a consistant high-pitched electrical whine when you were wearing the headset, and you couldn’t get rid of it. The Kave is much better in this regard, and is more comfortable too. It is ‘only’ 5.1 compared to the Tiamat’s 7.1, but really, 7.1 surround sound in a headset is just a marketing tactic anyway, you are never going to be able to appreciate the extra two channels when the speakers are that close together.


Speakers – KRK Rokit G2 5

[Link to Amazon]

speakers

These are great monitors, which I use for music production, and make for great speakers for general use as well. They are a little large for my desk and have a maximum volume which is far too loud for my flat, but I wouldn’t be without them.


Speaker Stands – IsoAcoustics L8R155 Speaker Stands

[Link to Amazon]

speaker stands
These are sound-isolating speaker stands, which I have setup to raise the speakers to ear height. I have to say, they’ve made the speakers sound so much better, and they look good too.


Mixer – Allen & Heath ZED 10

[Link to Amazon]

mixer

This is a good recording mixer which also doubles as a great USB soundcard. It has 4x mono RCA inputs and 2x stereo RCA inputs, which is enough to wire all my synths up, as well as feed the Rokit 5s.


Desktop Microphone – Pulse 50’s Retro Chrome Style Microphone

[Link to Amazon]

<a href="http://www.davidcraddock.net/wp-content/uploads/2014/mic

This is a 50’s style microphone that I liked the look of. I got it because I wanted a microphone that plugged into my mixing desk, which would allow me to talk hands-free on Skype. Also it is good for streaming games, as I can mix my voice in the microphone with the output the computer game sounds, and send both mixed to skype or Twitch.TV or wherever. It is also moderately useful for recording voices for music production and for routing via my FX pedal for some novelty vocal effects on Skype, such as delay, echo and reverb.


Back to Top of Post



Peripherals


Webcam – Logitech C920 HD Webcam

[Link to Amazon]

webcam

I use a webcam for Skype often, and I want to get more into recording for YouTube, so I bought a good quality HD webcam.


Keyboard – Das Keyboard Ultimate Silent EU

[Link to Amazon]

keyboard

This is the model with the quieter Cherry MX Brown key switches. It is a good mechanical keyboard for all purposes, looks good, and will encourage me to type more accurately. I grew annoyed at missing keys on the blank keycaps, so I added my own coloured keycaps for high visibility (see picture at top).


Gaming Keypad – Razer Orbweaver Stealth

[Link to Razer Site]

keypad

This is the ‘Stealth’ model with the Cherry MX Brown key switches. I was initially doubtful whether it was worth getting a gaming keypad, but so many people have recommended it, that I had to try it out. I actually have found it very useful, much more comfortable and accurate than using a normal keyboard for gaming. It also forces you to get into the habit of remapping keys for your own preference.


Mouse – Logitech G9X

[Link to Amazon]

g9x

I upgraded from my G500 mouse to this, smaller, G9X. I have started to adopt the claw grip for gaming and generally using my mouse. This makes the larger G500 difficult to use, as the larger size means that the mouse buttons don’t click properly when you have your fingers higher up on the mouse. So far, I am very happy with the G9X, the higher DPI scanning, and the easier to grip ‘precicse grip’ material means that it is better for other reasons for me.

I use the Corsair Vengeance mouse matt, it is made of metal unlike my previous Razor Golliathus cloth one, which I didn’t like because the edge of the matt would get stuck in the mouse when you moved it too far.


Back to Top of Post


My Aeron Chair

A good ergonomic chair is a wise investment if you’re going to spend a lot of time at your computer. One of the better known ergonomic models is the Herman Miller Aeron Task Chair.

Picture of Aeron Chair

What Other People Say About the Aeron

Jeff Atwood (from Coding Horror) says:

In fact, after browsing chairs for the last few years of my career, I’ve come to one conclusion: you can’t expect to get a decent chair for less than $500. If you are spending less than that on seating – unless you are getting the deal of the century on dot-bomb bankruptcy auctions – you’re probably making a mistake.

I still believe this to be true, and I urge any programmers reading this to seriously consider the value of what you’re sitting in while you’re on the job. In our profession, seating matters:

Chairs are a primary part of the programming experience. Eight hours a day, every day, for the rest of your working life – you’re sitting in one. Like it or not, whatever you’re sitting in has a measurable impact on your work experience.

Cheap chairs suck. Maybe I’ve become spoiled, but I have yet to sit in a single good, cheap chair. In my experience, the difference between the really great chairs and the cheap stuff is enormous. A quality chair is so comfortable and accommodating it effortlessly melts into the background, so you can focus on your work. A cheesy, cheap chair constantly reminds you how many hours of work you have left.

Chairs last. As I write this, I’m still sitting my original Aeron chair, which I purchased in 1998. I can’t think of any other piece of equipment I use in my job that has lasted me ten full years and beyond. While the initial sticker shock of a quality chair may turn you off, try to mentally amortize that cost across the next ten years or more.

Joel Spolsky (from Joel On Software) says:

Let me, for a moment, talk about the famous Aeron chair, made by Herman Miller. They cost about $900. This is about $800 more than a cheap office chair from OfficeDepot or Staples.

They are much more comfortable than cheap chairs. If you get the right size and adjust it properly, most people can sit in them all day long without feeling uncomfortable. The back and seat are made out of a kind of mesh that lets air flow so you don’t get sweaty. The ergonomics, especially of the newer models with lumbar support, are excellent.

They last longer than cheap chairs. We’ve been in business for six years and every Aeron is literally in mint condition: I challenge anyone to see the difference between the chairs we bought in 2000 and the chairs we bought three months ago. They easily last for ten years. The cheap chairs literally start falling apart after a matter of months. You’ll need at least four $100 chairs to last as long as an Aeron.

There was a post with a large set of comments on Hacker News about chairs and cheap alternatives to Aeron Chairs, with a lot of people saying that there is no cheap alternative to a good chair.

What I Say About the Aeron

I just got a refurbished Aeron chair for Christmas. I am really pleased with it. It comes in three base sizes, being larger than normal I went for the Size C. Once it arrived it was a good fit, but it was easily configurable to make it into an excellent fit. I prefer my chair not to recline, so I can’t comment on whether it is not a good chair for reclining, as Jeff Atwood says, but I will say it felt comfortable and stable reclining.

The real benefit so far has been on my posture; I did not know how much back pain I was having before with my really cheap office chair. Only when I got up from a 4 hour session on my Aeron chair did I notice how much better my back was feeling. It moulds your back into a good position, and I have noticed my back clicking and popping into its proper shape, after using the chair for a couple of days.

I ordered my chair from simplyaeron.co.uk, who I am very happy with, as they provided an MK2 Size C chair with lumbar support for less than £400, which is an absolute bargain compared to the retail price of a new chair. This also included delivery! It seems like new – there are no scuff marks and everything works perfectly.

I highly recommend getting an Aeron chair or an equivalent ergonomic chair recommended in the above resources, as I have really noticed a significant difference.

2013 Career Retrospective

2015 Update: The “Device Hive” project has now been changed to be called ‘Hive CI’ and it is being maintained by a team of developers at the BBC, which I am no longer a part of. It is in the process of being opensourced, see: http://bbc.github.io/hive-ci/

This year has been quite a busy and eventful one for me.

Connected Red Button
At the start of the year, I was working on the Connected Red Button team within the BBC. Connected Red Button is a major ongoing project in the Television and Mobile Platforms department at BBC North. Its aim is to replace the classic Red Button text service (which itself is the successor to Ceefax) with a new updated all-singing all-dancing interactive portal to internet content, available on Smart TVs and modern set top boxes. Currently Connected Red Button is live and accessible by pressing the Red Button on the new Virgin Media TiVo boxes. You can access the latest version of iPlayer, and the BBC News and BBC Sport smart TV apps from within one easy portal.

On CRB, I was working on the Java/Spring services layer, which connects to the various APIs of services like iPlayer that we have at the BBC, and gets all the content ready for the frontend. This data then gets passed to the very nice looking AS2 frontend to display, and that’s how it appears on your TV that is connected to your Virgin TiVo box in your living room.

The next version of CRB is being developed for Smart TVs with HTML browsers (so the frontend is in HTML5/JS instead of AS2). This type of Smart TV includes most of the new smart TVs that have come out recently, and will continue to be released in the future. The BBC (and the wider industry) is really anticipating that most TVs will be smart TVs in 5-10 years, and so the reach of Connected Red Button HTML will increase substantially so that most of the audience can be served by new applications that run on smart TVs.

Smartbridge

Smartbridge is the transitional frontend that is displayed to both 1) users that have smart TVs and internet connected STB (Set Top Boxes) capable of running our latest BBC applications such as Connected Red Button and the latest versions of iPlayer, and also 2) our traditional users that still have normal (un-smart) TVs that can only receive Broadcast Red Button (the service you get by pressing the red button on any BBC channel). Smartbridge is not a branded BBC product, it is the behind the scenes magic that helps ensure that we maintain the availability of traditional Red Button services as we simultaneously launch and develop Connected Red Button.

For several months this year I was working on Smarbridge. On Smartbridge, I was working to get the project released and out the door, which meant Java/Spring/Hibernate work, with MySQL database tinkering and some broadcast work configuring and testing the TVs that worked with Smartbridge. It was successfully released in October.

Device Hive

Device Hive is the working name for an BBC system that is an Android and iOS emulator and physical device testing platform. A server will run the Device Hive software, and mobile developers will be able to plug in their Android mobile or tablet or iPhone or iPad to the server via a USB cable, and choose an application to run on it, such as BBC iPlayer or BBC Radio Player. This application will then automatically be downloaded onto the device, and the automated Cucumber/Calabash test suite will be executed, which will step through every screen of the mobile application, triggering buttons, scrolling up and down and generally exercising every aspect of the mobile application. There will also be an option to run install and run applications on Android or iOS emulators, so we could have 10 emulators running at once, each running different segments of the automated test suite, and uploading the test results to a logging server.

Device Hive will mean, in particular, that we can test BBC mobile applications on the plethora of Android devices available, every make and model that we own of the different OS/hardware combinations can be plugged into a device hive server, and so we can see test runs for BBC iPlayer Android across all the different variations. This will mean we can help target a wider range of Android devices for new BBC iPlayer features, which will help ease the anger that some of our audience members feel because their specific Android iPlayer experience is not as good as the later models.

In October I moved departments from Television and Mobile Platforms to POD Test, and joined the new Device Hive team as lead developer. I am working in Ruby/Rails/Rspec/Cucumber and using Ubuntu Linux VMs and lots of Android and iOS devices to build up the system.

University Engagement

I have been continuing to work with Manchester University’s Ultimate Programming Society to organise and present talks to the students about working practices in software development that the BBC use. We have covered Behaviour Driven Development, Test Driven Development, Editors and IDEs and Agile Development Practices so far.

Generally I feel that I have worked on some pretty challenging projects this year, and I am very happy with being the lead developer on Device Hive, and look forward to making this project as useful and as powerful as I think it can be.

Android Debug Bridge failing to detect emulators under OSX

Android

I’ve been working on a project at the BBC where we are using the Android command-line tools from the Android Developer Tools, to spin up and terminate series of emulators. I noticed a big problem under OSX where ‘adb devices’ was failing to register emulators occasionally when we started them up, without any error message, even though they were loaded and quite clearly running in a window on OSX. This was a real problem for our project because we needed absolute parity between emulator process being launched and subsequently being detected by adb.

We switched to using adb with emulators in an Ubuntu 12.04 VM running under OSX, and we had no further problems with our setup. Emulators will now be programatically launched and torn down by our monitoring application. We now have an array of emulators which we can deploy to at will, which is very useful.

I don’t know what has caused this problem, my only hunch is that the Android toolkit was probably developed in a very Linux-heavy environment, and so adb on Linux was probably their first testing platform. All I can say is that Linux is much more stable than OSX, even as a VM, for Android emulation.

The Haiku Machine

550px-Write-a-Haiku-Poem-Intro

I found this awesome cut-up poetry generator, which takes the text of famous poets and builds structured poetry out of it. The guy that made it even developed the underlying algorithm as a research project. I have put a version of a free Amazon EC2 instance, wrote a little twitter bot in node.js, and wired the poetry generator with the twitter bot, and now I have this: https://twitter.com/haikumachine – a twitter bot that posts a haiku every five minutes, derived from Dylan Thomas’s poetry.

It could be improved, and there are sometimes erroneous tweets where the syllables aren’t counted quite right, or some of the punctuation doesn’t make sense once cut up, but damnit, it’s a bot that writes Haikus.

//platform.twitter.com/widgets.js

256 Color VIM on Crunchbang Waldorf

256 Colours in VIMTo get 256 colors working within terminator in Crunchbang Waldorf, I had to do the following:

  1. Add to ~/.bashrc
    export TERM=xterm-256color
  2. Install a 256 color VIM colorcheme, see desert256 for example.
  3. Add the following to ~/.vimrc:
    set t_Co=256
    set t_AB=^[[48;5;%dm
    set t_AF=^[[38;5;%dm
    

    ‘t_Co’ specifies exactly how many colours VIM can use. The other two lines seem to be Debian-specific color code escape sequences.

  4. If you want 256 color VIM for your root user when you sudo edit, then edit /usr/share/vim/vimrc and copy across your settings from your local ~/.vimrc and ~/.vim to this global environment.

Subversion 1.7 on Crunchbang Waldorf

I use the excellent http://www.smartsvn.com/ client from WANdisco. WANdisco have been releasing new open-source versions of SVN to the public with new improved reliability, and the client uses one of these versions, 1.7, to offer better performance.

Unfortunately if you choose to upgrade your entire repository to 1.7, this breaks compatibility with the default commandline SVN client on Waldorf which I like to use as well as Smart SVN, for quick ‘svn up’s and other commandline magic.

This means I have to download the latest commandline SVN client, the 1.7 version of subversion for Linux, available for free on the WANdisco site.

Unfortunately, you can’t install this version on the version of Debian that Crunchbang Waldorf is based on. There are broken dependencies on an old version of libsvn1, which is a requirement for another package that is part of the Debian base install.

Eventually I found this really helpful page, the instructions which will work 100% on Waldorf:

http://ymartin59.free.fr/wordpress/index.php/2012/11/25/how-to-install-subversion-1-7-from-wandisco-repository-on-debian-wheezy/

iTerm for OSX for a Colourful Terminal Experience

Screen Shot 2013-06-11 at 17.06.36

iTerm is much better than the standard OSX terminal client, not least because it has compatiablity with xterm256-color terminal emulation. xterm256-color emulation will give your terminal access to 256 colours instead of the usual 16. Much better, not just for looking pretty, but for distinguishing between different types of data in an editor like VIM or even in Cucumber output (see picture above). It’s also free.

http://iterm.sourceforge.net

Once installed, you will have to go in to the preferences and set your ‘Report Terminal Type’ to be ‘xterm256-color’. Then things should be more colourful. Then install a 256 color compatible theme in VIM to make use of that extra capacity. You can also edit your prompt and use 256 colour escape sequences, if you wish.

Tailing a log file and Running an Application at the Same Time

logviewing

A quick tip this, but a useful one. You can tail a log file in the background while running a script in the foreground. So for example, I frequently execute the following commands:

1.

tail -f /var/log/httpd.log &
/etc/init.d/apache restart

2. (The log file will spool onto the terminal as Apache is restarted.)

3. Once you are finished viewing the log file, foreground the log file process and kill it:

fg

Then terminate the foregrounded log tail with a control-c.

With this technique you can run as many commands as you want, and see the real-time effects on your log file, without having to open a new terminal. You will also see your program output interspersed with your log file output, which can be helpful when tracing down particular problems.

Monitoring a Slow Internet Connection in OSX

images

I am currently on holiday in Tenerife, and although I really like it here, one thing I do not like is the internet connection we have in our resort. Sometimes networked applications will just hang with no warning and there will be minutes where it’s not clear what is going on. Here are some ways you can find a little bit more about what is happening when an application is slow or seems to hang when you have a poor internet connection. Execute the following commands each in a separate terminal window.

Log Files

tail -f /var/log/*

This will give you an indication of what is happening in OSX. For example, I was installing the XCode Command Line Utils from within XCode. The installation progress information is severely lacking, it just shows a bar which moves from left to right. However I was able to find out what was happening by tailing the log files in /var/log, which provided me with an updated breakdown of the installer progress. You can exit tail by using Control-C which will return you to the shell.

Constant Ping

ping www.google.com

When I have problems with my internet connection, I always keep a ping running in the background in a terminal. The interesting information here is the ICMP RTT time shown as the milliseconds next to the ‘time’ label, and how many packets were dropped shown by the number of ‘request timeout’ messages. Google does not mind you pinging it, just like hundreds of thousands of other people do, and so you can keep this up constantly, monitoring problems with your internet connection. When you get ‘no route to host’ printed, this usually means that your gateway or wireless connection is down, which means you usually have to reestablish a connection manually.

tcpdump

sudo tcpdump en0 -vvv

Do you really want to see what is happening on your computers network connection? Turn the floodgates on then, and use tcpdump. This will output information on each packet that your computer sends out and is received in a slightly Matrix-style torrent of information. If you are downloading something via an application or have a number of active web connections such as AJAX Facebook pages loaded, you would expect to see a lot of traffic. If you don’t have a lot of traffic, and you’re expecting a lot, then something may be wrong. You can use tcpdump to get a general feel of what data is being passed around, and to what IP address, which you can then look up later for more clarity. You can also use grep and some basic TCP/IP networking knowledge to find out what exactly is happening on the network level.

Network Connection Status of Each Application

sudo lsof -i tcp

Want to find out information about applications are using your internet connection, and the connection state of each TCP connection? Use lsof. You will be given the name of the application that is using each TCP connection, the IP address to which it is connected to, and the TCP connection state (established is good, time wait can be a problem sign). Run this regularly to check on the connection state of your programs. This won’t monitor UDP connections, but should cover your web browsers.

Hopefully this information will give you a bit more insight into what is actually happening on your OSX machine when your internet connection is being unreliable and you want more information about what is going on. Once you have this information, you can use it to inform actions such as toggling the wireless off and on again to reestablish a connection, reloading webpages that have hung, restarting application downloads, or possibly finding a new hotel or resort with a better internet connection 🙂

TDD Talk

images

Recently, myself and two colleagues from the BBC, ran a session on Test Driven Development at the Manchester University ‘Ultimate Programming’ society. The society is a gathering where students discuss cool things they have done with programming, and occasionally have guest speakers from industry. I found the society online and thought it would be great to get the BBC more involved in the local university happenings.

It is the first outreach project that I have undertaken, and it required a lot of preparation. Our initial idea was to get students to implement the A* search algorithm in a practical session, using TDD. However after we had all implemented our own copy of the algorithm, and realised it had taken several hours each, we realised we’d not have enough time in the 2 hour slot that we had.

Instead then, we went back to the tried and tested FizzBuzz example, which is how I learned TDD at the BBC. This was nice and simple and relatively straightforward to implement in an hour practical session. The task was to implement FizzBuzz using write-the-tests-first TDD process, and we gave approximately 1 hour for the students to undertake this task. For the rest of the time we were going through our presentation and talking about how we use TDD and other development concepts at the BBC.

The session overall went quite well, and although it showed to me how difficult it is to present in front of a group of people for an hour, we had good feedback, and I think we really gave the students a different perspective on how to write code, one that a large section of them would not have been exposed to in a standard CS curriculum. We aim to do other talks, starting with the next one, which will be a session on how we use BDD (Behaviour Driven Development) at the BBC.

Here are the slides we put together for the presentation:

Here is the model solution to the TDD exercise, written by my colleague Jack Palfry:

Converting a single M2V frame into JPEG under OSX

I needed to view a single frame of a m2v file that had been encoded by our designers for playing out on TV. The file name was .mpg but in actuality it was a single .m2v frame renamed to be a .mpg. Windows Media Player classic used to display the frame fine when I opened the file normally, under Windows XP. However now I have switched to a Mac, I have found that Quicktime and VLC refused to display the single frame. I couldn’t find a video player that would open the single frame. So I resorted to the command line version of ffmpeg, which I installed via macports, to convert this single frame to a jpg file to view as normal. This line worked a treat:

ffmpeg -i north.mpg -ss 00:00:00 -t 00:00:1 -s 1024x768 -r 1 -f mjpeg north.jpg

Where ‘north.mpg’ was the m2v file, and ‘north.jpg’ was the output jpeg.

And this:

find -name *.mpg -exec ffmpeg -i {} -ss 00:00:00 -t 00:00:1 -s 1024x768 -r 1 -f mjpeg {}.jpg ;

Will go through all the mpg files in the current directory and below, and create their jpeg single frame equivalents, ie: for north.mpg it will create north.mpg.jpg.

Java 1.6 on RHEL4

After I wrote a Java application in JDK 1.6, I was stuck for a while when I realised that the target deployment machine was Red Hat Enterprise Linux 4. RHEL4 does not support Java 1.6 in its default configuration.

Luckily I found this article on the CentOS wiki which included instructions on how to install Java 1.6 on CentOS 4. Remembering that RHEL4 and CentOS 4 are almost identical, I tried the method supplied, and it worked. This is the page with the method:

http://wiki.centos.org/HowTos/JavaOnCentOS

Test Driven Systems Development with Nagios

Nagios can be seen as a automated test tool for systems, just as you would have automated tests for software projects. In test driven development (TDD), you write the tests first, and then use those tests to build up a software project that you can have confidence that it works. We can use this method to build up systems, or networks of systems. Plan out which services and processes should be running on your new systems, and then implement Nagios tests for every one. You can check the progress of your build by checking Nagios. I have been doing this at the BBC. It is a simple idea but one that seems to work.

JSoup Method for Page Scraping

Soup bowl

I’m currently in the process of writing a web scraper for the forums on Gaia Online. Previously, I used to use Python to develop web scrapers, with the very handy Python library BeautifulSoup. Java has an equivalent called JSoup.

Here I have written a class which is extended by each class in my project that wants to scrape HTML. This ‘Scraper’ class deals with the fetching of the HTML and converting it into a JSoup tree to be navigated and have the data picked out of. It advertises itself as a ‘web spider’ type of web agent and also adds a 0-7 second random wait before fetching the page to make sure it isn’t used to overload a web server. It also converts the entire page to ASCII, which may not be the best thing to do for multi-language web pages, but certainly has made the scraping of the English language site Gaia Online much easier.

Here it is:

import java.io.IOException;
import java.io.InputStream;
import java.io.StringWriter;
import java.text.Normalizer;
import java.util.Random;
import org.apache.commons.io.IOUtils;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

/**
* Generic scraper object that contains the basic methods required to fetch
* and parse HTML content. Extended by other classes that need to scrape.
*
* @author David
*/
public class Scraper {

        public String pageHTML = ""; // the HTML for the page
        public Document pageSoup; // the JSoup scraped hierachy for the page


        public String fetchPageHTML(String URL) throws IOException{

            // this makes sure we don't scrape the same page twice
            if(this.pageHTML != ""){
                return this.pageHTML;
            }

            System.getProperties().setProperty("httpclient.useragent", "spider");

            Random randomGenerator = new Random();
            int sleepTime = randomGenerator.nextInt(7000);
            try{
                Thread.sleep(sleepTime); //sleep for x milliseconds
            }catch(Exception e){
                // only fires if topic is interruped by another process, should never happen
            }

            String pageHTML = "";

            HttpClient httpclient = new DefaultHttpClient();
            HttpGet httpget = new HttpGet(URL);

                HttpResponse response = httpclient.execute(httpget);
                HttpEntity entity = response.getEntity();

                if (entity != null) {
                    InputStream instream = entity.getContent();
                    String encoding = "UTF-8";

                    StringWriter writer = new StringWriter();
                    IOUtils.copy(instream, writer, encoding);

                    pageHTML = writer.toString();
                    
                    // convert entire page scrape to ASCII-safe string
                    pageHTML = Normalizer.normalize(pageHTML, Normalizer.Form.NFD).replaceAll("[^\p{ASCII}]", "");

                }

                return pageHTML;
        }

        public Document fetchPageSoup(String pageHTML) throws FetchSoupException{
            
            // this makes sure we don't soupify the same page twice
            if(this.pageSoup != null){
                return this.pageSoup;
            }
            
            if(pageHTML.equalsIgnoreCase("")){
                throw new FetchSoupException("We have no supplied HTML to soupify.");
            }

            Document pageSoup = Jsoup.parse(pageHTML);

            return pageSoup;
        }
}

Then each class subclasses this scraper class, and adds the actual drilling down through the JSoup hierachy tree to get what is required:

...
this.pageHTML = this.fetchPageHTML(this.rootURL);
this.pageSoup = this.fetchPageSoup(this.pageHTML);

// get the first  section on the page
Element forumPageLinkSection = this.pageSoup.getElementsByAttributeValue("id","forum_hd_topic_pagelinks").first();
// get all the links in the above 
section Elements forumPageLinks = forumPageLinkSection.getElementsByAttribute("href"); ...

I’ve found that this method provides a simple and effective way of scraping pages and using the resultant JSoup tree to pick out important data.

Disabling Control-Enter and Control-B shortcut keys in Outlook 2003

At work, I still have to use Windows XP and Outlook 2003. I don’t particually mind this, except when I draft an email to someone and accidently I press Control-B instead of Control-V. Control-B will go ahead and send your partially composed email, resulting in some embarassment as you have to tell everyone to disregard it.

So I wanted to remove the ‘send email’ shortcut keys in Outlook 2003. There are two ways of doing this, one involves editing your group policy, which is something only my IT administration team can do, and I didn’t want to have to involve them. The other way is by making a change to your registry, which I will describe here.

  1. Open up regedit, and browse to the following registry key: HKEY_CURRENT_USER -> Software -> Policies -> Microsoft -> office -> 11.0 -> outlook
  2. Then create a new key called: “DisabledShortcutKeysCheckBoxes”.
  3. Under that key, create two new String Values:
    Name: CtrlB Data: 66,8
    Name: CtrlEnter Data: 13,8
  4. Then restart Outlook and those keys will be disabled.

Click on the thumbnail below to see what the finished edit should look like:

Directory names not visable under ls? Change your colours.

There is a problem I frequently encouter on Redhat/Fedora/CentOS systems with the output of the ls command. Under those distributions, the default setup is to display directories in a very dark colour. If you usually use a white foreground and a black background on your terminal client (such as Putty) then you will struggle to read the names of the directories under Redhat-based distributions.

There are two soloutions that I have used:

1. Change the colour settings in Putty

If you use Putty, ticking ‘Use System Colours’ here changes the “white foreground, black background” default into a “white background, black foreground”. This way you can at least read the console properly, good for a quick fix. You can also save these settings in putty to be the default for the host that you are connecting to, or even all hosts.

2. Change the LS_COLORS directive temporarily in the shell.

Alternatively, you can ask the ls command to display directories and other entries in colours that you specify. You could add these lines to the bottom of your .bashrc to make these changes permanent, or if you are using a shared machine, just copy and paste the following lines into the terminal and they will change the colours to a reddish more visable set, until you logout. :

alias ls='ls --color' # just to make sure we are using coloured ls
LS_COLORS='di=94:fi=0:ln=31:pi=5:so=5:bd=5:cd=5:or=31:mi=0:ex=35:*.rpm=90'
export LS_COLORS

(Original source for this particular LS_COLORS combo: http://linux-sxs.org/housekeeping/lscolors.html)

Scraping Gumtree Property Adverts with Python and BeautifulSoup

I am moving to Manchester soon, and so I thought I’d get an idea of the housing market there by scraping all the Manchester Gumtree property adverts into a MySQL database. Once in the database, I could do things like find the average monthly price for a 2 bedroom flat in an area, and spot bargains through using standard deviation from the mean on the price through using simple SQL queries via phpMyAdmin.

I really like the Python library BeautifulSoup for writing scrapers, there is also a Java version called JSoup. BeautifulSoup does a really good job of tolerating markup mistakes in the input data, and transforms a page into a tree structure that is easy to work with.

I chose the following layout for the program:

advert.py – Stores all information about each property advert, with a ‘save’ method that inserts the data into the mysql database
listing.py – Stores all the information on each listing page, which is broken down into links for specific adverts, and also the link to the next listing page in the sequence (ie: the ‘next page’ link)
scrapeAdvert.py – When given an advert URL, this creates and populates an advert object
scrapeListing.py – When given a listing URL, this creates and populates a listing object
scrapeSequence.py – This walks through a series of listings, calling scrapeListing and scrapeAdvert for all of them, and finishes when there are no more listings in the sequence to scrape

Here is the MySQL table I created for this project (which you will have to setup if you want to run the scraper):

--
-- Database: `manchester`
--

-- --------------------------------------------------------

--
-- Table structure for table `adverts`
--

CREATE TABLE IF NOT EXISTS `adverts` (
  `url` varchar(255) NOT NULL,
  `title` text NOT NULL,
  `pricePW` int(10) unsigned NOT NULL,
  `pricePCM` int(11) NOT NULL,
  `location` text NOT NULL,
  `dateAvailable` date NOT NULL,
  `propertyType` text NOT NULL,
  `bedroomNumber` int(11) NOT NULL,
  `description` text NOT NULL,
  PRIMARY KEY (`url`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

PricePCM is price per calendar month, PricePW is price per week. Usually each advert with have one or the other specified.

advert.py:

import MySQLdb
import chardet
import sys

class advert:

        url = ""
        title = ""
        pricePW = 0
        pricePCM = 0
        location = ""
        dateAvailable = ""
        propertyType = ""
        bedroomNumber = 0
        description = ""

        def save(self):
                # you will need to change the following to match your mysql credentials:
                db=MySQLdb.connect("localhost","root","secret","manchester")
                c=db.cursor()

                self.description = unicode(self.description, errors='replace')
                self.description = self.description.encode('ascii','ignore')
                # TODO: might need to convert the other strings in the advert if there are any unicode conversetion errors

                sql = "INSERT INTO adverts (url,title,pricePCM,pricePW,location,dateAvailable,propertyType,bedroomNumber,description) VALUES('"+self.url+"','"+self.title+"',"+str(self.pricePCM)+","+str(self.pricePW)+",'"+self.location+"','"+self.dateAvailable+"','"+self.propertyType+"',"+str(self.bedroomNumber)+",'"+self.description+"' )"

                c.execute(sql)

In advert.py we convert the unicode output that BeautifulSoup gives us into plain ASCII so that we can put it in the MySQL database without any problems. I could have used Unicode in the database as well, but the chances of really needing Unicode for representing Gumtree ads is quite slim. If you intend to use this code then you will also want to enter the MySQL credentials for your database.

listing.py:

class listing:

        url=""
        adverturls=[]
        nextLink=""


        def addAdvertURL(self,url):

                self.adverturls.append(url)

scrapeAdvert.py:

from BeautifulSoup import BeautifulSoup          # For processing HTML
import urllib2
from advert import advert
import time

class scrapeAdvert:

        page = ""
        soup = ""

        def scrape(self,advertURL):

                # give it a bit of time so gumtree doesn't
                # ban us
                time.sleep(2)

                url = advertURL
                # print "-- scraping "+url+" --"
                page = urllib2.urlopen(url)
                self.soup = BeautifulSoup(page)

                self.anAd = advert()

                self.anAd.url = url
                self.anAd.title = self.extractTitle()
                self.anAd.pricePW = self.extractPricePW()
                self.anAd.pricePCM = self.extractPricePCM()

                self.anAd.location = self.extractLocation()
                self.anAd.dateAvailable = self.extractDateAvailable()
                self.anAd.propertyType = self.extractPropertyType()
                self.anAd.bedroomNumber = self.extractBedroomNumber()
                self.anAd.description = self.extractDescription()

        def extractTitle(self):

                location = self.soup.find('h1')
                string = location.contents[0]
                stripped = ' '.join(string.split())
                stripped = stripped.replace("'",'"')
                # print '|' + stripped + '|'
                return stripped


        def extractPricePCM(self):

                location = self.soup.find('span',attrs={"class" : "price"})
                try:
                        string = location.contents[0]
                        string.index('pcm')
                except AttributeError: # for ads with no prices set
                        return 0
                except ValueError: # for ads with pw specified
                        return 0

                stripped = string.replace('£','')
                stripped = stripped.replace('pcm','')
                stripped = stripped.replace(',','')
                stripped = stripped.replace("'",'"')
                stripped = ' '.join(stripped.split())
                # print '|' + stripped + '|'
                return int(stripped)

        def extractPricePW(self):

                location = self.soup.find('span',attrs={"class" : "price"})
                try:
                        string = location.contents[0]
                        string.index('pw')
                except AttributeError: # for ads with no prices set
                        return 0
                except ValueError: # for ads with pcm specified
                        return 0
                stripped = string.replace('£','')
                stripped = stripped.replace('pw','')
                stripped = stripped.replace(',','')
                stripped = stripped.replace("'",'"')
                stripped = ' '.join(stripped.split())
                # print '|' + stripped + '|'
                return int(stripped)

        def extractLocation(self):

                location = self.soup.find('span',attrs={"class" : "location"})
                string = location.contents[0]
                stripped = ' '.join(string.split())
                stripped = stripped.replace("'",'"')
                # print '|' + stripped + '|'
                return stripped

        def extractDateAvailable(self):

                current_year = '2011'

                ul = self.soup.find('ul',attrs={"id" : "ad-details"})
                firstP = ul.findAll('p')[0]
                string = firstP.contents[0]
                stripped = ' '.join(string.split())
                date_to_convert = stripped + '/'+current_year
                try:
                        date_object = time.strptime(date_to_convert, "%d/%m/%Y")
                except ValueError: # for adverts with no date available
                        return ""

                full_date = time.strftime('%Y-%m-%d %H:%M:%S', date_object)
                # print '|' + full_date + '|'
                return full_date

        def extractPropertyType(self):

                ul = self.soup.find('ul',attrs={"id" : "ad-details"})
                try:
                        secondP = ul.findAll('p')[1]
                except IndexError: # for properties with no type
                        return ""
                string = secondP.contents[0]
                stripped = ' '.join(string.split())
                stripped = stripped.replace("'",'"')
                # print '|' + stripped + '|'
                return stripped

        def extractBedroomNumber(self):

                ul = self.soup.find('ul',attrs={"id" : "ad-details"})
                try:
                        thirdP = ul.findAll('p')[2]
                except IndexError: # for properties with no bedroom number
                        return 0
                string = thirdP.contents[0]
                stripped = ' '.join(string.split())
                stripped = stripped.replace("'",'"')
                # print '|' + stripped + '|'
                return stripped


        def extractDescription(self):

                div = self.soup.find('div',attrs={"id" : "description"})
                description = div.find('p')
                contents = description.renderContents()
                contents = contents.replace("'",'"')
                # print '|' + contents + '|'
                return contents

In scrapeAdvert.py there are a lot of string manipulation statements to pull out any unwanted characters, such as the ‘pw’ characters (short for per week) found in the price string, which we need to remove in order to store the property price per week as an integer.

Using BeautifulSoup to pull out elements is quite easy, for example:

ul = self.soup.find('ul',attrs={"id" : "ad-details"})

That finds all the HTML elements under the tag id=”ad-details”, so all the list elements in that list. More detail can be found in the Beautiful Soup documentation which is very good.

scrapeListing.py:

from BeautifulSoup import BeautifulSoup          # For processing HTML
import urllib2
from listing import listing
import time

class scrapeListing:

        soup = ""
        url = ""
        aListing = ""

        def scrape(self,url):
                # give it a bit of time so gumtree doesn't
                # ban us
                time.sleep(3)

                print "scraping url = "+str(url)

                page = urllib2.urlopen(url)
                self.soup = BeautifulSoup(page)

                self.aListing = listing()
                self.aListing.url = url
                self.aListing.adverturls = self.extractAdvertURLs()
                self.aListing.nextLink = self.extractNextLink()

        def extractAdvertURLs(self):

                toReturn = []
                h3s = self.soup.findAll("h3")
                for h3 in h3s:
                        links = h3.findAll('a',{"class":"summary"})
                        for link in links:
                                print "|"+link['href']+"|"
                                toReturn.append(link['href'])

                return toReturn

        def extractNextLink(self):

                links = self.soup.findAll("a",{"class":"next"})
                try:
                        print ">"+links[0]['href']+">"
                except IndexError: # if there is no 'next' link found..
                        return ""
                return links[0]['href']

The extractNextLink method here extracts the pagination ‘next’ link which will bring up the next listing page from the selection of listing pages to browse. We use it to step through the pagination ‘sequence’ of resultant listing pages.

scrapeSequence.py:

from scrapeListing import scrapeListing
from scrapeAdvert import scrapeAdvert
from listing import listing
from advert import advert
import MySQLdb
import _mysql_exceptions

# change this to the gumtree page you want to start scraping from
url = "http://www.gumtree.com/flats-and-houses-for-rent/salford-quays"

while url != None:
        print "scraping URL = "+url
        sl = ""
        sl = scrapeListing()
        sl.scrape(url)
        for advertURL in sl.aListing.adverturls:
                sa = ""
                sa = scrapeAdvert()
                sa.scrape(advertURL)
                try:
                        sa.anAd.save()
                except _mysql_exceptions.IntegrityError:
                        print "** Advert " + sa.anAd.url + " already saved **"
                sa.onAd = ""

        url = ""
        if sl.aListing.nextLink:
                print "nextLink = "+sl.aListing.nextLink
                url = sl.aListing.nextLink
        else:
                print 'all done.'
                break

This is the file you run to kick off the scrape. It uses an MySQL IntegrityError try/except block to pick out when an advert has already been entered into the database, this will throw an error because the URL of the advert is the primary key in the database. So no two records can have the same primary key.

The URL you provide it above gives you the starting page from which to scrape from.

The above code worked well for scraping several hundred Manchester Gumtree ads into a database, from which point I was able to use a combination of phpMyAdmin and OpenOffice Spreadsheet to analyse the data and find out useful statistics about the property market in said area.

Download the scraper source code in a tar.gz archive

Note: Due to the nature of web scraping, if – or more accurately, when – Gumtree changes its user interface, the scraper I have written will need to be tweaked accordingly to find the right data. This is meant to be an informative tutorial, not a finished product.

RESTful Web Services

Hammock with the background of a clear blue sky

REST (Representational State Transfer) is a way of delivering web services. When a web service conforms to REST, it is known as RESTful. The largest RESTful web service is the Hypertext Transfer Protocol (HTTP) which you use every day to send and receive information from web servers while browsing the internet.

To implement RESTful web services, you should implement four methods: GET, PUT, POST and DELETE. Resources on RESTful web services are typically defined as collections of elements. The REST methods can either act on a whole collection, or a specific element in a collection.

A collection is usually logically defined as a hierarchy on the URL, for example take this fictitious layout:

Collection: http://www.bbc.co.uk/iplayer/programmes/
Element: http://www.bbc.co.uk/iplayer/programmes/24
Element: http://www.bbc.co.uk/iplayer/programmes/25
Element: http://www.bbc.co.uk/iplayer/programmes/26

The REST methods you use do different things depending on whether you are interacting with a Collection resource or an Element resource. See below:

On a Collection: ie: http://www.bbc.co.uk/iplayer/programmes/
GET – Lists the URLs of the collection’s members.
PUT – Replace the entire collection with another collection.
POST – Create a new element in a collection, returning the new element’s URL.
DELETE – Deletes the entire collection.

On an Element: ie: http://www.bbc.co.uk/iplayer/programmes/24
GET – Retrieve the addressed element in the appropriate internet media type, ie: music file or image
PUT – Replace the addressed element of the collection, or if it doesn’t exist, create it in the parent collection.
POST – Treat the addressed element of the collection as a new collection, and add an element into it.
DELETE – Delete the addressed element of the collection.

REST is a simple and clear way of implementing the basic methods of data storage; CRUD (Create, Read, Update and Delete), see: http://en.wikipedia.org/wiki/Create,_read,_update_and_delete

‘Weather Forecast’ Calendar Service in PHP

The BBC provide 3 day weather RSS feeds for most locations in the UK. I thought it would be interesting to create a web service to turn the weather feed into calendar feed format, so I could have a constantly updated forecast of the next 3 days of weather mapped on to my iPhone’s calendar. Here it is on my iPhone:

Picture shows weather forecast on an iPhone calendar screenshot

Overview

The service is separated into five files:

  • ical.php – this contains the class ical which corresponds to a single calendar feed. A method called ‘addevent’ allows you to add new events to the calendar, and a method called ‘returncal’ redirects the resulting calendar file to the browser so people can subscribe to it using their calendar application.
  • forecast.php – this file contains the class forecast, which has properties for all aspects that we want to record for each day’s forecast, ie: Wind Speed and Humidity. It also contains the forecast set, which is a collection of forecast objects. The set class is serializable, which means each forecast object can be stored in a text file, including the Wind Speed, Humidity and all other things we want to record for each day.
  • scrape-weather.php – this file contains code that scrapes the weather feed, populates the forecast set with all the weather information for the next 3 days, and stores the result in a file called forecasts.ser.
  • forecasts.ser – this is all the data for the three day weather forecast, in serialized format. It is automatically deleted and recreated when the scrape-weather.php script is run.
  • reader.php – this file converts the forecasts.ser file into an iCal calendar, and outputs the iCal formatted result to the calendar application that accesses reader.php page.

It uses two external libraries:

  • MagpieRSS 0.72 – this popular library is used for reading the calendar RSS feed and converting it into a PHP object that is easier to manipulate by scrape-weather.php.
  • iCalcreator 2.8 – this is used for creating the output iCal format of the calendar in ical.php and outputting it to the browser in reader.php.

Files

init();
	}	

	function init(){
		$config = array( 'unique_id' => 'weather.davidcraddock.net' );
		  // set Your unique id
		$this->v = new vcalendar( $config );
		  // create a new calendar instance

		$this->v->setProperty( 'method', 'PUBLISH' );
		  // required of some calendar software
		$this->v->setProperty( "x-wr-calname", "Calendar Sample" );
		  // required of some calendar software
		$this->v->setProperty( "X-WR-CALDESC", "Calendar Description" );
		  // required of some calendar software
		$this->v->setProperty( "X-WR-TIMEZONE", "Europe/London" );
		  // required of some calendar software
	}

	function addevent($start_year,$start_month,$start_day,$start_hour,$start_min,
		  $finish_year,$finish_month,$finish_day,$finish_hour,$finish_min,
		  $summary,$description,$comment		
	){
		$vevent = & $this->v->newComponent( 'vevent' );
		  // create an event calendar component
		$start = array( 'year'=>$start_year, 'month'=>$start_month, 'day'=>$start_day, 'hour'=>$start_hour, 'min'=>$start_min, 'sec'=>0 );
		$vevent->setProperty( 'dtstart', $start );
		$end = array( 'year'=>$finish_year, 'month'=>$finish_month, 'day'=>$finish_day, 'hour'=>$finish_hour, 'min'=>$finish_min, 'sec'=>0 );
		$vevent->setProperty( 'dtend', $end );
		$vevent->setProperty( 'LOCATION', '' );
		  // property name - case independent
		$vevent->setProperty( 'summary', $summary );
		$vevent->setProperty( 'description',$description );
		$vevent->setProperty( 'comment', $comment );
		$vevent->setProperty( 'attendee', 'contact@davidcraddock.net' );
	}

	function returncal(){
		// redirect calendar file to browser
		$this->v->returnCalendar();
	}
}
?>
forecasts = new ArrayObject();
	}
}
scrapecurrent();
		$this->store();
	}

	function store(){
		$store_path = $this->store_path;
		unlink($store_path);
		file_put_contents($store_path, serialize($this->set));
	}

	function scrapecurrent(){
		$url = $this->feed_url;
		$rss = fetch_rss( $url );
		$message = "";
		if(sizeof($rss->items) != 3){
			die("Problem with BBC weather feed.. dying");
		}
		$i=0;
		$set = new forecast_set();
		$curdate = date("Y-m-d");
		echo $curdate;
		foreach ($rss->items as $item) {
			$href = $item['link'];
			$title = $item['title'];
			$description = $item['description'];
			print_r($item);
			$curyear = date('Y',strtotime(date("Y-m-d", strtotime($curdate)) . " +1 day"));
			$curmonth = date('m',strtotime(date("Y-m-d", strtotime($curdate)) . " +1 day"));
			$curday = date('d',strtotime(date("Y-m-d", strtotime($curdate)) . " +1 day"));
			preg_match('/:.+?,/',$title,$summary);
			preg_match('/Min Temp:.+?-*d*/',$title,$mintemp);
			preg_match('/Max Temp:.+?-*d*/',$title,$maxtemp);
			preg_match('/Wind Speed:.+?-*d*/',$description,$windspeed);
			preg_match('/Humidity:.+?-*d*/',$description,$humidity);
			$summary[0] = str_replace(': ','',$summary[0]);
			$summary[0] = str_replace(',','',$summary[0]);
			$mintemp[0] = str_replace('Min Temp: ','',$mintemp[0]);
			$maxtemp[0] = str_replace('Max Temp: ','',$maxtemp[0]);
			$windspeed[0] = str_replace('Wind Speed: ','',$windspeed[0]);
			$humidity[0] = str_replace('Humidity: ','',$humidity[0]);
			$mins[$i] = (int)$mintemp[0];	
			$maxs[$i] = (int)$maxtemp[0];
			$forecast = new forecast();
			$forecast->low = (int)$mintemp[0];
			$forecast->high = (int)$maxtemp[0];
			$forecast->year = (int)$curyear;
			$forecast->month = (int)$curmonth;
			$forecast->day = (int)$curday;
			$forecast->windspeed = $windspeed[0];
			$forecast->humidity = $humidity[0];
			$forecast->summary = ucwords($summary[0]);
			$set->forecasts->append($forecast);
			$i++;	
			$curdate = date('Y-m-d',strtotime(date("Y-m-d", strtotime($curdate)) . " +1 day"));
		}
		print_r($set);
		$this->set = $set;

	}

}
$s = new scrape3day();
<?php
require_once('ical.php');
require_once('forecast.php');

$c = new ical();
$f = unserialize(file_get_contents('forecasts.ser'));
for($i=0;$iforecasts[$i];
	$weather_digest = "Max: ".$curforecast->high." Min: ".$curforecast->low." Humidity: ".$curforecast->humidity."% Wind Speed: ".$curforecast->windspeed."mph.";
	$c->addevent($curforecast->year,$curforecast->month,$curforecast->day,7,0,$curforecast->year,$curforecast->month,$curforecast->day,7,30,$curforecast->summary,$weather_digest,$weather_digest);
}
$c->returncal();
?>

SVN Version

If you have subversion, you can check out the project from: http://svn.davidcraddock.net/weather-services/. There are a couple extra files in that directory for my automated freezing weather alerts, but you can safely ignore those.

Installation

You will have to add this entry to your crontab to run once per day. You could set the script to run at midnight through adding the following:

0 0 * * *  

For example, in my case:

0 0 * * * /usr/local/bin/php /home/david_craddock/work.davidcraddock.net/weather/scrape-weather.php 

You will then need to edit the contents of the $store_path and $feed_url variables in scrape-weather.php. Store_path should refer to a file path that the web server can create and edit files in, and feed_url should refer to the RSS feed of your local area that you have copied and pasted from the http://news.bbc.co.uk/weather/ site, don’t use mine because your area is likely different. After that, you’re set to go.

Find large files by using the OSX commandline

To quickly find large files to delete if you have filled your startup disk, enter this command on the OSX terminal:

sudo find / -size +500000 -print

This will find and print out file paths to files over 500MB. You can then go through them and delete them individually by typing rm “<file path>”, although there is no undelete so make sure you know you won’t miss them.

Finding files in Linux modified between two dates

You use the ‘touch’ command to create two blank files, with a last modified date that you specify – one with a date of the start of the range you want to specify, and the second with a date at the end of the range you want to specify. Then you reference to those two files in your find command:

touch /tmp/temp -t 200604141130
touch /tmp/ntemp -t 200604261630
find /data/ -cnewer /tmp/temp -and ! -cnewer /tmp/ntemp

Writing simple email alerts in PHP with MagpieRSS

I wrote an email alerter that sends me an email whenever the upcoming temperature may dip below freezing. It uses the Magpie RSS reader to pull down a 3 day weather forecast that is provided for my area in RSS form by the BBC weather site. It then parses this forecast and determines if either today’s or tomorrow’s weather may dip below freezing. If it might, it sends an email to my email address to warn me.

I scheduled this script to run every day by adding it as a daily cron job on my web host. You can set this up for any web hosts that support cron jobs.

items) != 3){
                $message .= 'Error: problem parsing BBC weather feed';
        }
        $i=0;
        foreach ($rss->items as $item) {
                $href = $item['link'];
                $title = $item['title'];
                preg_match('/Min Temp:.+?-*d*/',$title,$mintemp);
                preg_match('/Max Temp:.+?-*d*/',$title,$maxtemp);
                $mintemp[0] = str_replace('Min Temp: ','',$mintemp[0]);
                $maxtemp[0] = str_replace('Max Temp: ','',$maxtemp[0]);
                $mins[$i] = (int)$mintemp[0];
                $maxs[$i] = (int)$maxtemp[0];
                $i++;
        }

        // freezing warnings

        if($mins[0] < 0){
                $message .= "Today's temperature in W3 may go below freezing, anything down to ".$mins[0];
        }
        if($mins[1] 

You can right click on this link and ‘save as’ to download the script.

Reverting back to a previous version in CVS – the magic “undo” feature

If you’ve committed some code into to CVS, and made a mistake on that commit, you will want to know how to revert to a previously saved version. Here is the command line command for CLI versions of CVS:

$ cvs update -D '1 week ago'

Run this command in the main directory of your checked out working copy. This will revert your working copy to the version of the code that was checked in ‘1 week ago’ from the present date. You also use commands like “1 day ago” and “5 days ago”.

Then simply commit the changes with a log message:

$ cvs commit -m "Oops! Made a mistake, had to revert back to the 21/1/2011 version"

Netbeans for simple Java GUI Applications

I’ve been writing some simple Java GUI applications using the Netbeans IDE. It allows you to quickly make event-driven GUI applications, and generates a lot of skeleton code that you’ll need, but don’t necessarily want to type out. It reminds me of the IDE designer of Visual Basic 6, which allowed you to mock up simple GUIs with code in almost no time at all, although the VB language itself often proved difficult. With Netbeans you are using Java, and so you can make some powerful software with little effort.

Converting week numbers to dates

Here is some python code I adapted from this stackoverflow post to get the first day of a week specificed by a week number. This method includes leap year and summer time differences.

import time
def weeknum(num,year):
	instr = str(year)+" "+str(num-1)+" 1"
	print time.asctime(time.strptime(instr,'%Y %W %w'))

Here is me exectuting the code in Python’s IDLE shell:

See that the first week of 2009 actually started in 2008, but by the end of that week we are in 2009.

MediaMonkey allows you to transfer music from any computer onto your guest iPhone

MediaMonkey is a popular free media player for Windows. It has a great feature that allows you to transfer to and from an iPhone that is not registered with your computer. Normally only one iTunes install can be associated with your iPhone, but MediaMonkey allows you another way to transfer music and audio files with a ‘guest’ iPhone. Check it out, it works:

MediaMonkey

Applications I Reccomend

Software I use on my macbook & PC:

DVDRipper Pro for Mac – DVD ripping, can also rip to ISO
Handbrake for Mac – Transcoding from DVD rip to iPhone-playable file
iMovie for Mac – Video editing
BabasChess for Windows – Best chess client for internet play
Hypercam 2 – Best screencapture utility
Skype for both – For reliable messaging as well as voice and video chat
Virtual Clone Drive for Windows – For mounting ISO images
iTunes for Mac – Best music player, and keeps media synced with iPhone
VLC Player for both – For watching movies
DVD Player for Mac – For watching DVDs

iPhone Apps:

Skype – Best messenger
iBooks – Best ebook reader
London Buses – Best London transport router, can route via tube, bus, cycle path and foot
Tube Status – Displays the status of all lines, with any disruptions summarised
NextBuses – Great app that gives you lots of info on the buses and bus stops in your area.
Apple Remote – Apple remote, allows you to control the music on any wi-fi linked iTunes library
Chess.com’s Chess – Great chess game for vs. computer play
TasteKid – Type in a film, author, tv series.. and it will give you similar recommendations
Google Earth – Brilliant for navigational help, although I use iPhone’s inbuilt Maps first, for most things.
SomaFM – Chilled out relaxing electronica

Recording Game Videos on Windows 7

This is just a quick note to remind myself how I did this.

  • Hypercam2 is a good, free, video recorder that can cope with recording game videos. It’s freely available from http://www.hyperionics.com/hc/downloads.asp – just make sure when you install it you don’t tick on the spyware toolbar installation options.
  • My motherboard has a 5.1 digital soundcard built in. However the only way I can record off the soundcard is to plug in a standard audio cable from the speaker out (green) to the microphone in (orange).
  • The soundcard switches off the headphone output when it detects a speaker attached to the speaker out, so you have to go to the recording options in Windows 7 and right click on the microphone in. It will give you an option to ‘Monitor this input using the headphones’ – which will allow you to listen to anything coming into the microphone socket through the headphone socket on the front on my PC.
  • In hypercam, set the sound to record from the default input device, set the frame rate to 10/10
  • Record using the ‘select window to record from’ option, select the game window, and use the F2 button to start and stop the recording.
  • The video will be output in AVI format, but you can transcode or convert it into a quicktime MOV file for editing in iMovie, or you can use windows movie editor, which is free and quite good.

Insights into a modern Indie Music label

I read this remarkable post on a public mailing list I subscribe to. I thought it was such a great insight into running a music label, that I just had to post it here. It discusses issues facing modern music, such as DRM, DMCA, and other ways of making (or losing) money. Fascinating.

Here it is:

I work for a (fairly small) indie label – from witnessing this model in action I feel I have to stick up for the label given that I see the model working (or sometimes not so well) on a daily basis! Where we’ve done deals with artists in the past, they’ve almost always been a 50/50 arrangement – the artist receives 50% of net royalties. Where a label fronts recording costs, these can easily become £6-10,000 for an album session. Even an EP session can be upwards of £1,500 although these figures are a little pessimistic (though not unrealistic). (We actually designed, built and owned studios for ten years until 2001 but the project haemorrhaged money.)

With regards to CD pressing, a 1,000 run will cost around £800 including full colour print in a basic jewel case. The AP1/AP2a MCPS licence costs another amount on top. When getting your CDs pressed, add in other things (Super Jewel cases, slip / O-cards, digipaks or gatefolds with high quality card / fancy posters) and you can easily top the 1k mark, not even counting the artwork design costs. Of course, discount comes with with bulk, but almost nobody except the Big Four do >1k discs in a pressing. (To put things in perspective: when SyCo have done the X Factor Finalists CDs, they press up >10,000 of EACH finalist’s recording of the song – and shred the losers’ copies when the winner is announced!)

To put stuff into distro with someone like Universal, you have your line costs simply to have the title listed on their system – monthly recurring, per title – then handling costs, despatch costs, “salesforce” costs (even though really the only people they sell into are HMV now, and from last year they’ve stopped guaranteeing racking in all but the top 6 or so stores in the UK, it’s a joke). You can’t sell your discs through at full retail, you have your wholesale (Dealer) price. We’ve sold albums through at £6.65 and I’ve later seen them in a London HMV for £12.99. Oh, and did I mention that supermarkets and stores like HMV *DEMAND* what they call a “file discount” of up to 40% just to take stock? (which is on a non-negotiable sale or return basis with up to a six month returns period.)

If you end up in a position where you don’t sell stock through into shops, it usually costs less for your distro to SHRED your discs than it does to send it back to you! Ridiculous. The costs are stacked against the labels at all points – incredibly frustrating. And that’s even before you begin to contemplate any plugging, promo, advertising, miscellaneous online, merch, booking agent / gig costs… Or even an advance for the artist! But it gets better…

So, this figure of 63% which the old techdirt article might quote as truth where valid for major labels (who might also own distribution, management, publishing and studios under the same roof), the model quickly falls apart as soon as focus on a smaller label. I used to think the whole model was bullshit and the artists got shafted, but if anything it’s level pegging – smaller labels have just as tough a time as artists as the risk to them to fund any new release is proportionally WAY larger. Also, the techdirt article works on the basis of the artist receiving a 20% royalty – this is dismal, and the artist should be smacked for agreeing to such a pitiful rate like the chumps they probably (hypothetically) are.

Take one of our real world iTunes scenarios – from a 79p purchase, iTunes immediately keeps about 32p. For UK and most worldwide sales, this also includes the royalties which the label’s obliged to pay (in the UK, to the MCPS-PRS Alliance). However, the USA requires the selling party to pay the mechanical on each sale (an arse-about-tit form which has arisen from the disconnected Collection Agencies – Harry Fox Agency being the incumbent on Mechanicals and ASCAP, BMI and SESAC on the Performance royalties – which adds yet another level of complication.

From what’s left (47p), you halve the resulting amount on a 50/50 deal. Neither the label nor the artist gets much for their work. On some artists whom we’ve purely done digital distribution for (on a rolling licence agreement), we give the artist 80% of net. As you can imagine, we get virtually nothing – and our income’s directly tied to their success, so we have an interest in seeing them do well. It’s a tough environment to be in.

For receiving US/Canadian/Mexico/European/Australasian payments, we first have to receive the currency and have the bank convert it to GBP. Of course, we can’t get the Interbank rates, nobody but the banks get those – so more money’s immediately lost in conversion. The larger labels will have sweetheart deals with their banks (or almost certainly have accounts in each relevant territory) so this isn’t so much of a big deal, but the amount of administration just scales inordinately. If you deal with managing your artists’ Publishing rights, you can quickly become LITERALLY swamped in paperwork. The amount of time sucked up by adminning the release of music is extraordinary.

So please nobody think all music labels have it easy… I have no doubt that the Big Four have royally shafted artists in the past but they can largely lumber along based on a few artists doing exceptionally well for the rest of their current roster (with their back catalogue from very famous artists helping too). The problem they’re going to have is that almost none of the artists whose catalogue’s been released in the past two decades *really* has the staying power of the classic artists – Dire Straits, Genesis, Pink Floyd, The Who or Fleetwood Mac, just to name five off the top of my head. Don’t even get me started on the epic fail that is streaming revenues from Spotify, mFlow, We7 etc.

Now even with all of this, I still regard sites like YouTube as a promotional tool. Some of our most famous catalogue I’ve held off on issuing DMCA takedowns for, because it’s a genuinely beneficial promotional tool – it’s the pragmatic response. Where do people go first if they want to quickly listen to a track? YouTube! What happens if they only ever wanted to hear it once and never again? You’ve not lost that sale because it almost certainly would never have happened. What happens if they still want to have a copy of that track? They’ll go buy it from one of the easily accessible venues, it’s not expensive to do. The label’s job is to make the catalogue ubiquitous on all of the major (and some of the trendier niche stores) where at all possible. The digital distribution costs are another thing the label has to absorb – monthly, per track, per store usually, if not on an aggregation deal where it’s a percentage on each sale but the label usually ends up worse off. It’s a tough position because the label almost always feels the need to protect their ‘content’ (shudder – hate that word) but issuing takedowns for every instance of a track is more often than not a kneejerk reaction which harms longterm sales. I’m personally torn between leaving them, taking them down or even putting up better mashup/promo mix versions on the label’s official account!

Treat your customers like adults and I think you earn their respect a bit more. This applies to all forms of digital media, including tellybox shows. (thesis: DRM = genuinely unhelpful towards nurturing that unique supportive viewer-provider relationship. Trust your customers, they’ll not disrespect you.) In music, nobody wants to buy a track if they can never audition it, and 30sec samples aren’t really a good enough.

Restoring Ubuntu 10.4’s Bootloader, after a Windows 7 Install

I installed Windows 7 after I had installed Ubuntu 10.4. Windows 7 overwrote the Linux bootloader “grub” on my master boot record. Therefore I had to restore it.

I used the Ubuntu 10.4 LiveCD to start up a live version of Ubuntu. While under the LiveCD, I then restored the Grub bootloader by chrooting into my old install, using the linux command line. This is a fairly complex thing to do, and so I recommend you use this approach only if you’re are confident with the linux command line:

# (as root under Ubuntu's LiveCD)

# prepare chroot directory

mkdir /chroot
d=/chroot

# mount my linux partition

mount /dev/sda1 $d   # my linux partition was installed on my first SATA hard disk, on the first parition (hence sdA1).

# mount system directories inside the new chroot directory

mount -o bind /dev $d/dev
mount -o bind /sys $d/sys
mount -o bind /dev/shm $d/dev/shm
mount -o bind /proc $d/proc

# accomplish the chroot

chroot $d

# proceed to update the grub config file to include the option to boot into my new windows 7 install

update-grub

# install grub with the new configuration options from the config file, to the master boot record on my first hard disk

grub-install /dev/sda

# close down the liveCD instance of linux, and boot from the newly restored grub bootloader

reboot

Windows 7 Gaming on my Macbook

I have a 2006/2007 Core 2 Duo 2.6ghz white macbook, that I use regularly for internet, music, watching films, itunes and integration with my iPhone.

I wanted to turn my desktop PC into a ‘work only’ Ubuntu Linux machine, so that I don’t get distracted when I’m supposed to be doing something else.

But I still have a lot of PC games that I wanted to play on the Macbook, so I decided to try and setup a windows environment to play games on using Bootcamp 2.0 to create a dual-boot OSX/Windows 7 configuration.

It turns out it works really well. The Macbook runs Windows 7 64-bit edition fine, and although the integrated graphics card isn’t designed to run modern games very well, you can get a good gaming experience from small indie games and the older type of PC RPGs that I tend to play. My macbook got a 3.5 rating on the windows experience index for graphics, which is sufficient for many PC games.

First you need to partition your macbook’s HD using the Bootcamp assistant, in the OSX utilities section. Make sure you have your first OSX installation DVD to hand, the one that came with your Macbook. I chose to split the hard drive into two equally sized partitions. Then just place your W7 DVD in the drive, and Bootcamp takes care of the rest.

Once W7 is installed, you can access the Bootcamp menu on startup by holding down the option key. This brings up a menu where you can select to boot into OSX or Windows.

When you start W7 for the first time, you can install the windows driver set for your Macbook that Bootcamp provides you. Insert your OSX installation DVD 1, and run the setup.exe that is located in the Bootcamp folder. This will install native windows drivers for your Macbook hardware.

The only change I needed to make for my macbook, was to install the latest 64bit Realtek drivers for Vista/Windows 7, which are located on the Realtek website. This will fix any sound problems you might have while playing games.

Now don’t expect to run the latest 3D games, but if you’re happy enough with slightly older, classic, indie or retro games, you can get a good gaming experience on Windows 7 from your macbook. It does well with plenty of the indie games available on Value’s Steam distribution network.

Ripping Movies onto the iPhone

I’m currently watching Persepolis, the 2008 animated film about a tomboy anarchist growing up in Iran. I’m watching this on my new iPhone 3GS, and the picture and audio quality is very good.

Here’s what I used to convert my newly bought Persepolis DVD, for watching on the iPhone.

1x Macbook (but you can use any intel mac)
1x iTunes
1x RipIt – Commercial Mac DVD Ripper (rips up to 10 DVDs on the free trial, $20 after)
1x Handbrake 32 – Freely available transcoder
1x VLC 32 – Freely available media player
1x DVD

* Ripit – rips the video and audio from the DVD, onto your computer
* Handbrake 32 – ‘transcodes’ the ripped video and audio, meaning – it converts it into an iPhone compatible video file.
* VLC 32 – is used by Handbrake 32 to get past any problems with converting the media.

Go to the following sites to fetch the software:

1. Ripit – http://thelittleappfactory.com/ripit/
2. Handbrake 32 – http://handbrake.fr/downloads.php (get the 32 bit version)
3. VLC 32 – http://www.videolan.org/vlc/download-macosx.html (be sure to get the 32 bit version)

There’s currently a difficulty in getting the VLC 64 bit software for the Mac, and so although the 64 bit version is faster to use, you’re probably better off with 32 bit versions of both for now.

The Process

1) Rip the DVD.

Start RipIt. It will ask for a DVD, insert the DVD.. and point the resultant save location to the desktop. The ripping process takes about 40 minutes on my Macbook, you can check the progress by looking at the icon in the dock – it will be updated with the percentage of progress until completion. You can do other things on your mac while it’s ripping, even though the DVD drive will be occupied. Wait until it’s completed before continuing.

2) Transcode (convert) the ripped video file for use on the iPhone.

Start Handbrake. There are a bunch of transcoding settings called presets – those tell Handbrake what type of media player you want the converted video to work on. In handbrake on the right section of the window, select the iPhone preset. Then go to the file menu, select ‘Open’, and then select the video file that RipIt saved onto your desktop. Then select the destination for the converted video file. Then select the Start (green) button on Handbrake window, and it will start. You can now minimise handbrake and do other things. The transcoding process depends on the film, but takes about an hour on my Macbook. You can check on progress by maximizing the Handbrake window, and checking on the progress bar.

3) Move the converted video file onto your iPhone.

Once that’s done, you will have another media file on your desktop – this is the end result, a video file that will play on your iPhone. Simply connect your iPhone to your Mac, start up iTunes, and drag that file from your desktop into the iPhone icon on your iTunes window. It will take a couple of minutes to transfer, then eject the iPhone as normal

Now you can watch this new movie on your iPhone by going to the ‘Videos’ tab of your iPod app.

WordPress HTML edit mode inserts BR tags sometimes when you add a carriage return..

This is something that was quite annoying today, as I was struggling to use WordPress 2.9.2 to align some pictures in the HTML mode of editing a page, on a client’s website.

It turns out that WordPress was adding BR tags sometimes when I hit return.. and sometimes not. The annoying thing was, although the BRs were outputted in the resultant WordPress site, the BRs were not visible in the WordPress HTML edit mode itself.. meaning they were invisible and undetectable until I viewed the resultant website source and finally figured it out.

WordPress does insert some formatting tags now and then, it seems, but I would have thought it would tell you about the tags that would change the page layout! Apparently not. Anyway, something to be aware of for WordPress gurus..

Edit:

I don’t have time to report this as a bug, but this is the stack I’m using for anyone interested:

Browser: Google Chrome for Mac (5.0.342.9 beta)
TinyMCE Advanced Editor Plugin for WP (3.2.7)
Wordpress 2.9.2

The beta of Google Chrome is a bit unstable, although it may not be the source of the problem.