Performance of Java 2D drawing operations (part 3: image opacity)

Andy Balaam — Mon, 11 Feb 2019 23:51:05 +0000

Series: operations</a>, images</a>, opacity</p>

Not because I was enjoying it, I seemed compelled to continue my quest to understand the performance of various Java 2D drawing operations. I'm hoping to make my game Rabbit Escape</a> faster, especially on the Raspberry Pi, so you may see another post sometime actually trying this stuff out on a Pi.</p>

But for now, here are the results of my investigation into how different patterns of opacity in images affects rendering performance.</p>

You can find the code here: gitlab.com/andybalaam/java-2d-performance</a>.</p>

Results</h2>

Images with partially-opaque pixels are no slower than those with fully-opaque pixels</li>

Large transparent areas in images are drawn quite quickly, but transparent pixels mixed with non-transparent are slow</li> </ul>

Advice</h2>

Don't bother restricting pixels to be either fully transparent or fully opaque - partially-opaque is fine</li> </ul>

Opacity patterns in images</h2>

Non-transparent images drew at 76 FPS, and transparent ones dropped to 45 FPS.</p>

I went further into investigating transparency by creating images that were:</p>

Double the size of the original image, but the extra area is fully transparent, and the original area is non-transparent (41 FPS)</li> </ul>

I concluded that partial-opacity is not important to performance compared with full-opacity, but that large areas of transparency are relatively fast compared with images with complex patterns of transparency and opacity.</p>

Numbers</h2>

Transparency and opacity</h3>

Test</th>

FPS</th></tr></thead>

large nothing</td>

90</td></tr>

large images20 largeimages</td>

76</td></tr>

large images20 largeimages transparentimages</td>

45</td></tr>

large images20 largeimages transparent50pcimages</td>

34</td></tr>

large images20 largeimages transparent0pc100pcimages</td>

34</td></tr>

large images20 largeimages transparentareaimages</td>

41</td></tr> </tbody></table>

Feedback please</h2>

Please do get back to me with tips about how to improve the performance of my experimental code.</p>

Feel free to log issues</a>, make merge requests</a> or add comments to the blog post.</p>

Performance of Java 2D drawing operations (part 2: images)

Andy Balaam — Fri, 08 Feb 2019 02:05:29 +0000

Series: operations</a>, images, opacity</a></p>

In my previous post</a> I examined the performance of various drawing operations in Java 2D rendering. Here I look at some specifics around rendering images, with an eye to finding optimisations I can apply to my game Rabbit Escape</a>.</p>

You can find the code here: gitlab.com/andybalaam/java-2d-performance</a>.</p>

Results</h2>

Drawing images with transparent sections is very slow</li>
Drawing one large image is slower than drawing many small images covering the same area(!)</li>

Drawing images outside the screen</strong> is slower than not drawing them</strong> at all (but faster than drawing them onto a visible area)</li> </ul>
Advice</h2>

Avoid transparent images where possible</li>
Don't bother pre-rendering your background tiles onto a single image</li>
Don't draw images that are off-screen</li> </ul>
Images with transparency</h2>
All the images I used were PNG files with a transparency layer, but in most of my experiments there were no transparent pixels. When I used images with transparent pixels the frame rate was much slower, dropping from 78 to 46 FPS. So using images with transparent pixels causes a significant performance hit.</p>
I'd be grateful if someone who knows more about it can recommend how to improve my program to reduce this impact - I suspect there may be tricks I can do around setComposite or setRenderingHint or enabling/encouraging hardware acceleration.</p>
Composite images</h2>
I assumed that drawing a single image would be much faster than covering the same area of the screen by drawing lots of small images. In fact, the result was the opposite: drawing lots of small images was much faster than drawing a single image covering the same area.</p>
The code for a single image is:</p>
g2d.drawImage( singleLargeImage, 10, 10, null )</pre>
and for the small images it is:</p>
for (y in 0 until 40) { for (x in 0 until 60) { g2d.drawImage( compositeImages[(y*20 + x) % compositeImages.size], 10 + (20 * x), 10 + (20 * y), null ) } }</pre>
The single large image was rendered at 74 FPS, whereas covering the same area using repeated copies of 100 images was rendered at 80 FPS. I ran this test several times because I found the result surprising, and it was consistent every time.</p>
I have to assume some caching (possibly via accelerated graphics) of the small images is the explanation.</p>
Drawing images off the side of the screen</h2>
Drawing images off the side of the screen was faster than drawing them in a visible area, but slower than not drawing them at all. I tested this by adding 10,000 to the x and y positions of the images being drawn (I also tested subtracting 10,000 with similar results). Not drawing any images ran at 93 FPS, drawing images on-screen at 80 FPS, and drawing them off-screen only 83 FPS, meaning drawing images off the side takes significant time.</p>
Advice: check whether images are on-screen, and avoid drawing them if not.</p>
Numbers</h2>
Transparency</h3>
Test</th> FPS</th></tr></thead>

large nothing</td> 95</td></tr>
large images20 largeimages</td> 78</td></tr>
large images20 largeimages transparentimages</td> 46</td></tr> </tbody></table>
Composite images</h3>
(Lots of small images covering an area, or a single larger image.)</p>
Test</th> FPS</th></tr></thead>

large nothing</td> 87</td></tr>
large largesingleimage</td> 74</td></tr>
large compositeimage</td> 80</td></tr> </tbody></table>
Offscreen images</h3>
Test</th> FPS</th></tr></thead>

large nothing</td> 93</td></tr>
large images20 largeimages</td> 80</td></tr>
large images20 largeimages offscreenimages</td> 83</td></tr> </tbody></table>
Feedback please</h2>
Please do get back to me with tips about how to improve the performance of my experimental code.</p>
Feel free to log issues</a>, make merge requests</a> or add comments to the blog post.</p>

Performance of Java 2D drawing operations (part 1: types of operation)

Andy Balaam — Mon, 04 Feb 2019 03:21:06 +0000

Series: operations, images</a>, opacity</a></p>
I want to remodel the desktop UI of my game Rabbit Escape</a> to be more convenient and nicer looking, so I took a new look at game-loop-style graphics rendering onto a canvas in a Java 2D (Swing) UI.</p>
For more on images, see the next post</a>.</p>
Specifically, how fast can it be, and what pitfalls should I avoid when I'm doing it?</p>
Results</h2>

Larger windows are (much) slower</li>
Resizing images on-the-fly is very slow, even if they are the same size every time</strong></li>
Drawing small images is fast, but drawing large images is slow</li>
Drawing rectangles is fast</li>
Drawing text is fast</li>
Drawing Swing widgets in front of a canvas is fast</li>
Creating fonts on-the-fly is a tiny bit slow</li> </ul> Code</h2>
You can find the full code (written in Kotlin) at gitlab.com/andybalaam/java-2d-performance</a>.</p>
Basically, we make a JFrame and a Canvas and tell them not to listen to repaints (i.e. we control their drawing).</p>
val app = JFrame() app.ignoreRepaint = true val canvas = Canvas() canvas.ignoreRepaint = true</pre>
Then we add any buttons to the JFrame, and the canvas last (so it displays behind):</p>
app.add(button) app.add(canvas)</pre>
Now we make the canvas double-buffered and get hold of a buffer image for it:</p>
app.isVisible = true canvas.createBufferStrategy(2) val bufferStrategy = canvas.bufferStrategy val bufferedImage = GraphicsEnvironment .getLocalGraphicsEnvironment() .defaultScreenDevice .defaultConfiguration .createCompatibleImage(config.width, config.height)</pre>
Then inside a tight loop we draw onto the buffer image:</p>
val g2d = bufferedImage.createGraphics() try { g2d.color = backgroundColor g2d.fillRect(0, 0, config.width, config.height) ... the different drawing operations go here ...</pre>
and then swap the buffers:</p>
val graphics = bufferStrategy.drawGraphics try { graphics.drawImage(bufferedImage, 0, 0, null) if (!bufferStrategy.contentsLost()) { bufferStrategy.show() } } finally { graphics.dispose() } } finally { g2d.dispose() }</pre>
Results</h2>
Baseline: some rectangles</h3>
I decided to compare everything against drawing 20 rectangles at random points on the screen, since that seems like a minimal requirement for a game.</p>
My test machine is an Intel Core 2 Duo E6550 2.33GHz with 6GB RAM and a GeForce GT 740 graphics card (I have no idea whether it is being used here - I assume not). I am running Ubuntu 18.04.1 Linux, OpenJDK Java 1.8.0_191, and Kotlin 1.3.20-release-116. (I expect the results would be identical if I were using Java rather than Kotlin.)</p>
I ran all the tests in two window sizes: 1600x900 and 640x480. 640x480 was embarrassingly fast for all tests, but 1600x900 struggled with some of the tasks.</p>
Drawing rectangles looks like this:</p>
g2d.color = Color( rand.nextInt(256), rand.nextInt(256), rand.nextInt(256) ) g2d.fillRect( rand.nextInt(config.width / 2), rand.nextInt(config.height / 2), rand.nextInt(config.width / 2), rand.nextInt(config.height / 2) )</pre>
In the small window, the baseline (20 rectangles) ran at 553 FPS. In the large window it ran at 87 FPS.</p>
I didn't do any statistics on these numbers because I am too lazy. Feel free to do it properly and let me know the results - I will happily update the article.</p>
Fewer rectangles</h3>
When I reduced the number of rectangles to do less drawing work, I saw small improvements in performance. In the small window, drawing 2 rectangles instead of 20 increased the frame rate from 553 to 639, but there is a lot of noise in those results, and other runs were much closer. In the large window, the same reduction improved the frame rate from 87 to 92. This is not a huge speed-up, showing that drawing rectangles is pretty fast.</p>
Adding fixed-size images</h3>
Drawing pre-scaled images looks like this:</p>
g2d.drawImage( image, rand.nextInt(config.width), rand.nextInt(config.height), null )</pre>
When I added 20 small images (40x40 pixels) to be drawn in each frame, the performance was almost unchanged. In the small window, the run showing 20 images per frame (as well as rectangle) actually ran faster than the one without (561 FPS versus 553), suggesting the difference is negligible and I should do some statistics. In the large window, the 20 images version ran at exactly the same speed (87 FPS).</p>
So, it looks like drawing small images costs almost nothing.</p>
When I moved to large images (400x400 pixels), the small window slowed down from 553 to 446 FPS, and the large window slowed from 87 to 73 FPS, so larger images clearly have an impact, and we will need to limit the number and size of images to keep the frame rate acceptable.</p>
Scaling images on the fly</h3>
You can scale an image on the fly as you draw onto a Canvas. (Spoiler: don't do this!)</p>
My code looks like:</p>
val s = config.imageSize val x1 = rand.nextInt(config.width) val y1 = rand.nextInt(config.height) val x2 = x1 + s val y2 = y1 + s g2d.drawImage( unscaledImage, x1, y1, x2, y2, 0, 0, unscaledImageWidth, unscaledImageHeight, null )</pre>
Note the 10-argument form of drawImage is being used. You can be sure you have avoided this situation if you use the 4-argument form from the previous section.</p>
Note: the resulting image is the same size every time</strong>, and the Java documentation implies that scaled images may be cached by the system, but I saw a huge slow-down</strong> when using the 10-argument form of drawImage above.</p>
On-the-fly scaled images slowed the small window from 446 to 67 FPS(!), and the large window from 73 to 31 FPS, meaning the exact same rendering took over twice as long.</p>
Advice: check you are not using one of the drawImage overloads that scales images! Pre-scale them yourself (e.g. with getScaledInstance as I did here).</p>
Displaying text</h3>
Drawing text on the canvas like this:</p>
g2d.font = Font("Courier New", Font.PLAIN, 12) g2d.color = Color.GREEN g2d.drawString("FPS: $fpsLastSecond", 20, 20 + i * 14)</pre>
had a similar impact to drawing small images - i.e. it only affected the performance very slightly and is generally quite fast. The small window slowed from 553 to 581 FPS, and the large window from 87 to 88.</p>
Creating the font every time (as shown above) slowed the process a little more, so it is worth moving the font creation out of the game loop and only doing it once. The slowdown just for creating the font was 581 to 572 FPS in the small window, and 88 to 86 FPS in the large.</p>
Swing widgets</h3>
By adding Button widgets to the JFrame before the Canvas, I was able to display them in front. Their rendering and focus worked as expected, and they had no impact at all on performance.</p>
The same was true when I tried adding these widgets in front of images rendered on the canvas (instead of rectangles).</p>
Turning everything up to 11</h3>
When I added everything I had tested all at the same time: rectangles, text with a new font every time, large unscaled images, and large window, the frame rate reduced to 30 FPS. This is a little slow for a game already, and if we had more images to draw it could get even worse. However, when I pre-scaled the images the frame rate went up to 72 FPS, showing that Java is capable of running a game at an acceptable frame rate on my machine, so long as we are careful how we use it.</p>
Numbers</h2>
Small window (640x480)</h3>
Test</th> FPS</th></tr></thead>

nothing</td> 661</td></tr>
rectangles2</td> 639</td></tr>
rectangles20</td> 553</td></tr>
rectangles20 images2</td> 538</td></tr>
rectangles20 images20</td> 561</td></tr>
rectangles20 images20 largeimages</td> 446</td></tr>
rectangles20 images20 unscaledimages</td> 343</td></tr>
rectangles20 images20 largeimages unscaledimages</td> 67</td></tr>
rectangles20 text2</td> 582</td></tr>
rectangles20 text20</td> 581</td></tr>
rectangles20 text20 newfont</td> 572</td></tr>
rectangles20 buttons2</td> 598</td></tr>
rectangles20 buttons20</td> 612</td></tr> </tbody></table>
Large window (1200x900)</h3>
Test</th> FPS</th></tr></thead>

large nothing</td> 93</td></tr>
large rectangles2</td> 92</td></tr>
large rectangles20</td> 87</td></tr>
large rectangles20 images2</td> 87</td></tr>
large rectangles20 images20</td> 87</td></tr>
large rectangles20 images20 largeimages</td> 73</td></tr>
large rectangles20 images20 unscaledimages</td> 82</td></tr>
large rectangles20 images20 largeimages unscaledimages</td> 31</td></tr>
large rectangles20 text2</td> 89</td></tr>
large rectangles20 text20</td> 88</td></tr>
large rectangles20 text20 newfont</td> 86</td></tr>
large rectangles20 buttons2</td> 88</td></tr>
large rectangles20 buttons20</td> 87</td></tr>
large images20 buttons20 largeimages</td> 74</td></tr>
large rectangles20 images20 text20 buttons20 largeimages newfont</td> 72</td></tr>
large rectangles20 images20 text20 buttons20 largeimages unscaledimages newfont</td> 30</td></tr> </tbody></table>
Feedback please</h2>
Please do get back to me with tips about how to improve the performance of my experimental code.</p>
Feel free to log issues</a>, make merge requests</a> or add comments to the blog post.</p>

London Python Meetup January 2019 - Async Python and GeoPandas

Andy Balaam — Thu, 24 Jan 2019 14:14:46 +0000

It was a pleasure to go to the London Python Meetup</a> organised by @python_london</a>. There were plenty of friendly people and interesting conversations.</p>
I gave a talk "Making 100 million requests with Python aiohttp" (slides</a>, Blog post</a>) explaining the basics of writing async code in Python 3 and how I used that to make a very large number of HTTP requests.</p>
(Photo by CB Bailey</a>.)</p>
Hopefully it was helpful - there were several good questions, so I am optimistic that people were engaged with it.</p>
After that, there was an excellent talk by Gareth Lloyd</a> called "GeoPandas, the geospatial extension for Pandas" in which he explained how to use the very well-developed geo-spatial data tools available in the Python ecosphere to transform, combine, plot and analyse data which includes location information. I was really impressed with how easy the libraries looked to use, and also with the cool Jupyter</a> notebook Gareth used to explain the ideas using live demos.</p>
London Python Meetups seem like a cool place to meet Pythonistas of all levels of experience in a nice, low-pressure environment!</p>
Meetup link: aiohttp / GeoPandas</a></p>

Run bash inside any version of Linux using Docker

Andy Balaam — Tue, 08 Jan 2019 10:25:35 +0000

Docker is useful for some things, and not as useful as you think for others.</p>
Here's something massively useful: get a throwaway bash prompt inside any version of any Linux distribution in one command:</p>
docker run -i -t --mount "type=bind,src=$HOME/Desktop,dst=/Desktop" ubuntu:18.10 bash</pre>
This command downloads a recent Ubuntu 18.10 image, mounts my desktop as /Desktop in the container, and gives me a bash prompt. From here I can install any packages I want and then use them.</p>
For example, today I used it to decrypt a file that was encrypted with a cipher my main OS did not have a package for.</p>
When I exit bash, the container stops and I can find it with docker ps -a then remove it with docker rm. To really clean up I can find the downloaded images with docker image ls and remove them with docker image rm.</p>

Windows missing from grub menu after Ubuntu install

Andy Balaam — Thu, 03 Jan 2019 20:47:11 +0000

Windows was missing from my grub menu after I installed Ubuntu. Fortunately it was fixed by running:</p>
sudo grub-mkconfig -o /boot/grub/grub.cfg</pre>
and rebooting.</p>
Phew.</p>

New home page design

Andy Balaam — Sat, 22 Dec 2018 00:04:53 +0000

After years of getting around to it, I have redesigned my home page at artificialworlds.net</a>.</p>
It's basically intended to make me look clever or productive or interesting or something. Alternatively, it gives you somewhere to find that thing you know I made but can't find the link:</p>
</p>
The background image is "Read where you are" by delaram bayat.</a></p>
I am pleased with the page's responsive design, clarity, fast page-load, and colourfulness.</p>
What do you think?</p>

Code Like a Girl T-shirts

Andy Balaam — Mon, 17 Dec 2018 23:15:06 +0000

There are lots of people missing from the programming world: lots of the programmers I meet look and sound a lot like me. I'd really like it if this amazing job were open to a lot more people.</p>
One of the weird things that has happened is that somehow we seem to have the idea that programming is only for boys, and I'd like to fight against that idea by wearing a t-shirt demonstrating how cool I think it is to be a woman coder.</p>
So, I commissioned a design from an amazing artist called Ellie Mars</a>, who I found through her Mastodon.art page @elliemars@mastodon.art</a>. She did an amazing job, sending sketches and ideas back and forth, and finally she came up with this awesome design:</p>
</p>
I've printed a t-shirt for myself that I will give myself for Christmas, and I've made a page on Street Shirts so you can get one too!</p>
The link goes to an adult men's t-shirt, but after you click the link you can choose the t-shirt type (including women's and children's) and change the background colour.Â They are reasonably cheap:</p>

Code Like a Girl t-shirt </a></li> </ul>
If you have any questions, you are welcome to contact me (via DM or publicly) on twitter @andybalaam</a> or Mastodon @andybalaam@mastodon.social</a>, or by email, via a short test</a>.</p>
Ellie and I agreed to set up these t-shirts sales with no profit for us because we'd like to get the word out.Â If they are popular we might add a little, so get in fast for a good deal!</p>
10 points for anyone who can recognise the code in the background.Â It's from one of my favourite programs.</p>
Personally, I think we all spend too much of our time walking around advertising faceless corporations when we could be saying something a bit more useful on our clothes.Â What do you think of this idea?Â Maybe you could design a similar t-shirt?Â Let me know your thoughts in the comments below or on Twitter or Mastodon.</p>

Poor performance in Chrome (especially on mobile) - caused by SVG background images

Andy Balaam — Mon, 10 Dec 2018 01:08:19 +0000

I have spent the last few hours investigating abysmal performance in my latest little game project Cross The Road</a>. Firefox was fine, but Chromium and Chrome, especially on mobile, was rendering at about three frames per second.</p>
When I stopped using SVGs as background-images for my elements, and used PNGs instead, it improved to about 20-30 FPS.</p>
It seems fine to use SVGs as normal images, but for background-image, it really hurt performance.</p>

Interesting Characters (UTF-16, utf-8, Unicode, encodings) video

Andy Balaam — Sat, 24 Nov 2018 09:00:33 +0000

In which I try to express the weird joy I felt learning about all the odd quirks of the simple task of turning bytes into characters, and characters into bytes.</p> </iframe>
Slides: Interesting Characters slides</a></p>

Elm makes me happy (updated for Elm 0.19) video

Andy Balaam — Sat, 10 Nov 2018 02:45:18 +0000

Series: Snake in Elm</a>, Elm makes me happy, Elm Basics</a>, Elm Unit Test</a>, Elm JSON</a></p>
With up-to-date examples, and anecdotes from the last couple of years of continuing to enjoy writing web pages in Elm, here's a new version of my Elm advert:</p> </iframe>
Slides: Elm makes me happy slides</a>.</p>

Godot: make new objects at runtime (instancing) video

Andy Balaam — Sat, 10 Nov 2018 02:28:20 +0000

Series: 2D Shapes</a>, drag and drop</a>, new objects</p>
My Godot 3 game is progressing, and I am starting to think I am actually writing the level editor. Here's how I wrote code to make new versions of existing objects (by converting them to scenes):</p> </iframe>
Godot version: v3.0.6.stable.official.8314054</p>

You must rewind your incoming buffer when you fail to encode a character in a CharsetEncoder or you'll get an IllegalArgumentException

Andy Balaam — Fri, 09 Nov 2018 15:56:24 +0000

I am writing a CharsetEncoder</a> in Java, which is my kind of fun.</p>
I was getting a mysterious error when I identified that I could not encode certain characters:</p>
Exception in thread "main" java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:244) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:618) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:802) at java.nio.charset.Charset.encode(Charset.java:843) at java.nio.charset.Charset.encode(Charset.java:863)</pre>
After some investigation I realised the library code in Charset.encode was expecting me not to have consumed any characters of my incoming CharBuffer if I rejected the input by returning something like CoderResult.unmappableForLength.</p>
Of course, in order to discover the input was unmappable, I did have to read it, but I made this problem go away by stepping back one char when I found an error like this:</p>
@Override public CoderResult encodeLoop(CharBuffer in, ByteBuffer out) { char ch = in.get(); if(isUnmappable(ch)) { in.position(in.position() - 1);</span> return CoderResult.unmappableForLength(2); } // ... rest of method ...</pre>
I hope this helps someone.</p>

Godot: Dragging and dropping physics objects video

Andy Balaam — Thu, 08 Nov 2018 01:37:47 +0000

Series: 2D Shapes</a>, drag and drop, new objects</a></p>
Continuing to explore the Godot 3</a> game engine. I want to make a game where you drag blocks around and balance them on each other, but I couldn't find much documentation on how to drag-and-drop objects (except menu UI elements), and especially I found quite a few wrinkles when doing this with objects that are normally controlled by the physics engine.</p>
This time we actually write some code in Godot's programming language, GDScript.</p> </iframe>
Godot version: v3.0.6.stable.official.8314054</p>

Godot: 2D shapes bouncing off each other video

Andy Balaam — Tue, 06 Nov 2018 21:26:06 +0000

Series: 2D Shapes, drag and drop</a>, new objects</a></p>
In which I make a square and a triangle bounce on a rectangle using Godot 3</a>:</p> </iframe>
Godot version: v3.0.6.stable.official.8314054</p>

Elm JSON decoder examples

Andy Balaam — Fri, 19 Oct 2018 08:53:40 +0000

Series: Snake in Elm</a>, Elm makes me happy</a>, Elm Basics</a>, Elm Unit Test</a>, Elm JSON</p>
I find JSON decoding</a> in Elm</a> confusing, so here are some thoughts and examples.</p>
Setup</h2>
$ elm --version 0.19.0 $ mkdir myproj; cd myproj $ elm init ... $ elm install elm/json ... </pre>
To run the "Demo" parts of the examples below, type them into the interactive Elm interpreter. To try them out, start it like this:</p>
$ elm repl</pre>
and import the library you need:</p>
import Json.Decode as D</pre>
Scroll to "Concepts" at the bottom for lots of waffling about what is really going on, but if you're looking to copy and paste concrete examples, here we are:</p>
Examples</h2>
JSON object to Record</h3>
type alias MyRecord = { i : Int , s : String } recordDecoder : D.Decoder MyRecord recordDecoder = D.map2 MyRecord (D.field "i" D.int) (D.field "s" D.string)</pre>
Note: this uses a special trick - it turns out you can construct a record like MyRecord writing e.g. MyRecord 3 "qux" - the arguments have to be in the same order that the properties are declared in the record. The above only works because you can do this.</p>
Demo:</p>
> type alias MyRec = {i: Int, s: String} > myRecDec = D.map2 MyRec (D.field "i" D.int) (D.field "s" D.string) <internals> : D.Decoder MyRec > D.decodeString myRecDec "{\"i\": 3, \"s\": \"bar\"}" Ok { i = 3, s = "bar" } : Result D.Error MyRec </pre>
JSON array of ints to List</h3>
intArrayDecoder : D.Decoder (List Int) intArrayDecoder = D.list D.int</pre>
Demo:</p>
> myArrDec = D.list D.int <internals> : D.Decoder (List Int) > D.decodeString myArrDec "[3, 4]" Ok [3,4] : Result D.Error (List Int) </pre>
JSON array of strings to List</h3>
stringArrayDecoder : D.Decoder (List String) stringArrayDecoder = D.list D.string</pre>
Demo:</p>
> myArrDec2 = D.list D.string <internals> : D.Decoder (List String) > D.decodeString myArrDec2 "[\"a\", \"b\"]" Ok ["a","b"] : Result D.Error (List String)</pre>
JSON object to Dict</h3>
intDictDecoder : D.Decoder (Dict String Int) intDictDecoder = D.dict D.int</pre>
Demo:</p>
> myDictDecoder = D.dict D.int <internals> : D.Decoder (Dict.Dict String Int) > D.decodeString myDictDecoder "{\"a\": \"b\"}" Err (Field "a" (Failure ("Expecting an INT") <internals>)) : Result D.Error (Dict.Dict String Int) > D.decodeString myDictDecoder "{\"a\": 3}" Ok (Dict.fromList [("a",3)]) : Result D.Error (Dict.Dict String Int)</pre>
To build a Dict of String to String, replace D.int above withD.string.</p>
JSON array of objects to List of Records</h3>
type alias MyRecord = { i : Int , s : String } recordDecoder : D.Decoder MyRecord recordDecoder = D.map2 MyRecord (D.field "i" D.int) (D.field "s" D.string) listOfRecordsDecoder : D.Decoder (List MyRecord) listOfRecordsDecoder = D.list recordDecoder</pre>
Demo:</p>
> import Json.Decode as D > type alias MyRec = {i: Int, s: String} > myRecDec = D.map2 MyRec (D.field "i" D.int) (D.field "s" D.string) <internals> : D.Decoder MyRec > listOfMyRecDec = D.list myRecDec <internals> : D.Decoder (List MyRec) > D.decodeString listOfMyRecDec "[{\"i\": 4, \"s\": \"one\"}, {\"i\": 5, \"s\":\"two\"}]" Ok [{ i = 4, s = "one" },{ i = 5, s = "two" }] : Result D.Error (List MyRec)</pre>
Concepts</h2>
What is a Decoder?</h3>
A Decoder is something that describes how to take in JSON and spit out something. The "something" part is written after Decoder, so e.g. Decoder Int describes how to take in JSON and spit out an Int.</p>
The Json.Decode module contains a function that is a Decoder Int. It's called int:</p>
> D.int <internals> : D.Decoder Int</pre>
In some not-all-all-true way, a Decoder is sort of like a function:
-- This is a lie, but just pretend with me for a sec Decoder a : SomeJSON -> a -- That was a lie </pre></p>
To actually run your a Decoder, provide it to a function like decodeString:</p>
> D.decodeString D.int "45" Ok 45 : Result D.Error Int</pre>
So the actually-true way of getting an actual function is to combine decodeString and a decoder like int:</p>
> D.decodeString D.int <function> : String -> Result D.Error Int </pre>
When you apply decodeString to int you get a function that takes in a String and returns either an Int or an error. The error could be because the string you passed was not valid JSON:</p>
> D.decodeString D.int "foo bar" Err (Failure ("This is not valid JSON! Unexpected token o in JSON at position 1") ) : Result D.Error Int</internals></pre>
or because the parsed JSON does not match what the Decoder you supplied expects:</p>
> D.decodeString D.int "\"45\"" Err (Failure ("Expecting an INT") ) : Result D.Error Int</internals></pre>
(We supplied a String containing a JSON string, but the int Decoder expects to find a JSON int.)</p>
Side note: ints and floats are treated as different, even though the JSON Spec</a> treats them all as just "Numbers":</p>
> D.decodeString D.int "45.2" Err (Failure ("Expecting an INT") ) : Result D.Error Int</internals></pre>
What is a Value?</h3>
Elm has a type that represents JSON that has been parsed (actually, parsed and stored in a JavaScript object) but not interpreted into a useful Elm type. You can make one using the functions inside Json.Encode:</p>
> import Json.Encode as E > foo = E.string "foo" : E.Value </internals></pre>
You can even turn one of these into a String containing JSON using encode:</p>
> E.encode 0 foo "\"foo\"" : String</pre>
or interpret the Value as useful Elm types using decodeValue:</p>
> D.decodeValue D.string foo Ok "foo" : Result D.Error String</pre>
(When JSON values come from JavaScript, e.g. via flags</a>, they actually come as Values, but you don't usually need to worry about that.)</p>
However, what you can't do is pull Values apart in any way, other than the standard ways Elm gives you. So any custom Decoder that you write has to be built out of existing Decoders.</p>
How do I write my own Decoder?</h3>
If you want to make a Decoder that does custom things, build it from the existing magic Decoders, give it a type that describes the type it outputs, and insert your code using one of the mapN functions.</p>
For example, to decode only ints that are below 100:</p>
> under100 i = if i < 100 then D.succeed i else (D.fail "Not under 100") <function> : number -> D.Decoder number > intUnder100 = D.int |> D.andThen under100 <internals> : D.Decoder Int > D.decodeString intUnder100 "50" Ok 50 : Result D.Error Int > D.decodeString intUnder100 "500" Err (Failure ("Not under 100") <internals>) : Result D.Error Int</pre>
Here, we use the andThen function to transform the Int value coming from calling the int function into a Decoder Int that expresses success or failure in terms of decoding. When we do actual decoding using the decodeString funcion, this is transformed into the more familiar Result values like Ok or Err.</p>
If you want to understand the above, pay close attention to the types of under100 and intUnder100.</p>
If you want to write a Decoder that returns some complex type, you should build it using the mapN functions.</p>
For example, to decode strings into arrays of words split by spaces:</p>
> splitIntoWords = String.split " " <function> : String -> List String > words = D.map splitIntoWords D.string <internals> : D.Decoder (List String) > D.decodeString words "\"foo bar baz\"" Ok ["foo","bar","baz"] : Result D.Error (List String) </pre>
Above we used map to transform a Decoder String (the provided string function) into a Decoder (List String) by mapping it over a function (splitIntoWords) that transforms a String into a List String.</p>
Again, to understand this, look carefully at the types of splitIntoWordsand words.</p>
How do I build up complex Decoders?</h3>
Complex decoders are built by combining simple ones. Many functions that make decoders take another decoder as an argument. A good example is "JSON array of objects to List of Records" above - there we make a Decoder MyRecord and use it to decode a whole list of records by passing it as an argument to list, so that it returns a Decoder (List MyRecord) which can take in a JSON array of JSON objects, and return a List of MyRecords.</p>
Why is this so confusing?</h2>
Because Decoders are not functions, but they feel like functions. In fact they are opaque descriptions of how to interpret JSON that the Elm runtime uses to make Elm objects for you out of Values, which are opaque objects that underneath represent a piece of parsed JSON.</p>

Graft Animation Language on Raspberry Pi

Andy Balaam — Tue, 09 Oct 2018 23:44:40 +0000

Because the Rapsberry Pi uses a slightly older Python version, there is a special version of Graft</a> for it.</p>
Here's how to get it:</p>

Open a terminal window by clicking the black icon with a ">" symbol on it at the top near the left.</li>
First we need to install a couple of things Graft needs, so type this, then press Enter:
sudo apt install python3-attr at-spi2-core</pre></li>
If you want to be able to make animated GIFs, install one more thing:
sudo apt install imagemagick</pre></li>
To download Graft and switch to the Raspberry Pi version, type in these commands, pressing Enter after each line.
git clone https://codeberg.org/andybalaam/graft.git cd graft git checkout raspberry-pi</pre></li>
Now, you should be able to run Graft just like on another computer, for example, like this:
./graft 'd+=10 S()'</pre> If you're looking for a fun way to start, why not try the worksheet "Tell a story by making animations with code"</a>? For more info, see Graft Raspberry Pi Setup</a>.</li> </ul>

Worksheet: "Tell a story by making animations with code"

Andy Balaam — Mon, 24 Sep 2018 00:57:07 +0000

I'm running a workshop at the Egham Raspberry Jam</a> on 21st October</a>. The workshop will introduce my little animation language Graft</a>. We will tell a story using animations that we created ourselves using code.</p>
The worksheet for the workshop is here: PDF</a> or ODP</a>.</p>
I reproduced it as images below.</p>
Any feedback much appreciated.</p>
Comments</h2>

Graft Animation Language on Raspberry Pi – Andy Balaam's Blog 2018-10-09</span> </h3>
[…] If you’re looking for a fun way to start, why not try the worksheet “Tell a story by making animations with code”? […]</p> </div>

My experience upgrading to Elm 0.19

Andy Balaam — Wed, 12 Sep 2018 01:45:28 +0000

Elm is unstable, so upgrading to the next version can be painful. Here's what I needed to do to upgrade from 0.18 to 0.19.</p>

Replace elm-package.json and tests/elm-package.json with elm.json - e06f5a1728</a></li>
Switch to the new elm-test</a> - b964b7c7a</a></li>
Re-arrange Main, and how we call it from JavaScript - 0c118c49f</a></li>
Stop using eeue56/elm-all-dict</a> (since it's not ported to 0.19 and porting it looked hard due to a lack of Debug.crash) - fe100f256</a></li>
Replace toString with String.fromX or Debug.toString - 9e78163d0a3</a></li>
Stop "shadowing" names by making new variables with the same name as another in the scope - 9688a621de</a></li>
Adapt to the changed Html.style function - b991ab4f</a></li>
Stop using Debug.crash - f98a70ad1</a></li>
Adapt to the changes in the Regex module - 856762a4</a></li>
Stop using tuples with more than 3 parts - 472c0bb7</a></li> </ul>
The lack of Debug.crash is really, really painful, especially for a library like eeue56/elm-all-dict</a> that has lots of invariants that are hard or impossible to enforce via the type system. On the other hand, if Elm can give a hard guarantee that there will be no runtime errors, this seems pretty cool. The problem is that some code may well have to return the wrong answer silently, instead of crashing, which could be much worse than crashing in some use-cases.</p>
I was annoyed by the lack of more-than-3-part tuples, but even as I did the work to change my code, I saw it get better, so it's hard to argue with.</p>
The hardest part to work out was how to run the tests. Fortunately the tests themselves needed almost no changes. I just needed to do this:</p>
rm -r tests/elm-stuff rm tests/elm-package.json sudo npm install -g elm-test@0.19.0-beta8 elm-test install elm-test </pre>
My next job is to check out the --optimize compiler flag, and the advice on making the code smaller and faster</a>.</p>

Bulk adding items to Wunderlist using wunderline on Ubuntu MATE

Andy Balaam — Thu, 06 Sep 2018 19:53:04 +0000

If you use Wunderlist</a> and want to be able to bulk-add tasks from a text file, first install and set up wunderline</a>.</p>
Now, to be able to right-click a text file containing one task per line on Ubuntu MATE, create a file called "wunderlist-bulk-add" in ~/.config/caja/scripts/ and make it executable. Paste the code below into that file.</p>
(Note: it's possible that this would work in GNOME if you replaced "caja" with "nautius" in that path - let me know in the comments.)</p>
#!/usr/bin/env bash set -o pipefail function nonblank() { grep -v -e '^[[:space:]]*$' "$@" } for F in "$@"; do { COUNT=$(nonblank "$F" | wc -l | awk '{print $1}') if [ "$COUNT" = "" ]; then { zenity --info --no-wrap --text="File $F does not exist" } else { zenity --question --no-wrap \ --text="Add $COUNT items from $F to Wunderlist?" ANSWER=$? if [ "$ANSWER" = "0" ]; then { nonblank "$F" | \ wunderline add --stdin | \ zenity --progress --text "Adding $COUNT items to Wunderlist" SUCCEEDED=$? if [ "$SUCCEEDED" = "0" ]; then { zenity --info --no-wrap --text "Added $COUNT items." } else { zenity --error --no-wrap \ --text "Failed to add some or all of these items." }; fi }; fi }; fi }; done </pre>

Writing a new Flarum extension on Ubuntu

Andy Balaam — Thu, 06 Sep 2018 07:54:56 +0000

In a previous post I described how to install Flarum locally on Ubuntu</a>.</p>
Here is how I set up my development environment on top of that setup so I was able to write a new Flarum extension and test it on my local machine.</p>
Recap: I installed Apache and PHP and enabled mod_rewrite, installed MariaDB and made a database, installed Composer, and used Composer to install Flarum at /var/www/html/flarum.</p>
More info: Flarum Contribution Guide</a>, extension development guide</a>, extension development quick start</a>, workflow discussion</a>.</p>
I decided to call my extension "rabbitescape/flarum-ext-rabbitescape-leveleditor", so that is the name you will see below.</p>
Here's what I did:</p>
cd /var/www/html/flarum </pre>
Edit /var/www/html/flarum/composer.json (see the workbench explanation</a> for examples of the full file).</p>
At the end of the "require" section I added a line like this:</p>
"rabbitescape/flarum-ext-rabbitescape-leveleditor": "*@dev" </pre>
(Remembering to add a comma at the end of the previous line so it remained valid JSON.)</p>
After the "require" section, I added this:</p>
"repositories": [ { "type": "path", "url": "workbench/*/" } ], </pre>
Next, I made the place where my extension would live:</p>
mkdir workbench cd workbench mkdir flarum-ext-rabbitescape-leveleditor cd flarum-ext-rabbitescape-leveleditor </pre>
Now I created a file inside flarum-ext-rabbitescape-leveleditor called composer.json like this:</p>
{ "name": "rabbitescape/flarum-ext-rabbitescape-leveleditor", "description": "Allow viewing and editing Rabbit Escape levels in posts", "type": "flarum-extension", "keywords": ["rabbitescape"], "license": "MIT", "authors": [ { "name": "Andy Balaam", "email": "andybalaam@artificialworlds.net" } ], "support": { "issues": "https://github.com/andybalaam/flarum-ext-rabbitescape-leveleditor/issues", "source": "https://github.com/andybalaam/flarum-ext-rabbitescape-leveleditor" }, "require": { "flarum/core": "^0.1.0-beta.5" }, "extra": { "flarum-extension": { "title": "Rabbit Escape Level Editor" } } } </pre>
In the same directory I created a file bootstrap.php like this:</p>
<?php return function () { echo 'Hello, world!'; }; </pre>
Then I told Composer to refresh the project like this:</p>
cd ../.. # Back up to the main directory composer update </pre>
Among the output I saw a line like this which convinced me it had worked:</p>
- Installing rabbitescape/flarum-ext-rabbitescape-leveleditor (dev-master): Symlinking from workbench/flarum-ext-rabbitescape-leveleditor </pre>
I edited config.php in the main directory, and set 'debug' => true instead of false.</p>
Now I went to http://localhost/flarum, signed in as admin, adminpassword (as I set up in the previous post), clicked my username in the top right and chose Administration, then clicked Extensions on the left.</p>
I saw my extension in the list, and turned it on by clicking the check box below it. This made the whole web site disappear, and instead just the text "Hello world!" appeared. This showed my extension was being loaded.</p>
I edited bootstrap.php and commented out the line starting with "echo". I refreshed the page in my browser and saw the Flarum site reappear.</p>
Finally, to install what's needed for changing Flarum's UI, I needed to run:</p>
sudo apt install npm sudo npm install --global gulp </pre>
Now, my extension is installed and ready to be developed! See flarum.org/docs/extend/start</a> for how to get started making it do cool stuff.</p>

Ubuntu "compose" key for easy unicode character input

Andy Balaam — Wed, 05 Sep 2018 12:29:15 +0000

I found out on Mastodon</a> recently that the Compose key</a> exists, allowing you to enter special characters using easy-to-remember key sequences (e.g. "<compose></em>/=" gives you "â‰ ").</p>
To do this on Ubuntu (and probably many other systems), hold SHIFT, then press AltGr, release them both, and type something like "/=". You should see the â‰ symbol appear.</p>
In Ubuntu MATE, you can customise what key is the compose key by opening "Keyboard", clicking "Layouts", then "Options..." and checking a box under "Position of compose key". I checked "Right Alt", meaning my "AltGr" key is now the compose key.</p>
More info: Ubuntu ComposeKey docs</a>.</p>
Want to explore? See the list of the default sequences you can type</a>.</p>
Comments</h2>

Albert 2018-12-11</span> </h3>
Sounds cool, but doesn’t seem to work in Ubuntu 16.04. Maybe it only works with a specific locale or xkeyboard layout?</p> </div>

Andy Balaam 2018-12-12</span> </h3>
Hi Albert, I’m surprised to hear it doesn’t work for you, what a pity. Are you sure you pressed SHIFT and ALT GR together, then released them and then pressed "/="?</p> </div>

Albert 2018-12-12</span> </h3>
Found the problem. In System Settings / Keyboard / Shortcuts / Typing, I had "Compose Key" set to Disabled. The help text underneath says "To edit a shortcut, click the row and hold down the new keys …". That doesn’t work, even though it does for other shortcuts. However, clicking on Disabled gives a menu with no Shift-AltGr, but with "Right Alt" among others. After selecting that, it works (both with just AltGr, or Shift-AltGr). Thanks for this cool tip.</p> </div>

Andy Balaam 2018-12-12</span> </h3>
Brilliant, thank you!</p> </div>

Albert 2018-12-13</span> </h3>
Bad news: the Compose Key was disabled on my system for a good reason. If I set it to AltGr, I can no longer use it to type the various characters and accents which require it. So I guess this Compose Key trick only applies to a plain US keyboard layout. Sounds very useful for Americans, but only for Americans. Thanks anyway. It was intersting, and may be useful if I’m stuck some day in some Internet cafÃ© with a US layout.</p> </div>

Andy Balaam 2018-12-14</span> </h3>
Sorry to hear that. It works on a UK keyboard too. You could try re-mapping it to a different key entirely?</p> </div>

Redirecting all requests to https and www using .htaccess in Apache

Andy Balaam — Tue, 04 Sep 2018 07:23:57 +0000

I want all requests to artificialworlds.net/rabbit-escape/levels/</a> to get redirected to use the https protocol, and to include "www." at the beginning of the URL, and I found lots of Stack Overflow articles, but nothing there worked perfectly for me. Here is how I managed it.</p>
I edited the .htaccess file in the directory where I want this to apply, and added this at the top:</p>
<IfModule mod_rewrite.c> RewriteEngine On RewriteCond %{HTTP_HOST} ^artificialworlds\.net$ [NC] RewriteRule ^ https://www.artificialworlds.net%{REQUEST_URI} [R=301,L] RewriteCond %{HTTPS} off RewriteRule ^ https://www.artificialworlds.net%{REQUEST_URI} [R=301,L] </IfModule> </pre>

Example of a systemd service file

Andy Balaam — Thu, 23 Aug 2018 13:21:14 +0000

Here is an almost-minimal example of a systemd service file, that I use to run the Mastodon</a> bot of my generative art playground Graft</a>.</p>
I made a dedicated user just to run this service, and installed Graft into /home/graft/apps/graft under that username. Now, as root, I edited a file called /etc/systemd/system/graft.service and made it look like this:</p>
[Service] ExecStart=/home/graft/apps/graft/bot-mastodon User=graft Group=graft WorkingDirectory=/home/graft/apps/graft/ [Install] WantedBy=multi-user.target </pre>
Now I can start the graft service like any other service like this:</p>
sudo systemctl start graft</pre>
and find out its status with:</p>
sudo systemctl status graft</pre>
If I want it to run on startup I can do:</p>
sudo systemctl enable graft</pre>
and it will. Easy!</p>
If I want to look at its output, it's:</p>
sudo journalctl -u graft</pre>
As a reward for reading this far, here's a little animation you can make with Graft</a>:</p>
If you want the service to be restarted whenever it fails, under [Service] add something like this:</p>
Restart=always RestartSec=3</pre>
More info at: Jon Archer's blog post</a>.</p>

Installing Flarum on Ubuntu 18.04

Andy Balaam — Wed, 22 Aug 2018 19:44:25 +0000

I am setting up a forum for sharing levels for my game Rabbit Escape</a>, and I have decided to try and use Flarum</a>, because it looks really usable and responsive, has features we need like liking posts and following authors, and I think it will be reasonably OK to write the custom features we want.</p>
So, I want a dev environment on my local Ubuntu 18.04 machine, and the first step to that is a standard install.</p>
Warning: at the time of writing the Flarum docs say it does not work with PHP 7.2, which is what is included with Ubuntu 18.04, so this may not work. (So far it looks OK for me.)</p>
Here's how I got it working:</p>
sudo apt install \ apache2 \ libapache2-mod-php \ mariadb-server \ php-mysql \ php-json \ php-gd \ php-tokenizer \ php-mbstring \ php-curl php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" # Get the next line from https://getcomposer.org/download/ # Don't copy it exactly! php -r "if (hash_file('SHA384', 'composer-setup.php') === '544e09ee996cdf60ece3804abc52599c22b1f40f4323403c44d44fdfdd586475ca9813a858088ffbc1f233e9b180f061') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" mkdir $HOME/bin php composer-setup.php --install-dir=$HOME/bin/ --filename=composer rm composer-setup.php cd /var/www/html sudo mkdir flarum sudo chown $(whoami) flarum # Log out and in again here to get composer to be in your PATH cd flarum composer create-project flarum/flarum . --stability=beta sudo chgrp -R www-data . sudo chmod -R 775 . sudo a2enmod rewrite </pre>
Next I granted the permissions for Flarum's .htaccess file to change settings by modifying /etc/apache2/apache2.conf and adding this after the existing <Directory entries:</p>
<Directory "/var/www/html/flarum/"> AllowOverride All </Directory> </pre>
and restarted Apache:</p>
sudo systemctl restart apache2 </pre>
Now I created a database in MariaDB:</p>
$ sudo mariadb MariaDB [(none)]> CREATE USER flarumuser IDENTIFIED BY 'flarumpassword'; MariaDB [(none)]> CREATE DATABASE flarumdb; MariaDB [(none)]> GRANT ALL ON flarumdb.* TO flarumuser; </pre>
and went to http://localhost/flarum</a> in my browser, and entered these details:</p>
MySQL host: localhost MySQL database: flarumdb MySQL user: flarumuser MySQL password: flarumpassword table prefix: flarum_ Admin user: admin Admin email: <my email address> Admin password: adminpassword </pre>
Once I typed all those correctly and submitted, I had to wait a while, and then after it printed some details I refreshed the page and saw a local Flarum ready to customise.</p>
Next, I looked into how to get an extension development environment up and running</a>.</p>
If you want to find and share levels for Rabbit Escape, check up on our progress setting up the forum at https://artificialworlds.net/rabbit-escape/levels</a>.</p>

Rabbit Escape 0.11 out now!

Andy Balaam — Sun, 19 Aug 2018 16:57:34 +0000

The RABBOTS are coming!</p>
Get the latest version of Rabbit Escape:</p>

Android: Rabbit Escape on F-Droid</a> (free, no ads)</li>
Android: Rabbit Escape Free on the Play Store</a> (free, no ads)</li>
Android: Rabbit Escape on the Play Store</a> (Pay about 60p, identical to the free one!)</li>
Linux, Windows, Mac and Android: Direct from the Rabbit Escape web site</a> (free, no ads)</li> </ul>

Migrating videos from YouTube to PeerTube inside a Docker container

Andy Balaam — Wed, 15 Aug 2018 09:21:22 +0000

I have quite a few videos hosted on YouTube</a> that I would like to upload to my new PeerTube location</a>, but I don't want to install all the PeerTube dependencies on my machine, so I did it all inside a Docker image.</p>
First I built and started a Docker container:</p>
$ git clone https://github.com/chocobozzz/PeerTube /tmp/peertube $ cd /tmp/peertube $ docker build . -f ./support/docker/production/Dockerfile.buster --tag peertube $ docker run --tty --interactive peertube bash </pre>
Then I ran these commands inside it:</p>
# apt update && apt install git -y && apt upgrade -y # npm install -g npm n typescript # n stable # yarn upgrade yarn # NODE_ENV=develop NOCLIENT=1 yarn install --pure-lockfile --production=false # NODE_ENV=develop npm run setup:cli # node dist/server/tools/peertube-import-videos.js --url "https://peertube.mastodon.host" --username "andybalaam" --target-url "https://www.youtube.com/watch?v=TG0qRDrUPpA" --password=mypw </pre>
Of course, it would be better to write this up into its own Dockerfile to make this a one-liner.</p>
References: PeerTube Docker setup</a>, PeerTube video import</a>.</p>

How to write a programming language articles

Andy Balaam — Mon, 13 Aug 2018 11:10:11 +0000

Recent Overload</a> journal issues contain my new articles on How to Write a Programming Language.</p>
Part 1: How to Write a Programming Language: Part 1, The Lexer</a></p>
Part 2: How to Write a Programming Language: Part 2, The Parser</a></p>
PDF of the latest issue: Overload 146</a> containing part 2.</p>
This is all creative-commons licensed and developed in public at codeberg.org/andybalaam/articles-how-to-write-a-programming-language</a></p>

Allow drag-to-side, but not drag-to-top in Ubuntu MATE (Marco)

Andy Balaam — Fri, 13 Jul 2018 11:22:00 +0000

I love the "tiling" feature in many window managers including Marco</a> that means I can drag windows to the side of the screen and get them covering one half.</p>
However, I never use the similar feature that allows dragging a window to the top, and it often triggers when I just want to move a window upwards.</p>
Today I discovered that Marco (the non-fancy window manager in Ubuntu MATE and probably other places) does allow me to have one without the other, even though the UI configuration tools don't expose the option.</p>
Here's what I did:</p>
gsettings set org.mate.Marco.general allow-top-tiling false </pre>
Now my windows can be dragged to the side for half-screen maximisation, but not to the top for full-screen!</p>

Connecting to Slack from an IRC client using slirc

Andy Balaam — Fri, 15 Jun 2018 11:03:54 +0000

I tried to get back to an IRC interface to Slack using Matrix</a>, and it had some problems. Thanks to Colin Watson's comment on that post, I tried Daniel Beer's slirc</a>, and so far it seems to be working pretty well.</p>
Here's what I did:</p>
Get a Slack legacy token</a> which slirc will use to connect to Slack as you. Follow the instructions given at that link, and you should end up with a token that looks something like "abcd-123456768-ETC-ETC". Keep a note of it.</p>
Install the prerequisites for slirc, and download it:</p>
sudo cpan AnyEvent AnyEvent::HTTP AnyEvent::Socket AnyEvent::WebSocket::Client URI::Encode Data::Dumper JSON mkdir slirc cd slirc wget -q 'https://www.dlbeer.co.nz/articles/slirc/slirc-20180515.pl' chmod +x slirc-20180515.pl</pre>
Create a file in the slirc directory you created above, called rc.conf, and make it look like this:</p>
slack_token=abcd-123456768-ETC-ETC password=somepassword port=6667</pre>
Replace "abcd-123456768-ETC-ETC" with the Slack legacy token you noted down earlier.</p>
Replace "somepassword" with something you've made up (not your Slack password) - this is what you will type as the password in your IRC client.</p>
Run slirc and leave it running:</p>
./slirc-20180515.pl rc.conf</pre>
(Make sure you are inside the slirc dir when you run that.)</p>
Start your IRC client (e.g. HexChat</a>) and add a server with address "localhost" and port 6667, with your slack username and the password you added in the rc.conf (which you wrote instead of "somepassword").</p>
This mostly works for me, except it has a tendency to open a load of ad-hoc chats as channels, so I have to close them all to get a usable list.</p>

Using Matrix to connect to Slack from an IRC client on Ubuntu

Andy Balaam — Fri, 08 Jun 2018 16:08:35 +0000

Update:</strong> I found a better solution using slirc</a>.</p>
I like using HexChat</a> to talk to my colleagues, like one other guy</a>. It is fast, and it pops up a new window when someone sends me a direct/private message.</p>
Recently, Slack</a> shut down their IRC gateway</a>, forcing me to use their slow UI, and making me unable to be responsive to direct messages without spending my entire life checking said UI, or being disturbed by notifications.</p>
So, I decided to set up a Matrix</a> bridge, because how hard can it be?</p>
My system is Ubuntu MATE 18.04.</p>
What I did</h2>
Install Synapse</h3>
Install Synapse</a>, a Matrix homeserver.</p>
wget -qO - https://matrix.org/packages/debian/repo-key.asc | sudo apt-key add - sudo apt-add-repository http://matrix.org/packages/debian/ sudo apt update sudo apt install matrix-synapse</pre>
Edit /etc/matrix-synapse/homeserver.yaml to change the line containing "registration_shared_secret" to look something like:</p>
registration_shared_secret: "some secret password you made up"</pre>
(Replacing "some secret password you made up" with something secret.)</p>
Make a new user:</p>
register_new_matrix_user -c /etc/matrix-synapse/homeserver.yaml http://localhost:8008</pre>
Enter a username and password, and take a note of them.</p>
Install Riot</h3>
(Technically, this step is optional, but I definitely recommend it to check everything is working.)</p>
Install Riot</a>, a client that can talk to a homeserver.</p>
curl https://riot.im/packages/debian/repo-key.asc | sudo apt-key add - sudo apt-add-repository https://riot.im/packages/debian/ sudo apt update sudo apt install riot-web</pre>
Start Riot from the menu or by typing riot-web in a console.</p>
Choose to log in to a Custom Server (not create a new user), and enter the Home Server URL as http://localhost:8008, leave the Identity Server URL as it is, and enter the username and password you entered in the previous step.</p>
Riot should be able to connect to the Synapse server, even though it won't have any rooms in it yet.</p>
If this doesn't work, try running riot-web in a console, and check the Synapse log file at /var/log/matrix-synapse/homeserver.log</p>
Install matrix-puppet-slack</h3>
(Note: after this step, the homeserver can only be for you now, because it is logged into Slack as you. If you want to make a new Matrix server for lots of people, I found that matrix-appservice-slack</a> works, but this only connects channels, not direct messages.)</p>
Get a Slack legacy token</a> which matrix-puppet-slack will use to connect to Slack as you. Follow the instructions given at that link, and you should end up with a token that looks something like "abcd-123456768-ETC-ETC". Keep a note of it.</p>
Install matrix-puppet-slack</a> which allows you to connect a homeserver to a Slack instance, pretending it is you:</p>
cd git clone https://github.com/matrix-hacks/matrix-puppet-slack.git cd matrix-puppet-slack/ npm install cp config.sample.json config.json</pre>
Edit config.json so it looks something like this:</p>
{ "slack": [{ "team_name": "Name of my team", "user_access_token": "abcd-123456768-ETC-ETC" }], "registrationPath": "slack-registration.yaml", "port": 8090, "bridge": { "homeserverUrl":"http://localhost:8008", "domain": "localhost", "registration": "slack-registration.yaml" } }</pre>
Now generate a config file that Synapse will use, called slack-registration.yaml, by typing this command (still in the matrix-puppet-slack directory):</p>
node index.js -r -u "http://localhost:8090"</pre>
When asked, type in the username and password you entered in the first step (when you ran the register_new_matrix_user command).</p>
This creates a config file telling Synapse that matrix-puppet-slack is running on the local machine on port 8090.</p>
Now, run it:</p>
node index.js</pre>
and edit Synapse's config file /etc/matrix-synapse/homeserver.yaml to point at the new config file you just made, by editing the line mentioning app_service_config_files so that it looks like this:</p>
app_service_config_files: [ "/path/to/matrix-puppet-slack/slack-registration.yaml" ]</pre>
Make sure you replace "/path/to/matrix-puppet-slack" in the above with the directory where you put matrix-puppet-slack, which is /home/yourusername/matrix-puppet-slack if you followed the instructions exactly.</p>
Restart Synapse, and when you go back into Riot, you should see your Slack channels appear gradually (as people talk in them):</p>
sudo service matrix-synapse restart</pre>
If not, check the output from the node index.js command, and the Synapse log file at /var/log/matrix-synapse/homeserver.log.</p>
Install matrix-ircd</h3>
To allow connecting to Matrix from an IRC client, install matrix-ircd</a>:</p>
cd git clone https://github.com/matrix-org/matrix-ircd.git cd matrix-ircd cargo build cargo run -- --url "http://localhost:8008"</pre>
(Note this works on Ubuntu 18.04, but operating systems with older versions of Rust may need to get the latest first - see the matrix-ircd</a> page for details.)</p>
Connect from an IRC client</h3>
Now use an IRC client to connect to matrix-ird. I recommend HexChat</a> if you're looking for one.</p>
matrix-ircd runs on port 5999 by default, so you should be able to connect to it from your IRC client by setting the server to localhost/5999 and using the username and password from the first step (when you ran the register_new_matrix_user command). Use plain username/password authentication.</p>
If it works, you should see your Slack channels appear in your IRC client. If not check the logs mentioned before, and the error messages and logs from your IRC client.</p>
What works and doesn't work</h2>

Messages typed by me and others into IRC, Riot and Slack appear in all the other places, including direct messages.</li>
Synapse and Riot are installed fully, so Synapse will be running after a reboot, and Riot is available in the menus, but matrix-puppet-slack and matrix-ircd are just running in a console, and have to be manually started. It should be reasonably simple to make custom systemd files to start them automatically.</li>
matrix-puppet-slack crashed once and I had to restart it, so this may not be very reliable. I logged an issue</a>.</li>
The Slack channel names look terrible in my IRC client. I think matrix-ircd needs to find the pretty names (which are displayed correctly in Riot) and use them instead of the coded names used under the cover in Matrix.</li>
My first ever direct message to someone does not work, even though I can choose them and attempt to send a message. As soon as they say something to me, they appear as a channel in Riot and IRC, and I can talk back and forth no problem from then on.</li>
Direct messages look like normal channels in Matrix, which means I can't use HexChat to pop up notifications for them, so this was all pointless.</li> </ul>
Please leave comments below if you know how I can fix any of these problems.</p>
Conclusions</h2>

Matrix seems really cool</li>
It is basically possible to bridge IRC-Matrix-Slack as I wanted</li>
But, some remaining bugs mean I have to keep using Slack's UI for now :-(</li>
Please comment telling me how to do this better</li> </ul>
Comments</h2>

Colin Watson 2018-06-13</span> </h3>
I’ve been using slirc (https://www.dlbeer.co.nz/articles/slirc/index.html</a>) since the gateway shutdown. It’s pretty simple and doesn’t have too many moving parts to go wrong. Not everything’s perfect, but it’s been working tolerably well for me.</p> </div>

Andy Balaam 2018-06-15</span> </h3>
Colin, thanks! More success that way: http://www.artificialworlds.net/blog/2018/06/...</a></p> </div>

kenf 2021-02-15</span> </h3>
Hi,</p>
Can you please guide me into how to issue the link command after setting up the bridge. I don’t know where/how to set the administration control room</p> </div>

Andy Balaam 2021-02-15</span> </h3>
Hi kenf, I’m afraid I don’t understand the question, which means I am probably not the right person to ask, sorry. I suggest asking the matrix-puppet-slack people.</p> </div>

Clever Things People Do In Groovy So You Have To Know About Them video

Andy Balaam — Sun, 03 Jun 2018 16:34:00 +0000

Groovy</a> has lots of interesting syntax that can be used for domain-specific languages, such as Gradle</a> build files, and Jenkinsfiles</a>. I try to demystify the syntax tricks a bit so you have a chance to read and understand what the code is actually doing:</p> </iframe>
Slides</a> and source code</a> are available.</p>
Related: Gradle: what is a task, and how can I make a task depend on another task?</a></p>
Comments</h2>

Gradle: what is a task, and how can I make a task depend on another task? – Andy Balaam's Blog 2019-02-27</span> </h3>
[…] [Related: Clever things people do in Groovy so you have to know about them] […]</p> </div>

Examples of SQL join types (LEFT JOIN, INNER JOIN etc.)

Andy Balaam — Thu, 26 Apr 2018 11:22:04 +0000

I have 2 tables like this:</p>
> SELECT * FROM table_a; +------+------+ | id | name | +------+------+ | 1 | row1 | | 2 | row2 | +------+------+ > SELECT * FROM table_b; +------+------+------+ | id | name | aid | +------+------+------+ | 3 | row3 | 1 | | 4 | row4 | 1 | | 5 | row5 | NULL | +------+------+------+ </pre>
INNER JOIN cares about both tables</h2>
INNER JOIN cares about both tables, so you only get a row if both tables have one. If there is more than one matching pair, you get multiple rows.</p>
> SELECT * FROM table_a a INNER JOIN table_b b ON a.id=b.aid; +------+------+------+------+------+ | id | name | id | name | aid | +------+------+------+------+------+ | 1 | row1 | 3 | row3 | 1 | | 1 | row1 | 4 | row4 | 1 | +------+------+------+------+------+ </pre>
It makes no difference to INNER JOIN if you reverse the order, because it cares about both tables:</p>
> SELECT * FROM table_b b INNER JOIN table_a a ON a.id=b.aid; +------+------+------+------+------+ | id | name | aid | id | name | +------+------+------+------+------+ | 3 | row3 | 1 | 1 | row1 | | 4 | row4 | 1 | 1 | row1 | +------+------+------+------+------+ </pre>
You get the same rows, but the columns are in a different order because we mentioned the tables in a different order.</p>
LEFT JOIN only cares about the first table</h2>
LEFT JOIN cares about the first table you give it, and doesn't care much about the second, so you always get the rows from the first table, even if there is no corresponding row in the second:</p>
> SELECT * FROM table_a a LEFT JOIN table_b b ON a.id=b.aid; +------+------+------+------+------+ | id | name | id | name | aid | +------+------+------+------+------+ | 1 | row1 | 3 | row3 | 1 | | 1 | row1 | 4 | row4 | 1 | | 2 | row2 | NULL | NULL | NULL | +------+------+------+------+------+ </pre>
Above you can see all rows of table_a even though some of them do not match with anything in table b, but not all rows of table_b - only ones that match something in table_a.</p>
If we reverse the order of the tables, LEFT JOIN behaves differently:</p>
> SELECT * FROM table_b b LEFT JOIN table_a a ON a.id=b.aid; +------+------+------+------+------+ | id | name | aid | id | name | +------+------+------+------+------+ | 3 | row3 | 1 | 1 | row1 | | 4 | row4 | 1 | 1 | row1 | | 5 | row5 | NULL | NULL | NULL | +------+------+------+------+------+ </pre>
Now we get all rows of table_b, but only matching rows of table_a.</p>
RIGHT JOIN only cares about the second table</h2>
a RIGHT JOIN b gets you exactly the same rows as b LEFT JOIN a. The only difference is the default order of the columns.</p>
> SELECT * FROM table_a a RIGHT JOIN table_b b ON a.id=b.aid; +------+------+------+------+------+ | id | name | id | name | aid | +------+------+------+------+------+ | 1 | row1 | 3 | row3 | 1 | | 1 | row1 | 4 | row4 | 1 | | NULL | NULL | 5 | row5 | NULL | +------+------+------+------+------+ </pre>
This is the same rows as table_b LEFT JOIN table_a, which we saw in the LEFT JOIN section.</p>
Similarly:</p>
> SELECT * FROM table_b b RIGHT JOIN table_a a ON a.id=b.aid; +------+------+------+------+------+ | id | name | aid | id | name | +------+------+------+------+------+ | 3 | row3 | 1 | 1 | row1 | | 4 | row4 | 1 | 1 | row1 | | NULL | NULL | NULL | 2 | row2 | +------+------+------+------+------+ </pre>
Is the same rows as table_a LEFT JOIN table_b.</p>
No join at all gives you copies of everything</h2>
If you write your tables with no JOIN clause at all, just separated by commas, you get every row of the first table written next to every row of the second table, in every possible combination:</p>
> SELECT * FROM table_b b, table_a; +------+------+------+------+------+ | id | name | aid | id | name | +------+------+------+------+------+ | 3 | row3 | 1 | 1 | row1 | | 3 | row3 | 1 | 2 | row2 | | 4 | row4 | 1 | 1 | row1 | | 4 | row4 | 1 | 2 | row2 | | 5 | row5 | NULL | 1 | row1 | | 5 | row5 | NULL | 2 | row2 | +------+------+------+------+------+ </pre>
Comments</h2>

MySQL: Quick breakdown of the types of joins [duplicate] 2022-11-06</span> </h3>
… […]</p> </div>

Fixing Slack emojis in HexChat

Andy Balaam — Fri, 16 Feb 2018 15:53:05 +0000

If, like me, you are this person:</p>
(Source: xkcd.com/1782</a>)</p>
You may want to fix the stupid :slightly_smiling_face: messages you receive from Slack via the IRC gateway</a>. Obviously, I'd prefer they went away entirely, but it's still better to see a character than being spammed with colon abominations all over the place.</p>
You'll need the Python emoji package</a>, and a HexChat plugin like this:</p>
# Replace all the horrible :slightly_smiling_face: rubbish that Slack inserts # into horrible Unicode emoji symbols. # Author: Andy Balaam # License: CC0 https://creativecommons.org/publicdomain/zero/1.0/ # Requires https://pypi.python.org/pypi/emoji - I used 0.4.5 # I manually copied the emoji dir into: # /home/andrebal/.local/lib/python2.7/site-packages import emoji import hexchat __module_name__ = "slack-emojis" __module_version__ = "1.0" __module_description__ = "Translate emojis from Slack with colons into emojis" print "Loading slack-emojis" chmsg = "Channel Message" prmsg = "Private Message to Dialog" def preprint(words, word_eol, userdata): txt = word_eol[1] replaced = emoji.emojize(txt, use_aliases=True) if replaced != txt: hexchat.emit_print( userdata["msgtype"], words[0], replaced.encode('utf-8'), ) return hexchat.EAT_HEXCHAT else: return hexchat.EAT_NONE hexchat.hook_print(chmsg, preprint, {"msgtype": chmsg}) hexchat.hook_print(prmsg, preprint, {"msgtype": prmsg})</pre>
According to the page linked above, Slack are retiring the IRC gateway, which will make me very unhappy.</p>
Update:</strong> added support for private messages too.</p>
Comments</h2>
Victor 2018-04-29</span> </h3>
Hey there,</p>
I am not that coding expert, I just use wordpress and its plugins.</p>
Sorry for the noob question… Is there a way to make it work as a Wordpress plugin?</p>
(I have connected a slack channel to create posts in wordpress, through zapier. The posts are created with the abomination :smile: stuff… )</p>
Thanks in advance for your reply</p> </div>

Andy Balaam 2018-04-30</span> </h3>
Hi Victor, my plugin is written in Python, and WordPress plugins have to be PHP as far as I know, so there is no direct way to use it, I’m afraid :–(</p> </div>

Eitan Adler 2018-07-08</span> </h3>
I added this here for more visibility: https://github.com/hexchat/hexchat-addons/tre...</a></p> </div>

Andy Balaam 2018-07-09</span> </h3>
Awesome thanks!</p> </div>

Ideas on how lexing will work in Pepper3

Andy Balaam — Fri, 09 Feb 2018 13:14:48 +0000

I am trying to practice documentation-driven development in Pepper3</a>, so every time I start on an area, I will write documentation explaining how it works, and include examples that are automatically verified during the build.</p>
I've started work on lexing</a>, since you can't do much before you do that, but in fact, of course, I need to have a command line interface before I can verify any of the examples, so I'm working on that too.</p>
Lexing is the process that takes a stream of characters (e.g. from a file) and turns it into a stream of "tokens" that are chunks of code like a variable name, a number or a string. (There is more on lexing</a> in my mini programming language, Cell</a>.)</p>
My thoughts so far about lexing are in lexing.md</a>, and current ideas about command line interface are at command_line.md</a>. All very much subject to change.</p>
Headlines:</p>

Ordinary programmers can write their own lexing rules.</li>
Operators (functions like "+" that find their arguments on their left and right, instead of between brackets like normal functions) are defined at the lexing phase, so any symbol (e.g. "in") can be an operator if you want.</li>
Anything you might want to do with a pepper program, including running it, compiling it, packaging it for an distribution system, should be available as a sub-command of the main pepper3 command line.</li>
The command is "pepper3", never "pepper". If a new, incompatible version comes out, it will be called "pepper4", and they will be parallel-installable, with no confusion.</li> </ul>

Deleting commits from the git history

Andy Balaam — Tue, 06 Feb 2018 13:30:58 +0000

Today I wanted to fix a Git repo that contained some bad commits (i.e. git fsck complained about them). [I wanted to do this because GitLab was not allowing me to push the bad commits.]</p>
I wanted the code to look exactly as it did before, but the history to look different, so the bad commits disappeared, and (presumably) the work done in the bad commits to look like it was done in the commits following them.</p>
Here's what I ran:</p>
git filter-branch -f --commit-filter ' if [ "${GIT_COMMIT}" = "abdcef012345abcdef012345etcetcetc" ]; then echo "Skipping GIT_COMMIT=${GIT_COMMIT}" >&2; skip_commit "$@"; else git commit-tree "$@"; fi ' --tag-name-filter cat -- --all </pre>
(Where abdcef012345abcdef012345etcetcetc was the ID of the commit I wanted to delete.)</p>
Of course, you can make this cleverer to exclude multiple commits at a time, or run this several times, putting in the right commit ID each time.</p>

Questions and answers about Pepper3

Andy Balaam — Mon, 22 Jan 2018 09:58:22 +0000

Series: Examples</a>, Questions</p>
My last post Examples of Pepper3 code</a> was a reply to my friend's email asking what it was all about. They replied with some questions, and I thought the questions and answers might shed some more light:</p>

Questions!</p> </blockquote>
Brilliant ones, thanks.</p>

In general though you've said a lot about what Pepper can do without giving design decisions.</p> </blockquote>
Yep, total brain dump.</p>

Remind me again who this language is for :)</p> </blockquote>
It's a multi-paradigm (generic, functional, OO) language aimed at application programmers who want:</p>

"native" performance on their chosen platform (definitely including actual native machine code). This is inspired by C++.</li>
easy deployment (preferably a single binary containing everything, with an option to link most dependencies statically), including packaging of installers for major OSes. This is inspired by C++, and the pain of C++.</li>
perfect flexibility for creating types - "meta-programming" is just programming. Things you would have done using code generation (e.g. generating a class hierarchy from an XSD) are done by running arbitrary code at compile time. The powerful type system is inspired by Haskell and the book "Modern C++ Design", and the meta-programming is inspired by Lisp.</li>
Simple memory management without GC through ownership. This is inspired by modern C++, and then Rust came along and implemented it before I could, thus proving it works. However, I would remove a lot of the functionality in Rust (lifetimes) to make it much simpler.</li>
Strong support for functional programming if you want it. This is inspired by Haskell.</li>
The simplest possible core language, with application programmers able to expand it by giving them the same tools as the language designers - e.g. "for" is just a function, so you can make your own. I am hoping I can even make "class" a function. This is inspired by Lisp, and oppositely-inspired by Java.</li>
Separation between the idea of Interfaces, which I think I will call "type specifiers" (and will allow arbitrary code execution to determine whether a type satisfies the requirements) and structs/classes, allowing us to make new Interfaces and have old code satisfy them, meaning we can do generic stuff with e.g. ints even if no-one declared that "class Int : public Quaternion" or whatever.</li>
Lots of "nudges" towards things that are good: by default things will be functional and immutable - you will have to explicitly say if you want to use more dangerous constructs like side effects and mutable values.</li>
No implicit conversions, or really anything happening without you saying so.</li> </ul>

Can you assign floats to ints or vice versa?</p> </blockquote>
Yes, but you shouldn't.</p>

If you're setting types in code at the start of a file, is this only available in the main file? Are there multiple files per program? Can you have libraries? If so, do these decide the functionality of their types in the library or does this only happen in the main file?</p> </blockquote>
I haven't totally decided - either by being enforced, or as a matter of style, you will generally do this once at the beginning of the program (and choose on the compiler command line to do it e.g. the debug way or the release way) and it will affect all of your code.</p>
Libraries will be packaged as Pepper3 source code, so choices you make of the type of Int etc. will be reflected through the whole dependency tree. Cool, huh?</p>
This is inspired by Python.</p>

Can you group variables together into structs or similar?</p> </blockquote>
Yes - it will be especially easy to make "value types", and lots of default methods will be provided, that you will be strongly encouraged to use - e.g. copy and move operations. This is inspired by Elm.</p>

Why are variables immutable by default but mutable with a special syntax? It's the opposite of C++ const, but why that way around?</p> </blockquote>
This is one of the "nudges" - immutable stuff is much easier to think about, and makes parallel stuff easier, and allows optimisations and so on, so turning it on by default means you have to choose to take the bad path, and are inclined to take the virtuous one. This is inspired by Haskell and Rust.</p>

Why only allow assignments, function calls and operators? I'm sure you have good reasons.</p> </blockquote>
To be as simple as possible, so you only have those things to learn and the rest can be understood by just reading the code. This is inspired by Python.</p>
I wrote more of my (earlier) thoughts in this 4-post series, which is better thought through: Goodness in Programming Languages</a></p>

Examples of Pepper3 code

Andy Balaam — Fri, 19 Jan 2018 12:08:44 +0000

Series: Examples, Questions</a></p>
I have restarted my effort to make a new programming language that fits the way I like things. I haven't pushed any code yet, but I have made a lot of progress in my head to understand what I want.</p>
Here are some random examples that might get across some of the ways I am thinking:</p>
// You code using general types like "Int" but you can set what // they really are in the code (usually at the beginning), so // if you plan to use native ints in the production code, it's // a good idea to use: Int = CheckedNativeInt; // while in dev, since it will crash at runtime if you overflow. // Then, in production when you're sure you have no errors, // switch to an unchecked one: Int = NativeInt; // But, if you prefer correctness over efficiency, you can use // mathematical integer that never overflows: Int = ArbitrarySizeInt; // Variables are immutable by default, so: Int x = 4; x = 3; // this is a compile error // But this is OK Mutable(Int) y = 6; y = y + x; // Notice that you can call functions that return types that you // then use, like Mutable(Int) here. // Generally, code can run at either compile time or run time. // Code to do with types has to run at compile time. // By default, other code runs at run time, but you can force // it to run early if you want to. // A main method looks like this - you get hold of e.g. stdout through // a World instance - I try to avoid any global functions like print, or // global variables like sys.stdout. Auto main = {:(World world)->Int //... }; // (Although note that Int, String etc. actually are global variables, // which is a bit annoying) // I wish the main method were simpler-looking. The only saving grace // is that for simple examples you don't need a main method - // Pepper3 just calculates the expression you provide in your file and // prints it out. // Expressions in curly brackets are lambda functions, so: {3}; // is a function taking no arguments, returning 3, and: {:(Int x) x * 2 }; // is a function that doubles a value. Obviously, we can tie functions to names: Auto dbl = {:(Int x) x * 2 }; // Meaning we can call dbl like this: dbl(4); // Auto is a magic word to say ("use type inference"), so // this is equivalent to the above: fn([Int]->Int) dbl = {:(Int x) x * 2 }; // Because {} makes an anon function, things like "for" can be // functions instead of keywords. for(range(3), {:(Int x) world.stdout.println(to(String)(x)); }); // As far as possible, Pepper3 will only contain assignment statements: String s = "xx"; // and expressions containing function calls and operators: dbl(3) + 6; // This means we can make our own constructs like a different type of // for loop, which would need a new keyword in some languages: Auto parallel_for = import(multiprocess.parallel_for); </pre>

Recording gameplay videos on RetroPie

Andy Balaam — Sun, 07 Jan 2018 18:21:42 +0000

Stop!</strong> This blog post is out of date - see Recording gameplay videos on RetroPie</a> for a live version in codeberg that may be more helpful.</p>
Credits: this is a slightly corrected and shortened version of How To Record A GamePlay Video From A RetroPie</a> by selsine, which is itself based on Recording Live Gameplay in RetroPieâ€™s RetroArch Emulators Natively on the Raspberry Pi</a> by Retro Resolution.</p>
RetroPie</a> is based on RetroArch</a>. RetroArch has a feature to record gameplay videos, but the current version of RetroPie has it disabled, presumably because it was thought to be too intensive to run properly on a Raspberry Pi.</p>
These instructions tell you how to turn the recording feature on, and set it up. This works perfectly on my Raspberry Pi 3, allowing me to record video and sound from games I am playing.</p>
The code for this is here: codeberg.org/andybalaam/retropie-recording</a> - this was code written by RetroRevolution, with small corrections and additions by me.</p>
Before you start, you should have RetroPie working and connected to the Internet, and updated to the latest version.</p>
Note: you should make a backup of your RetroPie before you start, because if you type the command below you could completely break it, meaning you will have to wipe your SD card and start fresh.</strong></p>
Turning on the recording feature</h2> RetroArch uses the ffmpeg</a> program to record video. To turn on recording, we need to log into the Pi using ssh, download and compile ffmpeg, and then recompile RetroArch with recording support turned on.</p>
Log in to the Pi using ssh</h3>
Find out the IP address of your Pi by choosing "RetroPie setup" in the RetroPie menu and choosing "Show IP Address". Write down the IP address (four numbers with dots in between - for example: 192.168.0.3</strong>).</p>
On your Linux* computer open a Terminal and type:</p>
ssh pi@192.168.0.3</pre>
(put in the IP address you wrote down instead of 192.168.0.3)</p>
When it asks for your password, type: raspberry</strong></p>
If this works right, you should see something like this:</p>
* Note: if you don't have Linux, this should work OK on a Mac, or on Windows you could try using PuTTY</a>.</p>
Download and compile ffmpeg</h3>
Log in to the RetroPie as described above. The commands shown below should all be typed in to the window where you are logged in to the RetroPie.</p>
Download the script ffmpeg-install.sh by typing this:</p>
wget https://codeberg.org/andybalaam/retropie-recording/raw/branch/master/ffmpeg-install.sh</pre>
Now run it like this:</p>
bash ffmpeg-install.sh</pre>
(Note: DON'T use sudo to run this - just type exactly what is written above.)</p>
Now wait a long time for this to work. If it prints out errors, something went wrong - read what it says, and you may need to edit the ffmpeg-install.sh script to figure out what to do. Leave a comment and include the errors you saw if you need help.</p>
Hopefully it will end successfully and print:</p>
FFmpeg and Codec Installation Complete</pre>
If so, you are ready to move on to recompiling RetroArch:</p>
Recompile RetroArch with recording turned on</h3>
Download the script build-retroarch-with-ffmpeg.sh by typing this:</p>
wget https://codeberg.org/andybalaam/retropie-recording/raw/branch/master/build-retroarch-with-ffmpeg.sh</pre>
Now run it like this:</p>
bash build-retroarch-with-ffmpeg.sh</pre>
It should finish in about 10 minutes, and print:</p>
Building RetroArch with ffmpeg enabled complete</pre>
If it printed that, your RetroPie now has recording support! Restart your RetroPie:</p>
Restart the RetroPie</h3>
Restart your RetroPie.</p>
If you want to check that recording support is enabled, Look for "Checking FFmpeg Has Been Enabled in RetroArch" on the RetroResolution guide</a>.</p>
Now you need to set up RetroPie to record your emulator.</p>
Setting up recording for your emulator</h2>
To set up an emulator, you need a general recording config file (the same for all emulators), and a launch config for the actual emulator you are using.</p>
Create the recording config file</h3>
Log into the RetroPie as described in the first section, and type this to download the recording config file. If you want to change settings like what file format to record in, this is the file you will need to change.</p>
wget https://codeberg.org/andybalaam/retropie-recording/blob/master/recording-config.cfg</pre>
Create a launch config for your emulator</h3>
Each RetroPie emulator has a config file that describes how to launch it. For example, the NES emulator's version is in /opt/retropie/configs/nes/emulators.cfg.</p>
To get a list of all the emulators, log into your RetroPie and type:</p>
ls /opt/retropie/configs</pre>
In that list you will see, for example, "nes" for the NES emulators, and "gb" for the GameBoy emulators. Find the one you want to edit, and edit it with the nano editor by typing:</p>
nano /opt/retropie/configs/gb/emulators.cfg</pre>
(Instead of "gb" type the right name for the emulator you want to use, from the list you got when you typed the "ls" command above.)</p>
Now you need to add a new line in this file. Each line describes how to launch an emulator. You should copy an existing line, and add some more stuff to the end.</p>
For example, my version of this file looks like this:</p>
lr-gambatte = "/opt/retropie/emulators/retroarch/bin/retroarch -L /opt/retropie/libretrocores/lr-gambatte/gambatte_libretro.so --config /opt/retropie/configs/gb/retroarch.cfg %ROM%" lr-gambatte</span>-record</span> = "/opt/retropie/emulators/retroarch/bin/retroarch -L /opt/retropie/libretrocores/lr-gambatte/gambatte_libretro.so --config /opt/retropie/configs/gb/retroarch.cfg</span> --record /home/pi/recording_GB_$(date +%Y-%m-%d-%H%M%S).mkv --recordconfig /home/pi/recording-config.cfg</span> %ROM%"</span> default = "lr-gambatte" lr-tgbdual = "/opt/retropie/emulators/retroarch/bin/retroarch -L /opt/retropie/libretrocores/lr-tgbdual/tgbdual_libretro.so --config /opt/retropie/configs/gb/retroarch.cfg %ROM%"</pre>
The line I added is coloured: The green parts are things copied from the line above, and the red parts are new - those parts tell the launcher to use the recording config we made in the previous section.</p>
When you've made your edits, press Ctrl-X to exit nano, and type "Y" when it asks whether you want to save.</p>
Once you've done something similar to this for every emulator you want to record with, you are ready to actually do the recording!</p>
Actually doing a recording</h2>
Launching a game with recording turned on</h3>
In the normal RetroPie interface, go to your emulator and start it, but press the A button while it's launching, and choose "Select emulator for ROM". In the list that comes up, choose the new line you added in emulators.cfg. In our example, that was called "lr-gambatte-record".</p>
Now play the game, and exit when you are finished. If all goes well, the recording will have been saved!</p>
(Note: doing this means that every time you launch this game it will be recorded. To stop it doing this, press the "A" button while it's launching, choose "Select emulator for ROM" and choose the normal line - in our example that would be "lr-gambatte".)</p>
Getting the recorded files</h3>
To get your recording off the RetroPie, go back to your computer, open a terminal, and type:</p>
scp pi@192.168.0.3:recording_*.mkv ./</pre>
This will copy all recorded videos from your RetroPie onto your computer (into your home directory, unless you did a cd commmand before you typed the above).</p>
Now you should delete the files from your RetroPie. Log in to the RetroPie as described in the first section, and delete all recording files by typing this:</p>
Note: This deletes all your recordings, and you can't undo!</strong></p>
rm recording_*.mkv</pre>
Note: This deletes all your recordings, and you can't undo!</strong></p>
Safer: recording onto a USB stick</h3>
Note</strong>: recording directly onto the RetroPie like we described above is dangerous because you could fill up all the disk space or corrupt your SD card, which could make RetroPie stop working, meaning you need to wipe your SD card and set up RetroPie again.</p>
It's safer to record onto a separate USB disk. To find out how, read "Recording to an External Storage Device" in Retro Resolution's guide</a>.</p>
Comments</h2>

Steve 2018-02-07</span> </h3>
Hi, thanks for putting this together:)</p>
Step 2 is failing for me though. There seems to be a problem with the line number reference in /retroarch.sh but I can’t figure it out.</p>
When I launch build-retroarch-with-ffmpeg.sh, the script works up to this point before failing:</p>

echo ‘=== Patching RetroArch code to enable ffmpeg ===’ === Patching RetroArch code to enable ffmpeg === </li>
cd RetroPie-Setup </li>
patch -p1 patching file scriptmodules/emulators/retroarch.sh Hunk #1 FAILED at 40. 1 out of 1 hunk FAILED — saving rejects to file scriptmodules/emulators/retroarch.sh.rej </li> </ul>
In my retroarch.sh file, "function build_retroarch" is actually on line 41. As a test I tried to bring everything up by one line but that didn’t work either.</p>
Any ideas?</p> </div>

Andy Balaam 2018-02-09</span> </h3>
Ah, sorry to hear this. I am guessing the retroarch source code has been updated so my patch file doesn’t apply any more. I will need to update it when I get time, or if someone else can do it they should send me a merge request on GitHub.</p> </div>

Steve 2018-02-12</span> </h3>
Hey, I did a bit more digging and went back to retroresolution’s original how to. After running sudo ./retropie_packages.sh retroarch (to rebuild retroarch after removing the disable-ffmpeg flag), the script fails with: "Could not successfully build retroarch – RetroArch – frontend to the libretro emulator cores – required by all lr-* emulators (/root/RetroPie-Setup/tmp/build/retroarch/retroarch not found)."</p>
Found that some other people had the same issue but no solution was found. I suspect that the issue isn’t with your script so I think I’ll just give it another whirl with a fresh installation. I’ll let you know how it goes :)</p> </div>

Andy Balaam 2018-02-12</span> </h3>
Good luck Steve, and thanks for the info. Let us know.</p> </div>

Chris 2018-02-26</span> </h3>
Have you played around with Twitch streaming any from your pi? I have everything working except for N64 and PSP streaming. For some reason the game launches but the stream doesn’t start like it does for all other emulators. I read somewhere that those emulators may use more than the single core the others use, so maybe that has something to do with it?</p> </div>

Andy Balaam 2018-02-27</span> </h3>
Hi Chris, no I haven’t tried that – I’m surprised it works at all!</p> </div>

Chris 2018-02-27</span> </h3>
It actually works fairly well. Itâ€™s a similar setup to getting gameplay recording working.</p>
I followed a guide on another blog post I found (http://selsine.com/2016/10/16/how-to-live-str...</a>)</p> </div>

Andy Balaam 2018-02-27</span> </h3>
Good to hear, thanks for the link!</p> </div>

partTimeNinja 2018-05-14</span> </h3>
I had issues running the ffmpeg-install.sh . I was getting "There are problems and -y was used without —force-yes"</p>
Eventually I realized that the original folder I was in when I did step #1 was owned but root so I had to use "sudo wget…" I changed ownership of ffmpeg-install.sh and it seems to be running correctly now.</p> </div>

partTimeNinja 2018-05-14</span> </h3>
Never mind, same error.</p> </div>

salvatore 2019-05-26</span> </h3>
and if I wanted to transmit the recording later on twitch what should I do?</p> </div>

Nicolas Baumgardt 2019-08-06</span> </h3>
Hi. I followed your instructions, but got the following errors right after start:</p>
After I ran "bash ffmpeg-install.sh" I get this error:</p>
pi@retropie:~ $ bash ffmpeg-install.sh + echo ‘Installing FFmpeg Suite’ Installing FFmpeg Suite</p>
+ echo ———————————-</p>

echo</p></li>
echo ‘=== Updating APT repo ===’ === Updating APT repo ===</p></li>
sudo apt-get update Hit:1 http://archive.raspberrypi.org/debian</a> stretch InRelease Get:2 http://raspbian.raspberrypi.org/raspbian</a> stretch InRelease [15.0 kB] Get:3 http://raspbian.raspberrypi.org/raspbian</a> stretch/main armhf Packages [11.7 MB] Err:3 http://raspbian.raspberrypi.org/raspbian</a> stretch/main armhf Packages Hash Sum mismatch Hashes of expected file:</p>

Filesize:11662764 [weak] </li>
SHA256:d01c7c7b4963cb29fe508f90104ff697f7f68b127e1546e88f7f880098a69d85 </li>
SHA1:82848ac64aa65d092e7066352ddc5bc99d168135 [weak] </li>
MD5Sum:c46cdb694218cf7834c00a1493e13f5e [weak] Hashes of received file: </li>
SHA256:3138f9acc731b771b152ceeee83e3e9769784cedfb368e2545472df6d0bb0b0a </li>
SHA1:1ed901f11928bd6f5b2a09118f055c70d2fa05ac [weak] </li>
MD5Sum:f5fcd94cb6aed37c8ac05bd253081fde [weak] </li>
Filesize:1551993 [weak] Last modification reported: Mon, 05 Aug 2019 23:19:49 +0000 Release file created at: Mon, 05 Aug 2019 23:28:15 +0000 Fetched 1567 kB in 41s (38.1 kB/s) Reading package lists… Done E: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dist...</a> Hash Sum mismatch Hashes of expected file: </li>
Filesize:11662764 [weak] </li>
SHA256:d01c7c7b4963cb29fe508f90104ff697f7f68b127e1546e88f7f880098a69d85 </li>
SHA1:82848ac64aa65d092e7066352ddc5bc99d168135 [weak] </li>
MD5Sum:c46cdb694218cf7834c00a1493e13f5e [weak] Hashes of received file: </li>
SHA256:3138f9acc731b771b152ceeee83e3e9769784cedfb368e2545472df6d0bb0b0a </li>
SHA1:1ed901f11928bd6f5b2a09118f055c70d2fa05ac [weak] </li>
MD5Sum:f5fcd94cb6aed37c8ac05bd253081fde [weak] </li>
Filesize:1551993 [weak] Last modification reported: Mon, 05 Aug 2019 23:19:49 +0000 Release file created at: Mon, 05 Aug 2019 23:28:15 +0000 E: Some index files failed to download. They have been ignored, or old ones used instead. </li> </ul></li> </ul>
Then when trying to execute the second script "bash build-retroarch-with-ffmpeg.sh", I get this error:</p>
pi@retropie:~ $ bash build-retroarch-with-ffmpeg.sh + echo ‘Building RetroArch with ffmpeg enabled’ Building RetroArch with ffmpeg enabled</p>
+ echo ———————————————————</p>

echo</p></li>
echo ‘=== Downloading patch file ===’ === Downloading patch file ===</p></li>
wget https://raw.githubusercontent.com/andybalaam/...</a> —2019-08-06 14:52:06— https://raw.githubusercontent.com/andybalaam/...</a> Resolving raw.githubusercontent.com (raw.githubusercontent.com)… 151.101.240.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.240.133|:443… connected. HTTP request sent, awaiting response… 200 OK Length: 690 [text/plain] Saving to: ‘enable-ffmpeg.patch’</p></li> </ul>
enable-ffmpeg.patch 100%[===================================================================================================================>] 690 —.-KB/s in 0s</p>
2019-08-06 14:52:07 (3.95 MB/s) – ‘enable-ffmpeg.patch’ saved [^{690</sup>⁄_{690</sub>]</p>}}

echo ‘=== Patching RetroArch code to enable ffmpeg ===’ === Patching RetroArch code to enable ffmpeg === </li>
cd RetroPie-Setup </li>
patch -p1 patching file scriptmodules/emulators/retroarch.sh Hunk #1 FAILED at 40. 1 out of 1 hunk FAILED — saving rejects to file scriptmodules/emulators/retroarch.sh.rej </li> </ul>
Any help highly appreciated. Thank you!</p> </div>

Andy Balaam 2019-08-07</span> </h3>
Hi Nicolas, I’m afraid it looks like my script has gone out of date and I’m afraid I don’t know how to fix it, sorry.</p> </div>

Nicolas Baumgardt 2019-08-07</span> </h3>
Never mind, thanks for your response! Everything has its lifetime, scripts are no exception ;–)</p> </div>

Liam 2020-07-08</span> </h3>
i tried to run ffmpeg-install.sh, but halfway through it said "+ git clone git://git.videolan.org/x264 Cloning into ‘x264’… fatal: remote error: access denied or repository not exported: /x264 ". what do i do?</p> </div>

Andy Balaam 2020-07-09</span> </h3>
Hi Liam, I don’t know. It would be worth trying again since it might have been a momentary problem. However, I think this code is out-of-date now, so if you want to follow this you will need to understand and update it to suit the way things have changed in the meantime.</p> </div>

Yavuz Selim 2020-12-19</span> </h3>
Andy:</p>
Change the following line: git clone git://git.videolan.org/x264</p>
To: git clone https://code.videolan.org/videolan/x264.git</a></p> </div>

Andy Balaam 2020-12-22</span> </h3>
Thanks Yavuz Selim – fix applied in the repo!</p> </div>

Digital404 2024-12-26</span> </h3>
I tried to bash the build-retroarch-with-ffmpeg.sh and this is the error I get at the end.</p> </div>

Digital404 2024-12-26</span> </h3>
Hi there!</p>
I am new here I am trying to add recording to my emulators like what I see on here. I got passed everything beside "Bash build-retroarch-with-ffmpeg.sh" It giving me error message.</p>
bash build-retroarch-with-ffmpeg.sh + echo ‘Building RetroArch with ffmpeg enabled’ Building RetroArch with ffmpeg enabled</p>
+ echo ———————————————————</p>

echo</p></li>
echo ‘=== Downloading patch file ===’ === Downloading patch file ===</p></li>
wget https://raw.githubusercontent.com/andybalaam/...</a> able-ffmpeg.patch —2024-12-26 19:28:57— https://raw.githubusercontent.com/andybalaam/...</a> ecording/master/enable-ffmpeg.patch Resolving raw.githubusercontent.com (raw.githubusercontent.com)… 185.199.108.1 33, 185.199.109.133, 185.199.110.133, … Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108. 133|:443… connected. HTTP request sent, awaiting response… 200 OK Length: 690 [text/plain] Saving to: ‘enable-ffmpeg.patch.2’</p></li> </ul>
enable-ffmpeg.patch 100%[===================>] 690 —.-KB/s in 0.002s</p>
2024-12-26 19:29:03 (416 KB/s) – ‘enable-ffmpeg.patch.2’ saved [^{690</sup>⁄_{690</sub>]</p>}}

echo ‘=== Patching RetroArch code to enable ffmpeg ===’ === Patching RetroArch code to enable ffmpeg === </li>
cd RetroPie-Setup </li>
patch -p1 patching file scriptmodules/emulators/retroarch.sh Hunk #1 FAILED at 40. 1 out of 1 hunk FAILED — saving rejects to file scriptmodules/emulators/retroar ch.sh.rej </li> </ul>
Can someone help me.</p> </div>

TECH(K)NOW Day workshop on "Writing a programming language"

Andy Balaam — Tue, 10 Oct 2017 15:38:10 +0000

My OpenMarket</a> colleagues and I ran a workshop at TECH(K)NOW Day</a> on how to write your own programming language:</p>
</p>
A big thank you to my colleagues from OpenMarket who volunteered to help: Rowan, Jenny, Zach, James and Elliot.</p>
</p>
An extra thank you to Zach and Elliott for their impromptu help on the information desk for attendees:</p>
</p>
Hopefully the attendees enjoyed it and learned a bit:</p>

Making own programming language @TECHKNOWDay</a> pic.twitter.com/jfUFHq7iO3</a></p>
— 4theLoveOfCode (@4theLoveOfCode) October 10, 2017</a></p> </blockquote>

It was an amazing workshop thank you @andybalaam</a></p>
— 4theLoveOfCode (@4theLoveOfCode) October 10, 2017</a></p> </blockquote>
You can find the workshop slides, the full code, info about another simple language called Cell, and lots more links here: github.com/andybalaam/videos-write-your-own-language</a>, my blog at artificialworlds.net/blog</a>, and follow me on twitter @andybalaam</a>.</p>
Thanks to OpenMarket for supporting us in running this workshop!</p>

At @OpenMarket</a> we are honoured to support the excellent work of @WomenWhoCode</a>â€™s @TECHKNOWDay</a> and celebrate #AdaLovelaceDay</a> & #womenintech</a> https://t.co/udYXTLjOqL</a></p>
— OpenMarket (@openmarket) October 10, 2017</a></p> </blockquote>

FileZilla with proxy not working with strange characters in the password

Andy Balaam — Fri, 29 Sep 2017 12:35:37 +0000

Today I found I could not connect out through my proxy server with FileZilla</a></p>
It stopped working when I changed my password to something containing a double quote ".</p>
The solution? Change to use WinSCP</a>.</p>

HTML5 CSS Toolbar + zoomable workspace that is mobile-friendly and adaptive

Andy Balaam — Wed, 20 Sep 2017 23:42:11 +0000

I have been working on a prototype level editor</a> for Rabbit Escape</a>, and I've had trouble getting the layout I wanted: a toolbar at the side or top of the screen, and the rest a zoomable workspace.</p>
Something like this is very common in many desktop applications, but not that easy to achieve in a web page, especially because we want to take care that it adapts to different screen sizes and orientations, and, for example, allows zooming the toolbar buttons in case we find ourselves on a device with different resolution from what we were expecting.</p>
In the end I've gone with a grid-layout</a> solution and accepted the fact that sometimes on mobile devices when I zoom in my toolbar will disappear off the top/side. When I scroll back to it, it stays around, so using this setup is quite natural. On the desktop, it works how you'd expect, with the toolbar staying on screen at all zoom levels.</p>
Here's how it looks on a landscape display:</p> </iframe>
and portrait:</p> </iframe>
Read the full source code</a>.</p>
As you can see from the code linked above, after much fiddling I managed to achieve this with a relatively small amount of CSS, and no JavaScript. I'm hoping it will behave well in unexpected scenarios, because the code expresses what I want fairly closely.</p>
The important bits of the HTML are simple - a main div, a toolbar containing buttons, and a workspace containing some kind of work:</p>
<div id="main"> <div id="toolbar"> <button></button><button></button><button></button><button></button><button></button><button></button><button></button><button></button> </div> <div id="workspace"> <div id="work"> </div> </div> </div> </pre>
The keys bits of the CSS are:</p>
/* Ensure we take up the full height of the page. */ html, body, #main { height: 100%; } @media all and (orientation:landscape) { /* On a wide screen, it's a grid with 2 columns, and the toolbar can scroll downwards. */ #main { display: grid; grid-template-columns: 5em 1fr; } #toolbar { overflow-x: hidden; overflow-y: auto; } } @media all and (orientation:portrait) { /* On a tall screen, it's a grid with 2 rows, and the toolbar can scroll right. */ #main { display: grid; grid-template-rows: 5em 1fr; } #toolbar { overflow-x: auto; overflow-y: hidden; white-space: nowrap; } } </pre>
That replaces an awful lot of code in my first attempt, so I'm reasonably happy. If anyone has suggestions about how to make "100%" really mean 100% of the real device width and height, let me know. If I do some JavaScript I can make Mobile Firefox fit to the real screen size, but Mobile Chrome (and, I assume, Mobile Safari) lie to me about the screen size when zoomed in.</p>

maven-assembly-plugin descriptor for a simple tarball with dependencies

Andy Balaam — Thu, 14 Sep 2017 13:06:36 +0000

Today I was trying to make a simple tarball of a project + its dependent jar using the maven-assembly-plugin. I know this is a terrible way to do anything, but hey, just in case someone else wants to do something just as terrible, here are my pom.xml and assembly.xml (the assembly descriptor):</p>
$ tree . â”œâ”€â”€ assembly.xml â”œâ”€â”€ pom.xml â””â”€â”€ src â””â”€â”€ main â””â”€â”€ java â””â”€â”€ AssemblyExample.java</pre>
$ cat pom.xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd" > <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>myproject</artifactId> <name>My Projects</name> <version>0.1</version> <build><plugins><plugin> <artifactId>maven-assembly-plugin</artifactId> <version>2.2.1</version> <configuration><descriptors> <descriptor>assembly.xml</descriptor> </descriptors></configuration> <executions> <execution> <phase>package</phase> <goals><goal>attached</goal></goals> </execution></executions> </plugin></plugins></build> <dependencies> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <version>1.7.2</version> </dependency> </dependencies> </project> </pre>
$ cat assembly.xml <?xml version="1.0" encoding="UTF-8"?> <assembly> <id>ap3</id> <formats> <format>tar.gz</format> </formats> <includeBaseDirectory>true</includeBaseDirectory> <dependencySets> <dependencySet> <outputDirectory>jars</outputDirectory> <scope>runtime</scope> </dependencySet> </dependencySets> </assembly></pre>
$ mvn package ... [INFO] Compiling 1 source file to /home/andrebal/Desktop/assemblyexample/target/classes [INFO] ... [INFO] Building jar: /home/andrebal/Desktop/assemblyexample/target/myproject-0.1.jar ... [INFO] --- maven-assembly-plugin:2.2.1:attached (default) @ myproject --- [INFO] Reading assembly descriptor: assembly.xml [INFO] Building tar : /home/andrebal/Desktop/assemblyexample/target/myproject-0.1-ap3.tar.gz ... [INFO] BUILD SUCCESS</pre>
$ tar -tzf target/myproject-0.1-ap3.tar.gz myproject-0.1/jars/slf4j-log4j12-1.7.2.jar myproject-0.1/jars/slf4j-api-1.7.2.jar myproject-0.1/jars/log4j-1.2.17.jar myproject-0.1/jars/myproject-0.1.jar</pre>

Blog aggregator/planet in WordPress using FeedWordPress

Andy Balaam — Tue, 12 Sep 2017 13:55:34 +0000

I used to run Planet Code</a> using Venus</a> but that started breaking recently with 500 errors for SNI-enabled sites</a>, and it looks very unmaintained.</p>
I considered some other static aggregators, but on shared hosting it's hard to get the right Ruby version and gems installed and the like, so I went for another WordPress</a>, and found the very nice FeedWordPress</a> plugin to suck up all the feeds of coders somewhat related to ACCU</a> and/or coding generally.</p>
Let me know what you think.</p>

Adding a day in Python datetimes - use timedelta, not the datetime constructor

Andy Balaam — Mon, 07 Aug 2017 08:30:38 +0000

If you want "tomorrow" in Python datetimes, don't construct a datetime like this:</p>
from datetime import datetime, timedelta td = datetime.today() tm1 = datetime(td.year, td.month, td.day + 1</span>, 14, 0, 0) # Don't do this!</span> </pre>
Because it will work sometimes, but fail when today is the last day of the month:</p>
Traceback (most recent call last): File "./tomorrow", line 6, in tm1 = datetime(td.year, td.month, td.day + 1, 14, 0, 0) ValueError: day is out of range for month</module></pre>
Instead, use Python's timedelta, which is designed for this purpose:</p>
from datetime import datetime, timedelta td = datetime.today() tm2 = td + timedelta(days=1) print("tm2=%s" % str(tm2))</pre>
And it's easier to read too.</p>

Broken Levels Challenge - Egham Raspberry Pi Jam July 2017

Andy Balaam — Sun, 23 Jul 2017 16:42:58 +0000

Today at the Egham Raspberry Pi Jam</a> we did two things:</p>
1. The Broken Levels Challenge</h3>
Some nasty person came and broke our levels for our game Rabbit Escape</a> and we need you to fix them!</p>
To play this game you will need a PC version of Rabbit Escape</a>, our Broken Levels</a>, and the instruction sheets</a>. Let us know how you get on!</p>
2. Python Traffic Lights Programming Workshop</h3>
I ran a workshop to learn a bit of Python programming using this resource sheet Pi Stop Traffic Lights</a>.</p>
We had a lot of fun, and hopefully some people even learnt a little bit of coding.</p>

A lot of concentration. Python, GPIOZero. @4tronix_uk</a> pi-stop. @rjam_chat</a>
Thank you @andybalaam</a> pic.twitter.com/0cWAvSL6M8</a></p>
— Egham Raspberry Jam (@EghamJam) July 23, 2017</a></p> </blockquote>

Women Who Code workshop on "Write your own programming language"

Andy Balaam — Thu, 29 Jun 2017 09:20:21 +0000

On Wednesday 28th June 2017 a group of people from OpenMarket went to the Fora office space in Clerkenwell, London to run a workshop with the Women Who Code group, who work to help women achieve their career goals.</p>
OpenMarket provided the workshop "Write your own programming language" and funded the food, and the venue was provided gratis by Fora.</p>
We started the evening with some networking and food:</p>
but most of the time was spent coding:</p>
with lots of help from our OpenMarket helpers:</p>
The feedback we got was very positive:</p>

Thanks @andybalaam</a> for that great Write Your Own Programming Language workshop! & thanks @openmarket</a> and @fora_space</a> for sponsoring/hosting! pic.twitter.com/6U0DClAHMD</a></p>
— Women Who Code LDN (@WWCLondon) June 28, 2017</a></p> </blockquote>

"It's all about the cool" Thx @andybalaam</a>
Great workshop @coderinheels</a> pic.twitter.com/El3bAeRtK4</a></p>
— Sandrine Pataut (@SPataut) June 28, 2017</a></p> </blockquote>
Everyone seemed to be having fun, so we hope we might get invited back to do more in future.</p>
Why do this?</h2>
At OpenMarket we want to improve our diversity, and we have started by looking at gender diversity specifically. By being involved with events like this we hope to learn how we can make our company better at welcoming and supporting employees, encourage people from under-represented groups to apply to work here, and improve the general climate in our industry.</p>
Thank you</h2>
A huge thank you to the OpenMarket people (from London and Guadalajara!) who helped out - I think people felt welcome and there was plenty of help available for the attendees - you did a great job.</p>
Thank you also for the great response from everyone in our London office - several people in the office wanted to come but couldn't make it on the night - I am hoping we will get more opportunities in future.</p>
We're also really grateful to OpenMarket for funding the food, to Fora for providing the space, and to Women Who Code for doing such great work to improve our industry.</p>
Links</h2>

OpenMarket is hiring</a></li>
Fora</a></li>
Women Who Code LDN</a></li>
Write your own programming language</a> (workshop slides)</li>
datecalc</a> (my version of the language we wrote)</li> </ul>
[Photos by David Lawson.]</p>

Running a virtualenv with a custom-built Python

Andy Balaam — Tue, 27 Jun 2017 00:31:06 +0000

For my attempt</a> to improve the asyncio.as_completed Python standard library function I needed to build a local copy of cpython</a> (the Python interpreter).</p>
To test it, I needed the aiohttp</a> module, which is not part of the standard library, so the easiest way to get it was using virtualenv</a>.</p>
Here is the recipe I used to get a virtualenv and install packages using pip with a custom-built Python:</p>
$ ~/code/public/cpython/python -m venv env $ . env/bin/activate (env) $ pip install aiohttp (env) $ python mycode.py</pre>

Adding a concurrency limit to Python's asyncio.as_completed

Andy Balaam — Tue, 27 Jun 2017 00:16:09 +0000

Series: asyncio basics</a>, large numbers in parallel</a>, parallel HTTP requests</a>, adding to stdlib</p>
In the previous post</a> I demonstrated how the limited_as_completed</a> method allows us to run a very large number of tasks using concurrency, but limiting the number of concurrent tasks to a sensible limit to ensure we don't exhaust resources like memory or operating system file handles.</p>
I think this could be a useful addition to the Python standard library, so I have been working on a modification to the current asyncio.as_completed method. My work so far is here: limited-as_completed</a>.</p>
I ran similar tests to the ones I ran for the last blog post with this code to validate that the modified standard library version achieves the same goals as before.</p>
I used an identical copy of timed from the previous post</a> and updated versions of the other files because I was using a much newer version of aiohttp along with the custom-built python I was running.</p>
server looked like:</p>
#!/usr/bin/env python3 from aiohttp import web import asyncio import random async def handle(request): await asyncio.sleep(random.randint(0, 3)) return web.Response(text="Hello, World!") app = web.Application() app.router.add_get('/{name}', handle) web.run_app(app)</pre>
client-async-sem needed me to add a custom TCPConnector to avoid a new limit on the number of concurrent connections that was added to aiohttp in version 2.0. I also need to move the ClientSession usage inside a coroutine to avoid a warning:</p>
#!/usr/bin/env python3 from aiohttp import ClientSession, TCPConnector import asyncio import sys limit = 1000 async def fetch(url, session): async with session.get(url) as response: return await response.read() async def bound_fetch(sem, url, session): # Getter function with semaphore. async with sem: await fetch(url, session) async def run(r): with ClientSession(connector=TCPConnector(limit=limit)) as session: url = "http://localhost:8080/{}" tasks = [] # create instance of Semaphore sem = asyncio.Semaphore(limit) for i in range(r): # pass Semaphore and session to every GET request task = asyncio.ensure_future( bound_fetch(sem, url.format(i), session)) tasks.append(task) responses = asyncio.gather(*tasks) await responses loop = asyncio.get_event_loop() loop.run_until_complete(asyncio.ensure_future(run(int(sys.argv[1]))))</pre>
My new code that uses my proposed extension to as_completed looked like:</p>
#!/usr/bin/env python3 from aiohttp import ClientSession, TCPConnector import asyncio import sys async def fetch(url, session): async with session.get(url) as response: return await response.read() limit = 1000 async def print_when_done(): with ClientSession(connector=TCPConnector(limit=limit)) as session: tasks = (fetch(url.format(i), session) for i in range(r)) for res in asyncio.as_completed(tasks, limit=limit): await res r = int(sys.argv[1]) url = "http://localhost:8080/{}" loop = asyncio.get_event_loop() loop.run_until_complete(print_when_done()) loop.close()</pre>
and with these, we get similar behaviour to the previous post:</p>
$ ./timed ./client-async-sem 10000 Memory usage: 73640KB Time: 19.18 seconds $ ./timed ./client-async-stdlib 10000 Memory usage: 49332KB Time: 18.97 seconds</pre>
So the implementation</a> I plan to submit to the Python standard library appears to work well. In fact, I think it is better than the one I presented in the previous post, because it uses on_complete callbacks to notice when futures have completed, which reduces the busy-looping we were doing to check for and yield finished tasks.</p>
The Python issue is bpo-30782</a> and the pull request is #2424</a>.</p>
Note: at first glance, it looks like the aiohttp.ClientSession's limit on the number of connections</a> (introduced in version 1.0 and then updated in version 2.0) gives us what we want without any of this extra code, but in fact it only limits the number of connections, not the number of futures we are creating, so it has the same problem of unbounded memory use as the semaphore-based implementation.</p>
Comments</h2>

Alexander Mohr 2017-07-28</span> </h3>
I created this to solve this problem more generally: https://gist.github.com/thehesiod/7081ab165b9...</a></p> </div>

Andy Balaam 2017-07-29</span> </h3>
Hi Alexander, that is cool. In what way is it more general?</p> </div>

Reto Kaiser 2017-10-13</span> </h3>
Alexander’s script has the advantage that it allows to add coroutines after the loop has been started. Can be useful if one async operations should schedule other operations recursively. Here’s another solution: https://gist.github.com/njam/e19a497185a9f657...</a></p>
btw I guess for most usecases setting aiohttp’s connection limit should be sufficient: http://aiohttp.readthedocs.io/en/stable/clien...</a></p> </div>

Andy Balaam 2017-10-22</span> </h3>
Thanks Reto, good points.</p> </div>

Andrey Paramonov 2019-02-20</span> </h3>
Thanks for the inspiration! I’m currently using the following generalization of asyncio.gather</code> for similar problem:</p>
async</span> def</span> igather</span>(</span>tasks</span>,</span> limit</span>=</span>None</span>)</span>:</span> </span></span> pending</span> =</span> set</span>(</span>)</span> </span></span> while</span> True</span>:</span> </span></span> for</span> task</span> in</span> islice</span>(</span>tasks</span>,</span> limit</span> -</span> len</span>(</span>pending</span>)</span> if</span> limit</span> else</span> None</span>)</span>:</span> </span></span> pending</span>.</span>add</span>(</span>task</span>)</span> </span></span> if</span> pending</span>:</span> </span></span> done</span>,</span> pending</span> =</span> await</span> asyncio</span>.</span>wait</span>(</span>pending</span>,</span> return_when</span>=</span>asyncio</span>.</span>FIRST_COMPLETED</span>)</span> </span></span> for</span> task</span> in</span> done</span>:</span> </span></span> yield</span> task</span> </span></span> else</span>:</span> </span></span> break</span> </span></span></code></pre> </div>

Andrey Paramonov 2019-02-20</span> </h3>
Thanks for the inspiration!</p>
I’m currently using the following generalization of asyncio.gather</code> for similar problem:</p>
async def igather(tasks, limit=None): pending = set() while True: for task in islice(tasks, limit – len(pending) if limit else None): pending.add(task) if pending: done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED) for task in done: yield task else: break</p> </div>
Andy Balaam 2019-02-21</span> </h3> Nice, thanks Andrey!</p> </div> Sergey 2019-08-22</span> </h3> Maybe the "meant to be" way to limit number of workers is using queues: https://docs.python.org/3/library/asyncio-que...</a></p> </div> Andy Balaam 2019-08-22</span> </h3> Thanks Sergey – yes, maybe we could write some kind of limited-size task pool that pull from a queue.</p> </div> Raaga J 2020-06-02</span> </h3> What is the difference between a semaphore limit and TCP connector limit ?Can u please explain in detail with an example. I am a beginner in Asyncio. thanks</p> </div>
"git what" is "git status" on steroids Andy Balaam — Fri, 23 Jun 2017 01:40:41 +0000 For when git status is not enough, I wrote git what</a>:</p> If you often have a few branches on the go, it could be useful.</p> Making 100 million requests with Python aiohttp Andy Balaam — Mon, 12 Jun 2017 09:41:51 +0000 Series: asyncio basics</a>, large numbers in parallel</a>, parallel HTTP requests, adding to stdlib</a></p> Update</strong>: slides of a talk I gave at the London Python Meetup on this: Talk slides: Making 100 million HTTP requests with Python aiohttp</a>.</p> Update</strong>: see how Cristian Garcia improved on this code here: Making an Unlimited Number of Requests with Python aiohttp + pypeln</a>.</p> I've been working on how to make a very large number of HTTP requests using Python's asyncio</a> and aiohttp</a>.</p> PaweÅ‚ Miech's post Making 1 million requests with python-aiohttp</a> taught me how to think about this, and got us a long way, with 1 million requests running in a reasonable time, but I need to go further.</p> PaweÅ‚'s approach limits the number of requests that are in progress, but it uses an unbounded amount of memory to hold the futures that it wants to execute.</p> See also</strong>: 2 excellent related posts by Quentin Pradet: How do you rate limit calls with asyncio?</a>, How do you limit memory usage with asyncio?</a>.</p> We can avoid using unbounded memory by using the limited_as_completed function I outined in my previous post</a>.</p> Setup</h3> Server</h4> We have a server program "server":</p> (Note it differs from PaweÅ‚'s version because I am using an older version of aiohttp which has fewer convenient features.)</p> #!/usr/bin/env python3.5 from aiohttp import web import asyncio import random async def handle(request): await asyncio.sleep(random.randint(0, 3)) return web.Response(text="Hello, World!") async def init(): app = web.Application() app.router.add_route('GET', '/{name}', handle) return await loop.create_server( app.make_handler(), '127.0.0.1', 8080) loop = asyncio.get_event_loop() loop.run_until_complete(init()) loop.run_forever()</pre> This just responds "Hello, World!" to every request it receives, but after an artificial delay of 0-3 seconds.</p> Synchronous client</h4> As a baseline, we have a synchronous client "client-sync":</p> #!/usr/bin/env python3.5 import requests import sys url = "http://localhost:8080/{}" for i in range(int(sys.argv[1])): requests.get(url.format(i)).text</pre> This waits for each request to complete before making the next one. Like the other clients below, it takes the number of requests to make as a command-line argument.</p> Async client using semaphores</h4> Copied mostly verbatim from Making 1 million requests with python-aiohttp</a> we have an async client "client-async-sem" that uses a semaphore to restrict the number of requests that are in progress at any time to 1000:</p> #!/usr/bin/env python3.5 from aiohttp import ClientSession import asyncio import sys limit = 1000 async def fetch(url, session): async with session.get(url) as response: return await response.read() async def bound_fetch(sem, url, session): # Getter function with semaphore. async with sem: await fetch(url, session) async def run(session, r): url = "http://localhost:8080/{}" tasks = [] # create instance of Semaphore sem = asyncio.Semaphore(limit) for i in range(r): # pass Semaphore and session to every GET request task = asyncio.ensure_future(bound_fetch(sem, url.format(i), session)) tasks.append(task) responses = asyncio.gather(*tasks) await responses loop = asyncio.get_event_loop() with ClientSession() as session: loop.run_until_complete(asyncio.ensure_future(run(session, int(sys.argv[1]))))</pre> Async client using limited_as_completed</h4> The new client I am presenting here uses limited_as_completed from the previous post</a>. This means it can make a generator that provides the futures to wait for as they are needed, instead of making them all at the beginning.</p> It is called "client-async-as-completed":</p> #!/usr/bin/env python3.5 from aiohttp import ClientSession import asyncio from itertools import islice import sys def limited_as_completed(coros, limit): futures = [ asyncio.ensure_future(c) for c in islice(coros, 0, limit) ] async def first_to_finish(): while True: await asyncio.sleep(0) for f in futures: if f.done(): futures.remove(f) try: newf = next(coros) futures.append( asyncio.ensure_future(newf)) except StopIteration as e: pass return f.result() while len(futures) > 0: yield first_to_finish() async def fetch(url, session): async with session.get(url) as response: return await response.read() limit = 1000 async def print_when_done(tasks): for res in limited_as_completed(tasks, limit): await res r = int(sys.argv[1]) url = "http://localhost:8080/{}" loop = asyncio.get_event_loop() with ClientSession() as session: coros = (fetch(url.format(i), session) for i in range(r)) loop.run_until_complete(print_when_done(coros)) loop.close()</pre> Again, this limits the number of requests to 1000.</p> Test setup</h4> Finally, we have a test runner script called "timed":</p> #!/usr/bin/env bash ./server & sleep 1 # Wait for server to start /usr/bin/time --format "Memory usage: %MKB\tTime: %e seconds" "$@" # %e Elapsed real (wall clock) time used by the process, in seconds. # %M Maximum resident set size of the process in Kilobytes. kill %1</pre> This runs each process, ensuring the server is restarted each time it runs, and prints out how long it took to run, and how much memory it used.</p> Results</h3> When making only 10 requests, the async clients worked faster because they launched all the requests simultaneously and only had to wait for the longest one (3 seconds). The memory usage of all three clients was fine:</p> $ ./timed ./client-sync 10 Memory usage: 20548KB Time: 15.16 seconds $ ./timed ./client-async-sem 10 Memory usage: 24996KB Time: 3.13 seconds $ ./timed ./client-async-as-completed 10 Memory usage: 23176KB Time: 3.13 seconds</pre> When making 100 requests, the synchronous client was very slow, but all three clients worked eventually:</p> $ ./timed ./client-sync 100 Memory usage: 20528KB Time: 156.63 seconds $ ./timed ./client-async-sem 100 Memory usage: 24980KB Time: 3.21 seconds $ ./timed ./client-async-as-completed 100 Memory usage: 24904KB Time: 3.21 seconds</pre> At this point let's agree that life is too short to wait for the synchronous client.</p> When making 10000 requests, both async clients worked quite quickly, and both had increased memory usage, but the semaphore-based one used almost twice as much memory as the limited_as_completed version:</p> $ ./timed ./client-async-sem 10000 Memory usage: 77912KB Time: 18.10 seconds $ ./timed ./client-async-as-completed 10000 Memory usage: 46780KB Time: 17.86 seconds</pre> For 1 million requests, the semaphore-based client took 25 minutes on my (32GB RAM) machine. It only used about 10% of my CPU, and it used a lot of memory (over 3GB):</p> $ ./timed ./client-async-sem 1000000 Memory usage: 3815076KB Time: 1544.04 seconds</pre> Note: PaweÅ‚'s version only took 9 minutes on his laptop and used all his CPU, so I wonder whether I have made a mistake somewhere, or whether my version of Python (3.5.2) is not as good as a later one.</p> The limited_as_completed version ran in a similar amount of time but used 100% of my CPU, and used a much smaller amount of memory (162MB):</p> $ ./timed ./client-async-as-completed 1000000 Memory usage: 162168KB Time: 1505.75 seconds</pre> Now let's try 100 million requests. The semaphore-based version lasted 10 hours before it was killed by Linux's OOM Killer</a>, but it didn't manage to make any requests in this time, because it creates all its futures before it starts making requests:</p> $ ./timed ./client-async-sem 100000000 Command terminated by signal 9</pre> I left the limited_as_completed version over the weekend and it managed to succeed eventually:</p> $ ./timed ./client-async-as-completed 100000000 Memory usage: 294304KB Time: 150213.15 seconds</pre> So its memory usage was still very bounded, and it managed to do about 665 requests/second over an extended period, which is almost identical to the throughput of the previous cases.</p> Conclusion</h3> Making a million requests is usually enough, but when we really need to do a lot of work while keeping our memory usage bounded, it looks like an approach like limited_as_completed is a good way to go. I also think it's slightly easier to understand.</p> In the next post</a> I describe my attempt to get something like this added to the Python standard library.</p> Comments</h2> Jack Brown 2017-08-18</span> </h3> I am using this to scrape data from a website however the website will place a temp block on IP addresses if it detects excessive use. Is there a way I can place a delay between requests? I am assuming I can use the time.sleep(seconds) function but unsure of the best place to insert within script.</p> </div> Andy Balaam 2017-08-21</span> </h3> Jack, I think you could put a sleep inside the "async with" block in fetch(), but I’d seriously suggest you avoid spamming a web site against its terms and conditions! Maybe contact the site about getting hold of their data another way?</p> </div> Nam Vu 2017-10-02</span> </h3> How can we share cookies for all of 100 M requests?</p> </div> Andy Balaam 2017-10-02</span> </h3> Hi Nam, since many of the requests are happening in parallel, sharing cookies may not work the way you need, but apart from that caveat, you should be able to provide cookies the way you normally do it using the requests library.</p> </div> ruslan 2018-08-12</span> </h3> That is a nice experiment. Andy, the fact that your client-async-as-completed uses 100% CPU is not a good thing. It indicates a problem with using busy-waiting. Your "while True" loop continuously checks for task completion, and that causes 100% CPU utilization. A better approach is to use</p> while futures: done, futures = await asyncio.wait(futures, return_when=asyncio.FIRST_COMPLETED) # then re-fill the set of futures from the coros iterable</p> asyncio.wait does not do busy-waiting</p> </div> Andy Balaam 2018-08-13</span> </h3> Awesome, thanks ruslan!</p> </div> Shih-Wen Su 2018-08-31</span> </h3> This is a great post! I learned a lot from it. I’m also thinking about how to use aiohttp for a similar task. Inspired by this post, I’m using only the semaphore to control the concurrent requests without exhausting CPU or memory. The only thing I do differently is to acquire the semaphore before creating the future task and release the semaphore after fetching the result. Code: https://github.com/flyakite/100-million-reque...</a></p> </div> Andy Balaam 2018-09-04</span> </h3> Thanks Shih-Wen Su – looks great!</p> </div> Cristian Garcia 2018-09-19</span> </h3> Hi! I created this library called pypeln</code> – https://github.com/cgarciae/pypeln</a> for creating many kinds of concurrent data pipelines. It currently supports Processes, Threads and asyncio Tasks.</p> Your post was an inspiration when implementing the io module!</p> With pypeln you can easily solve the problem you showed like this:</p> from aiohttp import ClientSession </span></span>from pypeln import io </span></span>import sys </span></span> </span></span>r = int(sys.argv[1]) </span></span>url = "http://localhost:8080/{}" </span></span> </span></span>with ClientSession() as session: </span></span> data = range(r) </span></span> io.each(lambda i: fetch(url.format(i), session), data, workers=1000) </span></span></code></pre> Thanks for sharing you knowledge!</p> </div> Andy Balaam 2018-09-20</span> </h3> Wow Cristian that looks brilliant!</p> </div> Andy Balaam 2018-09-20</span> </h3> I was planning to write a follow-up using pypeln, but I got this:</p> cat client-pypeln #!/usr/bin/env python3.6 from aiohttp import ClientSession from pypeln import io import sys r = int(sys.argv[1]) url = "http://localhost:8080/{}" with ClientSession() as session: data = range(r) io.each(lambda i: fetch(url.format(i), session), data, workers=1000) </pre> $ ./client-pypeln 10 ./client-pypeln:10: UserWarning: Creating a client session outside of coroutine is a very dangerous idea with ClientSession() as session: Creating a client session outside of coroutine client_session: Traceback (most recent call last): File "./client-pypeln", line 10, in with ClientSession() as session: File "/usr/lib/python3/dist-packages/aiohttp/client.py", line 742, in __enter__ raise TypeError("Use async with instead") TypeError: Use async with instead Unclosed client session client_session: </pre> Can you give a more complete example?</p> </div> Cristian Garcia 2018-09-20</span> </h3> Hey Andy,</p> Glad you liked it!</p> The error is due to changes in the iohttp library, it seems that ClientSession now has to be run with "async with" instead of "with". Your original code should no longer work.</p> Here is a full working example:</p> from aiohttp import ClientSession </span></span>from pypeln import io </span></span>import asyncio </span></span>import sys </span></span> </span></span>async def fetch(url, session): </span></span> async with session.get(url) as response: </span></span> return await response.read() </span></span> </span></span>async def main(): </span></span> r = 10 </span></span> url = "http://google.com" </span></span> </span></span> # r = int(sys.argv[1]) </span></span> # url = "http://localhost:8080/{}" </span></span> </span></span> async with ClientSession() as session: </span></span> data = range(r) </span></span> await io.each(lambda i: fetch(url, session), data, workers=1000, run = False) </span></span> </span></span> </span></span> </span></span>loop = asyncio.get_event_loop() </span></span>loop.run_until_complete(main()) </span></span> </span></span>print("Finish") </span></span></code></pre> </div> Cristian Garcia 2018-09-20</span> </h3> BTW: how do you make a code block on the comments?</p> </div> Cristian Garcia 2018-09-20</span> </h3> Here is a gist for the code: https://gist.github.com/cgarciae/4d35bb9f79c6...</a></p> </div> Andy Balaam 2018-09-20</span> </h3> You can add "pre" tag around your code, I think. (I can, anyway.)</p> </div> Andy Balaam 2018-09-20</span> </h3> Thanks, I will test it out and hopefully write a new post.</p> </div> Andy Balaam 2018-09-20</span> </h3> Hmm, something is not working the way we expect – it took longer for this code to do 1000 requests than the client-async-as-completed took to do 10000:</p> </p> $ ./timed ./client-pypeln 1000 Memory usage: 38672KB Time: 21.66 seconds </pre></p> </div> Cristian Garcia 2018-09-20</span> </h3> Thank you very much for the feedback! If possible, can we continue the discussion in this issue: https://github.com/cgarciae/pypeln/issues/9</a></p> I looked at the code and made some optimizations, please update to the development code like this before you try:</p> pip install git+https://github.com/cgarciae/pypeln@develop</a></p> </div> Cristian Garcia 2018-09-20</span> </h3> Maybe include the -U flag:</p> pip install -U git+https://github.com/cgarciae/pypeln@develop</a></p> </div> Andy Balaam 2018-09-21</span> </h3> Continuing here: https://github.com/cgarciae/pypeln/issues/9</a></p> </div> Andy Balaam 2018-09-24</span> </h3> See https://medium.com/@cgarciae/making-an-infini...</a> for Christian’s new improved version.</p> </div> Increase amounts of requests per second – PythonCharm 2018-11-12</span> </h3> […] and aiohttp might be even more noteworthy, but, I guess, it would be easier to learn a version of an already […]</p> </div> Vishwa 2019-08-28</span> </h3> Hi ,</p> This is great analysis on timecomplexity. Looking for similar requirement with Python based Unit test framework: 1. Fire 100000 unique URL requests in parallel ( Not sequential and not repetitive requests) — Thread1 2. Verify HTTP response status code ( eg: 200, 403, 404 etc) of each requests in parallel , may be handled via different thread — Thread2 3. Verify Response headers of each URL requests —> may be handled via different thread – Thread3 4. Login to remote machine ( to which requests is fired) and verify logs —> Can this be done using RpyC — Thread4</p> any unit test framework like tornado or different framework using asyncio/aiohttp suitable with above requirements in optimized time fashion.</p> </div> marius-joe 2020-08-01</span> </h3> Hi Andy and Cristian, great development you did here, thank you.</p> I just have one question: Has Ruslan’s suggestion to lower the CPU consumption been also imported to the pypeln library ? https://www.artificialworlds.net/blog/2017/06...</a></p> </div> Andy Balaam 2020-08-10</span> </h3> Hi, I don’t know I’m afraid.</p> </div> Arthur 2020-09-17</span> </h3> Hi,</p> Nice post – I would suggest looking into Queues to manage more easily the memory consumption: – you can insert 100M jobs into an asyncio queue: queue = asyncio.Queue(); [queue.put_nowait(i) for i in range(r)]</code>. This will only consume a few 100s of MB of RAM. – then you can create a worker task that gets a job from the queue, run it, and moves on the the next one:</p> async def worker(queue): </span></span> while True: </span></span> job = await queue.get() </span></span> await actual_work(job) # you might want to wrap this in a try/except to handle crappy jobs </span></span> job.task_done() </span></span></code></pre> You can launch as many concurrent tasks as necessary with asyncio.create_task workers = [asyncio.create_task(worker(queue)) for _ in range(1000)]</code> </li> And wait for everything to be finished before cancelling the workers: await queue.join() for task in workers: task.cancel() with contextlib.suppress(asyncio.CancelledError): await task </code></li> </ul> </div> Lukasz 2021-01-28</span> </h3> Hi,</p> thanks for a great article! Aiohttp got my intrest, as I’m trying to gather some basic info (not the whole response, but things like status, redirect history, etc.), and async approach seems to be the best. I had some memory issues as I’m trying to run my script on over a milion urls, and found your solution extremely helpful.</p> I’m wondering however, have you ever tested it outside of the setup described in the article (i.e. with real urls, some of which might be broken or temporarily unavailable)? I guess I’m sort of reproducing your approach and adjusting it to my needs, however I’ve noticed that performance of my script dminishes greatly over time. For example, I was able to check 20K urls in roughly 20 mins, but then while running the script on a longer list, I got to 125K after 9 hours. Moreover, adjusting the task limit doesn’t seem to influence much – when I run it on 20K urls it was 20 mins regardless of whether I set the limit to 100 or 1000.</p> Would you be able to help me with that? I’m not an expert on general programming (I use python primarily for analytical stuff), so I could have missed something. Do you know what this performance degradation might be caused by?</p> Please see the code I’m using below (I’ve also tried to mix the ‘as completed’ with semaphores, but eventually, I think it doesn’t make too much sense).</p> import pandas as pd import asyncio, aiohttp import time from itertools import islice</p> def task_limiter(tasks, task_limit):</p> futures = [asyncio.ensure_future(c) for c in islice(tasks, 0, task_limit)] </span></span> </span></span>async def task_list_manager(): </span></span> while True: </span></span> await asyncio.sleep(0) </span></span> for f in futures: </span></span> if f.done(): </span></span> futures.remove(f) </span></span> try: </span></span> newf = next(tasks) </span></span> futures.append(asyncio.ensure_future(newf)) </span></span> except StopIteration: </span></span> pass </span></span> return f.result() </span></span> </span></span>while len(futures) > 0: </span></span> yield task_list_manager() </span></span></code></pre> async def collect_url_info(session, url, n):</p> try: </span></span> async with await session.get(url) as response: </span></span> </span></span> redirect_history = [] </span></span> </span></span> for i in range(len(response.history)): </span></span> if str(url) == str(response.history[i].url): </span></span> continue </span></span> else: </span></span> redirect_history.append(str(response.history[i].url)) </span></span> </span></span> print(f'Task {n} done') </span></span> return response.status, response.reason, redirect_history </span></span> </span></span>except Exception as e: </span></span> </span></span> msg = str(type(e)) + "|||" + str(e) </span></span> print(f'Task {n} done with status -1 {msg}') </span></span> return (-1, msg, list()) </span></span></code></pre> async def bound_collect(sem, session, url, n): async with sem: return await collect_url_info(session, url, n)</p> async def run_url_check(urls, task_limit, conn_limit, timeout):</p> sem</span> =</span> asyncio</span>.</span>Semaphore</span>(</span>conn_limit</span>)</span> </span></span>connector</span> =</span> aiohttp</span>.</span>TCPConnector</span>(</span>limit</span>=</span>None</span>)</span> </span></span> </span></span>headers</span> =</span> {</span> </span></span></span>"</span>Accept</span>"</span>:</span> </span>"</span>text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9</span>"</span>,</span> </span></span></span>"</span>Accept-Encoding</span>"</span>:</span> </span>"</span>gzip, deflate</span>"</span>,</span> </span></span></span>"</span>Accept-Language</span>"</span>:</span> </span>"</span>en-GB,en-US;q=0.9,en;q=0.8</span>"</span>,</span> </span></span></span>"</span>Dnt</span>"</span>:</span> </span>"</span>1</span>"</span>,</span> </span></span></span>"</span>Upgrade-Insecure-Requests</span>"</span>:</span> </span>"</span>1</span>"</span>,</span> </span></span></span>"</span>User-Agent</span>"</span>:</span> </span>"</span>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36</span>"</span>,</span> </span></span></span>"</span>Referer</span>"</span>:</span> </span>"</span>google.com</span>"</span> </span></span>}</span> </span></span> </span></span> </span></span>async</span> with</span> aiohttp</span>.</span>ClientSession</span>(</span>connector</span>=</span>connector</span>,</span> timeout</span>=</span>timeout</span>,</span> headers</span>=</span>headers</span>)</span> as</span> session</span>:</span> </span></span> tasks</span> =</span> (</span>collect_url_info</span>(</span>session</span>,</span> urls</span>[</span>i</span>]</span>,</span> i</span>)</span> for</span> i</span> in</span> range</span>(</span>len</span>(</span>urls</span>)</span>)</span>)</span> </span></span> </span></span> result</span> =</span> [</span>None</span> for</span> i</span> in</span> range</span>(</span>len</span>(</span>urls</span>)</span>)</span>]</span> </span></span> for</span> i</span>,</span> res</span> in</span> enumerate</span>(</span>task_limiter</span>(</span>tasks</span>,</span> task_limit</span>)</span>)</span>:</span> </span></span> result</span>[</span>i</span>]</span> =</span> await</span> res</span> </span></span> </span></span>return</span> result</span> </span></span></code></pre> def prep_output_df(df, url_col, url_info):</p> df['status_code'] = [x[0] for x in url_info] </span></span>df['status_reason'] = [x[1] for x in url_info] </span></span>df['redirect_history'] = [str(x[2]) for x in url_info] </span></span>df['final_url'] = [x[2][-1] if len(x[2]) != 0 else '' for x in url_info] </span></span>df['redirect_flag'] = df['final_url'].apply(lambda x: 1 if x != '' else 0, 1) </span></span> </span></span>return df </span></span></code></pre> ###SCRIPT start = time.time()</p> task_limit = 10000 #conn_limit = 1000 timeout = aiohttp.ClientTimeout(total=None, sock_connect=60, sock_read=60)</p> input_path = ‘input.csv’ data = pd.read_csv(input_path, nrows = 20000) url_col = ‘WEBSITE’ urls = data[url_col]</p> loop = asyncio.get_event_loop() url_info = loop.run_until_complete(run_url_check(urls, task_limit, conn_limit, timeout))</p> data = prep_output_df(data, url_col, url_info)</p> total_requests = len(data) error_requests = (data[‘status_code’] == -1).sum() print(‘NUMBER OF REQUESTS:’) print(total_requests) print(‘NUMBER OF ERRORS:’) print(error_requests) print(‘ERROR RATE:’) print((error_requests/total_requests))</p> output_path = ‘out.csv’ data.to_csv(output_path, index=False)</p> end = time.time()</p> print(f’Time taken: {end-start}s’)</p> </div> Andy Balaam 2021-01-28</span> </h3> Hi Lukasz, my first guess would be that the site(s) you are connecting to may slow down or throttle you if you hit them this hard. It might be worth logging the time between making a request and receiving a response. Another guess: if you’re storing the responses in memory, you may be starting to use up your computer’s memory space. If memory gets tight, the machine will slow right down. Does it seem unresponsive? The solution in that case would be to write responses to a file as you go, instead of doing it at the end.</p> </div> Tibabalase O 2021-09-07</span> </h3> Hey Andy, thanks for the detailed post! Sorry to ask if you could go into more detail, but I’m having trouble understanding the limited_as_completed</code> fx, are there any plans to maybe go a bit deeper eventually?</p> </div> Andy Balaam 2021-09-07</span> </h3> No plans to go deeper I’m afraid!</p> </div> Hacer solicitudes asincrónicas más rápido - python en Español 2022-09-08</span> </h3> […] #1: Realicé una prueba comparativa similar a una en este sitio y 5.000 solicitudes completadas en 3,5 segundos. Así que parece que el hardware no es el problema […]</p> </div> Python - printing UTC dates in ISO8601 format with time zone Andy Balaam — Fri, 09 Jun 2017 09:06:49 +0000 By default, when you make a UTC date from a Unix timestamp in Python and print it in ISO format, it has no time zone:</p> $ python3 >>> from datetime import datetime >>> datetime.utcfromtimestamp(1496998804).isoformat() '2017-06-09T09:00:04'</pre> Whenever you talk about a datetime, I think you should always include a time zone, so I find this problematic.</p> The solution is to mention the timezone explicitly when you create the datetime:</p> $ python3 >>> from datetime import datetime, timezone >>> datetime.fromtimestamp(1496998804, tz=timezone.utc).isoformat() '2017-06-09T09:00:04+00:00'</pre> Note, including the timezone explicitly works the same way when creating a datetime in other ways:</p> $ python3 >>> from datetime import datetime, timezone >>> datetime(2017, 6, 9).isoformat() '2017-06-09T00:00:00' >>> datetime(2017, 6, 9, tzinfo=timezone.utc).isoformat() '2017-06-09T00:00:00+00:00'</pre> Comments</h2> Derek 2018-08-23</span> </h3> Is there an easy way to use ‘Z’ instead of the more clumsy ‘+00:00’ at the end of UTC time?</p> </div> Python 3 - large numbers of tasks with limited concurrency Andy Balaam — Wed, 31 May 2017 02:09:43 +0000 Series: asyncio basics</a>, large numbers in parallel, parallel HTTP requests</a>, adding to stdlib</a></p> I am interested in running large numbers of tasks in parallel, so I need something like asyncio.as_completed, but taking an iterable instead of a list, and with a limited number of tasks running concurrently. First, let's try to build something pretty much equivalent to asyncio.as_completed. Here is my attempt, but I'd welcome feedback from readers who know better:</p> # Note this is not a coroutine - it returns # an iterator - but it crucially depends on # work being done inside the coroutines it # yields - those coroutines empty out the # list of futures it holds, and it will not # end until that list is empty. def my_as_completed(coros): # Start all the tasks futures = [asyncio.ensure_future(c) for c in coros] # A coroutine that waits for one of the # futures to finish and then returns # its result. async def first_to_finish(): # Wait forever - we could add a # timeout here instead. while True: # Give up control to the scheduler # - otherwise we will spin here # forever! await asyncio.sleep(0) # Return anything that has finished for f in futures: if f.done(): futures.remove(f) return f.result() # Keep yielding a waiting coroutine # until all the futures have finished. while len(futures) > 0: yield first_to_finish()</pre> The above can be substituted for asyncio.as_completed in the code that uses it in the first article</a>, and it seems to work. It also makes a reasonable amount of sense to me, so it may be correct, but I'd welcome comments and corrections.</p> my_as_completed above accepts an iterable and returns a generator producing results, but inside it starts all tasks concurrently, and stores all the futures in a list. To handle bigger lists we will need to do better, by limiting the number of running tasks to a sensible number.</p> Let's start with a test program:</p> import asyncio async def mycoro(number): print("Starting %d" % number) await asyncio.sleep(1.0 / number) print("Finishing %d" % number) return str(number) async def print_when_done(tasks): for res in asyncio.as_completed(tasks): print("Result %s" % await res) coros = [mycoro(i) for i in range(1, 101)] loop = asyncio.get_event_loop() loop.run_until_complete(print_when_done(coros)) loop.close()</pre> This uses asyncio.as_completed to run 100 tasks and, because I adjusted the asyncio.sleep command to wait longer for earlier tasks, it prints something like this:</p> $ time python3 python-async.py Starting 47 Starting 93 Starting 48 ... Finishing 93 Finishing 94 Finishing 95 ... Result 93 Result 94 Result 95 ... Finishing 46 Finishing 45 Finishing 42 ... Finishing 2 Result 2 Finishing 1 Result 1 real 0m1.590s user 0m0.600s sys 0m0.072s</pre> So all 100 tasks were completed in 1.5 seconds, indicating that they really were run in parallel, but all 100 were allowed to run at the same time, with no limit.</p> We can adjust the test program to run using our customised my_as_completed function, and pass in an iterable of coroutines instead of a list by changing the last part of the program to look like this:</p> async def print_when_done(tasks): for res in my_as_completed(tasks): print("Result %s" % await res) coros = (mycoro(i) for i in range(1, 101)) loop = asyncio.get_event_loop() loop.run_until_complete(print_when_done(coros)) loop.close()</pre> But we get similar output to last time, with all tasks running concurrently.</p> To limit the number of concurrent tasks, we limit the size of the futures list, and add more as needed:</p> from itertools import islice def limited_as_completed(coros, limit): futures = [ asyncio.ensure_future(c) for c in islice(coros, 0, limit) ] async def first_to_finish(): while True: await asyncio.sleep(0) for f in futures: if f.done(): futures.remove(f) try: newf = next(coros) futures.append( asyncio.ensure_future(newf)) except StopIteration as e: pass return f.result() while len(futures) > 0: yield first_to_finish()</pre> We start limit tasks at first, and whenever one ends, we ask for the next coroutine in coros and set it running. This keeps the number of running tasks at or below limit until we start running out of input coroutines (when next throws and we don't add anything to futures), then futures starts emptying until we eventually stop yielding coroutine objects.</p> I thought this function might be useful to others, so I started a little repo over here</a> and added it: asyncioplus/limited_as_completed.py</a>. Please provide merge requests and log issues to improve it - maybe it should be part of standard Python?</p> When we run the same example program, but call limited_as_completed instead of the other versions:</p> async def print_when_done(tasks): for res in limited_as_completed(tasks, 10)</span>: print("Result %s" % await res) coros = (mycoro(i) for i in range(1, 101)) loop = asyncio.get_event_loop() loop.run_until_complete(print_when_done(coros)) loop.close()</pre> We see output like this:</p> $ time python3 python-async.py Starting 1 Starting 2 ... Starting 9 Starting 10 Finishing 10</span> Result 10 Starting 11 ... Finishing 100 Result 100 Finishing 1 Result 1 real 0m1.535s user 0m1.436s sys 0m0.084s</pre> So we can see that the tasks are still running concurrently, but this time the number of concurrent tasks is limited to 10.</p> See also</h3> To achieve a similar result using semaphores, see Python asyncio.semaphore in async-await function</a> and Making 1 million requests with python-aiohttp</a>.</p> It feels like limited_as_completed is more re-usable as an approach but I'd love to hear others' thoughts on this. E.g. could/should I use a semaphore to implement limited_as_completed instead of manually holding a queue?</p> Comments</h2> David L 2017-11-30</span> </h3> This is golden, thanks for publishing it. Would love to see the limit</code> kwarg added to as_completed</code> in asyncio.</p> </div> Andy Balaam 2017-12-01</span> </h3> Thanks David – I guess it might be worth commenting on https://bugs.python.org/issue30782</a> ?</p> </div> Trahibidadido 2017-12-01</span> </h3> I’ve build some kind of object realization of it. Simply to inject everywhere in my projects, much tnx.</p> If you’re interested, I’ll create a pull asap.</p> </div> Bruce Wu 2017-12-02</span> </h3> I`m developing a web crawler framework with asyncio and I met memory high usage problem in testï¼Œso thx for you blog.</p> I notice you used loop to polling futures completion, why not use callback function as same as official "as_completed" method?</p> </div> Trahibidadido 2017-12-03</span> </h3> Ok, I’ve kinda lost my way and reimplement this for futures: https://gist.github.com/trahibidadido/3790adf...</a></p> </div> Andy Balaam 2017-12-08</span> </h3> Hi Trahibidadido, I’m not sure I understand. As far as I can see a function is the nicest way to provide this – maybe you can explain a bit more?</p> </div> Andy Balaam 2017-12-08</span> </h3> Hi Bruce, my understanding of the official as_completed is that it is intended to be called in a loop – am I mistaken?</p> </div> Andy Balaam 2017-12-08</span> </h3> Hi Trahibidadido, I think the version from my blog post will already work with futures, won’t it?</p> </div> Trahibidadido 2017-12-08</span> </h3> Yes. Your solution uses only one process, and executes coros with threads (threadPoolExecutor). If you try some cpu heavy tasks (in my case many regex text transforms) – it will be slow as hell.</p> I’ve tried to implement ProcessPoolExecutor and divide coros not by tasking %limit% coroutines and dealing with its futures, but tasking all coroutines at once and deal with all of the futures in processPool. This gives me some extra performance.</p> But my way, sadly, it doesnt work in oop ): https://stackoverflow.com/questions/47662040/...</a></p> </div> Bruce Wu 2017-12-09</span> </h3> Hi Andy, maybe my reply is not clear, sorry for confusing youã€‚</p> Let`s look into your blog code:</p> ... </span></span> async def first_to_finish(): </span></span> while True: # this is the loop I want to say </span></span> await asyncio.sleep(0) </span></span> for f in futures: </span></span> if f.done(): </span></span> futures.remove(f) </span></span>... </span></span></code></pre> The code use a loop to capture futuress completion, and I see the official â€œas_completedâ€ method use futures</code>s callback function.(https://github.com/python/cpython/blob/master...</a>)</p> </div> Andy Balaam 2018-01-18</span> </h3> Hi Bruce, I see that you mean inside</em> the as_completed function, sorry! Yes, the official version does it differently, so when I proposed to make this change in the standard library I did it that way: https://www.artificialworlds.net/blog/2017/06...</a> – see the MR here: https://github.com/python/cpython/pull/2424/files</a> . Personally, I prefer a for loop to callbacks, but that is a matter of taste I think.</p> </div> Ethan 2019-08-28</span> </h3> This is fantastic. Is there any reason you wouldn’t use BoundedSemaphore for rate limiting?</p> https://docs.python.org/3/library/asyncio-syn...</a></p> I’ve used it to rate limit aiohttp requests and it’s super clean.</p> </div> Andy Balaam 2019-09-05</span> </h3> Thanks Ethan, yeah that looks really useful.</p> </div> Multiple Async Requests Simultaneously - Programming Questions And Solutions Blog 2022-12-09</span> </h3> […] There are a number of ways to limit the rate of concurrency. For instance, see asyncio.semaphore in async-await function or large numbers of tasks with limited concurrency. […]</p> </div> Async IO en Python: un tutorial completo - Python super fácil 2023-03-14</span> </h3> […] https://www.artificialworlds.net/blog/2017/05...</a> […]</p> </div> Basic ideas of Python 3 asyncio concurrency Andy Balaam — Wed, 31 May 2017 01:46:32 +0000 Series: asyncio basics, large numbers in parallel</a>, parallel HTTP requests</a>, adding to stdlib</a></p> Update: see the Python Async Basics video</a> on this topic.</p> Python 3's asyncio module and the async and await keywords combine to allow us to do cooperative concurrent programming</a>, where a code path voluntarily yields control to a scheduler, trusting that it will get control back when some resource has become available (or just when the scheduler feels like it). This way of programming can be very confusing, and has been popularised by Twisted</a> in the Python world, and nodejs</a> (among others) in other worlds.</p> I have been trying to get my head around the basic ideas as they surface in Python 3's model. Below are some definitions and explanations that have been useful to me as I tried to grasp how it all works.</p> Futures</strong> and coroutines</strong> are both things that you can wait for.</p> You can make a coroutine by declaring it with async def: import asyncio async def</span> mycoro(number): print("Starting %d" % number) await</span> asyncio.sleep(1) print("Finishing %d" % number) return str(number)</pre></p> Almost always, a coroutine will await something such as some blocking IO. (Above we just sleep for a second.) When we await, we actually yield control to the scheduler so it can do other work and wake us up later, when something interesting has happened.</p> You can make a future out of a coroutine, but often you don't need to. Bear in mind that if you do want to make a future, you should use ensure_future, but this actually runs</strong></em> what you pass to it - it doesn't just create a future:myfuture1 = asyncio.ensure_future</span>(mycoro(1))</p> Runs mycoro!</pre></h1> But, to get its result, you must wait for it - it is only scheduled in the background:# Assume mycoro is defined as above myfuture1 = asyncio.ensure_future(mycoro(1))</span></p> We end the program without waiting for the future to finish</pre></h1> So the above fails like this:</p> $ python3 ./python-async.py Task was destroyed but it is pending! task: <Task pending coro=<mycoro() running at ./python-async:10>> sys:1: RuntimeWarning: coroutine 'mycoro' was never awaited</span></pre> The right way to block waiting for a future outside of a coroutine is to ask the event loop</strong> to do it:# Keep on assuming mycoro is defined as above for all the examples myfuture1 = asyncio.ensure_future(mycoro(1)) loop = asyncio.get_event_loop() loop.run_until_complete(myfuture1)</span> loop.close()</pre></p> Now this works properly (although we're not yet getting any benefit from being asynchronous):$ python3 python-async.py Starting 1 Finishing 1</pre></p> To run several things concurrently, we make a future that is the combination of several other futures. asyncio can make a future like that out of coroutines using asyncio.gather:</p> several_futures = asyncio.gather</span>( mycoro(1), mycoro(2), mycoro(3)) loop = asyncio.get_event_loop() print(loop.run_until_complete(several_futures)) loop.close()</pre> The three coroutines all run at the same time, so this only takes about 1 second to run, even though we are running 3 tasks, each of which takes 1 second:</p> $ python3 python-async.py Starting 3 Starting 1 Starting 2 Finishing 3 Finishing 1 Finishing 2 ['1', '2', '3']</pre> asyncio.gather won't necessarily run your coroutines in order, but it will return a list of results in the same order as its input.</p> Notice also that run_until_complete returns the result of the future created by gather - a list of all the results from the individual coroutines.</p> To do the next bit we need to know how to call a coroutine from a coroutine. As we've already seen, just calling a coroutine in the normal Python way doesn't run it, but gives you back a "coroutine object". To actually run the code, we need to wait for it. When we want to block everything until we have a result, we can use something like run_until_complete but in an async context we want to yield control to the scheduler and let it give us back control when the coroutine has finished. We do that by using await:</p> import asyncio async def f2(): print("start f2") await asyncio.sleep(1) print("stop f2") async def f1(): print("start f1") await</span> f2() print("stop f1") loop = asyncio.get_event_loop() loop.run_until_complete(f1()) loop.close()</pre> This prints:</p> $ python3 python-async.py start f1 start f2 stop f2 stop f1</pre> Now we know how to call a coroutine from inside a coroutine, we can continue.</p> We have seen that asyncio.gather takes in some futures/coroutines and returns a future that collects their results (in order).</p> If, instead, you want to get results as soon as they are available, you need to write a second coroutine that deals with each result by looping through the results of asyncio.as_completed and awaiting each one.</p> # Keep on assuming mycoro is defined as at the top async def print_when_done(tasks): for res in asyncio.as_completed</span>(tasks): print("Result %s" % await</span> res) coros = [mycoro(1), mycoro(2), mycoro(3)] loop = asyncio.get_event_loop() loop.run_until_complete(print_when_done(coros)) loop.close()</pre> This prints:</p> $ python3 python-async.py Starting 1 Starting 3 Starting 2 Finishing 3 Result 3 Finishing 2 Result 2 Finishing 1 Result 1</pre> Notice that task 3 finishes first and its result is printed, even though tasks 1 and 2 are still running.</p> asyncio.as_completed returns an iterable sequence of futures, each of which must be awaited, so it must run inside a coroutine, which must be waited for too.</p> The argument to asyncio.as_completed has to be a list of coroutines or futures, not an iterable, so you can't use it with a very large list of items that won't fit in memory.</p> Side note: if we want to work with very large lists, asyncio.wait won't help us here - it also takes a list of futures and waits for all of them to complete (like gather), or, with other arguments, for one of them to complete or one of them to fail. It then returns two sets of futures: done and not-done. Each of these must be awaited to get their results, so:</p> asyncio.gather # is roughly equivalent to: async def mygather(*args): ret = [] for r in (await asyncio.wait(args))[0]: ret.append(await r) return ret</pre> Further reading: realpython.com/async-io-python</a> (a very complete and clear explanation, with lots of links)</p> I am interested in running very large numbers of tasks with limited concurrency - see the next article</a> for how I managed it.</p> Comments</h2> Attila Homoki 2018-02-10</span> </h3> Just what I was looking for, very nicely presented…thank you</p> </div> Andy Balaam 2018-02-10</span> </h3> Thanks Attila!</p> </div> Dmitry 2018-02-23</span> </h3> Andy,</p> Your article explains the topic in very clear and neat way.</p> Thank you very much.</p> </div> Andy Balaam 2018-02-24</span> </h3> Thanks Dmitry, glad it helped.</p> </div> Hamad 2018-04-17</span> </h3> Wow dude. I think i might finally be able to understand async thanks to this brilliant article. Thanks a ton</p> </div> Andy Balaam 2018-04-18</span> </h3> Great to hear it Hamad!</p> </div> Kansai 2020-05-26</span> </h3> Thank you for the article. I learned a lot. I have a question though. In the function async def print_when_done(tasks): for res in asyncio.as_completed(tasks): print("Result %s" % await res) print("Argument is") #Here</p> is there a way to know the argument that is being passed? For example since I am passing coros = [mycoro(1), mycoro(2), mycoro(3)] can I somehow print !Argument is 1" "Argument is 2" "Argument is 3" somehow?</p> </div> Andy Balaam 2020-06-01</span> </h3> Hi Kansai, I don’t think so – you’ll need to print the argument inside mycoro.</p> </div> palau 2021-03-08</span> </h3> Andy your artical is very good Thanks</p> </div> C++ iterator wrapping a stream not 1-1 Andy Balaam — Wed, 17 May 2017 07:25:56 +0000 Series: Iterator</a>, Iterator Wrapper</a>, Non-1-1 Wrapper</p> Sometimes we want to write an iterator that consumes items from some underlying iterator but produces its own items slower than the items it consumes, like this:</p> ColonSep items("aa:foo::x"); // Prints "aa, foo, , x" for(auto s : items) { std::cout << s << ", "; }</pre> When we pass a 9-character string (i.e. an iterator that yields 9 items) to ColonSep, above, we only repeat 4 times in our loop, because ColonSep provides an iterable range that yields one value for each whole word it finds in the underlying iterator of 9 characters.</p> To do something like this I'd recommend consuming the items of the underlying iterator early, so it is ready when requested with operator*. We also need our iterator class to hold on to the end of the underlying iterator as well as the current position.</p> First we need a small container to hold the next item we will provide:</p> struct maybestring { std::string value_; bool at_end_; explicit maybestring(const std::string value) : value_(value) , at_end_(false) {} explicit maybestring() : value_("--error-past-end--") , at_end_(true) {} };</pre> A maybestring either holds the next item we will provide, or at_end_ is true, meaning we have reached the end of the underlying iterator and we will report that we are at the end ourself when asked.</p> Like the simpler iterators we have looked at, we still need a little container to return from the postincrement operator:</p> class stringholder { const std::string value_; public: stringholder(const std::string value) : value_(value) {} std::string operator*() const { return value_; } };</pre> Now we are ready to write our iterator class, which always has the next value ready in its next_ member, and holds on to the current and end positions of the underlying iterator in wrapped_ and wrapped_end_:</p> class myit { private: typedef std::string::const_iterator wrapped_t; wrapped_t wrapped_; wrapped_t wrapped_end_; maybestring next_;</pre> The constructor holds on the underlying iterator pointers, and immediately fills next_ with the next value by calling next_item passing in true to indicate that this is the first item:</p> public: myit(wrapped_t wrapped, wrapped_t wrapped_end) : wrapped_(wrapped) , wrapped_end_(wrapped_end) , next_(next_item(true)) { } // Previously provided by std::iterator typedef int value_type; typedef std::ptrdiff_t difference_type; typedef int* pointer; typedef int& reference; typedef std::input_iterator_tag iterator_category;</pre> next_item looks like this:</p> private: maybestring next_item(bool first_time) { if (wrapped_ == wrapped_end_) { return maybestring(); // We are at the end } else { if (!first_time) { ++wrapped_; } return read_item(); } }</pre> next_item recognises whether we've reached the end of the underlying iterator and saves the empty maybstring if so. Otherwise, it skips forward once (unless we are on the first element) and then calls read_item:</p> maybestring read_item() { std::string ret = ""; for (; wrapped_ != wrapped_end_; ++wrapped_) { char c = *wrapped_; if (c == ':') { break; } ret += c; } return maybestring(ret); }</pre> read_item implements the real logic of looping through the underlying iterator and combining those values together to create the next item to provide.</p> The hard part of the iterator class is done, leaving only the more normal functions we must provide:</p> public: value_type operator*() const { assert(!next_.at_end_); return next_.value_; } bool operator==(const myit& other) const { // We only care about whether we are at the end return next_.at_end_ == other.next_.at_end_; } bool operator!=(const myit& other) const { return !(*this == other); } stringholder operator++(int) { assert(!next_.at_end_); stringholder ret(next_.value_); next_ = next_item(false); return ret; } myit& operator++() { assert(!next_.at_end_); next_ = next_item(false); return *this; } }</pre> Note that operator== is only concerned with whether or not we are an end iterator or not. Nothing else matters for providing correct iteration.</p> Our final bit of bookkeeping is the range class that allows our new iterator to be used in a for loop:</p> class ColonSep { private: const std::string str_; public: ColonSep(const std::string str) : str_(str) {} myit begin() { return myit(std::begin(str_), std::end(str_)); } myit end() { return myit(std::end(str_), std::end(str_)); } };</pre> A lot of the code above is needed for all code that does this kind of job. Next time we'll look at how to use templates to make it useable in the general case.</p> How to write a programming language ACCU talk Andy Balaam — Wed, 17 May 2017 00:08:25 +0000 My talk from ACCU Conference 2017</a> where I describe a tiny programming language I wrote:</p> </a></p> Slides: How to write a programming language</a></p> Cell source code: github.com/andybalaam/cell</a></p> C++ iterator wrapper/adaptor example Andy Balaam — Fri, 12 May 2017 07:10:58 +0000 Series: Iterator</a>, Iterator Wrapper, Non-1-1 Wrapper</a></p> If you want to wrap an iterable range with another that transforms the underlying iterators in some way and allows looping or constructing other objects:</p> for (auto ch : Upper("abcdef")) { // Prints "ABCDEF" std::cout << ch; } Upper up(std::string("fOo")); std::string newfoo(std::begin(up), std::end(up)); assert(newfoo == "FOO"); </pre> then, similar to an ordinary iterable range</a> you will need to make a range class and a iterator class:</p> class myit { private: std::string::const_iterator wrapped_; class charholder { const char value_; public: charholder(const char value) : value_(value) {} char operator*() const { return value_; } }; public: // Previously provided by std::iterator typedef int value_type; typedef std::ptrdiff_t difference_type; typedef int* pointer; typedef int& reference; typedef std::input_iterator_tag iterator_category; explicit myit(std::string::const_iterator wrapped) : wrapped_(wrapped) {} value_type operator*() const { return std::toupper(*wrapped_); } bool operator==(const myit& other) const { return wrapped_ == other.wrapped_; } bool operator!=(const myit& other) const { return !(*this == other); } charholder operator++(int) { charholder ret(std::toupper(*wrapped_)); ++wrapped_; return ret; } myit& operator++() { ++wrapped_; return *this; } }; class Upper { private: const std::string str_; public: Upper(const std::string str) : str_(str) {} myit begin() { return myit(std::begin(str_)); } myit end() { return myit(std::end(str_)); } }; </pre> Notice the need to call the transforming/adapting functionstd::toupper in two places.</p> Update:</strong> std::iterator is deprecated in C++17, so removed.</p> Comments</h2> Ramyken 2019-10-18</span> </h3> Hello Andy, firstly many thanks for the great adaptor example. But what I don’t understand is why you have implemented the inner class charholder used in charholder operator++(int). Is it really necessary? Thanks</p> </div> Andy Balaam 2019-10-18</span> </h3> Hi Ramyken, Thanks! See the comments on the first post in this series here: https://www.artificialworlds.net/blog/2017/05...</a> . I thought very similarly to you when I wrote that, and Anthony Williams (author of C++ Concurrency In Action) very kindly put me straight. The postincrement operator has to return the value at the iterator before it was incremented. (That is why, in general, you should not use it…).</p> </div> Ramyken 2019-10-21</span> </h3> Hi Andy, many thanks, I did not know about it.</p> </div> C++ iterator example (and an iterable range) Andy Balaam — Thu, 11 May 2017 07:15:28 +0000 Series: Iterator, Iterator Wrapper</a>, Non-1-1 Wrapper</a></p> To make your own iterable range in C++ that you can loop over or make a vector out of (mine is called Numbers):</p> // Prints: // 3,4, for (auto n : Numbers(3, 5)) { std::cout << n << ","; } // Fills vec with 7, 8, 9 Numbers nums(7, 10); std::vector vec{std::begin(nums), std::end(nums)}; </int></pre> you need to write a class that is an Input Iterator, and provide an instance as the begin and end of your range:</p> class Numbers { private: const int start_; const int end_; public: Numbers(int start, int end) : start_(start) , end_(end) {} myit begin() { return myit(start_); } myit end() { return myit(end_); } }; </pre> The hard bit is the Input Iterator:</p> #include <iterator> class myit { private: int value_; class intholder { int value_; public: intholder(int value): value_(value) {} int operator*() { return value_; } }; public: // Previously provided by std::iterator - see update below typedef int value_type; typedef std::ptrdiff_t difference_type; typedef int* pointer; typedef int& reference; typedef std::input_iterator_tag iterator_category; explicit myit(int value) : value_(value) {} int operator*() const { return value_; } bool operator==(const myit& other) const { return value_ == other.value_; } bool operator!=(const myit& other) const { return !(*this == other); } intholder operator++(int) { intholder ret(value_); ++*this; return ret; } myit& operator++() { ++value_; return *this; } }; </pre> Update</strong>: thanks to Anthony Williams for the correction on the postincrement operator - see Generating Sequences</a> for more. Note the need to return a fiddly object that holds the answer we will give when the return value is dereferenced.</p> Using std::iterator as a base class actually only gives you some typedefs for free, but it's worth it to get the standard names, and to be clear what we are trying to do.</del> Update:</strong> std::iterator is deprecated in C++17</a> - just add the 5 typedefs yourself.</p> I suspect I might need to add some extra &s to make this good C++11 style?</p> Comments</h2> Anthony Williams 2017-05-11</span> </h3> Sadly, your postincrement operator isn’t right.</p> The requirement on postincrement is that *it++ returns the value at the iterator before the increment. Returning value_type from the postincrement operator doesn’t work for that: you would need to write just it++, which would be odd.</p> I explained how to write an iterator for exactly this scenario (numeric ranges) on my blog a couple of months ago: https://www.justsoftwaresolutions.co.uk/cplus...</a></p> </div> Andy Balaam 2017-05-11</span> </h3> Thanks Anthony – I wish I’d found your post when I was researching this. I was kind of hoping I’d get some corrections if I posted this!</p> </div> Make Android Gradle display unit test failure messages Andy Balaam — Wed, 10 May 2017 00:30:05 +0000 By default, Gradle does not show you what happened when a unit test failed:</p> $ ./gradlew test ... MyTest > Black_is_white FAILED org.junit.ComparisonFailure at MyTest.java:6 ^^^ WHAT ACTUALLY FAILED????</span> ...</pre> This is insane, and can be fixed (thanks to mrhaki</a>) by editing build.gradle to add:</p> // NOTE: this is the non-Android</span> solution - add to build.gradle test { testLogging { exceptionFormat = 'full' } }</pre> The above doesn't work with Android, but something similar does:</p> // Android</span> solution: add this to app/build.gradle android.testOptions.unitTests.all { testLogging { exceptionFormat = "full" } }</pre> And sanity prevails:</p> $ ./gradlew test ... MyTest > Black_is_white FAILED org.junit.ComparisonFailure: expected:<[black]> but was:<[white]> at org.junit.Assert.assertEquals(Assert.java:115) at org.junit.Assert.assertEquals(Assert.java:144) at MyTest.Black_is_white(MyTest.java:6) ...</pre> Files for plain Gradle:</h3> $ cat build.gradle</span> apply plugin: 'java' repositories { mavenCentral() } dependencies { testCompile 'junit:junit:[4,)' } test { testLogging { exceptionFormat = 'full' } } $ cat src/test/java/MyTest.java</span> public class MyTest { @org.junit.Test public void Black_is_white() { org.junit.Assert.assertEquals("black", "white"); } }</pre> Files for Android+Gradle</h3> $ cat build.gradle</span> buildscript { repositories { jcenter() } dependencies { classpath 'com.android.tools.build:gradle:2.3.1' } } allprojects { repositories { jcenter() } } $ cat app/src/main/AndroidManifest.xml</span> <?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="net.artificialworlds.testapp"> </manifest> $ cat app/src/test/java/MyTest.java</span> public class MyTest { @org.junit.Test public void Black_is_white() { org.junit.Assert.assertEquals("black", "white"); } } $ cat build.gradle</span> buildscript { repositories { jcenter() } dependencies { classpath 'com.android.tools.build:gradle:2.3.1' } } allprojects { repositories { jcenter() } } $ cat local.properties</span> sdk.dir=/home/andy/Android/Sdk $ cat settings.gradle</span> include ':app' </pre> Planet Code updates Andy Balaam — Tue, 02 May 2017 21:12:43 +0000 I maintain an unofficial aggregator of blogs by people who are involved with ACCU</a> and things they might be interested in.</p> It's called Planet Code</a>.</p> Today I made it look slightly less ugly, and added several new blogs after attending the excellent ACCU Conference</a>.</p> Subscribe, and enjoy.</p> A story about magic and how we treat each other Andy Balaam — Tue, 02 May 2017 08:29:19 +0000 For a lightning talk at the ACCU Conference</a> I wrote a little story:</p> A story about magic and how we treat each other</a></p> It describes one person's journey towards realising that we need to act to be kind to each other, and not to expect it to happen automatically.</p> In the tech community, a lot of people want to be kind, but find relating to people hard work, which can mean people are excluded when we stick to familiar groups. I think there was a widespread opinion that proper geeks were so uninterested in relationship stuff that we treated everyone equally by default, but that is clearly untrue given the makeup of many tech communities.</p> People are being excluded, and we need to be proactive in changing the behaviours that cause this.</p> Annual cost of clean water for every human Andy Balaam — Tue, 07 Mar 2017 15:53:23 +0000 According to a 2010 WHO-backed study</a> the cost of safe water for every human would be about 10 billion US dollars per year.</p> The UK government revenue for 2010-2011</a> was about 639 billion US dollars.</p> </p> I would vote for a political party proposing to raise this money and spend it for this purpose.</p> Iterating over the lines of a file in Java Andy Balaam — Thu, 09 Feb 2017 13:25:48 +0000 If you're trying to write a standard command-line tool in Java you are going to experience pain.</p> In Python, if we want to do something with each line on input to your program, we can do something like:</p> import sys for ln in sys.stdin: print(ln) # (We have an extra newline but who's counting?) </pre> In Java, after some fighting, the two closest alternatives I can find are:</p> import java.util.Scanner; class IterateInputScanner { public static void main(String[] args) { for(String ln : (Iterable)(() -> new Scanner(System.in).useDelimiter("\n"))) { System.out.println(ln); } } } </string></pre> or, less fashionably, and even more horribly:</p> import java.io.BufferedReader; import java.io.InputStreamReader; class IterateInputReader { public static void main(String[] args) throws java.io.IOException { BufferedReader reader = new BufferedReader(new InputStreamReader(System.in)); String ln; while((ln = reader.readLine()) != null) { System.out.println(ln); } } } </pre> Note that if you are reading from a stream other than stdin you will probably want to wrap this in a try-with-resources to close the stream when you've finished.</p> Take your pick, and I hope it saves you some fighting.</p> Comments</h2> Scott Shipp 2017-04-21</span> </h3> Check out java.nio.file.Files and java.nio.file.Paths. Something like:</p> Files.lines(Paths.get(fileToRead)).forEach((line) -> { System.out.println(line); } );</p> </div> Andy Balaam 2017-05-02</span> </h3> Nice, thanks Scott!</p> </div> Raspberry Pi Jam "Chaos Car!" Andy Balaam — Sun, 22 Jan 2017 13:01:33 +0000 Raspberry Pi 1 + battery pack + Bluetooth USB dongle + i-racer bluetooth car + Raspberry Pi camera + some Python code + loads of sellotape =</p> Chaos car!</p> Here's the code:</p> #!/usr/bin/env python2 import os import random import bluetooth import sys import time car_name = "DaguCar" def find_car_mac( car_name ): ret = None print "Scanning for a device called %s..." % car_name devices = bluetooth.discover_devices() for addr in devices: dev_name = bluetooth.lookup_name( addr ) if dev_name == car_name: print "Car found! (MAC=%s)" % addr time.sleep( 1 ) return addr else: print "Skipping device named '%s'." % dev_name sys.stderr.write( "Couldn't find a device called %s!\n" % car_name ) sys.exit( 1 ) #car_mac = find_car_mac( car_name ) print "Using hard-coded MAC address" car_mac = "20:13:04:23:05:71" print "Connecting to the car at %s..." % car_mac sock = bluetooth.BluetoothSocket( bluetooth.RFCOMM ) sock.connect( ( car_mac, 1 ) ) class Car: def turn_right(self): print("looking right") sock.send( '\x6A' ) def turn_left(self): print("looking left") sock.send( '\x5A' ) def roll_forward(self): print("watch out in front coming forward") sock.send( '\x1A' ) def roll_backward(self): print("beep beep going backward") sock.send( '\x2A' ) def wait(self): print("waiting") sock.send( '\x00' ) car = Car() instruction = ["turn right", "turn left", "roll forward", "roll backward", "wait"] while True: ci = random.choice(instruction) if ci == "turn right": car.turn_right() elif ci == "turn left": car.turn_left() elif ci == "roll forward": car.roll_forward() elif ci == "roll backward": car.roll_backward() elif ci == "wait": car.wait() time.sleep(0.5) </pre> Improving the code to avoid bumping into walls is left as an exercise for the reader.</p> Comments</h2> Raspberry Pi Jam â€œChaos Car!â€ Code #piday #raspberrypi @Raspberry_Pi « Adafruit Industries – Makers, hackers, artists, designers and engineers! 2017-01-27</span> </h3> […] From Andy Balaam’s Blog: […]</p> </div> Automated UI tests on Android Andy Balaam — Tue, 17 Jan 2017 22:37:43 +0000 I recently fought the Android emulator a lot to get my UI tests to run automatically during the build of Rabbit Escape</a>, so I thought I'd better write down what I did before I forget.</p> I already have tests that drive the Android UI (see e.g. SmokeTest.java</a> - it's not pretty, but it seems reliable, and it catches real problems) - this post is about how to run them automatically during our build.</p> Note that to run the Android emulator I'm fairly sure you need a windowing environment, so I don't think this could be moved to a headless build server. If course, you could always fight some kind of framebuffer thing.</p> Here is the part of our Makefile that launches the tests: android-smoke-tests: @echo ". Running Android smoke tests" ./build-scripts/android-start-emulator "android-8" "3.2in QVGA (ADP2)" ./build-scripts/android-test "free" "app-free-debug" ./build-scripts/android-test "" "app-paid-debug" ./build-scripts/android-stop-emulator </pre></p> and here is ./build-scripts/android-start-emulator - it starts up and emulator, waits for it to be ready, and unlocks its screen.:</p> #!/bin/bash set -x set -u set -e # Args TARGET="$1" # E.g. "android-8" DEVICE="$2" # E.g. "3.2in QVGA (ADP2)" # Setup ADB="${HOME}/Android/Sdk/platform-tools/adb" EMULATOR="${HOME}/Android/Sdk/tools/emulator" ANDROID="${HOME}/Android/Sdk/tools/android" NAME="rabbitescape-${TARGET}" TMP="/data/local/tmp" ${ANDROID} create avd \ --force \ --name "${NAME}" \ --target "${TARGET}" \ --abi "armeabi" \ --device "${DEVICE}" # Start the emulator ${EMULATOR} -avd "${NAME}" & # Wait for the device to boot and unlock it ${ADB} wait-for-device shell < ${TMP}/zero getprop dev.bootcomplete > ${TMP}/bootcomplete while cmp ${TMP}/zero ${TMP}/bootcomplete; do { echo -n "." sleep 1 getprop dev.bootcomplete > ${TMP}/bootcomplete }; done echo "Booted." exit ENDSCRIPT echo "Waiting 30 secs for us to be really booted" sleep 30 echo "Unlocking screen" ${ADB} shell "input keyevent 82" </endscript></pre> Now here is android-test - it launches the JUnit test code on the running emulator::</p> #!/bin/bash set -x set -u set -e PKGSUFFIX="$1" # E.g. "free" APKNAME="$2" # E.g. "app-free-debug" APPID="net.artificialworlds.rabbitescape${PKGSUFFIX}" TESTAPPID="net.artificialworlds.rabbitescape${PKGSUFFIX}.test" APK="rabbit-escape-ui-android/app/build/outputs/apk/${APKNAME}.apk" TESTAPK="rabbit-escape-ui-android/app/build/outputs/apk/${APKNAME}-androidTest.apk" ADB="${HOME}/Android/Sdk/platform-tools/adb" DIR="/data/local/tmp/${APPID}" TESTDIR="/data/local/tmp/${TESTAPPID}" function run_test() { TMPFILE=$(mktemp) ${ADB} shell am instrument \ -w \ -r \ -e class "$1" \ "${TESTAPPID}/android.test.InstrumentationTestRunner" \ | tee ${TMPFILE} egrep "OK (.* tests?)" ${TMPFILE} } ${ADB} push "${APK}" "${DIR}" ${ADB} push "${TESTAPK}" "${TESTDIR}" ${ADB} shell pm install -r "${DIR}" ${ADB} shell pm install -r "${TESTDIR}" run_test rabbitescape.ui.android.DialogsTest run_test rabbitescape.ui.android.SmokeTest run_test rabbitescape.ui.android.TestAndroidConfigUpgradeTo1 </pre> And here is android-stop-emulator - it shuts down the emulator:</p> #!/bin/bash set -x set -u echo -e "auth $(cat ~/.emulator_console_auth_token)\nkill" \ | telnet localhost 5554 echo "Emulator stopped." </pre> Submitting a package to F-Droid Andy Balaam — Tue, 17 Jan 2017 21:58:29 +0000 Here's what I needed to get a dev environment for F-Droid</a> up and running on Ubuntu 16.10, using F-Droid server version 0.7.0 (commit id 8147f9235), so that I could submit a package for inclusion in the F-Droid repository.</p> Doing this is apparently the best way to get your own package into the repository, since you can provide a direct merge request for the metadata about your package, making it easy for the maintainers.</p> References:</p> F-Droid manual</a></li> F-Droid's CONTRIBUTING.md</a></li> F-Droid Quickstart</a></li> Contribute to F-Droid</a></li> </ul> Setup</h2> Before you start, manually install the Android SDK at ~/Android/Sdk/ - see Download Android Studio</a>. I installed version 23.0.2, but you will probably have a later one and may need to adjust the version number below.</p> Note: If you're only planning to contribute a package I'm fairly certain you don't need to install the Android SDK at all - you can just use the build server by running ./makebuildserver as I outline below.</p> Also before you start, if you want to contribute to the server project you should fork the F-Droid server project by going to gitlab.com/fdroid/fdroidserver</a> and clicking Fork. When you've done that, the git clone command below will need to change to clone your own fork via SSH, instead of the HTTPS one cloning the main repo that is shown below. Do the same for the F-Droid data project</a>, which holds the information about the packages in F-Droid. It's the data project where you will want to make changes if you are submitting a package.</p> Run these commands:</p> # Prerequisites sudo apt-get install openjdk-8-jdk subversion git git-svn mercurial bzr virtualbox ruby ruby-dev vagrant python3 python3-paramiko python3-pil python3-pyasn1-modules python3-clint vagrant plugin install vagrant-cachier ln -s ~/Android/Sdk/build-tools/23.0.2/aapt ~/Android/Sdk/platform-tools/ # Get the code cd ~/code git clone https://gitlab.com/fdroid/fdroidserver.git git clone https://gitlab.com/fdroid/fdroiddata.git echo 'export PATH="~/code/fdroidserver:$PATH"' >> ~/.profile source ~/.profile # Config cd fdroiddata cp ../fdroidserver/examples/config.py ./ chmod 0600 config.py echo 'sdk_path = "$HOME/Android/Sdk"' >> config.py # Set up Vagrant build box cd ../fdroidserver cp ./examples/makebuildserver.config.py ./ ./makebuildserver # Now wait several hours for this to finish # Build a package (the F-Droid client) just to check it works cd ../fdroiddata mkdir repo fdroid update --create-key fdroid readmeta # Should give no output if it worked fdroid build --server org.fdroid.fdroid # Again, this could take several hours! </pre> Make your own package</h2> Below I'm using my own package, Rabbit Escape, as an example. Its Android code is inside rabbit-escape-ui-android/app, whereas many programs will just have it directly in a directory called "app".</p> Rabbit Escape also builds non-Android-specific Java and other things during its build, so your package may be simpler.</p> cd ../fdroiddata fdroid import --url https://github.com/andybalaam/rabbit-escape --subdir rabbit-escape-ui-android/app </pre> Now edit the new file that was created - in my case it was called metadata/net.artificialworlds.rabbitescape.txt.</p> I set the following info:</p> Provides:net.artificialworlds.rabbitescape Categories:Games License:GPLv2+ Author Name:Andy Balaam and the Rabbit Escape developers Author Email:rabbitescape@artificialworlds.net Web Site:http://artificialworlds.net/rabbit-escape Source Code:https://github.com/andybalaam/rabbit-escape Issue Tracker:https://github.com/andybalaam/rabbit-escape/issues Name:Rabbit Escape Summary:Lemmings-like puzzle/action game Description: 140 levels of puzzling action! blah blah blah . Repo Type:git Repo:https://github.com/andybalaam/rabbit-escape Binaries:https://github.com/andybalaam/rabbit-escape/releases/download/v%v/rabbit-escape-%v.apk Build:0.10.1,101 commit=v0.10.1 subdir=rabbit-escape-ui-android/app gradle=paid build=cd ../.. && make android-pre Auto Update Mode:Version v%v Update Check Mode:Tags v\d+\.\d+(\.\d+)? Current Version:0.10.1 Current Version Code:101 </pre> For more info, see the F-Droid manual</a>.</p> And then checked it all worked with:</p> cd ../fdroiddata fdroid lint net.artificialworlds.rabbitescape fdroid readmeta fdroid checkupdates net.artificialworlds.rabbitescape fdroid rewritemeta net.artificialworlds.rabbitescape </pre> When I got the version stuff right the checkupdates command printed:</p> INFO: Processing net.artificialworlds.rabbitescape... INFO: ...updating to version 0.10.1 (101) INFO: Finished. </pre> Then I made sure it built OK:</p> fdroid build --server -v -l net.artificialworlds.rabbitescape </pre> Actually, it didn't work, and I decided I had to request a new package (sox) be installed in the build machine environment (in the fdroidserver project). The relevant commit is here: 19e372026</a>. Actually though, after discussion with the F-Droid devs</a> we agreed I'd be better off not using sox during the build, so I didn't need this.</p> Side note: if you do end up needing to modify the build environment for F-Droid, make sure you delete the fdroiddata/buildserver directory when you re-try your build. That one had me stuck for a few days, with the old environment being used no matter what caches I cleared and vagrant commands I ran.</p> And now I was ready to request my package be included in F-Droid by committing and pushing the changes I had made to the fdroiddata project to my forked repo, and clicking the Merge Request button in the gitlab UI. My merge request is here: gitlab.com/fdroid/fdroiddata/merge_requests/1965</a></p> Comments</h2> Kar 2017-08-18</span> </h3> Note that you need a lot of space to build this.</p> </div> Mirek 2020-01-27</span> </h3> How long should the "fdroid build —server org.fdroid.fdroid" step take? It seems like it’s building every possible version of fdroid server in a virtual machine. Is that the desired outcome? Or should I just cancel after a few builds?</p> Could you clarify this step in the official guide?</p> Thanks in advance.</p> </div> Andy Balaam 2020-01-28</span> </h3> For me it took a very</em> long time (maybe even days!) for some steps, so I’d recommend leaving it and seeing what happens!</p> </div> Andy Balaam 2020-01-28</span> </h3> I added a note that the last step could also take several hours. Thanks Mirek!</p> </div> Resources for year 6 teachers on coding and programming Andy Balaam — Mon, 09 Jan 2017 16:23:47 +0000 I have been introducing some year 6 (UK) teachers to coding by showing them how to lay out a simple web page by writing HTML. I promised I would find some links to resources for them, so here it is:</p> HTML and JavaScript</h2> My examples of how to write HTML are at github.com/andybalaam/html-examples</a></p> There are several web sites that allow you to experiment with writing HTML and JavaScript and seeing the results immediately:</p> jsfiddle</a></li> repl.it</a> - this also has a "classroom" mode here: repl.it/site/classrooms</a></li> </ul> I also made some videos about how to make a snowflake animation in both JavaScript and Scratch here: Snowflake Christmas card</a>.</p> Raspberry Pi</h2> The Raspberry Pi is a cheap education-focussed computer that looks like a piece of circuit board the size of a credit card.</p> Their educational resources are at: raspberrypi.org/resources</a>.</p> There are lots of great videos about how to do different things with the Raspberry Pi, including the ones by The Raspberry Pi Guy</a>.</p> There are also my (boring, but comprehensive) videos teaching you to write a simple game in Python, from a starting point of no programming experience at all: My First Raspberry Pi Game</a>.</p> Graphical programming</h2> There are several tools and sites for learning programming by dragging and dropping blocks instead of typing code:</p> Scratch</a> - creative, unguided, a bit old-fashined looking but tried-and-trusted</li> code.org</a> - fun, attractive guided tasks featuring Disney characters, Minecraft etc.</li> blockly</a> - guided tasks with good progression</li> codeforlife.education</a> - I've not used this, but it looks like it could have potential</li> </ul> Other languages</h2> Hello, Ruby</a></li> Java - Alice</a></li> Swift - Playgrounds</a></li> </ul> Update:</strong> there is a huge list of similar resources here: Beginner's Resources to Learn Programming Languages</a>. Thanks to Erica and Sarah for emailing me the link!</p> Update 2:</strong> some more useful links: Learn to code from home</a> and Computer Language for beginners: HTML</a>. Thanks to Sarah for the suggestions, and Stacey for sending them to me!</p> Update 3:</strong> a comprehensive page of links to resources for learning how to code Virtual Reality</a>. Thanks to Katie for the suggestion, and Yazmin for sending it. Keep on being a computer nerd!</p> Comments</h2> Why I won’t link to AI resources – Andy Balaam's Blog 2024-05-24</span> </h3> […] received a very kind email today from someone who had found my page Resources for year 6 teachers on coding and programming helpful, and wanted to suggest another link for me to add, about AI […]</p> </div> Dana 2025-01-20</span> </h3> This is such a valuable resource!</a></p> </div> andyb 2025-01-21</span> </h3> Thank you!</p> </div> Setting up a sane Maven project Andy Balaam — Fri, 23 Dec 2016 16:07:17 +0000 Today I have spent hours trying to get wrangle Maven into letting me set up a sane Java project. The hardest parts were enforcing warnings-as-errors, and getting Maven to shut up a bit.</p> Some code that warns</h2> My first problem was writing some Java code that causes the compiler to emit warnings. For some reason I can't understand, my Java 1.8 compiler was not emitting warnings (even with -Xlint:all) for unused variables, use of deprecated methods, or unknown @SuppressWarnings types (suggestions from SO 1752607</a>).</p> Instead, I had to use an unnecessary cast:</p> $ cat src/tast/java/ExampleTest.java public class ExampleTest { public void warn() { String fixmePlease = (String)"Hello"; } }</pre> Now, finally, I got a warning:</p> $ javac -Xlint:all src/test/ExampleTest.java src/test/ExampleTest.java:3: warning: [cast] redundant cast to String String s = (String) "Hello!"; ^ 1 warning</pre> Maven compiler settings for warnings-as-errors</h2> I tried a vast set of combinations of properties like maven.compiler.failOnWarning and maven.compiler.fork (as specified in the maven compiler plugin docs</a>) before giving up on properties. Making a property called maven.compiler.failOnWarning seems to have no effect whatsoever, ever.</p> So I decided I must try the (very verbose) plugin tag containing a configuration tag, as also specified in the maven compiler plugin docs</a>. After a lot of messing about with flags that seemed incompatible, and Maven silently ignoring things it didn't understand, I came to a working config.</p> On the way, I discovered that setting the "fork" property to true is a non-starter, because Maven simply drops the compiler output in that case, meaning you can't see what is going wrong when it does.</p> Finally, I had a pom file like this:</p> cat pom.xml <?xml version="1.0"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>Example</artifactId> <packaging>jar</packaging> <version>0.0.1</version> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.6.0</version> <configuration> <source>1.8</source> <target>1.8</target> <failOnWarning>true</failOnWarning> <showWarnings>true</showWarnings> <compilerArgs> <arg>-Xlint:all</arg> </compilerArgs> </configuration> </plugin> </plugins> </build> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> </dependencies> </project> </pre> Which manages successfully to tell the compiler to show all warnings, and to fail when it sees one. (Thanks to SO 9192613</a>, among others.)</p> I dare not change anything, for fear that it will stop working again without any explanation.</p> Quieting Maven</h2> If you tell Maven to be quiet with -q or -e it will merrily fail the build because of a warning, but not tell you what the warning was.</p> Maven does not appear to have a command line option to set the log level to display warnings and errors only, but you can force it to do so by setting the environment variable MAVEN_OPTS like this:</p> MAVEN_OPTS=MAVEN_OPTS=-Dorg.slf4j.simpleLogger.defaultLogLevel=warn mvn clean test-compile</pre> (Thanks to SO 4782089</a>.)</p> And, with some guesswork (partly based on Configuring Maven</a>) I found that if I put something similar in .mvn/jvm.config I didn't have to type it every time:</p> $ cat .mvn/jvm.config -Dorg.slf4j.simpleLogger.defaultLogLevel=warn </pre> Failing on warnings, and seeing them!</h2> I don't know whether to feel triumphant or defeated, but, it works!</p> $ mvn clean test-compile [WARNING] COMPILATION WARNING : [WARNING] src/test/java/ExampleTest.java:[3,20] redundant cast to java.lang.String [ERROR] COMPILATION ERROR : [ERROR] src/test/java/ExampleTest.java: warnings found and -Werror specified [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.0:testCompile (default-testCompile) on project unmatcheddrrepair: Compilation failure [ERROR] src/test/java/ExampleTest.java: warnings found and -Werror specified [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException</pre> I wish I could suppress double-reporting of the error, and the extra rubbish at the end (and the super-long absolute paths of each file that push the actual errors off the side of the screen), but at this point, I must try and do what I was trying to do in the first place.</p> Guess how much I am liking Maven at this point.</p> Writing a unit test in Elm Andy Balaam — Tue, 01 Nov 2016 22:05:40 +0000 Series: Snake in Elm</a>, Elm makes me happy</a>, Elm Basics</a>, Elm Unit Test, Elm JSON</a></p> I've been having fun</a> with Elm</a> programming recently. Elm is a replacement for JavaScript that is pure functional and highly practical.</p> Here's how to go from nothing installed at all to writing a unit test that passes, in just over 10 minutes.</p> </iframe> The source code is here: github.com/andybalaam/elm-unit-test-example</a></p> Comments</h2> Snake in Elm – Andy Balaam's Blog 2016-11-01</span> </h3> […] Writing a unit test in Elm […]</p> </div> Elm Basics Video – Andy Balaam's Blog 2018-11-10</span> </h3> […] Snake in Elm, Elm makes me happy, Elm Basics, Elm Unit Test, Elm […]</p> </div> Elm makes me happy (updated for Elm 0.19) video – Andy Balaam's Blog 2018-11-10</span> </h3> […] Snake in Elm, Elm makes me happy, Elm Basics, Elm Unit Test, Elm […]</p> </div> How to write a programming language - Part 3, The Evaluator Andy Balaam — Sun, 16 Oct 2016 23:18:24 +0000 Series: Lexer</a>, Parser</a>, Evaluator.</p> Finally, we get onto the actual magic of the little language I wrote (Cell</a>) - the evaluator, which takes in syntax trees and finds their real value, in the context of the "environment": the symbols that are defined around it.</p> </iframe> Slides: How to write a programming language - Part 3, The Evaluator</a></p> If you want to, you can Support me on Patreon</a>.</p> Basic Haskell project setup (unit tests, code, formatting) Andy Balaam — Fri, 14 Oct 2016 01:56:17 +0000 To start a programming project, we need to be able to build, format code, and run unit tests.</p> Here's what I have found makes a sensible starting point for a Haskell project. Full code: hunit-example</a>.</p> To build and run tests, just do: make setup make test</pre></p> I tend to use a Makefile for all the commands I need to remember. The most important part here is the "setup" target which installs the cabal-install package (Cabal</a> is a package manager for Haskell), then uses Cabal to install hindent</a>, and finally installs all the dependencies of our actual project (which we specify in a .cabal file, shown further down). The format target uses hindent-all, which is shown below.</p> Update: added "happy" as a prerequisite - needed by hindent.</strong></p> Makefile: all: test</p> test: format cabal test</p> build: format cabal build</p> format: ./hindent-all</p> clean: cabal clean</p> setup: sudo apt-get install cabal-install happy cabal update cabal install hindent cabal install --run-tests</pre></p> Our production code is very simple - just two functions.</p> HUnitExample.hs:module HUnitExample where</p> double x = x * 2</p> half x = x / 2</pre></p> Our tests import our production code, and use tasty</a> (which includes HUnit</a>) to check it works.</p> test/Tests.hs: import HUnitExample (double, half) import Test.Tasty (defaultMain, testGroup) import Test.Tasty.HUnit (assertEqual, testCase)</p> main = defaultMain unitTests</p> unitTests = testGroup "Unit tests" [doublingMakesNumbersBigger, halvingMakesNumbersSmaller]</p> doublingMakesNumbersBigger = testCase "Double of 4 is 8" $ assertEqual [] 8 (double 4)</p> halvingMakesNumbersSmaller = testCase "Half of 9 is 4" $ assertEqual [] 4 (half 9)</pre></p> Each test is a testCase that uses assertions like assertEqual to verify the code. HUnit also provides lots of cryptic operators for obfuscating your tests.</p> Many projects will want to add property-based testing with QuickCheck</a> or SmartCheck</a> or similar.</p> The magic that makes all this work is the Cabal file, which can have any name that ends in ".cabal".</p> hunit-example.cabal: Name: hunit-example Version: 1.0.0 cabal-version: >= 1.8 build-type: Simple</p> Library Exposed-Modules: HUnitExample Build-Depends: base >= 3 && < 5</p> Test-Suite test-hunit-example type: exitcode-stdio-1.0 hs-source-dirs: tests Main-is: Tests.hs Build-Depends: base >= 3 && < 5 , tasty , tasty-hunit , hunit-example</pre></p> You will normally want lots more properties than this, but this is what I think is a minimal example. It defines the project properties, the production library module we are building, and a test suite that tasty uses to know what to run. Note that the test suite must depend on the production code module so you are allowed to import it.</p> To do code formatting, I made this bash script:</p> hindent-all:#!/bin/bash</p> Format all .hs files in the current directory tree with hindent</h1> HINDENT=${HOME}/.cabal/bin/hindent</p> function all_hs_files() { find ./ -name dist -prune -o -name "*.hs" -print }</p> for FILE in $(all_hs_files); do { NEWFILE=${FILE}.formatted cat ${FILE} | ${HINDENT} > ${NEWFILE}</p> if diff -q ${FILE} ${NEWFILE} >/dev/null; then { rm ${NEWFILE} } else { echo "hindent updated ${FILE}" mv ${NEWFILE} ${FILE} }; fi </code></pre> }; done</pre></p> I would be grateful if someone pointed out I didn't need to do that because someone has done it for me. Note: I chose not to use hfmt</a> because it hard-codes a long line length, whereas hindent behaves how I like with no configuration.</p> Last but not least, ignore the directory created by Cabal.</p> .gitignore: dist</pre></p> So, when we run we see this:</p> $ make test ... Running 1 test suites... Test suite test-hunit-example: RUNNING... Unit tests Double of 4 is 8: OK Half of 9 is 4: FAIL expected: 4.0 but got: 4.5 1 out of 2 tests failed (0.00s) ...</pre> Looks like we've got a bug to fix...</p> How to write a programming language - Part 2, The Parser Andy Balaam — Sun, 09 Oct 2016 23:47:28 +0000 Series: Lexer</a>, Parser, Evaluator</a></p> My little programming language, Cell</a> (Cell Elementary Learning Language) is designed to be simple. I want to use it to explain how to write a programming language. The parser is only 81 lines long</a>, so hopefully it's not too hard to understand.</p> Here's the explanation of the parser, which is the second part of a compiler or interpreter.</p> </iframe> Slides: How to write a programming language - Part 2, The Parser</a></p> If you want to, you can Support me on Patreon</a>.</p> Comments</h2> How to write a programming language – Part 1, The Lexer – Andy Balaam's Blog 2016-10-09</span> </h3> […] How to write a programming language – Part 2, The Parser […]</p> </div> Mousedown events delayed or sluggish on mobile Andy Balaam — Fri, 30 Sep 2016 20:47:22 +0000 I am learning about Elm</a> and web apps by writing a little game called sootl</a>.</p> It's all SVGs animated with requestAnimationFrame()</a>, and you control your player by clicking or touch the screen.</p> Everything seemed good on desktop browsers, but on mobile it took an age for the mousedown event to trigger when you touched.</p> Update: Thanks to @zatnosk on mastodon.social</a> for letting me know there is a better way: 300ms tap delay, gone away</a>. Short answer: add a <meta name="viewport" content="width=device-width"> tag inside your <head> tag.</strong></p> It turns out</a> that under certain circumstances, some mobile devices wait after the first touch happens to see whether you are going to swipe before they emit the mousedown event.</p> So, the short answer was to listen for touchstart events as well as mousedown ones. In Elm that looked like this change: e77e055e470c38489657ecaf5edc54a4e5f85782</a>.</p> How to write a programming language - Part 1, The Lexer Andy Balaam — Sat, 24 Sep 2016 07:39:14 +0000 Series: Lexer, Parser</a>, Evaluator</a></p> I wrote a little programming language, Cell</a> which is supposed to be simple enough to help explain how a programming language works.</p> Here's the explanation of the lexer, which is the first part of a compiler or interpreter.</p> </iframe> Slides: How to write a programming language - Part 1, The Lexer</a></p> If you want to, you can Support me on Patreon</a>.</p> Comments</h2> Neto chidozie 2020-01-16</span> </h3> You’re such a great teacher. Nice one, please do more videos on compilers</p> </div> Andy Balaam 2020-01-16</span> </h3> Thanks!</p> </div> Simple example of Netty 4 usage Andy Balaam — Thu, 15 Sep 2016 15:56:10 +0000 I feel the title of this post over-promises, since I was not able to make an example that seemed simple to me.</p> Anyway, here is a near-minimal example of how to use Netty</a> to make a server that shouts back at you whatever you say:</p> NettyExample.java: import io.netty.bootstrap.ServerBootstrap; import io.netty.buffer.ByteBuf; import io.netty.buffer.Unpooled; import io.netty.channel.ChannelHandlerContext; import io.netty.channel.ChannelInboundHandlerAdapter; import io.netty.channel.ChannelInitializer; import io.netty.channel.EventLoopGroup; import io.netty.channel.nio.NioEventLoopGroup; import io.netty.channel.socket.nio.NioServerSocketChannel; import io.netty.channel.socket.SocketChannel; import io.netty.util.CharsetUtil; import io.netty.util.ReferenceCountUtil; import java.nio.charset.Charset;</p> class NettyExample { public static void main( String[] args ) throws Exception { EventLoopGroup bossGroup = new NioEventLoopGroup(); EventLoopGroup workerGroup = new NioEventLoopGroup(); try { new ServerBootstrap()</span> .group( bossGroup, workerGroup ) .channel( NioServerSocketChannel.class ) .childHandler( new Init() ) .bind( 1337 )</span>.sync().channel().closeFuture().sync(); } finally { bossGroup.shutdownGracefully(); workerGroup.shutdownGracefully(); } }</p> private static class Init extends ChannelInitializer<socketchannel> { @Override public void initChannel( SocketChannel ch ) throws Exception { ch.pipeline().addLast( new ShoutyHandler() ); } } private static class ShoutyHandler extends ChannelInboundHandlerAdapter { @Override public void channelRead( ChannelHandlerContext ctx, Object msg ) { try { Charset utf8 = CharsetUtil.UTF_8; String in = ( (ByteBuf)msg ).toString( utf8 ); <span style="color:red;">String out = in.toUpperCase(); // Shout!</span> ctx.writeAndFlush( Unpooled.copiedBuffer( out, utf8 ) ); } finally { ReferenceCountUtil.release( msg ); } } @Override public void exceptionCaught( ChannelHandlerContext ctx, Throwable cause ) { cause.printStackTrace(); ctx.close(); } } </code></pre> }</socketchannel></pre></p> The lines that actually do something useful are highlighted in red. If anyone knows how to make it shorter, please comment below. It seems a lot to me.</p> To run this, do:</p> sudo apt-get install openjdk-8-jdk wget 'http://search.maven.org/remotecontent?filepath=io/netty/netty-all/4.1.5.Final/netty-all-4.1.5.Final.jar -O netty-all-4.1.5.Final.jar' javac -Werror -cp netty-all-4.1.5.Final.jar:. NettyExample.java && java -cp netty-all-4.1.5.Final.jar:. NettyExample</pre> Then in another terminal:</p> echo "Hello, world" | nc localhost 1337</pre> and observe the response:</p> HELLO, WORLD</pre> Comparison with Node.js</h2> Just for comparison, here is an approximate equivalent in Node.js:</p> shouty.js: var net = require('net');</p> var server = net.createServer</span>( function( socket ) { socket.setEncoding('utf8'); socket.on( 'data', function( data ) { socket.end( data.toUpperCase()</span> ); } ) } );</p> server.listen( 1337, "localhost" );</span></pre></p> To run it, do:</p> sudo apt-get install nodejs-legacy node shouty.js</pre> Then in another terminal:</p> echo "Hello, world" | nc localhost 1337</pre> and observe the response:</p> HELLO, WORLD</pre> Elm resizeable SVG canvas filling the screen Andy Balaam — Fri, 26 Aug 2016 02:10:43 +0000 I am toying with writing an SVG-based game</a> in (exciting-looking JavaScript-replacement) Elm</a>, and I wanted an SVG that filled the whole screen and resized when the screen resized. I found it harder than I expected, so here is what I came up with for your information and comment.</p> Try the demo</a>.</p> Because I was using Html.App.programWithFlags</a> I was not able to shortcut the process and use just elm-reactor - I needed to create an HTML file and compile my code with elm-make.</p> index.html sets up a full-screen app and passes in the window size:</p> <!DOCTYPE HTML> <html> <head> <meta charset="UTF-8"/> <title>Sootl</title> <script src="sootl.js"></script> <style> html, body, svg { margin: 0px; padding: 0px; border: 0px; overflow: hidden; } </style> </head> <body> </body> <script> var app = Elm.Main.fullscreen( { width: window.innerWidth, height: window.innerHeight } ); </script> </html></pre> elm-package.json</a> requires the Html, Svg and Window packages:</p> { "version": "1.0.0", "summary": "Stay out of the light!", "repository": "https://github.com/andybalaam/sootl.git", "license": "GPL2", "source-directories": [ "." ], "exposed-modules": [], "dependencies": { "elm-lang/core": "4.0.5 <= v < 5.0.0", "elm-lang/html": "1.1.0 <= v < 2.0.0", "elm-lang/svg": "1.1.1 <= v < 2.0.0", "elm-lang/window": "1.0.0 <= v < 2.0.0" }, "elm-version": "0.17.1 <= v < 0.18.0" }</pre> Main.elm</a> contains the Elm code, which starts off with the window size from JavaScript, and then listens to resize events using the Window module.</p> import Html exposing (Html) import Html.App exposing (programWithFlags) import Svg exposing (..) import Svg.Attributes exposing (..) import Window type alias Flags = { width : Int , height : Int } type Msg = Resize Int Int type alias Model = { screen : { width : Int , height : Int } } init : Flags -> (Model, Cmd Msg) init flags = ( { screen = { width = flags.width , height = flags.height } } , Cmd.none ) view : Model -> Html Msg view model = let sw = model.screen.width - 0 sh = model.screen.height - 0 in svg [ width <| toString sw , height <| toString sh ] [ rect [ x "0" , y "0" , width (toString model.screen.width) , height (toString model.screen.height) , fill "#eeffee" ] [] , text' [ x <| toString <| sw / 2 , y <| toString <| sh / 2 , fontSize <| toString <| sh / 10 , textAnchor "middle" ] [ text ((toString model.screen.width) ++ ", " ++ (toString model.screen.height)) ] ] update : Msg -> Model -> (Model, Cmd Msg) update msg model = let m = case msg of Resize w h -> {model | screen = {width = w, height = h}} in (m, Cmd.none) subscriptions : Model -> Sub Msg subscriptions model = Window.resizes (\size -> Resize size.width size.height) main = programWithFlags { init = init , view = view , update = update , subscriptions = subscriptions }</pre> I installed all the packages with:</p> elm-package install</pre> Then I compiled the code with:</p> elm-make --output=sootl.js Main.elm</pre> Now I launched elm-reactor:</p> elm-reactor</pre> And navigated my browser to http://localhost:8000/index.html</a> to see it working.</p> Ambiguous names in Java due to non-normalised unicode - but all OK in Python Andy Balaam — Thu, 11 Aug 2016 08:47:40 +0000 In Java and several other languages, identifiers (e.g. method names) are allowed to contain unicode characters.</p> Unfortunately, some combinations of unicode characters are logically identical. For example, Ã¡ (one character: Latin Small Letter a with Acute U+00E1</a>) is the same as aÌ (two characters: Latin Small Letter A U+0061</a>, and Non-spacing Acute Accent U+0301</a>). These combinations are not just similar - they are identical by definition.</p> Java does not do any normalisation</a> on your code before compiling it, so two identifiers containing equivalent but different unicode combinations are considered different (ref: JLS 7 section 3.8</a>).</p> $ cat U.java public class U { static String \u00e1() { return "A WITH ACUTE"; } static String a\u0301() { return "A + NON-SPACING ACUTE"; } public static void main(String[] a) { System.out.println(Ã¡()); System.out.println(aÌ()); } } $ javac U.java && java U A WITH ACUTE A + NON-SPACING ACUTE </pre> We can define and use two functions called Ã¡ and aÌ and they are totally independent entities.</p> But don't do this.</p> Python 3 also allows unicode characters in identifiers, but it avoids the above problem by normalising them (ref: Python 3 Reference, section 2.3</a>):</p> $ cat U.py #!/usr/bin/env python3 def Ã¡(): print("A WITH ACUTE") def aÌ(): print("A + NON-SPACING ACUTE") Ã¡() aÌ() $ hexdump -C U.py 23 21 2f 75 73 72 2f 62 69 6e 2f 65 6e 76 20 70 |#!/usr/bin/env p| 79 74 68 6f 6e 33 0a 0a 64 65 66 20 c3 a1</span> 28 29 |ython3..def ..</span>()| 3a 0a 20 20 20 20 70 72 69 6e 74 28 22 41 20 57 |:. print("A W| 49 54 48 20 41 43 55 54 45 22 29 0a 0a 64 65 66 |ITH ACUTE")..def| 20 61 cc 81</span> 28 29 3a 0a 20 20 20 20 70 72 69 6e | a..</span>():. prin| 74 28 22 41 20 2b 20 4e 4f 4e 2d 53 50 41 43 49 |t("A + NON-SPACI| 4e 47 20 41 43 55 54 45 22 29 0a 0a c3 a1</span> 28 29 |NG ACUTE")....</span>()| 0a 61 cc 81</span> 28 29 0a 0a |.a..</span>()..| $ ./U.py A + NON-SPACING ACUTE A + NON-SPACING ACUTE </pre> (Legend: A WITH ACUTE, A + NON-SPACING ACUTE) The second definition overwrites the first because they are considered identical. You can call it via either way of saying its name. Both ways of working are scary, but I'd definitely choose the Python 3 way if I had to.</p> Gracefully shutting down Firefox, to avoid the crash/session dialog Andy Balaam — Thu, 16 Jun 2016 12:47:22 +0000 I normally have several Firefox profiles open, and when I log out without closing the Firefox windows I get the "session restore" dialog on my next login.</p> This is because of Bug 336193</a> which says that Firefox should shut down gracefully when it receives a SIGTERM signal, which is what happens when I log out (in Ubuntu MATE anyway). At the moment, it shuts down ungracefully, meaning it is treated as if Firefox crashed when it restarts.</p> So, instead of my normal shutdown button, I have one that launches a script that closes all my Firefoxes first, like this:</p> #!/bin/bash # Prerequisites: # sudo apt-get install sysvinit-utils xdotool set -x set -u function close_firefoxes() { local PID local WID PID=$(pidof -s firefox) FOUND=$? while [ $FOUND ]; do { WID=$(xdotool search --pid "$PID" | tail -1) xdotool windowactivate --sync "$WID" xdotool key --clearmodifiers 'ctrl+q' sleep 1 PID=$(pidof firefox) FOUND=$? }; done } close_firefoxes mate-session-save --shutdown-dialog</pre> Snake in Python 3 + Qt 5 Andy Balaam — Fri, 10 Jun 2016 02:09:13 +0000 Series: Groovy</a>, Ruby</a>, BASIC</a>, Dart</a>, Elm</a>, Python3+Qt5</p> I'm writing the game Snake in lots of programming languages, for fun, and to try out new languages.</p> Python 3 broke compatibility to fix some mistakes - was it worth it? Qt 5 continues to offer more and more features - can it win me over?</p> </iframe> Slides: Snake in Python 3 + Qt 5</a></p> If you want to, you can Support me on Patreon</a>.</p> Basic Concepts of Christianity video Andy Balaam — Wed, 18 May 2016 08:35:45 +0000 Update: my ideas have changed quite a bit since I recorded this, especially about the idea that "every crime requires a punishment". If you'd like to explore more, try this sermon by my friend Jörg</a>.</strong></p> As a Christian myself I thought it might be interesting to try and outline what Christians believe, as I understand it. This is all from a personal perspective, but I hope it's helpful:</p> </iframe> Slides: Basic Concepts of Christianity</a></p> Elm Basics Video Andy Balaam — Fri, 13 May 2016 18:58:20 +0000 Series: Snake in Elm</a>, Elm makes me happy</a>, Elm Basics, Elm Unit Test</a>, Elm JSON</a></p> A lot of the documentation about the new language I am really excited about, Elm</a> focusses on the Elm Architecture</a> and the ideas of Functional Reactive Programming, and while those are the fundamental reasons I am interested in Elm, I found myself stuck on the syntax quite often.</p> So, in this video I review almost all the syntax and basic ideas you need to be able to read and write Elm code, treating it as a general programming language.</p> </iframe> Update: forgot to add the slides - here they are:</strong> Slides: Elm Basics</a></p> Comments</h2> Snake in Elm – Andy Balaam's Blog 2016-06-10</span> </h3> […] Elm Basics Video […]</p> </div> How to analyse a .phd heap dump from an IBM JVM Andy Balaam — Tue, 23 Feb 2016 10:31:57 +0000 If you have been handed a .phd file which is a dump of the heap of an IBM Java virtual machine, you can analyse it using the Eclipse Memory Analyzer Tool</a> (MAT), but you must install the IBM Monitoring and Diagnostic Tools</a> first.</p> Download MAT from eclipse.org/mat/downloads.php</a>. I suggest the Standalone version.</p> Unzip it and run the MemoryAnalyzer executable inside the zip. Add an argument to control how much memory it gets e.g. to give it 4GB:</p> ./MemoryAnalyzer -vmargs -Xmx4g</pre> Once it's started, go to Help -> Install new software.</p> Next to "Work with" paste in the URL for the IBM Developer Toolkit update site: http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/runtimes/tools/dtfj/</p> Click Add...</p> Type in a name like "IBM Monitoring and Diagnostic Tools" and click OK.</p> In the list below, an item should appear called IBM Monitoring and Diagnostic Tools. Tick the box next to it, click Next, and follow the wizard to accept the license agreements and install the toolkit.</p> Restart Eclipse when prompted.</p> Choose File -> Open Heap Dump and choose your .phd file. It should open in MAT and allow you to figure out who is using all that memory.</p> Comments</h2> Stevo 2016-09-29</span> </h3> Hey dude,</p> Very helpful guide.</p> </div> Douglas Cardoso 2017-03-15</span> </h3> Very nice buddy! Thank you!</p> =)</p> If need any help on HEAPDUMP and JAVACORE for WebSphere products, please contact me!</p> https://www.linkedin.com/in/dougcardoso21/</a></p> </div> Andy Balaam 2017-04-10</span> </h3> Thanks Douglas!</p> </div> Ryan Cox 2018-08-10</span> </h3> Thanks for this… IBM product is garbage (no pun intended).</p> </div> jmjeon 2018-11-29</span> </h3> Thanks you !!</p> </div> Erik Visser 2018-12-10</span> </h3> Thanks!</p> </div> Poonam Aswani 2019-02-22</span> </h3> When I tried to update ini file to 4g, it did not open MAT. I needed to reset to what it was that is 1024m and then when I opened this phd file, it gave an error that Error opening heap dump is encountered. Does someone know what to do?</p> </div> Stephen Nagle 2020-01-29</span> </h3> Very helpful , thanks.</p> Hi Poonam If you are on windows, type cmd in the search , go into command prompt. In the command prompt , change directory (cd) to the directory that MemoryAnalyser.exe is in. Then type MemoryAnalyzer -vmargs -Xmx4g and press enter.</p> </div> ¿Cómo crear un volcado de almacenamiento dinámico compatible con OpenJ9 a través de API? – stack 2022-09-07</span> </h3> […] Para detalles: https://www.artificialworlds.net/blog/2016/02...</a> […]</p> </div> Elm makes me happy (old: Elm 0.17) Andy Balaam — Mon, 08 Feb 2016 14:53:22 +0000 Updated version: Elm makes me happy (updated for Elm 0.19)</a></p> Elm lets you write interactive web sites in a really clean and safe-feeling way, which is a contrast to the feeling I get when using JavaScript and HTML. It's functional, but instead of that making everything seem hard, it makes me happy.</p> </iframe> Slides: Elm makes me happy</a></p> Comments</h2> Snake in Elm – Andy Balaam's Blog 2016-05-18</span> </h3> […] Elm makes me happy […]</p> </div> What's new in Java 8 Andy Balaam — Tue, 19 Jan 2016 08:40:36 +0000 Java 8 makes some functional programming ideas available to the rest of us, using Lambda, function references and a streaming style (map, filter, replace etc.):</p> </iframe> Slides: What's new in Java 8</a></p> Which Raspberry Pi photo was funniest? Andy Balaam — Mon, 18 Jan 2016 05:03:54 +0000 We had a great day at the Egham Raspberry Pi Jam</a>, and Rabbit Escape</a> and our Photo Booth</a>:</p> </p> seemed to go down well:</p> </p> But which photo was funniest? Here are some of the entries (I had to choose kids' ones without faces to go on here, but there were some other great ones!):</p> </p> But the winner has to be the eyeball wearing a hat!</p> </p> Thanks everyone, see you next time!</p> Raspberry Pi Funniest Photo Game Andy Balaam — Sat, 16 Jan 2016 17:03:16 +0000 For our latest Egham Raspberry Pi Jam</a>, we worked on a photo booth program, using the Raspberry Pi camera. Here's how we did it.</p> Downloads: funniest-photo-game.odp</a>, photo-booth.zip</a>.</p> Update 1: fixed a bug where it tried to create a directory that already exists</strong></p> Update 2: see the winning photo</a>!</strong></p> Equipment</h3> Raspberry Pi (1 or 2)</a></li> A USB mouse & keyboard</li> HDMI screen with HDMI cable</li> Raspberry Pi Camera Module</a></li> </ul> What we did</h3> Made and decorated a box that held the Pi and camera steady for taking photos: (Note the holes for the wires!)</li> Created some costumes (see "Costumes" below)</li> Wrote a Python program (see "Program" below) to display pictures on top of the camera picture, and take photos</li> Wrote up the instructions and competition rules</a></li> </ul> Costumes</h3> We made lots of costumes that show up over the picture. They all needed to be 1280x720 pixels, PNG files that use Indexed Color mode. (In Gimp we clicked "Image", then "Mode", then "Indexed Color" before choosing "File" then "Export" or "Overwrite" to save them as .png files.)</p> Here are some examples:</p> </a></p> </a></p> Once we'd made the costumes we put them in a directory called "costumes" next to the program file, photo-booth.py.</p> Setup</h3> To get the Pi ready to run our program we needed to type these commands:</p> sudo apt-get update sudo apt-get install python-picamera sudo apt-get install python-imaging </pre> Once we'd done this, we created the Python program and directories described in the next section, and then we ran the program with:</p> python photo-booth.py </pre> Program</h3> We made a directory to hold our program on the Desktop of our Raspberry Pi, which we called "photo-booth". Inside that, we made a "costumes" directory containing our costumes, and a "gallery" directory to hold the saved photos. Also inside "photo-booth" we saved this code as "photo-booth.py":</p> import io import picamera from PIL import Image import time import pygame import subprocess import os class PhotoBooth: def __init__( self, camera ): self.camera = camera self.costumes = os.listdir( 'costumes' ) self.current_overlay = None self.cos_num= 0 self.current_costume = None self.set_up_camera() self.change_costume(0) def set_up_camera(self): self.camera.resolution = ( 1280, 720 ) self.camera.framerate = 24 self.camera.start_preview() def change_costume(self, change): self.cos_num += change self.cos_num = self.cos_num%len(self.costumes) self.current_costume = Image.open( 'costumes/'+self.costumes[self.cos_num]) self.overlay(self.current_costume, 128) def overlay(self, image, alpha): pad = Image.new('RGB', ( ((image.size[0] + 31) // 32) * 32, ((image.size[1] + 15) // 16) * 16, )) pad.paste(image, (0, 0)) if self.current_overlay is not None: self.camera.remove_overlay(self.current_overlay) self.current_overlay = camera.add_overlay(pad.tostring(), size=image.size) self.current_overlay.alpha = alpha self.current_overlay.layer = 3 def take_photo(self): stream = io.BytesIO() camera.capture( stream, format='jpeg' ) stream.seek( 0 ) captimg = Image.open( stream ) imgrgba = self.current_costume.convert("RGBA") captimg.paste( imgrgba, ( 0, -100 ), imgrgba ) self.overlay(captimg, 255) captimg.save( 'gallery/photo%d.png' % time.time().real ) self.overlay(self.current_costume, 128) def run(self): while True: for event in pygame.event.get(): if event.type == pygame.QUIT: return if event.type == pygame.KEYDOWN: if event.key == pygame.K_LEFT: self.change_costume(1) if event.key == pygame.K_RIGHT: self.change_costume(-1) if event.key == pygame.K_SPACE: self.take_photo() if event.key == pygame.K_ESCAPE: return if event.key == pygame.K_g: subprocess.call('ls gallery/*.png | head -1 | xargs xdg-open', shell = True) return if event.type == pygame.MOUSEBUTTONDOWN: self.take_photo() pygame.init() pygame.display.set_mode() if not os.path.exists( 'gallery' ): os.makedirs( 'gallery' ) with picamera.PiCamera() as camera: booth = PhotoBooth( camera ) booth.run() </pre> Download code and costumes</h3> You can download the code and costumes we used here: photo-booth.zip</a>.</p> If you have any questions, feel free to leave a comment below!</p> Snake in Elm Andy Balaam — Thu, 07 Jan 2016 08:16:11 +0000 Snake Series: Groovy</a>, Ruby</a>, BASIC</a>, Dart</a>, Elm, Python3+Qt5</a></p> Elm Series: Snake in Elm, Elm makes me happy</a>, Elm Basics</a>, Elm Unit Test</a>, Elm JSON</a></p> I'm writing the game Snake in lots of programming languages, for fun, and to try out new languages.</p> Elm brings the comfortable feeling and scary syntax of a pure functional language to writing dynamic web sites.</p> </iframe> Slides: Snake in Elm</a></p> If you want to, you can Support me on Patreon</a>.</p> Comments</h2> Elm Basics Video – Andy Balaam's Blog 2016-05-18</span> </h3> […] Snake in Elm, Elm makes me happy, Elm […]</p> </div> Elm makes me happy – Andy Balaam's Blog 2016-05-18</span> </h3> […] Snake in Elm, Elm makes me happy, Elm […]</p> </div> Writing a unit test in Elm – Andy Balaam's Blog 2016-11-01</span> </h3> […] Snake in Elm, Elm makes me happy, Elm Basics, Elm Unit […]</p> </div> Out-of-school education settings Andy Balaam — Mon, 04 Jan 2016 20:59:28 +0000 Here is the comment I sent to the Government consultation on out-of-school education settings</a>. Feel free to re-use, modify, distribute as you like.</p> I am writing to express my opposition to the government's plans to regulate the content of out-of-school education.</p> I volunteer on an annual residential Christian holiday for young people which includes teaching on Christianity and the Bible. I am concerned that the work of this holiday would be regulated under these proposals. However, my concern is much wider than simply this and similar work.</p> I want to defend people I disagree with, as well as those with whom I agree.</p> I do not believe it is beneficial to society for governments to control the content of education provided outside of the state education system. In fact, I think it is a dangerous path to take, since while we may trust our current government and inspection regime, we have no reason to suppose it will always be tolerant and respectful of diverse views.</p> The use of terms such as "British values", "emotional harm" and "tolerance" in the proposals is concerning since these are open to widely-varying interpretations. In recent history, it could certainly have been considered "un-British" to lobby for the legalisation of homosexuality or the enfranchisement of women, or to support a communist political party.</p> Personally, I interpret "British values" to mean being tolerant of views very different from my own, and keeping state interference in individual decisions to a minimum. In that context, I believe these proposals are in conflict with British values, and my own values.</p> I believe that the best way to counteract violent religious extremism is to practice the tolerance that we preach: to demonstrate in words and actions that we respect individual freedom of speech and action. Where children are being taught values with which we disagree, we should argue in public against those values, not ban them from being taught.</p> New open source project on work time - git-history-data Andy Balaam — Mon, 14 Dec 2015 10:00:19 +0000 Announcing a little open source project that I have built at work and been allowed to publish Freely.</p> git-history-data</a> analyses a Git source code repository and dumps out data in a form that is easy to analyse.</p> I wrote an article demonstrating how to use it to find out some interesting information about the codebase of Git itself and got it published on IBM DeveloperWorks Open: Learning about the Git codebase using git-history-data</a>.</p> Java HTTPS to a server with a self-signed certificate Andy Balaam — Mon, 07 Dec 2015 15:55:37 +0000 Nothing is easy in Java, and nothing is more disproportionately non-easy than downloading something. If you add SSL to the equation, it becomes unfeasible for any human to navigate the twisted passages of the Java API, so here is a tiny fragment of map I have pieced together using the Internets.</p> Updated to link to my article about how to create a self-signed cert and trust store file</a>.</strong></p> If you have a server that you want to download something from, and you need to use SSL (i.e. your URL starts with "https://"), and it has a self-signed certificate, you will need to get hold of a "trust store" file (a .pkcs12 or .jks file) that tells Java it can trust your server. If you need to set this up yourself, try my article how to create a self-signed cert and trust store file</a>.</p> Note that this will only work if the self-signed certificate has the correct Subject Alternative Name (the hostname of the server) embedded in it. The article linked above tells you how to achieve this.</p> Assuming you have somehow magicked up a trust store file (let's call it trust.pkcs12), and you know the password for it (let's assume it's 000000, as we set up in the linked article), we can continue.</p> Let's write a little program Get.java that fetches a URL and tells us whether we got an error with the SSL connection:</p> import java.net.URL; public class Get { public static void main( String[] args ) throws Exception { try { new URL( args[0] ).openConnection().getInputStream(); System.out.println( "Succeeded." ); } catch( javax.net.ssl.SSLHandshakeException e ) { System.out.println( "SSL exception." ); } } } </pre> Compile this with:</p> javac Get.java</pre> And run it with:</p> $ java Get https://google.com Succeeded.</pre> This should succeed, because Java knows it can trust the benevolent Google deity, as we all do.</p> Now try it against your server with a self-signed (or otherwise untrusted) certificate and you should see an error:</p> $ java Get https://selfsigned.example.com SSL exception.</pre> And now for the answer you were waiting for. You don't need to use keytool. Repeat: you don't need to use keytool. To run Java telling it to trust your server, just do this:</p> $ java \ -Djavax.net.ssl.trustStore=/path/to/trust.pkcs12 \ -Djavax.net.ssl.trustStorePassword=000000 \ Get https://selfsigned.example.com Succeeded.</pre> That's it.</p> If it doesn't work, try:</p> $ java \ -Djavax.net.debug=SSL,trustmanager</strong> \ -Djavax.net.ssl.trustStore=/path/to/trust.pkcs12 \ -Djavax.net.ssl.trustStorePassword=000000 \ Get https://selfsigned.example.com </pre> and you should see some debugging output that may help.</p> Another useful debugging command is:</p> openssl s_client -showcerts -connect selfsigned.example.com</span></pre> Hope it helps.</p> Comments</h2> anant 2018-10-29</span> </h3> I was trying to import the public cert of selfsigned into existing java cacerts keystore (which is default used by the java), but it didn’t until I did above. My understanding is all it needs is to trust the cert that I did by importing.</p> </div> Eddie Kincaid 2019-08-15</span> </h3> Hi,</p> Thanks for this. It worked a charm! It has proved to be the best little tool for debugging issues, particularly when using self signed certificates, that I have come across.</p> </div> Andy Balaam 2019-08-15</span> </h3> Hi Eddie, glad it was helpful.</p> </div> Vim as editor for Claws Mail Andy Balaam — Thu, 03 Dec 2015 09:42:22 +0000 I got sick of not having Vim</a> as my text editor for emails in Claws Mail</a> but GVim makes my eyes bleed so I wanted proper Vim-in-a-terminal as my editor, which was not particularly straightforward.</p> The proper incantation was to make a script in my PATH called terminal-vim like this:</p> #!/bin/bash mate-terminal --disable-factory -x vim -f "$@" </pre> (gnome-terminal should work similarly if you're in GNOME.)</p> and to set it as my external editor in Claws Mail by going to Configuration, Preferences, External Programs, and setting Text Editor to terminal-vim '%s'.</p> Then under Writing in Preferences, I ticked the box "Automatically launch the external editor" and all is well.</p> Comments</h2> David 2017-04-09</span> </h3> Hello Andy,</p> I was trying to copy your approach for lxterminal (I am running lubuntu), creating an executable file in my PATH called terminal-vim:</p> #!/bin/bash lxterminal -e vim -f "$@"</p> I then followed your instructions within the claws-mail preference settings.</p> Now, when I press the ‘Compose’ button, vim does indeed pop up (in the terminal) and writes to this file: "~/.claws-mail/tmp/tmpmsg.0x140d930"</p> However, then I am lost. How do I get my text back into claws-mail?</p> Thanks for your guidance,</p> David</p> </div> Andy Balaam 2017-04-10</span> </h3> Hi David, I think if you just :wq the window should close and your text should end up in the Claws-mail window ready to send?</p> </div> David 2017-04-10</span> </h3> Hi Andy, thanks for you reply.</p> The transfer after :wq does not take place.</p> In any case, claws-mail expects input in various fields (addressee, title, mail text). How do I provide input to the right place? How does claws-mail know what to do with the text I have written in vim?</p> I am asking, because I could imagine that I am missing a crucial issue here…</p> Thanks for your help,</p> David</p> </div> Andy Balaam 2017-04-10</span> </h3> Hi David, for the To, Subject etc. I’m afraid you just have to type them into Claw-mail. The message body is what you get to edit in Vim, but if it’s not working I can’t think of why, sorry. One other possible thought: is lxterminal connection to some existing profile or session? If so, when you exit, the process would not really end so Claws would not know to continue. If that’s not it I’m afraid I am out of ideas – sorry.</p> </div> simon 2023-04-05</span> </h3> Hello, just posting this for anyone who may find it useful; to use vim with xfce4-terminal, enter the following directly into claws-mail under "set external editor to:"</p> xfce4-terminal —disable-server -x vim %s</p> Type your message, then :wq when done.</p> No script is required for this- thanks for pointing me in the right direction though. Presumably xfce4-terminal may be substituted with the terminal of your choice. Works fine on Claws 4.0.0</p> This was cobbled together with additional help from the linux mint forum: https://forums.linuxmint.com/viewtopic.php?t=...</a></p> Thanks to all!</p> </div> Using GitHub and GitLab Andy Balaam — Wed, 25 Nov 2015 09:55:17 +0000 Taking all the Git understanding we've developed through the series on Git</a> and applying it to how to work on and contribute to projects on GitHub and similar systems like GitLab:</p> </iframe> Avoiding postfix error "status=deferred (unable to look up host" Andy Balaam — Wed, 11 Nov 2015 12:47:20 +0000 My emails wouldn't send when I was working from home, giving me this message in /var/log/mail.log:</p> Nov 11 12:38:16 machinename postfix/smtp[20672]: CF5D6D41CE2: to=, relay=none, delay=14416, delays=14416/0.01/0.07/0, dsn=4.3.5, status=deferred (Host or domain name not found. Name service error for name=relay.example.com type=AAAA: Host not found) </name@example.com></pre> It turned out I was telling postfix not to use DNS to resolve the domain name of my mail relay server. There were several postfix config options I needed to remove to make this work:</p> # Make sure all of these are commented out or removed! #default_transport = smtp #relay_transport = smtp #inet_protocols = ipv4 #smtp_host_lookup = native </pre> The default values for these options are fine, but the ones shown above stop it working, so remove them or comment out with a # at the beginning of the line.</p> More at Postfix configuration parameters</a>.</p> Comments</h2> Alex M. 2016-04-01</span> </h3> Seems like the issue is that Postfix creates jail before network stack fully initialized, so /var/spool/postfix/etc/resolv.conf doesn’t have any servers to use and that’s why it fails. I have this issue on my raspberry pi Ubuntu setup and don’t have it on other PC.</p> </div> Andy Balaam 2016-04-05</span> </h3> Thanks Alex, yes – something weird is going on because I still see similar problems sometimes even with this config, but if I specify my relayhost by IP instead of domain name it works fine.</p> </div> Zhu Ye 2019-02-28</span> </h3> In my case, there is a legacy config from CentOS.</p> disable_dns_lookups = yes </span></span></code></pre> I use Ubuntu now, I have no idea how it worked in CentOS while it was set like this.</p> </div> Finding the download count of GitHub releases Andy Balaam — Fri, 16 Oct 2015 03:32:08 +0000 You can use the GitHub API to find out how many times the files in your releases have been downloaded.</p> For example, to find out how many downloads there have been of my Rabbit Escape project you can do:</p> curl -s https://api.github.com/repos/andybalaam/rabbit-escape/releases | egrep '"name"|"download_count"'</pre> Or you can look through the information manually by visiting a URL like https://api.github.com/repos/andybalaam/rabbit-escape/releases</a> in your browser.</p> To get the total I came up with this beautiful incantation:</p> curl -s https://api.github.com/repos/andybalaam/rabbit-escape/releases | egrep 'download_count' | cut '-d:' -f 2 | sed 's/,/+/' | xargs echo | xargs -I N echo N 0 | bc</pre> Comments</h2> Zmn 2017-04-01</span> </h3> Can I suggest this app to list all stats about your repository? http://github-analytics.apphb.com/</a></p> Disclaimer: I’m the owner of the application. Please share your opinion about it.</p> </div> Andy Balaam 2017-04-10</span> </h3> Zmn, thanks – it looks useful, but I would definitely like the projects to be sorted in descending order of downloads – I have a lot</em> of zeroes…. :–)</p> </div> Zmn 2017-04-10</span> </h3> Great, It’s nice you find it useful. I’m planning to improve usability and add some features… I’m working on it, stay tuned !</p> </div> Sarita 2021-04-20</span> </h3> Hi. I’m trying to use your total command on osx but I get an error:</p> (standard_in) 1: illegal character: N.</p> Any suggestion? thanks</p> </div> Andy Balaam 2021-04-20</span> </h3> Hi Sarita, it looks like xargs works differently on your platform. I’m afraid I don’t know what to do to fix it.</p> </div> Daniel 2025-08-13</span> </h3> Have a look at the following website, it shows detailed Github release stats: https://github-release-stats.ghostbyte.dev/</a></p> </div> Android: using a TextView to show rich text in an AlertDialog Andy Balaam — Sat, 10 Oct 2015 03:42:08 +0000 If you want to display a link or basic formatting in an AlertDialog on Android, you can do it by providing HTML.</p> The key parts you need are Html.fromHtml</a> and TextView.setMovementMethod</a>.</p> Make sure you pass the dialog's context in to the constructor of the TextView, not the context of the current activity. Otherwise the colours in your TextView will be wrong and you may well end up with black text on a dark grey background.</p> AlertDialog dialog = new AlertDialog.Builder( activity ) .setTitle( t( world.name ) ) .setPositiveButton( "Yes!" ) .setNeutralButton( "Maybe?" ) .create(); TextView view = new TextView( dialog.getContext() ); view.setText( Html.fromHtml( "<b>foo</b> <a href='#'>bar</a>" ) ); view.setMovementMethod( LinkMovementMethod.getInstance() ); view.setPadding( 10, 10, 10, 10 ); dialog.setView( view ); dialog.show(); </pre> If you are on API level 11+, you can use AlertDialog.Builder's getContext() method, so you don't have to create the dialog until the end.</p> Assassination of individuals by the state Andy Balaam — Wed, 30 Sep 2015 15:45:23 +0000 I wrote to my MP</a> (via writetothem.com</a>) about the British government assassinating people they suspected of planning terrorist attacks. He replied saying that the government believes the action was legal, proportional and necessary. Part of this justification was based on the fact that circumstances in Syria make it difficult to disrupt attacks in other ways. He stated the people killed were recruiting ISIL members and planning specific attacks in Britain. He also stated that this action was unconnected to general military action against ISIL in Syria.</p> My reply is below. Feel free to re-use and adapt it if you want to send a similar letter.</p> Dear Philip Hammond,</p> Thank you for your reply to my letter about the assassination of Junaid Hussain and Reyaad Khan.</p> I have 4 questions:</p> When did the British government start assassinating individuals suspected of planning terrorist attacks?</p> </li> What standard of evidence is required to assassinate someone?</p> </li> How imminent must a suspected attack be to warrant assassination?</p> </li> What oversight is there of each decision to perform an assassination?</p> </li> </ol> I continue to believe this practice is in conflict with British values and international law. You mention the UN Charter: I assume you are referring to Article 51 of the UN Charter, which refers to "armed attack" against a country. I do not accept that an individual suspected of planning a terrorist attack falls under the definition of an "armed attack" which was surely a reference to warfare by state-like entities, rather than plans being made by individuals to commit acts well-covered by UK criminal law.</p> We celebrated the 800th anniversary of Magna Carta in your constituency this year. Our reasons for celebration included the limitation of the state's power over the individual. When the state executes individuals without trial or even publication of evidence, is there any practical limit to its power?</p> Yours sincerely,</p> Andy Balaam</p> Difficult merges in Git - don't panic! Andy Balaam — Thu, 10 Sep 2015 10:38:13 +0000 A video in which I try to explain what merging and rebasing really are, to help you understand what is going on when Git presents you with scary-looking conflict messages. I also explain why you shouldn't panic because it's hard to lose your work, and how to get you work back if you really mess up:</p> </iframe> Slides here: Difficult Merges in Git</a>.</p> A commit represents the state of the world (and the history leading up to that state). A commit is not a diff.</p> Merging means making a new commit with two (or more) "parents" (previous commits) that represents the result of merging the changes from two different threads of development that happened separately. None of the already-committed commits are modified - you just get a new commit on top. History is more complicated, but true.</p> Rebasing means modifying the history of one thread of development so it looks like it happened after the other one. This involves modifying all the commits in that thread. There is no extra merge commit, so you lose the history of the merge that happened. History is simple, but it's a lie, and if you messed up the rebasing process, you can't get back to where you were (once your old commits have been garbage-collected).</p> ZX Spectrum BASIC Web Server Andy Balaam — Wed, 09 Sep 2015 10:22:11 +0000 Finally, you can write your web sites in ZX Spectrum BASIC, using my new project, ZX Spectrum BASIC CGI server </a>.</p> How it works</h3> Here's what happens when a request comes in:</p> Apache HTTPD receives the request, and runs the CGI script that does the magic.</li> The CGI script</a> (written in Bash) generates some BASIC code that provides the HTTP meta-vars (e.g. PATH_INFO for the path of the request) as DATA statements and wraps some other boilerplate around the code of the .basic file asked for in the request, and writes out another .basic file which is ready to be executed.</li> Then it uses BAS2TAP</a> (my mirror here</a>) to create an audio tape file representing the program.</li> Next it launches the Fuse</a> spectrum emulator (inside an Xvfb wrapper to make it think it has a display), telling it to load from the audio tape and run the program.</li> The program runs, and writes its HTTP responses to the Spectrum printer using LPRINT statements.</li> Fuse uses very basic OCR to understand what characters are printed, and writes them into a text file.</li> The CGI script monitors the output file, waiting for a line containing an end marker ("ENDSPECTRUMBASIC"). When it finds it, it kills the emulator (since it can't made to auto-exit, I think).</li> The CGI script sends the contents of the output file (minus the end marker) back to Apache, which returns it as a response.</li> </ul> Simple.</p> Why?</h3> Originally designed to demonstrate how Docker</a> can help isolate other services from "interesting" dependencies, this became a bit of a labour of love.</p> I'll never stop being grateful to the designers of the ZX Spectrum, or the authors of "Further Programming for the ZX Spectrum".</p> Comments</h2> ZX Spectrum BASIC CGI server – Retro Hardware 2017-05-17</span> </h3> […] Documentation:ZX Spectrum BASIC Web Server […]</p> </div> Letter to my MP on UK government assassination of British Citizens in Syria Andy Balaam — Tue, 08 Sep 2015 07:48:22 +0000 Below is a copy of the letter I sent to my MP this morning (via writetothem.com</a>). Feel free to re-use and adapt it if you want to send a similar letter.</p> Update</strong>: follow-up letter</a>.</p> Dear Philip Hammond,</p> I was extremely concerned to hear of the assassination of two British citizens by the UK armed forces [1].</p> The Prime Minister and Defence Secretary justified the attack on the basis of self-defence, which I believe is a reference to Article 51 of the UN Charter [2]. I consider the use of this article to cover action against individuals who are suspected of planning domestic terrorism to be wholly inappropriate. I support those who are calling for a legal challenge against this action.</p> Please pass on my concerns to the Prime Minister and Defence Secretary, and use your considerable influence in this area to make government policy focus on the use of criminal law to prevent domestic terrorism.</p> Please work to ensure the government abides by the decision of the UK parliament not to participate in military action in Syria, and thus not to treat this situation as a war in which the UK is participating. As you said on 2nd September 2013, "The House of Commons has ruled out military participation in any such response" (referring to our response to the use of chemical weapons).</p> Yours sincerely,</p> Andy Balaam</p> [1] http://www.bbc.co.uk/news/uk-34181475</a> [2] https://en.wikisource.org/wiki/Charter_of_the_United_Nations#Article_51</a> [3] http://www.publications.parliament.uk/pa/cm201314/cmhansrd/cm130902/debtext/130902-0001.htm#1309025000394</a></p> Rabbit Escape out now on Android! Andy Balaam — Sun, 30 Aug 2015 11:11:46 +0000 Watch the video:</p> </iframe> Buy the game on the Play Store</a>.</p> Changing the Docker daemon options in systemd on Ubuntu 15.04 Andy Balaam — Thu, 27 Aug 2015 11:48:16 +0000 Update: now documenting the better way, as described in issue 14513</a>.</strong></p> Update 2: I think this way is even better (works in Ubuntu 16.04, Docker 1.12.2, Reference: dockerd command line</a>):</strong> sudo -s -H echo '{"insecure-registries":["myreg.example.com:5000"]}' > /etc/docker/daemon.json exit sudo service docker restart</pre>End of update 2</strong></p> On earlier versions of Ubuntu (14.04 and before), changing the command line options to the Docker daemon (e.g. to allow using an insecure private registry) was just a matter of editing /etc/default/docker and uncommenting the line starting with #DOCKER_OPTS=.</p> On Ubuntu 15.04, even though that file still exists, editing it does not have any effect, and I found it quite tricky to work out how to change Docker's command line, so I wrote it up here.</p> I wanted to use an insecure private docker registry for Docker on Ubuntu 15.04, which uses systemd.</p> Under systemd, we must create a config file that overrides the default Docker command line by typing sudo systemctl edit docker. In the editor which pops up, type:[Service] ExecStart= ExecStart=/usr/bin/docker daemon -H fd:// --insecure-registry=myreg.example.com:5000 </pre></p> Note: the first "ExecStart=" line is necessary.</p> The second "ExecStart=" line should contain whatever command line options you want to include.</p> Note: for docker versions before 1.10, replace "daemon" with "-d".</p> With this config in place, restart the Docker service:</p> $ sudo systemctl restart docker </pre> Check everything looks right for the Docker service:</p> $ systemctl status docker </pre> And confirm the command line arguments have been applied with:</p> $ ps axwww | grep /usr/bin/docker </pre> The instructions here: Control and configure Docker with systemd</a> and issues: 14513</a> and 15859</a> suggest that the Docker team are not planning to make this any easier in the short term.</p> Docker fails to start on Ubuntu 15.04 Andy Balaam — Fri, 17 Jul 2015 09:32:58 +0000 I installed Docker on Ubuntu 15.04 using:</p> wget -qO- https://get.docker.com/ | sh</pre> as described at Install Docker on Ubuntu</a>.</p> I added myself to the docker group:</p> sudo usermod -aG docker balaaman</pre> Then I logged out and logged in again, and ran:</p> docker run hello-world</pre> and saw this:</p> Cannot connect to the Docker daemon. Is 'docker -d' running on this host?</pre> When I tried to start the Docker daemon like this:</p> sudo service docker start</pre> I got this error:</p> Failed to start docker.service: Unit docker.service is masked.</pre> Thanks to Yannick Lizzi on this thread</a> I found this fix:</p> systemctl unmask docker.service systemctl unmask docker.socket systemctl start docker.service </pre> After this, I logged out and logged in again (again) and "docker run hello-world" worked fine, and all was well.</p> Comments</h2> Nicholas Maietta 2016-06-23</span> </h3> You may want to consider updating this blog post to mention that the user will need to re-authenticate their session to pick up the changes if they cannot simply use the docker command without sudo, after making the aforementioned changes.</p> That is all, thank you.</p> </div> Andy Balaam 2016-07-13</span> </h3> Thanks Nicholas – I mentioned logging out and logging in again earlier (after the usermod command) – do you need to do it again after the systemctl commands? I don’t think I needed to…</p> </div> amin 2016-11-01</span> </h3> Thank you.</p> </div> fay 2016-12-28</span> </h3> it worked!! and very clearly described, thanks!</p> </div> Luciano 2018-03-22</span> </h3> Thank you very much, help appreciated :D</p> </div> Andy Balaam 2018-03-23</span> </h3> Glad it helped :–)</p> </div> JesÃºs Virseda Jerez 2018-09-05</span> </h3> "do you need to do it again after the systemctl commands?" Yes, you need ;) And teras and teras of thanks for your post Andy Balaam.</p> </div> Andy Balaam 2018-09-05</span> </h3> Thanks, updated!</p> </div> Renata Campos 2018-09-08</span> </h3> Perfeito!!</p> </div> brawaga 2019-06-05</span> </h3> Did not help. Still masked.</p> </div> Preventing Audacity from crashing when using PulseAudio Andy Balaam — Fri, 03 Jul 2015 14:44:45 +0000 I found that Audacity would crash whenever the Playback Device in the Devices section of Preferences was set to "pulse" or "default".</p> This can be fixed by launching Audacity like this:</p> Exec=env PULSE_LATENCY_MSEC=100 audacity</pre> I fixed it "permanently" by modifying my Audacity launcher file, which is in /usr/share/applications/audacity.desktop.</p> I changed the line starting "Exec=" to:</p> Exec=env PULSE_LATENCY_MSEC=100 audacity %F</pre> More info is on the Audacity forum</a>.</p> Movie podcast "The Good Robot Andys" Andy Balaam — Fri, 26 Jun 2015 15:04:17 +0000 My friend and I have launched a new podcast in which we discuss movies: The Good Robot Andys</a>.</p> </a></p> Enjoy.</p> Code for detecting when you leave an Android app Andy Balaam — Thu, 18 Jun 2015 08:22:35 +0000 Further to Detecting whether an Android app is stopping (or starting)</a>, I implemented code to decide when you are leaving or entering my game Rabbit Escape</a>.</p> The relevant class is called Lifecycle2SoundEvents</a>. (Yes, it's a terrible name. Yes, I spent a long time trying to name it, and this is the best I came up with.)</p> And the tests, which are in TestLifecycle2SoundEvents</a>, look like this:</p> @Test public void Press_the_home_button_api10_causes_pause() { Tester t = new Tester( activity1 ); t.in.onSaveInstanceState( activity1 ); t.in.onPause( activity1 ); t.in.onStop( activity1 ); // When we press home, we must at least pause (really we stop) t.assertPaused(); } </pre> which I was reasonably pleased with, because they match my original blog post Order of Android Activity lifecycle events</a> fairly well, without too much noise.</p> Mocks are Bad, Layers are Bad Andy Balaam — Wed, 10 Jun 2015 08:33:23 +0000 In which I argue that mocks are a code smell, and layers lead to increased coupling:</p> Mocks are Bad, Layers are Bad</a> (in ACCU</a>'s Overload Journal issue 127)</p> I also suggest some ways to avoid both mocks and layers, including Classical TDD, Selfish Object, Refactor to Functional and, of course, the Unix Philosophy. I work through a code example to demonstrate some of these things.</p> I also suggest that frameworks and inheritance hierarchies are bad, but the title was getting too long already.</p> You can also get the PDF of Overload 127</a>.</p> Prediction result: corporate Linux Andy Balaam — Mon, 08 Jun 2015 19:09:58 +0000 Ten years ago I predicted that 30% of corporate desktops would be Linux</a> or similar open source desktops.</p> It's very hard to find any accurate (or even confident) numbers, but it seems clear I was wrong.</p> This month, NetMarketShare puts Linux usage at 1.28%</a>, and w3schools puts Linux usage at 5.2%</a>.</p> I think companies almost certainly use Linux more than home consumers. Certainly, my own company has a very large section of its population using Linux - all users who have access to sensitive information are required to use it to increase security.</p> But, I would be very surprised if levels were at 30%.</p> Linux desktop usage does continue to creep upwards though, while everyone is distracted looking at phones and tablets. At some time there may be a tipping point.</p> Detecting whether an Android app is stopping (or starting) Andy Balaam — Fri, 05 Jun 2015 01:14:40 +0000 I am writing an Android app (called Rabbit Escape</a>), and I want it to start playing music when the user enters the app, and stop when the user leaves.</p> Not as easy as it sounds because Android largely doesn't think in apps, but Activities</a>.</p> Update:</strong> real-life code for this is here: Code for detecting when you leave an Android app</a>.</p> In my previous post Order of Android Activity lifecycle events</a> I tracked the methods that get called on activities on my two test devices when the user does various actions. There was some variation between devices, but we can determine some useful rules.</p> First, let's enumerate what we need to detect.</p> The user leaves by:</p> pressing "home" at any time</li> pressing the power button when the app is running</li> pressing "back" when in the first activity</li> launching an external activity (e.g. web browser)</li> </ul> The user enters by:</p> starting the app (e.g. from the home page, or the "running apps" panel)</li> pressing the power button when the app has been suspended</li> pressing "back" when in an external activity launched by us</li> </ul> From the previous post we can work out these rules:</p> When the user is entering, onResume is always called (on the activity that will be current)</li> When the user is leaving, onStop is always called (on the current activity), except when pressing the power button (to turn off) where you may only see an onPause call.</li> When the user is moving between activities, onResume is always called on the new activity, and onStop is always called on the old one. The onResume call is always before onStop</li> </ul> Thus, if we are OK with leaving ourselves running when the power button is pressed, we can detect leaving and entering the app using the code below.</p> If we need to stop something when the power button is pressed too, it seems we must be prepared to stop it when we receive onPause, and immediately restart it if we receive onResume, hoping that doesn't cause a problem (e.g. a break in the music).</p> // All activities have these two methods: @Override public void onResume() { super.onResume(); LeavingOrEntering.activityResumed( this ); } @Override public void onStop() { super.onStop(); LeavingOrEntering.activityStopped( this ); } // And we detect leaving and entering like this: class LeavingOrEntering { private static Activity currentActivity = null; public static void activityResumed( Activity activity ) { if ( currentActivity == null ) { // We were resumed and no-one else was running. notifyEntering() // Start the music! } currentActivity = activity; } public static void activityStopped( Activity activity ) { assert currentActivity != null; if ( currentActivity == activity ) { // We were stopped and no-one else has been started. notifyLeaving(); // Stop the music! } currentActivity == null; } } </pre> Order of Android Activity lifecycle events Andy Balaam — Fri, 05 Jun 2015 00:31:38 +0000 I noticed some variation between devices so I tested various user actions on various devices and recorded the Android Activity lifecycle methods (e.g. onResume, onPause, onCreate, onStop) that got called. My results are below.</p> [Why? I want to detect whether the user is leaving the app or just transitioning from one activity to another. See my next blog posts Detecting whether an Android app is stopping (or starting)</a> and Code for detecting when you leave an Android app</a> for how to do it.]</p> Press the home button</h3> Ensure the app process is stopped.</li> Start the app. It starts an Activity.</li> Start recording events.</li> Press "home" on the device.</li> </ul> HTC Wildfire S Android 2.3.5 (API 10)</th> onSaveInstanceStateonPauseonStop </pre></th></tr></thead> Nexus One Emulator (x86) Android L (API 20)</td> onPauseonSaveInstanceStateonStop </pre></td></tr> </tbody></table> When home was pressed, onPause and onStop were called in that order, and onSaveInstanceState was always called before onStop. The app was still running after the test.</p> Note that onSaveInstanceState can happen either before or after onPause.</p> This is consistent with the information given on the Android Activity API docs</a>:</p> Starting with Honeycomb, an application is not in the killable state until its onStop() has returned. This impacts when onSaveInstanceState(Bundle) may be called (it may be safely called after onPause() and allows and application to safely wait until onStop() to save persistent state.</p> </blockquote> Press the home button in a second activity</h3> Ensure the app process is stopped.</li> Start the app. It starts Activity1.</li> Press a button in the activity that launches Activity2.</li> Start recording events.</li> Press "home" on the device.</li> </ul> Results are identical to "Press the home button" above, with Activity2 receiving all the calls, and Activity1 not receiving any.</p> Press the back button to exit</h3> Ensure the app process is stopped.</li> Start the app. It starts an Activity.</li> Start recording events.</li> Press "back" on the device.</li> </ul> HTC Wildfire S Android 2.3.5 (API 10)</th> onPauseonStoponDestroy </pre></th></tr></thead> Nexus One Emulator (x86) Android L (API 20)</td> (same as above)</td></tr> </tbody></table> When back was pressed, onPause, onStop and onDestroy were called. The app process was still running after the test.</p> onSaveInstanceState was not called, because there is no current state to resume - when you re-enter you will be back at the beginning.</p> Start the app</h3> Ensure the app process is stopped.</li> Start recording events.</li> Start the app.</li> </ul> HTC Wildfire S Android 2.3.5 (API 10)</th> onCreateonStartonResume </pre></th></tr></thead> Nexus One Emulator (x86) Android L (API 20)</td> (same as above)</td></tr> </tbody></table> When the app was started, onCreate, onStart and onResume were called in that order.</p> Turn off with the power button</h3> Start the app.</li> Start recording events.</li> Press the power button (turn off).</li> </ul> HTC Wildfire S Android 2.3.5 (API 10)</th> onSaveInstanceStateonPause </pre></th></tr></thead> Nexus One Emulator (x86) Android L (API 20)</td> onPauseonSaveInstanceStateonStop</pre></td></tr> </tbody></table> When the phone was suspended, onSaveInstanceState and onPause were called in different orders, and onStop was called in some cases.</p> Turn on with the power button</h3> Start the app.</li> Press the power button (turn off).</li> Start recording events.</li> Press the power button (turn on).</li> Unlock the device.</li> </ul> HTC Wildfire S Android 2.3.5 (API 10)</th> onResume </pre></th></tr></thead> Nexus One Emulator (x86) Android L (API 20)</td> onRestartonStartonResume </pre></td></tr> </tbody></table> When the phone was resumed, onResume was always called, and onRestart and onStart were sometimes called.</p> Restart the app after pressing home</h3> Ensure the app process is stopped.</li> Start the app.</li> Press "home" on the device.</li> Start recording events.</li> Start the app.</li> </ul> HTC Wildfire S Android 2.3.5 (API 10)</th> onRestartonStartonResume </pre></th></tr></thead> Nexus One Emulator (x86) Android L (API 20)</td> (same as above)</td></tr> </tbody></table> When the app was restarted after pressing home, onCreate was not called, but onRestart, onStart and onResume were (in that order).</p> Restart the app after pressing back</h3> Ensure the app process is stopped.</li> Start the app.</li> Press "back" on the device.</li> Start recording events.</li> Start the app.</li> </ul> HTC Wildfire S Android 2.3.5 (API 10)</th> onCreateonStartonResume </pre></th></tr></thead> Nexus One Emulator (x86) Android L (API 20)</td> (same as above)</td></tr> </tbody></table> When the app was restarted after pressing back, onRestart was not called, but onCreate, onStart and onResume were (in that order).</p> This is consistent with the idea that pressing back effectively ends the app, but pressing home effectively pauses it.</p> Launch a second activity</h3> Ensure the app process is stopped.</li> Start the app. It starts Activity1.</li> Start recording events.</li> Press a button in the activity that launches Activity2.</li> </ul> HTC Wildfire S Android 2.3.5 (API 10)</th> Activity1.onSaveInstanceStateActivity1.onPauseActivity2.onCreateActivity2.onStartActivity2.onResumeActivity1.onStop </pre></th></tr></thead> Nexus One Emulator (x86) Android L (API 20)</td> Activity1.onPauseActivity2.onCreateActivity2.onStartActivity2.onResumeActivity1.onSaveInstanceStateActivity1.onStop </pre></td></tr> </tbody></table> The later version of Android called onSaveInstanceState much later in the process. This is consistent with the documentation for the Activity Lifecycle</a>.</p> In both cases Activity1's onPause was called before any methods were called on Activity2.</p> In both cases Activity1's onStop was called after all methods were called on Activity2.</p> Activity2's methods were called just as if it were the starting activity of a freshly-launched app (see "Start the app" above).</p> Activity1's methods were called just as if the home button had been pressed.</p> Launch an external activity</h3> Ensure the app process is stopped.</li> Start the app. It starts an Activity.</li> Start recording events (in our activity only).</li> Press a button in the activity that launches the Android web browser.</li> </ul> Results are identical to "Press the home button" above.</p> Go back from one activity to another</h3> Ensure the app process is stopped.</li> Start the app. It starts Activity1.</li> Press a button in the activity that launches Activity2.</li> Start recording events.</li> Press "back" on the device.</li> </ul> HTC Wildfire S Android 2.3.5 (API 10)</th> Activity2.onPauseActivity1.onRestartActivity1.onStartActivity1.onResumeActivity2.onStopActivity2.onDestroy </pre></th></tr></thead> Nexus One Emulator (x86) Android L (API 20)</td> (same as above)</td></tr> </tbody></table> Activity2 sees the same calls as in "Press the back button to exit" (i.e. it is destroyed).</p> Activity1 see the same calls as in "Restart the app after pressing home" (i.e. it is resumed).</p> Go back from an external activity to ours</h3> Ensure the app process is stopped.</li> Start the app. It starts an Activity.</li> Press a button in the activity that launches the Android web browser.</li> Start recording events (in our activity only).</li> Press "back" on the device.</li> </ul> Results are identical to "Restart the app after pressing home" above.</p> See my next blog post for how to detect whether we are leaving the app, or just transitioning between activities: Detecting whether an Android app is stopping (or starting)</a>.</p> Comments</h2> Code for detecting when you leave an Android app | Andy Balaam's Blog 2015-06-18</span> </h3> […] Order of Android Activity lifecycle events […]</p> </div> Anchal Maheshwari 2018-07-26</span> </h3> wow…. very nice information. its helpfull in interview prepration. thanks alot.</p> </div> Andy Balaam 2018-07-26</span> </h3> Thanks Anchal Maheshwari, I’m glad it was helpful.</p> </div> Snake in Dart Andy Balaam — Wed, 03 Jun 2015 22:06:00 +0000 Series: Groovy</a>, Ruby</a>, BASIC</a>, Dart, Elm</a>, Python3+Qt5</a></p> I'm writing the game Snake in lots of programming languages, for fun, and to try out new languages. This time, Dart, which is for people who love Java and wish they didn't have to do JavaScript.</p> </iframe> Slides: Snake in Dart</a></p> If you want to, you can Support me on Patreon</a>.</p> Preventing Kupfer switching to existing windows Andy Balaam — Fri, 15 May 2015 13:06:06 +0000 I am enjoying using Kupfer</a> but I don't like the way it switches to an existing window of an application, instead of launching a new instance.</p> I should write a patch for a config option or similar, but for now, here are my notes on how I disabled the behaviour in the source code:</p> sudo sensible-editor /usr/share/kupfer/kupfer/launch.py </pre> I just commented out 3 lines inside the launch_application method (lines 70-72 for me):</p> # if activate and svc.application_is_running(app_id): # svc.application_to_front(app_id) # return True </pre> This prevents Kupfer looking for open windows before launching.</p> Using Kupfer on MATE Andy Balaam — Tue, 12 May 2015 13:45:07 +0000 I am trying out MATE</a> desktop and really liking it.</p> I like to use a (GNOME-Do</a> style) keyboard-driven application launcher, and have found Kupfer</a> really good.</p> Kupfer as packaged for Ubuntu MATE</a> does not support MATE desktop, so the file manager ("Caja") does not show up by default, along with other things like the preferences dialogs, and there is no support for actions like Log Out or Shut Down.</p> To get MATE applications to show up in Kupfer, all I had to do was make up a config value that is not presented in the preferences, by editing the config file:</p> pluma ~/.config/kupfer/kupfer.cfg</pre> I found the line that started "desktop_type = " and changed it to say:</p> desktop_type = MATE</pre> Update</strong> [Thanks to Kristian Nygaard Jensen and Igor Santos]: if you can't find that line in the file, add this at the bottom of the file:</p> [plugin_applications] desktop_type = MATE </pre> To get MATE session commands in Kupfer, I copied the GNOME session management plugin and made a similar one for MATE.</p> cd /usr/share/kupfer/kupfer/plugin sudo cp session_gnome.py session_mate.py sudo pluma session_mate.py </pre> I just changed the 3 lines after "# sequences of argument lists", and made them look like this:</p> LOGOUT_CMD = (["mate-session-save", "--logout-dialog"],) SHUTDOWN_CMD = (["mate-session-save", "--shutdown-dialog"],) LOCKSCREEN_CMD = (["mate-screensaver-command", "--lock"],) </pre> When I restarted Kupfer I was able to open MATE applications and control the MATE session using Kupfer.</p> There is an open pull request from 2013 for Kupfer to add MATE session management: github.com/engla/kupfer/pull/15</a> - thanks to its author labero</a>, since this showed me what I needed to do for that part.</p> Comments</h2> Kristian Nygaard Jensen 2015-05-27</span> </h3> in which section in the config file […] did you find desktop_type. My kupfer.cfg does not contain desktop_type</p> </div> Igor Santos 2015-06-09</span> </h3> Same question here… There’s no desktop_type</code> field. In which section was it? I’ve thrown it in some places and it didn’t work out.</p> I’ve added a bunch of request in the project’s launchpad page asking for the maintainer to take a look at the bugs and PRs. If it’s possible, please, join there as well: https://answers.launchpad.net/kupfer/+questio...</a> https://bugs.launchpad.net/kupfer/+bug/1463500</a></p> </div> Igor Santos 2015-06-09</span> </h3> Actually, after some fiddling I found it: if the entry is not in the file, include this section:</p> [plugin_applications] desktop_type = MATE </code></p> This section can be created by Kupfer preferences if you change your default desktop from GNOME to something else on the Applications</code> plugin.</p> </div> Andy Balaam 2015-06-10</span> </h3> Ah, thanks Igor. That explains why it was there for me.</p> </div> Andy Balaam 2015-06-10</span> </h3> I have updated the post – thank you!</p> </div> Tweetable Art Code Andy Balaam — Mon, 04 May 2015 10:11:28 +0000 Picking apart some cool code to draw pretty pictures with code that fits into 3 tweets:</p> </iframe> This comes from a codegolf question here: Tweetable Mathematical Art</a>.</p> Slides: Tweetable Art Code</a>.</p> Rabbit Escape 0.3.1 - now with zoom! Andy Balaam — Sat, 02 May 2015 07:53:09 +0000 I've just release the latest version of Rabbit Escape, which makes things look a lot nicer because you can zoom in, getting you much closer to your rabbits:</p> There are still 60 levels of Lemmings and Pingus -like gameplay, all downloadable for free artificialworlds.net/rabbit-escape/</a>.</p> I've also improved performance significantly, so you should notice things get smoother on older devices.</p> All those zoomed images increase the download size to 9MB, which is a pity, but that's still pretty small.</p> Treat warnings as errors in a (Gnu) Makefile Andy Balaam — Wed, 22 Apr 2015 14:36:01 +0000 I got hit again last night by a bug in my Makefile that meant I effectively ran rm -rf /*, which was not fun.</p> Today I have been looking for how to make Make stop if a variable is undefined.</p> Here's my solution. This Makefile contains a bug ("SRCS" instead of "SRC"):</p> # Turn on the warning we want MAKEFLAGS += --warn-undefined-variables # Make sure MAKECMDGOALS is defined, so it doesn't cause an error itself ifndef MAKECMDGOALS MAKECMDGOALS = all endif SRC=hello.c all: compile # Fails if the Makefile contains any warnings. # Run this Makefile with the same goals, but with the -n flag. # Grep for warnings, and fail if any are found. no-make-warnings: ! make -n $(MAKECMDGOALS) 2>&1 >/dev/null | grep warning # Targets you want to check must depend on no-make-warnings compile: no-make-warnings gcc -o hello $(SRCS) </pre> When I run it I get:</p> $ make ! make -n all 2>&1 >/dev/null | grep warning Makefile:17: warning: undefined variable `SRCS' make: *** [no-make-warnings] Error 1 </pre> When I correct the bug I get:</p> $ make ! make -n all 2>&1 >/dev/null | grep warning gcc -o hello hello.c </pre> As expected.</p> To make a target warnings-resistant, you have to make it depend on the target no-make-warnings. If anyone has suggestions for how to avoid needing this, please comment.</p> I also posted this as a StackOverflow answer: How to treat a warning as an error in a Makefile?</a>.</p> Switching Xfce to use metacity Andy Balaam — Mon, 20 Apr 2015 16:01:50 +0000 I am trying out Xfce and liking it. However, I've never found a window manager better than Metacity, so I'd like to use it.</p> Here's how I switched:</p> # Install it sudo apt-get install metacity metacity-themes # Tell it how many workspaces I want gsettings set org.gnome.desktop.wm.preferences num-workspaces 9 # Of course, put my maximise, close buttons etc. in the right place gsettings set org.gnome.desktop.wm.preferences button-layout ':minimize,maximize,close' # See what themes are available ls /usr/share/themes/ # Set the theme I want gsettings set org.gnome.desktop.wm.preferences theme Dopple # Try it out immediately metacity --replace # Make the change permanent, log out and edit # ~/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-session.xml # and change this line: # <property name="Client0_Command" type="empty"/> # to this: # <property name="Client0_Command" type="array"><value type="string" value="metacity"/></property> </pre> Comments</h2> johnywhy 2018-08-03</span> </h3> hi, what’s the difference? thx</p> </div> Andy Balaam 2018-08-05</span> </h3> The difference is a few subtle behaviours when you drag windows around, and keyboard shortcuts like Alt-F7 for moving windows that I am used to. I use MATE now, which does everything I want very nicely.</p> </div> Eric Marceau 2019-03-10</span> </h3> Trying Metacity on Xubuntu 18.04.</p> Is anyone experiencing "jitter" and slow dragging when trying to drag a window around on the screen?</p> That is not experienced when using the XFCE on Xubuntu or straight Xubuntu.</p> </div> Java game programming: image rendering hints make no difference to rendering time Andy Balaam — Mon, 06 Apr 2015 22:50:39 +0000 I am writing an Open Source/Free Software Java desktop game (Rabbit Escape</a>) and am experimenting with allowing you to zoom in so that the images being rendered are fairly big, and I'm seeing some slow-down.</p> [Note: Rabbit Escape</a> is also available on Android, where it appears to have no problems with rendering speed...]</p> To speed it up, I have been experimenting with passing various values to Graphics2D</a>'s setRenderingHint</a> method.</p> My conclusion is that on my platform (OpenJDK Java 7 on Linux) it makes no difference whatsoever. Please leave a comment if I'm doing it wrong.</p> I am timing how long each frame takes to render, and comparing the time when I insert this code before my rendering code:</p> // As slow and high-quality as possible, please! g.setRenderingHint( RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON ); g.setRenderingHint( RenderingHints.KEY_ALPHA_INTERPOLATION, RenderingHints.VALUE_ALPHA_INTERPOLATION_QUALITY ); g.setRenderingHint( RenderingHints.KEY_COLOR_RENDERING, RenderingHints.VALUE_COLOR_RENDER_QUALITY ); g.setRenderingHint( RenderingHints.KEY_RENDERING, RenderingHints.VALUE_RENDER_QUALITY ); </pre> with inserting this before my rendering code:</p> // As fast and low-quality as possible, please! g.setRenderingHint( RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_OFF ); g.setRenderingHint( RenderingHints.KEY_ALPHA_INTERPOLATION, RenderingHints.VALUE_ALPHA_INTERPOLATION_SPEED ); g.setRenderingHint( RenderingHints.KEY_COLOR_RENDERING, RenderingHints.VALUE_COLOR_RENDER_SPEED ); g.setRenderingHint( RenderingHints.KEY_RENDERING, RenderingHints.VALUE_RENDER_SPEED ); </pre> [Documentation, such as it is, is here: RenderingHints</a>.]</p> I see no appreciable difference in rendering time or image quality between these two setups, or when I leave out the calls to setRenderingHint completely.</p> Platform details:</p> $ java -version java version "1.7.0_75" OpenJDK Runtime Environment (IcedTea 2.5.4) (7u75-2.5.4-1~trusty1) OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode) </pre> Snake in ZX Spectrum BASIC Andy Balaam — Fri, 27 Mar 2015 08:12:35 +0000 Series: Groovy</a>, Ruby</a>, BASIC, Dart</a>, Elm</a>, Python3+Qt5</a></p> I'm writing the game Snake in lots of programming languages, for fun, and to try out new languages. This time, the first language I ever learned:</p> </iframe> Slides: Snake in ZX Spectrum BASIC</a></p> If you want to, you can Support me on Patreon</a>.</p> fetchmail complaining about GoDaddy SSL certificate Andy Balaam — Thu, 26 Mar 2015 22:22:02 +0000 Update: I don't think this fixed the problem</strong></p> I was getting this every time I ran fetchmail.</p> fetchmail: Server certificate verification error: unable to get local issuer certificate fetchmail: Broken certification chain at: /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2 fetchmail: This could mean that the server did not provide the intermediate CA's certificate(s), which is nothing fetchmail could do anything about. For details, please see the README.SSL-SERVER document that ships with fetchmail. fetchmail: This could mean that the root CA's signing certificate is not in the trusted CA certificate location, or that c_rehash needs to be run on the certificate directory. For details, please see the documentation of --sslcertpath and --sslcertfile in the manual page. fetchmail: Server certificate verification error: certificate not trusted fetchmail: Server certificate verification error: unable to verify the first certificate fetchmail: Warning: the connection is insecure, continuing anyway. (Better use --sslcertck) </pre> I appear to have fixed it by running:</p> sudo c_rehash </pre> I found this by reading the documentation on --sslcertpath in the fetchmail man page. (As the error message told me to...)</p> Encapsulation as passing on responsibility Andy Balaam — Mon, 16 Mar 2015 14:17:55 +0000 I recently dealt with some code that I felt was not properly encapsulated, but in a sense that I've not seen articulated in this way before. Please enlighten me if I missed it.</p> Here's a snippet:</p> OutputThing manipulate( InputThing input ) { ProcessingThing proc( input ); moreProcessing( proc.intermediateResult, input ); return OutputThing( proc ); } </pre> This function takes in some input and returns some output, using a ProcessingThing to do some work.</p> I needed to make a change to ProcessingThing to support some functionality, and I fell foul of something I didn't expect.</p> I wanted to transform my input a bit, and wrote that code inside ProcessingThing. What I didn't expect was that the untransformed input would later be used (in the moreProcessing call). The moreProcessing code failed because the transformation I had done made the ProcessingThing inconsistent with the InputThing.</p> Obviously, I should have checked more carefully.</p> But, it struck me that there is a pattern here. We are processing something by passing it from step to step in a "production line" of code. By re-using something from an older step, we violate this metaphor of passing on responsibility, because suddenly there are two copies of our input - it wasn't passed on at all.</p> This feels a bit like we've violated the encapsulation of input by ProcessingThing, but maybe I'm stretching the word encapsulation too far?</p> I feel like if ProcessingThing properly encapsulated the details of InputThing, we wouldn't need to re-use input later.</p> In fact, the fix in my case was to put the moreProcessing logic into ProcessingThing, meaning there was no need to refer back to the original input. I think this supports the argument that we are talking about encapsulation.</p> Snake in Ruby Andy Balaam — Thu, 12 Mar 2015 02:09:01 +0000 Series: Groovy</a>, Ruby, BASIC</a>, Dart</a>, Elm</a>, Python3+Qt5</a></p> I'm writing the game Snake in lots of programming languages, for fun, and to try out new languages.</p> </iframe> Slides: Snake in Ruby</a></p> If you want to, you can Support me on Patreon</a>.</p> Batch-converting audio files to be louder (on Linux) Andy Balaam — Sat, 28 Feb 2015 07:14:01 +0000 My mp3 player is very quiet, so I wanted to make all my podcasts as loud as possible.</p> First I ran this to get the programs I needed:</p> sudo apt-get install libav-tools normalize-audio </pre> To convert each file I made a script that makes a "loud" directory, and puts the loud version of a file inside there. It uses the normalize-audio command to do it.</p> Note that this script encodes your (now louder) podcasts into Ogg Vorbis format at 50kb/s, which is quite low quality.</p> #!/bin/bash set -e set -u FILE="$1" DIR=`dirname "$FILE"` FILENAME=`basename "$FILE"` WAV_FILENAME="${FILENAME}.wav" LOUD_DIR="$DIR/loud" WAV_FILE="$LOUD_DIR/$WAV_FILENAME" LOUD_FILE="$LOUD_DIR/$FILENAME" mkdir -p "$LOUD_DIR" avconv -loglevel quiet -i "$FILE" "$WAV_FILE" normalize-audio -q -a 1 "$WAV_FILE" avconv -loglevel quiet -i "$WAV_FILE" -c:a libvorbis -b:a 50k "$LOUD_FILE" rm "$WAV_FILE" </pre> Finally I placed a Makefile in the directory containing podcasts directories, like this:</p> MP3S := $(wildcard *.mp3) LOUD_MP3S := $(MP3S:%.mp3=loud/%.mp3) OGGS := $(wildcard *.ogg) LOUD_OGGS := $(OGGS:%.ogg=loud/%.ogg) all: $(LOUD_OGGS) $(LOUD_MP3S) loud/%.ogg: %.ogg loud "$<" loud/%.mp3: %.mp3 loud "$<" </pre> Now I can make loud versions of all podcasts by just cding into the directory containing the Makefile, and typing make. By the power of make, it only converts files that have not already been converted.</p> Why Rabbit Escape is Open Source / Free Software Andy Balaam — Fri, 27 Feb 2015 08:00:32 +0000 Why I wanted to make Rabbit Escape Free Software, even though I also plan to sell it.</p> </iframe> Because I want to share it.</p> Comments</h2> Sebastian Silva 2017-04-06</span> </h3> Thanks for your game, I really enjoyed sharing it with my kids. Out of curiosity, how did your selling on Play go? Great game!</p> </div> Andy Balaam 2017-04-10</span> </h3> Hi Sebastian, thanks! Very few people have bought the paid version from the Play Store, and even fewer have downloaded the (identical) free version I recently released :–(</p> Possibly a failure of marketing…</p> </div> Sebastian Silva 2017-04-10</span> </h3> Thanks for your sincere answer! I thought Rabbit Escape is a very nice pre-programming game for kids (one is 6, but even my 3Â½ year old was enjoying it and succeeding in early levels).</p> I wish you success and inspiration and congratulate you on this beautiful game that I actively recommend to friends and fellow parents.</p> Regards from Peru</p> </div> Andy Balaam 2017-04-10</span> </h3> Sebastian thank you so much for the encouragement! I am very glad it has been fun for your kids – that is the real reason I wrote it, so I am satisfied :–)</p> </div> What is a good company? Andy Balaam — Tue, 24 Feb 2015 09:45:36 +0000 I've been trying to work out what I think would be a good company to work for. Here's what I've got so far. Please comment pointing out what I got wrong and missed out.</p> Be coo</h2> We believe a company should be a good place to work.</p> We sum that up in one rule:</p> "be coo"</p> </blockquote> Rules</h2> We define being coo like this:</p> Value humans</li> Be small, and do not pursue growth as a goal</li> Be privately owned</li> Make something that people want</li> Get paid for what you do</li> Contribute to wider society</li> </ul> More detail:</p> Value humans</h2> Be kind to people: all people, including your customers, your employees and your boss.</p> Use in-person communication whenever possible, and otherwise use live video chat. For talking about careers, performance etc. never settle for voice-only or text.</p> Regularly meet the people you manage and encourage them about what they are doing well. Rarely, talk to them about what didn't go well. Sometimes, if someone is not able to do a role it is better for everyone if their role changes or they get another job. When it looks like this might be the situation, communicate clearly and honestly about it with them both before that decision is made and afterwards.</p> In performance reviews, gently reprimand people who regularly work more than their contracted hours.</p> Have a policy on acceptable behaviour, which is enforced by humans telling other humans what they did was not OK, when it wasn't.</p> Do your best to employ people based on their ability to do the job. This includes not employing them because:</p> They look or sound right</li> They have spare time (e.g. don't exclude someone because they don't do any open source)</li> They seem like they are "one of us" or "get it"</li> </ul> Instead, look for evidence that they can do the job (this might include open source contributions, obviously), and that you can work with them. Being able to do the job will mean that they "get" things, and being able to work with them will mean that you can communicate effectively with them.</p> This is obviously hard. There are contradictions in the sentences above. Pro-actively invite feedback, accept negative feedback, and work to improve.</p> Be small, and do not pursue growth as a goal</h2> Growth is a stupid goal. Aim to do a good job. Aim to make enough money to pay your employees (and/or yourself) well. Aim to be the right size to do a good job for your customers.</p> Be privately owned</h2> If you are owned by shareholders, you have a duty to prioritise shareholder value. This is in conflict with some of the rules, so don't do it.</p> Make something that people want</h2> You have to make something that people will buy or pay for, otherwise you won't last long.</p> But, ask yourself whether people are glad that your company exists?</p> Make what you do something worthwhile: something that benefits people.</p> Get paid for what you do</h2> If you don't spend all day selling advertising, don't get paid for selling advertising.</p> Find customers who want to pay you for the work you do. If your customers are actually benefitting from a concealed side-effect of what you do, the temptation to lie to them or fool them will be strong. (In reality, they are not your customers, and you won't value them.)</p> Contribute to wider society</h2> Give people time off to do voluntary work. Within reason, don't tell them what that can or can't be.</p> Contribute bug reports, documentation and code to open source projects you use for your work.</p> Donate some of your profits to good causes supported by the staff.</p> Comments</h2> John Hutchinson 2015-03-23</span> </h3> Really interesting article. Do you mind if I circulate it in our company? (I got here after watching your videos on YT about Scheme, they are really good. I’ve never really understood why you would bother with lambdas in C# but now I do!)</p> </div> Andy Balaam 2015-03-23</span> </h3> John, thanks for the kind words – please do circulate it wherever you like.</p> </div> How to make your own levels for Rabbit Escape Andy Balaam — Thu, 19 Feb 2015 23:42:53 +0000 A little video showing you how to make levels for my new game Rabbit Escape</a>.</p> </iframe> There are more instructions on the Creating levels for Rabbit Escape</a> page.</p> Rabbit Escape v0.1 out now for Linux, Windows and Mac video Andy Balaam — Sun, 15 Feb 2015 21:05:04 +0000 A little video to announce my new game, Rabbit Escape</a>.</p> </iframe> Download it, play it, enjoy it, and let me know how I can improve it!</p> Rabbit Escape (a bit like Lemmings) v0.1 released for Linux, Windows and Mac Andy Balaam — Tue, 10 Feb 2015 11:27:50 +0000 Today I am releasing the first version of my new game, Rabbit Escape</a>.</p> It's an arcade puzzle game inspired by Lemmings</a> and Pingus</a>, but intended to be simpler and easier to control on a mobile device.</p> Your task is to guide a party of rabbits from the entrance to the exit by dropping tokens in front of them that give them special abilities such as building bridges or climbing walls.</p> Here's what it looks like:</p> An Android version will be coming soon, but for now I'm releasing the desktop version for Linux, Windows and Mac.</p> There are 60 levels, and I'm hoping people will be sending me lots more soon!</p> I plan to write a level editor, but for now you can create your own levels</a> by editing text files.</p> The game is Free Software</a> under GPL v2, and the graphics, levels etc. are released under the non-commercial Creative Commons BY-NC-SA licence. The choice of a non-commercial license for these parts is intended to prevent people copying the game wholesale onto an app store and making money from it. If you want to use it under a different license, please contact me.</p> I plan to charge the minimum price on the Android store, and offer the desktop version for free (providing an opportunity for donations).</p> Please try it out and let me know how you get on. It's a bit rough around the edges, but the game mechanics work, and it seems like it might be fun.</p> If you'd like to contribute, I'd be very excited! You can find the code at github.com/andybalaam/rabbit-escape</a>.</p> Snake in Groovy Andy Balaam — Thu, 05 Feb 2015 17:51:11 +0000 Series: Groovy, Ruby</a>, BASIC</a>, Dart</a>, Elm</a>, Python3+Qt5</a></p> I'm starting a series where I write the game Snake in lots of programming languages.</p> I almost always use writing Snake as my way in to understand a new language, so I'll share my thoughts about each language as I go.</p> </iframe> Slides: Snake in Groovy</a></p> If you want to, you can Support me on Patreon</a>.</p> Programmatic equivalents of web.xml sections for Tomcat Andy Balaam — Thu, 05 Feb 2015 12:48:56 +0000 Most documentation for J2EE configuration is based on having a web.xml file, but I want to configure my Tomcat programmatically. Here are some of the things I have found out.</p> Please use the comments below to correct what I got wrong, and mention equivalents for other parts of web.xml.</p> Getting started</h2> <web-app ... </pre> in code becomes something like:</p> import org.apache.catalina.Context; import org.apache.catalina.startup.Tomcat; ... Tomcat tomcat = new Tomcat(); Context context = tomcat.addContext( "", "WebContent" ); </pre> Adding a Servlet</h2> <web-app ... <servlet> ... <servlet-name>MyServlet</servlet-name> <servlet-class>com.example.MyServlet</servlet-class> ... </servlet> <servlet-mapping> <servlet-name>MyServlet</servlet-name> <url-pattern>/foo/*</url-pattern> </servlet-mapping> </pre> in code becomes something like:</p> ... Class servletClass = MyServlet.class; // MyServlet extends javax.servlet.http.HttpServlet String servletName = servletClass.getSimpleName(); // Or something else if you like Tomcat.addServlet( context, servletName, servletClass.getName() ); context.addServletMapping( "/foo/*", servletName ); </pre> Adding a filter</h2> <web-app ... <filter> <filter-name>MyFilter</filter-name> <filter-class>com.example.MyFilter</filter-class> </filter> <filter-mapping> <filter-name>MyFilter</filter-name> <url-pattern>/bar/*</url-pattern> </filter-mapping> </pre> in code becomes something like:</p> ... import org.apache.catalina.deploy.FilterDef; import org.apache.catalina.deploy.FilterMap; ... Class filterClass = MyFilter.class; // MyFilter implements javax.servlet.Filter String filterName = filterClass.getSimpleName(); // Or something else if you like FilterDef def = new FilterDef(); def.setFilterName( filterName ); context.addFilterDef( def ); FilterMap map = new FilterMap(); map.setFilterName( filterName ); map.addURLPattern( "/bar/*" ); context.addFilterMap( filterMap ); </pre> Adding a Listener</h2> <web-app ... <listener> <listener-class>com.example.MyContextListener</listener-class> </listener> </pre> in code becomes something like:</p> ... context.addApplicationListener( MyContextListener.class.getName() ); // MyContextListener implements javax.servlet.ServletContextListener </pre> This is for a ServletContextListener: it may be similar for other listeners, but I'm not sure.</p> Comments</h2> Terence Bandoian 2016-07-11</span> </h3> I’d like to add that Context.addApplicationListener also appears to work for ServletRequestListener implementations.</p> Thanks for posting!</p> </div> Andy Balaam 2016-07-12</span> </h3> Thanks Terence!</p> </div> Gustavo Castillo 2016-07-24</span> </h3> You have a typo in the first example.</p> Tomcat tomcat = new Tomcat(); Context context = m_tomcat.addContext( "", "WebContent" );</p> </div> Andy Balaam 2016-08-01</span> </h3> Thanks Gustavo – fixed.</p> </div> JAX-RS (using Apache CXF) in embedded Tomcat example Andy Balaam — Tue, 13 Jan 2015 17:24:02 +0000 I had serious trouble today firing up an embedded Tomcat server that serves up REST resources using JAX-RS via Apache CXF.</p> Here's minimal example, hopefully saving you the same trouble:</p> src/HelloJaxRs.java:</p> import org.apache.catalina.startup.Tomcat; import org.apache.catalina.Context; import org.apache.catalina.Wrapper; public class HelloJaxRs { public static void main( String[] args ) throws Exception { Tomcat tomcat = new Tomcat(); Context context = tomcat.addContext( "/", "" ); Wrapper servlet = context.createWrapper(); servlet.setName( "jaxrs" ); servlet.setServletClass( "org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet" ); servlet.addInitParameter( "jaxrs.serviceClasses", Hello.class.getName() ); servlet.setLoadOnStartup( 1 ); context.addChild( servlet ); context.addServletMapping( "/rest/*", "jaxrs" ); tomcat.start(); tomcat.getServer().await(); } } </pre> src/Hello.java:</p> import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.PathParam; @Path( "/hello/{name}" ) public class Hello { @GET public String getThing( @PathParam( "name" ) String name ) { return "Hello, " + name; } } </pre> build.xml: <?xml version="1.0" encoding="UTF-8"?> <project default="run"></p> <target name="run" depends="compile"> <mkdir dir="bin/tomcat.8080/webapps"/> <java classname="HelloJaxRs" fork="true" failonerror="true" dir="bin" > <classpath> <pathelement location="bin/"/> <fileset dir="lib"><include name="**/*.jar"/></fileset> </classpath> </java> </target> <target name="compile"> <mkdir dir="bin"/> <javac srcdir="src" destdir="bin"> <classpath> <fileset dir="lib"><include name="**/*.jar"/></fileset> </classpath> <compilerarg value="-Werror"/> </javac> </target> </code></pre> </project> </pre></p> To download the dependencies (or you can do something clever with Maven): $ mkdir lib $ cd lib $ wget http://www.webhostingjams.com/mirror/apache/tomcat/tomcat-7/v7.0.57/bin/apache-tomcat-7.0.57.tar.gz $ tar -xzf apache-tomcat-7.0.57.tar.gz $ wget http://www.apache.org/dyn/closer.cgi?path=/cxf/3.0.3/apache-cxf-3.0.3.tar.gz $ tar -xzf apache-cxf-3.0.3.tar.gz $ cd .. </pre></p> To run Tomcat: $ ant </pre></p> Then, in another terminal, you can see it's working: $ curl http://localhost:8080/rest/hello/andy Hello, andy </pre></p> Vim persistent buffer list combined with saved sessions Andy Balaam — Thu, 08 Jan 2015 11:47:06 +0000 I like to use several saved Vim</a> sessions to remember what files I was working on in a particular project.</p> I also like to have a list of buffers I have open on the left-hand side of the screen that is always visible and up-to-date.</p> Sometimes, the existence of a generated buffer like the list of buffers can confuse the session restore code, but I have found a way to make this work nicely using the Buffergator</a> plugin and the built-in mksession</a> command.</p> Here are the relevant snippets from my .vimrc.</p> I am a beginner: please comment if you see bugs or possible improvements.</p> " Prerequisites: install the Buffergator plugin " Save session on exit, excluding problematic stuff like options set sessionoptions=blank,buffers,curdir,tabpages,resize,winpos,winsize au VimLeavePre * if v:this_session != '' | exec "mks! " . v:this_session | endif " Open Buffergator when you press F5 nnoremap <F5> :BuffergatorOpen<CR> " Various Buffergator options to make it persistent, " and displayed how I like let g:buffergator_autodismiss_on_select = 0 let g:buffergator_display_regime = "bufname" let g:buffergator_sort_regime = "filepath" let g:buffergator_autoupdate = 1 " Suppress the error message about the Buffergator buffer " when you restore a session function! OpenBuffergatorIfItsBufferExists() if bufnr("[[buffergator-buffers]]") > '' BuffergatorOpen normal o endif endfunction autocmd VimEnter * call OpenBuffergatorIfItsBufferExists() </pre> Now when I start vim with vim -S path/to/session.vim the session is restored and saved when I exit, with a persistent buffer list on the left, and no errors during the session restore.</p> Comments</h2> Val 2020-02-21</span> </h3> Hi,</p> unfortunately, this did not work for me. BufferGator still complained ("buffergator: only one buffer available") after having restored the session.</p> To make BufferGator happy after a session restore (performed using ":source Session.vim"), I’ve added the function below to my vimrc. The trick is to automatically call "bnext" as many times as available buffers. The snippet below also defines the command "LoadSession" while can load any saved session (ex: "LoadSession projectA.vim"). The keyboard shortcut "l" loads the default session (equivalent to LoadSession Session.vim) Best.</p> function! FuncLoadSession(name) exe "source " . a:name let i = len(getbufinfo({‘buflisted’:1})) let j = 0 while j <= 1 bnext let j+=1 endwhile endfunction</p> command! -nargs=1 LoadSession call FuncLoadSession() map l :call FuncLoadSession("Session.vim")</p> </div> Val 2020-02-21</span> </h3> so the angle brackets of my previous comment were removed automatically. Let’s try one more time using html character codes this time.</p> The last line, which defines the keyboard shortcut â€œlâ€, was supposed to be:</p> map l :call FuncLoadSession(â€œSession.vimâ€) <</p> </div> Snowflake Christmas card web page on the Raspberry Pi Andy Balaam — Tue, 16 Dec 2014 02:39:17 +0000 In this video I will show you how to make an electronic Christmas card for your friends or family using HTML and JavaScript, which means it will be a little web site that anyone can see by going to it in their Internet browser.</p> I'm doing this on the Raspberry Pi, but you can do the same thing on almost any computer that exists. All you need is a web browser (like Firefox or Internet Explorer) and a text editor (like Notepad or Gedit).</p> If this looks very difficult, try the video I made making a similar card using Scratch, which is a lot easier: Snowflake Christmas card in Scratch on the Raspberry Pi</a>.</p> If you'd like to use the snowflake picture I drew, right-click this link and choose "Save link as..." or similar: snowflake.svg</a>.</p> </iframe> If you'd like to compare my code against yours, right-click this link and choose "Save link as..." or similar: snowflakes.html</a>.</p> If you'd like to see what the finished product looks like, just left-click on snowflakes.html above, instead of right-clicking.</p> Snowflake Christmas card in Scratch on the Raspberry Pi Andy Balaam — Tue, 16 Dec 2014 02:27:52 +0000 In this video I will show you how to make an electronic Christmas card for your friends or family using Scratch.</p> Scratch can work on most computers - you can download it from http://scratch.mit.edu/</a>.</p> Scratch is already installed on your Raspberry Pi if you've got Raspian</a> or NOOBs</a> on your SD card.</p> </iframe> (If you want to get a Pi try my Raspberry Pi: Before we start</a> video.)</p> If you want to try something more advanced, you can make a similar card as a web page: Snowflake Christmas card web page on the Raspberry Pi</a>.</p> Encoding URLs in Java Andy Balaam — Thu, 04 Dec 2014 13:05:12 +0000 To encode a URL in Java, create a new instance of java.net.URI</a> and then call toString() or toASCIIString() on it.</p> DO NOT use java.net.URLEncoder</a>. It is for HTML form encoding, not encoding URLs, despite its name.</p> Each part of a URL must be escaped differently, so use the multi-argument constructors for URI where you have multiple parts you want to stick together. You can pass in null for arguments you want to omit, and it works the way you'd hope.</p> This program:</p> class EncodeUrls { public static void main( String[] args ) throws java.net.URISyntaxException { url( "http", "example.com", "/x/y/x", "x=y", "pos" ); url( "http", "exa\uD83D\uDCA9mple.com", "/x/\uD83D\uDCA9/x", "x=\uD83D\uDCA9", "po\uD83D\uDCA9" ); } private static void url( String scheme, String host, String path, String query, String fragment ) throws java.net.URISyntaxException { java.net.URI uri = new java.net.URI( scheme, host, path, query, fragment ); System.out.println( "." + uri.toString() ); System.out.println( "." + uri.toASCIIString() ); System.out.println(); } } </pre> prints:</p> .http://example.com/x/y/x?x=y#pos .http://example.com/x/y/x?x=y#pos .http://exaðŸ’©mple.com/x/ðŸ’©/x?x=ðŸ’©#poðŸ’© .http://exa%F0%9F%92%A9mple.com/x/%F0%9F%92%A9/x?x=%F0%9F%92%A9#po%F0%9F%92%A9 </pre> Books that changed my life Andy Balaam — Tue, 02 Dec 2014 11:56:22 +0000 Further Programming for the ZX Spectrum</strong> by Ian Stewart, Robin Jones</p> </li> Foundation</strong> by Isaac Asimov</p> </li> Crime and Punishment</strong> by Fyodor Dostoyevsky</p> </li> Structure and Interpretation of Computer Programs</strong> by Harold Abelson, Gerald Jay Sussman, Julie Sussman</p> </li> Slaughterhouse 5</strong> by Kurt Vonnegut</p> </li> Possibly: Children of Men</strong> by P.D. James (too early to say)</p> </li> </ul> Why I use Linux Andy Balaam — Mon, 01 Dec 2014 09:52:14 +0000 I have used Linux at home for quite a few years, and a couple of years ago I changed my work machine from Windows to Ubuntu Linux</a>. It's made life at work better for me.</p> Better for me</h2> This move has made me much more productive, saving valuable time and money. Here is why:</p> Security updates happen automatically, without hitches, and rarely requiring reboots.</h3> Since I changed, security updates are applied almost as soon as they are made available with no interaction from me. I have never seen a security upgrade that broke my system and prevented me working, whereas before I saw this several times.</p> My computer works faster.</h3> My computer feels much faster to work with. In recent tests for our project, we found that the time spent waiting for a compile on my Linux machine was half that spent waiting on an identical system using Windows.</p> I work faster.</h3> Because my system is more customisable than before, I can work the way that feels more comfortable to me. Support for keyboard shortcuts is more comprehensive in Linux, and I can work using the keyboard for most things, which helps prevent recurrence of RSI, from which I have suffered in the past.</p> I can automate repetitive tasks.</h3> It's much easier to write small scripts on Linux for things that I do every day. This means I spend more time thinking about the work I have to do, instead of all the button-clicking required to do it.</p> I can have more programs open.</h3> My computer now works comfortably simultaneously running 3 or 4 heavy corporate applications, at least 2 copies of Eclipse, Firefox with many tabs open, several Chrome windows, 3 text editor windows, a 1GB virtual machine and many other programs, without slowing down or crashing. When I used Windows I needed fewer programs open, but because Linux manages memory better and many of the programs are less memory-hungry, I can do more things at the same time.</p> I can use "virtual desktops".</h3> It's hard to explain why until you've tried it, but virtual desktops are an incredibly addictive feature. I avoid neck strain by using a single monitor directly in front of me*, but I am able to have lots of separate work spaces and can switch very quickly between them. I use one desktop for email and instant messaging, another for the Internet, and several more for programming in different environments. Switching between them happens as fast as I can press the relevant keyboard shortcut. This is a killer feature of modern Linux desktops in my opinion.</p> [*Multiple-monitor setups are well-supported too, of course.]</p> Modern tools work better.</h3> Modern software - especially programming software - is frequently developed on Linux, and has its "home" there, meaning it is easier to install and update than on other platforms. Examples include: Python, Node.js, Perl, Vim, Emacs, Bash and Ruby, all of which are much easier to use and install on Linux than on Windows.</p> Better for my company</h2> Because I use Linux, my company saves time and money.</p> Administering machines is easier.</strong> I get Ubuntu and corporate security updates automatically through the built-in security update mechanisms. The corporate updates are maintained by volunteers at my work in their spare time.</li> It's cheaper.</strong> There is no license fee to use Ubuntu, and no added-value version to buy. Support can be paid for from several companies, including the company behind Ubuntu, Canonical.</li> I am more productive.</strong> As explained above, I am able to do more work in the same time because I use Linux.</li> </ul> LibreOffice spell check not working in Lubuntu Andy Balaam — Wed, 12 Nov 2014 20:28:39 +0000 I installed LibreOffice into Lubuntu, but the spell checking didn't work.</p> It turns out I need to install a dictionary for my locale:</p> sudo apt-get install myspell-en-gb </pre> I needed to restart LibreOffice, and then I got spell checking as I typed.</p> Is it ok to represent dates as Unix time (seconds since the epoch) in JSON? Andy Balaam — Thu, 06 Nov 2014 14:00:15 +0000 Yes.</p> If your JSON contains integer numbers that represent Unix time</a> (seconds since the "epoch"), and you parse your JSON to JavaScript, the range of integers that can accurately be represented is -9007199254740992 to 9007199254740992</a> (ref: EcmaScript standard</a>, section 8.5).</p> The year 4000AD starts at unix time 64060588815, which is 9007135194152192 less than the maximum.</p> The year 1AD started at unix time -2177452800, which is 9007197077288192 more than the minimum.</p> So there are enough numbers.</p> Using hexdump to represent a binary file in Java source code Andy Balaam — Wed, 05 Nov 2014 17:03:23 +0000 To embed a binary file (e.g. a png image) into Java source code:</p> $ hexdump -v -e '/1 "%3i, "' myfile.png -119, 80, 78, 71, 13, 10, 26, 10, 0, 0, 0, 13, 73, 72, 68, 82, ... </pre> Copy and paste the output into Java code like this:</p> private static final byte[] myfile = { (byte)-119, 80, 78, 71, ... }; </pre> Disabling screensaver on LXDE Andy Balaam — Sat, 18 Oct 2014 09:17:10 +0000 Caffeine</a> wasn't working for me. I got an error like this:</p> dbus.exceptions.DBusException: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.ScreenSaver was not provided by any .service files </pre> Bug 1280449</a> contained the solution for me. Thank you Paulius Vitkus:</p> sudo apt-add-repository ppa:behda/ppa sudo apt-get remove caffeine sudo apt-get update && sudo apt-get install caffeine </pre> Android development - saving state Andy Balaam — Fri, 17 Oct 2014 07:16:19 +0000 Series: Setup</a>, Menu</a>, Drawing</a>, non-Android</a>, Working</a>, Saving state.</p> Android apps like Rabbit Escape</a> need to save their state when asked, and restore themselves later. This happens when you rotate the screen, and could happen at various other times. Here's how I handled that in Rabbit Escape:</p> </iframe> Android development - Rabbit Escape really working on Android Andy Balaam — Tue, 07 Oct 2014 20:04:46 +0000 Series: Setup</a>, Menu</a>, Drawing</a>, non-Android</a>, Working, Saving state</a>.</p> Up until now, you weren't sure to believe my promises that Rabbit Escape</a> really was going to be an Android game, since I hadn't actually got it running on Android. Well, the wait is over:</p> </iframe> Showing urgent (flashing) windows on all desktop in LXDE's taskbar (window list) in Lubuntu Andy Balaam — Thu, 25 Sep 2014 23:55:24 +0000 As it stands, lxpanel's taskbar plugin</a> (part of the LXDE desktop you get with Lubuntu) does not show urgent (flashing) windows if they are on a different virtual desktop from the one you are looking at.</p> This seems wrong to me (I logged Bug 682</a>), and the fix is a little one-liner:</p> --- lxpanel.orig/src/plugins/taskbar.c 2013-08-27 23:57:55.000000000 +0100 +++ lxpanel/src/plugins/taskbar.c 2014-09-26 00:48:25.026855589 +0100 @@ -202,10 +202,10 @@ tk->flash_timeout = g_timeout_add(interval, (GSourceFunc) flash_window_timeout, tk); } -/* Determine if a task is visible considering only its desktop placement. */ +/* Determine if a task is visible considering only its desktop placement and urgency. */ static gboolean task_is_visible_on_current_desktop(TaskbarPlugin * tb, Task * tk) { - return ((tk->desktop == ALL_WORKSPACES) || (tk->desktop == tb->current_desktop) || (tb->show_all_desks)); + return ((tk->desktop == ALL_WORKSPACES) || (tk->desktop == tb->current_desktop) || (tb->show_all_desks) || tk->urgency); } /* Recompute the visible task for a class when the class membership changes. </pre> To install this patch into Lubuntu, do something like this: sudo apt-get install build-essential fakeroot dpkg-dev sudo apt-get build-dep lxpanel mkdir lxpanel cd lxpanel apt-get source lxpanel cd lxpanel-* wget https://sourceforge.net/p/lxde/bugs/682/attachment/show-urgent-windows-on-all-desktops.patch patch -p1 < show-urgent-windows-on-all-desktops.patch dpkg-buildpackage -rfakeroot -b cd .. sudo dpkg -i lxpanel_*.deb killall lxpanel lxpanel --profile Lubuntu </pre></p> Launch an urgent window using Python and Xlib with the UrgencyHint flag Andy Balaam — Thu, 25 Sep 2014 23:25:47 +0000 I am trying to fix a bug in lxpanel's taskbar plugin, and needed to launch an urgent window. Here's how I did it in a little python.</p> #!/usr/bin/python # urgent.py -- launch an urgent window (Copyright messages are at the bottom) # To use: # sudo apt-get install python-xlib # ./urgent.py import sys import Xlib from Xlib import X, display, Xutil d = display.Display() s = d.screen() w = s.root.create_window( 50, 50, 300, 200, 2, s.root_depth, X.InputOutput, X.CopyFromParent, background_pixel = s.white_pixel, ) w.set_wm_name( 'Urgent!' ) w.set_wm_hints( flags = Xutil.UrgencyHint ) w.map() try: while 1: e = d.next_event() except Xlib.error.ConnectionClosedError: pass # This code is based on: # examples/xrandr.py -- demonstrate the RandR extension # from http://python-xlib.sourceforge.net/ # # Copyright (C) 2014 Andy Balaam # Copyright (C) 2009 David H. Bronke # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA </pre> Android programming - a non-Android, Android Game Andy Balaam — Wed, 24 Sep 2014 06:04:03 +0000 Series: Setup</a>, Menu</a>, Drawing</a>, non-Android, Working</a>, Saving state</a></p> We're planning to write an Android game. So why would we deliberately avoid Android while we're writing it? To make sure we're not overly-dependent on the Android ways of doing things, and are able to run our tests etc. on the local machine:</p> </iframe> Android programming - drawing bitmaps in a game loop Andy Balaam — Sun, 21 Sep 2014 19:12:36 +0000 Series: Setup</a>, Menu</a>, Drawing, non-Android</a>, Working</a>, Saving state</a></p> I will be talking about how to write Android programs that share code with non-Android programs. The program I will be writing is a simple game.</p> This time we're writing a game loop and drawing images onto a Canvas using a SurfaceView:</p> </iframe> Don't use email auto-forwarding (emails will go missing) Andy Balaam — Fri, 19 Sep 2014 02:47:35 +0000 I've been trying to chase down a mysterious problem with emails going missing. The problem with things being missing is that you can't see them, so it has been difficult to track down exactly what's going on, but it became clear that we were receiving no email at all from Amazon or Play.com for several months, and that some other mails had disappeared too. I was secretly assuming that I'd screwed something up in the terrifying postfix+dovecot+fetchmail+spamassassin+procmail config on our server, but I couldn't find any indication of what could be going wrong.</p> I finally got around to considering whether it was someone else's fault, and very surprisingly, my ISP Plusnet</a> and my web hosts Bluehost</a> were both very responsive and helpful when I contacted them.</p> I thought an over-enthusiastic spam filter might be swallowing mails, but it turns out the problem was that I was "forwarding" emails from my domain to my ISP-provided mailbox.</p> Since the SPF</a> anti-spam system has begun to be implemented, emails from people like Amazon come with SPF policies that make plain message forwarding</a> fail, because forwarding makes it look like emails come from the server forwarding them, when actually they come from the original sender. This makes it impossible to do SPF, so the emails were getting silently deleted.</p> Now that I know what the problem is, I have even found a sentence describing this on the Wikipedia page on SPF</a>: "SPF breaks plain message forwarding" and this useful article: Why forwarding your email is a BAD idea</a>.</p> The only solution I can see is to set up a mailbox on my domain host's site, and stop using the mailbox provided by my ISP. I was only using this mailbox as a holding area before I pulled down the mail via POP3, so it doesn't actually make much difference to me.</p> If you're using GMail you can set it up to pull from a POP3 server, and some other webmail services let you do that too, so using your domain host's web server doesn't have to be inconvenient.</p> Losing email makes me stressed.</p> Comments</h2> Marie Seltenrych 2017-08-23</span> </h3> I read this with interest as I have a similar problem. I thought I fixed forwarding emails but thus far no emails are getting through. I do also have a Gmail account (plus 2 others). Your comments give me hope once again that I am not a complete idiot. Thanks a lot. Marie :)</p> </div> Andy Balaam 2017-08-23</span> </h3> Yes, I live in perpetual confusion about how it really works, and perturb it until it seems to work when it stops. This is obviously not a good security posture :–)</p> </div> Fetchmail "mda" option makes emails bypass the spamassassin filter in postfix Andy Balaam — Fri, 19 Sep 2014 02:19:56 +0000 I have spent several months with no spam filtering in my postfix+dovecote+spamassassin+fetchmail+procmail setup, and no idea why. I had spamassassin enabled and working when I piped emails into it manually, and I had lines like this in my postfix master.cf:</p> # Listen on port 2025, not 25, and filter with the spamassassin service 2025 inet n - - - - smtpd -o content_filter=spamassassin # ... # Definition of the spamassassin service: pipe to spamc, then sendmail spamassassin unix - n n - - pipe user=spamd argv=/usr/bin/spamc -x -e /usr/sbin/sendmail -oi -f ${sender} ${recipient} </pre> But no matter what I did, I saw no spamassassin headers in my emails, and no mention of using it in the logs. Frustratingly, it was filtering perfectly on outgoing mail, which I really don't need...</p> Today I worked out what the problem was - nothing to do with postfix or spamassassin config, but to do with my .fetchmailrc.</p> Because I was running on a non-standard port, and also because I had tried not allowing unencrypted connections to postfix even from localhost, I was using telling fetchmail to deliver with /usr/sbin/sendmail, instead of allowing it to connect to postfix on port 25 as it normally does. So my .fetchmailrc contained:</p> poll example.com with proto POP3 user 'x' there with password 'y' is 'z' here mda "/usr/sbin/sendmail -i -f %F -- %T" </pre> It turns out this completely bypasses all the filters, and delivers the mail immediately. So, what I needed was the syntax to connect on a non-standard port, which looks like:</p> poll example.com with proto POP3 user 'x' there with password 'y' is 'z' here smtphost localhost/2025 </pre> (If your postfix listens on port 25, you don't need the smtphost line at all.)</p> And now my email is being filtered for spam, and put into a Junk folder by procmail.</p> Android programming - a menu using Activities and Intents Andy Balaam — Wed, 17 Sep 2014 07:04:22 +0000 Series: Setup</a>, Menu, Drawing</a>, non-Android</a>, Working</a>, Saving state</a></p> I will be talking about how to write Android programs that share code with non-Android programs. The program I will be writing is a simple game.</p> This time we're making a simple menu structure:</p> </iframe> Raspberry Pi Minecraft Python summer project screenshots Andy Balaam — Tue, 16 Sep 2014 20:39:08 +0000 [Screenshots of Minecraft on the Raspberry Pi can't be made with VNC</a> or screenshot tool like scrot</a>, but they can be made with the excellent raspi2png</a>.]</p> </blockquote> We did some Python programming in Minecraft on the Raspberry Pi</a> for our summer projects.</p> Child 2 made some lovely houses and enjoyed destroying things much more efficiently than when you do it by hand:</p> and Child 0 made a spell book. You can see the "elements" spell has been cast in the background (earth, air, water and fire), the "topsy-turvy" spell on the right, the "frozen" spell on the left, and on the far left you can just see a bit of the "river" spell:</p> To cast spells you must first utter the magical incantations:</p> python </pre> and then:</p> from spells import * </pre> then each spell can be cast by simply saying its name followed by the double brackets of power</em>, for example:</p> topsyturvy() </pre> Android programming - setting up Android Studio Andy Balaam — Thu, 11 Sep 2014 06:57:59 +0000 Series: Setup, Menu</a>, Drawing</a>, non-Android</a>, Working</a>, Saving state</a></p> I will be talking about how to write Android programs that share code with non-Android programs. The program I will be writing is a simple game.</p> First, how to set up Android Studio:</p> </iframe> What is a string? Andy Balaam — Sat, 23 Aug 2014 08:34:10 +0000 Most programming languages have some wrinkles around unicode and strings*. In my ficticious language Pepper</a>, there are no wrinkles of any kind, and everything is perfect.</p> *E.g. JavaScript</a>, Java</a>, Haskell</a>, Ruby</a>, Python</a>.</p> </blockquote> There are several key concepts. The most important are an interface AnyString and the variable** String which is what you should use when you are writing code with strings.</p> **String is a variable that refers to a type, so you just use it like a type and don't worry about it.</p> </blockquote> interface AnyString { def indexable(CodePoint) code_points( implements(AnyString) string ) } </pre> In Pepper an interface can describe what free functions exist as well as what member function a class must have, and here we just require that a code_points function exists that gives us a collection of CodePoint objects that may be indexed (i.e. is random-access).</p> When your Pepper program starts, the String variable will refer to something that implements this interface, and probably some other interfaces too. Most Pepper programs will use a String that is implemented as an array of bytes representing a string in UTF-8, but the programmer doesn't need to be aware of that, and in a situation where something different is needed (e.g. where we know lots of non-Latin characters will be used and UTF-16 will be more efficient) String can be set to something different in the configuration settings used by the compiler.</p> When you want to do something with a string, there will be functions that only rely on the AnyString interface and deal with CodePoints internally, but there will be other overloads that are potentially more efficient, for example there are two versions of the standard print function:</p> def void print( implements(AnyString) string ) def void print( NativeUtf8String string ) </pre> The NativeUtf8String class is implemented as a std::string in the C++ code emitted by the Pepper compiler, and the most efficient way to represent an array of bytes when compiling onto other platforms, so the version of print that uses it can be quite efficient.</p> Because all these types are known at compile time, the C++ code generated by the Pepper compiler can use the native types directly (and be efficient), even though the programmer is writing code using just the AnyString and String types, meaning their code can be adapted to other platforms by using a different configuration.</p> The Pepper environment exposes standard-out and standard-in as UTF-8 streams, and takes care of converting to the platform encoding for you (at runtime).</p> Absolute Truth in programming languages Andy Balaam — Fri, 22 Aug 2014 09:38:00 +0000 Is enforcing truthfulness the opposite of beauty?</p> Can 2 + 2 = 5?</p> Improvements, corrections, further contributions are welcome.</p> $ cat five.cpp #include <iostream> int operator+( int x, int y ) { return 5; } int main() { std::cout << 2 + 2 << std::endl; } $ g++ five.cpp five.cpp:2:29: error: â€˜int operator+(int, int)â€™ must have an argument of class or enumerated type </pre> $ python >>> int.__add__ = lambda y: 5 TypeError: can't set attributes of built-in/extension type 'int' </pre> $ cat five.hs import Prelude hiding ((+)) x + y = 5 main = print ( 2 + 2 ) $ ghc five.hs && ./five 5 </pre> $ cat five.rb class Fixnum def +(y) 5 end end print 2 + 2 $ ruby five.rb 5 </pre> $ mzscheme > (define (+ x y) 5) > (+ 2 2) 5 </pre> Options for code reviews with Git Andy Balaam — Mon, 18 Aug 2014 10:19:24 +0000 We're thinking about switching to Git for my work, and I want to be confident we can still support good code reviews if we make the switch.</p> I am a big fan of in-person reviews, and for that, git difftool is enough but sometimes you need to do it asynchronously, and then you need a tool or a process or something.</p> Here are the options as I see them so far (please comment if you know others I should consider):</p> Emailing patches.</strong> Git has git format-patch and git bundle that allow creating a file containing changes that can be sent by email or message. These can be reviewed as patches or applied to the working tree and reviewed in context.</li> Feature branch and pull request.</strong> Devs to push their changes to a branch in a shared repo and send an email or message asking a colleague to pull the branch. The reviewer looks at the changes in the repo or pulls them, then either sends back comments, or merges the branch into their own and delivers to the master branch.</li> Tools.</strong> There are several extra tools that sit in front of Git and deliver changes when they are reviewed. These include: Gerrit</a>, Critic</a>, Review Board</a>.</li> </ol> References:</p> StackOverflow: Using git for Code Reviews?</a></li> A successful Git branching model</a></li> Integrating topic branches in Git</a></li> </ul> Desktop sharing not working in Lubuntu 14.04 Andy Balaam — Tue, 12 Aug 2014 14:57:07 +0000 On my recently-upgraded to 14.04 Lubuntu machine, desktop sharing didn't work.</p> Here's how I made it work:</p> Once (to allow some Windows clients to connect without encryption):</p> gsettings set org.gnome.Vino require-encryption false</pre> Then every time I want to share my desktop:</p> /usr/lib/vino/vino-server</pre> and leave that running while I'm sharing.</p> Best GCC warning flags for compiling C++ Andy Balaam — Fri, 18 Jul 2014 09:01:29 +0000 A recent discussion on ACCU-general</a> gave people an opportunity to share the warning flags they like to use with g++.</p> I thought I'd write down the consensus as I understood it, mainly for my own reference:</p> -Wredundant-decls -Wcast-align -Wmissing-declarations -Wmissing-include-dirs -Wswitch-enum -Wswitch-default -Wextra -Wall -Werror -Winvalid-pch -Wredundant-decls -Wmissing-prototypes -Wformat=2 -Wmissing-format-attribute -Wformat-nonliteral </pre> We were advised by Jonathan Wakely</a> that -Weffc++ is not very useful since it is mostly based on the first edition of the book Effective C++, many of whose recommendations were improved in the second edition, and also apparently GCC doesn't do a great job of warning about them.</p> Update:</strong> thanks to an article[1] by Roger Orr in CVu</a> these flags are highly recommended in GCC 5.2+:</p> -flto -Wodr </pre> [1] Orr, Roger "One Definition Rule</a>", in CVu Volume 27, Issue 5 p16 (editor: Steve Love)</p> Comments</h2> No Name 2019-05-09</span> </h3> -Wredundant-decls is duplicated</p> </div> Martin Moene 2014-07-19</span> </h3> Thanks Andy,</p> Would be nice to have each one linked to the right spot in the documentation [1], but that seems unsupported.</p> What advised the ordering of this list?</p> cheers, Martin</p> [1] e.g. http://gcc.gnu.org/onlinedocs/gcc/Warning-Opt...</a>, -Wcast-align</p> </div> Andy Balaam 2014-07-20</span> </h3> Hi Martin, I don’t think the order matters does it? I just copied in from people’s emails in the order I found them…</p> </div> Martin Moene 2014-07-21</span> </h3> Hi Andy,</p> The compiler probably cares less :)</p> -Wall -Wcast-align -Werror -Wextra -Wformat-nonliteral -Wformat=2 -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-prototypes -Wredundant-decls -Wredundant-decls -Wswitch-default -Wswitch-enum</p> </div> What git server should I use? Andy Balaam — Tue, 15 Jul 2014 15:40:35 +0000 At work we are considering whether we can use Git for our source control. I am a big fan of Git, so I'd like to see this happen.</p> We only need to work against a central repository most of the time, so I'm looking at what servers might work for us.</p> Update: This StackExchange question may help: Self-hosted replacement for Github</a>.</p> Update: Added software from the StackExchange answers to the list.</p> Features we will need:</p> User management</li> Repository management</li> Browsing code and diffs via the web</li> Hosted in-house</li> </ul> Features we might want:</p> External user authentication e.g. via LDAP</li> Code review</li> Integrating with an issue tracker</li> </ul> Most of my use of Git so far has been against large servers like GitHub</a> (which I really like) and SourceForge</a>, but recently I set up a test Git server using gitolite</a> and gitweb</a>, which gives me my 4 "needs" above but not my 3 "wants". It also requires command-line use of git to administer SSH keys for users, so might not suit our system administrators or all our developers.</p> So, lazyweb, what server should I recommend?</p> Here is my research so far:</p> Free git server software</h2> GitLab</a> - looks a lot like GitHub, and appears to satisfy all 4 of my needs and all 3 of my wants. Might be a bit decentralised (ironically) for our usage e.g. the docs talk about using merge requests for code review whereas I'd expect we'd want a commit-gating style which is what I believe Gerrit provides.</p> SCM-Manager</a> - looks very corporate. Likely it could satisfy my needs and my wants.</p> Gerrit</a> + Gitblit</a> - lots of code review features, used by major projects including the Android open source project. Weird that Gerrit doesn't include a code browser and you have to add something like Gitblit. I think this will give me all 4 of my needs and all 3 of my wants.</p> Gitolite</a> + gitweb</a> - this is what I am using at the moment, and it works well, satisfying the needs above, but not the wants. Gitolite configuration is done by editing config files and pushing them into a special git repository on the server. Adding users means adding a user's SSH key to the config repository, so requires tech-savvy users and admins. gitweb is fast and clear. My only complaint is that you don't seem to be able to control the amount of context you see in a diff (often I want to see the full files).</p> Gitorious</a> - the software behind a mature public site that some people really like. Doesn't appear to do LDAP authentication, and may not integrate with issue tracking.</p> tuleap</a> - project planning, chat, issue tracking, builds, document management, discussion board, news all in one product. Includes Gerrit for code reviews, Jenkins for build management. Supports LDAP and OpenID authentication.</p> Phabricator</a> - code hosting and review, issue tracking, wiki, alerts, message boards, blogs, Q&A, polls all in one product. Supports LDAP and OAuth authentication.</p> GitPrep</a> - explicitly a clone of GitHub. Seems to look nice, but a young project and not talked about much on the Internet yet.</p> GitBucket</a> - explicitly a clone of GitHub. Couldn't find much information beyond that.</p> GitList</a> - only a repository viewer, but could possibly be used with gitolite instead of plain gitweb - may have more features such as full-file diffs, but I'm not sure. Looks pretty, and doesn't have much documentation.</p> CGit</a> - only a repository viewer, but definitely allows specifying the amount of context in diffs (and if you edit the URL directly you can ask for as much as you want). Very plain interface, and minimal documentation. Claims to be fast.</p> Cydra</a> - may turn out to be good but no web site at the moment, so probably not mature enough to consider.</p> Gitosis</a> appears to be a dead project.</p> Paid (in-house) git server software</h2> Very unlikely that we will pay for anything, but here are the options I have found so far:</p> Atlassian Stash</a> - one-time payment e.g. $6,000 for 100 users.</p> GitHub Enterprise</a> - apparently they do in-house installations but I couldn't find any information. GitHub has an excellent interface and features.</p> Microsoft Team Foundation Server</a> - a larger system that offers Git integration as a feature.</p> RhodeCode</a> - 50 users for $199/month, supports LDAP and Active Directory authentication, and code review. Claims to be highly secure.</p> Comments</h2> jesper hogstrom 2015-07-17</span> </h3> You can hook gitolite to an LDAP/AD server. We use it extensively, and use the groups to provide access rights. We use Crucible for reviews and not much integration with an issue tracker, but I think it could be done.</p> That said, I am investigating options. Gitlab is high on the list, though I believe Visual Studio Online might be a big bang replacement for all of it (Jira, Crucible, TeamCity, git/gitolite).</p> </div> Andy Balaam 2015-07-19</span> </h3> Thanks jesper.</p> </div> Jesper HÃ¶gstrÃ¶m 2015-07-21</span> </h3> Obviously, if you want the glue scripts to hook up gitolite to AD, just mail me!</p> </div> Andy Balaam 2015-07-21</span> </h3> Thanks – GitLab does seem to be emerging as a winner for me too, especially as it now seems to provide a continuous integration server too.</p> </div> Centralised version control repositories | dazdaztech blog 2016-04-04</span> </h3> […] This is also worth a quick read http://www.artificialworlds.net/blog/2014/07/...</a> […]</p> </div> DOSUNMU ADETOKUNBO NASIR 2017-08-01</span> </h3> gitblit any day……now has AD</p> </div> SUNSAN 2014-12-09</span> </h3> Which option did you go with? Would you recommend it to a corp with a staff of approx 50 devs?</p> </div> Andy Balaam 2014-12-11</span> </h3> Sadly, we haven’t got there yet, but I was thinking GitLab or Gerrit would probably be the right choice. TBH if I had my own way (which I won’t) I’d go for GitHub itself – not the hosted version, but the real one, with private repos.</p> </div> thoroc 2015-01-08</span> </h3> I am looking into this for work as well, and I have found at least three more candidates:</p> https://www.deveo.com/</a> (commercial) – not tested, but their page looks good </li> http://gogs.io/</a> (FOSS) – tested, can be buggy at time, but cross platform and have all the basic you could expect from a GitHub replacement </li> http://bonobogitserver.com/</a> (FOSS) – that one is a bit of an ugly duckling as it is a .Net based one so windows/IIS server only </li> </ul> I have tried Stash and Rhodecode, and although I have heard people raving how good the former is, I have yet to have a good experience with it. Same with Rhodecode. But maybe it was due to the fact that I have to run both on a weak VM.</p> </div> Andy Balaam 2015-01-09</span> </h3> Thanks for the extra info, useful.</p> </div> Renewing self-signed certificate for ejabberd Andy Balaam — Tue, 15 Jul 2014 08:44:41 +0000 I run an ejabberd</a> server on an Ubuntu 12.10 box and this week I started getting notified by my IM client that the server's certificate had expired.</p> Here's how I managed to generate a new certificate.</p> WARNING: this process backs up, deletes and then restores your ejabberd database, so it is probably fairly risky.</p> # Move any previous backups out of the way sudo mv /var/backups/ejabberd-* ~/Desktop/ # Move the expired certificate out of the way sudo mv /etc/ejabberd/ejabberd.pem /etc/ejabberd/ejabberd.pem.old # Reconfigure the ejabberd package (WARNING: backs up and deletes your database!) sudo dpkg-reconfigure ejabberd # Make the database backup file readable sudo chmod a+rx /var/backups/ejabberd-*/ sudo chmod a+r /var/backups/ejabberd-*/* # Restore the backup sudo ejabberdctl restore /var/backups/ejabberd-*/ejabberd-database </pre> If you're lucky, your server will now be back up with a new self-signed certificate.</p> In general, the policy of using dpkg-reconfigure to handle creating new self-signed certificate seems to work nicely.</p> Comments</h2> Jim 2015-09-15</span> </h3> Thanks. That seems to have worked OK for me too. However I’m not sure about the restoring of the database. I get a file not found error at that stage, but I’m not clear what you’re restoring. Are you restoring the database you just created, or should that last line reference the stuff you backed up to ~/Desktop? I have a working system so I’m not /too/ concerned :–)</p> </div> Andy Balaam 2015-09-15</span> </h3> Hi Jim, that should be restoring the database you just created. Moving stuff to Desktop is just getting any old backups out of the way.</p> </div> Andy 2015-10-26</span> </h3> Why not use the routine creating the cert? In postinstall of debian ejabberd package it is (modified with the remove part):</p> $: cat /root/scripts/regenerate-ejabberd.pem.sh #!/bin/bash</p> script to generate new ejabberd.pem</h1> Making /etc/ejabberd/ejabberd.pem if it does not exist.</h1> cd /etc/ejabberd if [ -f ejabberd.pem ] rm ejabberd.pem fi if [ ! -f ejabberd.pem ] then echo "Generating SSL certificate /etc/ejabberd/ejabberd.pem…" EHOSTNAME=$(hostname -s 2>/dev/null || echo localhost) EDOMAINNAME=$(hostname -d 2>/dev/null || echo localdomain) openssl req -new -x509 -days 365 -nodes -out ejabberd.pem -keyout ejabberd.pem > /dev/null 2>&1 < last line is fi) and then make it executeable with chmod a+x</p> </div> Andy Balaam 2015-10-27</span> </h3> Thanks Andy, good suggestion.</p> </div> Christian 2020-04-20</span> </h3> This no longer works. Is there an update on how to do it with the latest ejabberd available with raspbian?</p> </div> Jeff Cook 2014-09-19</span> </h3> Worked for me! :) Thanks</p> </div> Letter to my MP on DRIP Andy Balaam — Mon, 14 Jul 2014 10:19:21 +0000 Sent via WriteToThem.com</a>.</p> More Info</a></p> Dear Mr Hammond,</p> I am writing to express my concern about the Data Retention and Investigatory Powers Bill (DRIP).</p> I feel that forcing companies to store data on their customers is an unacceptable breach of the right to privacy, and not justified by the need to combat terrorism or other crime. I think a better balance could be struck by requiring agencies to get a court order to engage in such invasive surveillance.</p> I am also concerned about the elements of DRIP that amend the Regulation of Investigatory Powers Act (RIPA), including the expansion of the definition of a communications service provider.</p> What emergency are we facing that requires parliament to move so quickly on this issue? The only emergency I can see is that UK government agencies are currently acting illegally.</p> These agencies should stop acting illegally, and laws should be drafted in the normal way to authorise proportionate surveillance.</p> I am particularly concerned that DRIP contains many similarities to legislation that has already been rejected by parliament.</p> I look forward to your response.</p> Yours sincerely,</p> Andy Balaam</p> [Inspired by Leo McArdle</a>, thanks.]</p> [Feel free to re-use in whole or in part.]</p> Comments</h2> Chris 2014-07-14</span> </h3> I hope you don’t mind but I’v copied your letter and sent it to my MP. Thank you.</p> </div> Andy Balaam 2014-07-14</span> </h3> Really glad to hear it was helpful! I’ve added a comment making it explicit you can copy it.</p> </div> What is node.js? Andy Balaam — Mon, 14 Jul 2014 07:21:38 +0000 Node.js is a way of writing your web applications wholly in JavaScript, which sounds like a terrible idea.</p> However, it is also an asynchronous programming environment that has been proven to scale really well, and it's a collection of libraries many of which are excitingly small and well-encapsulated.</p> Here we just cover the basics of what it is:</p> </iframe> Slides: What is node.js?</a></p> A mini mocking framework Andy Balaam — Mon, 07 Jul 2014 07:18:08 +0000 Ever wondered how a mocking framework like JMock or GoogleMock works?</p> We'll look at how to write a very basic example, and find out that Java is more flexible than you may have thought.</p> </iframe> Slides: A mini mocking framework</a>.</p> Simple template programming Andy Balaam — Mon, 30 Jun 2014 07:16:56 +0000 C++ template meta-programming sounds harder than it is.</p> So long as you can fight through some horrific syntax and understand recursive functions, you can write any algorithm you like and have it run at compile time in your C++ programs.</p> </iframe> Slides: Simple Template Programming</a></p> Andrei Alexandrescu's amazing book on using template meta-programming for really useful, cool stuff is: Modern C++ Design</a>.</p> Five Quines video Andy Balaam — Wed, 25 Jun 2014 07:34:28 +0000 A quine is a program that prints out its own source code. I will describe five examples:</p> </iframe> Slides: Five Quines</a></p> Arguably the greatest program ever written:</p> Qlobe</a>.</p> More info on quines:</p> http://www.madore.org/~david/computers/quine.html</a></li> https://en.wikipedia.org/wiki/Quine_%28computing%29</a></li> http://rosettacode.org/wiki/Quine</a></li> http://c2.com/cgi/wiki?QuineProgram</a></li> update:</strong> just, wow: github.com/taylorconor/quinesnake</a></li> </ul> OpenID security in a REST API Andy Balaam — Mon, 23 Jun 2014 07:21:06 +0000 Series: What is REST?</a>, A Basic REST API</a>, Paging</a>, Search</a>, Security</a>, Token</a>, OpenID.</p> We are implementing a REST API. Today, using OpenID to do our authentication, instead of doing it ourselves:</p> </iframe> Slides: OpenID security in a REST API</a>.</p> Code: poemtube on codeberg</a>.</p> avconv performance slow in Ubuntu 14.04 trusty Andy Balaam — Thu, 15 May 2014 00:50:02 +0000 I noticed that my TV conversion scripts were suddenly running very slowly after upgrading to Lubuntu 14.04.</p> After some investigation it turned out it had started defaulting to using libx264 encoding when the output file was named *.mp4.</p> When I ask it to use mpeg4 encoding instead, it goes fast again:</p> avconv -i MyProg.ts -acodec copy -b:v 2500 -c:v mpeg4</span> MyProg.mp4</pre> I assume in previous releases it was defaulting to mpeg4, unless there's been some huge performance bug introduced into libx264.</p> Interestingly, when I built the latest version of avconv from their git repo, it didn't include libx264 at all, and happily defaulted to mpeg4.</p> Does test-driven development harm clarity? Andy Balaam — Fri, 09 May 2014 13:57:58 +0000 In a recent keynote at RailsConf called Writing Software</a>*, David Heinemeier Hansson</a> argues that test-driven development (TDD) can harm the clarity of our code, and that clarity is the key thing we should be aiming for when writing software.</p> (*contains swearing)</p> It's an excellent talk, and I would highly recommend watching it, especially if you are convinced (like me) that TDD is a good thing.</p> I was inspired by watching the video. Clarity certainly is extremely important, and the name he coins, Software Writer, sits better with me than Software Engineer or Software Developer. I have often felt Programmer was the best name for what I am, but maybe I will adopt Software Writer.</p> The real goal</h3> I would argue that clarity is not our goal in writing software. I think our goal is:</p> Working, modifiable software</p> </blockquote> Clarity helps us feel confident that our software works because we can read the code and understand what it does.</p> Clarity helps us modify our software because we can understand what needs to be changed and are less likely to make mistakes when we change it.</p> A good set of full-system-level tests helps us feel confident that our software works because they prove it works in certain well-defined scenarios. A good set of component-level and unit tests gives us confidence that various parts work, but as David points out, confidence in these separate parts does not give us much meaningful confidence that the whole system works.</p> Good sets of tests at all levels help us modify our software because we are free to refactor (or re-draft as David puts it). Unit and component tests give us confidence that structural changes we are making don't modify the external behaviour of the part we are re-structuring. Once we have made enabling structural changes, the changes we make that actually modify the system's behaviour are smaller and easier to deal with. The tests that break or must be written when we modify the system's behaviour help us understand and explain the behaviour changes we are making.</p> So both clarity and tests at all levels can contribute to our goal of writing working, modifiable software.</p> But David wasn't talking about tests - he was talking about TDD - driving the design of software by writing tests.</p> How TDD encourages clarity</h3> Before I talk about how we should accept some of what David is saying, let's first remind ourselves of some counter-points. TDD is explicitly intended to improve our code.</p> I agree with David when he defines good code as clear code, so how does TDD encourage clarity?</p> TDD encourages us to break code into smaller parts, with names.</strong> Smaller, named classes and functions are clearer to me than large blocks of code containing sub-blocks that do specific jobs but are not named. When I write in a TDD style I find it encourages me to break code into smaller parts.</p> TDD encourages us to write code that works at a single level of abstraction.</strong> Code that mixes different levels is less clear than code at a single level. I find that using TDD helps me resist the temptation to mix levels because it encourages me to compose two pieces that deal separately with each level, rather than linking them together.</p> It is vital to point out here that TDD doesn't push you towards clarity unless you already wanted to go there. I have seen TDD code that is unclear, stuffed full of boilerplate, formed by copy-paste repetition, and is generally awful. As a minimal counter-example, TDD doesn't stop you using terrible names for things.</p> But, when you care about clarity, and have an eye for it, I feel TDD can help you achieve it.</p> How TDD hurts clarity</h3> David's argument against TDD is that it makes us write code that is less clear. His main argument, as I understand it, is:</p> TDD forces us to use unnecessary layers of abstraction.</strong> Because we must never depend on "the world," TDD forces us to inject dependencies at every level. This makes our code more complex and less clear.</p> At its core, we must acknowledge that this argument is true. Where TDD causes us to inject dependencies that we otherwise would not inject, we are making our code more complex.</p> However, there are elements of a straw man here too. Whenever I can, I allow TDD to drive me towards systems with fewer dependencies, not injected dependencies. When I see a system with fewer dependencies, I almost always find it clearer.</p> Test against the real database?</h3> David frequently repeats his example of testing without hitting the database. He points out that allowing this increases complexity, and that the resulting tests do not have anything like as much value as tests that do use the database.</p> This hurts, because I think his point is highly valid. I have seen lots of bugs, throughout systems (not just in code close to the database) that came from wrong assumptions about how the database would behave. Testing a wide range of functionality against the real database seems to be the only answer to this problem. Even testing against a real system that is faster (e.g. an in-memory database) will not help your discover all of these bugs because the faster database will have different behaviour from the production one.</p> On the other hand, tests that run against a real database will be too slow to run frequently during development, slowing everything down and reducing the positive effects of TDD.</p> I don't know what the answer is, but part of it has got to be to write tests at component level, testing all the behaviour that is driven by database behaviour, but not requiring huge amounts of other systems to be spun up (e.g. the web server, LDAP, other HTTP endpoints) and run these against the real database as often as possible. If they only take about 5 minutes maybe it's reasonable to ask developers to run them before they commit code.</p> But my gut tells me that running tests at this level should not absolve us from abstracting our code from the production database. It just feels Right to write code that works with different storage back ends. We are very likely to have to change the specific database we use several times in the history of our code, and we may well need to change the paradigm e.g. NoSQL->SQL.</p> In a component that is based on the database, I think you should unit test the logic in the standard TDD way, unit test the database code (e.g. code that generates SQL statements) against a fast fake database, AND comprehensively test the behaviour of the component as a whole against a real database. I admit this looks like a lot of tests, but if you avoid "dumb" unit tests that e.g. check getters and setters, I think these 3 levels have 3 different purposes, and all have value*.</p> *Writing the logic TDD encourages smaller units of logic, and gives confidence that it is correct. Testing the database code against a fake database gives confidence that our syntax is right, and testing the whole component against the real database gives us confidence that our behaviour really works.</p> Injecting the database as a dependency gives us the advantage of our code having two consumers, which is one of the strongest arguments put forward by proponents of TDD that it gives us better code. All programmers know that there are only three numbers: 0, 1 and more. By having "more" users of our database code, we (hopefully) end up with code that speaks at a single level e.g. there are no SQL statements peppered around code which has no business talking direct to the database.</p> Clarity of tests</h3> In order for tests to drive good APIs in our production code, and for them to serve as documentation, they must be clear. I see a lot of test code that is full of repetition and long methods, and for me this makes it much less useful.</p> If our tests are complex and poorly factored, they won't drive good practice in our production code. If we view unclear tests as a smell, the fixes we make will often encourage us to improve the clarity of our production code.</p> If our tests resemble (or actually are) automatically-generated dumb callers of each method, we will have high coverage and almost no value from them. If we try to change code that is tested in this way, we will be thwarted at every turn.</p> If, on the other hand, we write tests that are clear and simple expressions of behaviour we expect, we will find them easy to understand and maintain, they will drive clear code in production, and sometimes we may realise we are writing them at a higher level than unit tests. When this happens, we should "float freely" with David and embrace that. They are testing more. That is good.</p> Higher-level tests</h3> David (with the support of Jim Coplien</a>) encourages us to test at a coarser grain than the unit. I strongly agree that we need more emphasis on testing at the component and system levels, and sometimes less unit testing, since we don't want write two tests that test the same thing.</p> However, there are some problems with larger tests.</p> First, larger tests make it difficult to identify what caused a problem. When a good unit test fails, the problem we have introduced (or the legitimate change in behaviour) is obvious. When a component or system test fails, often all we know is that something is wrong, and the dreaded debugging process must begin. In my experience, this is not just a myth. One of the pieces of code that I am most proud of in my career was a testing framework allowed you to write concise and clear tests of a horrible tangled mess of a component. The tests ran fairly quickly, with no external dependencies being used, but if one of them failed your heart sank.</p> Second, large tests can be fragile. Sometimes they fail because the database or network is down. Sometimes they fail because your threads did stuff in an unexpected order. Threading is a really good example: when I want to debug a weird threading issue I want to write an unthreaded test that throws stuff at a component in a well-defined but unusual order. When I can get that test to fail I know I've found a real problem, and I can fix it. If the code is not already unit tested (with dependencies being injected, sometimes) then writing that test can be really difficult.</p> TDD makes our code testable, and while testability (like clarity) is not an end in itself, it can be darn useful.</p> Conclusion</h3> Good things</h4> Clarity is good because it supports working, modifiable code.</p> Tests are good because they support working, modifiable code.</p> Testability is good because it supports tests, especially debugging tests.</p> TDD is good when it supports clarity, testability and tests.</p> TDD is bad when it hurts clarity.</p> If you throw out TDD, try not to throw out tests or testability. You will regret it.</p> What to do</h4> Write tests at the right level. Don't worry if the clearest level is not the unit level.</p> Use tests to improve clarity.</p> If your tests are unclear, there is something wrong. Fix it.</p> Steps to success</h4> Get addicted to TDD.</p> </li> Wean yourself off TDD and start to look at the minimal set of tests you can write to feel that sweet, sweet drug of confidence.</p> </li> Do not skip step 1.</p> </li> </ol> Comments</h2> Phil Nash 2014-05-13</span> </h3> As I was reading this this morning I just got to the part about testing against "real" databases when I finished my breakfast and got back to work. The task in progress was investigating a bug that only happens when we connect to the real database. Ha! The trouble is all our dev databases are currently being refreshed (with data from production) so I can’t work on the issue for an hour or so at this point – and when I can there’s no guarantee that the data that caused the issue will be the same! D’oh! In our case I believe it’s the difference in the data that caused the issue, rather than the database itself. This is why we run higher level integration and system regression test suites against the dev DBs at least nightly (and where this showed up). But once t does show up I’ll try and capture it in a deterministic unit test (or low-level integration test – by capturing a subset of the DB data). Clearly both are needed – for all sorts of reasons. But it’s primarily the unit tests that drive the low-level design. System tests drive the high-level design (as they show how our code behaves when it hits "the real world". There’s a dance between these levels as we try to capture things we learn at the system test level back at the unit test level (where appropriate). In short TDD is driven by tests. Which level those tests are pitched at varies as appropriate to the project, time etc – but you typically need multiple levels that all influence the design in some respect. (I’d also say that a lot of what I label "unit tests" in our code are really low-level integration tests – but they largely meet Michael Feathers’ requirements for unit tests, so I continue to call them that).</p> </div> Andy Balaam 2014-05-13</span> </h3> Definitely some of my best tests have been "component" tests – things that check real behaviour of relatively large areas of code, but still satisfy Feathers’ requirements for unit tests.</p> Also, I think driving high-level design from high-level tests and low-level design from low-level tests is the right way to think about things.</p> </div> Token-based security in a REST API Andy Balaam — Tue, 29 Apr 2014 07:20:04 +0000 Series: What is REST?</a>, A Basic REST API</a>, Paging</a>, Search</a>, Security</a>, Token, OpenID</a>.</p> We are implementing a REST API. Today, adding to the security we already have by making a security token that is stored in a cookie:</p> </iframe> Slides: Token-based security in a REST API</a>.</p> Code: poemtube on codeberg</a>.</p> Comments</h2> Dieu Merci 2017-12-07</span> </h3> Good evening Andy, I have some problem to understand Rest security. I don’t know if you can help me. Firstly I’m confuse for making the different between API and the Backend, and secondly I don’t know what points to concern the Rest API security. By following your video you focused on authentication and autorization, my question for this is what about injections, CSRF, XSS, …</p> Thanks.</p> </div> Andy Balaam 2017-12-08</span> </h3> Hi Dieu Merci, can you ask a more specific question? I don’t know how to answer this.</p> </div> Dieu Merci 2017-12-08</span> </h3> how to protect an Rest API from Sql injections, CSRF attack?</p> </div> Andy Balaam 2017-12-08</span> </h3> I suggest looking at https://www.owasp.org/index.php/Category:OWAS...</a> – the security aspects of a REST API are similar to other projects, except that there is usually no JavaScript running in the process. However, often the results of an API request are processed inside JavaScript, so many of the same ideas are relevant.</p> </div> Security in a REST API – Andy Balaam's Blog 2014-04-29</span> </h3> […] Series: What is REST?, A Basic REST API, Paging, Search, Security, Token. […]</p> </div> Avoid mocks by refactoring to functional Andy Balaam — Fri, 11 Apr 2014 16:23:55 +0000 At this week's ACCU</a> Conference I went to an excellent talk by Sven Rosvall</a> entitled "Unit Testing Beyond Mock Objects".</p> The talk covered the newer Java and C# unit testing frameworks that allow inserting mock objects even where legacy code is using some undesired dependency directly, meaning there is no seam</a> where you can insert a different implementation.</p> These tools solve a real problem, and could be useful.</p> However, I want to move the discussion in a different direction: can we avoid mocking altogether, and end up with better code?</p> Sven gave us an example of some legacy code a little bit like this. (I translated into C++ just for fun.)</p> // Return true if it's between 12:00 and 14:00 now. bool is_it_lunch_time_now() { system_clock::time_point now = system_clock::now(); time_t tt = system_clock::to_time_t( now ); tm local_tm = ( *localtime( &tt ) ); int hour = local_tm.tm_hour; return ( 12 <= hour && hour < 14 ); } </pre> To test this code, we would have something like this:</p> TEST_CASE( "Now: It's lunchtime now (hopefully)" ) { // REQUIRE( is_it_lunch_time_now() ); // NOTE: only run at lunch time! REQUIRE( !is_it_lunch_time_now() ); // NOTE: never run at lunch time! } </pre> Which is agony. But the production code is fine:</p> if ( is_it_lunch_time_now() ) { eat( sandwich ); } </pre> So, the normal way to allow mocking out a dependency like this would be to add an interface, making our lunch-related code take in a TimeProvider. To avoid coupling the choice of which TimeProvider is used to the calling code, we pass it into the constructor of a LunchCalculator that we plan to make early on:</p> class TimeProvider { public: virtual tm now_local() const = 0; }; class RealTimeProvider : public TimeProvider { public: RealTimeProvider() { } virtual tm now_local() const { system_clock::time_point now = system_clock::now(); time_t tt = system_clock::to_time_t( now ); return ( *localtime( &tt ) ); } }; class HardCodedHourTimeProvider : public TimeProvider { public: HardCodedHourTimeProvider( int hour ) { tm_.tm_hour = hour; } virtual tm now_local() const { return tm_; } private: tm tm_; }; class LunchCalc { public: LunchCalc( const TimeProvider& prov ) : prov_( prov ) { } bool is_it_lunch_time() { int hour = prov_.now_local().tm_hour; return ( 12 <= hour && hour < 14 ); } private: const TimeProvider& prov_; }; </pre> and now we can write tests like this:</p> TEST_CASE( "TimeProvider: Calculate lunch time when it is" ) { HardCodedHourTimeProvider tp( 13 ); // 1pm (lunch time!) REQUIRE( LunchCalc( tp ).is_it_lunch_time() ); } TEST_CASE( "TimeProvider: Calculate lunch time when it isn't" ) { HardCodedHourTimeProvider tp( 10 ); // 10am (not lunch :-( ) REQUIRE( ! LunchCalc( tp ).is_it_lunch_time() ); } </pre> Innovatively, these tests will pass at all times of day.</p> However, look at the price we've had to pay: 4 new classes, inheritance, and class names ending with unKevlin</a>ic words like "Provider" and "Calculator". (Even worse, in a vain attempt to hide my embarrassment I abbreviated to "Calc".)</p> We've paid a bit of a price in our production code too:</p> // Near the start of the program: config.time_provider = RealTimeProvider(); // Later, with config passed via PfA</a> if ( LunchCalc( config.time_provider ).is_it_lunch_time() ) { eat( sandwich ); } </pre> The code above where we create the RealTimeProvider is probably not usefully testable, and the class RealTimeProvider is also probably not usefully testable (or safely testable in the case of some more dangerous dependency). The rest of this code is testable, but there is a lot of it, isn't there?</p> The advantage of this approach is that we have been driven to a better structure. We can now switch providers on startup by providing config, and even code way above this stuff can be fully exercised safe in the knowledge that no real clocks were poked at any time.</p> But. Sometimes don't you ever think this might be better?</p> tm tm_local_now() // Not testable { system_clock::time_point now = system_clock::now(); time_t tt = system_clock::to_time_t( now ); return ( *localtime( &tt ) ); } bool is_lunch_time( const tm& time ) // Pure functional - eminently testable { int hour = time.tm_hour; return ( 12 <= hour && hour < 14 ); } </pre> The tests look like this:</p> TEST_CASE( "Functional: Calculate lunch time when it is" ) { tm my_tm; my_tm.tm_hour = 13; // 1pm (lunch time!) REQUIRE( is_lunch_time( my_tm ) ); } TEST_CASE( "Functional: Calculate lunch time when it isn't" ) { tm my_tm; my_tm.tm_hour = 10; // 10am (not lunch time :-( ) REQUIRE( ! is_lunch_time( my_tm ) ); } </pre> and the production code looks very similar to what we had before:</p> if ( is_lunch_time( tm_local_now() ) ) { eat( sandwich ); } </pre> Where possible, let's not mock stuff we can just test as pure, functional code.</p> Of course, the production code shown here is now untestable, so there may well be work needed to avoid calling it in a test. That work may involve a mock. Or we may find a nicer way to avoid it.</p> Comments</h2> Peter Bindels 2014-04-12</span> </h3> … What about just mocking what you use? Stop trying to make C++ into Java with EasyMock or C# with RhinoMock, use something tailored to C++ (and I mean all of C++).</p> #include "hippomocks.h"</p> TEST_CASE( "Calculate lunch time when it is" ) { MockRepository mocks; mocks.ExpectCall(system_clock::now).Return(system_clock::lunchtime); REQUIRE( is_lunch_time() ); }</p> TEST_CASE( "Calculate lunch time when it isn’t" ) { MockRepository mocks; mocks.ExpectCall(system_clock::now).Return(system_clock::dinnertime); REQUIRE( ! is_lunch_time() ); }</p> Stop using C++ as if it’s a badly implemented Java or a badly implemented Haskell. Use it for what it’s worth.</p> </div> GermÃ¡n 2014-04-13</span> </h3> I have been thinking on how to mock avoiding to,write an interface.</p> You could do this, though it is not superior to the last solution:</p> extern std::function is_it_lunch_time_now;</p> Default implementation goes in .cpp.</p> In test code:</p> is_lunch_time_now = /mock impl</em>/;</p> </div> GermÃ¡n 2014-04-13</span> </h3> function is bool() but the website swallowed things between less than and greater than symbols due to markup I guess.</p> </div> German Diago 2014-04-13</span> </h3> You can avoid writing an interface doing this, but needs source code modification:</p> extern std::function is_it_lunch_time_now;</p> .cpp file:</p> is_it_lunch_time_now = []() -> bool { … };</p> test file, rebind function:</p> is_it_lunch_time_now = /mock implementation</em>/</p> </div> Andy Balaam 2014-04-14</span> </h3> Peter: it looks like hoppomocks gives you the ability to insert a seam where there isn’t one, like PowerMock which was the subject of the talk that inspired this post. This sounds like a helpful thing to be able to do when you’re in a situation where there is a lot of code already using untestable code, but I am trying to think about how we would strucure code ideally. In my opinion, in the ideal case we wouldn’t rely on "magic" like that. Of course, that argument relies on us sharing understanding of what is "magic" and what is perfectly good and normal code.</p> </div> Andy Balaam 2014-04-14</span> </h3> GermÃ¡n: thank you, I fixed the less-thans.</p> </div> Andy Balaam 2014-04-14</span> </h3> GermÃ¡n: yes, that is another way to avoid mocks. But isn’t the functional style for is_lunch_time() nicer anyway, even if we also implement is_it_lunch_time_now() based on it?</p> </div> Peter Bindels 2014-04-14</span> </h3> > Of course, that argument relies on us sharing understanding of what is â€œmagicâ€ and what is perfectly good and normal code.</p> To me it is the discussion of what exactly an interface is. Is it only an interface if you have a C++ class with pure virtual methods that you can replace with a mock object? Or is a set of C functions with a coherent goal also an interface?</p> I tend to the latter. Having code that does not require mocks to test is inherently better, but you will have to tie those bits of code together somehow</em>, and you’ll need to check that the whole of those bits of code does what you expect it to do. You can check a key and a lock, but until you check them together you don’t know you have the right</em> key.</p> </div> Andy Balaam 2014-04-14</span> </h3> Hi Peter, yes – it is legitimate to use natural seams (e.g. a mock implementation that matches the declared function signature of the real dependency) that are available in C++ that are not in a language like Java. However, I’d still prefer to write code whose behaviour depends only on its input where possible, and doesn’t use an untestable depenency call in its implementation. I do admit you’ve suggested a good way to replace that dependency call, but I’d still rather not have it.</p> </div> Phil Nash 2014-04-17</span> </h3> First – good to see you using Catch for your test framework :–)</p> Second – I completely agree with your thought processes here. It’s one of the reasons that truly functional code is so much easier to reason about (once you get your head around the paradigm in the first place) – but it all has it’s start in straightforward stuff like this.</p> Thirdly – despite a different language and focus – it echoes a lot of what I wrote here: http://www.levelofindirection.com/journal/201...</a> (which I know you’ve seen because you were the first commenter ;–) )</p> </div> Andy Balaam 2014-04-17</span> </h3> Phil – yes, this was my first go with CATCH. I enjoyed it. We are in total agreement :–) If only everyone listened to us …? (That link should be http://www.levelofindirection.com/journal/201...</a> )</p> </div> Using the final keyword in interface method parameters does nothing Andy Balaam — Wed, 02 Apr 2014 09:13:15 +0000 Consider the following Java code:</p> class FinalInInterface { private static interface WithFinal { public void run( final int x ); } private static class WithoutFinal implements WithFinal { public void run( int x ) { x = 4; System.out.println( x ); } } public static void main( String[] args ) { new WithoutFinal().run( 3 ); } } </pre> This code compiles, and when it runs, it prints "4".</p> So adding "final" to the x parameter of the run() method in the interface WithFinal has no effect - the implementor of this interface, WithoutFinal is allowed to declare its own run() method without "final", and modify x as much as they want.</p> This makes more sense if you realise that in Java every method argument is passed by value</a>. When you pass an object reference, the method and the calling code are talking about the same object, but the reference is passed by value, so if you reassign it inside the method, the original object is unaffected, and the reference to it in the calling code is also unaffected. The semantics are basically identical to passing a pointer in C or C++ - the pointer is copied, but the pointed-to object is the same.</p> If you declare method parameters final in your interface, people implementing that interface don't need to declare them final, and can modify them in their implementations. It's possible that by declaring them final you are trying to communicate to implementors of the interface that they should also declare them final, but as the designer of the interface, it's really none of your business how the implementor implements it.</p> There is no "const" in Java, so you have no way of preventing the implementor of your interface from modifying an object that is passed in (by copying a reference to it).</p> If that makes you sad, join the club.</p> Using xscreensaver to lock the screen in Lubuntu Andy Balaam — Wed, 26 Mar 2014 09:17:30 +0000 Recent versions of Lubuntu seem to have broken screen locking</a>.</p> To get screen locking via xscreensaver</a> do this:</p> # Install xscreensaver sudo apt-get install xscreensaver # Start xscreensaver when you log in echo "xscreensaver -nosplash" >> ~/.config/lxsession/Lubuntu/autostart # Edit the "screen lock" launcher sudo leafpad /usr/share/applications/lubuntu-screenlock.desktop </pre> Change lubuntu-screenlock.desktop to look like this:</p> [Desktop Entry] Name=ScreenLock (xscreensaver)</span> Name[zh_TW]=éŽ–ä½èž¢å¹• Comment=Lock your screen Icon=system-lock-screen Exec=xscreensaver-command -lock</span> NoDisplay=true </pre> If you want a keyboard shortcut to lock the screen, add this entry (under another <keybind> tag) to ~/.config/openbox/lubuntu-rc.xml:</p> <keybind key="C-A-l"> <action name="Execute"> <command>xscreensaver-command -lock</command> </action> </keybind> </pre> Comments</h2> Michael Day 2016-06-15</span> </h3> Awesome. This helped me a LOT today on Lubuntu 16.04.</p> Thanks a ton.</p> </div> Andy Balaam 2016-06-16</span> </h3> Hi Michael, I’m glad it helped!</p> </div> Security in a REST API Andy Balaam — Tue, 04 Mar 2014 18:09:47 +0000 Series: What is REST?</a>, A Basic REST API</a>, Paging</a>, Search</a>, Security, Token</a>, OpenID</a>.</p> We are implementing a REST API. Today, providing security using HTTP Basic Authorization:</p> </iframe> Slides: Security in a REST API</a>.</p> Code: poemtube on codeberg</a>.</p> Comments</h2> Token-based security in a REST API – Andy Balaam's Blog 2014-04-29</span> </h3> […] What is REST?, A Basic REST API, Paging, Search, Security, […]</p> </div> Searching in a REST API Andy Balaam — Fri, 31 Jan 2014 08:24:30 +0000 Series: What is REST?</a>, A Basic REST API</a>, Paging</a>, Search, Security</a>, Token</a>, OpenID</a>.</p> We are implementing a REST API. Today, we provide a basic search.</p> </iframe> Slides: Searching in a REST API</a>.</p> Code: poemtube on codeberg</a>.</p> Comments</h2> Paging in a REST API – Andy Balaam's Blog 2014-01-31</span> </h3> […] in the morning, still writing Free Software « A Basic REST API Searching in a REST API […]</p> </div> What is REST? – Andy Balaam's Blog 2014-03-04</span> </h3> […] What is REST?, A Basic REST API, Paging, Search, […]</p> </div> A Basic REST API – Andy Balaam's Blog 2014-04-29</span> </h3> […] What is REST?, A Basic REST API, Paging, Search, Security, […]</p> </div> Paging in a REST API Andy Balaam — Thu, 05 Dec 2013 08:26:48 +0000 Series: What is REST?</a>, A Basic REST API</a>, Paging, Search</a>, Security</a>, Token</a>, OpenID</a>.</p> We consider some choices of how to provide paging through a list resource in a REST API.</p> </iframe> Slides: Paging in a REST API</a>.</p> Code: poemtube on codeberg</a>.</p> A Basic REST API Andy Balaam — Wed, 06 Nov 2013 21:08:35 +0000 Series: What is REST?</a>, A Basic REST API, Paging</a>, Search</a>, Security</a>, Token</a>, OpenID</a>.</p> I'll walk you through the code of a simple REST API I am writing to be a kind of YouTube for poetry.</p> </iframe> Slides: A Basic REST API</a>.</p> Code: poemtube on codeberg</a>.</p> Avoid backslashes anywhere in Java code (Java error "illegal unicode escape") Andy Balaam — Thu, 10 Oct 2013 10:16:43 +0000 Did you know you can insert unicode-escaped characters, anywhere in a Java program?</p> Most of us are familiar with using unicode escapes like this:</p> String pound = "\u00A3";</pre> but in fact constructs like \u00A3 can go anywhere, including in a comment.</p> This is all fine so long as they're valid, but what if you're generating Java code without due care and attention?</p> And what if you're inserting file paths into your generated code?</p> And what if one of your directories has a name starting with a "u"?</p> Then you get code like this (edited example from our real project!):</p> // DO NOT EDIT! This file was generated from: // C:\usr\foo.xml </pre> And, only on the machine with the dir called "usr", we got a compile error like this:</p> MyClass.java:14: illegal unicode escape // C:\usr\foo.xml ^ 1 error </pre> Which took me a while to track down.</p> Reference: JLS-3.3 Unicode Escapes</a>.</p> Note this only applies to unicode escapes, not others like \n or \t - they are only processed within character or string literals (JLS-3.10.6</a>).</p> What is REST? Andy Balaam — Tue, 01 Oct 2013 07:08:53 +0000 Series: What is REST?, A Basic REST API</a>, Paging</a>, Search</a>, Security</a>, Token</a>, OpenID</a>.</p> The basic concepts of REST APIs:</p> </iframe> Slides: What is REST?</a>.</p> Code: poemtube on codeberg</a>.</p> Bash arrays Andy Balaam — Wed, 18 Sep 2013 13:45:21 +0000 Bash arrays are a lot like Bash Associative Arrays</a>, but with numbers as keys.</p> Here's a quick reference.</p> Basics</h3> $ declare -a MYARR # Create an array $ MYARR[3]=foo # Put a value into an array $ echo ${MYARR[3]} # Get a value out of an array foo $ echo MYARR[3]</span> # WRONG MYARR[0] $ echo $MYARR[3]]</span> # WRONG [3] </pre> Creating, adding</h3> $ declare -a MYARR # Explicitly declare $ MYARR[3]=foo # Or this line implicitly makes it an array $ MYARR[4]=bar # Can add values one by one </pre>$ declare -a MYARR=(a b c) # Initialise all at once $ echo ${MYARR[0]} a $ echo ${MYARR[1]} b $ echo ${MYARR[2]} c </pre>$ declare -a MYARR # Or declare separately $ MYARR=(a b c) # Then initialise $ echo ${MYARR[0]} a $ echo ${MYARR[1]} b $ echo ${MYARR[2]} c </pre>$ declare -a MYARR=(a b c) $ MYARR=("${MYARR[@]}" d) # Add an element $ echo ${MYARR[@]} a b c d $ declare -a MYARR2=(e f g) $ MYARR=("${MYARR[@]}" "${MYARR2[@]}") # Concatenate arrays $ echo ${MYARR[@]} a b c d e f g </pre></p> Keys/Indices</h3> $ declare -a MYARR $ MYARR[3]=foo $ echo ${MYARR[0]} # Unassigned values are empty $ echo ${MYARR[4]} # Unassigned values are empty $ MYARR[seven]=bar # A text index is treated as 0 $ echo ${MYARR[0]} bar $ echo ${MYARR[seven]} # A text index is treated as 0 bar </pre>$ K=3 $ MYARR[$K]=baz # Variables containing numbers work like numbers $ echo ${MYARR[$K]} baz $ echo ${MYARR[3]} # Obviously the value is accessible via the actual index baz $ K=foo $ MYARR[$K]=bash # Variables containing text are treated as 0 $ echo ${MYARR[0]} bash </pre></p> Length</h3> $ declare -a MYARR=(a b c) $ echo ${#MYARR[@]} # Length of an array 3 $ echo $#MYARR[@]</span> # WRONG 0MYARR[@] $ echo ${#MYARR}</span> # WRONG 1 </pre>$ MYARR[7]=x $ echo ${#MYARR[@]} # Only existing indices count in the length 4 </pre> $ declare -a MYARR=(a bb ccc) $ echo ${#MYARR[0]} # Length of an individual element 1 $ echo ${#MYARR[1]} 2 $ echo ${#MYARR[2]} 3 </pre></p> Looping</h3> $ declare -a MYARR=("a 1" b c) $ # Loop through array values $ for V in "${MYARR[@]}"; do echo $V; done a 1 b c $ for V in ${MYARR[@]}; do echo $V; done</span> #WRONG a 1 b c </pre>$ echo "${!MYARR[@]}" # Print all indices - quoted, but quotes removed by echo 0 1 2 $ echo "${MYARR[@]}" # Print all values - quoted, but quotes removed by echo a 1 b c </pre></p> Clearing</h3> $ declare -a MYARR $ MYARR[3]=x $ echo ${MYARR[3]} x $ unset MYARR $ declare -a MYARR $ echo ${MYARR[3]} </pre> Deleting</h3> $ MYARR[2]=foo $ echo ${MYARR[2]} foo $ unset ${MYARR[2]}</span> # WRONG $ echo ${MYARR[2]} foo $ unset MYARR[2] # To delete from an array, use "unset" with similar syntax to assigning $ echo ${MYARR[2]} $ MYARR[3]=quux $ echo ${MYARR[3]} quux $ K=3 $ unset MYARR[$K] # Can unset using a variable for the key too $ echo ${MYARR[3]} </pre> $ declare -a MYARR=(a b c d e f) $ MYARR=("${MYARR[@]:0:3}" "${MYARR[@]:4}") # Remove element 3, leaving no gap $ echo ${MYARR[@]} </pre> Cool stuff</h3> $ declare -a MYARR=(a b c d e f g) $ echo ${MYARR[@]:2:3} # Extract a sub-array c d e </pre> $ declare -a MYARR=(a b c d e f g) $ echo ${MYARR[@]/d/FOO} # Replace elements that match a b c FOO e f g </pre> Scope</h3> $ unset MYARR $ function createmap() { MYARR[5]=bar; } # Implicit creation puts it in the global scope $ echo ${MYARR[5]} $ createmap $ echo ${MYARR[5]} bar </pre>$ unset MYARR $ function createmaplocal() { declare -a MYARR; MYARR[3]=bar; } # Explicit creation puts it in the local scope $ echo ${MYARR[3]}</p> $ createmaplocal $ echo ${MYARR[3]}</p> </pre> Links</h3> Bash manual: Arrays</a></li> Bash Array Tutorial</a></li> Bash Hackers Wiki</a></li> Linux Journal</a></li> Superuser: Test if element is in array in Bash</a></li> </ul> Comments</h2> Marcos FRM 2016-04-20</span> </h3> On Keys/Indices’ second box there is a typo:</p> $ K=3 $ MYARR[$K]=baz # Variables containing numbers work like numbers $ echo ${MYARR[$K]} bar <——————– THIS</p> It will print 'baz' there.</p> </div> Andy Balaam 2016-04-26</span> </h3> Marcos, thank you – fixed.</p> </div> Bob 2013-09-20</span> </h3> Note well that these should never be used in portable scripts. /bin/sh may not be bash and bash may not be installed at all.</p> </div> Andy Balaam 2013-09-20</span> </h3> Hi Bob, yes this is specific to bash.</p> </div> Using if:set, unless:set etc. with macrodefs in Ant Andy Balaam — Fri, 13 Sep 2013 15:53:41 +0000 In a previous post I outlined how to use if:set and unless:set</a> to execute tasks conditionally in Ant 1.9.1.</p> Unfortunately, this does not work with macrodefs. When I try to execute a macrodef conditionally like this:</p> <project xmlns:if="ant:if" xmlns:unless="ant:unless" default="build" > <macrodef name="mymacro"> <sequential> <echo message="inside mymacro"/> </sequential> </macrodef> <target name="build"> <mymacro if:set="doit"/> </target> </project> </pre> When I set the "doit" property and run like this, it fails:</p> $ ant -Ddoit=true Buildfile: build.xml build: BUILD FAILED build.xml:14: Unknown attribute [ant:if:if:set] Total time: 0 seconds </pre> It looks to me like this is a bug: the if:set attribute is getting passed into the macro, which is complaining that it doesn't expect an attribute with that name. (If you try to create an attribute with that name, you'll find that "if:set" is an illegal name...)</p> However, there is a workaround. You can wrap the call to your macrodef in a <sequential> tag:</p> <project xmlns:if="ant:if" xmlns:unless="ant:unless" default="build" > <macrodef name="mymacro"> <sequential> <echo message="inside mymacro"/> </sequential> </macrodef> <target name="build"> <sequential if:set="doit"> <mymacro/> </sequential> </target> </project> </pre> And now it works!</p> $ ant -Ddoit=true Buildfile: build.xml build: [echo] inside mymacro BUILD SUCCESSFUL Total time: 0 seconds </pre>$ ant Buildfile: build.xml</p> build:</p> BUILD SUCCESSFUL Total time: 0 seconds </pre></p> Conditional execution even when calling a macrodef. Enjoy.</p> Using if:set, unless:set etc. to execute a task conditionally in Ant Andy Balaam — Fri, 13 Sep 2013 15:41:31 +0000 I'm not sure anyone except me is still struggling on with using Ant</a>, but just in case, here is a nice thing.</p> In Ant 1.9.1 they added a useful feature: instead of needing to use the <if></a> tag and similar from ant-contrib</a>, you can conditionally execute "any" task.</p> (In fact, this appears to mean "any task except a macrodef" - see my post Using if:set, unless:set etc. with macrodefs in Ant</a>.)</p> You need to add these namespaces at the top of your project:</p> <project xmlns:if="ant:if" xmlns:unless="ant:unless" > </pre> and then make any task conditional by adding one of these attributes:if:set if:true if:blank unless:set unless:true unless:blank </pre></p> if:set and unless:set take a property name</strong> and run the task if it is set at all (or unless it is set at all).</p> <mytask if:set="property.name" other_attrs="" .../> </pre> Where mytask is the name of any ordinary Ant task like echo, jar etc.</p> (Note there is no ${} around the property name for if:set.)</p> if:true and unless:true take a value</strong> and run the task if it is true (or unless it is true), so they need to be used like this:</p> <mytask if:true="${property}" other_attrs="" .../> </pre> (Note the ${} here, but not above.)</p> Similarly, if:blank and unless:blank take a value</strong> and run the task if it is blank (or unless it is blank), e.g.:</p> <mytask if:blank="${property}" other_attrs="" .../> </pre> NOTE: The example in the documentation is wrong (at this time, 2013-09-13) - it uses the property name, but this does not work - you must surround it with ${} to get its value.</strong></p> The properties can be specified in your build file as normal, or supplied on the command line to ant with -Dproperty.name=value.</p> Here's an example of using set:</p> build.xml:<project xmlns:if="ant:if" xmlns:unless="ant:unless" default="build" ></p> <property name="setinxml" value="true"/> <target name="build"> <echo if:set="setinxml" message="if:set=setinxml"/> <echo unless:set="setinxml" message="unless:set=setinxml"/> <echo if:set="notset" message="if:set=notset"/> <echo unless:set="notset" message="unless:set=notset"/> <echo if:set="setincmd" message="if:set=setincmd"/> <echo unless:set="setincmd" message="unless:set=setincmd"/> </target> </code></pre> </project> </pre></p> And here's the output:</p> $ ant -version Apache Ant(TM) version 1.9.2 compiled on July 8 2013 $ ant -Dsetincmd=true Buildfile: build.xml build: [echo] if:set=setinxml [echo] unless:set=notset [echo] if:set=setincmd BUILD SUCCESSFUL Total time: 0 seconds </pre> The documentation for this, such as it is, is here: If And Unless</a>.</p> Comments</h2> fabricio 2016-05-12</span> </h3> saved my day</p> </div> Dev 2016-07-13</span> </h3> Thanks for the info, it proved to be very useful and simple to use!!</p> Dev</p> </div> Andy Balaam 2016-07-13</span> </h3> Thanks Dev, glad it was useful.</p> </div> Priyasha 2018-06-19</span> </h3> Can i do an if on entire project some how ? is that possible ?</p> </div> Andy Balaam 2018-06-20</span> </h3> Hi Priyasha, not that I know of :–(</p> </div> Priyasha Praik 2018-06-23</span> </h3> Hi andy, thanks for this. I have three targets in one project in my ant script. and i just want to add an if statement (effectively) which control all the three targets. so for example if i set that variable to true, then all three targets should execute. if i set it to false, nothing should execute.Can i do something like that?</p> str of my file: very simple:</p> The above is how it looks, i just want to conditionally either execute them all or not execute any. Is there any way out ???</p> I am so grateful for your response again, thanks very much</p> </div> Andy Balaam 2018-06-24</span> </h3> Hi Priyasha, inside each of your targets you could have a block, with if:set attributes for the variable you want to use to control them. Then they would execute, but actually do nothing. Hopefully that will work for what you want.</p> </div> Priyasha Pratik 2018-06-24</span> </h3> Hey, thanks a lot, could you give me a small example. I am very new to ant and having a big hard time with this. Thanks, I really appreciate your help</p> </div> Vim for Fun Andy Balaam — Wed, 11 Sep 2013 06:59:47 +0000 I have recently got quite excited about Vim (the text editor), so here is a video showing you the basics. I use it because I enjoy learning it.</p> </iframe> Slides: Vim for Fun</a>.</p> Working with others in git Andy Balaam — Wed, 24 Jul 2013 18:38:47 +0000 Series: Why git?</a>, Basics</a>, Branches</a>, Merging</a>, Remotes</p> Now I've covered almost everything you need to understand the basics of how git works. In this video I discuss how to clone, pull from and fork remote git repositories, and how to send patches and make pull requests to a project maintainer.</p> </iframe> Slides: Working with others in git</a>.</p> Merging in git Andy Balaam — Thu, 11 Jul 2013 07:29:18 +0000 Series: Why git?</a>, Basics</a>, Branches</a>, Merging, Remotes</a></p> The last video covered how to make branches and simple merges.</p> This time we look at what to do when you want just one change from another branch (cherry-picking) and how to merge two branches when there are conflicts between them. We also look at rebasing, which is a different way to combine the work done on two branches, which hides complexity from other people at the expense of changing history.</p> </iframe> Slides: Merging in git</a>.</p> Behaviour of Java String.split() when some answers are the empty string Andy Balaam — Thu, 23 May 2013 13:51:52 +0000 Can you guess the output of this program?</p> class SplitTest { static void split( String s ) { System.out.println( s.split( ";" ).length ); } public static void main( String[] args ) { split(""); split(";"); split("x;"); split(";y"); } } </pre> Here it is:</p> $ javac SplitTest.java && java SplitTest 1 0 1 2 </pre> Wow. Docs: String.split</a>.</p> Side note: It's not easy to fit Java examples into tweets.</p> class X{static void x(String s){System.out.println(s.split(";").length);}public static void main(String[]a){x("");x(";");x("x;");x(";y");}}</p> — Andy Balaam (@andybalaam) May 23, 2013</a></p> </blockquote> Checking the case of a filename on Windows Andy Balaam — Thu, 23 May 2013 12:30:22 +0000 Windows generally uses a case-insensitive but not case-preserving</a> file system.</p> When writing some code that is intended to be used on Linux as well as Windows, I wanted it to fail on Windows in the same cases that it would fail on Linux, and this meant detecting when the case of a filename differed from its canonical case on the file system.</p> I want to ask "is this file name correct in terms of case?"</p> I was working in Java, but I think this issue would be similar in other languages: it's difficult to ask for the canonical case version of a file name when we currently have a filename with abitrary case.</p> The only solution I came up with was to list the contents of the parent directory and check whether my arbitrary filename is listed with the correct case in the results:</p> // CaseCheck.java import java.util.Arrays; import java.io.File; import java.io.IOException; class CaseCheck { private static File parentFile( File f ) { File ret = f.getParentFile(); if ( ret == null ) { ret = new File( "." ); } return ret; } private static boolean existsAndCaseCorrect( String fileName ) { File f = new File( fileName ); return Arrays.asList( parentFile( f ).list() ).contains( f.getName() ); } public static void main( String[] args ) throws IOException { System.out.println( existsAndCaseCorrect( args[0] ) ); } } </pre> Checking it on its own source file: javac CaseCheck.java && java CaseCheck cASEcheck.java false</p> javac CaseCheck.java && java CaseCheck CaseCheck.java true </pre></p> It seems to work.</p> Note that this also returns false if the file doesn't exist, and will throw an error if the file name specifies a parent directory that doesn't exist.</p> Branches in git Andy Balaam — Mon, 13 May 2013 20:00:21 +0000 Series: Why git?</a>, Basics</a>, Branches, Merging</a>, Remotes</a></p> Last time we discussed starting a project and committing changes.</p> Now we look at how to create branches, which are one of the main reasons for having source control. We'll cover creating branches, switching between them, and the simplest parts of merging from one to another.</p> </iframe> Slides: Branches in git</a>.</p> Passing several values through a pipe in bash Andy Balaam — Fri, 10 May 2013 09:13:46 +0000 I have been fiddling with some git-related shell scripts, and decided to try and follow the same approach as git in their structure. This means using the Unix system where each piece of functionality is a separate script (or executable) that communicates by using command-line arguments, reading from the standard input stream, and writing output to the standard output stream.</p> This allows each piece of functionality to be written in any programming or scripting language. In git's case this has allowed initial versions to be written in bash or perl and later optimised versions (sometimes written in C) to be dropped in, piece by piece. It's an incredibily flexible way of working and can also be very efficient.</p> Most of my prototyping has been in bash, and I've found sometimes I need to write out multiple values from a script and collect them as input in another script.</p> Writing the output is simple:</p> #!/bin/bash # outputter.bash # Imagine A, B and C have been created by some complex process: A="foo bar" B=" bar" C="baz " # At the end of our script we simply write them out on separate lines in a known order echo "${A}" echo "${B}" echo "${C}" </pre> But reading them in somewhere else gave me some trouble until I learned this recipe:</p> #!/bin/bash # inputter.bash # Read in the values one per line: IFS=$'\n' read A IFS=$'\n' read B IFS=$'\n' read C # Now we can use them. echo "A='${A}'" echo "B='${B}'" echo "C='${C}'" </pre> And now the values transfer succesfully, preserving whitespace:</p> $ ./outputter.bash | ./inputter.bash A='foo bar' B=' bar' C='baz ' </pre> The recipe uses bash's built-in read command to populate the variables, but sets the IFS variable (Internal Field Separator) to a newline, meaning all the whitespace in the line is treated as part of the value to be read. The $'\n' syntax is a literal newline.</p> Comments</h2> Bob 2013-05-11</span> </h3> #!/bin/bash</p> is a bug.</p> Use #!/usr/bin/env bash intead</p> However, I believe none of this is bash specific so</p> #!/bin/sh is best</p> </div> Andy Balaam 2013-05-11</span> </h3> Hi Bob, citation needed?</p> </div> Goodness in programming languages, part 4 - Ownership & Memory Andy Balaam — Wed, 01 May 2013 09:00:31 +0000 Posts in this series: Syntax</a>, Deployment</a>, Metaprogramming</a>, Ownership</em></p> There is often a trade-off between programming language features and how fast (and predictably) the programs run. From web sites that serve millions of visitors to programs running on small devices we need to be able to make our programs run quickly.</p> One trade-off that is made in many modern programming languages (including Python, Ruby, C#, Java and JVM-based languages) is that the system owns all the memory. This avoids the need for the programmer to think about how long pieces of memory need to live, but it means a lot of memory can hang around a lot longer than it really needs to. In addition, it can mean the CPU has to jump around to lots of different memory locations to find pieces of dynamically-allocated memory in different locations. Where this jumping around causes caches to be invalidated that can really slow things down.</p> While these garbage collection-based languages have been evolving, C++ has been developing along a different track. C++ allows the programmer to allocate and free up memory manually (as in C), but over time the community of C++ programmers has been developing a new way of thinking about memory, and developing tools in the C++ language to make it easier to work in this way.</p> Modern C++ code rarely or never uses "delete" or "free" to deallocate memory, but instead defines clearly which object owns each other object. When the owning object is no longer needed, everything it owns can be deleted, immediately freeing their memory. The top-level objects are owned by the current scope, so when the function or block of code we are in ends, the system knows these objects and the ones they own can be deleted. Objects that last for the whole life of the program are owned by the scope of the main function or equivalent.</p> One advantage of explicit ownership is that the right thing happens automatically when something unexpected happens (e.g. an exception is thrown, or we return early from a function). Because the objects are owned by a scope, as soon as we exit that scope they are automatically deleted, and no memory is "leaked".</p> Because ownership is explicit, we can often group owned objects in memory immediately next to the objects that own them. This means we jump around to different memory locations less often, and we have to do less work to find and delete regions of memory. This makes our programs faster.</p> Here are some things I like:</p> Modern C++'s clarity about who owns what.</strong> By expressing ownership explicitly we make clear our intentions, and avoid memory leaks.</li> Modern C++'s fast and cache-friendly memory handling.</strong> Allocating memory for several objects together reduces time spent looking for space, and means caches are more likely to be used.</li> </ul> In my experience, the most frequent performance problems I have had to solve have really been memory problems. Explicit ownership can reduce unnecessary memory management overhead by taking back the work from the system (the garbage collector) and allowing programmers to be explicit about who owns what.</p> How to use git (the basics) Andy Balaam — Sat, 20 Apr 2013 08:05:54 +0000 Series: Why git?</a>, Basics, Branches</a>, Merging</a>, Remotes</a></p> Git is a very powerful tool, but somewhat intimidating at first. I will be making some videos working through how to use it step by step.</p> First, we look at how to track your own code on your own computer, and then get a brief look at a killer feature: stash, which lets you pause what you were doing and come back to it later.</p> </iframe> Slides: How to use git (the basics) slides</a>.</p> Using gnome-mplayer to play DVB radio without asking whether you want to resume Andy Balaam — Mon, 15 Apr 2013 19:12:31 +0000 When I launch gnome-mplayer to play back radio over my TV card (DVB), it asks me whether I want to resume from where I left off, which doesn't make sense for this kind of stream.</p> I couldn't find a way to switch this off, but a little hacking with gnome-mplayer's sqlite database does the trick.</p> Here's my Radio 4 launch script:</p> #!/bin/bash URI="dvb://BBC Radio 4" sqlite3 ~/.config/gnome-mplayer/gnome-mplayer.db "DELETE FROM media_entries WHERE uri='${URI}'" gnome-mplayer "${URI}" </pre> C++14 "Terse" Templates - an argument against the proposed syntax Andy Balaam — Sat, 13 Apr 2013 01:22:13 +0000 Today I attended two excellent talks by Bjarne Stroustrup</a> at the ACCU Conference 2013</a>. The first was an inspiring explanation of the recent C++11 standard, and the second, "C++14 Early thoughts"</a> was an exciting description of some of the features that might go into the next standard.</p> One of those features, which Bjarne called "Terse" Templates, might be a good idea, but the syntax Bjarne proposed seems like a bad idea to me, because it leaks unwanted names into the namespace containing the function you are writing.</p> Allow me to explain.</p> Background - Concepts Lite</h3> I attended another excellent talk before Bjarne's, called "Concepts Lite-Constraining Templates with Predicates"</a> by Andrew Sutton</a>, introducing "Concepts Lite", which is an attempt to salvage a manageable language feature from the very large "Concepts" feature that failed to make it into C++11.</p> My (so far very basic) understanding of Concepts Lite is that it is a way of defining conditions that state whether a template will be expanded for a given type.</p> So, in C++11 (and C++98), we can declare a (stupid) template function like so:</p> template<typename ListOfInt> int first( ListOfInt& list ) { return list.size() > 0 ? list[0] : 0; } </pre> The code in this function template assumes that list has a size method, and an operator[] method. We tried to "suggest" this, by naming our template parameter ListOfInt, but the poor programmer may not realise exactly what we meant.</p> If we do the wrong thing, and try to use the first function with an int argument:</p> int i = 3; first( i ); </pre> It goes wrong, because ints don't have a size method:</p> In function 'int first(ListOfInt&) [with ListOfInt = int]': error: subscripted value is neither array nor pointer error: request for member 'size' in 'list', which is of non-class type 'int' </pre> This error is not too obscure, but in complex cases the errors can be extremely long, and point to problems that appear to be unrelated to the code we are writing.</p> Really what we want to know is that int is not a ListOfInt.</p> Concepts Lite give us the ability to define what a ListOfInt means, and only expand the template for types that match that definition.</p> In our example we would do something like this:</p> template<typename ListOfInt> requires SizeAndIndex<ListOfInt>()</span> int first( ListOfInt& list ) { return list.size() > 0 ? list[0] : 0; } </pre> (There is actually a neater syntax, but we'll do it like this for now because we need the more verbose form later.)</p> What this means is that this template function will only be expanded for types that satisfy the constraint.</p> The definition of SizeAndIndex is outside the scope of this article - it allows us to check whether types satisfy some conditions. In this case we assume it checks that the type contains the methods we use.</p> Now when we do the wrong thing:</p> int i = 3; first( i ); </pre> We get a simple error message, that properly tells us what's wrong:</p> error: no matching call to â€˜first(int list)â€™ note: candidate is â€˜first(ListOfInt& list)â€™ note: where ListOfInt = int note: template constraints not satisfied note: â€˜ListOfIntâ€™ is not a/an â€˜SizeAndIndexâ€™ type since note: â€˜list.size()â€™ is not valid syntax </pre> (The above is fiction, but Andrew assures us he gets real errors like this with his prototype.)</p> So Concepts Lite gives us the optional ability to check that our template parameters are what we expected them to be, giving a decent error message, instead of waiting for something to fail much later when we compile the instantiated template.</p> So far so utterly cool. (And, in my ill-informed opinion, the only bit of Concepts I really wanted anyway.)</p> There's more information on this feature here: Concepts Lite: Constraining Templates with Predicates</a> and here: Concepts-Lite</a>.</p> Constraints on multiple types</h3> The Concepts Lite feature as proposed allows us to specify constraints that describe how multiple types relate to each other, by doing something like this:</p> template<typename Victim1, typename Victim2> requires Lakosable<Victim1, Victim2> void lakos( Victim1 a, Victim2 b ); </pre> Here the Lakosable constraint can specify conditions that describe how the two types relate to each other, for example that Victim1::value_type is equal to the type of Victim2.</p> This is very good.</p> Now, the bit I want to argue against.</p> "Terse" Templates - the syntax I don't like</h3> Bjarne gave us an example of the std::merge function, which has lots of arguments, and very complex constraints on them. He showed us that these could all be nicely wrapped into a single Mergeable constraint (similar to the Lakosable constraint above) but he argued that there was still too much repetition. The repetition comes from the fact that several functions in the standard library have the exact same template parameters, with the exact same constraints on them, and that you have to mention the whole list of template parameters twice: once after the template keyword, and once in the requires condition.</p> This led him to look for a terser syntax.</p> So, he proposed a modest new construct that looks like this:</p> using Lakosable{Victim1,Victim2}; // (1) </pre> that allows a radical departure from everything that has gone before in terms of declaring templates. After we've made the declaration (1), we can declare the exact function we declared above with this little line:</p> void lakos( Victim1 a, Victim2 b ); // (2) </pre> The using declaration in (1) makes the names Victim1 and Victim2 available in the current namespace, and gives them special powers that mean functions taking parameters of type Victim1 or Victim2 are automatically function templates, even though the template keyword is nowhere to be seen.</p> There was some resistance in the room to this proposal. Most of it focussed on (2), and the fact that templates were being declared without it being visible because of the lack of the template keyword.</p> I'm actually ok with (2). In fact, my ficticious programming language Pepper</a> (which represents everything I think is Right in programming languages) provides a feature very much like this - all non-definite parameter types act as "implicit" templates in Pepper (see "implicit_templates.pepper" on the Examples</a> page).</p> Bjarne made a reasonable defence of (2), arguing that we often want new features to be "signposted" by new keywords (he cited user-defined types as an example - apparently some people wanted to require "class MyClass" instead of just "MyClass" every time we referred to a user-defined type) but later when they are familiar we want less verbose syntax. (Presumably the "new" feature he was talking about here is templates.)</p> My problem is with (1).</p> As my neighbour in the talk (whose name I missed, sorry) pointed out, what (1) does is dump 2 new names Victim1 and Victim2 in the namespace containing the lakos function template.</p> No-one wants these names.</p> In fact, why are we doing any of this?</p> The sole purpose of the exercise is to constrain the lakos function template. Why is the result putting 2 names into the namespace?</p> More seriously, in the case of the standard library, these names will go into the std:: namespace, and there could easily be clashes. If the std::merge function uses the name For for one of its template parameters (a Forward_iterator), and std::copy wants to use one with the same name, but with different constraints, it will override the definition of For.</p> I.e. If we do this:</p> namespace std { using Mergeable{For,For2,Out}; // define std::merge } // and somewhere else: namespace std { using Copyable{For,Out}; // define std::copy } </pre> then the (useless) value of std::For will be different depending on the order in which we import the header files.</p> I Think</h3> I think.</p> Please correct me if I'm wrong.</p> Conclusion</h3> If I'm right, this all seems bad and Wrong.</p> What was wrong with:</p> template<typename Victim1, typename Victim2> requires Lakosable<Victim1, Victim2> void lakos( Victim1 a, Victim2 b ); </pre> anyway?</p> Comments</h2> Phil Nash 2013-04-13</span> </h3> As I said on twitter, I think the point Bjarne didn’t really get across in the talk – but he said to me after (if I understood him correctly) – was that this not just about making some library corner case a bit terser. His vision is that this would be a new, default, approach to writing any kind of templates! I still need to chew on this bit overall I like the idea of overhauling the template syntax. For me it’s always been unnecessarily verbose – even without introducing constraints (and don’t get me started on the current constraints workarounds!). After all look at generics in something like C#:</p> void doSomethingWith( … ) { /* … */ }</p> compared to C++</p> template void doSomethingWith( … ) { /* … */ }</p> Do we really need the template keyword? Do we really need all those typenames? (that was my question to Bjarne and his response was that with his terse templates proposal you would never need to write another "typename").</p> I’m all for eliminating boilerplate. But I share the concern where it introduces different issues. One is polluting the namespace. This is unfortunate, but not a show-stopper for me. I agree with Bjarne that this is akin to introducing other names, such as class names. You just "deal with it". Having said that I prefer the suggestion from the floor (I don’t know who from) to require that the concept names be qualified by the constraint names (not sure if that’s the appropriate use of those terms). However I think even the constraint name should not pollute the namespace if it only has a single use.</p> My beef is more with the non-local effects. If this is the default way to write templates – not just how you might share constraints across templates – then there seems to be a disconnect between the constraints and the template functions and classes that use them.</p> Interesting times…</p> </div> Andy Balaam 2013-04-15</span> </h3> His vision is that this would be a new, default, approach to writing any kind of templates!</p> </blockquote> Sounds good to me – I just don’t like the way it’s done here.</p> I agree with Bjarne that this is akin to introducing other names, such as class names.</p> </blockquote> But those class names are potentially useful, whereas these aren’t?</p> I agree about non-local effects.</p> I think Bjarne is mixing together a general idea that templates could/should become lighter on syntax with a specific idea about how to express constraints. I don’t see how they go together at all.</p> </div> setUp and tearDown considered harmful Andy Balaam — Wed, 27 Mar 2013 15:08:45 +0000 Some unit test frameworks provide methods (often called setUp and tearDown, or annotated with @Before and @After) that are called automatically before a unit test executes, and afterwards.</p> This structure is presumably intended to avoid repetition of code that is identical in all the tests within one test file.</p> I have always instinctively avoided using these methods, and when a colleague used them recently I thought I should try to write up why I feel negative about them. Here it is:</p> Update: note I am talking about unit tests here. I know these frameworks can be used for other types of test, and maybe in that context these methods could be useful.</strong></p> 1. It's action at a distance</h3> setUp and tearDown are called automatically, with no indication in your code that you use them, or don't use them. They are "magic", and everyone hates magic.</p> If someone is reading your test (because it broke, probably) they don't know whether some setUp will be called without manually scanning your code to find out whether it exists. Do you hate them?</p> 2. setUp contains useless stuff</h3> How many tests do you have in one file? When you first write it, maybe, just maybe, all the tests need the exact same setup. Later, you'll write new tests that only use part of it.</p> Very soon, you grow an uber-setUp that does all the setup for various different tests, creating objects you don't need. This adds complexity for everyone who has to read your tests - they don't know which bits of setUp are used in this test, and which are cruft for something else.</p> 3. They require member variables</h3> The only useful work you can do inside setUp and tearDown is creating and modifying member variables.</p> Now your tests aren't self-contained - they use these member variables, and you must make absolutely sure that your test works no matter what state they are in. These member variables are not useful for anything else - they are purely an artifact of the choice to use setUp and tearDown.</p> 4. A named function is better</h3> When you have setup code to share, write a function or method. Give it a name, make it return the thing it creates. By giving it a name you make your test easier to read. By returning what it creates, you avoid the use of member variables. By avoiding the magic setUp method, you give yourself the option of calling more than one setup function, making code re-use more granular (if you want).</p> 5. What goes in tearDown?</h3> If you're using tearDown, what are you doing?</p> Are you tearing down some global state? I thought this was a unit test?</p> Are you ensuring nothing is left in an unpredictable state for future tests? Surely those tests guarantee their state at the start?</p> What possible use is there for this function?</p> Conclusion</h3> A unit test should be a self-contained pure function, with no dependencies on other state. setUp and tearDown force you to depend on member variables of your test class, for no benefits over named functions, except that you don't have to type their names. I consider typing the name of a properly-named function to be a benefit.</p> Comments</h2> Phil Nash 2013-03-27</span> </h3> Good post. You’re very much in tune with the thinking of James Newkirk, who led the NUnit team, and later wrote x-Unit:</p> http://jamesnewkirk.typepad.com/posts/2007/09...</a></p> It’s also something that gets consideration in CATCH (my C++ test framework). In CATCH you can write a self contained function that sets up state, on the stack (as this is C++), and inline with a number a Sections that run in isolation but share that set-up code. Because you can nest Sections you can share just the setup that is common to child Sections and even have bits of setup shared between more specific bits of setup – and all in a way that is a lot simpler than I just made it sound :–)</p> Come to my ACCU 2013 talk to see it in action ;–)</p> </div> Andy Balaam 2013-03-28</span> </h3> Hi Phil, thanks! As ever, your talk clashes with about 3 others I want to see. I am really interested in CATCH :(</p> </div> Phil Nash 2013-03-28</span> </h3> try catch me after (see what I did there?)</p> </div> Jonathan Allen 2013-03-28</span> </h3> No, its not a "unit test". Stop assuming that every test has to be a unit test.</p> </div> Andy Balaam 2013-03-29</span> </h3> Fair enough. If it’s not a unit test maybe there is a use for these functions.</p> </div> Andy Balaam 2013-03-29</span> </h3> Phil: will do :)</p> </div> Why use git for source control? Andy Balaam — Wed, 13 Mar 2013 08:20:53 +0000 Series: Why git?, Basics</a>, Branches</a>, Merging</a>, Remotes</a></p> Putting your code in git is fast, flexible and powerful. You can track versions on a single machine, or scale up to thousands of people working together, with sub-teams, reviews and cherry-picking of changes. Don't fear branching any more:</p> </iframe> Slides: Why use git for source control? slides</a>.</p> Set the volume in OpenBox/LXDE (or on the command line) with PulseAudio and Ubuntu Andy Balaam — Tue, 05 Mar 2013 11:09:13 +0000 I am switching to LXDE</a>, and enjoying it, but a few things require some manual config before it's just how I like it.</p> To control the sound volume with the volume buttons, the default LXDE config in ~/openbox/lxde-rc.xml contains an entry like this:</p>  <keybind key="XF86AudioRaiseVolume"> <action name="Execute"> <command>amixer -q sset Master 3%+</command> </action> </keybind> </pre> (Inside the <keyboard> section.)</p> This doesn't work for me, but we can do it by sending a command to PulseAudio, using the pactl command. The command to increase the volume is:</p> pactl -- set-sink-volume 0 +5% </pre> To decrease the volume, put "-5%" instead of "+5%". Note that if you have more than one enabled audio sink you might need to change the "0" to a "1" or something else. Running pactl stat should help you here.</p> [Beware of bug 686667</a>, meaning you can't use pacmd and you must have the -- at the beginning.]</p> So the correct recipe for your OpenBox config in ~/openbox/lxde-rc.xml is:</p> <keybind key="XF86AudioRaiseVolume"> <action name="Execute"> <command>pactl -- set-sink-volume 0 +5%</command> </action> </keybind> <keybind key="XF86AudioLowerVolume"> <action name="Execute"> <command>pactl -- set-sink-volume 0 -5%</command> </action> </keybind> </pre> After editing this file you can run:</p> openbox --reconfigure </pre> To update without restarting OpenBox.</p> Of course, because I don't have volume control buttons, I want my volume to change with Ctrl-Alt-PageUp and Ctrl-Alt-PageDown, so I use this recipe:</p> <keybind key="C-A-Prior"> <action name="Execute"> <command>pactl -- set-sink-volume 0 +5%</command> </action> </keybind> <keybind key="C-A-Next"> <action name="Execute"> <command>pactl -- set-sink-volume 0 -5%</command> </action> </keybind> </pre> but that's just me.</p> Comments</h2> Dakota 2015-05-28</span> </h3> much appreciated! I’m using an old laptop so I switched from Cinnamon (which I prefer, but it’s bulky) to LXDE. I had the same problem. reconfiguring Openbox gives me flashbacks to when I used Crunchbang, as the media buttons didn’t work by default.</p> </div> Everybody loves build.xml (test-driven Ant) Andy Balaam — Sat, 16 Feb 2013 14:39:53 +0000 In the previous post</a> we looked at how it is possible to write reasonable code in Ant</a>, by writing small re-usable blocks of code.</p> Of course, if you're going to have any confidence in your build file you're going to need to test it. Now we've learnt some basic Ant techniques, we're ready to do the necessary magic that allows us to write tests.</p> </iframe> Slides: Everybody loves build.xml slides</a>.</p> First, let me clear up what</em> we're testing:</p> What do we want to test?</h2> We're not testing our Java code. We know how to do that, and to run tests, if we've written them using JUnit</a>, just needs a <junit> tag in our build.xml. (Other testing frameworks are available and some people say they're better).</p> The things we want to test are:</p> build artifacts</strong> - the "output" of our builds i.e. JAR files, zips and things created when we run the build,</li> build logic</strong> - such as whether dependencies are correct, whether the build succeeds or fails under certain conditions, and</li> units of code</strong> - checking whether individual macros or code snippets are correct.</li> </ul> Note, if you're familiar with the terminology, that testing build artifacts can never be a "unit test", since it involves creating real files on the disk and running the real build.</p> Below we'll see how I found ways to test build artifacts, and some ideas I had to do the other two, but certainly not a comprehensive solution. Your contributions are welcome.</p> Before we start, let's see how I'm laying out my code:</p> Code layout</h2> build.xml - real build file asserts.xml - support code for tests test-build.xml - actual tests </pre> I have a normal build file called build.xml, a file containing support code for the tests (mostly macros allowing us to make assertions) called asserts.xml, and a file containing the actual tests called test-build.xml.</p> To run the tests I invoke Ant like this:</p> ant -f test-build.xml test-name </pre> test-build.xml uses an include to get the assertions:</p> <include file="asserts.xml"/> </pre> Tests call a target inside build.xml using subant, then use the code in asserts.xml to make assertions about what happened.</p> Simple example: code got compiled</h2> If we want to check that a <javac ...> task worked, we can just check that a .class file was created:</p> Here's the test, in test-build.xml:</p> <target name="test-class-file-created"> <assert-target-creates-file target="build" file="bin/my/package/ExampleFile.class" /> </target> </pre> We run it like this:</p> ant -f test-build.xml test-class-file-created</pre> The assert-target-creates-file assertion is a macrodef in asserts.xml like this:</p> <macrodef name="assert-target-creates-file"> <attribute name="target"/> <attribute name="file"/> <sequential> <delete file="@{file}" quiet="true"/> <subant antfile="build.xml" buildpath="." target="@{target}"/> <assert-file-exists file="@{file}"/> </sequential> </macrodef> </pre> It just deletes a file, runs the target using subant, then asserts that the file exists, which uses this macrodef:</p> <macrodef name="assert-file-exists"> <attribute name="file"/> <sequential> <echo message="Checking existence of file: @{file}"/> <fail message="File '@{file}' does not exist."> <condition> <not><available file="@{file}"/></not> </condition> </fail> </sequential> </macrodef> </pre> This uses a trick I've used a lot, which is the fail task, with a condition inside it, meaning that we only fail if the condition is satisfied. Here we use not available which means fail if the file doesn't exist.</p> Harder example: JAR file</h2> Now let's check that a JAR file was created, and has the right contents. Here's the test:</p> <target name="test-jar-created-with-manifest"> <assert-target-creates-file target="build" file="dist/MyProduct.jar" /> <assert-file-in-jar-contains jarfile="dist/MyProduct.jar" filename="MANIFEST.MF" find="Main-Class: my.package.MyMain" /> </pre> This just says after we're run the target, the file MyProduct.jar exists, and it contains a file called MANIFEST.MF that has the right Main-Class information in it.</p> assert-file-in-jar-contains looks like this:</p> <macrodef name="assert-file-in-jar-contains"> <attribute name="jarfile"/> <attribute name="filename"/> <attribute name="find"/> <sequential>  <delete dir="${tmpdir}/unzip"/> <unzip src="@{jarfile}" dest="${tmpdir}/unzip"/> <fail message="@{jarfile}:@{filename} should contain @{find}"> <condition> <resourcecount when="equal" count="0"> <fileset dir="${tmpdir}/unzip"> <and> <filename name="**/@{filename}"/> <contains text="@{find}"/> </and> </fileset> </resourcecount> </condition> </fail> <delete dir="${tmpdir}/unzip"/> </sequential> </macrodef> </pre> Which basically unzips the JAR into a directory, then searches the directory using fileset for a file with the right name and contents, and fails if it's not found (i.e. if the resourcecount of the fileset is zero. These are the kinds of backflips you need to do to bend Ant to your will.</p> Or, you can choose the Nuclear Option.</p> The Nuclear Option</h2> If ant tasks just won't do, since Ant 1.7 and Java 1.6 we can drop into a <script> tag.</p> You ain't gonna like it:</p> <script language="javascript"><![CDATA[ system.launchMissiles(); // Muhahahaha ]]></script> </pre> The script tag allows us to use a scripting language as provided through the JSR 223</a> Java feature directly within our Ant file, meaning we can do anything.</p> In all the JVMs I've tried, the only scripting language actually available is JavaScript, provided by the Rhino</a> virtual machine, which is now part of standard Java.</p> When using the script tag, expect bad error messages. Rhino produces unhelpful stack traces, and Ant doesn't really tell you what went wrong.</p> So now we know how to test the artifacts our build produces, but what about directly testing the logic in build.xml?</p> Testing build logic</h2> We want to:</p> Confirm that targets succeed or fail under certain conditions</p> </li> Check indirect dependencies are as expected</p> </li> Test a unit of Ant logic (e.g. a macrodef)</p> </li> </ul> Success and failure</h2> Here's a little macro I cooked up to assert that something is going to fail:</p> <macrodef name="expect-failure"> <attribute name="target"/> <sequential> <local name="ex.caught"/> <script language="javascript"><![CDATA[ try { project.executeTarget( "@{target}" ); } catch( e ) { project.setProperty( "ex.caught", "yes" ) } ]]></script> <fail message="@{target} succeeded!!!" unless="ex.caught"/> </sequential> </macrodef> </pre> I resorted to the Nuclear Option of a script tag, and used Ant's Java API (through JavaScript) to execute the target, and catch any exceptions that are thrown. If no exception is thrown, we fail.</p> Testing dependencies</h3> To check that the dependencies are as we expect, we really want to run ant's dependency resolution without doing anything. Remarkably, ant has no support for this.</p> But we can hack it in:</p> <target name="printCdeps"> <script language="javascript"><![CDATA[ var targs = project.getTargets().elements(); while( targs.hasMoreElements() ) { var targ = targs.nextElement(); targ.setUnless( "DRY.RUN" ); } project.setProperty( "DRY.RUN", "1" ); project.executeTarget( "targetC" ); ]]></script> </target> </pre> (See Dry run mode for Ant</a> for more.)</p> Now we need to be able to run a build and capture the output. We can do that like this:</p> <target name="test-C-depends-on-A"> <delete file="${tmpdir}/cdeps.txt"/> <ant target="printCdeps" output="${tmpdir}/cdeps.txt" /> <fail message="Target A did not execute when we ran C!"> <condition> <resourcecount when="equal" count="0"> <fileset file="${tmpdir}/cdeps.txt"> <contains text="targetA:"/> </fileset> </resourcecount> </condition> </fail> <delete file="${tmpdir}/cdeps.txt"/> </target> </pre> We use ant to run the build, telling it to write to a file cdeps.txt. Then, to assert that C depends on A, we just fail if cdeps.txt doesn't contain a line indicating we ran A. (To assert a file contains a certain line we use a load of fail, condition, resourcecount and fileset machinery as before.)</p> So, we can check that targets depend on each other, directly or indirectly. Can we write proper unit tests for our macrodefs?</p> Testing ant units</h3> To test a macrodef or target as a piece of logic, without touching the file system or really running it, we will need fake versions of all the tasks, including <jar>, <copy>, <javac> and many more.</p> If we replace the real versions with fakes and then run our tasks, we can set up our fakes to track what happened, and then make assertions about it.</p> If we create a file called real-fake-tasks.xml, we can put things like this inside:</p> <macrodef name="jar"> <attribute name="destfile"/> <sequential> <property name="jar.was.run" value="yes"/> </sequential> </macrodef> </pre> and, in build.xml we include something called fake-tasks.xml, with the optional attribute set to true:</p> <include file="fake-tasks.xml" optional="true"/> </pre> If the target we want to test looks like this (in build.xml):</p> <target name="targetA"> <jar destfile="foo.jar"/> </target> </pre> Then we can write a test like this in test-build.xml:</p> <target name="test-A-runs-jar" depends="build.targetA"> <fail message="Didn't jar!" unless="jar.was.run"/> </target> </pre> and run the tests like this:</p> cp real-fake-tasks.xml fake-tasks.xml ant -f test-build.xml test-A-runs-jar rm fake-tasks.xml </pre> If fake-tasks.xml doesn't exist, the real tasks will be used, so running your build normally should still work.</p> This trick relies on the fact that our fake tasks replace the real ones, which appears to be an undocumented behaviour of my version of Ant. Ant complains about us doing this, with an error message that sounds like it didn't work, but actually it did (on my machine).</p> If we wanted to avoid relying on this undocumented behaviour, we'd need to write our real targets based on special macrodefs called things like do-jar and provide a version of do-jar that hands off to the real jar, and a version that is a fake. This would be a lot of work, and pollutes our production code with machinery needed for testing, but it could work with Ant's documented behaviour, making it unlikely to fail unexpectedly in the future.</p> Summary</h2> You can write Ant code in a test-driven way, and there are even structures that allow you to write things that might be described as unit tests.</p> At the moment, I am using mostly the "testing artifacts" way. The tests run slowly, but they give real confidence that your build file is really working.</p> Since I introduced this form of testing into our build, I enjoy working with build.xml a lot more, because I know when I've messed it up.</p> But I do spend more time waiting around for the tests to run.</p> Comments</h2> Per Arnold Blaasmo 2013-09-12</span> </h3> Hi, I have been working in a similar scope trying to do TDD on the ANT build scripts themselves. I have used antunit (http://ant.apache.org/antlibs/antunit/</a>) for doing asserts on macrodefs.</p> Have you considered using Antunit?</p> Regards Per A.</p> </div> Andy Balaam 2013-09-12</span> </h3> Hi Per, no, I didn’t find it when I was looking for something like this. Thanks for pointing it out. It looks pretty similar to what I’ve home-grown myself.</p> </div> Everybody hates build.xml (code reuse in Ant) Andy Balaam — Fri, 08 Feb 2013 08:03:59 +0000 If you're starting a new Java project, I'd suggest suggest considering the many alternatives to Ant, including Gant</a>, Gradle</a>, SCons</a> and, of course, Make</a>. This post is about how to bend Ant to work like a programming language, so you can write good code in it. It's seriously worth considering a build tool that actually is a programming language.</p> If you've chosen Ant, or you're stuck with Ant, read on.</p> </iframe> Slides: Everybody hates build.xml slides</a>.</p> Most projects I've been involved with that use Ant have a hateful build.xml surrounded by fear. The most important reason for this is that the functionality of the build file is not properly tested, so you never know whether you've broken it, meaning you never make "non-essential" changes i.e. changes that make it easier to use or read. A later post and video will cover how to test your build files, but first we must address a pre-requisite:</p> Can you write good code in Ant, even if you aren't paralysed by fear?</p> One of the most important aspects of good code is that you only need to express each concept once. Or, to put it another way, you can re-use code.</p> I want to share with you some of the things I have discovered recently about Ant, and how you should (and should not) re-use code.</p> But first:</p> What is Ant?</h2> Ant is 2 languages:</p> A declarative language to describe dependencies</li> A procedural language to prescribe actions</li> </ul> In fact, it's just like a Makefile (ignore this if Makefiles aren't familiar). A Makefile rule consists of a target (the name before the colon) with its dependencies (the names after the colon), which make up a declarative description of the dependencies, and the commands (the things indented by tabs) which are a normal procedural description of what to do to build that target.</p> # Ignore this if you don't care about Makefiles! target: dep1 dep2 # Declarative action1 # Procedural action2 </pre> The declarative language</h3> In Ant, the declarative language is a directed graph of targets and dependencies:</p> <target name="A"/> <target name="B" depends="A"/> <target name="C" depends="B"/> <target name="D" depends="A"/> </pre> This language describes a directed graph of dependencies. I.e. they say what depends on what, or what must be built before you can build something else. Targets and dependencies are completely separate from what lives inside them, which are tasks.</p> The procedural language</h3> The procedural language is a list of tasks:</p> <target ...> <javac ...> <copy ...> <zip ...> <junit ...> </target> </pre> When the dependency mechanism has decided a target will be executed, its tasks are executed one by one in order, just like in a programming language. Except that tasks live inside targets, they are completely separate from them. Essentially each target has a little program inside it consisting of tasks, and these tasks are a conventional programming language, nothing special (except for the lack of basic looping and branching constructs).</p> I'm sorry if the above is glaringly obvious to you, but it only recently became clear to me, and it helped me a lot to think about how to improve my Ant files.</p> Avoiding repeated code</h2> Imagine you have two similar Ant targets:</p> <target name="A"> <javac srcdir="a/src" destdir="a/bin" classpath="myutil.jar" debug="false" /> </target> <target name="B"> <javac srcdir="b/code" destdir="b/int" classpath="myutil.jar" debug="false" /> </target> </pre> The classpath and debug information are the same in both targets, and we would like to write this information in one single place. Imagine with me that the code we want to share is too complex for it to be possible to store it as the values of properties in some properties file.</p> How do we share this code?</p> The Wrong Way: antcall</h3> Here's the solution we were using in my project until I discovered the right way:</p> <target name="compile"> <javac srcdir="${srcdir}" destdir="${destdir}" classpath="myutil.jar" debug="false" /> </target> <target name="A"> <antcall target="compile"> <param name="srcdir" value="a/src"/> <param name="destdir" value="a/bin"/> </antcall> </target> <target name="B"> <antcall target="compile"> ... </pre> Here we put the shared code into a target called compile, which makes use of properties to access the varying information (or the parameters, if we think of this as a function). The targets A and B use the <antcall> task to launch the compile target, setting the values of the relevant properties.</p> This works, so why is it Wrong?</p> Why not antcall?</h3> antcall launches a whole new Ant process and runs the supplied target within that. This is wrong because it subverts the way Ant is supposed to work. The new process will re-calculate all the dependencies in the project (even if our target doesn't depend on anything) which could be slow. Any dependencies of the compile target will be run even if they've already been run, meaning some of your assumptions about order of running could be incorrect, and the assumption that each target will only run once will be violated. What's more, it subverts the Ant concept that properties are immutable, and remain set once you've set them: in the example above, srcdir and destdir will have different values at different times (because they exist inside different Ant processes).</p> Basically what we're doing here is breaking all of Ant's paradigms to force it to do what we want. Before Ant 1.8 you could have considered it a necessary evil. Now, it's just Evil.</p> The Horrific Way: custom tasks</h3> Ant allows you to write your own tasks (not targets) in Java. So our example would look something like this:</p> Java:</p> public class MyCompile extends Task { public void execute() throws BuildException { Project p = getProject(); Javac javac = new Javac(); javac.setSrcdir( new Path( p, p.getUserProperty( "srcdir" ) ) ); javac.setDestdir( new File( p.getUserProperty( "destdir" ) ) ); javac.setClasspath( new Path( p, "myutil.jar" ) ); javac.setDebug( false ); javac.execute(); } } </pre> Ant:</p> <target name="first"> <javac srcdir="mycompile"/> <taskdef name="mycompile" classname="MyCompile" classpath="mycompile"/> </target> <target name="A" depends="first"> <mycompile/> </target> <target name="B" depends="first"> <mycompile/> </target> </pre> Here we write the shared code as a Java task, then call that task from inside targets A and B. The only word to describe this approach is "cumbersome". Not only do we need to ensure our code gets compiled before we try to use it, and add a taskdef to allow Ant to see our new task (meaning every target gets a new dependency on the "first" target), but much worse, our re-used code has to be written in Java, rather than the Ant syntax we're using for everything else. At this point you might start asking yourself why you're using Ant at all - my thoughts start drifting towards writing my own build scripts in Java ... anyway, I'm sure that would be a very bad idea.</p> The Relatively OK Way: macrodef</h3> So, enough teasing. Here's the Right Way:</p> <macrodef name="mycompile"> <attribute name="srcdir"/> <attribute name="destdir"/> <sequential> <javac srcdir="@{srcdir}" destdir="@{destdir}" classpath="myutil.jar" debug="false" /> </sequential> </macrodef> <target name="A"> <mycompile srcdir="a/src" destdir="a/bin"/> </target> <target name="B"> <mycompile srcdir="b/code" destdir="b/int"/> </target> </pre> Since Ant 1.8, we have the macrodef task, which allows us to write our own tasks in Ant syntax. In any other language these would be called functions, with arguments which Ant calls attributes. You use these attributes by giving their name wrapped in a @{} rather than the normal ${} for properties. The body of the function lives inside a sequential tag.</p> This allows us to write re-usable tasks within Ant. But what about re-using parts from the other language - the declarative targets and dependencies?</p> Avoiding repeated dependencies?</h2> Imagine we have a build file containing targets like this:</p> <target name="everyoneneedsme"... <target name="A" depends="everyoneneedsme"... <target name="B" depends="everyoneneedsme"... <target name="C" depends="everyoneneedsme"... <target name="D" depends="everyoneneedsme"... </pre> In Ant, I don't know how to share this. The best I can do is make a single target that is re-used whenever I want the same long list of dependencies, but in a situation like this where everything needs to depend on something, I don't know what to do. (Except, of course, drop to the Nuclear Option of the <script> tag, which we'll see next time.)</p> I haven't used it in anger, but this kind of thing seems pretty straightforward with Gradle</a>. I believe the following is roughly equivalent to my example above, but I hope someone will correct me if I get it wrong:</p> task everyoneneedsme tasks.whenTaskAdded { task -> task.dependsOn everyoneneedsme } task A task B task C task D </pre> (Disclaimer: I haven't run this.)</p> So, if you want nice features in your build tool, like code-reuse and testability, you should consider a build tool that is integrated into a grown-up programming language where all this stuff comes for free. But, if you're stuck with Ant, you should not despair: basic good practice is possible if you make the effort.</p> Comments</h2> Ti 2017-08-14</span> </h3> BTW, "proscribe" and "prescribe" are two very different words. I think you wanted the second but wrote the first by mistake.</p> </div> Andy Balaam 2017-08-17</span> </h3> Thanks Ti, I fixed the pr[oe]scribe problem.</p> </div> Gilbert 2014-07-23</span> </h3> Good point about antcall – scripts using antcall are a PITA ! btw. The Macrodef Task exists since Ant 1.6.0 ^{12</sup>⁄_{2003</sub>, see http://ant.apache.org/manual/Tasks/macrodef.html</a></p> </div>}} Andy Balaam 2014-07-23</span> </h3> Hi Gilbert, thanks for the correction.</p> </div> Dry run mode for Ant (ant -n, ant --dry-run) Andy Balaam — Thu, 31 Jan 2013 16:47:57 +0000 I am working on the problem of writing Ant build files in a test-driven way. One thing I found myself needing was a "dry run" mode, like many Unix tools have. For example, make has the -n or --dry-run option, which shows what it would have done, but doesn't really do it.</p> Today I found a partial solution to this problem, so that you can at least see which dependencies will be run when you run a particular ant target.</p> It's an horrific hack, but it's the best I can do at the moment.</p> We write some code in a <script> tag to hack all the targets in our project (at runtime). We modify the targets so they all have an "unless" attribute, set to a property name of "DRY-RUN". Then we set the "DRY-RUN" property, and execute our target.</p> Ant prints out the names of all the targets in the dependency chain, even if they are not executed because of an unless attribute.</p> Note: this code makes use of the Ant <script> script tag, which is an Ant 1.8+ feature. Using JavaScript inside this tag seems to be supported in Oracle, OpenJDK and IBM versions of Java, but is not guaranteed.</p> <?xml version="1.0" encoding="UTF-8"?> <project default="build"> <target name="targetA"/> <target name="targetB" depends="targetA"> <echo message="DON'T RUN ME"/> </target> <target name="targetC" depends="targetB"/> <target name="build" depends="targetB"/> <target name="dry-run"> <do-dry-run target="build"/> </target> <macrodef name="do-dry-run"> <attribute name="target"/> <sequential> <script language="javascript"><![CDATA[ var targs = project.getTargets().elements(); while( targs.hasMoreElements() ) { var targ = targs.nextElement(); targ.setUnless( "DRY.RUN" ); } project.setProperty( "DRY.RUN", "1" ); project.executeTarget( "@{target}" ); ]]></script> </sequential> </macrodef> </project> </pre> Running this build file normally, the tasks in the targets execute, so we can see that the <echo> happens:</p> $ ant Buildfile: build.xml targetA: targetB: [echo] DON'T RUN ME build: BUILD SUCCESSFUL Total time: 0 seconds </pre> But when we run the dry-run target, only the target names are printed, and the <echo> task (and any other tasks) don't:</p> $ ant dry-run Buildfile: build.xml dry-run: targetA: targetB: build: BUILD SUCCESSFUL Total time: 0 seconds </pre> A lot of pain, for a partial implementation of very simple functionality that you'd expect to be a built-in feature? I couldn't possibly comment.</p> Comments</h2> Russel Winder 2013-02-01</span> </h3> Isn’t the real problem here that you are using Ant? Perhaps try a modern build tool such as Gradle? (http://www.gradle.org</a>).</p> </div> Andy Balaam 2013-02-01</span> </h3> Hi Russell, yes it is. But, in case others are similarly doomed, I hope this might help them.</p> </div> My First Raspberry Pi Game " Part 12 " Scoring, done! Andy Balaam — Wed, 16 Jan 2013 08:19:58 +0000 Parts: 1</a>, 2</a>, 3</a>, 4</a>, 5</a>, 6</a>, 7</a>, 8</a>, 9</a>, 10</a>, 11</a>, 12.</p> Writing your first ever computer program on the Raspberry Pi.</em></p> Today, we finish!</p> </iframe> Our game is almost done. All we need to do now is let you play several times, and give you a score at the end.</p> First, because we're going to use it lots of times, we need to make the ready_screen function set its background colour properly. Open redgreen.py in LeafPad, and add a single line to the function ready_screen, making it look like this:</p> def ready_screen(): screen.fill( pygame.Color( "black" ) )</span> white = pygame.Color( "white" ) write_text( screen, "Ready?", white, True ) pygame.display.flip() </pre> Previously, ready_screen was always the first thing we did, so we got away with not drawing a background colour because it starts off plain black. Now, we need to do it.</p> Next, let's do the really interesting part. We want to play the game several times, and whenever we want to do something several times, we need a loop. This time we'll use a for loop, letting us go through a list of things. Scroll to the very bottom, and change the code to look like this:</p> # We start from here</span> start() for i in range( 10 ):</span> ready_screen()</span> wait()</span> shape()</span> end() </pre> The new lines are green above, and lines that haven't changed except being indented by putting four spaces at the beginning are blue.</p> A for loop lets you run through a list of things, running the same code each time. A for loop always looks like for NAME in LIST where NAME is the name of a new variable, and LIST is a list of things. What we've done here is make a list of 10 numbers by calling the range function and giving it an argument of 10, and told Python to put the particular item of the list that we're working on now into a variable called i.</p> So, the ready_screen, wait and shape functions will each get called 10 times. Each time they are called, i will be a different number. We're not using i yet, so all that matters for the moment is that the code runs 10 times. Try it out by opening LXTerminal and typing ./redgreen.py, and you'll see that you can play the game 10 times, and then it will finish.</p> Playing 10 times is all very well, but it's not a lot of fun if I can't see how well I've done at the end. Let's keep track of our score.</p> We'll award the player 1 point for every time they get it right, and no points if they get it wrong. The places where we know which of these has happened are in red_shape and green_shape. Let's change them to pass back a score (either 1 or 0) depending on what you did:</p> def green_shape(): ...the rest of green_shape is still here...</span> pressed = shape_wait() if pressed: green_success() return 1</span> else: green_failure() return 0</span> </pre> def red_shape(): ...the rest of green_shape is still here...</span> pressed = shape_wait() if pressed: red_failure() return 0</span> else: red_success() return 1</span> </pre> I've abbreviated it above, but we're not changing anything in these functions except at the very bottom, where we're adding two return lines to each function.</p> Whenever the player succeeds, we return a score of 1 point, and whenever they fail we return 0 points.</p> We're not doing anything with this score yet. We call the green_shape and red_shape functions from inside shape, so first let's make sure shape passes back the answer to where we need it:</p> def shape(): GREEN = 0 RED = 1 shape = random.choice( [GREEN, RED] ) if shape == GREEN: return</span> green_shape() else: return</span> red_shape() </pre> shape doesn't need to do anything special here - just take the answer coming from green_shape or red_shape and use the return statement to pass it back to us.</p> Now shape is giving us back an answer, we can use it in the main code right at the bottom:</p> start() correct = 0</span> for i in range( 10 ): ready_screen() wait() correct += </span>shape() end( correct</span> ) </pre> We've made a variable called correct that keeps hold of how many correct answers we've been given (i.e. the score). It starts off as zero, and every time we call shape we add on the answer that comes back. shape will either return 0 or 1, so correct will increase by either 0 or 1 each time.</p> The last thing we've done here is pass the answer (the player's final score) into the end function so we can display it. To use this answer, we need to change end a bit:</p> def end( correct</span> ): print "You got %d correct answers" % correct</span> screen.fill( pygame.Color( "black" ) ) white = pygame.Color( "white" ) write_text( screen, "Thanks for playing!", white, True ) msg = "Score: %d Press a key to exit" % correct</span> write_text( screen, msg</span>, white, False ) pygame.display.flip() pygame.event.clear() timed_wait( 0, press_events ) </pre> We changed the def line to allow us to pass in the score, giving it the same name we used below, correct. Then we added a line that prints out the answer into the terminal, just for good measure, and we modified the write_text line, splitting it into 2 parts - creating a variable called msg containing our message, and then using it on the next line.</p> Twice above we've used a nice feature of Python that makes building our own messages quite simple. If you write a string like "Score: %d Press a key to exit" you can substitute a number into it using the % "operator" as we've done (an operator is something like + or / that combines 2 things). Where the %d appears in the string, it gets replaced by the number inside the variable you supply (correct in our case). You can also substitute in other strings (using %s) and lots of other things if you want to. This allows us to put the score into a string and then print it on the screen.</p> If you try your game now you will see it counts how many right answers you got and tells you at the end. Wouldn't it be better, though, if it told you how you were doing all the way through?</p> Scroll up to the ready_screen function and modify it to take two arguments and use them to keep us informed:</p> def ready_screen( go_number, correct</span> ): screen.fill( pygame.Color( "black" ) ) white = pygame.Color( "white" ) write_text( screen, "Ready?", white, True ) go_number_str = "Turn: %d Score: %d" % ( ( go_number + 1 ), correct )</span> write_text( screen, go_number_str, pygame.Color( "white" ), False )</span> pygame.display.flip() </pre> The arguments we take are called go_number and correct. correct will be the current score, as we've seen before, and go_number is the counter telling us how far we've got.</p> We use a slightly different form of the % operator here to substitute two values into a string instead of one. To do this, we put a list of values on the right instead of just one: ( ( go_number + 1 ), correct ). We need brackets around the outside so that Python knows it is a list and doesn't just take the first value on its own. When we use a list like this, the values will be substituted in order, one for each %d (or %s or similar) that is in the string. You must always have the same number of %ds in the string as values in the list.</p> You may be wondering why we have to add one to go_number. We'll see in a moment.</p> To be able to provide the two new arguments to ready_screen we need to change the code right at the bottom to look like this:</p> start() correct = 0 for i in range( 10 ): ready_screen( i, correct</span> ) wait() correct += shape() end( correct ) </pre> Remember when we made the for loop I mentioned that i would be a different number each time we ran the code inside the loop? We pass that number in to ready_screen where it will be used as the go_number. We also pass in the current score, correct.</p> The reason why we needed to add 1 to go_number inside ready_screen is that when you have a loop like for i in range( 10 ), the variable i actually gets the values 0, 1, 2, ... with the last value being 9, instead of ranging from 1 to 10 as you might expect. The reasoning behind this is kind of lost in the mists of time, and kind of makes perfect sense, depending how you look at it. Anyway, believe me when I tell you that once you've got used to it you're going to find it warm and comforting, but for now you may find it a bit weird.</p> And, on that typically strange note, we have finished! Try out your program, and you should find it tells you what go you're on, and what your score is all the way through.</p> Something else you might like to do now is make your game run in full-screen mode (like many games). You can do that by changing the start function like this:</p> def start(): global screen pygame.init() screen = pygame.display.set_mode( screen_size, pygame.FULLSCREEN</span> ) </pre> If you have any problems, compare your version with mine here: redgreen.py</a></p> I've made a slightly extended version of the game that measures your reaction speed and gives you a score based on how quickly you press. In future I may even add more features. If you'd like to follow the project, you can find it here: redgreen on github</a>.</p> I'll be doing more series in the future, some for beginners like this one, and some more advanced topics. If you'd like to find out what I'm doing, subscribe to the blog RSS feed</a>, follow me on Twitter</a> or go to my YouTube page</a> and subscribe.</p> </iframe> My First Raspberry Pi Game " Part 11 " Being less rude Andy Balaam — Fri, 11 Jan 2013 08:14:59 +0000 Parts: 1</a>, 2</a>, 3</a>, 4</a>, 5</a>, 6</a>, 7</a>, 8</a>, 9</a>, 10</a>, 11, 12</a>.</p> Writing your first ever computer program on the Raspberry Pi.</em></p> </iframe> We've nearly finished our game. Next on our list is to fix that bug where you can't exit some of the time, and make our code a bit tidier in the process.</p> The first thing I want to do is make the Esc</strong> key quit the game. This is fairly normal behaviour, and will help if we run in full screen mode, where there is no close button to click.</p> Open up redgreen.py in LeafPad as usual, and find the function shape_wait and the line if evt.type == pygame.QUIT:. Replace the whole line with this:</p> if is_quit( evt ): </pre> We've replaced the code asking whether the event was a quit event (i.e. the user closed the window) with a call to a function. Let's write that function. Just above shape_wait type in this function:</p> def is_quit( evt ): return evt.type == pygame.QUIT </pre> You've just done another bit of refactoring. Instead of writing the code directly in the if line, we've added a call to the function, and the function does exactly the same thing as we did before: it returns True if the event is a quit event, and False otherwise. If you try the program now (by saving in LeafPad, opening LXTerminal and typing ./redgreen.py) you should see it behaves exactly as it did before.</p> You may well ask why we did it. The answer is because now we can change the is_quit function to do something extra. Replace it with this:</p> def is_quit( evt ): return ( evt.type == pygame.QUIT or ( evt.type == pygame.KEYDOWN and evt.key == pygame.K_ESCAPE ) ) </pre> This is a more complicated bit of logic, saying that we will return True if either of two things is true: EITHER the event is a quit event (as before), OR the event is a KEYDOWN, and the specific key that was pressed was the Escape key ("Esc") on the keyboard. Notice that the brackets around the "and" part help us know which bits go together - we don't want to quit for any keypress event, only one where the key is Escape.</p> If you try your program again, you should find you can press Escape to exit when you're looking at a red or green shape.</p> The shape_wait function is quite a useful one, and the next thing we want to do is use it in a few more places. Before we can do that, we need to refactor it to make it a bit more flexible.</p> Make a new function called timed_wait further up, just before start, and cut the entire body of shape_wait and paste it into timed_wait. So it looks like this:</p> def timed_wait(): event_types_that_cancel = pygame.KEYDOWN, pygame.MOUSEBUTTONDOWN ... all the rest of shape_wait here ...</span> pygame.time.set_timer( finished_waiting_event_id, 0 ) </pre> Now change shape_wait to look like this:</p> def shape_wait(): """ Wait while we display a shape. Return True if a key was pressed, or false otherwise. """ return timed_wait() </pre> As usual, we've just replaced some code with a call to a function that contains the exact same code, so hopefully our program will work exactly as before.</p> Now we're going to make timed_wait a bit more general, while still preserving all the same behaviour. We do this by changing some of the variables we use in timed_wait into arguments we pass in. Change shape_wait to look like this:</p> def shape_wait(): """ Wait while we display a shape. Return True if a key was pressed, or false otherwise. """ press_events = pygame.KEYDOWN, pygame.MOUSEBUTTONDOWN return timed_wait( 2000, press_events ) # 2 seconds </pre> and modify timed_wait to accept those arguments (notice I also added a description of what it does):</p> def timed_wait( time_to_wait, event_types_that_cancel ): """ Wait for time_to_wait, but cancel if a relevant event happens. Return True if cancelled, or False if we waited the full time. """ finished_waiting_event_id = pygame.USEREVENT + 1 pygame.time.set_timer( finished_waiting_event_id, time_to_wait ) pygame.event.clear() pressed = False waiting = True while waiting: evt = pygame.event.wait() if is_quit( evt ): quit() elif evt.type in event_types_that_cancel: waiting = False pressed = True elif evt.type == finished_waiting_event_id: waiting = False pygame.time.set_timer( finished_waiting_event_id, 0 ) return pressed </pre> Again, after these changes our program should work exactly as before - we're passing values in as arguments that are exactly what we used to make as variables. But now, timed_wait is a lot more flexible, and we'll use that flexibility very soon.</p> But first, we need to make a change to cover the unexpected. Inside timed_wait we've made a timer using pygame.time.set_timer and at the end we've cancelled it by calling pygame.time.set_timer again. However, if something goes wrong in between where we create the timer, and where we cancel it, it's possible that something called an "exception" will be "thrown". When an exception is thrown, the program stops running normally, line by line, and jumps out to somewhere else.</p> I'm not going to explain any more about exceptions here, but I am going to show you how to make absolutely sure that something will happen, even if an exception is thrown. The way to do that is to use a try ... finally block. We want to make sure our timer is always cancelled, so as soon as we've made it, we start a try block, and at the end we say finally. Anything inside that finally block will be run, even if an exception was thrown in the code inside the try block. The changes look like this:</p> def timed_wait( time_to_wait, event_types_that_cancel ): """ Wait for time_to_wait, but cancel if a relevant event happens. Return True if cancelled, or False if we waited the full time. """ finished_waiting_event_id = pygame.USEREVENT + 1 pygame.time.set_timer( finished_waiting_event_id, time_to_wait ) try:</span> pygame.event.clear() pressed = False waiting = True while waiting: evt = pygame.event.wait() if is_quit( evt ): quit() elif evt.type in event_types_that_cancel: waiting = False pressed = True elif evt.type == finished_waiting_event_id: waiting = False</span> finally:</span> pygame.time.set_timer( finished_waiting_event_id, 0 )</span> return pressed </pre> The lines in green are new, and the ones in blue are just indented by four more spaces to make them part of the try and finally blocks. Now, we know that even if something goes wrong while we're waiting, we will always cancel the timer we set up. Yet more good manners!</p> Now, after all that work, we finally have a timed_wait function that is flexible enough to be used everywhere we want to wait for something. Let's start with the wait function. Change it to look like this:</p> def wait(): time_to_wait = random.randint( 1500, 3000 ) # Between 1.5 and 3 seconds timed_wait( time_to_wait, () )</span> </pre> By using our clever timed_wait function instead of the built-in pygame.time.wait we gain some extra politeness: we can now quit the program on the "Ready?" screen by closing the window or pressing the Escape key. Try it!</p> Notice that we passed in () as the second argument to timed_wait. This argument is called event_types_that_cancel and is normally a list of types of event that will stop us waiting. () is Python's way of saying an empty list, so we're saying we don't want to stop for any normal events (such as key presses) - only for quit events, or when the time is up.</p> Before we change lots more code to use timed_wait, we are going to make a new variable that we can use in lots of places in the code. Quite a few times, we want to wait for either a key press or a mouse click. We want this when we're showing a red or green square, and when we're at the end saying goodbye, and ideally we also want it when we're telling the user how they did, so they can skip it if they're impatient. So, right near the top, add a new line just below where we create screen_size:</p> screen_width = 640 screen_height = 480 screen_size = screen_width, screen_height press_events = pygame.KEYDOWN, pygame.MOUSEBUTTONDOWN</span> </pre> This variable press_events will be our list of normal event types that we consider to be a "press" - essentially, the player doing something. Now that we've defined this at the top, we can take out the variable with the same name from shape_wait - it will use the global one instead:</p> def shape_wait(): """ Wait while we display a shape. Return True if a key was pressed, or false otherwise. """ return timed_wait( 2000, press_events ) # 2 seconds </pre> We can also re-use press_events in the end function, and at the same time call our new timed_wait function:</p> def end(): screen.fill( pygame.Color( "black" ) ) white = pygame.Color( "white" ) write_text( screen, "Thanks for playing!", white, True ) write_text( screen, "Press a key to exit", white, False ) pygame.display.flip() pygame.event.clear() timed_wait( 0, press_events )</span> </pre> Notice that this time we pass zero as the time to wait - this just means we will never time out on this screen - the zero gets passed in and used in the pygame.time.set_timer call, but passing in zero for the time there means "cancel this event", and is harmless if the event doesn't actually exist, so no timer will be set up - we will only stop waiting when the player presses something, which is what we want here.</p> Now we can make our success and failure functions more polite. Change them all to look like this:</p> def green_success(): tick() green = pygame.Color( "green" ) white = pygame.Color( "white" ) write_text( screen, "Well done!", green, True ) write_text( screen, "You pressed on green!", white, False ) pygame.display.flip() timed_wait( 2000, press_events ) # 2 seconds</span> def green_failure(): cross() red = pygame.Color( "red" ) white = pygame.Color( "white" ) write_text( screen, "Bad Luck!", red, True ) write_text( screen, "Green means press something!", white, False ) pygame.display.flip() timed_wait( 2000, press_events ) # 2 seconds</span> def red_success(): tick() green = pygame.Color( "green" ) white = pygame.Color( "white" ) write_text( screen, "Well done!", green, True ) write_text( screen, "You didn't press on red!", white, False ) pygame.display.flip() timed_wait( 2000, press_events ) # 2 seconds</span> def red_failure(): cross() red = pygame.Color( "red" ) white = pygame.Color( "white" ) write_text( screen, "Bad Luck!", red, True ) write_text( screen, "Red means don't press anything!", white, False ) pygame.display.flip() timed_wait( 2000, press_events ) # 2 seconds</span> </pre> They all call timed_wait saying wait for 2 seconds, but skip if a key is pressed because the player is impatient to get on to the next round. This change means not only can you skip past these success and failure screens, but also you can quit while they are visible, and the last vestige of rudeness has been wiped out from our game.</p> Well done - just one job left, which is to allow several rounds, and count the player's score as they play. We'll do that next time.</p> In the meantime, you can fix a bug I made - I typed get_width instead of get_height, which made my circles too big. Change the line inside green_shape that looks like radius = screen.get_width() / 3 to look like this:</p> radius = screen.get_height() / 3 </pre> There we are - much better</p> You can check your verson against mine here: redgreen.py</a></p> See you next time, when hopefully we'll finish the game!</p> Comments</h2> mason 2013-02-25</span> </h3> in part 12 mine wold not work so i copied and pasted yours but it did not work?</p> </div> Andy Balaam 2013-02-26</span> </h3> Hi mason, what error or wrong behaviour did you see?</p> </div> My First Raspberry Pi Game " Part 10 " Red square Andy Balaam — Fri, 14 Dec 2012 23:31:33 +0000 Parts: 1</a>, 2</a>, 3</a>, 4</a>, 5</a>, 6</a>, 7</a>, 8</a>, 9</a>, 10, 11</a>, 12</a>.</p> Writing your first ever computer program on the Raspberry Pi.</em></p> </iframe> We're writing a really simple game - you have to press a key when you see green, and not press a key when you see red.</p> I've been promising for a while that there will be a red square as well as a green circle, and this time we're going to make that dream a reality.</p> The code we've written so far has this overall structure:</p> start() ready_screen() wait() shape() end() </pre> It gets started, tells you to get ready, waits a random time, shows you a shape and collects your keypresses (or not), and then it ends.</p> Previously, the shape function just showed a green shape every time, by calling a function called green_shape. Not any more - change it to look like this:</p> def shape(): GREEN = 0 RED = 1 shape = random.choice( [GREEN, RED] ) if shape == GREEN: green_shape() else: red_shape() </pre> The first 2 lines just make two variables for us called GREEN and RED. They can have any values, so long as they're not the same, so I've chosen 0 and 1.</p> The reason we've made these variables is so that we can make a random choice of one or the other. To do this, we call the choice function from the random module (which we already have listed as imported at the top). choice takes in a list of things to choose from, and returns the one it chose randomly.</p> So the shape variable will contain the value of either GREEN or RED. We do an if to decide what to do based on which it is.</p> If we chose GREEN, we do what we used to do, and call green_shape but if we chose RED we will end up in the else part of the if, and call a new function we will call red_shape.</p> So, what will red_shape look like? Quite a lot like green_shape actually. Just above the shape function, add this:</p> def red_shape(): red = pygame.Color( "red" ) height = 2 * ( screen.get_height() / 3 ) left = ( screen.get_width() / 2 ) - ( height / 2 ) top = screen.get_height() / 6 screen.fill( pygame.Color( "white" ) ) pygame.draw.rect( screen, red, ( left, top, height, height ), 0 ) write_text( screen, "Don't press!", pygame.Color( "black" ), False ) pygame.display.flip() pressed = shape_wait() if pressed: red_failure() else: red_success() </pre> Most of this function is taken up with drawing a red rectangle, which we do by working out what size it should be, then calling pygame.draw.rect with the right dimensions and colour. After that we write some text encouraging the player to leave their keyboard alone, and then we do the normal pygame.display.flip to show this on the screen.</p> Once we've drawn the shape, we do something very similar to what we did inside green_shape - we wait to see what happens, by calling the already-existing shape_wait function, and get the answer back from it saying whether or not the player pressed something.</p> This time, if they pressed something they got it wrong, so we call a new function called red_failure, and if they did nothing they did the right thing, so we call another new function called red_success.</p> These two functions are also quite simple. Type them in above green_shape:</p> def red_success(): tick() green = pygame.Color( "green" ) white = pygame.Color( "white" ) write_text( screen, "Well done!", green, True ) write_text( screen, "You didn't press on red!", white, False ) pygame.display.flip() pygame.time.wait( 2000 ) # Can't quit or skip! def red_failure(): cross() red = pygame.Color( "red" ) white = pygame.Color( "white" ) write_text( screen, "Bad Luck!", red, True ) write_text( screen, "Red means don't press anything!", white, False ) pygame.display.flip() pygame.time.wait( 2000 ) # Can't quit or skip! </pre> These functions re-use lots of existing code - they draw a tick or a cross and then use write_text to tell the player what happened, and then they do the normal flip and wait for a bit.</p> Try your program - it should now show you a red square about half of the time, instead of a green circle every time, and it should give you feedback about whether you did the right or the wrong thing. Feel free to try it a few times, and make sure you've run through all the combinations. If you made a mistake somewhere you may not see it until you actually run the relevant bit of code.</p> Once you're happy with that, let's fix a little bug while we're here. Somehow I missed a bit from the shape_wait function, so if you press a key on the ready screen, it will register as you pressing the key really quickly when the shape appears. Try running your program and hammering a key when it says "Ready?". You'll see it thinks you pressed immediately the shape appears (whether red or green). This is annoying, and could even allow cheating, but we can prevent it by adding a single line to shape_wait:</p> def shape_wait(): """ Wait while we display a shape. Return True if a key was pressed, or false otherwise. """ event_types_that_cancel = pygame.KEYDOWN, pygame.MOUSEBUTTONDOWN time_to_wait = 2000 # Display the shape for 2 seconds finished_waiting_event_id = pygame.USEREVENT + 1 pygame.time.set_timer( finished_waiting_event_id, time_to_wait ) pygame.event.clear()</span> pressed = False waiting = True while waiting: evt = pygame.event.wait() if evt.type == pygame.QUIT: quit() elif evt.type in event_types_that_cancel: waiting = False pressed = True elif evt.type == finished_waiting_event_id: waiting = False pygame.time.set_timer( finished_waiting_event_id, 0 ) return pressed </pre> As we've seen before, pygame.event.clear tells PyGame to forget all the events that have happened recently, which prevents this problem.</p> If you want to check your version against mine, you can find it here: redgreen.py</a>.</p> We've nearly built a fully-working game. We've got two main tasks ahead of us: fix the "can't exit" bug, and allow multiple rounds with a score at the end. We'll do them in that order, so next time it's bug-fixing, and some more refactoring to help us do it.</p> Comments</h2> lemuel akinsete 2018-04-09</span> </h3> how do you make music and videos on raspberry pi</p> </div> Axel 2014-06-24</span> </h3> Could you post the final program at the end. I am having trouble finding out what I did wrong (whenever I try running this program it says, "./redgreen.py File./redgreen.py</em>. line 127 write_text( screen, "Don’t"t Press!", pygame.Color( "black" ), False )</p> SyntaxError: invalid syntax"</p> but I can’t find out what is wrong with my script line. That line looks the same as the one you wrote.</p> </div> Axel 2014-06-24</span> </h3> The file says there is a problem with the word Text</p> </div> Andy Balaam 2014-06-25</span> </h3> Hi Axel, the full program listing at the end of the last video looks like: https://raw.githubusercontent.com/andybalaam/...</a>. There is a link "redgreen.py" near the bottom of the blog post that shows what it should look like after this video.</p> </div> Andy Balaam 2014-06-25</span> </h3> Axel, looking at your error message, it might be that you have an extra quote after the t of Don’t.</p> </div> My First Raspberry Pi Game " Part 09 " Lots more words Andy Balaam — Sat, 08 Dec 2012 08:48:12 +0000 Parts: 1</a>, 2</a>, 3</a>, 4</a>, 5</a>, 6</a>, 7</a>, 8</a>, 9, 10</a>, 11</a>, 12</a>.</p> Writing your first ever computer program on the Raspberry Pi.</em></p> </iframe> We're writing a really simple game - you have to press a key when you see green.</p> This time we're going to add lots of instructions on each page so the player knows what to do.</p> Up until now, we've only had one piece of writing - the word "Ready?" when the game starts up. To make the game easy to use we want to write instructions on each page. We'll take the code we wrote to say "Ready?" and turn it into a function that we can re-use for lots of different writing.</p> The first step we're going to do will be "refactoring", which just means changing our program without changing what it does.</p> Have a look at this code, that we've already got:</p> def start(): global screen, ready_text pygame.init() screen = pygame.display.set_mode( screen_size ) font = pygame.font.Font( None, screen_height / 5 ) ready_text = font.render( "Ready?", 1, pygame.Color( "white" ) )</span> def quit(): pygame.quit() sys.exit() def ready_screen(): textpos = ready_text.get_rect( centerx = screen.get_width() / 2, centery = screen.get_height() / 2 ) screen.blit( ready_text, textpos )</span> pygame.display.flip() </pre> The blue code here works together to write something on the screen. The top part inside start creates an image called ready_text, and the bottom part in ready_screen works out where to draw it, then draws it.</p> We're going to extract these two bits out from where they are, and put them into a function we can re-use whenever we want to write something.</p> Make a function above start and call it write_text. Cut the lines in blue, and paste them into write_text. Rename the variable ready_text to rend (in three places) because it's a rendered image of our writing. You should end up with something like this:</p> def write_text(): font = pygame.font.Font( None, screen_height / 5 ) rend = font.render( "Ready?", 1, pygame.Color( "white" ) ) textpos = rend.get_rect( centerx = screen.get_width() / 2, centery = screen.get_height() / 2 ) screen.blit( rend, textpos )</span> </pre> Now make the code work almost exactly as before by making a call to this function from within ready_screen:</p> def ready_screen(): write_text()</span> pygame.display.flip() </pre> You can try your program now and you should find it works exactly as it did before.</p> So far we haven't make a very useful function, because it always writes "Ready?". We can change that by adding arguments to the write_text function for the screen to write on, the text to write, and the colour of the writing. Now write_text looks like this:</p> def write_text( screen, text, color</span> ): font = pygame.font.Font( None, screen.get_height() / 5 ) rend = font.render( text</span>, 1, color</span> ) pos = rend.get_rect( centerx = screen.get_width() / 2, centery = screen.get_height() / 2 ) screen.blit( rend, pos ) </pre> and ready_screen looks like this:</p> def ready_screen(): white = pygame.Color( "white" )</span> write_text( screen, "Ready?", white</span> ) pygame.display.flip() </pre> (Notice that I made a variable called white and then passed it in.)</p> At this point, you can also delete the line above write_text that says ready_text = None because we don't need that variable any more (we are just using the variable rend inside write_text) and you can remove ready_text from the first line of the start function, so it just reads global screen.</p> Again, you can try your program now and you should find it works exactly as it did before.</p> By working in gradual steps like this, we give ourselves small enough chunks of things to think about that we're relatively unlikely to get confused and screw things up. As you work with longer programs I hope you'll see how useful this technique can be.</p> The next step we want to take is to allow two different types of writing: either in the middle, big or at the bottom, smaller. Let's add another argument, called big and make it control how big the writing is, and where it goes.</p> def write_text( screen, text, color, big</span> ): if big: height = screen.get_height() / 5 up = screen.get_height() / 2</span> font = pygame.font.Font( None, height</span> ) rend = font.render( text, 1, color ) pos = rend.get_rect( centerx = screen.get_width() / 2, centery = up</span> ) screen.blit( rend, up ) </pre> And let's pass in True so we behave just as we did before:</p> def ready_screen(): white = pygame.Color( "white" ) write_text( screen, "Ready?", white, True</span> ) pygame.display.flip() </pre> So far we've only said what happens when we pass True for big - if we pass False, height and pos will not be set, and the program will go wrong. Let's fix that now:</p> def write_text( screen, text, color, big</span> ): if big: height = screen.get_height() / 5 up = screen.get_height() / 2 else: height = screen_height / 12 up = screen.get_height() - ( screen_height / 24 )</span> font = pygame.font.Font( None, height ) rend = font.render( text, 1, color ) pos = rend.get_rect( centerx = screen.get_width() / 2, centery = up ) screen.blit( rend, pos ) </pre> So now we can write small text at the bottom of the sreen. Let's do that on the green_shape page, so the player knows to press a key. Change green_shape to look like:</p> def green_shape(): green = pygame.Color( "green" ) centre = ( screen.get_width() / 2, screen.get_height() / 2 ) radius = screen.get_width() / 3 screen.fill( pygame.Color( "white" ) ) pygame.draw.circle( screen, green, centre, radius, 0 ) write_text( screen, "Press something!", pygame.Color( "black" ), False )</span> pygame.display.flip() pressed = shape_wait() if pressed: green_success() else: green_failure() </pre> Now we can add some more text to green_success and green_failure saying well done when you won and bad luck when you lost:</p> def green_success(): tick() green = pygame.Color( "green" ) white = pygame.Color( "white" ) write_text( screen, "Well done!", green, True ) write_text( screen, "You pressed on green!", white, False ) pygame.display.flip()</span> pygame.time.wait( 2000 ) # Can't quit or skip! def green_failure(): cross() red = pygame.Color( "red" ) white = pygame.Color( "white" ) write_text( screen, "Bad Luck!", red, True ) write_text( screen, "Green means press something!", white, False ) pygame.display.flip()</span> pygame.time.wait( 2000 ) # Can't quit or skip! </pre> Notice that we have added pygame.display.flip() to the end of each of these sections, so we can remove that line from the end of tick and cross. Also, we need to move our tick and cross pictures up so they're not on top of the writing, so we'll divide by 4 instead of 2 to get the middle of the picture:</p> def tick(): colour = pygame.Color( "green" ) w = screen.get_width() / 2 h = screen.get_height() / 4</span> points = ( ( w - w/5, h - h/9 ), ( w, h + h/5 ), ( w + w/3, h - h/3 ), ) screen.fill( pygame.Color( "black" ) ) pygame.draw.lines( screen, colour, False, points, 20 ) pygame.display.flip()</span> def cross(): colour = pygame.Color( "red" ) w = screen.get_width() / 2 h = screen.get_height() / 4</span> left = w - w/3 right = w + w/3 top = h - h/3 bottom = h + h/3 start1 = left, top end1 = right, bottom start2 = left, bottom end2 = right, top screen.fill( pygame.Color( "black" ) ) pygame.draw.line( screen, colour, start1, end1, 20 ) pygame.draw.line( screen, colour, start2, end2, 20 ) pygame.display.flip()</span> </pre> And finally, let's say goodbye at the end. Change end to look like this:</p> def end(): screen.fill( pygame.Color( "black" ) ) white = pygame.Color( "white" ) write_text( screen, "Thanks for playing!", white, True ) write_text( screen, "Press a key to exit", white, False ) pygame.display.flip()</span> pygame.event.clear() event_types_that_cancel = pygame.KEYDOWN, pygame.MOUSEBUTTONDOWN waiting = True while waiting: evt = pygame.event.wait() if evt.type == pygame.QUIT: quit() elif evt.type in event_types_that_cancel: waiting = False </pre> Now we've got much more enlightened players of our game!</p> We've made lots of changes in lots of different places today. If something doesn't work, check your version again mine: redgreen.py</a>.</p> Next time we'll show red squares as well as green circles, making the game significantly more interesting.</p> My First Raspberry Pi Game " Part 08 " Success and failure Andy Balaam — Tue, 04 Dec 2012 08:29:19 +0000 Parts: 1</a>, 2</a>, 3</a>, 4</a>, 5</a>, 6</a>, 7</a>, 8, 9</a>, 10</a>, 11</a>, 12</a>.</p> Writing your first ever computer program on the Raspberry Pi.</em></p> </iframe> We're writing a really simple game - you have to press a key when you see green.</p> Today we're going to wait for a key press. If we get one, we'll tell the player they did well. If not, we'll tell them they are a bad person.</p> We're going to change the green_shape function first, to make it wait for a key press (or give up waiting) and then tell the player what happened.</p> Find the green_shape function and add the new bit that I've highlighted in green, at the end:</p> def green_shape(): green = pygame.Color( "green" ) centre = ( screen.get_width() / 2, screen.get_height() / 2 ) radius = screen.get_width() / 3 screen.fill( pygame.Color( "white" ) ) pygame.draw.circle( screen, green, centre, radius, 0 ) pygame.display.flip() pressed = shape_wait() if pressed: green_success() else: green_failure()</span> </pre> green_shape is the function that shows a green shape to the player.</p> This new code does 2 things. First, it calls a function shape_wait (that we haven't written yet) that waits for a key press. We are expecting this function to give us back an answer, which we will store inside a new variable, pressed.</p> Second, it checks the value of pressed, and calls a different function in each case. If a key was pressed, this is good (because we're showing a green shape, so you're supposed to press a key) so we call the green_success function (which we haven't written yet either). If no key was pressed because we gave up waiting, we call the green_failure function (which we haven't written yet!).</p> That covers everything we want to do today - all we have to do is write those 3 missing functions.</p> Let's start with the hardest one - shape_wait. Go up to just above the green_shape function, and type this:</p> def shape_wait(): """ Wait while we display a shape. Return True if a key was pressed, or false otherwise. """ event_types_that_cancel = pygame.KEYDOWN, pygame.MOUSEBUTTONDOWN time_to_wait = 2000 # Display the shape for 2 seconds finished_waiting_event_id = pygame.USEREVENT + 1 pygame.time.set_timer( finished_waiting_event_id, time_to_wait ) </pre> There are a few things to explain here. First, the writing at the top just below the def line. This is the way we explain in Python what a function does and what it's for. It's optional, and we haven't done it before, but I thought this function was interesting enough for us to provide some explanation. Notice the triple-quotes """ at the beginning and end. That is a way Python allows us to write longer strings of text that cover more than one line. The string starts at the first triple-quote, and ends at the last.</p> After our documentation string, we create a familiar variable event_types_that_cancel that holds on to all the types of event we are interested in - key presses and mouse clicks. Next we remember how long we are going to wait before giving up in another variable time_to_wait.</p> After that we do something a bit interesting. Up to now we have been dealing with "events" - things that happen such as mouse clicks, key presses and mouse movements, but we have only been responding to them, not creating them. The next 2 lines are how we create our own event, that we want to respond to later.</p> What we want to do is make an event happen in 2 seconds' time, so that we can give up waiting when it comes. The way we do that is first create an "ID" for it. This is just a numeric "name" that we can use to talk about the same type of event later. In PyGame the right ID to choose for an event you created yourself is pygame.USEREVENT + 1 (and higher numbers if you need more than one). We don't know what number PyGame has stored inside its own variable pygame.USEREVENT, and we don't care - all we care about is that PyGame says if we use numbers bigger than that, we'll be fine. If we use smaller numbers, we are going to clash with the built-in events like pygame.KEYDOWN that we have already seen.</p> Once we have an appropriate ID stored inside finished_waiting_event_id we are ready to ask PyGame to create an event that will happen in 2 seconds' time. We do that by calling the set_timer function inside pygame.time.</p> Now continue the function by typing all this:</p> pressed = False waiting = True while waiting: evt = pygame.event.wait() if evt.type == pygame.QUIT: quit() elif evt.type in event_types_that_cancel: waiting = False pressed = True elif evt.type == finished_waiting_event_id: waiting = False pygame.time.set_timer( finished_waiting_event_id, 0 ) return pressed </pre> This is the code that waits for something to happen. It's quite similar to the loop we saw in part 6</a>, where we were also waiting for something to happen, but it's slightly more complicated because we have to handle more possibilities.</p> This function will provide an answer to the code that called it, and the answer is going to be whether or not the player pressed a key. Providing an answer like this is called "returning a value" and we do it by writing a line like the last one here, using the return statement. The first line above creates a variable called pressed, which starts off set to False, meaning they haven't pressed anything, and somewhere in between it might get set to True, and then the last line returns this answer - True or False for whether or not a key was pressed.</p> In between we have a loop similar to part 6 - we create a variable called waiting which tells us whether to keep looping, and then we loop using the while line through all the lines indented below it. The inside of the loop (the part that gets repeated) waits for an event to happen with pygame.event.wait and then has a series of if and elif sections, that do different things depending what type of event happened.</p> First (the if part), we check whether the player closed the window. If so, we call our function quit, that stops everything immediately.</p> Next (the first elif), we check whether a key or mouse button was pressed. If so, we make sure the value we will return inside pressed is updated to say a key was pressed (i.e. we make it True), and then we set waiting to False so that we will stop looping at the end of this repeat.</p> Now (the second elif), we check whether what happened was the special event we created earlier when we called set_timer. If so, we need to end the loop (so we set waiting to False), but no key was pressed, so we leave pressed as it was.</p> Finally, if the event that happened didn't fit any of our categories (for example it might have been a mouse movement event), we do absolutely nothing because none of the if or elif sections was triggered. We jump straight back to the start of the loop and start waiting for the next event to happen.</p> So, eventually, either an interesting event happens, or the "we've been waiting too long" event we created happens, and we come out of the while loop. The last thing we have to do is cancel the "we've been waiting too long" event, just in case it hasn't happened yet - we don't want it confusing us later. We do that by calling set_timer again, with the same ID as before, but with 0 for the amount of time to wait - this tells PyGame we're not interested in that event any more.</p> Once we've done that we return the answer about whether a key was pressed, and we're done with shape_wait.</p> Next up are green_success and green_failure. These tell the player whether they succeeded or failed - did they manage to press when they saw green?</p> They're both quite simple. Type these just above green_shape:</p> def green_success(): tick() pygame.time.wait( 2000 ) # Can't quit or skip! def green_failure(): cross() pygame.time.wait( 2000 ) # Can't quit or skip! </pre> If a key was pressed on green, we want to draw a "tick" mark on the screen, so we call a function tick that we'll write in a moment. Similarly, if a key wasn't pressed, we will draw a cross.</p> Drawing shapes is fairly straightforward, but a bit verbose. Just above green_success type these 2 functions:</p> def tick(): colour = pygame.Color( "green" ) w = screen.get_width() / 2 h = screen.get_height() / 2 points = ( ( w - w/5, h - h/9 ), ( w, h + h/5 ), ( w + w/3, h - h/3 ), ) screen.fill( pygame.Color( "black" ) ) pygame.draw.lines( screen, colour, False, points, 20 ) pygame.display.flip() def cross(): colour = pygame.Color( "red" ) w = screen.get_width() / 2 h = screen.get_height() / 2 left = w - w/3 right = w + w/3 top = h - h/3 bottom = h + h/3 start1 = left, top end1 = right, bottom start2 = left, bottom end2 = right, top screen.fill( pygame.Color( "black" ) ) pygame.draw.line( screen, colour, start1, end1, 20 ) pygame.draw.line( screen, colour, start2, end2, 20 ) pygame.display.flip() </pre> Both of these functions get some variables ready, do some maths on them to decide where on the screen to start and end the lines they are drawing, and then draw the lines (after making a black background with screen.fill).</p> The tick is drawn by passing in a list of 3 points on the screen to the pygame.draw.lines function, and the cross is drawn using two separate calls to pygame.draw.line, one for each line. After we've drawn our lines, we call pygame.display.flip as normal to show them on the screen.</p> With those two functions in place, we're ready to try it out. Open LXTerminal in the usual way, and type our usual incantation:</p> ./redgreen.py </pre> If all has gone well, you should see the green shape as before, but when you press a key a tick should appear. If you don't press a key, after a while a red cross should appear.</p> If that doesn't happen, check your typing really carefully, and compare your version with mine: redgreen.py</a>.</p> Next time, we'll add some writing explaining what you should do at each step.</p> My First Raspberry Pi Game " Part 07 " A green circle Andy Balaam — Wed, 28 Nov 2012 09:34:37 +0000 Parts: 1</a>, 2</a>, 3</a>, 4</a>, 5</a>, 6</a>, 7, 8</a>, 9</a>, 10</a>, 11</a>, 12</a>.</p> Writing your first ever computer program on the Raspberry Pi.</em></p> </iframe> We're going to write a game that tests your reactions - press a key when you see green, but don't when you see red.</p> Today we see some of what we have been waiting for</a> - a genuine bona-fide green circle, made by you!</p> We're going to need some random numbers, so edit your program in LeafPad, and add a line, just before import pygame near the top:</p> import random </pre> This makes the "random" module available to us so we can make some numbers later.</p> Remember we had a function called "wait", but it never did anything? It was supposed to wait for a random amount of time before we showed our green or red shape. Let's write it now. Find the empty wait function and replace it with:</p> def wait(): time_to_wait = random.randint( 1500, 3000 ) # Between 1.5 and 3 seconds pygame.time.wait( time_to_wait ) # Note bug: can't quit during this time </pre> The first line makes a variable time_to_wait and puts a random number into it. The random.randint function gives us a random number between the two numbers we supplied, so here between 1,500 and 3,000. time_to_wait is a time in milliseconds, so this means between one and half and three seconds.</p> After the closing bracket, we have a hash symbol #, and then some writing. This is a "comment", and it is completely ignored by Python. It's just for us.</p> [As time goes on, I hope you will begin to see programming more and more as talking to other people, not just to the computer. It's fairly easy to write a computer program, but much harder to understand one written by someone else. Most programs live a long time, and people need to understand them to keep them up-to-date, so making them as easy to understand as possible is very important. Comments are one way to help people understand, but in a way they are a last resort - if possible, the code itself should be so easy to understand that you don't need many comments. Here, I thought that the translation between seconds and milliseconds might be helpful to someone looking at this later.]</p> The next line uses a function inside PyGame's time module to wait for the amount of time we give it (in milliseconds, stored in time_to_wait). Note that this is not the same wait function we have seen before, pygame.event.wait. That one waits forever for an event to happen, but this one waits (and can't be interrupted) for the amount of time we say.</p> I've added another comment to this line saying that there's a bug in our program: if we write it like this, you can't actually quit the game by closing the window while we're waiting. The pygame.time.wait function won't be interrupted by the window being closed, so we'll ignore it. This is almost unbearably rude, but don't worry - we'll fix it soon (ish).</p> And now for the really exciting part: we're going to draw a green shape on the screen. Let's make a function, just above the shape function, called green_shape:</p> def green_shape(): green = pygame.Color( "green" ) centre = ( screen.get_width() / 2, screen.get_height() / 2 ) radius = screen.get_width() / 3 screen.fill( pygame.Color( "white" ) ) pygame.draw.circle( screen, green, centre, radius, 0 ) pygame.display.flip() </pre> This code makes a variable green holding onto the colour green, one called centre holding the co-ordinates of the centre of the screen, and one called radius holding the size of the circle we want to draw.</p> Then it uses the fill function on screen to colour in the screen white, and then draws our circle with a call to pygame.draw.circle, using the variables we have prepared as arguments, telling it where to draw the circle, in what colour, and what size.</p> Finally it uses flip as before to tell PyGame we have finished.</p> The last piece of today's jigsaw is just to call the function we created above. Find the empty shape function, and make it look like this:</p> def shape(): green_shape() </pre> This literally just means run the green_shape function.</p> Take a deep breath, prepare to be excited, open LXTerminal and run our new program in the usual way:</p> ./redgreen.py </pre> If all has gone well, the ready screen will appear for a couple of seconds, before a white screen with a big green circle on it appears. This will then go away when you press a key.</p> If something goes wrong, check back what you typed, and compare your version against mine: redgreen.py</a>.</p> Next time, we'll find out whether you pressed a key or were too slow!</p> Launching an OSGi app on the command line Andy Balaam — Tue, 27 Nov 2012 10:29:54 +0000 I am a total beginner on this topic, but here is what I have found so far. Please correct me, advise me etc. in the comments.</p> A colleague has written a program based on OSGi</a>, which is a framework for obfuscating everything</del> "modular" software, which I think means loading code at runtime, even if it requires a different classloader, and even if it depends on other things which also need loading at runtime. I may be wrong though.</p> Anyway, all I want to do is run it (on windows only, for now).</p> OSGi is quite closely linked to the Eclipse</a> development environment - Eclipse uses OSGi to manage its plugins, which are an infinitely complex web of incompatible interdependencies.</p> My colleague showed me how to set up their Eclipse environment, and how to run the program from within Eclipse using an OSGi launch configuration. I also managed to build the JAR files of the program itself.</p> What I wanted to do was run it outside Eclipse, on the command line (and eventually in an Ant</a> build).</p> I found it almost impossible to understand anything others have written about OSGi, but somehow I muddled my way through to being able to do it.</p> Here's how I managed it.</p> Find out the dependencies</h3> If you have an Eclipse launch configuration that successfully runs your program inside Eclipse, you can find out all the OSGi bundles (which are JAR files containing extra config) that it needs. Go to the Run menu and click Run Configurations. Find the relevant run configuration, which will be inside the OSGi Framework tree item on the left. Click on it, and choose the Bundles tab. The tree view will show you all the bundles needed to run this program. If you check "Only show selected" you can see only the ones you need.</p> If you want a text version of this list, look for a file here:</p> WORKSPACE/.metadata/.plugins/org.eclipse.pde.core/LAUNCHERNAME/org.eclipse.simpleconfigurator/bundles.info</pre> where WORKSPACE is the location of your Eclipse workspace, and LAUNCHERNAME is the name of the run configuration you found in the tree view in the Run Configurations page.</p> The bundles.info file contains a comma-separated list of bundle name (which is a reversed-url type name, like a package name - don't confuse it with a package name though), some version information, and the path to the actual JAR file that is this bundle.</p> We will also need the Equinox</a> OSGi bundles, including the SimpleConfigurator (oh, the irony). I am using these versions - yours might be different:</p> org.eclipse.equinox.launcher.win32.win32.x86_1.1.2.R36x_v20101222/* org.eclipse.equinox.app_1.3.1.R36x_v20100803.jar org.eclipse.equinox.common_3.6.0.v20100503.jar org.eclipse.equinox.concurrent_1.0.100.v20100503.jar org.eclipse.equinox.ds_1.2.1.R36x_v20100803.jar org.eclipse.equinox.event_1.2.0.v20100503.jar org.eclipse.equinox.frameworkadmin.equinox_1.0.200.v20100505.jar org.eclipse.equinox.frameworkadmin_2.0.0.v20100503.jar org.eclipse.equinox.http.jetty_2.0.0.v20100503.jar org.eclipse.equinox.http.registry_1.1.1.R36x_v20101103.jar org.eclipse.equinox.http.servlet_1.1.0.v20100503.jar org.eclipse.equinox.launcher_1.1.1.R36x_v20101122_1400.jar org.eclipse.equinox.p2.artifact.repository_1.1.1.R36x_v20100901.jar org.eclipse.equinox.p2.console_1.0.200.v20100601.jar org.eclipse.equinox.p2.core_2.0.3.R36x_v20110111.jar org.eclipse.equinox.p2.director.app_1.0.202.R36x_v20110608.jar org.eclipse.equinox.p2.directorywatcher_1.0.203.R36x_v20101220.jar org.eclipse.equinox.p2.director_2.0.4.R36x_v20120223.jar org.eclipse.equinox.p2.engine_2.0.1.R36x_v20110201.jar org.eclipse.equinox.p2.extensionlocation_1.2.0.v20100518.jar org.eclipse.equinox.p2.garbagecollector_1.0.100.v20100503.jar org.eclipse.equinox.p2.jarprocessor_1.0.200.v20100503a.jar org.eclipse.equinox.p2.metadata.repository_1.1.1.R36x_v20110302.jar org.eclipse.equinox.p2.metadata_2.0.1.R36x_v20101202.jar org.eclipse.equinox.p2.operations_2.0.0.v20100510.jar org.eclipse.equinox.p2.publisher_1.1.2.v20100824-2220.jar org.eclipse.equinox.p2.ql_2.0.0.v20100503a.jar org.eclipse.equinox.p2.reconciler.dropins_1.1.3.R36x_v20111213.jar org.eclipse.equinox.p2.repository.tools_2.0.1.R36x_v20100823.jar org.eclipse.equinox.p2.repository_2.0.2.R36x_v20110111-1500.jar org.eclipse.equinox.p2.touchpoint.eclipse_2.0.4.R36x_v20110908.jar org.eclipse.equinox.p2.touchpoint.natives_1.0.201.R36x_v20110111.jar org.eclipse.equinox.p2.ui.sdk.scheduler_1.0.0.v20100507-1815.jar org.eclipse.equinox.p2.ui.sdk_1.0.100.v20100513.jar org.eclipse.equinox.p2.ui_2.0.0.v20100518.jar org.eclipse.equinox.p2.updatechecker_1.1.101.R36x_v20100823.jar org.eclipse.equinox.p2.updatesite_1.0.201.R36x_v20100823.jar org.eclipse.equinox.preferences_3.3.1.R36x_v20110302.jar org.eclipse.equinox.registry_3.5.0.v20100503.jar org.eclipse.equinox.security.ui_1.0.200.v20100503.jar org.eclipse.equinox.security.win32.x86_1.0.200.v20100503.jar org.eclipse.equinox.security_1.0.200.v20100503.jar org.eclipse.equinox.simpleconfigurator.manipulator_2.0.0.v20100503.jar org.eclipse.equinox.simpleconfigurator_1.0.200.v20100503.jar org.eclipse.equinox.util_1.0.200.v20100503.jar org.eclipse.osgi.services_3.2.100.v20100503.jar org.eclipse.osgi.util_3.2.100.v20100503.jar org.eclipse.osgi_3.6.50.R36x_v20120315-1500.jar </pre> (I have no idea whether you need all of these - sorry.)</p> Make a fake environment</h3> Imagine you have a program that consists of 2 JARs itself, called my_program_main_1.0.jar and my_program_lib_1.1.jar, 2 dependency bundles called com.dep1_3.6v20091112.jar and org.dep2_1.0.jar and a directory containing a dependency bundle called org.foo.dep3_1.4.</p> Create a directory that will contain your program and its dependencies. Structure it like this:</p> main_dir/ my_program/ my_program_main_1.0.jar my_program_lib_1.1.jar osgi_install_area/ plugins/ com.dep1_3.6v20091112.jar org.dep2_1.0.jar org.foo.dep3_1.4/ ... more stuff in here ... org.eclipse.equinox.launcher.win32.win32.x86_1.1.2.R36x_v20101222/ ... more stuff in here ... org.eclipse.equinox.app_1.3.1.R36x_v20100803.jar ... lots more here ... configuration/ config.ini bundles.info </pre> Copy your program's JARs into the my_program directory (feel free to give it a different name!).</p> Copy the Equinox bundles and the other dependencies into the plugins directory. You can find them in the directory listed in the bundles.info file you found in the previous section. The Equinox bundles should be alongside the others (hopefully).</p> The contents of bundles.info should be something like this:</p> #version=1 MyProgramMain,1.0,file:../my_program/my_program_main_1.0.jar,4,true MyProgramLib,1.1,file:../my_program/my_program_lib_1.1.jar,4,true com.dep1,3.6v20091112,file:./plugins/com.dep1_3.6v20091112.jar,4,true org.dep2,1.0,file:./plugins/org.dep2_1.0.jar,4,true org.foo.dep3,1.4,file:./plugins/org.foo.dep3_1.4/,4,true org.eclipse.equinox.launcher.win32.win32.x86,1.1.2.R36x_v20101222,file:./plugins/org.eclipse.equinox.launcher.win32.win32.x86_1.1.2.R36x_v20101222/,4,true org.eclipse.equinox.app,1.3.1.R36x_v20100803,org.eclipse.equinox.app_1.3.1.R36x_v20100803.jar,4,true ... Plus similar lines for all the equinox and OSGi bundles you need ... </pre> Make sure that the version number after the first comma exactly matches the version number encoded in the JAR filename.</p> The contents of config.ini should be something like this:</p> osgi.bundles=reference\:file\:C\:/absolute/path/to/main_dir/osgi_install_area/plugins/org.eclipse.equinox.simpleconfigurator_1.0.200.v20100503.jar@1\:start org.eclipse.update.reconcile=false osgi.bundles.defaultStartLevel=4 osgi.install.area=file\:C\:/absolute/path/to/main_dir/osgi_install_area osgi.framework=file\:C\:/absolute/path/to/main_dir/osgi_install_area/plugins/org.eclipse.osgi_3.6.50.R36x_v20120315-1500.jar org.eclipse.equinox.simpleconfigurator.configUrl=file\:/C\:/absolute/path/to/main_dir/bundles.info osgi.configuration.cascaded=false </pre> Note that these paths have to be absolute. I plan to write a small ant script to create the config.ini file when needed.</p> Run the program</h3> Now, to run your program, it's a simple matter of getting into the right directory:</p> cd "C:\absolute\path\to\main_dir"</pre> and typing this simple command:</p> java -Declipse.ignoreApp=true -Dosgi.noShutdown=true -Dfile.encoding=Cp1252 -classpath "osgi_install_area\plugins\org.eclipse.equinox.launcher_1.1.1.R36x_v20101122_1400.jar" org.eclipse.equinox.launcher.Main -configuration "file:configuration/" -os win32 -ws win32 -arch x86 -nl en_GB -console ACTUAL_ARGS_TO_PROGRAM </pre> Where ACTUAL_ARGS_TO_PROGRAM is where you type the actual command-line arguments (if any) you want to pass to your program.</p> Disclaimer</h3> I have no idea what most of this means, and I've probably told you a lot of things that are wrong. Please, people who know about this stuff, correct me and I'll update this post.</p> This worked for me, but I feel like I must be doing something wrong - can it really be this difficult?</p> Comments</h2> Teu 2016-12-22</span> </h3> Hi there, simple question (hooping you still read this old post)</p> Where is the point where you tell which application framwork must run? i suppose is main class in you MyProgramMain.jar ..where ACTUAL_ARGS_TO_PROGRAM goes.</p> Becaouse in you command line there is not reference to you MyProgramMain, in bundles.info line with MyProgramMain.jar seems similar to others line and in config.ini i don’t see anything can help.</p> I try to replicate but nothing invoke my main class.</p> </div> Andy Balaam 2016-12-23</span> </h3> Hi Teu, I don’t think OSGi uses Main classes or methods but to be honest I found it totally confusing and am very glad I never have to touch it again. Good luck, and sorry I can’t help!</p> </div> Neil Bartlett 2014-12-14</span> </h3> You can run an OSGi framework with just "java -jar org.eclipse.osgi_.jar".</p> The problems you have encountered seem related to the vast number of bundles you have added to the framework. You admit yourself that you don’t know what they do, so why are you bothering with them?</p> </div> Andy Balaam 2014-12-22</span> </h3> Hi Neil, thanks for the information. The bundles were needed by the code I was running, which I didn’t write. The main problem I was having was constructing a directory structure containing all the right stuff in order to be able to run the program, rather than the actual command line needed to run it (although that was a challenge too). I’d appreciate any tips on how to "build" something running inside Eclipse into a "executable" project (i.e. a file or set of files that can be run using the command you gave or a similar one).</p> </div> My First Raspberry Pi Game " Part 06 " A better class of waiting Andy Balaam — Sat, 24 Nov 2012 08:44:10 +0000 Parts: 1</a>, 2</a>, 3</a>, 4</a>, 5</a>, 6, 7</a>, 8</a>, 9</a>, 10</a>, 11</a>, 12</a>.</p> Writing your first ever computer program on the Raspberry Pi.</em></p> </iframe> Today we wait, but better, while we show the ready screen</a> we made last time. In the process we cover two of the most important things you need to understand to write computer programs.</p> The program we've written so far can wait around for something to happen, but it can't distinguish between the various possible "somethings" that might happen. Specifically, it finishes very abruptly if we even move our mouse over the window!</p> In this part we'll teach our program to ignore the movements of the mouse, but keep looking out for events it is interested in. We'll also make it able to exit quickly if someone closes its window.</p> First, we need to get hold of the sys object (actually it's called a "module") which allows us to do things with the system, like quit, which is what we're going to do with it.</p> Open up LeadPad again, and immediately above the import pygame line, add this:</p> import sys </pre> Now we're going to write a new function that allows us to exit the game immediately, whenever we want to. Just above the def ready_screen() line, add this new function:</p> def quit(): pygame.quit() sys.exit() </pre> This function tells PyGame we are finishing (by calling pygame.quit), then exits: sys.exit() stops everything dead.</p> Finally, we make our waiting a bit posher, by editing the end function look like this:</p> def end(): pygame.event.clear() event_types_that_cancel = pygame.KEYDOWN, pygame.MOUSEBUTTONDOWN waiting = True while waiting: evt = pygame.event.wait() if evt.type == pygame.QUIT: quit() elif evt.type in event_types_that_cancel: waiting = False </pre> There's quite a lot here, but we'll go through it line by line.</p> First, we clear the events as before using pygame.event.clear to make sure anything that has already happened doesn't interfere with us.</p> Next we make a variable called event_types_that_cancel that is a list of all the types of events that we care about. They are key presses (KEYDOWN), and mouse clicks (MOUSEBUTTONDOWN). We'll use this variable later.</p> Now we make another variable, called waiting. We use this to decide when we've finished. We set it to True, which means we are waiting, i.e. we haven't finished.</p> The next line while waiting is the start of a loop, which means a piece of code we run lots of times. A "while" loop keeps on running until something happens. In this case, the lines after while waiting: will run again and again until the waiting variable becomes False.</p> The rest of the lines are all "inside" the loop, meaning they run again and again until the loop finishes. Notice that they are indented by a further 4 spaces to show that they are inside. If we had more lines later that were not indented, they would run after the loop had finished.</p> The first line creates a variable called evt that holds on to what came back when we called the function pygame.event.wait(). This function waits until an event happens. When an event happens, it could be a mouse movement, or a key press, or one of several other types of events. The old version of this code just called pygame.event.wait(), but didn't look at what type of event happened. That's why it finished even when you moved the mouse over the window. This new version captures the event that happened in the evt variable, and then does different things depending on what type of event it was.</p> The line starting with if lets us decide what to do based on what is happening. The first decision we make is evt.type == pygame.QUIT, which means if the type of event we have found is a special QUIT event. This type of event happens if the user closes the window by clicking the "X" in the top right-hand corner. If this happened, we go on to the next line, which calls the quit() function we created earlier. If this didn't happen, we skip that line, and go on to the next one that is only indented as much as the original if line.</p> The next line after we skip is the one starting with elif. The "condition" we are checking (decision we are making) is whether evt.type in event_types_that_cancel which means is the event we have seen one of the ones we are interested it - is its type one of the things in event_types_that_cancel. event_types_that_cancel is a list containing the event type for mouse clicks, and the one for key presses, so if something interesting happened we will go to the next line, which is waiting = False. This sets waiting to False, meaning we will stop the loop. If the event was a different type (e.g. a mouse movement), we will skip the waiting = False line as well.</p> If neither of the conditions was true (so the event was not the user closing the window, or pressing a key or clicking the mouse) then we do nothing in the if and elif bit, and because we're in a loop, we go back to the beginning and wait() again for another event. This goes on forever until one of the events we're interested in happens. When it does, we either quit() immediately, or stop looping because waiting becomes False (meaning the while line lets us escape). When we've stopped looping we carry on, leaving the end function, and get to the end of the program.</p> If you can understand this you have just grasped two of the most important parts of programming - looping and conditional execution.</p> I suggest a long cup of tea.</p> While you're sipping, try out our new program by opening LXTerminal as before, and typing the usual incantation:</p> ./redgreen.py </pre> Notice this time it doesn't quit when you move the mouse, but it does if you click, or press a key, or close the window.</p> If it doesn't work compare your program with my version: redgreen.py</a>.</p> Next time we'll make an actual green shape appear, as promised a long time ago!</p> Comments</h2> bruce 2014-11-26</span> </h3> Andy, thanks for the video series.. in my "pi" world, and i now understand the redgreen thanks to you. i am wanting to do a project for my son and my trainset where my train passes through the if or hall effect( i dont care which sensor), but then pi reads that break and sends a command to turn on the lights at the crossing gates, before the train gets there. once the trains clears the sensor, it waits so many seconds and releases the lights. so i can use your approach of "assume its magic and it knows what to do", but i haven’t a clue as to how to call pins on and off, nor then how to take that state of ^{1</sup>⁄_{0</sub> and turn on another relay from it..so from redgreen, this is level maybe 105…. any thoughts of the "library" of commands? it wold be interesting to design an image of the raspberry pi and when you (mouseover) a pin or part of the raspberry, it calls up the command associated with it. well in conclusion- i guess i gave examples of two different programs..</p>}} your thoughts are very much appreciated..</p> Bruce Humphrey</p> </div> Andy Balaam 2014-11-26</span> </h3> Hi Bruce, this might help: http://makezine.com/projects/tutorial-raspber...</a> Good luck with your project!</p> </div> My First Raspberry Pi Game " Part 05 " Say something Andy Balaam — Thu, 22 Nov 2012 08:31:40 +0000 Parts: 1</a>, 2</a>, 3</a>, 4</a>, 5, 6</a>, 7</a>, 8</a>, 9</a>, 10</a>, 11</a>, 12</a>.</p> Writing your first ever computer program on the Raspberry Pi.</em></p> </iframe> Today we will write some writing on that blank screen</a> we made last time.</p> But first, a couple of tricks (we are doing magic</a> after all). We're going to make our program know it is a Python program, without needing to be told. To do this we need to do 2 things.</p> First, open up LeadPad as normal and add this line at the very (very) top of the file. Make sure there are no empty lines above it:</p> #!/usr/bin/env python </pre> That's a "hash" (or, for Americans, "pound") symbol, followed by an exclamation mark. Note that all the slashes are forward slashes, not backward.</p> This tells our Raspberry Pi that this is a Python program. Now we need to tell our Pi that this program is allowed to run by itself, instead of its name being passed to the python program like we have been doing before.</p> To do this, open up LXTerminal as before, and type exactly this (and press Return):</p> chmod +x redgreen.py </pre> If all of this worked correctly, you should be able to run our program in a new way. Instead of typing python redgreen.py like we were before, we can type this in to LXTerminal:</p> ./redgreen.py </pre> That's a dot, followed by a forward slash, followed by the name of our program.</p> This means run the program in this directory (that is the "./" part) called "redgreen.py". Your Pi will look at redgreen.py and find the line we added that starts with "#!" and know to use Python to run it.</p> Now let's get on with writing something on the screen. Go back to LeafPad and change the line starting with screen_size = to be these 3 lines:</p> screen_width = 640 screen_height = 480 screen_size = screen_width, screen_height </pre> This creates 3 variables - screen_width, screen_height and the one we had before screen_size, which is now made by putting the first 2 together. We're going to use screen_height later.</p> Just below those 3 lines, type this:</p> screen = None ready_text = None </pre> This gets 2 variables ready for us, and makes them empty. We're going to fill them in inside start.</p> Change the start function to look exactly like this:</p> def start(): global screen, ready_text</span> pygame.init() screen = pygame.display.set_mode( screen_size ) font = pygame.font.Font( None, screen_height / 5 ) ready_text = font.render( "Ready?", 1, pygame.Color( "white" ) )</span> </pre> The green lines above are the bits we've added. The line beginning global tells Python we want to work with those variables we got ready earlier inside this function, even though we created them outside it. (Without saying they were "global" we would be in danger of working with versions of them that only existed while we were inside the function, and disappeared as soon as we left.)</p> The font line makes a new font (a font is a typeface, or way of writing text). The first argument we passed was None because we don't care at the moment which font we use (e.g. "Arial" or "Times New Roman") - we are happy with the default. The second argument is for the size of the font we want, and we passed in screen_height / 5, which means the value of the screen_height variable we created near the top, divided by 5. The "/" character is how we write division in Python - it is supposed to look a bit like a fraction.</p> Finally, on the last line we create another variable called ready_text, which contains the "rendered" version of the word "Ready?", using the font we created, in white. "Render" means we create a picture showing the writing we wanted. We'll draw this picture onto the screen in a second.</p> Now that all our preparation is over, we can finally write a fuller version of the ready_screen function. Change it to look like this:</p> def ready_screen(): textpos = ready_text.get_rect( centerx = screen.get_width() / 2, centery = screen.get_height() / 2 ) screen.blit( ready_text, textpos ) pygame.display.flip() </pre> The first 4 lines (textpos up to the closing bracket all on its own) are really all one "line of code" - they are like one sentence of our program, that happens to span multiple lines. In Python if we want to continue a sentence (we call it a "statement") we just leave a bracket unclosed. Python knows we have finished when we have closed all the brackets.</p> This "line" from textpost = up to ) creates a variable called textpos, which contains the place on the screen where we want to put our writing. We look inside ready_text (which is our rendered writing that we created above) for a function called get_rect that calculates a rectangle for us that is the right place on the screen to put the writing. The arguments we pass to get_rect are screen.get_width() / 2 and screen.get_height() / 2, which are telling it that it should calculate the rectangle by putting its centre in the middle of the screen. The middle of the screen is half-way across its width, and half-way down its height, which is why we are dividing the width and the height of the screen by 2.</p> Something worth noticing here is that the arguments to get_rect have names - we wrote centerx = and centery =. In Python we are allowed to give the names of arguments, or sometimes we can miss them out if we are happy just to put them in the right order. The get_rect function can actually take lots of different arguments, so it needs to know which ones you mean, which is why we named them.</p> Finally the last 2 lines do the real work. The screen.blit line tells PyGame to write the ready_text picture (the rendered writing) onto the screen at the position stored inside textpos. pygame.display.flip() is what we do to tell PyGame we've finished messing about with the screen, and we're ready for it to display what we've done.</p> [The word "blit" is an oddity from the olden days</a> which I'm afraid you'll just have to memorise, and "flip" comes from the fact that behind the scenes there are really two screens - the one we are displaying, and the one we are working on. flip() switches them over, displaying the one we were working on, and making the other one ready to be worked on.]</p> If you've got this far, well done! With any luck, we're going to see the word "Ready?" on the screen in big letters.</p> Switch to LXTerminal again and type our new spell:</p> ./redgreen.py </pre> Don't move the mouse or press anything: a window should appear, with the word "Ready?" written in big white letters. When you press a key or move the mouse over it, it should disappear.</p> If something goes wrong, check back over the instructions carefully, and compare your file with this one: redgreen.py</a>.</p> Comments</h2> will 2013-11-01</span> </h3> i followed everything then i tryed it once and it worked then i tryed again to show my dad and it didnt work</p> </div> Andy Balaam 2013-11-05</span> </h3> Hi will, what happened when it went wrong?</p> </div> Louis 2013-12-30</span> </h3> hi Andy, thanks for all the tutorials they are great and help a ton!!!</p> one problem is for some reason since i tried the shortcut on EP.5 NONE of my programs will work. can i fix it or do i need to reset my raspberry pi?</p> </div> Andy Balaam 2014-01-02</span> </h3> Hi Louis, what error messages do you see? Try deleting the #! bit at the beginning and continuing without it – you can still launch it with python redgreen.py every time – the other way is just supposed to be easier.</p> </div> Piet 2014-08-20</span> </h3> Hi Andy I did every thing right and it is not working i checked everything three times and it says traceback (most recent call last): Screen = none Name error: name "none" is not defined</p> PLEASE REPLY I HAVE COME TOO FAR</p> </div> Piet 2014-08-20</span> </h3> Hi Andy I did every thing right and it is not working i checked everything three times and it says traceback (most recent call last): File "redgreen.py", line 9, in Screen = none Name error: name "none" is not defined</p> PLEASE REPLY I HAVE COME TOO FAR</p> </div> Andy Balaam 2014-08-21</span> </h3> Hi Piet, don’t give up! You need to spell "None" with a capital "N". Also, make sure you check the case of all the other parts as well – I notice you’ve put a capital "s" for "screen", but it should be lower case.</p> </div> Nigel Hughes 2012-12-08</span> </h3> Thanks for giving us the text version. The video is fine, but takes a lot of time and it is much easier to go back through the text version when you have blundered!</p> </div> Andy Balaam 2012-12-10</span> </h3> No problem! Yes, I find typing from video pretty error-prone. I also sometimes explain things better in the text, or vice-versa.</p> </div> Michael 2013-01-22</span> </h3> Hi Andy. Just wanted to thank you for your series of tutorials. As a complete novice I am very grateful for the step by step introduction. So is my 11 year old son, who has proudly published hi first "Hello world" on his diy website. He feels very accomplished. https://diy.org/sparkboy123/000sb9?utm_source...</a></p> Quick question though. Is there a way to go back and change the set-up for the keyboard? So of the keys are mixed up (the " and the @ for example are reversed). No worries if you don’t have time for random questions.</p> Thanks again.</p> </div> Andy Balaam 2013-01-22</span> </h3> Hi Michael, please let your son know that I was so excited he got his "Hello world" program working I updated Part 2</a> to link to his entry!</p> To change your keyboard settings (I think you have a US keyboard, but it defaulted to UK probably) I think the best thing to do is re-run the program you saw first time you booted, which you can do by opening LXTerminal and typing "sudo raspi-config" (without the quotes). Let me know whether it works!</p> Thank you so much for your positive feedback.</p> </div> Andy Balaam 2013-01-22</span> </h3> More details here by the way http://elinux.org/RPi_raspi-config</a></p> </div> My First Raspberry Pi Game " Part 04 " A small black screen Andy Balaam — Tue, 20 Nov 2012 19:01:29 +0000 Parts: 1</a>, 2</a>, 3</a>, 4, 5</a>, 6</a>, 7</a>, 8</a>, 9</a>, 10</a>, 11</a>, 12</a>.</p> Writing your first ever computer program on the Raspberry Pi.</em></p> </iframe> Today we will get an actual window to appear, with nothing on it!</p> Last time we wrote a magic spell</a> describing the bare outline of what how our game will work. It's going to pause, show a shape, and then the player must either press a key (if it's green) or not press a key (if it's red).</p> The program we've written calls several mini-programs called functions, but they don't exist yet, and if we run it, it fails with an error. The names of the functions we have imagined are start, ready_screen, wait, shape and end.</p> The first thing we need to do is write those functions, so that the program runs correctly, even though it does nothing.</p> Open up LeafPad again and open redgreen.py from inside the pi folder. At the top, above the code we've already written (which uses the functions) type exactly this:</p> def start(): pass def ready_screen(): pass def wait(): pass def shape(): pass def end(): pass </pre> Notice that the word pass is "indented" by 4 spaces. This is how Python knows which lines of code are part of your function, and which are another part of the program. It's very important that all lines are indented by the same amount, and that you never mix "tab" characters with spaces. To make life simple, I suggest always using 4 spaces.</p> What we've done is define 5 functions (the ones we call further down). def just means define a function, and we've followed that by the name of the function, then (). So far, each of our functions is empty: they are all just one line, which says pass, which means "do nothing".</p> If we've typed this correctly, our program should now run correctly, and do nothing. Let's try it. Open LXTerminal and type:</p> python redgreen.py </pre> If it's working, the computer should say absolutely nothing to you, and give you another prompt, ending with $.</p> If you get an error message, read it (especially the last line) and try to work out what you typed wrong.</p> Now we're going to write some actual useful code, and make a black screen appear.</p> At the very top, above everything else we've done today, add this line:</p> import pygame </pre> This tells Python we want to use the PyGame</a> library, which is some code written by someone else, designed to help us write games in Python. The Raspberry Pi comes with PyGame already installed, so to use it all we need to do is add this line.</p> Just below that line, type this:</p> screen_size = 640, 480 </pre> This is how we define a "variable". A variable is a thing with a name and a value that we can refer back to later. The variable's name is screen_size and its value is "640, 480". We're going to use this variable to remember how big we want our window to be.</p> Now we need to modify our start function to bring up a black screen. Replace both lines of the start function with this:</p> def start(): pygame.init() screen = pygame.display.set_mode( screen_size ) </pre> The first line, starting with def is the same as before - it means we are defining a function called "start".</p> Where we used to have just pass, we now have 2 lines (both indented by 4 spaces as before). The pygame.init() line just tells PyGame to get ready to start. We need something like this in our program whenever we use PyGame. The second line creates a window of the size we want. It calls a function inside pygame. The dots mean we should "look inside" the thing on the left, so pygame.display.set_mode means something like "look inside pygame for something called display, then look inside display for something called set_mode". set_mode is a function, which we are calling (that is what the brackets mean).</p> Unlike other functions we have called, which just use "()", set_mode takes an "argument" which means we are passing information to it. The information we are passing is the size of the screen, which we have already stored in the variable screen_size. So the value "640, 480" gets passed in to set_mode, telling it how big to make the window.</p> We've made a window, but if we leave things as they are now, we will open it, then immediately finish, so it will go away again. We need to wait a while.</p> Edit the end function to look like this:</p> def end(): pygame.event.clear() pygame.event.wait() </pre> This code means "wait until something happens".</p> The first line (clear) tells PyGame we're not interested in anything that happened before - clear the list of events that has built up - and the second line (wait) means wait until something happens. In this case we are waiting until the person using the program presses a key, closes the window, or even moves the mouse in front of the window.</p> Let's try it out. Open up LXTerminal as before, and type the same thing we've typed before:</p> python redgreen.py </pre> Make sure you don't move the mouse or press any keys after you've pressed Return. If all has gone well, a small, black, empty window will appear. Then if you press a key or move the mouse in front of it, it should disappear.</p> If this isn't what happens, check back and make sure you typed everything exactly as above. Your complete file should look like this: redgreen.py</a>.</p> Well done! You made a window appear. Next time, if we're lucky, we'll put something in it...</p> Comments</h2> Nate 2017-07-13</span> </h3> Hello, Andy. I can’t get the black small screen to appear. The terminal on my pi says import: command not found screen_size: command not found and some other syntax errors. The terminal is running python 3.4.2 and I’ve tried upgrading the pip and pygame to no avail.</p> </div> Andy Balaam 2017-07-23</span> </h3> Hi Nate, it sounds like you are trying to type your program directly into the terminal. You need to edit your program in the text editor (Leafpad), save it, and run it by typing "python redgreen.py" in the terminal.</p> </div> Jimmy Saldivias 2018-01-06</span> </h3> Hello Andy. I edited the program, using 4 spaces. When I asked to run the message on line 2 there was IndentationError: unexpected indent. So I removed the 4 spaces from the line 2 and kept the 4 spaces for the rest of lines. The message of error in line 2 dissapeared. But the error appeared again this time for line 5. So I removed the 4 spaces from all lines. The error messaged changed for a the one that said NamError: name ‘start’ is not defined. I am using Python2. Any suggestions?</p> </div> Jimmy Saldivias 2018-01-06</span> </h3> Andy. My mistake. I have not used def before the names and (): after the names. Now it is working. Thanks.</p> </div> Jimmy Saldivias 2018-01-06</span> </h3> Andy. The link to PREVIOUS is not correct. The same happens to the windows to NEXT of the forecoming lesson.</p> </div> Andy Balaam 2018-01-07</span> </h3> Hi Jimmy, glad to hear you got it working!. Those Next and Previous links are just whichever blog posts I wrote at that time, not next and previous in this series, and I can’t change them because they are made automatically by my blog software (WordPress). The links at the top ("1", "2", …) are correct though!</p> </div> Length of Open Source licenses Andy Balaam — Thu, 15 Nov 2012 13:34:34 +0000 I have been choosing a license for my ficticious programming language, Pepper</a>. One consideration is the complexity of the (combination of) license(s) used. Complexity may be related to length, so for your enjoyment, here are the lengths of some popular licenses:</p> </p> Is it a coincidence I've decided to use the MIT License?</p> Here's how I made it (on my Ubuntu Linux 12.04 system):</p> #!/bin/bash declare -A LICENSES LICENSES[Apache]=file:///usr/share/common-licenses/Apache-2.0 LICENSES[GPL2]=file:///usr/share/common-licenses/GPL-2 LICENSES[GPL3]=file:///usr/share/common-licenses/GPL-3 LICENSES[BSD]=file:///usr/share/common-licenses/BSD LICENSES[Artistic]=file:///usr/share/common-licenses/Artistic LICENSES[MIT]=http://www.jclark.com/xml/copying.txt LICENSES[Mozilla]=http://www.mozilla.org/MPL/2.0/index.txt LICENSES[Python]=http://hg.python.org/cpython/raw-file/tip/LICENSE LICENSES[Eclipse]=http://www.eclipse.org/legal/epl-v10.html function wordcounts() { for K in "${!LICENSES[@]}"; do { echo `lynx -dump ${LICENSES[$K]} | wc -w` $K }; done } wordcounts | sort -n | awk '{print $2 " " $1}' > data.txt gnuplot <<END set terminal png large enhanced font "Helvetica,11" set output 'license-length.png' unset key set style fill solid set title "Number of words in common Free and Open Source licenses" plot 'data.txt' using (column(0)):2:(0.5):(\$0):xtic(1) with boxes lc variable; END rm data.txt </pre> Comments</h2> Phil Nash 2012-11-17</span> </h3> Shame you didn’t include the boost licence. I’d like to see how that stacks up (It would certainly be on the far left somewhere)</p> </div> Andy Balaam 2012-11-17</span> </h3> BOOST is 215 – just to the right of BSD I think. I considered BOOST, but went for MIT as it’s more widely used.</p> </div> My First Raspberry Pi Game " Part 03 " It's like a magic spell Andy Balaam — Wed, 14 Nov 2012 08:27:28 +0000 Parts: 1</a>, 2</a>, 3, 4</a>, 5</a>, 6</a>, 7</a>, 8</a>, 9</a>, 10</a>, 11</a>, 12</a>.</p> Writing your first ever computer program on the Raspberry Pi.</em></p> </iframe> Today we will write the very basic outline of our game. When we've finished it won't actually do anything. In fact, it won't even run.</p> Writing a computer program is a lot like doing magic. To write programs well it is really helpful to treat it like magic, and not think about how it works (until you have to...). We're going to do that - write down in very simple terms what our program will do, but not yet think about how it will do it.</p> The program we are going to write is a very simple test of your reactions. It will show you either a green circle or a red square. If you see a green circle, you have to press a key as quickly as you can. If you see a red square, you must not press anything.</p> That's the whole of the game (for now).</p> To make this game, we need to start up, then show a ready screen, then wait for a while showing the ready screen, then show a shape and wait for a key press, then finish.</p> So, let's write our magic spell. Start up LeafPad just like we did in part 2</a>. Click File, then Open, click on "pi" on the left, then double-click redgreen.py to load up our Hello, world program. Delete everything that's there, and type exactly this instead:</p> start() ready_screen() wait() shape() end() </pre> The brackets after each word mean "do it" - what they really mean is find a "function" with this name, and run it. A function is like a mini-program.</p> So, now our program is finished, right?</p> Let's run it. Go to the File menu in LeafPad and click Save, then open LXTerminal just like in part 2 and run the program by typing "python redgreen.py" as before. Here's what happens:</p> $ python redgreen.py Traceback (most recent call last): File "redgreen.py", line 3, in start() NameError: name 'start' is not defined </module></pre> We told python to run a function called "start", but we haven't written any functions yet, so it couldn't find it.</p> Our spell is cast, but we next time we need to start on the ingredients it uses. See you then.</p> Comments</h2> Andy Balaam's Blog » Blog Archive » My First Raspberry Pi Game â€“ Part 02 â€“ Saying hello 2012-11-14</span> </h3> […] Free Software « My First Raspberry Pi Game – Part 01 – Before we start My First Raspberry Pi Game â€“ Part 03 â€“ It’s like a magic spell […]</p> </div> Bill Gentles 2012-11-18</span> </h3> This is very helpful. I’m a total newbie at Linux. Thanks for making things so clear. Hope there’s more to come.</p> </div> Andy Balaam 2012-11-18</span> </h3> Hi Bill, thanks! I am working on the next one…</p> </div> Andy Balaam's Blog » Blog Archive » My First Raspberry Pi Game – Part 01 – Before we start 2012-11-19</span> </h3> […] My First Raspberry Pi Game – Part 01 – Before we start Parts: 1, 2, 3. […]</p> </div> Mark Kelly 2012-11-20</span> </h3> This is very easy to follow. I look forward to the next one and many more after that. You have a talent for teaching. So many others have this look at lot more complicated than it is. You have done the clever thing of making it seem simple; not an easy task to do.</p> </div> Andy Balaam 2012-11-20</span> </h3> Thanks Mark!</p> </div> My First Raspberry Pi Game " Part 02 " Saying hello Andy Balaam — Tue, 06 Nov 2012 08:45:20 +0000 Parts: 1</a>, 2, 3</a>, 4</a>, 5</a>, 6</a>, 7</a>, 8</a>, 9</a>, 10</a>, 11</a>, 12</a>.</p> Writing your first ever computer program on the Raspberry Pi. See Part 1</a> for how to get and set up the Pi.</em></p> Today we will find out how to write a computer program, and how to run it.</p> </iframe> We're going to write one of the simplest programs you can write - we're going to get the computer to say hello to us.</p> First, we need a text editor to write down our program. Click the weird aeroplane-y thing in the bottom left - that brings up the menu (like the Start button on Windows), then choose Accessories, then Leafpad. Leafpad is the text editor we will be using.</p> Leafpad will start, and show you an empty page. This is where we will write our program.</p> Type in exactly this:</p> print "Hello, world!" </pre> and then click the File menu at the top of the Leafpad window and choose Save As. Click the word "pi" on the left and then click in the empty box next to the word Name, and type the name of our program, which is:</p> redgreen.py</pre> "redgreen" is the name and the ".py" means this is a program written in the language Python</a>. We'll be finding out more about Python as we go on.</p> Click the Save button.</p> Our program is finished! Now we need to run it.</p> Click the aeroplane-y thing again, then Accessories, then LXTerminal. A terminal is a program you use to run other programs.</p> When LXTerminal has started, your cursor will appear next to a $ sign. This means it is ready for you to tell it what to do.</p> Type exactly this:</p> python redgreen.py </pre> What this means is run the program called "python", and pass the name of our program (redgreen.py) to it. This is how you run Python programs.</p> Now press the RETURN key.</p> If all goes well, our program will talk back to us, and say what we told it to say:</p> Hello, world! </pre> Let's look again at our program.</p> It's just one "statement", a print statement. A statement is something to do.</p> We pass one "argument" to print, "Hello, world!". An argument is some information you give to a statement.</p> "print" doesn't mean print to the printer, but write to the screen. So our program did exactly what we told it to do - it wrote our message to the screen.</p> Next time, we'll map out the whole of our real program - a simple game.</p> Update: congratulations to sparkboy123 on getting this working</a>!</em></p> Comments</h2> Jimmy Saldivias 2018-01-06</span> </h3> Hi Andy. I decided to do the exercises. I found out that they don’t work with python 3. But as you mentioned in the comments on the previous chapter, it works fine with python 2. I did another mistake -in case anyone else does the same-. I used uppercase. So, things did not work until I used lower case for the command and for the name of the file. Thanks for the videos and the tutorial.</p> </div> My First Raspberry Pi Game - Part 01 - Before we start Andy Balaam — Tue, 30 Oct 2012 20:50:00 +0000 Parts: 1, 2</a>, 3</a>, 4</a>, 5</a>, 6</a>, 7</a>, 8</a>, 9</a>, 10</a>, 11</a>, 12</a>.</p> In this series I intend to guide you through writing your first ever computer program.</p> </iframe> We will write our own game on the Raspberry Pi, which is a cheap-as-chips computer designed for learning about computers.</p> Get a Raspberry Pi</h3> To follow along, here's what you will need:</p> A Raspberry Pi (about Â£25) - I got it from RS</a></li> An SD card (about Â£10) - be careful - not all of them work. I use: Kingston Technology 16GB</a></li> If your TV supports it, an HDMI cable (about Â£1) - I got: HDMI to HDMI Connector</a>. If your TV doesn't support HDMI, get a composite cable, but it won't look as good or work as well.</li> A power supply (about Â£4) e.g. a smartphone charger (micro USB, at least 5V, 1A) - I got Micro USB Mains Charger</a> but my existing HTC Wildfire S charger worked too.</li> USB keyboard and mouse (about Â£7) - I had them lying around, but a quick search suggests this one might be ok: CiT USB Keyboard and Mouse</a>.</li> </ul> (Total cost, very approximately: Â£47)</p> Install "Raspian"</h3> To use the Pi you will need to install some software onto your SD card.</p> To do this you will need a PC or laptop. If you don't have one, or you'd prefer not to download and install software to an SD card, check out The Pi Hut</a>. They sell SD cards that already have Raspian installed on them.</p> Raspian is the name of the software we will use to start up and run our Raspberry Pi. You need to download it and install it onto your SD card before you can put the SD card into the Pi and turn it on. To do this, you'll need a way of writing to the SD card. Lots of laptops (and some desktops) have built-in SD card readers, or you can get a USB reader (I got this one: SD Card Reader USB 2.0</a>).</p> To install Raspian "wheezy" (wheezy is the name of the latest version) go to the Raspberry Pi download page at www.raspberrypi.org/downloads</a> and click the link in the Raspian "wheezy" section next to the words "Direct download". Follow the instructions on how to install Raspian to your SD card here: elinux.org/RPi_Easy_SD_Card_Setup</a>.</p> (There are also some helpful instructions here: reviews.cnet.co.uk/desktops/how-to-get-started-with-the-raspberry-pi-50009845/</a>.)</p> Start the Pi</h3> Once you've got an SD card with Raspian on it, insert it into your Pi (the SD card slot is underneath, which surprised me a bit). Plug the Pi into your TV by connecting the HDMI cable to it and plugging the other end into the TV's HDMI port. Plug your keyboard and mouse into the 2 USB slots.</p> Take a deep breath, and plug the power supply into the micro-USB port.</p> If all goes well, some lights will appear on the Pi, you will be able to switch your TV to HDMI mode and your screen will show some writing and possibly pictures of raspberries. Wait for it all to settle down, and (hopefully) eventually you'll see the setup screen.</p> First time setup</h3> The first time your Pi boots it will ask you to do some setup. Read the raspi-config menu items and see whether there's anything you want to change. You might want to change your keyboard and language settings, but I didn't need to change anything at all. I just pressed TAB and then right-arrow to move onto the word Finish, then pressed RETURN.</p> There's more information about how to set everything up at elinux.org/RPi_raspi-config</a>, and there's a nice detailed video here: First boot and Raspi-config</a>.</p> Wait a bit more, and eventually you should see a huge raspberry, with a mouse cursor and desktop. If so, you're ready for the next part!</p> Update</strong>: a real person really following this series!:</p> @andybalaam</a> my lad @irrelevawesome</a> just got a Rpi. He's using your tutorial now. Thanks! twitter.com/foobarber/statâ€¦</a></p> — Pete Barber (@foobarber) January 26, 2013</a></p> </blockquote> Comments</h2> Thrifting with Raspberry Pi | Dive Into the Mind of a Middle School Teacher 2013-08-05</span> </h3> […] I found an awesome website to create my first game! It took me two hours to complete and was pretty simple. If a green circle […]</p> </div> Jimmy Saldivias 2018-01-03</span> </h3> This is a great place to be. I have been working with Raspberry Pi 3 for a year now. And nowhere i found as structured approach to programming as yours. It is easy to follow. Each step is explained thoroughly. At the same time you do not delve more than is necessary to explain a concept. Directing us to places where we can find more info if needed or wanted. I particularly like the "magic" approach. Thanks.</p> </div> Colin Brearley 2018-01-04</span> </h3> Hi Andy, hope you are all well. We got J a Pi for Christmas so will be getting him to try this out this weekend!!</p> </div> Andy Balaam 2018-01-04</span> </h3> Hi Colin, we are very well, good luck with it, and leave comments if things are unclear. It’s a bit out of date, but hopefully most of it still makes sense. It uses Python 2, so if Python 3 is the default on the Pi 3, that will cause you problems. if that is the situation, you can make it all work by just typing "python2" to launch, instead of "python".</p> </div> Ken Roberts 2012-10-31</span> </h3> Andy, This is wonderful. You are a great teacher.</p> </div> Andy Balaam 2012-10-31</span> </h3> Thanks Ken :)</p> </div> Cool Raspberry Pi Projects – Andy Balaam's Blog 2012-12-21</span> </h3> […] Update: I made some videos on how to write your first ever game on your Raspberry Pi! […]</p> </div> Running Dojo DOH tests in a browser without a web server Andy Balaam — Tue, 30 Oct 2012 10:46:09 +0000 Dojo's DOH requires a web server to run tests in a browser. But never fear:</p> $ cd ~/code/dojo $ ls docs dojo util $ python -m SimpleHTTPServer & $ xdg-open http://localhost:8000/util/doh/runner.html </pre> Note that you will see some test failures, because the python web server doesn't do PHP.</p> When finished:</p> $ kill %1 </pre> to stop your web server.</p> On Python 3, use this instead of the SimpleHTTPServer line:</p> python3 -m http.server & </pre> Yes, Python includes a little web server that serves files in your current directory. Batteries included. Thanks to Lyle Backenroth</a> and commandlinefu</a> for making me aware of this.</p> Comments</h2> Paul Lysak 2012-11-17</span> </h3> Thank you, it helped me a lot to make a testable distribution of my CSV parsing library.</p> </div> Bash associative array examples Andy Balaam — Wed, 17 Oct 2012 12:30:50 +0000 Quick reference of things I discovered about how to use associative arrays in bash. Note: bash version 4 only.</strong></p> (See also Bash Arrays</a>.)</p> Basics</h3> $ declare -A MYMAP # Create an associative array $ MYMAP[foo]=bar # Put a value into an associative array $ echo ${MYMAP[foo]} # Get a value out of an associative array bar $ echo MYMAP[foo]</span> # WRONG MYMAP[foo] $ echo $MYMAP[foo]</span> # WRONG [foo] </pre> Creating</h3> $ declare -A MYMAP # Explicitly declare $ MYMAP[foo]=bar # Or this line implicitly makes it an associative array (in global scope, bash 4.2+ only) $ MYMAP[baz]=quux # Can add multiple values one by one $ MYMAP[corge]=grault </pre>$ declare -A MYMAP=( [foo]=bar [baz]=quux [corge]=grault ) # Initialise all at once $ echo ${MYMAP[foo]} bar $ echo ${MYMAP[baz]} quux </pre>$ declare -A MYMAP # Or declare separately $ MYMAP=( [foo]=bar [baz]=quux [corge]=grault ) # Then initialise $ echo ${MYMAP[foo]} bar $ echo ${MYMAP[baz]} quux </pre></p> Variables as keys</h3> $ K=baz $ MYMAP[$K]=quux # Use a variable as key to put a value into an associative array $ echo ${MYMAP[$K]} # Use a variable as key to extract a value from an associative array quux $ echo ${MYMAP[baz]} # Obviously the value is accessible via the literal key quux </pre> Quoting keys</h3> $ declare -A MYMAP $ MYMAP[foo A]="bar B" # Quoting keys makes no difference $ MYMAP["corge X"]="grault Y" # Quoting keys makes no difference $ echo ${MYMAP["corge X"]} # You can access by quoting grault Y $ echo ${MYMAP[corge X]} # Or not bother grault Y $ echo ${MYMAP[foo A]} bar B $ MYMAP['waldo 1']="fred 2" # Single quotes also make no difference $ echo ${MYMAP['waldo 1']} fred 2 $ echo ${MYMAP[waldo 1]} fred 2 $ K=plugh $ MYMAP['$K']=xyzzy # Except single quotes prevent variable expansion, as usual $ echo ${MYMAP[plugh]} $ echo ${MYMAP['$K']} xyzzy </pre> Missing keys</h3> $ MYMAP[foo]=bar $ echo ${MYMAP[missing]} # Accessing a missing value gives "" $ # Testing whether a value is missing from an associative array $ if [ ${MYMAP[foo]+_} ]; then echo "Found"; else echo "Not found"; fi Found $ if [ ${MYMAP[missing]+_} ]; then echo "Found"; else echo "Not found"; fi Not found </pre> Looping</h3> $ declare -A MYMAP=( [foo a]=bar [baz b]=quux ) $ echo "${!MYMAP[@]}" # Print all keys - quoted, but quotes removed by echo foo a baz b $ # Loop through all keys in an associative array $ for K in "${!MYMAP[@]}"; do echo $K; done foo a baz b $ for K in ${!MYMAP[@]}; do echo $K; done</span> # WRONG foo a baz b $ # Looping through keys and values in an associative array $ for K in "${!MYMAP[@]}"; do echo $K --- ${MYMAP[$K]}; done foo a --- bar baz b --- quux $ # Loop through all values in an associative array $ for V in "${MYMAP[@]}"; do echo $V; done bar quux </pre> Clearing</h3> $ declare -A MYMAP $ MYMAP[foo]=bar $ echo ${MYMAP[foo]} bar $ declare -A MYMAP</span> # Re-declaring DOES NOT clear an associative array $ echo ${MYMAP[foo]} bar $ unset MYMAP # You need to unset and re-declare to get a cleared associative array $ declare -A MYMAP $ echo ${MYMAP[foo]} </pre> Deleting keys</h3> $ MYMAP[foo]=bar $ echo ${MYMAP[foo]} bar $ unset ${MYMAP[foo]}</span> # WRONG $ echo ${MYMAP[foo]} bar $ unset MYMAP[foo] # To delete from an associative array, use "unset" with similar syntax to assigning $ # BUT see next section if key contains spaces $ echo ${MYMAP[foo]} $ MYMAP[baz]=quux $ echo ${MYMAP[baz]} quux $ K=baz $ unset MYMAP[$K] # Can unset using a variable for the key too $ # BUT see next section if variable may contain spaces - always better to quote $ echo ${MYMAP[baz]} </pre> Deleting keys containing spaces</h3> $ declare -A MYMAP $ MYMAP[foo Z]=bar $ echo ${MYMAP[foo Z]} bar $ unset MYMAP[foo Z]</span> # WRONG bash: unset: `MYMAP[foo': not a valid identifier bash: unset: `Z]': not a valid identifier $ unset MYMAP["foo Z"] # You must quote keys containing spaces when you unset in an associative array $ echo ${MYMAP[foo Z]} $ MYMAP[foo Z]=bar $ unset MYMAP['foo Z'] # Single quotes work too $ echo ${MYMAP[foo Z]} $ MYMAP[baz A]=quux $ echo ${MYMAP[baz A]} quux $ K="baz A" $ unset MYMAP[$K]</span> # WRONG bash: unset: `MYMAP[baz': not a valid identifier bash: unset: `A]': not a valid identifier $ unset MYMAP["$K"] # You must quote variables whose values may contain spaces when you unset in an associative array $ echo ${MYMAP[baz A]} $ MYMAP[baz A]=quux $ unset MYMAP['$K'] # Curiously, single quotes work here too, although I'd suggest avoiding them $ echo ${MYMAP[baz A]} </pre> Length</h3> $ declare -A MYMAP=( [foo a]=bar [baz b]=quux ) $ echo ${#MYMAP[@]} # Number of keys in an associative array 2 $ echo $#MYMAP[@]</span> # WRONG 0MYMAP[@] $ echo ${#MYMAP}</span> # WRONG 0 </pre> Numeric indexing</h3> $ declare -A MYMAP=( [foo a]=bar [baz b]=quux ) $ KEYS=("${!MYMAP[@]}") # Make a normal array containing all the keys in the associative array $ echo ${KEYS[0]} # Find a key via an index foo a $ echo ${MYMAP[${KEYS[0]}]} # Find a value via an index bar $ # Loop through using an index $ for (( I=0; $I < ${#MYMAP[@]}; I+=1 )); do KEY=${KEYS[$I]}; echo $KEY --- ${MYMAP[$KEY]}; done foo a --- bar baz b --- quux </pre> Scope</h3> $ unset MYMAP $ function createmap() { MYMAP[foo]=bar; } # Implicit creation puts it in the global scope $ echo ${MYMAP[foo]} $ createmap $ echo ${MYMAP[foo]} bar </pre>$ unset MYMAP $ function createmaplocal() { declare -A MYMAP; MYMAP[foo]=bar; } # Explicit creation puts it in the local scope $ echo ${MYMAP[foo]}</p> $ createmaplocal $ echo ${MYMAP[foo]}</p> </pre> Checking Bash version</h3> $ bash --version # Must be at least version 4 to have associative arrays GNU bash, version 4.2.24(1)-release (x86_64-pc-linux-gnu) ... </pre> Links</h3> Bash manual: Arrays</a></li> Linux Journal: Associative Arrays in Bash</a></li> Bash Hackers Wiki: Arrays</a></li> Superuser: Test if element is in array in Bash</a></li> Stackoverflow: How to iterate over associative array in bash</a></li> </ul> Comments</h2> Walter 2015-06-09</span> </h3> Wow, just have learned how to use associative arrays, via this very handy page! A clear HowTo. And what I also especially like about it, is that along with examples how to do things, it also gives the examples how to NOT do certain things. It caught me before falling into a few pitfalls: you have predictive mind. :–)</p> </div> Mark Allan 2015-10-22</span> </h3> Great site… but I am looking for an explanation of the code below? HOW DOES THIS WORK WITHOUT AN ASSIGN??? I am totally confused, it works, it inits and declares, it’s simple you can see the values but well… it’s like an awk 1 to me??? Except I can’t see the syntax in any manual or search I’ve done.</p> echo 1 | awk ‘{ sorex["W"] sorex["B"] sorex["TH"] sorex["FR"] for (i in sorex) print i }’</p> Many thanks in case you can explain it.</p> Mark (Manchester)</p> </div> Andy Balaam 2015-10-23</span> </h3> Hi Mark, that code inside the single quotes is all Awk code, not bash. This might help: https://www.gnu.org/software/gawk/manual/gawk.html</a></p> </div> coderofsalvation 2016-02-13</span> </h3> Use this higher order function to prevent the pyramid of doom:</p> foreach(){ arr="$(declare -p $1)" ; eval "declare -A f="${arr#*=}; for i in ${!f[@]}; do $2 "$i" "${f[$i]}"; done }</p> example:</p> $ bar(){ echo "$1 -> $2"; } $ declare -A foo["flap"]="three four" foo["flop"]="one two" $ foreach foo bar flap -> three four flop -> one two</p> </div> Bash association arrays | Jacek Kowalczyk MyBlog 2016-02-24</span> </h3> […] I found a very good article about bash association arrays. Here is a link to it : http://www.artificialworlds.net/blog/2012/10/...</a> […]</p> </div> Craig Strickland 2013-07-28</span> </h3> For the benefit of future visitors to this page (like me) that are running pre-4.2 bash, the comment in your statement:</p> "$ MYMAP[foo]=bar # Or this line implicitly makes it an associative array (in global scope)"</p> is not true for bash versions <4.2 wherein associative arrays MUST be explicitly created with "declare -A".</p> If not pre-declared, then your example (if NOT preceded by "declare -A"):</p> "$ MYMAP[foo]=bar"</p> implicitly performs arithmetic evaluation of the expression "foo", which produces a numeric result of "0", thereby assigning element "0" of indexed</em> array "MYMAP".</p> You can see the result:</p> $ declare -p MYMAP declare -a MYMAP='([0]="bar")'</p> The subscript is "0", not the string "foo".</p> This is in:</p> $ echo $BASH_VERSION 4.1.5(1)-release</p> —tgi</p> </div> Andy Balaam 2013-09-12</span> </h3> Hi Craig, thanks for the very informative addition.</p> </div> Dave 2016-09-22</span> </h3> I’m confused about scope. The following doesn’t work as I expect. Even though I explicitly declare fruit to be an associative array, and it acts like it inside the while loop, the values added during the loop are not present outside the loop. Thanks for any clarification.</p> $ cat /tmp/t.bash #!/bin/bash echo "a apple" > /tmp/fruit echo "b banana" >> /tmp/fruit echo "c cranberry" >> /tmp/fruit</p> declare -A fruit fruit[p]=pumpkin cat /tmp/fruit | while read line; do t=$(echo $line|sed -e ’s/ .//‘); f=$(echo $line|sed -e ’s/.</em> //’); fruit[$t]=$f ; echo "fruit[$t] = ‘${fruit[${t}]}’; fruit[p]=${fruit[p]}." ; done</p> echo ""</p> echo "fruit[a]=${fruit[‘a’]}" echo "fruit[b]=${fruit[‘b’]}" echo "fruit[c]=${fruit[‘c’]}" $ /tmp/t.bash fruit[a] = ‘apple’; fruit[p]=pumpkin. fruit[b] = ‘banana’; fruit[p]=pumpkin. fruit[c] = ‘cranberry’; fruit[p]=pumpkin.</p> fruit[a]= fruit[b]= fruit[c]= fruit[p]=pumpkin $ bash —version GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu) Copyright © 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later</p> This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.</p> </div> Andy Balaam 2016-09-24</span> </h3> Hi Dave, if you set a variable value inside the do .. done it does not leak out of the scope:</p> $ cat /tmp/t.bash #!/bin/bash echo "a apple" > /tmp/fruit x=2 cat /tmp/fruit | while read line; do x=3; done echo $x</p> $ /tmp/t.bash 2</p> </div> daveadams 2016-09-24</span> </h3> In order to get the scope to work how you expect, @Dave, you need to invert the operations. It doesn’t work because you are piping the output of cat /tmp/fruit</code> into a while loop, but the second command in the pipe (the while loop) spawns in a new process. So in that subprocess, the variables are being set, but when the while loop terminates the subprocess terminates and the changes to the variables are lost. So in order to do what you want, the while loop needs to be in the process with the rest of the script. So, instead you can do:</p> $ cat test.sh #!/bin/bash</p> cat >/tmp/fruit < a apple b banana c cranberry FRUITS</p> declare -A fruit=( [p]=pumpkin )</p> while read t f; do fruit[$t]="$f" echo "fruit[$t] = '${fruit[${t}]}'; fruit[p]=${fruit[p]}." done < /tmp/fruit</p> echo "" for i in "${!fruit[@]}"; do echo "fruit[$i] = '${fruit[$i]}'" done</p> $ bash test.sh fruit[a] = 'apple'; fruit[p]=pumpkin. fruit[b] = 'banana'; fruit[p]=pumpkin. fruit[c] = 'cranberry'; fruit[p]=pumpkin.</p> fruit[a] = 'apple' fruit[b] = 'banana' fruit[c] = 'cranberry' fruit[p] = 'pumpkin'</p> </div> Andy Balaam 2016-09-27</span> </h3> Excellent, thanks Dave!</p> </div> mop 2013-11-24</span> </h3> Passing to a function and how to assign between variables (both arrays) is missing IMHO.</p> Regards, and thank you, Matteo.</p> </div> Andy Balaam 2013-12-02</span> </h3> Hi Matteo, thanks – yes those would be useful.</p> </div> Harro Merkus 2013-12-23</span> </h3> Thanks for your post.</p> I was looking for a way to delete a variable key from an associative array, where that variable may be a single space. Then these do not work: unset MYMAP[ ] unset MYMAP[’ ‘] unset MYMAP[" "] K=’ ‘ unset MYMAP[$K] unset MYMAP["$K"]</p> However, this one does work: unset MYMAP[’$K’]</p> Also, if K is a single or double quote, only the latter one works!</p> Using GNU bash, version 4.2.25(1)-release (x86_64-pc-linux-gnu).</p> </div> Andy Balaam 2014-01-02</span> </h3> Hi Harro, thanks – useful information.</p> </div> Sharon 2017-02-27</span> </h3> Great post!!! Thanks a lot,</p> Can you please explain why do you add "+_" when you trying to test value existing? It works for me without this addition: /home/ubuntu# if [ ${MYMAP[blablabla]} ]; then echo yes; else echo no;fi</p> no</p> </div> Andy Balaam 2017-02-27</span> </h3> Hi Sharon, I don’t actually know why I added +_ – I am wondering whether this is an artefact of copying and pasting from somewhere else… Thanks for the comment!</p> </div> Greywolf 2014-01-07</span> </h3> Default variable test/expansion rules apply:</p> $ declare -A ax; $ ax[foo]="xkcd"; $ echo ${ax[foo]:-MISSING}; xkcd $ echo ${ax[bar]:-MISSING}; MISSING $ echo ${ax[foo]:+SET}; SET $</p> </div> Andy Balaam 2014-01-08</span> </h3> Thanks Greywolf, very helpful.</p> </div> Sameer Oak 2014-02-11</span> </h3> Thank you very much for such a priceless post.</p> At present, I’m struggling to find solution to either of the following problems: 1> how to convert a nornal array (indexed array with index starting at 0) into an associative array where value becomes a key and value itself is the value. 2> Create a new assoc array from indexed array where values are keys. And this in a single statement. I know it can very well be done using a loop but for a huge sized array containing almost 500,000 elements, a loop is an overhead. 3> Create an assoc array from the result of sql query. I normally create an indexed array from the sql query result as below: mapfile -t a_dummy <<< "$(mysql -u root —disable-column-names —silent -B -e "select * from dummy_tbl;" "$DB_NAME")" where $DB_NAME is the variable pointing to DB name string.</p> Thank you very much in advance.</p> </div> just_an_abstraction 2017-05-09</span> </h3> Really useful, I was stuck declaring an associative implicitly inside a function, apparently you need declare -A for that to work fine. And it apparently stays in local scope too.</p> </div> Andy Balaam 2017-05-09</span> </h3> Thanks, glad it was useful!</p> </div> CPRitter 2014-03-17</span> </h3> Andy: Thanks for the informative write-up! I wish I had found it before I spent an hour figuring it out myself. Now, I was brought to your site while searching for a solution to this …</p> Is there a less clumsy method of sorting keys than this (spaces in keys must be preserverd)…</p> bash-4.1$ declare -A ARY=( [fribble]=frabble [grabble]=gribble [co bb le]=cribble [babble]=bibble [zibble]=zabble [n o bbl e]=nibble [mobble]=mibble ) bash-4.1$ keys=( ${!ARY[@]} ) bash-4.1$ IFS=$‘\n’ sorted_keys=( $( echo -e "${keys[@]/%/\n}" | sed -r -e ’s/^ *//’ -e ‘/^$/d’ | sort ) ) bash-4.1$ for key in "${sorted_keys[@]}"; do echo "$key: ${ARY[$key]}"; done babble: bibble co bb le: cribble fribble: frabble grabble: gribble mobble: mibble n o bbl e: nibble zibble: zabble bash-4.1$</p> Thanks!</p> </div> Andy Balaam 2014-03-18</span> </h3> Hi CPRitter, that looks like a pretty good way to do this, but I think at this point I’d be reaching for Perl or Python…</p> </div> Mac OS X Bash – upgrade – Open Source Refinery 2017-06-25</span> </h3> […] Bash associative array examples […]</p> </div> Doron 2018-01-11</span> </h3> @Sharon, Andy,</p> Re Missing Keys and the "+_" in the examples: this is in fact quite important, and it’s a good thing you quoted it in this guide.</p> It differentiates between the case where a key does not exist, and the case where it does exist but its value is null.</p> To use Sharon’s example, this indeed seems to work:</p> if [ ${MYMAP[blablabla]} ]; then echo yes; else echo no;fi</p> no</p> BUT if you</p> MYMAP[blablabla]=""</p> Then</p> if [ ${MYMAP[blablabla]} ]; then echo yes; else echo no;fi</p> no</p> But</p> if [ ${MYMAP[blablabla]+_} ]; then echo yes; else echo no;fi</p> yes</p> Et voila.</p> </div> Rahul 2018-01-12</span> </h3> Nice Way to show examples. Other examples of Array Basics Shell Script: https://blog.prakhar.info/array-basics-shell-...</a></p> </div> Andy Balaam 2018-01-12</span> </h3> Thanks Rahul!</p> </div> Andy Balaam 2018-01-12</span> </h3> Thanks Doron!</p> </div> Avi 2018-12-01</span> </h3> declare -A MYMAP doesn’t work and throws an error: declare: -A: invalid option declare: usage: declare [-afFirtx] [-p] [name[=value] …]</p> using the quotes around the values throws an error like this: array[wow]: command not found san francisco</p> </div> Andy Balaam 2018-12-03</span> </h3> Avi, are you sure you are using bash? I just tried declare -A MYMAP here and it worked.</p> </div> Will 2014-05-03</span> </h3> I’m jealous of this. Thanks for the write up but would you consider wrapping "bash version 4 only" at the start of the article in strong tags? :) I just bashed (cough) my head against the keyboard for 10 minutes because I’m on bash 3.2.8 (OSX 10.7.5).</p> </div> david 2019-06-07</span> </h3> Will people ever learn that</p> bash --version </span></span></code></pre> is not the way to check the version of your current bash? You’re only checking the version of the bash which is found first in your path, not necessarily the one you’re currently running. You can and should use</p> echo $BASH_VERSION </span></span></code></pre> or just the keyboard shortcut C-x C-v.</p> </div> Andy Balaam 2019-06-07</span> </h3> Thanks david, good point. I would prefer it phrased less rudely though.</p> </div> david 2019-06-07</span> </h3> Maybe, but in these attention dearth times formulating in sharply pointed way is often the only way to get people notice and remember. I found the rest of the article quite good, so it was a disappointment to see this tip at the end. As you can guess it was not the first time I saw it, but in an article like this, people will copy it, as you can also see in the comments above.</p> The problem with such tips is that they will give the right answer most of the time, leading to even more confusion and frustration when they don’t.</p> </div> Akito 2020-03-09</span> </h3> Amazing! Explains everything about associative arrays in a single article. Answered all my questions at once.</p> </div> Andy Balaam 2020-03-10</span> </h3> Hi Akito, glad it helped!</p> </div> Self-Perfection 2020-07-10</span> </h3> There is an error in "Numeric indexing" section Example KEYS=(${!MYMAP[@]})</p> Should actually be KEYS=("${!MYMAP[@]}")</p> otherwise keys with spaces would split to separate array items.</p> </div> Andy Balaam 2020-07-10</span> </h3> Awesome, thank you Self-Perfection – I have fixed it.</p> </div> Andy Balaam 2014-05-08</span> </h3> Thanks Will, updated. Sorry you can’t use it!</p> </div> R Sriram 2020-12-02</span> </h3> Thanks a million for the page and examples. It’s been a L.O.N.G time since I went to the net for ‘just bash’ questions (:=), so it was great to hear that bash now has ass.arrays. I used to do a lot of bash+cmdline-perl (perl -e) to do what a simple ass.array in bash could have done. Now, I have my task cut out. The way you have written the examples is just as one developer talks to another.. Simple, neat, to the point. Thanks again.</p> </div> BC 2020-12-19</span> </h3> Nice article – thanks</p> </div> bash dynamic (variable) variable names 2022-06-14</span> </h3> […] [VERY GOOD TUTORIAL!] *****https://www.artificialworlds.net/blog/2012/10...</a> […]</p> </div> Associative Arrays in Shell Scripts 2022-06-14</span> </h3> […] I found a nice, short tutorial on this here. […]</p> </div> osirisgothra 2014-10-30</span> </h3> Its good to note that:</p> >item=( [item1]="one" [item2]="two )</p> if done on a un[define]d variable, will treat it like an -a instead of an -A, which causes the last entry only to be recognized as the first indexer (zero) unless, of course, those items have value,</p> in the above example, if the variables $item1 and $item2 are un[define]d, then the result would be:</p> > declare -p item item=([0]="two")</p> this happened because undeclared variables have an implicit value of 0 when used as an indexer, it would be so these two lines are identical:</p> >item=( [item1]="one" [item2]="two ) >declare -p item item=([0]="two")</p> >item=( [0]="one" [0]="two ) >declare -p item item=([0]="two")</p> As you can see on the second line, the index ‘0’ gets defined twice, of course the last being the final value for that index.</p> The case is quite different if you have defined values for $item1 and $item2:</p> >item1=12 >item2=24 >item=( [item1]="one" [item2]="two )</p> then you get this:</p> > declare -p item item=( [12]="one" [24]="two )</p> and verify it like this:</p> >echo ${item[12]} one >echo ${item[24]} two</p> Of course, if you had already had values in the other index 0, it would have been erased by this though not touching index 0 you are still resetting the value of the variable —– unless you used += instead of =. Furthermore, if the values of $item1 and $item2 were not integers (strings), the values would go back to being implicitly 0 again.</p> This is important because many programmers expect that because integer arrays are implicit, that the associative arrays should be</em> too. And it even appears that way if the array was [declare]d one previously.</p> As a RULE, it is good to just declare ALL variables. I make it a habit to use "shopt -o -s nounset" in my scripts. However, interactive scripts like .bashrc or completion scripts do not always have this luxury, because it’s a pain to set it, and then unset it, also saving the value which is overhead in the sense of time taken to implement/reimplement each time. In those cases, hopefully the habit of doing it in scripts rubs off on you enough to have it done in the interactive ones as well :)</p> Hope that helped (someone) this font is so small i can hardly read it for some reason today, so if i made a mistake that’s why ( too lazy to zoom :) ) <– double chin!</p> </div> Andy Balaam 2014-10-30</span> </h3> Thanks osirisgothra, very helpful.</p> </div> qwe 2015-01-10</span> </h3> great article!</p> </div> Running Dojo 1.7+ DOH unit tests on the command line with Rhino Andy Balaam — Tue, 09 Oct 2012 16:19:38 +0000 To run your own DOH</a>-based unit tests on the command line using Rhino</a>:</p> NOTE: this is Dojo 1.7 and above. For 1.6, there was a whole other cryptic incantation.</strong></p> Project layout</h3> Imagine your code is somewhere different from dojo, and another library you use is somewhere else:</p> C:/code/mycode/org/me/mytests/ ... mytestmodule.js ... C:/code/mycode/org/them/nicelib/ ... C:/libs/dojo/dojo/ ... dojo.js ... dijit/ ... dojox/ ... util/doh/ ... main.js ... </pre> Config file</h3> Yes, you need a config file. Imagine it's at C:/code/mycode/dohconfig.js and it looks like this:</p> require({ paths: { "org/me" : "../../../code/mycode/org/me", "org/them" : "../../../code/mycode/org/them/nicelib" } }); </pre> Command line</h3> Now you can run your tests like this:</p> java -jar C:/libs/dojo/util/shrinksafe/js.jar C:/libs/dojo/dojo/dojo.js baseUrl=file:///C:/libs/dojo/dojo load=file:///C:/code/mycode/dohconfig.js load=doh test=org/me/mytests/mytestmodule </pre> Explanation</h3> java -jar - run Java and execute a JAR file.</li> C:/libs/dojo/util/shrinksafe/js.jar - path to the JAR file that is Rhino, a JavaScript interpreter written in Java (and included in Dojo's source distribution).</li> C:/libs/dojo/dojo/dojo.js - the Dojo "loader" - unlike in 1.6 and earlier, you don't run DOH's runner.js. Instead you run dojo.js and pass "load=doh" as an argument.</li> baseUrl=file:///C:/libs/dojo/dojo - a URL form of the location of the directory that contains dojo.js.</li> load=file:///C:/code/mycode/dohconfig.js - the location of your config file, which defines the "paths" variable, previously (in 1.6) known as registerModulePaths. This variable helps Dojo find your code based on its module name (here "org/me").</li> load=doh - after you've read (actually, executed) the config file, execute DOH.</li> test=org/me/mytests/mytestmodule - the module name of your test (not the path - a module name which can be found using the lookups in the paths defined in your config file).</li> </ul> Comments</h2> rajesh 2013-04-16</span> </h3> Hi, Is it possible to load the dojo into the java application with the help of Rhino adn envjs? i’m in a need to render the dojo charts on server side. when trying to load the dojo into java via rhino, an Exception rises which is as follows, "org.mozilla.javascript.EcmaError: TypeError: Cannot read property "parentNode" from undefined " Im struggling to find a way to load Dojo. Please help me</p> </div> Andy Balaam 2013-04-16</span> </h3> Hi rajesh, I’d be surprised if that worked. I suggest using a hidden web browser control?</p> </div> m 2012-10-16</span> </h3> Excellent article…is there a way to read the unit test results from the command line and log them to a text/xml file? Thanks</p> </div> Andy Balaam 2012-10-17</span> </h3> Thanks! To log the output into a text file, just add " > log.txt" at the end of the command (or is that not what you meant?)</p> </div> Ricardo Lucio 2013-01-21</span> </h3> Excellent article!!</p> One question: when I try to run a test that touch the dom (e.g.: when I create a dojox.mobile.Button in my test) it complains that window is undefined. After some tests I noticed that the rhino engine didn’t provide a dom implementation. Am I missing someting?</p> </div> Andy Balaam 2013-01-22</span> </h3> Hi Ricardo, thanks! You are not missing something. There is no DOM when running in Rhino. env.js (http://www.envjs.com/</a>) might do what you need, if you want to do tests of DOM-manipulating code on the command line. There’s still a lot you can’t do without a real-live browser, though.</p> </div> Lambda functions timeline Andy Balaam — Fri, 14 Sep 2012 13:34:25 +0000 I did a talk at work about lambda functions (anonymous functions), and something possessed me to make a timeline of when they were introduced into various languages. Some languages were born with them, and some grew them later - in the latter case I give the language version and date in which they were introduced.</p> It's probably entirely wrong, but here it is:</p> </a></p> Note that C# had varying levels of support for lambda functions in different versions. I used the version (3.0) in which Wikipedia describes its support as "full".</p> Goodness in programming languages, part 3 " not doing the same thing more than once Andy Balaam — Wed, 05 Sep 2012 07:36:21 +0000 Posts in this series: Syntax</a>, Deployment</a>, Metaprogramming, Ownership</a></em></p> I'm going to use a word here - don't stop reading: Metaprogramming</strong>. Does the language provide what you need to avoid repeating yourself?</p> Repeating boilerplate code, algorithms and most importantly ideas, slows you down, makes maintenance difficult, and allows all kinds of mistakes to creep in. If a language provides the ability to abstract, name and re-use all the different types of structure it contains, you can avoid harmful repetition.</p> Here are some things I like:</p> Python, JavaScript and Scheme's ability to treat functions like any other object.</strong> A massive step towards sharing code is being allowed to pass around something that can be called without worrying about what it is.</li> Scheme's ability to define an algorithm independently of types.</strong> In Scheme, there is never a need to write another version of the same function because it deals with different types of thing.</li> Python's ability to read and modify classes just like any other object.</strong> Want a class just like your current one, except it logs every method call? Write a function that copies and modifies the class definition.</li> Scheme's ability write code about code.</strong> In Scheme, code is just some nested lists. It's trivial to build and modify code without stepping out of the language.</li> C++'s ability to write code that runs at compile time.</strong> If you can stand the syntax and (lack of) debugging, C++ template metaprogramming allows you to build C++ code at compile time without stepping out of the compiler environment.</li> Scheme and C's macro systems.</strong> Both Scheme and C (and C++) allow you to write macros that build commonly-repeated code. Scheme's syntax for this is much easier to work with.</li> </ul> Until you've experienced the freedom of totally generic code in a language like Scheme it's hard to explain why the "Generics" features of some languages are so lacking. Of course, static typed languages work under different constraints. Would it be possible to write a language with very strong generic programming features, but which still allows static typing and compiling to native, non-generic code? I think so.</p> Tail Call Optimisation in C++ - lightning talk video Andy Balaam — Mon, 30 Jul 2012 08:49:18 +0000 You can watch the Tail Call Optimisation in C++ lightning talk video</a>, which I gave at the ACCU 2012 conference</a> in April.</p> You can also read the (clearer and more correct) writeup I did later: Tail Call Optimisation in C++</a> or the subsequent article published in Overload 109</a>.</p> Comments</h2> Andy Balaam’s Blog » Blog Archive » Lighting talk – Tail call optimisation in C++ 2012-07-30</span> </h3> […] Update: watch the video […]</p> </div> Cool Raspberry Pi Projects Andy Balaam — Fri, 20 Jul 2012 12:19:20 +0000 Someone claimed that everyone who has a Raspberry Pi has done what I did with it: try it out then file it in a drawer marked "must play with later". To try and debunk this claim, here are some cool projects I have seen.</p> Update: I made some videos on how to write your first ever game on your Raspberry Pi</a>!</strong></p> Raspberry Pi Accessories You Already Have!</h3> This guy uses an old PSOne as a screen+audio, a phone battery for power, and a hard drive or mp3 player for storage.</p> </a></p> Raspberry Pi Robot Test!</h3> A robot carrying a RPi on its back, controllable via the Internet. It runs a node.js web server, and receives HTTP requests to control it.</p> </a></p> Raspberry Pi - Quake 3 demo</h3> A playable Quake 3.</p> </a></p> RaspberryPi: XBMC Android Remote Control Demo</h3> A video player, with a mobile phone as the remote control.</p> </a></p> PIE1 -Â– Raspberry Pi Sends Live Images from Near Space</h3> Story</a> | Photos of launch</a> | Photos from space</a></p> Balloon flying up to 40km taking photos using a USB webcam and saving them to an SD card.</p> VNC Viewer on Raspberry Pi</h3> A VNC viewer allowing you to create a thin client/terminal in other rooms of your house.</p> </a></p> Also see the Raspberry Pi Thin Client project</a>, for a small distro containing VMWare, Remote Desktop, Citrix Receiver & a browser.</p> Raspberry Pi running the Fuse ZX Spectrum Emulator</h3> A ZX Spectrum Emulator, taking the nostalgia to the max.</p> </a></p> Make A Raspberry Pi Web Server</h3> Make your own Â£25 web server.</p> </a></p> Voice Controlled Robot using a Raspberry Pi</h3> A voice-controlled robot arm.</p> </a></p> Pi Plays Poppins</h3> A Pi playing the Glockenspeil.</p> </iframe> Pi Plays Poppins</a> from Mike Cook</a> on Vimeo</a>.</p> Raspberry Tank - First Day Out</h3> A tank, currently directly controlled by typing into ssh. They are working on an autonomous version.</p> </a></p> Learn how to program</h3> My own tutorial on how to write computer programs in Python on your Pi. You will make a very simple game.</p> </a></p> Not-yet-implemented ideas</h3> There are lots of people talking about ideas for projects. Here are some I'd like to see come off:</p> Voice-activated wall-mounted screen</li> Advanced alarm clock</li> Motion-activated lights and/or camera</li> Internet radio</li> Coffee machine</a></li> Phone system</li> Web browser on your TV</li> Autonomous boat</a></li> </ul> Scheme 7: Macros video Andy Balaam — Mon, 02 Jul 2012 07:33:08 +0000 Series: Feel the cool</a>, Basics</a>, Closures</a>, Recursion</a>, Quotation</a>, Lambda</a>, Macros.</p> Continuing the series on Scheme, this video explains the ultimate alternative - when nothing else is flexible enough, we can create our own bits of lanugage using macros.</p> </iframe> Slides for Scheme 7: Macros</a></p> Comments</h2> Jennii 2020-05-23</span> </h3> Hi Andy. I did computer science-y stuff years and years ago. After other careers, I retired and thought it would be fun to remember. Your Scheme videos did a wonderful job of nudging my memory. Thanks ever so much.</p> </div> Andy Balaam 2020-06-01</span> </h3> Thanks Jennii, so encouraged to hear that!</p> </div> Scheme 6: Lambda video Andy Balaam — Tue, 26 Jun 2012 16:55:04 +0000 Series: Feel the cool</a>, Basics</a>, Closures</a>, Recursion</a>, Quotation</a>, Lambda, Macros</a>.</p> Continuing the series on Scheme, this video explains the lambda function, which allows you to define anonymous functions. It goes on to bend your mind with 2 examples of the enormous power of functions and closures in Scheme.</p> </iframe> Slides for Scheme 6: Lambda</a></p> Tail Call Optimisation in C++ published in Overload journal Andy Balaam — Fri, 22 Jun 2012 08:53:35 +0000 You read it here first</a>, but now you can have a paper version of "Tail Call Optimisation in C++", published almost as-is, in Overload 109</a> the journal of ACCU</a>.</p> Scheme 5: Quotation video Andy Balaam — Thu, 21 Jun 2012 07:26:10 +0000 Series: Feel the cool</a>, Basics</a>, Closures</a>, Recursion</a>, Quotation, Lambda</a>, Macros</a>.</p> Continuing the series on Scheme, this video explains on of the most powerful and unfamiliar features of Scheme: quotation. Quoting allows us to talk about and manipulate code in code, as easily as we deal with strings or numbers.</p> </iframe> Slides for Scheme 5: Quotation</a></p> Scheme 4: Recursion - how to write functions in Scheme video Andy Balaam — Sat, 16 Jun 2012 18:47:55 +0000 Series: Feel the cool</a>, Basics</a>, Closures</a>, Recursion, Quotation</a>, Lambda</a>, Macros</a>.</p> Continuing the series on Scheme, this video describes the way you write code in Scheme: recursion. Recursive functions can be pretty efficient in Scheme, if you structure them right, and the video covers the concept of a tail call, which is the most important way of doing this.</p> </iframe> Slides for Scheme 4: Recursion</a></p> Scheme 3: Closures video Andy Balaam — Fri, 01 Jun 2012 18:38:22 +0000 Series: Feel the cool</a>, Basics</a>, Closures, Recursion</a>, Quotation</a>, Lambda</a>, Macros</a>.</p> Continuing the series on Scheme, this video describes an important feature of functions in Scheme: closures. Closures allow functions to carry their "environment" around with them, and even change it later. This allows all kinds of exciting things, including implementing something similar to classes in other languages.</p> </iframe> Slides for Scheme 3: Closures</a></p> Scheme 2: Basics - pairs and lists video Andy Balaam — Fri, 25 May 2012 21:31:17 +0000 Series: Feel the cool</a>, Basics, Closures</a>, Recursion</a>, Quotation</a>, Lambda</a>, Macros</a>.</p> Continuing the series on Scheme, this video describes some of the things you will need to understand to be able to write real programs. I introduce pairs, lists, and navigation through lists using recursive procedures, before discussing the map and foldr procedures, which you will use in most programs.</p> </iframe> Slides for Scheme 2: basics - pairs and lists</a></p> Scheme 1: Feel the cool video Andy Balaam — Tue, 22 May 2012 17:16:24 +0000 Series: Feel the cool, Basics</a>, Closures</a>, Recursion</a>, Quotation</a>, Lambda</a>, Macros</a>.</p> Here's me trying to persuade you to be interested in a 40-year-old programming language, called Scheme, which is a type of Lisp:</p> </iframe> Slides for Scheme 1: Feel the cool</a></p> Comments</h2> Johnny 2023-02-27</span> </h3> Hello Andy, What implementation of Scheme are you using in this fantastic video, is it the one you typically use? I would like to begin learning this implementation. Thank you so much for your time.</p> </div> Andy Balaam 2023-02-27</span> </h3> Hi Johnny, I’m using mzscheme: https://github.com/racket/mzscheme</a> . Enjoy!</p> </div> Pete Barber 2012-06-12</span> </h3> Cool:–) just watched part 1. Enjoyable intro. Look fwd to watching the next couple. Other than fun & cool what was your motivation for learning scheme (writing emacs macros?)</p> </div> Andy Balaam 2012-06-13</span> </h3> Hi Pete, thanks! A friend recommended Structure and Interpretation of Computer Programs, and after a couple of pages I was hooked. Also I am interested, one day, in writing my own language, so learning Lisp was a necessary prerequisite…</p> </div> Generalising tail call optimised C++ Andy Balaam — Tue, 08 May 2012 22:41:50 +0000 This series: Lightning talk</a>, Explanation</a>, Performance</a>, Generalisation.</p> In previous posts I discussed the construction of some C++ that does the same job that the tail call optimisation does in some other languages. The example code given showed the case where we know that every function in the recursion will take two long integer parameters, and return a long as well.</p> In fact, it is perfectly possible to generalise this code to cover more complex cases. Fortunately, the trampoline function doesn't need to know the arguments taken by the functions being called, only the return value. It looks like this:</p> template<typename RetT> const RetT trampoline_templ( std::auto_ptr< IAnswer<RetT> > answer ) { while( !answer->finished() ) { answer = answer->tail_call()(); } return answer->value(); }</pre> So we can call this with the type of the return value as our template parameter, and supply an object which satisfies the IAnswer<RetT> interface:</p> template<typename RetT> class IAnswer { public: virtual const bool finished() const = 0; virtual const ICallable<RetT>& tail_call() const = 0; virtual const RetT value() const = 0; };</pre> where ICallable looks like this:</p> template<typename RetT> class ICallable { public: typedef std::auto_ptr< IAnswer<RetT> > AnswerPtr; virtual AnswerPtr operator()() const = 0; };</pre> The concrete classes that implement these interfaces need to know the types (and number) of the arguments, but that's ok because they only get instantiated by code that would otherwise (in the standard, non-tail-call recursion case) call the functions themselves. In the toy case we are using of repeatedly adding up numbers to multiply by two, the outer function looks like this:</p> const long times_two_tail_call_templ( const long n ) { typedef Answer2<long, long, long> AnswerType; return trampoline_templ( AnswerType::newFn( times_two_tail_call_impl, 0, n, 0 ) ); }</pre> and the inner one looks like this:</p> std::auto_ptr< IAnswer<long> > times_two_tail_call_impl( const long acc, const long i ) { typedef Answer2<long, long, long> AnswerType; if( i == 0 ) { return return AnswerType::newAns( acc ); } else { return AnswerType::newFn( times_two_tail_call_impl, acc + 2, i - 1, 0 ); } }</pre> Both of the above use static convenience methods newFn and newAns that I have defined on Answer2 to create smart pointers to newly-allocated Answer2 objects. newAns creates an Answer2 that contains a final answer, and newFn creates an Answer2 specifying another function (with arguments) to call.</p> Answer2 looks like this:</p> template<typename RetT, typename Arg1T, typename Arg2T> class Answer2 : public IAnswer<RetT> { private: typedef FnPlusArgs2<RetT, Arg1T, Arg2T> FnArgs; typedef std::auto_ptr< IAnswer<RetT> > AnswerPtr; const bool finished_; const FnArgs tail_call_; const RetT value_; private: Answer2( const bool finished, const FnArgs tail_call, const RetT value ) : finished_( finished ) , tail_call_( tail_call ) , value_( value ) { } static AnswerPtr newPtr( const bool finished, const FnArgs tail_call, const RetT value ) { return AnswerPtr( new Answer2<RetT, Arg1T, Arg2T>( finished, tail_call, value ) ); } public: static AnswerPtr newFn( const typename FnArgs::fn_type fn, const Arg1T arg1, const Arg2T arg2, const RetT zero_val ) { return newPtr( false, FnArgs( fn, arg1, arg2 ), zero_val ); } static AnswerPtr newAns( const RetT value ) { return newPtr( true, FnArgs::null(), value ); } virtual const bool finished() const { return finished_; }; virtual const FnArgs& tail_call() const { return tail_call_; }; virtual const RetT value() const { return value_; }; };</pre> and uses FnPlusArgs2, which looks like this:</p> template<typename RetT, typename Arg1T, typename Arg2T> class FnPlusArgs2 : public ICallable<RetT> { private: typedef typename ICallable<RetT>::AnswerPtr AnswerPtr; public: typedef AnswerPtr (*fn_type)( const Arg1T, const Arg2T ); private: const fn_type fn_; const Arg1T arg1_; const Arg2T arg2_; public: FnPlusArgs2( const fn_type fn, const Arg1T arg1, const Arg2T arg2 ) : fn_( fn ) , arg1_( arg1 ) , arg2_( arg2 ) { } virtual AnswerPtr operator()() const { return fn_( arg1_, arg2_ ); } static FnPlusArgs2<RetT, Arg1T, Arg2T> null() { return FnPlusArgs2<RetT, Arg1T, Arg2T>( NULL, 0, 0 ); } };</pre> I have continued with the 2 long arguments, and long return value example here, but with the above code it is possible to construct recursive code using more than one function, and the functions can have different numbers of arguments, and different argument types, so long as they all co-operate to produce an answer of the required type. The Source code for this article</a> includes an example, in the file tail_call_templ_2fns.cpp</a>, of two different functions that call each other recursively, and take different arguments, using the trampoline function and interfaces listed above, and Answer3 and FnPlusArgs3 class templates similar to Answer2 and FnPlusArgs2 shown above. Implementing the N-args case using variadic templates (C++11) or template meta-programming is left as an exercise for the reader.</p> This more realistic case where the number and types of arguments are not known beforehand forces us to use dynamic memory to store our AnswerN objects, and causes more pointer dereferences and virtual function calls, and these do hurt performance. In tests on my machine, this code ran approximately 10 times slower than the version using only stack memory. Perhaps we C++ programmers should comfort ourselves that many languages supporting tail-call optimisation require dynamic memory, virtual functions and pointer indirection to do absolutely everything.</p> JavaScript WTFs Videos Andy Balaam — Fri, 04 May 2012 23:16:46 +0000 I recorded some videos of my JavaScript WTFs presentations:</p> </iframe> </iframe> </iframe> </iframe> </iframe> </iframe> You can get the JavaScript WTFs slides</a>.</p> Closures in Scheme Andy Balaam — Fri, 04 May 2012 15:11:38 +0000 Update: watch the video</a></strong></p> In this series on Scheme: Intro</a>, Basics</a>, Closures.</p> Here's a presentation I did recently, on Closures in the Scheme programming language. Closures are the way the environment in which a function was created hangs around with it as long as the function itself exists. They allow many flexible programming styles, and in this presentation I demonstrate how a simple object-oriented class structure can be implemented using closures.</p> Closures in Scheme</a></p> Performance of tail call optimised C++ Andy Balaam — Mon, 30 Apr 2012 21:21:10 +0000 This series: Lightning talk</a>, Explanation</a>, Performance, Generalisation</a>.</p> After I wrote a version of tail-call optimised code in C++</a> I became interested in its performance relative to normal recursion. The tail call version can process arbitrarily large input, but how much do you pay for that in terms of performance?</p> Recap: there are 4 different versions of our function, called times_two. The first, "hardware", uses the "*" operator to multiply by 2. The second, "loop" uses a for loop to add up lots of 2s until we get the answer. The third, "recursive", uses a recursive function to add up 2s. The fourth, "tail_call" is a reimplementation of "recursive", with a manual version of the tail call optimisation - see the original article for details.</p> Let's look first at memory usage. Here is the stack memory usage over time as reported by Massif</a> of calling the four functions for a relatively small input value of 100000:</p> </p> The recursive function uses way more memory than the others (note the logarithmic scale), because it keeps all those stack frames, and the tail_call version takes much longer than the others (possibly because it puts more strain on Massif?), but keeps its memory usage low. Let's look at how that affects its performance, for different sizes of input:</p> </p> For these much larger input values, the recursive and tail_call functions take similar amounts of time, until the recursive version starts using all the physical memory on my computer. At this point, its execution times become huge, and erratic, whereas the tail_call function plods on, working fine.</p> So the overhead of the infrastructure of the tail call doesn't have much impact on execution time for large input values, but it's clear from the barely-visible green line at the bottom that using a for-loop with a mutable loop variable instead of function calls is way, way faster, with my compiler, on my computer, in C++. About 18 times faster, in fact.</p> And, just in case you were wondering: yes those pesky hardware engineers with their new-fangled "*" operator managed to defeat all comers with their unreasonable execution times of 0 seconds every time (to the nearest 10ms). I suppose that shows you something.</p> Tail call optimisation in C++ Andy Balaam — Mon, 30 Apr 2012 20:03:21 +0000 This series: Lightning talk</a>, Explanation, Performance</a>, Generalisation</a>.</p> Some programming languages make recursive programming more practical by providing the tail call optimisation. For a tiny talk at the recent ACCU conference I looked at how we might do something similar in C++.</p> Update:</strong> Source code for this article</a> is available.</p> Multiplying by two</h3> Here's a toy problem we will use as our example.</p> Imagine for a second that you want to write a function that multiplies a number by two. OK, we can do that:</p> long times_two_hardware( long value ) { return value * 2; }</pre> Now imagine that you don't have the "*" operator. We're going have to use "+":</p> long times_two_loop( long value ) { long ret = 0; for ( long i = 0; i < value; ++i ) { ret += 2; } return ret; }</pre> (Obviously, this is just a silly example designed to be easy to follow.)</p> Now imagine that you read somewhere that state was bad, and you could always replace a loop with recursion. Then you might get something like this:</p> long times_two_naive_recursive( long value ) { if ( value == 0 ) { return 0; } else { return 2 + times_two_naive_recursive( value - 1 ); } }</pre> This is fine, but what happens when you run it for a large input?</p> $ ulimit -S -s 16 $ ./times_two naive_recursive 100000 Segmentation fault</pre> Note that I set my stack size to be very small (16K) to make the point - actually, this will run successfully for very large arguments, but it will eat all your memory and take a long time to finish.</p> Why does this fail? Because every time you call a function, the state of the current function is saved, and new information is pushed onto the stack about the new function. When you call a function from within a function multiple times, the stack grows and grows, remembering the state all the way down to the place where you started.</p> So is programming like this useless in practice?</p> Tail call optimisation</h3> No, because in several programming languages, the compiler or interpreter performs the "tail call optimisation".</p> When you call a function from within some other code, you normally need the state of the current code to be preserved. There is a special case where you don't need it, though, and this is called a tail call. A tail call is just the situation where you call a function and immediately return its return value as your return value. In this case, we don't need any of the state of the current code any more - we are just about to throw it away and return.</p> The tail call optimisation throws away this unneeded state before calling the new function, instead of after.</p> [In practice, in compiled code, this involves popping all the local variables off the stack, pushing the new function parameters on, and jmping to the new function, instead of calling it. This means that when we hit the ret at the end of the new function, we return to the original caller, instead of the location of the tail call.]</p> Many recursive functions can be re-cast as tail-call versions (sometimes called iterative versions). The one we're looking at is one of those. Here is the tail-call version:</p> long times_two_recursive_impl( long total, long counter ) { if ( counter == 0 ) { return total; } else { return times_two_recursive_impl( total + 2, counter - 1 ); } } long times_two_recursive( long value ) { return times_two_recursive_impl( 0, value ); }</pre> It consists of an outer function times_two_recursive which just hands off control to the inner function times_two_recursive_impl. The inner function uses a counter variable and calls itself recursively, reducing that counter by one each time, until it reaches zero, when it returns the total, which is increased by 2 each time.</p> The key feature of this implementation is that the recursive function times_two_recursive_impl uses a tail call to do the recursion: the value of calling itself is immediately returned, without reference to anything else in the function, even temporary variables.</p> So, let's see what happens when we compile and run this:</p> $ ulimit -S -s 16 $ ./times_two recursive 100000 Segmentation fault</pre> Did I mention that C++ doesn't do the tail call optimisation?*</p> * Tail call optimisation isn't in the C++ standard. Apparently, some compilers, including MS Visual Studio and GCC, do provide tail call optimisation under certain circumstances (when optimisations are enabled, obviously). It is difficult to implement for all cases, especially in C++ since destruction of objects can cause code to be executed where you might not have expected it, and it doesn't appear to be easy to tell when a compiler will or will not do it without examining the generated assembly language. Languages which have this feature by design, like Scheme (and D?) can do it more predictably.</p> </blockquote> So how would we write code that is tail call optimised in C++? Possibly of more interest to me personally: if we were generating C++ as the output format for some other language, what code might we generate for tail call optimised functions?</p> Tail call optimised C++</h3> Let's imagine for a second we have some classes, which I'll define later. FnPlusArgs holds a function pointer, and some arguments to be passed to it. Answer holds on to one of 2 things: either a FnPlusArgs to call later, or an actual answer (return value) for our function.</p> Now we can write our function like this:</p> Answer times_two_tail_call_impl( long acc, long i ) { if ( i == 0 ) { return Answer( true, null_fn_plus_args, acc ); } else { return Answer( false, FnPlusArgs( times_two_tail_call_impl, acc + 2, i - 1 ), 0 ); } } long times_two_tail_call( long n ) { return trampoline( Answer( false, FnPlusArgs( times_two_tail_call_impl, 0, n ), 0 ) ); }</pre> This has the same structure as times_two_recursive, if a little more verbose. The important point to note, though, is that times_two_tail_call_impl doesn't call itself recursively. Instead, it returns an Answer object, which is a delegate saying that we have more work to do: calling the provided function with the supplied arguments.</p> The Trampoline</h3> All we need now is some infrastructure to call this function, and deal with its return value, calling functions repeatedly until we have an answer. This function is called a "trampoline", and you can sort of see why:</p> long trampoline( Answer answer ) { while ( !answer.finished_ ) { answer = answer.tail_call_(); } return answer.value_; }</pre> While the answer we get back tells us we have more work to do, we call functions, and when we're finished we return the answer.</p> Now all we need to get this working is the definition of Answer and FnPlusArgs:</p> struct Answer; typedef Answer (*impl_fn_type)( long, long ); struct FnPlusArgs { impl_fn_type fn_; long arg1_; long arg2_; FnPlusArgs( impl_fn_type fn, long arg1, long arg2 ); Answer operator()(); }; impl_fn_type null_fn = NULL; FnPlusArgs null_fn_plus_args( null_fn, 0, 0 ); struct Answer { bool finished_; FnPlusArgs tail_call_; long value_; Answer( bool finished, FnPlusArgs tail_call, long value ); }; FnPlusArgs::FnPlusArgs( impl_fn_type fn, long arg1, long arg2 ) : fn_( fn ) , arg1_( arg1 ) , arg2_( arg2 ) { } Answer FnPlusArgs::operator()() { return fn_( arg1_, arg2_ ); } Answer::Answer( bool finished, FnPlusArgs tail_call, long value ) : finished_( finished ) , tail_call_( tail_call ) , value_( value ) { }</pre> The only notable thing about this is that we use operator() on FnPlusArgs to call the function it holds.</p> Results</h3> Now, when we run this code, we get what we wanted:</p> $ ulimit -S -s 16 $ ./times_two tail_call 100000 200000</pre> (I.e. it doesn't crash.)</p> So, it turns out that the tail call optimisation is just a while loop. Sort of.</p> Comments</h2> C++ Tail Recursion Using 64-bit variables – Part 2 | Coding Adventures 2015-05-15</span> </h3> […] In my previous postÂ I talked aboutÂ recursion problems in a Fibonacci function using 64-bit variables as function parameters , compiling with the Microsoft Visual C++ compiler. It turned out that while thatÂ worked well using 32-bit types it didn’t really usingÂ 64-bit ones. Just as a reminder, Tail RecursionÂ is an optimization performed by the compiler. It is the process ofÂ transforming certain types of tail callsÂ into jumps instead of function calls. More about tail recursion here. […]</p> </div> C++ Tail Recursion Using 64-bit variables – Part 2 – Coding Adventures 2020-01-04</span> </h3> […] In my previous post I talked about recursion problems in a Fibonacci function using 64-bit variables as function parameters, compiled using the Microsoft Visual C++ compiler. It turned out that while tail recursion was enabled by the compiler using 32-bit types it didn’t really when switching to 64-bit ones. Just as a reminder, Tail Recursion is an optimization performed by the compiler. It is the process of transforming certain types of tail calls into jumps instead of function calls. More about tail recursion here. […]</p> </div> Konstantin 2013-01-15</span> </h3> Function times_two_tail_call( long n ) run tail_call( Answer(… which is not defined in example I suppose that function trampoline should run instead or corresponding typedef for tail_call as trampoline should be defined</p> </div> Andy Balaam 2013-01-15</span> </h3> Hi Konstantin, thanks – that’s right – it should be trampoline inside times_two_tail_call. I’ll change it.</p> </div> Lighting talk - Tail call optimisation in C++ Andy Balaam — Fri, 27 Apr 2012 20:03:00 +0000 This series: Lightning talk, Explanation</a>, Performance</a>, Generalisation</a>.</p> Update: watch the video</a></strong></p> Here's the lightning talk I gave at the ACCU 2012 Conference</a>:</p> Tail Call Optimisation in C++</a></p> It's about how you would generate C++ code that represents a recursive function, without running out of stack space.</p> Scheme: pairs, lists and recursion Andy Balaam — Tue, 24 Apr 2012 09:01:00 +0000 Update: watch the video</a></strong></p> In this series on Scheme: Intro</a>, Basics, Closures</a>.</p> Here's a presentation I did recently, on the basics of the Scheme programming language.</p> Scheme: pairs, lists and recursion</a></p> What timezone Python thinks I'm in Andy Balaam — Wed, 18 Apr 2012 22:50:37 +0000 sudo aptitude install python-dateutil python -c "import dateutil.tz; import datetime; print datetime.datetime.now( dateutil.tz.tzlocal() ).tzname()" </pre> Freeview dvb-t scan gives "tuning failed!!!" after 18th April switchover Andy Balaam — Wed, 18 Apr 2012 08:01:22 +0000 Usually when the channel lineup changes, I just have to do this: $ scan /usr/share/dvb/dvb-t/uk-CrystalPalace > ~/.mplayer/channels.conf </pre>But this time it is failing with:>>> tune to: 505833000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE WARNING: >>> tuning failed!!! >>> tune to: 505833000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE (tuning failed) WARNING: >>> tuning failed!!! </pre>(snip)</p> However, a more low-level scan with w_scan seems to be working today:</p> $ sudo apt-get install w-scan # Install w_scan $ w_scan -X > ~/.mplayer/channels.conf </pre> I still can't see BBC Four at the moment...</p> Update</strong>It looks like I have to scan when the channels are actually on air. I did this: $ cd ~/.mplayer $ w_scan -X > channels.conf.morning # I ran this in the morning $ w_scan -X > channels.conf.evening # I ran this in the evening $ cat channels.conf.morning channels.conf.evening | sort | uniq > channels.conf </pre></p> Now I have both BBC Four and CBeebies, which pretty much covers us!</p> Comments</h2> Another way to scan DVB channels: w_scan | The Angry Technician 2013-05-08</span> </h3> […] messages after that, and no scan results. I’m not sure exactly what is causing this yet, but somebody else in the UK noticed this too, and suspects it began right after a very recent transmitter retune. Luckily, he also had a […]</p> </div> Myatu 2012-06-12</span> </h3> Two words: THANK YOU! :)</p> </div> Scheme: feel the cool Andy Balaam — Fri, 13 Apr 2012 16:44:16 +0000 Update: watch the video</a></strong></p> In this series on Scheme: Intro, Basics</a>, Closures</a>.</p> Here's a presentation I did recently, introducing the Scheme programming language (a dialect of Lisp).</p> Scheme: feel the cool</a></p> Metaprogramming</a> is very powerful, but is usually so hard that we shy away from it. In Scheme, most metaprogramming is just "normal" programming - you can do it in the same language, and in a macro language that is very similar the normal language.</p> Comments</h2> Mitch 2013-02-12</span> </h3> Hi, I would like to know if you are using the Pretty Big language in drracket (in your scheme tutorial videos)? Thank you!</p> </div> Andy Balaam 2013-02-13</span> </h3> Hi, no I’m using mzscheme, which is apparently the backwards-compatible environment, almost identical to older versions of PLT Scheme.</p> </div> Anushka Yadav 2023-04-29</span> </h3> I watched your video in 2023, 10 years later thought to comment it was really nice …</p> </div> Andy Balaam 2023-05-01</span> </h3> Thank you!</p> </div> abpccpba 2013-01-01</span> </h3> Just finished your first video about how cool Scheme is. Good job. Now to watch the rest.</p> </div> Andy Balaam 2013-01-02</span> </h3> Thanks abpccpba, enjoy them :)</p> </div> JavaScript WTFs Andy Balaam — Fri, 30 Mar 2012 13:41:58 +0000 I recently did a presentation at work on "surprising" things you may encounter in JavaScript. I present it here for your delectation:</p> JavaScript WTFs</a></p> Sorting arrays was the one that wasted me the most time. Arrays in JavaScript are a neat hack on top of objects. Given that, presumably, all modern implementations have a separate type for an array as opposed to an object, you would hope that at some point they would grow up into something really different in the language.</p> You can watch the videos here: JavaScript WTFs videos</a></p> Comments</h2> jeffjohnsondicc 2024-11-28</span> </h3> JavaScript quirks never fail to surprise! Sorting arrays being tied to their object-like nature is a classic “gotcha.” Your presentation must have been enlightening for many devs navigating these nuances. It’s a reminder that even mature languages have their peculiarities, keeping us learning and on our toes. Great insight! 🚀 Regards, DICC INSTITUTE</a></p> </div> andyb 2024-11-28</span> </h3> Thanks, glad you enjoyed it!</p> </div> Goodness in programming languages, part 2 " getting your code running Andy Balaam — Thu, 08 Mar 2012 10:03:21 +0000 Posts in this series: Syntax</a>, Deployment, Metaprogramming</a>, Ownership</a></em></p> The fancy word for what I'm talking about here is Deployment</strong>. How easy is it, once you've written your code, to get it running on someone else's computer? What barriers are there to someone else using your program?</p> Examples of potential barriers include:</p> Having to download some other program first, e.g. a runtime or some kind of dependency.</li> The actual program being a huge download.</li> Something mysteriously not working because of different versions of something on the person's computer.</li> The program simply not working on the operating system or machine architecture of the computer.</li> </ul> Anything that can be done to remove these barriers makes my life as the supporter of the program easier, and makes it more likely people will use it.</p> Here are some things I like:</p> Java's ability to run almost anywhere.</strong> Once you have the runtime, Java code is relatively easy to get running on many different operating systems and architectures, using almost-identical code. Other runtime-based languages are also strong here.</li> Java and Python's large built-in libraries.</strong> Both Java and Python include large amounts of functionality in their standard libraries, reducing the need to depend on third-party components.</li> Python and Perl's ability to work out of the box on most Linux systems.</strong> If you are developing for Linux, you can pretty much guarantee that an up-to-date runtime will be available, meaning no dependencies at all are needed.</li> Many languages' easy integration with Linux packaging.</strong> Most of the above barriers disappear if you can install dependencies using your operating system's built-in package manager.</li> Quickly</a>'s easy way to build your program into the packaging system.</strong> Things really become easy if your program itself is integrated into the packaging system.</li> C and C++'s lack of dependencies.</strong> It is possible to write C and C++ programs that depend on nothing except standard runtime libraries, which are almost guaranteed to exist on most machines.</li> </ul> One way to handle dependencies is to package them together with your code. This is what most corporate Java software does - the specific Java runtime that is known to work is packaged with the program itself. This makes for big downloads, and defeats the concept of providing a single package for all platforms, but it does work. It also makes for huge headaches to do with licensing, and is often impossible for software producers who don't employ legions of lawyers. It also feels bad and wrong.</p> When packaging and deploying software, I subscribe to the philosophy of "Find the dependencies - and eliminate them</a>". Until someone can click a single button and have your software running, you haven't finished.</p> Comments</h2> Andy Balaam’s Blog » Blog Archive » Goodness in programming languages, part 1 – syntax and layout 2012-03-08</span> </h3> […] Posts in this series: Syntax, Deployment […]</p> </div> Building jEdit from source Andy Balaam — Sun, 12 Feb 2012 21:17:11 +0000 I have recently agreed to take over development of the JSLint</a> plugin</a> for my favourite editor, jEdit</a>.</p> To start developing I decided to build jEdit and its plugins from source, which wasn't as easy as I would have expected (thanks to deffbeff</a> for some pointers). A few small changes to the setups for the plugins would make it much easier - at some point I may try to help out there, but for now, here is how I built jEdit and the set of plugins I use regularly:</p> sudo apt-get install openjdk-6-jdk ant git-core subversion cd ~/code/public mkdir jedit cd jedit svn co https://jedit.svn.sourceforge.net/svnroot/jedit/jEdit/trunk jEdit cd jEdit; ant; cd .. mkdir plugins</pre> Now I made a file inside plugins called build.properties which looked like this:</p> install.dir = ../../jEdit/build/settings/jars jedit.plugins.dir = ../../jEdit/build/settings/jars jedit.install.dir = ../../jEdit/build build.support = ../build-support</pre> And then continued like this:cd plugins svn co https://jedit.svn.sourceforge.net/svnroot/jedit/build-support/trunk build-support</p> svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/ErrorList/trunk ErrorList svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/CommonControls/trunk CommonControls svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/InfoViewer/trunk InfoViewer svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/Console/trunk Console svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/BufferList/trunk BufferList svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/FindFile/trunk FindFile svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/GnuRegexp/trunk GnuRegexp svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/OpenIt/trunk OpenIt svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/RecentBufferSwitcher/trunk RecentBufferSwitcher svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/Sessions/trunk Sessions svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/WhiteSpace/trunk WhiteSpace svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/ScriptEnginePlugin/trunk ScriptEnginePlugin svn co https://jedit.svn.sourceforge.net/svnroot/jedit/plugins/JavascriptScriptEnginePlugin/trunk JavascriptScriptEnginePlugin</p> git clone git://jedit.git.sourceforge.net/gitroot/jedit/ProjectViewer git clone git://github.com/andybalaam/jslint-plugin-for-jedit.git jslint</p> cd ErrorList; ant; cd .. cd CommonControls; ant; cd .. cd InfoViewer; ant; cd .. cd ProjectViewer; ant; cd .. cd Console; ant; cd .. cd BufferList; ant; cd .. cd FindFile; ant; cd .. cd GnuRegexp; ant; cd .. cd OpenIt; ant; cd .. cd RecentBufferSwitcher; ant; cd .. cd Sessions; ant; cd .. cd WhiteSpace; ant build; cd .. cd ScriptEnginePlugin; ant; cd .. cd JavascriptScriptEnginePlugin; ant; cd .. cd jslint; ant; cd ..</pre></p> To find out the dependencies of all plugins and therefore what order to build them, I did this:find ./ -name "*.props" | xargs grep "depend.*plugin"</pre>Surely something should do this for me automatically?</p> The problem I haven't solved yet is that the FTP plugin requires JUnit. I could have copied the junit.jar into the jars dir and I think it would have worked, but I want to do it "properly"...</p> Finally, I made a .desktop shortcut file in ~/.local/share/applications/jedit.desktop like this:</p> [Desktop Entry] Name=jEdit GenericName=Programmer's Text Editor Comment=Edit text files Exec=java -jar /home/andy/code/public/jedit/jEdit/build/jedit.jar -settings=/home/andy/code/public/jedit/jEdit/build/settings %U Icon=/home/andy/code/public/jedit/jEdit/doc/jedit.png Terminal=false Type=Application Categories=Development;TextEditor; StartupNotify=true MimeType=text/plain; StartupWMClass=org-gjt-sp-jedit-jEdit</pre> Which enabled me to run the jEdit I had built from my system menu.</p> Note that I am running jEdit under OpenJDK because Sun/Oracle Java is annoying to get hold of at the moment, and generally somewhat evil. Thanks Oracle for killing Java. jEdit does not work</a> brilliantly with OpenJDK, which is frustrating. Maybe I'll try to fix it?</p> Comments</h2> Warren P 2013-03-29</span> </h3> I’m curious if anyone who uses jedit from source, ever got jEdit to build from inside either netbeans or eclipse? I can get jEdit to build from the command line directly with ant but it seems that jedit’s project structure and both Netbeans and Eclipse’s features to import a java project from existing sources or using an existing ant build.xml are seriously limited in ways that prevents them from working at all with the jedit sources.</p> I asked a question about netbeans+jedit here:</p> http://stackoverflow.com/questions/15693837/h...</a></p> Netbeans even has a wiki page addressing this very subject, but like all things, it’s out of date and no longer works. (UI changes in Netbeans UI especially around how it works with existing Java projects have rendered Netbeans 7.3 less functional in this area than earlier versions, it seems.)</p> Warren</p> </div> Andy Balaam 2013-04-08</span> </h3> Hi Warren, I haven’t tried this I’m afraid.</p> </div> Andy Balaam 2012-03-14</span> </h3> Note on a Lucid machine I had to uninstall Eclipse and install the ant 1.8 packages for Natty by manually downloading from here: https://launchpad.net/~eclipse-team/+archive/...</a></p> </div> Goodness in programming languages, part 1 - syntax and layout Andy Balaam — Fri, 10 Feb 2012 09:42:03 +0000 In this series I will comment on what I like in some of the languages I use. I will cover things that I find convenient, things that might lead me to write correct code, things that tend to make my code more readable, and possibly other things that I just like for no clearly-articulated reason. The purpose of this series is to help me think about what features I would put in a language if I created my own.</em></p> Posts in this series: Syntax, Deployment</a>, Metaprogramming</a>, Ownership</a></em></p> What aspects of syntax and layout - how the code appears in your text editor - do I like? This is entirely irrelevant to the computer that is running the code, and an implementation detail of the language compiler or interpreter, but is extremely important to me.</p> Python and Scheme's low-symbol approach.</strong> Python and Scheme tend towards the use of English words, rather than symbols to express concepts. This makes code easier to read for someone unfamiliar with the language, and also for me.</li> Python's unambiguous block structure.</strong> Python uses indentation to express block structure (e.g. for variable scope, namespaces and logic structures). Programmers in most languages (notably C-like and Lisp-like ones) tend to use indentation to help humans understand the block structure of their code, but the computer uses different symbols to understand the same thing. This can lead to ambiguity where a human may mis-read the real block structure of some code because the indentation is inconsistent with the symbols. Python avoids the problem by making indentation the syntax understood by the compiler. [Go avoids the same problem by stating that code with inconsistent indenting (as defined by the gofmt program) is invalid.]</li> Scheme's simplicity.</strong> Scheme (like other Lisp dialects) has very simple rules to describe its syntax. This means you need to learn very little before you can understand the structure of Scheme programs, and it is unlikely you will be confused by rare structures.</li> </ul> Of course, there are trade-offs in all these points. Using fewer symbols can make programs verbose (although I find Python feels very concise). The lack of an end-of-block symbol makes short Python snippets look clean, but can make longer sections hard to understand (perhaps better editor support would help with this?). Scheme's use of brackets for everything means there are a lot of brackets, so working out which does what job can be difficult.</p> Nevertheless, the goals of reducing the number of symbols with special meaning, allowing humans and computers to use the same ways of understanding code, and being as simple as possible are good goals to have.</p> Comments</h2> Andy Balaam’s Blog » Blog Archive » Goodness in programming languages, part 2 â€“ getting your code running 2012-03-08</span> </h3> […] in this series: Syntax, […]</p> </div> Ethernet not working on Ubuntu with Realtek RTL8111/8168B Andy Balaam — Sat, 21 Jan 2012 08:22:40 +0000 Thanks to rvdavid, the network on my new PC is now working, despite Ubuntu misrecognising it:</p> How to Get Gigabit Speeds from RTL8111/8168B PCI Express Gigabit Ethernet controller on Ubuntu Linux</a>.</p> This is the kind of thing I had to do 10 years ago. I thought we'd moved on...</p> FreeGuide usage stats Andy Balaam — Sun, 16 Oct 2011 07:06:11 +0000 A long time ago I mentioned</a> I'd added some (opt-in) usage tracking for FreeGuide. Here are the latest stats. I think they've held up pretty well given how much I've neglected the project...</p> </a></p> </a></p> </a></p> Entertain your baby using your Android phone Andy Balaam — Thu, 01 Sep 2011 13:18:55 +0000 Last night my friend and I uploaded the first version of Baby Distraction</a>, an Android app featuring charming cartoon animals, vibration and sounds.</p> I helped almost exclusively in an advisory capacity, but still am inexplicably proud of the result: it's incredibly simple, but polished, and, most importantly, their baby loves it.</p> This morning I searched for "baby games" on the Android Marketplace, and it was the top hit. Fame and fortune await.</p> rdesktop swallows keyboard events Andy Balaam — Wed, 06 Jul 2011 12:03:04 +0000 The Linux remote desktop client rdesktop</a> grabs all keyboard events by default, which is good, because it allows e.g. you to Alt-Tab between applications and Ctrl-Alt-Del as if you were in front of the remote machine.</p> However, it is also bad because you can't use the standard keyboard shortcuts to switch between virtual desktops. Varius Googling had convinced me that this problem was not soluble, since X only provides a global keyboard-grabbing function XGrabKeyboard</a>, and once the event has been grabbed and fed to rdesktop, there doesn't appear to be a way to re-emit it back up to the window manager so that it can be used to switch desktops.</p> Kudos to Sunner</a>, therefore, for coming up with a workaround</a>. It's really simple: you have to press the key twice, and the first time, rdesktop simply ungrabs the keyboard. The second time, the window manager receives the event as normal.</p> Here's my version of Sunner's patch, against rdesktop 1.6.0. It removes Alt-Tab support since I don't need that, and adds Ctrl-Alt-Up and -Down, because I do need those:</p> --- rdesktop-1.6.0/xkeymap.c 2011-07-06 12:49:01.000000000 +0100 +++ rdesktop-1.6.0-mod/xkeymap.c 2011-07-06 11:40:24.000000000 +0100 @@ -598,6 +598,25 @@ ui_seamless_toggle(); break; + case XK_Left: + case XK_Right: + case XK_Up: + case XK_Down: + if ((get_key_state(state, XK_Alt_L) || get_key_state(state, XK_Alt_R)) + && (get_key_state(state, XK_Control_L) + || get_key_state(state, XK_Control_R))) + { + /* Ctrl-Alt-Left/Right/Up/Down: + * Ungrab the keyboard so that user can use Windows manager's hot keys */ + extern RD_BOOL g_fullscreen; + if (g_fullscreen) { /* Turn to normal window. Otherwise, rdesktop will be always on top */ + xwin_toggle_fullscreen(); + } + + XUngrabKeyboard(g_display, CurrentTime); + return True; + } + break; } return False; }</pre> (Download it here: rdesktop-1.6.0-ungrab-on-ctrl-alt-direction.patch</a>)</p> And as usual here is how to patch your Ubuntu system to include this patch:</p> sudo apt-get install build-essential fakeroot dpkg-dev sudo apt-get build-dep rdesktop mkdir tmp cd tmp apt-get source rdesktop cd rdesktop-* wget http://www.artificialworlds.net/blog/wp-content/uploads/rdesktop-1.6.0-ungrab-on-ctrl-alt-direction.patch patch -p1 < rdesktop-1.6.0-ungrab-on-ctrl-alt-direction.patch dpkg-buildpackage -rfakeroot -b cd .. sudo dpkg -i rdesktop*.deb</pre> Now someone just needs to add support for reading your window manager settings to determine automatically what shortcuts you use to switch desktops, and/or support for a config file listing keys that make rdesktop ungrab the keyboard.</p> Comments</h2> Scott Smedley 2016-02-24</span> </h3> rdesktop -K</p> Do not override window manager key bindings.</p> </div> Andy Balaam 2016-02-26</span> </h3> Thanks Scott – I want most shortcuts to work in rdesktop, but not all – this is why I had to patch rdesktop, but -K could well be useful for others.</p> </div> Reclaiming desktop space with Ubuntu's window-picker-applet Andy Balaam — Thu, 16 Jun 2011 14:15:16 +0000 For a long time I've customised GNOME to have a vertical task bar on the left and launcher buttons across the top. I've struggled with a long-standing bug with using a vertical panel</a>, and I've sacrificed quite a lot of screen space to make enough room to see the window titles.</p> Recently I've found myself using smaller screens, and finding certain aspects of the Ubuntu Netbook Remix</a> quite good. In particular, I like the combination of Maximus</a> and window-picker-applet</a>, which I combine with GNOME Do</a>. This setup allows me to have almost all of my screen set aside for programs I am using, without feeling lost or inconvenienced.</p> I need one setting to prevent Maximus from automatically maximising windows - I just want it to remove the title bar when a window is maximised, because window-picker-applet shows the title in my panel instead:</p> gconftool --type bool --set /apps/maximus/no_maximize true</p> and then I add Maximus to my Startup Applications (Click Add, give it a name, and set command to "maximus"), and set GNOME Do's "Start GNOME Do at login" preference (the Preferences screen can be found on the drop-down at the top right of the GNOME Do window).</p> Now I am happy, except for a little bug fix and visual tweak I want for window-picker-applet. The relevant bugs are Bug #425813</a> (see my patch at the bottom) and Bug #798197</a>. On an Ubuntu system it's amazingly easy to use a patched version:</p> sudo apt-get install build-essential fakeroot dpkg-dev sudo apt-get build-dep window-picker-applet mkdir tmp cd tmp apt-get source window-picker-applet cd window-picker-applet-* wget http://bugs.launchpad.net/ubuntu/+source/window-picker-applet/+bug/798197/+attachment/2171550/+files/show-windows-requring-attention-on-all-workspaces.patch patch -p1 < show-windows-requring-attention-on-all-workspaces.patch wget http://bugs.launchpad.net/ubuntu/+source/window-picker-applet/+bug/425813/+attachment/2171630/+files/faster-deeper-attention-flash.patch patch -p1 < faster-deeper-attention-flash.patch dpkg-buildpackage -rfakeroot -b cd .. sudo dpkg -i window-picker-applet_*.deb killall gnome-panel</p> Comments</h2> Lanoxx 2012-03-26</span> </h3> Hi,</p> If you are interested, there is now a port of window-picker-applet available on https://github.com/lanoxx/window-picker-applet</a></p> Regards Lanoxx</p> </div> Andy Balaam 2012-03-27</span> </h3> Hi Lanoxx, I certainly am interested – thank you very much!</p> </div> My Address Book 1.9.0 - rewritten from scratch Andy Balaam — Wed, 11 May 2011 12:58:03 +0000 Ordinarily, my motivations for doing open source work are clear: peer recognition and the satisfaction of knowing people are using my work.</p> However, I've been distracted from that stuff recently because of my desire to scratch my own itch, by re-writing My Address Book</a> from scratch to allow sharing the same address book between the web interface and the various email programs I use on different computers.</p> The only way to make it work with those email programs is to store the data in an LDAP server, instead of the custom MySQL database I had used in the original version (which is still available, of course: My Address Book version 1</a>).</p> While I was doing it, I took the opportunity (or made the mistake) of re-writing in my favourite language, Python. Joel would not approve</a>, but that's the fun of open source: I can do whatever I like, no matter how unwise.</p> I learned a lot of things on the way:</p> I like web.py</a>. It is a simple, and helpful library in the Python tradition of being concise but powerful.</li> Setting up Python running as FastCGI on either Apache or lighttpd is much</em> harder than creating a page of PHP. However, once it's done, it's done. (The results of my research are shown at the bottom of the Install Guide</a>.)</li> Setting up an LDAP server to act as a little address book is unbelievably complicated. See the Install Guide</a> for how to do it. One day, I or someone else will turn all those instructions into the preinstall step of a .deb, and no-one will ever have to worry about it again. Volunteers, please.</li> Templetor</a>, web.py's templating system is fine, but too slow for large pages. I had to reimplement the main address list rendering</a> in plain Python, which made me sad.</li> I am compulsive about getting my work out into the world.</li> </ol> It has been a struggle to write all the documentation and think through the installation procedure and all the other stuff that comes with making a public release, but I have been unable to move on to other things until I have got it done. I hope I've done an ok job - the installation procedure is far too complex and should be automated, but if the volume of documentation is any indicator of how helpful it is, it should be reasonable.</p> I think I found it more difficult than normal because, unlike normal, getting it out to other people was not my main motivation for doing the project. I'm glad I've worked through it though, and I really hope some people get interested enough to help me make it much easier to use.</p> I think this project is a good example of how it can be much better to live in the open source world than the proprietory one. If I want to create a small address book for my family in the open source world, I end up with an industrial-strength LDAP server providing it, meaning that so long as someone does the work to make it simple to use for my simple use case, it can scale to solve pretty much any problem I might have in the future. So if I start a small business that eventually grows and needs to track millions of addresses, I can keep using the same LDAP server, on the same hardware, as my original address book.</p> Of course, My Address Book would have broken long before that. It doesn't even do paging, and the home page lists all your addresses in one page.</p> I must write a rant one day about how I hate paging.</p> Switching workspace in GNOME via the command line Andy Balaam — Fri, 04 Mar 2011 17:09:42 +0000 I use rdesktop</a> to connect to some Windows machines, and it works beautifully. I like to allow it to grab keyboard input so I can switch and close windows on my Windows desktop without fear of accidentally doing something to the rest of my Linux desktop.</p> However, I have never found a way of letting some keystrokes escape - specifically Ctrl-Alt-Left, Right, Up, Down so that I can switch workspaces (virtual desktops) away from the workspace containing rdesktop.</p> I have finally found what I hope is a good workaround - I have installed easystroke</a> and defined some mouse gestures I can use to switch desktop. These mouse gestures are not swallowed by rdesktop, so they still work, even when it has focus.</p> When I asked easystroke to send the Ctrl-Alt-Left etc. keystrokes, they got swallowed by rdesktop, so all I needed to be able to do complete the story was a command-line tool to switch workspace.</p> That turned out to be trickier than you might imagine, but here is my solution, using the amazing wmctrl</a>.</p> This should work on GNOME with the metacity window manager, and the standard GNOME workspace switcher. It should be easy to adapt to other window managers and workspace tools.</p> First, install some tools. On Ubuntu you need:</p> sudo apt-get install bc wmctrl coreutils grep bash</p> Now create a file with gedit ~/bin/workspace-switcher & and edit it to look like this:</p> #!/bin/bash CMD="$1" NUM_WORKSPACES=`gconftool-2 --get /apps/metacity/general/num_workspaces` NUM_COLS=`gconftool-2 --get /apps/panel/applets/workspace_switcher_screen0/prefs/num_rows` NUM_ROWS=`echo "$NUM_WORKSPACES / $NUM_COLS" | bc` CURRENT_WS=`wmctrl -d | grep \* | cut -d " " -f 1` MOVE_LEFT="- $NUM_ROWS" MOVE_RIGHT="+ $NUM_ROWS" MOVE_UP="-1" MOVE_DOWN="+1" case $CMD in "Left" ) NEW_WS=`echo $CURRENT_WS "-" $NUM_ROWS | bc` if [[ $NEW_WS -lt 0 ]]; then NEW_WS=$CURRENT_WS; fi ;; "Right" ) NEW_WS=`echo $CURRENT_WS "+" $NUM_ROWS | bc` if [[ $NEW_WS -ge $NUM_WORKSPACES ]]; then NEW_WS=$CURRENT_WS; fi ;; "Up" ) WS_COL=`echo $CURRENT_WS "%" $NUM_ROWS | bc` if [[ $WS_COL -eq 0 ]]; then { NEW_WS=$CURRENT_WS } else { NEW_WS=`echo $CURRENT_WS "- 1" | bc` }; fi ;; "Down" ) NEW_WS=`echo $CURRENT_WS "+ 1" | bc` NEW_WS_COL=`echo $NEW_WS "%" $NUM_ROWS | bc` if [[ $NEW_WS_COL -eq 0 ]]; then NEW_WS=$CURRENT_WS; fi ;; * ) NEW_WS=$CMD esac wmctrl -s $NEW_WS</pre> Make it executable with chmod +x ~/bin/workspace-switcher and make sure ~/bin is in your PATH.</p> You can run it like this:</p> switch-workspace Left</p> to move left - the other possiblities are, obviously, Right, Up and Down.</p> Or like this:</p> switch-workspace 3</p> to move to a workspace by number.</p> If you want to use it with easystroke, create an action, and for the command simply enter switch-workspace Left and similar as the command.</p> Easystroke can be installed on Ubuntu like this:</p> sudo apt-get install easystroke</p> Comments</h2> Jason 2015-09-27</span> </h3> Just the piece of magic I was after, thanks.</p> </div> thevtm 2016-03-28</span> </h3> I modified the code for just the Up/Down</p> https://gist.github.com/thevtm/0519bfeaab408c...</a></p> </div> Andy Balaam 2016-04-05</span> </h3> Awesome, thanks thevtm!</p> </div> Amacvar 2014-02-19</span> </h3> Thank you, Thank you, Thank you!! Now I can have many more workspaces than the 12 I was limiting myself to … :) ;) I’m a visual person …</p> </div> Jon 2018-08-29</span> </h3> Thanks, this gave me the information I needed to fix my workspace key bindings for Pop!_OS</p> </div> Bohous Mach 2021-09-14</span> </h3> I have modified it a bit but thanks a lot for the idea! No more struggle with workspace grid/matrix in gnome shell</p> </div> Andy Balaam 2021-09-15</span> </h3> Glad it helped!</p> </div> how to change workspace by command line - Page 2 2011-03-04</span> </h3> […] This might be helpful – it is a script to switch workspace using the command line: http://www.artificialworlds.net/blog…-command-line/</a> […]</p> </div> Dan Connor 2014-05-27</span> </h3> This is awesome. I am using xmonad on Debian in VirtualBox. OSX is capturing the super key so I have to use Alt for xmonad. But this means that the default gnome switching shortcuts are not captured. But I was able to set up shortcuts to each workspace with your script. Great!</p> </div> Andy Balaam 2014-06-01</span> </h3> Hi Dan, I’m really glad it was helpful.</p> </div> Andrew P. 2012-03-22</span> </h3> When I check for existence of utilities included by default on my Ubuntu 10.04 system (e.g., "which bash"), I find that bc, grep and bash are already there. When I try to install coreutils with apt-get, it reports that the latest version is already installed. The only thing that is missing is wmctrl.</p> </div> Andy Balaam 2012-03-22</span> </h3> Thanks Andrew.</p> </div> Alexander Sorokin 2014-08-18</span> </h3> Thanks for the script! You saved me a bunch of time.</p> </div> Andy Balaam 2014-08-19</span> </h3> Hi Alexander, glad it was helpful!</p> </div> Martin Vahi 2014-10-04</span> </h3> That was useful, unfortunately. :-D</p> </div> Anatomy of an interpreter: the Evaluator Andy Balaam — Sat, 12 Feb 2011 01:54:01 +0000 Posts in this series: Lexer</a>, Parser</a>, Evaluator</em></p> I'm still really enjoying writing my Scheme interpreter Subs</a>, which can now succesfully run all the example code from SICP</a> up to section 2.3.4. I've made the changes I mentioned I would in the Lexer</a> article, so now the Lexer returns Tokens that contain information about their basic types, and I've gone through a significant refactoring to replace one of the several massive switch statements with a virtual function call (Martin Fowler would be proud</a>).</p> Last time I explained how the Parser</a> takes the stream of tokens coming from the Lexer and returns a hierarchical tree of Values, each of which represents an operation or thing in the program.</p> The Evaluator takes in a tree of Values, "evaluates" it, and returns another Value object, which is the answer. The Evaluator class is by far the most complex part of Subs, so in this post we'll start with an overview of how it works. Future posts will break down the different parts in more detail.</p> The most interesting parts of the Evaluator</a> class interface look like this:</p> class Evaluator { public: std::auto_ptr<Value> EvalInContext( const Value* value, boost::shared_ptr<Environment>& environment ); };</pre> The EvalInContext method takes in a Value to evaluate, and an "environment" *in which to evaluate it. Note that the in the real code it takes a couple more arguments, including a mysterious and annoying boolean called is_tail_call which will be explained later.</p> * More on environments later. All you need to know for now is that they provide a way of keeping hold of all the things we currently know about, identified by name.</p> </blockquote> A very simplified version of EvalInContext would look like this:</p> std::auto_ptr<Value> Evaluator::EvalInContext( const Value* value, boost::shared_ptr<Environment>& environment ) { if( is_symbol( value ) ) { return eval_symbol( value, environment ); } if( !is_combination( value ) ) { return auto_ptr<Value>( value->Clone() ); } const CombinationValue* combo = to_combination( value ); CombinationValue::const_iterator it = combo->begin(); auto_ptr<Value> evaldoptr = EvalInContext( *it, environment ); if( special_symbol( evaldoptr ) ) { return process_special_symbol( evaldoptr, combo ); } else { ++it; CombinationValue argvalues; for( ; it != combo->end(); ++it ) { argvalues.push_back( EvalInContext( *it, environment ).release() ); } return run_procedure( evaldoptr.get(), &argvalues, *this, environment ); } }</pre> If the Value to be evaluated is just a symbol, we call eval_symbol which basically looks up the symbol's name in the environment and returns the value it finds.</p> If the Value is not a combination (i.e. the root of a tree of other values) it must be a basic type such as a string or an integer. In this case we simple copy the Value and return it.</p> Otherwise, it's a combination. To evaluate a combination, we follow the "eval-apply" pattern. The principle is to evaluate all the Values in the combination separately, and then "apply" (run) the first value (the "operator") as a procedure, using the other values as arguments. The first value must evaluate to something that is recognisable as a procedure, or this doesn't make sense and we will throw an error.</p> In practice it's a tiny bit more complicated. We evaluate the first Value in the combination (by calling EvalInContext recursively), then we check whether it's a special symbol such as if or let and if so, deal with it separately. Otherwise, we evaluate all the other Values (calling EvalInContext recursively again) and put them into a new CombinationValue, and pass the operator and the arguments to run_procedure, which looks something like this:</p> std::auto_ptr<Value> run_procedure( const Value* operator, const CombinationValue* args, Evaluator& ev, boost::shared_ptr<Environment>& environment ) { if( is_builtin_procedure( operator ) ) { return handle_builtin_procedure( operator, args, environment ); } else { std::auto_ptr<Value> ret; const CompoundProcedureValue* proc = to_compound_procedure( operator ); boost::shared_ptr<Environment> new_env = proc->ExtendEnvironmentWithArgs( args ); for( CombinationValue::const_iterator it = proc->GetBody()->begin(); it != proc->GetBody()->end(); ++it ) { ret = ev.EvalInContext( *it, new_env ); } return ret; } }</pre> Running a procedure means either doing something built-in (for example adding up two numbers and returning the result) or evaluating some other code, which comes from the definition of the procedure being run. First we call ExtendEnvironmentWithArgs to create a new Environment, which contains the argument Values that were supplied, and then we loop through all the sections of the body of the procedure, evaluating each one. Note that we throw away the returned Values for all sections except the last one (this is how Scheme works).</p> Once we've evaluated and applied our procedure, which of course potentially includes numerous recursive calls to EvalInContext, we end up with a Value that is returned, and we're done.</p> Simple eh?</p> But now I must make a confession: almost everything I have written above is lie. Why would I lie to you? Because I missed out something wonderful and strange called "tail-call optimisation". I'll explain next time.</p> How to ask technical questions in person Andy Balaam — Sat, 12 Feb 2011 00:27:38 +0000 In a healthy team performing a technical task, there will be a lot of questions. Those questions will sometimes be asked by those with less technical knowledge, but (in a healthy team) there will be plenty of questions going back the other way too. Questions that come up in the normal course of your work are the main way knowledge is going to be transferred amongst your team members. They are to be encouraged, and you should never feel like a nuisance, or a technical wimp, because you're asking them.</p> It's easy to imagine, when you're asking someone a question, that whether you get a good answer depends mainly on the person you're asking.</p> In fact, an awful lot depends on you.</p> To get a good answer to a question, you need to prepare, to manage the Zone, and above all, to make the person you are asking feel clever.</p> What follows are some tips on how to ask a technical person a technical question when you're in the same room as them. Some of them may also apply to asking on the phone, or even via instant messaging.</em></p> 1. Prepare</h3> You're about to ask someone a question.</p> Breathe.</p> Think about what you are hoping to find out. This might sound obvious, but when you're in the thick of trying to solve a problem, you may find you're actually quite confused about what you're doing. If you can, you want to avoid wasting the other person's time with your confusion. Even worse, you might confuse them, and then they won't feel clever.</p> This doesn't need to take long - it may take half a second, but just let it cross your mind - what do I want to find out?</p> Think about the other person: what are they doing right now? Is it related? What do they know about? From what angle will they be approaching your question? Again, this will probably take almost no time at all, but it will help you ask your question well.</p> Now you need to zoom out. You are deep into a problem, so deep in fact that you're stuck on a very specific part of it. This part of it is burning into your consciousness, obscuring the wider picture of what you're doing. The person you're asking is going to need that wider picture, so remind yourself what it is.</p> Once you've zoomed out, you will need to engage with the Zone.</p> 2. Manage the Zone</h3> The biggest problem you have when asking a technical person a technical question is that they may well be in the Zone. The Zone is a state of intense concentration in which you get really good work done (Rands elaborates in A Nerd in a Cave</a>). The thing about the Zone is that it is difficult (sometimes even painful) to come out of.</p> If the person you're asking is in the Zone, they will need to be coaxed out. When you start talking, even if they turn and look at you and say something encouraging, they are somewhere far away, solving a problem. Your problem is an unwelcome intrusion.</p> The first lesson about the Zone is: don't say much at first.</p> When you start talking the other person doesn't know whether you're going to offer them a cup of tea, talk politics, or ask them how to implement quicksort. They haven't disengaged at all from the problem they are working on. You need to let them know you are going to ask them a technical question, and start the painful process of pulling them out of their problem, and into yours.</p> Often the best thing to do is pick a single word or phrase that lets them know the general area you are talking about: for example, "import", or "the C++ standard", or "garbage collection". If you say something like that, and watch their eyes, you can almost see the glaze of the Zone lifting.</p> If you can see they're still elsewhere, give them a second - maybe they'll even say something like, "I just want to finish this line." If they do, let them - step back, take the pressure off - they will be much better able to answer you when the thing they need to remember is written down.</p> Now your job is to zoom them in on your problem, at their pace. It's very important to watch and listen to the other person while you do this. Give them a problem area, and check they understood what you meant. Now give them closer context - maybe the directory you're talking about, or the DLL you're working on. Now tell them what class you're editing, or what file.</p> As soon as you can see it's needed, be quick to show them the exact thing you're doing on your screen or paper - that way they can add any extra context they need themselves. Be very wary of drawing vague diagrams on paper or a whiteboard - they confuse more often than they enlighten - much better to show them the real code or document you are working on. They can handle it.</p> All the time, watch how they are reacting - are they remembering what you're talking about, or completely in the dark? You may need to choose a different way into the problem. Remember, at this stage it's your job to explain what you mean, not their job to guess.</p> If you do this well, they're slightly in the Zone again, but this time focussed on your problem area. Now, and only now, hit them with your question. They'll know exactly what you're talking about, and feel really clever, especially if they know the answer.</p> There are a couple other things you need to know about the Zone. As we know, it can be painful to be pulled out of the Zone. That means that the person you're talking to, no matter how nice they are, and how much they like you, may feel a distinct feeling of irritation when you start talking. To make matters worse, they were concentrating, so probably frowning already. Even if they don't feel any irritation at all, they may look like they do. What can you do about this? Not a lot, except don't take it personally - it's normally momentary, and experienced technical people will recognise it's a false feeling, almost a physical reaction, and put it to one side immediately.</p> Finally, understanding the Zone explains why preparation is so important. It's very likely you are in the Zone yourself at the beginning of this process. You are submerged in a difficult problem, and quite possibly deeply irritated by your inability to solve it. To ask a question effectively (and thus get a good answer) you need to pull yourself out of the Zone and back into human interaction mode. This includes ensuring your annoyance is fully dissipated, or you're going to say something you shouldn't.</p> Throughout all this you need to make it your mission to make the other person feel clever.</p> 3. Make them feel clever</h3> Why would you want to make the person you're asking feel clever?</p> I certainly don't mean flatter them. If they're deep in their work they don't want you to waste their time telling them how good they are at something.</p> Think about it this way: how likely is it that you'll get an answer if you've made them feel stupid? I don't mean you won't get an answer because they're too annoyed with you to bother - I mean they're feeling stupid because you've confused them, and so they have no idea what the answer is.</p> Bear in mind that it's really easy to confuse someone. In your technical problem area there are thousands, or possibly even millions, of tiny micro-contexts that make sense within themselves, but sound like gibberish if you're outside that context.</p> Turning to someone and saying something like, "How do I Frobnish the Pernicator?" without first reminding them that Pernicators get created every time the TunableBeadnest is RePreppered is very hard work for them: they need to ask you a series of questions before they can work out what you're asking. Making them do this work makes them feel like they aren't clever enough to hold the entire knowledge of your arena in the front of their mind simultaneously. Of course, no-one is able to do this, but everyone feels like they ought to be. Don't make them feel stupid.</p> Even worse is to say something like, "How do I call this method?" In this case, even if they had total front-of-brain memory of the entire arena, they still couldn't answer this question, without interrogating you about what you meant.</p> You goal should be that the first thing they have to say in the whole interaction is when they give you the answer, or at least ask a question that shows they have grasped what you're asking, and makes them feel clever.</p> Of course, making them feel clever also makes them more inclined to talk to you next time you have a question, but that's merely a side benefit. The real reason you want to make them feel clever is that when they feel like that, it means you have given them the information they need, in the order they want, with timing that works for them, so that they can give you a great answer.</p> Comments</h2> Molly 2014-04-25</span> </h3> Andy,</p> You clearly explained what I have dimly sensed (from NOT asking questions right, or at the right moment) for years. Thank you for explaining this to me and everyone else. Now, on to asking things the right way!</p> </div> Andy Balaam 2014-04-25</span> </h3> Hi Molly, really glad you found it helpful!</p> </div> Anatomy of an interpreter: the Parser Andy Balaam — Thu, 30 Sep 2010 23:38:17 +0000 Posts in this series: Lexer</a>, Parser, Evaluator</a></em></p> Subs</a> has reached version 1.3.4, which means that it can successfully run all the tests from chapter 1 of SICP</a>. This is very exciting.</p> Last time I explained a bit about the Lexer</a>, which takes in a stream of characters and emits a stream of tokens: individual elements of code such as a bracket, a keyword or a symbol.</p> Generally, parsers emit some kind of tree structure - they understand the raw tokens as a hierarchical structure which (conceptually, at least) will be executed from the bottom up, with each branch-point in the tree being an operation of some kind.</p> Our parser takes in a stream of tokens, and emits a stream of parsed trees.</p> Parsing Scheme is very easy, because (except for a couple of exceptions I haven't implemented yet) there is essentially one rule: start with an open bracket, see a list of things, and then find a close bracket. Of course, one of the "things" you see may itself be another bracketted list, so after parsing you get a tree structure of nested lists.</p> The parser in Subs looks like this:</p> class Parser { public: Parser( ILexer& lexer ); std::auto_ptr<Value> NextValue(); private: ILexer& lexer_; };</pre> We supply a Lexer in the constructor, which we know will provide us with tokens when we need them via its NextToken() method. The Parser's NextValue() method returns a pointer to a Value, which is the base class for all the "things" in the Subs interpreter.</p> There are lots of types of things that inherit from the Value class, but the "parse tree" (the output of the parser) will only consist of a very small subset of them:</p> CombinationValue</li> DecimalValue</li> IntegerValue</li> StringValue</li> SymbolValue</li> </ul> The CombinationValue class forms the tree structure. Its declaration looks like this:</p> class CombinationValue : public Value, public std::vector<Value*> { // ... };</pre> It is simply a list of other Values.</p> Note that it "owns" those Values in the sense that it deletes them when it is deleted. I have recently made the jump to make Subs depend on BOOST, so it's on my TODO list to make containers like this use the BOOST smart containers to manage this job for me.</p> </blockquote> DecimalValue, IntegerValue and StringValue are relatively self-explanatory: they contain numbers and strings that were found as literals in the source code.</p> SymbolValue is essentially everything else - if the code that recognises the type of a token can't recognise it as a bracket, a number or a string, we assume it is a symbol, and tuck it away in a SymbolValue to be understood later.</p> The core of the Parser looks like this (with some error-checking removed):</p> std::auto_ptr<Value> next_value( ILexer& lexer, Token token ) { if( token.Name() == "(" ) { auto_ptr<CombinationValue> ret( new CombinationValue ); while( true ) { token = lexer.NextToken(); if( token.Name() == ")" ) { break; } // Recursive call ret->push_back( next_value( lexer, token ).release() ); } return auto_ptr<Value>( ret.release() ); } else { return ValueFactory::CreateValue( token ); } }</pre> (Full code here: Parser.cpp</a>) It's a simple recursive function that creates a CombinationValue whenever it finds a bracket, and otherwise uses a ValueFactory to create an individual value.</p> Side note: the wisdom of using recursion could certainly be questioned, since it limits the depth of bracketting we can handle to the size of the C++ stack, but the only other way to get the same result would be to keep our own manual stack of unfinished combinations, and it just seems perverse to re-implement language features like that. What might well be more interesting would be to consider whether we can actually evaluate parts of the tree as we go, without parsing it all at once. This might make the whole setup scale rather better, but would most likely be quite complex. The implementation presented here will work fine for almost any imaginable program - remember we would need not just code whose execution</em> is deeply nested, but whose expression</em> in code had thousands of levels of nesting before the parser would fail.</p> </blockquote> The ValueFactory</a> uses some basic rules such as "starts and ends with a quote" or "consists of only numbers and a decimal point" to recognise what type of Value to create, and if no rules match it defaults to a SymbolValue.</p> When we have completed a bracketted expression, we return a complete tree of numbers, strings and symbols, and it is ready to be evaluated, which you can think of as simply expanding the tree we already have into the full expression of itself, and then reducing it back down again to an answer.</p> Next time, the Evaluator and the famous eval-apply loop.</p> Anatomy of an interpreter: the Lexer Andy Balaam — Wed, 01 Sep 2010 23:27:39 +0000 Posts in this series: Lexer, Parser</a>, Evaluator</a></em></p> I have been having a lot of fun recently writing my Scheme interpreter Subs</a>. I have never implemented a full programming language before, so I am learning fast (mostly through mistakes) and wanted to write down some of the stuff I am discovering.</p> Note: if you want to learn more about what Scheme is I recommend Scheme (Wikipedia)</a> and the book SICP</a>, which is the inspiration for all this.</p> I am writing everything from scratch, just because it's fun (certainly not because I think it is in any way better to do it that way...). As we will see, that gives me opportunities to do things in different ways from the normal way such things are done. So far, every time I find out I've deviated from the normal way I've quickly discovered why I am wrong, and had to learn the true path.</p> Text-based programming languages, whether interpreted or compiled, need a lexer. A lexer takes in characters and spits out "tokens", which are groups of characters that represent a single thing, such as a bracket, a variable name or a number. (Those tokens are then passed on to the parser, which I will cover in a different post.)</p> Scheme (and other Lisp variants) are fairly easy to lex because they don't have much syntax - you just need to be able to understand round brackets, numbers and strings, and a couple of special cases that I won't go into because I haven't actually implemented them yet. (Mind you, I haven't implemented strings yet either...)</p> When I started Subs I took my normal approach of doing whatever I wanted without any research or even much thought, and wrote something that I called a lexer, but which was really something else. It took in a stream of characters, read it one "word" at a time (using whitespace as separators), broke up the word if it contained bracket characters, and emitted a tree structure with each branch representing a bracketted list. It just seemed sensible, while I was watching the brackets flow by, to understand them and create a tree structure.</p> However, for a number of reasons, that approach turned out to be wrong.</p> First, reading a "word" at a time made things much harder than simply stepping through each character. It made my initial implementation slightly faster, but as soon as I realised I cared about white space (e.g. keeping track of what line we are on) it had to go. When it went it also turned out to be easier to deal with unusual code layout - for example "a(b" should be lexed as 3 tokens, but would be handed to us as a single word.</p> Second, and more importantly, creating a tree structure at this point was a waste of time. Creating tree structures is normally the job of a parser, and mixing these responsibilities gave me some pointless inefficiency: the lexer emitted a tree of tokens, which the parser then translated into another tree (of fully-understood code objects). It turned out that walking the tree of tokens and copying that structure in the parser was at least as hard as just taking in a flat stream of tokens and constructing the tree just once.</p> So, I re-wrote Lexer</a> into something that is starting to become worthy of the name. The most interesting parts of its signature look like this:</p> class Lexer { public: Lexer( std::istream& instream ); Token NextToken(); }; </pre> It takes in a reference to a stream, which will provide the characters, and when NextToken is called, it reads enough characters to determine what the next token will be, and returns it.</p> Side note: Subs is written using Test-Driven Development</a>. I re-implemented the Lexer and Parser from scratch (naming the new classes NewLexer and NewParser until they were ready), modified the code that used them to use the new interfaces, ran the tests, and immediately knew that the new classes worked as well as the old ones. That level of confidence is incredibly freeing. I can't imagine how I would ever have convinced myself the new classes were ready had I not had that safety net of 100s of tests that ensure the interpreter correctly responds to each type of input.</p> </blockquote> Currently the Token</a> class it returns is pretty much just a string, with some information attached about where that string was in the original text. In researching this article I realised that most lexers attach more information than that to their tokens - they understand its basic type such as integer, decimal, string, bracket or symbol. At the moment in Subs, this work is done in the parser (so for the lexer each token is just a string), but I can see why it is helpful to do this work in the lexer, because for most types we have the information anyway. For example, in order to recognise that '"foo (bar"' (where the double-quotes are in the real code) is a single token we must understand that it is a string. Since we know it at this stage, we might as well record it in the Token object so we don't have to work it out again later. When Subs supports strings, I will probably add a "type" field to the token and move this work from the parser.</p> On a more general programming point, following on from comments I made in a previous post</a>, it is worth noting that the structure of the lexer (and, as we will see later, the parser) uses a technique called "streams". What this means is that we write functions like NextToken that process a small part of the total problem, and return their answer. If we chain together functions like that (for example the parser's equivalent function calls NextToken whenever it needs a new token) we can process arbitrarily large input without using lots of memory or slowing down. The lexer is able to process any number of tokens using a very small amount of memory, and will only fall over if it encounters a single token that is ridiculously large.</p> The stream style can be very useful for some problems, not only because it can be efficient with memory, but also because it can help break problems into neat, small pieces that are easier to implement correctly. It is also useful for writing code in a functional style, because it allows us to avoid having any internal state (e.g. we could easily implement NextToken to take in the stream it should read from and avoid any member variables at all in Lexer), by pushing state out into the input and output, instead of having it in our program. In this case that means instead of reading, storing and then processing a program, our lexer can simply process a few characters and emit a token without knowing anything about the surrounding code or wider context. This makes it much easier to test, and (potentially) easier to do clever things with, like prove mathematically that it is correct (!)</p> Next time, the parser.</p> Comments</h2> Muhammad 2018-06-22</span> </h3> Really helpful. Recently, I have been trying to write a markdown parser. And I got stuck on some points which was clearly explained in the lexer and parser articles. Would you please update the link to the code? Thanks</p> </div> Andy Balaam 2018-06-22</span> </h3> Hi Muhammad, I’m glad it was helpful. Thanks for pointing out the broken links – I have fixed them in all three articles.</p> </div> Muhammad 2018-06-22</span> </h3> Thank you</p> </div> Subs Scheme Lisp Interpreter Andy Balaam — Wed, 18 Aug 2010 16:05:16 +0000 Why would you write a Lisp interpreter?</p> I find that question difficult to answer, but the joy of open source is that I don't have to answer it.</p> Subs</a> is a Scheme interpreter written in C++, and growing out of the excitement I have felt while reading Structure and Interpretation of Computer Programs</a>.</p> Subs is very incomplete, and will probably never be otherwise, but it is exciting how quickly you can write a functional Lisp interpreter. I plan to go through SICP section by section and ensure all the examples work in Subs. So far I am failing on the last example in section 1.1.1 (because I don't support new-lines inside statements yet!) but in reality most of the work is done to support quite a lot of stuff (NOT including mutable variables...yuck).</p> One possible explanation, if such a thing were necessary, is that one day I want to write a new programming language that has all the simplicity and metaprogramming capability of Lisp, with the native performance and deployability of C++, and the syntactic elegance of Python. I've got a lot to learn first.</p> Scalable Graph Coverage Andy Balaam — Mon, 21 Jun 2010 00:25:47 +0000 If you're interested in dealing with large directed graphs of dependent objects and want some tips on how to process them in a way that scales in terms of memory usage, you may be interested in the article I wrote for Overload</a>, which is a journal of the ACCU</a> (an organisation promoting "professionalism in programming").</p> The article is Scalable Graph Coverage</a>, or, in my original title, "Comparing scalable algorithms for walking all nodes of a dependency graph".</p> It contains lots of code, written in C++, using BOOST Graph Library. The code demonstrates some of the algorithms that are available for choosing batches of nodes to be processed together to reduce the number of nodes that are loaded several times, and without running out of memory.</p> Ubuntu Lucid upgrade - a catalogue of disaster and pain Andy Balaam — Fri, 14 May 2010 10:37:33 +0000 I recently upgraded 3 machines to Lucid. I don't recommend it.</p> Broken after upgrade</h3> 2 out of 3 machines did not boot after upgrade to Lucid.</p> One, an upgrade from Hardy, crashed hard during the upgrade (the screen was frozen with a prompt about lmsensors visible). It's possible this was caused by a hardware issue of some kind, but it's behaved itself since. When I powered off the machine it failed to boot because grub couldn't see my hard drive. I believe this was because the file system had been changed in some way (to ext4?) but grub hadn't been upgraded because of the crash, so it couldn't handle it.</p> Anyway, I spent a day trying to recover the system, and then a night rebuilding from scratch. I'm still configuring the system and regretting the upgrade.</p> The second, a netbook with a fairly standard Karmic install on it, upgraded smoothly and worked.</p> The third, my work machine, upgraded smoothly from Karmic, and frigged my fstab so my second drive wasn't found. It changed my main drive to be identified by a UUID (instead of being /dev/sda), and this meant my second drive, which used to be /dev/sdb was now called /dev/sda. The upgrade failed to correct this in my fstab and gave me a very rude error during boot.</p> Fortunately I had the technical knowledge and found the time to fix these things, but it's hardly something I'd like someone like my parents to have to deal with.</p> Annoying after upgrade</h3> My volume applet disappeared. Bug 552221</a></li> Empathy doesn't tell me about messages any more. When I enable the stupid indicator applet, I get a tiny yellow star when a new message has arrived. I can't get an Empathy applet to appear in the status area.</li> Thunderbird 3 thrashed my disk and network for a couple of days to download and index all my IMAP mail. It is faster now, though.</li> I can't get Thunderbird to give me a "Calendar" link in the bottom left to switch to the Lightning plugin. I did find the "LightningButton" add-on that allows me to have a button on the toolbar. Yes, you are reading that correctly.</li> </ul> I loved Karmic. It got me all excited about Ubuntu again. I wish I'd stuck with it.</p> Comments</h2> Gerry 2010-05-17</span> </h3> Wish I’d read this before, instead of later. My Karmic laptop upgrade went fine. But my desktop from Hardy to Lucid… enough dpkg dependency errors that the system crashed hard. Unless I find some shortcut, it’ll have to be rebuilt. It’s still ext3 and grub is intact and the old images, but it apparently changed enough that it’s a mess. They supplied no clue on how to approach fixing this. The abort requested I report a but against upgrade-manager, but launchpad doesn’t have that package. And this one’s my work machine for app development, so it’s pretty loaded.</p> Looks like smash and burn’s the only way, so I’m downloading lucid to a flash drive. At least the drives can still be mounted:–)</p> </div> Andy Balaam 2010-05-17</span> </h3> Hi Gerry, good luck with the recovery. I wonder whether some kind of dependency problem caused my crash. By the time I got back to it it was just frozen solid.</p> </div> Andy Balaam 2010-05-24</span> </h3> Thanks to a post by Will McGugan, I fixed the problems I was having with the flash plugin failing and losing sound: http://www.willmcgugan.com/blog/tech/2010/5/2...</a></p> (Summary: rm ~/.mozilla/plugins/libflashplayer.so)</p> </div> FreeGuide 0.10.11 Andy Balaam — Tue, 24 Nov 2009 06:58:58 +0000 I've made a new release of FreeGuide</a> today, which contains minor bug fixes.</p> I changed the stability rating on sourceforge to "Production/Stable" (previously it was "Alpha"). Maybe I should bump the version number next time to 1.0. Certainly all this 0.10.x stuff is crazy.</p> I managed to get rid of quite a bit of unused and unworking code this release, so I'm feeling somewhat more positive about developing FreeGuide. My next plan is to spruce up the ant build so you can easily use it in Eclipse, and convert the whole lot to a single JAR, which may improve startup time.</p> The program seems to work for quite a few people, and I don't change it much these days, so it might be time to call it 1.0. The only major feature I am planning (one day...) is to combine the Find dialog with the vertical view and possibly the favourites editor to make them different views onto the same underlying implementation. I'd like to make all of them show filter results as you type.</p> Other than that all I really want to do it fix user pain in terms of bugs and UI wrinkles.</p> Don't design for performance until it's too late Andy Balaam — Fri, 20 Nov 2009 06:16:40 +0000 There is a piece of ancient wisdom which states:</p> Premature optimisation is the root of all evil</a></p> </blockquote> This ancient wisdom is, like all ancient wisdom, correct.</p> However.</p> It appears to have been reinterpreted as essentially meaning:</p> Don't design for performance until it's too late</p> </blockquote> which is clearly, and very importantly, very wrong.</p> Performance is a feature</h3> Before I begin I want us all to agree that performance is a feature</a>.</p> I work on a real-life "enterprise</a>" application. Its features are entirely driven by the need for immediate cash, not by developers following pipe dreams. And yet, for the last 6-12 months the majority of my time has been spent trying to retrofit performance into this application. Believe me, this is not because we have users who are obsessive about wasting valuable seconds - it's because our performance sucks so hard it's deeply embarrassing.</p> What is your favourite program? How well does it perform? What is your least favourite? Why?</p> For me, and many other people, their answers to those questions demonstrate the importance of performance. Firefox was launched to improve the performance of Mozilla. People love git because of how fast it is. Lotus Notes is hated so much</a> partly because of its performance. My main complaints about programs I use involve performance (e.g. Thunderbird is too slow for IMAP email</a>.)</p> A fast response to the user is one of those crucial inches</a> on the journey to software that makes people happy. Making people happy gives you the kind of scary fanboyism that surrounds git. Wouldn't you like that for your product?</p> What is optimisation?</h3> When my hero said that premature optimisation was the root of all evil, he was talking in the days when you had to hand-optimise your C in assembly language. Or, more likely in his case, you had to hand-optimise your assembly language into faster assembly language. Optimisation like that very often obfuscates your code.</p> These days, 99% of the time, your compiler does all of this work for you, so you can have relatively comprehensible code in whatever trendy language you like, and still have the fastest possible local implementation of that in machine code.</p> Meanwhile, Knuth knew that 99% of your code is simply not performance-critical - it only runs a few times, or it's just so much faster than some other bit that it doesn't matter. The lesson we all learn eventually is that the slow bit is never quite what you thought, and you have to measure to find out where to concentrate your effort.</p> So, if optimisation is obfuscation, and 99% of your code isn't the bit you need to make faster, it becomes clear that premature optimisation is the root of much evil.</p> But optimisation is not designing for performance.</p> Design for performance</h3> Fundamentally, to get good performance, you are going to need to measure the time spent in various parts of your code (I suggest Very Sleepy</a> if you're on Windows) and make the slow bits faster (or happen less often). However, there are still some principles you can follow that will mean you need to spend less time doing this*.</p> *Which is a pity, because I really love doing it.</p> If you don't design for performance you are almost certainly going to need to restructure large parts of your program later, which is very difficult and time-consuming.</p> There are two aspects to designing for performance: writing good local code, and creating good global structure.</p> Write good local code</h4> Before you write an algorithm, think for a few minutes about how to make it work efficiently. e.g. if you're writing C++, consider whether a deque</a> or a list</a> would be better than a vector</a> for how you're going to use it.</p> Think about what is idiomatic for your language and why. Think about what the computer really has to do to produce the results you are asking for. Are there going to be a lot of objects about? Maybe you can avoid copying them too many times. Are you creating and deleting a lot of objects? Can you reuse some instead? (Exercise caution with that one, though - if you start obfuscating you come into conflict with the ancient wisdom.)</p> Often, if you think through what you are doing, and the most efficient way to do it, you will end up with a faster and more memory-efficient algorithm, that expresses your intention better</em> than if you'd written the first thing that came into your head. There is no downside to that.</p> Try to minimise the number of times you need to ask the operating system for a chunk of memory: this is surprisingly slow. E.g. in C++, prefer creating by-value data members instead of pointers to objects allocated with their own call to new. By the way, don't worry if this sounds intimidating. The way to learn this stuff is to measure what you have done and then work out why it is slow. Next time you'll jump straight to the fast solution without the detour to the slow one. Of course, none of this will matter if you don't have good global structure.</p> Create good global structure</h4> The hardest and most important work you need to do to have good performance is to have good structure in the ways the different parts of your program interact. This means thinking about how classes and components communicate with and control each other. It may be helpful to use a streaming style of communication - can you send little chunks of information to be processed one by one instead of a huge great blob? Try to make sure your components to use a common infrastructure: if different parts use different string classes you are going to spend most of your time copying from one to the other. The hardest and deepest mystery in getting good performance (and in programming generally) is choosing the right fundamental data structures. I'll never forget the lesson I learnt** when a friend of mine had a conversation with me about a toy project I was doing (that was particularly focussed on trying to be fast) and then went away and produced code that was orders of magnitude faster, simply because he had chosen the right data structure. **The lesson I learnt was that I am not as good as I think I am. To be honest this section is a little shorter than I'd like because I know I don't have a lot of answers about how to do this well. I do know, though, that if you don't think about it now you will have the pain of restructuring your program later, when it's full of bug fixes that are going to get rebroken by the restructuring. Of course, if you do think about it now you're still pretty likely to need to change it later...</p> Ancient wisdom</h3> Ancient wisdom is usually right, but misinterpreting it and using it as a license to write bad code is a bad idea. Carry on.</p> Comments</h2> Anatomy of an interpreter: the Lexer – Andy Balaam's Blog 2018-06-22</span> </h3> […] a more general programming point, following on from comments I made in a previous post, it is worth noting that the structure of the lexer (and, as we will see later, the parser) uses a […]</p> </div> Technorati appear to have broken openid delegation Andy Balaam — Tue, 03 Nov 2009 12:08:16 +0000 I have just changed to myopenid</a> and can delegate to them from my own openid URL by following the instructions here: https://www.myopenid.com/help#own_domain</a>.</p> Google don't appear to let you delegate to them (i.e. have your own url).</p> I'm really glad I chose to set up a delegate now, since the brokenness of Technorati would have screwed me if I hadn't done that.</p> NNDB 0.1 Andy Balaam — Fri, 23 Oct 2009 20:57:36 +0000 I've managed to get NNDB</a>, my C++ data storage library which is almost, but not entirely unlike SQL, into a fit state for a release.</p> You can create tables, set indices on columns, insert data, retrieve data using something like a SELECT, filter it using something like WHERE (which uses indices where available), and order it using something like an ORDER BY.</p> So far it has been a fantastic way to get my hands dirty with some Template metaprogramming</a>, and some C++ as it should be*, but the reason why I started this was to help me think about how databases work, so I'm really looking forward to getting into how to implement JOINs. At the moment I have only very vague ideas.</p> NNDB is based heavily on the STL</a> (part of C++'s standard library), BOOST</a> (a playground for things that might one day be in C++'s standard library, and hang-out for some of the cleverest people alive), and Loki</a> (the continuation of Andrei Alexandrescu's Template metaprogramming (but used for good, not evil?) library written for and explained in Modern C++ Design</a>. This book ranks in the top 5 most exciting books I have read). I continue to be more impressed by all three the more I learn.</p> I have even been having discussions with the Loki devs</a> about some code I needed for NNDB that I think might be helpful for other people using Loki. It's called ForEachType</a> and it allows you to loop (at runtime) through all the types in a Typelist and do something for each one.</p> The project is already working in terms of helping me think about databases. For example, I really hadn't thought before about how expensive ORDER BY is. To implement it I needed to create a temporary std::map covering the entire result set - in a real database this obviously requires reading every single row before we can even begin to return any results. The way to avoid this is to have an index. Which reminds me: the next thing I need to do is make ORDER BY able to use indices (at the moment it's only WHEREs that take advantage of them).</p> So next on my list are:</p> ORDER By uses indices</li> Non-unique indices (presumably implemented with a std::multimap)</li> Joins</li> </ul> I am still very excited so you may see more releases over the next few months.</p> [* NNDB so far contains zero (0) calls to new and zero (0) calls to delete. Obviously the code it uses (e.g. std::vector) calls them, but that code manages all the memory for me, and most of it uses custom allocators to make it very fast. I have no idea how fast NNDB is, but maybe it could be quite fast. I am pretty confident it doesn't contain any memory errors. Famous last words...]</p> Comments</h2> Anonymous 2010-02-17</span> </h3> You mention in your article that you need TypelistForEach, which (if I understood correctly), doesn’t exist in Loki yet; did you have a look at Boost.MPL (MetaProgramming Library)? It proposes a huge set of type containers along with several algorithms to operate on them at compile-time, such as transform, sort, fold and so on.</p> You could also check out Boost.Fusion, a library making the link between pure compile-time computations (cf. MPL) and pure runtime computations (cf. STL). It contains several heterogeneous containers that can be manipulated either at compile-time with metafunctions or at runtime with "classic" functions.</p> </div> Andy Balaam 2010-02-18</span> </h3> Hi, I haven’t looked at MPL yet – I only discovered it recently. Fusion also sounds very interesting – thanks!</p> By the way, I submitted my code and it was accepted into Loki: http://loki-lib.svn.sourceforge.net/viewvc/lo...</a></p> </div> NNDB's Not a Database Andy Balaam — Sat, 10 Oct 2009 08:22:04 +0000 My latest project is called NNDB</a>.</p> I've worked with databases for quite a long time now, and for a while I've been thinking about how they work under the hood. I know very little about it, but I thought I could learn a bit by trying to implement something similar myself.</p> I'm interested in how queries work against joined tables, how to implement indices and so on.</p> I've also been feeling that I want to do some C++ as an open source project. I do it all day at work, and for some problems it feels like the right tool for the job.</p> NNDB is sort-of like an in-memory database, but it works with C++ types for its columns, instead of a fixed set like varchar, int etc. You can put your own value-typed classes in the columns, and all values are type-checked at compile time.</p> It's always struck me as strange that with a traditional code+SQL setup you have to keep your SQL in sync with your code manually. Of course, there are lots of trendy Object-Relational-Mapping thingies that solve that problem, but I felt it could be approached from another direction: instead of generating code to match your data, or generating SQL to match your code, why not specify your data structure in</em> code?</p> In NNDB you define a table something like this:</p> typedef nndb::Values< unsigned long, std::string, std::string, MyDate > PersonValues; class PersonTable : public nndb::Table { public: enum Columns { id, first_name, last_name, date_of_birth }; };</personvalues></pre> Actually, defining your own class is unnecessary, but it's nice to have an enum to name your columns, and making a class gives you a nice place to put it.</p> To insert a row you do something like this:</p> PersonTable person_table; person_table.Insert( PersonValues( 0, "Andy", "Balaam", MyDate( 12000000 ) ) );</pre> You can do simple queries with WHERE and ORDER BY clauses, and I'm working on indexes.</p> After that will come JOINs, and anything else that takes my fancy.</p> I don't anticipate NNDB being useful to anyone - it's really for me to understand why things are as they are in the world of databases. However, you never know - it may turn out to be a fast and convenient way to store data in the C++ world. I think some of the applications that use databases don't really need the kind of concurrent multi-user network-accessible features they have, but really just want to search, join and store reliably, and NNDB might one day grow into something that can find a niche.</p> To explore more, check out the complete example</a>.</p> Comments</h2> SG 2009-10-29</span> </h3> "instead of generating code to match your data, or generating SQL to match your code, why not specify your data structure in code?"</p> See LISP :)</p> PS: Really cool C++ project!</p> </div> Andy Balaam 2009-10-29</span> </h3> Thanks! I am actually currently reading http://mitpress.mit.edu/sicp/full-text/book/b...</a> and loving it. It turns out pretty much all the interesting stuff in template metaprogramming is almost identical to how you would do the same thing in LISP.</p> </div> Guillaume Yziquel 2010-02-09</span> </h3> Two projects that might be interesting to you, though quite far from C++ userland.</p> PG’OCamlm by Richard Jones: http://pgocaml.berlios.de/</a></p> A statically typed relational algebra by Mauricio Fernandez: http://eigenclass.org/hiki/addressing-orm-pro...</a></p> </div> Andy Balaam 2010-02-10</span> </h3> Thanks Guillaume – really interesting projects, and further really interesting links at the bottom of the second one. I knew someone else must have done something like this before…</p> In self-augmenting language it is possible to build something like this on top of a real database, which I think would be very difficult in C++.</p> I have been having thoughts about what my ideal language would be. Recently I have been wondering about whether it’s possible to create a self-augmenting language that is also compiled in a traditional way, and type-safe in the C++ sense. Essentially I want to be able to extend the C++ compiler at compile time.</p> </div> Diffident 0.3 Andy Balaam — Sat, 10 Oct 2009 07:54:48 +0000 My original plan for Diffident</a>, the side-by-side diff viewer and editor that works in a terminal, was to implement basic editing capabilities before making another release.</p> Of course, that turned out to be quite ambitious. It involves essentially implementing a full text editor, which is not really what I want to do. I may actually implement a "jump out to $EDITOR" option before the basic text editing facilities.</p> What I have implemented for this release is the ability to add and remove lines, and copy lines from one side to the other. For my personal use, this covers about 90% of cases, so I think it's worthy of a release.</p> There is no undo/redo as yet, but the framework for that is in place, so I may make another release sometime soonish that is just for that.</p> So the dream of a diff viewer and editor</em> is starting to come true...</p> CBeebies and other channels not working with mplayer Andy Balaam — Fri, 09 Oct 2009 16:08:03 +0000 It turns out the scan command from the linuxtv-dvb-apps package (on Ubuntu Hardy Heron, anyway) isn't writing the correct channels.conf, and this makes some channels not work for me, including CBeebies, CBBC Channel and CITV.</p> The fix is to do as this person did: http://ubuntuforums.org/showthread.php?t=686829</a>.</p> Basically, the kaffeine TV viewer scans the channels fine so you can manually copy the values you need from its config file.</p> The incorrect channels have "0:0" as the second-last and third-last numbers in their row in ~/.mplayer/channels.conf .</p> Those two zeros should be replaced by the first two numbers you find in the row for this channel in ~/.kde/share/apps/kaffeine/channels.dvb . You can ignore the numbers in brackets in this file.</p> Firefox keyword search for finding C++ keywords Andy Balaam — Wed, 23 Sep 2009 09:58:37 +0000 I often want to search the SGI C++ reference for a keyword. The best way I have found to jump straight to the page I want is to use Google's "I'm Feeling Lucky" search limited to searching within sgi.com.</p> You can create a Firefox keyword search to allow you to do this quickly from the location bar. Now I just type Ctrl-L then e.g. "c vector" to jump straight to the page about std::vector.</p> To do this, make a bookmark (to anything) and then right-click it in the Bookmarks menu and choose Properties. Edit it to look like this:</p> The Location field is set to http://www.google.com/search?as_q=%s&as_sitesearch=sgi.com&btnI=1, and the Keyword field is just c.</p> This post was largely for my own benefit to remember this next time I need to set it up, but I thought it might be useful to others as the information about how to set these searches up is not easy to find.</p> If you want to go to a google search results page, instead of jumping to the "I'm Feeling Lucky" result, remove the &btnI=1 part.</p> Comments</h2> Charles Bailey 2009-09-23</span> </h3> Strangely, I still find my hardcopy of ISO/IEC/BS 14882:2003 the most convenient reference for C++. Ctrl-F in the pdf version at work also works quite well but is somehow not as satisfyingly tactile.</p> </div> Flaws in the software patents system Andy Balaam — Thu, 10 Sep 2009 12:56:55 +0000 Note: I am in favour of patents on inventions.</em></p> I believe the current software patents system has a number of flaws (in decreasing order of importance to me):</p> They attempt to close off the use of knowledge rather than open it up. If we were not filing patents for our work, we could be writing papers or books to share that knowledge. What would have happened if the Gang of Four had filed patent applications instead of writing Design Patterns</a>?</li> They can verge on the dishonest. Describing most software development work as "novel" and "non-obvious" is simply incorrect. There is a huge community of software developers, and the idea that none of them would find what I am doing "obvious" beggars belief.</li> They are a tool that big companies can use to squash small companies. Small companies can't afford to develop a large defensive patent portfolio. Patents don't even need to be valid or high-quality: if there are enough of them, a company that can afford a lot of lawyers can destroy one that can't.</li> They are a tool that big companies can use to squash open source projects. Many open source projects have no money at all. If a large company doesn't like some project, they can simply sue for patent infringement (even for patents that are blatantly irrelevant) and the project will be toast.</li> </ol> So I don't want to be involved in patent applications at my work, even though my employer is keen on patents.</p> I am now considering how I should act on this opinion. I don't want to screw my colleagues but I think I want to refuse to be a part of any patent applications as far as I can do that without screwing them.</p> IGCC - a real-eval-print loop for C/C++ Andy Balaam — Mon, 31 Aug 2009 02:56:50 +0000 When you first hear about the Read-Eval-Print Loop</a> you might well think "So what?" as I did.</p> What's so great about being able to type commands interactively?</p> But the thing is that it creeps up on you.</p> Everyone already knows programming is an interactive thing - we need constant feedback to validate our ideas. Programming on paper is incredibly frustrating because you have to plough on with assumptions that are probably wrong.</p> It's just so comfortable to be able to try out ideas in an interactive interpreter.</p> I mean, it's really not much hassle to create a new directory, make a new file, edit the file to contain the code you want to try, remember the right command to compile it, then run the program and see the results, is it?</p> Well, no, it isn't, but it's enough of a hassle that sometimes you don't bother and you try it out in the code you are really working on, and if your work is like mine that means a minimum of 5 minutes to compile and link, and there you are playing foosball again when you could be getting something done.</p> The REPL gives you a place to try throwaway things extremely quickly, and when you're working with something beautiful like Python it's easy to get addicted.</p> So my mind started to wander and it struck me that a pale imitation of the REPL could be made for us poor C++ programmers, and it would generally serve the purposes I've described above.</p> So IGCC</a> was born. Its name means "Interactive GCC" and it's a read-eval-print loop for C++ (and, for most cases it will work for C too).</p> It uses the real GCC underneath, so you know you are running the exact code you would be (and it's somewhat easier to write than a custom C/C++ interpreter) and all it does is take away the hassle of creating a simple program and compiling it with GCC.</p> It wraps your code in a standard C program, includes some common dependencies, and compiles it, printing the results of running them immediately. Using it looks like this:</p> $ ./igcc g++> int a = 5; g++> a += 2; g++> cout << a << endl; 7 g++> --a; g++> cout << a << endl; 6 g++> </pre> Apart from all the sugar that I'd love to add, the main missing features are some kind of equivalent of the Python dir command</a>, and code completion.</p> It's not rocket science, but it might make you a little bit more interactive in your C and C++ coding, which might save you valuable foosball time.</p> Enjoy, improve, etc. IGCC</a>.</p> Foosball image taken from http://en.wikipedia.org/wiki/File:Baby_foot_artlibre_jnl.jpg</p> Diffident - command line side-by-side diff editor Andy Balaam — Wed, 26 Aug 2009 20:13:19 +0000 I really like Beyond Compare</a>, which is a proprietary diff program with all those little touches that make it a joy to use*. The way I write code at work generally involves a bit of hacking in jEdit</a>, checking the code myself, and then reviewing the code with a colleague.</p> *Recently, though, they've brought out a newer version that seems to overcomplicate things.</p> Both my own checking and the code review with a colleague generally involve comparing the code with the previous version in the (perforce) repository. Beyond Compare integrates nicely with the perforce tools and allows you to see a change as diffs of each file involved.</p> None of this is anything new to the Free Software world, of course. All the version control programs I've used allow you to do the equivalent of cvs diff</code> and see what changes you've made. Git</a> has a particularly good git diff</code> mode which by default colours your diff and pipes it to less, making it easy to read and use.</p> What I have found myself missing recently, though, is the ability to edit the files as you diff them. The whole point of reviewing what you've done is to make changes as they occur to you, and with perforce + Beyond Compare it is really natural to make those changes within the diff tool.</p> Incidentally, I also really like the side-by-side style of Beyond Compare. By default it inserts "missing" lines so that all the similar lines are aligned, rather than trying to indicate with balloon-like lines that code has been inserted or transformed. I find those balloons very annoying and confusing (and they take up space on my screen, grr).</p> Having a side-by-side view also makes copying lines from one side to the other much easier. I often copy from one side to the other - especially when I realise what I have done is stupid and I want to revert a section back to how it was.</p> So, to curtail an already-long story, I decided there was yet another area of my life where the only solution was to run software I had written, rather than relying on the shoddy stuff put out by others, so I wrote a diff tool.</p> Diffident</a> is a diff tool inspired by Beyond Compare and git. It shows you a side-by-side diff of your files in a terminal (one day it may have a GTK+ interface too) and allows you to edit them. The editing part is in development - expect a release fairly soon, or get the code from the git repository. The output is coloured, and you can see the whole of both files, using keyboard shortcuts to jump to the next and previous differences.</p> You might ask "Why not just use Beyond Compare?" For a number of reasons:</p> It is not Free Software. I can't improve it or trust it.</li> It's sort of Windows-y. I know there is a Linux version, but I bet it's not very Linux-y. (Disclaimer: I've never tried it ;)</li> It doesn't work in a terminal.</li> The inline editing support is not great. Its real strength is copying from side to side.</li> It doesn't feel right when used with git. I have got this set up in my Cygwin environment - it works, but it's no fun.</li> It isn't written by me.</li> </ol> I'd really like Diffident to become the de-facto diff tool for git people (and everyone else). That proves to be a bit trickier than I'd like because of the way git interacts with diff tools, but I've got a decent solution for using it as the GIT_EXTERNAL_DIFF tool, and I hope to improve it in the future. (For those who are interested, the kind of thing I'm thinking about is how to get git diff --cached</code> to allow me to edit the files in the index.)</p> So anyway, check out Diffident</a> and if you like it, help me make it better.</p> Random background in GNOME Andy Balaam — Fri, 31 Jul 2009 08:50:24 +0000 I've written a little script to show a different random background in GNOME every day, by looking inside your Photos directory.</p> It will rotate them to be the right way up, if you have exiftran installed.</p> Get it here: Random Background</a>.</p> Analog literals Andy Balaam — Wed, 20 May 2009 09:17:23 +0000 I love this: C++ Multi-Dimensional Analog Literals</a>.</p> I quote:</p> Have you ever felt that integer literals like "4" don't convey the true size of the value they denote? If so, use an analog integer literal instead: unsigned int b = I---------I;</pre></p> </blockquote> It goes on to explain that you can use 2- and 3-dimensional "analog literals". Genius. Read the article. Try to read the code :)</p> Isn't C++ ... erm ... powerful?</p> You'll notice that there are 9 dashes used to denote 4. This is because the trick it is using uses operator--. I'm sure the original author did this in his/her sleep and thought it was too trivial to post (or posted it before?) but I thought: if we can use operator! instead, can't we create analog literals that use the same number of symbols as the number we want?</p> The answer is yes, and it's pretty simple:</p> notliterals.h: class NotLiteral { public: NotLiteral( unsigned int ival ) : val_( ival ) { }</p> NotLiteral operator!() const { return NotLiteral( val_ + 1 ); } operator unsigned int() const { return val_; } unsigned int val_; </code></pre> };</p> const NotLiteral NL( 0 ); </pre></p> test_notliterals.cpp: #include "notliterals.h" #include <cassert></p> int main() { assert( !!!!NL == 4 ); assert( !!NL == 2 );</p> assert( !!!!!!!!!!!!!!!NL == 15 ); </code></pre> } </pre></p> With this simpler form, it's almost believable that there might be some kind of useful application?</p> Extending this to 3 dimensions is left as an exercise for the reader. For 2 dimensions, if you just want the area (not the width and height), how about this?:</p> assert( !!! !!! !!!NL == 9 ); </pre> Update:</strong> By the way, if you don't like all the emphasis! of! using! exclamation! marks! you can do the same thing with the unary one's complement operator, ~. Just replace "!" everywhere above with "~" and you're done. Unfortunately, you can't do the same with - or + because the parser recognises "--" as the decrement operator instead of seeing that it is clearly</em> two calls to the unary negation operator.</p> Separate regular expressions, or one more complex one? Andy Balaam — Fri, 03 Apr 2009 06:44:21 +0000 I have asked myself this question several times, so I thought it was about time I did a test and found an answer.</p> If the user of your program can supply you with a list of regular expressions to match against some text, should you combine those expressions into one big one, or treat them separately?</p> In my case I need an OR relationship, so combining them just means putting a pipe symbol between them.*</p> So: one expression made by ORing, or looping through several - which is better? There's only one way to find out</a>:</p> import re, sys line_with_match_foo = "This line contains foo." line_with_match_baz = "This line contains baz." line_without_match = "This line does not contain it." re_strings = ( "foo", "bar1", "bar2", "baz", "bar3", "bar4", ) piped_re = re.compile( "|".join( re_strings ) ) separate_res = list( re.compile( r ) for r in re_strings ) NUM_ITERATIONS = 1000000 def piped( line ): for i in range( NUM_ITERATIONS ): if piped_re.search( line ): print "match!" # do something def separate( line ): for i in range( NUM_ITERATIONS ): for s in separate_res: if s.search( line ): print "match!" # do something break # stop looping because we matched arg = sys.argv[1] if arg == "--piped-nomatch": piped( line_without_match ) elif arg == "--piped-match-begin": piped( line_with_match_foo ) elif arg == "--piped-match-middle": piped( line_with_match_baz ) elif arg == "--separate-nomatch": separate( line_without_match ) elif arg == "--separate-match-begin": separate( line_with_match_foo ) elif arg == "--separate-match-middle": separate( line_with_match_baz ) </pre> And here are the results:</p> $ time python re_timings.py --piped-nomatch > /dev/null real 0m0.987s user 0m0.943s sys 0m0.032s $ time python re_timings.py --separate-nomatch > /dev/null real 0m3.695s user 0m3.641s sys 0m0.037s </pre> So when no regular expressions match, the combined expression is 3.6 times faster.</p> $ time python re_timings.py --piped-match-middle > /dev/null real 0m1.900s user 0m1.858s sys 0m0.033s $ time python re_timings.py --separate-match-middle > /dev/null real 0m3.543s user 0m3.439s sys 0m0.042s </pre> And when an expression near the middle of the list matches, the combined expression is 1.8 times faster.</p> $ time python re_timings.py --piped-match-begin > /dev/null real 0m1.847s user 0m1.797s sys 0m0.035s $ time python re_timings.py --separate-match-begin > /dev/null real 0m1.649s user 0m1.597s sys 0m0.032s </pre> But in the (presumably much rarer) case where all lines match the first expression in the list, the separate expressions are marginally faster.</p> A clear win for combing the expressions, unless you think it's likely that most lines will match expressions early in the list.</p> Note also if you combine the expressions the performance is similar when the matching expression is at different positions in the list (whereas in the other case list order matters a lot), so there is probably no need for you or your user to second-guess what order to put the expressions in, which makes life easier for everyone.</p> I would guess the results would be similar in other programming languages. I certainly found it to be similar in C# on .NET when I tried it a while ago.</p> By combining the expressions we ask the regular expression engine to do the heavy lifting for us, and it is specifically designed to be good at that job.</p> Open questions:</p> Have I made a mistake that makes these results invalid?</p> </li> * Can arbitrary regular expressions be ORed together simply by concatenating them with a pipe symbol in between?</p> </li> Can we do something similar if the problem requires us to AND expressions?</p> </li> </ol> New Baby Andy Balaam — Sat, 21 Mar 2009 20:53:22 +0000 Finally the number of people in our family is prime again.</p> Mum, baby, siblings etc. are all really well and enjoying ourselves.</p> Talk in code Andy Balaam — Thu, 05 Mar 2009 10:31:00 +0000 Last week we had an extended discussion at work about how we were going to implement a specific feature.</p> This discussion hijacked our entire Scrum</a> sprint planning meeting</a> (yes, I know, we should have time-boxed</a> it). It was painful, but the guy who was going to implement it (yes, I know, we should all collectively own our tasks) needed the discussion: otherwise it wasn't going to get implemented. It certainly wasn't going to get broken into short tasks until we knew how we were going to do it.</p> Anyway, asides aside, I came out of that discussion bruised but triumphant. We had a plan not only on how to write the code, but also how to test it. I believe the key thing that slowly led the discussion from a FUD</a>-throwing contest into a constructive dialogue was the fact that we began to talk in code</strong>.</p> There are two facets to this principle:</p> 1. Show me the code</h3> As Linus once said, "Talk is cheap. Show me the code.</a>".</p> If you are at all disagreeing about how what you're doing will work, open up the source files in question. Write example code - modify the existing methods or sketch a new one. Outline the classes you will need. Code is inherently unambiguous. White board diagrams and hand-waving are not.</p> Why wouldn't you do this? Fear you might be wrong? Perhaps you should have phrased your argument a little less strongly?</p> Is this slower than drawing boxes on a whiteboard? Not if you include time spent resolving the confusion caused by the ambiguities inherent in line drawings.</p> Does UML</a> make whiteboards less ambiguous? Yes, if all your developers can be bothered to learn it. But why learn a new language when you can communicate using the language you all speak all day - code?</p> 2. Create a formal language to describe the problem</h3> If your problem is sufficiently complex, you may want to codify the problem into a formal (text-based) language.</p> In last week's discussion we were constantly bouncing back and forth between different corner cases until we started writing them down in a formal language.</p> The language I chose was an adaptation of a Domain-specific language</a> I wrote to test a different part of our program. I would love to turn the cases we wrote down in that meeting into real tests that run after every build (in fact I am working on it) but their immediate value was to turn very confusing "what-if"s into concrete cases we could discuss.</p> Before we started using the formal language, the conversations went something like this:</p> Developer:</strong> "If we implement it like that, this bad thing will happen."</p> Manager:</strong> "That's fine - it's a corner case that we can tidy up later if we need it."</p> Developer:</strong> (Muttering) "He clearly doesn't understand what I mean."</p> Repeat</strong></em></p> After we started using the formal language they went something like this:</p> Developer:</strong> "If we implement it like that, this bad thing will happen."</p> Me:</strong> "Write it down, I tell you."</p> Developer:</strong> (Typing) "See, this will happen!"</p> Manager:</strong> "That's fine - it's a corner case that we can tidy up later if we need it."</p> Developer:</strong> (Muttering) "Flipping managers."</p> Summary</h3> The conversation progresses if all parties believe the others understand what they are saying. It is not disagreement that paralyses conversations - it is misunderstanding.</p> To avoid misunderstanding, talk in code - preferably a real programming language, but if that's too verbose, a text-based code that is unambiguous and understood by everyone involved.</p> Note on whiteboards</h3> You can't copy and paste them, and you can't (easily) keep what you did with them, and you can't use them to communicate over long distances.</p> And don't even try and suggest an electronic whiteboard. In a few years they may solve all of the above problems, but not now. They fail the "can I draw stuff?" test at the moment.</p> Even when electronic whiteboards solve those problems, they won't solve the fact that lines and boxes are more ambiguous and less detailed than code in text form.</p> If you all know and like UML, that makes your diagrams less ambiguous, but still they often don't allow enough detail: why bother?</p> Fixing the vertical panel window list on Ubuntu Hardy Andy Balaam — Wed, 10 Dec 2008 22:20:49 +0000 For a long time now I've been bugged by GNOME Bug 86382</a>.</p> Recent versions of GNOME have featured a window list applet which behaves badly when situated on a vertical panel (which is where I like to have mine). Older versions were ok except the buttons would expand and contract to make enough vertical space to hold the window title if written vertically, even though the window titles were written horizontally. Recent versions have much worse behaviour where once you get over about 7 windows open the window list will flicker rapidly between displaying in 1 or two columns making it almost impossible to click on the button you want.</p> There have been various patches attached to the bug, and today I managed to get the latest installed on my Ubuntu Hardy 8.04 machine by building custom .debs for the relevant components. I'm posting exactly what I did here in case it's useful to anyone who wants to get this patch installed, and also as a general reference for what you need to do to apply a patch and install it using Ubuntu's (or Debian's) package management system as a well-behaved package.</p> The patches are to libwnck and gnome-panel, and they implement the behaviour I want, which is horizontal text, and buttons of fixed height in a window list which expands as needed to fill the available space. With this patch installed, I find a vertical panel is the best way to manage a large number of open windows.</p> I don't recommend running this script - better to read and understand it yourself. Nevertheless (and with no guarantees whatsoever) if you do run it as is on a Hardy machine, it should install patched versions of libwnck and gnome-panel which work better with vertical panels.</p> # Get all the libraries and source code we need sudo apt-get install build-essential fakeroot dpkg-dev sudo apt-get build-dep libwnck gnome-panel # Create a directory to do all our messing about in mkdir build-libwnck cd build-libwnck/ # Get the source code for the 2 components we are building apt-get source libwnck apt-get source gnome-panel # Do libwnck first cd libwnck-2.22.3 # Create a changelog entry cat <<END > new-entry.txt libwnck (2.22.3-0ubuntu2) hardy-proposed; urgency=low * applied libwnck vertical window list fix from Alexey Rusakov and Vincent Untz -- Anon <anon@example.com> Wed, 10 Dec 2008 20:50:00 +0000 END cat new-entry.txt debian/changelog > debian/changelog-new rm debian/changelog mv debian/changelog-new debian/changelog rm new-entry.txt # Get the libwnck patch and apply it wget -O libwnck-fix-vertical.patch "http://bugzilla.gnome.org/attachment.cgi?id=124112" patch -p2 < libwnck-fix-vertical.patch # Build the package aclocal automake dpkg-buildpackage -rfakeroot -b # Install the libwnck packages cd .. sudo dpkg -i libwnck*.deb # Now do gnome-panel cd gnome-panel-2.22.2 # Create a changelog entry cat <<END > new-entry.txt gnome-panel (1:2.22.2-0ubuntu1.2) hardy-proposed; urgency=low * applied libwnck vertical window list fix from Alexey Rusakov and Vincent Untz -- Anon <anon@example.com> Wed, 10 Dec 2008 20:50:00 +0000 END cat new-entry.txt debian/changelog > debian/changelog-new rm debian/changelog mv debian/changelog-new debian/changelog rm new-entry.txt # Get the gnome-panel patch and apply it wget -O gnome-panel-fix-vertical.patch "http://bugzilla.gnome.org/attachment.cgi?id=107133" patch -p2 < gnome-panel-fix-vertical.patch # Build the package aclocal automake dpkg-buildpackage -rfakeroot -b # Install the gnome-panel packages cd .. sudo dpkg -i *panel*.deb </pre> Update: fixed incorrect comment - thanks Stefan.</strong></p> Update: changed -p0 to -p2 in the patch commands.</strong></p> Comments</h2> Alexander Kojevnikov 2008-12-10</span> </h3> You are my hero Andy, thanks!</p> </div> Andy Balaam 2008-12-11</span> </h3> Save your praise for Alexey Rusakov and Vincent Untz!</p> </div> Andy Balaam 2008-12-11</span> </h3> To use my version of the patch which makes the taskbar buttons taller (but in a very bad way for which I should be punished), replace the first wget command with this one:</p> wget -O libwnck-fix-vertical.patch "http://bugzilla.gnome.org/attachment.cgi?id=124396"</a></p> </div> Stefan Huszics 2008-12-11</span> </h3> Thanks a bunch Andy. This will allow me to upgrade my laptop to 8.04.</p> The bonus of learning how to apply patches to .deb is appreciated as well. :)</p> </div> Stefan Huszics 2008-12-11</span> </h3> BTW, you have an error above "Install the gnome-panel packages"</p> cd .. sudo dpkg -i libwnck*.deb</p> obviously "gnome-panel" in the comment is wrong</p> </div> Stefan Huszics 2008-12-11</span> </h3> Also, I needed to install automake 1.9 for gnome-panel (default on system was 1.7.9, and latest 1.10.1) and then run aclocal-1.9 automake-1.9</p> </div> Andy Balaam 2008-12-11</span> </h3> Thanks for the info Stephan – I have made the correction. The biggest bonus for me too was learning how build custom debs. It lowers the barrier a lot to hacking on GNOME if I don’t have to build an entire new version from SVN, but can just apply little tweaks to the installed version.</p> Of course, that comes with risks. I did break my panel completely at one point and had to do:</p> links2 http://archive.ubuntu.com/ubuntu/pool/main/</a></p> and find the right versions of the packages I was replacing and install them over the top of the ones I had broken.</p> Of course, the advantage of using debs is that I knew exactly which packages I needed to download and install to get back to a working system. If I’d been building from tarballs it would have been much harder to work out.</p> </div> cb474 2008-12-11</span> </h3> It turned out I had to install automake, because I was getting a "command not found" error when I tried to run aclocal. This is on a Debian Testing system.</p> Then I got to the step where I have to run aclocal the first time and I got these two errors:</p> configure.in:34: warning: macro AM_PROG_LIBTOOL' not found in library configure.in:73: warning: macro</code>AM_GLIB_GNU_GETTEXT’ not found in library</p> The first warning was solved by installing libtool. But I’m still getting the second one about gettext and can’t figure out how to solve that problem. There is a "gettext.m4" file in my /usr/share/aclocal. Any suggestions about how to fix this error and move on to the next step?</p> (Thanks for the Howto, by the way, I really appreciate it.)</p> </div> Stefan Huszics 2008-12-11</span> </h3> > It lowers the barrier a lot to hacking on GNOME</p> Indeed, used to have Gentoo myself and compiled everything from source, so was quite puzzled of how to do things in Ubuntu without breaking the entire desktop (including with future security updates). Only messing with the exact .debs lowers that risk by a lot.</p> Now I just need to figure out how to hack the patch so it doesnt rotate the text already at <80px , <30px would fit me a lot better.</p> </div> Billy Buerger 2008-12-15</span> </h3> I’ve been following that bug for a while now. But didn’t know where to start for applying the patch. Thanks for the help. Although I’m running 8.10 which uses V2.24.1 of the gnome packages. So my first attempt to apply the patch failed and broke things pretty good. Maybe there’s a better way, but I went through the changes in the patch and made the modifications manually. Now I finally have my window list working correctly in Ubuntu. About freakin’ time.</p> Now if only some of the other applets played better in vertical panels. At least those are mostly cosmetic. And since your little article here pointed me in the right direction for installing modifications, maybe I’ll play around with editing some of them myself.</p> </div> Andy Balaam 2008-12-15</span> </h3> Hi Billy – glad to be of help! Have fun making the other applets behave. Which ones cause you problems? I don’t tend to run many myself.</p> </div> Billy Buerger 2008-12-16</span> </h3> Main Menu button is pretty big. </li> Quick Lounge (Launchers List) seems to always want to be square. But at least it displays the icons horizontally first. </li> Notification icons sometimes are big. But it doesn’t seem like they always are. Also stacks them vertically instead of horizontal. </li> Many of them stretch the image to the full width of the panel. Similar to what the main menu seems to do. Trash and Volume come to mind. Although neither are particularly important. Especially since the sound doesn’t work correctly on my laptop. Also seems that these aren’t always a problem depending on your icon set. If it doesn’t have a big icon in the theme, then it stays small.</li> </ul> </div> Andy Balaam 2008-12-19</span> </h3> @cb474: Sorry, I missed your comment in the spam moderation queue.</p> I don’t know how to fix that, sorry. Is gettext itself a separate package that can be installed?</p> </div> Walter Huf 2009-04-26</span> </h3> I just upgraded my laptop to Jaunty, and your walkthrough sadly does not work on Jaunty. When I try to run the first dpkg-buildpackage, it compiles and then fails after installing a bunch of html files. The exact error message is: ……. — Installing ./html/WnckTasklist.html — Installing ./html/WnckWindow.html — Installing ./html/WnckWorkspace.html /bin/bash: line 20: —relative: command not found make[3]: *** [install-data-local] Error 127 make[3]: Leaving directory /home/hufman/wnk.new/libwnck-2.26.0/doc' make[2]: *** [install-am] Error 2 make[2]: Leaving directory</code>/home/hufman/wnk.new/libwnck-2.26.0/doc’ make[1]: *** [install-recursive] Error 1 make[1]: Leaving directory `/home/hufman/wnk.new/libwnck-2.26.0’ make: *** [common-install-impl] Error 2 dpkg-buildpackage: failure: fakeroot debian/rules binary gave error exit status 2</p> Do you have any suggestions? Everything worked fine in Intrepid… I don’t know what’s wrong.</p> </div> Andy Balaam 2009-04-27</span> </h3> @Walter, I’m sorry I don’t have any idea what’s causing this :( Is it just the docs that are going wrong? If so, there might be a ./configure flag that disables building the documentation?</p> </div> John Bartholomew 2009-04-29</span> </h3> First, thank you so much for writing this, doing it this way was much easier than a manual build from the gnome git repositories (which I tried first).</p> @Walter: I had exactly the same problem. The reason (well, immediate reason at least) is that the GTKDOC_REBASE variable isn’t being set up, and make is trying to run ‘$(GTKDOC_REBASE) —relative …’, which expands into just ‘—relative …’, which bash then complains about since there is no —relative command. I don’t know what the real underlying cause is though. My solution was: make sure the gtkdoc package is installed, check that /usr/bin/gtkdoc-rebase exists, and then run ‘export GTKDOC_REBASE=/usr/bin/gtkdoc-rebase’ before re-running dpkg-buildpackage. The build should now succeed. I didn’t have any other problems with Andy’s instructions, on a fresh Jaunty install.</p> </div> John Bartholomew 2009-04-29</span> </h3> nb: Just for completeness (not that it really matters), all these ‘—relative’ things are two</em> dashes, then ‘relative’ — wordpress is just turning double-dash into an em-dash (which is quite right, just not good for code :P)</p> </div> Andy Balaam 2009-04-30</span> </h3> @John, thanks for the encouragement, and thanks for the advice for Walter!</p> </div> amnuxoll 2009-05-01</span> </h3> cb474:</p> I also got the AM_GLIB_GNU_GETTEXT error. I just fixed it by installing the libglib2.0-dev package.</p> Now let’s see if the rest of this works…</p> </div> amnuxoll 2009-05-01</span> </h3> No go for me. I got this error installing the patched gnome-panel:</p> dpkg: dependency problems prevent configuration of gnome-panel: gnome-session (2.26.0svn20090408-0ubuntu2) breaks gnome-panel (<< 1:2.23) and is installed.</p> Apparently I need to update gnome-session too? I can tell I’m in up to my neck already and don’t want to tinker too much for fear of trashing the whole thing.</p> </div> amnuxoll 2009-05-01</span> </h3> Ok, it does</em> work. I just needed to ignore the scary errors and have faith. Upon reboot the panel bug is gone on my system. Thank you, Andy!</p> The Synaptic Package Manager complains that I’ve got a broken dependency now but I’m pretty sure I’ll just have to live with that.</p> </div> Alexander Kojevnikov 2009-09-06</span> </h3> Carey Underwood posted a patch in bgo#86382 which works quite well. Instructions for Arch Linux are here: http://versia.com/2009/09/05/arch-linux-yaour...</a></p> </div> Andy Balaam 2009-09-06</span> </h3> Thanks Alexander – how does this patch improve on the older ones?</p> </div> Alexander Kojevnikov 2009-09-08</span> </h3> Both do the job, but Carey’s patch is much simpler and smaller; and changes only libwnck</p> </div> Dylan 2010-11-15</span> </h3> does anyone have an updated patch for the latest gnome-panels? I can’t get it working…</p> </div> Andy Balaam’s Blog » Blog Archive » Reclaiming desktop space with Ubuntu’s window-picker-applet 2011-06-16</span> </h3> […] a vertical task bar on the left and launcher buttons across the top. I’ve struggled with a long-standing bug with using a vertical panel, and I’ve sacrificed quite a lot of screen space to make enough room to see the window […]</p> </div> An actual difficult bug fixed Andy Balaam — Wed, 08 Oct 2008 03:21:04 +0000 Of course, I am bound to get a bug report immediately I have posted this telling me my fix breaks everything, but for the moment I am chuffed that I found, tested, and fixed a genuinely difficult bug</a>.</p> I am particularly proud because I wrote an automated test to ensure it can never happen again, and I used that test to make the debugging process much easier than it otherwise would be. The code that reads, processes and stores listings in FreeGuide is a spider's web of interfaces and helper classes (because of the arguably over-engineered plugins framework used for all the moving parts), and tracking this down with plain old-fashioned debugging would have been a huge job.</p> Anyway, I bet you are dying to hear what the bug was, aren't you?</p> When you have a programme already stored in FreeGuide, and then a new one comes along that overlaps it, the old programme is deleted. For example if we start off with:</p> ... 19:00 Top Gear ................... 20:00 Charlie and Lola .............</pre> but then later download listings again and get these programmes:</p> ... 19:00 Pocoyo .. 19:15 Round the World ...... </pre> Then Top Gear</em> will disappear, and be replaced by the 2 new programmes. In fact, any old programme that overlaps any new incoming programme will be automatically deleted.</p> At least, that is what is supposed to happen. In fact, the real situation is a little more complex because the programmes are stored in separate files (.ser files) for different days and times. Actually, there are 4 files for each day, named things like "day-2008-09-15-A.ser", where the suffix A, B, C and D indicate which part of each day a file is for.</p> So imagine what happens when the first set of programmes comes in looking like this:</p> 19:00 Programme 1A ......... 21:15 Programme 1B .. 21:30 Programme 1C ........... 22:00</pre> and then the second comes in like this:</p> 19:00 Programme 2A......................................... 21:45 Programme 2C .. 22:00</pre> So obviously the old 3 programmes should be completey deleted, and the new 2 should be what you see.</p> But you don't. In fact what you see is programme 1B and programme 2C, before 1B and between the two. Weird huh?</p> "Why?" I hear you ask. Well, it's simple when you consider how the programmes are split into files.</p> Programme 1A goes into file day-2008-09-14-D.ser, and programmes 1B and 1C go into day-2008-09-15-A.ser.</p> [Side note: this is true in this case because the bug reporter is in the GMT -0400 timezone and the file boundaries are quarters of a day in GMT.]</p> Then, when the new programmes come along, 2A goes into 14-D - wiping out 1A, and 2C goes into 15-A - wiping out 1C but not 1B.</p> Then, when the files get read back in again later, 2A is read from 14-D, but then 1B is read from 15-A, wiping out 2A, and finally 2C is read in as well from 15-A, so we end up with 1B and 2C.</p> How to fix it? Well, what I did was leave everything as it is, and then do the final read in the reverse order. This means we read in 1B and 2C, but then we read 2A later, and it wipes out 1B, leaving 2A and 2C as we would expect.</p> Neat fix eh? It works because this kind of wrongness in the .ser files will only exist when a programme hanging off the end should have wiped out something in a later file. Because programmes are classified into files by their start time, they can only hang off the end of a file, not the beginning, so reading the files in backwards will always read the hanging-over file last, wiping out anything which should have been wiped out earlier.</p> There is a little bug/feature remaining, but it only applies when you get some really weird listings from your provider. If you had a programme like 1A (19:00 - 21:15), and downloaded new listings, which ONLY contained a programme overlapping it, but falling into a later file (so maybe it starts at 21:00), and didn't contain any programme starting at 19:00, then the backwards reading would mean you would never see your new programme because it would be wiped out by 1A.</p> This is a very unusual case though, since normally if you get a new programme at 21:00, you will also get new programmes leading up to it, if only to reflect the fact that 1A is now a different length. So this is really a theoretical bug, which explains why I've decided not to fix it...</p> Anyway, by the time I'd fiddled with my test for this to get the bug to trigger (which took a long time - working out which bits to fake out and which to test at all was tricky), the actual fix was easily implemented (1 line of code I chose to break out into 3), and then validated in a single click.</p> Just in case I hadn't mentioned it, I love tests.</p> FreeGuide 0.10.9 Andy Balaam — Tue, 09 Sep 2008 22:33:01 +0000 So let's quietly forget FreeGuide 0.10.8, which basically didn't work.</p> I fixed the problem (which was some nasty threading thing) by making the initial "Choose Channels for XMLTV" call from the first time wizard not do the clever stuff of displaying errors to the user. Instead it dumps any errors to the console (which is an improvement over swallowing them completely as it did before), and when you run "Choose Channels for XMLTV" from the menu, or "Configure" from the Options screen, you still get the errors displayed in a nice dialog as in 0.10.8.</p> More excitingly, new in this release: some unit tests!</p> You can run them by doing this in the src directory:</p> java freeguide.test.slow.FreeGuideSlowTest</code></p> They test the fixes I made in 0.10.9 for XMLTV files that have certain attributes missing, and have different date formats.</p> The date formats were supposed to be already working, but a fix I made several years (yes, years</em>) ago broke them. Why didn't I catch the problem? No tests.</p> The second age of FreeGuide is coming. I shall call it The Time of Testing.</p> Wii repaired free! Andy Balaam — Tue, 09 Sep 2008 22:19:38 +0000 A few weeks ago my Nintendo Wii started not accepting any Wii disks (but was still fine for GameCube ones... interesting). It made a clicking noise for a bit and then declared "Bad Disk" or something like that.</p> I need my Wii. I watch all my TV through it, using recordtv</a>. I had to get it fixed.</p> I looked up the official support options, and the deal in the UK is that you go to http://www.nintendoservicecentre.co.uk and they send you an envelope. You put your Wii in the envelope (postage paid) and they have a look at it. Then they give you one of two fixed prices: either Â£28 or Â£70 depending on the severity of the problem. (I guess Â£70 is about the cost price of a Wii?). If you agree on the price, you pay them and they send it back fixed, postage paid.</p> So I sent mine off, worried and entertainment-less. I heard nothing for about a week, and then it returned, fixed, with a little sticker on it saying "Repair: Wii minor, Â£0.00."</p> Paint me satisfied.</p> +1 Nintendo.</p> Another one Andy Balaam — Mon, 01 Sep 2008 21:08:46 +0000 </p> Due in March. Very excited.</p> FreeGuide 0.10.8 Andy Balaam — Mon, 28 Jul 2008 20:00:38 +0000 I am still working slowly on moving FreeGuide forward. Somehow it seems my itches for FreeGuide are all about making it less annoying for people who are trying it the first time. I guess this is motivated by my desire for world domination.</p> Anyway, we are one small step closer to my mum being able to use FreeGuide - when the "Choose channels" step (i.e. the XMLTV grabber configuration) goes wrong, you can now see a real genuine error message, and hopefully figure out what went wrong.</p> Actually, it always used to work that way but the error-catching got refactored away at some point. Anyway, I am slowly taking the ground back...</p> As I do more and more test-driven development at work I am becoming completely addicted. For this FreeGuide code I wrote a couple of unit tests but they are not within a proper framework, and can't be launched easily as a test suite. I am considering JUnit.</p> I also want to set up some component-level tests e.g. for downloading listings for each country and checking everything works as expected. It's brilliant fun having tests in place, but when you have as little time as I have for FreeGuide at the moment, it's difficult to decide to spend a long time working on a test framework when I could be fixing a "real user problem" or adding a cool new feature.</p> But I've got the testing bug badly, so watch this space.</p> Debugging memory use and fragmentation on Windows using Address Space Monitor Andy Balaam — Fri, 18 Apr 2008 10:29:11 +0000 At work at the moment we are putting a lot of effort into making our program not crash. Sensible, eh?</p> It's crashing because a) it uses an enormous amount of memory and b) it tends to fragment your remaining memory. One of the characteristics of this program is that from time to time it needs a large contiguous chunk of memory so that it can pass a big bit of XML to someone else. This tends to be a problem when your memory is fragmented.</p> For example today I was running it, and it was using 1GB of memory (+about 0.3GB reserved), leaving 0.7GB free in the 2GB address space. However, the largest available contiguous block of memory was about 60MB.</p> The long-term answer is obviously to use less memory, but in the shorter term we have had quite a bit of success by writing code that works around the fragmentation. For example, we have a custom mutable string-like class that can store its chars in several separate blocks of memory instead of all in one contiguous block.</p> Debugging this kind of thing is always a very difficult process. For me it has transformed from an almost-impossible task into a tractible one because of Charles Bailey</a>'s Address Space Monitor</a> tool. This gives you a clear picture of your process' address space, and version 0.6a</a> adds recording functionality that makes reports to impress managers a lot easier to produce.</p> Without it, we wouldn't know where to start.</p> C++ is an expert language Andy Balaam — Mon, 07 Apr 2008 09:26:44 +0000 Update: I should point out at the beginning that I love C++. Anything below which sounds bitter or critical is borne of a deep and growing love. C++ is a journey into worlds of beauty and strength.</strong></p> I assert that C++ is an expert language. What do I mean?</p> You shouldn't be allowed to use C++ in anger unless you've used it for 2 years in anger.</p> </blockquote> Aside: what should we do about this?</em></p> In practice this means that no-one should recruit newbie C++ developers.</em></p> The only alternative is to have some kind of apprenticeship system, where all the code written by a newbie is re-written by their mentor for about 2 years. This is a great learning experience, and could weed out people with insufficient capacity for humility to be a C++ programmer. (Note I say capacity for humility, because the actual humility will be provided by the constant crushing of your spirit provided by someone tearing your code apart every day.)</em></p> In Java, to create an "array", and add an object to it, you do this:</p> MyObj obj = new MyObj(); ArrayList<MyObj> arr = new ArrayList<MyObj>(); arr.add( obj ); </pre> In Python, you do this:</p> obj = MyObj() arr = [] arr.append( obj ) </pre> In C, you do this:</p> MyStruct obj; MyStruct arr[50] = { obj }; </pre> In Perl, you do this:</p> my $obj = MyObj->new(); my @arr; push(@arr,$obj); </pre> In Pascal, you do this:</p> var arr : ARRAY [1..50] of int; begin arr[1] := 7; end </pre> In Haskell, you might do something like this (thanks to Neil Mitchell):</p> [myObj] </pre> Don't get het up about the fact that these examples do different things: my point is, they are reasonable ways of performing the task (add something to an array) in the languages chosen. (Please do send in corrections, though - none of these were checked and they are probably wrong.)</p> Note that the C example hides a little more complexity because you need to make sure you tidy up your memory afterwards.</p> Now, what do you do in C++? It's just as easy, right?</p> MyObj obj; std::vector<MyObj> arr; arr.push_back( obj ); </pre> WRONG!</p> In the examples above, you don't need to know what is going on under the covers</strong>.</p> In fact, in general I suggest there are broadly two types of programming language around at the moment: those where you have to know how things work, but where how things work is quite simple (e.g. C, assembly languages, maybe FORTRAN and COBOL) and those which isolate you from how things work (all the rest?).</p> Where does C++ fit in to this scheme? It is in the unique position of being a language which has incredibly complex things going on under the covers, and you have to know about it!</strong></p> What do I mean by saying you have to know what's going on under the covers?</p> Let's look at our example again, and ask this question: what types of object can you put in the array? In the other programming languages above, you can essentially put any "normal" objects (where normal is different for each example) into the array.</p> In C++, here are some of the rules you need to understand about objects you can put into std::vector. You should understand these before you try using std::vector. If you can't understand them, you should think hard until you do.</p> Default constructor</h3> If you want to give the size of the vector when you create it (or resize it later), your object must have a default constructor [Stroustrup §16.3.4</em>].</p> (Note: if you don't define any other constructors, the compiler will automatically define a default constructor for you (which may or may not do what you want). If you do, the default constructor is the one that can be called without any arguments [Stroustrup §10.4.2</em>].</p> Example:</p> $ cat default_constructor_required.cpp #include <vector> class MyObject { public: MyObject( int num ) { } }; int main( int argc, const char* argv[] ) { std::vector<MyObject> arr( 5 ); } $ g++ default_constructor_required.cpp /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_vector.h: In constructor â€˜std::vector<_Tp, _Alloc>::vector(size_t) [with _Tp = MyObject, _Alloc = std::allocator<MyObject>]â€™: default_constructor_required.cpp:12: instantiated from here /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_vector.h:219: error: no matching function for call to â€˜MyObject::MyObject()â€™ default_constructor_required.cpp:5: note: candidates are: MyObject::MyObject(int) default_constructor_required.cpp:4: note: MyObject::MyObject(const MyObject&) </pre> Nice error message, eh?</p> (Note: if your array contains a built-in type (e.g. int) it will be initialised to 0 in its default constructor [Stroustrup §4.9.5</em>].</p> If you don't want to give the size of the vector when you create it (not recommended), then you don't need a default constructor in your object [Stroustrup §16.3.4</em>].</p> Copy constructor</h3> Your object must also have a copy constructor. Example:</p> $ cat copy_constructor_required.cpp #include <vector> class MyObject { public: MyObject() { } private: MyObject( const MyObject& ); }; int main( int argc, const char* argv[] ) { std::vector<MyObject> arr( 5 ); } $ g++ copy_constructor_required.cpp /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_vector.h: In constructor â€˜std::vector<_Tp, _Alloc>::vector(size_t) [with _Tp = MyObject, _Alloc = std::allocator<MyObject>]â€™: copy_constructor_required.cpp:15: instantiated from here copy_constructor_required.cpp:10: error: â€˜MyObject::MyObject(const MyObject&)â€™ is private /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_vector.h:219: error: within this context /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_construct.h: In function â€˜void std::_Construct(_T1*, const _T2&) [with _T1 = MyObject, _T2 = MyObject]â€™: /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_uninitialized.h:194: instantiated from â€˜void std::__uninitialized_fill_n_aux(_ForwardIterator, _Size, const _Tp&, __false_type) [with _ForwardIterator = MyObject*, _Size = unsigned int, _Tp = MyObject]â€™ /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_uninitialized.h:218: instantiated from â€˜void std::uninitialized_fill_n(_ForwardIterator, _Size, const _Tp&) [with _ForwardIterator = MyObject*, _Size = unsigned int, _Tp = MyObject]â€™ /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_uninitialized.h:310: instantiated from â€˜void std::__uninitialized_fill_n_a(_ForwardIterator, _Size, const _Tp&, std::allocator<_Tp2>) [with _ForwardIterator = MyObject*, _Size = unsigned int, _Tp = MyObject, _Tp2 = MyObject]â€™ /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_vector.h:219: instantiated from â€˜std::vector<_Tp, _Alloc>::vector(size_t) [with _Tp = MyObject, _Alloc = std::allocator<MyObject>]â€™ copy_constructor_required.cpp:15: instantiated from here copy_constructor_required.cpp:10: error: â€˜MyObject::MyObject(const MyObject&)â€™ is private /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_construct.h:81: error: within this context </pre> (Aside: this program is 181 bytes, and the error message is 1950 bytes.)</p> Assignment operator</h3> You also need operator=. Example:</p> $ cat assignment_operator_required.cpp #include <vector> class MyObject { private: MyObject& operator=( const MyObject& ); }; int main( int argc, const char* argv[] ) { MyObject obj; std::vector<MyObject> arr; arr.push_back( obj ); } $ g++ assignment_operator_required.cpp /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/vector.tcc: In member function â€˜void std::vector<_Tp, _Alloc>::_M_insert_aux(__gnu_cxx::__normal_iterator<typename _Alloc::pointer, std::vector<_Tp, _Alloc> >, const _Tp&) [with _Tp = MyObject, _Alloc = std::allocator<MyObject>]â€™: /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_vector.h:610: instantiated from â€˜void std::vector<_Tp, _Alloc>::push_back(const _Tp&) [with _Tp = MyObject, _Alloc = std::allocator<MyObject>]â€™ assignment_operator_required.cpp:13: instantiated from here assignment_operator_required.cpp:6: error: â€˜MyObject& MyObject::operator=(const MyObject&)â€™ is private /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/vector.tcc:260: error: within this context /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_algobase.h: In static member function â€˜static _BI2 std::__copy_backward<_BoolType, std::random_access_iterator_tag>::copy_b(_BI1, _BI1, _BI2) [with _BI1 = MyObject*, _BI2 = MyObject*, bool _BoolType = false]â€™: /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_algobase.h:443: instantiated from â€˜_BI2 std::__copy_backward_aux(_BI1, _BI1, _BI2) [with _BI1 = MyObject*, _BI2 = MyObject*]â€™ /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_algobase.h:482: instantiated from â€˜static _BI2 std::__copy_backward_normal<true, true>::copy_b_n(_BI1, _BI1, _BI2) [with _BI1 = __gnu_cxx::__normal_iterator<MyObject*, std::vector<MyObject, std::allocator<MyObject> > >, _BI2 = __gnu_cxx::__normal_iterator<MyObject*, std::vector<MyObject, std::allocator<MyObject> > >]â€™ /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_algobase.h:517: instantiated from â€˜_BI2 std::copy_backward(_BI1, _BI1, _BI2) [with _BI1 = __gnu_cxx::__normal_iterator<MyObject*, std::vector<MyObject, std::allocator<MyObject> > >, _BI2 = __gnu_cxx::__normal_iterator<MyObject*, std::vector<MyObject, std::allocator<MyObject> > >]â€™ /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/vector.tcc:257: instantiated from â€˜void std::vector<_Tp, _Alloc>::_M_insert_aux(__gnu_cxx::__normal_iterator<typename _Alloc::pointer, std::vector<_Tp, _Alloc> >, const _Tp&) [with _Tp = MyObject, _Alloc = std::allocator<MyObject>]â€™ /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_vector.h:610: instantiated from â€˜void std::vector<_Tp, _Alloc>::push_back(const _Tp&) [with _Tp = MyObject, _Alloc = std::allocator<MyObject>]â€™ assignment_operator_required.cpp:13: instantiated from here assignment_operator_required.cpp:6: error: â€˜MyObject& MyObject::operator=(const MyObject&)â€™ is private /usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../include/c++/4.0.3/bits/stl_algobase.h:412: error: within this context </pre> (Aside: program is 202 bytes, error is 2837 bytes.)</p> Excuses</h3> Don't give me that "the compiler will provide them for you" excuse. What the compiler provides is often wrong, unless you've been careful to ensure you don't own any members by holding pointers to them: i.e. if you've fully understood the problem I am setting out and avoided the pitfalls.</p> Conclusion</h2> I assert that C++ is an expert language. Quite apart from the fact that the method names on STL objects use archane phrases like "push_back" rather than "add", and the error messages you get from popular compilers are huge and almost incomprehensible, my main point is that you have to understand the basics of how the standard library is implemented, before you can use it. This is expert behaviour.</p> I have illustrated this point by showing what you need to know to use the standard resizeable array type in C++. You need to know a lot.</p> More on whether the fact that C++ is an expert language is a bad thing, later.</p> Update: simplified the C example thanks to Edmund's suggestion.</strong></p> Update 2: corrected the Java example thanks to Anon's comment.</strong></p> Update 3: corrected the Haskell example thanks to Neil Mitchell's comment.</strong></p> Comments</h2> Perl Coding School » Blog Archive » perl code [2008-04-07 11:39:45] 2008-04-07</span> </h3> […] C++ is an expert language By Andy Balaam The only alternative is to have some kind of apprenticeship system, where all the code written by a newbie is re-written by their mentor for about 2 years. This is a great learning experience, and could weed out people with insufficient … Andy Balaam’s Blog – http://www.artificialworlds.net/blog</a> […]</p> </div> Tac-Tics 2008-04-07</span> </h3> Your Haskell and Python examples are whack. Those are the only two I checked, but you should probably be more careful if you’re going to make a point about language inexperience =–)</p> > obj = MyObj() > arr = [] > arr.append( obj )</p> No one would do this. This style is too much like Java =-P A real Pythonist would do it simply like this, unless they needed the handle to obj.</p> arr = [MyObj()]</p> The Haskell code is more less good. > [ get_my_obj :: MyObj ] In what you have, you annotate the type of get_my_obj inside a list. There are many subtle errors here. First, annotations in Haskell are almost always omitted except for top-level functions. The type checker can infer it all. Next, it’s very awkward to put the annotation INSIDE the list (though, it is totally legal). If the type checker did indeed require get_my_obj to be annotated, it would almost always be written as:</p> [get_my_obj] :: [MyObj]</p> Showing that the list, as a whole, is a list containing MyObj objects. Lastly, and this one is the biggest issue, get_my_obj is named as if it were a factory for a MyObj object. But things are weird in Haskell, and get_my_obj is a constant</em>. That is, it doesn’t create a new object every time this line gets run. It returns the same one every time. What you want to do is probably: 1) apply some arguments to it; 2) use a constructor and apply arguments to that, not a function; or 3) use a monad for a side-effectful object creation.</p> 1) [sin 1.0]</p> 2) [Person "Haskell" "Curry"]</p> 3) do { input</p> </div> T4C 2008-04-07</span> </h3> However, using C++ has the huge advantages that make it all worth it:</p> [ ] easier to manage [ ] performs faster [ ] faster to code in [ ] easier to debug [ ] better object oriented logic [x] has been around for a long time [x] shouldn’t be used by anyone who hasn’t been coding in it for two years</p> That sounds sarcastic but "has been around for a long time" is damning — it curses us with all that legacy code.</p> The frustrating aspect of C++’s "expert"-ness is that it’s a volume of details, not concepts that are required to be a competent C++ coder. Internalizing "push_back" is not going to contribute to any lifelong coding skill, that’s for sure.</p> </div> markus 2008-04-07</span> </h3> But one problem is that, with languages like Ruby and Python becoming more popular, and C still being the oldschool powerhorse – why is the complexity inside C++ needed?</p> I mean newbies produce more errors everywhere, and in C++ you shoot at your foot and the whole neighbor village is gone. But why use C++ when one has C available and can combine it with i.e. Ruby anyway?</p> To me it seems that the language is just too complex, because if you continually surprise humans, than who is at fault – the language, or the human?</p> </div> noah 2008-04-07</span> </h3> perl could be simplifies to push @arr, MyObj->new();</p> The temproary variable isn’t required. Python quite likely would support similar shortening</p> </div> kL 2008-04-07</span> </h3> in D: auto obj = new MyObj; MyObj[] array; array ~= obj;</p> </div> Riaan Minne 2008-04-07</span> </h3> I personally like to believe your point that you have to understand how things work under the hood of a language should be applied to any language. Basic logical computer science concepts like arrays and stack and so forth are arguably universal in the world of computer programming, the language of choice does not change it.</p> I have been coding Java for almost 8 years now, and yes I admit due to its high level nature I am still learning a thing or two now and then. The difference is that I fully understand these experts "concepts", and fully confident I can implement it in almost any language of my choice since the computational logic does not change (barring language implementation differences, but not even that should be an issue).</p> I agree with you main point of C++ being a expert language, but I believe an expert should necessarily be distinguished by his/hers language of choice.</p> </div> Riaan Minne 2008-04-07</span> </h3> I personally like your point, but the idea that you have to understand how things work under the hood of a language should be applied to any language. Basic logical computer science concepts like arrays and stacks and so forth are arguably universal in the world of computer programming, the language of choice does not change it.</p> I have been coding Java for almost 8 years now, and yes I admit due to its high level nature I am still learning a thing or two about JVM inner workings now and then. The difference is that I fully understand these experts "concepts", and fully confident I can implement it in almost any language of my choice since the computational logic does not change (barring language implementation differences, but not even that should be an issue).</p> I agree with you main point of C++ being a expert language, but I believe an expert should necessarily be distinguished by his/hers language of choice.</p> </div> rrwood 2008-04-07</span> </h3> I think you are right. I worked hard at becoming a good C++ programmer, and it wasn’t until I had a fairly deep understanding of the language, compiler, and runtime libs that it all started making any sense.</p> 2 years is probably a little long, but 6 months of fairly intense work with the help of someone who already Knows The Way would be a reasonable minimum.</p> Personally, now that I’ve actually been working with Python, I can’t ever see myself choosing C++ for new work. Python with C modules for the compute-intensive stuff is WAY more attractive for anything I can see myself doing.</p> I suspect that there are cases where C++ really makes sense. Anybody out there have any good examples? And I’m not talking about cases where the reason for C++ is the existence of a huge body of legacy code that can’t be changed…..</p> </div> Err 2008-04-07</span> </h3> You don’t have to know how the STL is implemented, because the requirements on the type traits of vector elements are specified in the standard. It would be ridiculous to require you to know how the STL is implemented, because there are many implementations; their focal point, however, is the standard.</p> </div> Anon 2008-04-07</span> </h3> I’m no Java pro but do you mean ArrayList (http://java.sun.com/j2se/1.4.2/docs/api/java/...</a> ) not Array (http://java.sun.com/j2se/1.4.2/docs/api/java/...</a> )?</p> </div> erm.... 2008-04-07</span> </h3> if people are not going to hire newbs, how are they going to become !newbs ? (i leave to you what the negation of newb is :) )</p> as a side comment, i know at least one guy that knew more c++ that some "experienced" c++ developers…</p> everyone was a newb once, right??</p> </div> Ric 2008-04-07</span> </h3> Like the C example, the PErl can be simplified from this:</p> my $obj = MyObj->new(); my @arr; push(@arr,$obj);</p> To this:</p> my $obj = MyObj->new(); my @arr = ( $obj );</p> Or even (if you are careful):</p> my @arr = ( MyObj->new() );</p> </div> Neil Mitchell 2008-04-07</span> </h3> In Haskell you would do: [myObj] – type signatures are almost never needed as they are usually inferred.</p> </div> fname lname 2008-04-07</span> </h3> for x=1 to 10 print "Hello World!" next x</p> </div> Mike 2008-04-07</span> </h3> Your example cites only references to the Standard Template Library, which is not an integral part of the C++ language itself. I would agree that the STL is overly complex and does not work in what would seem to be a logical manner, but I don’t know that I would go so far as to claim that "C++ is an expert language."</p> </div> Edgardo Portal 2008-04-07</span> </h3> "I have illustrated this point by showing what you need to know to use the standard resizeable array type in C++. You need to know a lot."</p> The three things you note about C++ are about the first thing most books mention (along with providing a (possibly virtual) destructor), so I’d venture to guess that most people venturing to use the STL would have that under their belts by then.</p> </div> Sgt. Pepper 2008-04-07</span> </h3> Another stupid and pointless C++ flame…</p> </div> Max 2008-04-07</span> </h3> To me it sounds more like you are saying C++ is a language for people that have actually used it for longer than a week.</p> </div> B.G. 2008-04-07</span> </h3> I have to call bollux on this entire post. The truth is, most developers writing C++ code are doing so using MFC, or the Borland equivalent (C++ Builder) – especially in Europe. I’ve worked for 4 different companies (all successful) over the past 10 years. All used C++ as the primary programming language, and all used either MFC or VCL classes as the backbone of all development. None of them used the STL in any way.</p> For my very first C++ job I learned the STL, (read the books and so on). But this learning was never used, as the ‘simplicity’ of the other established libraries always won out at the higher, decision making level.</p> Yes, it’s true that for certain jobs you’re better off writing your own string list, or whatever – but for most Enterprise software (Read: money making software), this is simply not EVER necessary!</p> This post reads as old-school, pretentious, bullcrap! What’s next – every car should have a Porsche engine? Every medical procedure MUST be carried out in the number 1 US hospital? Every wife should appear in Maxim’s top 100 list before the ring is offered, otherwise they’re too ugly and should be dumped at the alter?</p> Get a grip! – on reality that is.</p> </div> Kyle 2008-04-07</span> </h3> I can’t say I agree with this, for one simple reason:</p> Every class in C++ must have a default constructor, a copy constructor, and an assignment operator.</p> Granted, the compiler will let you CREATE a class without these, but even a C++ newbie, even somebody who’s had a single class in the language, knows that every class in C++ must have a default constructor, a copy constructor, and an assignment operator.</p> Once it becomes automatic to create these methods (and they’re usually very simple to create) for every class you write, you don’t really run into these odd problems with the STL.</p> </div> Andy Balaam 2008-04-07</span> </h3> Apologies for the delay approving all your posts. I have been drowning under a torrent of spam for the last couple of weeks. I will reply to each comment personally this evening or tomorrow.</p> </div> Misc 2008-04-07</span> </h3> Yeah .. You need to know what you’re doing, which is completely unlike every other language on the planet. Or, perhaps, you can just use the C example in the same old way and move on with your life.</p> I agree with B.G. – get a grip.</p> </div> Karl 2008-04-07</span> </h3> There are two senses of the phrase "how it works." One is how it works on the inside — in software terms, how it is implemented. The other is how it works from the perspective of someone using it — in software terms, the interface.</p> In all of the languages that you cited, the array feature has an interface with which the programmer must operate. The interface to the C++ std::vector is certainly complicated, and you could argue that it is overcomplicated, but the details you cited as things you have to know about how it works have nothing to do with the implementation of std::vector, they are only parts of the complicated interface of std::vector.</p> So, while you can argue that C++ is complicated, your example only shows that C++ is quantitatively more complicated. If you want to argue that C++ occupies an entirely separate "complication class," you will need a different argument. Such an argument is probably possible.</p> </div> Frank 2008-04-07</span> </h3> I agree that newbee c++ coders can be detrimental in the core of a complex project. Bad programmers are detrimental, regardless of the language.</p> The issue raised in this article is less about cpp and more about garbage collection/reference counting.</p> Of the ‘simple’ examples given, most of them use reference types and garbage collection/reference counting.</p> Pascal used array of int, nothing complex there in cpp either</p> C example uses value type copied by value. Ownership of members is ‘undefined’, a crash waiting to happen.</p> Most of the CPP errors result from passing the object by value. The compiler is warning you that it can’t do it correctly without being told how. Change to something like this for semantics comparable to the other examples.</p> typedef smart_ptr myobj_ptr std::vector arr; arr.push_back( new MyObj );</p> </div> Nick G 2008-04-07</span> </h3> You can simplify the python example as well: arr = [myObj()]</p> </div> raincat 2008-04-07</span> </h3> To C++’s defence here: the program you have written here is semantically very different from the other languages. In C++ you have a vector of MyObject, in all other languages (besides C) you have an array of pointers to objects. The pointer variant in C++ would be:</p> MyObj obj; std::vector arr; arr.push_back( &obj );</p> No default constructors, copy constructors or assignment operators needed. Actually, I don’t know of any other language than C/C++ where you can have arrays with your own objects as value types.</p> However, I wouldn’t argue that you’re wrong. C++ is an arcane and complex language. You can gain a little efficiency and direct control of the computer hardware, but it is rarely worth all the problems.</p> </div> The rant rambler 2008-04-07</span> </h3> The non-C++ examples are a perfect illustration why there shouldn’t be a C++ newbie anymore on this planet. Like, if you know everything about the inner workings of a combustion engine would that make you a better driver? In my 20 years as a C programmer I’ve seen worse C++ programmers (and they weren’t newbies) than C ones.</p> </div> Ryan Walker 2008-04-07</span> </h3> in ruby, one line:</p> arr = []</p> </div> Leonardo 2008-04-07</span> </h3> ArrayList arr = Arrays.asList(new MyObj());</p> And still… strongly typed, compiler checked…</p> </div> Ryan Walker 2008-04-07</span> </h3> in ruby, one line:</p> arr = [] << Object.new</p> </div> Leonardo 2008-04-07</span> </h3> Wooops… to fast with the copy’n’paste. Should be:</p> List arr = Arrays.asList(new MyObj());</p> </div> Andy Balaam 2008-04-07</span> </h3> Tac-Tics</strong>:</p> arr = [MyObj()]</blockquote> I was expecting to use the MyObj later, and/or do some more stuff to it before I added it to the array. I write quite a lot of Python, so obviously at least one people does use this kind of style. I admit it is probably quite Java-y.</p> The Haskell code is more less good.</blockquote> :) The Haskell example was added for fun and is my first ever line of Haskell. Sounds like I made quite a few mistakes. I have corrected it based on another comment. Please let me know if the corrected version is wrong in any way.</p> you should probably be more careful if youâ€™re going to make a point about language inexperience</blockquote> Point taken :).</p> T4C</strong>:</p> itâ€™s a volume of details, not concepts that are required</blockquote> I think there is a large number of very complex concepts too – more than most languages.</p> markus</strong>:</p> But one problem is that, with languages like Ruby and Python becoming more popular, and C still being the oldschool powerhorse - why is the complexity inside C++ needed?</blockquote> Ruby or Python with C for the fast parts is a compelling alternative in some situations, but C++ can really express some things you can’t do in other languages… and isn’t it just more exciting to code in?</p> noah</strong>:</p> Thanks, yes, I was assuming I’d be doing something to the MyObj before putting it into the array. I don’t know why I was assuming that…</p> kL</strong>:</p> array ~= obj;</blockquote> Wow, cool operator!</p> Riaan Minne</strong>:</p> the idea that you have to understand how things work under the hood of a language should be applied to any language</blockquote> My argument is that in C++ it is necessary</em> to know how things work before you even start. Of course, one can become an expert in any programming language, and yes, many of the concepts you acquire are transferrable. By no means all, though.</p> rrwood</strong>:</p> I suspect that there are cases where C++ really makes sense</blockquote> Yes, and I’m not at all saying you shouldn’t use it – I’m mainly saying you should make sure you hire gurus, I think. Actually, I’m not completely sure what I am saying you should do – I’m basically just saying that C++ is for experts.</p> Incidentally, one compelling area for using C++ is embedded. But I think it can be great for anything.</p> Err</strong>:</p> You donâ€™t have to know how the STL is implemented, because the requirements on the type traits of vector elements are specified in the standard.</blockquote> That is technically true. However, the only way I can think of to remember those otherwise arbitrary restrictions is to have some kind of mental model of what is going on underneath. That’s certainly how I remember them (when I do). If you prefer to remember them by memorising the standard, then my argument is simply that there is a large amount of arbitrary information which needs to be learnt before you use STL containers.</p> Anon</strong>:</p> do you mean ArrayList</blockquote> Thank you, yes, I am an idiot. Example corrected.</p> erm….</strong>:</p> if people are not going to hire newbs, how are they going to become !newbs ?</blockquote> Yes, that is a major problem. I started learning C++ during my PhD, which helped because I had no employer as such, but I have become a lot better (certainly not expert…) since I started using it for my job. I am very grateful to my employer for taking me on, but very regretful about a lot of the code I wrote in the past – I wish I had been mentored much more.</p> Ric</strong></p> Like the C example, the PErl can be simplified from this: ...snip... To this: my $obj = MyObj->new(); my @arr = ( $obj );</blockquote> Simplified or shortened? I prefer the longer version to clarify what each step does (especially in an example…).</p> fname lname</strong>:</p> DIM a$ INPUT "Enter your name", a$ PRINT "Hello, ", a$</p> Mike</strong>:</p> Your example cites only references to the Standard Template Library, which is not an integral part of the C++ language itself.</blockquote> Since it’s part of the standard, I would argue that it is.</p> I would agree that the STL is overly complex</blockquote> It remains an open question as to whether it is possible to implement a powerful library in C++ that does not require you to have a good feel for how it works under the covers before you use it. Maybe someone can suggest an example of such a library.</p> Perhaps the compromise would be in terms of performance, which would take away a lot of people’s reasons for using C++ in the first place.</p> BTW, I never said that STL is overly complex, only that it is complex.</p> Edgardo Portal</strong></p> The three things you note about C++ are about the first thing most books mention (along with providing a (possibly virtual) destructor), so Iâ€™d venture to guess that most people venturing to use the STL would have that under their belts by then.</blockquote> I certainly didn’t have this under my belt when I first used STL containers – I just got scared and put pointers into STL containers and had to manage the memory myself. Understanding by-value semantics came later for me. Perhaps it was because I didn’t really read books. (I’m trying to correct that now.)</p> Sgt. Pepper</strong>:</p> Another stupid and pointless C++ flameâ€¦</blockquote> It’s not a flame – I love C++!</p> I hope it’s not pointless – maybe it will help us think about things like:</p> How to teach C++? (e.g. Charles Bailey and I had an interesting chat today about whether you can teach C++ by starting with by-value semantics and only covering pointers later. Hopefully he’ll blog about that, along with the myriad ways he disagrees with me :)</p></li> How to write libraries?</p></li> The need for mentoring of new developers?</p></li> </ul> Max</strong>:</p> To me it sounds more like you are saying C++ is a language for people that have actually used it for longer than a week.</blockquote> I have used it for several years and have loads left to learn. I also know people who are way, way stronger than me and they say they have a lot left to learn as well. You could be an incredibly fast learner.</p> B.G.</strong>:</p> The truth is, most developers writing C++ code are doing so using MFC, or the Borland equivalent (C++ Builder) - especially in Europe.</blockquote> I work as a programmer in Europe and I use STL whenever I can, which is a lot. All the underlying classes we use company-wide are based on STL templates and algorithms, where they are appropriate.</p> STL is excellent! I’ve heard that MFC is horrible, but I’ve never used it.</p> One reason to use the standard facilities is portability. Surely for "Enterprise" software this is important? Our product runs on 4 platforms, only one of which has MFC.</p> Lots of people at last week’s ACCU conference were very interested in STL, so some people are definitely using it.</p> This post reads as old-school, pretentious, bullcrap! Whatâ€™s next - every car should have a Porsche engine? Every medical procedure MUST be carried out in the number 1 US hospital? Every wife should appear in Maximâ€™s top 100 list before the ring is offered, otherwise theyâ€™re too ugly and should be dumped at the alter?</blockquote> I don’t understand your point. Please could you be more explicit?</p> </div> Andy Balaam 2008-04-07</span> </h3> Kyle</strong>:</p> Granted, the compiler will let you CREATE a class without these, but even a C++ newbie, even somebody whoâ€™s had a single class in the language, knows that every class in C++ must have a default constructor, a copy constructor, and an assignment operator.</blockquote> No, every class with value semantics needs these. John Lakos’ talk "Toward a Common Intuition and Reusable Testing Methodology" at the ACCU conference (which inspired my use of this example here) was very interesting on this topic. I don’t know whether he has put his slides online – they might be interesting, but really you’d need to see the video to get the full force of it!</p> Misc</strong>:</p> Yeah .. You need to know what youâ€™re doing, which is completely unlike every other language on the planet.</blockquote> I presume this is meant sarcastically. I think there are degrees of "knowing what you’re doing", and I think C++ requires a higher degree.</p> Karl</strong>:</p> the details you cited as things you have to know about how it works have nothing to do with the implementation of std::vector, they are only parts of the complicated interface of std::vector.</blockquote> Yes, agreed in theory. As I mentioned in another comment, though, in practice, the only way I have found to remember such details is to think (in general terms) about how the artefacts I am using are implemented. Stroustrup encourages this approach, probably partly because it is a book intended to help those who implement the language as well as those who use it.</p> If you want to argue that C++ occupies an entirely separate "complication class," you will need a different argument.</blockquote> I agree that it is perfectly possible to remember requirements like these without thinking about the implementation, but I can’t think of an easier way.</p> I think I’d be happy to say that C++ is more complex than other languages, rather than belonging to a separate class.</p> Frank</strong>:</p> Bad programmers are detrimental, regardless of the language.</blockquote> Agreed. What about inexperienced coders? How damaging are they? How easy is it to spot their errors?</p> The issue raised in this article is less about cpp and more about garbage collection/reference counting.</blockquote> I’d say it was a more general point than that. In general I have found that people documenting C++ libraries talk about the implementation (in general terms) quite often, whereas I have found that that is much more rare when documenting libraries for other languages. Perhaps it is purely a cultural issue.</p> Pascal used array of int, nothing complex there in cpp either</blockquote> My point here is that Pascal is simpler. If you want more complex semantics you have to write them yourself.</p> C example uses value type copied by value. Ownership of members is â€˜undefinedâ€™, a crash waiting to happen.</blockquote> Yes, if your struct contains pointers.</p> Most of the CPP errors result from passing the object by value. The compiler is warning you that it canâ€™t do it correctly without being told how.</blockquote> Yes, the existence (and use in STL) of by-value semantics in C++ is an example of one of the ways it is more complex than less flexible languages.</p> raincat</strong>:</p> To C++â€™s defence here: the program you have written here is semantically very different from the other languages.</blockquote> Actually, I donâ€™t know of any other language than C/C++ where you can have arrays with your own objects as value types.</blockquote> Yes, that is exactly my point – having these difficult to understand facilities available makes C++ complex.</p> However, I wouldnâ€™t argue that youâ€™re wrong. C++ is an arcane and complex language. You can gain a little efficiency and direct control of the computer hardware, but it is rarely worth all the problems.</blockquote> I think I really must have put myself across badly if it sounded like I was criticising C++.</p> I agree that C++ is complex, and sometimes arcane, but it is also an amazing phenomenon, full of power and beauty, and incredibly useful for almost anything.</p> </div> Glen Goldsmith 2008-04-07</span> </h3> Wow. Good info.</p> Are there some good books that discuss this type of thing ? I’ve had a couple of C++ classes at college… and you know I’ve never run into that information before.</p> Thanks for the heads up…. Glen</p> </div> Andy Balaam 2008-04-07</span> </h3> Hi Glen, glad to hear you found it useful. As someone else pointed out, don’t forget about the destructor (although if you don’t have any raw pointer members, the compiler-provided one may well be fine).</p> I use Stroustrup</a>, but I’m not convinced it is the best place to start. Whenever anyone asks me about books I usually just say "always choose an O’Reilly", but in this case I happen to think that Practical C++ Programming</a> is not that good (it’s the book I used to learn, and I didn’t find it inspiring).</p> This one looks quite cool: How Not to Program in C++</a>, but I’ve never read it.</p> If you want to keep going deeper, I’ve found some really interesting stuff on the Dr Dobb’s Journal</a> web site, and I can recommend ACCU</a>’s publications Overload and CVu.</p> Scott Meyers</a>’ books "Effective C++" and "More Effective C++" are excellent, but not aimed at beginners. They cover exactly the type of thing I described above in this blog entry.</p> Perhaps others have suggestions for good books they used to learn C++?</p> </div> Justin 2008-04-07</span> </h3> MFC does suck. And sucks hard (at least back in 2001 when I had to use it).</p> As far as understanding the under-workings of the STL, I think, instead, that a basic understanding of how different data structures work and where they are appropriate is sufficient, especially since, as previously pointed out, that the STL is implemented differently all over the place.</p> Also, PHP:</p> $arr = array( new MyObj() );</p> and JavaScript:</p> var MyObj = {}, arr = [ MyObj ];</p> </div> jlnr 2008-04-07</span> </h3> Article- I agree that many details make this complicated, but it’s not that</em> fundamentally bad. For most of the other languages, you inevitably have to understand the difference between references and values; Ruby tries to turn everything into a reference, but if you want to be clever and create a list of ten empty strings via [""] * 10, the fundamental problem comes up again as soon as you try to modify one of them. In Java you also have to understand the (annoying) difference between primitives and objects. But once you understand to divide your classes into value types and noncopyable reference types, and use tr1 pointers to handle the reference types, it gets so much easier and the whole design suddenly makes a lot of sense, though technically, even more things are happening under the surface.</p> I don’t understand why everything surrounding copying is brought up so often, while I haven’t read much criticism for completely failed concepts such as argument dependent lookup, which is PHP-class bad in that it makes a seemingly harmless feature (calling a function) overly complex. Templates being complex is just something you should expect ;)</p> Kyle- Totally disagree. Define copying/assignment at the lowest level, i.e. smart pointers, containers etc., then just do nothing in the upper levels. Things owning an object that can’t be copied have a scoped_ptr member so aren’t copyable (rightly so!), structs that only have some strings and ints can be copied. Problem solved ;)</p> </div> Glen Goldsmith 2008-04-08</span> </h3> Hey Andy…</p> THanks so much.</p> The Scott Meyers’ books are probably what I’m looking for. After taking a few C++ classes at school, I’ve been through several books — and very important information such as the above, aren’t covered in beginning books.</p> Most of these beginning books cover syntax, a little OOP theory, a dash of UML 2 and after 800 pages and 15 chapters… congrats! You can put C++ on your resume now!</p> So this is very helpful stuff.</p> The web of course is an extremely helpful tool —– but to answer specific questions without knowing why, it’s great. On example: What’s the fastest way to copy very large amounts of small files. The web has that answer, but usually never "why" the code is that answer. They don’t go into the depth and background you need to know… to get some experience so you can take that knowledge forth and prosper.</p> Keep up the great blog.</p> </div> ben 2008-04-08</span> </h3> In Ruby:</p> arr = [MyObj.new]</p> </div> kg 2008-04-08</span> </h3> At college I was taught how to program using C++, but I didn’t really learn</em> C++ till I read C++ Primer (4th ed) by Stan Lippman, cover-to-cover. I think it’s probably the best book for learning the fundamentals and how to use C++ "the right way". Admittedly, it’s a very long read, but it’s a good place to start if you want to grok C++ instead of just getting by.</p> Stoustrup’s book seems more focused on the language itself and less focused on how to use it. But, if you want to know why C++ is the way it is (although probably most people don’t), check out his other book The Design and Evolution of C++.</p> Nowadays, I have to consult the C++ standard itself once in a while (the only way to get a definitive answer on some very specific questions). ANSI sells it for about $30 (which is still highway robbery for a language standard). Don’t be afraid to get yourself a copy and have it as a reference.</p> I haven’t read the Scott Meyers books, but they are probably very good, not so much for learning the basics, but for how to solve the kinds of problems that come up a lot when using C++ in the real world. For this type of thing, also check out Herb Sutter’s books and his Guru of the Week articles.</p> In any case, you are definitely right, C++ is geared more toward experts. Hopefully C++0x will change that somewhat. However, I’m of the opinion that becoming a C++ expert is worth the effort. (Can you be an expert user of a language without being an expert in the language itself?)</p> Warning: Use of RAII may cause your debugging skills to falter, because you won’t need to use the debugger as often.</p> </div> Enlaces del 28/04/2008 | El ChigÃ¼ire Literario 2008-04-28</span> </h3> […] quÃ© se usa C++ en los videojuegos? y C++ es un lenguaje para expertos. Enlaces robados a […]</p> </div> GCC's Unhelpful Error Messages | Cardinal Peak 2019-04-10</span> </h3> […] Balaam has a humorous blog post whereâ€”almost as an asideâ€”he generates a 1950-byte error message from a source program of only […]</p> </div> GCC’s Unhelpful Error Messages – Cardinal Peak 2019-12-27</span> </h3> […] Balaam has a humorous blog post whereâ€”almost as an asideâ€”he generates a 1950-byte error message from a source program of only […]</p> </div> Templated test code? Andy Balaam — Wed, 19 Mar 2008 10:51:39 +0000 At work at the moment, as part of an initiative to get with the 21st century, we are waking up to testing our code.</p> Thus, I am writing a lot of unit tests for old code, which can be soul-destroyingly repetitive and very pointless-feeling (even though really I do see a great value in the end result - tested code is refactorable code).</p> Often, tests have a lot in common with each other, so it feels right to reduce code repetition, and factor things into functions etc. The Right Way of doing this is to leave your tests as straightforward as possible, with preferably no code branches at all, just declarative statements.</p> Contemplating writing unit tests for the same method on 20+ very similar classes, using a template function "feels" right, for normal code values of "feel". However, for test code, maybe it's wrong?</p> My question is: is it ok to write a test function like this?:</p> void test_all_thingies() { test_One_Thingy<Thingy1>(); test_One_Thingy<Thingy2>(); test_One_Thingy<Thingy3>(); test_One_Thingy<Thingy4>(); } template< class T > void test_One_Thingy() { T thingy; thingy.doSomething(); TEST_ASSERT( thingy.isSomething() ); } </pre> Worse still, is this ok?</p> void test_all_thingies() { test_One_Thingy<Thingy1>( "Thingy1 expected output" ); test_One_Thingy<Thingy2>( "Thingy2 expected output" ); test_One_Thingy<Thingy3>( "Thingy3 expected output" ); test_One_Thingy<Thingy4>( "Thingy4 expected output" ); } template< class T > void test_One_Thingy( std::string expected_output ) { T thingy; thingy.doSomething(); TEST_ASSERT( thingy.getOutput() == expected_output ); } </pre> Reasons for: otherwise I'm going to be writing huge amounts of copy-pasted code (unless someone can suggest a better way?).</p> Reasons against: how clear is it going to be which</em> class failed the test when it fails?</p> Update: fixed unescaped diagonal brackets.</strong></p> Comments</h2> Charles Bailey 2008-03-20</span> </h3> I presume a clerical error has left out the template parameter specification on the test_One_Thingy function calls.</p> ‘Reasons against’ can be rebutted fairly fast. The thing with templates is that they are ‘real code’, so you can’t use any preprocessor code in the template function to distinguish different instantiations.</p> Assuming that TEST_ASSERT is a macro that makes use of FILE</strong> and LINE</strong> for its helpful diagnostic, the obvious way is to make a more configurable macro which does essentially the same thing, but allows you to specify the file and line. For example, make TEST_ASSERT_FL macro so that these two lines are equivalent. TEST_ASSERT_FL( x, FILE</strong>, LINE</strong> ) TEST_ASSERT( x )</p> Then, change the signature of test_One_Thingy as follows. template void test_One_Thingy( const std::string& ex, const char* file, unsigned int line )</p> And have it use the new macro. TEST_ASSERT_FL( thingy.getOutput() == expected_output, file, line );</p> Then create a new macro. #defined TEST_ONE_THINGY( cls, expect ) test_One_Thingy( expect, FILE</strong>, LINE</strong> )</p> void test_all_thingies() { TEST_ONE_THINGY( ClassA, "Thingy1 expected output" ); TEST_ONE_THINGY( ClassB, "Thingy2 expected output" ); TEST_ONE_THINGY( ClassC, "Thingy3 expected output" ); TEST_ONE_THINGY( ClassD, "Thingy4 expected output" ); }</p> File and line now give you which class caused the failure, at the expense of not telling you the file and line of the template code where the actual test is. (This may not matter much if the template code only has a single assert macro.)</p> Now the funky way :) .</p> Assuming you have a TestAssert underlying function on which the TEST_ASSERT macro is built…</p> template inline std::string SuffixType( const std::string& in ) { std::ostringstream s; s ( #x ).c_str(), FILE</strong>, LINE</strong>)</p> template void test_One_Thingy( const std::string& expected_output ) { T thingy; thingy.doSomething(); TEST_ASSERT_TEMPL( T, thingy.getOutput() == expected_output ); }</p> Warning! The type suffix is not guaranteed to be human readable.</p> </div> Charles Bailey 2008-03-20</span> </h3> Grrrr, WordPress has completely garabled all of the angle brackets from my comments.</p> </div> Andy Balaam 2008-03-26</span> </h3> I think I get the idea, and it certainly helps us tell which test fails, but it still feels a little complicated for test code, which I should have added as a further reason against.</p> </div> Andy Balaam 2008-03-26</span> </h3> Charles emailed the ungarbled version to me:</p> </p> #include "hshgtest.h" #include #include </p> struct ClsA { const char* str() const { return "My name is A"; } };</p> struct ClsB { const char* str() const { return "My name is B"; } };</p> struct ClsC { const char* str() const { return "My Name is C"; } };</p> struct ClsD { const char* str() const { return "My name is D"; } };</p> namespace {</p> template< class T > inline std::string SuffixType( const std::string& in ) { std::ostringstream s; s << in << " : " << typeid( T ).name(); return s.str(); }</p> #define TEST_ASSERT_TEMPL( t, x ) HSHGTest::TestAssert( x, SuffixType< t >( #x ).c_str(), FILE</strong>, LINE</strong>)</p> template< class T > void ClassTest( const std::string& expect ) { T test; TEST_ASSERT_TEMPL( T, expect == test.str() ); }</p> void testall() { ClassTest< ClsA >( "My name is A" ); ClassTest< ClsB >( "My name is B" ); ClassTest< ClsC >( "My name is C" ); ClassTest< ClsD >( "My name is D" ); }</p> }</p> HSHG_BEGIN_TESTS HSHG_TEST_ENTRY( testall ) HSHG_END_TESTS</p> HSHG_TEST_MAIN </p></pre> </p> </div> Andy Balaam 2008-03-26</span> </h3> What about just plain old this?:</p> </p> </p> #define TEST_ONE_THINGY(CLS,OUTPUT) test_One_Thingy( #CLS, OUTPUT );</p> void test_all_thingies() { TEST_ONE_THINGY( Thingy1, "Thingy1 expected output" ); TEST_ONE_THINGY( Thingy2, "Thingy2 expected output" ); TEST_ONE_THINGY( Thingy3, "Thingy3 expected output" ); TEST_ONE_THINGY( Thingy4, "Thingy4 expected output" ); }</p> template< class T > void test_One_Thingy( std::string class_name, std::string expected_output ) { T thingy; thingy.doSomething(); TEST_ASSERT_CLS( class_name, thingy.getOutput() == expected_output ); }</p> </p></pre> </p> </div> Andy Balaam 2008-03-26</span> </h3> Where TEST_ASSERT_CLS does the obvious thing.</p> </div> Andy Balaam 2008-03-26</span> </h3> Stringificizationalism seems the most reliable way to get the class name here.</p> </div> Public bzr branch of FreeGuide Andy Balaam — Wed, 12 Mar 2008 09:56:19 +0000 On the subject of distributed source code management, Dan Watkins has just informed me that the launchpad team have created a bazaar branch of FreeGuide's code, so if you're into that kind of thing, you can download the code from that instead of our central subversion repo.</p> The link is here: http://code.edge.launchpad.net/~vcs-imports/freeguide-tv/trunk</a>.</p> If you're into git, I'd suggest git-svn. It's what I use for FreeGuide development now. Let me know if you have trouble getting it working.</p> Public git repo for GSSMP Andy Balaam — Wed, 12 Mar 2008 09:30:39 +0000 Git is supposed to work when you upload your repository to an http server, but in reality, no-one except me seems to use it, and it's nothing like as useful as having a proper repository that people can commit to etc.</p> So, here it is: http://repo.or.cz/w/gssmp.git</a>.</p> Enjoy.</p> Finally released last.fm support for GSSMP Andy Balaam — Sun, 20 Jan 2008 09:34:37 +0000 There was a weird bug in my last.fm support for GSSMP</a>, where it would crash when run from the GNOME panel, but be fine when run from a terminal.</p> After two bugs fixes from Charles Bailey</a>, the problem seems to have gone away, so I've finally been able to make a release.</p> Now I have to resist the temptation to choose songs to make last.fm think I'm cool, instead of songs I actually want to listen to.</p> And I really need to fix the debian/Ubuntu package of lastfmsubmitd so that the client exe lastfmsubmit is included in the package. Without that, using last.fm with GSSMP is a major hassle.</p> lighttpd on Ubuntu - alias.url doesn't work when included in a module config file Andy Balaam — Sun, 09 Dec 2007 01:00:06 +0000 I've decided to use lighttpd</a> on my new machine (which was generously given to me and features a 700GB raid array).</p> My main reason for choosing lighttpd was the FLV streaming plugin</a> for it that will make my Wii TV viewing experience a lot better. (It works - I've tried it!)</p> However, if I'm going to use lighttpd for streaming TV to my Wii, I want to use it for everything, rather than having Apache running as well. In any case, I think given my load pattern (average 0.01 users at any time, approx) a lower-memory (but maybe less stable under heavy load) solution seems a good choice.</p> So I have to get my webmail, home-grown address book</a>, and mediawiki installations all working with lighttpd. How hard can it be?</p> Actually, harder than it should be.</p> Lighttpd is supposed to listen to a directive that looks like this:</p> alias.url += ( "/mediawiki/" => "/usr/share/mediawiki1.10/" )</pre> and on Ubuntu, being Debian-derived, you can split your lighttpd config into separate files, allowing apt to modify the main config file during an upgrade without blatting your customisations.</p> So I created a file in /etc/lighttpd/conf-available/ called 50-mediawiki.conf that contained the alias-url line above, and then I ran:</p> sudo lighty-enable-mod mediawiki</pre> and a symbolic link was duly created at /etc/lighttpd/conf-enabled/50-mediawiki.conf.</p> I restarted lighttpd like so:</p> sudo /etc/init.d/lighttpd force-reload</pre> and navigated to localhost/mediawiki, but got a 404 error.</p> A lot of experimentation later, and it turns out to be a stupid thing.</p> Lighttpd has a stupid "by design" behaviour, which I think I can summarise as "if you encounter a change to alias.url that comes AFTER a conditional that modifies alias.url, ignore it."</p> So, to explain by example, this code in /etc/lighttpd/lighttpd.conf will successfully redirect /test/ to /home/andy/test/:</p> alias.url += ( "/test/" => "/home/andy/test/" ) $HTTP["remoteip"] == "127.0.0.1" { alias.url += ( "/doc/" => "/usr/share/doc/", "/images/" => "/usr/share/images/" ) $HTTP["url"] =~ "^/doc/|^/images/" { dir-listing.activate = "enable" } } </pre> but this code:</p> $HTTP["remoteip"] == "127.0.0.1" { alias.url += ( "/doc/" => "/usr/share/doc/", "/images/" => "/usr/share/images/" ) $HTTP["url"] =~ "^/doc/|^/images/" { dir-listing.activate = "enable" } } alias.url += ( "/test/" => "/home/andy/test/" ) </pre> Will leave you scratching your head as to why it doesn't work.</p> Since the bit of lighttpd.conf that is put there by Debian (or Ubuntu?) to include all files in conf-enabled/ comes at the very end of the file, any alias.url lines in any module config are ignored. Clever eh?</p> To skip to the end, my fix was to move the Debian documentation code to the very end of the config file, so the end of lighttpd.conf now looks like this:</p> ... #### external configuration files ## mimetype mapping include_shell "/usr/share/lighttpd/create-mime.assign.pl" ## load enabled configuration files, ## read /etc/lighttpd/conf-available/README first include_shell "/usr/share/lighttpd/include-conf-enabled.pl" #### handle Debian Policy Manual, Section 11.5. urls #### and by default allow them only from localhost $HTTP["remoteip"] == "127.0.0.1" { alias.url += ( "/doc/" => "/usr/share/doc/", "/images/" => "/usr/share/images/" ) $HTTP["url"] =~ "^/doc/|^/images/" { dir-listing.activate = "enable" } }</pre> Now my separate file for mediawiki works fine, and my 404 error has changed to a much more healthy 403. (If that causes pain, expect a blog entry on that too.)</p> As for why lighttpd works in such a stupid way, apparently it's by design: lighttpd bug 1427</a></p> And here's the relevant debian bug, to which I will try to remember to reply: Debian bug 445459</a></p> Incidentally, running this might be helpful:</p> lighttpd -tp -f /etc/lighttpd/lighttpd.conf | less</pre> It shows you how lighttpd has understood your configuration files.</p> Lighttpd seems to be light on end-user documentation, and concern for ease of use. I hope I'm not making a big mistake...</p> Comments</h2> Charles Bailey 2007-12-10</span> </h3> lighttpd can’t be too bad, after all it is the default git instaweb solution. It seems to work for that, although I had difficulty with the current git. It seems that instaweb isn’t quit insta as I had to poke a config file.</p> </div> Steve Greenland 2007-12-13</span> </h3> I’ve been mostly happy with lighttpd. The conditional configuration is nifty, the re-write syntax is, I think, easier to deal with than apache’s, and I have no regrets in converting from apache to lighttpd. Yes, the documentation could be better, but the mailling list is pretty responsive, and getting people to write docs is always hard.</p> But this "design decision" is really weird.</p> </div> Andy Balaam 2007-12-15</span> </h3> It’s good to know that the mailing list is useful – thanks for the input. I like lighttpd’s config file syntax so far. Although apache’s style has become familiar, it is pretty obscure.</p> No arguments about the weird design decision. I assume it is something that feels natural to the developers because of the way they’ve implemented the config file parsing.</p> </div> Duncan Idaho’s Weblog » Aliasy, lighttpd i Debian Lenny 2008-04-20</span> </h3> […] Jak zwykle jestem otwarty na uwagi i propozycje dotyczÄ…ce moich wpisÃ³w. Jako ÅºrÃ³dÅ‚o podam tu Andy Balaam’s Blog, na ktÃ³rym znalazÅ‚em rozwiÄ…zanie tego problemu. kwiecieÅ„ 20th 2008 Posted to Debian, […]</p> </div> FreeGuide updated to latest XMLTV, and bug fixes Andy Balaam — Fri, 16 Nov 2007 04:03:05 +0000 Check out a release candidate here: http://freeguide-tv.sourceforge.net/rc/</a> .</p> So far I've only uploaded an RPM, but hopefully there'll be a Windows exe and some other packages soon.</p> Update: Windows installer now uploaded too.</strong></p> Test it! It's got better capturing and reporting of download errors, more sensible "download in background" and "show output" behaviour, and its list of XMLTV grabbers is synched with the latest XMLTV (0.5.50).</p> However, apparently XMLTV 0.5.50 is causing problems in FreeGuide for at least some people, so it may not be a flawless experience...</p> I really must make a one-button build for Ubuntu .debs, like I have for RPMs. I have been using Ubuntu for some time now...</p> I'd forgotten to update the version page on the FreeGuide web site for 0.10.5, so users weren't notified of that release. I'll do it for 0.10.6, and hopefully get some stats together to find out how many people are using it. Now that the stats are working, we may be starting to get some better information.</p> Yes, I couldn't sleep tonight.</p> No, I won't be very clever tomorrow.</p> But hey, I was clever tonight - dipping into some of the spaghetti that makes FreeGuide tick these days and not only making a change that works, but also making things a tiny bit better</a> - _that's_ an achievement.</p> Comments</h2> Andy Balaam 2007-11-16</span> </h3> Of course, calling myself clever was a big mistake: http://freeguide-tv.svn.sourceforge.net/viewv...</a></p> </div> duckmaze 0.2 Andy Balaam — Sat, 10 Nov 2007 00:23:53 +0000 I've finally released to the world the secret extra duckmaze</a> levels. They're contained in the 0.2 release, which is also faster, and doesn't freeze time.</p> (As far as I know.)</p> It's got 18 levels, and a level editor, which means you can make more.</p> Do it.</li> Send them to me.</li> ???</li> Profit.</li> </ul> Caution: numbers that feel prime may be divisible by 3 Andy Balaam — Thu, 08 Nov 2007 09:08:55 +0000 Caution: numbers that feel prime may be divisible by 3. For example, 51 and 57.</p> Conversely, numbers that feel divisible by 3 may be prime. For example, 61, 89 and 97.</p> Nominations for least prime-feeling numbers that actually are prime: 89 and 101.</p> Nominations for best prime number: 2 and 101.</p> Both have digits that sum to 2.</p> Yes, I was trying to count sheep last night.</p> Comments</h2> Midge 2007-11-30</span> </h3> A Prime, primes and half a prime</em></p> I was thinking this morning about 34 (nearly my birthday) and how it feels like it should be prime! It is midway between the two nearest primes tho, and two times a prime.</p> </div> Andy Balaam 2007-12-03</span> </h3> Two times a prime does often feel special.</p> </div> Announcing Record TV Andy Balaam — Mon, 24 Sep 2007 09:08:42 +0000 Last night I uploaded the first public version of my latest project, Record TV</a>. Record TV is a system for recording TV (on a Linux desktop computer) that is designed to allow lots of different user interfaces all to use the same back end. It is currently only useful for people who are quite familiar with the Linux command line. It essentially has no user interface at all, but the back end stuff works for recording TV.</p> Perhaps more excitingly, I have also managed to get my recorded programmes to play back on my Nintendo Wii, so I can watch them on my TV.</p> Find out more on the project page linked above. I've released this code very early, in the spirit of "release early, release often," so expect to hack on it a bit to get it working.</p> If you think MythTV just goes about things the wrong way, and you'd like to help do it right, it might be of interest.</p> It's mostly Python, with some PHP and shell scripts.</p> A job I'd like to do is to be able to use FreeGuide</a> as a UI for selecting programmes to record.</p> Comments</h2> Karel 2007-11-16</span> </h3> I am really interested to use your program as "freevo recordserver plugin ". In fact, I use an NSLU2 and freevo recordserver is too heavy. If you have any idea on how to do this, could you please send me and email. It will be great. I’am going to investigate and if I found something , I ll send you an email</p> Thanks for your program</p> </div> Andy Balaam 2007-11-16</span> </h3> Hi Karel, I’m afraid I have no idea but let me know how you get on!</p> </div> FreeGuide 0.10.5 Andy Balaam — Wed, 19 Sep 2007 14:17:35 +0000 After over a year, and several U-turns, the next version of FreeGuide is out.</p> I was dead-set on getting recording functionality into it, and when some students turned up asking to help, I directed them towards doing that. They produced a reasonable framework for how to do it (without any of the dirty details of actually recording stuff), but we ended up throwing it away in favour of the way Alex wanted to do it, which was as an extension of the current reminder code.</p> This way of doing things never reached a stable state, so the trunk was festering for a long time with broken code in it that lost your settings, and tried to record programmes but started too late, and with loads of other usability bugs that I'd hesitate to call minor since any one of them could put off someone trying the program for the first time and persuade them never to come back.</p> In the end, since this just wasn't getting fixed, I decided we needed to give up on recording and get back to what we're good at, which is being a TV guide. There is, of course, quite a lot of overlap in these two functions, but I just didn't have time to think through properly what a recording program should do, and how to move FreeGuide over to a sensible model.</p> So, I reverted the recording code, and it's there in the SVN history if anyone wants to resurrect it.</p> Meanwhile, the thing that motivated me to do this and actually get a release out was the fact that the US listings provider Zap2It has wound up their service, and it's been replaced with a new one, called Schedules Direct</a>, for which you have to pay money.</p> Hopefully, any US and Canadian users of FreeGuide should be able to download the latest version of FreeGuide (and update to the latest XMLTV if they're on Linux) and start viewing their listings as soon as they've paid for them from Schedules DIrect. Do let us know on the mailing list if this isn't the case.</p> So, hopefully the crisis has been averted, and now we can get back to doing things right. I've implemented the Benevolent Fascist Dictator</a> rule, and so far, trunk is better than 0.10.5, and we could make a release from it any time. I plan to make one every couple of months, to keep things ticking along.</p> FreeGuide 0.10.5 contains millions of small features and bug fixes, that I didn't manage to keep up with while trunk was broken. The next release will be after a much shorter delay, and will have a much better change log.</p> FreeGuide SVN now useable Andy Balaam — Sat, 18 Aug 2007 10:14:16 +0000 Why not try the latest SVN version of FreeGuide, which contains lots of bug fixes over 0.10.4? There are detailed instructions on how to Build FreeGuide from SVN</a>, and do ask on the developers' list if you have problems.</p> I've fixed the horrible bug that deleted all your preferences, and I think the code's in pretty good shape, with all the recording code gone, but lots of nice additions put in over the last year.</p> Try it out, and let us know on the developers' list how you get on - good or bad.</p> Heartfelt apologies for the feed URL change Andy Balaam — Wed, 15 Aug 2007 12:20:43 +0000 Sorry, sorry, sorry, sorry. I decided to change to wordpress' pretty-URLs mode, and it changed the feed URL, which I was not expecting, and which I think is fundamentally borken.</p> FreeGuide - putting recording on hold Andy Balaam — Wed, 15 Aug 2007 11:00:26 +0000 FreeGuide has been stalled for about a year because the basic code for recording programmes has been sitting in SVN, but it contained far too many bugs to be useable, even by the developers. This kind of situation completely takes the wind out of development, since any little bug fixes or features people want to send it are either against very old code, or never get written because when they download the latest code it doesn't work.</p> I am more and more of the opinion that the code in your trunk (or HEAD, in CVS terminology) should ALWAYS by _better_ than the latest release. I.e. it should only contain bug fixes and properly tested new features. Anything else (i.e. code that is still being knocked into shape) should live in a branch. Branches are pain, but not as much pain as your project dying because new developers who try and do the right thing (download the latest code) are confronted by non-working code and move on to something more fun.</p> So, I've reverted the recording code. If we want to resurrect it, we can branch from just before where I reverted, and then merge from the trunk into that branch any other fixes that apply.</p> The next release of FreeGuide will be from the trunk, and will contain bug fixes and small new features submitted by a couple of people who have stepped up to do the work. I hope, when my life settles slightly, that I might even be one of them.</p> New baby resolution: never allow non-completely-working code in trunk. Without new developers, your project will die.</p> Small hypocritical extra note: I completely broke the code in trunk by reverting the recording code, so now it loses all your settings when you exit. Use with extreme caution. This, of course, completely breaks the above resolution. I will fix it when I can, and in the meantime, Rick is looking into it, so hopefully it will be fixed soon.</p> New Baby arrived safely Andy Balaam — Wed, 15 Aug 2007 10:38:23 +0000 I won't waste too much of your time, but here she is:</p> </p> She's excellent, and mum is doing really well. Big brother appears to be a little bothered, but presumably that will pass.</p> (Badly wrong) usage statistics for FreeGuide Andy Balaam — Tue, 31 Jul 2007 13:53:12 +0000 A long time ago I added functionality to FreeGuide so people could opt to report their use of FreeGuide to under a username, or anonymously. This all happens as part of the latest version check, and you can turn it off, obviously.</p> Quite a long time ago, Alex broke this functionality, and I haven't yet made a release that fixes it. Stupid, really, because the usage statistics we've got are all-but useless:</p> </a></p> </a></p> </a></p> Hopefully I'll get a release out soon that fixes it, and these graphs will become more sensible.</p> Comments</h2> Andy Balaam’s Blog » Blog Archive » FreeGuide usage stats 2011-10-16</span> </h3> […] long time ago I mentioned I’d added some (opt-in) usage tracking for FreeGuide. Here are the latest stats. I think they […]</p> </div> Procmail Maildir messages disappearing Andy Balaam — Tue, 26 Jun 2007 08:08:44 +0000 Google-friendly title, no?</p> Executive summary:</p> In your ~/.procmailrc</p> MAILDIR=$HOME/Maildir - is BAD, whereas</p> MAILDIR=$HOME/Maildir/ - is good.</p> The slash at the end appears to tell procmail that this is a Maildir directory, rather than something else, elusive.</p> For a week my messages were appearing in $HOME/Maildir, but they were just in there with names like msg.zFS, instead of going into $HOME/Maildir/new and having a name like 1282544959.Q22432E173.192.0.54.2:1,T.</p> This meant they effectively disappeared, and I had a quiet, relaxing week.</p> Followed by a panicked and expletive-filled hour or so.</p> When I put the slash back, new messages started appearing in my Inbox again.</p> I was also able just to move the msg.zFS files into "new", and they appeared fine, which was nice.</p> Comments</h2> Andy Balaam 2007-06-27</span> </h3> I feel much more comfortable now reality has been restored.</p> </div> Charles Bailey 2007-06-26</span> </h3> I think you should have: MAILDIR=$HOME/Maildir and DEFAULT=${MAILDIR}/</p> This specifies where your Maildir "tree" is, and that by default you want to deliver into that directory as a top level Maildir format folder.</p> </div> Andy Balaam 2007-06-26</span> </h3> I’m not going to touch it ever again. It made me too frightened.</p> </div> Charles Bailey 2007-06-26</span> </h3> OK, I admit it, perhaps I’m wrong. I’ve just checked my /etc/procmail. MAILDIR=$HOME/Maildir/ DEFAULT=$MAILDIR</p> </div> Andy Balaam 2007-06-26</span> </h3> Today truly is a special day.</p> </div> Charles Bailey 2007-06-27</span> </h3> Right, in order not to be wrong I’ve changed it to this:</p> MAILDIR=$HOME/Maildir DEFAULT=$MAILDIR/</p> and am so far suffering from no adverse affects.</p> </div> last.fm in GSSMP Andy Balaam — Thu, 07 Jun 2007 02:38:47 +0000 Obviously I should have gone to bed, but I've got last.fm</a> support working in GSSMP</a>. It's in the git repo, and I should get a release out as soon as I've written a little bit of documentation.</p> lastfmsubmitd on Ubuntu Dapper Andy Balaam — Thu, 07 Jun 2007 00:31:00 +0000 So I had been thinking about putting last.fm</a> support into GSSMP</a>, and then I got an email to the mailing list suggesting it, and then I couldn't sleep tonight, and, as ever, something that sounded easy has turned into a saga.</p> I decided to use lastfmsubmitd</a> to handle communicating with last.fm, since it seems silly to re-implement this in every app, and, after worrying that the web site didn't mention any binary packages, I was glad to discover that it has indeed been packaged for Ubuntu Edgy, Feisty and Gutsy.</p> Of course, since I use an distro version that is *gasp* almost ONE YEAR OLD (Ubuntu Dapper) there were no packages for my version.</p> So I downloaded the Edgy version and tried to install it, but it complained that my version of python-support was too old.</p> After quite a bit of faffing I extracted the .deb file using ar (1), and then untarred the file inside there called control.tar.gz and edited the file inside there called control so that it said python-support (>= 0.1) instead of 0.2. Simple, eh?</p> [Of course, all of this assumes that the packager had no good reason for asking for version 0.2. So far, it looks like that assumption was correct.]</p> After some careful re-tarring and re-arring (if that's a word) I had a new .deb, which you can download if you're interested, from here: lastfmsubmitd_0.32.1-1-friggedbyandy_all.deb</a> which appears to install cleanly (so long as I create a config file /etc/lastfmsubmitd.conf BEFORE I install...).</p> That config file, by the way, just looks like this:</p> [account] password = mypassword user = myusername </pre> and is owned by the lastfm user, in the root group, and is not readable by "all". Note that you can't make it owned by the lastfm user until after you've installed the deb, because that user won't exist until then.</p> Are you following so far?</p> Of course, for some unknown reason that deb only contains the daemon lastfmsubmitd, and not the helper program lastfmsubmit, which allows you to actually submit tracks on the command line, despite the fact that the deb installs the man page for that command.</p> So, I downloaded the relevant original source package</a>, and just extracted the file lastfmsubmit and put it in my PATH.</p> After executing a simple command like this:</p> sudo chmod a+w /var/spool/lastfm </pre> (obviously) it seems to actually work when I run from the command line like so:</p> lastfmsubmit --artist "Propellerheads" --album "Decksandrumsandrockandroll" --title "A Number of Microphones" --length "45"</p> Now on to doing what I actually wanted to do...</p> Update:</strong> note also the bug in these packages that prevents cron.daily from running: Ubuntu 136717</a></p> Obviously, there was a bug Andy Balaam — Fri, 01 Jun 2007 19:29:32 +0000 So I've released GSSMP</a> 1.1, which is guaranteed to contain one fewer bugs, and is relatively unlikely to contain any new ones. Also, thanks to JoÃ£o Pinto, it should build without warnings on AMD64 too!</p> If anyone wants to make an AMD64 package, I'd be really happy to put that up on the download page. It should be a simple matter of unzipping the source, cd-ing into the gssmp directory, and running make pkg-all.</p> Go on, it'll be fun...</p> Comments</h2> Charles Bailey 2007-06-04</span> </h3> How about just installing a cross-compiler on one of your outdated systems?</p> Oh wait, you’ve already heard how enjoyable it is.</p> </div> Andy Balaam 2007-06-04</span> </h3> I have occasional fantasies about installing about 10 virtual machines – one for each Debian, Ubuntu, Fedora, SuSe etc. version, and writing a scripting interface to automatically build native packages for each system inside the virtual machines.</p> Is it normal to have fantasies about such things?</p> </div> Charles Bailey 2007-06-04</span> </h3> No. I could tell you about normal fantasies, then you could exercise your post editing powers.</p> What you really want is a cross packager. Packages are just streams of bytes (on all platforms), so given some source code, and some sort of package specification file you should be able to write a C program to produce a set of packages.</p> </div> Watching TV: a simpler way Andy Balaam — Wed, 16 May 2007 19:00:06 +0000 MPlayer is a black art. GMPlayer, doubly so.</p> It turns out this does the job much better than my previous hacks:</p> #!/bin/bash gmplayer -cache 2048 -zoom "dvb://$1"</pre> It's still not perfect for radio over digital TV, though - you get an empty window where the picture should be, and the cache is far too large for radio.</p> GSSMP 1.0 Andy Balaam — Sat, 12 May 2007 06:32:37 +0000 Get it here: http://gssmp.sourceforge.net</a>.</p> This is the first time I've declared one of my projects stable, and given it a 1.0 version number.</p> Prove me wrong by downloading it and breaking it.</p> Enjoy.</p> Now I must dash - I've just got to finish off my other projects.</p> Choose channel dialog Andy Balaam — Thu, 10 May 2007 03:57:06 +0000 Following on from the last post about how I got my TV card working, here is a little script I wrote that allows me to choose a channel and start watching. It uses zenity for a rudimentary GUI:</p> $ cat bin/tv_choose_channel #!/bin/bash awk -F ':' '{print $1}' < .mplayer/channels.conf | \ zenity --list --title "Choose Channel" \ --text "Choose a TV channel to watch..." \ --column "Channels" | xargs -IREP tv_watch "REP"</pre> Digital TV on Dapper with my Hauppauge WinTV Nova-T card Andy Balaam — Thu, 10 May 2007 03:44:18 +0000 My card reports itself like this:</p> $ lspci *snip* 0000:00:0a.0 Multimedia controller: Philips Semiconductors SAA7146 (rev 01) *snip</pre> Everything seemed to work out of the box:</p> $ dmesg | grep DVB -A 0 -B 3 [17179585.872000] saa7146: register extension 'budget dvb'. [17179585.872000] ACPI: PCI Interrupt 0000:00:0a.0[A] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11 [17179585.872000] saa7146: found saa7146 @ mem e0aa2000 (revision 1, irq 11) (0x13c2,0x1005). [17179585.872000] DVB: registering new adapter (TT-Budget/WinTV-NOVA-T PCI). [17179585.908000] adapter has MAC addr = 00:d0:5c:20:33:92 [17179585.908000] DVB: registering frontend 0 (LSI L64781 DVB-T)...</pre> So, many, many hours of Googling later, and after many blind alleys and red herrings, I remembered (yes, I have indeed been through all of this before...) what I needed to do next:</p> scan /usr/share/doc/dvb-utils/examples/scan/dvb-t/uk-CrystalPalace > \ ~/.mplayer/channels.conf</pre> and, finally, to watch TV I just do this:</p> mplayer -zoom "dvb://BBC ONE"</pre> That seems to fail sometimes, and this is more reliable:</p> $ cat bin/tv_watch #!/bin/bash TMPFILE=~/Desktop/tv.mpg mplayer -dumpstream -dumpfile "$TMPFILE" "dvb://$1" & sleep 5 mplayer -zoom "$TMPFILE" kill %1 rm -f "$TMPFILE"</pre> which I run by simply typing:</p> tv_watch "BBC FOUR"</pre> Update: the "scan" command is provided by the "dvb-utils" package on Ubuntu.</strong></p> Comments</h2> Charles Bailey 2007-05-14</span> </h3> What about: #!/bin/bash TMPFILE=~/Desktop/tvfifo mkfifo $TMPFILE mplayer -dumpstream -dumpfile "$TMPFILE" "dvb://$1" & sleep 5 mplayer -zoom "$TMPFILE" kill %1 rm -f "$TMPFILE" Might be lighter on disc usage. No particular reason to delete and recreate the fifo every time, though.</p> </div> Andy Balaam 2007-05-14</span> </h3> I’ll try it tonight. If it works, you rule.</p> </div> Andy Balaam 2007-05-15</span> </h3> It works except gmplayer complains "Seek failed" as it starts. If I use mplayer instead it’s fine but for radio channels that gives me no window to close when I want the noise to stop.</p> Why is gmplayer so annoying?</p> </div> Charles Bailey 2007-05-15</span> </h3> You only tried gmplayer to take the shine of my ‘rulage’.</p> </div> Andy Balaam 2007-05-15</span> </h3> Some Googling suggests the "-cache" option. I’ll try that tonight. Seems like the best way to do it if gmplayer co-operates.</p> </div> Restoring a backup with Duplicity Andy Balaam — Wed, 02 May 2007 23:38:00 +0000 See the last post for how I back up my important files to a remote server. To restore them I do this:</p> duplicity --file-to-restore path/to/file.txt scp://user@example.com//home/example/bkp/home/andy/Maildir file.txt</pre> and enter the passphrase when prompted.</p> Note that I can restore this backup even if I lose my entire home hard drive because it doesn't rely on any key file - just my passphrase. This is less secure than using a private key file, but where do you back up your key file to? You know it will be lost when you need it ... safer just to write down your passphrase somewhere, I reckon.</p> Remote encrypted backup with duplicity Andy Balaam — Wed, 02 May 2007 06:55:42 +0000 My father-in-law's hard drive died the other day and I was just able to recover the data using Knoppix</a> and my iRiver</a>. Since I recently discovered a superb program that lets me create encrypted backups on a remote server over ssh, I thought I'd spread the word. The program is called Duplicity</a> and it is simplicity itself to use. Here is my remote backup script:</p> #!/bin/bash # Copy this to /etc/cron.weekly and make it executable and owned by root # Back up my most important stuff from the local machine to a remote # server, encrypted with a passphrase function run_duplicity { echo Backing up '$1' ... PASSPHRASE=`cat /root/.blah/passphrase` duplicity $1 \ scp://user@example.com//home/example/bkp$1 } run_duplicity /var/lib/mysql/dbtobackup run_duplicity /home/andy/Maildir </pre> I have one "run_duplicity" line for each directory I want to back up securely. I have my passphrase in a file only reeadable by root in /root/.blah/passphrase, and you either have to type your ssh password each time you run the script, or you can set up a key to let you log in automatically using ssh-keygen.</p> GSSMP goes beta Andy Balaam — Tue, 20 Mar 2007 10:10:20 +0000 I've just released version 0.9 of GSSMP</a>, and since I've fixed all the bugs I know about (except working around a gstreamer bug, which I will try to do before 1.0), I've marked this release as beta.</p> This is the second project I've marked as beta, and I am determined I am going to get to 1.0 this time. That means no more features, just bug fixes, and hopefully a 1.0 release in a few weeks, when it's had a bit of testing.</p> Really, I think FreeGuide should have had a 1.0 release a couple of years ago, but I've always had a lot of features in mind before I feel it is complete. I think this was a mistake: it was pretty stable at one point, and it might have been helpful to label it as such.</p> GSSMP is not going to fall into that trap! It's supposed to be minimal, and that is what it is.</p> It's surprising how many features would be nice to have in a minimal app, though, like being a drag and drop target and checking the MIME types of files before playing them, not to mention cool stuff like gapless playback and volumne balancing...</p> But no, these will have to come after 1.0. My focus needs to go back onto FreeGuide after GSSMP 1.0, as I've been neglecting it, and it needs some love if it's ever going to get to 0.11 ... and 0.12 ... and so on for infinity ... and then after that, 1.0.</p> Recording in FreeGuide Andy Balaam — Wed, 14 Mar 2007 12:55:22 +0000 If you download the nightly builds</a> of FreeGuide, you can try out the new recording functionality.</p> There is quite a bit to do before this is ready for release. There are several bugs in the TODO list</a> and you have to create your own scripts at the moment to launch mplayer or whichever recording software you want to use.</p> When the release happens, we will incude scripts that are useful for common TV cards (actually, it's beginning to look like mplayer will handle all this for us, so all we need is one script) and make it easy to set up, with a default setup that should work for most people out of the box. Or that's the idea.</p> The good news is that Alex appears to be back and working on fixing the bugs, so the ever-receding horizon of the next release is somewhat less receding.</p> I've not quite managed to get it to actually record something yet, (mainly due to mplayer being extremely awkward) but it shoud work...</p> My daily backup Andy Balaam — Fri, 23 Feb 2007 06:26:37 +0000 I recently recovered my files from my daily backup, which had actually worked!</p> (Pop quiz: this is how I deleted half my home dir - what is wrong with this Makefile?</p> TMP_DIR=/tmp/gssmp blah: blah1 blah ... blah7: ... rm -rf $TMPDIR/ ... </pre>) Anyway, I am extremely grateful that the backup did work, but a slight problem was that I have been backing up absolutely everything, and never deleting anything that was backed up until I ran out of disk space.</p> So when I restored from backup, every file I had ever deleted (since I started backing up) re-appeared. This included files and dirs (and email) that had just been moved somewhere else - I now had 2 copies of them.</p> I've spent the night deleting stuff that I didn't actually want restored, and this has encouraged me to create an improved backup script that puts deleted stuff into a separate folder, named after the day it was deleted. I can delete those folders at my leisure when they get too big, but when a disaster happens like it did on Wednesday, I can restore back to a sane state by just ignoring any irrelevant deleted folders.</p> It's quite simple, but I hadn't thought of it before, so it might be helpful to others. Enjoy:</p> #!/bin/bash # Copy this to /etc/cron.daily and make it owned by root, and executable # Back up everything important from my main drive (hda) to the backup # drive (hdb). # Keep stuff that has been deleted from the real drive in a separate # backup dir, named after the day it was deleted. BKP_DRIVE=/media/hdb1 FLAGS="-a --delete-during --max-delete 100 --backup" DT=`date +%F` DELETED_DIR=deleted-$DT function run_rsync { THIS_DIR=$1 EXCLUDES=$2 echo rsync $FLAGS --backup-dir=$BKP_DRIVE/$DELETED_DIR$THIS_DIR \ $EXCLUDES $THIS_DIR/ $BKP_DRIVE$THIS_DIR/ rsync $FLAGS --backup-dir=$BKP_DRIVE/$DELETED_DIR$THIS_DIR \ $EXCLUDES $THIS_DIR/ $BKP_DRIVE$THIS_DIR/ } run_rsync /home/andy \ "--exclude /iRiver --exclude /downloads --exclude /.Trash --exclude /misc/qemu" run_rsync /etc run_rsync /var run_rsync /boot </pre> Creating .deb and RPM packages Andy Balaam — Fri, 23 Feb 2007 05:37:25 +0000 Since I've managed to get it to work for GSSMP, I thought I might share how I am generating .deb and RPM packages using a Makefile. These make it much easier for Ubuntu and Fedora Linux users (respectively) to install the program. It should also be helpful for people using Debian, who can try the .deb, and other RPM-based distros, who can try the RPM.</p> This project is all autotool</a>ed-up, so this may not be very helpful if you don't use autotools.</p> (Side tip: I don't understand autotools at all, but since I created the project with glade</a>, it did all the hard work for me.)</p> I created these files for the Ubuntu .deb package:</p> changelog:</p> gssmp (0.8-0ubuntu1) edgy; urgency=low * First release packaged for ubuntu -- Andy Balaam <myemailaddress> Thu, 15 Feb 2007 12:33:11 +0000</pre> copyright:</p> This package was debianized by Andy Balaam <myemailaddress> on Thu, 15 Feb 2007 12:33:11 +0000. It was downloaded from http://sourceforge.net/project/showfiles.php?group_id=187144&package_id=219138 Copyright Holder: Andy Balaam <myemailaddress> License: This package is free software; you can redistribute it and/or modify blah blah blah GPL boilerplate</pre> control:</p> Source: gssmp Section: sound Priority: optional Maintainer: Andy Balaam <myemailaddress> Build-Depends: debhelper (>= 4.0.0), autotools-dev, libgstreamer0.10-dev, libgnomeui-dev Standards-Version: 3.6.2 Package: gssmp Architecture: any Depends: libgstreamer0.10-0, libgnomeui-0 Description: Gnome Simple Stateful Music Player GSSMP is a music player designed to work with Gnome. It does not store a database of all your music, but is designed to work with music files organised into directories. It is small, unobtrusive, and tries to do what you would expect, instead of asking you questions. . It remembers which file was playing on exit, and continues that file from the same place when you start again. It remembers tracks you have been listening to recently and displays them in the "Recent" menu.</pre> I put those three files in my source tree under install/deb/.</p> For the RPM, I just needed one file, gssmp.spec:</p> ### ### RPM spec file for Gnome Simple Stateful Music Player ### ### Adapted from jEdit's spec file http://www.jedit.org ### Summary: A music player that doesn't interfere. Name: gssmp Provides: gssmp Version: 0.8 Release: 1 License: GPL Group: Applications/Multimedia/ Source0: %{name}-%{version}.tar.bz2 URL: http://gssmp.sourceforge.net/ Vendor: Andy Balaam <myemailaddress> Packager: Andy Balaam <myemailaddress> BuildArch: i386 BuildRoot: %{_builddir}/%{name}-root Requires: libgnomeui, gstreamer BuildRequires: libgnomeui-devel, gstreamer-devel %description GSSMP is a music player designed to work with Gnome. It does not store a database of all your music, but is designed to work with music files organised into directories. It is small, unobtrusive, and tries to do what you would expect, instead of asking you questions. It remembers which file was playing on exit, and continues that file from the same place when you start again. It remembers tracks you have been listening to recently and displays them in the "Recent" menu. %prep %setup -n %{name}-%{version} %build ./configure --prefix=/usr make %install make DESTDIR=%{buildroot} install %clean rm -rf %{buildroot} %files /usr/bin/gssmp /usr/share/doc/gssmp/AUTHORS /usr/share/doc/gssmp/COPYING /usr/share/doc/gssmp/INSTALL /usr/share/doc/gssmp/NEWS /usr/share/doc/gssmp/README /usr/share/doc/gssmp/TODO</pre> which I put in install/rpm</p> Then I modified my Makefile.am to build the deb and RPM when I told it to:</p> APP_NAME=gssmp # Create a deb package pkg-deb: pkg-src - rm -r $(TMP_DIR) mkdir -p $(TMP_DIR) cp pkg/$(APP_NAME)-$(VERSION).tar.bz2 $(TMP_DIR)/ tar --directory $(TMP_DIR)/ \ -xjf $(TMP_DIR)/$(APP_NAME)-$(VERSION).tar.bz2 cd $(TMP_DIR)/$(APP_NAME)-$(VERSION)/; \ echo | dh_make --single --copyright gpl -e myemailaddress -f \ ../$(APP_NAME)-$(VERSION).tar.bz2 cp install/deb/changelog install/deb/control \ install/deb/copyright $(TMP_DIR)/$(APP_NAME)-$(VERSION)/debian/ cd $(TMP_DIR)/$(APP_NAME)-$(VERSION)/; \ rm debian/README.Debian debian/*.ex debian/*.EX; \ ./configure; \ dpkg-buildpackage -rfakeroot; \ mv ../*.deb $(PWD)/pkg/ rm -r $(TMP_DIR); # Create an RPM package pkg-rpm: pkg-src mkdir -p $(HOME)/.rpm/RPMS/i386 mkdir -p $(HOME)/.rpm/SRPMS mkdir -p $(HOME)/.rpm/BUILD mkdir -p $(HOME)/.rpm/SOURCES mkdir -p $(HOME)/.rpm/tmp - rm -r $(HOME)/.rpm/BUILD/$(APP_NAME)-root - rm -r $(HOME)/.rpm/RPMS/$(APP_NAME)-* - rm -r $(HOME)/.rpm/SRPMS/$(APP_NAME)-* - rm -r $(HOME)/.rpm/SOURCES/$(APP_NAME)-* cp pkg/$(APP_NAME)-$(VERSION).tar.bz2 $(HOME)/.rpm/SOURCES/ rpmbuild --quiet --nodeps --define="_topdir $(HOME)/.rpm" \ -ba install/rpm/gssmp.spec mv $(HOME)/.rpm/RPMS/i386/$(APP_NAME)-$(VERSION)-*.i386.rpm pkg/</pre> Now I just needed to run ./autogen.sh to regenerate my Makefile, and then make pkg-deb or make pkg-rpm and the deb or RPM appears magically in the pkg/ directory.</p> Autotools, for all its faults, does make this stuff relatively easy. All the code for installing in different places (installing at all) and general flexibility is done for you. Of course, it's at the expense of ever understanding what's going on, but you can't have everything.</p> Now I just need to make a man page ... and fix all the bugs ... oh, and make an icon ... and make it accept a filename on the command line ... and sort out MIME types ... etc ... so it should be finished really soon.</p> Comments</h2> myITcorner: Tomasz Gebarowski programming blog » Blog Archive » Making Debian library package with dh_make 2008-10-12</span> </h3> […] Debian New Maintainers’ Guide Andy Balaam’s Blog Tags: debian, gnu, library, […]</p> </div> Moved to WordPress Andy Balaam — Fri, 09 Feb 2007 14:16:09 +0000 I've moved my blog to WordPress, which will allows you to comment on my posts.</p> I guess it's a sad day for DIYBlog</a>, but hey, it was just a workaround until I got some web space that supported PHP.</p> Happy interaction!</p> Comments</h2> Matt 2007-03-28</span> </h3> Please fix http://www.artificialworlds.net/diyblog/</a> so that it no longer says "To see what you can make with DIYBlog, see my blog."</p> I was interested in seeing what DIYblog looks like so I loaded your blog and said to myself: "wow, his DIYblog skin looks just like WordPress!" Only then I scrolled to the bottom and realised that it wasn’t DIYblog afterall!</p> If you still intend to keep your DIYblog page then it would be nice for that "my blog" link to go to an archive of your old blog so people can see what kind of output DIYblog produces. If you haven’t saved an old copy of your blog then you could use one of the copies from the Wayback Machine:</p> http://web.archive.org/web/20060521070058/htt...</a></p> </div> Andy Balaam 2007-03-28</span> </h3> Hi Matt, sorry about that!</p> I’ve modified the page to point at my personal blog aggregation page, which is created using diyblog.</p> </div> Two new babies Andy Balaam — Wed, 17 Jan 2007 23:11:43 +0000 I have 2 new babies to tell you about:</p> Gnome Simple Stateful Music Player</a> which is a music player that just stays out of your face, playing music off your hard drive without making a database or anything like that. It uses GNOME</a>/GTK</a>, GConf</a> and gstreamer</a> and is written in C, which is great fun.</li> </ol> I think C and Python are actually very similar: they are direct languages that are designed to be simple, and allow you to specify what you want to happen in a very straightforward way.</p> GSSMP is nearly ready for a 0.1 release - it's simple, but it does most of what I want. Its killer feature is that it remembers what you were listening to and continues from there when you restart. Sounds simple, I know, but sorely lacking from some music programs. It also starts quickly, as opposed to, say, slowly or, for example, very slowly.</p> Another one:</li> </ol> </p> New games! Andy Balaam — Thu, 21 Dec 2006 17:48:45 +0000 I've been really enjoying the GP2X</a> I got for our 5-year anniversary (!) and have got DVD ripping working, along with Quake playing at a decent speed, and lots of downloadable games, notably Spout</a>, which is excellent, and (unlike many games which are really too visually complex to see on a small screen) simple enough to work on a handheld. There are a few other good games too, and lots that I immediately deleted off my SD card. I'd like to see some of the very polished LGames</a> ported - maybe I'll get around to it myself sometime.</p> Anyway, all of that is peripheral really, because what I really like doing is writing games, not playing them. So far I have imposed two games on the unsuspecting public:</p> mop(e)snake</a> - a classic snake game with no featuritis - just a snake, apples (pain, actually - read the plot if you're interested...) and you. Try to "fill the screen with green"^{TM</sup></em>. It's reasonably mature now and works on GP2X, Windows and Linux. There's not a lot left to do, except a bit of polish and fixing any bugs that come up.</p>} duckmaze</a> - a maze game where you can move walls. This is based on a game I played years ago on a handheld, whose name I have forgotten. It's potentially quite fun, but I need to design some more levels - so far it only has 8, and several of them are pretty noddy. The first release includes a level editor (go me), so you can help out with designing more. Actually, you don't even really need the level editor, since the level file format is designed to be fairly human-readable, so potentially you could create levels in a text editor. I'm quite pleased with that, even though no-one is ever going to appreciate it now I've written the level editor.</p> Download, enjoy, write to the mailing lists with ideas, bugs, code etc.</p> Linux Journal - Forgiveness Andy Balaam — Mon, 18 Dec 2006 22:15:33 +0000 Dear LJ,</p> A few months ago I wrote to you saying we had to break up - you had become too chatty and opinion-filled for my taste.</p> I have to admit that it turns out I was bluffing.</p> I love you too much, and today I renewed my subscription.</p> You did take a significant dip in quality, but things have got better over the last couple of months.</p> More important, though, is that fact that you started from such a high quality base that even when it dipped, you were still the only Linux publication for me.</p> Yours forever,</p> Andy Balaam</p> Another FreeGuide release candidate Andy Balaam — Wed, 19 Jul 2006 09:16:48 +0000 I screwed up the RPM, so if you tried that, try the latest one from here: freeguide-tv.sf.net/rc</a></p> FreeGuide 0.10.4 release candidate Andy Balaam — Tue, 04 Jul 2006 16:47:38 +0000 Lots and lots of bug fixes, and some cool features sneaked in by people who obviously didn't read my TODO list ;) means that FreeGuide 0.10.4 is nearly ready.</p> You can test it by downloading the relevant installer from freeguide-tv.sf.net/rc</a>. I've put up a Linux RPM and tarball, and hopefully a Windows installer and other OS/distro packages will be up soon.</p> Reasons to try it:</p> There are loads of bug fixes in it.</li> You can now search for programmes!</li> You can try the new vertical list view.</li> It will help us if we find bugs before the release, and that will make you feel good.</li> </ul> A Quick and desperate introduction to data recovery Andy Balaam — Thu, 08 Jun 2006 04:07:29 +0000 No, I did not just develop a passing interest in data recovery as a leisure activity. Presumably like everyone else, I learnt about it because it was necessary.</p> A year or two ago my friend's computer was packing up, and he asked me to extract and store all their files for him until he got hold of a new one. These files were basically the whole history of their life in photo, movie, word document etc. form - it totalled about 3.9GB. We were pretty successful about extracting the files from his dying machine by booting into Linux and copying them onto my iRiver. I've been keeping hold of them ever since, with a copy on my iRiver and another on my home machine to be sure.</p> During the fairly disastrous upgrade of my machine from Fedora Core 3 to Ubuntu Dapper Drake that I did last week (of which, probably, more later) I managed to delete all the little scripts I keep in my home directory to do useful things. (They represent quite a lot of collected wisdom, and losing them has been annoying, but anyway.) One of those scripts called other scripts to delete old stuff I wasn't interested in from my iRiver, and synchronise my local copy of my iRiver's contents with the machine itself.</p> Obviously, I made a small mistake in re-writing that script, which meant that it started deleting everything older than 2 weeks old off my entire home directory. Since I had gone off to bed while this was running, I was extremely lucky that it hit a read-only file fairly soon so I didn't lose much, but what I did lose was the zip file into which I had put all my friend's files.</p> Of course, my synchronisation program worked like a dream, running as scheduled that night and merrily deleting the zip from my iRiver too. Lesson 1: Don't let your backup program delete files automatically</strong>.</p> If it had been my own data I would have been gutted, but the thought that I had lost someone else's lifetime collections of photos and sentimental things was pretty horrible.</p> So I began several long nights of investigating data recovery. I assumed it would be easier to get the files off my Linux ext3 partition, since it is well documented and hackable, but in fact, I learned lesson number 2: You can't recover deleted files off an ext3 partition.</strong> That is probably not strictly true, but in practice, it is. The reason is that when you delete a file in ext3 it blanks the record of it from somewhere, so all that's left is the actual data with no index entry. This makes it pretty much impossible to recover anything.</p> I tried lots of different things, including using grep and strings to search the device directly, and debugfs which appears only to be useful for ext2. The closest I got was a program called foremost</a> which knows what the start and end of each type of file looks like, and searches the raw data for things matching the type of file you are looking for. Foremost seems really cool, and it found lots of things that looked like zip files, but almost none of them actually were valid zips, and it didn't find anything big enough to be the file I wanted anyway.</p> So much for my Linux drive. Lesson 3: Consider ext2 for a backup partition</strong>. It is apparently easy to undelete deleted files on ext2. On the other hand, if the computer crashes while writing to an ext2 drive, you are much more likely to have corrupted data than if you use ext3. Quelle dommage.</p> I find it quite pleasing that the fact that I had a backup meant not that my files were backed up, which would of course be too easy for a person who has recently been branded Master of Pain (Receiving)</em> by his colleagues, but it did mean I had another deleted file on a different filesystem which I could attempt to recover.</p> My iRiver (which, incidentally, I love even more after this incident than I did before) uses a FAT32 filesystem, which everyone agrees is rubbish, but which has the highly relevant advantage of being simple and thus less difficult to recover data from.</p> I made a copy of the entire disk by doing dd /dev/sd1 > myhd.raw and then I was free to attempt recovery on this raw file without fear that I would overwrite something on the real disk.</p> I hoped it would be easy to get the file back off FAT32, but it turned out to be difficult as well, mainly because the file was so huge. I tried lots of different programs, most of which crashed unceremoniously, or couldn't find my file. Lesson 4: Don't give up</strong>. Many of these programs were lying to me - telling me to give up. Had it been my own data, I would have given up. We are several days into my nightly trial now. I was tired.</p> Then, like a shining beacon of professionalism and respect for data, over the horizon came Autopsy</a> and its underlying Sleuth Kit</a> tools. I installed them, and started the server by typing autopsy and then pointed my web browser at http://localhost:9999 and followed the instructions. It could see my file! I told it to recover it, and my browser said it was downloading the file.</p> And then it failed. I tried again. It failed.</p> So I tried with wget. It failed. I tried with curl. It failed.</p> I realised the problem - the file was over 2GB, which is not supported in some downloading software! I tried with wget again, after confirming that it supported large files. It failed again.</p> Lesson 5, which really I should have figured out ages ago is: Don't use large files</strong>. Especially not large zip files containing all your precious data, obfuscating the files from your recovery program.</p> Now, I knew autopsy was running on my local machine and then providing the file I needed as a download, so I knew I could access the file directly by using the tools autopsy was using underneath.</p> Autopsy, may the Lord bless you for making those tools accessible separately.</p> Autopsy had told me the inode number of the file I wanted, so a quick read of the icat tool's man page showed me what to do. I ran the command and ended up with a 3.9GB file!</p> So I unzipped it.</p> Obviously, it failed.</p> Full of optimism, I assumed that this really was my file, but that it had been corrupted slightly by having some of its data over-written. I started looking for zip file recovery programs.</p> To cut a long story short, there are a lot of zip file recovery programs for Windows and Linux, and some of the Windows ones appear to work using Wine, but in the end (which I should have thought of from the beginning) the one that actually worked, rather than tantalising you with the names of the files and then either crashing or charging you money to recover them, is the one written by the guy who invented the zip file. The trial version of PKZip for Windows</a>, running on a Windows machine successfully opened and repaired my damaged archive, with this summary message:</p> Extracted 3,151 files</p> Skipped 0 files</p> 62 errors/warnings</p> Yes, out of 3,151 files, 62 were damaged, and the rest were fine.</p> It was a long journey, but we made it. Finally, lesson 6: If it's important, have 2 backups</strong>.</p> Comments</h2> Grant 2008-02-03</span> </h3> Brilliant write up, and much sypathy, and congratulations!</p> The funny thing is, I have messed up more files, when backing up, then if I had left them alone!</p> I’m currently looking for 6hrs of work I deleted accidently after migrating a machine. Proably will take me more time to recover them!</p> My mistake was to use a GUI copying program in linux, not as root! (should have used the CL!) when moving files.</p> Now I have to try and recover some files from an ext3 partition. I also foolishly didn’t partition of my data directory, which would have saved me some of the agro.</p> Wish me luck!</p> </div> Andy Balaam 2008-02-03</span> </h3> Hi Grant, good luck!</p> I’m afraid you’re going to need it – ext3 seems to be extremely difficult.</p> My fingers are crossed for you.</p> </div> FreeGuide fan mail Andy Balaam — Thu, 18 May 2006 22:41:44 +0000 Really encouraged to receive this:</p> Hello Mr Balaam,</p> A quick email to thank you for giving the world the brilliant Freeguide. (I saw in your blog that you have a high volume of email - no reply needed).</p> I tried Digiguide for a while, loved the principle, found the interface too cluttered. Went back to downloading pages from here and there, and dipping into the Radio Times timeline, which is not quite customisable enough.</p> I happen to live in a place where I get an unusual mix of channels - UK free satellite + UK terrestrial +Irish terrestrial. I'd never found a single listings service which covers the options.</p> Then, by chance, I found Freeguide, and I love it. I couldn't ask for more (except, perhaps, for customisable colours for program categories :-) ) Incidentally, one of your blog entries wondered how many painless installs there currently were. Here's one more to be counted (WIn2K, Sun Java v5Update6)</p> I don't know much about software, but I'm sure this must have taken a vast amount of your time. Thank you for every moment of it. This program will be useful every day, and I'll be passing it on at every chance I get.</p> Best wishes Nick Nixon</p> Over-engineering gone mad Andy Balaam — Mon, 15 May 2006 15:44:47 +0000 Picture the scene: you are writing a Java application and you're trying to do things right, so you use the java.util.logging code to do your logging. To create a logger you do this:</p> Logger log = Logger.getLogger( "name" );</p> Everything goes fine until you want to set the logging level from a command-line argument. You set the logging level like so:</p> log.setLevel( lev );</p> Where lev is, e.g. Level.FINE.</p> When you run your program, none of your log messages appear except the ones that appeared before (SEVERE, WARNING, INFO). Nothing appears to have changed.</p> You read some API docs, and find out that each Logger has a list of handlers. So you call getHandlers() on your Logger.</p> It doesn't have any.</p> You take a breath.</p> You read some more, muttering about Sun engineers having spent too much time at university, and find out that there is a hierarchy of Loggers containing other Loggers.</p> You simultaneously wonder how that could be useful, and whether some stupid thing is happening here to do with that.</p> You try log.getParent() and find that it is not null. Eureka! You've got it now:</p> log.getParent().setLevel( lev );</p> You hastily recompile, palms sweaty with finally cracking a problem that should not have taken this long.</p> Nothing has changed. You still don't get your log messages.</p> OK, that's fine: you know about handlers now, so it's a simple change - we'll do this instead:</p> log.getParent().getHandlers()[0].setLevel( lev );</p> You recompile, trying not to get your hopes up.</p> Nothing.</p> You consider re-writing your application (which has been in development over 5 years) in any other language. You decide to try one more thing before giving up. What if you need to tell both the Logger and the handler what level to use? It would be pretty awkward, but not outside the realms of possibility.</p> log.getParent().setLevel( lev ); log.getParent().getHandlers()[0].setLevel( lev );</p> Not even very hopeful now, you recompile.</p> Nothing.</p> This is the moment where you go and find anyone who will understand (or failing that anyone at all) and tell them about it. They wipe the spittle from their faces and appear keen to leave.</p> You will not be defeated by this. You try every combination you can think of. You start looking into how to log a bug with Sun. You curse and curse again.</p> A very long time later, you know you should have given up, and you're trying things just out of bloody-mindedness, and you stumble across this:</p> log.setLevel( lev ); log.getParent().setLevel( lev ); log.getParent().getHandlers()[0].setLevel( lev );</p> Yes, that's right. To set the log level of the default logger in Java you have to learn about handlers and the logger hierarchy, and you have to set the level in three places.</p> THREE PLACES.</strong></p> You swear never to feel superior again when people start having "language wars" about which programming language is better. They are not all the same.</p> You engage in a relaxing pasttime, such travelling to Sun's development centre and punching the person who "designed" that interface in the face.</p> Keyboard layout change in Ubuntu Andy Balaam — Thu, 11 May 2006 18:03:24 +0000 I've had this problem a couple of times in my Ubuntu machine. I installed it with US keyboard layout, and then changed it to UK later (for my user). Even when I select the correct keyboard model (Generic 105-key (Intl) PC), and layout (United Kingdom International) some keys still act weirdly (e.g. pressing " does nothing, and if I press it twice I get a strange "-like symbol). I fixed it by running:</p> sudo xmodmap /usr/share/xmodmap/xmodmap.uk</p> Comments</h2> Andre 2007-08-01</span> </h3> Thanks Andy! Lost my swedish Ã…Ã¥Ã„Ã¤Ã–Ã¶, and found them here ;–)</p> </div> Andy Balaam 2007-08-01</span> </h3> Great news Andre – wouldn’t want you to have to survive without them…</p> </div> Vince 2008-01-15</span> </h3> Great! Works fine with a UK Keyboard stuck in French mode for historical reasons…</p> </div> Nick Mann 2015-07-08</span> </h3> http://askubuntu.com/questions/434849/change-...</a> has an answer that worked for me</p> sudo apt-get install console-common sudo dpkg-reconfigure console-data</p> </div> Andy Balaam 2015-07-08</span> </h3> Thanks Nick.</p> </div> Tony 2008-06-04</span> </h3> I have a Compaq Presario R3000 lap top. I can not seem to get the layout right no matter what. I purchased it in Canada. I guess the layout is the same as US. Non of my SHIFT characters are correct and i am missing @ from the keyboard entirely. Any suggestions?</p> </div> Ozbo 2008-07-29</span> </h3> Thanks, this solved my problem when nothing else could. By the way I needed my slovenian configuration by changing last two letters into si. THANKS REALLLLY</p> </div> arun 2008-11-01</span> </h3> thanx dude it ran</p> </div> Peter 2008-12-15</span> </h3> I’d like to use ubuntu with at least 3 language layouts without having to write scripts if possible. Is there any option like in windows where you can change with alt-tab?</p> </div> Andy Balaam 2008-12-15</span> </h3> Hi Peter, the "Keyboard Indicator" applet might help, but I’ve no idea really I’m afraid. You can install that by right-clicking on the Panel, choosing "Add to Panel…" and then clicking "Keyboard Indicator", then "Add". Good luck!</p> </div> Peter 2008-12-16</span> </h3> Muchas gracias :) That’s what i’ve been looking for. If i find a hotkey for the function, i’ll post it here. thx again :)</p> </div> Andy Balaam 2008-12-16</span> </h3> Glad to be of help :)</p> </div> jess 2009-04-06</span> </h3> thanks andy!!:)got really scared for a min!!thank you sooo much</p> </div> gowtham 2009-04-20</span> </h3> Hi andy and others, i have installed ubuntu in my DELL INSPIRON-1525, my keys are not working inside terminals.. Backspace is not functioning and many more strange symbols.. i have a layout of USA and DELL in my settings.. help me out.. thank you..</p> </div> Chris 2009-06-06</span> </h3> Cheers mate, another problem solved ;)</p> </div> Malgosia 2009-06-24</span> </h3> Thanks, that solved my problem of not having all this bountiful polish characters as well!</p> </div> random guy 2009-08-17</span> </h3> Thank You very much !!!!!</p> </div> chris lanz 2009-09-16</span> </h3> I was so glad to get your message about sudo xmodmap</p> Now at least I can do ‘ and ` and ~ and ^ but now " and @ are reversed. I’ll try some other layouts… .I guess I can learn to substitute shift-2 for shift-</p> I’m on a Dell Inspiron</p> </div> chris lanz 2009-09-16</span> </h3> Actually, it seems I’ll have to run the sudo xmodmap every time I boot. Is there any way I can get it into a config.sys or autoexec.bat, to use a couple of ancient terms?</p> </div> ulkas 2009-10-22</span> </h3> 2chris lanz: just put it into automatic sessions: menu-system-preferences-sessions</p> </div> miki 2010-01-26</span> </h3> andy balaam, yes!, this was a great help indeed, should be more widely known one, I am using Linux for 10 years and never got to it…Thanks! Croatian and Polish, with mandarin (zhujin), really good.</p> </div> Pox 2007-03-18</span> </h3> the Â¨Â¨ thing is caused by the International layout – pressing ‘ and then a will make Ã¡.</p> </div> Michael 2007-04-21</span> </h3> thanks Andy :–) that solved a few issues</p> cheers Mate</p> </div> Sean 2007-05-15</span> </h3> Thanks. This has been a big help!</p> </div> Andy Balaam 2007-05-15</span> </h3> Glad to be of service!</p> </div> Andreas Zieritz 2007-12-01</span> </h3> Seems to fix a problem i had on my Norwegian PC as well. Ã¸Ã¸Ã¸Ã¦Ã¦Ã¦Ã¦Ã†Ã†Ã† back again.</p> </div> FreeGuide source layout (and SVN migration) Andy Balaam — Tue, 02 May 2006 22:59:58 +0000 Christian and others have pointed out lots of problems with the source code layout in FreeGuide. The biggest problem is that each plugin has its own source tree which makes it very difficult to set up in an IDE like Eclipse. I knew it needed doing, and the other night I couldn't sleep and I felt like making it happen. Since SVN is so much better than CVS for re-arranging source code (because you can move files and keep their history) and because sourceforge are now offering SVN, I decided the right thing to do would be to migrate to SVN before making the changes.</p> So far, I have managed to get all the source code to build from the command line (instructions here: freeguide-tv.sourceforge.net/dev/index.php/Build_from_SVN</a>) and I'm working on making the Ant build work again, although there's still a bit to do there.</p> My aim is for it to be extremely easy to start developing FreeGuide: just download or check out the source code, fire up your editor or IDE, compile and run. That means FreeGuide needs to handle loading plugins from either JARs or directories (without being told which to do), and it needs to know where to look for its lib and doc directories if they are not specified.</p> Soon, it will Just Work. Then hopefully this will encourage more developers to get involved.</p> Meanwhile I've decided if I want to build up some momentum I need to balance my limited time between code and email, instead of just fire-fighting the email all the time. Some individual users may not get their questions answered, which I hate, but in the end it would be worse if the project lost interest and faded away. I need to inject some excitement!</p> Letter to Linux Journal Andy Balaam — Fri, 17 Mar 2006 16:06:30 +0000 Dear LJ,</p> I'd like to start by saying thank you very much for the years of quality articles and features that your magazine has provided. I eagerly await the arrival of each month's copy as it jets over the Atlantic to my doorstep. LJ has always provided detailed and informative articles on topics both mainstream and more quirky, but always with the deeply technical edge no other magazine or web site I have found can provide.</p> I am writing because the April issue proved a grave disappointment. Gone were the technical articles and gone were the interesting projects. In their place I found a plethora of opinion pieces most of which provided no new insight into their subject that I could not have found on Slashdot or OSNews, along with some beginner's articles on how to install a couple of PHP applications. Even the Doc had nothing interesting to say about the yesterday's-news trying-to-be-trendy refrain of "blogs, wikis and podcasts" which seemed to be repeated several times on each page. The articles which did contain decent technical content (notably "Work the shell" and "Remote Temperature Monitoring with Linux") were swamped beneath the river of fluff through which I found myself wading.</p> When Nicholas Petreley wrote his first uninformed and ill-judged "/etc/rant" I felt I could let it pass since it was just one page of an excellent magazine, but now that his style of writing - opinion rather then content - seems to be spreading to the rest of the magazine, I feel the need to express my disquiet at what he has been doing with this column.</p> /etc/rant is an article put aside each month for criticising free software projects. The fact that an editor would allow such an article to exist, let alone write it himself, displays a profound and worrying misunderstanding of the community to which he is trying to contribute. Free software projects live and breath through encouragement and just plain getting on with it. As a member of this community, the right way to express dissatisfaction with the way something is being done is to contribute to the project, or (if necessary) launch or encourage an equivalent project that does things differently. By criticising from the sidelines Mr. Petreley reveals himself to be an outsider - as a writer a more appropriate way to contribute to the community (and hence form a useful part of it) would be to praise and encourage projects which are doing the right thing, rather than sucking energy out of valuable and innovative projects which have involved large amounts of volunteer effort such as GCJ and Gnome.</p> I never thought I would say it, but if this continues I will be cancelling my subscription to LJ. Sadly, I don't think I'll ever find another magazine that will reach the level of quality I have come to expect from you.</p> Yours sincerely,</p> Andy Balaam</p> Linux Journal slides Andy Balaam — Tue, 14 Mar 2006 23:52:36 +0000 jdub notes</a> that Linux Journal is going to the dogs. If the only change were to the back page I could get over it and move on, but so far I am about half way through the latest issue (March 2006?) and I've read nothing but opinion pieces.</p> I can get all the rantish uninformed opinions I need on OSNews. When I want quality technical articles written by people at the top of their field which cover hardcore problem solving and creative projects, then I turn to LJ. That is why I pay REAL MONEY for a paper journal like this. Because it has substance.</p> If I have to cancel my subscription, where should I go?</p> Donation Andy Balaam — Wed, 08 Mar 2006 16:37:45 +0000 Forgot to mention: I got my first non-anonymous donation to the FreeGuide</a> project - Steve Fuller sent in a donation along with a request (but with no implication of linkage ;). Thanks Steve! It never ceases to amaze me how Americans will donate to projects like this. My feeling about British people (certainly myself) is that I would always think if I were going to donate to something I'd make it something "important" like Christian Aid, but actually I do think Free Software is important too (actually in order to achieve some of the same goals that Christian Aid have...). The idea of mixing philanthropy with self-interest, which seems to be natural to Americans, can feel a little dirty to us Europeans, but actually that is stupid.</p> Anyway, enough philosophy: Steve, your donation will be used to buy myself something nice (e.g. a GameCube game or some online poker money) and it has significantly encouraged me to get stuck back into FreeGuide</a> development (see the previous post ... guilt is rising). You never know, it might even subconsciously encourage me to get to your specific feature request sooner ;).</p> (The feature request was allowing favourites to match on programme descriptions and other, more specialised fields. I am wondering about how to keep the UI simple for this, but it should be do-able given some time to implement it.)</p> FreeGuide update Andy Balaam — Wed, 08 Mar 2006 16:28:24 +0000 I have very little time for FreeGuide</a> at the moment. Part of the problem is that there are lots of boring bugs to fix instead of interesting new features, and it's hard to motivate myself to get into it. I am really keen on making it Just Work every time, but since other people have been working on the code and quite a lot of it is unfamiliar, it's a lot of work to work out where to look to fix a problem.</p> Hopefully when I've purged this poker thing from my system I will find time to get into the latest TODO list</a>. At the moment all my FreeGuide time is going into trying to keep up with helping people on the mailing lists. Joe Blow is doing a fantastic job of providing an initial response to lots of mails, but often people are encountering weird bugs that need a developer to get involved. It would be great if some of the other people involved in development were able to help out, but it's quite a thankless task, and takes away time from developing...</p> It's frustrating when quite a few of the bugs we have are things I fixed in a previous version that have crept in with new implementations of existing functionality. It's taken ages to get FreeGuide stable, and recently it seems to have taken some steps backwards. My feeling is that the percentage of people who are able to install it with no problems and download listings straight away has gone down considerably. I _really_ want to fix this.</p> That's why the TODO list for 0.10.4 is pure bugs. We need to make it _work_ before we make it do new stuff. Otherwise people are just going to walk away.</p> Saying that, the addition of a new experimental vertical listings view by Christian is really exciting. I haven't even found time to try it out yet: I must examine my priorities. FreeGuide has always been my "main" project but I have been neglecting it as time presses in.</p> The Boy (as I call him) is doing well and has slept well the last 2 nights, giving us a break from the all-over-body pain of extended tiredness. If this is part of a trend towards more sleep, I may get more time to make FreeGuide better.</p> Faster Poker Analysis Andy Balaam — Tue, 07 Mar 2006 17:41:11 +0000 Forgot to mention that I also have one other optimisation up my sleeve: instead of fully analysing each hand I could only do that when I need it i.e. when the hand type is the same, so I could just return "flush" and only if your opponent also has a flush would I need to say "flush ace high" or whatever. This provision of the detail is actually quite time-consuming, so it may be a significant improvement to avoid it in most cases.</p> Current timings (without the above optimisation) on a P4 2.4GHz machine: [andy@andy-p4-linux src]$ time ./poker 5 # Analyse all possible 5-card hands Straight flush: 0.001539% Four of a kind: 0.024010% Full house: 0.144058% Flush: 0.196540% Straight: 0.392465% Three of a kind: 2.112845% Pair of pairs: 4.753902% Pair: 42.256903% High card: 50.117739%</p> real 0m4.058s user 0m3.805s sys 0m0.002s [andy@andy-p4-linux src]$ time ./poker 7 # Analyse all possible 7-card hands Straight flush: 0.016811% Four of a kind: 0.168067% Full house: 2.555057% Flush: 3.039766% Straight: 4.475084% Three of a kind: 4.873941% Pair of pairs: 23.512489% Pair: 43.946865% High card: 17.411920%</p> real 3m43.479s user 3m42.963s sys 0m0.047s</pre>This is nice, because there are 51 times as many 7-card hands as 5-card ones, and the analysis only takes 55 times as long even though it's more complex to analyse larger hands, so it means my analysis scales pretty well with the number of cards.</p> Poker Analysis Andy Balaam — Tue, 07 Mar 2006 16:59:00 +0000 I've just requested sourceforge space for my latest mini-project: Poker Analysis. I've been a big fan of poker for years, and am pleased to see everyone is catching up with me finally with the boom of Internet poker. I've started playing a little online, and it set me thinking about the algorithmic aspects of analysing the game. Obviously, the real skills of poker are about predicting people's actions, but before you can do that you need to be able to see what hand you've got and what possibilities are out there, and that stuff is entirely deterministic.</p> I like small, self-contained challenges, so I set about writing a program to play every two-person game of Texas Hold 'Em poker that could ever happen, and ranks the two-card hands you can have in order of how many games they win. Obviously most people have a pretty good idea of which hands are good or bad, but I bet there would be some interesting results to see the full ranking.</p> Of course, I'm sure lots of people have done this before, but I want to do it too! Also, of course, it doesn't matter how good your hand is in the grand scheme of things, just whether it beats the other hands out there, but bear with me: I'm interested in the results.</p> The major challenge is getting it to run fast enough, and that is what I'm finding interesting. I've already changed my algorithm several times and made huge improvements in execution time, but to run through all the trillion or so (I think) hands would still take several years to execute on my home machine, so I've got some work still to do.</p> There are 2 major parts to the algorithm: enumerating all the hands you could get, which sounds easy but is in fact slightly tricky, and difficult to do perfectly if you want to avoid repeating redundant analyses, and finding out what hand you have (e.g. pair, flush). I'm pretty happy with my current implementation of the second, but I've got quite a lot of work to do on the first.</p> To analyse what hand you've got I'm taking an approach where all the real work is done when you sort the hand, and then looking for pairs or straights is easy since the cards are already next to each other. I then re-sort the cards so that the same suits are next to each other, and that makes it easy to find flushes. The part I'm really happy with is that the way I choose hands to analyse means they are already sorted, and when I transform them into the sorted-by-suit instead of by number order, I have very little work to do the way I've done it. (Some slight extra complication is added by the need for aces to be either high or low in straights.) This means that deciding what hand you've got is pretty quick, and I am able to enumerate and analyse every 5-card poker hand (2.6 million hands) in 4 seconds on my home machine.</p> I believe I might be able to speed up the analysis a little by changing my encoding of cards which at the moment is just consecutive integers (e.g. 0=2c, 1=3c, 2=4c, ..., 12=Ac, 13=2d, ...) but might be better to have each suit starting at a power of two to allow the use of the shift operator instead of %13 or something, but I'm happy for the moment. Where I need to improve is in writing the code that enumerates all the hands. As I have it at the moment, it plays every 2-card hand against every other 2-card hand for every 5-card set of community cards (but obviously preventing repeats of the same card) but this is hugely redundant (something like 24x?) since the suits are symmetrical. Exactly how much redundancy I can squeeze out of it is what I am thinking about now.</p> Anyway, hopefully soon the code will be in sourceforge CVS, and when I've achieved my goal of examining all games I'll make a release. Hopefully in the future I'll make it accept as arguments the current situation in a game and output the probabilities of each player winning.</p> Podcasts I listen to Andy Balaam — Mon, 27 Feb 2006 17:43:24 +0000 I use a modestly-modified BashPodder</a> to download my podcasts (Internet radio shows) automatically every night, and sort them into different directories. If anyone is interested in my modifications, do ask and I can send them to you. I haven't sent them to Linc since there seem to be a million modifications, which probably do everything mine do, but better. The only modestly interesting thing I added was the ability to convert from m4a format to mp3 automatically, and that doesn't seem to work, and the Adam Curry PodFinder show is so bad that I realised I didn't want to be able to do it anyway (note no link since it's really not worth it).</p> Anyway, that brings me on to my main point, which is that I have subscribed to a lot of podcasts, and unsubscribed to a lot too. I have now honed my list to ones that I really like, so why not share?</p> LugRadio</a> - excellent Linux show that even my wife enjoys (she is not in the least interested in Linux) because the four (British) presenters are really funny.</li> Poker Diagram</a> - two British guys playing online poker and talking about it. I've learnt a lot about poker from this, and really enjoy listening to it. Of no interest to someone who doesn't know Texas Hold'Em poker pretty well, as it's tricky to follow just by audio description.</li> The Linux Link Tech Show</a> - US Linux show that is broadcast live, and then podcast unedited. Not in the least professional, but entertaining if you are interested in Linux and/or like listening to the ramblings of some totally ordinary Americans. It's not so often that you get to hear from real people in America, instead of TV stereotypes, and I like that aspect of it as well as some interviews with interesting people.</li> Mark Kermode</a> - really funny film reviews originally broadcast on the UK's Radio 5 (which I would never listen to, since all my radios are permanently tuned to Radio 4). I really trust his opinions, and I love to hear him destroy bad films. It's remarkable that a true horror fan has managed to get himself a show on national radio, and not just reviewing horror films, but all films. He, as he is the first to admit, is always right.</li> From Our Own Correspondent</a> - fascinating "personal reflections" by BBC correspondents who are living in all kinds of interesting places. Journalists, who often have to hold back poetic thoughts to be impartial, are allowed to do whatever they like on this programme, and the results are often very moving, and almost always the kind of thing you talk to people at work about the next day.</li> Mashup Town</a> - because I am in with the kids. Occasionally has a really good song e.g. the mashup of Eminem with The White Stripes that was amazing and I accidentally deleted. Email it to me if you've got it.</li> </ul> Basically everything else I've listened to is almost completely not worth it.</p> Look out this year for loads of corporate AdvertCasts and avoid them.</p> Setting up my ideal email system 4 Andy Balaam — Mon, 27 Feb 2006 17:14:00 +0000 Sending mail</h3> I used to do something clever with sendmail, but then spamcop blocked my computer because it apparently wasn't set up properly, so now I use my ISP's smtp server like everyone else. When I need to tunnel in I simply create a tunnel from localhost's port 25 to my isp's smtp server's port 25, and set localhost as my smtp server on the machine from which I am tunneling. It works fine, and there was really no need to use my own sendmail in the first place, it turns out.</p> Next time: how I made my own webmail server.</p> TODO for this blog:</p> FreeGuide update</li> My poker analysis program</li> </ul> I spawn Andy Balaam — Mon, 06 Feb 2006 21:38:46 +0000 </p> Setting up my ideal email system 3 Andy Balaam — Mon, 12 Sep 2005 18:11:11 +0000 Fetchmail setup</h3> My ISP provides my email by POP3, so I use fetchmail to download it every 10 minutes. I created a file .fetchmailrc file in my home directory that looks like this:</p> # Configuration created Wed Jul 27 06:43:07 2005 by fetchmailconf set postmaster "andy" set bouncemail set no spambounce set properties "" poll mail.myisp.co.uk with proto POP3 user 'andy@myisp.co.uk' there with password 'password' is 'andy' here</p> This tells fetchmail to use the username and password given to me by my ISP to download mail into the default location for the user andy on the local machine. This would normally put my mail into /var/spool/mail/andy, or something like that, but to get it into my Maildir (which I talked about in a previous blog entry</a>), I added these lines to a .procmailrc in my home directory:</p> MAILDIR=$HOME/Maildir/ DEFAULT=$MAILDIR LOGFILE=$MAILDIR/log</p> Now when I type the command fetchmail (logged in as andy) I get something like this:</p> $ fetchmail fetchmail: Server CommonName mismatch: localhost != mail.myisp.co.uk fetchmail: Server CommonName mismatch: localhost != mail.myisp.co.uk fetchmail: Server CommonName mismatch: localhost != mail.myisp.co.uk fetchmail: No mail for andy@myisp.co.uk at mail.myisp.co.uk</p> (No idea how to get rid of those errors, but they don't seem to cause any problems.) And if any mail was found, it appears in my Inbox in Thunderbird, as if by magic.</p> Next time: how I send mail, using sendmail.</p> FreeGuide 0.10pre2 Andy Balaam — Mon, 12 Sep 2005 17:59:43 +0000 So I've put out FreeGuide</a> version 0.10pre2, and then a little bug-fix release because the Linux RPMs were broken. By calling this 0.10pre2-2 I managed to annoy my faithful Gentoo packager Christian, and myself, but at the time it seemed too much hassle to bump everytjing up to 0.10pre3.</p> The next release, 0.10.3, is</strong> going to be stable, and everyone is finally going to be using and testing our latest code stream. We've had a few bugs reported again 0.10pre2, and we've already got a couple to fix, and then we're going to push it out asap and get everyone migrated up.</p> I hope people will like the new version - it's a lot faster, and the UI experience is better (in my opinion). I just hope there aren't too many wrinkles in the upgrade path.</p> Alex has written some good upgrade code, but some of the concepts have changed, and I'm not entirely happy with the depths you have to delve into the Options screen just to play with your XMLTV grabber, but most things should "Just Work" for normal users.</p> After 0.10.3, I really want to get recording working, using Reuel's plugin. If anyone wants to send me a UK digital TV card, you can be sure it would be put to good use...</p> Setting up my ideal email system 2 Andy Balaam — Wed, 31 Aug 2005 20:19:10 +0000 Setting up Mozilla Thunderbird</h3> On the same machine that was running Dovecot, I set up Thunderbird</a> to be able to access my email. I did this by creating a new email account (which I called "Andy IMAP"), choosing "localhost" as the server, and entering the username I created in /etc/imap.passwd ('brian' in the example I gave) as the username. I set up all the other settings as I wanted them, and clicked OK.</p> When I clicked "Get New Messages," Thunderbird asked for my password, and I entered the password I had put into /etc/imap.passwd ('sausages' in the example I gave). Now I could see some folders had appeared under the new account I had created, but there was nothing in them.</p> I tested whether I could read and write emails to the account by dragging an email from another folder (in an old account) to the "Inbox" folder in the new account "Andy IMAP". It copied OK, and when I clicked on Inbox I could see the email in the list, and when I clicked the email its contents appeared in the preview pane. Success! My IMAP account existed.</p> Next time, how I made emails that people sent to me appear in my new Inbox.</p> Setting up my ideal email system Andy Balaam — Fri, 19 Aug 2005 19:23:15 +0000 Recently I have managed to get pretty much my ideal email system set up. This is the kind of thing that would simply be impossible on a Windows box, but (when you know how) is a quick job on Linux. I didn't know how, but now I do (ish). I'm going to do it in separate installments, trying to make them slightly self-contained.</p> Setting up an IMAP server</h3> I chose Dovecot</a> because it looked fairly good, and it came high up on Google. Installing it on Fedora Core 3 was just:</p> yum install dovecot</p> But then I needed to set up the config file. The only thing I changed looked like this:</p> default_mail_env = maildir:~/Maildir</p> I did this because after reading the scare stories about locking when using an mbox-type mailbox, I decided Maildir sounded safer. Maildir holds each mail in a separate file, which sounds good for preventing corruption if something goes wrong, so I chose it. It seems to work ok.</p> I left the passwords to be plain text for the moment, which meant my /etc/imap.passwd had to look like this:</p> brian:{PLAIN}sausages:500:500::/home/brian</p> Where 'brian' would be your username and 'sausages' the password you have decided to use for mail. If you're using plain text passwords, don't use the same password as your login password, because any old packet sniffer will be able to see the mail password.</p> With this set up, I had a working IMAP server which stored messages in my home dir under the ~/Maildir directory. Of course, there were no messages in there at the moment - that's coming up later. Next time, how I made Mozilla Thunderbird see the messages stored there!</p> Just a little bit longer... Andy Balaam — Tue, 09 Aug 2005 15:56:21 +0000 Now that my Christian youth holiday is over, I've got just one more major job to do before I can get back to working on FreeGuide</a>: my thesis corrections. I don't know how long it's going to take, but it will certainly be painful. It might be best to avoid me for a few weeks if you don't like "grumpy Andy".</p> BashPodder Andy Balaam — Fri, 22 Jul 2005 15:08:54 +0000 I really must get BashPodder</a> working, not only to download my two favourite Linux shows TLLTS</a> and LugRadio</a> but also because Radio 4</a> has started making Today</a> and the best radio programme out there, From Our Own Correspondent</a> available for podcast download.</p> Python objects in C++ Andy Balaam — Tue, 19 Jul 2005 15:12:30 +0000 Sometime I find myself thinking in Python</a>. Or, accurately, wistfully thinking of Python. Raw C++ is completely hamstrung when it comes to simple concepts like handling strings or dealing with lists (maybe BOOST</a> is better?). Why can't I just do this:</p> remaining_args = sys.argv[3:7]</p> I don't even mind if it looks like this:</p> PyList argv = new PyList( argv, argc ); PyList remaining_args = argv.slice( 3, 7 );</p> Is this kind of thing possible? If not, someone really should package up the Python library in such a way that it is.</p> Of course, I'm speaking from a position of total ignorance here: maybe someone has done what I'm talking about with the Python stuff. After a brief look at BOOST, I can say for sure that it continues the STL's tactic of "under-use through obscurity" and "surprisingly missing all the useful features". No slice operator, anyway...</p> If they have done/do do the Python thing, the most frustrating part then is going to be wishing you could have all the bits of the Python library that are written in Python. Does anyone fancy starting the "GCP" project?</p> CD writing on Fedora Core 3 Andy Balaam — Sun, 17 Jul 2005 18:06:04 +0000 I'm now thing of moving off Fedora Core 3 (probably to Ubuntu, or maybe FC4 for the Java stuff) and it turns out I've never written a CD from it, because when I tried it didn't work: I kept getting the message "cdrecord: Success. Cannot open SCSI driver." I finally managed to find a solution. For some reason I need to do this:</p> cdrecord dev=ATA:1,1,0 -eject -v speed=2 ubuntu-5.04-live-i386.iso</p> Note the ATA: in the dev bit. No idea why.</p> Prince of Persia - The Sands of Time Andy Balaam — Fri, 15 Jul 2005 16:47:47 +0000 arrived through the post from ebay yesterday. It is superb</strong>. Exactly my kind of game, and with loads of features from the old game (one of my favourites of all time), all updated and made even more beautiful. It is incredibly atmospheric, and so far the gameplay has been utterly compelling. Am I more excited about this than the pervious post? Surely not.</p> Personal stuff is boring Andy Balaam — Fri, 15 Jul 2005 16:46:28 +0000 ...and I am not a big fan of pictures in blogs, but this is a special occasion:</p> </p> World domination is nigh. Soon we shall all be Balaams.</p> Sourceforge Andy Balaam — Wed, 13 Jul 2005 15:46:29 +0000 I have requested a Sourceforge</a> project for DIYBlog</a>, and it just struck me again how utterly fantastic Sourceforge is. What would the Free and Open Source Software world be like now without Sourceforge? A lot poorer, I reckon.</p> I'm desperately trying to find time to test FreeGuide</a> for a 0.10.2 release. So far it looks like although my favourites have been imported, they are not working. In fact, when I create a new favourite it works for that run but when I restart the program they are not highlighted. Hmm.</p> London Andy Balaam — Fri, 08 Jul 2005 15:23:01 +0000 London is a little quieter than normal this morning, but still pretty busy. It seems like most people are going to work, although some are having to walk. There are definitely fewer tourists on the streets.</p> London terrorist attacks Andy Balaam — Thu, 07 Jul 2005 19:46:28 +0000 As a largely unmilitarised country, Britain (like many other Western countries) is a soft target for terrorists. But that very softness makes attacking us in this way pointless: it's like punching a sponge. We won't change - we like our country how it is.</p> It's possible that by drawing attention to themselves, the people who did this may have changed some aspect of the world, but as for advancing their cause - how can they when the only discernable motive they have is hatred? Does murder really win them support at home? I doubt it.</p> We in Britain have done and continue to do some terrible things. I wish we would stop, but today isn't going to make any difference either way.</p> If you or anyone you know was hurt today, I'm sorry.</p> The dangers of learning to code Andy Balaam — Thu, 07 Jul 2005 15:45:15 +0000 I was speaking to someone over the weekend who works in HR and has been enjoying making various Microsoft Access databases to do useful things for work. I warned her she might get the programming bug and then never be able to stop. Then I realised she already had it when she summed it up beautifully:</p> "You can make it do</em> things."</p> </blockquote> I love coding.</p> Train gaming Andy Balaam — Wed, 06 Jul 2005 15:34:15 +0000 For my train journey, I ended up downloading a few emulators that didn't work, and a few games that didn't work, but what did work was the PC version of Prince of Persia, and the PC version of Cannon Fodder, both of which were very enjoyable.</p> Especially good were some new levels someone had created for PoP - a lot harder than the original ones! It did show how much work went into creating games, even back then, though, since there were a couple of subtle bugs in the new levels that led to you getting trapped with no means of killing yourself, which means your entire game is over and you have to start again, which is quite frustrating.</p> I did dig out the code for my track and field game, but when it appeared all on one line with weird symbols for the carriage returns, and I contemplated the idea of fixing this (and coding generally) in notepad, I decided to go back to PoP instead.</p> More on the track and field game later. It's in Python...</p> Constantly changing passwords Andy Balaam — Mon, 04 Jul 2005 15:24:38 +0000 I know it's supposed to be best practice to force users to change their passwords regularly, but how are we supposed to remember them? I wonder whether anyone has done any research into passwords becoming lower quality as people are asked to change them. It just becomes increasingly difficult to think up memorable things that are not obvious, or look for ways of fooling the computer into letting you have a similar password every month.</p> Gaming on the move Andy Balaam — Fri, 01 Jul 2005 15:14:47 +0000 We're taking a 4.5 hour train ride to Leeds at the weekend, and then even longer on the way home, so I am hoping to entertain myself using a Windows 2000 laptop with no CD drive. The options I'm thinking of are:</p> Pingus</a></li> Prince of Persia</a> on WinUAE</a></li> Continuing work on my secret Track and Field game by porting the UI stuff to Windows.</li> Booting some kind of Linux off my iRiver. Not at all sure this is possible.</li> </ul> The easy way would have been to run Games Knoppix</a>, but there's no CD drive. Gutted.</p> ATV Palm export Andy Balaam — Tue, 28 Jun 2005 20:10:51 +0000 We've already had a bug report/feature request for the ATV Palm plugin Alex wrote, so it's obviously going to be pretty popular. It's really great that Alex is doing so much work on FreeGuide</a> at the moment. We are having our disagreements here and there, but FreeGuide</a> is getting a lot better all the time.</p> Nearing a new release Andy Balaam — Tue, 28 Jun 2005 20:02:16 +0000 FreeGuide</a> is hopefully nearing a new release. Alex has been working on the plugins system, and hopefully I will be able to test it this week. If it works ok, we'll release it as unstable, and then with a few bug fixes we should be able to make the next release stable. Alex has refactored the code so that the main program is just another plugin. This is pretty cool as it means we can upgrade everything without restarting the program (hopefully, although in practice we think we may hit problems with resources not being freed). We are currently discussing exactly who should be able to upgrade what plugin when. I am keen that users be able to get a new listings grabber as soon as they need one without needing their admin to do it for them. On the other hand, probably only the admin should be able to upgrade the main app. The way jEdit</a> does these things seems to work pretty well.</p> DIYPlanet Andy Balaam — Fri, 24 Jun 2005 08:27:34 +0000 When I wrote DIYBlog</a> it occurred to me that what I was writing would translate into a blog aggregator pretty easily, and so I'm working on making that so. I'd like to start a little community of Java developers who are working on small free and open source software Java programs, so we can start to communicate through our blogs and help each other out with thorny issues e.g. how to get our programs working with an all free software systems (more on that later).</p> For the moment I'm working on aggregating my own favourite feeds, which you can see here: Planet Andy</a>. All this is/will be running on pure FTP web space, with a little cron job running on my own machine to keep it updated. Of course, that does mean it only works when my machine is turned on...</p> Re-implementing Andy Balaam — Thu, 23 Jun 2005 20:26:11 +0000 Been discussing with Alex about the dangers of re-implementing - he wants to simplify the command-line arguments code in FreeGuide</a>, but rather than modifying the existing code, he's re-written it. The problem with re-writing is that old code usually has lots of non-obvious bug fixes in it which you will have to re-implement all over again as they are reported. Alex has re-written a few things, and generally he's improved the code structure, but we have both then had extra work to do fixing the bugs that are inevitably introduced. That can seem like a waste of time if they're bugs that I have already fixed in the past.</p> It always feels easier to write your own new version instead of understanding what's already there, but in reality, unless you're changing the output, it's never a good idea completely to re-write the internals. There's a good article by Joel Spolsky</a> on this.</p> Honestly my last post about my viva Andy Balaam — Wed, 22 Jun 2005 18:09:07 +0000 ...for a while.</p> Woke up at 4.30am today having awful dreams about how none of the code worked. My dreams transposed the problems I am trying to solve at work onto the code I need to write for my corrections. I couldn't stand it any longer so I got up and started work on it, and I've launched an experiment now which should hopefully provide all I need in terms of experimental work. So, it's not as bad as my dreams...</p> The idea of getting back into the writing part also terrifies me. I guess another few weeks of getting up in the small hours might get it done, but I'm likely to lose all my friends. Obsessing about this is already making me totally inward-focussed. I am trying to make an effort to think about what other people might be feeling, to try and keep my self-pity in check.</p> Corrections Andy Balaam — Tue, 21 Jun 2005 15:07:16 +0000 Well, I passed, so I should be happy, but I've got to do some corrections before I become a doctor, so I'm feeling disappointed, but really I'm very close to finally achieving what I've worked so hard for. Just one last push.</p> Viva today Andy Balaam — Mon, 20 Jun 2005 12:56:56 +0000 Today I find out whether I get a PhD. I've been strangely not-nervous about it, although I've woken up pretty early this morning. It's difficult to `revise' since most of what we'll be talking about is what I wrote. I've been trying to refresh my memory of papers I cited (especially those of my examiners...) and of what I wrote, but it's been so boring I haven't done much. Let's hope I don't regret it later.</p> I'm fairly hopeful about passing - I would hope my supervisors would not have let me submit if it wasn't going to pass, but I see the viva as being both a rite of passage everyone has to go through (i.e. a horrible experience that my examiners had to go through, so they want to inflict it on me, and a negotiation process about how many corrections I need to do.</p> I'm not worried about having a hard time today: I tend to stand up reasonably well under face-to-face pressure like this, and I hope I can robustly defend points of which I am confident, and give way in places where I should refine my thinking. I'm more worried about the corrections. I just can't imagine getting back into writing my thesis - I've left it so far behind my thinking, I'm so sick of it, that the idea of writing another chapter or two is horrific. It was bad enough the first time.</p> Mnemonic Andy Balaam — Fri, 17 Jun 2005 18:53:07 +0000 I made a page for the band Pete Beresford and me had when we were at uni. You can download the 3 songs I was getting around to sending to John Peel. If you like your metal instrumental and a tiny bit Propellorheads, it might be for you: Mnemonic</a>. If you like it, let me know and I'll send you some FLAC</a> recordings of those songs (still not great quality, I'm afraid), and/or some of our other songs.</p> FreeGuide's TODO list Andy Balaam — Thu, 16 Jun 2005 23:19:19 +0000 Take a look at the latest TODO</a> list. There are several categories of things on it:</p> Already done e.g. "Add firefox to the possible browsers."</li> Completeness features e.g. "Use Java's printing capabilities to print"</li> Feature creep e.g. "Automatically make a channel set when you right-click a channel and choose "Move up" (or even drag!)"</li> Difficult-to-trace bugs e.g. "Make it not say no listings after midnight"</li> Difficult features e.g. "Minimize to system tray"</li> Subtle UI improvements e.g. "Remember where you were in the Options screen"</li> Major features we are in the middle of e.g. "Run an exe when a programme is about to start and/or end" (this is actually recording)</li> </ol> I should go through and make sure everything is relevant, but also I need to prioritise what to do before 1.0 if we are ever going to get there. 7s are being done by Alex and others, 2s really need doing if we are going to call it finished, 4s should get cleared up during the beta stage (hopefully), most 5s are likely to get dropped, and 3s and 6s are good potentials for a public vote - then the most wanted ones will get done and the others will have to be post-1.0.</p> I may go through the list and actually classify them this way, and then we can see what we're doing.</p> FreeGuide packaging and plugins Andy Balaam — Wed, 15 Jun 2005 14:53:00 +0000 I couldn't sleep last night, I think because my viva for my DPhil (PhD) is coming up on Monday. Work is going to be a struggle, but at least I got a chance to think about FreeGuide</a>.</p> Alex is working on a plugins system (it's sort-of-working in CVS) that will allow us to split the grabbers out from the main program, and allow people to innovate with new plugins without delving into all the code, which is quite large and a bit messy to say the least. Both of these things are really good, especially because my own time is very limited, so allowing people to work on plugins seperately means progress can continue without me being such a bottleneck.</p> However, handling plugins means we need to change the way FreeGuide is packaged. I've had some thoughts about how to do it, and I've put them together into an OpenOffice.org Impress presentation</a> (or try the PDF</a>). I'm looking for comments on this before we finalise the system. Once it's established it will be hard to change, so I want to try and get it right.</p> If there are a couple more sleepless nights we might even see FreeGuide's 0.10.x branch going stable ... Alex has a few more bugs to fix, but we're getting there.</p> DIYBlog Andy Balaam — Tue, 14 Jun 2005 16:17:47 +0000 I've made a page about how I'm making this blog, here: DIYBlog</a>. I may even get around to putting it on freshmeat</a> at some point, and if others want to contribute I may even put it on SourceForge</a>.</p> RSS working? Andy Balaam — Mon, 13 Jun 2005 23:17:00 +0000 I think I've managed to work out a system to make RSS work. Basically I write my blog entries into a raw RSS file, and then run a little Python script to add ids, links etc. and generate an HTML page from that. It's been a hassle to write (mainly due to timezones, which I've now decided to ignore) and if anyone's interested in some highly-customisable blogging software that only requires FTP on your web host, get in touch, especially if you're the kind of person who thinks its cool to hand-code your RSS.</p> Prediction: corporate Linux Andy Balaam — Thu, 09 Jun 2005 20:23:00 +0000 By Thursday 9th June 2015 30% of corporate desktop computers will be running operating systems (and desktop environments) which satisfy the Open Source Definition</a>.</p> Guilty Expression Andy Balaam — Wed, 08 Jun 2005 16:03:00 +0000 Having previously been the anonymous benevolent dictator of Guilty Expression</a> I have decided to become non-anonymous and I'll just have to make sure what I write there is either acceptable or I'll have to log out before I write it. Guilty Expression is a web site for people who want somewhere to express the thoughts and feelings they have about God and Christianity that they can't express in their church (if they have one) or with people in real life.</p> I set up the sire after talking to several of my friends who used to be church-going Christians, but were hurt in one way or another (by their church, or by God in the sense of things happening to them) and are now almost-Christians, or ex-Christians but who still think a lot about these things and want to talk about them without being branded unsound or a heretic. A couple of these people have very genuine relationships with God, which, by their very messed-up-ness are very real, but they obviously suffer from not being part of a community. GE is supposed to provide a way back in to talking to people like them and people who feel more confident in their faith (but are very tolerant of those who aren't).</p> GE is also a place for Christians to complain about their churches and the attitudes of other Christians, and other people, and almost to let off steam about things in a way that does not have very bad consequences in terms of hurting people or causing division in their church or group.</p> We've had articles written by people of lots of different ages and backgrounds (from pastors to non-Christians) on loads of topics (from anorexia to the bible to science) and lots of different viewpoints.</p> My excuse for writing about it on what I hope to be a mainly technical blog is that it presents an interesting challenge in terms of how to build a community. The site is supposed to be for people who are feeling disposessed and out of it, and so they're not likely to be looking for it. The vast majority of visitors and contributors to the site are people I have told about it personally, and it has spread a little from there by word of mouth. My being anonymous has not helped, because it has made it difficult for me to spread the word through my other web presences, and that ti why I've decide to own up. If you have any ideas about where it might be useful to spread the word, let me know by email.</p> If you're interested in GE, go to the site: www.guiltyexpression.org</a> and consider signing up for the mailing list to hear every now and then when a new article is published.</p> RSS Andy Balaam — Wed, 08 Jun 2005 15:22:00 +0000 I really should provide an RSS feed. Does anyone know of a good reference on the simplest form of RSS that I can use to auto-generate some RSS from the HTML of this page? I'm prepared to do a little python coding to make it happen. Give me an email if so.</p> Speaking of giving me an email, wouldn't it be great if I could have a proper blog that allowed comments? Well, I'd need a web host that allowed PHP or something, and that would cost money. If anyone knows of a good free hosting service (built on Free Software, preferably) let me know. I've registered on advogato.org</a> but although I have finally managed to work out how to post to a blog, it doesn't allow comments, so the only benefit it offers is an RSS feed, and the lack of control is not worth that to me yet. I may get frustrated with my home-grown HTML/FTP solution, but for the moment I'm quite enjoying it.</p> A real job Andy Balaam — Tue, 07 Jun 2005 15:12:00 +0000 Getting used to working is an interesting thing. I haven't been as tired as I thought I might be, probably partly because I got into a routine of getting up at 7am with my wife (she was working while I did my PhD). However, the weirdness of how much of a job is to do with social pressure is interesting. What I mean is that it takes a lot to get fired (in my job, anyway - I hope...) and you could probably get away with being pretty mediocre and keep your current salary. The things motivating you to do well and work hard are mainly desire for promotion (and within reason your salary will probably increase over time even if you're mediocre) and social pressure. The main way in my work you are motivated is that if you don't do what you were supposed to do one week, you have to survive a difficult moment on Monday when we have our status meeting.</p> Anyway, none of this is relevant to me because I am really motivated by how fun it is to be working with other people on something that I am reasonably good at, and also because I never have to read or write academic papers again.</p> Except, of course that I have my viva later this month, and I will have corrections out of that. But I'm trying not to think about it.</p> What is going on with FreeGuide? Andy Balaam — Sat, 04 Jun 2005 23:12:00 +0000 I thought I'd write a little bit about FreeGuide</a>, since I'm planning to link this page off the FreeGuide page.</p> I'm the lead developer on FreeGuide, and having just started a new job (which so far is excellent) my free time is seriously limited, especially as my commute is 1.5 hours at the moment. I expect as I settle in I'll find more time, and I also hope my commute may reduce in a few months. In the meantime I've had to fall back into maintenance mode, but fortunately there are several contributors stepping up to add long-requested features. The most prolific contributor has been Alex, who has been working on a number of excellent new features in our development branch. That branch started when Risto refactored the code into an MVC style and optimised the display to speed up massively operation with more than about 10 channels.</p> That branch has been around _far_ too long, as it's a maintenance headache to look after 2 quite different branches, and it causes confusion when people send in patches against old code.</p> I really want to make the development branch reasonably bug-free and then declare it stable and retire the 0.8.x branch once and for all. I hope we can do that pretty soon.</p> Meanwhile, Alex is adding more features, the most notable of which is the plugins system. That will eventually allow you to download and install new listings grabbers without upgrading FreeGuide itself, and plug in loads of other things like exporters, recording schedulers, and loads more. Also excitingly soon to go in is the recording scheduler system developed by Reuel and integrated by Alex.</p> I want to make 0.10.x stable next release, then get those features in (probably under a 0.11.x name), stabilise them, go through the TODO and make a _short_ list of things to do before 1.0, do them and NOTHING else, and then finally release 1.0. It would be great if the plugin system turned out to be flexible enough to handle all the new thigns people suggest, but if not I plan to keep the 1.0.x branch stable long term, and allow further changes in a 1.1 branch.</p> The features I think are needed for 1.0 are:</p> Recording (through a TV card) while FG is running</li> Scheduling recording</li> Coloured favourites</li> A manual and help</li> </ul> Hopefully not a lot else...</p> Google Andy Balaam — Fri, 03 Jun 2005 23:07:00 +0000 Can't decide what to think about Google. I was very suspicious about gmail, and this whole portal business</a> is very boring, and _so_ corporate, but you really can't argue with sponsoring open source contributions</a>, can you? If doing this kind of thing starts being seen as the way to impress the developer community (and therefore recruit the geekiest geeks) then everyone wins, and world domination is one step closer.</p> Beginning blog Andy Balaam — Thu, 02 Jun 2005 15:23:00 +0000 I've decided to start writing a blog, not least to convince people that although my work on open source projects (especially FreeGuide</a>) has slowed down, it has not slipped out of my mind. Quite the opposite: I'm finding my lack of time very frustrating, especially when the train journey in to work every day gives me time to think of lots of excellent ideas, none of which I am going to be able to put into practice. I'll log them here, and maybe someone else will do them. If not, I'll do them when I retire.</p> I'm hoping to keep this blog as technical as possible, with only occasional daliances into personal matters, since it does my head in to be reading something like Planet Gnome</a> and have to read about people's cats being sick etc.</p>

Andy Balaam's Blog

A full (slow) solution to the Billion Row Challenge

The billion row challenge: do we have a bug?

Writing a bit-wise iterator (with Jez!) for our gunzip implementation

Keep the faith

Implementing gunzip from scratch in Rust

Jez returns for programming language chat!

Announcing Changelog Builder - a towncrier clone for building changelogs

Using anyhow to handle errors in my Rust changelog builder

A simpler blog comment system

Andy 2026-04-28</span> </h3> Thanks Benedikt - sounds good but I think I'm happy with the very-manual approach for now!</p> </div>

Building a Rust commandline tool: a towncrier clone

The billion row challenge: splitting lines using SIMD in Rust

Rust: reading very large files for the billion row challenge

Parsing whole lines of the Billion Row Challenge

Improving our billion-row parsers and benchmarks

Making our own String type in Rust

Writing a Snake game in Rust/WASM

Talking to localhost from inside an Android emulator

Rust: compiling to WASM to make a browser-based game using canvas

Set up</h2> Before we start, we need to install:</p>

Follow along</h2> You can watch me create this setup live here:</p> </iframe>

Billion row challenge: parsing temperatures fast(ish)

Rust: building arbitrary precision integers (badly)

Rust: how to return a closure from a function, and what are closures anyway?

FizzBuzz 2: FizzBuzz Harder

Why I don't use AI

Converting a string to an int in Rust (for no reason)

Why I don't use AI (expanded)

Resizing and moving an encrypted LVM volume

Implementing FizzBuzz in Rust (for no reason)

Reversing a string in Rust (for no reason)

Mini-rust in Rust 050: Parsing function definitions

at0m 2026-05-18</span> </h3> Looking forward to more episodes!</p> </div> Andy Balaam 2026-05-20</span> </h3> Thanks at0m! I'm not sure many other people were keen, so I'm not planning any more episodes yet, but you never know ... maybe I'll get back into this one day...</p> </div>

Andy Balaam 2026-05-20</span> </h3> Thanks at0m! I'm not sure many other people were keen, so I'm not planning any more episodes yet, but you never know ... maybe I'll get back into this one day...</p> </div>

Why do I have 2 passwords? How to talk about encryption in Matrix

Invisible Crypto: can Matrix be both secure and easy to use?

Mini-rust in Rust 049: Lexing an empty function

Mini-rust in Rust 048: Handling variable scope

Mini-rust in Rust 047: Updating types as we get more information

Jez and Andy write a Parser

Jez learns Rust by writing a Lexer

Mini-rust in Rust 046: Type-checking mutations

Mini-rust in Rust 045: Combining two types when we add things

Why I don't use AI

Mini-rust in Rust 044: Finding the type of a symbol

Mini-rust in Rust 043: Expressions on the left of operators

Mini-rust in Rust 042: An unexpected token means stop this expression

jezhiggins 2025-11-28</span> </h3> That was a journey!</p> </div>

Mini-rust in Rust 041: While conditions that are not operations

Mini-rust in Rust 040: While conditions should be bool

Mini-rust in Rust 039: Spans should be ranges

Mini-rust in Rust 038: Avoiding a String allocation for each token

Mini-rust in Rust 037: Complex expressions in mutation statements

Mini-rust in Rust 036: Evaluating code blocks

Mini-rust in Rust 035: Tidying up the evaluator

Mini-rust in Rust 034: Our first program, actually working

Mini-rust in Rust 033: Evaluating operators like plusequals and timesequals

Mini-rust in Rust 032: Supporting a Unit type and type checking mutations

Mini-rust in Rust 031: Getting "while" through the type checker

Mini-rust in Rust 030: Expressions after while loops

Mini-rust in Rust 029: Parsing plus-equals

Mini-rust in Rust 028: Lexing plus-equals

jezhiggins 2025-08-15</span> </h3> Not cross, just disappointed</p> </div> andyb 2025-08-15</span> </h3> Not intimidated, just devastated.</p> </div>

andyb 2025-08-15</span> </h3> Not intimidated, just devastated.</p> </div>

Standing again for the Matrix Governing Board

Purpose of the board</h2> The board operates on an advisory basis, but is responsible for approving budgets, projects and partnerships that are significant to the future of the foundation.</p> I see the Governing Board as having two major purposes:</p>

Testing nonexistence

Mini-rust in Rust 027: Parsing blocks of code

jezhiggins 2025-08-15</span> </h3> Blimey Andy, that was journey</p> </div> andyb 2025-08-15</span> </h3> Yeah, am I ever going to stop?</p> </div>

andyb 2025-08-15</span> </h3> Yeah, am I ever going to stop?</p> </div>

Mini-rust in Rust 026: Parsing an empty while loop

Mini-rust in Rust 025: Parsing let mut

Mini-rust in Rust 024: Lexing the <= sign

Mini-rust in Rust 023: Refactor to share structure of the syntax tree

Rust tracing basic setup

Comments</h2> andyb 2026-02-09</span> </h3> Great, I hope it’s useful!</p> </div> ghenry 2026-02-08</span> </h3> Stealing this for use in OneTalker 👌 Thanks!</p> </div>

andyb 2026-02-09</span> </h3> Great, I hope it’s useful!</p> </div> ghenry 2026-02-08</span> </h3> Stealing this for use in OneTalker 👌 Thanks!</p> </div>

ghenry 2026-02-08</span> </h3> Stealing this for use in OneTalker 👌 Thanks!</p> </div>

Mini-rust in Rust 022: Stop lying about types in errors

Andy 2026-04-28</span> </h3>
Thanks Benedikt - sounds good but I think I'm happy with the very-manual approach for now!</p> </div>

Follow along</h2>
You can watch me create this setup live here:</p> </iframe>
and you can find the full source code at: codeberg.org/andybalaam/wasm-game</a>.</p>
Follow me on mastodon: @andybalaam@mastodon.social</a></p>
The first thing we need to do is create a Rust+WASM project.</p>

at0m 2026-05-18</span> </h3>
Looking forward to more episodes!</p> </div>

Andy Balaam 2026-05-20</span> </h3>
Thanks at0m! I'm not sure many other people were keen, so I'm not planning any more episodes yet, but you never know ... maybe I'll get back into this one day...</p> </div>

Andy Balaam 2026-05-20</span> </h3>
Thanks at0m! I'm not sure many other people were keen, so I'm not planning any more episodes yet, but you never know ... maybe I'll get back into this one day...</p> </div>

jezhiggins 2025-11-28</span> </h3>
That was a journey!</p> </div>

jezhiggins 2025-08-15</span> </h3>
Not cross, just disappointed</p> </div>

andyb 2025-08-15</span> </h3>
Not intimidated, just devastated.</p> </div>

andyb 2025-08-15</span> </h3>
Not intimidated, just devastated.</p> </div>

jezhiggins 2025-08-15</span> </h3>
Blimey Andy, that was journey</p> </div>

andyb 2025-08-15</span> </h3>
Yeah, am I ever going to stop?</p> </div>

andyb 2025-08-15</span> </h3>
Yeah, am I ever going to stop?</p> </div>

`Comments</h2> andyb 2026-02-09</span> </h3> Great, I hope it’s useful!</p> </div> ghenry 2026-02-08</span> </h3> Stealing this for use in OneTalker 👌 Thanks!</p> </div>`

andyb 2026-02-09</span> </h3>
Great, I hope it’s useful!</p> </div>

ghenry 2026-02-08</span> </h3>
Stealing this for use in OneTalker 👌 Thanks!</p> </div>

ghenry 2026-02-08</span> </h3>
Stealing this for use in OneTalker 👌 Thanks!</p> </div>

jezhiggins 2025-01-24</span> </h3>
I have been waiting for you to peek!</p> </div>

andyb 2025-01-24</span> </h3>
Ha, well technically I am not going to peek :–)</p> </div>

andyb 2025-01-24</span> </h3>
Ha, well technically I am not going to peek :–)</p> </div>