Still Not Perfect

It has been a long long time since I’ve written anything here. I think I’m going to finally get back to it. Plenty has happened since I last wrote. Work has been consistently crazy - who would expect anything less from a company as fast paced as it is ridden with standard giant corporate silliness. Ian has grown into a full blown toddler - running, jumping, astounding all of us with his grasp of language, and generally being a lot of fun. We’ve had some big ups and downs in our family life in the meantime - the biggest up being the birth of Chava Leigh, a beautiful daughter, on March 31, 2007.

I’m still very much into all things high-tech, especially open-source. But, those things are no longer as high on my list of priorities to write about. I hardly have any time anymore to devote to producing any neat tools. Instead I think you’ll see posts from now on on family, AP, politics, and my particular hot-button issues. In fact, the things that have finally pushed me over the edge to post again are in no particular order:

• Iraq war - Why it is failing, was that failure inevitable, and what parallels I see with another famous military engagement.
• Al Qaeda - Are their motivations and intentions really as the Bush administration spins them (they hate our freedom and our women’s rights, the terrorists will follow us home, etc.)?
• Our rights are being destroyed right under our noses at home - How did it get this far without anyone who objected noticing? And what hope do we have to get them back?
• Our kids aren’t turning into spoiled brats because of AP - in fact, we’re continually surprised at how much respect we earn from them by wholly respecting them, and relating to and guiding them accordingly. And other parents are noticing too, at least they keep telling us so.
• Immigration - I think the approach this country is most likely to end up with is all wrong, but not for the reasons you think. This one might be my most radical opinion, so watch out.

So, I’m back, with no guarantee of regularity. But with some things to say, and some strong opinions. Now I just have to figure out which one to mention first.

A couple people commented on my project idea for instant messaging over HTTP. After looking at the open source IM market in more depth I’ve found that the pieces I was looking for are actually all there, mostly.

• Jabberd 2 - an open source XMMP server implementation. There are also some great extensions (called transports) like PyAIM-t and PyMSNt that allow your server to interface with other IM providers. Any jabber server would probably do, but I liked the looks of Jabberd 2 best. Adopting an already existing IM server will save a bunch of time trying to design and write a backend for my project.
• Punjab - an open source HTTP jabber client interface. Kind of like a session-intelligent proxy between the statelessness of the HTTP and the statefullness of XMMP. Version 0.6 is is the latest release, but it has a couple problems. I’ve had more success going with the latest code from CVS. Even that has some issues though, mostly that it does not implement JEP-0124 completely, and binding connections simply get turned into polling connections with a very fast (read instantaneous) polling rate. This problem is where I’m currently focusing my efforts.
• JWChat - an open source browser based chat client. No ActiveX or Java required - 100% Javascript, which is native in all major browsers. Along with Apache on the server side to proxy requests to Punjab it works pretty well. Not really a fan of the appearance currently, but that is easy to change.

So, even though it appears nobody had really ventured into exactly the space I was thinking about, there are enough tools out there that I can pretty much construct my idea out of pre-existing code. Progress is being made, but it isn’t ready for public yet. Punjab seems to be the most important piece of the puzzle to me and it still has a couple show stopping bugs (but I think I know how to fix them - except for the scalability problems). Once I get it patched I’ll release the patches and a guide to build your own HTTP to Jabber to AIM/MSN/Y!/etc setup.

The past couple of months I’ve been pondering an idea. It isn’t a new one, or anywhere close to groundbreaking, but I don’t think anyone has accomplished anything quite like it before. I want to write an IM/chat system that is as responsive and real-time as native IM/chat programs, and that operates solely over HTTP, with vanilla browsers - no Java applets, no ActiveX plug-ins, just Javascript that all modern browsers support. The advantage of such a system is that it could traverse firewalls and proxies traditionally used to block users from using IM/chat, and that users would never be prompted to install applets or plug-ins that many users distrust (for good reason too).

So, why hasn’t it been done before? When I first had the idea I thought surely there would be at least 10 similar projects already. I found several attempts, but none seemed to fit the bill - some communicated entirely over HTTP but required Java applets, some didn’t use any applets or plug-ins on the client but weren’t responsive or real-time (updates were polled for ever 2 secs or so). Looking at the current field of attempts there seemed to be a couple of challenges that I would have to find a way around.

• HTTP/1.1 supports keep-alive connections, but they don’t work well for 2-way communication. The client still must initiate every exchange, so the server can’t send messages to the client on demand.
• A client repeatedly polling a server for messages (even when there are none for it) puts a lot of unnecessary load on the server, and generally results in a poor user experience. The programs I looked at that used this model often felt very laggy and sluggish - they certainly didn’t feel like native applications.

Read the rest of this entry »

We had a couple of days of downtime (really just 1) because of the move. Like an idiot I didn’t take enough care when shutting Ridcully down and some MySQL tables got corrupted. Hex didn’t wake up right away after the move either. It wouldn’t even power on, and normally it is the DNS server, so it was kind of hard to research how to fix the MySQL problems I was having on Ridcully. Finally after just letting it sit for a little while Hex woke up, and then a quick ‘repair table wp_comments’ later on Ridcully we’re back up and running error free. Just looks like we lost a few comments (probably the last 25 or so).

Today I watched Garden State again. It is a wonderful film. One I can really relate to. I love how it doesn’t depend on formulaic explosions, sex, violence, etc. Instead it shows real life, with all it’s confusion, pain/release and quirky characters. Zach Braff writes/directs/stars along with Natalie Portman and a small but very effective supporting cast.

What I like so much about this film is that it doesn’t try to cram real life into the standard 3 act Hollywood template. Instead it adapts to the characters and their stories. That helps it to express the deep emotions of loss and release, and then to show the beginnings of Largeman’s re-finding himself. It shows intense emotion without being sappy. It resonates with the generations who have been drugged numb - pumped full of Ritalin, anti-depressants, etc. It helps them to realize that maybe there wasn’t anything actually wrong with them - maybe the problems their parents were trying to solve by drugging them into submission or sending them away were just regular growing up phases.

On top of that it is artfully composed. It’s visuals aren’t groundbreaking, but they fit together, and help tell the human side of the story. They show life in it’s reality, not through rose colored glasses. At the same time they help the tone not to be too dark.

The characters, though quirky and sometimes shallow, are quite lovable. They give us opportunities to laugh and wear a knowing smile, and a couple scenes later have us connecting with them at such a level that even I teared up (which is rare). The acting makes the struggles the story is about believable. We see the conflict that has been brewing for years between father and son, how the trauma of death brings it into the open again, and how absolute resolution isn’t the only way to deal with the situation (I love how it doesn’t cop out and give in to the obvious “everything’s alright” solution here). We see a couple falling in love. We see drugged up high-school buddies who haven’t moved on with their lives yet, but we also see them realize they’ve got more to look forward to.

I think it is one of those rare films that can connect with a whole generation. If you were born in the seventies to eighties this one is almost certainly for you. I think for many of us in those generations this film is as much a homecoming for us as it is for the main character - even if we didn’t grow up in New Jersey. No, this one isn’t probably going to win major acclaim, and no it isn’t groundbreaking in style or technique. But it is solid, dependable, and real. It doesn’t pull it’s punches, and it throws in just the right mix of Braff’s odd humor.

Sorry about our recent outage. One of the Memorial Day weekend lightning storms zapped our firewall’s motherboard. It took me a little while to figure out that the problem was the motherboard instead of the power supply. However, once I figured it out the part was suprisingly cheap ($9.99 for a motherboard with onboard UltraSCSI and onboard server class NIC, who would have guessed?). I guess the hardware is so old that the few vendors who still have parts left are trying to liquidate them. My UPS should have protected the whole system from the lightning strike, but for some reason it didn’t work as well as it should have. It kept the other two servers plugged into it up and running just fine, but the firewall got zapped. Maybe it is time to look for a better UPS, or maybe the surge came in through the cable modem (???). Or maybe I should count it as an acceptable risk for hosting at home instead of in a real data center.

Anyway, we’re back on line now thanks to cheap parts.

Groklaw reports that the DC Circuit of the US Court of Appeals has ruled that the FCC does not have the authority to impose the Broadcast Flag requirement on digital TV consumers and equipment manufacturers. The Court decided that the FCC had overstepped it’s bounds and that it did not have any authority to make regulations regarding the Broadcast Flag until Congress grants them that power.

The FCC Broadcast Flag was an attempt by the MPAA and other content distributors to limit the legal uses of digital TV broadcasts to initial viewing. This means that recording for later viewing or otherwise archiving the broadcasts would be illegal (at least in their original digital quality), and it would therefore be illegal for manufacturers of consumer electronics and computer equipment to ship devices that allowed recording of content “protected by” the Broadcast Flag. Naturally the Broadcast Flag would really put a damper on the development of innovative technologies like Tivo that actually enhance and improve our lives instead of complicating them.

Of course I’m sure the MPAA and other proponents of the Broadcast Flag will not give up just yet. I expect that they will make every effort to get the ruling overturned by higher courts. If that is not successful I expect that they will return to lobbying Congress for legislation that grants the FCC the power to control what consumers do with content after it has been broadcast. I personally believe that the FCC’s power should only apply to the communication of information, which includes transmission and any interference with transmission. What a consumer does with the signal and the content after their device (which is their personal property) has received the transmission should be up to them (as long as their uses of the content respect the copyright). I do not believe that the FCC’s power should apply to devices that can be used to receive broadcasts (specific purpose consumer electronics or general purpose computer devices), other than regulating their interference with transmission of those broadcasts and other’s receipt of those broadcasts.

If you value your freedom to do what you want with your digital TV signal for your personal use after you have received it into your home please consider writing to your Senator and House Representative to make sure they know it. Because if the Broadcast Flag decision is upheld in higher courts, you can bet the MPAA lobbyists will start in on Congress. You can also contribute to the fight for your freedom in the digital world by supporting the EFF.

The last couple of weeks have been really tough for me and my family. It started with me forgetting to reorder by medicine until I only had a day or two’s worth left. Now my refills usually get here within a couple days of me ordering them, but of course this time it had to take almost two weeks. Even though it’s a relatively mild medicine (lexapro), the withdrawal can wreak havoc on you.

To add to that my sister-in-law came into town just when things were starting to get hard. That made it even harder because she was staying in my “study” which is where I like to wind down and level myself out, stuff that is even more important to try to do when you are suddenly coming off of your medicine.

Then, a couple of days after she arrived my wife started to get really sick (high fever, etc). Turns out she had a gallbladder infection and a couple of stones. Time for emergency surgery. A couple of days in the hospital to recover and she is feeling much better. I’m thankful my sister-in-law was around to help some during that time. Once we get my wife home from the hospital we’ve got the house to ourselves again, and my medicine had arrived so I could start ramping up on it again.

And then things get rattled up at work. I had known a management change was in the works, and even that it would be happening soon. It is just hard knowing that I am leaving what will probably be the best manager I will ever work for. I’m getting used to the new team now, but it just isn’t the same (even though 4 of my teammates are the same).

Of course I haven’t really had the energy to even record everything that has been going on until now, almost a month after most of it. Oh well, that’s how life is sometimes I guess.

As pretty much everyone who reads anyone‘s blog knows by now, Yahoo is buying Flickr. I hope this doesn’t change Flickr’s wonderful open philosophy. Their API should be a model for all sites, no matter what service they provide. It is well documented (but not overly verbose), and extremely flexible. Hopefully Yahoo’s $ can help Flickr deal with some of the capacity and stability challenges that have come with their incredible growth. Flickr may also be able to contribute a lot on the usability front to Yahoo’s different sites, some of which haven’t improved in this area for 3-4 years. We’ll just have to see how it all works out.

Our son Ian Randall was born 2/17 at 1:39AM. He is 8 lbs 11oz and 20.5″ long. He’s got a full head of dark hair already. Unfortunately he’s been having a lot of trouble with his oxygen saturation. The official diagnosis is pneumonia, but the doctors aren’t quite sure that’s exactly what’s wrong. He’s also had a bit of a heart murmur, but that seems to be clearing up. He’s still in the NICU, and will be at least until next Sunday. Obviously I’ve let things get a bit stale on my site during all the craziness, and don’t expect me to get too much up-to-date for a bit. We’ll be spending a lot of time at the hospital visiting Ian. Please keep us all in your prayers.

Eventually I will catch up with the WP world too and update my Flickr gallery plugin to be WP 1.5 compatible. But it might still be a while.

Update: I’ve posted some pictures of Ian here and here.