Cooking With Technology

LostAFollower Alpha Release

Managing relationships in social networks is difficult, just like managing real-world relationships is difficult. If someone unfriends you on Facebook, or stops following you on Twitter, is it because of something you posted? If a friend in real life stops talking to you, is it because of something you said? It can be difficult to tell, and the best course of action is probably to ask your friends what is going on, whether on social networking sites or in the real world.

But there is a difference here: the breaking-point status online is binary. Either someone is friends with you or they aren’t, they were following you but now they’ve stopped. And while sites like Twitter and Facebook are happy to tell you when you have a new friend request or a new follower, they sweep under the rug when someone reverses that action – all you notice is you have one less follower, or one less friend. Honestly, it bugs me – and I’m not the only one. Others have written services to monitor a user’s followers on Twitter, but they all fail in some function or another. Qwitter is an example of this – their service is so overloaded that your user only gets hit every couple *months.* Pretty terrible.

Which is why I wrote Lost A Follower. Lost A Follower addresses the problems I’ve described thus far – it is a service that monitors your Twitter profile. When you lose a follower, Lost A Follower gathers your last post to Twitter and the username of the individual and sends them both to you. Now you get a notification sent directly to your email that tells you who stopped following and, importantly, *why.* It is in its initial release right now, so I expect plenty of bugs on the server side – but I’m excited to get it off the ground. I have a lot planned for future versions and extensions of the Lost A Follower platform, but all that will come in due time :)

So if you are a stat-monger like me, or you just wish Twitter would help out a bit with managing your social network, give Lost A Follower a shot. It is free, developed by yours truly, and is currently in need of test subjects :)

mod_python Project Configuration Files

In my last post I brought up how to get around mod_python’s default inability to handle relative paths for imported modules. The next night after writing that, I faced a very similar issue with configuration files.

It is a sin to duplicate data in a project (except when caching), so most sane desktop applications will have a central store that contains the configuration parameters for the application (usually packaged as a resource with the application, or in the registry for Windows users). Mod_python can’t do this because, as was discussed in my last post, it views the current working directory of whatever script it is executing as the root directory of the filesystem (”/”). So where do you put the configuration file for your mod_python based project if you can’t refer to it relatively, don’t want to hardcode an absolute path, and want to distribute the configuration with the project?

The solution hit me like a bucket of cold water. Give up on the property or ini format file, and make your configuration a python dict object that you can directly import:

# config.py
#
# This is the project's configuration. Import it when a class needs to access
# project configuration variables.
#

_config = {
    # The name of my app so pathHack knows what string to look for
    'rootname': 'sampleWebApp',
    # Database options
    'db_host': 'localhost',
    'db_dbname': 'sampleWebApp',
    'db_user': 'sampleWebAppUser',
    'db_password': 's0m3p455w0rd',
    # Debug logfile path
    'debug_logfile': '/tmp/lostafollower.log'
    };

This file needs to live in your project’s root directory as far as mod_python is concerned. For more information, read my last post. If you’ve already read it, then you’ll understand when I say that config.py lives in the same folder as pathHack.py.

Once you can import it from your main source files, use it like this:

<%
# sample usage of config
from config import _config as _c
req.write(_c['rootname'])
%>

Tada! Extraordinarily simple configuration file that is easily accessible and can be distributed with your mod_python application. I’m pretty happy about it :)

Relative Import Paths Using Mod_Python

A long while back (a year already? Jeeze!) David Brenner and I wrote a web application without using any frameworks, only mod_python and python server pages (PSP). We ran into a few problems along the way, one of which being that the import path mod_python used had to be composed of absolute paths. It isn’t a huge problem to just hard code a path to your project’s lib directory, but it is hardly portable and adds to the fragility of the system.

I’m now doing another project (more on that later), again using mod_python and psp. This time around however, I have a hack that gets around the absolute path problem. The solution looks like this:

# This file has only one function: to add directories to the path
# so we can import other python files/modules.

import sys

# Default directories to add to the path
_dirs = ["/src", "/lib"]
_applicationRoot = "sampleWebApp"

# Will go through the list and add the directories given relative to the
# lostafollower base path. Allows us to do paths relative to our installation
# instead of hard-coding absolute paths.
def addLibs( dirs ):

  # If [] was provided, just use the default.
  if not dirs:
    dirs = _dirs

  base = ""
  for entry in sys.path:
   if entry.find(_applicationRoot) != -1:
     base = entry
     break

  for dir in dirs:
    try:
      # Test if the path contains the path we want
      sys.path.index(base+dir)
    except:
      # The path doesn't already contain our path
      sys.path = sys.path+[base+dir]

The reason we normally can’t use relative paths when doing imports is because the mod_python handler considers the root (”/”) of the filesystem to be the current working directory of the application, regardless of where the application is actually running from. However, Apache will put whatever alias it has for your application’s directory [1] in the PYTHONPATH for mod_python, which is what allows us to do this hack. By finding the path that Apache has in the PYTHONPATH, we can add whatever paths we want relative to that. No .htaccess PythonPath directives, no hard coding of paths to your project’s module locations. Just:

1. Put the above code in a file (I call it pathHack.py) in your application’s root directory (root as far as Apache is concerned) [1]
2. import pathHack (or whatever you called it) in whichever file needs your other modules
3. Call pathHack.addLibs([]) to add the defaults (/path/to/sampleWebApp/lib, etc), or pathHack.addLibs(”/lib/3rdParty”) to add /path/to/sampleWebApp/lib/3rdParty to your path
4. import any python module in those newly added directories!

That is really it. Hard coded paths begone!

[1]

When I say “the alias it [Apache] has for your application’s directory,” what I mean is that even though, for example, my application runs out of /home/robertp/src/sampleApp/src, mod_python sees the root as being /home/robertp/src/sampleApp because of some unknown magic. It doesn’t really matter what the path is, in all honesty – when specifying a value for _applicationRoot in the solution code above, you are just looking for a string unique to the path to your application (just be sure to make all supplied paths relative from that base). Try just your project’s deployment folder for starters. If you are still having trouble finding it, create a quick psp file in the same folder as your other psp files, and do:

import sys
from mod_python import apache
req.write(str(sys.path))

That will just print out all the path elements to your browser window when visited. Pick out your application’s name or path from there, and set _applicationRoot to that. It isn’t completely ideal, but it is better than hard-coding ;)

Part 2: Connections and Configuration

In part one of this series, I covered the hardware and basic operating system installation for a home theater PC running Mythbuntu. In other words, part one covered the easy stuff. Running the audio cables between the PC and the TV is actually a more complicated affair than I had imagined, yielding three major issues that had to be overcome before ending up with an acceptable configuration.

The blue lights are very subtle, but neat in the dark

1. Audio

I should clarify that connecting the cables isn’t the problem, it is the technical limitations and configuration issues accompanying their connection that poses challenges. My audio connection is straight stereo-out to stereo-in via a 10 foot 3.5mm interconnect. Upon plugging it in and turning on the TV however, no audio was present. The reason behind the problem ended up being with how ALSA was loading modules, and was easily fixed with a configuration change. If you copy my setup and are using a Creative Audigy 2 sound card with an Ubuntu variant, try the following:

1. Run ‘alsamixer’ on the command line and verify your Master and PCM channels are turned up.
2. Verify you have your cable plugged in to the correct slot on your sound card.
3. My fix: another sound device (my motherboard’s onboard sound) was stealing ALSA’s focus. Adding the following to /etc/modprobe.d/blacklist and rebooting fixed my problem: blacklist snd_intel8x0

2. Video – Analog Connection via DSUB15

The Viera has an analog mini-DSUB 15 port for hooking up a PC directly to the TV. I had never had an issue with signal quality before on smaller monitors, so I didn’t forsee any problems with hooking a ten foot VGA cable from the computer to the TV. There are a multitude of issues with this setup, however.

First, read the specifications for the Viera carefully (in the manual, I couldn’t find the information anywhere else), and Panasonic states the TV is capable of producing a maximum resolution of 1280×1024@60hz. This is a slight problem, because in order to display content at 1080p, 1080 lines of vertical resolution are needed. Furthermore, with a 10′ run of analog signal 1024×768 is clean, but pushing 1280×1024 causes a significant amount of signal and color bleeding.

The pits of this is that if you stick with an analog connection, you are really stuck with these problems. You can clean up the signal by running a shorter cable from the HTPC and the TV, but that limits the placement of the machine. The resolution limitation is the real kicker, as spending all of that money on a HDTV capable of producing 1080p becomes a complete waste if you can only fully display 720p content. I refused to believe that the TV couldn’t produce its native resolution of 1920×1080 from a computer source, so the next fix comes from addressing that issue.

3. Video – Digital Connection via DVI -> HDMI (and More Audio Solutions)

The TV is capable of producing 1080p from a source connected through component or HDMI cables, displaying at 1920×1080. My solution to the problems found in section two of this post was to then find a way to get my HTPC’s video output to an HDMI cable. HDMI is really only DVI with added support for sound, so another $15 purchase of a DVI -> HDMI cable and a week of waiting was all I needed to make the connection. The benefit was immediate – being digital, there is no signal bleeding or interference. The drawbacks presented themselves just as quickly, however. I found that my analog audio connection had cut out, I could no longer produce sound on the TV. Additionally, the HTPC was displaying at 1920×1080, but the TV was only displaying a portion of it, with my mouse going off-screen to access common menus.

The video display was the most immediately annoying part. Reading many online forums suggested that it was a problem with overscanning, and there were even guides on how to access the service menus of the Viera to grant a 1:1 pixel match (effectively disabling overscanning). I haven’t found this documented anywhere else, so I want to say it clearly: ADJUSTING YOUR VIERA’S OVERSCAN SETTINGS IS UNNECESSARY (is it still bad design to use marquee and blink tags? I’m tempted to use them here). The easy and smart fix is documented in the service manual (not the user manual), and is as easy as the following:

1. Plug in your DVI -> HDMI cable, ensure your source HTPC is on and change the TV’s video input so the HTPC’s screen is visible.
2. On your remote, press “Menu,” then go to “Picture”
3. Scroll down to the second screen, and go to “Advanced Picture”
4. Go to the last option, “HD size,” and change it from “1″ to “2″

That fixed it for me. Restarting Xorg and a change of the resolution to 1920×1280@55hz had me running in full high-def glory. But without sound.

The sound issue was a nuisance. HDMI normally carries a digital audio signal with the digital video feed, so the Viera is nice enough to automatically ignore all analog audio input sources when your display comes from an HDMI source. Unfortunately, getting audio spit out over DVI was more difficult than I cared to pursue (with a lot of forums claiming the task impossible), and buying a DVI + analog audio -> HDMI converter is in excess of 200 euros. Again, the simpler and smarter solution is again in the Viera service manual (not the user manual), and can again be done by following these steps:

1. Plug in your HTPC and get it set up again so the screen is visible.
2. On your remote, press “Menu,” then go to “Audio”
3. Select “Advanced audio”
4. Go to “HDMI 1 in” (or to whichever port your HTPC is plugged in to)
5. Scroll over until the port your audio source is in is selected (I selected “PC”)

You should now have overridden the default of using the digital signal from your HDMI cable to now use the analog input.

For me, these three steps resolved the major issues (and relieved the major headaches) I encountered while setting up my HTPC. I sincerely hope that these steps will help people avoid making the mistakes I did, and save them the many days spent polling forums looking for answers to these issues. In part three of this series of posts, I will address setting up the system to make retrieving media especially easy, and some other configuration tweaks that make my system an easy-to-use content retrieving monster. Subscribing to my RSS feed will ensure you don’t miss it.

I’m off to watch The Dark Knight in 1080p – until next time!

Part 1: Mythbuntu HTPC and a Panasonic Viera

I purchased a 46″ Panasonic Viera 1080p plasma TV at the beginning of January, and the resulting fun I’ve been having with it is largely responsible for my lack of recent blog updates. I, being the cheap geek I am, have no desire to either pay for an HD package from my cable company, nor do I want to go out and buy an HD source and then buy or rent movies. The solution I came up with to view my existing media and new HD content thus revolves around a relatively cheap solution – a home theatre PC (HTPC). There were significant challenges in getting everything set up how I wanted it, and to prevent others from having to repeat my failures, I’m going to post a multi-part guide to repeating my final configuration – which looks a little like this. Part 1 after the jump!

Yes, that is 30 Rock in the background :)

Continue reading ‘Part 1: Mythbuntu HTPC and a Panasonic Viera’

Unsubstantiated Hype: Python and Web Development

This past academic quarter I elected to do an independent study. A lot of hype has been surrounding web development over the past few years, and the python scripting language for even longer than that. Figuring it was time to immerse myself in both subjects and the technologies related to them, the independent study turned into a full four credit project aimed toward creating a web application for administering quizzes. Because I only had ten weeks to learn everything involved with developing a web application on top of the language, I coerced my good friend David Brenner into doing it with me. Development is done, and I now feel like I can talk intelligently about my experiences.

The project was developed using straight python, utilizing the mod python Apache module to serve Python Server Page (psp) files. Python isn’t my favorite language to code with, it is lacking some things that I’ve come to expect from a scripting language, and some things are just silly. Examples of lacking features include the ability to get an iterator for a collection. Iterators that grant references both to the next and previous element of a collection grant superior control and the ability to keep state while iterating. Python only allowing you to move forward through a collection with its “for x in collection:” construct really hurts while parsing large sequences of text, and having to revert to array notation and keep an index is much less human readable than just manipulating an iterator.

Annoyances with the language include python being a scripting language that doesn’t default to printing the string value of non-string objects, and using procedural style method calls intermixed with object-oriented style. For example, to print an object (say an array), you can’t just do print(“Array contents: “ + arrayName). You have to manually return the string value by using print(“Array contents:” + str(arrayName)) in order to avoid an error that print() was expecting an object of type string. I know python’s mantra is “explicit over implicit,” but languages are tools that you learn how to use. As I learn more, I expect to gain proficiency with the tool in order to save me time in the future. Python is easy to learn and easy to use, but I have yet to find any tricks with the language that save me time or make my job easier than it would be with another language. One last thing that I simply do not understand is the necessity to include a reference to “self” in method and constructor definitions. I strongly prefer the method signature to match the required call, python’s decision to branch from that convention is one that I can’t imagine will have a justification strong enough to alter my opinion of the practice.

As it goes with web development, we also used javascript, a MySQL database backed with InnoDB table engines, LDAP for authentication, HTML, and CSS. Development was utterly boring. Being a computer science student, I enjoy working on new things that present challenges and reward creative solutions. The most thought intensive part of this project was the parser run on imported quizzes using a format we decided on – and that was developed in the fourth week. Everything else was the brainless transfer of requirements to code. Some of the people I go to school with are attracted this kind of development, and I think I know why: frontend web dev is easy. Writing the backend logic isn’t bad, but the interface code and associated handlers could be composed by a high school student with some spare time and only rudimentary knowledge. The things learned in computer science definitely aren’t applied in that domain.

Finally, to wrap up loose ends: object relational database design is neat and very handy, javascript is odd at first but easy to pick up, dealing with web requests and forms is tedious, debugging web applications is a *bitch,* and Debian Linux with Apache, Subversion, screen, and vim makes for a great development and hosting environment.

Web development has its place, but it isn’t for me. I’m capable with python, but I’ll stick with ruby. And if you introduce yourself to me, try to be more descriptive than “I’m a web developer” so my experiences don’t tarnish my opinion of you – because I bet you are more skilled at writing code than that title would suggest.

Adventures With s/re/g

While working on our independent study, Dave suggested we use foreign keys to establish relationships between our databases. We were using the MyISAM engine by default, which doesn’t support foreign keys. The fix to the problem was simple. Since our database doesn’t have much useful data in it, popping open the createdb.sql script we are using in Vim and issuing one quick command took care of everything:

:%s/);/) ENGINE=InnoDB;/g

Now all of our tables are using the InnoDB storage engine. I love Vim!

DocBrowser Launch (and Shameless Plug)

I’m one of those people that tends to get distracted when I’m procrastinating. Most often, this comes out when I’m doing a programming assignment and I go to open documentation in my browser. For those of you that weren’t aware, Firefox and Stumbleupon are a developer’s worst nightmare – so many interesting things, only a click away! It had to stop.

So I hacked up DocBrowser, a minimalist’s web browser created solely for the purpose of quickly accessing documentation and APIs. It is free of clutter and distractions, leaving only you and the information you seek on the screen. It is not extendable, and its feature set is limited. It is therefore exactly what I had in mind when I started coding it.

I’ve created a page for it here. It is free, and is only 1.5mb, so why not download it and give it a shot?

Windows only, requires .NET 2.0.

Geek Adventures: Project Management – Implementation

Last time, I talked about the requirements and initial division of an attendance database project for Computer Science House. If you recall, we left off with the project having been split into parts that will eventually be coupled into a complete system – these parts being an iButton interface, iButton back end, user interface, and client back end.

Mark, having been on the last team’s attempt to create this database, was tasked with getting our preliminary idea of what the database should look like up and running. Additionally, since he’ll be working directly with the database, I’ve assigned him the task of creating a PHP interface for interacting with the database. This interface will be called from the user interface as a means of centralizing the database calls for easier updates and maintenance.

Dave offered his help with whatever module needed developing, so he is primarily responsible with developing the user interface. He’ll be using PHP, JavaScript, and CSS as his development tools, utilizing Ajax-ian techniques for creating an interactive interface. This is going to be the most time consuming part of the project, so while Dave is responsible for the primary design and page layout, both Mark and I will be jumping on board to implement specific features when our parts are complete.

As for myself, being the only one with the iButton hardware for now, I am implementing the iButton interface and back end. The technologies I am using are Java, JavaScript, and LiveConnect. Java allows for the development of both the iButton communication application (the iButton interface) and the applet (iButton back end), the first of which was necessary to use the iButton Java SDK, and the second of which allows us to use LiveConnect to make the iButton IDs available to our web user interface invisibly.

The division of the project into smaller parts allows each member of our team to develop in parallel. With an emphasis placed on clean, self-explanatory interfaces existing at the points in which the modules come together, core development should proceed smoothly. As a bonus, each member is granted the freedom to develop in their own style at their own speed, with the only required communication occurring when it is time for members to link up their pieces.

For now, we code. See you when it comes time to integrate!

Geek Adventures: Project Management – Requirements

In my short career as a software developer, I’ve gained experience in multiple programming languages, development styles, and team roles. One role I have yet to play however is project manager. Transforming an idea into an end product using several stages of development with multiple people using many necessarily distinct technologies has always intrigued me (and here is my first public announcement-) an ideal job. Since there is no room for me to gain experience in this arena at my current job, I’ve decided to take on a small project and recruit two others to help me mature the concept and realization of a meeting attendance database for Computer Science House (CSH).

The basic idea of the attendance database falls in CSH’s need to record when members attend committe and general house meetings so we may conduct evaluations of our members. It is a simple idea until the requirements are analyzed. They are as listed:

1. The system must be able to store and retrieve which members were present at a specific meeting.
2. The database will be on a central server, but the client must be multi-platform and portable.
3. The system must be able to able to verify on some level that a member was physically present at a meeting.
4. Corrections to the database shall only be made by the executive board member presiding over the meeting.

And once these basic requirements are met, features will include:

1. The system must be accessible in the absence of an Internet connection.
2. The system will make the job of the executive board members easier by not requiring their involvement in recording a member’s presence at the meeting.
3. The system will include functionality for the evaluations director and chairman to generate reports of meeting attendance for all members to aid in quarterly evaluations.

Once equipped with the requirements and feature requests, I set out to find a couple of intelligent members that could aid in designing and development of the project. They are David Brenner (chairman, spring quarter ‘07) and Mark Schillaci (evaluations director, ‘07-’08). Dave learns as naturally and relentlessly as his heart beats and is capable of grasping abstract concepts easily. This makes him great for picking something up and applying it quickly. Mark is less experienced than Dave or I, but is also sharp. The reason I asked him to be a part of this project has two factors. First, he was on the last attempt to create the attendance database, and second, because he isn’t afraid to dream. This second factor paired with his lack of fear of being wrong yields great ideas that, when grounded and refined, become solid and actionable contributions.

With a team assembled, analysis of the requirements and how to move them into implementable items began. The first step was a separation of the system into components: the database and the client. Following this, an analysis of the components was made in order to fit the requirements. The database being hosted centrally is a given, so it really doesn’t need further discussion. The client requirements state that it must be portable and multi-platform, and obviously must be able to communicate with a database remotely. Additionally, it should be able to verify the physical presence of a member. This one is more interesting.

When developing for multiple platforms, the most efficient way to proceed is to create something that has a single interface for all platforms and a codebase that requires little or no modification to make available to other platforms. This results in creating a single user experience regardless of their operating system, and most importantly as a developer, an easy to maintain system. These objectives are most easily achieved with a web application, where any user with access to a graphical web browser that supports javascript will be able to use our application.

 The issue of verification on the client required more thought, and came to me while I was working on another project. The first method proposed by the developers of a previous attempt at creating the attendance database was to have the executive board member or house secretary log in, then manually type in or check off the names of people who were present. This system is flawed – it relies on the individual entering the information to see each member, which in a crowded room with members sitting behind couches can be unreliable. It also leaves the members not knowing if the person taking attendance noticed them, resulting in their asking “did you get me?” Immediate feedback to the member is necessary to prevent this. My solution to this problem is to use iButtons.

iButtons are a technology that in their most simple form are capable of carrying a guaranteed unique 64 bit ID. We already use them on CSH in our Drink projects Big Drink and Little Drink, as well as in prototype implementations of our iButton doorlocks, so the technology is not foreign to our members as an identifying token. In our case, a USB iButton reader would be connected to the laptop from which the attendance database client was running, and members upon entering the meeting place would click their iButton into the reader which would then record their attendance at the meeting. This solves the requirement of ensuring a member is physically present at the meeting and easily addresses the feature requesting that little involvement from the executive board member be had in recording attendance.

 The final subdivison of the client ends up looking like this with the above taken into consideration: the user interface (graphical interface), client backend (database communications,) iButton interface (physical reader and accompanying software), and iButton backend (local communications with client). In the next part of this series, we’ll discuss the technologies involved, the solutions decided upon, and the initial roadmap of development.