Archive for the 'Programming' Category
In my last post I brought up how to get around mod_python’s default inability to handle relative paths for imported modules. The next night after writing that, I faced a very similar issue with configuration files.
It is a sin to duplicate data in a project (except when caching), so most sane desktop applications will have a central store that contains the configuration parameters for the application (usually packaged as a resource with the application, or in the registry for Windows users). Mod_python can’t do this because, as was discussed in my last post, it views the current working directory of whatever script it is executing as the root directory of the filesystem (”/”). So where do you put the configuration file for your mod_python based project if you can’t refer to it relatively, don’t want to hardcode an absolute path, and want to distribute the configuration with the project?
The solution hit me like a bucket of cold water. Give up on the property or ini format file, and make your configuration a python dict object that you can directly import:
# config.py
#
# This is the project's configuration. Import it when a class needs to access
# project configuration variables.
#
_config = {
# The name of my app so pathHack knows what string to look for
'rootname': 'sampleWebApp',
# Database options
'db_host': 'localhost',
'db_dbname': 'sampleWebApp',
'db_user': 'sampleWebAppUser',
'db_password': 's0m3p455w0rd',
# Debug logfile path
'debug_logfile': '/tmp/lostafollower.log'
};
This file needs to live in your project’s root directory as far as mod_python is concerned. For more information, read my last post. If you’ve already read it, then you’ll understand when I say that config.py lives in the same folder as pathHack.py.
Once you can import it from your main source files, use it like this:
<%
# sample usage of config
from config import _config as _c
req.write(_c['rootname'])
%>
Tada! Extraordinarily simple configuration file that is easily accessible and can be distributed with your mod_python application. I’m pretty happy about it :)
A long while back (a year already? Jeeze!) David Brenner and I wrote a web application without using any frameworks, only mod_python and python server pages (PSP). We ran into a few problems along the way, one of which being that the import path mod_python used had to be composed of absolute paths. It isn’t a huge problem to just hard code a path to your project’s lib directory, but it is hardly portable and adds to the fragility of the system.
I’m now doing another project (more on that later), again using mod_python and psp. This time around however, I have a hack that gets around the absolute path problem. The solution looks like this:
# This file has only one function: to add directories to the path
# so we can import other python files/modules.
import sys
# Default directories to add to the path
_dirs = ["/src", "/lib"]
_applicationRoot = "sampleWebApp"
# Will go through the list and add the directories given relative to the
# lostafollower base path. Allows us to do paths relative to our installation
# instead of hard-coding absolute paths.
def addLibs( dirs ):
# If [] was provided, just use the default.
if not dirs:
dirs = _dirs
base = ""
for entry in sys.path:
if entry.find(_applicationRoot) != -1:
base = entry
break
for dir in dirs:
try:
# Test if the path contains the path we want
sys.path.index(base+dir)
except:
# The path doesn't already contain our path
sys.path = sys.path+[base+dir]
The reason we normally can’t use relative paths when doing imports is because the mod_python handler considers the root (”/”) of the filesystem to be the current working directory of the application, regardless of where the application is actually running from. However, Apache will put whatever alias it has for your application’s directory [1] in the PYTHONPATH for mod_python, which is what allows us to do this hack. By finding the path that Apache has in the PYTHONPATH, we can add whatever paths we want relative to that. No .htaccess PythonPath directives, no hard coding of paths to your project’s module locations. Just:
- Put the above code in a file (I call it pathHack.py) in your application’s root directory (root as far as Apache is concerned) [1]
- import pathHack (or whatever you called it) in whichever file needs your other modules
- Call pathHack.addLibs([]) to add the defaults (/path/to/sampleWebApp/lib, etc), or pathHack.addLibs(”/lib/3rdParty”) to add /path/to/sampleWebApp/lib/3rdParty to your path
- import any python module in those newly added directories!
That is really it. Hard coded paths begone!
[1]
When I say “the alias it [Apache] has for your application’s directory,” what I mean is that even though, for example, my application runs out of /home/robertp/src/sampleApp/src, mod_python sees the root as being /home/robertp/src/sampleApp because of some unknown magic. It doesn’t really matter what the path is, in all honesty – when specifying a value for _applicationRoot in the solution code above, you are just looking for a string unique to the path to your application (just be sure to make all supplied paths relative from that base). Try just your project’s deployment folder for starters. If you are still having trouble finding it, create a quick psp file in the same folder as your other psp files, and do:
import sys
from mod_python import apache
req.write(str(sys.path))
That will just print out all the path elements to your browser window when visited. Pick out your application’s name or path from there, and set _applicationRoot to that. It isn’t completely ideal, but it is better than hard-coding ;)
A lot of developeres I knew back in college loved using Macs for their development machines, and you don’t have to look very far to see this opinion reflected on the Internet. In total, I’ve been using a Mac as a development machine for about five months. And now that I have a spare moment to write, I want to share my experience coming from using Debian Linux as a development machine to coding Java on a Mac.
The Environment
OSX as an environment is just different initially. Using the Apple key instead of the control key is odd to get used to, so there was some loss of productivity (and a lot of frustration) when my muscle-memory keyboard shortcuts weren’t working. I got over the change after a couple weeks, but it still is a pain. The placement of the Apple key is two keys further in than control, and I now have to move my hand a decent amount to hit my very frequently keyboard shortcuts.
The windowing system also takes some getting used to. Close and application and it doesn’t actually close? What gives? I haven’t found a single instance where that has been a useful feature to have. It just adds a separate action to closing a process.
On those lines, windows can only be resized from the bottom right corner. If you want to expand something by grabbing an edge, or making something bigger by dragging an upper edge, you can’t. After growing up on Windows and then Debian with KDE, the instinctual grab and drag resulting in a non-action is annoying. The borders are there, but they aren’t functionally doing anything other than separating the window from other windows.
OSX is also missing virtual desktops. From a developer’s standpoint, all I can do is issue a giant WTF. Having separate desktops allows for the separation of distracting tasks and productive tasks. Having e-mail, instant messenger, and iTunes sharing a space with Eclipse, documentation, and my xterms (more on that in a sec) is extremely counter-productive. The argument can always be made that only applications you are using should be open, but switching workflows should be as simple as hitting a shortcut – not by manually managing windows.
A very minor point is that OSX does not feel snappy. My machine is a couple years old, so that could have something to do with it – but having a few xterms open and eclipse in the background should not cause a noticeable delay when switching to and typing into a new text field.
The Tools
The biggest thing that initially irked me about OSX was its lack of several very common tools. For example, OSX offers curl as a web retrieval tool instead of wget, claiming they are functionally equivalent. They absolutely are not. Wget has a mirroring feature that I found myself sorely missing when trying to pull down a multi-leveled source tree without version control from a web page.
Also missing isn’t as much of a tool, but a feature that is completely missing: the /proc filesystem. On a Linux box, a /proc filesystem can be easily enabled (if it isn’t by default) which contains information about the kernel, processes, devices, the processor, and a HUGE amount of infromation that comes in pretty handy in unexpected ways while developing. Not having this resource available causes headaches at inopportune times.
Other tools are available, but modified. Tools such as netstat show different information and have slightly modified options than the Debian counterparts. The result: figuring out how to display which process was tied to a port took nearly 15 minutes to figure out instead of taking a couple seconds to issue a command. Another example is top: processes can’t be listed as trees in the OSX version of the tool. It is small subtleties such as this that interrupt me when I am deep in a mental debugging stack – subtleties that kill productivity and add frustration.
A smaller issue was the lack of X11 and xterm. Console is fine for small tasks, but lacks the power and customizability of an xterm. Plus, I don’t want to reconfigure a console to match my xterm. I just want to pull over my .Xdefaults and have things the way I had them before. Luckily, Apple provides X11 in their developer pack, so installation wasn’t an issue.
Installation and Package Management
Application installation on a Mac is incredibly easy for user applications. Things like Omnigraffle (my very favorite diagramming software EVER) install with a drag and drop from a disk image to the applications folder.
However, installing common Unix tools (like wget, for reasons mentioned above) was a problem completely ignored by Apple. There is no package manager that comes with OSX by default for these kind of tools, and compilation is left up to the user. As a developer, this matters to me. In Debian, the power of the APT package management system made installing a new tool a single command process. Apple very easily loses here in terms of ease of use.
Libraries are also managed in a way that was initially odd. OSX considers applications like Java to be a framework, so access to something like JAVA_HOME is hidden behind a symlink. Not being in a standard location was initially annoying (/usr/lib or something similar), but once found, I was already a huge fan of the system. Keeping a symlink to a library is a Good Thing ™, because upgrades can be made seamlessly to new versions without requiring any updates to the shortcuts that point to the library. However, it could be done better. Debian uses an ‘alternatives’ system for all major libraries. Now multiple versions of any library can be set as the system-wide default at any time without requiring shortcut updates by allowing the alternatives system to manage the symlinks for you. This actually becomes useful in java development for allowing your application to be easily tested using different virtual machine implementations.
Conclusion
The conclusion shouldn’t come as any surprise to anyone: I hate developing on a Mac. I do no Mac-specific development, so the ease of writing Cocoa applications with Xcode means nothing to me. The features that I have come to rely on are either missing or implemented differently on OSX, leaving me with hurdles and interruptions where I absolutely do not need them.
When I am developing, I want things to just work – and I mean that from a developer’s perspective. My user experience matters less to me than having my platform enable my development productvity, and in this department OSX fails. I eagerly await the time when I can switch back to a Debian Linux workstation for my professional development, with fluid workflows and familiar tools to fuel my debugging speed and code output.
I just wasted an hour tracking down a problem with an application that controls Sharp projectors via a RS-232 (serial) connection. On sending the projectors a “POWR 1″ code to turn the projector on, we were getting “ERR” in response. The problem ended up being that the bulb on the projector was burnt out, so the projector was intelligently unable to turn itself on.
This is one case where a simple problem turned out to be a fairly annoying one.
GCJ, the GNU compiler with java extensions, does a great job at compiling Java into bytecode, but still has some bugs in its libraries when dealing with Swing components. Installing Sun’s Java packages on Debian thus is occasionally necessary, and has historically been a chore. I won’t list the process here to even show my distaste for it – it just wasn’t very fun.
Things are much easier now, though. Just make sure a non-free package repository is listed in your sources.list, and things become magic:
sudo echo "deb http://ftp.us.debian.org/debian/ lenny non-free" >> /etc/apt/sources.list
(Note that if you aren’t using lenny, you should change that. Also, feel free to choose a different mirror.)
Now update your package repository:
sudo apt-get update
And finally install whichever Sun Java packages you want!
sudo apt-get install sun-java6-jdk sun-java6-jre sun-java6-plugin
Cheers to Matthias Klose (Ubuntu), Juergen Kreileder (Blackdown), Barry Hawkins, Jeroen van Wolffelaar, and the other folks behind debian-java for adding these packages to Debian’s repositories. It is another push that greatly enhances the usability of the project for both developers and users alike.
My coolest project from work as of late has been to write a program to control the Sharp LCD projectors in RIT’s Software Engineering department through a serial interface. I really like writing code that interfaces with hardware, so this was one I enjoyed doing.
One thing I learned quickly is that debugging serial applications is hard. When my code wasn’t working, I didn’t know if it was because of the hardware, the software, or the command messages I was sending. If serial consoles or line printers were still widely available, it wouldn’t be hard to see what was going on. Since they aren’t, an alternate solution was devised utilizing GNU screen.
With two Linux boxes running Debian, I set one up with my code to send data from, and the other with screen attached to the serial device at /dev/ttyS0. Screen functions normally, but with the added bonus of displaying messages recieved from the serial device, and when entered, sending messages to the device. Screen makes it possible to visualize serial communications and send test messages without having to write or modify any code, making it an infinitely useful tool in debugging serial applications.
My bug turned out to be twofold: one hardware, and one software. The first was that the serial connection to the projector was not complete; there is a break somewhere in the wall. The second was that the manual states only that the command message be followed by a newline character. I was sending Unix newlines, like ‘\n’. The projector was written expecting Windows style newlines, which is a newline prepended by a carriage return, or ‘\r\n’. This simple fix, which I would have never stumbled over without the help of screen, was the source of my software based problems.
Yesterday I gave the second lecture in my “hacking” series. We’ve progressed beyond general descriptions and terminology and moved into the technical aspects of the stack, vulnerable code, and crafting exploits. The lecture ran about 45 minutes, and was accompanied by a live demonstration of exploiting vulnerable code.
For those that came late or missed it, the slides are available here (pdf format), although once again the real content was in the accompanying talk.
The demonstration used three source files. The shellcode was written by BreeZe of binbash.org, while the other two are my own. They are:
Running the Exploit
First, I only tested the exploit running on Debian Linux, running a 2.6.23 kernel on a 32bit x86 machine. It probably won’t work on Windows or a Mac, and it definitely won’t run on your SPARC.
Second, I used version 3.3 of gcc’s compiler. Newer versions contain checks to make stack smashing harder, but 3.3 is free of any of these security features. You’ll have to use it when compiling these, which can be done by issuing gcc-3.3 -o <name> <name.c>, like:
gcc-3.3 -o exploit exploit.c
Third, Linux kernels after 2.6.12 do virtual address space randomization, which will prevent the exploit from running successfully after being compiled. To disable this, issue (as root):
sysctl -w kernel.randomize_va_space = 0
Fourth and finally, if you want to trigger a core dump in the vuln program, you have to lift any restrictions on dumping core files. Something like the following should work:
ulimit -c 100000
Note that the shellcode file is superfluous, I only included it as an example and guide.
Next Time
At this point, I’m not sure what I’ll be covering next week. There is a lot I could do – writing shellcode, showcase more advanced shellcode, demonstrate gaining a remote shell (remote exploitation is a whole other beast), secure coding, polymorphic and self-modifying shellcode… there are a lot of topics, and only one lecture left before the end of the series.
Regardless, it will be fun, and we’ll all learn a hell of a lot.
I’ll be at Barcamp Rochester 3 this Saturday in GCCIS, and more likely than not I’ll be talking about anything and everything computer security. If you have any questions anything or find something I’ve said interesting, hit me up! I’m friendly, and always up for talking about computer security :)
This past academic quarter I elected to do an independent study. A lot of hype has been surrounding web development over the past few years, and the python scripting language for even longer than that. Figuring it was time to immerse myself in both subjects and the technologies related to them, the independent study turned into a full four credit project aimed toward creating a web application for administering quizzes. Because I only had ten weeks to learn everything involved with developing a web application on top of the language, I coerced my good friend David Brenner into doing it with me. Development is done, and I now feel like I can talk intelligently about my experiences.
The project was developed using straight python, utilizing the mod python Apache module to serve Python Server Page (psp) files. Python isn’t my favorite language to code with, it is lacking some things that I’ve come to expect from a scripting language, and some things are just silly. Examples of lacking features include the ability to get an iterator for a collection. Iterators that grant references both to the next and previous element of a collection grant superior control and the ability to keep state while iterating. Python only allowing you to move forward through a collection with its “for x in collection:” construct really hurts while parsing large sequences of text, and having to revert to array notation and keep an index is much less human readable than just manipulating an iterator.
Annoyances with the language include python being a scripting language that doesn’t default to printing the string value of non-string objects, and using procedural style method calls intermixed with object-oriented style. For example, to print an object (say an array), you can’t just do print(“Array contents: “ + arrayName). You have to manually return the string value by using print(“Array contents:” + str(arrayName)) in order to avoid an error that print() was expecting an object of type string. I know python’s mantra is “explicit over implicit,” but languages are tools that you learn how to use. As I learn more, I expect to gain proficiency with the tool in order to save me time in the future. Python is easy to learn and easy to use, but I have yet to find any tricks with the language that save me time or make my job easier than it would be with another language. One last thing that I simply do not understand is the necessity to include a reference to “self” in method and constructor definitions. I strongly prefer the method signature to match the required call, python’s decision to branch from that convention is one that I can’t imagine will have a justification strong enough to alter my opinion of the practice.
As it goes with web development, we also used javascript, a MySQL database backed with InnoDB table engines, LDAP for authentication, HTML, and CSS. Development was utterly boring. Being a computer science student, I enjoy working on new things that present challenges and reward creative solutions. The most thought intensive part of this project was the parser run on imported quizzes using a format we decided on – and that was developed in the fourth week. Everything else was the brainless transfer of requirements to code. Some of the people I go to school with are attracted this kind of development, and I think I know why: frontend web dev is easy. Writing the backend logic isn’t bad, but the interface code and associated handlers could be composed by a high school student with some spare time and only rudimentary knowledge. The things learned in computer science definitely aren’t applied in that domain.
Finally, to wrap up loose ends: object relational database design is neat and very handy, javascript is odd at first but easy to pick up, dealing with web requests and forms is tedious, debugging web applications is a *bitch,* and Debian Linux with Apache, Subversion, screen, and vim makes for a great development and hosting environment.
Web development has its place, but it isn’t for me. I’m capable with python, but I’ll stick with ruby. And if you introduce yourself to me, try to be more descriptive than “I’m a web developer” so my experiences don’t tarnish my opinion of you – because I bet you are more skilled at writing code than that title would suggest.
I’m one of those people that tends to get distracted when I’m procrastinating. Most often, this comes out when I’m doing a programming assignment and I go to open documentation in my browser. For those of you that weren’t aware, Firefox and Stumbleupon are a developer’s worst nightmare – so many interesting things, only a click away! It had to stop.
So I hacked up DocBrowser, a minimalist’s web browser created solely for the purpose of quickly accessing documentation and APIs. It is free of clutter and distractions, leaving only you and the information you seek on the screen. It is not extendable, and its feature set is limited. It is therefore exactly what I had in mind when I started coding it.
I’ve created a page for it here. It is free, and is only 1.5mb, so why not download it and give it a shot?
Windows only, requires .NET 2.0.
As previously mentioned in this blog, I’m taking a programming skills course focusing on Aspect Oriented Programming (AOP) using AspectJ. I am using eclipse with the AspectJ plugin to develop, and recently came across a situation where I wanted to define a pointcut based on an object that uses generics. In attempting to do so, I got the following error in eclipse:
parameterized types not supported for this and target pointcuts (erasure limitation)
While the error puzzled me momentarily, a quick hop over to the AspectJ developer notebook sorted things out. The full description of the solution is there if you want to read, but the quick fix is easy: remove the generic identifier of whatever you are trying to match against.
For example, this is incorrect:
pointcut badPointcut(): target( List<String> );
While this is the fixed version:
pointcut goodPointcut(): target( List );
Now get back to work :)
