X-Sendfile2 Added in Lighttpd 1.4.24

Regarding my previous post about X-Sendfile and FastCGI improvements for Lighttpd, this feature was recently added in the Lighttpd 1.4.24 release. It is now possible to build an entire pseudostreaming system using Lighttpd and FastCGI, that is much faster than the previous mod_magnet alternative.

Getting Arch Linux fully working with a Samsung N130 Netbook

Recently, I purchased an Asus EEE 1101HA netbook. My experience getting the video to work on it was horrible and unsuccessful (Poulsbo chipset, ugh,) and other features were really difficult to get working. Last night I returned it to the store, and picked up a Samsung N130 (in black) for $50 less than the EEE.

There are two versions of the N130 laptop. One version of the laptop includes an Atheros wireless device, supported by the ath9k kernel module. The other version of the laptop (which I received from Best Buy in the US,) includes a Realtek 8192 wireless device which does not have a native Linux driver at this time. I hear there’s one scheduled for the 2.6.32 kernel, but the release date on that version is still unknown. To work around this problem we’ll use ndiswrapper to wrap the Windows XP version of the Realtek 8192E driver, which is a great workaround for now until 2.6.32 comes out.

This post details how to setup Arch Linux on an N130, along with wireless support, xmonad, and a bunch of apps. This setup is specific to what I use, but it works very well for me. I especially recommend this setup (read: xmonad) because of the N130’s small screen; xmonad is a great tiling window manager to use on a small laptop screen that helps you to avoid having to use the touchpad. Additionally, this post gives my specific config files and scripts that I use every day in my desktop environment (modified here for laptop use.) These have been requested of me before by users on Reddit, so I thought it’d be appropriate to publish them here.

The first thing I did after getting home with the N130 was make an Arch flash drive and install Arch from it. At first, I installed Arch with the root and swap partitions encrypted. Let me mention that if you want to be at all productive on the N130, you cannot encrypt the root or swap partitions (this was also true for the 1101HA.) If you do encrypt the root and swap partitions, you’ll notice your disk wait jump through the roof to more than 40% when doing simple operations like `pacman -Ss xorg` and kcryptd will be the top process on your system, for sure. When I realized I couldn’t do anything productive with this lag, I reinstalled with unencrypted root and swap partitions, and just encrypted the /home partition. This turned out to be much faster, obviously, and I recommend this route, despite the unencrypted swap being a potential vulnerability.

So, run through the Arch installer as you normally would, nothing fancy here. You’ll need a wired connection to download packages until we get the wireless working. Remember that when using the 2009.08 Arch installer to setup encrypted partitions, you must create your /etc/crypttab file yourself before booting into your new system.

Boot into your new system, login as root, and install relevant packages and configurations.

You must now configure networkmanager. This is very well documented, so just follow the steps listed here.

Logout as root, and login as your normal user. I’ve a few configuration files and scripts to make your life with X, Xmonad, vim, the system tray, and dmenu easier. First, setup the appropriate directories.

Finally, reboot your system (this will make slim and wireless come up on boot,) and you should be all set to login and start using xmonad. Once logged in, you should see a working battery monitor, network manager applet (with wireless and wired connections working,) xmobar on top, and a bunch of applications ready to go.

Here are some useful key combinations to get you going if you’re unfamiliar with either xmonad or my specific configuration:

• Alt+1: Takes you to the “code” workspace where gvim goes by default
• Alt+2: Takes you to the “term” workspace where terminal goes by default
• Alt+3: Takes you to the “web” workspace where Firefox goes by default
• Alt+4: Takes you to the “msg” workspace where Pidgin goes by default
• Alt+5: Takes you to the “media” workspace where Banshee goes by default
• Alt+Ctrl+l: Locks the screen with Xscreensaver
• Alt+Shift+Enter: Opens a terminal
• Alt+Shift+c: Close the current window
• Alt+k: Focus the next window in the workspace
• Alt+p: Run dmenu, from which you can launch applications
• F2 while in vim/gvim: Open NERDTree
• Alt+p cheese: Run cheese, allowing you to take photos/videos using the webcam

The only things I haven’t gotten working yet are the Fn (function) keys that control brightness, volume, etc. If I figure this out I’ll add it here.

If you find any errors in this post or have any questions please let me know. Good luck with your Samsung N130.

JWZ, Mozilla

I relate to this journal so well that I feel like I wrote it.

I just had to post that statement somewhere.

Recent Activity

I’ll use recent activity in no particular order to suffice an update.

I submitted this Django snippet.

I created this fork of this project.

I increased the font size on shellsage.com because of this very good blog post.

I submitted this Django bug, which, despite protest, has been closed. I don’t think “we don’t really have a good way to track ‘things we need to do for v2.0’” is a valid reason to close (aka discard) a ticket that details a valid enhancement. Besides this, the Django project should drop trac and use redmine, github, or bitbucket.

I switched to gvim from emacs.

I experimented with Pylons and hated it, not because I disliked its architecture or what it had to offer, but because it was extraordinarily difficult for me to implement user authentication without completely rolling my own; this in my mind is a completely pointless redundancy. I tried the suggested packages too, there were SQLAlchemy version incompatibilities. I wish this hadn’t been the case, I’d like to try mako templates instead of django templates, I think (yes, I know I can use mako in django, I’ll try it sometime.)

I finished up my work on bil. Unfortunately this probably means the project is about to die.

I’m finishing up my work on pcann-time-series. The only thing I have left to do is deal with some array indexing issues that have a lot of variables involved.

Bigger Fonts

I found this blog post to be of amazing value, I really do see the difference in 16px fonts. I’ve changed the fonts on shellsage.com to be 16px and must say I’m in love. Thanks to Wilson.

lighttpd, X-Sendfile-Extended

There is a problem in serving high volume large data files on something like a media web site. The issue comes in when trying to both manage files on the backend and deliver files quickly to frontend users.

A common filesystem used to manage large numbers of files is mogilefs, yet mogilefs has no built-in security mechanisms, nor is it capable of serving high volume well. This is not a fault of mogilefs, as obviously some sort of caching should be used.

Lighttpd is a common web server used to server large numbers of static files. It’s very good at its job.

If mogilefs is the backend in our scenario, and lighttpd is the frontend, the major concern comes in getting lighttpd to communicate with mogilefs only when needed, and also getting the two to communicate efficiently.

Lighttpd supports one mechanism for FastCGI apps to tell it what local file to serve, the X-Sendfile response header. This is pretty straight forward to use: a request comes in, the FastCGI app parses the URL and pulls the relevant file out of the mogilefs cluster to a local filesystem cache, and then passes the X-Sendfile header to lighttpd, at which point lighttpd serves the file from the local filesystem cache. This all seems well and good, but when you try to do more advanced serving this solution has drawbacks.

The first drawback is how to serve a range of data to a user. This is required for video pseudostreaming: a user tries to seek to a spot in a video in their browser that has not yet been buffered, and thus a new request must be initiated beginning at that new spot in the file. Lighttpd supports pseudostreaming via the mod_flv_streaming module, but that module really has no ability to cooperate with mod_fastcgi, and in fact is intended entirely to work with only static files being served directly off of disk with no mod_fastcgi middle man.

The second drawback is how to serve a range of data to a user with an FLV file header prepending the actual file data. You can’t just send it back as the normal response, because lighttpd will overwrite that data with the contents of the file specified by X-Sendfile.

Thus, there was no good way to connect mogilefs, a local filesystem cache, and lighttpd. The requirements for a solution were to allow a FastCGI app to fetch a file out of a mogilefs cluster if it did not exist in the local cache, and tell lighttpd to serve the file, optionally from a specific point in the file with all required file headers in place.

I was able to get a nearly ideal solution in place, but it involved mod_magnet which is slow (processed in lighttpd core.) Also, since the lua memcache module can’t be used within a mod_magnet script because of naming conflicts between the lua socket module and lighttpd’s internal lua entities, there is no way to securely generate a temporary name for files in the local system cache. To work around this, I had to generate file names based on time and the incoming file key from the request which is certainly not ideal and also does not allow the cache to be expired or cleaned gracefully.

The obvious solution to me was to patch lighttpd. Initially, I thought that I’d only have to serve a portion of a file via X-Sendfile to allow for pseudostreaming. So, I submitted a patch to allow for an X-Sendfile-Range header that simply specified a range of bytes to serve from the file. This patch was applied for the 1.4.23 release, but then removed because…

I forgot though that this solution had a major drawback (and it was less than ideal for stbuehler, the current main developer of lighttpd.) One would actually need to have lighttpd serve two files consecutively, in order. The first file contains the FLV header sent when pseudostreaming, and the second file would be the actual video file (whose range to send would also need to be specified to lighttpd.)

With some help from the lighttpd developers, we came to the conclusion that the X-Sendfile-Extended header should be implemented, allowing a FastCGI app to specify the response header one or more times in order to have lighttpd serve the respective files off of the local filesystem. I have recently submitted a patch for this feature as well. The only things remaining on it are for the feature to support the standard HTTP range syntax, and also to have lighttpd interpret comma delimited header values (e.g. X-Sendfile-Extended: 15-5000 /tmp/myfile, 777-2239 /tmp/myotherfile) to conform with the HTTP specification.

Although a quite fast and capable system is in place using a simple python flup script and lighttpd’s mod_magnet, implementing this mechanism once lighttpd stable is patched should drastically improve performance.

Trying to find time to develop bil, neural nets

I’ve two outstanding projects to take care of that have been on hold for awhile due to lack of time. The only thing that’s changed is that they can’t be on hold much longer.

bil was started in November 2008, and must be finished by August 7, 2009. I have yet to implement typing, functions, and arrays, and those three things are going to be a pain in the ass, especially given how little time I’ve got.

I’m also trying to complete a project determining the most efficient partially connected neural network to learn the next consecutive output of the sine function, given three consecutive, immediately prior points. To do this, I must evaluate all graphs between 3-1 (3 inputs, 1 output) and 3-3-1 (3 inputs, 3 hidden, 1 output, fully connected.) It’s probably that either 3-1 or a fully connected 3-3-1 will be best, but if there is a partially connected best, it will probably be publish worthy. The issue with this is finding a library that can implement designed partially connected graphs, of which in my experience, there are none. My Master’s dissertation will hopefully change this in that I plan to extend fann to support this prior to exploring grid computing of distributed neural nets. But for now, I have to implement this manually, which again requires time I don’t have.

After these, in the near future, I have to complete a project for visualizing data provided by sensors on a sensor network, and also formally submit my pre-proposal for my Master’s disseration.

Textie, gedit-vcs transferred to Jesse van den Kieboom

I’ve decided to stop work on the gedit-vcs plugins project, and go my own route. I was uncomfortable with some of the routes that I’d have had to take to implement those plugins, and I don’t think they were in line with my own goals. I’ve transferred ownership of the project to Jesse van den Kieboom (jessevdk) of gedit.

I’ve begun work on a minimal IDE that I call Textie. It’s written with pygtk and pygtksourceview. My base planned feature set includes:

• A multi-document interface (MDI) via tabs
• Project support via external project files
• All configuration options available on a per-project basis
• SVN integration
• Syntax highlighting themes
• A project file browser, file system browser, open file browser, and symbol browser
• Language independent code completion based on project’s configuration
• Configurable keyboard shortcuts on a per-project basis
• The ability to assign keyboard shortcuts to shell scripts to do things like build and run

My current status is a working window with menus, toolbar, and editor. When I have something that can actually edit a file and save it (CRUD,) I will start a Google Code Project for it.

gedit-svn changed to gedit-vcs

After numerous discussions on the gedit mailing list, and more private discussions with gedit developers, we’ve decided to pursue a set of plugins that share a common interface. These plugins are the gedit-vcs project.