Programming

U.S. Census Bureau Offers Public API for Data Apps

For any software developers with an urge to play around with demographic or socio-economic data: the U.S. Census Bureau has launched an API for Web and mobile apps that can slice that statistical information in all sorts of nifty ways.

The API draws data from two sets: the 2010 Census (statistics include population, age, sex, and race) and the 2006-2010 American Community Survey (offers information on education, income, occupation, commuting, and more). In theory, developers could use those datasets to analyze housing prices for a particular neighborhood, or gain insights into a city’s employment cycles.

The APIs include no information that could identify an individual.

/*
For those itchin' to get to the the APIs, click the link. There's also an "app gallery."

I, for one, am looking forward to the unique and useful ways in which this data will tell us new things about ourselves.
*/

PostgreSQL 9.1 Released

/*
Among the many new features, here's a snippet from the News page:
*/

Advancing the State of the Art
Our community of contributors innovates with cutting-edge features. Version 9.1 includes several which are new to the database industry, such as:

K-Nearest-Neighbor Indexing: index on "distance" for faster location and text-search queries

Serializable Snapshot Isolation: keeps concurrent transactions consistent without blocking, using "true serializability"

Writeable Common Table Expressions: execute complex multi-stage data updates in a single query

Security-Enhanced Postgres: deploy military-grade security and Mandatory Access Control

/*
The SE-Postgres looks to be particularly interesting. It allows you to use SELinux Mandatory Access Controls on the PostgreSQL users and data. Neat. Particularly the SECURITY LABEL.
*/

/*
Beware, however, that there are some incompatibilities with previous releases!

From the 9.1 Release Notes:
*/

Change the default value of standard_conforming_strings to on (Robert Haas)

By default, backslashes are now ordinary characters in string literals, not escape characters. This change removes a long-standing incompatibility with the SQL standard. escape_string_warning has produced warnings about this usage for years. E'' strings are the proper way to embed backslash escapes in strings and are unaffected by this change.

/*
Also from the Release Notes, it mentions the addition of synchronous replication.
*/

PostgreSQL streaming replication is asynchronous by default. If the primary server crashes then some transactions that were committed may not have been replicated to the standby server, causing data loss. The amount of data loss is proportional to the replication delay at the time of failover.

Synchronous replication offers the ability to confirm that all changes made by a transaction have been transferred to one synchronous standby server. This extends the standard level of durability offered by a transaction commit. This level of protection is referred to as 2-safe replication in computer science theory.

When requesting synchronous replication, each commit of a write transaction will wait until confirmation is received that the commit has been written to the transaction log on disk of both the primary and standby server. The only possibility that data can be lost is if both the primary and the standby suffer crashes at the same time. [...] ...it also necessarily increases the response time for the requesting transaction. The minimum wait time is the roundtrip time between primary to standby.

/*
There's also a What's New in PostgreSQL 9.1 wiki page that explains much of these new features in detail.
*/

Opa

Opa is a new member in the family of languages aiming to make web programming transparent by automatically generating client-side Javascript and handling communication and session control. Opa is written in OCaml. A hierarchical database and web server are integrated with the language. The distribution model is based on a notion of a session, a construct roughly comparable to process definitions in the join-calculus or to concurrent objects in a number of formalisms.

A good place to start is here. And here you can find several example programs with accompanying source code.

/*
This looks interesting. Although I don't know OCaml, it might be worth learning it to use this. The automatic creation of the client-side JavaScript for maintaining sessions and communication sounds pretty slick.

Here is the link to the documentation, which looks to be pretty decent.
*/

ldd Arbitrary Code Execution

/*
This article explores a documented, though largely unknown, feature of the Linux dynamic linker. It also appears that BSD, Solaris, and HP-UX might also fall victim to this "trick."

I'm hesitant to call it a vulnerability since it's a documented feature, and requires some social engineering to succeed. It is, however, behavior that many experienced Unix admins may not be familiar with.
*/

Shellcode Collection

/*
This site contains hundreds of shellcodes ranging from the standard execve(/bin/sh) to bindshells; from Linux to IRIX to Cisco IOS. Pretty neat collection!
*/

PostgreSQL 8.3 Cheat Sheet

/*
A very handy cheat-sheet of common PostgreSQL programming features.
*/

GSSP (GIAC Secure Software Programmer): C Secure Coding Tasks, Skills and Knowledge

/*
This is a PDF file from the SANS Institute on secure C-coding practices.
From the PDF...
*/

This document enumerates common C coding tasks and identifies rules, recommendations, and guidelines for accomplishing these tasks securely.

/*
A lot of the material I've seen so far has been common-sense stuff (input validation, mistrust of environment variables, etc), but it's nice having it in a form not unlike a checklist. You write a new function or new class, and then run down the list; making sure you've followed each item. There's also one of these for Java ("booo") and .Net ("hisssssss").

Weighing in at only 10 pages, I think this one is worth wasting dead trees on.
*/

Five More PHP Design Patterns

/*
A follow-up to the previous Five Common PHP Design Patterns on IBM's DeveloperWorks.

This one covers the Adaptor, Iterator, Decorator, Delegate, and State patterns.

Both articles are definitely worth a read!
*/

Five Common PHP Design Patterns

/*
Yet another IBM DeveloperWorks article. This is a good read, for any programmer, especially a PHP programmer.

This article covers the following design methods:
Factory
Singleton
Observer
Chain-of-Command
Strategy pattern

I thought I'd already linked to this article ages ago. It's been in my bookmarks for ages; along with the next article.
*/

pl/pgSQL Programming Guide

With PL/pgSQL you can group a block of computation and a series of queries inside the database server, thus having the power of a procedural language and the ease of use of SQL, but with considerable savings of client/server communication overhead.

Extra round trips between client and server are eliminated

Intermediate results that the client does not need do not have to be marshaled or transferred between server and client

Multiple rounds of query parsing can be avoided

This can result in a considerable performance increase as compared to an application that does not use stored functions.

Also, with PL/pgSQL you can use all the data types, operators and functions of SQL.

/*
This is a link to the PostgreSQL 8.3 documentation for the pl/pgSQL procedural programming language. You can greatly speed up application performance by moving much of the decision-making to the database.
*/

/*
Miscellaneous Microsoft Docs

Occasionally I come across some Microsoft articles that are of use to myself or those I know. I've gathered a list of Windows Server 2K3 and IIS 6.0 commands and tools that will help in automating processes.

How to Restart IIS
Additional Resources for the IIS 6.0 Metabase
Command-Line Tools Included in IIS
Using Command-Line Administration Scripts
Starting and Stopping Services (IIS 6.0)
*/

Speed Up Your Ajax Applications While Dodging Web Services Vulnerabilities

Deploying bandwidth-efficient Ajax applications does not guarantee that the service levels in a Service Level Agreement will stay high. No matter how well you change code in the Ajax format to make it more bandwidth efficient, there will be always risks and vulnerabilities you'll need to watch out for and mitigate. Regular developerWorks author Judith Myerson gives a brief Ajax recap, shows what Web services vulnerabilities are and why Service Level Agreements (SLA) are important, and suggests some solutions for speeding up Ajax applications.

/*
This DeveloperWorks article covers AJAX concepts such as vulnerabilities (excessive bandwidth, corrupted data, frequent small requests, and memory leaks), SLAs, and improvements.

Improvement topics include speeding up applications, web services standards, and traffic monitoring.
*/

AJAX Security Tools

Certain vulnerabilities within Ajax applications can allow malicious hackers to reek havoc with your applications. Identity theft, unprotected access to sensitive information, browser crashes, defacement of Web applications, and Denial of Service attacks are just a few of the potential disasters Ajax applications can be prone to and which developers need to guard against when building Ajax capabilities into their applications.

/*
This DeveloperWorks article has some interesting tools. One, aSSL, is an AJAX add-on that allows you to tunnel your AJAX requests back to the server over AES encryption. Very nice. It's licensed under the MIT license.

There are also links to several Firefox add-ons and extensions, such as Session Manager, that allows you to re-establish sessions on demand.
*/

PHP Security Consortium :: PHP Security Guide

"If you do not design your application with security in mind, you are doomed to be constantly addressing new security vulnerabilities. Careful programming cannot make up for a poor design."
-- PHP Security Guide, Chapter 1, Section 1.1

/*
This guide certainly gets to the point. The PDF is only 37 pages long, so it's a fairly short read. There's coverage on global variables, data filtering, error reporting, form processing, XSS, CSRF, SQL injection, session hijacking, and file-system concerns. Sample code is used several times as well.
*/

Perl's Curses::UI

/*
I was writing a perl script today to migrate database data, and realized it would be very handy to have some type of status on the screen. I started looking through documentation for Perl's Curses::UI family of modules and found that the Curses::UI::Progressbar was exactly what I was looking for!

I've been writing in perl for better than 10 years now, but this is the first that I've really attempted much in the way of a CLI-based "user interface" for much of my code. It was fed by either CGI over the web or using Getopt::Std and just writing formatted text out to STDOUT.

There's a small tutorial called Curses::UI::Tutorial that will allow you to get your feet wet. It only covers a small handful of available "widgets", but it covers the basic structure of "windows" on the screen, and what input/output fields each of those may contain.

The code reminds me somewhat of Perl/Tk code where you design most of the interface, throw some logic into a few event handlers, and then call MainLoop to spin until an event occurs.

Here's a list of most/all of the more common elements within Curses::UI:

Curses::UI::Widget
Curses::UI::Container
Curses::UI::Color

Curses::UI::Buttonbox
Curses::UI::Calendar
Curses::UI::Checkbox
Curses::UI::Label
Curses::UI::Listbox
Curses::UI::Menubar
Curses::UI::PasswordEntry
Curses::UI::Popupmenu
Curses::UI::Progressbar
Curses::UI::Radiobuttonbox
Curses::UI::TextEditor
Curses::UI::TextEntry
Curses::UI::TextViewer
Curses::UI::Window

Curses::UI::Dialog::Basic
Curses::UI::Dialog::Error
Curses::UI::Dialog::Filebrowser
Curses::UI::Dialog::Status
*/

Prototype JavaScript Framework

Prototype is a JavaScript Framework that aims to ease development of dynamic web applications.

/*
I am just starting to dig into AJAX and Prototype, more specifically. So far, it seems fairly intuitive and less bulky than the Google Web Toolkit (GWT).

For those wishing to jump straight to the API docs, click here. You might also be interested in the Tips and Tutorials section.
*/

perltoot - Tom's Object-Oriented Tutorial for Perl

/*
This is excellent documentation for Perl Object-Oriented Programming; also known as Camel POOP.

The site is in Russia, but the documentation appears to be entirely in English.

The index of sections are listed below:
*/

NAME

DESCRIPTION

Creating a Class

o Object Representation
o Class Interface
o Constructors and Instance Methods
o Planning for the Future: Better Constructors
o Destructors
o Other Object Methods

Class Data

o Accessing Class Data
o Debugging Methods
o Class Destructors
o Documenting the Interface

Aggregation

Inheritance

o Overridden Methods
o Multiple Inheritance
o UNIVERSAL: The Root of All Objects

Alternate Object Representations

o Arrays as Objects
o Closures as Objects

AUTOLOAD: Proxy Methods

o Autoloaded Data Methods
o Inherited Autoloaded Data Methods

Metaclassical Tools

o Class::Template
o Data Members as Variables
o NOTES
o Object Terminology

Wikipedia Articles in Regards to Various Types of Exploits

/*
First, there is the Wiki page detailing exactly what an exploit is. This is a very good read and should acquaint anyone with what an exploit is and can be capable of.

Second, there are Wiki pages detailing several types of exploits and detailed information as to how they work. Below, you'll find some great examples of exploits ranging from low-risk web browser information disclosure on up to full system-level compromises.

• Buffer Overflow
  
• Heap Overflow
  
• Stack Buffer Overflow
  
• Integer Overflow
  
• Return-to-libc Attack
  
• Format String Attack
  
• Race Condition
  
• Code Injection
  
• SQL Injection
  
• Cross-Site Scripting (XSS)
  
• Cross-Site Request Forgery (CSRF)
  

Some of these methods were even new to me. The Sea-Surf (CSRF - Cross-Site Request Forgery) method is something I had at least considered as possible, but I had no idea that the method had a name and was actively in use in-the-wild.

Some of these methods require some social engineering to trick the end-user (target) into activating the payload; whereas others require no interaction by the target, and they are often unaware that they have been or are being compromised.

There are many other interesting documents covering everything from shellcode to polymorphic code to a NOP (Null OPeration) to NOP Sleds.

Happy hacking!
*/

/*
In searching for other information, I came across a nice Endianness chart on Sun.com.

This table describes, by platform, if it's big-endian or little-endian. For a description of the difference, check out the Wikipedia page on Endianness.

The platforms are listed below, and which type they use:

Big Endian:
Sun SPARC
IBM 360/370
Motorola 68000 & 88000
PA-RISC
PowerPC

Little Endian:
Intel family (8086, 80286, 80386, 80486, 80586, and 80686, and newer)
PDP-11
VAX
Alpha

I've seen MIPS processors in both varieties.
*/

Linus Torvalds Slams C++

From: Linus Torvalds linux-foundation.org
Subject: Re: [RFC] Convert builin-mailinfo.c to use The Better String Library.
Newsgroups: gmane.comp.version-control.git
Date: 2007-09-06 17:50:28 GMT (1 day, 1 hour and 59 minutes ago) On Wed, 5 Sep 2007, Dmitry Kakurin wrote:
>
> When I first looked at Git source code two things struck me as odd:
> 1. Pure C as opposed to C++. No idea why. Please don’t talk about portability,
> it’s BS.

*YOU* are full of bullshit.

C++ is a horrible language. It’s made more horrible by the fact that a lot
of substandard programmers use it, to the point where it’s much much
easier to generate total and utter crap with it. Quite frankly, even if
the choice of C were to do *nothing* but keep the C++ programmers out,
that in itself would be a huge reason to use C.

/*
While I can't say I disagree with him, it's always funny to see someone so highly regarded saying this publicly. Linus is widely known not only for being an excellent programmer, but also an excellent project manager in maintaining the worldwide mass of patches that is known as Linux.
*/

First AMD/ATI Specs Released

/*
It looks as if AMD has made good on their word to release the specs so the open-source community can create a proper 3D driver. There was some speculation as to whether or not they would actually come through. Apparently they had committed to this ages ago, then backed out. They're finally out, and no NDA.
*/

42589_rv630_rrg_1.01o.pdf
RRG-216M56-03oOEM.pdf

/*
There have been quite a few bullet-points regarding CFS in the past week or so. There are a few things worth mentioning.
*/

CFS, Focusing on Simplification and Performance

Ingo posted some results from the lmbench benchmark noting around a 16% speedup on both the 32-bit and 64-bit x86 architectures. He added, "we are now a bit faster than the O(1) scheduler was under v2.6.22 - even on 32-bit. The main speedup comes from the avoidance of divisions (or shifts) in the wakeup and context-switch fastpaths."

/*
That's damn impressive. If you can shave, say, 10% off your context-switching time, that will reap huge rewards.

Let's look at some numbers...

There have been more than 1241498 context switches in the 88 minutes this machine has been up (kernel build, reboot). To knock that much time off something that occurs that frequently amounts to quite a bit of savings. A faster context-switch also results in much snappier performance.
*/

Benchmarking CFS

"Looking at these graphs (and the fixed one from your second email), it sure looks a lot like CFS is doing at *least* as well as the old scheduler in every single test, and doing much better in most of them (in addition it's much more consistent between runs)."

/*
The graphs do show quite encouraging numbers. A quick peek at some of the numbers. All times are in microseconds.

lat_ctx -s 0 2:

2.6.23-rc1      2.6.23-rc6-cfs  2.6.23-rc6-cfs-devel

   5.15            4.91                 5.05

   5.23            5.18                 4.85

   5.19            4.89                 5.17

   5.36            5.23                 4.86

   5.35            5.00                 5.13

   5.34            5.05                 5.12

   5.26            4.99                 5.06

   5.11            5.04                 4.96

   5.29            5.19                 5.18

   5.40            4.93                 5.07

*/

Greenplum Database

Greenplum Database is the first open source powered database software that can scale to support multi-terabyte data warehousing demands. Greenplum Database allows organizations to analyze vast amounts of business data 10 to 100 times faster than traditional data warehouse solutions at a fraction of the cost.

Key Features of Greenplum Database

• Faster Reporting and Analytics
  
• Dramatic Cost Reduction
  
• Centralized Cluster Management and Administration
  
• Linear Scalability and Flexibility
  
• Modular Performance and Capacity Scale-Out Capability

Greenplum Database's fundamental breakthrough is its ability to store and process terabytes of data using clusters of low-cost servers. Greenplum Database moves processing power as close as possible to the data, so processing always occurs in parallel, delivering a dramatic boost in query and load performance. In addition, Greenplum Database's Dynamic Provisioning technology makes it easy to add incremental data warehouse capacity when needed, avoiding costly appliance upgrades.

/*
It looks like there might finally be a decent option for PostgreSQL scalability. I have not personally used this software, yet, but I will be installing it shortly. I intend to do some benchmarks versus a stand-alone PostgreSQL database and see how the numbers match up.

If this solution is half as good as the website touts it to be, then Oracle may end up losing some serious ground in the SMB (Small-to-Medium Business) market.
*/

High-Level Architecture: A database in Greenplum is actually an array of individual databases, usually running on different servers or hosts, all working together to present a single database image. The Greenplum master is the primary entry point to the Greenplum Database System. It is the database instance where users connect to the database and execute SQL statements. The master coordinates the work amongst the other database instances in the system-the Greenplum segments, which is where the user data resides.

Mirroring and Fault Tolerance: When you deploy your Greenplum Database system, you have the option to configure mirror segments. Mirror segments allow database queries to fail over to a backup segment if the primary segment is unavailable.

Greenplum Database is able to detect when a host is unavailable or when a segment database server process is down. When this occurs the master will mark the primary segments on that host as out-of-service and immediately switch over to the mirror segments so that the operation can continue.

...

Why is Greenplum Database better-suited to business intelligence and data warehousing than databases like Oracle?
Greenplum Database’s “shared-nothing” architecture is optimal for fast queries and loads because it places processors as close as possible to the data itself, and performs queries and other operations with the maximum degree of parallelism possible. “OLTP” architectures like Oracle’s were designed and built with an entirely different purpose and are not capable of the kinds of parallelism, or performance, that Greenplum Database delivers.

How is that Greenplum can deliver a product like Greenplum Database, but no other company or organization has to date?
When the founders of Greenplum converged in 2003, they set out to change the game. They saw that enterprise software, and particularly database software, was far too expensive and performed badly. Our uniquely capable team includes some of the best minds in the industry, with experts from Oracle, Teradata, Sybase, Informix, Netezza, PostgreSQL, HPTi, CalTech, MIT, Stanford University, and other leading companies, organizations and institutions. Greenplum Database is the result of the confluence of Greenplum’s unique vision and vast experience in the midst of undeniable industry trends.

Top Ten of Programming Advice NOT to Follow

/*
I came across this article on Digg a few days ago. Just got to finish reading it.

This article presents some very well stated points on why some of these "standard" rules should not be followed. It's actually somewhat reassuring to know that I'm not the only one that thinks some of these "standards" are no more than constraints.
*/

Tuning LAMP Systems, Part 2: Optimizing Apache and PHP

/*
Another great IBM DeveloperWorks! article. It also links to the first article, which appears to be a 3-part series.

The article seems to require Apache 2.x. Most of my installs tend to be 1.3.x with PHP 5.x, mod_security, and sometimes mod_perl. YMMV.
*/