Doug Bryant

Tech thoughts and notes

OSCON 2005 - MetaProgramming

presenation by Glenn Vanderburg

What is meta programming? It’s Programming your programming language

Rubyist have been discovering metaprogramming. Ruby style and idioms are still changing and adapting

Ruby good for metaprogramming b/c

  • Dynamic and reflexive – everything is open – blocks allow writing new control structures – most declarations are executable statements – only slightly less malleable than lisp (no macros) – unobtrusive

Examples…

attr_reader, attr_writer, and attr_accessor.

if written in ruby attr_reader would be written like (actually written in C ) precode class Module def attr_reader(*syms) syms.each do |sym| class_eval %{ def #{sym} @#{sym} end } end end /code/pre

Speaker goes through several implementations over time of different ways different people did metaprogramming with ruby.

How to think about metaprogramming

  • Definiting new constructs for your programming Language
    • so what do the constructs to? whatever you domain needs it to do.

Another way to think about metaprogramming is a new set of conceptual tools for eliminating duplication (and other smells) from your code.

And another way to think about it is how rails does it – almost as if you can talk you code – PersonTable has_a :name

Most DSLs also deal with other things ou don’t usually find in general-purpose languages

  • Context dependence
  • commands and sentences
  • Units
  • Large vocabularies
  • Heirachy

Contexts – context for a new set of statements – a new scope (not in 1.8, but in 1.9) precode Struct.new(“Interval”, :start, :end) do def length @start – @end end end /code/pre

Backend code looks like if you wanted to add it to 1.8 precode class Struct initialize(*args, block) struct_class = #define struct using args struct_class_class_eval(block) if block_given? end end /code/pre

Another example of context from Systir system testing tool

precode add_user { name “Charles” password “secret” priviliges normal } /code/pre

Commands and Sentences

Multipart complex statements

ex. field(autoinc, :reg_id, pk) Overall, it’s just a methodcall – the first parameter – the type – is a method call precode def autoinc return FieldType::AutoInc.instance end /code/pre

Units

Domain specific – general purpose language deals with scalars – programs must maintain their knowledge ex 3.days.from_now Watch out for operator overloading

precode class Numeric def days self * 60 end end /code/pre

Large Vocabularies

override method_missing

Usage: Roman.XXII Roman.CCIX precode class Roman def self.missing_method(method_id) str = method_id.id2name roman_to_int(str) end def roman_to_int(string) … end end /code/pre

Resources:

http://www.vanderburg.org/Speaking/Stuff/oscon05.pdf http://hypermetrics.com/rubyhacker/coralbook/

OCON2005 - the Latest and Craftiest Attacks and Penetration Techniques and Tools

by Nitesh Dhanjani

Closed source tools not good for monitoring your own networks. Lots of times they give false positives and there is no way to verify whether the positive is correct or not.

  • Methodology
  • finding vulnerabilities using Google.com
  • Using the Nessus framework
  • Web application vulnerabilities
  • Other useful AP tools

Attack and penetration methodology

  • Discovery (whois, traceroute, search engines, etc)
  • Scanning (ping sweeping, port scanning – find the live computers and scan)
  • Enumeration (service Identification, banner grabbing)
  • Exploiting known vulnerabilities (research on internet)
  • Installing rootkits/cleaning logs (ador on linux)

Googling for Vulnerabilities

  • Find private information that inadvertently have been made public
  • Stealth: find info on google – does not tell site you are looking at them (especially w/ google cache)
  • looking for patterns /*/admin
  • looking for error messages
  • find vnc desktops “VNC Desktop” inurl:5800 – no username – only passwords – user password generators to crack
  • Webcams – /view/view.shtml axis /ViewerFrame?Mode=Motion /home/homeJ.html sample/LvAppl/ etc

Go to oreilly.net and search for his name for article on how to do this.

The Nessus Framework

  • utomated vulnerability scanner
  • Opensource
  • Client Server Arch
  • Extend (write plugins) using NASL (Nessus Attack Scripting Language)

Writing a simple NASL Plugin
Web application serves /src/passwd.inc

This file contains username and passwords

the plugin will scan for this vulnerability and report it as a security whole (severe)
precode
if(description)
BLOCK
script_category (ACT_GATHER_INFO)
script_family (english:“CGI abuses”)
script_copyright(english:“foo bar baz”)

include (“http_func.inc”); port = get_http_port(default:80) if (… vunerability…)

report it
end
/code/pre

SQL Injection

causes: Dynamic SQL and lack of input validation

Authors preference for SQL injection prevention are stored procedures.

There is also Blind SQL Injection. Does not rely on verbose SQL error messages. Attempt to fetch database data. Check out Absinthe ( http://0×90.org/ )

Cross Site Scripting (XSS)

  • Occurrs when a webapplication does not html output encode user supplied data
  • Example http://example.com/cgi-bin/welcome?cgi?text=lt;scriptgt;alert(document.cookie);lt;/scriptgt;
  • Replace above example with any JS Code (steal cookies, hijack users sessions)

Burp Proxy

  • Java based HTTP web proxy
  • Alter http GET and POST requests on the fly
  • get it from http://

Other tools -

  • Metasploit (point, click,root) Framework for developing and testing exploit code – http://metasploit.com
  • Wikto: automated google and webserver vulnerability scanning (and much more)
  • http://sensepost.com/research/wikto/
  • Ettercap Network MITM attacks, content filtering, sniffing, etc http://ettercap.sourceforge.net/
  • Whax – live linux distro – Most AP tools/exploits included – http://iwhax.{com/net/org}?
  • good book is Network Security Tools – O’Reilly

Update Doh! Guess what – typo does not escape the content of a blog post. So when I posted the straight text, I started getting javascript popups with my session id an name from the above javascript code – updated to escape with lt gt symbols.

Customizing Mac OSX Using Opensource

by Jordan Hubbard + Kevin Van Vechten

recompiling software which come pre-built w/ osx

Darwin is the os core of osx. Includes kernel, IOKit families, and BSD commands and libraries.

Darwin source – developer.apple.com/darwin || darwinsource.opendarwing.org
.tar.gz snapshots organized by release.

OpenDarwin – community run site w/ standard opensource tools (bugzilla, cvs, irc, etc)

Webkit – webkit.opendarwind.org. Based on khtml/kjs
can progress live

DarwinPorts – similar to FreeBSD ports system. 2,500+ ports. Easy customization wiht “Variants”

Fink – Based on debian packages. 5K+ ports – offers pre-built packages.

Building Darwin

  • know objective
  • only replace system when necessary
  • beware of software updates (have to re-apply mods)
  • make backups

Potential Problems

  • Default compiler problems (different gcc version)
  • environment variables
  • build aliases
  • no private headers
  • no static libraries

user __gcc_select__ to change and report compiler versions

environment variables (where system looks for some things)

  • SRCROOT (src files)
  • OBJROOT (object files)
  • SYSROOT (debug bin)
  • DSTROOT (final bin)

Variables (compile)

  • MACOSX_DEPLOYMENT_TARGET (10.4)
  • RC_RELEASE (Tiger)
  • UNAME_RELEASE (8.0)
  • RC_ARCH (ppci386)
  • RC_ProjectName (name of project being built)

And many more compile time variables

Missing header files – private headers (ex. /usr/local/streams/streams.h /usr/local/lib/libkld.a)

No internal tools /usrl/local/kextool? – for building kernel modules

DarwinBuild – handles all the above incompatibilities

precode

  1. darwinbuild -init AC46 (ac46 is engineering build number)
  2. darwinbuild project_name
  3. darwinbuild bash
    /code/pre
    This will download sources build (missed point in which you could patch source) and install new binary

OSCON 2005 - Yield to the Block: The Power of Blocks in Ruby

with Matz (ruby’s father)

Presentation Slides

Interesting presentation. Valuable information, but slides moved really quickly.

One of ruby’s biggest strengths is blocks.

Increasing in popularity. 65 attendees at rubyconf in 2004 – over 200 pre-registrations for rubyconf in 2005. Reflects growing popularity of ruby.

Why? Hacker preference and killer application

Rails brought ruby to the limelight.

Blocks are unique and powerful in ruby. You can think of blocks as high order function

Martin Fowler reference is a good intoduction to closures and blocks.

Most of the rest of the presentation were code examples which Matz went over very quickly.

Quote from Matz

You (the audience) should have better japanese than I have english.

OSCON 2005 - Thursday Keynote

Arrived about 15 minutes late…

Transforming Software Architecture into Internetwork Architecture

  • Not tied to any platform
  • Extensible
  • Generic – general purpose
  • Federated
  • Identifier, Formats, and Protocol Standards

Internetwork Architecture of Global trade mirrors Internet/Sofware architecture. Standard containers and standard port protocol.

DHH

Secrets behind Ruby on Rails (same talk given at FOSCON)

Ruby on rails is an integrated stack of integrated frameworks. Ships with O/R mapper, MVC controller, etc. In other words a bunch of stuff that makes web developers happy!

Has had ~100k downloads in the last year.

Has created an ecosystem of people either partly or fully earning a living from developing w/ Rails.

250+ professionals from 36 countries. More than 5K+ weblog posts says Technorati. First book already has 6K+ orders. 7 more books coming. Definite interest and buzz around Rails.

Why is rails interesting?

  • convention over configuration (no xml situps – how many time do you have to tell the computer to do the same thing?)
    • As long as you follow the rails conventions, you don’t really have to do any configuration. But if you need to step out of the convention, rails supports it.

precode
class Post ActiveRecord::Base
belongs_to :weblog
has_many :comments
has_one :author, :class = Person
end
/code/pre
For example, there is no Author object in the database, so you can override the default and map it to Person (has_one line above)

  • Change is instant. Goal for rails was instant changes. Make a code or db change, refresh webbrowser, and you see your changes. No redeployment, no re-compilation, just save and referesh browser.
    • This is built in functionality of Ruby, not Rails. Ruby has Introspection, Open Classes, and you can execute code in class definitions.
  • Rails ships as a complete, integrated, and familiar stack. Gives you everything you need out of the box. Might seem like that reduces your freedom and flexibility but actually gives you more. Kinda like Apple computer. You buy the way they do things from hardware to software. Everything works and works well. That eliminates many of the problems with OS such as windows or linux and allows you to run and solve problems. (may have paraphrased the Apple computer comparison from last night FOSCON)
  • Flexibility is overrated. Too many technologies are chasing technologies as if flexibility were free. Rails trumps flexibility and you get alot in return. Constraints are liberating. Don’t have to worry about all that infastructure and can solve problems.

HP w/ Linux (more vendor wind…)

and more wind, and more wind, and more wind touting how important opensource software is and how cool linux is. Thank you Mr. Obvious. Now talking about how much HP does for opensource. Thank you, that’s very valuable but I don’t think it earns you a spot as a keynote speaker.

Computational Origami

Origami is the japanese art of paperfolding. Decorative abstract stapes. The modern extenstion is sculptural art achieved by folding paper, usually folded from one piece of paper.

Showing picture of origami folded from one piece of paper – incredible. One sheet, no cuts, only folding. Wow!

So what changed in the world of origami that change the old way of using several sheets of paper to the artsy form of today? Mathmatics.

Can model on computer and translate to paper. Take a stickfigure, measure all the lengths and construct a set of equasions. Then can solve for the crease pattern. Has a program called TreeMaker available for download that aids with this.

Applications in the real world

Space telescopes, automotive applications to name a few. Speaker designed a lense for a sattelite/space telescope. Needed to compact it to put it into space. How did they get it on the shuttle? Origami creases to reduce size.

Airbags – used to design how an airbag flattens inside the steering wheel of a car.

Mitchell Baker Chief lizard wrangler

Portland U. Now distributes mozilla/firefox downloads – bouncer – distributes load for mozilla/firefox downloads. Portland U. is the “hub” for all the downloads.

Started a commercial (for profit) mozilla arm to pay ongoing development, be able to accept money from online ads, etc.

Lastly, Dick ? from Identity20.com (perhaps .org) was speaking. Very good speaker about and good presentation about the next generation of identity management and why stuff like Passport did not work.

FOSCON2005 @ FreeGeek

For anyone who was or was not able to attend OSCON for Ruby and Rails related things, FreeGeek (think through the PDXLUG) sponsored some of the ruby/rails speakers from OSCON to come down and give a talk. This was very cool.

I arrived a few minutes late and DHH was speaking about rails. Very cool, especially because it was not one of the talks he had given so far at OSCON (atleast that I had seen).

Also there was Vanerburg speaking about metaprogramming in Ruby. The material he presented was pretty interesting. His talk plus Dave Thomas’ talk at the beginning of the week really gets me excited about Ruby. WOW! It is such an incredible language. it has so much more to offer than what you can get out of the Pickaxe book (Book is excellent, just does not push all the boundaries of Ruby. That may have require another 500 pages.)

West spoke about recreating the NextStep API for flash called ActiveStep and integrating it with Rails. Pretty interesting, although I did not get some of what he was talking about when it was flash specific. I have never done anything w/ flash. Very interesting.

And lastly there was White Lucky Stiff. They were slightly beyond description. Two guys and a girl showing homemade artsy computer animations, playing music and telling jokes. All material, w/ the exception of a couple of jokes, was about… Ruby. And most of it was funny. Definitely not politically correct, but that’s what made it so fun.

Met Robby Russell and talked to him for a bit. Great guy.

FreeGeek rocks. Turns out FreeGeek is a non-profit that recycles computer and teaches people how to build computers and run linux (job training) in the Portland area. Their location is very cool. Kinda like a computer garage. I was very comfortable there. It was a good fit.

Also in attendance was Matz – He did not give a presentation but accepted a gift from FreeGeek. A Japanese to English Dictionary.

Update Phil corrected me. Sorry for the misinformation. Rich Kilmer talked about the ActiveStep Flash API. Why the lucky stuff rather than “White Luck Stiff”. And Matz was given a dictionary of American idioms.

Don’t trust my memory!

OSCON 2005 - Extracting Rails From Basecamp

Last session of the day. More rails!

Extracting Rails from Basecamp

Basecamp came from 37 Signals.

  • Less people, more power
  • Less money, more value
  • Less resources, better use
  • Less time, better time

Trust – the best technical decisions come from technical people when it comes down to developing applications.

Start w/ least amount of resource and then start to scale.

Tried to build half a product rather than a half ass product.

Less Software

Approach to software engineering under constraints was to write less software.

So why another framework?

Experienced PHP background w/ some java.

Basecamp originally written in PHP. Hit wall with PHP – it fought back against what he wanted it to do.

Inspired by java but it was too much enterprise. Too much focus on the 99.999% and not enough on the 98%. Geard toward building the amazon.com of software, not the single developer hacking away.

So… Liberate the good ideas from the different languages. Ruby was the new place to liberate those ideas and make them not hurt.

Calls himself a shallow programmer

  • Aesthetics
  • Joy
  • Less

So halfway through Basecamp he realized this might be useful to other people. And now we need an extraction.

Doesn’t work to design a framework before you start coding. That’s the cart before the horse. You are then limited to what you designed.

Need application driven development to make good frameworks.

So why opensource it.

  • Selfish. Let other people do some of the work. Write it youself or let some one else help you. Only works when there are selfish reasons on the other end (they gain something too)
  • (and two other reasons)

And it works. 1000 patches in 9 months. Extract, pass, reap, rinse, repeat.

Now need to get traction for your opensource project. If nobody is looking, it does not matter.

Greatest fear as an opensource developer is obscurity. Need a network to reap the effects.

Opensource programmers need to get rid of their academic humbleness and bring rave about the good stuff. If you are not passionate, nobody else will. Need to set a baseline of excitement. Passion is infectious. You decide how high to set the bar. David set the bar very high!

Self delusions work!

Go Looking for Trouble. Tout advantages over the known. “If it bleeds, it leads”

Dealing with Traction

Now you need to scale your culture

  • Early influx can bend you out of shape.
  • Release not so early, then often. (get it mostly like you want it then release and release often. That way there is less debate about the way things should work)
  • Set a viral example of kindness. Care and show direction for newbies. Don’t tell anyone to RTFM. That will drive your users off and set and example for your culture.

Items will be added to rails by necessity. Example i18n. Not many people using it. Some people have tried it, but once enough people need and are using i18n, it will appear almost automagically and be useful.

Best Hacker of the Year

Congrats to David Heinemeier Hansson for the O’Reilly/Google best hacker of the year

OSCON 2005 - State of the Opensource Databases

Ingres _ by CA_

Ingres r3 is the version which was put into opensource.

Features

  • Value based table partitioning (a-c go here d-x go here, etc)
    • can partition tables based on value of records
    • No application change required
  • parallel query execution
    • may utlize more than one cpu on MP machines for a single query.
    • single processor machines also benefit
  • Advanced query optimization techniques (query decomposition, flattening, and rewrite)
  • Federation through ingres star
    • support two phase commits for distributed transactions
  • Replication of master at one or more slaves
    • can also have multiple masters where the other master acts as a slave when doing updates to one or the other
  • next release to support grids
  • admittly driver support is lacking
  • winblows only db gui

MySQL with David and Monty

  • runs on 87 gazillion platforms
  • Connector/MXJ – embedded jdbc within java (embedded java database???)
  • Storage engine slides…
  • Special storage engines
    • Archive – logging data you don’t want to delete or update
    • Blackhole – allows replication but throws away all data
    • MyISAM for logging: prefix locking allows fully concurrent inserts and reads

MySQL 5.0

<ul>
    <li>Stored procedures</li>
    <li>Triggers</li>
    <li>Views</li>
    <li>XA &#8211; distributed commits across different databases</li>
    <li>Data dictionary</li>
    <li>Server side cursors</li>
    <li>Precision Math &#8211; exact calculations with well defined rounding and atleast 56 digits precision.  Very fast w/ static memory allocation</li>
    <li>Strict mode &#8211; ability to get rollback/errors instead of closest value/warning messages (eh? can tell you when it is a bad date, 02/31/2005)</li>
    <li>Federated storage engine</li>
    <li>greedy optimizer (fast multi table joins)</li>
    <li>instance manager (replaces mysqld_safe script)</li>
    <li>extenstion to <span class="caps">LOAD</span> <span class="caps">DATA</span> for doing transformations/calculations at time of load</li>
    <li>5.0 still has 327 bugs but 16 really bad bugs at time of this writing.</li>
</ul><ul>
<li>Upcoming features
<ul>
    <li>partitioning (needed for 20 petabytes that one user is planning)</li>
    <li>replication additions &#8211; row based (physical) replication (normal is logical) + multi source replication</li>
    <li>global backup api</li>
    <li>mysql cluster w/ disk data (non indexed columns)</li>
    <li>Hash  Merge joins</li>
</ul></li>

Firebird w/ Ann ?

  • High concurrency, high performance
  • low administration costs
  • flexible architecture
  • active project

Development seems to be fairly slow going. Borland reniged the opensource license and Firebird is a fork. First releases were mostly bug fixes. 1.0 and 1.5 can share same data file.

Firebird Strengths

<ul>
    <li>Flexible architecture</li>
    <li>processed based (fork)</li>
    <li>multi-threaded shared server</li>
    <li>embedded</li>
    <li>low admin cost
    <ul>
        <li>self balancing indexes</li>
        <li>cooperative garbage collection</li>
        <li>single file database</li>
        <li>transportable backup</li>
    </ul></li>
</ul><p><i>Firebird future (2.0)</i></p>
  • currently in alpha 3
  • global temporary tables
  • execute block
  • physical backup

Vulcan

  • in parallel development
  • fork to work on 64bit arch
  • internal sql

Postgresql 8.1

8.1 features are frozen now. beta release in mid-august or early september.

Key New Features:

  • Indexes combined using bitmaps – index any attributes you want to query on later. then when you do a join across different attributes, the engine bitmaps the indexes and can perform very fast lookups (knows where to find pages and other data structures)
  • Two-phase commit for distributed transactions
  • Automatic vaccuum process
  • Global shared buffer lock removed, improves SMP support
  • Functions returning multiple out parameters (oracle feature)
  • Shared row locks

Pervasive, Green Plum and EnterpriseDB and Unisys building products/businesses around Postgres. Approx 1.5 million downloads of 8.0.×. and lots of new users (notable NWS and Ameritrade)

OSCON 2005 - Webwork vs. Spring Spackdown

Matt Raible and Matthew Porter

Not much to report here. Mostly a bitch session about what’s wrong w/ java frameworks. I could not help but feel sorry after using rails for the last couple of months.

Matt and Matthew are both good presenters. Good light presentation for the day.