# RubyLit - This README is a program!

The output of this README.md file is a _program_ that turns documents into:

* <a href="html/README.rb.html">README.rb</a> - a Ruby program
* <a href="html/README.html.html">README.html</a> - a (hideously) formatted HTML document

It's a
<a href="https://en.wikipedia.org/wiki/Literate_programming">literate program</a>
(wikipedia.org)
and the concept comes from Donald Knuth.
(Update: I've written about what it _felt like_ to make this little program:
<a href="http://ratfactor.com/cards/literate-programming">literate-programming</a>.)

In order to get the process rolling, there's a non-literate `stage1.rb` script
that does the initial "tangling". You'll see what _that_ means in a moment.

To make this literate programming stuff work, all source is indented. That not
only makes it easy to read in source form, but by chosing that method,
I've also made this document a valid Markdown file, so the README will be
properly formatted when you view it on the Web as HTML output via tools such as
<a href="http://ratfactor.com/repos/reporat/">RepoRat</a>.

Here's how it starts. The program takes the root filename as a command
line argument:

    fname = ARGV[0]

I intially wanted to use the file extension ".lit", but the real magic happened
when I realized I could use ".md" and have the README itself be the program:

    fname_input = "#{fname}.md"

Next, I "tangle" and "weave" the input document:

    <<Tangle>>
    <<Weave>>

"Tangle" creates the program and "Weave" creates the documentation.
Those little `<<bracket things>>` are literate programming macros that
include source code from other sections of the document (identified by
Markdown subheadings).


## Tangle

What's neat about literate programming is that the source can be presented
in any order, so you can explain it however you like. The tangling process
puts it back into order so it can actually run.

The _full_ literate programming concept as imagined by Knuth and implemented
in his initial 'WEB' system not only allows you to include bits of code,
but even lets you define parametric macros, so the literate document is
actually a **meta-language** on top of the underlying programming language!

I've just implemented a crude "include" macro for this demonstration, but that
alone gives me a ton of flexibility!

Here's how I've made that work. I've got a `segments` hash that will store
all of the lines of a "segment" in the literate program. I have the program
begin in a segment identified with the `:start` symbol:

    myseg = :start
    segments = {}
    segments[myseg] = []

Then I loop through all of the lines in the source document and handle
just two special cases. (All other lines are treated as the "document"
part of the literate program and are completely ignored here!)

    File.open(fname_input).each do |line|
        <<Handle segments>>
        <<Handle code lines>>
    end

And after gathering the lines into segments, I recursively follow
the includes to write out the final program to a file:

    <<Write the program>>

## Handle segments

When I see a "## " at the beginning of the line, I know it's a Markdown
level 2 heading, which I'm using to indicate new code segments. They
don't _have_ to include code and even if they do, they don't _have_ to
be used.

Here you can see that I'm setting the current segment to the name of
the heading and initializing a new array to store the code lines:

      if(line.start_with?('## '))
        myseg = line[3..].chomp
        segments[myseg] = []
      end

## Handle code lines

When I see a space at the beginning of the line, I know it's indented source
code. This is extremely strict and not flexible and is just one example of the
non-industrial nature of this demonstration program. :-)

      if(line.start_with?(' '))
        segments[myseg].push(line)
      end

## Write the program

Here's the fun part! I've got this recursive method called `put_lines` that
takes an open destination file, the hash of named code segments (each segment
being an array of lines), and a target segment to print.

I'm looking at each line to see if it's a `<<macro thingy>>` (include request).
If it is, I recurse into the requested segment. Otherwise, I just output the
current line to the file:

    def put_lines(file, segments, sname)
      segments[sname].each do |line|

        if(m = /^\s*<<([^>]+)>>\s*$/.match(line))

          put_lines(file, segments, m[1])

        else

          file.puts line

        end

      end
    end

To start the above recursive process, I open the ".rb" output file and request
the `:start` segment:

    File.open("#{fname}.rb", 'w') do |out|
      put_lines(out, segments, :start)
    end

That's it!


## Weave

The "weave" part of the application is the documentation creation portion.

Since my scheme for this literate program is to encode it as pure Markdown,
I could just rely on an external tool to create the HTML (in fact, that's
probably how you're reading this README right now).

But since Ruby comes with a Markdown parser as part of it's Standard Library,
I figured I might as well include it. The parser and generator are part of
the RDoc (Ruby documentation) module:

    require 'rdoc'

The markdown source is the literate program document (yeah, we read it in
the Tangle process and we'll read it again for Weave):

    data = File.read(fname_input)

Then some boilerplate. RDoc is like a mini-Pandoc in that it can take input
and produce output in a bunch of different formats, and we pay for that
flexibility with some complexity:

    formatter = RDoc::Markup::ToHtml.new(RDoc::Options.new, nil)
    html = RDoc::Markdown.parse(data).accept(formatter)

And then I just write that out to a ".html" file, bookended by start and
end document tags:

    File.open("#{fname}.html", 'w') do |out|
      out.puts("<html><body>")
      out.print(html)
      out.puts("</body></html>")
    end

And that's it!

## Running it!

To turn this README into a program starts with "stage1", which only includes
the "tangle" part of the process (no documentation output):

    $ ruby stage1.rb README
    Found segment 'Tangle'
    Found segment 'Handle segments'
    Found segment 'Handle code lines'
    Found segment 'Write the program'
    Found segment 'Weave'
    Found segment 'Comparing this document with its output'
    Fetching segment 'start'
    Fetching segment 'Tangle'
    Fetching segment 'Handle segments'
    Fetching segment 'Handle code lines'
    Fetching segment 'Write the program'
    Fetching segment 'Weave'

(As you can see, I also gave it some output to help me debug segment names.)

That produces `README.rb`, which is now the Ruby program we've described
above, which can be used to process itself again:

    ruby README.rb README

And running that _again_ proves that we're **fully bootstrapped**. We're
running the output of the README against the README:

    ruby README.rb README

(Note that this part of the document you're reading right now has indented
"code" blocks to show the command line and output. Those are not valid Ruby, so
why is that okay? That's okay because they're never explicitly included in the
program!  The Ruby interpreter never sees them.)

## Bootstrapping

This repo includes two simple literate test programs (Markdown files) I used to
get the intial "stage1" program working:

* `hello.md`
* `hello-segments.md`

When stage1 was done, I copied it to use as the basis for the final
document/program you're reading now. Then I followed the "Running it!" process
exactly as shown above and it worked! :-)

## A batch processing extension

Parker Glynn-Adey has extended this program to support the exporting of
segments of a literate source file as separate output files...which, in turn,
can be literate programs in their own right! Woah.

See `README-BATCH.md` in this repository for that example.