There's no such thing as a dumb question! The documentation for the HTML visualizer can be found here: http://www.dotnetnuke.com/Support/Help/tabid/787/TID/3025/cid/416/Default.aspx
However, it's simple enough to explain right here in the thread for ya :)
Basically, you provide an HTML file (unfortunately, at this point it must be a file, you can't just type in HTML). The Reports module will use that template for every row in the query results and inject a chunk of HTML for each row, one after the other.
The template is just plain HTML, except that it contains little tokens of the form: "[ColumnName]", where "ColumnName" is a column in your query results. The module will replace that whole token with the value of that column for the specified row.
While the module will automatically loop through the record set and inject a copy of the HTML for each row, it does not (at this point) have a header or footer feature. If you want something more complex than simple HTML injection, you'd have to use XSLT. At this point, the HTML visualizer is a little limited, but the XSLT Visualizer can achieve almost all of the more complex scenarios that the HTML Visualizer can't at the moment (though it is, unfortunately, a little more complex :( ).