cgo: support preprocessor macros passed on the command line

Go code might sometimes want to use preprocessor macros that were passed
on the command line. This wasn't working before and resulted in the
following error:

    internal error: could not find file where macro is defined

This is now supported, though location information isn't available
(which makes sense: the command line is not a file).

I had to use the `clang_tokenize` API for this and reconstruct the
original source location. Apparently this is the only way to do it:
https://stackoverflow.com/a/19074846/559350
In the future we could consider replacing our own tokenization with the
tokenizer that's built into Clang directly. This should reduce the
possibility of bugs a bit.

cgo/const.go

@@ -195,7 +195,9 @@ func (t *tokenizer) Next() {
 	t.curValue = t.peekValue
 
 	// Parse the next peek token.
-	t.peekPos += token.Pos(len(t.curValue))
+	if t.peekPos != token.NoPos {
+		t.peekPos += token.Pos(len(t.curValue))
+	}
 	for {
 		if len(t.buf) == 0 {
 			t.peekToken = token.EOF
@@ -207,7 +209,9 @@ func (t *tokenizer) Next() {
 			// Skip whitespace.
 			// Based on this source, not sure whether it represents C whitespace:
 			// https://en.cppreference.com/w/cpp/string/byte/isspace
-			t.peekPos++
+			if t.peekPos != token.NoPos {
+				t.peekPos++
+			}
 			t.buf = t.buf[1:]
 		case len(t.buf) >= 2 && (string(t.buf[:2]) == "||" || string(t.buf[:2]) == "&&" || string(t.buf[:2]) == "<<" || string(t.buf[:2]) == ">>"):
 			// Two-character tokens.