Extending Visual Studio 2010 to Support Additional Programming Languages

Got a new language? Or maybe an old language that you're tired of writing with nothing but a similarly ancient text editor? By writing a Visual Studio® 2010 extension, you can bring that language into the Visual Studio IDE as a supported content type—and bring editor features such as syntax highlighting and IntelliSense® to the IDE for your language.

Visual Studio's language support includes these features and a number of other powerful tools long enjoyed by Visual C#® and Visual Basic® developers. In this article we'll concentrate on supporting four key editor functions:

• Syntax Highlighting Code is color-coded according to language classification.
• Error Tags Terms that are associated with syntax errors are marked with a red squiggle.
• Quick Info Pop-up tooltips provide help text when the cursor hovers over a term.
• Statement Completion The editor suggests a list of valid completions as the user types.

In Visual Studio 2010, you can support these capabilities with a series of MEF (Managed Extensibility Framework) component parts that plug into the editor. MEF is a general-purpose extension framework, supported by the new editor in Visual Studio 2010, in which components export and import parts from other components based on contracts. MEF, along with WPF formatting, changes the way you implement extensions in Visual Studio 2010.

To illustrate the process of adding language support to Visual Studio 2010, we'll walk through the code from the Ook! language integration sample. Ook! is an example language that consists of just three valid tokens separated by spaces (namely, "Ook!," "Ook.," and "Ook?"). The example was created by Chris Granger and the code is available from MSDN Code Gallery. For those interested, Ook is an esoteric but Turing-complete language. The complete language definition is available here. In this introduction to extending the editor for a new language, we will focus on Ook!, but you can also find more extensive example code in the IronPython Integration sample. In what follows, we'll talk a lot about different parts of the Ook! code sample. You'll need to download the code in order to follow along.

Figure 1. The Visual Studio editor, made Ook!-aware by the sample extension. The screenshot shows syntax highlighting, error tags, and statement completion for Ook!.

Tag-Based

The four features we're extending here fall into two categories. Syntax highlighting and error tags are both based on tagging, while Quick Info and statement completion are both aspects of IntelliSense. We'll work with the IntelliSense features a little further on; for now, let's focus on tagging.

In the context of the Visual Studio editor, "tags" are data attributes that are associated with spans of text. Error tags and classification tags (used for syntax highlighting, among other things) are both tag types that are directly supported by the editor. To provide syntax highlighting for a new language, you need to create a tagger class that will apply classification tags to text spans, and then define a visual presentation for these classifications. For error tagging, you need a tagger that will apply the pre-defined error tag type to errors in the text. Text with the error tag type is displayed with a red squiggle underline.

So we need two taggers, one to tag text spans with classification tags, and a second to tag errors. For Ook!, and probably for most real languages, too, it makes sense to run the raw text through a lexer before attempting to apply classification or error tags. But we don't want to duplicate this common tokenization behavior in both taggers.

Fortunately, this is MEF, and MEF is all about composability. We'll define a common tagging service that will provide token tags, and then run the output of the token tagger through both the classification tagger and the error tagger.

Figure 2. Simplified interaction between Ook! taggers and the Visual Studio 2010 editor. The extension's Token Tagger supplies the editor with spans of text tagged according to their Ook! token type. The Classification Tagger and Error Tagger then consume those token tags through a tag aggregator obtained from the editor.

All taggers need at least an implementation of ITagger, which will supply tags through its GetTags() method, and an implementation of ITaggerProvider, which is the MEF component part that provides the tagger to Visual Studio. This is Ook!'s token tag provider:

[Export(typeof(ITaggerProvider))]
[ContentType("Ook!")]
[TagType(typeof(OokTokenTag))]
internal sealed class OokTokenTagProvider : ITaggerProvider
{
public ITagger<T> CreateTagger<T>(ITextBuffer buffer) where T : ITag
{
return new OokTokenTagger(buffer) as ITagger<T>;
}
}

The Export, ContentType, and TagType attributes tell Visual Studio that the token tagger provider is available for spans of the Ook! content type. (The "Ook!" content type is defined and registered by exporting a ContentTypeDefinition, and associated with the ".ook" extension through the export of a FileExtensionToContentTypeDefinition.)

The main part of the token tagger itself is the GetTags() method, which returns all the tags for a given set of text spans. GetTags returns TagSpan<T>s, which associate a tag of a given type with a text span.In its GetTags() implementation, the Ook! token tagger identifies Ook!'s three tokens and returns a TagSpan<T> for each occurrence, tagged with an OokTokenTag.

public IEnumerable<ITagSpan<OokTokenTag>>
GetTags(NormalizedSnapshotSpanCollection spans)
{
foreach (SnapshotSpan curSpan in spans)
{
ITextSnapshotLine containingLine = curSpan.Start.GetContainingLine();
int curLoc = containingLine.Start.Position;
string[] tokens = containingLine.GetText().ToLower().Split(' ');
foreach (string ookToken in tokens)
{
if (_ookTypes.ContainsKey(ookToken))
{
var tokenSpan = new SnapshotSpan(curSpan.Snapshot, new Span
(curLoc, ookToken.Length));
if( tokenSpan.IntersectsWith(curSpan) )
yield return new TagSpan<OokTokenTag>(tokenSpan,
new OokTokenTag(_ookTypes[ookToken]));
}
curLoc += ookToken.Length + 1;
}
}
}

In the code above, _ookTypes is a dictionary that maps strings (e.g., "ook?") to token types (e.g., OokQuestion). OokTokenTag is an ITag implementation that takes the token type in its constructor and stores it for later use by tag consumers.

Now we've got a token tagger that will mark legal Ook! tokens with Ook! tags. We can use the tag spans returned by this tagger in our classification and error taggers.

Classification for Syntax Highlighting

The classification tagger will be similar in structure to the token tagger. There will need to be a classifier provider that exports the tagger to Visual Studio, and a GetTags() implementation that returns tagged spans. The main differences are that the classification tagger will consume the tag spans output from the token tagger, and it will return tags of type ClassificationTag that can be associated with a presentation format.

The classifier provider needs to import two services from the framework:

[Import]
internal IBufferTagAggregatorFactoryService aggregatorFactory = null;
[Import]
internal IClassificationTypeRegistryService ClassificationTypeRegistry = null; 

The tag aggregator factory service allows the classifier provider to construct a tag aggregator for Ook! token types, which the classifier can use to read the tags supplied by the token tagger. The classification type registry service provides the mechanism for associating classification types with presentation formats.

Ook! exports three classification types, as you might expect. Here is one of the three definitions:

[Export(typeof(ClassificationTypeDefinition))]
[Name("ook.")]
internal static ClassificationTypeDefinition ookPeriod = null; 

It assigns a name ("ook.") to the classification type. That name is then used in the presentation format definition, which says that spans of this classification type should be displayed with an orange foreground.

[Export(typeof(EditorFormatDefinition))]
[ClassificationType(ClassificationTypeNames = "ook.")]
[Name("ook.")]
internal sealed class OokP : ClassificationFormatDefinition
{
public OokP()
{
this.ForegroundColor = Colors.Orange;
}
}

In its constructor, the OokClassifier creates a dictionary ("_ookTypes") that maps token types to named classification types, using the classification type registry. Its GetTags() implementation can now use the tag aggregator to get token tags and then provide matching classification tags with classification types from the _ookTypes dictionary.

public IEnumerable<ITagSpan<ClassificationTag>>
GetTags(NormalizedSnapshotSpanCollection spans)
{
foreach (var tagSpan in this._aggregator.GetTags(spans))
{
var tagSpans = tagSpan.Span.GetSpans(spans[0].Snapshot);
yield return
new TagSpan<ClassificationTag>(tagSpans[0],
new ClassificationTag(_ookTypes[tagSpan.Tag.type]));
}
}

When the string "Ook." appears in an Ook! buffer, the token tagger tags the span that contains the string with an OokTokenTag with type OokPeriod. The OokClassifier reads those tags and assigns a classification tag of the type "ook." to the span. Because the editor has a presentation format registered for that type, the span appears with the orange foreground as in the screenshot above.

Creating Squiggles

The Ook! sample project doesn't include error tagging, but because the project has been factored into a token tagger and classifier, it's easy enough to add the feature. The error tagger becomes another tagger, like the classifier, that consumes spans with token tags and returns spans with the ErrorTag type.

To keep things simple, we'll just say that any tokens that aren't one of the three recognized types are an error. To do that, we can create an additional token type ("OokOther") that the token service will use to tag any token that is not in the legal token list. Then the error tagger will map OokOther tokens to ErrorTags. In a more fully-developed language service, layers above the lexer (e.g., parser, semantic analyzer) would also contribute ErrorTags.

If we make this modification to the token service, the error tagger GetTags() implementation looks almost like the OokClassifier implementation, except that it provides spans with error tags:

public IEnumerable<ITagSpan<IErrorTag>> GetTags(NormalizedSnapshotSpanCollection spans)
{
foreach (var tagSpan in this._aggregator.GetTags(spans))
{
if (tagSpan.Tag.type == OokTokenTypes.OokOther)
{
var tagSpans = tagSpan.Span.GetSpans(spans[0].Snapshot);
yield return
new TagSpan<ClassificationTag>(tagSpans[0],
new ErrorTag(OokErrorType));
}
}
}

Spans tagged with ErrorTag are presented in the editor with the red squiggle underline.

Providing IntelliSense

IntelliSense features are implemented using a set of related IntelliSense components.

The main components that you'll work within the IntelliSense model are:

• IntelliSense Source The source object provides the actual content, such as the Quick Info tooltip text, or the completion list.
  
  
• IntelliSense Controller The controller manages the lifetime of the IntelliSense session.
  
  
• IntelliSense Session The session object represents the active IntelliSense process, generally triggered in response to a user action on a trigger point in the buffer.
  
  
• IntelliSense Presenter The presenter determines the presentation of content in the editor.
  
  
• IntelliSense Broker The broker is imported from the framework and manages the interaction between the other component types.

At a high level, the IntelliSense process works like this: Your extension provides a controller to Visual Studio to associate with each view in the editor. The controller listens for relevant events in the view, and starts an IntelliSense session in response to those events. The session asks the source object for the actual content, and displays this content in a format appropriate for the content type.

Quick Info Tool Tips

For Quick Info, the controller listens for mouse hover events and the source provides content that can be displayed in the tool tip. The Ook! Quick Info source uses the token tagger we described earlier. If the mouse is hovering in a span tagged with the exclamation token type, the Quick Info source provides a string ("Exclaimed Ook!") as the tool tip content.

The Ook! extension includes an implementation of IIntellisenseController and IQuickInfoSource. It also includes the corresponding providers for these objects, which are the MEF component parts that connect them to the rest of Visual Studio. We'll skip over the details of the providers here and instead concentrate on the core functionality of the controller and source objects.

The main job of the controller is to start up a Quick Info session in response to mouse hover events. Here is the event handler:

private void OnTextViewMouseHover(object sender, MouseHoverEventArgs e)
{
SnapshotPoint? point = this.GetMousePosition
(new SnapshotPoint(_textView.TextSnapshot, e.Position));
if (point != null)
{
ITrackingPoint triggerPoint = point.Value.Snapshot.CreateTrackingPoint
(point.Value.Position,
PointTrackingMode.Positive);
if (!_componentContext.QuickInfoBroker.IsQuickInfoActive(_textView))
{
_session = _componentContext.QuickInfoBroker.CreateQuickInfoSession
(_textView, triggerPoint, true);
_session.Start();
}
}
}

The handler translates the mouse point to a point in the current buffer. It then finds the IntelliSense broker for the view and uses the broker to start a new Quick Info session if there isn't already an active session.

You don't need to implement a session object for Quick Info; there's a built-in QuickInfoSession object that you should use for this purpose. The standard QuickInfoSession will locate Quick Info sources that are registered for the content type of the buffer.

The Quick Info source provides content in its AugmentQuickInfoSession method. Ook!'s implementation uses the token tagger to determine what content should be shown for the trigger point. The main part of AugmentQuickInfoSession for OokQuickInfoSource is a loop that adds content only if the span has the specified token tag.

foreach (IMappingTagSpan<OokTokenTag> curTag in _aggregator.GetTags
(new SnapshotSpan(triggerPoint, triggerPoint)))
{
if (curTag.Tag.type == OokTokenTypes.OokExclaimation)
{
var tagSpan = curTag.Span.GetSpans(_buffer).First();
applicableToSpan = _buffer.CurrentSnapshot.CreateTrackingSpan
(tagSpan, SpanTrackingMode.EdgeExclusive);
quickInfoContent.Add("Exclaimed Ook!");
}
}

There are a few items worth noting in the code shown above. First, applicableToSpan is returned from the method that tells the editor the span for which the tool tip should be active. The session will end when the mouse cursor leaves the applicable span. Second, although we've added simple text content here, you can add WPF graphical content to provide styled text or other presentation effects.

IntelliSense Statement Completion

To handle statement completion, we need to trigger the completion session in response to keystrokes and other types of commands. For this reason, we use a CommandFilter class (an implementation of IOleCommandTarget) as the completion controller. When the command filter receives a command that should initiate a completion session, it uses the completion broker to create a new session and then starts it up. The completion broker is imported from Visual Studio when the controller is created for a view.

The CommandFilter's Exec() method contains a big switch statement that determines what completion action should be taken in response to commands. A few examples should help to illustrate how the Exec() method works.

case VSConstants.VSStd2KCmdID.COMPLETEWORD:
handled = StartSession();
break;
case VSConstants.VSStd2KCmdID.TAB:
handled = Complete(true);
break;
case VSConstants.VSStd2KCmdID.CANCEL:
handled = Cancel();
break; 

StartSession(), Complete(), and Cancel() are methods that actually provide the IntelliSense completion behavior. StartSession(), for example, uses the completion broker to find the active session or to create a new one. In response to COMPLETEWORD (usually mapped to Ctrl-Space), the controller starts a completion session. In response to TAB or CANCEL, the controller commits or dismisses the completion session.

The Exec() method handles normal character keys like this:

case VSConstants.VSStd2KCmdID.TYPECHAR:
char ch = GetTypeChar(pvaIn);
if (ch == ' ')
StartSession();
else if (_currentSession != null)
Filter();
break; 

If the user types a space (the token separator), the Exec() method starts a new completion session. Otherwise, if there is an active session in progress, the controller filters the completion list based on the addition of the new character. The Filter() method uses the session object to update the completion set.

The last piece of the statement completion picture is the completion source object. The completion source provides a CompletionSet in its AugmentCompletionSession method. The CompletionSet specifies a list of possible completions and a text span to which it applies. For a real language, you'd likely create this list dynamically, based on some documentation source. Ook! just creates a three-element list for every session with the three valid strings, and returns this along with the calculated text span:

var applicableTo = snapshot.CreateTrackingSpan
(new SnapshotSpan(start, triggerPoint), SpanTrackingMode.EdgeInclusive);
completionSets.Add
(new CompletionSet
("All", "All", applicableTo, completions, Enumerable.Empty<Completion>()));

Extensibility Through MEF

Visual Studio 2010's support of extensibility through MEF means you can create a variety of component parts to customize much of the editor's behavior. When integrating a new language into the Visual Studio IDE, you can package all the MEF parts that provide language-specific behaviors into an extension project.

Ook! is an example of such an extension project, one that provides classification, syntax highlighting, Quick Info, and statement completion. Whether you're creating an extension to deliver these capabilities or other language features, the techniques outlined above and detailed in the Ook! sources are a good starting point.

* This article was commissioned by and prepared for Microsoft Corporation. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.