There are all sorts of different inline tags, and I haven't found a place that explains them all in one place, so here is the quick and dirty...

*UDPATED 2012-02-22: Thanks to Conrad Buck and Austin for updates!

<% ... %>

The most basic inline tag, basically runs normal code:

<% if (User.IsInRole("admin")) { %>
    You can see this
<% } else { %>
    You are no admin fool!
<%} %>

http://msdn2.microsoft.com/en-us/library/ms178135(vs.80).aspx

<%= ... %>

Used for small chunks of information, usually from objects and single pieces of information like a single string or int variable:

The Date is now <%= DateTime.Now.ToShortDateString() %>
The value of string1 is <%= string1 %>

http://msdn2.microsoft.com/en-us/library/6dwsdcf5(VS.71).aspx

*note: <%= is the equivalent of Response.Write() - Courtesy of Adam from the US,thanks!

<%# .. %>

Used for Binding Expressions; such as Eval and Bind, most often found in data controls like GridView, Repeater, etc.:

<asp:Repeater ID="rptMeetings" DataSourceID="meetings" 
    runat="server">
    <ItemTemplate>
        <%# Eval("MeetingName")%>
    </ItemTemplate>
</asp:Repeater>

http://msdn2.microsoft.com/en-us/library/ms178366.aspx

<%$ ... %>

Used for expressions, not code; often seen with DataSources:

<asp:SqlDataSource ID="party" runat="server" 
    ConnectionString="<%$ ConnectionStrings:letsParty %>" 
    SelectCommand="SELECT * FROM [table]"
/>

http://msdn2.microsoft.com/en-us/library/d5bd1tad.aspx

<%@ ... %>

This is for directive syntax; basically the stuff you see at the top your your aspx pages like control registration and page declaration:

<%@ Page Language="C#" 
    MasterPageFile="~/MasterPage.master" 
    AutoEventWireup="true" CodeFile="Default.aspx.cs" 
    Inherits="_Default" Title="Untitled Page" %>
<%@ Register TagPrefix="wp" Namespace="CustomWebParts" %>

http://msdn2.microsoft.com/en-us/library/xz702w3e(VS.80).aspx

<%-- ... --%>

This is a server side comment, stuff you don't want anyone without code access to see:

<asp:Label ID="lblAwesome" runat="server"/>
<%-- sometimes end users make me angry --%>

<asp:LinkButton ID="lbEdit" Text="Edit"
    OnClick="Edit_Click" runat="server" />

http://msdn2.microsoft.com/en-us/library/4acf8afk.aspx

<%: ... %>

This tag is identical to the "<%= ... %>" tag except that it auto-html-encodes the value within the tag. As Phil Haack said: I often tell people it’s <%= but with the = seen from the front.:

<%: Html.TextBox("FirstName") %>

http://haacked.com/archive/2009/09/25/html-encoding-code-nuggets.aspx

And that's that.

inline asp.net tags... sorting them all out (<%$, <%=, <%, <%#, etc.)