I'm toying with the idea of providing some kind of Git access to resources used by a SaaS application to it's users. This will allow users to edit and push content through the Git interface as well as through the application's native web-based interface. My main concern is how to reconcile merge conflicts when a user edit's the content in the web-based interface (I'm not so concerned about merge conflicts from the Git interface since Git and their Git client should handle that). This is very similar to how GitHub allows both Git-based and web-based access to their Wikis, and I'm curious how they handle this situation as a pattern for others to follow when providing both web-based and Git-based access to content.

If a user goes to edit a Wiki page on GitHub through the web interface and another user pushes a change to the Wiki repository branch before they finish, what happens when they save their changes? Does it use a "last one wins" or a "sorry, redo all your changes on the new version"?

I found a related SO post that discussed the problem in general, but I'm very curious how GitHub specifically handles it since it's backed by Git which already has some merging capabilities baked in.

I've had the need for a simple ExpandoObject initializer several times before and typically use the following two extension methods to accomplish something like initializer syntax:

public static KeyValuePair<string, object> WithValue(this string key, object value)
{
    return new KeyValuePair<string, object>(key, value);
}

public static ExpandoObject Init(
    this ExpandoObject expando, params KeyValuePair<string, object>[] values)
{
    foreach(KeyValuePair<string, object> kvp in values)
    {
        ((IDictionary<string, Object>)expando)[kvp.Key] = kvp.Value;
    }
    return expando;
}

Then you can write the following:

dynamic foo = new ExpandoObject().Init(
    "A".WithValue(true),
    "B".WithValue("Bar"));

In general I've found that having an extension method to build KeyValuePair<string, object> instances from a string key comes in handy. You can obviously change the name to something like Is so that you can write "Key".Is("Value") if you need the syntax to be even more terse.

I've been pondering this for a couple hours and I'm a little surprised at how challenging it is. I'm trying to initialize a static cache of reflection data for a given object from a base class that will be accessed from multiple threads. I'm having a hard time coming up with the correct pattern for initializing the cache.

My first thought was that I'll just initialize the static cache to null, check if it's null in the constructor, and then build and set it if it's not. I.e.:

class TestBase
{
  private static ConcurrentDictionary<string, PropertyInfo> Cache;

  protected TestBase()
  {
    if(Cache == null)
    {
      ConcurrentDictionary<string, PropertyInfo> cache =
        new ConcurrentDictionary<string, PropertyInfo>();
      // Populate...
      Cache = cache;
    }
  }
}

This has a flaw in that if I construct another object while the first one is still populating the cache, I'll end up constructing two caches and the second will (presumably, though not always) overwrite the first. This is probably okay since they'll both be complete caches, but it seems hacky and I would hope we can do better.

So my second thought was to initialize the cache in a static constructor which only gets called once per AppDomain before any instances are created. StackOverflow seems to have several answers to similar questions that point in this direction. This seemed great until I realized that the static constructor won't have access to the reflection data for the derived type.

I could always synchronize access in the constructor to insure only one thread is creating/populating the cache and any other access while that's happening should block, but then I'm locking on every construction just to protect an operation that should only happen once. I don't like the performance implications of that.

What I have right now is a flag that is set using Interlocked.Exchange and a ManualResetEventSlim. It looks like this:

class TestBase
{
  private static ConcurrentDictionary<string, PropertyInfo> Cache;
  private static volatile int BuildingCache = 0;
  private static ManualResetEventSlim CacheBuilt =
    new ManualResetEventSlim();

  protected TestBase()
  {
    if(Interlocked.Exchange(ref BuildingCache, 1) == 0)
    {
      Cache = new ConcurrentDictionary<string, PropertyInfo>();
      // Populate...
      CacheBuilt.Set();
    }
    CacheBuilt.Wait();
  }
}

I suspect that there might already be an accepted, or at least known, way of doing this kind of thing - is this it? If not, is there a better way to synchronize the cache initialization? Note that the question is not about how to make cache access thread-safe, that can be assumed by using ConcurrentDictionary (or similar).

I have noticed that while debugging my site, any request that results in a 404 and appears to refer to a path on the disk relative to my configured virtual directory is intercepted by Cassini and rudely replaced with a directory listing. I'm using Nancy Framework, but given that this problem appears to be at the web server level, I suspect Cassini would act the same way for MCV applications. I can't find any documentation on this "feature" other than a related commit message on the Cassini source that says "...directory listing only overrides 404 responses for directories".

I would much rather my development web server stop trying to outsmart my framework. I makes the debugging experience more than a little jarring. In an MVC framework the request URLs have nothing at all to do with file locations, so the fact I'm getting directory listings for some invalid requests and the correct 404 page for others gets annoying. Not to mention it's making several of my unit tests fail because they rely on auto-generated content in my 404 error pages (which I can't manually test either).

Is there any way to disable this functionality in Cassini? I know I could install IIS Express, but I'd rather not. Especially since my unit test runner and hosts file (this is a multi-domain application) are already configured just right.

Batch put and delete are available now (they call it BatchWriteItem): http://docs.amazonwebservices.com/amazondynamodb/latest/developerguide/API_BatchWriteItems.html

However, you still can not update multiple items in a single request.

I have an idea for a new web application and have been looking for tools to help me get a jump start. Essentially, I would like to provide a service to "my" users that would let them easily create a site that includes blogging and some other specialized functionality while allowing them to maintain their own users. The best analogy is that I'd like to create a site similar to WordPress.com (the service, not the software), TypePad, or Blogger but way simpler (I'm only one guy after all!) and designed for users in a very specific niche. I'm guessing this concept is general enough that answers should benefit others looking to do something similar.

Which brings me to my question: what tools exist to help develop multi-tenant, multi-user web applications? I would personally prefer something open source and written in .NET, but for the benefit of everyone else I think non-open source and/or non-.NET systems would also prove valuable. My first thought was to look for open source CMS systems that support multi-tenancy and have a robust extension mechanism. The most promising that I've found so far is Orchard. Has anyone used it (or something similar) for this kind of purpose?

In general, requirements for this type of application would probably be akin to:

• Multi-site/multi-tenant
  • Allow the creation of multiple sites that are fully customizable (such as themes) but that share a common set of overall features and functionality.
• Different types of users
  • Support for "master" users that can configure the entire ecosystem.
  • Support for "administrators" that have control over their own site and users, but not those of others.
  • Support for "users" that are tied to an individual site, possibly with multiple access levels or types (as defined by the site's administrator).
• Robust extensibility mechanisms
  • Since the whole point is to provide a hosting application for individual sites in a specific niche, the setup has to be highly extensible to support additional features, changes to the platform, etc.

(I asked this over at Web Applications, but got closed as off-topic. Hopefully SO is the right site this time. Forgive me if not - it can be difficult to determine the appropriate SE site sometimes).

If you think XML might fit the bill, my company just released an open source embedded XML database for the .NET platform called Nxdb. It's under the Apache 2.0 license and has been in development and use internally for several years. It's basically a binding to a cross-compiled (using IKVM) version of BaseX (a fantastic Java XML database) along with extra functionality for the embedded use case and the .NET environment. The project page is here: https://dracorp.assembla.com/spaces/nxdb

Deployment is fairly simple, though it does require a number of assemblies due to the way IKVM is linked and deployed (I think 10 or so at last count). The documentation is somewhat sparse, though there are some examples in the Wiki. We're working on adding more examples and better documentation right now (and are open to suggestions).

(Hopefully linking to my own project doesn't irk the mods, but I figured since this post directly answers the question and provides a link to a permissive OS project it would be okay).

This is an old question, but I thought I'd add an answer in case anyone stumbles on it. My company just released an open source embedded XML database for the .NET platform called Nxdb. It's under the Apache 2.0 license and has been in development and use internally for several years. It's basically a binding to a cross-compiled (using IKVM) version of BaseX (a fantastic Java XML database) along with extra functionality for the embedded use case and the .NET environment. The project page is here: https://dracorp.assembla.com/spaces/nxdb

(Hopefully linking to my own project doesn't irk the mods, but I figured since this post directly answers the question and provides a link to a permissive OS project it would be okay).

This is an old question, but I thought I'd add an answer in case anyone stumbles on it. My company just released an open source embedded XML database for the .NET platform called Nxdb. It's under the Apache 2.0 license and has been in development and use internally for several years. It's basically a binding to a cross-compiled (using IKVM) version of BaseX (a fantastic Java XML database) along with extra functionality for the embedded use case and the .NET environment. The project page is here: https://dracorp.assembla.com/spaces/nxdb

XML works well for this type of data store given that as long as the content you're trying to store is serializable to text you can store complex hierarchical trees. In fact, if you access the database directly, you don't ever even have to touch "XML". It can also be queried with XQuery, a powerful and complete query language.

(Hopefully linking to my own project doesn't irk the mods, but I figured since this post directly answers the question and provides a link to a permissive OS project it would be okay).

I'm noticing an odd behavior with the XmlSerializer and generic lists (specifically List<int>). I was wondering if anyone has seen this before or knows what's going on. It appears as though the serialization works fine but the deserialization wants to add extra items to the list. The code below demonstrates the problem.

Serializable class:

public class ListTest
{
    public int[] Array { get; set; }
    public List<int> List { get; set; }

    public ListTest()
    {
        Array = new[] {1, 2, 3, 4};
        List = new List<int>(Array);
    }
}

Test code:

ListTest listTest = new ListTest();
Debug.WriteLine("Initial Array: {0}", (object)String.Join(", ", listTest.Array));
Debug.WriteLine("Initial List: {0}", (object)String.Join(", ", listTest.List));

XmlSerializer serializer = new XmlSerializer(typeof(ListTest));
StringBuilder xml = new StringBuilder();
using(TextWriter writer = new StringWriter(xml))
{
    serializer.Serialize(writer, listTest);
}

Debug.WriteLine("XML: {0}", (object)xml.ToString());

using(TextReader reader = new StringReader(xml.ToString()))
{
    listTest = (ListTest) serializer.Deserialize(reader);
}

Debug.WriteLine("Deserialized Array: {0}", (object)String.Join(", ", listTest.Array));
Debug.WriteLine("Deserialized List: {0}", (object)String.Join(", ", listTest.List));

Debug output:

Initial Array: 1, 2, 3, 4
Initial List: 1, 2, 3, 4
XML: <?xml version="1.0" encoding="utf-16"?>
<ListTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Array>
    <int>1</int>
    <int>2</int>
    <int>3</int>
    <int>4</int>
  </Array>
  <List>
    <int>1</int>
    <int>2</int>
    <int>3</int>
    <int>4</int>
  </List>
</ListTest>
Deserialized Array: 1, 2, 3, 4
Deserialized List: 1, 2, 3, 4, 1, 2, 3, 4

Notice that both the array and list appear to have serialized to XML correctly but on deserialization the array comes out correct but the list comes back with a duplicate set of items. Any ideas?

I am having a very odd problem with local variables being out of context in the Visual Studio 2010 debugger for a C# console application targeting .NET 4.0. I've searched for other similar questions on SO, but while some have the same symptoms, none seem to apply directly to this problem (they all appear to have other root causes).

The problem is that for some variables (but not all) I do not get a tooltip with their value, they do not appear in the Locals window, and I get "The name 'xyz' does not exist in the current context" if I add them to the Watch window. It appears to affect some variables but not others, and I can't figure out a pattern (it doesn't seem to be based on member vs. local, class vs. struct, or any other differentiator). I've restarted my computer and Visual Studio, verified I'm in a clean Debug build, made sure the debugging frame is correct, made sure to refresh the variables in the watch screen, and attempted various spells and incantations.

I've included a screenshoot below (bigger version at http://i.stack.imgur.com/JTFBT.png).

Any thoughts?

EDIT:

Adding some additional information:

The problem is repeatable. The exact same variables either work or don't work, even after completely shutting down and restarting Visual Studio. This leads me to believe there's actually something systematic going wrong rather than just memory corruption or something.

I've also discovered that it appears to be related to the try-catch block. If I position the breakpoint outside the try statement I can see any of the in-scope variables normally. As soon as the execution point enters the try statement all the variables outside the try block become inaccessible and I can only access the ones inside the try statement. It's almost as though the debugger is treating the try block as a separate method (though you can see the code/compiler still does have access to in-scope variables). Has anyone seen this behavior before?

ANOTHER EDIT:

I (partially) take back what I said about the try-catch being suspect - it appears that in this portion of the code the debugger exhibits this odd taking stuff out of context for any enclosing block. For example, if I set a breakpoint directly inside the foreach statement in the screenshot I can see the "port" variable value on each iteration, but none of the variables outside the foreach statement (which disappear as soon as I enter the foreach block). Then as soon as you enter the try block, the "port" variable suddenly goes away. This is getting really weird.

Also, as requested, the code for the entire method is below.

private void ConfigureAnnouncerSockets(XDocument configDocument)
{
    XElement socketsElement = configDocument.XPathSelectElement("/Configuration/Network/AnnouncerSockets");
    bool useDefault = true;
    if (socketsElement != null)
    {
        //Use the default announcers? (they will be added at the end)
        XAttribute defaultAttribute = socketsElement.Attribute("useDefault");
        if (defaultAttribute != null)
        {
            useDefault = Convert.ToBoolean(defaultAttribute);
        }

        //Get the default frequency
        int defaultFrequency = Announcer.DefaultFrequency;
        XAttribute frequencyAttribute = socketsElement.Attribute("frequency");
        if (frequencyAttribute != null)
        {
            defaultFrequency = Convert.ToInt32(frequencyAttribute.Value);
        }

        //Get all sockets
        foreach (XElement socketElement in socketsElement.XPathSelectElements("./Socket"))
        {
            //Get the address
            IPAddress address = IPAddress.Broadcast;
            string addressAttribute = (string)socketElement.Attribute("address");
            if(!GetAddress(addressAttribute, ref address, true))
            {
                Intelliplex.Log.Warn("Invalid announcer socket address: " + addressAttribute);
                continue;
            }

            //Get the local address
            IPAddress localAddress = null;
            string localAddressAttribute = (string)socketElement.Attribute("localAddress");
            if(!GetAddress(localAddressAttribute, ref localAddress, false))
            {
                Intelliplex.Log.Warn("Invalid announcer socket local address: " + localAddressAttribute);
                continue;
            }

            //Get the port(s)
            List<int> ports = new List<int>();
            string[] ranges = ((string)socketElement.Attribute("port")).Split(new[] { ',' });
            foreach (string range in ranges)
            {
                string[] portPair = range.Split(new[] { '-' });
                int firstPort = Convert.ToInt32(portPair[0]);
                int lastPort = portPair.Length > 1 ? Convert.ToInt32(portPair[1]) : firstPort;
                do
                {
                    ports.Add(firstPort);
                } while (++firstPort <= lastPort);
            }

            //Get the local port
            int localPort = socketElement.Attribute("localPort") != null
                ? Convert.ToInt32((string)socketElement.Attribute("localPort")) : 0;

            //Get the frequency
            int frequency = socketElement.Attribute("frequency") != null
                ? Convert.ToInt32((string)socketElement.Attribute("frequency")) : defaultFrequency;

            //Create the socket(s) and add it/them to the manager
            foreach (int port in ports)
            {
                try
                {
                    IPEndPoint endPoint = new IPEndPoint(address, port);
                    IPEndPoint localEndPoint = localAddress == null
                        ? new IPEndPoint(IPAddress.Any, 0) : new IPEndPoint(localAddress, localPort);
                    Announcer socket = new Announcer(frequency, endPoint, localEndPoint);
                    AnnouncerSockets.Add(socket);
                }
                catch (Exception ex)
                {
                    Intelliplex.Log.Warn("Could not add announcer socket: " + ex.Message);
                }
            }
        }
    }

    //Add default announcement sockets?
    if (useDefault)
    {
        ConfigureDefaultAnnouncerSockets();
    }
}

So it turns out this is related to a bug in PostSharp. I had been using PostSharp but removed all aspects from my code and ensured that none were applied. I also verified with Reflector that the methods were intact in the assembly. However, it appears simply referencing PostSharp triggers some kind of manipulation of the debugging symbols that causes this problem. A (little) more information can be found here:

http://www.sharpcrafters.com/forum/Topic5794-21-1.aspx#bm7927

Also, in the release notes for the latest PostSharp hotfix states one of the fixed issues in hotfix 2.1.5.6 is "Debugging symbols: local variable symbols lost in implicit iterators."

When I installed the latest and greatest PostSharp the problem went away and the universe returned to normal. Hopefully this question/answer will help anyone else using PostSharp who stumbles on this odd behavior before the next official PostSharp release. Make sure you're on hotfix 2.1.5.6 or greater (given the severity of the bug, this probably should have been an actual release).

Thanks for all the help everyone.

It's been a long time since this question was posted, but since there's no answer and I stumbled here too, I figured I'd add one. I copied the same factory code you did (from the StackOverflow answer here) and had the same problem. I found the solution at this StackOverflow answer.

It turns out Visual Studio 2010 (which I'm assuming you're using) has a problem with std::make_pair. Just use std::pair<std::string,LevelObject*(*)()> instead and you'll be good to go. At least that resolved this exact same problem for me.

I am considering using Qt for developing a new XML-based application that makes heavy use of XQuery. I am especially excited about the way QtXmlPatterns apparently lets you query any data with an appropriate model as if it were XML. I've used XML databases and other query engines in the past (mainly Saxon and BaseX) and am wondering about the performance of XQuery under Qt. I've seen some older evidence that it's a little slow, but that's probably out of date by now.

Does anyone have any (recent) benchmarks comparing Qt XQuery to other XQuery engines. What about anecdotal evidence? Does the Qt XQuery engine allow the use of indexes or other ways to speed up queries through preprocessing (I couldn't find any)?

I am currently working on a network tool that needs to decode/encode a particular protocol that packs fields into dense bit arrays at arbitrary positions. For example, one part of the protocol uses 3 bytes to represent a number of different fields:

Bit Position(s)  Length (In Bits)    Type
0                1                   bool
1-5              5                   int
6-13             8                   int
14-22            9                   uint
23               1                   bool

As you can see, several of the fields span multiple bytes. Many (most) are also shorter than the built-in type that might be used to represent them, such as the first int field which is only 5 bits long. In these cases, the most significant bits of the target type (such as an Int32 or Int16) should be padded with 0 to make up the difference.

My problem is that I am having a difficult time processing this kind of data. Specifically, I am having a hard time figuring out how to efficiently get arbitrary length bit arrays, populate them with the appropriate bits from the source buffer, pad them to match the target type, and convert the padded bit arrays to the target type. In an ideal world, I would be able to take the byte[3] in the example above and call a method like GetInt32(byte[] bytes, int startBit, int length).

The closest thing in the wild that I've found is a BitStream class, but it appears to want individual values to line up on byte/word boundaries (and the half-streaming/half-indexed access convention of the class makes it a little confusing).

My own first attempt was to use the BitArray class, but that proved somewhat unwieldy. It's easy enough to stuff all the bits from the buffer into a large BitArray, transfer only the ones you want from the source BitArray to a new temporary BitArray, and then convert that into the target value...but it seems wrong, and very time consuming.

I am now considering a class like the following that references (or creates) a source/target byte[] buffer along with an offset and provides get and set methods for certain target types. The tricky part is that getting/setting values may span multiple bytes.

class BitField
{
    private readonly byte[] _bytes;
    private readonly int _offset;

    public BitField(byte[] bytes)
        : this(bytes, 0)
    {
    }

    public BitField(byte[] bytes, int offset)
    {
        _bytes = bytes;
        _offset = offset;
    }

    public BitField(int size)
        : this(new byte[size], 0)
    {
    }

    public bool this[int bit]
    {
        get { return IsSet(bit); }
        set { if (value) Set(bit); else Clear(bit); }
    }

    public bool IsSet(int bit)
    {
        return (_bytes[_offset + (bit / 8)] & (1 << (bit % 8))) != 0;
    }

    public void Set(int bit)
    {
        _bytes[_offset + (bit / 8)] |= unchecked((byte)(1 << (bit % 8)));
    }

    public void Clear(int bit)
    {
        _bytes[_offset + (bit / 8)] &= unchecked((byte)~(1 << (bit % 8)));
    }

    //startIndex = the index of the bit at which to start fetching the value
    //length = the number of bits to include - may be less than 32 in which case
    //the most significant bits of the target type should be padded with 0
    public int GetInt32(int startIndex, int length)
    {
        //NEED CODE HERE
    }

    //startIndex = the index of the bit at which to start storing the value
    //length = the number of bits to use, if less than the number of bits required
    //for the source type, precision may be lost
    //value = the value to store
    public void SetValue(int startIndex, int length, int value)
    {
        //NEED CODE HERE
    }

    //Other Get.../Set... methods go here
}

I am looking for any guidance in this area such as third-party libraries, algorithms for getting/setting values at arbitrary bit positions that span multiple bytes, feedback on my approach, etc. I included the class above for clarification and am not necessarily looking for code to fill it in (though I won't argue if someone wants to work it out!).

As promised, here is the class I ended up creating for this purpose. It will wrap an arbitrary byte array at an optionally specified index and allowing reading/writing at the bit level. It provides methods for reading/writing arbitrary blocks of bits from other byte arrays or for reading/writing primitive values with user-defined offsets and lengths. It works very well for my situation and solves the exact question I asked above. However, it does have a couple shortcomings. The first is that it is obviously not greatly documented - I just haven't had the time. The second is that there are no bounds or other checks. It also currently requires the MiscUtil library to provide endian conversion. All that said, hopefully this can help solve or serve as a starting point for someone else with a similar use case.

internal class BitField
{
    private readonly byte[] _bytes;
    private readonly int _offset;
    private EndianBitConverter _bitConverter = EndianBitConverter.Big;

    public BitField(byte[] bytes)
        : this(bytes, 0)
    {
    }

    //offset = the offset (in bytes) into the wrapped byte array
    public BitField(byte[] bytes, int offset)
    {
        _bytes = bytes;
        _offset = offset;
    }

    public BitField(int size)
        : this(new byte[size], 0)
    {
    }

    //fill == true = initially set all bits to 1
    public BitField(int size, bool fill)
        : this(new byte[size], 0)
    {
        if (!fill) return;
        for(int i = 0 ; i < size ; i++)
        {
            _bytes[i] = 0xff;
        }
    }

    public byte[] Bytes
    {
        get { return _bytes; }
    }

    public int Offset
    {
        get { return _offset; }
    }

    public EndianBitConverter BitConverter
    {
        get { return _bitConverter; }
        set { _bitConverter = value; }
    }

    public bool this[int bit]
    {
        get { return IsBitSet(bit); }
        set { if (value) SetBit(bit); else ClearBit(bit); }
    }

    public bool IsBitSet(int bit)
    {
        return (_bytes[_offset + (bit / 8)] & (1 << (7 - (bit % 8)))) != 0;
    }

    public void SetBit(int bit)
    {
        _bytes[_offset + (bit / 8)] |= unchecked((byte)(1 << (7 - (bit % 8))));
    }

    public void ClearBit(int bit)
    {
        _bytes[_offset + (bit / 8)] &= unchecked((byte)~(1 << (7 - (bit % 8))));
    }

    //index = the index of the source BitField at which to start getting bits
    //length = the number of bits to get
    //size = the total number of bytes required (0 for arbitrary length return array)
    //fill == true = set all padding bits to 1
    public byte[] GetBytes(int index, int length, int size, bool fill)
    {
        if(size == 0) size = (length + 7) / 8;
        BitField bitField = new BitField(size, fill);
        for(int s = index, d = (size * 8) - length ; s < index + length && d < (size * 8) ; s++, d++)
        {
            bitField[d] = IsBitSet(s);
        }
        return bitField._bytes;
    }

    public byte[] GetBytes(int index, int length, int size)
    {
        return GetBytes(index, length, size, false);
    }

    public byte[] GetBytes(int index, int length)
    {
        return GetBytes(index, length, 0, false);
    }

    //bytesIndex = the index (in bits) into the bytes array at which to start copying
    //index = the index (in bits) in this BitField at which to put the value
    //length = the number of bits to copy from the bytes array
    public void SetBytes(byte[] bytes, int bytesIndex, int index, int length)
    {
        BitField bitField = new BitField(bytes);
        for (int i = 0; i < length; i++)
        {
            this[index + i] = bitField[bytesIndex + i];
        }
    }

    public void SetBytes(byte[] bytes, int index, int length)
    {
        SetBytes(bytes, 0, index, length);
    }

    public void SetBytes(byte[] bytes, int index)
    {
        SetBytes(bytes, 0, index, bytes.Length * 8);
    }

    //UInt16

    //index = the index (in bits) at which to start getting the value
    //length = the number of bits to use for the value, if less than required the value is padded with 0
    public ushort GetUInt16(int index, int length)
    {
        return _bitConverter.ToUInt16(GetBytes(index, length, 2), 0);
    }

    public ushort GetUInt16(int index)
    {
        return GetUInt16(index, 16);
    }

    //valueIndex = the index (in bits) of the value at which to start copying
    //index = the index (in bits) in this BitField at which to put the value
    //length = the number of bits to copy from the value
    public void Set(ushort value, int valueIndex, int index, int length)
    {
        SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
    }

    public void Set(ushort value, int index)
    {
        Set(value, 0, index, 16);
    }

    //UInt32

    public uint GetUInt32(int index, int length)
    {
        return _bitConverter.ToUInt32(GetBytes(index, length, 4), 0);
    }

    public uint GetUInt32(int index)
    {
        return GetUInt32(index, 32);
    }

    public void Set(uint value, int valueIndex, int index, int length)
    {
        SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
    }

    public void Set(uint value, int index)
    {
        Set(value, 0, index, 32);
    }

    //UInt64

    public ulong GetUInt64(int index, int length)
    {
        return _bitConverter.ToUInt64(GetBytes(index, length, 8), 0);
    }

    public ulong GetUInt64(int index)
    {
        return GetUInt64(index, 64);
    }

    public void Set(ulong value, int valueIndex, int index, int length)
    {
        SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
    }

    public void Set(ulong value, int index)
    {
        Set(value, 0, index, 64);
    }

    //Int16

    public short GetInt16(int index, int length)
    {
        return _bitConverter.ToInt16(GetBytes(index, length, 2, IsBitSet(index)), 0);
    }

    public short GetInt16(int index)
    {
        return GetInt16(index, 16);
    }

    public void Set(short value, int valueIndex, int index, int length)
    {
        SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
    }

    public void Set(short value, int index)
    {
        Set(value, 0, index, 16);
    }

    //Int32

    public int GetInt32(int index, int length)
    {
        return _bitConverter.ToInt32(GetBytes(index, length, 4, IsBitSet(index)), 0);
    }

    public int GetInt32(int index)
    {
        return GetInt32(index, 32);
    }

    public void Set(int value, int valueIndex, int index, int length)
    {
        SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
    }

    public void Set(int value, int index)
    {
        Set(value, 0, index, 32);
    }

    //Int64

    public long GetInt64(int index, int length)
    {
        return _bitConverter.ToInt64(GetBytes(index, length, 8, IsBitSet(index)), 0);
    }

    public long GetInt64(int index)
    {
        return GetInt64(index, 64);
    }

    public void Set(long value, int valueIndex, int index, int length)
    {
        SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
    }

    public void Set(long value, int index)
    {
        Set(value, 0, index, 64);
    }

    //Char

    public char GetChar(int index, int length)
    {
        return _bitConverter.ToChar(GetBytes(index, length, 2), 0);
    }

    public char GetChar(int index)
    {
        return GetChar(index, 16);
    }

    public void Set(char value, int valueIndex, int index, int length)
    {
        SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
    }

    public void Set(char value, int index)
    {
        Set(value, 0, index, 16);
    }

    //Bool

    public bool GetBool(int index, int length)
    {
        return _bitConverter.ToBoolean(GetBytes(index, length, 1), 0);
    }

    public bool GetBool(int index)
    {
        return GetBool(index, 8);
    }

    public void Set(bool value, int valueIndex, int index, int length)
    {
        SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
    }

    public void Set(bool value, int index)
    {
        Set(value, 0, index, 8);
    }

    //Single and double precision floating point values must always use the correct number of bits
    public float GetSingle(int index)
    {
        return _bitConverter.ToSingle(GetBytes(index, 32, 4), 0);
    }

    public void SetSingle(float value, int index)
    {
        SetBytes(_bitConverter.GetBytes(value), 0, index, 32);
    }

    public double GetDouble(int index)
    {
        return _bitConverter.ToDouble(GetBytes(index, 64, 8), 0);
    }

    public void SetDouble(double value, int index)
    {
        SetBytes(_bitConverter.GetBytes(value), 0, index, 64);
    }
}

I am in the process of developing a special-purpose network tool with some packet sniffing and decoding capabilities. I am looking for languages designed to assist in the dissection/decoding of arbitrary packet formats. Idealy, the solution should be based on open standards. There are related questions on SO, but most deal with the full lifecycle of packet sniffing (I don't care so much about the capture, there are other libraries that do that well).

In general, what I'm looking for is a language and supporting framework for the declaritive definition of packet formats and corresponding run-time decoding. Because this problem can be generalized to any non-network binary data, a solution that does this for arbitrary binary streams would also be in scope. I am a little surprised no such standard currently exists in a mature and robust state (at least that I could find) - though there seem to be a lot of interesting but not-quite-right and almost-there projects (see below). Perhaps that speaks to the difficulty of the problem, or maybe to a lack of demand.

By way of example, I'm interested in technologies and ideas similar to the following (in no particular order):

• Packet.Net - Does the job of converting from binary packet representations to structures, but the dissectors are all hard-coded and it doesn't appear to be able to handle more complex formats.
• DFDL - I've been following this one for a while and was even participating in teleconferences a year or so ago. The standard seems to be reaching maturity, but implementation appears to be challenging. Not that I mind getting my hands dirty, but I'm not sure I've got the resources on this project to implement such a wide-ranging standard from scratch for this purpose.
• Network Monitor Open Source Parsers - This project describes packets using a C-like syntax for use by Microsoft Network Monitor. It has a lot of packets already defined and the language appears robust enough to support complex structures. Unfortunately, the only implemention of an execution engine is in NetMon and while the grammar for the language could probably be reverse-engineered, implementing a processing engine might be very difficult. I also worry that because of the explicit tie between the parser language and the NetMon tool there are non-general aspects to the language that would make it inappropriate for uses in other tools.
• NetPDL - This one looks very interesting, but development seems to have languished. It's also not totally clear how to make use of the execution engine outside of their own environment.
• Wireshark Dissectors - I've thought about wrapping/using native Wireshark Dissectors for this purpose, but they are tied pretty closely to Wireshark itself. The dissectors also use code to perform most of the decoding, which is a little counter to what I'm looking for - I'd prefer something that's a little more declaritive (though obviously there's a balance since complex packet structures often require switching and other logic to determine the final makeup).
• BSDL - An academic language similar in concept to DFDL (see above). Interesting and in the right direction, but outside of a couple papers nothing else seems to exist.

I'm not necessarily looking for a complete solution here (though if someone knows of one I haven't covered, that would be great). I'm more interested in comments or anecdotes about the technologies I've indicated above as well as pointers or ideas for routes I haven't thought of or covered.

You can probably use XPath to easily get to the right data and manually fill in the Location properties (or even better, add the code to the Location class). The key is the XPath extension methods for Linq to XML. Particularly look at XPathSelectElement and XPathEvaluate:

loc.Name = elements.XPathSelectElement("//Name").Value;
loc.Latitude = Convert.ToInt64(elements.XPathSelectElement("//Latitude").Value);
loc.Longitude = Convert.ToInt64(elements.XPathSelectElement("//Longitude").Value);
//loc.AddressLine = ??? (Not sure what the intended value is here...)
loc.FormattedAddress = elements.XPathSelectElement("//FormattedAddress").Value;
loc.PostalCode = elements.XPathSelectElement("//PostalCode").Value;

Here is an answer as a single Linq expression:

List<string> source = new List<string>();
source.Add("Cat, Animal, 2");
source.Add("Dog, Animal, 3");
source.Add("Luke, Human, 1");
source.Add("Owl, Animal, 0");

List<string> dest = new List<string>(
    source
        .Select(s => s.Split(new []{',', ' '}, StringSplitOptions.RemoveEmptyEntries))
        .Select(s => { if(s[1] == "Animal") s[2] = (Convert.ToInt32(s[2]) + 1).ToString(); return s; })
        .Select(s => String.Join(", ", s))
    );

In my current project I use IKVM to cross-compile several Java libraries that deal with various aspects of XML. These libraries are then integrated with several .NET libraries and my mainline code. Everything works fine, but I suspect that there are several inefficiencies, especially in the area of stream-based data access.

Many of the Java libraries can accept streaming SAX classes or other streaming objects such as OutputStream, etc. In some cases I can wrap the appropriate Java class in a coorisponding .NET subclass to bridge the gap and provide seamless streaming between the two languages. For example, creating a class that derives from both the .NET MemoryStream and the Java OutputStream. In most cases however, the interface is challenging and I am left passing whole strings around - even though I have streams available on the .NET side and the Java side accepts (different) stream classes (and vice versa).

In general my question is if anyone has encountered similar problems passing data to/from IKVM compiled libraries using streams and how were they solved or mitigated? Do any third-party solutions exist to help bridge this gap? For example, code that provides Java SAX wrappers for .NET XmlReader and/or XmlWriter would be very useful.

In general, the unsafe keyword allows you direct access to memory and therefore bypasses all verification and safety checks by the CLR.

Here's a good article on the use and impact of unsafe code: http://blogs.msdn.com/b/sebby1234/archive/2006/04/05/565090.aspx

If the part that needs to be colored is unique in the original in terms of hue (and you could make it unique by changing it in the source image through Photoshop or something), this should do the trick. It works by locking an input bitmap (so each pixel can be manipulated) and then converting each pixel to HSB so that hue (the "color") of the pixel can be adjusted if it falls within a certain range of hues. This will have a nice effect because gradients such as shadow and slight variances will also be colorized correctly.

Colorizing code:

Bitmap bmp = ...
byte minHue = 0;
byte maxHue = 10;
byte newHue = 128;

...

BitmapData bmpData = bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height),
    ImageLockMode.ReadWrite, bmp.PixelFormat);
IntPtr ptr = bmpData.Scan0;
int bytes = bmpData.Stride * bmpData.Height;
byte[] rgbValues = new byte[bytes];
System.Runtime.InteropServices.Marshal.Copy(ptr, rgbValues, 0, bytes);
for (int c = 0; c < rgbValues.Length; c += 4)
{
    HSBColor hsb = new HSBColor(Color.FromArgb(
        rgbValues[c + 3], rgbValues[c + 2],
        rgbValues[c + 1], rgbValues[c]));
if(hsb.H > minHue && hsb.H < maxHue)
    {
        hsb.H = Convert.ToByte(newHue);
    }
    Color color = hsb.ToRGB();
    rgbValues[c] = color.B;
    rgbValues[c + 1] = color.G;
    rgbValues[c + 2] = color.R;
    rgbValues[c + 3] = color.A;
}
System.Runtime.InteropServices.Marshal.Copy(rgbValues, 0, ptr, bytes);
bmp.UnlockBits(bmpData);

HSBColor.cs (from ZedGraph):

/// <summary>
/// Hue-Saturation-Brightness Color class to store a color value, and to manage conversions
/// to and from RGB colors in the <see cref="Color" /> struct.
/// </summary>
/// <remarks>
/// This class is based on code from http://www.cs.rit.edu/~ncs/color/ by Eugene Vishnevsky.
/// This struct stores the hue, saturation, brightness, and alpha values internally as
/// <see cref="byte" /> values from 0 to 255.  The hue represents a fraction of the 360 degrees
/// of color space available. The saturation is the color intensity, where 0 represents gray scale
/// and 255 is the most colored.  For the brightness, 0 represents black and 255
/// represents white.
/// </remarks>
[Serializable]
public struct HSBColor
{
    /// <summary>
    /// The color hue value, ranging from 0 to 255.
    /// </summary>
    /// <remarks>
    /// This property is actually a rescaling of the 360 degrees on the color wheel to 255
    /// possible values.  Therefore, every 42.5 units is a new sector, with the following
    /// convention:  red=0, yellow=42.5, green=85, cyan=127.5, blue=170, magenta=212.5
    /// </remarks>
    public byte H;
    /// <summary>
    /// The color saturation (intensity) value, ranging from 0 (gray scale) to 255 (most colored).
    /// </summary>
    public byte S;
    /// <summary>
    /// The brightness value, ranging from 0 (black) to 255 (white).
    /// </summary>
    public byte B;
    /// <summary>
    /// The alpha value (opacity), ranging from 0 (transparent) to 255 (opaque).
    /// </summary>
    public byte A;

    /// <summary>
    /// Constructor to load an <see cref="HSBColor" /> struct from hue, saturation and
    /// brightness values
    /// </summary>
    /// <param name="h">The color hue value, ranging from 0 to 255</param>
    /// <param name="s">The color saturation (intensity) value, ranging from 0 (gray scale)
    /// to 255 (most colored)</param>
    /// <param name="b">The brightness value, ranging from 0 (black) to 255 (white)</param>
    public HSBColor(int h, int s, int b)
    {
        this.H = (byte)h;
        this.S = (byte)s;
        this.B = (byte)b;
        this.A = 255;
    }

    /// <summary>
    /// Constructor to load an <see cref="HSBColor" /> struct from hue, saturation,
    /// brightness, and alpha values
    /// </summary>
    /// <param name="h">The color hue value, ranging from 0 to 255</param>
    /// <param name="s">The color saturation (intensity) value, ranging from 0 (gray scale)
    /// to 255 (most colored)</param>
    /// <param name="b">The brightness value, ranging from 0 (black) to 255 (white)</param>
    /// <param name="a">The alpha value (opacity), ranging from 0 (transparent) to
    /// 255 (opaque)</param>
    public HSBColor(int a, int h, int s, int b)
        : this(h, s, b)
    {
        this.A = (byte)a;
    }

    /// <summary>
    /// Constructor to load an <see cref="HSBColor" /> struct from a system
    /// <see cref="Color" /> struct.
    /// </summary>
    /// <param name="color">An rgb <see cref="Color" /> struct containing the equivalent
    /// color you want to generate</param>
    public HSBColor(Color color)
    {
        this = FromRGB(color);
    }


    /// <summary>
    /// Implicit conversion operator to convert directly from an <see cref="HSBColor" /> to
    /// a <see cref="Color" /> struct.
    /// </summary>
    /// <param name="hsbColor">The <see cref="HSBColor" /> struct to be converted</param>
    /// <returns>An equivalent <see cref="Color" /> struct that can be used in the GDI+
    /// graphics library</returns>
    public static implicit operator Color(HSBColor hsbColor)
    {
        return ToRGB(hsbColor);
    }

    /// <summary>
    /// Convert an <see cref="HSBColor" /> value to an equivalent <see cref="Color" /> value.
    /// </summary>
    /// <remarks>
    /// This method is based on code from http://www.cs.rit.edu/~ncs/color/ by Eugene Vishnevsky.
    /// </remarks>
    /// <param name="hsbColor">The <see cref="HSBColor" /> struct to be converted</param>
    /// <returns>An equivalent <see cref="Color" /> struct, compatible with the GDI+ library</returns>
    public static Color ToRGB(HSBColor hsbColor)
    {
        Color rgbColor = Color.Black;

        // Determine which sector of the color wheel contains this hue
        // hsbColor.H ranges from 0 to 255, and there are 6 sectors, so 42.5 per sector
        int sector = (int)Math.Floor((double)hsbColor.H / 42.5);
        // Calculate where the hue lies within the sector for interpolation purpose
        double fraction = (double)hsbColor.H / 42.5 - (double)sector;

        double sFrac = (double)hsbColor.S / 255.0;
        byte p = (byte)(((double)hsbColor.B * (1.0 - sFrac)) + 0.5);
        byte q = (byte)(((double)hsbColor.B * (1.0 - sFrac * fraction)) + 0.5);
        byte t = (byte)(((double)hsbColor.B * (1.0 - sFrac * (1.0 - fraction))) + 0.5);


        switch (sector)
        {
            case 0:         // red - yellow
                rgbColor = Color.FromArgb(hsbColor.A, hsbColor.B, t, p);
                break;
            case 1:         // yellow - green
                rgbColor = Color.FromArgb(hsbColor.A, q, hsbColor.B, p);
                break;
            case 2:         // green - cyan
                rgbColor = Color.FromArgb(hsbColor.A, p, hsbColor.B, t);
                break;
            case 3:         // cyan - blue
                rgbColor = Color.FromArgb(hsbColor.A, p, q, hsbColor.B);
                break;
            case 4:         // blue - magenta
                rgbColor = Color.FromArgb(hsbColor.A, t, p, hsbColor.B);
                break;
            case 5:
            default:        // magenta - red
                rgbColor = Color.FromArgb(hsbColor.A, hsbColor.B, p, q);
                break;
        }

        return rgbColor;
    }

    /// <summary>
    /// Convert this <see cref="HSBColor" /> value to an equivalent <see cref="Color" /> value.
    /// </summary>
    /// <remarks>
    /// This method is based on code from http://www.cs.rit.edu/~ncs/color/ by Eugene Vishnevsky.
    /// </remarks>
    /// <returns>An equivalent <see cref="Color" /> struct, compatible with the GDI+ library</returns>
    public Color ToRGB()
    {
        return ToRGB(this);
    }

    /// <summary>
    /// Convert a <see cref="Color" /> value to an equivalent <see cref="HSBColor" /> value.
    /// </summary>
    /// <remarks>
    /// This method is based on code from http://www.cs.rit.edu/~ncs/color/ by Eugene Vishnevsky.
    /// </remarks>
    /// <returns>An equivalent <see cref="HSBColor" /> struct</returns>
    public HSBColor FromRGB()
    {
        return FromRGB(this);
    }

    /// <summary>
    /// Convert a <see cref="Color" /> value to an equivalent <see cref="HSBColor" /> value.
    /// </summary>
    /// <remarks>
    /// This method is based on code from http://www.cs.rit.edu/~ncs/color/ by Eugene Vishnevsky.
    /// </remarks>
    /// <param name="rgbColor">The <see cref="Color" /> struct to be converted</param>
    /// <returns>An equivalent <see cref="HSBColor" /> struct</returns>
    public static HSBColor FromRGB(Color rgbColor)
    {
        double r = (double)rgbColor.R / 255.0;
        double g = (double)rgbColor.G / 255.0;
        double b = (double)rgbColor.B / 255.0;

        double min = Math.Min(Math.Min(r, g), b);
        double max = Math.Max(Math.Max(r, g), b);

        HSBColor hsbColor = new HSBColor(rgbColor.A, 0, 0, 0);

        hsbColor.B = (byte)(max * 255.0 + 0.5);

        double delta = max - min;

        if (max != 0.0)
        {
            hsbColor.S = (byte)(delta / max * 255.0 + 0.5);
        }
        else
        {
            hsbColor.S = 0;
            hsbColor.H = 0;
            return hsbColor;
        }

        double h;
        if (r == max)
            h = (g - b) / delta;        // between yellow & magenta
        else if (g == max)
            h = 2 + (b - r) / delta;    // between cyan & yellow
        else
            h = 4 + (r - g) / delta;    // between magenta & cyan

        hsbColor.H = (byte)(h * 42.5);
        if (hsbColor.H < 0)
            hsbColor.H += 255;

        return hsbColor;
    }

}

There are several embeddable C# web servers, but none that I know of that will load and use Apache plugins.

• http://cassinipp.codeplex.com/
• http://webserver.codeplex.com/
• XSP (From Mono) can be embedded, but there isn't much documentation (http://weblogs.asp.net/britchie/archive/2005/07/25/420512.aspx)
• http://www.codeproject.com/KB/IP/SimpleHttpServer.aspx

My first thought for this problem is to create a simple TextReader derivative that is responsible for buffering input from the stream. This class would then be used to feed an XmlReader. The TextReader derivative could fairly easily scan the incoming content looking for complete "blocks" of XML (a complete element with starting and ending brackets, a text fragment, a full attribute, etc.). It could also provide a flag to the calling code to indicate when one or more "blocks" are available so it can ask for the next XML node from the XmlReader, which would trigger sending that block from the TextReader derivative and removing it from the buffer.

Edit: Here's a quick and dirty example. I have no idea if it works perfectly (I haven't tested it), but it gets across the idea I was trying to convey.

public class StreamingXmlTextReader : TextReader
{
    private readonly Queue<string> _blocks = new Queue<string>();
    private string _buffer = String.Empty;
    private string _currentBlock = null;
    private int _currentPosition = 0;

    //Returns if there are blocks available and the XmlReader can go to the next XML node
    public bool AddFromStream(string content)
    {
        //Here is where we would can for simple blocks of XML
        //This simple chunking algorithm just uses a closing angle bracket
        //Not sure if/how well this will work in practice, but you get the idea
        _buffer = _buffer + content;
        int start = 0;
        int end = _buffer.IndexOf('>');
        while(end != -1)
        {
            _blocks.Enqueue(_buffer.Substring(start, end - start));
            start = end + 1;
            end = _buffer.IndexOf('>', start);
        }

        //Store the leftover if there is any
        _buffer = end < _buffer.Length
            ? _buffer.Substring(start, _buffer.Length - start) : String.Empty;

        return BlocksAvailable;
    }

    //Lets the caller know if any blocks are currently available, signaling the XmlReader can ask for another node
    public bool BlocksAvailable { get { return _blocks.Count > 0; } }

    public override int Read()
    {
        if (_currentBlock != null && _currentPosition < _currentBlock.Length - 1)
        {
            //Get the next character in this block
            return _currentBlock[_currentPosition++];
        }
        if(BlocksAvailable)
        {
            _currentBlock = _blocks.Dequeue();
            _currentPosition = 0;
            return _currentBlock[0];
        }
        return -1;
    }
}

If name is a List<T> then you probably want name.RemoveAt(i) if I understand the question correctly.

Alternatively, you could just use name.RemoveAll(n => n == selectName); instead of the for loop.

At least for the check part you could use an Enum with the [Flags] attribute to create bit field. That might be a little more extensible if you add more methods in the future. You could then use a simple lookup table and do away with the PredicateOptionSet class. Example:

[Flags]
public enum PredicateOption
{
    IsCaseSensitive, IsRegularExpression, IsMultipleTerms
};

...

public Dictionary<PredicateOption, Func<SearchResult, bool>> _predicates
    = new Dictionary<PredicateOption, Func<SearchResult, bool>>();
_predicates.Add(PredicateOption.IsCaseSensitive, result => Search(result.Name));
_predicates.Add(PredicateOption.IsCaseSensitive | PredicateOption.IsMultipleTerms,
    result => SearchCaseInsensitiveMultiple(result.Name));

....

PredicateOption option = PredicateOption.IsCaseSensitive | PredicateOption.IsMultipleTerms;
SearchResults.Where(_predicates[option]);