The most recent version of Nxdb includes a complete object persistence framework and this post explains some of the motivation behind it and provides some insight into how it was implemented.
Wikipedia sums it up nicely by saying “a persistence framework is middleware that assists and automates the storage of program data into databases.” More to the point, a persistence framework allows the state of object instances to be stored externally (usually, but not necessarily, in a database) and recalled at a later time (such as a future program execution). The “state” of an object could be everything that is necessary to fully represent the object (such as all of it’s fields), or it could be some important subset of object data.
Persistence frameworks are related to, but not the same, as object-relational mapper (ORM) tools. A persistence framework usually manages the conversion of an object to some format the external storage medium can understand, the storage of that data into the external medium, the fetching of data from the external medium, and the instantiation and population of instances based on that data. The problem with persistence frameworks when they use a relational database (or other highly structured storage) is that the database needs to know something about the objects being stored so that tables, columns, etc. can be created to support them. ORM tools assist with this process by mapping the data to be stored (I.e., the object) to the storage medium (I.e., the database). They often automate the process of creating appropriate tables and columns, managing foreign keys, etc. An ORM is only needed when the storage medium cannot natively support the type or structure of data needing to be stored.
In general, I wouldn’t really recommend rolling your own persistence framework. There are so many good ones out there (including NHibernate, SubSonic, and mybatis) and it is doubtful you’ll be able to improve on them enough to make the exercise worthwhile. So why ignore this advice and build a new one for Nxdb? The answer is that almost all of the existing persistence framework are based on SQL, talk to SQL databases, and either use or have built-in ORM capabilities. XML is a completely different storage medium, one that is able to better represent the hierarchical nature of object-oriented classes. While a couple of .NET XML persistence frameworks do exist (such as Proetus) and several articles have been written at CodeProject and elsewhere on how to write XML persistence capabilities, they tend to be somewhat conceptual and abstract the storage layer by producing XML that becomes the developer’s responsiblity to store and retrieve. We saw an opportunity to unlock the potential of a native XML persistence layer by having it be tied directly to an XML database. This allows automation of the storage and retrieval of objects and opens up the door for interesting use cases such as querying the object graph using XQuery.
When we started thinking about an XML persistence layer for Nxdb we considered what the most important requirements should be and narrowed them down to a few key ones:
The main entry point for the persistence framework is the Manager class. All objects are initially persisted or retrieved through one of it’s methods. Objects can be attached to the manager, which means that they are stored in a cache and the manager will return a reference to the same instance of the object if the same XML content is used. Additionally, attached objects can be automatically updated when the database changes, making the objects more of a programmatic representation of the database rather than the database simply serving as a backing store. Each object is persisted to and from the database using a Persister. The persister is primarily responsible for determining what parts of an object should be stored in the database, how those parts should be represented (for example as XML elements or attributes), and performing the actual serialization and deserialization of an object. There are several persisters including one that supports the native .NET XmlSerializer, one that provides a great deal of control through the use of attributes on class members, and one that allows completely custom behavior through the implementation of an interface. Should additional techniques be required, a new persister is easy to create and implement. Overall, the architecture and process by which objects are stored to the database and fetched from the database is presented below.
More information about the Nxdb persistence framework as well as details on how to use it can be found at the Nxdb Wiki.
For the second time in as many weeks, I have the pleasure of announcing a new open source project. This time I’ve released Nxdb, a high-performance embedded XML database for .NET with full XQuery support. It is essentially a .NET binding for the excellent BaseX Java XML database. However, Nxdb goes well beyond a simple wrapper by providing a native .NET API suitable for embedding (the primary focus of BaseX is on client/server uses), providing additional functionality to interface directly with .NET classes and objects, and rethinking several aspects of the BaseX design for the embedded use case. The underlying BaseX code is cross-compiled to IL using IKVM making Nxdb 100% native .NET, usable on all .NET platforms including Mono.
This project has a long history. Around the fall of 2008, my company, DRAC was tasked with developing an entirely new cross-platform graphical user interface for one of the US Air Force’s most complex models. The model itself used over a hundred different input files, each with slightly different grammar and syntax. What we needed was a way to store, query, and manipulate all of this data in a consistent way. XML seemed like an ideal choice given it’s ability to represent data hierarchically and powerful XQuery query language. Having already made the decision to use Mono as our platform, we scoured both the open source community and commercial vendors to find a database backend that was a good fit. We were surprised to discover that there weren’t very many suitable embedded XML databases for .NET and the ones we did find were either too slow for processing and querying the volume of data we needed to work with or had cumbersome and limited APIs that didn’t allow the control we needed.
Thankfully, we stumbled on BaseX when it was a fairly early project. We were immediatly impressed with the completeness of it’s query engine and the fast performance. Unfortunatly it was written in Java and we were using .NET and had already sunk considerable resources into the GUI shell. Having used IKVM to cross-compile Java code before, we decided to give it a shot for BaseX and were pleased to discover it converted and ran quite well. Over the intervening years we’ve separated our BaseX binding from the original project, applied it to many other projects, and watched as BaseX has flourished. Nxdb is now used in production code every day and we’re excited to share it with the rest of the open source community.
Nxdb is designed to be easy to use. It provides two primary means of working with the database. The first is by directly working with the database through classes that represent XML nodes. The other main interface is through the evaluation of XQuery expressions.
The Database class serves as the starting point for working with an Nxdb database. It encapsulates all of the operations on a database instance and provides methods for getting node objects and evaluating queries. Before use, Nxdb must be initialized as follows using the static Database.Initialize() method (path is where all of the database files will be stored):
Database.Initialize(path);
Once that’s done, a new or existing database can be obtained via the static Database.Get() method:
Database.Get(databaseName);
Once a Database instance has been obtained, XML content can be added, documents fetched, and queries executed. A very simple example usage might look like this:
using Nxdb;
Database.Initialize("C:\Temp\NxdbData");
using(Database db = Database.Get("Example"))
{
db.Add("DocA", "<A><B>b</B><C>c</C></A>");
Document doc = db.GetDocument("DocA");
Console.Write(doc.OuterXml);
}
Nxdb is open source and released under the Apache 2.0 license. It can be obtained here: https://dracorp.assembla.com/spaces/nxdb
NiceThreads is threading utility library designed to make different threading primitives easier to use with a more consistent API. It started out of frustration with the different options (and more specifically, the different APIs) for enabling thread safety and locks in the .NET framework and how much code was required to use some of them. NiceThreads provides a consistent interface for standard Monitor locks and the ReaderWriterLockSlim class (and possibly others in the future). It also provides support for activating and deactivating these locking primitives through the disposable pattern. Finally, it provides wrappers that can easily provide thread-safety to unsafe objects.
ILocker locker = new ReaderWriterLockSlimLocker(); locker.EnterReadLock(); // Do work... locker.ExitReadLock(); locker = new MonitorLocker(); locker.EnterReadLock(); // Do work... locker.ExitReadLock();
ILocker locker = new ReaderWriterLockSlimLocker();
using(new ReadLock(locker))
{
// Do work...
}
</pre>
SyncObject<int> num = new SyncObject<int>(10);
num.Sync = 20; // Access as a property with a thread-safe setter
int value = num.Sync; // Access as a property with a thread-safe getter
using(num.WriteLock())
{
// We can now access using unsafe code
num.UnsyncField++; // Provides direct field access
value = num.Unsync; // Access as a property with an unsafe getter
}
num.DoWrite(n => n + 10); // Thread-safe write with an Action
value = num.DoRead(n => n + 100); // Thread-safe read with a Func
<pre>
NiceThreads is open source and released under the Apache 2.0 license. It can be obtained here: https://dracorp.assembla.com/spaces/nicethreads
I’m back. Don’t worry (my one reader), I haven’t given up on this blogging idea. I’ve just been very busy recently.
Today’s post is an XQuery function designed to get a count of the number of week (or work) days between two dates. It’s designed to mimic the Excel NETWORKDAYS function. I got the algorithm from Bernal Schooley in this thread and then adapted it to XQuery. It also makes use of the FunctX day-of-week function, so if you have FunctX functions already referenced you can take that part out.
declare namespace functx = "http://www.functx.com";
declare function functx:day-of-week
($date as xs:anyAtomicType?) as xs:integer? {
if (empty($date))
then ()
else
xs:integer((xs:date($date) - xs:date('1901-01-06')) div xs:dayTimeDuration('P1D')) mod 7
};
declare function local:weekdays
($start as xs:anyAtomicType?, $end as xs:anyAtomicType?) as xs:integer? {
if(empty($start) or empty($end))
then()
else
if($start > $end)
then -local:weekdays($end, $start)
else
let $dayOfWeekStart := functx:day-of-week($start)
let $dayOfWeekEnd := functx:day-of-week($end)
let $adjDayOfWeekStart := if($dayOfWeekStart = 0) then 7 else $dayOfWeekStart
let $adjDayOfWeekEnd := if($dayOfWeekEnd = 0) then 7 else $dayOfWeekEnd
return
if($adjDayOfWeekStart <= $adjDayOfWeekEnd)
then xs:integer((xs:integer(days-from-duration(xs:date($end) - xs:date($start)) div 7) * 5)
+ max(((min((($adjDayOfWeekEnd + 1), 6)) - $adjDayOfWeekStart), 0)))
else xs:integer((xs:integer(days-from-duration(xs:date($end) - xs:date($start)) div 7) * 5)
+ min((($adjDayOfWeekEnd + 6) - min(($adjDayOfWeekStart, 6)), 5)))
};
Usage:
local:weekdays('2009-06-01', '2010-06-30')
I ran across this while working on some complex GtkSharp grabbing behavior to mimic window focusing for a docking framework. Turns out there is a weird inconsistency in the way Gtk+ manages the grab stack. While the list of grabbed Widgets is indeed a stack, a flag on each Widget (Widget.HasGrab) is used to check if a Widget has the grab or not. The problem is that Grab.Add (which calls the Gtk+ method gtk_grab_add) never clears the flag for the currently grabbed Widget if you’re nesting grabs. That means that every Widget in the grab stack will have Widget.HasGrab set to true. If you try to add a Widget to the grab stack and it’s already in the stack (even if there are multiple other grabbed Widgets after it in the stack), it won’t get added. Because the flag is set though, it will get removed at the first place it was in the stack on a call to Grab.Remove (gtk_grab_remove).
While it may not be a bug, this is certainly odd behavior. The solution (at least for me) was to write two small utility methods. SafeAdd first removes the Widget.HasGrab flag to ensure that the Widget always gets added to the grab stack, regardless of if it’s previously in it. SafeAdd should be paired with SafeRemove which checks the newly grabbed Widget after removing one to make sure it still has the Widget.HasGrab flag set. Note that it doesn’t clear the Widget.HasGrab flag for grabbed Widget getting replaced by a new grabbed Widget on the grab stack as you might expect. This maintains compatibility with all the other Gtk code that might be expecting Widgets anywhere in the stack to have the flag set.
public static void SafeAdd(Widget widget)
{
if (widget == null)
{
throw new ArgumentNullException("widget");
}
if (widget.Sensitive)
{
widget.ClearFlag(WidgetFlags.HasGrab);
Grab.Add(widget);
}
}
public static void SafeRemove(Widget widget)
{
if (widget == null)
{
throw new ArgumentNullException("widget");
}
Grab.Remove(widget);
Widget current = Grab.Current;
if( current != null && !current.HasGrab )
{
current.SetFlag(WidgetFlags.HasGrab);
}
}
To show a context menu in GtkSharp (or “popup” as they’re called in Gtk land), you would normally add an event handler for Widget.PopupMenu, create or use a Menu instance, and then call Menu.Popup. The only problem is that for many widgets, the right-click doesn’t trigger the Widget.PopupMenu event. This is fine for systems where there is no right mouse button or where a right-click isn’t the customary way of initiating context menus. However, on systems where there is a user expectation that the way to open a context menu is through a right-click (such as Windows), we need some way to trigger one.
The situation is complicated a little bit by the existing behavior of Widget.PopupMenu. According to the Gtk manual, “By default, the key binding mechanism is set to emit this signal when the Shift+F10 or Menu keys are pressed while a widget has the focus.” There is a recommendation in the manual that if a developer wants context menus on right-click they should handle the Widget.ButtonPressEvent, listen for the appropriate clicks, and launch the menu using the Menu.Popup method. This is all fine except that now you’ve got two things to listen to to get proper context menu handling: Widget.PopupMenu and Widget.ButtonPressEvent. It would be nice if there were just one event to handle that got raised anytime a context menu needed to be displayed.
The following class does just that. You can “attach” it to any Widget and it will listen for the Widget.PopupMenu event to work with the default context menu handling and the Widget.ButtonPressEvent to also work with right-clicks. When either of these occur, it will first propagate the event through to the underlying Widget (in case there are other things that are supposed to be triggered by whatever event caused the context menu) and then raise a ContextMenuHelper.ContextMenu event that you can handle and use to display the context menu regardless of what triggered it. This method should ensure proper event handling and ordering while reducing duplication of code by enabling a single event for context menu handling.
using System;
using Gdk;
using GLib;
using Gtk;
namespace Somedave
{
public class ContextMenuEventArgs : EventArgs
{
private Widget widget;
public Widget Widget { get { return widget; } }
private bool rightClick;
public bool RightClick { get { return rightClick; } }
public ContextMenuEventArgs(Widget widget, bool rightClick)
{
this.widget = widget;
this.rightClick = rightClick;
}
}
public class ContextMenuHelper
{
public event EventHandler<ContextMenuEventArgs> ContextMenu;
public ContextMenuHelper()
{}
public ContextMenuHelper(Widget widget)
{
AttachToWidget(widget);
}
public ContextMenuHelper(Widget widget, EventHandler handler)
{
AttachToWidget(widget);
ContextMenu += handler;
}
public void AttachToWidget(Widget widget)
{
widget.PopupMenu += Widget_PopupMenu;
widget.ButtonPressEvent += Widget_ButtonPressEvent;
}
public void DetachFromWidget(Widget widget)
{
widget.PopupMenu -= Widget_PopupMenu;
widget.ButtonPressEvent -= Widget_ButtonPressEvent;
}
[GLib.ConnectBefore]
private void Widget_PopupMenu(object o, PopupMenuArgs args)
{
RaiseContextMenuEvent(args, (Widget)o, false);
}
[GLib.ConnectBefore]
private void Widget_ButtonPressEvent(object o, ButtonPressEventArgs args)
{
if (args.Event.Button == 3 && args.Event.Type == EventType.ButtonPress)
{
RaiseContextMenuEvent(args, (Widget)o, true);
}
}
private bool propagating = false; //Prevent reentry
private void RaiseContextMenuEvent(SignalArgs signalArgs, Widget widget, bool rightClick)
{
if (!propagating)
{
//Propagate the event
Event evnt = Gtk.Global.CurrentEvent;
propagating = true;
Gtk.Global.PropagateEvent(widget, evnt);
propagating = false;
signalArgs.RetVal = true; //The widget already processed the event in the propagation
//Raise the context menu event
ContextMenuEventArgs args = new ContextMenuEventArgs(widget, rightClick);
if (ContextMenu != null)
{
ContextMenu.Invoke(this, args);
}
}
}
}
}
I recently had to create some functionality to export a TreeView widget to a CSV file for further analysis. Since I tend to think about generic behavior, I decided to code up a method that would take any arbitrary TreeView and perform the export operation. Luckily, the TreeView widget and the attached TreeModel both contain a lot of functionality for accessing the data and it’s presentation. I decided that I wanted the exported CSV file to represent the perspective of the model as currently represented in the TreeView including column visibility and sort order. This led to the trickiest part of the process. Because a CellRenderer can be customized using cell data functions (such as those added by a call to TreeViewColumn.SetCellDataFunc), I had to pull the content to export from the CellRenderer as opposed to pulling directly from the TreeModel. Turns out there’s a method to take the TreeIter from a TreeModel and apply it to all the CellRenderers in a given TreeViewColumn. Since I really only care about textual content, I decided to only export those columns that contain CellRendererText renderers.
After working out the algorithm to fetch what needed to be exported I thought I was ready to roll. Turns out that the CSV pseudo-standard is pretty complex though (the RFC is here), and I quickly got bogged down in writing all kinds of special cases for escaping, quoting, etc. Thankfully, someone else had already been down this road and I was able to find the excellent KBCsv library which will write and read formatted CSV files. My only complaint was that it used another utility library purely for convenience in exception generation and null checking (I already use a ton of libraries in our application and I’d prefer not to add any unnecessarily). I replaced the calls to the utility library with the language equivalents, but that’s totally a personal preference.
Without further adieu, I present the TreeViewHelper.ExportToCsv and TreeViewHelper.ExportToCsvFile methods…
using System;
using System.Collections.Generic;
using System.IO;
using Gtk;
using Kent.Boogaart.KBCsv;
namespace Somedave
{
public static class TreeViewHelper
{
public static bool ExportToCsv(TreeView treeView, Window parent)
{
FileChooserDialog fcd = new FileChooserDialog("Export File", parent, FileChooserAction.Save,
"Cancel", ResponseType.Cancel, "Export", ResponseType.Accept);
fcd.DoOverwriteConfirmation = true;
FileFilter filter = new FileFilter { Name = "CSV File" };
filter.AddPattern("*.csv");
fcd.AddFilter(filter);
if (fcd.Run() == (int)ResponseType.Accept)
{
string path = fcd.Filename;
fcd.Destroy();
return ExportToCsvFile(treeView, path);
}
fcd.Destroy();
return false;
}
public static bool ExportToCsvFile(TreeView treeView, string path)
{
//Get the iterator
TreeIter iter;
if (treeView.Model.GetIterFirst(out iter))
{
//Create the stream
using (StreamWriter streamWriter = new StreamWriter(path, false))
{
//Create the CSV writer
using (CsvWriter csvWriter = new CsvWriter(streamWriter))
{
List<string> headers = new List<string>();
List<string> values = new List<string>();
//Traverse the tree
do
{
values.Clear();
foreach (TreeViewColumn column in treeView.Columns)
{
//Only output visible columns
if (column.Visible)
{
//Loop through CellRenderers to make sure we have a CellRendererText
string value = null;
column.CellSetCellData(treeView.Model, iter, false, false);
foreach (CellRenderer renderer in column.CellRenderers)
{
CellRendererText text = renderer as CellRendererText;
if (text != null)
{
//Setting value indicates this column had a CellRendererText and should be included
if (value == null)
{
value = String.Empty;
}
//Add the header if the first time through
if (headers != null)
{
headers.Add(column.Title);
}
//Append to the value
if (text.Text != null)
{
value += text.Text;
}
}
}
if (value != null)
{
values.Add(value);
}
}
}
//Output the header
if (headers != null)
{
csvWriter.WriteHeaderRecord(headers.ToArray());
headers = null;
}
//Output the values
csvWriter.WriteDataRecord(values.ToArray());
} while (treeView.Model.IterNext(ref iter));
}
}
return true;
}
return false;
}
}
}
My name is Dave Glick and I live in Northern Virginia with my wife, son, and dog. I’ve been developing software in one form or another for most of my life, starting with a programming class given by the dad of a friend when we were in third or fourth grade (unfortunately, my budding geek cred was unappreciated on the playground at recess). I can still remember sitting in their basement and pouring over sheet-fed printouts of BASIC code from my Guess The Number game trying to figure out why the computer always chose the same two numbers. I’m sure I learned a valuable lesson about properly seeding random number generators or something, but what I managed to take with me was that debugging code from a printout is a royal pain.
I started programming as a job in high school. After school was out I would drive over to a local ISP and code up web pages for their clients in between answering support calls. I learned most of what I know about Linux and Unix while grepping log files and rebooting servers trying to figure out why the old lady on the phone couldn’t get her thirty images of cats emailed to her entire extended family. Believe me, every horror story you’ve ever heard about technical support calls is probably true. In any case, I managed to do a good enough job that they took me off of the support desk to work on web projects exclusively. I stayed with them through college honing my skills and earning some spending money. At one point, they even got me an on-site contract with a hot events management startup in DC during the frenzy of the first dotcom boom. I remember there was an office there where the door was always closed. On the wall outside the door there was a little white board with lots of tick marks. When I noticed the number of tick marks were always varying, I asked what was going on. I was told that one of their best software guys was in the room and the ticks represented how many hours he had been in there without stepping out. When I came to the office one evening and there were seven or eight ticks on the whiteboard, I couldn’t help but wonder if the guy had expired out of frustration while looking at printouts of code. I never did see him the entire four months I worked there.
Once the dotcom bubble burst, web work was a lot more difficult to find and the ISP I worked for was having problems of its own competing against the rising popularity of cheap and available broadband. I ended up getting a job in one of the thousands of small defense contracting companies in the Northern Virginia area. I’ve stayed in that industry for almost ten years now while completing an undergraduate degree in Computer Graphics Design, a graduate degree in Software Engineering, a graduate certificate in Web-Based Software Engineering, and an MCSE certification. I currently work for Data Research and Analysis Corporation developing cross-platform networking and distributed modeling and simulation tools using .NET, Mono, GtkSharp, and XML.
In doing the research for starting this blog I noticed almost every blog author I respect has a paragraph or two in their introductory posts about why they blog. For me the answer is simple, I do a lot of cool stuff and work out a lot of hard problems that people never see. Though my Google-fu is pretty strong, I’m always frustrated when I have a problem and can’t find any mentions of it. I always end up thinking “well, someone must have run into this before.” Hopefully I can spare others some of that frustration by publishing things I learn about and work out.
I also want to educate and inform about my chosen architecture stack. I’ve been doing development with .NET, Mono, GtkSharp, and XML for a while now and have come to appreciate the simplicity and power that these technologies provide. Unfortunately, the area of cross-platform and Microsoft-independant .NET development doesn’t get as much attention as I think it should. I’m hopeful that I can add my voice to those already out there and help continue to make people aware that there is .NET life beyond Microsoft (though I don’t have anything against Microsoft, and I like a lot of what they do).
I can’t deny it’s also good for a little ego boost :)
I'm toying with the idea of providing some kind of Git access to resources used by a SaaS application to it's users. This will allow users to edit and push content through the Git interface as well as through the application's native web-based interface. My main concern is how to reconcile merge conflicts when a user edit's the content in the web-based interface (I'm not so concerned about merge conflicts from the Git interface since Git and their Git client should handle that). This is very similar to how GitHub allows both Git-based and web-based access to their Wikis, and I'm curious how they handle this situation as a pattern for others to follow when providing both web-based and Git-based access to content.
If a user goes to edit a Wiki page on GitHub through the web interface and another user pushes a change to the Wiki repository branch before they finish, what happens when they save their changes? Does it use a "last one wins" or a "sorry, redo all your changes on the new version"?
I found a related SO post that discussed the problem in general, but I'm very curious how GitHub specifically handles it since it's backed by Git which already has some merging capabilities baked in.
I've had the need for a simple ExpandoObject initializer several times before and typically use the following two extension methods to accomplish something like initializer syntax:
public static KeyValuePair<string, object> WithValue(this string key, object value)
{
return new KeyValuePair<string, object>(key, value);
}
public static ExpandoObject Init(
this ExpandoObject expando, params KeyValuePair<string, object>[] values)
{
foreach(KeyValuePair<string, object> kvp in values)
{
((IDictionary<string, Object>)expando)[kvp.Key] = kvp.Value;
}
return expando;
}
Then you can write the following:
dynamic foo = new ExpandoObject().Init(
"A".WithValue(true),
"B".WithValue("Bar"));
In general I've found that having an extension method to build KeyValuePair<string, object> instances from a string key comes in handy. You can obviously change the name to something like Is so that you can write "Key".Is("Value") if you need the syntax to be even more terse.
I've been pondering this for a couple hours and I'm a little surprised at how challenging it is. I'm trying to initialize a static cache of reflection data for a given object from a base class that will be accessed from multiple threads. I'm having a hard time coming up with the correct pattern for initializing the cache.
My first thought was that I'll just initialize the static cache to null, check if it's null in the constructor, and then build and set it if it's not. I.e.:
class TestBase
{
private static ConcurrentDictionary<string, PropertyInfo> Cache;
protected TestBase()
{
if(Cache == null)
{
ConcurrentDictionary<string, PropertyInfo> cache =
new ConcurrentDictionary<string, PropertyInfo>();
// Populate...
Cache = cache;
}
}
}
This has a flaw in that if I construct another object while the first one is still populating the cache, I'll end up constructing two caches and the second will (presumably, though not always) overwrite the first. This is probably okay since they'll both be complete caches, but it seems hacky and I would hope we can do better.
So my second thought was to initialize the cache in a static constructor which only gets called once per AppDomain before any instances are created. StackOverflow seems to have several answers to similar questions that point in this direction. This seemed great until I realized that the static constructor won't have access to the reflection data for the derived type.
I could always synchronize access in the constructor to insure only one thread is creating/populating the cache and any other access while that's happening should block, but then I'm locking on every construction just to protect an operation that should only happen once. I don't like the performance implications of that.
What I have right now is a flag that is set using Interlocked.Exchange and a ManualResetEventSlim. It looks like this:
class TestBase
{
private static ConcurrentDictionary<string, PropertyInfo> Cache;
private static volatile int BuildingCache = 0;
private static ManualResetEventSlim CacheBuilt =
new ManualResetEventSlim();
protected TestBase()
{
if(Interlocked.Exchange(ref BuildingCache, 1) == 0)
{
Cache = new ConcurrentDictionary<string, PropertyInfo>();
// Populate...
CacheBuilt.Set();
}
CacheBuilt.Wait();
}
}
I suspect that there might already be an accepted, or at least known, way of doing this kind of thing - is this it? If not, is there a better way to synchronize the cache initialization? Note that the question is not about how to make cache access thread-safe, that can be assumed by using ConcurrentDictionary (or similar).
I have noticed that while debugging my site, any request that results in a 404 and appears to refer to a path on the disk relative to my configured virtual directory is intercepted by Cassini and rudely replaced with a directory listing. I'm using Nancy Framework, but given that this problem appears to be at the web server level, I suspect Cassini would act the same way for MCV applications. I can't find any documentation on this "feature" other than a related commit message on the Cassini source that says "...directory listing only overrides 404 responses for directories".
I would much rather my development web server stop trying to outsmart my framework. I makes the debugging experience more than a little jarring. In an MVC framework the request URLs have nothing at all to do with file locations, so the fact I'm getting directory listings for some invalid requests and the correct 404 page for others gets annoying. Not to mention it's making several of my unit tests fail because they rely on auto-generated content in my 404 error pages (which I can't manually test either).
Is there any way to disable this functionality in Cassini? I know I could install IIS Express, but I'd rather not. Especially since my unit test runner and hosts file (this is a multi-domain application) are already configured just right.
Batch put and delete are available now (they call it BatchWriteItem): http://docs.amazonwebservices.com/amazondynamodb/latest/developerguide/API_BatchWriteItems.html
However, you still can not update multiple items in a single request.
I have an idea for a new web application and have been looking for tools to help me get a jump start. Essentially, I would like to provide a service to "my" users that would let them easily create a site that includes blogging and some other specialized functionality while allowing them to maintain their own users. The best analogy is that I'd like to create a site similar to WordPress.com (the service, not the software), TypePad, or Blogger but way simpler (I'm only one guy after all!) and designed for users in a very specific niche. I'm guessing this concept is general enough that answers should benefit others looking to do something similar.
Which brings me to my question: what tools exist to help develop multi-tenant, multi-user web applications? I would personally prefer something open source and written in .NET, but for the benefit of everyone else I think non-open source and/or non-.NET systems would also prove valuable. My first thought was to look for open source CMS systems that support multi-tenancy and have a robust extension mechanism. The most promising that I've found so far is Orchard. Has anyone used it (or something similar) for this kind of purpose?
In general, requirements for this type of application would probably be akin to:
(I asked this over at Web Applications, but got closed as off-topic. Hopefully SO is the right site this time. Forgive me if not - it can be difficult to determine the appropriate SE site sometimes).
If you think XML might fit the bill, my company just released an open source embedded XML database for the .NET platform called Nxdb. It's under the Apache 2.0 license and has been in development and use internally for several years. It's basically a binding to a cross-compiled (using IKVM) version of BaseX (a fantastic Java XML database) along with extra functionality for the embedded use case and the .NET environment. The project page is here: https://dracorp.assembla.com/spaces/nxdb
Deployment is fairly simple, though it does require a number of assemblies due to the way IKVM is linked and deployed (I think 10 or so at last count). The documentation is somewhat sparse, though there are some examples in the Wiki. We're working on adding more examples and better documentation right now (and are open to suggestions).
(Hopefully linking to my own project doesn't irk the mods, but I figured since this post directly answers the question and provides a link to a permissive OS project it would be okay).
This is an old question, but I thought I'd add an answer in case anyone stumbles on it. My company just released an open source embedded XML database for the .NET platform called Nxdb. It's under the Apache 2.0 license and has been in development and use internally for several years. It's basically a binding to a cross-compiled (using IKVM) version of BaseX (a fantastic Java XML database) along with extra functionality for the embedded use case and the .NET environment. The project page is here: https://dracorp.assembla.com/spaces/nxdb
(Hopefully linking to my own project doesn't irk the mods, but I figured since this post directly answers the question and provides a link to a permissive OS project it would be okay).
This is an old question, but I thought I'd add an answer in case anyone stumbles on it. My company just released an open source embedded XML database for the .NET platform called Nxdb. It's under the Apache 2.0 license and has been in development and use internally for several years. It's basically a binding to a cross-compiled (using IKVM) version of BaseX (a fantastic Java XML database) along with extra functionality for the embedded use case and the .NET environment. The project page is here: https://dracorp.assembla.com/spaces/nxdb
XML works well for this type of data store given that as long as the content you're trying to store is serializable to text you can store complex hierarchical trees. In fact, if you access the database directly, you don't ever even have to touch "XML". It can also be queried with XQuery, a powerful and complete query language.
(Hopefully linking to my own project doesn't irk the mods, but I figured since this post directly answers the question and provides a link to a permissive OS project it would be okay).
I'm noticing an odd behavior with the XmlSerializer and generic lists (specifically List<int>). I was wondering if anyone has seen this before or knows what's going on. It appears as though the serialization works fine but the deserialization wants to add extra items to the list. The code below demonstrates the problem.
Serializable class:
public class ListTest
{
public int[] Array { get; set; }
public List<int> List { get; set; }
public ListTest()
{
Array = new[] {1, 2, 3, 4};
List = new List<int>(Array);
}
}
Test code:
ListTest listTest = new ListTest();
Debug.WriteLine("Initial Array: {0}", (object)String.Join(", ", listTest.Array));
Debug.WriteLine("Initial List: {0}", (object)String.Join(", ", listTest.List));
XmlSerializer serializer = new XmlSerializer(typeof(ListTest));
StringBuilder xml = new StringBuilder();
using(TextWriter writer = new StringWriter(xml))
{
serializer.Serialize(writer, listTest);
}
Debug.WriteLine("XML: {0}", (object)xml.ToString());
using(TextReader reader = new StringReader(xml.ToString()))
{
listTest = (ListTest) serializer.Deserialize(reader);
}
Debug.WriteLine("Deserialized Array: {0}", (object)String.Join(", ", listTest.Array));
Debug.WriteLine("Deserialized List: {0}", (object)String.Join(", ", listTest.List));
Debug output:
Initial Array: 1, 2, 3, 4
Initial List: 1, 2, 3, 4
XML: <?xml version="1.0" encoding="utf-16"?>
<ListTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Array>
<int>1</int>
<int>2</int>
<int>3</int>
<int>4</int>
</Array>
<List>
<int>1</int>
<int>2</int>
<int>3</int>
<int>4</int>
</List>
</ListTest>
Deserialized Array: 1, 2, 3, 4
Deserialized List: 1, 2, 3, 4, 1, 2, 3, 4
Notice that both the array and list appear to have serialized to XML correctly but on deserialization the array comes out correct but the list comes back with a duplicate set of items. Any ideas?
This is an old question, but I thought I'd add an answer in case anyone stumbles on it. My company just released an open source embedded XML database for the .NET platform called Nxdb. It's under the Apache 2.0 license and has been in development and use internally for several years. It's basically a binding to a cross-compiled (using IKVM) version of BaseX (a fantastic Java XML database) along with extra functionality for the embedded use case and the .NET environment. The project page is here: https://dracorp.assembla.com/spaces/nxdb
I am having a very odd problem with local variables being out of context in the Visual Studio 2010 debugger for a C# console application targeting .NET 4.0. I've searched for other similar questions on SO, but while some have the same symptoms, none seem to apply directly to this problem (they all appear to have other root causes).
The problem is that for some variables (but not all) I do not get a tooltip with their value, they do not appear in the Locals window, and I get "The name 'xyz' does not exist in the current context" if I add them to the Watch window. It appears to affect some variables but not others, and I can't figure out a pattern (it doesn't seem to be based on member vs. local, class vs. struct, or any other differentiator). I've restarted my computer and Visual Studio, verified I'm in a clean Debug build, made sure the debugging frame is correct, made sure to refresh the variables in the watch screen, and attempted various spells and incantations.
I've included a screenshoot below (bigger version at http://i.stack.imgur.com/JTFBT.png).
Any thoughts?
EDIT:
Adding some additional information:
The problem is repeatable. The exact same variables either work or don't work, even after completely shutting down and restarting Visual Studio. This leads me to believe there's actually something systematic going wrong rather than just memory corruption or something.
I've also discovered that it appears to be related to the try-catch block. If I position the breakpoint outside the try statement I can see any of the in-scope variables normally. As soon as the execution point enters the try statement all the variables outside the try block become inaccessible and I can only access the ones inside the try statement. It's almost as though the debugger is treating the try block as a separate method (though you can see the code/compiler still does have access to in-scope variables). Has anyone seen this behavior before?
ANOTHER EDIT:
I (partially) take back what I said about the try-catch being suspect - it appears that in this portion of the code the debugger exhibits this odd taking stuff out of context for any enclosing block. For example, if I set a breakpoint directly inside the foreach statement in the screenshot I can see the "port" variable value on each iteration, but none of the variables outside the foreach statement (which disappear as soon as I enter the foreach block). Then as soon as you enter the try block, the "port" variable suddenly goes away. This is getting really weird.
Also, as requested, the code for the entire method is below.
private void ConfigureAnnouncerSockets(XDocument configDocument)
{
XElement socketsElement = configDocument.XPathSelectElement("/Configuration/Network/AnnouncerSockets");
bool useDefault = true;
if (socketsElement != null)
{
//Use the default announcers? (they will be added at the end)
XAttribute defaultAttribute = socketsElement.Attribute("useDefault");
if (defaultAttribute != null)
{
useDefault = Convert.ToBoolean(defaultAttribute);
}
//Get the default frequency
int defaultFrequency = Announcer.DefaultFrequency;
XAttribute frequencyAttribute = socketsElement.Attribute("frequency");
if (frequencyAttribute != null)
{
defaultFrequency = Convert.ToInt32(frequencyAttribute.Value);
}
//Get all sockets
foreach (XElement socketElement in socketsElement.XPathSelectElements("./Socket"))
{
//Get the address
IPAddress address = IPAddress.Broadcast;
string addressAttribute = (string)socketElement.Attribute("address");
if(!GetAddress(addressAttribute, ref address, true))
{
Intelliplex.Log.Warn("Invalid announcer socket address: " + addressAttribute);
continue;
}
//Get the local address
IPAddress localAddress = null;
string localAddressAttribute = (string)socketElement.Attribute("localAddress");
if(!GetAddress(localAddressAttribute, ref localAddress, false))
{
Intelliplex.Log.Warn("Invalid announcer socket local address: " + localAddressAttribute);
continue;
}
//Get the port(s)
List<int> ports = new List<int>();
string[] ranges = ((string)socketElement.Attribute("port")).Split(new[] { ',' });
foreach (string range in ranges)
{
string[] portPair = range.Split(new[] { '-' });
int firstPort = Convert.ToInt32(portPair[0]);
int lastPort = portPair.Length > 1 ? Convert.ToInt32(portPair[1]) : firstPort;
do
{
ports.Add(firstPort);
} while (++firstPort <= lastPort);
}
//Get the local port
int localPort = socketElement.Attribute("localPort") != null
? Convert.ToInt32((string)socketElement.Attribute("localPort")) : 0;
//Get the frequency
int frequency = socketElement.Attribute("frequency") != null
? Convert.ToInt32((string)socketElement.Attribute("frequency")) : defaultFrequency;
//Create the socket(s) and add it/them to the manager
foreach (int port in ports)
{
try
{
IPEndPoint endPoint = new IPEndPoint(address, port);
IPEndPoint localEndPoint = localAddress == null
? new IPEndPoint(IPAddress.Any, 0) : new IPEndPoint(localAddress, localPort);
Announcer socket = new Announcer(frequency, endPoint, localEndPoint);
AnnouncerSockets.Add(socket);
}
catch (Exception ex)
{
Intelliplex.Log.Warn("Could not add announcer socket: " + ex.Message);
}
}
}
}
//Add default announcement sockets?
if (useDefault)
{
ConfigureDefaultAnnouncerSockets();
}
}
So it turns out this is related to a bug in PostSharp. I had been using PostSharp but removed all aspects from my code and ensured that none were applied. I also verified with Reflector that the methods were intact in the assembly. However, it appears simply referencing PostSharp triggers some kind of manipulation of the debugging symbols that causes this problem. A (little) more information can be found here:
http://www.sharpcrafters.com/forum/Topic5794-21-1.aspx#bm7927
Also, in the release notes for the latest PostSharp hotfix states one of the fixed issues in hotfix 2.1.5.6 is "Debugging symbols: local variable symbols lost in implicit iterators."
When I installed the latest and greatest PostSharp the problem went away and the universe returned to normal. Hopefully this question/answer will help anyone else using PostSharp who stumbles on this odd behavior before the next official PostSharp release. Make sure you're on hotfix 2.1.5.6 or greater (given the severity of the bug, this probably should have been an actual release).
Thanks for all the help everyone.
It's been a long time since this question was posted, but since there's no answer and I stumbled here too, I figured I'd add one. I copied the same factory code you did (from the StackOverflow answer here) and had the same problem. I found the solution at this StackOverflow answer.
It turns out Visual Studio 2010 (which I'm assuming you're using) has a problem with std::make_pair. Just use std::pair<std::string,LevelObject*(*)()> instead and you'll be good to go. At least that resolved this exact same problem for me.
I am considering using Qt for developing a new XML-based application that makes heavy use of XQuery. I am especially excited about the way QtXmlPatterns apparently lets you query any data with an appropriate model as if it were XML. I've used XML databases and other query engines in the past (mainly Saxon and BaseX) and am wondering about the performance of XQuery under Qt. I've seen some older evidence that it's a little slow, but that's probably out of date by now.
Does anyone have any (recent) benchmarks comparing Qt XQuery to other XQuery engines. What about anecdotal evidence? Does the Qt XQuery engine allow the use of indexes or other ways to speed up queries through preprocessing (I couldn't find any)?
I am currently working on a network tool that needs to decode/encode a particular protocol that packs fields into dense bit arrays at arbitrary positions. For example, one part of the protocol uses 3 bytes to represent a number of different fields:
Bit Position(s) Length (In Bits) Type
0 1 bool
1-5 5 int
6-13 8 int
14-22 9 uint
23 1 bool
As you can see, several of the fields span multiple bytes. Many (most) are also shorter than the built-in type that might be used to represent them, such as the first int field which is only 5 bits long. In these cases, the most significant bits of the target type (such as an Int32 or Int16) should be padded with 0 to make up the difference.
My problem is that I am having a difficult time processing this kind of data. Specifically, I am having a hard time figuring out how to efficiently get arbitrary length bit arrays, populate them with the appropriate bits from the source buffer, pad them to match the target type, and convert the padded bit arrays to the target type. In an ideal world, I would be able to take the byte[3] in the example above and call a method like GetInt32(byte[] bytes, int startBit, int length).
The closest thing in the wild that I've found is a BitStream class, but it appears to want individual values to line up on byte/word boundaries (and the half-streaming/half-indexed access convention of the class makes it a little confusing).
My own first attempt was to use the BitArray class, but that proved somewhat unwieldy. It's easy enough to stuff all the bits from the buffer into a large BitArray, transfer only the ones you want from the source BitArray to a new temporary BitArray, and then convert that into the target value...but it seems wrong, and very time consuming.
I am now considering a class like the following that references (or creates) a source/target byte[] buffer along with an offset and provides get and set methods for certain target types. The tricky part is that getting/setting values may span multiple bytes.
class BitField
{
private readonly byte[] _bytes;
private readonly int _offset;
public BitField(byte[] bytes)
: this(bytes, 0)
{
}
public BitField(byte[] bytes, int offset)
{
_bytes = bytes;
_offset = offset;
}
public BitField(int size)
: this(new byte[size], 0)
{
}
public bool this[int bit]
{
get { return IsSet(bit); }
set { if (value) Set(bit); else Clear(bit); }
}
public bool IsSet(int bit)
{
return (_bytes[_offset + (bit / 8)] & (1 << (bit % 8))) != 0;
}
public void Set(int bit)
{
_bytes[_offset + (bit / 8)] |= unchecked((byte)(1 << (bit % 8)));
}
public void Clear(int bit)
{
_bytes[_offset + (bit / 8)] &= unchecked((byte)~(1 << (bit % 8)));
}
//startIndex = the index of the bit at which to start fetching the value
//length = the number of bits to include - may be less than 32 in which case
//the most significant bits of the target type should be padded with 0
public int GetInt32(int startIndex, int length)
{
//NEED CODE HERE
}
//startIndex = the index of the bit at which to start storing the value
//length = the number of bits to use, if less than the number of bits required
//for the source type, precision may be lost
//value = the value to store
public void SetValue(int startIndex, int length, int value)
{
//NEED CODE HERE
}
//Other Get.../Set... methods go here
}
I am looking for any guidance in this area such as third-party libraries, algorithms for getting/setting values at arbitrary bit positions that span multiple bytes, feedback on my approach, etc. I included the class above for clarification and am not necessarily looking for code to fill it in (though I won't argue if someone wants to work it out!).
As promised, here is the class I ended up creating for this purpose. It will wrap an arbitrary byte array at an optionally specified index and allowing reading/writing at the bit level. It provides methods for reading/writing arbitrary blocks of bits from other byte arrays or for reading/writing primitive values with user-defined offsets and lengths. It works very well for my situation and solves the exact question I asked above. However, it does have a couple shortcomings. The first is that it is obviously not greatly documented - I just haven't had the time. The second is that there are no bounds or other checks. It also currently requires the MiscUtil library to provide endian conversion. All that said, hopefully this can help solve or serve as a starting point for someone else with a similar use case.
internal class BitField
{
private readonly byte[] _bytes;
private readonly int _offset;
private EndianBitConverter _bitConverter = EndianBitConverter.Big;
public BitField(byte[] bytes)
: this(bytes, 0)
{
}
//offset = the offset (in bytes) into the wrapped byte array
public BitField(byte[] bytes, int offset)
{
_bytes = bytes;
_offset = offset;
}
public BitField(int size)
: this(new byte[size], 0)
{
}
//fill == true = initially set all bits to 1
public BitField(int size, bool fill)
: this(new byte[size], 0)
{
if (!fill) return;
for(int i = 0 ; i < size ; i++)
{
_bytes[i] = 0xff;
}
}
public byte[] Bytes
{
get { return _bytes; }
}
public int Offset
{
get { return _offset; }
}
public EndianBitConverter BitConverter
{
get { return _bitConverter; }
set { _bitConverter = value; }
}
public bool this[int bit]
{
get { return IsBitSet(bit); }
set { if (value) SetBit(bit); else ClearBit(bit); }
}
public bool IsBitSet(int bit)
{
return (_bytes[_offset + (bit / 8)] & (1 << (7 - (bit % 8)))) != 0;
}
public void SetBit(int bit)
{
_bytes[_offset + (bit / 8)] |= unchecked((byte)(1 << (7 - (bit % 8))));
}
public void ClearBit(int bit)
{
_bytes[_offset + (bit / 8)] &= unchecked((byte)~(1 << (7 - (bit % 8))));
}
//index = the index of the source BitField at which to start getting bits
//length = the number of bits to get
//size = the total number of bytes required (0 for arbitrary length return array)
//fill == true = set all padding bits to 1
public byte[] GetBytes(int index, int length, int size, bool fill)
{
if(size == 0) size = (length + 7) / 8;
BitField bitField = new BitField(size, fill);
for(int s = index, d = (size * 8) - length ; s < index + length && d < (size * 8) ; s++, d++)
{
bitField[d] = IsBitSet(s);
}
return bitField._bytes;
}
public byte[] GetBytes(int index, int length, int size)
{
return GetBytes(index, length, size, false);
}
public byte[] GetBytes(int index, int length)
{
return GetBytes(index, length, 0, false);
}
//bytesIndex = the index (in bits) into the bytes array at which to start copying
//index = the index (in bits) in this BitField at which to put the value
//length = the number of bits to copy from the bytes array
public void SetBytes(byte[] bytes, int bytesIndex, int index, int length)
{
BitField bitField = new BitField(bytes);
for (int i = 0; i < length; i++)
{
this[index + i] = bitField[bytesIndex + i];
}
}
public void SetBytes(byte[] bytes, int index, int length)
{
SetBytes(bytes, 0, index, length);
}
public void SetBytes(byte[] bytes, int index)
{
SetBytes(bytes, 0, index, bytes.Length * 8);
}
//UInt16
//index = the index (in bits) at which to start getting the value
//length = the number of bits to use for the value, if less than required the value is padded with 0
public ushort GetUInt16(int index, int length)
{
return _bitConverter.ToUInt16(GetBytes(index, length, 2), 0);
}
public ushort GetUInt16(int index)
{
return GetUInt16(index, 16);
}
//valueIndex = the index (in bits) of the value at which to start copying
//index = the index (in bits) in this BitField at which to put the value
//length = the number of bits to copy from the value
public void Set(ushort value, int valueIndex, int index, int length)
{
SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
}
public void Set(ushort value, int index)
{
Set(value, 0, index, 16);
}
//UInt32
public uint GetUInt32(int index, int length)
{
return _bitConverter.ToUInt32(GetBytes(index, length, 4), 0);
}
public uint GetUInt32(int index)
{
return GetUInt32(index, 32);
}
public void Set(uint value, int valueIndex, int index, int length)
{
SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
}
public void Set(uint value, int index)
{
Set(value, 0, index, 32);
}
//UInt64
public ulong GetUInt64(int index, int length)
{
return _bitConverter.ToUInt64(GetBytes(index, length, 8), 0);
}
public ulong GetUInt64(int index)
{
return GetUInt64(index, 64);
}
public void Set(ulong value, int valueIndex, int index, int length)
{
SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
}
public void Set(ulong value, int index)
{
Set(value, 0, index, 64);
}
//Int16
public short GetInt16(int index, int length)
{
return _bitConverter.ToInt16(GetBytes(index, length, 2, IsBitSet(index)), 0);
}
public short GetInt16(int index)
{
return GetInt16(index, 16);
}
public void Set(short value, int valueIndex, int index, int length)
{
SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
}
public void Set(short value, int index)
{
Set(value, 0, index, 16);
}
//Int32
public int GetInt32(int index, int length)
{
return _bitConverter.ToInt32(GetBytes(index, length, 4, IsBitSet(index)), 0);
}
public int GetInt32(int index)
{
return GetInt32(index, 32);
}
public void Set(int value, int valueIndex, int index, int length)
{
SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
}
public void Set(int value, int index)
{
Set(value, 0, index, 32);
}
//Int64
public long GetInt64(int index, int length)
{
return _bitConverter.ToInt64(GetBytes(index, length, 8, IsBitSet(index)), 0);
}
public long GetInt64(int index)
{
return GetInt64(index, 64);
}
public void Set(long value, int valueIndex, int index, int length)
{
SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
}
public void Set(long value, int index)
{
Set(value, 0, index, 64);
}
//Char
public char GetChar(int index, int length)
{
return _bitConverter.ToChar(GetBytes(index, length, 2), 0);
}
public char GetChar(int index)
{
return GetChar(index, 16);
}
public void Set(char value, int valueIndex, int index, int length)
{
SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
}
public void Set(char value, int index)
{
Set(value, 0, index, 16);
}
//Bool
public bool GetBool(int index, int length)
{
return _bitConverter.ToBoolean(GetBytes(index, length, 1), 0);
}
public bool GetBool(int index)
{
return GetBool(index, 8);
}
public void Set(bool value, int valueIndex, int index, int length)
{
SetBytes(_bitConverter.GetBytes(value), valueIndex, index, length);
}
public void Set(bool value, int index)
{
Set(value, 0, index, 8);
}
//Single and double precision floating point values must always use the correct number of bits
public float GetSingle(int index)
{
return _bitConverter.ToSingle(GetBytes(index, 32, 4), 0);
}
public void SetSingle(float value, int index)
{
SetBytes(_bitConverter.GetBytes(value), 0, index, 32);
}
public double GetDouble(int index)
{
return _bitConverter.ToDouble(GetBytes(index, 64, 8), 0);
}
public void SetDouble(double value, int index)
{
SetBytes(_bitConverter.GetBytes(value), 0, index, 64);
}
}
I am in the process of developing a special-purpose network tool with some packet sniffing and decoding capabilities. I am looking for languages designed to assist in the dissection/decoding of arbitrary packet formats. Idealy, the solution should be based on open standards. There are related questions on SO, but most deal with the full lifecycle of packet sniffing (I don't care so much about the capture, there are other libraries that do that well).
In general, what I'm looking for is a language and supporting framework for the declaritive definition of packet formats and corresponding run-time decoding. Because this problem can be generalized to any non-network binary data, a solution that does this for arbitrary binary streams would also be in scope. I am a little surprised no such standard currently exists in a mature and robust state (at least that I could find) - though there seem to be a lot of interesting but not-quite-right and almost-there projects (see below). Perhaps that speaks to the difficulty of the problem, or maybe to a lack of demand.
By way of example, I'm interested in technologies and ideas similar to the following (in no particular order):
I'm not necessarily looking for a complete solution here (though if someone knows of one I haven't covered, that would be great). I'm more interested in comments or anecdotes about the technologies I've indicated above as well as pointers or ideas for routes I haven't thought of or covered.
In both cases you're using an extension method, returning an object, and then using that object as the input to the next method in the chain. The second example is close to a Fluent Interface (but not quite). I think in your two examples the critical distinction is one of mutability. Do you want the original input to be modified (the second example) or not (the first example)? The "correct" answer depends on the situation and context.
You can probably use XPath to easily get to the right data and manually fill in the Location properties (or even better, add the code to the Location class). The key is the XPath extension methods for Linq to XML. Particularly look at XPathSelectElement and XPathEvaluate:
loc.Name = elements.XPathSelectElement("//Name").Value;
loc.Latitude = Convert.ToInt64(elements.XPathSelectElement("//Latitude").Value);
loc.Longitude = Convert.ToInt64(elements.XPathSelectElement("//Longitude").Value);
//loc.AddressLine = ??? (Not sure what the intended value is here...)
loc.FormattedAddress = elements.XPathSelectElement("//FormattedAddress").Value;
loc.PostalCode = elements.XPathSelectElement("//PostalCode").Value;
Here is an answer as a single Linq expression:
List<string> source = new List<string>();
source.Add("Cat, Animal, 2");
source.Add("Dog, Animal, 3");
source.Add("Luke, Human, 1");
source.Add("Owl, Animal, 0");
List<string> dest = new List<string>(
source
.Select(s => s.Split(new []{',', ' '}, StringSplitOptions.RemoveEmptyEntries))
.Select(s => { if(s[1] == "Animal") s[2] = (Convert.ToInt32(s[2]) + 1).ToString(); return s; })
.Select(s => String.Join(", ", s))
);
In my current project I use IKVM to cross-compile several Java libraries that deal with various aspects of XML. These libraries are then integrated with several .NET libraries and my mainline code. Everything works fine, but I suspect that there are several inefficiencies, especially in the area of stream-based data access.
Many of the Java libraries can accept streaming SAX classes or other streaming objects such as OutputStream, etc. In some cases I can wrap the appropriate Java class in a coorisponding .NET subclass to bridge the gap and provide seamless streaming between the two languages. For example, creating a class that derives from both the .NET MemoryStream and the Java OutputStream. In most cases however, the interface is challenging and I am left passing whole strings around - even though I have streams available on the .NET side and the Java side accepts (different) stream classes (and vice versa).
In general my question is if anyone has encountered similar problems passing data to/from IKVM compiled libraries using streams and how were they solved or mitigated? Do any third-party solutions exist to help bridge this gap? For example, code that provides Java SAX wrappers for .NET XmlReader and/or XmlWriter would be very useful.
XDocument inDoc = XDocument.Parse(
"<parent><child id='1'>test1</child><child id='2'>test2</child></parent>");
XDocument outDoc = XDocument.Parse(
String.Format("<parent><children>{0}</children></parent>",
String.Join(";",
inDoc.Root.Elements()
.Select(e => e.Attribute("id").Value + "|" + e.Value)
.ToArray())));
In general, the unsafe keyword allows you direct access to memory and therefore bypasses all verification and safety checks by the CLR.
Here's a good article on the use and impact of unsafe code:
http://blogs.msdn.com/b/sebby1234/archive/2006/04/05/565090.aspx
If the part that needs to be colored is unique in the original in terms of hue (and you could make it unique by changing it in the source image through Photoshop or something), this should do the trick. It works by locking an input bitmap (so each pixel can be manipulated) and then converting each pixel to HSB so that hue (the "color") of the pixel can be adjusted if it falls within a certain range of hues. This will have a nice effect because gradients such as shadow and slight variances will also be colorized correctly.
Colorizing code:
Bitmap bmp = ...
byte minHue = 0;
byte maxHue = 10;
byte newHue = 128;
...
BitmapData bmpData = bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height),
ImageLockMode.ReadWrite, bmp.PixelFormat);
IntPtr ptr = bmpData.Scan0;
int bytes = bmpData.Stride * bmpData.Height;
byte[] rgbValues = new byte[bytes];
System.Runtime.InteropServices.Marshal.Copy(ptr, rgbValues, 0, bytes);
for (int c = 0; c < rgbValues.Length; c += 4)
{
HSBColor hsb = new HSBColor(Color.FromArgb(
rgbValues[c + 3], rgbValues[c + 2],
rgbValues[c + 1], rgbValues[c]));
if(hsb.H > minHue && hsb.H < maxHue)
{
hsb.H = Convert.ToByte(newHue);
}
Color color = hsb.ToRGB();
rgbValues[c] = color.B;
rgbValues[c + 1] = color.G;
rgbValues[c + 2] = color.R;
rgbValues[c + 3] = color.A;
}
System.Runtime.InteropServices.Marshal.Copy(rgbValues, 0, ptr, bytes);
bmp.UnlockBits(bmpData);
HSBColor.cs (from ZedGraph):
/// <summary>
/// Hue-Saturation-Brightness Color class to store a color value, and to manage conversions
/// to and from RGB colors in the <see cref="Color" /> struct.
/// </summary>
/// <remarks>
/// This class is based on code from http://www.cs.rit.edu/~ncs/color/ by Eugene Vishnevsky.
/// This struct stores the hue, saturation, brightness, and alpha values internally as
/// <see cref="byte" /> values from 0 to 255. The hue represents a fraction of the 360 degrees
/// of color space available. The saturation is the color intensity, where 0 represents gray scale
/// and 255 is the most colored. For the brightness, 0 represents black and 255
/// represents white.
/// </remarks>
[Serializable]
public struct HSBColor
{
/// <summary>
/// The color hue value, ranging from 0 to 255.
/// </summary>
/// <remarks>
/// This property is actually a rescaling of the 360 degrees on the color wheel to 255
/// possible values. Therefore, every 42.5 units is a new sector, with the following
/// convention: red=0, yellow=42.5, green=85, cyan=127.5, blue=170, magenta=212.5
/// </remarks>
public byte H;
/// <summary>
/// The color saturation (intensity) value, ranging from 0 (gray scale) to 255 (most colored).
/// </summary>
public byte S;
/// <summary>
/// The brightness value, ranging from 0 (black) to 255 (white).
/// </summary>
public byte B;
/// <summary>
/// The alpha value (opacity), ranging from 0 (transparent) to 255 (opaque).
/// </summary>
public byte A;
/// <summary>
/// Constructor to load an <see cref="HSBColor" /> struct from hue, saturation and
/// brightness values
/// </summary>
/// <param name="h">The color hue value, ranging from 0 to 255</param>
/// <param name="s">The color saturation (intensity) value, ranging from 0 (gray scale)
/// to 255 (most colored)</param>
/// <param name="b">The brightness value, ranging from 0 (black) to 255 (white)</param>
public HSBColor(int h, int s, int b)
{
this.H = (byte)h;
this.S = (byte)s;
this.B = (byte)b;
this.A = 255;
}
/// <summary>
/// Constructor to load an <see cref="HSBColor" /> struct from hue, saturation,
/// brightness, and alpha values
/// </summary>
/// <param name="h">The color hue value, ranging from 0 to 255</param>
/// <param name="s">The color saturation (intensity) value, ranging from 0 (gray scale)
/// to 255 (most colored)</param>
/// <param name="b">The brightness value, ranging from 0 (black) to 255 (white)</param>
/// <param name="a">The alpha value (opacity), ranging from 0 (transparent) to
/// 255 (opaque)</param>
public HSBColor(int a, int h, int s, int b)
: this(h, s, b)
{
this.A = (byte)a;
}
/// <summary>
/// Constructor to load an <see cref="HSBColor" /> struct from a system
/// <see cref="Color" /> struct.
/// </summary>
/// <param name="color">An rgb <see cref="Color" /> struct containing the equivalent
/// color you want to generate</param>
public HSBColor(Color color)
{
this = FromRGB(color);
}
/// <summary>
/// Implicit conversion operator to convert directly from an <see cref="HSBColor" /> to
/// a <see cref="Color" /> struct.
/// </summary>
/// <param name="hsbColor">The <see cref="HSBColor" /> struct to be converted</param>
/// <returns>An equivalent <see cref="Color" /> struct that can be used in the GDI+
/// graphics library</returns>
public static implicit operator Color(HSBColor hsbColor)
{
return ToRGB(hsbColor);
}
/// <summary>
/// Convert an <see cref="HSBColor" /> value to an equivalent <see cref="Color" /> value.
/// </summary>
/// <remarks>
/// This method is based on code from http://www.cs.rit.edu/~ncs/color/ by Eugene Vishnevsky.
/// </remarks>
/// <param name="hsbColor">The <see cref="HSBColor" /> struct to be converted</param>
/// <returns>An equivalent <see cref="Color" /> struct, compatible with the GDI+ library</returns>
public static Color ToRGB(HSBColor hsbColor)
{
Color rgbColor = Color.Black;
// Determine which sector of the color wheel contains this hue
// hsbColor.H ranges from 0 to 255, and there are 6 sectors, so 42.5 per sector
int sector = (int)Math.Floor((double)hsbColor.H / 42.5);
// Calculate where the hue lies within the sector for interpolation purpose
double fraction = (double)hsbColor.H / 42.5 - (double)sector;
double sFrac = (double)hsbColor.S / 255.0;
byte p = (byte)(((double)hsbColor.B * (1.0 - sFrac)) + 0.5);
byte q = (byte)(((double)hsbColor.B * (1.0 - sFrac * fraction)) + 0.5);
byte t = (byte)(((double)hsbColor.B * (1.0 - sFrac * (1.0 - fraction))) + 0.5);
switch (sector)
{
case 0: // red - yellow
rgbColor = Color.FromArgb(hsbColor.A, hsbColor.B, t, p);
break;
case 1: // yellow - green
rgbColor = Color.FromArgb(hsbColor.A, q, hsbColor.B, p);
break;
case 2: // green - cyan
rgbColor = Color.FromArgb(hsbColor.A, p, hsbColor.B, t);
break;
case 3: // cyan - blue
rgbColor = Color.FromArgb(hsbColor.A, p, q, hsbColor.B);
break;
case 4: // blue - magenta
rgbColor = Color.FromArgb(hsbColor.A, t, p, hsbColor.B);
break;
case 5:
default: // magenta - red
rgbColor = Color.FromArgb(hsbColor.A, hsbColor.B, p, q);
break;
}
return rgbColor;
}
/// <summary>
/// Convert this <see cref="HSBColor" /> value to an equivalent <see cref="Color" /> value.
/// </summary>
/// <remarks>
/// This method is based on code from http://www.cs.rit.edu/~ncs/color/ by Eugene Vishnevsky.
/// </remarks>
/// <returns>An equivalent <see cref="Color" /> struct, compatible with the GDI+ library</returns>
public Color ToRGB()
{
return ToRGB(this);
}
/// <summary>
/// Convert a <see cref="Color" /> value to an equivalent <see cref="HSBColor" /> value.
/// </summary>
/// <remarks>
/// This method is based on code from http://www.cs.rit.edu/~ncs/color/ by Eugene Vishnevsky.
/// </remarks>
/// <returns>An equivalent <see cref="HSBColor" /> struct</returns>
public HSBColor FromRGB()
{
return FromRGB(this);
}
/// <summary>
/// Convert a <see cref="Color" /> value to an equivalent <see cref="HSBColor" /> value.
/// </summary>
/// <remarks>
/// This method is based on code from http://www.cs.rit.edu/~ncs/color/ by Eugene Vishnevsky.
/// </remarks>
/// <param name="rgbColor">The <see cref="Color" /> struct to be converted</param>
/// <returns>An equivalent <see cref="HSBColor" /> struct</returns>
public static HSBColor FromRGB(Color rgbColor)
{
double r = (double)rgbColor.R / 255.0;
double g = (double)rgbColor.G / 255.0;
double b = (double)rgbColor.B / 255.0;
double min = Math.Min(Math.Min(r, g), b);
double max = Math.Max(Math.Max(r, g), b);
HSBColor hsbColor = new HSBColor(rgbColor.A, 0, 0, 0);
hsbColor.B = (byte)(max * 255.0 + 0.5);
double delta = max - min;
if (max != 0.0)
{
hsbColor.S = (byte)(delta / max * 255.0 + 0.5);
}
else
{
hsbColor.S = 0;
hsbColor.H = 0;
return hsbColor;
}
double h;
if (r == max)
h = (g - b) / delta; // between yellow & magenta
else if (g == max)
h = 2 + (b - r) / delta; // between cyan & yellow
else
h = 4 + (r - g) / delta; // between magenta & cyan
hsbColor.H = (byte)(h * 42.5);
if (hsbColor.H < 0)
hsbColor.H += 255;
return hsbColor;
}
}
There are several embeddable C# web servers, but none that I know of that will load and use Apache plugins.
My first thought for this problem is to create a simple TextReader derivative that is responsible for buffering input from the stream. This class would then be used to feed an XmlReader. The TextReader derivative could fairly easily scan the incoming content looking for complete "blocks" of XML (a complete element with starting and ending brackets, a text fragment, a full attribute, etc.). It could also provide a flag to the calling code to indicate when one or more "blocks" are available so it can ask for the next XML node from the XmlReader, which would trigger sending that block from the TextReader derivative and removing it from the buffer.
Edit: Here's a quick and dirty example. I have no idea if it works perfectly (I haven't tested it), but it gets across the idea I was trying to convey.
public class StreamingXmlTextReader : TextReader
{
private readonly Queue<string> _blocks = new Queue<string>();
private string _buffer = String.Empty;
private string _currentBlock = null;
private int _currentPosition = 0;
//Returns if there are blocks available and the XmlReader can go to the next XML node
public bool AddFromStream(string content)
{
//Here is where we would can for simple blocks of XML
//This simple chunking algorithm just uses a closing angle bracket
//Not sure if/how well this will work in practice, but you get the idea
_buffer = _buffer + content;
int start = 0;
int end = _buffer.IndexOf('>');
while(end != -1)
{
_blocks.Enqueue(_buffer.Substring(start, end - start));
start = end + 1;
end = _buffer.IndexOf('>', start);
}
//Store the leftover if there is any
_buffer = end < _buffer.Length
? _buffer.Substring(start, _buffer.Length - start) : String.Empty;
return BlocksAvailable;
}
//Lets the caller know if any blocks are currently available, signaling the XmlReader can ask for another node
public bool BlocksAvailable { get { return _blocks.Count > 0; } }
public override int Read()
{
if (_currentBlock != null && _currentPosition < _currentBlock.Length - 1)
{
//Get the next character in this block
return _currentBlock[_currentPosition++];
}
if(BlocksAvailable)
{
_currentBlock = _blocks.Dequeue();
_currentPosition = 0;
return _currentBlock[0];
}
return -1;
}
}
Intermediate code is very similar to assembly in that it contains a limited set of instructions. The runtime is then able to reliably and consistently operate on this (somewhat) small set of instructions without having to worry about parsing the language, etc. It can therefore gain performance improvements and make optimizations.
If name is a List<T> then you probably want name.RemoveAt(i) if I understand the question correctly.
Alternatively, you could just use name.RemoveAll(n => n == selectName); instead of the for loop.
At least for the check part you could use an Enum with the [Flags] attribute to create bit field. That might be a little more extensible if you add more methods in the future. You could then use a simple lookup table and do away with the PredicateOptionSet class. Example:
[Flags]
public enum PredicateOption
{
IsCaseSensitive, IsRegularExpression, IsMultipleTerms
};
...
public Dictionary<PredicateOption, Func<SearchResult, bool>> _predicates
= new Dictionary<PredicateOption, Func<SearchResult, bool>>();
_predicates.Add(PredicateOption.IsCaseSensitive, result => Search(result.Name));
_predicates.Add(PredicateOption.IsCaseSensitive | PredicateOption.IsMultipleTerms,
result => SearchCaseInsensitiveMultiple(result.Name));
....
PredicateOption option = PredicateOption.IsCaseSensitive | PredicateOption.IsMultipleTerms;
SearchResults.Where(_predicates[option]);
@CharlesB - I tried some different scenarios but could never get it to show me anything in the web interface. I assume it was either auto-merging the differences (which appeared to happen in most cases) or otherwise resolving by choosing one or the other, but that doesn't really give me a definitive answer since my few tests probably didn't cover all possible cases. My guess is that it attempts a merge first and then uses the most recent on a conflict, but I'm not positive.
That's excellent - hadn't come across that class before. Does Lazy<T>.Value block while the value is being initialized (I assume it would have to)? If so, this looks perfect.
@JamieSee - I actually thought about that, but my concern was that after the first item is added, any subsiquent instantiations will be allowed to proceed (since at least one item is in the cache), even if the additional cache items are still being added in the first constructor. This could lead to the following instances accessing the incomplete cache while it's still under construction.
@payo - How did you get a reference to the Type instance for the derived type? this.GetType() isn't available (no "this") in the static constructor. Perhaps I wasn't clear enough - I don't know the derived types (and there are more than one). I.e., I can't do Type type = typeof(SomeDerivedType); in the base class static constructor to build the cache.
And thanks to you for keeping the accepted answer updated.
I know it's at least available in the .NET API (see the changelog here: aws.amazon.com/releasenotes/.NET/6888944574638729)
Okay, okay - fine. I installed IIS Express and it's working fine now. It wasn't hard to configure either. Except I ran into a permissions problem getting alternate testing hostnames to work. I found the solution here: learn.iis.net/page.aspx/1005/…
It does support multi-tenancy, though seems a little odd based on this SO question: stackoverflow.com/questions/5677694/…
You are correct about not working on Windows CE - I wasn't thinking and missed the connection there between CE and .NET CF. Though the Nxdb code itself would probably run (or could be made to run) under the compact framework, it relies heavily on IKVM which is not currently compatible with .NET CF.
Thanks for the answers - as I mentioned in a comment below, I didn't realize the deserializer was able to mutate the state of members during deserialization (in most cases I think it just sets new values, right?). Is List<T> the only class it can do this with? Is there some reference indicating which classes the deserializer can mutate?
Thanks, moving the initialization out of the constructor did indeed solve the problem. I misunderstood how the deserializer works. I didn't realize it mutates existing objects, I thought it always used the set accessor to write entirely new instances of children to the newly constructed object (in which case the list created in the constructor would have just been forgotten). You learn something everyday.
@Yahia While I can't post the entire source repository as Hans requested, I can go ahead and post the content of the method in question. It's not very exotic - mainly just reading in an XML file and configuring some networking classes based on that. I'll put the code up in the question.
One more bit of intel - the problem is repeatable. The exact same variables either work or don't work, even after completely shutting down and restarting Visual Studio. This leads me to believe there's actually something systematic going wrong rather than just memory corruption or something.
@Hans I am running SP1 (just checked the About window to be sure - actual version is 10.0.30319.1). Also on Windows 7 64 if that might help. Unfortunately, I can't publish the entire repository - it's both company proprietary and pretty complex (I.e., it's not set up for someone else to just take build right now). I work with several very complex solutions though and this is the first time I've seen this issue.
@phoog Thanks for the suggestion. I just checked the project properties and "Optimize code" is unchecked.