2010年9月24日星期五

C# Producer-Consumer Threading

由於有大量資料要從Database query出來, 但又不想使User Interface 停下來, 在Stackoverflow得到幫助, 用Producer-Consumer model 來解決

流程是: User Interface 需要query資料 >> producer 就produce一個delegate(就是c++ 的function pointer) >> 一大堆的query(其實是fucntion), 就由consumer來執行 >> 等所有有關的query都做完, 就invoke main thread(interface的thread) 的有關control refresh一下, 讓它顯示query了回來的資料

e.g. 有一個label要顯示用戶名稱

labelA.Text = queryUserName(userId, labelA);

public static string queryUserName(uint id, Control control)
{
    QueryFunction threadStart = delegate { StaticRes.queryUserNameThread(id, control); };
    Program.g_QueryThread.Enqueue(threadStart);
    return "Querying User Name...";
}

StaticRes.queryUserNameThread 就是雖要時間較長, 用來query資料的function

以下是class QueryThread的implementation:

public delegate void QueryFunction();

/// <summary>
/// Act as a consumer to the queries produced by the DataGridViewCustomCell
/// </summary>
public class QueryThread
{
    private struct QueueItem
    {
        public Delegate _target;
        public UInt64 _id;

        public QueueItem(Delegate target, UInt64 id)
        {
            _target = target;
            _id = id;
        }
    }
    private Object _syncEvents = new Object();
    private Queue<QueueItem> _queryQueue = new Queue<QueueItem>();
    private EventWaitHandle _waitHandle = new EventWaitHandle(false, EventResetMode.AutoReset);

    Producer queryProducer;
    Consumer queryConsumer;

    public QueryThread()
    {
        queryProducer = new Producer(_queryQueue, _syncEvents, _waitHandle);
        queryConsumer = new Consumer(_queryQueue, _syncEvents);

        Thread producerThread = new Thread(queryProducer.ThreadRun);
        Thread consumerThread = new Thread(queryConsumer.ThreadRun);

        producerThread.IsBackground = true;
        consumerThread.IsBackground = true;

        producerThread.Start();
        consumerThread.Start();
    }

    public void Enqueue(Delegate item, UInt64 id)
    {
        QueueItem queueItem = new QueueItem(item, id);
        _queryQueue.Enqueue(queueItem);

        _waitHandle.Set();
    }
}

按下來就是producer 跟consumer thread
producer主要是fire event 使consumer thread 開始digest那些query
consumer就是執行enqueue了的query function

class Producer
{
    private readonly Queue<QueueItem> _queue;
    private Object _sync;
    private EventWaitHandle _handle;

    public Producer(Queue<QueueItem> q, Object sync, EventWaitHandle waitHandle)
    {
        _queue = q;
        _sync = sync;
        _handle = waitHandle;
    }
    
    public void ThreadRun()
    {
        lock (_sync)
        {
            while (true)
            {
                //wait until item is enqueued
                _handle.WaitOne();

                //enqueued, tell worker thread
                if (_queue.Count > 0)
                {
                    Monitor.Pulse(_sync);
                    Monitor.Wait(_sync,0);
                 }
             }
        }
    }
}

class Consumer
{
    private readonly Queue<QueueItem> _queue;
    private Object _sync;

    public Consumer(Queue<QueueItem> q, Object sync)
    {
        _queue = q;
        _sync = sync;
    }

    public void ThreadRun()
    {
        lock (_sync)
        {
            Delegate query;
            while (true)
            {
                while (_queue.Count == 0)
                {
                    if (Program.g_CustomDialog.Visible == true)
                    {
                        Program.g_CustomDialog.DialogResult = DialogResult.OK;
                    }

                    Monitor.Wait(_sync);

                }

                QueueItem item = _queue.Dequeue();
                query = item._target;
                query.DynamicInvoke(null);
            }
        }
    }
}

後來有人在stackoverflow上說 lock後的 while(true) loop 會容易引致deadlock
由於我的threading implementation比較簡單, 就不作改動了, 下次記緊就好

msdn上對c# threading的教學: How to: Synchronize a Producer and a Consumer Thread
但不要跟他的implementation, 看一看這裡

2010年9月10日星期五

Code Syntax HighLight on Blogger

Code Highlighting on Blogger (from CraftyFella's Blog)

Add the following code before the </head> tag in the css template of your Blogger
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css' rel='stylesheet' type='text/css'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCpp.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCSharp.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCss.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPhp.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPython.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushRuby.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushVb.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPerl.js' type='text/javascript'></script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.config.clipboardSwf = 'http://alexgorbatchev.com/pub/sh/current/scripts/clipboard.swf';
SyntaxHighlighter.all();
</script>

How to use:
<pre class="brush: csharp">
// Comment
public class Testing {
public Testing() {
}
 
public void Method() {
/* Another Comment
on multiple lines */
int x = 9;
}
}
</pre>

Escaped HTML:
Before posting, try to translate it into escaped html is better
Convert Raw HTML to Escaped HTML

2010年9月9日星期四

[.NET C#] DataTable GROUP BY

由於有大量Transaction資料(超過十萬行)在資料庫, 同時想做多個不同的統計, 為免加重資料庫的負荷, 唯有一次過query所需資料, 再在client那邊進行統計運算

Google一下後, 發現.net中的DataTable竟然沒有GROUP BY的功能
找了一些用foreach loop + Rows.Add 來自行製作一個新Table的教學
類似:
private DataTable GroupBy(DataTable myTable)
{
    DataTable dt = new DataTable();
    dt.Columns.Add("ColumnName");
    foreach (DataRow row in myTable.Rows)
    {
        string s = row["ColumnName"].ToString();
        DataRow[] drx = dt.Select("ColumnName= '" + s + "'");
        if (drx.Length == 0)
        {
            dt.Rows.Add(new object[] { s });
        }
    }
    return dt;
}

可是這麼一來就每一項統計都要自行製作一套Group by的運作, 延續性太低, 於是我在arstechnica forum發現了一個有用的function, 就拿了來改改, 比較dynamic一點點

public struct StatGroupByInfo
{
    public string table_Filter;
    public string[] column_FilterExpr;
    public string[] column_names;
    public string[] column_functions;
    public Type[] column_types;
    public string[] group_by;       //also sort by the first group_by column

    public StatGroupByInfo(string table_filter, string[] col_filters, string[] col_names, string[] col_functions, Type[] col_type, string[] group)
    {
        table_Filter = table_filter;
        column_FilterExpr = col_filters;
        column_names = col_names;
        column_functions = col_functions;
        column_types = col_type;
        group_by = group;
    }
}

static class DataSetFunctions
{
    public static DataTable GroupBy(DataTable table, StatGroupByInfo info)
    {
        return GroupBy(table, info.table_Filter, info.column_FilterExpr, info.column_names, info.column_functions, info.column_types, info.group_by);
    }

    public static DataTable GroupBy(DataTable table, string table_filter, string[] filterExpr, string[] aggregate_columns, string[] aggregate_functions, Type[] column_types, string[] group_by_columns)
    {
        //Filter the table
        DataView dvView = new DataView(table, table_filter, group_by_columns[0] + " ASC", DataViewRowState.CurrentRows);

        //Group the table by column
        DataTable dtGrouped = dvView.ToTable(true, group_by_columns);

        //Implement aggregate functions
        for (int i = 0; i < aggregate_columns.Length; i++)
        {
            dtGrouped.Columns.Add(aggregate_columns[i], column_types[i]);
        }

        foreach (DataRow row in dtGrouped.Rows)
        {
            List<string> filter_parts = new List<string>();

            for (int i = 0; i < group_by_columns.Length; i++)
            {
                filter_parts.Add(String.Format("[{0}] = '{1}'", group_by_columns[i], row[group_by_columns[i]].ToString().Replace("'", "''")));
            }

            string filter = String.Join(" AND ", filter_parts.ToArray());

            for (int i = 0; i < aggregate_columns.Length; i++)
            {
                string columnFilter = filter;
                if (!string.IsNullOrEmpty(filterExpr[i].Trim()))
                {
                    columnFilter = columnFilter + " AND " + filterExpr[i];

                    if (!string.IsNullOrEmpty(table_filter.Trim()))
                    {
                        columnFilter = table_filter + " AND " + columnFilter;
                    }
                }

                row[aggregate_columns[i]] = table.Compute(aggregate_functions[i], columnFilter);

                if (row[aggregate_columns[i]] == DBNull.Value && column_types[i] == typeof(uint))
                {
                    row[aggregate_columns[i]] = 0;
                }
            }
        }

        return dtGrouped;
    }
}

用法:
StatGroupByInfo groupByInfo = new StatGroupByInfo( "", new string[] { "[action] = 1 or [action] = 2",  "[action] = 3 or [action] = 4", "[action] = 5 or [action] = 6"}, new string[]{ "itemmallPoint", "nonItemmallPoint", "totalPoint" }, new string[]{ "sum(usePoint)", "sum(usePoint)", "sum(usePoint)" }, new Type[]{ typeof(uint), typeof(uint), typeof(uint) }, new string[]{ "date" } ),

DataTable _dt = DataSetFunctions.GroupBy(DbRes.Transaction, groupByInfo);

解說:
要customize 一個group by 的operation, 主要是用StatGroupByInfo裡的variables
public string table_Filter: 就是原來那個table的filter, 先濾掉不在統計範圍內的entry
public string[] column_FilterExpr: 每個column不同的filter
public string[] column_names: column的名稱, 可以自訂
public string[] column_functions: 每個column的aggregate function, 如 count, max, avg 等, 可參考MSDN
public Type[] column_types: 每個column的type, 要跟上面aggregate的return type對應
public string[] group_by: 要進行 GROUP BY 的columns


[後記]
後來發現有很多人寫的DataHelperClass, 用來manage dataset的data:
geekswithblogs一堆Helper Class 中的DataSetHelper Class that I am using
jiezhi@cnblogs
MSDN 上的一些文章

2010年9月6日星期一

C# Get Enum from Number or String

這裡找到一個由number/string 回傳 Enum 的 generic function

public static T StringToEnum<T>(string name)
{
  return (T)Enum.Parse(typeof(T), name);
}

public T NumToEnum<T>(int number)
{
   return (T)Enum.ToObject(typeof(T), number);
}

Example:
public enum DaysOfWeek
{
   Monday,
   Tuesday,
   Wednesday,
   Thursday,
   Friday,
   Saturday,
   Sunday
}

public enum MonthsInYear
{
   January,
   February,
   March,
   April,
   May,
   June,
   July,
   August,
   September,
   October,
   November,
   December
}

DaysOfWeek d = StringToEnum<DaysOfWeek>("Monday");
//d is now DaysOfWeek.Monday

MonthsInYear m = StringToEnum<MonthsInYear>("January");
//m is now MonthsInYear.January


如果Enum中沒有相對的value, 就會throw exception:
DaysOfWeek d = StringToEnum<DaysOfWeek>("Katillsday");
//throws an ArgumentException
//Requested value "Katillsday" was not found.

它提供了一個簡單的argument check, 這就行了:
if(Enum.IsDefined(typeof(DaysOfWeek), "Katillsday"))
    StringToEnum<DaysOfWeek>("Katillsday");


可是轉眼又看到有關performance及安全性的問題:The danger of over simplification
MSDN上也說不要用Enum.IsDefine(), M$大哥啊, 忽悠我啊?!

既然問題對我影響好像不大......就當過關了
但還是看了相關的資料, 說不定日後再要面對這個問題(希望不要)
C# Enum.Parse() Bug
More C# 'enum' Wackiness
Common Type System—Type Safety Limitations 這個讚

2010年9月3日星期五

C# Google Chart API

昨天找到一個for C#的 Google Chart API: GoogleChartSharp
很多東西都基本上齊全, 但裡面好像沒有提供 text format with scaling 的功能(找不到"chds"字串)
我做了個簡單的湊合一下

在Chart.cs裡加了一段 :

private string dataScale;
public void SetDataScale(int xLowerBound, int xUpperBound, int yLowerBound, int yUpperBound)
{
    this.dataScale += String.Format("{0},{1},{2},{3}", xLowerBound, xUpperBound, yLowerBound, yUpperBound);
}


protected virtual void collectUrlElements()
{
    .........

    // data scale
    if(dataScale != null)
    {
        urlElements.Enqueue(String.Format("chds={0}", this.dataScale));
    }

    .........
}
我後來才發現google提供的encode方法最高支援4095的int, 而且用extended encoding的話,  scaling好像是自動的, 即不能用"chds"字串來設定data scale("chds"只適用於text format)

所以只好把原來的data 乘以一個scalar(maxValue就是圖表資料中的最大值)

double scalar = 4095 / (double)maxValue;
dataValue *= scalar;
這樣就可(不知是不是完全)解決scaling的問題

2010年9月2日星期四

C# DataColumn To Array

在網上找到一段將 DataColumn 轉為 Array的Code
沒有Iteration, 很簡潔

String[] rowValuesForColumn =
    Array.ConvertAll<DataRow, String>(
    dataTable.Select(),
    delegate(DataRow row) { return (String) row[columnName]; }
 );