Category Archives: Development – .NET
Hourly Contract vs Full Time Proper Calculation
Some of my friends at work quitted the full time job and went for contract (paid by hour without benefits) for more money and flexibility. When I asked them the equivalent annual salary, they showed me a pretty good number. Is that the actual money going into their pocket? Wait…
There are so many on-line “contractor calculator” and “rate formulas” on the Internet, however many of them are wrong because many items have not been taken into account. Therefore, I came up with this spreadsheet and it can be fully customized. Hope this helps other people.
Download the Excel file
- ContractVsFullTime.xlsx (Excel 2010 Xlsx, RIGHT-CLICK and SAVE AS)
In Canada, I.T. industry for experienced contractors has the average hourly rate from $60 to $150/hr. Let’s take the lower boundary $60/hr job as an example. With other formulas/online calculcators, they show you more than $100K/yr. But when you put that on my spreadsheet, you are only getting around $89K/yr. Play with it, don’t let the job agent fool you. Compare this with your full time salary, negotiate for better rate and decide if it is a risk to quit. Good luck.
Entity Framework – Table Per Hierarchy (Single Table Inheritance)
In the past, I have worked with some ORM software such as X-tensive DataObject .NET, which generated the database schema based on the Entity models. This type of ORM is called Table-per-Type Inheritance, i.e. each table is mapped to single Entity in the model (*note 1).
However, this doesn’t always work if we have an existing database or we want to maximize the performance of the relational queries or we want to save space in database. In many cases, we don’t want the relational database to look like the Entity model.
Entity Framework supports Table-per-Hierarchy Inheritance, i.e. different types in the Entity model that inherits from a base class and all mapped to a single table (*note 1).
This is an extension to the articles I found on the Internet:
- Inheritance and Associations with Entity Framework (Moses’s Blog)
- Single Table Inheritance in Entity Framework (Zeeshan Hirani)
(*note 1 were quotes from Moses’s Blog)
School Example
Let’s look at a very traditional “Person”, “Student” and “Instructor” example, I have created a simple database. With “PersonTypeID” defined as a foreign key in a separated table and this is the major difference from the 2 links above.
In Visual Studio 2010, I used the “Generate Model from Database” to create the initial EDMX file for my web application. It helped 50% of the work and I strongly recommended. Yet, there are some manual work to do because Visual Studio doesn’t know what the conceptual level would look like.
At this point, it is pure relational-to-object. If you start coding in LINQ, it’s like LINQ-to-SQL. However, our objective is to make it object oriented. So, let’s add 2 new Entities in the design tool, namely “Student” and “Instructor” which inherits from base class “Person“. Also, we have to delete the “HireDate” and “EnrollmentDate” from PERSON entity and reproduce the 2 properties.
If you compile right away, you will get the error:
Error 3005: Problem in mapping fragments starting at line xxx:Must specify mapping for all types in…. An Entity with Key (PK) will not round-trip when: Entity is type [xxxxxxxxx.yyyyyyyyy]
Conditional Mapping
To fix this, we have to add conditional mapping for the 2 Entities (Student and Instructor). This tells the engine to save the value 1 (PersonTypeID) to database (Person table) when we create a Person object.
We now have the conditional mapping. Let’s compile again, but you will get this error:
Error 3032: Problem in mapping fragments starting at line xxx:Condition member ‘Person.PersonTypeID’ with a condition other than ‘IsNull=False’ is mapped. Either remove the condition on Person.PersonTypeID or remove it from the mapping.
In conceptual world, we don’t actually need to use “PersonTypeID” since we have already used conditional mapping to define the type of the person. So, we need to delete the “PersonalTypeID” from “Person” Entity. Then, we need to delete the whole “PersonType” Entity too! Here is the final model should look like:
If you are not using the “PeronalType” table to store the value, i.e. you are using integer or string, probably you may need to set the “Person” table to “Abstract”, otherwise you will get another annoying error:
Error 3023: Problem in Mapping Fragments starting at line xx,xx,xx : Column xxxx has no default value and is not nullable. A column value is required to store entity data.
Create object and Save to DB
You should now be able to compile without any error. In the following sample ASPX code, we have created “Student” object and assigned some properties to it. Note that we have NEVER specified the “PersonalTypeID”.
using System.Web; using System.Web.UI; using System.Web.UI.WebControls; using System.Transactions; using System.Data.Objects; namespace EntityTest { public partial class Test : System.Web.UI.Page { protected void Page_Load(object sender, EventArgs e) { SchoolEntities dataContext = new SchoolEntities(); using (TransactionScope transactionScope = new TransactionScope()) { Student student = new Student(); student.EnrollmentDate = DateTime.Today; student.FirstName = "John"; student.LastName = "Smith"; dataContext.People.AddObject(student); dataContext.SaveChanges(SaveOptions.AcceptAllChangesAfterSave); transactionScope.Complete(); dataContext.AcceptAllChanges(); } } } } |
After running the above code, the “Student” is automatically saved with “PersonTypeID” assigned to “2″. It worked!
Using LINQ Group By and String.Join() / Aggregate() in Entity Framework 3.5
“Group By” in LINQ with “String.Join(), Aggregate()” Problem
First of all, if you ever need to use “Group By” in LINQ on Entity Framework (EF), and you want to use “String.Join()” to convert a LIST to a delimited string, you may bump into this error easily:
- “LINQ to Entities does not recognize the method ‘System.String Join(System.String, System.String[])’ method, and this method cannot be translated into a store expression. “
If you use “Aggregate()”, you will get another error message.
- LINQ to Entities does not recognize the method ‘System.String Aggregate[String](System.Collections.Generic.IEnumerable`1[System.String], System.Func`3[System.String,System.String,System.String])’ method, and this method cannot be translated into a store expression.
The most annoying part is, this works fine in LINQ-To-SQL, but not LINQ-To-Entity. After searching on the Internet, one of the solutions is to convert your object query using .ToList() and you can use String.Join(). Here is an example, Users and Roles are Entities in Data Context which are mapped as Many-To-Many (In relational database behind the scene, there are 3 tables).
For such data, we have:
jSmith Administrator
jSmith Users
We want to group them and display it in Grid:
jSmith Administrator, User
It is easy to do it in SQL and TSQL, however LINQ in Entity Framework, it is a little bit tricky.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | var result1 = (from a in users b in roles where (a.RoleCollection.Any(x => x.RoleId = b.RoleId)) group a by a.UserName into userGroup select new { UserName = userGroup.FirstOrDefault().UserName, RoleNames = userGroup.FirstOrDefault().RoleCollection.Select(x => x.RoleName) }); var result2 = (from a in result1.ToList() select new { UserName = a.UserName, RoleNames = String.Join(", ", a.RoleNames.ToArray()) }); |
The above code actually worked fine when it was binded to the Grid, “RoleNames” showed properly by separated with a delimiter, i.e. “Administrator, User”.
Performance Issue
When I looked at the code, at first I didn’t find anything wrong with it. However when I threw 50,000 records on the Users table and ran the above code again, we got serious performance issue. It took so long for the page to load and everytime you flip to another page on the Grid, it took more than 20 seconds. That is totally not usable.
LINQ’s “Group By” does not directly translated into SQL’s “Group By”, they can be very expensive. Please refer to MSDN for details. Let’s take a look at the following by moving the LINQ group by to result2:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | var result1 = (from a in users b in roles where (a.RoleCollection.Any(x => x.RoleId = b.RoleId)) select new { UserName = a.UserName, RoleNames = b.RoleName) }); var result2 = (from a in result1.ToList() group a by a.UserName into userGroup select new { UserName = userGroup.FirstOrDefault().UserName, RoleNames = String.Join(", ", (userGroup.Select(x => x.RoleNames)).ToArray()) }); |
This time, it only took 2 seconds to flip through each page on the Gird, with the same 50,000 records. That’s a 10 times difference.
To further improve on the speed, replace String.Join() with Aggregate():
1 2 3 4 5 6 | // String.Join() RoleNames = String.Join(", ", (userGroup.Select(x => x.RoleNames)).ToArray()) // Aggregate() RoleNames = (userGroup.Select(x => x.RoleNames)).Aggregate((a,b) => (a + ", " + b)) |
This improved the speed by 1 more second on getting 50,000 records.
Interesting Findings
In the 2nd code above, if you change “result1.ToList()” to “result1″ and then comment out “RoleNames” line like the following. The performance is going slower again, it took about 10 seconds to flip to next page on the Grid. Interesting…
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | var result1 = (from a in users b in roles where (a.RoleCollection.Any(x => x.RoleId = b.RoleId)) select new { UserName = a.UserName, RoleNames = b.RoleName) }); var result2 = (from a in result1 group a by a.UserName into userGroup select new { UserName = userGroup.FirstOrDefault().UserName, // RoleNames = String.Join(", ", (userGroup.Select(x => x.RoleNames)).ToArray()) }); |
It seems .ToList() and group by on the same query will make the performance faster, probably it translates to SQL properly.
If you have LINQ Profiler, you can see it behind the scene and then compare the 3 cases on SQL level. Unfortunately I don’t have such tool and I am pretty happy with the result, I ain’t gonna spend more time on this to further optimize it (perhaps it is possible).
Case Study: Error only happens on production, cannot even re-produce in debug mode?
I love debugging and reverse engineering, especially challenging one. The following C# code is an interesting one that our team came across today.
The symptom: It occasionally throws an exception only on production server, we even ran debugger on production, stepped through the code line by line and couldn’t reproduce the error. Of course, we couldn’t reproduce on developers’ machines either. Assuming all the environment are the same, except production is being a very fast and powerful server.
Business requirement: Changing password for a user in Active Directory through our application by calling a custom WCF web service.
Technical limitation: In Active Directory, administrator can only “Reset” password, they cannot “Change” password because administrator does not know the user’s old password. That’s the standard group policy. If administrator “Reset” password, the end user will get prompted and force them to change their assigned password. Business does not want this, to prevent this from happening, we can first “Reset” with a temporary password and then “Change” to new password.
There is another rule, the last 10 used password in history cannot be used again.
1 2 3 4 5 6 7 8 | // ------------------------------------------- // Code to change password tempPassword = CreatePassword(); newPassword = CreatePassword(); service.ResetPassword(email, tempPassword); service.ChangePassword(email, tempPassword, newPassword); |
1 2 3 4 5 6 7 8 9 10 11 | // ------------------------------------------- // Simplied version of create password public string CreatePassword() { Random rnd = new Random(); string str1 = GetDictionary(rnd.Next(5000)); string str2 = GetSpecialChar(rnd.Next(10)); string str3 = rnd.Next(1000).ToString(); return str1 + str2 + str3; } |
Okay, everything seems to make sense and it DOES work in DEV, QA and UAT. Now, it goes to production, the “ChangePassword()” routine occasionally throws an error saying “Password already in History”. What is wrong?
The analysis: After hours of tracing, I could finally re-produced the error. By writing everything to the LOG file without running in debug mode, I found the problem. In 1 out of 10 cases, the “tempPassword” and “newPassword” are having exactly the same value!!!!
The next question is, is it something wrong with the “CreatePassword()” function? What is the chance of creating the same password using Random? The chance should be very rare!
All of a sudden, the light bulb came up. It’s must be the RANDOM SEED! So I checked the Ticks by writing to log file.
1 2 | Random rnd = new Random(); WriteLog(DateTime.Now.Ticks.ToString()); |
Bingo!! In some cases, they have exact SAME ticks!!!! This explains why.
Why would this happen? Because the production server is so fast, calling “Random()” twice – can be executed AT THE SAME TIME in nanoseconds, so it is getting the same seed from the clock, which generates the exact same random number as well as the same password.
Random in all programming languages are based on “seed”, which is based on server’s clock. In C#, Microsoft tries to make it easy for developers by making seed parameter optional. I remember in other languages I used before, like Java and PHP, they forced you to provide your own SEED.
Solution 1 : Once I found out the problem, I shared with the team members. One of the developers suggested a solution like this:
1 2 3 | tempPassword = CreatePassword(); System.Threading.Thread.Sleep(2000); // sleep for 2 seconds newPassword = CreatePassword(); |
It will work, but…. it’s bad code and can cause performance issue. There is a better solution.
Solution 2: Forget about Random and Seeding, you can simply write a LOOP and compare the tempPassword and newPassword, if they have the same value loop it again until they get a different value. There is no way for a server to execute the 2 lines at the same time with 100% chances.
Solution 3 : To prove the finding was correct, I then tested the following code on production and it worked flawlessly, simply making the 2nd seed not the same as 1st one. However, there is possibilities of runtime overflow with seed+1. It works most of the time, but it is still dangerous.
1 2 3 | int seed = (int)DateTime.Now.Ticks; tempPassword = CreatePassword(seed); newPassword = CreatePassword(seed+1); |
The CreatePassword method needs to take one parameter when calling Random:
1 | Random rnd = new Random(seedvalue); |
Solution 4: MSDN article talks about this. [UPDATED]
http://msdn.microsoft.com/en-us/library/aa329890(VS.71).aspx
As you can see from the MSDN article, AutoSeedRandoms() passes the random instance to create the number, we can do the same:
1 2 3 4 5 6 7 8 9 10 11 12 | Random rnd = new Random(); tempPassword = CreatePassword(rnd); newPassword = CreatePassword(rnd); public string CreatePassword(Random rnd) { string str1 = GetDictionary(rnd.Next(5000)); string str2 = GetSpecialChar(rnd.Next(10)); string str3 = rnd.Next(1000).ToString(); return str1 + str2 + str3; } |
This is working perfectly fine and guarantees both password will not be the same, but you have to make sure “new Random()” is being called once. Since this is a web application, it has challenge to maintain one instance of Random throughout the whole application/session. One way is to store the Random instance in user’s SESSION or APPLICATION variable without worrying about seeding, on the other hand, this is generally bad practice for web development to use session/app variables.
Yet, there is one interesting paragraph on the same MSDN article that worths reading.
Quote from MSDN: If your application runs on a fast computer the system clock might not have time to change between invocations of this constructor; the seed value might be the same for different instances of Random. In that case, apply an algorithm to differentiate the seed value in each invocation.
For instance, the following C# expressions use a bitwise complement operation to generate two different seed values even if the system time value is the same.
1 2 3 | Random rdm1 = new Random(unchecked((int)DateTime.Now.Ticks)); Random rdm2 = new Random(~unchecked((int)DateTime.Now.Ticks)); |
Conclusion: Depending on how you want to solve this Random problem, there is always multiple ways to get from point A to point B.
An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException
One of the projects that we are working on is using Windows Communication Foundation (WCF) for web service. This *.SVC is much more powerful than standard ASP.NET *.ASMX web service. Everything is working perfectly and has been running fine in QA/UAT and we are ready to go for production launch. One day before the launch, we got this error:
“An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.
(System.ServiceModel.Security.MessageSecurityException)
Server stack trace:
at System.ServiceModel.Channels.SecurityChannelFactory
1.SecurityRequestChannel.ProcessReply(Message reply,
SecurityProtocolCorrelationState correlationState,
TimeSpan timeout)”
We were thinking that it was related to SSL, security mode or http binding but we couldn’t fingure it out. Finally, I found it in MSDN forum – it was due to system clock not sync between the web server and the web service server!!! According to Microsoft the 2 servers can only have certain time difference for security reason.
401 Access Denied error on reading a MOSS document from web part/.NET application
I am working on a web part that reads an XML template (applies to any files) from a “Document List” on MOSS. Here is the code:
Pitt is the hostname when setting up the “Web Application” in MOSS, in this case the web part is trying to connect to itself (localhost, pitt).
When the Web Part comes up, it throws an 401 Exception on the xmlDocument.Load(). I thought it was something to do with permission and should be very easy, ended up spending more than 2 days to figure this out.
Configuration:
Windows 2003 Enterprise with SP2, MOSS 2007, Visual Studio 2008, .NET 3.5 SP1
What I have tried:
1. Using “System.Net.NetworkCredential” to hardcode the username/password for the resolver, still got 401 error
2. Checked the Identity Application Pool, MOSS Service, Farm Administrator, Web Application Policy (Central Admin), Site Collection administrator, all permission related area, etc.
3. Deleted the MOSS Web Application and re-created it, didn’t help
Symptoms:
1. Internet Explorer on the MOSS server worked fine (connecting to pitt, locally), and was able to get the XML document (IIS Log showed proper credential was passed). However if you take a look closely, you can see first 2 attempts failed (401 error), however 3rd time the credential was passed (alpha/rini):
2. Web Part running locally, gets 401 error (On the IIS Log, the Credential was NOT even being passed, it was missing from the Http Request!!!!)
Problem Isolation
To isolate the problem, I wrote a very simple Windows .NET application.
Extremely Strange:
When Windows Application ran locally, get 401 Access Denied error:
When Windows Application ran remotely from another computer on the LAN, it worked ???
Workaround
After spending so much time, I came up with a workaround:
1. Delete the HOST header on IIS for your MOSS web application
2. In the code, do not use the host name. Use “localhost” instead:

This works for both web part and windows application!
Final Solution
With help from the coworkers in our team to solve this weird problem, no one had a clue why this was happening. Finally one of our team members recalled that there was similar authentication issue on production server a while ago and our Operation group changed one registry and fixed it. This lead us to a new keyword, called “Registry”. Google on this important keyword, BINGO! Found the Microsoft article: http://support.microsoft.com/kb/896861
Following the article and edited the registry, and it FIXED the problem!!!!
Further reading the article, this explains why my workaround worked, because HOST NAME did not match with the MACHINE NAME! It was a new security feature being introducted in Windows 2003 Service Pack !! Damn it.
Temperature Alert Alternative – Brando TempPer USB Hygro-Thermometer
I was looking for USB device that measures temperature of my server room remotely on the web from my cell phone. Also I want to make sure the heating system works fine when I was on vacation during winter to prevent frozen water pipes.
Searching on the Internet, I found couple of them:
- Temperature Alert (USB version $129 US, Wi-Fi $319 US)
- Brando USB Hygro-Thermometer (Measures Tempeature, Humidity – $23 US)
- Brando USB Thermometer (Measures Tempeature only, $19 US)
- UJ503 USB Thermometer (Technology company, wholesale only?)
- Sensor Soft ($199 US)
- Oregon Scientific Thermometer (Desktop display with USB, $45 US)
Which one did I buy?
Price price price. I bought the Brando USB Hygro-Themometer. Compare to Temperature Alert, I can buy 5 of them. Shipping was fast, it took around 1 week from Hong Kong to Canada.
Pros and Cons of Brando
Well, you paid for what you get. But for $20 US, what do you expect? It does what it claimed, I have no complain.
1. It’s good price, even free shipping
2. Monitoring software sucks badly. However .NET API downloadable
3. You will get “Maybe Data Error” if USB Hub is used or USB cable is being too long, wtf
4. The sensor is NOT pre-calibrated, enter the adjustment yourself
5. It is not extremely accurate, it’s about +/- 3 degree, it’s ok for my purpose
6. It measures Tempeature, Humidity (.DLL API calculates Dew Point too)
7. Both Tempeature alert and Brando indeed is RS232 device, they make it as USB. So, the driver comes with the hardware is basically a USB-SERIAL driver. It has to find the right virtual “COM” port in order to talk to the device.
Web and Mobile phone Support – I wrote my own software
None of the above products has web support or mobile phone support, I can only “Remote Desktop” to monitor my home temperature. That is not acceptable, so I wrote my own application. The sample web application is extremely simple, took me less than an hour to do it. It is a very simple web project, you have to create your own Solution in Visual Studio. It has 3 versions:
1. HTML version usage:
TempCheckHTML.aspx?TemperatureAdjustment=0&HumidityAdjustment=0
2. Plain Text version (Just show plain-text without HTML)
Get Temperature: TempCheckPlainText.aspx?TypeName=Temperature&Adjustment=0
Get Humidity : TempCheckPlainText.aspx?TypeName=Humidity&Adjustment=0
3. Web Service version (develop your own Application by talking to Web Service)
TempCheckService.asmx
Source Code download
Click to download the complete source code (Right-click, Save Target As)
(The script is totally free to use/distribute as long as you keep the copyright line)
1. Download API kit from Brando USB web site (I do not have rights to distribute the DLL)
2. Add TEMPerDll.dll to Visual Studio Reference (.NET 2.0 Framework or above is required)
3. Compile the code
Now, I can easily monitor my home temperature and humidity using my cell phone:
If you know .NET, you can easily write an NT service, write the log to SQL server and render the stats using Reporting Server, it’s way better than the software comes with the hardware. You can also use .NET compact framework to write a Windows Mobile application and talks to the web service. It’s fun.
What are other usages?
1. Monitor temperature of your Wine Cellar
2. Going for vacation during winter time where outside temperature can go as low as -30c degree, you may want to check the tempeature of your house, in case of the heating system failure. You can call a friend to help out to prevent flooding due to frozen water pipes.
3. If you have pets in your house, it’s good to remote monitor the temperature
4. Collect data and you can do your own analysis on energy usage by compare to multiple years
“DropDownList has a SelectedIndex which is invalid because it does not exist in the list of items”
I have a project that needs to dynamically build the dropdownlist, so I used DataBind. Well, for the first time it is fine, however if I reuse the control without a postback, I get the “DropDownList has a SelectedIndex which is invalid because it does not exist in the list of items” error. I tried to find the solution on Internet but none worked.
My solution is extremely simple, just add 2 lines before the actual binding, and set the DataSource to null and it solved the problem! ClearSelection() won’t work in this case.
DropDownList1.DataSource = null;
DropDownList1.DataBind();
DropDownList1.DataSource = myDataSource;
DropDownList1.DataBind();
Hope this helps other people





























