Black Magic Code

A good thing to think about.

2008-12-03T15:09:00.000-08:00

This is short and to the point. Productivity. My personal reflection to this, he blames the Operating System(OS). It is not actually that where the problems is. You can get a productive experience on most OS:es. The problem is the software put on the OS.

Opinions, everybody has them.

2008-09-01T13:18:00.000-07:00

There is plenty of C++ hate to go around. There are forums, blogs and social bookmarking sites where the hate sometimes appear. This is usually posted through a web browser. The post are about how bad the language is in managing memory(because it doesn't), how hard it is to parse(there are some fun ambiguities) and so on. People claim in these posts that it takes a genius to get all these things right. But the thing is that bothers me is that... Ehm.... the majority of the web browsers are implemented in C++. In other words, they rely on the very language that they hate so much.

Further, this is the fun part. By anecdotal evidence, the people that spread the most hate about C++ have their head up so far the "Web 2.0" ass, that they don't see that their beloved web experience is run by C++.

Sorry and hate to break it the "Web 2.0" fanboys C++ is good for something. And it does not take a genius to program with it. Your favorite language of the hour is not good enough to build a complete web browser, that can bring the "Web 2.0" experience to people. Prove me wrong please.

This is getting out of hand now.

2008-07-07T11:32:00.000-07:00

Adobe has released their latest Adobe Reader. Funny stuff can be read here and here. Yes if you want to read PDF:s with the adobes own reader you must have atleast +200 megabytes available on your hard drive.

This is a sorry state of affairs. This is not about bloat anymore. It is intrusive behavior by the software company. It is not new in the industry as a whole. Microsoft has done it, the redistributable for .NET 3.5 is a whopping 197 megabytes. Yeah old Realplayer stole file associations for media files types(persistent rumors say that they even didn't support). Apple did a nice trick with an update.

This is not about bad programming with way too many features A.K.A. bloat. This is wasting the resources of their users without "asking" them what they want. If they even bother to ask what the users want they have to take an action to avoid the feature(s).

Now ask yourselves this! What gives Adobe, Apple, Microsoft and RealNetworks the right, to steal disc real estate, your file associations and install other software that you downloaded?

Skills shortage?

2008-06-18T11:51:00.000-07:00

Found on the web: Skills shortage hits games firms. Poor poor game companies, they can't find skilled slaves anymore. They blame it on math education and whatnot. Well, sorry but your reputation is not that good. If the reputation was better and it would be a good job, *gasp* people would actually bother with the necessary math.

It is about eight years ago I visited a game company with a friend. My friend knew a member of the team that did work on the Refactor 2 engine for Battlefield 1942. We visited him at his office, in the evening around eight or nine. Because he was still working. About two years prior to the release date.

Usually what I've met game programmers they are insanely skilled at what they do. I can be humble and say that I'm not nearly as good of a programmer as most game developers are. But also AFAIK the average game developer drone is not paid that well compared to the skill set and the hours he puts in.

Is there a shortage, I would not say so. Most people don't want to work under poor conditions for a lousy pay and they even have to know advanced math. People that know advanced math usually are smarter than that.

About outperforming STL

2008-05-31T12:27:00.000-07:00

I can't really say that I'm chocked by the conclusion in the article by Sergey Solyanik called STL vector performance. But it is a fine piece on what is wrong with microbenchmarking. It is also a case study of people who critique C++/STL should actually learn it before they critique it.

I've spent a great deal of time in learning C++ and STL. It is complex language and library where there a number of finer points to remember. There are a number of other things that I cringe on but I'll limit myself to "microbenchmarking" and actually learning the library/language.

You can make any language/library look bad with a microbenchmark. Because it limits itself to a couple of features and a particular way of solving a problem. The biggest problem in the STL vector performance post is that he is comparing a narrow problem of filling an array with values and filling the same values into a much more complex and capable vector template. All that complexity is adding abstractions that cost a fair bit of cycles that in the case of integers can be outperformed with realloc or if you have known size preallocating. The post revolves around a very unclear problem. What exactly is vector bad at? This is where the language and library knowledge comes in.

As a C++ programmer I am in the habit of using vectors and arrays the same way. Because the STL library implementation gives a nice little detail with vectors. The elements of a vector are stored in a continuous area of memory exactly like an array. You can access the internal array of a vector through the [] and it will have same speed as a regular array access. But to keep this continuous array of objects this way it needs to reallocate the memory when you need more space for your objects. i.e. insertions of objects at the end of vector is not constant. This the complexity of push_back, check if you need more space before you insert and actually doing an insert. For a vector of int:s it could be quite large compared to a array of int:s reallocated with realloc(). If I'm ready to sacrifice access speed to the elements I could use something like deque from STL instead. Insertion in the front and back is guaranteed to be "constant", because it does not provide the guarantee of continuous storage. Internally it looks quite different, read all about it in vector vs. deque.

What if you need fast access by index? You should use a STL vector and fine tune your growth with reserve to increase capacity. Adding to the vector with push_back might still be costly depending on what you are pushing. Because when you reserve you don't actually add anything to the vector you just make sure it has enough memory(a.k.a. capacity) when you add more objects onto it. Read all about it in Uses and Abuses of Vector.

Further comment on "How software companies die!"

2008-02-18T10:46:00.000-08:00

People that work with sales and marketing are for software developers a strange bunch. For a software developer it is totally nonsensical why anyone would come up with ideas like: Gluing a couple of eyes on a rock and call it a Pet Rock. But marketing people see how brilliant it is. They think: "You can sell anything with the right marketing!".

But you can't sell everything with marketing. It depends on the nature of the "product". As soon as the buyer expects it to something and be usable the meme of "anything can be sold with marketing" becomes useless. Marketing and sales do not understand the nature of the beast of software. When you cram in more features in the poor developers schedule two things happen. They have less time to fix old bugs and the buyers (sometimes referred to as the market) get confused by too many options and features. If you ever try to fight a marketing department when they say "the market demands more options and features", you will get slammed each and every time with "developers are too dumb too understand the users" or "developers don't understand marketing". So when they get their way the result is an unmaintainable mess of buggy bloatware with features no one knows how to use. Thanks marketing.

Are there any solutions for this mess?

Lets look at a related business of hardware. Processors to be precise. I feel like an old fart when I think back to the dancing clean room guys. But that was exactly what happened with Intel. Marketing took over because they had a successful campaign with brightly colored people dancing telling about MMX enabled processors. They started to butt in to the engineering decisions of the company. Intel had an CEO that allowed that. To make a long story short, it resulted in a Pentium IV with a magical number of 3.2. But a lot of engineering compromises where necessary to reach the magical number. Competitors caught up, they had engineers making R&D decisions. Resulted in a different gamble from their part. AMD trounced Intel with good engineering, they wouldn't have had the chance if Intel had made decisions based on technical merits than reaching a magical marketing number. Intel shifted to an engineering based decisions and they are making solid products ever since.

This is also very true for software. It is better to make solid products rather than market the crap out of compromises.

Sounds awfully familiar.

2008-02-18T07:08:00.000-08:00

This pops up now and again when you read about software engineering and programming on the web. How Software Companies Die by Orson Scott Card. It is scary how often you see this pattern repeated. It seems to be repeated at my current workplace. Considering it is written in 1995, the remaining question is will people ever learn.

I'll post a longer comment on it later.

Here is a scary thought for you.

2008-02-01T09:57:00.000-08:00

The faith of billions of euros(or dollars or whatever currency you prefer) are decided by Excel sheets with quick n' dirty VBA hacks. These hacks are sometimes built by people with little or no training in software engineering. This is the scariest bit of information that I have got from the SocGen scandal. Jeromé Kerviel isn't a computer genius, he knows a little VBA. Don't believe me read this.

Who would have known, an Excel bug could trigger the next stock market crash. Microsoft decides the faith of more money than they are themselves worth.

I don't know about you, but to me this looks like an accident waiting to happen. And if we don't do anything about, it will not be pretty.

This can't be for real...

2007-10-25T15:00:00.000-07:00

I have huge amounts of power available on computer. A dual core with a lot of RAM available. This dual core is 64-bit capable so of course my OS(the latest Ubuntu if your curious) supports it. On occasion I use it to browse the web. I landed on "Aftonbladet" and the JavaScript coupled with a couple of Flash ads. Guess what, my beautiful dual core with 2 gigabytes of RAM memory became sluggish! From visiting a web page!

Why would anyone in their right mind think that everything in the future of computing is going to be Rich Internet Applications? Seriously, we have enormous amounts of hardware in our homes to browse the web. But the crazy thing here is that our monster machines idles when we use them to browse the web. We are wasting clock cycles on inefficient software that is supposed run the wave of the future by enabling new whizzbang Rich Internet Applications! Think about it... is this really the way to go?

I hate Windows Vista.

2007-05-17T11:16:00.000-07:00

I have for the first time professionally crossed paths with Vista. I hate the Vista experience. How many of those mind numbing dialogs of Allow and Continue buttons I have to press? How about the thing where you drag and drop a file into a folder where it already exists and it asks you every time that if you want to Move and Replace or Copy and Replace, with the cancel where the continue used to be on XP.

But you can customize away most of these dialogs, but is it easy? No.... after I have seen the guides for disabling the dialogs. It has left me wondering why people insist on that Vista is granny-ready but not the nearest Ubuntu.

About PHP security...

2007-04-06T04:02:00.000-07:00

I saw a link on programming.reddit that was about PHP security. It was a few slides of which vulnerabilities a inexperienced PHP coder can introduce. Take a look for yourself. These are the most common and quite easily exploitable holes in your garden variety of PHP "application".

Now the holes in them selves aren't that interesting. It is this comment I found in the discussion on reddit about the pdf slides above.

That's one of the things I don't like about PHP's culture and philosophy. They think it's the programmer's fault. The programmer should have to do all this by hand every single time. The programmer shouldn't be given any help or supplied with any set of intelligent defaults.

That is the best thing ever written about secure coding in PHP. The PHP tool providers has a flawed culture when it comes to aiding the programmer in security. It has a very low barrier of entry to start programming in and it means that inexperienced people will use it to start coding. But the security aspects of writing less vulnerable code in PHP has a steep learning curve. Also this has to be done manually each and every time you accept user input. Do you see budding PHP coders thinking about security when they start out. Do you think that your average PHP monkey is thinking about security when sitting in his hot little room at the local web dev shop with his boss breathing down his neck and asking "is it done yet?" every five seconds.

Of course the programmer has some responsibility to see that his code is securely written, but the programming language should help him if he does a few mistakes in writing secure code. And that is up to Zend company to change their culture and philosophy when it comes to this.

Managment using bad metrics.

2007-02-28T12:03:00.000-08:00

I have always thought that I'm one of the lucky ones that is not measured with Lines Of Code or something similarly bad metric on my day job as a software developer. Managers has always tried to measure the productivity of a developer, and some of the metrics are notoriously bad. Like Lines Of Code(LOC). LOC counts are really bad at measuring productivity when it comes to comparing developer A who does nice concise well designed code and developer B that is a notorious copy paste programmer without even considering design. With LOC measuring developer B is more productive. Most of the software industry has realized that developers B are bad for you, without things like design and forethought when developing software is bad for the end result. But the message has not gone through to various marketing, sales and PR people out there.

As it turns out marketing/sales at development shops has their own metric. I'll call it "Ass Heat Index". This is the metric where you as a developer is measured by the time spent with your ass in the office chair and warming it up. It seems that they have a covert sensor sensing the heat that your ass i radiating to the office chair and plots graphs accordingly. The more heat over a longer period your ass is giving away to the chair means that you are more productive than the guy sitting shorter time in his chair.

The funny thing is that we wasted over an hour in an meeting with our PR director learning about this index and thus lowering all the Ass-Heat-Indexes for our developers.

Seriously if your schedule is slipping, do something else about than complain to your developers that they aren't spending enough time in their chairs.

I've seen them all...

2006-12-04T11:00:00.000-08:00

I've got a link for you, continuing on what is wrong with the software industry. This time it is Steve Mcconnell filling in on the Classic Mistakes you can do in a project.

This is almost to good to be true....

2006-11-28T14:33:00.000-08:00

I was over at reddit today and found a very interesting link. An interview with Bjarne Stroustrup the designer of C++. Titled: "The Problem with Programming". Now if you have a hard time figuring out my last rambling about the software industry. This is the article to read.

Bjarne states:

I think the real problem is that "we" (that is, we software developers) are in a permanent state of emergency, grasping at straws to get our work done. We perform many minor miracles through trial and error, excessive use of brute force, and lots and lots of testing, but--so often--it's not enough.
Software developers have become adept at the difficult art of building reasonably reliable systems out of unreliable parts. The snag is that often we do not know exactly how we did it: a system just "sort of evolved" into something minimally acceptable. Personally, I prefer to know when a system will work, and why it will.

And when asked about how to fix it.

In theory, the answer is simple: educate our software developers better, use more-appropriate design methods, and design for flexibility and for the long haul. Reward correct, solid, and safe systems. Punish sloppiness.

But he is not naive and continues with

In reality, that's impossible. People reward developers who deliver software that is cheap, buggy, and first.

Read the whole interview it is worth your time.

One thing wrong with the software industry.

2006-11-22T10:43:00.000-08:00

This is a post I have wanted to write a long time. This will be an article about what is so depressing about the craft of creating computer software and hopefully I'll be writing a few more of these articles.

The thing is that I know the secrets of why so many software projects just fail. Well they aren't exactly secrets, they are more like lessons that many people in positions to make decisions about software development just don't learn. If they where secrets, I would be a billionaire by now with a streak of successful projects behind me.

Lesson one: Artificial deadlines, milestones, timelines or time related whatever that is way too short, have a habit of biting you in the ass in the end. This is the most costly mistake that many companies do. Examples of this practice is: "This can't possibly take more than a couple of hours!" or "This must be done in x months, or the company must close up shop!". If a developer or developer manager buys into this then the impending train-wreck is almost always is a fact.

The single most important reason why it is a train-wreck just waiting to happen is that: Developers under stress is almost always more prone to produce low quality code. It could be possible that the code that was produced under pressure actually works, sort of. Lets say that it produces the desirable result. Of course there are a few bugs, but those get fixed... no big wreck that I warned you of earlier. This is a possible outcome, also in my experience not too uncommon. So where is the train-wreck?

It is in the maintenance phase of your application. All of a sudden when you have stressed out code it lacks some things like uhm... design because there was no time, comments because the developer that was under stress thought "I'll fill in that later, after the project is shipped and done!" and the next crisis surfaced(it was those pesky bugreports that poured in!).

The psychology behind the keyboard that produces this is that with an artificial timeline/deadline it is treated like a 100 m dash instead of a marathon. This is the important thing, when you impose stress on a developer he will code like there is no tomorrow and the result has no tomorrow either. He expends all his energy on the very beginning of the race and just codes away without thinking of the rest of it. What happens if the bug that surfaces six months after delivery and it turns out that it is a huge design flaw? There is no simple way of "patching" it, it takes a lot of time and money. What happens if the feature request that come in after one month is crucial for your customer/customers and it doesn't fit in your design without rewriting significant portions of the code? Same result. You might lose the customer(s), because they lack patience and/or find a cheaper vendor that (can) provide what they want. That is expensive isn't it. That is the wreck in all its glory...

This was an entry inspired by parts of this article.

This looks like great fun....

2006-11-20T04:35:00.000-08:00

I've been keeping my eye on Damn Vulnerable Linux for a while now. It looks like a terrific educational tool... who knows it might teach a trick or two for experienced people also.

kernel.randomize_va_space....

2006-11-14T13:33:00.000-08:00

This is actually a quite a cool feature that I recently ran across on my Linux box at home. It is an Ubuntu 6.10 and I tried to run some old shellcode. For the life of me I didn't get it to work. In this was examples with specially crafted vulnerable software just to demonstrate. So I forgot about for a week or so, and one day last week I started to hunt down what was wrong. I found it... all never 2.6 linux kernels have the nasty habit of randomizing the address space of new processes. Why would you want to do that may you ask...

Because classic shellcode exploits rely on a known address space to execute its malicious payload. When you randomize your address space on the new processes means that your computer becomes more secure against some exploits.

Notice the some because it is not a panacea. It is not going to fix all your vulnerabilities. Also it is possible to turn it off if you by some odd chance experience problems. "sysctl -w kernel.randomize_va_space=0". And I suspect that there is going to be a number of System administrators out there just turn it of "because it could pose a problem", if they knew about it.

Why I program linux in the evening...

2006-10-26T11:46:00.000-07:00

Maybe it is time for an update of the blog.

Being a developer you can almost guess what platform I develop for on my day job. You don't need superior mind reading skills to know that it is windows. Because the majority of the computers out there are running Windows. So it is natural that some program for that platform.

But looking at the development on the Windows world for developers I've recently becoming more and more worried on what direction it is taking. It is not something highly technical as the newfangled WinFX API that is the new "managed" interface to work with windows. The scariest part of the new direction is the Microsoft commitment of having the biggest media delivery platform. It means that they pay more attention to please the big media companies than the actual needs of their customers. You know the people that are actually buying the Microsoft products.

I know it is a conspiracy theory, but look at the circumstantial evidence: Microsoft patched the "vulnerability" that allowed removal of DRM in Windows Media faster than the VML exploit in the worlds most widely used browser. And of course there is the difference in the media capabilities of the different windows editions, there is more possibilities with patchguard than without. I'm going to hold my view what I think of patchguard for myself. But I'll tell you this much, I think it doesn't help to keep the questionable software out of your computer, even if it is marketed as such. The Sony rootkit fiasco should have taught us that, what happens if Sony f***s up again?

With Linux I don't have to worry of the direction of the company that delivers your favorite Linux distribution, because you can always change distribution.

"Are you incompetent, it is just adding a single button!"

2006-09-13T11:07:00.000-07:00

Oh no not again. It has been years since I last encountered the "it is just a button!". For f****ks sake, adding a button to a Graphical User Interface is no actual work, but making that button do something is.... uhm, anything from quite easy to impossible. The most annoying thing, is that there is no way of explaining to the person that can be nearly impossible especially when the person refuses to provide a spec for it.

The discussion starts with a "add a button", when the developer asks what the button is supposed to do and the response is "it shouldn't be that difficult". Now the developer thinks "great, for the first time in my career I get a proper spec!" and all he gets is dissappointment. He is provided with basicly what the button is supposed to be labeled and he is left by his manager to figure out what the label means. No spec, no cursory requirements, just a button called something and alot of questions...

And I have run into it too many times to count by now...

On development of security applications.

2006-07-29T04:05:00.000-07:00

Over at assa's blog an interresting post appeared... I'm not going to comment on the whole thing just to say that he is righteous. I will comment on a small passage on the business case of the anti - virus/spyware/hacker software companies.

"How about education of PC users? That is something mostly neglected by anti- vendors, just because educated users making educated decisions could possibly harm their business." - Assarbad

This quote shows the short-sightedness of the whole anti- industry. Education of users to have good habits when using internet services like mail or the web would improve the experience for everyone. Because some of the spam you receive daily in your inbox is from someones computer who clicked on a link that he/she shouldn't have clicked on. The solution is not relying on out dated definition files of your particular security software. I hope that you run some security software but, it is the last line of defense, first and foremost you must have good habits. What those good habits are... Maybe I'll run a series on that later...

But I know it when I see it...

2006-05-23T11:52:00.000-07:00

A swedish television investigative journalism "show" called "Insider" is currently running a investigation about the money behind porn industry. Now why I use the term "show" for the television program is that it is not totally unbiased journalism. The first episode of this investigation was little about the background on the history and starting to follow the money. One of the points was that Internet Service Providers(ISP) makes a bundle out of this. I could almost buy their point if it wasn't for their "expert": Per Hellquist. I'll translate a quote for you: "It is possible to screen pictures for skin colored pixels, and from there determine if a picture is pornographic or not!". This in the context that it is feasible for ISP:s to intercept porn traffic by this method. This is so wrong at so many levels.

To start of Per works at Symantec in sweden. Nothing wrong with working at symantec ( or is there? ), but it begs the question how he got his job in the first place? He lacks some fundamental knowledge about how the internet works and lets not forget some common sense.

Lets try to break down what is wrong with the statement.

1. When you transfer a file on the internet, the file is divided in to chunks called packets and routed through the network!

This means that to analyze a picture all packets of that picture must be captured ( also rerequest a packet if it was malformed during transport ) and then assembled to the file, first after that you can start your analysis of the picture. But if you capture your packets during transit then it is too late to stop the transfer! To solve these problems we must somehow proxy the transfer. If by some luck we succeseed in capturing and analyzing the content of the image the next problem rears it's ugly head.

2. Picture analysis is not cheap!

It takes enourmous amounts of memory and processing power to keep up with just the http traffic of a connection. But these companies are usually rich and they could actually take a slightly higher fee for a slower connection just to filter it for the customers. Which customer would not like to pay more and get less content???

3. Picture analysis is hard!

Really, thats why it is so computationally expensive. First of all what is skin color, the shade of colors differs slightly with race. That makes it a little harder but lets assume that we can solve this problem. How would the computer determine image comes from a lingerie webshop or porn page? Number of pixels that are skin colored? The problem here is actually very well known. A human can actually tell what is smut or not, but it is a matter of personal taste. A computer has a very hard time to "see the difference".

How about C...

2006-05-03T11:35:00.001-07:00

Recently I started studying the deep stuff in C++ with things like meta template programming(MTP) and so on. But also realized the problem with the cool stuff. In my previous post I described programmers that are not proficient enough with C++ and why I'd like to see them move on to the managed land. You should not use MTP in environments where people are not ready to learn it. Because you may not be around until End of Life of that particular code to maintain it. I only know a few people that I would dare get near such code unless it was really carefully crafted.

Maybe we should go back to C instead. Don't get me wrong, C++ is extremly powerful and versatile programming language but maybe it has too much power. C has a less features but it has plenty of power. Less features means less worry about something biting your fellow programmers in the *ss.

Maybe someone says: "This is 2006 and C is very old!"... Yes it is 2006 and C is old, but what is wrong with that? I'll tell you whats wrong with that! Nothing... There is no expiry date on programming languages. It is fairly portable and has a stable standard and has plenty of power.

The C++ Illuminati...

2006-03-28T12:04:00.000-08:00

There almost seems to be a secret society of C++ programmers out there (I'm not a member... even if I was I wouldn't tell you). They are rooting for the success of Java and C#(pronounced C-Suck). You would ask yourselves why?

If you ever had attempted any significant C++ development you should have developed a love-hate relationship for this programming language. This because there a number of things that you need to understand and remember when programming. Not only all obvious things like copy-constructors and virtual destructors, also things like what methods are silently created, their default behaviour and how static variables must be created to exactly determine when they are initialized. Of course you have worked with programmers that doesn't know or care about these details. They think that they can't be bothered with this, but if you are a better programmer than they are you know how important these "finer-points" are. So you end up hating the language instead of the programmers who can't program in it. But yet you love its power.

This is the reason why the C++ illuminati is hoping for success of the managed languages. Someone once said to me "The devil is in the details!". Managed environments can't give that close attention to details as the native languages can. The most mindless and booring development projects out there are fine with managed langauges. These projects are good money and the talentless developers who couldn't code their way out of a wet paper bag need a language and environment that saves them from themselves. When more talentless cubicle drones move to languages that suites them, the more real work talented C++ programmers can do instead of cleaning up the miserable messes by the talentless programmers. This is where C++ gets a real revolution... atleast that is what the C++ illuminati hopes.

A short introduction.

2006-03-22T13:02:00.000-08:00

It is important to remember what software is about. The things that code does. At the lowest level it only manipulates 1 and 0. To this dual state we attach layers of abstraction. Grouping these primary pieces to numbers. These numbers are sometimes turned into characters. A series of characters are grouped together into strings and so on. My goal when trying to become a good developer is to effortlessly move between these abstraction levels efforlessly.

I'm a C++ programmer by trade. By night I use Python, Assembler, C and of course C++.