So using my poor mans sampler (and the awesome shark tool on OS X) we found that my application spends a lot (90%) of its time serializing and deserializing json. That's "not good".
I've had a lot of pain finding a good json library for C or C++. Some are one-way (encode or decode only). Some wont handle large numbers (uint64_t at least). Some can't handle unicode.
The boost version based on boot::property tree looked very promising at first. But it has a lot of nasty edge cases. (Unicode is broken unless you're using wchar_t, value nodes lose their type and become strings, certain edge cases wont serialize properly). We managed to work around all these issues, patching the boost libraries and putting hacks into our code. But the insurmountable problem is that it is SLOW.
Now I could look for another library and work around its idiosyncrasies, but boost is already the second library that we've tried to work around.. and none of the other libraries look promising (or have licenses we can work with...)
What can you do when one of your core libraries is causing most of your pain. (Bugs, performance, hacks) - rewrite it ;)
This is not a path we took lightly. But we have an alpha/beta working version of the library, and its about to go into our live servers. The code is a bit disorganized and could do with a bit of a clean up.. but our goals have been reached. It seems to parse and write json about 20x faster than our corresponding boost wrapped code. It performs far less allocations. It doesn't throw away type information. It does support arbitrary sized numbers. It does the right thing with unicode (except one edge case of properly converting some UTF8 into surrogate escaped UTF16 pairs).
Anyway I'll do a proper post on how to use it soon, but for now here's the github link:
https://github.com/mikeando/fastjson
I'd love people to play with it and leave feedback.
There's also a post about it on my company blog.
Wednesday, March 2, 2011
Monday, February 14, 2011
Poor-man sampler saves the day again.
So one of my applications was being very slow to start up when running in a VM.
Running the poormans sampler detailed in my previous post. I found that I got a heap of samples looking like this:
Thread 1 (Thread 0xb6f436f0 (LWP 1657)):
#0 0xb7828430 in __kernel_vsyscall ()
#1 0xb7370f93 in read () from /lib/tls/i686/cmov/libc.so.6
#2 0xb731aedb in _IO_file_underflow () from /lib/tls/i686/cmov/libc.so.6
#3 0xb731dcc8 in __underflow () from /lib/tls/i686/cmov/libc.so.6
#4 0xb731a888 in ?? () from /lib/tls/i686/cmov/libc.so.6
#5 0xb731c7b8 in _IO_sgetn () from /lib/tls/i686/cmov/libc.so.6
#6 0xb73103be in fread () from /lib/tls/i686/cmov/libc.so.6
#7 0x0811028f in main (argc=2, argv=0xbffccf54) at src/server.cpp:252
That is I was waiting for a read to complete. Sometimes for 5-10 seconds, sometimes for a few minutes.
What was this troublesome read? An unexpected socket call? Nope - something I would never have expected. Here's the "offending" code.
Reading from /dev/random blocks until there is enough entropy in its internal entropy pool to complete the read. On an isolated VM there's not much system noise generating entropy and so the pool was emptying quickly.
Turns out theres a non-blocking random device that uses feedback of hashed values to prevent blocking when the entropy pool is low, so switching this to using "/dev/urandom" and all was OK.
Now maybe I could have found this by doing some traditional debugging, (maybe not as that may have generated enough noise to stop /dev/random blocking!) But using dumps from gdb the problem was found in a few minutes of time.
Running the poormans sampler detailed in my previous post. I found that I got a heap of samples looking like this:
Thread 1 (Thread 0xb6f436f0 (LWP 1657)):
#0 0xb7828430 in __kernel_vsyscall ()
#1 0xb7370f93 in read () from /lib/tls/i686/cmov/libc.so.6
#2 0xb731aedb in _IO_file_underflow () from /lib/tls/i686/cmov/libc.so.6
#3 0xb731dcc8 in __underflow () from /lib/tls/i686/cmov/libc.so.6
#4 0xb731a888 in ?? () from /lib/tls/i686/cmov/libc.so.6
#5 0xb731c7b8 in _IO_sgetn () from /lib/tls/i686/cmov/libc.so.6
#6 0xb73103be in fread () from /lib/tls/i686/cmov/libc.so.6
#7 0x0811028f in main (argc=2, argv=0xbffccf54) at src/server.cpp:252
That is I was waiting for a read to complete. Sometimes for 5-10 seconds, sometimes for a few minutes.
What was this troublesome read? An unexpected socket call? Nope - something I would never have expected. Here's the "offending" code.
FILE * f = fopen("/dev/random", "r");
unsigned seed;
fread( &seed, sizeof(seed), 1, f);
fclose(f);
srandom(seed);
This code seeds the random number generators using some values pulled from the random device /dev/random.Reading from /dev/random blocks until there is enough entropy in its internal entropy pool to complete the read. On an isolated VM there's not much system noise generating entropy and so the pool was emptying quickly.
Turns out theres a non-blocking random device that uses feedback of hashed values to prevent blocking when the entropy pool is low, so switching this to using "/dev/urandom" and all was OK.
Now maybe I could have found this by doing some traditional debugging, (maybe not as that may have generated enough noise to stop /dev/random blocking!) But using dumps from gdb the problem was found in a few minutes of time.
Sunday, February 6, 2011
Poor mans sampling profiler for live processes using GDB
While working on OSX I've got used to having the shark system profiler at my fingertips. I love being able to see what's going on in a live process, where all the threads are stuck. and what's taking up all the time on my system.
On linux you can use the oprofile kernel module, or the commercial zoom profiler (which uses a modifier oprofile under the hood I believe)
However if these aren't available to you then you can attach to your process using gdb and manually CTRL-C and backtrace / continue to get a feel for what's going on. This is suggested in several posts on stackoverflow( here and here )
A neater way to do this without ever pausing the application is..
gdb -batch -x stackdumper.gdb ./a.out 123456 > stack.0
where ./a.out is the binary you are interested and 123456 is the PID.
If you set stackdumper.gdb to contain
thread apply all backtrace
Then you'll get a backtrace on all threads. The advantage of this over the manual method is that the binary is stopped for as little time as possible.
I used this to find that all our threads were waiting on some JSON writing code that should have been fast.
i.e. a sample of about 10 runs of the sampler showed one thread deep in json decoding and 2-7 other threads all waiting in pthread mutex / condition code.
On linux you can use the oprofile kernel module, or the commercial zoom profiler (which uses a modifier oprofile under the hood I believe)
However if these aren't available to you then you can attach to your process using gdb and manually CTRL-C and backtrace / continue to get a feel for what's going on. This is suggested in several posts on stackoverflow( here and here )
A neater way to do this without ever pausing the application is..
gdb -batch -x stackdumper.gdb ./a.out 123456 > stack.0
where ./a.out is the binary you are interested and 123456 is the PID.
If you set stackdumper.gdb to contain
thread apply all backtrace
Then you'll get a backtrace on all threads. The advantage of this over the manual method is that the binary is stopped for as little time as possible.
I used this to find that all our threads were waiting on some JSON writing code that should have been fast.
i.e. a sample of about 10 runs of the sampler showed one thread deep in json decoding and 2-7 other threads all waiting in pthread mutex / condition code.
Thursday, January 27, 2011
Lean Development : Focus on Learning
So continuing on with my notes on Lean Development, we're going to look at point 3 from Tom Poppendiecks business card.
3. Focus on Learning - Scientific Method; Challenge Standards; Feedback; Continuous Improvement
Lets start from the back of the list and work to the front, as I find that to be the logical progression of the sub-points.
Continuous Improvement is about continually tweaking your production pipeline and/or the product so that you become more efficient in producing what the customer wants.
Feedback is how you evaluate the changes you make to your pipeline. If your metrics improve, then accept the change, if not work out why it didn't and undo the change.
Challenge Standards - This is a cultural attitude of not accepting the status quo. Its the need for the people involved to have the desire and ability to change the system. If something is a standard then there needs to be a reason for why. If not its arbitrary and subject to getting changed (if the change can be demonstrated to be beneficial)
Scientific Method - This is how we go about getting change implemented. Firstly you hypothesise measurable results from the change will do, then you make the change. If what you hypothesised didn't happen then you need to undo it and reevaluate. "Switching to a frooble compiler should reduce test execution times by 50%, increasing throughput by 3 story-points per week". Results and experiments should be documented somewhere so that the company can learn from what was done (positive or negative).
If we do these things we will understand how our process works and will have a system so that the process can change to meet the changing environment in which the development pipeline exists.
3. Focus on Learning - Scientific Method; Challenge Standards; Feedback; Continuous Improvement
Lets start from the back of the list and work to the front, as I find that to be the logical progression of the sub-points.
Continuous Improvement is about continually tweaking your production pipeline and/or the product so that you become more efficient in producing what the customer wants.
Feedback is how you evaluate the changes you make to your pipeline. If your metrics improve, then accept the change, if not work out why it didn't and undo the change.
Challenge Standards - This is a cultural attitude of not accepting the status quo. Its the need for the people involved to have the desire and ability to change the system. If something is a standard then there needs to be a reason for why. If not its arbitrary and subject to getting changed (if the change can be demonstrated to be beneficial)
Scientific Method - This is how we go about getting change implemented. Firstly you hypothesise measurable results from the change will do, then you make the change. If what you hypothesised didn't happen then you need to undo it and reevaluate. "Switching to a frooble compiler should reduce test execution times by 50%, increasing throughput by 3 story-points per week". Results and experiments should be documented somewhere so that the company can learn from what was done (positive or negative).
If we do these things we will understand how our process works and will have a system so that the process can change to meet the changing environment in which the development pipeline exists.
Monday, January 24, 2011
Lean Development : Build Quality In
This is the fourth post in my mini-series about Lean Development.
The bullet point I'm going to be writing about today is:
Continuous Integration means that your code should be being built and tested on every check in. Broken builds should be addressed immediately. Implicit in this is some kind of version control system. Often this is a larger batch of tests than the specific unit / functional tests used while designing a single feature. Ideally this will run on multiple test systems, for each system you deploy to.
The bullet point I'm going to be writing about today is:
- Mistake-Proof with TDD; Write No New Legacy; Continuous Integration
TDD is "Test Driven Design". This means not only testing all your code, but writing your tests before you code. And letting your design be driven by the issues that arise while making these tests pass. This also holds for bug fixing -- create a test case that reproduces and narrows down the bug, then make that test pass.
"Write No New Legacy" means don't write code that is hard to maintain. This means modular and easily testable code. Make sure the hard bits of the code are documented.
Continuous Integration means that your code should be being built and tested on every check in. Broken builds should be addressed immediately. Implicit in this is some kind of version control system. Often this is a larger batch of tests than the specific unit / functional tests used while designing a single feature. Ideally this will run on multiple test systems, for each system you deploy to.
One idea behind these points is to make it hard for an unnoticed error to reach production. TDD makes it difficult to make the error in the first place. "Write No New Legacy" means you shouldn't have tricky untestable code paths to trip you up. "Continuous Integration" means you should never have a "broken system" on your hands.
The bigger idea behind these points is continuous improvement of the code-base. If all your tests from TDD get plugged into the continuous integration tests, then you should never have a bug reappear.This should give you confidence that your code is doing what it is supposed to do.
The final point is that these processes make it fast to track down a bug, less time hunting bugs means more time delivering real product.
Why use saru for testing?
I've had a few questions about why I use my own little testing framework, saru, for testing, rather than using something standard like cxxunit or whatever.
One of the key problems is testing across multiple languages. We use C++, PHP and python for various pieces of our pipeline. Saru was designed so that plugging in a new language is easy. So if someday we need to support Java then I'm not worried. Now we could use cxxunit for C++, phpunit for php, etc, but I like to have unified reporting and a bit more integration of these tests.
Another problem I have with using a pure C++ framework is sometimes I want to test the condition "Class Foo should not be default constructible." The easiest way for me to check that is to have a fragment of code that should fail to compile with given error messages. You can't test that with a C++ testing framework. (In this particular case you may be able to do something with SFINAE style template hackery, or fork and call a compiler, but these just feel very hackish)
My third reason for using saru is that sometimes I want to do things that are not easily expressed in C++, but easily expressed in other languages. Things like "are the source files correctly formatted?", or "do all C++ exceptions have a corresponding PHP handler?", or "are there tests for each of the classes in this directory?". These are all C++ related questions, but much easier expressed in bash using awk/sed/grep or python than in pure C++.
So thats why I use saru.
One of the key problems is testing across multiple languages. We use C++, PHP and python for various pieces of our pipeline. Saru was designed so that plugging in a new language is easy. So if someday we need to support Java then I'm not worried. Now we could use cxxunit for C++, phpunit for php, etc, but I like to have unified reporting and a bit more integration of these tests.
Another problem I have with using a pure C++ framework is sometimes I want to test the condition "Class Foo should not be default constructible." The easiest way for me to check that is to have a fragment of code that should fail to compile with given error messages. You can't test that with a C++ testing framework. (In this particular case you may be able to do something with SFINAE style template hackery, or fork and call a compiler, but these just feel very hackish)
My third reason for using saru is that sometimes I want to do things that are not easily expressed in C++, but easily expressed in other languages. Things like "are the source files correctly formatted?", or "do all C++ exceptions have a corresponding PHP handler?", or "are there tests for each of the classes in this directory?". These are all C++ related questions, but much easier expressed in bash using awk/sed/grep or python than in pure C++.
So thats why I use saru.
Tuesday, January 18, 2011
Lean Development : Eliminate Waste II
This is going to be another short post. Luckily we only have three points to cover...
The remaining 3 points from my earlier post on the first principle of lean software development are:
Over production in a production environment, means producing more stock than the consumer will consume or at a rate that leads to a build up of inventory. In a software situation I consider the features to be analogous to the product. If you are producing features that are not needed by the customer then you are over producing. In a software sense this is very similar to the over-processing.
Defects and Rework, in my experience, are where a large chunk of waste in a traditional software development lie. I seem to spend a lot of time fixing bugs. So what do we do about it? Test Driven Design, Continuous Integration and "Stop the Line" are some of the tools advocated by Lean software. I guess I'll have to write about some of them in another post.
As I said this was going to be a short one. I'll move on to the next of the principles tomorrow...
The remaining 3 points from my earlier post on the first principle of lean software development are:
- Over Processing
- Over Production
- Defects and Rework
Over production in a production environment, means producing more stock than the consumer will consume or at a rate that leads to a build up of inventory. In a software situation I consider the features to be analogous to the product. If you are producing features that are not needed by the customer then you are over producing. In a software sense this is very similar to the over-processing.
Defects and Rework, in my experience, are where a large chunk of waste in a traditional software development lie. I seem to spend a lot of time fixing bugs. So what do we do about it? Test Driven Design, Continuous Integration and "Stop the Line" are some of the tools advocated by Lean software. I guess I'll have to write about some of them in another post.
As I said this was going to be a short one. I'll move on to the next of the principles tomorrow...
Subscribe to:
Posts (Atom)