Skye Sweeney's Build Log

After several very frustrating days, I have found the root cause of my bad readings. It was not in the sensor or in its setup. It was in fact the the TWI/I2C driver code that I took from the ATMEL SAM7 libraries. So far every bug that has taken me more than 2 days to find has been in the ATMEL SAM7 libraries. They are riddled with bugs and simplifications that are not mentioned.

The issue here was that the TWI/I2C peripheral is rather different than something like a UART or SPI bus. You start the thing and then you need to hang on for the ride as the peripheral clocks in data. As a byte arrives, you need to be right there to pull it out of the receive buffer. When the last but arrives, you then need to set the STOP condition bit to have the peripheral stop clocking in/out data.

Most of the time this works just fine. You activate the peripheral and sit in a tight loop watching the status register for new bytes. But what happens when an interrupt occurs in the middle of the polling loop? Perhaps an interrupt that takes more 23 microseconds (1 byte at 400kHz) to execute? Perhaps the RTOS tick timer that might need to swap in and out a task and might take several hundred micro seconds? Well the TWI peripheral continues to march along clocking in bytes that get overrun because you are not there to pull them out of the receive register. And because the LSM303 has an auto incrementing address scheme it wraps around back to the start of the data registers this is why I would seem to get all the bytes, but just in funky orders.

It certainly would have been nice had ATMEL included the interrupt disabling code, or at least MENTIONED that any interrupts could screw you over royally. From now on no ATMEL supplied SAM7 code is going to be used without a major scrubbing.

So the solution was simply to disable interrupts for the short amount time needed to clock in the 6 bytes of data (135 micro seconds). A better approach should I fell like being an overachiever would be to make the whole thing interrupt driven with a higher priority interrupt that the RTOS tick timer.

I ran the system for four hours today without a single fault. Before the fix, the problem would phase in and out (as the timer tick overlapped the TWI transfer) and would happen at least every 20 minutes.

“Thank you ATMEL”

I have a Pololu 9 DOF IMU (version 2) that uses an LSM303DLLC for the magnetometer and accelerometer. With these two sensors you can create a tilt compensated compass. It was my plan to do exactly that to get heading for my robot as I do not fully trust the GPS heading especially moving at slow speeds. But discussion on the DiyRovers forum may make me change my mind.

In any case, it did not take much effort to rehost the sample code from Pololu for my ARM processor. The biggest issue was writing the low level TWI/I2C device driver. Somehow I got suckered into trusting the sample code from Atmel. Even the latest library for the SAM7 was riddled with errors and simplifications. When I re wrote the driver, I was talking to the LSM303 in short order.

Since the IMU also has a L3GD20 3 axis gyro, I quickly wrote the interface for that and added. I then added to my navigation thread the code to sample the sensors and combine the accelerometer and magnetometer data into a heading. On the surface this seemed to work fairly well in a benign magnetic environment.

But in the presence of ferrous materials, the accuracy quickly goes out the window unless you perform some calibration. So I sent a block of time writing the calibration routine to find the various max and min values and rescale the data accordingly. It is not as good a fitting a 3D spheroid to the point cloud, but the math is radically simpler.

It was during the testing of the calibration, that I discovered that on occasion the values from the magnetometer would be completely bogus. Looking at the data more carefully it was obvious that bytes were being lost, corrupted, or swapped. I have now spent way too much time trying to track this problem down.

I have a suspesion of what is going on, but I refuse to make the software change till I get to root cause. Since this error only occurs a few times every 10 thousand samples, I could not see it for a while and think I had solved the problem.

Time passes….

Ok, Now I am no longer sure. I figured that I was reading the data registers at the same time they were being updated. The magnetometer is set for 15 Hz update rate and my nav update thread runs at 5Hz. Over time they were bound to phase in and cause problems.  But watching the I2C data relative to the “data ready” and error condition, I find no correlation.

Next I commented out the reading of the accelerometer  and Gyro and the problem goes away. But now I put them back exactly as they were and the problem still is not occurring. Great Snakes!