Data collection. 1 of n

Today I took my robot to my test area. I collected two battery packs worth of data. I am now begging the long process of analyzing 15 megabytes of binary log files. Oh Joy! No sarcasm here. I love data.

So the first thing I did was to take one run of 2400 samples and plot the speed provided by the GPS and that I compute from the encoder. Here I was a bit worried that I got all the math right with gear ratios and wheel diameters. But as you can see, I was pretty much spot on.

One thing to notice is that the encoder speed is actually much more accurate at slow speeds and particularly at a dead stop. The GPS is never content to say it is not moving but the encoder is.


More data later!

Posted in AVC | Leave a comment

Long time no post

It has been a while since I last posted. A lot and a little has happened in the meantime. I went on vacation to London and had a great time, but I have not made a huge amount of progress on my AVC entry.

What I have gotten is a PID loop module that I will use for controlling speed and heading. I am trying to tune the speed parameters and found that trial and error is not the way to go. I really needed a way to collect some sudden changes and record the output. That allows you to use some math to get close.

I was not about to collect this data under computer control. The safest way would be under RC control. But to do this, I needed a way to record the PWM output to the ESC from the micro. Since my ARM processor has a Timer/Counter peripheral, I wrote some code to measure the pulse width. Amazingly, this seemed to work right out of the gate other than the counts were 1/2 of what I had calculated them to be. I almost let it go, but something told me to dig deeper.

Well I found the issue in the LowLevel initialization routine where the PLL and the main clocks get setup. I had divided the PLL by 4 and not the desired 2. I have been running my processor at 1/2 speed all this time. I changed the dividor expecting nothing to work properly, but was pleasently surprised to see everything working. This means I had properly normalized everything by the clock rate.

I hope to get out this weekend and collect the data I need to tune the speed PID loop. If I get this far, I will then try to stabilize the heading loop.

Speaking of heading, I have too many sources of heading. I have headings from the GPS, the magnetometer, the IMU, and by solar compass. I think I have given up trying to figure out Kalman filters so I am going to implement a Complementory filter. Should be simpler to implement. Just need to figure out a measure of accuracy for each source. I think GPS is best when moving at high speed and the EHE is low. Magnetometer is best when the car is not moving at all. The IMU is ok for short periods of time, and the solar compass is best if it is locked on the sun. Should be doable.

Posted in ARM, AVC, Uncategorized | Leave a comment

Major milestones fall!

Today I was able to check out a few major milestones today. The first is that I determined that I can indeed control the steering servo and ESC via the ARM processor. I was a bit worried that they would not take a 3V3 signal over the normal 5V0. But I can put that worry behind me.

The second milestone was my first test of controlling the steering as a function of cross track error. Although I initially got the sign backwards, as I moved the robot from one side of the course to the other, the steering wheels turned appropriately. Admittedly this was a horribly simple test, but it dis verify the basics of a whole bunch of math.

Next major milestone is to get something to provide a heading (gyro, GPS, magnetometer, …) to start working on the steering and waypoint navigation logic.

Posted in AVC | Leave a comment

LSM303 bug fixed!

After several very frustrating days, I have found the root cause of my bad readings. It was not in the sensor or in its setup. It was in fact the the TWI/I2C driver code that I took from the ATMEL SAM7 libraries. So far every bug that has taken me more than 2 days to find has been in the ATMEL SAM7 libraries. They are riddled with bugs and simplifications that are not mentioned.

The issue here was that the TWI/I2C peripheral is rather different than something like a UART or SPI bus. You start the thing and then you need to hang on for the ride as the peripheral clocks in data. As a byte arrives, you need to be right there to pull it out of the receive buffer. When the last but arrives, you then need to set the STOP condition bit to have the peripheral stop clocking in/out data.

Most of the time this works just fine. You activate the peripheral and sit in a tight loop watching the status register for new bytes. But what happens when an interrupt occurs in the middle of the polling loop? Perhaps an interrupt that takes more 23 microseconds (1 byte at 400kHz) to execute? Perhaps the RTOS tick timer that might need to swap in and out a task and might take several hundred micro seconds? Well the TWI peripheral continues to march along clocking in bytes that get overrun because you are not there to pull them out of the receive register. And because the LSM303 has an auto incrementing address scheme it wraps around back to the start of the data registers this is why I would seem to get all the bytes, but just in funky orders.

It certainly would have been nice had ATMEL included the interrupt disabling code, or at least MENTIONED that any interrupts could screw you over royally. From now on no ATMEL supplied SAM7 code is going to be used without a major scrubbing.

So the solution was simply to disable interrupts for the short amount time needed to clock in the 6 bytes of data (135 micro seconds). A better approach should I fell like being an overachiever would be to make the whole thing interrupt driven with a higher priority interrupt that the RTOS tick timer.

I ran the system for four hours today without a single fault. Before the fix, the problem would phase in and out (as the timer tick overlapped the TWI transfer) and would happen at least every 20 minutes.

“Thank you ATMEL”

Posted in ARM, AVC, Pololu 9DOF IMU | Leave a comment

Woes with a LSM303DLHC

I have a Pololu 9 DOF IMU (version 2) that uses an LSM303DLLC for the magnetometer and accelerometer. With these two sensors you can create a tilt compensated compass. It was my plan to do exactly that to get heading for my robot as I do not fully trust the GPS heading especially moving at slow speeds. But discussion on the DiyRovers forum may make me change my mind.

In any case, it did not take much effort to rehost the sample code from Pololu for my ARM processor. The biggest issue was writing the low level TWI/I2C device driver. Somehow I got suckered into trusting the sample code from Atmel. Even the latest library for the SAM7 was riddled with errors and simplifications. When I re wrote the driver, I was talking to the LSM303 in short order.

Since the IMU also has a L3GD20 3 axis gyro, I quickly wrote the interface for that and added. I then added to my navigation thread the code to sample the sensors and combine the accelerometer and magnetometer data into a heading. On the surface this seemed to work fairly well in a benign magnetic environment.

But in the presence of ferrous materials, the accuracy quickly goes out the window unless you perform some calibration. So I sent a block of time writing the calibration routine to find the various max and min values and rescale the data accordingly. It is not as good a fitting a 3D spheroid to the point cloud, but the math is radically simpler.

It was during the testing of the calibration, that I discovered that on occasion the values from the magnetometer would be completely bogus. Looking at the data more carefully it was obvious that bytes were being lost, corrupted, or swapped. I have now spent way too much time trying to track this problem down.

I have a suspesion of what is going on, but I refuse to make the software change till I get to root cause. Since this error only occurs a few times every 10 thousand samples, I could not see it for a while and think I had solved the problem.

Time passes….

Ok, Now I am no longer sure. I figured that I was reading the data registers at the same time they were being updated. The magnetometer is set for 15 Hz update rate and my nav update thread runs at 5Hz. Over time they were bound to phase in and cause problems.  But watching the I2C data relative to the “data ready” and error condition, I find no correlation.

Next I commented out the reading of the accelerometer  and Gyro and the problem goes away. But now I put them back exactly as they were and the problem still is not occurring. Great Snakes!


Posted in ARM, AVC, Pololu 9DOF IMU | Leave a comment

Been to busy to post

Between the work on my robot, my day job, and various projects at home I have not had the chance to post. And yet a lot of things have happened. I will make some attempt to catch up.

Posted in Uncategorized | Leave a comment

Agoraphobic robot

I have been frustrated by a problem with my robot. Each time I would go outside to test, the robot’s processor would hang. I tried various tactics in getting to the bottom of the problem. The real bummer was that my log files would always be empty files. I had nothing to go on.

Well over the last few days, I realized the empty log files were as a result of not commiting or syncing the data to the FAT. I was just writting and writting and only at the end of a run would I press the magic button to close the file. So I added a sync command every second to at least capture up to the last second before the crash.

What I found in my new log files was troubling. It was logging thousands of invalid commands to my debug serial port. I had a phantom hacker!

Well when I go outside to test, I disconnect the FTDI USB to logic level serial cable. When I did that, I left the processor’s RX line disconnected. Any noise on that line would look like a byte. That byte would then get logged to the SD card. But the SD SPI lines ran close to the RX line. In short order I would get a firestorm of bad bytes that would overwhelm the processor.

Solution: A single 10K pullup resistor on the RX line keeps out the noise when the cable is disconnected.

Posted in ARM, AVC | Leave a comment

Slight pause in the development

The problem with being an embedded software engineer by day and doing prety much the same think at home is that you can get burned out. Not normaly an issue when work is slow or boring. But when you work 10-12 hours a day on complex technical issues, its hard to put in a few more hours when you ge home. Somedays, all I want is dinner, and to watch something mindless on TV (there is no shortage of mindles crap of TV).

So that is where I am now. Been waiting for months to get hardware. It was delayed for many reasons. Now I get it, they want me to turn it around overnight. Here is a multimillion dollar hardware design effort that on Thursday was a box of parts. Friday lunch was a built unit, and could we please have it working by Sunday evening so that it could be taken to a trade show? Really? Hardware gets 6 months and software gets a short weekend? But for those in the trade, this is not uncommon.

There are also other interesting issue working for a large company like I do. The best approximation of the AVC competition area is the overflow parking lot at work. Same size and shape with a good view of the south sky. It also has a survey benchmark for great GPS comparisons. During the week, the only cars there are hot rods people don’t want scratched. On weekends it is deserted. So I made the request to use the parking lot for testing. I hate to think how much that simple request has cost the company. Security, Facilities, Ethics, Legal, Environmental Health, Building Mayor, and every project manager in the building had to bless the request. Took nearly a month and a signed disclaimer form to get approval. But on the other hand, I should be proud in that they just did not say no out of ease.

One of the first things I want to do is to collect GPS data for extended periods at the benchmark. That should start to tell me if the GPS is at all useful. Next I will collect GPS data of the car in motion but tightly constrained to a straight line. I plan on stretching a rope on the ground that will pass through loops on the car.

Posted in AVC, Uncategorized | Leave a comment

Writing to FLASH

You would think that writing to FLASH would not be such a big thing. But it took several days to accomplish. The first issue is that your reprogram code can’t live in the same FLASH as you want to program. With only one bank of FLASH that meant running the code from RAM. That took a fair amount of learning including a jaunt into assembler and GLU linker file syntax. Next was writing the device driver to use the SAM7’s EFC peripheral. This was quite easy since a good shell exists in the ATMEL supplied library. The real problem came from trying to understand the sequence of events to write to FLASH. Get this, you need to write the data to the desired addresses inside a page AND THEN request a reprograming. Seems backwards to me and from previous experience. The other issue was a bug I found when I turned off interrupts as needed during the reprograming. When I turned them back on, any interrupts that occured during the outage where not queued. I therefore lost my timer tick interrupt and the OS stoped running. Turns out You need to use level triggering on interrupts and not edge. Strange.

Posted in ARM, AVC | Leave a comment

AVC update

I have been working pretty hard on my AVC entry. In recent days I have a first draft of the navigation code. I have been slowed down by my inability to write configuration data to FLASH. The fisrt step in that is to be able to run code from RAM during the programming. This was more difficult that I imagined taking several evenings to get working. I should be able to get FLASH writing down tonight.

With that out of the way, I can read and write configuration parameters (waypoints, speed ration, PID coefficients, …) via the serial port user interface. Then I can take the board outside.

Meanwhile I ran a 24 hour test with my uBlox running inside logging all the binary UBX data to an SD Card at a 5Hz update rate. Not a byte lost and the results were amazing. Inside my house with only limited view out a single window, the GPS tracked 6 plus SVs and gave be horizontal errors in the 20 meter range with occational 60 meter errors. Pretty dang amazing. Can’t wait to take to my test site.

I can smell a first test run in the air. Almost like spring!

Speaking of test runs, I have been thinking about how to measure truth on those first runs. I will not be allowed to video tape or take pictures at the test site. I also can’t make any permanent changes to the parking lot nor use any toxic chemicals. I was thinking of pulling a small trailer with a pressurized water bottle. I could then leave a water trail behind. Might just work.

Posted in Uncategorized | Leave a comment