Saving data to SD memory card

[jlucas]

I’m using a PIC 18F4520 in a GPS vehicle tracking application. I’m collecting data via radio from moving vehicles.

I want to use a micro SD card for event logging. I know there’s a TON of information on this site about how to do this. That’s my problem – information overload. Perhaps if I can get answers to a few questions, I can get unstuck.

The first regards the hardware interface. I don’t need to read anything back to the PIC from the SD card. Can I get by with 3 wires to the SD card: data in, +3.3 V, and ground? This would save having to add a 3.3 to 5 V level translator to up-convert the voltage for data back to the PIC.

Moving on to the firmware, first I would like to create a folder named with the MAC address of the RF module. Then I would like to save the data from each event to a separate file named with the time and date of the event. I can use csv, tab-delimited, or fixed-field-length format, as long as I can pull the SD card, stick it into a PC, and open the file in Excel.

I have enough memory in the PIC to buffer the data for an entire event, then create and write the file to the SD card after the event is over. Can someone point me to some information on how to 1) create a folder, 2) create a file, and 3) write data to that file? The data will contain both text and numbers as well as the field separator and a Return at the end of each 1-sec record.

Some other questions:

Can I use SD cards preformatted with FAT32?

Is there any limit to the size of the SD card? I'm thinking of using 2-8 Gb.

Can I create and name a folder on the SD card to hold all the files?

Can I use long file names?

Can the SD cards be hot-pluggable/unplugable?

I don’t expect you guys to write my code for me, but if anyone can point me to a source of help for this, I would be most appreciative.

Thanks much,

Jim
_________________
Always remember, things are never so bad that they can't get worse.

[temtronic]

Short answer.....use another chip for data logging.

Longer answer.. You'll spend a LOT of time researching/looking for code, cutting code, R&Ding a workable solution using mismatched devices(5V vs 3V). Yes, most , if not all of the 'code' has been cut but you'll have to work out the nasty details and time is money, don't kid yourself.

Longest answer...
There is an EASY alternative to SD cards for data logging. Use a FTDI Vinculum module. For $30 out the door you get a 5v PIC compatible USB flash drive data logging module.EASY serial command/data transfers.You can be 'up and running' in less than a 1/2 a day. Been there doing that.

Once you figure out the record format for storing data, it's as easy as one print command.Data gets stored as CSV format making it a piece of cake to import into a spreadsheet.Currently I have one recording data at 115200 baud, zero problems,about 200 bytes of data per record,every second,24/7.

Ok, so it's a 'big' chunk of money up front for the module but you get instant compatabilty with 5V PICs, simpler serial data transfers,huge storage,and all the messy FAT,files,dir,etc is done for you so you don't need to have 'SD drivers' in the PIC, saving space,more room for 'features' .Consider what your R&D cost is in 'engineering labour'. For less than 1 hour( @$50 /hr) you can have a viable, working solution up and running.Work smarter..not harder.

[temtronic]

Short answer.....use another chip for data logging.

Longer answer.. You'll spend a LOT of time researching/looking for code, cutting code, R&Ding a workable solution using mismatched devices(5V vs 3V). Yes, most , if not all of the 'code' has been cut but you'll have to work out the nasty details and time is money, don't kid yourself.

Longest answer...
There is an EASY alternative to SD cards for data logging. Use a FTDI Vinculum module. For $30 out the door you get a 5v PIC compatible USB flash drive data logging module.EASY serial command/data transfers. You can be 'up and running' in less than a 1/2 a day. Been there doing that.

Once you figure out the record format for storing data, it's as easy as one print command.Data gets stored as CSV format making it a piece of cake to import into a spreadsheet. Currently I have one recording data at 115200 baud, zero problems, about 200 bytes of data per record, every second, 24/7.

Ok, so it's a 'big' chunk of money up front for the module but you get instant compatabilty with 5V PICs, simpler serial data transfers,huge storage,and all the messy FAT, files, dir, etc is done for you so you don't need to have 'SD drivers' in the PIC, saving space,more room for 'features'. Consider what your R&D cost is in 'engineering labour'. For less than 1 hour( @$50 /hr) you can have a viable, working solution up and running. Work smarter..not harder.

[asmboy]

i use this FABULOUS module as a backup data logger in several different instruments i've designed.

works like a charm with up to 4GB micro SD cards i'ved tested - and is very fault tolerant - as it does a FREQUENT buffer commit to any open file you are writing too-power failure has never to my knowledge corrupted a disk either

http://www.sparkfun.com/products/9530

attaches to the TX -// or RX pins of a pic EZ 2
we buy them by the 100's and at $20 bux - is a bargain for what it can do

[ckielstra]

Why is it that so many people in this forum want to interface a 5V PIC to a 3.3V SD card?
For making this work you will need a 3.3V to 5V translation because data can only be written to the card when you check the response answer from the card.
Some work arounds for this problem are possible but none of them brilliant.
My advise, use another PIC that can run at 3.3V directly and you won't have to invest in the extra cost for voltage converters and second power supply. As a bonus you save some power as well. For example the 'LF' version of your current PIC of one of the newer models.

Then about you saying you have enough memory available; you realise that writing data to the SD card happens in blocks of 512 bytes so the FAT library might use more of your 1536 bytes RAM than you anticipated.

About available FAT libraries:
- CCS supplies one for free. It works but some problems have been reported with the format function (it doesn't write a correct partition table or something like that causing Windows not being able to read the files). Once the CCS format is used the effect can only be undone by hard writing data to card address 0-512.
- The code library of this forum has at least two free implementations. Check the threads for comments on quality.
- Andrew, nickname asmallri, is a member of this forum and sells a commercial library with good reviews.

[jlucas]

Thanks, everybody!

temtronic, at $30 per unit, the FTDI Vinculum module is rather expensive compared to the SD card and holder. Since I’m hoping to sell these thing by the 1000s, it's worth spending $1000 on R&D if it saves me $1 per unit. It still might be the best solution, though.

asmboy, your $20 solution looks intriguing. It’s still more than I would like to spend, but it’s definitely on the table at this time.

ckielstra, I guess because they don’t make SD cards that run 5V like everything else on my board. I can get down to 3.3 V with resistors. Up-converting requires another part, but that should be less than $1. If I can get by without additional hardware other than that, the card, and card holder, I'll be money ahead.

I’ll only need one or at the most two 512-byte blocks of data for each file, and once it’s written to the SD card, I don’t need to keep it in RAM. If the data goes over 512 bytes, I guess I can write it in two operations and append the second block to the file.

A 2-Gb SD card should be plenty.

File names longer than 8.3 would be very nice; I can get by with much fewer than 65,536 entries.

By hot plugging and unplugging, I mean while the unit is under power, but not during a write operation.

Again, many, many thanks to all. I’ll get to work studying these new possibilities.
_________________
Always remember, things are never so bad that they can't get worse.

[temtronic]

The Vinculum module I was talking about is a complete unit.VNC1L chip, PCB, USB connector,misc resistors,caps, headers,etc. designed for rapid prototyping/ proof of concept. Chip alone can be bought for about $7.50 /1k.

Your math is interesting. If you only save $1 /unit and sold 1000 units that's $1,000 you saved but it cost you $1,000 in R&D so you broke even,moneywise. Time is a critical factor in getting a product to market.That $1,000 R&D money is only 10 hrs for me(call it 2 days) and frankly there is no way anyone can cut code equal to what the Vinculum can do in that time frame.
We all 'know' that a bigger PIC is better but most try to squeeze 'everything' into the cheapest unit trying to save a few pennies meanwhile R&D time has consumed 100s if not 1,000s of $.Same thing applies to say voltage regulators, while a 78L05 may do the job(on paper) using a 7805 can easily save on repair costs,service calls,etc. as it'll take 'abuse' better.

Just something to consider....

[jlucas]

Thanks again, tt. I have some work ahead of me now.
_________________
Always remember, things are never so bad that they can't get worse.

[ckielstra]

Why spend money to get the SD card to interface with 5V and not run everything else at 5V? I'm wondering which 5V components you have in your design that are not available in a 3.3V version.

For example you could replace the 18F4520 chip by a 18F45K20 which has a lot more features, can run up to 64MHz at 3V (versus 25MHz at 3.3V and 40MHz at 5V for the older 4520). Is also cheaper because produced with newer teechnology ($1.60 v.s. $3.29 at high volumes).

[temtronic]

did a check at digikey and the Vinculum-II version is about $3!

[asmallri]

[jlucas]

[asmboy]

[jlucas]

Sounds like good advice, asmboy. I have a lot of good information to work with now. Thanks!
_________________
Always remember, things are never so bad that they can't get worse.

[n-squared]