Now that you have your RAMADDA server installed you may ask how do I get go about populating it with data and other content. We covered how to make folders and upload files (or archives of files) through the web interface. However, RAMADDA also supports ways of providing access to the files that are already on the server machine. RAMADDA provides a harvester facility that allows an administrator to scan the local file system on the server and add entries into the database for those files. For example, say you have a large directory of data (and/or other) files (e.g., /project/data) that you want RAMADDA to provide access to. You could simply create a Server Side File view of the directory but this does not actually add entries into the repository database for the files and you cannot add metadata, search etc., with those entries.

The File Harvester can walk the directory tree and add entries into the repository for the files it finds.

The first thing you need to do is to specify in the Site Administration area that it is OK to serve up files from that directory tree. Go to the Admin->Settings->Access page in RAMADDA and enter the directory path, e.g., /project/data:

Next, create a File Harvester by going to the Harvesters tab of the Admin pages and press the "New Harvester" button. Specify a name (e.g., "Test"), leave the type as Local Files and hit create:

This will take you to the Harvester Edit form. The minimal configuration you need to do here is to enter the file system directory to scan and the RAMADDA folder to add the harvested entries to. In this example we are harvesting the directory tree /project/data and we are adding the entries into the RAMADDA folder RAMADDA/Case Studies/Test: When you have entered this information press "Change" which will apply the changes and then take you back to the Harvester Edit Form. You can start this Harvester with the "Start" link in the upper right. Or you can go to the main Harvesters page (the top Harvesters tab) and this will list your Harvester. From there you can start the Harvester and monitor and/or stop its progress. When it is finished the Stop link will change to Start.

With the above settings the directory tree of your file system will be used to create the folder hierarchy in RAMADDA. Every file the Harvester finds will result in a file entry. You can run a Harvester any number of times and it will only add the new files that it has not seen before.

The File Harvester has many other settings described in more detail here.

If the server machine you have RAMADDA running on has a large collection of data files then you can very easily provide RAMADDA access to those files with the "Local File View" entry type. This provides a view of the file system. The Folders and Entries it creates are not in the database but they provide a way to access the data through all of the RAMADDA services.

For example, on Unidata's motherlode server we have a large archive (10's of millions of files) of real time data from Unidata's Internet Data Distribution system (IDD). This would be too large to harvest into the database. Instead, on Unidata's RAMADDA server there is a Local File View entry: IDD Data that provides a synthetic view of this large directory tree.

You create one of these entry types through the File->New Entry menu. The create form looks like:

You specify a directory path on the server machine (e.g,. /data/ldm/pub) and other configuration options (e.g., what to show, what not to show, etc.)

Note: for security reasons the local file path that is specified has to be a subdirectory of one of the directory paths specified in the Site Administration page under Settings->Access->File System Access

You end up then with a view of that directory that looks like: