How Can I Partition a Large Dataset?

 

Overview:

In some situations data sets can be defined as too large to upload. This is indicated if the application or site that the data is being uploaded to churns on loading for a very long time. This can lead to failures and issues processing the data. 

This article covers: 

• Creating a Copy of the Raw Data.
• Creating the Folder Structure.
• Copying the Data.

Requirements:

• Data from a supported monitor. 
• Knowledge of the folder structure for monitor data.

Process:

Below you can find the process in 3 parts: Creating a Copy of the Raw Data, Creating the Folder Structure, and Copying the Data.

Creating a Copy of the Raw Data:

Note: For best speed copy the data set to your computer's desktop or a folder location. It is not recommended to make edits to the folders in the original data set from the monitor.

1. Select the folder on the data card.
2. Right Click on the folder that contains the data.
3. Select Copy from the menu.
   
4. Navigate to the destination folder.
5. Right Click, select Paste.
   

6. Open folder containing the data.

 

Creating the folder structure:

1. Navigate to the folder containing the data.
2. Open another File Explorer  for the same folder.
3. Place the windows side-by-side to make it easy to see both windows.
   Note: Window 1 will be the copy of raw data, Window 2 will be the newly partitioned folders.
4. Navigate to the location of the folder that was downloaded from the monitor in window 1. 
5. In window 2 open the same location and create a new folder with a name like 'Part 1'.
   
6. Open the new folder.
7. In the folder 'Part 1', recreate your folder structure using the copy of the data you made. 
   • For John Deere GS4 data with the JD-Data folder the structure will look like this: Part 1\JD-Data\log\2020_Farm_Field_01\__.JDL
   • For Precision 2020 and Exports from FieldView the folder structure will look like this: Part 1\Farm_Field\xx3505\__.dat
8. Prepare your files for upload:
   • Folders for Sync Pro should not exceed 150MB and do not need to be compressed.
   • Folders for Web Upload should not exceed 2GB and do need to be compressed.

Note: This process will be repeated until you have all parts copied to the new folders that are small enough to be uploaded. The original duplicate can be copied and renamed as 'Part 2' and reused for the next copy of the data.

Copying the Data:

1. In Window 1, the original copy of Raw Data.
2. Navigate to the 'log' folder.
3. Select some of the folders (5 to 10 is a good starting point). Use a Shift+Click to select multiple,
   
   
4. Right Click, select Copy.
5. In window 2, right click and select Paste.

Note: Repeat the copy process until all files have been copied into the Part folders.