There are different options to upload data to Big Query depending on our needs.
Uploading data from a couple of months compared to several years requires different approaches. Here below are the various options available for transferring data using Dataslayer.
If you are not familiar yet with our Big Query product see the intro in this FAQ.
BigQuery Menu
For uploading historical data, the manual option is typically recommended. However, depending on the volume of data you're dealing with, manually splitting it might sometimes be a more efficient choice.
Once the query is created, you must pay attention to the date range that you have selected and the amount of data you have for this date range.
Manual option
If you have a small volume of data (one month, or one year with a small amount of information) you can use the manual option for immediate data uploading. By choosing this option you will upload all the data at once.
One-month date range
Manually Split uploading option
The manually split uploading option is ideal for handling a large volume of data, such as a month with extensive data across multiple accounts or data spanning over a year. This feature divides queries into groups and executes them periodically, mitigating timeouts, interruptions, and errors. This way your data is uploaded gradually and without interruption.
Important: Note that this function can ONLY be executed when the query has TEMPORAL DIMENSIONS added.
When this option is used, the query can take a long time, so once executed, even if the browser is closed, the query will continue to run.
Five years - date range
Once you have your historical data, the best practice is to modify the transfer's date range to yesterday and set the daily schedule option.
Warning: Make sure that your dataset has the default expiration date set to never otherwise the tables will be deleted automatically. This is explained here.
Schedule option
Select this option to schedule periodic uploads. Ideal for keeping the data updated periodically.
Schedule option
As always, please contact us via our live chat on our website or via email if you still have doubts or questions. We are happy to help!