Blobs and Datagrids

[sid]

Hi Guys

I have a few questions on BLOBS and datagrids and will appreciate some sample code here.

I have managed to create a form with a datagrid on it , and a image object that retrieves the images from the cloud when a user hilites a line on the grid. This is designed as a cloudcall and not a local call , so its a tad slow.

here is a vid on how this works : https://www.loom.com/share/d609c74551df48309f51723192353039

here is the code:

Code:

local  tBlobGridRef, tPathToBlob, theLine, theSelectedID, tPHtender

getProp uSelectedPicID
 
  put the dgHilitedLines of me into theLine
  return GetDataOfLine(theLine,"blobReference")
 
end uSelectedPicID


on selectionChanged pHilitedIndex, pPrevHilitedIndex
 
  show widget "PHSpinner"
  put empty into tBlobGridRef
  put empty into tPathToBlob
 
  put the uSelectedPicID of group "Tenders" into theSelectedID
  put cdb_readBlob(theSelectedID,"cloud") into tPathToBlob
  set the filename of image "tenderPicViewer" to tPathToBlob
  //set the text of image "tenderPicViewer" to tPathToBlob
  put the text of image "PHTender" into tPHtender
  if tPathToBlob is empty then set the text of image "tenderPicViewer" to tPHtender
  hide widget "PHSpinner"
 
end selectionChanged
 

Here is  question 1. Its a cloud call but the path to the image ends up being a local path? Its pretty slow ,( perhaps because I'm living in a rural farm town) Does the system download the image if its a cloud based call or just look in the local Blobs table if it exists?

so I tried to resize all images to under 10kb and that helped a lot. Much faster, so I tried to put the text of the image into a table key, (not a BLOB table) figuring I can just store the text into the key and then that should speed it up more if its just a 100x80 pixel .png thumbnail. LCM did not seem to allow this? can we actually do that? Or will it totally mess up the performance?

Question 3 , Is it possible to put a thumbnail of a BLOB into a DataGrid? and if so, is it possible to only load the images that will be visible to the user (IE. only when the lines are visible on his screen. Will appreciate some help with this.
I am designing a system with at least 70000 product items and God help a tablet trying to load ...lets say 30000 thumbnails if a query is run. It will probably die...although it will be interesting to test. So the idea is run a LCM Query and put the results into a DataGrid table with a BLOB in one of the columns.

Question 4. Is it possible to have more than 1 Blobs table? because I feel that it important could separate things like documents and High Res images from thumbnails, because you do not want to sync the High Res Blob table with a thing like a POS system or a tablet.

Would really appreciate some sample code for question 3

I have to say that this is the easiest , most productive database design I have ever seen. I'm a total beginner and did not have to learn a single sql query with the code snippet feature . Thanks Livecloud !!!

Keep safe everyone!
Sid

[mark_talluto]

Q1: If you read the blob with a target of 'cloud' then the call will go to the cloud. It stores the data locally for quick access. The time to get the data will be dependent on the size of the blob and the performance of your internet connection. Resizing the images to be smaller will improve performance. You can store binary and text data in your non-blob tables. You have more blob space than database space. Thus, it behooves you to take advantage of the blob space for non-queryable data. The blobs are stored in an S3 bucket. If you store binary data in a standard key, remember to treat the CRUD operations as you would if the data was text. You will not be able to use the BLOB APIs on data stored as a standard record. LCM will not be able to display that data as it does when viewing the blob table.

Q3: You can put images in a datagrid. You will have to download the image and load the local blob into memory. Put the imageData into an image in your stack. I am not sure if a datagrid can reference an image on disk directly. But you can have your images reference image data on disk. The datagrid pages data, much like how LCM pages data. As you scroll through your data in LCM, it is reading the data in real-time from the cloud and displaying it in the table view. Our table is not a datagrid. The DG should be relatively efficient in its display of data.

For your large data sets, you will want to develop a cursor like system where you are tracking your place in the table. You can efficiently load only the portion of data that is to be displayed at one time. When your user page flips or scrolls, you would then do another cloud call on that data and display it. You can forget the previous data by deleting that variable. This method will allow a mobile device to access millions of records. The trick is to not store millions of records in your view control nor in memory at one time. Only show what they can see at any moment in time.

Q4: You can have only one Blob table. This is not a limitation in performance since the data is stored efficiently in an S3 bucket. These buckets scale for you automatically and hold an almost infinite amount of data. You might consider syncing ranges of records instead of the whole table. Your performance will increase dramatically. There are multiple methods for doing this. One method would be to create ranges of records and store them in a record in a table just for that purpose. You can then read on a range of those records. They would hold the line delimited recordIDs of the blobs that you would like to group together. Then do a batch read on those recordIDs and enjoy better performance.

Thoughts on your video: When doing reads, you may consider blocking access to your UI until the read is complete. I can see that you are allowing the user to click a button that will generate a cloud call. LiveCode is not completely blocking the internet transactions. So, you have to prevent a user from clicking a button that will cause another call. Otherwise, your current downloads can be interrupted because a new read has been initiated. You can do this by displaying a control over your UI that will catch all user interaction.

In another post, we discussed our caching system. When we release that, the cloud reads are done using async methods. Your use will be able to request multiple reads at the same time. You have control over sync or async in uploads. You have to use the command cdb_setUploadMethod with a parameter of 'async' or 'sync'. We default to sync. I just noticed that we do not have this API in our docs. I'll work on that in the future.

[sid]

Thanks Mark

This will help. Thats quite a bit to digest

If you have time in the furture, I would appreciate a code sample of how you can view a BLOB on a datagrid. I can't get that to work at all

I'm not in a hurry for this. I have lots to do yet.