speed & reliability

[sltfn]

Hello,

I have a question similar to simon.schvartzmann's in the thread "Livecloud x Dropbox performance."

The app I hope to get working with acceptable performance & reliability will replace an app that reads from and writes to a server-based MySQL database. I've not generated comparative numbers but am finding queries and updates to my LiveCloud table to be significantly slower than with MySQL. I'm trying to decide if the slower performance will be within the range of acceptability or not.

In addition to the speed issue, I'm finding that updates to records sometimes fail. In my most recent test, updating c730 records in rapid succession, looping through a tab-delimited list in LiveCode, updates for 5 of the records failed, so a failure rate of about 0.7%.

Of course, the only acceptable failure rate overall is 0%. I've built in an extra loop to take the list of failed updates and try updating again. In this test the 5 updates that failed initially all went through on second try but I'll need to incorporate a system that repeats until there are no failures and all records get updated.

Back to the speed issue. The pattern seems to be that the update process will run relatively quickly for some number of records and then there is a "hang," in which I'll receive messages for several seconds to the effect the server response cannot be downloaded at this time.

Is there anything I can do in my LiveCode scripts—perhaps wait for a fraction of a second between updates?—that might improve reliability? Is LiveCloud being overwhelmed by updates in rapid succession? If a wait command might help, what length of time? Each fraction of a second is, after all, a fraction of a second one would rather not add to processing time.

For my app to work acceptably well, I'll need to be able to update as many as 3000–4000 records in sequence without having to hold things up until the whole batch, including retries on failures, is complete.

What should I try? Thanks in advance.

[simon.schvartzman]

@sltfn, good question. I've also seen reliability issues but wasn't able to identify them as well as you have.
I really thinks LiveCloud has a great potential but I've faced many problems since I started testing it and what worries me more is that the team takes a long time to answer or they don't answer at all...

Will see what the answer to your question is.

Regards

[sltfn]

@sltfn, good question.  I've also seen reliability issues but wasn't able to identify them as well as you have.
I really thinks LiveCloud has a great potential but I've faced many problems since I started testing it and what worries me more is that the team takes a long time to answer or they don't answer at all...

Will see what the answer to your question is.

Regards

Agree @simon.schvartzmann. Great potential. Really hope it proves reliable or I'll need to seek a different solution. Crossing fingers.

Regards

[efrain.c]

Hi, sltfn

If possible, can I see the code for your update routine? Are you using cdb_update or cdb_batchUpdate? I recommend using the batch APIs whenever possible instead of using a loop for each record.

I created a test stack with a create button and an update button. The create button created 10,000 records using cdb_batchCreate and the update button updated those 10,000 records using cdb_batchUpdate. There were no failures for either. The create took 6.118 seconds and the update took 8.978 seconds. I've included my code for reference. Let me know if you have any questions.


This is the create button script:

on mouseUp
local tInputA, tTableID, tCounter, tOutputA, tRepetitions
local tStart, tEnd

put cdb_tableID("testTable") into tTableID
put 10000 into tRepetitions
put 0 into tCounter
repeat tRepetitions
add 1 to tCounter
put "test" & tCounter into tInputA[tTableID][tCounter]["firstName"]
put "name" & tCounter into tInputA[tTableID][tCounter]["lastName"]
put "test" & tCounter & "@mail.com" into tInputA[tTableID][tCounter]["email"]
put random(100) into tInputA[tTableID][tCounter]["age"]
put item random(2) of "M,F" into tInputA[tTableID][tCounter]["sex"]
end repeat

put the milliseconds into tStart
put cdb_batchCreate(tInputA, "cloud") into tOutputA
put the milliseconds into tEnd

put tEnd - tStart && "milliseconds"

put tOutputA[tTableID] into tOutputA
if the number of lines of the keys of tOutputA is not tRepetitions then
answer "There were" && tRepetitions - the number of lines of the keys of tOutputA && "failures."
else
answer "There were no failures."
end if
end mouseUp



This is the update button script:

on mouseUp
local tInputA, tTableID, tOutputA, tResultsA, tFailures
local tStart, tEnd

put cdb_tableID("testTable") into tTableID
put cdb_read(tTableID, "*", "cloud") into tOutputA

repeat for each key xRecordID in tOutputA
put tOutputA[xRecordID]["age"] + 1 into tInputA[tTableID][xRecordID]["age"]
end repeat

put the milliseconds into tStart
cdb_batchUpdate tInputA, "cloud"
put the milliseconds into tEnd

put cdb_read(tTableID, "*", "cloud") into tResultsA

put 0 into tFailures
repeat for each key xRecordID in tOutputA
if tResultsA[xRecordID]["age"] is not tOutputA[xRecordID]["age"] + 1 then add 1 to tFailures
end repeat

answer "There were" && tFailures && "failures."
put tEnd - tStart && "milliseconds"
end mouseUp

[clarencemartin]

efrain,
although your posting is directed towards sltfn, I appreciate seeing code samples that may be useful to others.
Thanks for the code sample.
I appreciate it.
I would like to see more code samples. Can we have a section in the forum for Scripting Samples?

[sltfn]

Yes, thank you, efrain. I am not using the batch APIs but will look at them more closely and hopefully figure out how to use them. These code samples will be helpful.

[mark_talluto]

The value of the batch routines is that you can communicate with multiple records on multiple tables in the same call. Using batch can turn 10k record transactions into a single call.

It may be of interest to know that the basic methods feed internally to the batch routines. Looping through each request should not be any less reliable than batching your calls into a single transaction. Looping your calls will be slower.

That said, you should see 100% accuracy using either method. Did you get an error message in your message box indicating there was a problem with the five records that failed? It would be helpful if you would elaborate on your testing so we can locate possible bugs.

Calls that fail to reach the cloud are automatically cached. The next attempt to make a cloud call will flush the cache for you. If you have time, please send us a test stack so we can reproduce the issue. Thanks.

[mark_talluto]

I added a new forum for everyone to paste sample code.

[mark_talluto]

Hello,

I have a question similar to simon.schvartzmann's in the thread "Livecloud x Dropbox performance."

The app I hope to get working with acceptable performance & reliability will replace an app that reads from and writes to a server-based MySQL database. I've not generated comparative numbers but am finding queries and updates to my LiveCloud table to be significantly slower than with MySQL. I'm trying to decide if the slower performance will be within the range of acceptability or not.

In addition to the speed issue, I'm finding that updates to records sometimes fail. In my most recent test, updating c730 records in rapid succession, looping through a tab-delimited list in LiveCode, updates for 5 of the records failed, so a failure rate of about 0.7%.

Of course, the only acceptable failure rate overall is 0%. I've built in an extra loop to take the list of failed updates and try updating again. In this test the 5 updates that failed initially all went through on second try but I'll need to incorporate a system that repeats until there are no failures and all records get updated.

Back to the speed issue. The pattern seems to be that the update process will run relatively quickly for some number of records and then there is a "hang," in which I'll receive messages for several seconds to the effect the server response cannot be downloaded at this time.

Is there anything I can do in my LiveCode scripts—perhaps wait for a fraction of a second between updates?—that might improve reliability? Is LiveCloud being overwhelmed by updates in rapid succession? If a wait command might help, what length of time? Each fraction of a second is, after all, a fraction of a second one would rather not add to processing time.

For my app to work acceptably well, I'll need to be able to update as many as 3000–4000 records in sequence without having to hold things up until the whole batch, including retries on failures, is complete.

What should I try? Thanks in advance.

Hi Simon,

Our first implementation of Blobs is not fully optimized. It is definitely a version 1. We have plans to work on the performance now that we worked on the features we wanted. I am working on an update for NurseNotes that is going to rely on our blob features. We are going to need them to be as fast as possible. I know where the bottleneck is. Hang in there. 

We manage all the waiting for you in the libraries. You are free to fire away on your transactions. I would like to see the portion of your code that writes your records to the cloud and any other related code you do after that. It might be us some insight into what is going on. Thx.

[simon.schvartzman]

Mark, good to know your are working to optimize the blob performance.

Regarding "...I would like to see the portion of your code ..." I have to say that my code has been siting (attached to my original post) on the forum waiting for a response for more than 6 days...

https://forums.livecloud.io/showthread.p...ht=dropbox

Best...