kpedersen
Posts: 12
|
| Posted: 06/13/2012, 12:49 PM |
|
Both list views and editable grids are behaving rather badly, when the amount of lines rise.
Example server latency for editable grids
17k data = 151 ms
80 k data = 1230 ms
800 k data = 1,7 MINUTES
It seems the problem is getting exponentially worse.
The regular lists are better, but still slows down exponentially by the number of lines (300 lines = 16 seconds).
Checked possible problem sources
- Disabled antivirus scanner / used proper browser
- Data returned is gzipped and connection is highspeed.
- The underlying query including network latency is for the large one 151ms.
Environment
- Java Servlets
- Tomcat running + MySQL
Please advise
Kasper Pedersen
PS.
I have a single evnt code running Before Show Row
---------------------------------------------
String name = (String) e.getPage().getComponent("formfieldList").getControl("FeltNavn").getValue( );
if( name.trim().equals("") )
e.getPage().getComponent("formfieldList").getControl("FeltNavn").setValue("< empty >");
|
 |
 |
kpedersen
Posts: 12
|
| Posted: 06/13/2012, 3:34 PM |
|
I ran some additional tests on a local machine with no possible bottlenecks in RAM/CPU
items latency latency/item
100 ____ 289 ms ___ 2,89
200 ____ 885 ms ___ 4,425
300 ___ 1998 ms ___ 6,66
400 ___ 3270 ms ___ 8,175
Could it be a problem with the template rendering code ?
/Kasper
|
|
|
|
Oper
Posts: 1195
|
| Posted: 06/13/2012, 4:22 PM |
|
How much data the grid display?
10/15/20 at time?
if that the case then the database/table design/index and QRY construction need to be looked.
* Just make sure you are using LIMITs
* Avoid View as much as possible
* Use Merge Index
no matter how big the data is should be really fast
_________________
____________________________
http://www.7bz.com (Free CMS,CRM Developed in CCS)
http://www.PremiumWebTemplate.com
Affiliation Web Site Templates
Please do backup first |
|
|
|
laneoc
Posts: 154
|
| Posted: 06/13/2012, 7:54 PM |
|
I have similar trouble which is related to my antivirus.
Once I disabled the antivirus -- or used a different browser -- the slowness went away.
_________________
Lane |
|
|
|
kpedersen
Posts: 12
|
| Posted: 06/13/2012, 10:46 PM |
|
Thanks for fast response
I am on purpose not using paging, because i need all the data in a single page.
The above numbers ("items") are the number of records returned. About 100 records is fine, but at 300+ everything slows very fast.
I am fully aware that this amount of records would result in slower pages, but i fail to understand that this progression should scale by N*N. The above example from a regular list shows almost perfect correlation of: Latency (ms) = square( Record count ) * 0.02
Is it possible to disable / manually rewrite the code for this behaviour ? I dont need a fix to code generator, just some hint to what is causing this and then i can try and fix the issue by overriding that part of the code.
/Kasper
Quote Oper:
How much data the grid display?
10/15/20 at time?
if that the case then the database/table design/index and QRY construction need to be looked.
* Just make sure you are using LIMITs
* Avoid View as much as possible
* Use Merge Index
no matter how big the data is should be really fast
|
|
|
|
Lucius
Posts: 220
|
| Posted: 06/14/2012, 3:50 AM |
|
Just out of curiosity... What is the need / business case for needing to display so huge editable grid?
|
|
|
|
E43509
Posts: 283
|
| Posted: 06/14/2012, 4:55 AM |
|
Can you isolate the query times?
How long does the sql take to return all the rows (not just the first set) for the various scenarios? Delays like this are usually due to the query and the amount of data being returned and not necessarily the transport layer.
Good luck.
|
|
|
|
Oper
Posts: 1195
|
| Posted: 06/14/2012, 8:08 AM |
|
oh,
* you have Listbox on that Grid
* every lixbox has Populated Data
* that make your page bigger not slow.
so maybe the size of the page is similar to downloading 1 or 2 MB file each display
do you have a workable link i could check?
_________________
____________________________
http://www.7bz.com (Free CMS,CRM Developed in CCS)
http://www.PremiumWebTemplate.com
Affiliation Web Site Templates
Please do backup first |
|
|
|
kpedersen
Posts: 12
|
| Posted: 06/14/2012, 5:16 PM |
|
It's actually mainly the normal list thats the problem (editable grid is just worse).
The business need for long lists is printing: Paged setup would not work, as this would require a lot of clicks ... 3 per page actually: Printing af list of 400 items (20 a page) would be 120 clicks.
Quote Lucius:
Just out of curiosity... What is the need / business case for needing to display so huge editable grid?
|
|
|
|
kpedersen
Posts: 12
|
| Posted: 06/14/2012, 5:20 PM |
|
The query time for the first example is 155 ms.
The query times scale by N, so the N*N problem must logically be in the CodeCharge code.
Also the content is gzipped a factor 20, and compared to other transport latencys this cannot account for large total latencys.
Quote E43509:
Can you isolate the query times?
How long does the sql take to return all the rows (not just the first set) for the various scenarios? Delays like this are usually due to the query and the amount of data being returned and not necessarily the transport layer.
Good luck.
|
|
|
|
Oper
Posts: 1195
|
| Posted: 06/14/2012, 5:26 PM |
|
Use Firefox, Right Click your grid after is finsihed displaying
Choose Page Info
a Post that?
Something like this:
_________________
____________________________
http://www.7bz.com (Free CMS,CRM Developed in CCS)
http://www.PremiumWebTemplate.com
Affiliation Web Site Templates
Please do backup first |
|
|
|
kpedersen
Posts: 12
|
| Posted: 06/14/2012, 5:28 PM |
|
No, the N*N rendering time is also there on plain lists/tables - using editable grids just make the problem worse. The example values i posted above is from a regular list/table ("Grid").
I really appreciate your interest in this matter, but unfortunately i cannot disclose the online example: I will however will put some effort into constructing an example along with some sample data.
Using the older CCS version + PHP this was not an issue (lists with 1000+ items), so i expect the problem i mainly an issue in the generated Servlet code (guessing).
Quote Oper:
oh,
* you have Listbox on that Grid
* every lixbox has Populated Data
* that make your page bigger not slow.
so maybe the size of the page is similar to downloading 1 or 2 MB file each display
do you have a workable link i could check?
|
|
|
|
Oper
Posts: 1195
|
| Posted: 06/14/2012, 5:30 PM |
|
I'm positive sure is not CCS issue.
you could send the link privaly in PM if you want.
_________________
____________________________
http://www.7bz.com (Free CMS,CRM Developed in CCS)
http://www.PremiumWebTemplate.com
Affiliation Web Site Templates
Please do backup first |
|
|
|
kpedersen
Posts: 12
|
| Posted: 06/14/2012, 5:35 PM |
|
Document sizes from Chrome:
19.76KB
451.73KB
1 = Total size (unzipped)
2 = Transferred (zipped)
Time to fetch from server: 16,2 seconds
Quote Oper:
Use Firefox, Right Click your grid after is finsihed displaying
Choose Page Info
a Post that?
Something like this:
|
|
|
|
Oper
Posts: 1195
|
| Posted: 06/14/2012, 7:27 PM |
|
so you are downloading 500K every post +
What was the Render Mode Choosed by the browser?
_________________
____________________________
http://www.7bz.com (Free CMS,CRM Developed in CCS)
http://www.PremiumWebTemplate.com
Affiliation Web Site Templates
Please do backup first |
|
|
|
kpedersen
Posts: 12
|
| Posted: 06/14/2012, 8:16 PM |
|
I expect the rendering mode to be irellevant:
The numbers i have provided are the load times (Chrome > Developer tools > Network). Just to make sure I have testet the load time of a local filesystem copy of the page, and the loadtime is displayed as 2 ms - likewise rendering is extremely fast.
Note that only 15 kB (zipped) are actually transferred. The 500K is the unzipped size (after download).
Quote Oper:
so you are downloading 500K every post +
What was the Render Mode Choosed by the browser?
|
|
|
|
kpedersen
Posts: 12
|
| Posted: 06/14/2012, 8:22 PM |
|
I ran the tests on the page again
records ms kB
1 30 18
100 283 140
200 882 265
300 1884 390
400 3310 515
The results are illustrated here
http://www.p2e.dk/ccsPerformance.xlsx
|
|
|
|
kpedersen
Posts: 12
|
| Posted: 06/18/2012, 8:51 AM |
|
It seems there is a problem with the BeforeShowRow event.
The render time scaling seems to drop from N*N to N as soon as the event is removed.
I havn't had the time investigate what code is causing the problem, but i guess the codecharge team will be able to take it from here (N*N scalabilty IS a problem).
|
|
|
|
kpedersen
Posts: 12
|
| Posted: 06/18/2012, 8:57 AM |
|
Quote kpedersen:
It seems there is a problem with the BeforeShowRow event.
The render time scaling seems to drop from N*N to N as soon as the event is removed.
I havn't had the time investigate what code is causing the problem, but i guess the codecharge team will be able to take it from here (N*N scalabilty IS a problem).
Ups ... jumping to conclusions:
Either the BeforeShowRow event OR the code provided above is the source of the bad performance.
|
|
|
|
Lucius
Posts: 220
|
| Posted: 06/18/2012, 11:46 AM |
|
Quote :The business need for long lists is printing: Paged setup would not work, as this would require a lot of clicks ... 3 per page actually: Printing af list of 400 items (20 a page) would be 120 clicks.
So if you are simply in need of printing, then why use editable grid?
|
|
|
|
Lucius
Posts: 220
|
| Posted: 06/19/2012, 2:06 AM |
|
Another thought. If it's really only printing values queried from DB, without much parameters and you need it to be very fast with lot of data, I would consider doing this outside of CCS. I don't know what is the exact CCS overhead for processing DB data, but all RAD tools have it. It's a trade of speed of development and code optimization/simplicity.
Simple HTML output (tables) page with queries written in language you use, will always be faster than RAD tool generated code. All depends on how complicated is the functionality behind it...
|
|
|
|
mor
Posts: 119
|
| Posted: 06/19/2012, 5:12 AM |
|
PHP. Datasource: joined tables with few text and numeric fields.
Records count - Page size - Time
1 record - 1,1k - 96ms
10 records - 2,4k - 98ms
100 records - 14,7k - 178ms
1000 records - 138,7k - 705ms
10000 records - 1,35Mb - 16,8s
Good speed.
Do u rly need more then 10k records per page? :)
If 'yes' - please use cashing, correctly configured and more powerful servers.
If 'no' - add page navigator and configure page size for component :)
_________________
Mor ve Ötesi |
|
|
|
