How to Get the Callback Parameter of a Json File?

[Cyrix]

I want to use ZP to download financial data from morningstar.com
The URL to return the financial data is like this
http://mschart.morningstar.com/chartweb/defaultChart?type=getcc&secids=F00000H3VG;XI&dataid=8225&startdate=2018-03-28&enddate=2019-03-28&currency=&format=1&callback=jQuery17209497126185594382_1553795038840&_=1553795041139

It's the source data for a chart on the quote page
http://quotes.morningstar.com/indexquote/quote.html?t=F00000H3VG

The main question is: how can we get the callback parameter part "&callback=jQuery17209497126185594382_1553795038840&_=1553795041139" in order to create the complete URL?

It's not on the source code of the quote page.
We can find that URL from the "F12" debug tab of Chrome on the quote page when the chart is being loaded, but is there a way to get it in ZP?


Any ideas?

Thanks.

 

[Cyrix]

Another simpler way to ask this question is

I want to scrape financial data behind the main chart on http://quotes.morningstar.com/indexquote/quote.html?t=F00000H3VG.

The URL of the json data is like this http://mschart.morningstar.com/chartweb/defaultChart?type=getcc&secids=F00000H3VG;XI&dataid=8225&startdate=2018-03-28&enddate=2019-03-28&currency=&format=1&callback=jQuery17209497126185594382_1553795038840&_=1553795041139

I found it manually on the "F12" debug tab of Chrome when the chart is being loaded.
Is there a way in ZP to obtain this URL automatically?

Thanks.

 

[lokiys]

Hi. Those JSON URLs You gave do not work for me. Simple nothing loads.
BUT there is feature Traffic monitor in zenno, so You can see all Requests being sent from browser to server and back.
This is where You can get this URL, but I'm very sure that You can scrape out all URL parts from the source text.
Keep digging.
https://help.zennolab.com/en/v5/zennoposter/5.18.0.0/webframe.html#topic531.html
https://zennolab.com/wiki/en:traffic

Cheers

 

[Cyrix]

The reason why the json URL returns a blank page on your computer is because you are trying to directly open it.
You first need to first go to the chart page, obtain the json URL, then open it.
I don't know if it's because of cookie or the callback parameter, but if we don't first go to the quote page, the json URL will be blank.
I think it's morningstar's trying to prevent people directly getting the data without visiting the page.

I tried the first example codes in your link but got this

Why?
Thank you.

 

[Cyrix]

The code I tried was

instance.SetContentPolicy("BlockList", new []{ "mc.yandex.ru" }, null);
instance.UseTrafficMonitoring = false;
Tab tab = instance.ActiveTab;
// navigate to url
instance.ClearCache();
instance.ClearCookie();
tab.Navigate("http://lessons.zennolab.com");
if (tab.IsBusy) tab.WaitDownloading();
// get list of requests
var traffic = instance.ActiveTab.GetTraffic();
// print count of items, it will be more than 0
project.SendInfoToLog("First count of traffic elements = " + traffic.Count());
// get some data from items and log it
foreach(var t in traffic)
    project.SendInfoToLog(string.Format("Url: {0}\r\n Method: {1}\r\n Result: {2}", t.Url, t.Method, t.ResultCode));
// get list of request second time
traffic = instance.ActiveTab.GetTraffic();
// print count of items, it will be 0, because the previous method call removed the old items
project.SendInfoToLog("Second count of traffic elements = " + traffic.Count());

and got the above errors.

 

[Cyrix]

I updated to latest version of ZP and the example code runs without errors.
Let me see if I can get the json URL.
Thanks.

 

[Cyrix]

I was able to get the json URL of the chart on http://quotes.morningstar.com/indexquote/quote.html?t=F00000H3VG using the example code now, but the problem is that I need to click the "Maximum" button on the chart to get the full history of the data.
How to insert the "clicking" action into the code?

The code I am using is below:

Tab tab = instance.ActiveTab;
instance.ClearCache();
instance.ClearCookie();
tab.Navigate("http://quotes.morningstar.com/indexquote/quote.html?t=F00000OMB5");
if (tab.IsBusy) tab.WaitDownloading();
var traffic = instance.ActiveTab.GetTraffic();
foreach(var t in traffic)
if (t.Url.Contains("defaultChart") && t.Url.Contains("8225"))
{
return t.Url;
}


How do I do a clicking of the "Maximum" button between tab.Navigate and GetTraffic?
Can I insert an action block within the c# code?

Thanks.

 

[lokiys]

You can do all your actions on the page, click buttons, etc. and only then get traffic.

BUT I would suggest You to try to find all parameters one by one, then build full URL with all parameters in it and then use GET request to download this JSON string.
BUT as You said this URL works only if You go to the main page then most probably You need to do GET request to the main page to get cookies and also get some parameters to build the second URL. Then do GEt request to second URL (data URL) and You will have all You need. In right way

UPD: Link to assembly example is not correct code You have to use.
I have just shown an example METHOD you have to use to get what You want.
This code has to be edited.

 

[Cyrix]

You can do all your actions on the page, click buttons, etc. and only then get traffic.

BUT I would suggest You to try to find all parameters one by one, then build full URL with all parameters in it and then use GET request to download this JSON string.
BUT as You said this URL works only if You go to the main page then most probably You need to do GET request to the main page to get cookies and also get some parameters to build the second URL. Then do GEt request to second URL (data URL) and You will have all You need. In right way

UPD: Link to assembly example is not correct code You have to use.
I have just shown an example METHOD you have to use to get what You want.
This code has to be edited.

Thank you very much.

 

[lokiys]

Thank you very much.

This button is the best for telling Thank You

 

[Cyrix]

This button is the best for telling Thank You

Got it, thanks.

 

[Cyrix]

A side question:

I need ZP to click the "Maximum" button after the chart is loaded, but ZP always tries to click it before the page finishes loading and misses.
Now I have to add a pause of 10 secs before clicking the "Maximum" to increase the chance that "Maximum" is already shown for clicking, but this way is unreliable.

Is there a way to make ZP really wait automatically for the page actually loaded?
Thanks.